diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C2-verdicts.md b/documentation/audits/dataloss-hunt-2026-10-20/C2-verdicts.md index 709a7509..6da8133f 100644 --- a/documentation/audits/dataloss-hunt-2026-10-20/C2-verdicts.md +++ b/documentation/audits/dataloss-hunt-2026-10-20/C2-verdicts.md @@ -16,7 +16,7 @@ Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searche | **D1** shares second-drive copy pruned by another drive's group | **CONFIRMED** (unit test, production code, fake mirror seam — `C3-D1-shares-tier2/`) | defect | **Worse than reported:** with shares on two drives that pick the same target, EVERY night the second group's reconcile deletes the first group's copy (`removed stale dest dir hdd_2/b_share`) — one drive's shares never have a second copy while the run says ok. And the night a drive goes away, its copy is deleted (`removed stale dest dir hdd_3/csaladi_fotok`). | | D1b an AWAY drive is still chosen as the second-drive target | CONFIRMED-CODE (+ the same unit test: `hdd_3/iskola => hdd_1/...` after hdd_1 was marked disconnected) | defect | `tier2.go:151-180` loops `GetSchedulableStoragePaths()` (`settings.go:1593-1603` excludes only Decommissioned); no Disconnected check. On a real box the away drive's empty mount point is on the guest's root disk | | **K1** reinstall after „delete the old host" loses the route to the old escrow | **CONFIRMED** at the hub store (unit test against production code, `C3-K1-escrow-after-host-delete/`); the controller half (new key minted, history set aside on night one) CONFIRMED-CODE | defect (03 §8a / 07 §7.0 promise the route; D19+R-1154 removed it unrecorded) | hub `store.go:3164-3229` demotes into `host_escrow_superseded` under the OLD host_id and deletes the `hosts` row; `GetEscrowStatusForCustomer` `store.go:3499-3527` joins on `hosts` → nil for the new box; `customer_reset.go:48,155` and `store.go:4189` DO include `host_deletions`. Live proof needs Tester4 off-site ON (B7) | -| K2 abandoning one set-aside copy purges every retained package | PLAUSIBLE | defect (breaks `store.go:4176-4179`'s own invariant) | `store.go:4185-4190` customer-wide delete read true | +| K2 abandoning one set-aside copy purges every retained package | CONFIRMED-CODE (`store.go:4185-4190`) → R-1182 | defect (breaks `store.go:4176-4179`'s own invariant) | `store.go:4185-4190` customer-wide delete read true | | **R1/D4** a failed LOCAL unit restore: no copy of the live state, app started anyway | **CONFIRMED LIVE** on 9202 (`C3-R1-local-restore-9202/`): live row destroyed, DB crash-loops, no undo copy, raw English Docker text to the household | **design question** for the operator: decision 154 (R-638 A) chose „start anyway"; D20 (R-893 A) gave „put back exactly" to the OFF-SITE path only | `restore_unit.go:428-489` | | R2 a crash mid-restore starts the app on a mixed state; the undo copy is never read at start | PLAUSIBLE | defect | `appstop_marker.go:307`, `offbox_restore_undo.go:137` | | R3 a cut-off shares restore can be placed and reported restored | CONFIRMED-CODE | defect (R-358 class, fixed for apps, not shares) | `shares_restore.go:131-140` readiness = non-empty dir | @@ -29,7 +29,7 @@ Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searche | R12 interrupted whole restore leaves `*.felhom-restoring`, every retry fails | CONFIRMED-CODE | defect | `tier2_whole.go:86-87` O_EXCL; removed only on an in-process error | | **D2** a missing / emptied app folder deletes its second-drive copy within one night | **CONFIRMED** for a MISSING folder (unit test, production code + real copy seam: `C3-D2-missing-leg/` — mirror of `photos` deleted, run „ok", the warning says only „nem kerültek a másodlagos mentésbe"); the EMPTIED case CONFIRMED-CODE (`rsyncMirror` = `rsync -a --delete`, `tier2.go:744-751`; the test seam does not delete) | defect-shaped corner of a choice: 07 §8.2 „data legs are NOT guarded" is about a leg LEAVING the class set; this is a declared leg that is MISSING, which the code itself logs as a gap (`tier2_capture.go:55-61`) and then reconciles away (`tier2.go:454`) | P1 for optional legs (no other tier holds them) | | D3 / T11 a host clock ahead silences the whole-guest tier; the hub reads the same clock | PLAUSIBLE (B6 #8 on the rehearsal VM is its live test) | defect | agent `localapi/server.go:1178-1190` | -| D5 a dump that fails its own check has already replaced the good one | CONFIRMED-CODE | defect for the ORDER | `appbackup/dbdump.go:388-422`: size check, rename (`:399`), then `ValidateDump` | +| D5 a dump that fails its own check has already replaced the good one | **CONFIRMED** (unit test with a docker stub: `C3-D5-dump-replaced/`) → R-1185 | defect for the ORDER | `appbackup/dbdump.go:388-422`: size check, rename (`:399`), then `ValidateDump` | | D6 kept-data page can list a running app's folder | PLAUSIBLE | defect | needs an unreadable app.yaml | | D7 unit restore can point the app at an old drive | PLAUSIBLE (low) | defect | | | D8 local whole-guest keep=1 | design (07 §6.1 R-684 chose it) | design | **NOT-A-FINDING: the operator chose keep=1 for demo-hp; recorded in 07** | @@ -37,7 +37,7 @@ Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searche | T2 controller image without tzdata → UTC schedules | **NOT-A-FINDING:** measured on 9202 (controller 0.331.0, Debian 13 base): `/usr/share/zoneinfo/Europe/Budapest` present, scheduler logs „scheduled for … CEST" | — | | | T3 whole-guest start creeps ~5 min/night | PLAUSIBLE | defect | needs the agent journal over weeks | | T4 whole-guest quiesce stops apps mid-restore | PLAUSIBLE | defect | `quiesce.go:213-275` | -| T5 nightly off-site copy skipped SILENTLY when the flag is held; catch-up never retries | CONFIRMED-CODE | defect (the code chose nil for the nightly path; the catch-up gap is not chosen) | `offbox.go:981-996`, `main.go:1246` | +| T5 nightly off-site copy skipped SILENTLY when the flag is held; catch-up never retries | CONFIRMED-CODE → R-1186 | defect (the code chose nil for the nightly path; the catch-up gap is not chosen) | `offbox.go:981-996`, `main.go:1246` | | T6 proof / integrity never run on a box off at 05:30–06:00 | PLAUSIBLE | defect (comment `main.go:1575-1577` says the opposite) | | | T7 hub false „no DB dump" for late or evening W | PLAUSIBLE | defect | hub `monitor/deadline.go:416-419, 556-565` | | T8 the OS leg follows a daytime valve backup | design question | design | 07 §6.1 / 11 §8.1 set no time bound | diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/go-test-output.txt b/documentation/audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/go-test-output.txt new file mode 100644 index 00000000..6ca273a8 --- /dev/null +++ b/documentation/audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/go-test-output.txt @@ -0,0 +1,9 @@ +# go test at controller 7ca59e2 (v0.336.0), scratch clone; a docker STUB on PATH (no real docker reached); 2026-10-20T11:07:38Z +=== RUN TestRehearsalD5_InvalidDumpReplacesTheGoodOne + zz_rehearsal_d5_test.go:26: POSITIVE CONTROL: last night's dump, 152 B, ValidateDump={Valid:true TableCount:1 Error: FileSize:152 ModTime:2026-10-20 13:07:39.022566077 +0200 CEST UserTableFound:false UserRows:0 LooksEmpty:false} +2026/10/20 13:07:39 [WARN] [backup] ValidateDump FAIL: /tmp/TestRehearsalD5_InvalidDumpReplacesTheGoodOne2769942191/002/unit/app-postgres.sql — dump file too small (< 100 bytes) + zz_rehearsal_d5_test.go:31: tonight's dump: err= validation={Valid:false TableCount:0 Error:dump file too small (< 100 bytes) FileSize:99 ModTime:2026-10-20 13:07:39.026566071 +0200 CEST UserTableFound:false UserRows:0 LooksEmpty:false} + zz_rehearsal_d5_test.go:32: FAULT: the file on disk now = 99 B, still holds CREATE TABLE photos: false +--- PASS: TestRehearsalD5_InvalidDumpReplacesTheGoodOne (0.00s) +PASS +ok gitea.dooplex.hu/admin/felhom-controller/internal/appbackup 0.008s diff --git a/documentation/audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/zz_rehearsal_d5_test.go b/documentation/audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/zz_rehearsal_d5_test.go new file mode 100644 index 00000000..d117fa09 --- /dev/null +++ b/documentation/audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/zz_rehearsal_d5_test.go @@ -0,0 +1,35 @@ +package appbackup + +// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D5 / R-1185). Never committed. A docker STUB on PATH +// (no real docker is reached) prints a header-only dump, as a database that was re-initialised empty would. +import ( + "context" + "io" + "log" + "os" + "path/filepath" + "testing" +) + +func TestRehearsalD5_InvalidDumpReplacesTheGoodOne(t *testing.T) { + bin := t.TempDir() + stub := "#!/bin/sh\ncase \"$1\" in\n inspect) echo true ;;\n exec) printf -- '-- PostgreSQL database dump\\nSET statement_timeout = 0;\\n\\n--\\n-- PostgreSQL database dump complete\\n--\\n' ;;\nesac\n" + if err := os.WriteFile(filepath.Join(bin, "docker"), []byte(stub), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin) + dumpDir := filepath.Join(t.TempDir(), "unit") + final := filepath.Join(dumpDir, "app-postgres.sql") + good := "-- PostgreSQL database dump\nCREATE TABLE public.photos (id int);\nCOPY public.photos (id) FROM stdin;\n1\n2\n\\.\n\n--\n-- PostgreSQL database dump complete\n--\n" + os.MkdirAll(dumpDir, 0o755) + os.WriteFile(final, []byte(good), 0o644) + t.Logf("POSITIVE CONTROL: last night's dump, %d B, ValidateDump=%+v", len(good), ValidateDump(final, DBTypePostgres)) + + db := DiscoveredDB{ContainerName: "app-db", ContainerID: "0123456789abcdef", DBType: DBTypePostgres, DBUser: "u", DBName: "d", StackName: "app"} + res := DumpOneTo(context.Background(), db, final, log.New(io.Discard, "", 0), false) + b, _ := os.ReadFile(final) + t.Logf("tonight's dump: err=%v validation=%+v", res.Error, res.Validation) + t.Logf("FAULT: the file on disk now = %d B, still holds CREATE TABLE photos: %v", len(b), contains(string(b), "CREATE TABLE public.photos")) +} + +func contains(s, sub string) bool { return len(s) >= len(sub) && (func() bool { for i := 0; i+len(sub) <= len(s); i++ { if s[i:i+len(sub)] == sub { return true } }; return false })() } diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/dst-replay/README.md b/documentation/audits/rehearsal-2026-10-20/B6-failures/dst-replay/README.md new file mode 100644 index 00000000..df7d38b0 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/dst-replay/README.md @@ -0,0 +1,16 @@ +# DST — what each scheduler does on Sunday 2026-10-25 (03:00 CEST → 02:00 CET), read from code and replayed + +No architecture document mentions DST (`07` has no hit for DST, daylight or timezone — checked again 2026-10-20 with +ASCII fragments `DST`, `daylight`, `time.?zone`, `ora.?allit`; positive control: `Europe/Budapest` is found in 07). + +| scheduler | 2026-10-25 | 2027-03-28 (spring) | how known | +|---|---|---|---| +| controller `sched.Daily` (db-dump W, second copy W+60, off-site W+105, fill-watch, metrics, asset-sync, abandon-sweep, proof, integrity) | **once each**; a 02:xx time runs at the SECOND (CET) occurrence | a 02:xx time runs at 03:xx CEST — **the default 02:30 dump lands on 03:30, the same instant as the 03:30 second-drive copy** | PRODUCTION `nextDailyRunAt` replayed (`replay.txt`) | +| hub `scheduleDaily` (db-snapshot 02:00, prune 04:30, deadline-check 05:00, key-audit 07:10) | **TWICE** — e.g. the alarm check at 04:00 CET and 05:00 CET | once each, one hour late | the hub's arithmetic replayed (`../../../dataloss-hunt-2026-10-20/C3-T9-hub-dst/replay.txt`) → **R-1183** | +| agent (whole-guest due, restore-test, OS leg, cap) | unaffected (UTC durations) | unaffected | code read (C1 reviewer 4) | +| controller quiesce gate [W+2h, W+6h) by wall minute | 04:30–08:30 CET for W = 02:30 (4 h real) | 04:30–08:30 CEST | code read | +| host timers on a household box | monotonic only | monotonic only | code read | +| DooPlex timers | Oct 25 is a Sunday: both weekly restore tests run on the DST day; the 02:30 hub-db-backup timer is inside the repeated hour — not verified | — | code read | + +The live check of the box itself (B6 #7: the rehearsal VM's clock set to 2026-10-25 01:30 CEST and run to 04:00 CET) +is recorded separately when it runs. diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/dst-replay/replay.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/dst-replay/replay.txt new file mode 100644 index 00000000..93069b9b --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/dst-replay/replay.txt @@ -0,0 +1,25 @@ +# controller 7ca59e2 (v0.336.0) scheduler.nextDailyRunAt replayed; go version go1.27.2 linux/amd64; 2026-10-20T11:06:36Z +=== RUN TestRehearsalDST_ControllerDailyTimes + zz_rehearsal_dst_test.go:24: 02:00: 10-25 02:00 CET(01:00Z) 10-26 02:00 CET(01:00Z) 10-27 02:00 CET(01:00Z) + zz_rehearsal_dst_test.go:24: 02:15: 10-25 02:15 CET(01:15Z) 10-26 02:15 CET(01:15Z) 10-27 02:15 CET(01:15Z) + zz_rehearsal_dst_test.go:24: 02:30: 10-25 02:30 CET(01:30Z) 10-26 02:30 CET(01:30Z) 10-27 02:30 CET(01:30Z) + zz_rehearsal_dst_test.go:24: 02:59: 10-25 02:59 CET(01:59Z) 10-26 02:59 CET(01:59Z) 10-27 02:59 CET(01:59Z) + zz_rehearsal_dst_test.go:24: 03:00: 10-25 03:00 CET(02:00Z) 10-26 03:00 CET(02:00Z) 10-27 03:00 CET(02:00Z) + zz_rehearsal_dst_test.go:24: 03:30: 10-25 03:30 CET(02:30Z) 10-26 03:30 CET(02:30Z) 10-27 03:30 CET(02:30Z) + zz_rehearsal_dst_test.go:24: 04:15: 10-25 04:15 CET(03:15Z) 10-26 04:15 CET(03:15Z) 10-27 04:15 CET(03:15Z) + zz_rehearsal_dst_test.go:24: 05:30: 10-25 05:30 CET(04:30Z) 10-26 05:30 CET(04:30Z) 10-27 05:30 CET(04:30Z) + zz_rehearsal_dst_test.go:24: 06:00: 10-25 06:00 CET(05:00Z) 10-26 06:00 CET(05:00Z) 10-27 06:00 CET(05:00Z) + zz_rehearsal_dst_test.go:24: 22:00: 10-24 22:00 CEST(20:00Z) 10-25 22:00 CET(21:00Z) 10-26 22:00 CET(21:00Z) + zz_rehearsal_dst_test.go:24: 02:00: 03-28 03:00 CEST(01:00Z) 03-29 02:00 CEST(00:00Z) 03-30 02:00 CEST(00:00Z) + zz_rehearsal_dst_test.go:24: 02:15: 03-28 03:15 CEST(01:15Z) 03-29 02:15 CEST(00:15Z) 03-30 02:15 CEST(00:15Z) + zz_rehearsal_dst_test.go:24: 02:30: 03-28 03:30 CEST(01:30Z) 03-29 02:30 CEST(00:30Z) 03-30 02:30 CEST(00:30Z) + zz_rehearsal_dst_test.go:24: 02:59: 03-28 03:59 CEST(01:59Z) 03-29 02:59 CEST(00:59Z) 03-30 02:59 CEST(00:59Z) + zz_rehearsal_dst_test.go:24: 03:00: 03-28 03:00 CEST(01:00Z) 03-29 03:00 CEST(01:00Z) 03-30 03:00 CEST(01:00Z) + zz_rehearsal_dst_test.go:24: 03:30: 03-28 03:30 CEST(01:30Z) 03-29 03:30 CEST(01:30Z) 03-30 03:30 CEST(01:30Z) + zz_rehearsal_dst_test.go:24: 04:15: 03-28 04:15 CEST(02:15Z) 03-29 04:15 CEST(02:15Z) 03-30 04:15 CEST(02:15Z) + zz_rehearsal_dst_test.go:24: 05:30: 03-28 05:30 CEST(03:30Z) 03-29 05:30 CEST(03:30Z) 03-30 05:30 CEST(03:30Z) + zz_rehearsal_dst_test.go:24: 06:00: 03-28 06:00 CEST(04:00Z) 03-29 06:00 CEST(04:00Z) 03-30 06:00 CEST(04:00Z) + zz_rehearsal_dst_test.go:24: 22:00: 03-27 22:00 CET(21:00Z) 03-28 22:00 CEST(20:00Z) 03-29 22:00 CEST(20:00Z) +--- PASS: TestRehearsalDST_ControllerDailyTimes (0.00s) +PASS +ok gitea.dooplex.hu/admin/felhom-controller/internal/scheduler 0.004s diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/dst-replay/zz_rehearsal_dst_test.go b/documentation/audits/rehearsal-2026-10-20/B6-failures/dst-replay/zz_rehearsal_dst_test.go new file mode 100644 index 00000000..950bdf8e --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/dst-replay/zz_rehearsal_dst_test.go @@ -0,0 +1,27 @@ +package scheduler + +// THROWAWAY replay for the 2026-10-20 rehearsal (B6 #7, DST) — never committed. Replays the PRODUCTION +// nextDailyRunAt over the two DST changes for every controller daily time; the loop re-reads the clock at each +// fire instant exactly as the scheduler does. +import ( + "testing" + "time" +) + +func TestRehearsalDST_ControllerDailyTimes(t *testing.T) { + loc := getBudapestLocation() + for _, win := range []struct{ from, to time.Time }{ + {time.Date(2026, 10, 24, 12, 0, 0, 0, loc), time.Date(2026, 10, 26, 12, 0, 0, 0, loc)}, + {time.Date(2027, 3, 27, 12, 0, 0, 0, loc), time.Date(2027, 3, 29, 12, 0, 0, 0, loc)}, + } { + for _, hm := range []string{"02:00", "02:15", "02:30", "02:59", "03:00", "03:30", "04:15", "05:30", "06:00", "22:00"} { + now, line := win.from, "" + for now.Before(win.to) { + next := nextDailyRunAt(hm, now) + line += " " + next.In(loc).Format("01-02 15:04 MST") + "(" + next.UTC().Format("15:04Z") + ")" + now = next + } + t.Logf("%s:%s", hm, line) + } + } +} diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index c2c0cd7c..1b2b3d05 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -165,7 +165,7 @@ stopping line that lies. | **R-1179** | Backup & restore | P2 | **A drive marked „away" can still be chosen as the second-drive backup target.** `tier2.go:151-180` loops `GetSchedulableStoragePaths()`, which drops only decommissioned drives (`settings.go:1593-1603`); nothing checks `Disconnected`. Seen in the R-1175 unit test (a share copied to a drive marked disconnected). On a real box the away drive's mount point is an empty folder on the guest's root disk (or the host's root via the bind), so the night's copy fills the wrong disk and is not where the household thinks. CONFIRMED in code; not run live. Data-loss hunt 2026-10-20. | OPEN | — | Fix: skip a disconnected target like a decommissioned one; live check on a scratch box with a removed drive | CC | | **R-1180** | Backup & restore | P2 | **The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete files not yet copied back.** `RunAllTier2` (`tier2.go:479-520`) takes no backup flag and does not ask about a running restore; the whole restore holds only the backup flag (`tier2_whole.go:301`), the night leg only `LegLock` (`cmd/controller/main.go:1348-1352`). A long restore of a photo library at 03:30, or 15 min after a missed night, overlaps. PROVEN against production code with the real `rsync -a --delete` mirror (`audits/dataloss-hunt-2026-10-20/C3-T1-mirror-during-restore/`): with the backup flag held by a restore, `RunAllTier2` still mirrors, and the mirror goes from [a,b,c,d] to [a,b] — the two files the restore was bringing back are gone from the only copy. Data-loss hunt 2026-10-20 (C1 reviewer 4, T1). | OPEN | — | Fix: the mirror leg skips (and says so) while any restore runs; test with the existing tier-2 seams | CC | | **R-1181** | Backup & restore | P2 | **A missing or emptied app folder deletes its second-drive copy within one night; for OPTIONAL folders (audiobooks, comics, ROMs, Immich's external library) that copy was the only one.** A missing declared leg is skipped (`tier2_capture.go:55-61`, silent for optional legs) and then reconciled away (`tier2.go:454`); an emptied one is mirrored with `rsync -a --delete` (`:744-751`). PROVEN for a missing folder against production code (`audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/`): the mirror's `photos` deleted, run „ok", warning only „nem kerültek a másodlagos mentésbe". 07 §8.2 chose „data legs are not guarded" for a leg that LEAVES the class set; a declared leg that is MISSING is not that case. Optional legs are in no off-site tier (07 §6.2). | WAITING-ON-OPERATOR | decision: keep a missing leg's mirror (and say so) until the household releases it? And: does an accidental folder deletion need a second-drive history for optional legs? | Operator: answer in STATUS | operator | -| **R-1182** | Backup & restore | P2 | **Abandoning ONE set-aside off-site copy deletes EVERY retained key package of the customer, including ones that open other set-aside copies still stored.** hub `api/handler.go:626-627` → `PurgeSupersededEscrowForCustomer` (`store.go:4185-4190`) deletes customer-wide; the comment right above (`:4176-4179`) states the invariant this breaks. PLAUSIBLE (code read by reviewer and main session; not run). Data-loss hunt 2026-10-20 (C1 reviewer 3, K2). | VERIFY | — | Prove with a hub store test (two superseded rows, one abandon); then purge only the package that covers the abandoned path | CC | +| **R-1182** | Backup & restore | P2 | **Abandoning ONE set-aside off-site copy deletes EVERY retained key package of the customer, including ones that open other set-aside copies still stored.** hub `api/handler.go:626-627` → `PurgeSupersededEscrowForCustomer` (`store.go:4185-4190`) deletes customer-wide; the comment right above (`:4176-4179`) states the invariant this breaks. CONFIRMED in code (the SQL deletes every retained row of the customer, live and deleted hosts alike; nothing ties a row to the abandoned path). The R-1176 reinstall makes a second set-aside copy likely. Data-loss hunt 2026-10-20 (C1 reviewer 3, K2). | OPEN | — | Fix: purge only the package whose key opens the abandoned path (record the path or repo id with each retained row); hub store test as red-proof | CC | | **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** **READY 2026-10-20 — measured and planned, nothing changed; the decision is the operator's (STATUS D21).** Re-measured on 9202 (Immich v3.2.4): off-site ~1.33 GB = model cache 824 MB (62 %), raw database folder 283 MB (21 %), Immich's own backups 115 MB (9 %), logical dump 54 MB, originals 49 MB, thumbnails 2 MB — 102 MB must survive. A: leave out all three re-fetchable parts (−92 %; needs a never-run restore from the dump alone on the vectorchord image). B: only the model cache (−62 %; needs one restore with an empty cache). Pick: B now, A's other parts after their proof (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`). | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC | | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC | | **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC | @@ -190,7 +190,7 @@ stopping line that lies. | **R-1172** | Backup & restore | P4 | **After option A (R-893): a failed off-site restore that was PUT BACK is logged at ERROR, and an empty `backups/restore-undo/` folder stays on disk.** Measured on 9202 2026-10-20 (rc.1): the web handler logs `[ERROR] [web] off-box reconstitute rallly (async): … Nem változott semmi…` — good news at ERROR; the put-back removes the per-app folder but not the empty `restore-undo/` parent. Small; 0.336.0 was already pushed (a version is pushed once). | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P2/live/11-controller-log-window.txt`, `12-after-failed-restore.txt` | Next controller release: the put-back outcome logged at WARN (its own sentence is the household's), the empty parent removed; a test | CC | | **R-1173** | Backup & restore | P3 | **R-921's consequence (no app stop for an off-site tier the agent will refuse) has not been seen live.** Agent 0.161.0 serves `host_busy` (proven live on demo-hp 2026-10-20 during a press: busy `backup:local`), and controller ≥ 0.335.0 reads it — but the case only arises on the night schedule (a manual press backs up the local tier only), so the no-stop branch is proven by its red-proved test alone. | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P1/1b-host-busy-poll.txt` | Read demo-hp's controller log after the night its off-site (pbs) tier is due (~2026-10-22): a `host is busy … no app is stopped (R-921)` line, or the off-site tier starting with nothing in flight | CC | | **R-1184** | Backup & restore | P3 | **A cut-off shared-folder restore can be „placed" and reported restored while files are missing; the download is then deleted.** Readiness is only „the scratch folder is not empty" (`internal/backup/shares_restore.go:131-140`) — the R-358 class fixed for apps, not for shares; 30-min download window (`web/offbox_handlers.go:716`); fixed 2 GiB free-space gate that fails open (`:107`). CONFIRMED in code. Data-loss hunt 2026-10-20 (C1 reviewer 1, R3). | OPEN | — | Fix: a completion marker after restic succeeds, as for apps | CC | -| **R-1185** | Backup & restore | P3 | **A database dump that fails its own content check has already replaced last night's good dump, and the second drive copies it an hour later.** `appbackup/dbdump.go:388-422`: size check, rename over the good file (`:399`), then `ValidateDump` (`:419`). Volume tars are promoted with no content check either (`backup.go:902`). Tier 1 is one point per app by design (07 §6.1); the ORDER is the defect. CONFIRMED in code. Data-loss hunt 2026-10-20 (D5). | OPEN | — | Fix: validate the temp file before the rename; keep the last good dump on a failed check | CC | +| **R-1185** | Backup & restore | P3 | **A database dump that fails its own content check has already replaced last night's good dump, and the second drive copies it an hour later.** `appbackup/dbdump.go:388-422`: size check, rename over the good file (`:399`), then `ValidateDump` (`:419`). Volume tars are promoted with no content check either (`backup.go:902`). Tier 1 is one point per app by design (07 §6.1); the ORDER is the defect. PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/`, a docker stub prints a header-only dump): `ValidateDump` says invalid („dump file too small"), `DumpOneTo` returns no error, and last night's good dump (with its `CREATE TABLE`) is gone from disk. Data-loss hunt 2026-10-20 (D5). | OPEN | — | Fix: validate the temp file before the rename; keep the last good dump on a failed check | CC | | **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC | | **R-1187** | Backup & restore | P3 | **`.fab` export/import candidates from the data-loss hunt, not yet proven:** (a) an export silently drops a drive folder that holds a symlink (`appexport/fabplan.go:292`, `export.go:714`); (b) the import never sets file owners and drops symlinks (`restore.go:1157-1176`); (c) import into a STOPPED app merges into the existing volumes, and the default export (stop ticked) carries no SQL dump (`restore.go:361-374`, `export.go:262,366`, `appbackup/dbdump.go:142`); (d) no concurrency guard, no undo, no record (`restore.go:181-186`, `web/handler_export.go:312-353`); (e) PROVEN and split out as R-1191 (the import drops the family gate); (f) it trusts the bundle's HDD_PATH (`restore.go:589-631`); (g) the DB import runs psql without ON_ERROR_STOP and misses postgis (`restore.go:1052-1053, 928-937`). Read by reviewer 1; main session spot-checked (a); (e) proven live → R-1191. Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Prove (c) first on 9202 with a throwaway app; split into rows only what reproduces | CC | | **R-1188** | Backup & restore | P3 | **Restore-path candidates from the data-loss hunt, not yet proven (one confirmed in code):** (a) a crash or power cut mid-restore restarts the app on a mixed state and never reads the undo copy (`appstop_marker.go:307`, `offbox_restore_undo.go:137`, `bootrecon/bootrecon.go:210-213`); (b) a failed `volume rm` is ignored before the untar (`restore.go:218`); (c) CONFIRMED in code: an interrupted second-drive whole restore leaves `.felhom-restoring`, and every retry fails „file exists" (`tier2_whole.go:86-87`, O_EXCL); (d) the full off-site download has a 30-min cap per press and the household is not told to retry (`web/offbox_handlers.go:455`); (e) place and full restore re-read the LATEST snapshot instead of the downloaded one (`offbox_restore.go:675-679`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Prove (a) with the B6 power-cut-mid-restore scenario; (c) with a kill test on 9202 | CC |