Data-loss hunt: R-1180 proven against production code (the nightly rsync --delete mirror runs while a restore holds the backup flag and deletes the files being restored)

2026-10-20 13:02:50 +02:00
parent a57a8363df
commit 20a77d367b
4 changed files with 49 additions and 2 deletions
@@ -33,7 +33,7 @@ Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searche
 | D6 kept-data page can list a running app's folder | PLAUSIBLE | defect | needs an unreadable app.yaml |
 | D7 unit restore can point the app at an old drive | PLAUSIBLE (low) | defect | |
 | D8 local whole-guest keep=1 | design (07 §6.1 R-684 chose it) | design | **NOT-A-FINDING: the operator chose keep=1 for demo-hp; recorded in 07** |
-| T1 the nightly mirror (`rsync --delete`) can run during a restore from that mirror | CONFIRMED-CODE | defect | `RunAllTier2` (`tier2.go:479-520`) takes no backup flag; the whole restore holds only `acquireRunning` (`tier2_whole.go:301`) |
+| T1 the nightly mirror (`rsync --delete`) can run during a restore from that mirror | **CONFIRMED** (unit test, production rsync mirror: `C3-T1-mirror-during-restore/`) → R-1180 | defect | `RunAllTier2` (`tier2.go:479-520`) takes no backup flag; the whole restore holds only `acquireRunning` (`tier2_whole.go:301`) |
 | T2 controller image without tzdata → UTC schedules | **NOT-A-FINDING:** measured on 9202 (controller 0.331.0, Debian 13 base): `/usr/share/zoneinfo/Europe/Budapest` present, scheduler logs „scheduled for … CEST" | — | |
 | T3 whole-guest start creeps ~5 min/night | PLAUSIBLE | defect | needs the agent journal over weeks |
 | T4 whole-guest quiesce stops apps mid-restore | PLAUSIBLE | defect | `quiesce.go:213-275` |
@@ -0,0 +1,8 @@
+# go test at controller 7ca59e2 (v0.336.0), scratch clone; the PRODUCTION rsyncMirror (rsync -a --delete) on local temp dirs; 2026-10-20T11:02:49Z
+=== RUN   TestRehearsalT1_NightlyMirrorRunsDuringARestore
+    zz_rehearsal_t1_test.go:26: POSITIVE CONTROL: mirror after the night = [a.jpg b.jpg c.jpg d.jpg]
+    zz_rehearsal_t1_test.go:34: restore in progress: backup flag held = true
+    zz_rehearsal_t1_test.go:37: FAULT: mirror DURING the restore = [a.jpg b.jpg]  (c.jpg, d.jpg — the files the restore is bringing back — are gone from the only copy)
+--- PASS: TestRehearsalT1_NightlyMirrorRunsDuringARestore (0.22s)
+PASS
+ok  	gitea.dooplex.hu/admin/felhom-controller/internal/backup	0.230s
@@ -0,0 +1,39 @@
+package backup
+
+// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate T1 / R-1180). Never committed.
+import (
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+type t1Provider struct{ *t2v2Provider }
+
+func (p t1Provider) ListDeployedStacks() []StackSummary { return []StackSummary{{Name: "app"}} }
+
+func TestRehearsalT1_NightlyMirrorRunsDuringARestore(t *testing.T) {
+	m, src, target, prov := newTier2V2(t, "app")
+	prov.has["app"] = true
+	prov.binds["app"] = []ClassifiedBind{mHDD("appdata/app/photos")}
+	m.stackProvider = t1Provider{prov}
+	m.tier2Mirror = rsyncMirror // the PRODUCTION mirror (rsync -a --delete), local temp dirs only
+	for _, n := range []string{"a.jpg", "b.jpg", "c.jpg", "d.jpg"} {
+		mustWrite(t, filepath.Join(src, "appdata", "app", "photos", n), n)
+	}
+	m.RunAllTier2()
+	mir := filepath.Join(target, "backups", "secondary", "app", "hdd", "appdata", "app", "photos")
+	ls := func(d string) []string { e, _ := os.ReadDir(d); var o []string; for _, x := range e { o = append(o, x.Name()) }; return o }
+	t.Logf("POSITIVE CONTROL: mirror after the night = %v", ls(mir))
+
+	// The household deleted half the photos and starts the second drive's restore; the restore holds the backup flag.
+	os.Remove(filepath.Join(src, "appdata", "app", "photos", "c.jpg"))
+	os.Remove(filepath.Join(src, "appdata", "app", "photos", "d.jpg"))
+	if err := m.acquireRunning(); err != nil {
+		t.Fatal(err)
+	}
+	t.Logf("restore in progress: backup flag held = %v", m.acquireRunning() != nil)
+	// 03:30 — the nightly second-drive leg fires while the restore is still copying.
+	m.RunAllTier2()
+	t.Logf("FAULT: mirror DURING the restore = %v  (c.jpg, d.jpg — the files the restore is bringing back — are gone from the only copy)", ls(mir))
+	m.releaseRunning()
+}
@@ -163,7 +163,7 @@ stopping line that lies.
 | **R-1177** | Backup & restore | P2 | **A local restore that fails destroys the household's live data and keeps no copy of it: the app is left broken.** PROVEN LIVE on scratch 9202 (`audits/dataloss-hunt-2026-10-20/C3-R1-local-restore-9202/`): Rallly + Postgres; control restore OK; then a live row C, the unit's DB volume tar damaged (half), the household's restore → the live volume removed, `tar: short read`, the DB replay correctly skipped, the app started → Postgres crash-loops (`initdb: directory … exists but is not empty`); row C gone; NO undo or safety copy anywhere; the household reads „Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1 stderr: …" (raw English Docker output). Mechanism `internal/backup/restore_unit.go:428-489`, `restore.go:215-246`. The OFF-SITE path got „put back exactly as it was" in v0.336.0 (D20 = A, R-893); the LOCAL path — 07 §3's main customer restore — did not. Two reviewers found it independently (R1, D4). | WAITING-ON-OPERATOR | decision: extend D20's put-back (safety dump + live-volume copy before the first change, rollback on failure) to the local and second-drive unit restores? | Operator: answer the D-question in STATUS; then CC builds it with the R-893 harness | operator |
 | **R-1178** | Backup & restore | P2 | **The second drive's „Teljes visszaállítás" can change files, then refuse the database half, leave the app STOPPED, and tell the household „the app and its data are untouched".** `tier2_whole.go:301-340` stops the app and merges the files first, then calls the unit restore, whose refusals (cut-off dump, mixed versions, allow-list, data key — `restore_unit.go:366-423`) all return before its own stop/start. PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-R5-whole-restore/`): 1 file restored + 1 replaced (the live one kept beside), calls `[stop]`, no start, message „…A visszaállítás biztonsági okból nem indult el — az alkalmazás és az adatai érintetlenek." Data-loss hunt 2026-10-20 (C1 reviewer 1). | OPEN | — | Fix: run the unit's pre-checks BEFORE the file half (or start the app and say what changed on a refusal); red-proof with the C3 test | CC |
 | **R-1179** | Backup & restore | P2 | **A drive marked „away" can still be chosen as the second-drive backup target.** `tier2.go:151-180` loops `GetSchedulableStoragePaths()`, which drops only decommissioned drives (`settings.go:1593-1603`); nothing checks `Disconnected`. Seen in the R-1175 unit test (a share copied to a drive marked disconnected). On a real box the away drive's mount point is an empty folder on the guest's root disk (or the host's root via the bind), so the night's copy fills the wrong disk and is not where the household thinks. CONFIRMED in code; not run live. Data-loss hunt 2026-10-20. | OPEN | — | Fix: skip a disconnected target like a decommissioned one; live check on a scratch box with a removed drive | CC |
-| **R-1180** | Backup & restore | P2 | **The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete files not yet copied back.** `RunAllTier2` (`tier2.go:479-520`) takes no backup flag and does not ask about a running restore; the whole restore holds only the backup flag (`tier2_whole.go:301`), the night leg only `LegLock` (`cmd/controller/main.go:1348-1352`). A long restore of a photo library at 03:30, or 15 min after a missed night, overlaps. CONFIRMED in code; not run live. Data-loss hunt 2026-10-20 (C1 reviewer 4, T1). | OPEN | — | Fix: the mirror leg skips (and says so) while any restore runs; test with the existing tier-2 seams | CC |
+| **R-1180** | Backup & restore | P2 | **The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete files not yet copied back.** `RunAllTier2` (`tier2.go:479-520`) takes no backup flag and does not ask about a running restore; the whole restore holds only the backup flag (`tier2_whole.go:301`), the night leg only `LegLock` (`cmd/controller/main.go:1348-1352`). A long restore of a photo library at 03:30, or 15 min after a missed night, overlaps. PROVEN against production code with the real `rsync -a --delete` mirror (`audits/dataloss-hunt-2026-10-20/C3-T1-mirror-during-restore/`): with the backup flag held by a restore, `RunAllTier2` still mirrors, and the mirror goes from [a,b,c,d] to [a,b] — the two files the restore was bringing back are gone from the only copy. Data-loss hunt 2026-10-20 (C1 reviewer 4, T1). | OPEN | — | Fix: the mirror leg skips (and says so) while any restore runs; test with the existing tier-2 seams | CC |
 | **R-1181** | Backup & restore | P2 | **A missing or emptied app folder deletes its second-drive copy within one night; for OPTIONAL folders (audiobooks, comics, ROMs, Immich's external library) that copy was the only one.** A missing declared leg is skipped (`tier2_capture.go:55-61`, silent for optional legs) and then reconciled away (`tier2.go:454`); an emptied one is mirrored with `rsync -a --delete` (`:744-751`). PROVEN for a missing folder against production code (`audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/`): the mirror's `photos` deleted, run „ok", warning only „nem kerültek a másodlagos mentésbe". 07 §8.2 chose „data legs are not guarded" for a leg that LEAVES the class set; a declared leg that is MISSING is not that case. Optional legs are in no off-site tier (07 §6.2). | WAITING-ON-OPERATOR | decision: keep a missing leg's mirror (and say so) until the household releases it? And: does an accidental folder deletion need a second-drive history for optional legs? | Operator: answer in STATUS | operator |
 | **R-1182** | Backup & restore | P2 | **Abandoning ONE set-aside off-site copy deletes EVERY retained key package of the customer, including ones that open other set-aside copies still stored.** hub `api/handler.go:626-627` → `PurgeSupersededEscrowForCustomer` (`store.go:4185-4190`) deletes customer-wide; the comment right above (`:4176-4179`) states the invariant this breaks. PLAUSIBLE (code read by reviewer and main session; not run). Data-loss hunt 2026-10-20 (C1 reviewer 3, K2). | VERIFY | — | Prove with a hub store test (two superseded rows, one abandon); then purge only the package that covers the abandoned path | CC |
 | **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** **READY 2026-10-20 — measured and planned, nothing changed; the decision is the operator's (STATUS D21).** Re-measured on 9202 (Immich v3.2.4): off-site ~1.33 GB = model cache 824 MB (62 %), raw database folder 283 MB (21 %), Immich's own backups 115 MB (9 %), logical dump 54 MB, originals 49 MB, thumbnails 2 MB — 102 MB must survive. A: leave out all three re-fetchable parts (−92 %; needs a never-run restore from the dump alone on the vectorchord image). B: only the model cache (−62 %; needs one restore with an empty cache). Pick: B now, A's other parts after their proof (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`). | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC |