C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (v0.183.0)
Both are the system reporting healthy while the customer is not, and both live in the same status-derivation code. Neither is fixed by making the system quieter. C9-F1 (HIGH) — Tier-2 writes recovery-unit/ on EVERY run and RestoreTier2Files has never read it (tier2_restore.go:101-104 reads hdd/ + userdata/ only). Phase 0 enumerated all 53 catalog templates against both demo boxes: 43 apps have NO readable subtree, so the button stopped the app, restored 0 files, restarted it and said "Nincs hiányzó fájl — minden fájl megvan a helyén." — at the moment the customer pressed it because files were missing, with 156 MB of BookStack's data unread in the same copy. 9 apps have file legs but never their DB or volumes, so the same sentence was also a clean bill of health over data never opened (immich: 1.3 GB Postgres unit). Honesty half shipped: a pre-flight coverage check refuses UP FRONT without stopping the app and NAMES the action that works; a run that proceeds claims only what it EXAMINED and discloses that the database and volumes are not covered. Completeness is filed as C9-F1b — routing to the Tier-1 unit restore puts a destructive operation behind a non-destructive button, so its confirm copy has to carry that difference. C9-F4 filed: nothing reads the Tier-2 recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable by any customer action. C9-F2 (HIGH) — a crash loop was counted as working. StateRestarting is deliberately NOT added to IsDownState (that alarms on every deploy fleet-wide, the over-correction F-A1 nearly cost us); a sustained run becomes down after crashLoopAfter = 5m, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace. The dashboard counter uses the same predicate, so it no longer contradicts the alarm on the same screen. README's claim that faults "still surface as restarting" was a wish with no test — corrected in place; it is the seventh such instance. Six red-proofs observed, including the one that matters most: adding StateRestarting to IsDownState fails the brief-restart test with "every deploy and update would page the operator". go test ./... rc=0, 27 packages, run and read separately from this commit.
This commit is contained in:
@@ -1,5 +1,66 @@
|
||||
## Changelog
|
||||
|
||||
### v0.183.0 — C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (2026-07-28)
|
||||
|
||||
Both are the same shape — the system reporting healthy while the customer is not — and both were
|
||||
found by Campaign 9 on live hardware.
|
||||
|
||||
**C9-F1 (HIGH).** Tier-2 writes TWO things on every run: the capture legs (`hdd/`, `userdata/`) and,
|
||||
always, a full `recovery-unit/` — the app's DB dumps and named-volume tarballs. `RestoreTier2Files`
|
||||
reads **only the two legs** (`tier2_restore.go:101-104`) and has never opened `recovery-unit/`. For an
|
||||
app whose data lives entirely in named volumes that is its ENTIRE dataset, so pressing
|
||||
„Fájlok visszaállítása" stopped the app, restored 0 files, restarted it, and reported
|
||||
„Nincs hiányzó fájl — minden fájl megvan a helyén." — at the exact moment the customer pressed it
|
||||
BECAUSE files were missing, while 156 MB of BookStack's data sat unread in the same copy.
|
||||
|
||||
**Phase 0 enumerated all 53 catalog templates** (cross-checked against both demo boxes' actual copies):
|
||||
**43 apps** have no readable subtree at all — the restore is a guaranteed no-op for them, forever —
|
||||
**9** have file legs but never their database or volumes, and 1 is stateless. Four apps (`plex`,
|
||||
`jellyfin`, `emby`, `navidrome`) are in the 43 only because their single bind is a `:ro` media mount,
|
||||
which `ClassifyBinds` correctly excludes.
|
||||
|
||||
Fixed on the honesty axis (completeness is filed as C9-F1b, see below):
|
||||
- a **pre-flight coverage check refuses UP FRONT** — no op begun, and the app is **not stopped**;
|
||||
- the refusal **names the action that works** instead of dead-ending 81% of the catalog:
|
||||
„Ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza — az alkalmazás nem állt
|
||||
le. Használd a Visszaállítás indítása gombot a Biztonsági mentés → Visszaállítás oldalon.";
|
||||
- where the restore DOES run it now claims only what it **examined** —
|
||||
„Minden vizsgált fájl megvan a helyén." — plus, whenever a unit is present,
|
||||
„Az alkalmazás adatbázisa és belső kötetei nem tartoznak ebbe a visszaállításba." That second string
|
||||
closes the QUIET half: immich's 1.3 GB Postgres unit is not covered, so the old blanket sentence was
|
||||
a clean bill of health over data the operation never opened.
|
||||
|
||||
New seam: `Manager.Tier2RestoreCoverage` + `Tier2Coverage{Legs, HasUnit}`, computed from the RECORDED
|
||||
copy on disk rather than the catalog, so an app whose template changed is judged by what it actually has.
|
||||
|
||||
**C9-F2 (HIGH).** `IsDownState` excludes `restarting` as "self-recovering", but with the catalog's
|
||||
standard `restart: unless-stopped` Docker retries forever — so a crash loop was counted as working.
|
||||
Campaign 9 watched docmost loop for nine minutes (restartcount 18) while F-OBS's heartbeat printed
|
||||
„180 scans since boot, 4 deployed app(s) evaluated, **0 currently down**". No banner, no
|
||||
`app_start_failed`, no email, no hub event, indefinitely.
|
||||
|
||||
`StateRestarting` is deliberately **NOT** added to `IsDownState` — that would alarm on every deploy and
|
||||
update fleet-wide, the over-correction F-A1 nearly cost us. Instead a sustained restarting run becomes
|
||||
down after `crashLoopAfter = 5m`, justified against three numbers already in this codebase: the deploy
|
||||
flow's **120 s** health timeout, Mealie's **60 s** `start_period` (the slowest catalog healthcheck), and
|
||||
R-97b's **180 s** quiesce grace — which the threshold must exceed so the two windows compose into one
|
||||
bounded delay instead of leaving a gap. Docker's own backoff caps at 60 s, so a real crash loop
|
||||
registers ≥4 attempts inside the window. New `Stack.RestartingSince` (not persisted, same reasoning as
|
||||
the R-88 breaker) + `Stack.CrashLooping(now)`, used by BOTH the alarm and the dashboard counter — which
|
||||
previously counted `restarting` as running, contradicting the alarm on the same screen.
|
||||
|
||||
Red-proofs, all observed: crash-loop term removed → A fails; **StateRestarting naively added to
|
||||
IsDownState → B fails** („every deploy and update would page the operator"); quiesce term removed →
|
||||
C fails; coverage guard removed → D fails with the app STOPPED; guard made unconditional → E fails
|
||||
(the paperless regression guard); old blanket message restored → F fails.
|
||||
|
||||
**Filed, not fixed:** **C9-F1b** (route class-B apps to the Tier-1 unit restore — it puts a destructive
|
||||
operation behind a button reached via a non-destructive one, so the confirm copy has to carry that
|
||||
difference) and **C9-F4** (`backups/secondary/<stack>/recovery-unit/` is written by every Tier-2 run and
|
||||
read by NOTHING — `RecoveryUnitPath` resolves to `backups/primary/`, so the second local copy that
|
||||
exists precisely for drive loss is unreachable by any customer action).
|
||||
|
||||
|
||||
### v0.182.0 — R-101 + F-DIAG: the customer must not be told a failed backup is a copy (2026-07-28)
|
||||
|
||||
**R-101.** `Tier2LastRun` is the ATTEMPT clock — `recordTier2Failure` writes it too — and it was
|
||||
|
||||
Reference in New Issue
Block a user