felhom.eu/REPORT.md

REPORT — C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (2026-07-28)

Overwritten per the standing rule. Shipped: controller v0.183.0, live on both demo boxes. Fleet: hub v0.80.0, agent v0.110.0, controller 0.183.0. peti-felhom untouched.

Both defects are the same shape — the system reporting healthy while the customer is not — and both live in the same status-derivation code.

Phase 0 — the asymmetry, sized before designing anything

Tier-2 writes two things on every run: the capture legs (hdd/, userdata/) and, always, a full recovery-unit/ (DB dumps + named-volume tarballs). RestoreTier2Files reads only the two legs (tier2_restore.go:101-104) and has never opened recovery-unit/.

All 53 catalog templates enumerated, cross-checked against both boxes' actual copies:

class	count	what the restore can return
A	9	the file legs only — never their database or named volumes
B	43	nothing at all — a guaranteed no-op, forever
C	1	bentopdf, stateless

Four apps (plex, jellyfin, emby, navidrome) are in B only because their single bind is a :ro media mount, which ClassifyBinds correctly excludes. 81% of the catalog.

The asymmetry is Tier-2's alone. Tier-1 (RestoreFromRecoveryUnit) and offsite both restore the unit and replay volume dumps — so BookStack already had a working restore; only this button lied.

Shipped — Part 1a (honesty)

• Refuses UP FRONT. Tier2RestoreCoverage is consulted before any op begins; a class-B app is refused without being stopped. Live: BookStack uptime stayed Up About an hour (Campaign 9 left it at Up 25 seconds).
• Names the working action rather than dead-ending 81% of the catalog:

  „Ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza — az alkalmazás nem állt le. Használd a Visszaállítás indítása gombot a Biztonsági mentés → Visszaállítás oldalon."

• Claims only what was examined (the QUIET half — immich's 1.3 GB Postgres unit is not covered, so the old blanket sentence was a clean bill of health over data never opened):

  „Minden vizsgált fájl megvan a helyén." + „Az alkalmazás adatbázisa és belső kötetei nem tartoznak ebbe a visszaállításba."

Shipped — Part 2 (C9-F2)

StateRestarting is deliberately NOT added to IsDownState — that alarms on every deploy and update fleet-wide. A sustained run becomes down after crashLoopAfter = 5m, set above the three real numbers already in the codebase: the deploy flow's 120 s health timeout, Mealie's 60 s start_period, and R-97b's 180 s quiesce grace (so the windows compose into one bounded delay instead of leaving a gap). Docker's backoff caps at 60 s, so a real loop registers ≥4 attempts inside it. Carried by Stack.RestartingSince (not persisted) + Stack.CrashLooping(now), used by both the alarm and the dashboard counter — which previously counted restarting as running and so contradicted the alarm on the same screen.

Live replay (demo-hp + demo-felhom)

#	scenario	result
1	crash loop alarms	Ten consecutive 30 s samples silent through the threshold window, then 17:05:40 Event pushed: app_start_failed (warn) — Telepített alkalmazás nem fut: Uptime Kuma — 5m25s after the loop began (5 min + one scan).
1b	heartbeat COUNTS it	17:06:10 [deadapp] check alive: 20 scans since boot, 2 deployed app(s) evaluated, **1 currently down** — Campaign 9's evidence was 0 currently down while an app looped.
2	normal deploy is silent	A real uptime-kuma deploy produced only app_deployed (info); no alarm, with deadapp-check running every 30 s throughout.
3	restore refuses without an outage	The honest message rendered; [WARN] Tier-2 file restore refused up front: stack=bookstack has no restorable subtree in its copy (unit_present=true) — app NOT stopped; BookStack uptime unbroken.
4	paperless still restores (regression guard on Campaign 9's headline)	3 files deleted → restored → 43/43 byte-identical to the pre-deletion sha256 set, documents_ok 16 of 16 problems [].

Filed, not fixed

• C9-F1b — route class-B apps to the Tier-1 unit restore from the card the customer already opened. Its own task deliberately: it puts a DESTRUCTIVE operation (overwrites live data with the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry that difference.
• C9-F4 — nothing reads the Tier-2 copy's recovery-unit/ mirror. Written by every Tier-2 run (tier2.go:369), read by no path: RecoveryUnitPath resolves to backups/**primary**/ (appbackup/paths.go:46-48), and the only reader of the secondary tree is tier2_restore.go:79. Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary unit is gone while this mirror survives on the second drive, unreachable by any customer action, leaving offsite as the only route. Potentially larger than C9-F1.

Tests

go test ./... rc=0, 27 packages — run and rc read separately from the commit. Six red-proofs all observed, including the one that matters most: adding StateRestarting to IsDownState fails the brief-restart test with "every deploy and update would page the operator".

Observations

• A seventh shipped-invariant-comment. controller/README.md stated that faults "still surface as exited/degraded/restarting/unhealthy" — but restarting was in no down set at all. The sentence was a wish with no test pinning it. Corrected in place, with the threshold rule documented beside it.
• Pre-existing gate failure, not mine: scripts/docker_run_volume_path_gate.py fails on internal/appexport/estimate.go:179 (an unreviewed docker run -v). Verified it fails identically on clean HEAD; left alone as out of scope rather than silently "fixed".
• The other six gates pass (template_id, emoji, mojibake, native_confirm, app_row_dedup, offbox_rename).

NOT yet live-validated — carried forward

• Tier-1 content recovery after real loss — still the most valuable unproven item (Campaign 9's A2 ran against an intact app; A3 used Tier-2). Unchanged by this work.
• The C9-F2 threshold under a quiesce cycle (Scenario C) is unit-proven but was not replayed live; it needs a backup window plus an app that fails to come back.
• C9-F1's refusal for the other 42 class-B apps is proven by enumeration and by BookStack live, not app-by-app.
• Host reboot mid-backup, three-way concurrency with GC, Scenario C live, offsite_stale firing, F-HUB — all still open from Campaign 9.