C9-F1 (honesty half) + C9-F2 shipped in controller v0.183.0; C9-F1b and C9-F4 filed

Phase 0 sized C9-F1 properly before anything was designed: 43 of the 53 catalog apps have NO
subtree the Tier-2 restore can read (not 2), 9 are covered only for their file legs and never
their database or volumes, 1 is stateless. The asymmetry is Tier-2's alone — Tier-1 and offsite
both restore the unit and replay volume dumps, so BookStack always had a working restore and
only this button lied.

Shipped: the restore refuses BEFORE stopping the app and names the action that does work; a run
that proceeds claims only what it EXAMINED and discloses that the database and volumes are not
covered. C9-F2 alarms after a 5-minute sustained-restarting threshold, set above the 120s deploy
timeout, Mealie's 60s start_period and R-97b's 180s grace; StateRestarting is deliberately NOT
added to IsDownState.

Live: silent through ten 30s samples then app_start_failed at 5m25s, heartbeat now reads
"1 currently down" where Campaign 9 recorded 0; a real deploy stayed silent; bookstack refused
with its uptime unbroken; paperless re-restored 43/43 byte-identical, 16/16 docs clean.

Filed, not fixed: C9-F1b (route to the Tier-1 restore — its own task because it puts a
destructive operation behind a non-destructive button) and C9-F4 (nothing reads the Tier-2
copy's recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable
by any customer action — potentially larger than C9-F1).

REPORT.md

@@ -1,85 +1,103 @@
-# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28)
+# REPORT — C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (2026-07-28)
 
-**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign,
-and findings are recorded, never fixed inline. Full write-up:
-`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`.
-Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously).
+**Overwritten** per the standing rule. **Shipped: controller v0.183.0**, live on **both** demo boxes.
+Fleet: hub v0.80.0, agent v0.110.0, controller **0.183.0**. `peti-felhom` untouched.
 
-Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes.
-**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact.
+Both defects are the same shape — the system reporting healthy while the customer is not — and both
+live in the same status-derivation code.
 
-## The headline — two never-proven restore paths are now proven
+## Phase 0 — the asymmetry, sized before designing anything
 
-Driven through the **real endpoints the UI posts to**, over https through traefik with a real session
-and CSRF token, on live hardware.
+Tier-2 writes **two** things on every run: the capture legs (`hdd/`, `userdata/`) and, always, a full
+`recovery-unit/` (DB dumps + named-volume tarballs). `RestoreTier2Files` reads **only the two legs**
+(`tier2_restore.go:101-104`) and has never opened `recovery-unit/`.
 
-| proof | result |
-|---|---|
-| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) |
-| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) |
-| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy |
-| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 |
-| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy |
-| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 |
+All 53 catalog templates enumerated, cross-checked against both boxes' actual copies:
 
-The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return
-at their last-backup state.**
-
-## Findings — 3 defects, ranked (none fixed)
-
-| # | finding | severity |
+| class | count | what the restore can return |
 |---|---|---|
-| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** |
-| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** |
-| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** |
+| **A** | **9** | the file legs only — **never** their database or named volumes |
+| **B** | **43** | **nothing at all** — a guaranteed no-op, forever |
+| C | 1 | `bentopdf`, stateless |
 
-Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog
-sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an
-artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed
-so live.
+Four apps (`plex`, `jellyfin`, `emby`, `navidrome`) are in B only because their single bind is a
+`:ro` media mount, which `ClassifyBinds` correctly excludes. **81% of the catalog.**
 
-## Mechanisms confirmed working, live
+**The asymmetry is Tier-2's alone.** Tier-1 (`RestoreFromRecoveryUnit`) and offsite both restore the
+unit and replay volume dumps — so BookStack already had a working restore; only this button lied.
 
-R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target
-breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app
-thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of
-the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on
-demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's
-sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the
-hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable;
-F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests).
+## Shipped — Part 1a (honesty)
 
-## Where it stopped, and what remains
+- **Refuses UP FRONT.** `Tier2RestoreCoverage` is consulted before any op begins; a class-B app is
+  refused **without being stopped**. Live: BookStack uptime stayed `Up About an hour` (Campaign 9
+  left it at `Up 25 seconds`).
+- **Names the working action** rather than dead-ending 81% of the catalog:
+  > „Ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza — az alkalmazás nem
+  > állt le. Használd a Visszaállítás indítása gombot a Biztonsági mentés → Visszaállítás oldalon."
+- **Claims only what was examined** (the QUIET half — immich's 1.3 GB Postgres unit is not covered,
+  so the old blanket sentence was a clean bill of health over data never opened):
+  > „Minden vizsgált fájl megvan a helyén." + „Az alkalmazás adatbázisa és belső kötetei nem
+  > tartoznak ebbe a visszaállításba."
 
-Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host
-reboot mid-backup) was deliberately not started — a large new fault class against boxes that are
-remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown.
+## Shipped — Part 2 (C9-F2)
 
-**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's
-threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time
-`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign
-forbids. Reported rather than worked around. The no-code-change alternative (age the controller's
-reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended
-method next time.
+`StateRestarting` is deliberately **NOT** added to `IsDownState` — that alarms on every deploy and
+update fleet-wide. A **sustained** run becomes down after `crashLoopAfter = 5m`, set above the three
+real numbers already in the codebase: the deploy flow's **120 s** health timeout, Mealie's **60 s**
+`start_period`, and R-97b's **180 s** quiesce grace (so the windows compose into one bounded delay
+instead of leaving a gap). Docker's backoff caps at 60 s, so a real loop registers ≥4 attempts inside
+it. Carried by `Stack.RestartingSince` (not persisted) + `Stack.CrashLooping(now)`, used by **both**
+the alarm and the dashboard counter — which previously counted `restarting` as running and so
+contradicted the alarm on the same screen.
 
-**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an
-intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way
-concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`.
+## Live replay (demo-hp + demo-felhom)
 
-## Recovery
+| # | scenario | result |
+|---|---|---|
+| 1 | **crash loop alarms** | Ten consecutive 30 s samples **silent** through the threshold window, then `17:05:40 Event pushed: app_start_failed (warn) — Telepített alkalmazás nem fut: Uptime Kuma` — **5m25s** after the loop began (5 min + one scan). |
+| 1b | **heartbeat COUNTS it** | `17:06:10 [deadapp] check alive: 20 scans since boot, 2 deployed app(s) evaluated, **1 currently down**` — Campaign 9's evidence was `0 currently down` while an app looped. |
+| 2 | **normal deploy is silent** | A real `uptime-kuma` deploy produced only `app_deployed (info)`; no alarm, with deadapp-check running every 30 s throughout. |
+| 3 | **restore refuses without an outage** | The honest message rendered; `[WARN] Tier-2 file restore refused up front: stack=bookstack has no restorable subtree in its copy (unit_present=true) — app NOT stopped`; BookStack uptime unbroken. |
+| 4 | **paperless still restores** (regression guard on Campaign 9's headline) | 3 files deleted → restored → **43/43 byte-identical to the pre-deletion sha256 set**, `documents_ok 16 of 16 problems []`. |
 
-Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive
-observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active),
-windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on
-either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and
-the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots).
+## Filed, not fixed
 
-One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box.
-Its `until` is long past so it blocks nothing; it clears on the next successful backup or any
-controller restart (by design, not persisted). Clearing it would have cost another app outage for no
-benefit.
+- **C9-F1b** — route class-B apps to the Tier-1 unit restore from the card the customer already
+  opened. Its own task **deliberately**: it puts a DESTRUCTIVE operation (overwrites live data with
+  the backup state) behind a button reached via a NON-destructive one, so the confirm copy must carry
+  that difference.
+- **C9-F4** — **nothing reads the Tier-2 copy's `recovery-unit/` mirror.** Written by every Tier-2 run
+  (`tier2.go:369`), read by no path: `RecoveryUnitPath` resolves to `backups/**primary**/`
+  (`appbackup/paths.go:46-48`), and the only reader of the secondary tree is `tier2_restore.go:79`.
+  Tier-2 exists for the case where the PRIMARY drive is lost — and in exactly that case the primary
+  unit is gone while this mirror survives on the second drive, unreachable by any customer action,
+  leaving offsite as the only route. **Potentially larger than C9-F1.**
 
-**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in
-`/opt/docker/stacks/<app>` starts a Felhom app **without its secrets** — they are injected by the
-controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a
-crash loop during recovery. Manual recovery must go through `POST /api/stacks/<name>/restart`.
+## Tests
+
+`go test ./...` **rc=0, 27 packages** — run and `rc` read *separately* from the commit. Six red-proofs
+all observed, including the one that matters most: adding `StateRestarting` to `IsDownState` fails the
+brief-restart test with *"every deploy and update would page the operator"*.
+
+## Observations
+
+- **A seventh shipped-invariant-comment.** `controller/README.md` stated that faults "still surface as
+  `exited`/`degraded`/**`restarting`**/`unhealthy`" — but `restarting` was in no down set at all. The
+  sentence was a wish with no test pinning it. Corrected in place, with the threshold rule documented
+  beside it.
+- **Pre-existing gate failure, not mine:** `scripts/docker_run_volume_path_gate.py` fails on
+  `internal/appexport/estimate.go:179` (an unreviewed `docker run -v`). Verified it fails identically
+  on clean HEAD; left alone as out of scope rather than silently "fixed".
+- The other six gates pass (`template_id`, `emoji`, `mojibake`, `native_confirm`, `app_row_dedup`,
+  `offbox_rename`).
+
+## NOT yet live-validated — carried forward
+
+- **Tier-1 content recovery after real loss** — still the most valuable unproven item (Campaign 9's A2
+  ran against an intact app; A3 used Tier-2). Unchanged by this work.
+- The C9-F2 threshold under a **quiesce** cycle (Scenario C) is unit-proven but was not replayed live;
+  it needs a backup window plus an app that fails to come back.
+- C9-F1's refusal for the other 42 class-B apps is proven by enumeration and by BookStack live, not
+  app-by-app.
+- Host reboot mid-backup, three-way concurrency with GC, Scenario C live, `offsite_stale` firing,
+  F-HUB — all still open from Campaign 9.