Campaign 9: the Tier-2 restore paths are PROVEN; 3 defects filed, none fixed

Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.

Three defects, recorded not fixed:
  C9-F1 (HIGH)   the Tier-2 restore button is offered for apps it can never restore, takes a
                 real outage, and reports "nothing was missing" — indistinguishable from a
                 genuine result, while 156 MB of that app's data sits unread in the same copy.
  C9-F2 (HIGH)   an app in a crash loop never alarms on any channel; StateRestarting is in no
                 down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
  C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
                 is dead until a human unlocks and the operator is told "unknown reason".
                 This answers Phase C item 8.

Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.

Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.

documentation/backlog/OPEN-ITEMS.md

@@ -28,6 +28,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
 | **F-A1** | ~~A restore-test in flight made a healthy backup report as FAILED (HTTP 409 read as a tier failure): breaker armed + operator emailed, on both boxes~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → **0** operator emails, real failure → **1** | — |
 | **R-100** | ~~A restic offsite tier that fails every night never goes stale on the hub — `isStale` counted from `LastRun`, which the controller writes unconditionally on failure~~ | **SHIPPED + PROVEN-LIVE** (controller v0.181.0 + hub v0.80.0, 2026-07-28) | — | Anchored on a new `last_success`. **Severity corrected during Phase 0: this was NOT a silence** — `backup_failed` does fire nightly and reaches the operator (live DB: 5 sends). The real defect is **defeated defence in depth**: the hub-side *pull* net was anchored on a field the failing controller keeps refreshing, so it could not compensate for a lost *push* (cf. F-HUB). Live on demo-hp: induced failure → `last_run` advanced 11:25:48Z, `last_success` **held** 11:24:20Z; demo-felhom healthy → anchor advanced. Legacy degrade logged once per customer, live | — |
 | **R-101** | ~~Tier-2 `LastRun` is written on failure and rendered to the customer as „Legutóbbi másolat" — including in the restore confirm dialog~~ | **SHIPPED + PROVEN-LIVE** (controller v0.182.0, 2026-07-28) | — | `CrossDriveBackup.LastSuccess` + `SuccessTracked`; the dialog names the last **successful** copy and discloses a failed newest attempt. Legacy rows migrate truthfully on first touch (an `ok` row adopts its time; an `error` row seeds nothing) — without the marker all 7 fleet rows would have flipped to „Még nincs sikeres másolat" on deploy. **Part 2:** the three `record*` sites rebuilt the whole struct; replaced by `tier2Update` (copy-and-overlay, safe by construction) — the naive fix would have had `recordTier2Failure` CLEAR the anchor. Live on demo-hp, rendered dialog read in both states | — |
+| **C9-F1** | Tier-2 „Fájlok visszaállítása" is offered for apps whose Tier-2 copy has **no restorable file leg** (BookStack, Docmost = 2 of the 4 Tier-2 apps on demo-felhom). Clicking it stops the app, restores 0 files, and reports **„Nincs hiányzó fájl — minden fájl megvan a helyén."** — indistinguishable from a genuine "nothing was missing", on every click, forever, while 156 MB of that app's data sits unread in `recovery-unit/` in the same copy | **READY (HIGH)** | — | Either don't render the button when the copy has no `hdd/`+`userdata/` leg, or make the message distinguish "nothing missing" from "nothing restorable here". `tier2_restore.go:101-104` reads only those two subtrees; `tier2_restore_test.go:219` pins 0-files-as-success and explicitly accepts the outage; `tier2_honesty_test.go` Scenario D asserts the button MUST render for exactly BookStack's shape | CC |
+| **C9-F2** | An app in a **Docker crash loop never alarms on any channel** — no dashboard banner, no `app_start_failed`, no customer email, no hub event. `aggregateState` returns `StateRestarting` (`manager.go:753-756`, checked before the degraded branch) and `IsDownState` (`manager.go:54-56`) excludes it, so `classifyRunStates` computes `down=false`. With `restart: unless-stopped` this is **permanent** silence. F-CRIT-1's fix covers `StateStopped` only | **READY (HIGH)** | — | Add `StateRestarting` to the down set with a debounce, or watch `RestartCount`. Proven live on demo-felhom: 9+ min crash loop, restartcount 18, while the F-OBS heartbeat printed **"180 scans since boot, 4 deployed app(s) evaluated, 0 currently down"** | CC |
+| **C9-F3** | An **interrupted offsite run leaves an exclusive restic lock the self-heal cannot reach**: `resticStep` (`offbox.go:634-648`) has `unlock --remove-all`, but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`offbox.go:77-93`) has no lock case → `"other"` → fail-fast. Tier dead until a human unlocks; `ClassifyOffsiteFailure` likewise has no lock case so the operator is told **„A távoli mentés ismeretlen okból nem sikerült"** for a precisely-known, self-healable condition | **READY (MEDIUM)** | — | Add a lock case to both classifiers and let the probe path escalate to `unlock --remove-all`. Answers Phase C item 8: the repo is NOT usable after a killed run. Cleared manually this run; tier proven working again (`ok`, 1m35s). Reachable by any interruption — container restart, OOM, **host reboot mid-backup** | CC |
 | **F-DIAG** | ~~Four distinct offsite failure causes collapse into two operator-visible strings~~ | **SHIPPED** (controller v0.182.0, 2026-07-28) | — | `ClassifyOffsiteFailure` → quota / orphaned / no_repo / no_units / transport / **unknown**, each with its own Hungarian message. Unclassifiable says so rather than being folded into a neighbour. **Secrets:** the old message was a raw `err.Error()` passthrough carrying `sftp:<user>@<host>:<path>`; redaction is now by the target's **actual** host/user/path (a first regex-only attempt leaked on a bare hostname and its own test caught it). Unit-proven; **not** yet exercised by a live offsite failure of each class | — |
 | **F-OPS** | ~~A manual `pct restore` inherits the source guest's bind mounts — during a real DR, on a different host, under pressure~~ | **DOCUMENTED** (2026-07-28) | — | `documentation/runbooks/RUNBOOK-manual-guest-restore.md`: which `mpN` are volumes vs host binds, the `mp9` source-VMID trap (it can bind **another guest's bootstrap credentials**), strip-and-re-add before first boot, and a positive pre-start verification. Docs only by design — the agent already neutralises binds on its own restore paths, and a second implementation would drift | — |
 | **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — |