felhom.eu/documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md

CAMPAIGN 9 — proof, not fixes: the restore paths and the interactions

Adversarial, destructive, unattended. 2026-07-28 17:12 CEST → in progress.

Nodes exercised: demo-felhom (+ guest 9201), demo-hp (+ guest 9201), ep0 (felhom-hetzner). Controllers on 0.182.0 both boxes; agent 0.110.0; hub 0.80.0.

peti-felhom was never touched — not its data, not its namespace, not u629488-sub2. The ep0 rollback copy /srv/pbs-felhom was never deleted.

Evidence: DooPlex:~/campaign9/evidence/, written continuously by 7 collectors (6 log streams + a 2-minute state snapshot) started before the first mutation and verified emitting live lines before Phase A began.

---

Headline

The two never-proven restore paths are now proven, on live hardware, through the real endpoints the UI posts to. A customer who deletes files — or their entire app data directory — gets everything back, byte-identical, and the app works afterwards.

One HIGH-severity defect fell out of proving it: F-1. For apps whose data lives entirely in Docker named volumes, the „Fájlok visszaállítása" button is rendered, stops the app, restores nothing, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — a message that positively asserts the customer's files are all present. It says this on every click, on any day, no matter how much data was lost, while 156 MB of that app's data sits unread in the same Tier-2 copy the button claims to restore from.

---

1. Phase 0 — questions, answers, and skips

Asked in one batch, before touching anything. All four answered; nothing was skipped for lack of an answer.

#	Question	Answer received
1	Which targets may I destroy data on?	Throwaway apps AND existing demo apps on BOTH boxes.
2	May the hub's staleAfter be compressed for Phase D item 11?	Yes — compress, then revert.
3	How may ep0 be taken down?	Power-cycle allowed programmatically (widened from Campaign 8's "no power-cycle").
4	drill-r50 destroy/rebuild? Host reboot for Fault 12?	drill-r50 free to destroy; host reboot OK on BOTH demo hosts.

---

2. Phase A — the restore proofs (the headline)

A1 — Tier-2 cross-drive restore of ordinary app data — PROVEN

Target. paperless-ngx on demo-hp guest 9201 — chosen because it is the only app in the fleet that combines real user content (16 documents), a complete Tier-2 copy with a real user-file leg, and an R-101 success anchor. Live root /mnt/felhom-drives/nvme-1tb; Tier-2 copy /mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx.

Method. The real endpoint the UI posts to — POST /backup/tier2/restore with stack_name — driven through traefik over https with the vhost Host header, from inside the guest, with a real session cookie and a real CSRF token scraped from the rendered /backups/apps page. No server logic was skipped; only browser rendering.

The rendered dialog (evidence/phaseA/A1-dialog-rendered.txt) — this is the R-101 branch, live:

Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:43.

Baseline. All 43 files in the copy were byte-identical to live before anything was touched (A1-copy-inv-BEFORE.txt vs A1-live-inv-BEFORE.txt, diff empty).

The mutation exercised all three of the dialog's promises in one run — because the dialog makes three distinct claims and proving them separately would have been three outages:

class	files	promise under test
DELETED	6 (3 originals, 1 archive, 2 thumbnails, spanning all subdirs, incl. one complete document)	must come back byte-identical
CREATED	2 (one in an existing dir, one in a new dir)	„NEM törlődnek" — must survive
MODIFIED	1 thumbnail, appended marker	„NEM módosulnak" — must not be overwritten

Result — all three hold (A1-verify.txt):

=== PROMISE 1: deleted files came back BYTE-IDENTICAL ===
originals/0000002.pdf: OK
originals/0000005.pdf: OK
originals/0000012.pdf: OK
archive/0000012.pdf: OK
thumbnails/0000012.webp: OK
thumbnails/0000003.webp: OK

=== PROMISE 2: created files were NOT deleted ===
PRESENT  originals/c9-created-file.txt  be279d439b852e78c6d10379c4f569360283d4ce77523f4848723beb5b78997e
PRESENT  c9-newdir/c9-keep.txt          4fa680f76eeac33369fefec88d1a216cea7d16255573e439302d40c63fa9d6a9

=== PROMISE 3: locally-modified file was NOT overwritten ===
pre-edit  fd9b92dd77df6bd9e6474d238ec86e029e7f562c820ed1172d71ad6cb9eccb42  <- must NOT match
post-edit 15c61ab7ddd1bfcfecc7ac7f43d12357de123b42d3ab127e3a23a46af4bd4fd2  <- must match
actual    15c61ab7ddd1bfcfecc7ac7f43d12357de123b42d3ab127e3a23a46af4bd4fd2
marker still present: CAMPAIGN9-LOCAL-EDIT-MUST-NOT-BE-OVERWRITTEN
live file count: 45     (43 baseline + 2 created — arithmetic closes)

Stop/start and health, from the controller (A1-controller-log.txt):

15:21:59 [WARN] [web] Tier-2 file restore requested (async): stack=paperless-ngx
15:21:59 [INFO] [backup] Tier-2 file restore for paperless-ngx: …/secondary/paperless-ngx (v2) → /mnt/felhom-drives/nvme-1tb (additive-only)
15:22:20 … Post-restore health check: paperless-ngx not yet running, waiting…   (×5)
15:22:45 [DEBUG] [backup] Post-restore health check: paperless-ngx is running
15:22:45 [INFO] [backup] Tier-2 file restore completed for paperless-ngx: 6 file(s) restored (39s)

Container uptime confirmed the real stop/start independently (Up 5 seconds (health: starting) → Up 27 seconds (healthy) polled at 10-second intervals).

The restored data is usable BY THE APP, not merely present on disk (A1-usable.txt) — two independent checks:

=== CHECK 1: paperless resolves each restored doc and its content matches the DB checksum ===
doc 2   invoice-114750664          original_exists=True checksum_match=True thumb_exists=True
doc 5   Project Overtime 2023.11   original_exists=True checksum_match=True thumb_exists=True
doc 12  Hotel                      original_exists=True checksum_match=True archive_exists=True archive_match=True thumb_exists=True

=== CHECK 2: served over paperless's own HTTP API, hashing what the app returns ===
doc 2  http=200  served_sha256=87226ff2…13244f   == recorded before deletion
doc 5  http=200  served_sha256=1395403a…f2f2df   == recorded before deletion
doc 12 http=200  served_sha256=1dd4cd16…dd924b   == recorded before deletion (archive; /download/ serves the archive when one exists)

A1 verdict: PROVEN. Byte-identical recovery, both non-destruction promises kept, app healthy and serving the restored bytes over its own API.

A2 — Tier-1 local recovery-unit restore — DISTINCT, and exercised

It is a distinct path, not subsumed: POST /backup/restore (stack_name + snapshot_id) → RestoreFromRecoveryUnit (internal/web/handlers.go:1230, internal/backup/restore_unit.go:96). It reads the app's on-drive recovery unit, recovers secrets from the guest's live app.yaml, restores named-volume data from the unit's tars, and redeploys. Tier-2 copies are not reachable through it.

Exercised on calibre-web (demo-felhom), A2-restore-status.txt / A2-log.txt:

15:28:34 [INFO] [backup] Restoring calibre-web from recovery unit: images=1, secrets recovered=0/0, data_keys=0
15:28:39 [INFO] [backup] Restoring Docker volume calibre-web_calibre_web_config for calibre-web
15:28:39 [INFO] [backup] Restored 1 Docker volume(s) for calibre-web
15:28:52 [INFO] [backup] Restore-from-unit completed: calibre-web        (18.25s)

App healthy 29 s later; volume fingerprint unchanged (6b10ec90…), consistent with restoring a tar that matched current content.

Honest limit: this run proves the path executes and the app survives it. It does not prove content recovery, because nothing was lost first. The content-recovery proof is A3.

One thing checked and deliberately NOT reported as a finding. snapshot_id is required by the handler and then never passed to RestoreFromRecoveryUnit, while the success flash names it — which looks like "you picked A and got B". It is neither: restore_points.go:14-18 states there is exactly one keep-side restore point and that snapshot_id is logging-only, and the live API confirms exactly one point per app with short_id: "helyi" — the value the flash prints. Correct by design, pinned by its own comment. ARTIFACT of reading, not a defect.

A3 — restore after REAL loss — PROVEN

The harder shape: not gap-fill, but deleting the app's data directory entirely, while the app ran, as an "I deleted my files" event actually looks.

=== BEFORE: paperless serves doc 12 ===      source_exists True
=== DESTROYING /mnt/felhom-drives/nvme-1tb/appdata/paperless ===
exists after rm: NO
=== AFTER ===                                source_exists False
                                             doc12_download_http=404
=== doc count in DB is untouched ===         16

The loss was verified with a positive observable — the document download going 200 → 404 — not merely by the directory being absent.

Tier used: Tier-2, via the same customer button. Result (A3-restore-status.txt, A3-verify.txt):

message: "paperless-ngx: 43 fájl visszaállítva a másodlagos másolatból."   (46 s)

=== 1. every restored file byte-identical to the Tier-2 copy? ===
copy files: 43 / live files: 43
IDENTICAL — every file in the copy is back, byte-for-byte, and nothing extra

=== 2. does PAPERLESS work? ===
documents_ok 16 of 16
problems []

=== 3. serve doc 12 over HTTP again (the observable that was 404 after the loss) ===
doc 2  http=200 sha256=87226ff2…13244f
doc 5  http=200 sha256=1395403a…f2f2df
doc 12 http=200 sha256=1dd4cd16…dd924b

A3 verdict: PROVEN — the customer gets their app back, not "only some files".

The honest residue, stated plainly. Three things did not come back, and none of them is a defect — each is the correct consequence of not being in the last successful copy:

• the two files created after the last Tier-2 run — never backed up, correctly gone;
• the local edit to thumbnails/0000001.webp — the file returned in its pristine backed-up form (fd9b92dd…), i.e. the post-loss restore legitimately reverted an unbacked-up edit. This is the same file that A1 proved is never overwritten while it still exists; the two results are consistent and together delimit the guarantee exactly: existing files are untouched; destroyed files return at their last-backup state.
• appdata/paperless/export/ — absent from the copy, recreated empty by paperless on startup.

---

3. Findings

SHIPPED 2026-07-28 — controller v0.183.0. F-1's honesty half and F-2 in full. Phase 0 of the fix sized F-1 properly: 43 of the 53 catalog apps have no restorable subtree (not 2), 9 are covered only for their file legs, 1 is stateless. The restore now refuses before stopping the app and names the action that works; a run that proceeds claims only what it examined and discloses that the database and volumes are not covered. F-2 alarms after a 5-minute sustained-restarting threshold — proven live: silent through ten 30 s samples, then app_start_failed at 5m25s, and the F-OBS heartbeat now reads 1 currently down where Campaign 9 recorded 0. Completeness is filed as C9-F1b, and C9-F4 was filed during the fix: nothing reads the Tier-2 copy's recovery-unit/ mirror at all. See REPORT.md and CHANGELOG v0.183.0.

F-1 — the restore button reports success for apps it can never restore — DEFECT, HIGH

Applies to: every app whose data lives entirely in Docker named volumes and which therefore has no bind-mounted user-file leg. On demo-felhom that is BookStack and Docmost — 2 of the 4 apps with a Tier-2 copy, i.e. half the fleet's Tier-2-configured apps.

Action. Clicked „Fájlok visszaállítása" for bookstack on demo-felhom via the real endpoint.

Observation (A1b-bookstack-post.txt, A1b-bookstack-log.txt):

15:26:05 [WARN] [web] Tier-2 file restore requested (async): stack=bookstack
15:26:05 [INFO] [backup] Tier-2 file restore for bookstack: …/secondary/bookstack (v2) → /mnt/sys_drive/felhom-data (additive-only)
15:26:28 [INFO] [backup] Tier-2 file restore completed for bookstack: 0 file(s) restored (19s)

restore-status: {"ok":true, … "message":"Nincs hiányzó fájl — minden fájl megvan a helyén."}

BEFORE: bookstack|Up 4 hours (healthy)
AFTER:  bookstack|Up 25 seconds (healthy)        <- a real outage was taken

And the copy the button claims to restore from is not empty — it holds the app's entire data:

16K   …/secondary/bookstack/recovery-unit/compose
60K   …/secondary/bookstack/recovery-unit/db-dumps
156M  …/secondary/bookstack/recovery-unit/volume-dumps      <- BookStack's actual data
4.0K  …/secondary/bookstack/recovery-unit/manifest.json

ls …/secondary/bookstack/  ->  recovery-unit          (no hdd/, no userdata/)

Why it can never succeed. RestoreTier2Files merges exactly two subtrees — destBase/hdd → liveNsRoot and destBase/userdata → liveNsRoot/userdata (internal/backup/tier2_restore.go:101-104) — and os.Stat-skips each when absent (:114-116). recovery-unit/ is never read by this path. For an app with no bind legs, tier2CaptureSet correctly yields zero legs (internal/backup/tier2_capture.go:42-91), so neither subtree is ever written. The loop therefore restores 0 files on every click, forever, and backupTier2RestoreHandler maps n == 0 to the reassuring message (internal/web/handlers.go:1313-1316).

Expectation. Either the button is not offered when the app's Tier-2 copy contains no restorable file leg, or the message distinguishes "nothing was missing" from "this copy holds no restorable files for this app". The customer must not be told their files are all in place by a code path that never looked at where their files live.

Severity: HIGH. This is the customer's own recovery action. The failure mode is the project's own documented trap class twice over: the message is a presence-not-success claim (0 files copied is read as "nothing was missing"), and it is unfalsifiable — identical output whether the app is pristine or has lost everything. It is also the exact shape of the six shipped-invariant comments in the workspace CLAUDE.md: tier2_restore.go:14-24 scopes the helper to "class-C data — HDD bind-mount user files", which is true of the helper and silently false of the button the UI renders on top of it. No test pins the UI-level claim.

DEFECT. Not an artifact: no cadence was compressed and no state was manufactured — BookStack was in its normal, healthy, production configuration.

---

4. What came off the never-proven list

item	status
Tier-2 cross-drive restore of ordinary app data (was: only R-7b share definitions, 2026-07-18)	PROVEN (A1)
Tier-2 „existing files are not modified or deleted" promise	PROVEN (A1, both halves)
Tier-2 restore after total loss of an app data directory	PROVEN (A3)
Restored data usable by the app rather than present on disk	PROVEN (A1/A3, via paperless's own API)
Tier-1 recovery-unit restore is a distinct path	ESTABLISHED; path exercised end-to-end (A2)
Tier-1 content recovery after real loss	NOT PROVEN — A2 ran against an intact app; A3 used Tier-2

---

5. Phase B — do the six recent fixes interact?

Setup, and what is therefore an ARTIFACT

knob	real	campaign	box
local (vzdump) cadence	0 ⇒ 24 h	900 s	both
PBS tier cadence	604800 s (7 d)	900 s	both
restore-test cadence	302400 s (84 h)	120 s (demo-hp) / 3600 s (demo-felhom)	both
backup_window_start	02:30 / unset	14:00 ⇒ gate [16:00, 20:00)	both
PBS reachability	up	iptables -j REJECT on 10.77.0.1:8007	both

Everything about FREQUENCY below is an artifact of that compression. What is not an artifact is the ORDER of events, which mechanism fired, and what the operator was told.

B3 — both tiers due, one succeeds, one fails — COHERENT

This arose on the very first cycle after the gate opened (B3-mixed-outcome-cycle1.txt):

15:42:58 [quiesce] backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx]
15:43:05 [quiesce] tier local: backup job …006689159 started — polling
15:44:25 [quiesce] tier local: backup job …006689159 done — next tier may start (app still quiesced)
15:44:25 [quiesce] tier felhom-pbs: backup job …020207536 started — polling
15:44:35 [quiesce] unquiescing (backup failed): restarting 1 stack(s)
15:44:50 [quiesce] tier felhom-pbs: backup job …020207536 failed
15:44:50 [quiesce] tier felhom-pbs has now failed 1 time(s) in a row — deferring its next quiesce
                   by 15m0s (cap 4h0m0s) so the apps are not stopped again for a backup that cannot succeed
15:44:50 Event pushed: whole_guest_backup_failed (error) — … (next attempt in 15m0s)

• R-82's one-quiesce rule holds under mixed outcomes: two due tiers, apps stopped once, both tiers attempted inside the single window.
• The breaker is per-target as designed: local succeeded and was untouched; only felhom-pbs armed. A broken offsite tier did not suppress a healthy local one.
• Exactly ONE event, edge-triggered at the first failure, naming the tier, the reason and the retry interval. The hub received exactly one row (id 1915) — no duplicates.
• App downtime was ~97 s and paperless came back healthy.

Operator picture: coherent. One event, one cause, one next step.

F-CRIT-2's size filter, confirmed live against REAL damage — not manufactured

Unprompted, before any Phase B injection, the demo-hp agent logged:

17:37:59 WARN backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a
successful backup  target=felhom-pbs vmid=9201 volid=felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z
size_bytes=1 reason="size 1 B is below the 1048576 B plausibility floor"

Independently confirmed on ep0's filesystem — the snapshot is a genuine aborted archive:

/mnt/pbs-datastore/ns/demo-hp/ct/9201/2026-07-28T05:31:14Z/
  catalog.pcat1.tmp_didx   0 bytes
  root.pxar.tmp_didx       0 bytes        <- no manifest, no index.json.blob
  pct.conf.blob          425 bytes

This is F-CRIT-2 doing exactly its job on a real leftover phantom, in production, without being asked. PROVEN, not manufactured.

Incidental proof: the local whole-guest archive restores and boots

The restore-test that ran at 17:47:32 selected the local vzdump written by the B3 cycle minutes earlier, restored it into scratch guest 990000, booted it, and tore it down:

17:49:01 restore-test: scratch guest torn down  vmid=990000
17:49:01 backup: scheduled restore-test passed  archive=local:backup/vzdump-lxc-9201-2026_07_28-17_43_05.tar.zst duration_s=89.19

pct list afterwards shows no 990000 guest — F-LEAK's fenced destroy is holding.

B1 — contention, then the breaker — COHERENT; the contention path does NOT swallow the transition

Forced by tightening the restore-test cadence to 120 s against its ~89 s duration, so the R-85 gate is held most of the time. Both sides of the seam were captured in the same second:

Agent (agent-demo-hp.log):

18:03:09 local-api: backup refused — a heavy operation is already in flight  vmid=9201 requested_target=local      busy=restore-test
18:03:09 local-api: backup refused — a heavy operation is already in flight  vmid=9201 requested_target=felhom-pbs  busy=restore-test

Controller (ctrl-demo-hp.log):

16:02:58 [quiesce] backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx]
16:03:09 [quiesce] tier local is BUSY — the agent refused the backup because a concurrent heavy
                   operation holds it. This is contention, NOT a failure: the tier stays due and retries in 15m0s
16:03:09 [quiesce] tier felhom-pbs is BUSY — … (contended for 0s)
16:03:09 [quiesce] unquiescing (last tier is busy — deferring to a later cycle): restarting 1 stack(s)

• No breaker armed and NO event pushed for the contention — F-A1's whole point, confirmed.
• The app was restarted immediately rather than left down for the restore-test's duration — the second trap F-A1's comment names ("app thrash") is avoided.
• Note felhom-pbs was contended in a cycle where it would certainly have failed (its path is blocked). The 409 precedes the attempt, so contention wins that cycle and the tier stays due — correct, and it does not reset the breaker: noteTierContention never touches breaker state, and the earlier armed failure count survives.

Operator picture: coherent, and correctly silent. Contention produced no operator noise at all, while the genuine failure minutes earlier produced exactly one event.

The breaker's retry discipline, over four polls

15:44:50 tier felhom-pbs has now failed 1 time(s) in a row — deferring … by 15m0s (cap 4h0m0s)
15:47:59 [DEBUG] tier felhom-pbs is in backoff after 1 consecutive failure(s) — not quiescing until 2026-07-28T15:59:50Z
15:52:58 [DEBUG] … not quiescing until 2026-07-28T15:59:50Z
15:58:00 [DEBUG] … not quiescing until 2026-07-28T15:59:50Z

Three skipped polls, one WARN when the breaker armed, DEBUG for each skip, one event. The apps were not stopped once during the backoff. This is the exact behaviour R-88 was built for, and the pre-R-88 bug (a stop/start cycle every 5 minutes forever) did not recur.

Both boxes independently, same result

demo-felhom armed its own breaker at 16:00:00 with its own single event (whole_guest_backup_failed, hub id 1916), against demo-hp's id 1915. Per-box and per-target isolation both hold.

B4 / B5 — and what they turned up

The demo-felhom rig (docmost made unstartable, PBS blocked, cadences compressed) produced a finding rather than a clean pass — see F-2 below. R-100's success anchor was confirmed correct along the way: after the induced offsite failure on demo-hp, last_run advanced to the failing run while last_success stayed at the last genuine success:

"last_run":     "2026-07-28T16:01:56Z",
"last_status":  "error",
"last_success": "2026-07-28T16:00:18Z",

That is precisely R-100's contract — a failure no longer refreshes the staleness clock.

Not reached

B2's combined form (a phantom created during a quiesce that ALSO fails to restart an app) and B4's full form (a stale anchor while the breaker is backed off, observed as one operator picture) were not reached. Both need a longer run than this session had.

---

6. Findings

F-2 — an app in a crash loop never alarms, on any channel — DEFECT, HIGH

This is F-CRIT-1's exact shape in a state F-CRIT-1's fix does not cover. Campaign 8 closed "an app that fails to restart after a quiesce never alarms" for StateStopped. A crash-looping app is not StateStopped, and nothing sees it.

Action. On demo-felhom, docmost/docmost:0.95.0 was retagged to an image that exits immediately, and the stack was started. (The first attempt — editing the compose image tag — was reverted by the catalog sync within 3 minutes, which is how O-1 below was found.)

Observation. docmost crash-looped continuously. Sampled every 10 s through the controller's own /api/stacks — i.e. exactly the value the alarm logic reads, not docker ps:

16:05:53 docmost_state=restarting restartcount=14
16:06:24 docmost_state=restarting restartcount=15
16:07:04 docmost_state=restarting restartcount=15
16:08:05 docmost_state=restarting restartcount=16
16:08:25 docmost_state=restarting restartcount=16     … unbroken for the whole observation

The restart policy makes this permanent, not transient:

docker inspect docmost -> unless-stopped maxretry=0 restartcount=16

And across that entire period:

dashboard page mentions of "docmost":                 0
dashboard warning words (Hiba|Figyel|nem fut):        0
app_start_failed events pushed:                       0
hub events for demo-felhom:                           only whole_guest_backup_failed (the tier, not the app)

The decisive evidence is the detector's own heartbeat. F-OBS (v0.180.0) exists so that a stalled dead-app check cannot hide behind silence — it prints a POSITIVE observable every ~10 minutes. Here it proves the opposite of what it was built to reassure about: the detector is alive, running, and blind.

15:59:17 [deadapp] check alive: 160 scans since boot, 4 deployed app(s) evaluated, 0 currently down
16:09:17 [deadapp] check alive: 180 scans since boot, 4 deployed app(s) evaluated, 0 currently down
                                                                                 ^^^^^^^^^^^^^^^^^
docmost at 16:09:17:  status=restarting  restartcount=18  policy=unless-stopped

Twenty scans ran between those two lines. Every one of them looked at a permanently crash-looping app and counted 0 down. This is not an absent signal being read as health — it is the health signal itself, positively asserting that nothing is wrong.

The causal chain, pinned to the code.

1. aggregateState (internal/stacks/manager.go:753-756) — "Any restarting → stack is restarting". This test runs before the partial/degraded logic, so one crash-looping container plus two healthy ones yields StateRestarting, never StateDegraded.
2. IsDownState (internal/stacks/manager.go:54-56) is StateStopped || StateExited || StateDegraded — StateRestarting is absent.
3. classifyRunStates (cmd/controller/main.go:1292-1294) computes down := stacks.IsDownState(st.State) && !userStopped && !quiesced[…] → false.

So the app never enters dead (no dashboard banner) and never enters states as Down (no app_start_failed, hence no customer email and no hub event). There is no crash-loop detector anywhere in the controller — RestartCount is read in exactly one unrelated place (internal/web/intermediary.go:105,343), and StateRestarting appears in no alarm path.

Why this is not covered by the existing suppression logic. R-97b's grace window is irrelevant here: it expired at 16:01:56 and the app was still silent minutes later. The failure is not a suppression that fails to expire — it is a state that is never considered "down" in the first place.

Expectation. A container that Docker is restarting indefinitely is a dead app. Either StateRestarting belongs in the down set (with a debounce so a normal restart does not alarm), or the restart count must be watched. As written, docker restart: unless-stopped — the catalog's standard policy — converts every crash into permanent silence.

Severity HIGH. It is the same class the project has now shipped twice (F-CRIT-1, and the classifyRunStates I1 comment in the workspace CLAUDE.md table), it affects the most common real-world app failure mode, and it is indefinitely silent rather than delayed.

DEFECT. The trigger (a retagged image) was manufactured; crash loops are not — a bad image, a failed migration, a corrupt config or a repeatedly OOM-killed process all produce exactly this state, and no cadence was compressed to make it happen.

---

O-1 — recovery-unit capture does not validate, but the catalog sync bounds the damage — OBSERVATION, not a defect

This was very nearly filed as a MEDIUM defect. The live system corrected me, and the correction is the more useful result.

What was observed first. The docmost compose was edited to an unpullable image tag; the controller's periodic refresh captured the recovery unit 7 minutes later with no validation, so the live app and its only restore point disagreed — and the restore point was the broken one:

15:50:07 [INFO] [backup] Recovery unit captured for docmost → …/backups/primary/docmost (images=3, …)
live:      docmost|docmost/docmost:0.95.0|Up 4 hours (healthy)
manifest:  ['docmost/docmost:0.95.0-campaign9-nonexistent', …]

That looked serious, because ListRestorePoints returns exactly one keep-side restore point per app (restore_points.go:14-18, confirmed live) — an unvalidated refresh replaces the only one there is. CaptureRecoveryUnit (recovery_unit.go:78-130) does indeed gate its write on a config checksum change alone: no run-state check, no health check, no test that the pinned images resolve.

What actually happened next. Three minutes later the catalog git-sync reverted the drift on its own, and the recovery unit followed within the same pass:

15:53:17 [INFO] [sync] Updated docmost/docker-compose.yml
15:53:17 [INFO] [sync] Periodic sync: Sablonok frissítve — frissítve: docmost
15:53:17 [INFO] [backup] Recovery unit captured for docmost → …/backups/primary/docmost (images=3, …)

Verified afterwards on all three surfaces:

live compose image:            image: docmost/docmost:0.95.0
Tier-1 recovery unit pins:     ['docmost/docmost:0.95.0', 'postgres:16-alpine', 'redis:7-alpine']
Tier-2 copy's unit pins:       ['docmost/docmost:0.95.0', 'postgres:16-alpine', 'redis:7-alpine']

Conclusion. Compose drift is self-healing and the restore point follows it, bounded by the sync interval — the poisoned window here was ~3 minutes. The system does not leave you in the state I manufactured, so reporting this as a defect would have been reporting an artifact. The absence of validation in CaptureRecoveryUnit is real, but it is compensated, and the compensating control was proven live rather than assumed.

The one residue worth an operator's attention — narrow, and NOT demonstrated here: the sync makes the catalog the source of truth, so a bad image tag in the catalog would propagate to every box's only restore point unhealed, because sync would be restoring it rather than reverting it. That is a catalog-quality concern owned by the catalog gates, not a controller defect.

F-3 — an interrupted offsite run leaves a lock the self-heal cannot reach, reported as "unknown" — DEFECT, MEDIUM

This answers Phase C item 8 ("is the repo usable after a killed run, or does it need an unlock?"). It was found by accident — the campaign severed its own in-flight run — which makes it a genuine interruption rather than a constructed one.

Sequence. An offsite run started at 15:58:12 and took the repo's exclusive lock at 15:59:00. At 15:59:33 its transport was severed. The run died holding the lock. Every later run then failed:

16:15:48 [offbox] pre-push dump leg completed in 20.056s — snapshot pair is coherent
16:15:50 [ERROR] [offbox] backup failed after 23s: offbox repo unreachable: exit status 1:
         unable to create lock in backend: repository is already locked exclusively by PID 14328 on demo-hp by root
         lock was created at 2026-07-28 15:59:00 (16m49s ago)
         the `unlock` command can be used to remove stale locks

The remedy exists in the code and was never reached. resticStep (internal/backup/offbox.go:634-648) detects repository is already locked, escalates to unlock --remove-all, and retries once — its comment states plainly that this is required because "the recreated container has a new hostname, so restic can't verify the dead PID and won't treat the lock as stale for ~30 min (the overnight-campaign C2 finding)".

But the failure happens before any resticStep call. ensureOffboxRepo (offbox.go:676-696) probes the repo first; that probe hits the lock, classifyResticProbe (offbox.go:77-93) has no case for a lock error so it returns "other", and the default branch returns offbox repo unreachable: … — a fail-fast with no unlock escalation. The self-heal sits behind a gate the failure never opens.

And the operator is told the wrong thing. ClassifyOffsiteFailure likewise has no lock case, so it falls through to OffsiteFailUnknown. Hub event 1919:

"A távoli mentés ismeretlen okból nem sikerült (23s): offbox repo unreachable: … repository is
already locked exclusively by PID 14328 … the `unlock` command can be used to remove stale locks"

"The remote backup failed for an unknown reason" — for a condition that is precisely known, named in the attached detail, and self-healable by code already in the binary.

Impact. The offsite tier stays dead until a human intervenes. Verified: it took a manual restic unlock --remove-all to restore it (evidence/phaseC/C8-unlock.txt).

Reachability is ordinary, not exotic. Any interruption of an in-flight run leaves this lock — a controller container restart, an OOM, a network drop, or a host reboot mid-backup, which is Phase C item 6's scenario. The code's own comment describes exactly this ("the C2 crash lock").

Severity MEDIUM, argued rather than asserted. Against HIGH: an error event is raised immediately on the operator channel, its detail literally contains the remedy, and R-100's 48 h offsite_stale is a second backstop, so this is loud-but-mislabelled rather than silent — unlike F-1 and F-2. Against LOW: a backup tier is down until manual intervention, and the headline actively misdirects triage.

DEFECT. No cadence compression was involved in the mechanism; only the interruption was deliberate.

---

O-2 — a hand-run docker compose up -d starts a Felhom app WITHOUT its secrets — OPERATIONAL, worth a runbook line

Encountered during recovery, by me. Restoring docmost with a plain cd /opt/docker/stacks/docmost && docker compose up -d produced a container that crash-looped on:

{"minLength":"APP_SECRET must be longer than or equal to 32 characters","isNotEmpty":"APP_SECRET should not be empty"}

Secrets are injected by the controller's stackEnv at start time (StartStack, internal/stacks/manager.go:912-948); they are not in a .env file beside the compose. A manual up -d therefore recreates the container with empty secrets — turning a healthy app into a broken one, and looking exactly like the fault you were trying to fix. Recovery must go through the controller (POST /api/stacks/<name>/restart), which is what finally fixed it.

---

7. Phase C / D / E — reached, and not reached

#	item	status
C6	host reboot mid-backup	NOT REACHED. Approved by the operator; not attempted. Its most likely consequence was nonetheless observed by another route — see F-3, which is exactly the state a reboot mid-backup would leave.
C7	three-way concurrency (backup + restore-test + GC)	PARTIALLY. Backup-vs-restore-test contention was proven (B1); GC was not added as a third party.
C8	restic transport interrupted mid-run — is the repo usable after?	ANSWERED: no, it needs an unlock. See F-3.
D9	Scenario C — never-succeeded Tier-2 on a throwaway app	NOT REACHED. No throwaway app was created.
D10	F-DIAG's failure classes + no credential/repo-URL/raw-error passthrough	PROVEN for the surfaces that leave the box — see below.
D11	R-100's threshold — see offsite_stale actually fire	NOT REACHED, and the approved method turned out not to exist — see below.
E12	F-HUB SQLITE_BUSY reproduction	NOT REACHED (explicitly lowest priority).

D10 — the sanitiser, proven on the exact case that defeated its first version

Six distinct causes map to six distinct Hungarian messages (offbox.go:201-208). The transport case was induced live by blocking port 23 from inside the guest, and the resulting hub event's details_json was:

"A távoli tárhely nem érhető el (hálózat vagy bejelentkezés) (51s): offbox repo unreachable:
 exit status 1: subprocess ssh: ssh: connect to host <repo> port 23: Network is unreachable
 Fatal: unable to open repository at <repo> unable to start the sftp session, error: EOF"

• the correct distinct class, not a generic string;
• the bare hostname — the precise shape the sanitiser's first version leaked on, per its own comment (offbox.go:170-181) — is replaced with <repo>;
• the full sftp:user@host:/path repo URL is replaced with <repo>.

Nothing raw leaves the box. The hub report carries no error text at all: OffboxReportStatus (offbox.go:1149-1174) has no error field by construction.

One precise caveat, not a finding. The last_error persisted in the guest's settings.json is raw, and the customer's own /backups/remote page renders it verbatim under „Utolsó hiba:", including the bare hostname and the sftp: URL. That is local-only and discloses nothing to that viewer: the same host, user and repo path are displayed two rows above as configuration on the same page. Sanitisation is applied on the paths that leave the box, which is where the contract puts it.

D11 — the approved compression does not exist as a knob

Phase 0 approved compressing the hub's staleAfter for this item. It is not configurable: cmd/hub/main.go:552 calls monitor.NewOffsiteChecker(dataStore, 0, …) and 0 selects the compile-time defaultOffsiteStaleAfter = 48 * time.Hour (internal/monitor/offsite.go:41,60-62). Compressing it would have required editing, building and deploying hub code — which this campaign is forbidden to do ("no production code ships", "no inline fixes"). Reported rather than worked around, per the standing rule that a recommendation not followed gets a line saying why.

The alternative that needs no code change — ageing the controller's reported last_success anchor past 48 h and letting the hub judge it at its real threshold — was identified but not executed. That is the recommended method for a future run.

Along the way R-100's producer half was confirmed correct twice, at real settings: after each induced failure last_run advanced while last_success held.

"last_run":     "2026-07-28T16:15:50Z",
"last_status":  "error",
"last_success": "2026-07-28T16:00:18Z",

---

8. Phase F — recovery

Full log: evidence/PHASE-F-recovery.log. Every revert was verified with a positive observable.

reverted	verification
agent cadences, both hosts	cadences restored -> 0 302400 [604800]; systemctl is-active = active
backup windows, both boxes	set back to 02:30 through the same endpoint that changed them
PBS path block, both hosts	pvesm status shows felhom-pbs … active on both
offsite transport block (guest)	ssh reaches the storage box again (host key exchanged)
docmost broken image	real 0.95.0 re-pulled; restarted via the controller after O-2
stale restic lock	unlock --remove-all; list locks empty; 8 snapshots readable
campaign scratch files	0 remaining c9-* on both guests; host /tmp cleared
collectors	all 7 stopped; 0 remaining remote tails

The offsite tier was not merely unblocked but proven working again — a full run through the real endpoint after the unlock:

"last_run":     "2026-07-28T16:20:57Z",
"last_status":  "ok",
"last_success": "2026-07-28T16:20:57Z",
"last_duration": "1m35s",
"snapshot_count": 8

Final fleet state — both boxes fully healthy:

demo-felhom : docmost + docmost-postgres + docmost-redis, immich (×4), calibre-web,
              bookstack (×2), filebrowser, samba, cloudflared, traefik   — ALL healthy
demo-hp     : paperless (×3), filebrowser, samba, cloudflared, traefik   — ALL healthy
agents      : active on both hosts
iptables    : 0 campaign9 rules on either host and in either guest
scratch VMs : none in the 990000 band on either host
ep0         : /srv/pbs-felhom rollback copy intact (13G); datastore 14% used
peti-felhom : never touched

One benign residue, stated for completeness. The in-memory R-88 breaker on each box still holds a felhom-pbs consecutive-failure count from the induced failures. Its until timestamps are long past, so it blocks nothing; the only effect is that a next failure would back off 30 m instead of 15 m. It clears on the next successful whole-guest backup, and by design it also clears on any controller restart (breaker.go — deliberately not persisted). Nothing was done about it because clearing it would have meant taking another app outage for no benefit.

---

9. Where this stopped, and why

Stopped at the end of Phase B, with Phase D item 10 additionally completed, then Phase F.

Phase A — the campaign's stated headline and its whole justification — is complete and proven. Phase B produced two of its five combinations cleanly, plus a HIGH finding. Phase C item 8 was answered by an accident that turned out to be a genuine defect. Phase D item 10 is proven; D11's approved method did not exist.

Phase C item 6 (host reboot mid-backup) was deliberately not started. It is a large new fault class against two boxes that are remote until ~08-02, and beginning it would have meant either rushing it or leaving the fleet in an unknown state. The campaign's own rule — stop cleanly at a phase boundary; a broken test node is fine, an unreported one is not — made the choice: finish the recovery properly rather than start a fault I could not supervise to completion. F-3 partly covers its most likely consequence anyway, since an interrupted run is exactly what a reboot mid-backup produces.

---

10. Findings, ranked

#	finding	severity	class
F-1	the Tier-2 restore button reports success for apps it can never restore (BookStack, Docmost — half the fleet's Tier-2 apps); takes a real outage; message is indistinguishable from a genuine "nothing was missing"	HIGH	DEFECT
F-2	an app in a crash loop never alarms on any channel — StateRestarting is in no down-set, and the dead-app heartbeat positively reports "0 currently down" while it loops	HIGH	DEFECT
F-3	an interrupted offsite run leaves an exclusive lock the existing self-heal cannot reach; tier dead until manual unlock; reported to the operator as "unknown reason"	MEDIUM	DEFECT
O-1	recovery-unit capture validates nothing, but the catalog sync reverts drift within ~3 min and the unit follows	—	OBSERVATION (nearly mis-filed as a defect)
O-2	a hand-run docker compose up -d starts a Felhom app without its secrets	—	OPERATIONAL

Mechanisms confirmed working, on live hardware: R-82's one-quiesce rule under mixed outcomes; R-88's breaker (per-target, edge-triggered, one event, no app thrash); F-A1's contention deferral (no breaker, no event, prompt restart); F-CRIT-2's size filter (against a real 1-byte phantom, not a manufactured one); R-100's success anchor (twice); F-DIAG's sanitiser (on the exact bare-hostname case that defeated its first version); F-OBS's positive observable (which is what made F-2 provable); F-LEAK's fenced destroy (no leaked scratch guests across ~10 restore-tests).

---

11. What came off the never-proven list — and the honest residue

Off the list:

• Tier-2 cross-drive restore of ordinary app data — the customer's actual button (A1)
• Tier-2's "existing files are NOT modified or deleted" promise, both halves (A1)
• Tier-2 restore after total loss of an app's data directory (A3)
• Restored data usable by the app, proven through the app's own API (A1/A3)
• Tier-1 recovery-unit restore is a distinct path, and it executes end-to-end (A2)
• The six recent fixes do compose: R-82 + R-88 + F-A1 + F-CRIT-2 + R-100 produced one coherent operator story in every combination reached
• Whether an interrupted restic run leaves a usable repo (C8 — it does not)

Still not proven — the residue:

• Tier-1 content recovery after real loss. A2 ran against an intact app; A3 used Tier-2. Nobody has yet destroyed an app's data and recovered it from the recovery unit. This is now the single most valuable unproven item.
• Host reboot mid-backup (C6) — untested, and F-REBOOT proved the guest-reboot sibling was real.
• Three-way concurrency with GC as the third party (C7).
• Scenario C — a never-succeeded Tier-2, live (D9).
• offsite_stale actually firing (D11) — never observed. Needs the anchor-ageing method above, not the staleAfter compression that was approved but does not exist.
• F-HUB SQLITE_BUSY characterisation (E12).
• Whether F-1 and F-2 have siblings: both are "a surface reports health it did not establish". Nothing in this campaign searched systematically for others.