felhom.eu/documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md

# CAMPAIGN 9 — proof, not fixes: the restore paths and the interactions

**Adversarial, destructive, unattended. 2026-07-28 17:12 CEST → in progress.**

Nodes exercised: `demo-felhom` (+ guest 9201), `demo-hp` (+ guest 9201), `ep0`
(`felhom-hetzner`). Controllers on **0.182.0** both boxes; agent 0.110.0; hub 0.80.0.

**`peti-felhom` was never touched** — not its data, not its namespace, not `u629488-sub2`. The ep0
rollback copy `/srv/pbs-felhom` was never deleted.

Evidence: **`DooPlex:~/campaign9/evidence/`**, written continuously by 7 collectors (6 log streams +
a 2-minute state snapshot) started **before** the first mutation and verified emitting live lines
before Phase A began.

---

## Headline

**The two never-proven restore paths are now proven, on live hardware, through the real endpoints
the UI posts to.** A customer who deletes files — or their entire app data directory — gets
everything back, byte-identical, and the app works afterwards.

**One HIGH-severity defect fell out of proving it: F-1.** For apps whose data lives entirely in
Docker named volumes, the „Fájlok visszaállítása" button is rendered, stops the app, restores
nothing, and reports **„Nincs hiányzó fájl — minden fájl megvan a helyén."** — a message that
positively asserts the customer's files are all present. It says this on every click, on any day,
no matter how much data was lost, while 156 MB of that app's data sits unread in the same Tier-2
copy the button claims to restore from.

---

## 1. Phase 0 — questions, answers, and skips

Asked in one batch, before touching anything. **All four answered; nothing was skipped for lack of
an answer.**

| # | Question | Answer received |
|---|---|---|
| 1 | Which targets may I destroy data on? | **Throwaway apps AND existing demo apps on BOTH boxes.** |
| 2 | May the hub's `staleAfter` be compressed for Phase D item 11? | **Yes — compress, then revert.** |
| 3 | How may ep0 be taken down? | **Power-cycle allowed programmatically** (widened from Campaign 8's "no power-cycle"). |
| 4 | `drill-r50` destroy/rebuild? Host reboot for Fault 12? | **drill-r50 free to destroy; host reboot OK on BOTH demo hosts.** |

---

## 2. Phase A — the restore proofs (the headline)

### A1 — Tier-2 cross-drive restore of ordinary app data — **PROVEN**

**Target.** `paperless-ngx` on **demo-hp** guest 9201 — chosen because it is the only app in the
fleet that combines real user content (16 documents), a complete Tier-2 copy with a real user-file
leg, and an R-101 success anchor. Live root `/mnt/felhom-drives/nvme-1tb`; Tier-2 copy
`/mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx`.

**Method.** The real endpoint the UI posts to — `POST /backup/tier2/restore` with `stack_name` —
driven **through traefik over https with the vhost Host header**, from inside the guest, with a real
session cookie and a real CSRF token scraped from the rendered `/backups/apps` page. No server logic
was skipped; only browser rendering.

**The rendered dialog** (`evidence/phaseA/A1-dialog-rendered.txt`) — this is the R-101 branch, live:

```
Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM
törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:43.
```

**Baseline.** All 43 files in the copy were byte-identical to live before anything was touched
(`A1-copy-inv-BEFORE.txt` vs `A1-live-inv-BEFORE.txt`, `diff` empty).

**The mutation exercised all three of the dialog's promises in one run** — because the dialog makes
three distinct claims and proving them separately would have been three outages:

| class | files | promise under test |
|---|---|---|
| DELETED | 6 (3 originals, 1 archive, 2 thumbnails, spanning all subdirs, incl. one complete document) | must come back byte-identical |
| CREATED | 2 (one in an existing dir, one in a **new** dir) | „NEM törlődnek" — must survive |
| MODIFIED | 1 thumbnail, appended marker | „NEM módosulnak" — must not be overwritten |

**Result — all three hold** (`A1-verify.txt`):

```
=== PROMISE 1: deleted files came back BYTE-IDENTICAL ===
originals/0000002.pdf: OK
originals/0000005.pdf: OK
originals/0000012.pdf: OK
archive/0000012.pdf: OK
thumbnails/0000012.webp: OK
thumbnails/0000003.webp: OK

=== PROMISE 2: created files were NOT deleted ===
PRESENT  originals/c9-created-file.txt  be279d439b852e78c6d10379c4f569360283d4ce77523f4848723beb5b78997e
PRESENT  c9-newdir/c9-keep.txt          4fa680f76eeac33369fefec88d1a216cea7d16255573e439302d40c63fa9d6a9

=== PROMISE 3: locally-modified file was NOT overwritten ===
pre-edit  fd9b92dd77df6bd9e6474d238ec86e029e7f562c820ed1172d71ad6cb9eccb42  <- must NOT match
post-edit 15c61ab7ddd1bfcfecc7ac7f43d12357de123b42d3ab127e3a23a46af4bd4fd2  <- must match
actual    15c61ab7ddd1bfcfecc7ac7f43d12357de123b42d3ab127e3a23a46af4bd4fd2
marker still present: CAMPAIGN9-LOCAL-EDIT-MUST-NOT-BE-OVERWRITTEN
live file count: 45     (43 baseline + 2 created — arithmetic closes)
```

**Stop/start and health**, from the controller (`A1-controller-log.txt`):

```
15:21:59 [WARN] [web] Tier-2 file restore requested (async): stack=paperless-ngx
15:21:59 [INFO] [backup] Tier-2 file restore for paperless-ngx: …/secondary/paperless-ngx (v2) → /mnt/felhom-drives/nvme-1tb (additive-only)
15:22:20 … Post-restore health check: paperless-ngx not yet running, waiting…   (×5)
15:22:45 [DEBUG] [backup] Post-restore health check: paperless-ngx is running
15:22:45 [INFO] [backup] Tier-2 file restore completed for paperless-ngx: 6 file(s) restored (39s)
```

Container uptime confirmed the real stop/start independently (`Up 5 seconds (health: starting)` →
`Up 27 seconds (healthy)` polled at 10-second intervals).

**The restored data is usable BY THE APP, not merely present on disk** (`A1-usable.txt`) — two
independent checks:

```
=== CHECK 1: paperless resolves each restored doc and its content matches the DB checksum ===
doc 2   invoice-114750664          original_exists=True checksum_match=True thumb_exists=True
doc 5   Project Overtime 2023.11   original_exists=True checksum_match=True thumb_exists=True
doc 12  Hotel                      original_exists=True checksum_match=True archive_exists=True archive_match=True thumb_exists=True

=== CHECK 2: served over paperless's own HTTP API, hashing what the app returns ===
doc 2  http=200  served_sha256=87226ff2…13244f   == recorded before deletion
doc 5  http=200  served_sha256=1395403a…f2f2df   == recorded before deletion
doc 12 http=200  served_sha256=1dd4cd16…dd924b   == recorded before deletion (archive; /download/ serves the archive when one exists)
```

**A1 verdict: PROVEN.** Byte-identical recovery, both non-destruction promises kept, app healthy
and serving the restored bytes over its own API.

### A2 — Tier-1 local recovery-unit restore — **DISTINCT, and exercised**

**It is a distinct path**, not subsumed: `POST /backup/restore` (`stack_name` + `snapshot_id`) →
`RestoreFromRecoveryUnit` (`internal/web/handlers.go:1230`, `internal/backup/restore_unit.go:96`).
It reads the app's on-drive recovery unit, recovers secrets from the guest's live `app.yaml`,
restores named-volume data from the unit's tars, and redeploys. Tier-2 copies are **not** reachable
through it.

Exercised on `calibre-web` (demo-felhom), `A2-restore-status.txt` / `A2-log.txt`:

```
15:28:34 [INFO] [backup] Restoring calibre-web from recovery unit: images=1, secrets recovered=0/0, data_keys=0
15:28:39 [INFO] [backup] Restoring Docker volume calibre-web_calibre_web_config for calibre-web
15:28:39 [INFO] [backup] Restored 1 Docker volume(s) for calibre-web
15:28:52 [INFO] [backup] Restore-from-unit completed: calibre-web        (18.25s)
```

App healthy 29 s later; volume fingerprint unchanged (`6b10ec90…`), consistent with restoring a tar
that matched current content.

**Honest limit:** this run proves the *path executes and the app survives it*. It does **not** prove
content recovery, because nothing was lost first. The content-recovery proof is A3.

**One thing checked and deliberately NOT reported as a finding.** `snapshot_id` is required by the
handler and then never passed to `RestoreFromRecoveryUnit`, while the success flash names it —
which looks like "you picked A and got B". It is neither: `restore_points.go:14-18` states there is
exactly one keep-side restore point and that `snapshot_id` is logging-only, and the live API
confirms exactly one point per app with `short_id: "helyi"` — the value the flash prints. Correct by
design, pinned by its own comment. **ARTIFACT of reading, not a defect.**

### A3 — restore after REAL loss — **PROVEN**

The harder shape: not gap-fill, but deleting the app's data directory **entirely**, while the app
ran, as an "I deleted my files" event actually looks.

```
=== BEFORE: paperless serves doc 12 ===      source_exists True
=== DESTROYING /mnt/felhom-drives/nvme-1tb/appdata/paperless ===
exists after rm: NO
=== AFTER ===                                source_exists False
                                             doc12_download_http=404
=== doc count in DB is untouched ===         16
```

The loss was verified with a **positive observable** — the document download going 200 → **404** —
not merely by the directory being absent.

**Tier used: Tier-2**, via the same customer button. Result
(`A3-restore-status.txt`, `A3-verify.txt`):

```
message: "paperless-ngx: 43 fájl visszaállítva a másodlagos másolatból."   (46 s)

=== 1. every restored file byte-identical to the Tier-2 copy? ===
copy files: 43 / live files: 43
IDENTICAL — every file in the copy is back, byte-for-byte, and nothing extra

=== 2. does PAPERLESS work? ===
documents_ok 16 of 16
problems []

=== 3. serve doc 12 over HTTP again (the observable that was 404 after the loss) ===
doc 2  http=200 sha256=87226ff2…13244f
doc 5  http=200 sha256=1395403a…f2f2df
doc 12 http=200 sha256=1dd4cd16…dd924b
```

**A3 verdict: PROVEN — the customer gets their app back, not "only some files".**

**The honest residue, stated plainly.** Three things did not come back, and none of them is a
defect — each is the correct consequence of not being in the last successful copy:

- the two files created *after* the last Tier-2 run — never backed up, correctly gone;
- the local edit to `thumbnails/0000001.webp` — the file returned in its **pristine** backed-up
  form (`fd9b92dd…`), i.e. the post-loss restore legitimately reverted an unbacked-up edit. This is
  the same file that A1 proved is *never overwritten* while it still exists; the two results are
  consistent and together delimit the guarantee exactly: **existing files are untouched; destroyed
  files return at their last-backup state.**
- `appdata/paperless/export/` — absent from the copy, recreated empty by paperless on startup.

---

## 3. Findings

> **SHIPPED 2026-07-28 — controller v0.183.0.** F-1's honesty half and F-2 in full. Phase 0 of the fix
> sized F-1 properly: **43 of the 53 catalog apps** have no restorable subtree (not 2), 9 are covered
> only for their file legs, 1 is stateless. The restore now refuses **before** stopping the app and
> names the action that works; a run that proceeds claims only what it **examined** and discloses that
> the database and volumes are not covered. F-2 alarms after a 5-minute sustained-restarting threshold
> — proven live: silent through ten 30 s samples, then `app_start_failed` at 5m25s, and the F-OBS
> heartbeat now reads `1 currently down` where Campaign 9 recorded `0`. Completeness is filed as
> **C9-F1b**, and **C9-F4** was filed during the fix: nothing reads the Tier-2 copy's `recovery-unit/`
> mirror at all. See `REPORT.md` and CHANGELOG v0.183.0.


### F-1 — the restore button reports success for apps it can never restore — **DEFECT, HIGH**

**Applies to:** every app whose data lives entirely in Docker named volumes and which therefore has
no bind-mounted user-file leg. On demo-felhom that is **BookStack and Docmost** — 2 of the 4 apps
with a Tier-2 copy, i.e. half the fleet's Tier-2-configured apps.

**Action.** Clicked „Fájlok visszaállítása" for `bookstack` on demo-felhom via the real endpoint.

**Observation** (`A1b-bookstack-post.txt`, `A1b-bookstack-log.txt`):

```
15:26:05 [WARN] [web] Tier-2 file restore requested (async): stack=bookstack
15:26:05 [INFO] [backup] Tier-2 file restore for bookstack: …/secondary/bookstack (v2) → /mnt/sys_drive/felhom-data (additive-only)
15:26:28 [INFO] [backup] Tier-2 file restore completed for bookstack: 0 file(s) restored (19s)

restore-status: {"ok":true, … "message":"Nincs hiányzó fájl — minden fájl megvan a helyén."}

BEFORE: bookstack|Up 4 hours (healthy)
AFTER:  bookstack|Up 25 seconds (healthy)        <- a real outage was taken
```

And the copy the button claims to restore from is **not empty** — it holds the app's entire data:

```
16K   …/secondary/bookstack/recovery-unit/compose
60K   …/secondary/bookstack/recovery-unit/db-dumps
156M  …/secondary/bookstack/recovery-unit/volume-dumps      <- BookStack's actual data
4.0K  …/secondary/bookstack/recovery-unit/manifest.json

ls …/secondary/bookstack/  ->  recovery-unit          (no hdd/, no userdata/)
```

**Why it can never succeed.** `RestoreTier2Files` merges exactly two subtrees —
`destBase/hdd` → `liveNsRoot` and `destBase/userdata` → `liveNsRoot/userdata`
(`internal/backup/tier2_restore.go:101-104`) — and `os.Stat`-skips each when absent
(`:114-116`). `recovery-unit/` is never read by this path. For an app with no bind legs,
`tier2CaptureSet` correctly yields **zero legs** (`internal/backup/tier2_capture.go:42-91`), so
neither subtree is ever written. The loop therefore restores 0 files **on every click, forever**,
and `backupTier2RestoreHandler` maps `n == 0` to the reassuring message
(`internal/web/handlers.go:1313-1316`).

**Expectation.** Either the button is not offered when the app's Tier-2 copy contains no restorable
file leg, or the message distinguishes "nothing was missing" from "this copy holds no restorable
files for this app". The customer must not be told their files are all in place by a code path that
never looked at where their files live.

**Severity: HIGH.** This is the customer's own recovery action. The failure mode is the project's
own documented trap class twice over: the message is a **presence-not-success** claim (0 files
copied is read as "nothing was missing"), and it is **unfalsifiable** — identical output whether the
app is pristine or has lost everything. It is also the exact shape of the six shipped-invariant
comments in the workspace `CLAUDE.md`: `tier2_restore.go:14-24` scopes the helper to "class-C data —
HDD bind-mount user files", which is true of the *helper* and silently false of the *button* the UI
renders on top of it. No test pins the UI-level claim.

**DEFECT.** Not an artifact: no cadence was compressed and no state was manufactured — BookStack
was in its normal, healthy, production configuration.

---

## 4. What came off the never-proven list

| item | status |
|---|---|
| Tier-2 cross-drive restore of **ordinary app data** (was: only R-7b share definitions, 2026-07-18) | **PROVEN** (A1) |
| Tier-2 „existing files are not modified or deleted" promise | **PROVEN** (A1, both halves) |
| Tier-2 restore after **total loss** of an app data directory | **PROVEN** (A3) |
| Restored data **usable by the app** rather than present on disk | **PROVEN** (A1/A3, via paperless's own API) |
| Tier-1 recovery-unit restore is a **distinct** path | **ESTABLISHED**; path exercised end-to-end (A2) |
| Tier-1 **content recovery** after real loss | **NOT PROVEN** — A2 ran against an intact app; A3 used Tier-2 |

---

## 5. Phase B — do the six recent fixes interact?

### Setup, and what is therefore an ARTIFACT

| knob | real | campaign | box |
|---|---|---|---|
| local (vzdump) cadence | `0` ⇒ 24 h | 900 s | both |
| PBS tier cadence | 604800 s (7 d) | 900 s | both |
| restore-test cadence | 302400 s (84 h) | 120 s (demo-hp) / 3600 s (demo-felhom) | both |
| `backup_window_start` | 02:30 / unset | 14:00 ⇒ gate [16:00, 20:00) | both |
| PBS reachability | up | `iptables -j REJECT` on `10.77.0.1:8007` | both |

**Everything about FREQUENCY below is an artifact of that compression.** What is not an artifact is
the ORDER of events, which mechanism fired, and what the operator was told.

### B3 — both tiers due, one succeeds, one fails — **COHERENT**

This arose on the very first cycle after the gate opened (`B3-mixed-outcome-cycle1.txt`):

```
15:42:58 [quiesce] backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx]
15:43:05 [quiesce] tier local: backup job …006689159 started — polling
15:44:25 [quiesce] tier local: backup job …006689159 done — next tier may start (app still quiesced)
15:44:25 [quiesce] tier felhom-pbs: backup job …020207536 started — polling
15:44:35 [quiesce] unquiescing (backup failed): restarting 1 stack(s)
15:44:50 [quiesce] tier felhom-pbs: backup job …020207536 failed
15:44:50 [quiesce] tier felhom-pbs has now failed 1 time(s) in a row — deferring its next quiesce
                   by 15m0s (cap 4h0m0s) so the apps are not stopped again for a backup that cannot succeed
15:44:50 Event pushed: whole_guest_backup_failed (error) — … (next attempt in 15m0s)
```

- **R-82's one-quiesce rule holds under mixed outcomes**: two due tiers, apps stopped **once**, both
  tiers attempted inside the single window.
- **The breaker is per-target as designed**: `local` succeeded and was untouched; only `felhom-pbs`
  armed. A broken offsite tier did not suppress a healthy local one.
- **Exactly ONE event**, edge-triggered at the first failure, naming the tier, the reason and the
  retry interval. The hub received exactly one row (`id 1915`) — no duplicates.
- App downtime was ~97 s and paperless came back healthy.

**Operator picture: coherent.** One event, one cause, one next step.

### F-CRIT-2's size filter, confirmed live against REAL damage — **not manufactured**

Unprompted, before any Phase B injection, the demo-hp agent logged:

```
17:37:59 WARN backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a
successful backup  target=felhom-pbs vmid=9201 volid=felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z
size_bytes=1 reason="size 1 B is below the 1048576 B plausibility floor"
```

Independently confirmed on ep0's filesystem — the snapshot is a genuine aborted archive:

```
/mnt/pbs-datastore/ns/demo-hp/ct/9201/2026-07-28T05:31:14Z/
  catalog.pcat1.tmp_didx   0 bytes
  root.pxar.tmp_didx       0 bytes        <- no manifest, no index.json.blob
  pct.conf.blob          425 bytes
```

This is F-CRIT-2 doing exactly its job on a real leftover phantom, in production, without being
asked. **PROVEN, not manufactured.**

### Incidental proof: the local whole-guest archive restores and boots

The restore-test that ran at 17:47:32 selected the local vzdump written by the B3 cycle minutes
earlier, restored it into scratch guest `990000`, booted it, and tore it down:

```
17:49:01 restore-test: scratch guest torn down  vmid=990000
17:49:01 backup: scheduled restore-test passed  archive=local:backup/vzdump-lxc-9201-2026_07_28-17_43_05.tar.zst duration_s=89.19
```

`pct list` afterwards shows no `990000` guest — **F-LEAK's fenced destroy is holding.**

### B1 — contention, then the breaker — **COHERENT; the contention path does NOT swallow the transition**

Forced by tightening the restore-test cadence to 120 s against its ~89 s duration, so the R-85 gate
is held most of the time. Both sides of the seam were captured in the same second:

Agent (`agent-demo-hp.log`):

```
18:03:09 local-api: backup refused — a heavy operation is already in flight  vmid=9201 requested_target=local      busy=restore-test
18:03:09 local-api: backup refused — a heavy operation is already in flight  vmid=9201 requested_target=felhom-pbs  busy=restore-test
```

Controller (`ctrl-demo-hp.log`):

```
16:02:58 [quiesce] backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx]
16:03:09 [quiesce] tier local is BUSY — the agent refused the backup because a concurrent heavy
                   operation holds it. This is contention, NOT a failure: the tier stays due and retries in 15m0s
16:03:09 [quiesce] tier felhom-pbs is BUSY — … (contended for 0s)
16:03:09 [quiesce] unquiescing (last tier is busy — deferring to a later cycle): restarting 1 stack(s)
```

- **No breaker armed and NO event pushed for the contention** — F-A1's whole point, confirmed.
- The app was **restarted immediately** rather than left down for the restore-test's duration —
  the second trap F-A1's comment names ("app thrash") is avoided.
- Note `felhom-pbs` was contended in a cycle where it would certainly have *failed* (its path is
  blocked). The 409 precedes the attempt, so contention wins that cycle and the tier stays due —
  correct, and it does **not** reset the breaker: `noteTierContention` never touches breaker state,
  and the earlier armed failure count survives.

**Operator picture: coherent, and correctly silent.** Contention produced no operator noise at all,
while the genuine failure minutes earlier produced exactly one event.

### The breaker's retry discipline, over four polls

```
15:44:50 tier felhom-pbs has now failed 1 time(s) in a row — deferring … by 15m0s (cap 4h0m0s)
15:47:59 [DEBUG] tier felhom-pbs is in backoff after 1 consecutive failure(s) — not quiescing until 2026-07-28T15:59:50Z
15:52:58 [DEBUG] … not quiescing until 2026-07-28T15:59:50Z
15:58:00 [DEBUG] … not quiescing until 2026-07-28T15:59:50Z
```

Three skipped polls, **one** WARN when the breaker armed, **DEBUG** for each skip, **one** event.
The apps were not stopped once during the backoff. This is the exact behaviour R-88 was built for,
and the pre-R-88 bug (a stop/start cycle every 5 minutes forever) did not recur.

### Both boxes independently, same result

demo-felhom armed its own breaker at 16:00:00 with its own single event
(`whole_guest_backup_failed`, hub id 1916), against demo-hp's id 1915. Per-box and per-target
isolation both hold.

### B4 / B5 — and what they turned up

The demo-felhom rig (docmost made unstartable, PBS blocked, cadences compressed) produced a finding
rather than a clean pass — see **F-2** below. R-100's success anchor was confirmed correct along the
way: after the induced offsite failure on demo-hp, `last_run` advanced to the failing run while
`last_success` **stayed** at the last genuine success:

```
"last_run":     "2026-07-28T16:01:56Z",
"last_status":  "error",
"last_success": "2026-07-28T16:00:18Z",
```

That is precisely R-100's contract — a failure no longer refreshes the staleness clock.

### Not reached

B2's *combined* form (a phantom created during a quiesce that ALSO fails to restart an app) and B4's
full form (a stale anchor while the breaker is backed off, observed as one operator picture) were not
reached. Both need a longer run than this session had.

---

## 6. Findings

### F-2 — an app in a crash loop never alarms, on any channel — **DEFECT, HIGH**

This is F-CRIT-1's exact shape in a state F-CRIT-1's fix does not cover. Campaign 8 closed
"an app that fails to restart after a quiesce never alarms" for `StateStopped`. A **crash-looping**
app is not `StateStopped`, and nothing sees it.

**Action.** On demo-felhom, `docmost/docmost:0.95.0` was retagged to an image that exits immediately,
and the stack was started. (The first attempt — editing the compose image tag — was reverted by the
catalog sync within 3 minutes, which is how O-1 below was found.)

**Observation.** docmost crash-looped continuously. Sampled every 10 s through the controller's own
`/api/stacks` — i.e. exactly the value the alarm logic reads, not `docker ps`:

```
16:05:53 docmost_state=restarting restartcount=14
16:06:24 docmost_state=restarting restartcount=15
16:07:04 docmost_state=restarting restartcount=15
16:08:05 docmost_state=restarting restartcount=16
16:08:25 docmost_state=restarting restartcount=16     … unbroken for the whole observation
```

The restart policy makes this **permanent**, not transient:

```
docker inspect docmost -> unless-stopped maxretry=0 restartcount=16
```

And across that entire period:

```
dashboard page mentions of "docmost":                 0
dashboard warning words (Hiba|Figyel|nem fut):        0
app_start_failed events pushed:                       0
hub events for demo-felhom:                           only whole_guest_backup_failed (the tier, not the app)
```

**The decisive evidence is the detector's own heartbeat.** F-OBS (v0.180.0) exists so that a stalled
dead-app check cannot hide behind silence — it prints a POSITIVE observable every ~10 minutes. Here
it proves the opposite of what it was built to reassure about: the detector is **alive, running, and
blind**.

```
15:59:17 [deadapp] check alive: 160 scans since boot, 4 deployed app(s) evaluated, 0 currently down
16:09:17 [deadapp] check alive: 180 scans since boot, 4 deployed app(s) evaluated, 0 currently down
                                                                                 ^^^^^^^^^^^^^^^^^
docmost at 16:09:17:  status=restarting  restartcount=18  policy=unless-stopped
```

Twenty scans ran between those two lines. Every one of them looked at a permanently crash-looping
app and counted **0 down**. This is not an absent signal being read as health — it is the health
signal itself, positively asserting that nothing is wrong.

**The causal chain, pinned to the code.**

1. `aggregateState` (`internal/stacks/manager.go:753-756`) — *"Any restarting → stack is restarting"*.
   This test runs **before** the partial/degraded logic, so one crash-looping container plus two
   healthy ones yields `StateRestarting`, never `StateDegraded`.
2. `IsDownState` (`internal/stacks/manager.go:54-56`) is
   `StateStopped || StateExited || StateDegraded` — **`StateRestarting` is absent**.
3. `classifyRunStates` (`cmd/controller/main.go:1292-1294`) computes
   `down := stacks.IsDownState(st.State) && !userStopped && !quiesced[…]` → **false**.

So the app never enters `dead` (no dashboard banner) and never enters `states` as `Down` (no
`app_start_failed`, hence no customer email and no hub event). There is **no crash-loop detector
anywhere in the controller** — `RestartCount` is read in exactly one unrelated place
(`internal/web/intermediary.go:105,343`), and `StateRestarting` appears in no alarm path.

**Why this is not covered by the existing suppression logic.** R-97b's grace window is irrelevant
here: it expired at 16:01:56 and the app was still silent minutes later. The failure is not a
suppression that fails to expire — it is a state that is never considered "down" in the first place.

**Expectation.** A container that Docker is restarting indefinitely is a dead app. Either
`StateRestarting` belongs in the down set (with a debounce so a normal restart does not alarm), or
the restart count must be watched. As written, `docker restart: unless-stopped` — the catalog's
standard policy — converts every crash into permanent silence.

**Severity HIGH.** It is the same class the project has now shipped twice (F-CRIT-1, and the
`classifyRunStates` I1 comment in the workspace `CLAUDE.md` table), it affects the most common
real-world app failure mode, and it is *indefinitely* silent rather than delayed.

**DEFECT.** The trigger (a retagged image) was manufactured; **crash loops are not** — a bad image,
a failed migration, a corrupt config or a repeatedly OOM-killed process all produce exactly this
state, and no cadence was compressed to make it happen.

---

### O-1 — recovery-unit capture does not validate, but the catalog sync bounds the damage — **OBSERVATION, not a defect**

**This was very nearly filed as a MEDIUM defect. The live system corrected me, and the correction is
the more useful result.**

What was observed first. The docmost compose was edited to an unpullable image tag; the controller's
periodic refresh captured the recovery unit **7 minutes later** with no validation, so the live app
and its only restore point disagreed — and the restore point was the broken one:

```
15:50:07 [INFO] [backup] Recovery unit captured for docmost → …/backups/primary/docmost (images=3, …)
live:      docmost|docmost/docmost:0.95.0|Up 4 hours (healthy)
manifest:  ['docmost/docmost:0.95.0-campaign9-nonexistent', …]
```

That looked serious, because `ListRestorePoints` returns **exactly one** keep-side restore point per
app (`restore_points.go:14-18`, confirmed live) — an unvalidated refresh replaces the only one there
is. `CaptureRecoveryUnit` (`recovery_unit.go:78-130`) does indeed gate its write on a config
*checksum change* alone: no run-state check, no health check, no test that the pinned images resolve.

**What actually happened next.** Three minutes later the catalog git-sync reverted the drift on its
own, and the recovery unit followed within the same pass:

```
15:53:17 [INFO] [sync] Updated docmost/docker-compose.yml
15:53:17 [INFO] [sync] Periodic sync: Sablonok frissítve — frissítve: docmost
15:53:17 [INFO] [backup] Recovery unit captured for docmost → …/backups/primary/docmost (images=3, …)
```

Verified afterwards on all three surfaces:

```
live compose image:            image: docmost/docmost:0.95.0
Tier-1 recovery unit pins:     ['docmost/docmost:0.95.0', 'postgres:16-alpine', 'redis:7-alpine']
Tier-2 copy's unit pins:       ['docmost/docmost:0.95.0', 'postgres:16-alpine', 'redis:7-alpine']
```

**Conclusion.** Compose drift is self-healing and the restore point follows it, bounded by the sync
interval — the poisoned window here was **~3 minutes**. The system does not leave you in the state I
manufactured, so **reporting this as a defect would have been reporting an artifact.** The absence
of validation in `CaptureRecoveryUnit` is real, but it is compensated, and the compensating control
was proven live rather than assumed.

**The one residue worth an operator's attention** — narrow, and NOT demonstrated here: the sync makes
the *catalog* the source of truth, so a bad image tag **in the catalog** would propagate to every
box's only restore point unhealed, because sync would be restoring it rather than reverting it. That
is a catalog-quality concern owned by the catalog gates, not a controller defect.

### F-3 — an interrupted offsite run leaves a lock the self-heal cannot reach, reported as "unknown" — **DEFECT, MEDIUM**

**This answers Phase C item 8** ("is the repo usable after a killed run, or does it need an
unlock?"). It was found by accident — the campaign severed its own in-flight run — which makes it a
genuine interruption rather than a constructed one.

**Sequence.** An offsite run started at 15:58:12 and took the repo's exclusive lock at 15:59:00. At
15:59:33 its transport was severed. The run died holding the lock. Every later run then failed:

```
16:15:48 [offbox] pre-push dump leg completed in 20.056s — snapshot pair is coherent
16:15:50 [ERROR] [offbox] backup failed after 23s: offbox repo unreachable: exit status 1:
         unable to create lock in backend: repository is already locked exclusively by PID 14328 on demo-hp by root
         lock was created at 2026-07-28 15:59:00 (16m49s ago)
         the `unlock` command can be used to remove stale locks
```

**The remedy exists in the code and was never reached.** `resticStep`
(`internal/backup/offbox.go:634-648`) detects `repository is already locked`, escalates to
`unlock --remove-all`, and retries once — its comment states plainly that this is required because
"the recreated container has a new hostname, so restic can't verify the dead PID and won't treat the
lock as stale for ~30 min (the overnight-campaign C2 finding)".

But the failure happens **before** any `resticStep` call. `ensureOffboxRepo`
(`offbox.go:676-696`) probes the repo first; that probe hits the lock, `classifyResticProbe`
(`offbox.go:77-93`) has no case for a lock error so it returns `"other"`, and the default branch
returns `offbox repo unreachable: …` — a fail-fast with **no unlock escalation**. The self-heal sits
behind a gate the failure never opens.

**And the operator is told the wrong thing.** `ClassifyOffsiteFailure` likewise has no lock case, so
it falls through to `OffsiteFailUnknown`. Hub event 1919:

```
"A távoli mentés ismeretlen okból nem sikerült (23s): offbox repo unreachable: … repository is
already locked exclusively by PID 14328 … the `unlock` command can be used to remove stale locks"
```

"The remote backup failed **for an unknown reason**" — for a condition that is precisely known,
named in the attached detail, and self-healable by code already in the binary.

**Impact.** The offsite tier stays **dead until a human intervenes**. Verified: it took a manual
`restic unlock --remove-all` to restore it (`evidence/phaseC/C8-unlock.txt`).

**Reachability is ordinary, not exotic.** Any interruption of an in-flight run leaves this lock — a
controller container restart, an OOM, a network drop, or **a host reboot mid-backup, which is
Phase C item 6's scenario**. The code's own comment describes exactly this ("the C2 crash lock").

**Severity MEDIUM, argued rather than asserted.** Against HIGH: an error event *is* raised
immediately on the operator channel, its detail literally contains the remedy, and R-100's 48 h
`offsite_stale` is a second backstop, so this is loud-but-mislabelled rather than silent — unlike
F-1 and F-2. Against LOW: a backup tier is down until manual intervention, and the headline actively
misdirects triage.

**DEFECT.** No cadence compression was involved in the mechanism; only the interruption was
deliberate.

---

### O-2 — a hand-run `docker compose up -d` starts a Felhom app WITHOUT its secrets — **OPERATIONAL, worth a runbook line**

Encountered during recovery, by me. Restoring docmost with a plain
`cd /opt/docker/stacks/docmost && docker compose up -d` produced a container that crash-looped on:

```
{"minLength":"APP_SECRET must be longer than or equal to 32 characters","isNotEmpty":"APP_SECRET should not be empty"}
```

Secrets are injected by the controller's `stackEnv` at start time (`StartStack`,
`internal/stacks/manager.go:912-948`); they are **not** in a `.env` file beside the compose. A manual
`up -d` therefore recreates the container with empty secrets — turning a healthy app into a broken
one, and looking exactly like the fault you were trying to fix. Recovery must go through the
controller (`POST /api/stacks/<name>/restart`), which is what finally fixed it.

---

## 7. Phase C / D / E — reached, and not reached

| # | item | status |
|---|---|---|
| C6 | host reboot mid-backup | **NOT REACHED.** Approved by the operator; not attempted. Its most likely consequence was nonetheless observed by another route — see F-3, which is exactly the state a reboot mid-backup would leave. |
| C7 | three-way concurrency (backup + restore-test + GC) | **PARTIALLY.** Backup-vs-restore-test contention was proven (B1); GC was not added as a third party. |
| C8 | restic transport interrupted mid-run — is the repo usable after? | **ANSWERED: no, it needs an unlock.** See **F-3**. |
| D9 | Scenario C — never-succeeded Tier-2 on a throwaway app | **NOT REACHED.** No throwaway app was created. |
| D10 | F-DIAG's failure classes + no credential/repo-URL/raw-error passthrough | **PROVEN for the surfaces that leave the box** — see below. |
| D11 | R-100's threshold — see `offsite_stale` actually fire | **NOT REACHED**, and the approved method turned out not to exist — see below. |
| E12 | F-HUB `SQLITE_BUSY` reproduction | **NOT REACHED** (explicitly lowest priority). |

### D10 — the sanitiser, proven on the exact case that defeated its first version

Six distinct causes map to six distinct Hungarian messages (`offbox.go:201-208`). The transport case
was induced live by blocking port 23 from inside the guest, and the resulting hub event's
`details_json` was:

```
"A távoli tárhely nem érhető el (hálózat vagy bejelentkezés) (51s): offbox repo unreachable:
 exit status 1: subprocess ssh: ssh: connect to host <repo> port 23: Network is unreachable
 Fatal: unable to open repository at <repo> unable to start the sftp session, error: EOF"
```

- the correct distinct class, not a generic string;
- the **bare hostname** — the precise shape the sanitiser's first version leaked on, per its own
  comment (`offbox.go:170-181`) — is replaced with `<repo>`;
- the full `sftp:user@host:/path` repo URL is replaced with `<repo>`.

**Nothing raw leaves the box.** The hub *report* carries no error text at all: `OffboxReportStatus`
(`offbox.go:1149-1174`) has no error field by construction.

**One precise caveat, not a finding.** The `last_error` persisted in the guest's `settings.json` is
raw, and the customer's own `/backups/remote` page renders it verbatim under „Utolsó hiba:",
including the bare hostname and the `sftp:` URL. That is local-only and discloses nothing to that
viewer: the same host, user and repo path are displayed two rows above as *configuration* on the
same page. Sanitisation is applied on the paths that leave the box, which is where the contract puts
it.

### D11 — the approved compression does not exist as a knob

Phase 0 approved compressing the hub's `staleAfter` for this item. It is **not configurable**:
`cmd/hub/main.go:552` calls `monitor.NewOffsiteChecker(dataStore, 0, …)` and `0` selects the
compile-time `defaultOffsiteStaleAfter = 48 * time.Hour` (`internal/monitor/offsite.go:41,60-62`).
Compressing it would have required editing, building and deploying hub code — which this campaign is
forbidden to do ("no production code ships", "no inline fixes"). **Reported rather than worked
around**, per the standing rule that a recommendation not followed gets a line saying why.

The alternative that needs no code change — ageing the controller's reported `last_success` anchor
past 48 h and letting the hub judge it at its real threshold — was identified but not executed.
That is the recommended method for a future run.

Along the way R-100's producer half was confirmed correct twice, at real settings: after each
induced failure `last_run` advanced while `last_success` held.

```
"last_run":     "2026-07-28T16:15:50Z",
"last_status":  "error",
"last_success": "2026-07-28T16:00:18Z",
```

---

## 8. Phase F — recovery

Full log: `evidence/PHASE-F-recovery.log`. Every revert was verified with a positive observable.

| reverted | verification |
|---|---|
| agent cadences, both hosts | `cadences restored -> 0 302400 [604800]`; `systemctl is-active` = `active` |
| backup windows, both boxes | set back to `02:30` through the same endpoint that changed them |
| PBS path block, both hosts | `pvesm status` shows `felhom-pbs … active` on both |
| offsite transport block (guest) | ssh reaches the storage box again (host key exchanged) |
| docmost broken image | real `0.95.0` re-pulled; **restarted via the controller** after O-2 |
| stale restic lock | `unlock --remove-all`; `list locks` empty; 8 snapshots readable |
| campaign scratch files | `0` remaining `c9-*` on both guests; host `/tmp` cleared |
| collectors | all 7 stopped; `0` remaining remote tails |

**The offsite tier was not merely unblocked but proven working again** — a full run through the real
endpoint after the unlock:

```
"last_run":     "2026-07-28T16:20:57Z",
"last_status":  "ok",
"last_success": "2026-07-28T16:20:57Z",
"last_duration": "1m35s",
"snapshot_count": 8
```

**Final fleet state — both boxes fully healthy:**

```
demo-felhom : docmost + docmost-postgres + docmost-redis, immich (×4), calibre-web,
              bookstack (×2), filebrowser, samba, cloudflared, traefik   — ALL healthy
demo-hp     : paperless (×3), filebrowser, samba, cloudflared, traefik   — ALL healthy
agents      : active on both hosts
iptables    : 0 campaign9 rules on either host and in either guest
scratch VMs : none in the 990000 band on either host
ep0         : /srv/pbs-felhom rollback copy intact (13G); datastore 14% used
peti-felhom : never touched
```

**One benign residue, stated for completeness.** The in-memory R-88 breaker on each box still holds
a `felhom-pbs` consecutive-failure count from the induced failures. Its `until` timestamps are long
past, so it blocks nothing; the only effect is that a *next* failure would back off 30 m instead of
15 m. It clears on the next successful whole-guest backup, and by design it also clears on any
controller restart (`breaker.go` — deliberately not persisted). Nothing was done about it because
clearing it would have meant taking another app outage for no benefit.

---

## 9. Where this stopped, and why

**Stopped at the end of Phase B, with Phase D item 10 additionally completed**, then Phase F.

Phase A — the campaign's stated headline and its whole justification — is complete and proven. Phase
B produced two of its five combinations cleanly, plus a HIGH finding. Phase C item 8 was answered by
an accident that turned out to be a genuine defect. Phase D item 10 is proven; D11's approved method
did not exist.

Phase C item 6 (host reboot mid-backup) was deliberately **not started**. It is a large new fault
class against two boxes that are remote until ~08-02, and beginning it would have meant either
rushing it or leaving the fleet in an unknown state. The campaign's own rule — *stop cleanly at a
phase boundary; a broken test node is fine, an unreported one is not* — made the choice: finish the
recovery properly rather than start a fault I could not supervise to completion. F-3 partly covers
its most likely consequence anyway, since an interrupted run is exactly what a reboot mid-backup
produces.

---

## 10. Findings, ranked

| # | finding | severity | class |
|---|---|---|---|
| **F-1** | the Tier-2 restore button reports success for apps it can never restore (BookStack, Docmost — half the fleet's Tier-2 apps); takes a real outage; message is indistinguishable from a genuine "nothing was missing" | **HIGH** | DEFECT |
| **F-2** | an app in a crash loop never alarms on any channel — `StateRestarting` is in no down-set, and the dead-app heartbeat positively reports "0 currently down" while it loops | **HIGH** | DEFECT |
| **F-3** | an interrupted offsite run leaves an exclusive lock the existing self-heal cannot reach; tier dead until manual `unlock`; reported to the operator as "unknown reason" | **MEDIUM** | DEFECT |
| O-1 | recovery-unit capture validates nothing, but the catalog sync reverts drift within ~3 min and the unit follows | — | OBSERVATION (nearly mis-filed as a defect) |
| O-2 | a hand-run `docker compose up -d` starts a Felhom app without its secrets | — | OPERATIONAL |

**Mechanisms confirmed working, on live hardware:** R-82's one-quiesce rule under mixed outcomes;
R-88's breaker (per-target, edge-triggered, one event, no app thrash); F-A1's contention deferral
(no breaker, no event, prompt restart); F-CRIT-2's size filter (against a real 1-byte phantom, not a
manufactured one); R-100's success anchor (twice); F-DIAG's sanitiser (on the exact bare-hostname
case that defeated its first version); F-OBS's positive observable (which is what made F-2
provable); F-LEAK's fenced destroy (no leaked scratch guests across ~10 restore-tests).

---

## 11. What came off the never-proven list — and the honest residue

**Off the list:**

- Tier-2 cross-drive restore of **ordinary app data** — the customer's actual button (A1)
- Tier-2's "existing files are NOT modified or deleted" promise, both halves (A1)
- Tier-2 restore after **total loss** of an app's data directory (A3)
- Restored data **usable by the app**, proven through the app's own API (A1/A3)
- Tier-1 recovery-unit restore is a **distinct** path, and it executes end-to-end (A2)
- The six recent fixes **do** compose: R-82 + R-88 + F-A1 + F-CRIT-2 + R-100 produced one coherent
  operator story in every combination reached
- Whether an interrupted restic run leaves a usable repo (C8 — it does not)

**Still not proven — the residue:**

- **Tier-1 content recovery after real loss.** A2 ran against an intact app; A3 used Tier-2. Nobody
  has yet destroyed an app's data and recovered it *from the recovery unit*. This is now the single
  most valuable unproven item.
- **Host reboot mid-backup** (C6) — untested, and F-REBOOT proved the guest-reboot sibling was real.
- **Three-way concurrency** with GC as the third party (C7).
- **Scenario C** — a never-succeeded Tier-2, live (D9).
- **`offsite_stale` actually firing** (D11) — never observed. Needs the anchor-ageing method above,
  not the `staleAfter` compression that was approved but does not exist.
- **F-HUB** `SQLITE_BUSY` characterisation (E12).
- **Whether F-1 and F-2 have siblings**: both are "a surface reports health it did not establish".
  Nothing in this campaign searched systematically for others.