From 955083c0fc0ab3d4c4dc092bfb6f39e6d6e076ac Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 28 Jul 2026 18:24:06 +0200 Subject: [PATCH] Campaign 9: the Tier-2 restore paths are PROVEN; 3 defects filed, none fixed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase A is the headline and it passed on live hardware, through the real endpoints the UI posts to: a customer who deletes files — or their entire app data directory — gets everything back byte-identical, and the app works afterwards (paperless served the restored bytes over its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold. Three defects, recorded not fixed: C9-F1 (HIGH) the Tier-2 restore button is offered for apps it can never restore, takes a real outage, and reports "nothing was missing" — indistinguishable from a genuine result, while 156 MB of that app's data sits unread in the same copy. C9-F2 (HIGH) an app in a crash loop never alarms on any channel; StateRestarting is in no down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes. C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier is dead until a human unlocks and the operator is told "unknown reason". This answers Phase C item 8. Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have been reporting an artifact. Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real cadences, offsite tier proven working again, no leaked scratch guests, peti untouched. D11's approved staleAfter compression turned out not to be a knob; reported, not worked around. --- REPORT-campaign9.md | 85 ++ REPORT.md | 198 ++--- .../CAMPAIGN-9-restore-proof-2026-07-28.md | 825 ++++++++++++++++++ documentation/backlog/OPEN-ITEMS.md | 3 + 4 files changed, 981 insertions(+), 130 deletions(-) create mode 100644 REPORT-campaign9.md create mode 100644 documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md diff --git a/REPORT-campaign9.md b/REPORT-campaign9.md new file mode 100644 index 0000000..7cee9fb --- /dev/null +++ b/REPORT-campaign9.md @@ -0,0 +1,85 @@ +# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28) + +**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign, +and findings are recorded, never fixed inline. Full write-up: +`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`. +Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously). + +Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes. +**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact. + +## The headline — two never-proven restore paths are now proven + +Driven through the **real endpoints the UI posts to**, over https through traefik with a real session +and CSRF token, on live hardware. + +| proof | result | +|---|---| +| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) | +| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) | +| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy | +| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 | +| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy | +| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 | + +The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return +at their last-backup state.** + +## Findings — 3 defects, ranked (none fixed) + +| # | finding | severity | +|---|---|---| +| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** | +| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** | +| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** | + +Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog +sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an +artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed +so live. + +## Mechanisms confirmed working, live + +R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target +breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app +thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of +the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on +demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's +sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the +hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable; +F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests). + +## Where it stopped, and what remains + +Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host +reboot mid-backup) was deliberately not started — a large new fault class against boxes that are +remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown. + +**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's +threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time +`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign +forbids. Reported rather than worked around. The no-code-change alternative (age the controller's +reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended +method next time. + +**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an +intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way +concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`. + +## Recovery + +Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive +observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active), +windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on +either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and +the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots). + +One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box. +Its `until` is long past so it blocks nothing; it clears on the next successful backup or any +controller restart (by design, not persisted). Clearing it would have cost another app outage for no +benefit. + +**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in +`/opt/docker/stacks/` starts a Felhom app **without its secrets** — they are injected by the +controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a +crash loop during recovery. Manual recovery must go through `POST /api/stacks//restart`. diff --git a/REPORT.md b/REPORT.md index 0b4b400..7cee9fb 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,147 +1,85 @@ -# REPORT — closing the backup arc: seam sweep + live outage validation (2026-07-27) +# REPORT — CAMPAIGN 9: the restore paths, proven (2026-07-28) -**Overwritten** per the standing rule. **No version shipped** — the sweep found no defect, so the -witnesses are compile-time-only source changes riding the next release. Agent `9842c52`→`0236553`, -controller `ca013c8`→`8f46495`. Live fleet unchanged: hub v0.79.0, agent v0.105.0, controller v0.178.0. +**Overwritten** per the standing rule. **No production code shipped** — this was a proof campaign, +and findings are recorded, never fixed inline. Full write-up: +`documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md`. +Evidence: `DooPlex:~/campaign9/evidence/` (69 files, 221 MB, 7 collectors, written continuously). -## Part 1 — the seam table +Fleet unchanged and healthy at close: hub v0.80.0, agent v0.110.0, controller v0.182.0 on both boxes. +**`peti-felhom` was never touched.** The ep0 rollback copy `/srv/pbs-felhom` (13 G) is intact. -Every optional-capability type assertion across all four repos. `StackDataProvider`, `BackupReporter` -and `RestoreTestReporter` are **not** in this table: they are interface-typed struct *fields*, which -the compiler already enforces at assignment. The dangerous shape is `x.(Interface)`. +## The headline — two never-proven restore paths are now proven -| Interface | Asserted at | On failure | Silent? | Witness now | -|---|---|---|---|---| -| `quiesce.TieredBackend` | `tiers.go:109` (`resolveDueTiers`) | falls back to the untargeted single-tier path | **SILENT** | ✅ moved to production | -| `quiesce.TieredBackend` | `quiesce.go:375` (`allTiersForManualRun`) | manual run covers only the untargeted tier | **SILENT** | ✅ same witness | -| `quiesce.TieredBackend` | `tiers.go:191`, `:199` | **unchecked — would panic** | no (loud) | ✅ same witness | -| `localapi.BackupArchiveLister` | `server.go:1155` (`newestArchiveOn`) | degrades to `archiveAbsent` = pre-R-84 in-memory-only | **SILENT** | ✅ added | -| `agentapi.AgentVersionReporter` | `features.go:157` | version gate → live probe | silent, benign (both decide correctly) | ✅ added | -| anonymous `GuestMemory` iface | `features.go:86` | returns `errNoMemoryProbe` → `SupportUnknown` | **no** — distinct error | n/a (anonymous) | +Driven through the **real endpoints the UI posts to**, over https through traefik with a real session +and CSRF token, on live hardware. -**Was `TieredBackend` the only one, or the first one? It was the first one.** -`BackupArchiveLister` has the identical shape and a worse blast radius: a silent degrade to -`archiveAbsent` is the R-84 bug returning — a cold store after a restart reading as "no backup ever" — -with nothing in any log. It had **no witness of any kind**, and in fact **zero compile-time witnesses -existed in production code anywhere in either repo** before today. - -**No defect found.** Every implementation satisfies its interface today, so per the task's rule there -is no version bump and no deploy. What was missing was the *guard*, not the behaviour. - -**The guard is proven, not assumed.** Breaking `BackupRunner.NewestArchiveTime`'s signature now fails -`go build`: - -``` -cmd/felhom-agent/seam_witnesses.go:28:38: *backup.BackupRunner does not implement - localapi.BackupArchiveLister (wrong type for method NewestArchiveTime) -``` - -Before the witness, that same break compiled and vetted clean. The controller's witness also **moved -out of a `_test.go` file** — a test-file witness fires on `go test`/`go vet` but not on `go build` -alone, which is exactly the path the R-88 Part 2 near-miss would have taken. - -## Part 2 — the live outage - -Pre-flight cleared: 0 tasks in flight on ep0, 0 backup/restore processes on either box (counted by -`COMM`, since `pgrep -f` self-matches and gave a false `1` first), 0 scratch guests, no quiesce marker. - -### The headline — `age_state=unknown` on real hardware, first ever - -```json -{"vmid":9201,"due":true, - "reason":"backup storage unreadable and no in-memory record — age UNKNOWN, treating as due", - "target":"felhom-pbs","age_state":"unknown"} -``` - -Identical on both boxes. The transition is timestamped on demo-felhom: - -``` -18:35:30 felhom-pbs/content 200 ← before power-off -18:39:26 felhom-pbs/content 500 ← after -18:39:11 local/content 200 ← local tier, same window, unaffected -``` - -Agent journal, both boxes: `could not read the backup storage for the due-check … target=felhom-pbs` -— the degrade path genuinely executed. **Positive observables throughout**, per standing rule 3: an -empty quiesce log alone would have been equally consistent with a dead loop. - -### The behaviour it gates - -demo-felhom's `local` tier was **genuinely due** (25.8 h) for the whole window, so the due set was -`[local(known), felhom-pbs(unknown)]` → `oldestAge` nil → `valveLicensed` false → **defer**. This is -the 09:02 incident's exact condition replayed with the fix in place. - -| | demo-felhom | demo-hp | -|---|---|---| -| App containers running | **10**, unchanged | **3**, unchanged | -| Quiesce cycles | **0** | **0** | -| Quiesce marker | absent | absent | -| vzdump attempted | none since 07:13 | none since 09:37 | - -**Not one app stack was stopped.** Per-tier isolation demonstrated on the same box in the same -minute: `local` → 200 with a real age, `felhom-pbs` → 500 → unknown. - -### Item 4 — conditions did not arise, and that is confirmed, not assumed - -`felhom-pbs` was not due on either box (28 h / 7 h against a 168 h cadence), so no backup ran, nothing -failed, **the R-88 breaker did not arm**, and **no `whole_guest_backup_failed` travelled**. I did not -force one. Confirmed positively: the newest hub event is `16:10:33`, *before* the window opened at -16:24, and 0 events exist in the window. - -## Part 2.2 — post-boot - -| Check | Result | +| proof | result | |---|---| -| Mount | `/mnt/pbs-datastore ext4 rw,relatime,discard` — **relatime present, `noatime` ABSENT** ✅ | -| Swap | `/swapfile 4G` back — **survived its first reboot** (it was added and verified without one) ✅ | -| PBS services | `proxmox-backup` + `proxmox-backup-proxy` both active ✅ | -| Datastore | `felhom-offsite → /mnt/pbs-datastore`, online ✅ | -| Snapshots | demo-felhom **2**, demo-hp **3** — **exact match**, same timestamps ✅ | -| Schedules | `gc-schedule sun 04:30`, `verify-new 1`, 2 prune jobs ✅ | -| Filesystem | `EXT4-fs (sda): recovery complete` — journal replayed after the unclean power-off, then mounted r/w; **0** I/O or EXT4 errors ✅ | +| **A1** — Tier-2 restore of ordinary app data (`paperless-ngx`, demo-hp) | 6 deleted files back **byte-identical** (`sha256sum -c` all OK) | +| A1 — „A meglévő fájlok NEM módosulnak és NEM törlődnek" | 2 created files survived; 1 locally-edited file **not overwritten** (edit marker intact) | +| A1 — app stopped/restarted and healthy | stop→copy→start in 39 s, `paperless-webserver` healthy | +| A1 — data **usable by the app**, not just on disk | paperless resolved all 3 docs, checksums matched its own DB, and **served the restored bytes over its own HTTP API** at the exact pre-deletion sha256 | +| **A2** — Tier-1 recovery-unit restore is a **distinct** path | `POST /backup/restore` → `RestoreFromRecoveryUnit`; ran end-to-end in 18 s, 1 volume restored, app healthy | +| **A3** — restore after **total loss** (whole appdata dir `rm -rf`) | loss proven by doc download going **200 → 404**; restore returned **43/43 files byte-identical**, `documents_ok 16 of 16`, downloads back to 200 | -### The unplanned finding: the volume changed device name +The honest boundary A1+A3 together establish: **existing files are untouched; destroyed files return +at their last-backup state.** -``` -before: volume=/dev/sdb root=/dev/sda1 -after: volume=/dev/sda root=/dev/sdb1 -``` +## Findings — 3 defects, ranked (none fixed) -The kernel enumerated the disks in the opposite order across the reboot. The mount survived only -because fstab uses `/dev/disk/by-id/scsi-0HC_Volume_106469259`. **A `/dev/sdb` entry would have failed -to mount — or mounted the root disk's device node.** That choice was load-bearing and had never been -tested; this reboot tested it. +| # | finding | severity | +|---|---|---| +| **C9-F1** | The Tier-2 restore button is offered for apps it can **never** restore (BookStack, Docmost). It takes a real app outage, restores 0 files, and reports „Nincs hiányzó fájl — minden fájl megvan a helyén." — while 156 MB of that app's data sits unread in the same copy | **HIGH** | +| **C9-F2** | An app in a **crash loop never alarms on any channel**. `StateRestarting` is in no down-set, so the dead-app heartbeat printed *"180 scans … 0 currently down"* while the app had been looping for 9 minutes | **HIGH** | +| **C9-F3** | An **interrupted offsite run** leaves an exclusive restic lock the existing self-heal cannot reach; the tier is dead until a human unlocks, and the operator is told *"unknown reason"* | **MEDIUM** | -### Recovery +Two things were deliberately **not** filed as defects: a recovery-unit poisoning that the catalog +sync self-healed within ~3 minutes (proven live — reporting it would have been reporting an +artifact), and a `snapshot_id` that looked ignored but is documented as logging-only and confirmed +so live. -Both boxes returned to `age_state=known` on both tiers within ~60 s of boot; `felhom-pbs` on -demo-felhom back to `due=false, within cadence window, age_seconds=102102`. No breaker was armed, so -none needed clearing; no failure event fired, so no recovery event was expected — and none appeared. +## Mechanisms confirmed working, live -## Part 3 — CX33 +R-82's one-quiesce rule under mixed outcomes (2 tiers due, apps stopped **once**, per-target +breaker); R-88's breaker (edge-triggered, one WARN, one event, three silent DEBUG skips, **no app +thrash**); F-A1's contention deferral (409 → no breaker, no event, prompt restart — both sides of +the seam captured in the same second); **F-CRIT-2's size filter against a real 1-byte phantom** on +demo-hp, confirmed independently on ep0's filesystem; R-100's success anchor twice; **F-DIAG's +sanitiser on the exact bare-hostname case that defeated its first version** (nothing raw reaches the +hub event or the report); F-OBS's positive observable — which is precisely what made C9-F2 provable; +F-LEAK's fenced destroy (no leaked `990000` guests across ~10 restore-tests). -**Not available even with the server powered off**, so the powered-on theory is settled: it is the -Cost-Optimized line's "Limited availability", not the power state. No rescale happened. R-90 stays -`BLOCKED`/`WATCHING` with the 4 GiB swap as the interim — now a *proven* interim, since it survived a -reboot. Escape hatch if 8 GB becomes urgent: the **Regular Performance (CPX)** / **General Purpose -(CCX)** lines carry no availability warning, at higher cost. Operator's call. +## Where it stopped, and what remains -## Comes off the never-live-validated list +Stopped at the **end of Phase B**, plus Phase D item 10, then full recovery. Phase C item 6 (host +reboot mid-backup) was deliberately not started — a large new fault class against boxes that are +remote until ~08-02, and starting it would have meant rushing it or leaving the fleet unknown. -- **`age_state=unknown` observed on real hardware** — the path R-88 Part 2 exists for. -- **The window gate refusing an unknown-driven valve, with a genuinely due tier present** — zero app - stacks stopped. -- **Per-tier isolation under a real offsite outage.** -- **The by-id mount surviving a reboot** (and surviving a device rename, which was not anticipated). -- **The 4 GiB swap surviving a reboot.** -- **Datastore integrity across an unclean power-off** — journal replay, exact snapshot match. +**Approved but impossible:** Phase 0 cleared compressing the hub's `staleAfter` for R-100's +threshold test. It is **not a knob** — `cmd/hub/main.go:552` passes `0`, selecting the compile-time +`defaultOffsiteStaleAfter = 48h`. Compressing it needed a hub code change, which the campaign +forbids. Reported rather than worked around. The no-code-change alternative (age the controller's +reported `last_success` past 48 h and let the hub judge at its real threshold) is the recommended +method next time. -## Honestly cannot come off it +**The honest residue — still not proven:** Tier-1 **content** recovery after real loss (A2 ran on an +intact app; A3 used Tier-2) — now the most valuable open item; host reboot mid-backup; three-way +concurrency with GC; Scenario C live; `offsite_stale` actually firing; F-HUB `SQLITE_BUSY`. -- **The R-88 breaker has still never armed on real hardware.** No backup was due, so none failed. -- **`whole_guest_backup_failed` has never travelled from a REAL failure** — only from the induced - POST on 2026-07-27 15:04:57. The quiesce→notify link remains unit-proven. -- **R-97b's app-alarm suppression has never fired live** — it needs a real quiesce cycle. -- **`age_state=absent` has never been observed live** — it needs a genuinely fresh tier. -- **The legacy-agent path (Scenario C) is now unobservable on this fleet** — both agents are v0.105.0. -- **The first in-window scheduled backup** lands 2026-07-28 at the earliest. +## Recovery + +Every config reverted from `evidence/config-before/REVERT.md`, each verified with a **positive +observable**: agent cadences back to `0 / 302400 / 604800` on both hosts (`is-active` = active), +windows back to `02:30`, `pvesm` shows `felhom-pbs active` on both, 0 campaign iptables rules on +either host or guest, 0 scratch guests in the `990000` band, all stacks healthy on both boxes, and +the offsite tier not merely unblocked but **proven working again** (`ok`, 1m35s, 8 snapshots). + +One benign residue: the in-memory R-88 breaker still holds a `felhom-pbs` failure count on each box. +Its `until` is long past so it blocks nothing; it clears on the next successful backup or any +controller restart (by design, not persisted). Clearing it would have cost another app outage for no +benefit. + +**One operational lesson worth a runbook line:** a hand-run `docker compose up -d` in +`/opt/docker/stacks/` starts a Felhom app **without its secrets** — they are injected by the +controller's `stackEnv` at start time, not stored in a `.env`. It turned a healthy docmost into a +crash loop during recovery. Manual recovery must go through `POST /api/stacks//restart`. diff --git a/documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md b/documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md new file mode 100644 index 0000000..0880b73 --- /dev/null +++ b/documentation/audits/CAMPAIGN-9-restore-proof-2026-07-28.md @@ -0,0 +1,825 @@ +# CAMPAIGN 9 — proof, not fixes: the restore paths and the interactions + +**Adversarial, destructive, unattended. 2026-07-28 17:12 CEST → in progress.** + +Nodes exercised: `demo-felhom` (+ guest 9201), `demo-hp` (+ guest 9201), `ep0` +(`felhom-hetzner`). Controllers on **0.182.0** both boxes; agent 0.110.0; hub 0.80.0. + +**`peti-felhom` was never touched** — not its data, not its namespace, not `u629488-sub2`. The ep0 +rollback copy `/srv/pbs-felhom` was never deleted. + +Evidence: **`DooPlex:~/campaign9/evidence/`**, written continuously by 7 collectors (6 log streams + +a 2-minute state snapshot) started **before** the first mutation and verified emitting live lines +before Phase A began. + +--- + +## Headline + +**The two never-proven restore paths are now proven, on live hardware, through the real endpoints +the UI posts to.** A customer who deletes files — or their entire app data directory — gets +everything back, byte-identical, and the app works afterwards. + +**One HIGH-severity defect fell out of proving it: F-1.** For apps whose data lives entirely in +Docker named volumes, the „Fájlok visszaállítása" button is rendered, stops the app, restores +nothing, and reports **„Nincs hiányzó fájl — minden fájl megvan a helyén."** — a message that +positively asserts the customer's files are all present. It says this on every click, on any day, +no matter how much data was lost, while 156 MB of that app's data sits unread in the same Tier-2 +copy the button claims to restore from. + +--- + +## 1. Phase 0 — questions, answers, and skips + +Asked in one batch, before touching anything. **All four answered; nothing was skipped for lack of +an answer.** + +| # | Question | Answer received | +|---|---|---| +| 1 | Which targets may I destroy data on? | **Throwaway apps AND existing demo apps on BOTH boxes.** | +| 2 | May the hub's `staleAfter` be compressed for Phase D item 11? | **Yes — compress, then revert.** | +| 3 | How may ep0 be taken down? | **Power-cycle allowed programmatically** (widened from Campaign 8's "no power-cycle"). | +| 4 | `drill-r50` destroy/rebuild? Host reboot for Fault 12? | **drill-r50 free to destroy; host reboot OK on BOTH demo hosts.** | + +--- + +## 2. Phase A — the restore proofs (the headline) + +### A1 — Tier-2 cross-drive restore of ordinary app data — **PROVEN** + +**Target.** `paperless-ngx` on **demo-hp** guest 9201 — chosen because it is the only app in the +fleet that combines real user content (16 documents), a complete Tier-2 copy with a real user-file +leg, and an R-101 success anchor. Live root `/mnt/felhom-drives/nvme-1tb`; Tier-2 copy +`/mnt/sys_drive/felhom-data/backups/secondary/paperless-ngx`. + +**Method.** The real endpoint the UI posts to — `POST /backup/tier2/restore` with `stack_name` — +driven **through traefik over https with the vhost Host header**, from inside the guest, with a real +session cookie and a real CSRF token scraped from the rendered `/backups/apps` page. No server logic +was skipped; only browser rendering. + +**The rendered dialog** (`evidence/phaseA/A1-dialog-rendered.txt`) — this is the R-101 branch, live: + +``` +Visszaállítja a hiányzó fájlokat a másodlagos másolatból? A meglévő fájlok NEM módosulnak és NEM +törlődnek. Az alkalmazás a művelet idejére leáll. Legutóbbi sikeres másolat: 2026-07-28 16:43. +``` + +**Baseline.** All 43 files in the copy were byte-identical to live before anything was touched +(`A1-copy-inv-BEFORE.txt` vs `A1-live-inv-BEFORE.txt`, `diff` empty). + +**The mutation exercised all three of the dialog's promises in one run** — because the dialog makes +three distinct claims and proving them separately would have been three outages: + +| class | files | promise under test | +|---|---|---| +| DELETED | 6 (3 originals, 1 archive, 2 thumbnails, spanning all subdirs, incl. one complete document) | must come back byte-identical | +| CREATED | 2 (one in an existing dir, one in a **new** dir) | „NEM törlődnek" — must survive | +| MODIFIED | 1 thumbnail, appended marker | „NEM módosulnak" — must not be overwritten | + +**Result — all three hold** (`A1-verify.txt`): + +``` +=== PROMISE 1: deleted files came back BYTE-IDENTICAL === +originals/0000002.pdf: OK +originals/0000005.pdf: OK +originals/0000012.pdf: OK +archive/0000012.pdf: OK +thumbnails/0000012.webp: OK +thumbnails/0000003.webp: OK + +=== PROMISE 2: created files were NOT deleted === +PRESENT originals/c9-created-file.txt be279d439b852e78c6d10379c4f569360283d4ce77523f4848723beb5b78997e +PRESENT c9-newdir/c9-keep.txt 4fa680f76eeac33369fefec88d1a216cea7d16255573e439302d40c63fa9d6a9 + +=== PROMISE 3: locally-modified file was NOT overwritten === +pre-edit fd9b92dd77df6bd9e6474d238ec86e029e7f562c820ed1172d71ad6cb9eccb42 <- must NOT match +post-edit 15c61ab7ddd1bfcfecc7ac7f43d12357de123b42d3ab127e3a23a46af4bd4fd2 <- must match +actual 15c61ab7ddd1bfcfecc7ac7f43d12357de123b42d3ab127e3a23a46af4bd4fd2 +marker still present: CAMPAIGN9-LOCAL-EDIT-MUST-NOT-BE-OVERWRITTEN +live file count: 45 (43 baseline + 2 created — arithmetic closes) +``` + +**Stop/start and health**, from the controller (`A1-controller-log.txt`): + +``` +15:21:59 [WARN] [web] Tier-2 file restore requested (async): stack=paperless-ngx +15:21:59 [INFO] [backup] Tier-2 file restore for paperless-ngx: …/secondary/paperless-ngx (v2) → /mnt/felhom-drives/nvme-1tb (additive-only) +15:22:20 … Post-restore health check: paperless-ngx not yet running, waiting… (×5) +15:22:45 [DEBUG] [backup] Post-restore health check: paperless-ngx is running +15:22:45 [INFO] [backup] Tier-2 file restore completed for paperless-ngx: 6 file(s) restored (39s) +``` + +Container uptime confirmed the real stop/start independently (`Up 5 seconds (health: starting)` → +`Up 27 seconds (healthy)` polled at 10-second intervals). + +**The restored data is usable BY THE APP, not merely present on disk** (`A1-usable.txt`) — two +independent checks: + +``` +=== CHECK 1: paperless resolves each restored doc and its content matches the DB checksum === +doc 2 invoice-114750664 original_exists=True checksum_match=True thumb_exists=True +doc 5 Project Overtime 2023.11 original_exists=True checksum_match=True thumb_exists=True +doc 12 Hotel original_exists=True checksum_match=True archive_exists=True archive_match=True thumb_exists=True + +=== CHECK 2: served over paperless's own HTTP API, hashing what the app returns === +doc 2 http=200 served_sha256=87226ff2…13244f == recorded before deletion +doc 5 http=200 served_sha256=1395403a…f2f2df == recorded before deletion +doc 12 http=200 served_sha256=1dd4cd16…dd924b == recorded before deletion (archive; /download/ serves the archive when one exists) +``` + +**A1 verdict: PROVEN.** Byte-identical recovery, both non-destruction promises kept, app healthy +and serving the restored bytes over its own API. + +### A2 — Tier-1 local recovery-unit restore — **DISTINCT, and exercised** + +**It is a distinct path**, not subsumed: `POST /backup/restore` (`stack_name` + `snapshot_id`) → +`RestoreFromRecoveryUnit` (`internal/web/handlers.go:1230`, `internal/backup/restore_unit.go:96`). +It reads the app's on-drive recovery unit, recovers secrets from the guest's live `app.yaml`, +restores named-volume data from the unit's tars, and redeploys. Tier-2 copies are **not** reachable +through it. + +Exercised on `calibre-web` (demo-felhom), `A2-restore-status.txt` / `A2-log.txt`: + +``` +15:28:34 [INFO] [backup] Restoring calibre-web from recovery unit: images=1, secrets recovered=0/0, data_keys=0 +15:28:39 [INFO] [backup] Restoring Docker volume calibre-web_calibre_web_config for calibre-web +15:28:39 [INFO] [backup] Restored 1 Docker volume(s) for calibre-web +15:28:52 [INFO] [backup] Restore-from-unit completed: calibre-web (18.25s) +``` + +App healthy 29 s later; volume fingerprint unchanged (`6b10ec90…`), consistent with restoring a tar +that matched current content. + +**Honest limit:** this run proves the *path executes and the app survives it*. It does **not** prove +content recovery, because nothing was lost first. The content-recovery proof is A3. + +**One thing checked and deliberately NOT reported as a finding.** `snapshot_id` is required by the +handler and then never passed to `RestoreFromRecoveryUnit`, while the success flash names it — +which looks like "you picked A and got B". It is neither: `restore_points.go:14-18` states there is +exactly one keep-side restore point and that `snapshot_id` is logging-only, and the live API +confirms exactly one point per app with `short_id: "helyi"` — the value the flash prints. Correct by +design, pinned by its own comment. **ARTIFACT of reading, not a defect.** + +### A3 — restore after REAL loss — **PROVEN** + +The harder shape: not gap-fill, but deleting the app's data directory **entirely**, while the app +ran, as an "I deleted my files" event actually looks. + +``` +=== BEFORE: paperless serves doc 12 === source_exists True +=== DESTROYING /mnt/felhom-drives/nvme-1tb/appdata/paperless === +exists after rm: NO +=== AFTER === source_exists False + doc12_download_http=404 +=== doc count in DB is untouched === 16 +``` + +The loss was verified with a **positive observable** — the document download going 200 → **404** — +not merely by the directory being absent. + +**Tier used: Tier-2**, via the same customer button. Result +(`A3-restore-status.txt`, `A3-verify.txt`): + +``` +message: "paperless-ngx: 43 fájl visszaállítva a másodlagos másolatból." (46 s) + +=== 1. every restored file byte-identical to the Tier-2 copy? === +copy files: 43 / live files: 43 +IDENTICAL — every file in the copy is back, byte-for-byte, and nothing extra + +=== 2. does PAPERLESS work? === +documents_ok 16 of 16 +problems [] + +=== 3. serve doc 12 over HTTP again (the observable that was 404 after the loss) === +doc 2 http=200 sha256=87226ff2…13244f +doc 5 http=200 sha256=1395403a…f2f2df +doc 12 http=200 sha256=1dd4cd16…dd924b +``` + +**A3 verdict: PROVEN — the customer gets their app back, not "only some files".** + +**The honest residue, stated plainly.** Three things did not come back, and none of them is a +defect — each is the correct consequence of not being in the last successful copy: + +- the two files created *after* the last Tier-2 run — never backed up, correctly gone; +- the local edit to `thumbnails/0000001.webp` — the file returned in its **pristine** backed-up + form (`fd9b92dd…`), i.e. the post-loss restore legitimately reverted an unbacked-up edit. This is + the same file that A1 proved is *never overwritten* while it still exists; the two results are + consistent and together delimit the guarantee exactly: **existing files are untouched; destroyed + files return at their last-backup state.** +- `appdata/paperless/export/` — absent from the copy, recreated empty by paperless on startup. + +--- + +## 3. Findings + +### F-1 — the restore button reports success for apps it can never restore — **DEFECT, HIGH** + +**Applies to:** every app whose data lives entirely in Docker named volumes and which therefore has +no bind-mounted user-file leg. On demo-felhom that is **BookStack and Docmost** — 2 of the 4 apps +with a Tier-2 copy, i.e. half the fleet's Tier-2-configured apps. + +**Action.** Clicked „Fájlok visszaállítása" for `bookstack` on demo-felhom via the real endpoint. + +**Observation** (`A1b-bookstack-post.txt`, `A1b-bookstack-log.txt`): + +``` +15:26:05 [WARN] [web] Tier-2 file restore requested (async): stack=bookstack +15:26:05 [INFO] [backup] Tier-2 file restore for bookstack: …/secondary/bookstack (v2) → /mnt/sys_drive/felhom-data (additive-only) +15:26:28 [INFO] [backup] Tier-2 file restore completed for bookstack: 0 file(s) restored (19s) + +restore-status: {"ok":true, … "message":"Nincs hiányzó fájl — minden fájl megvan a helyén."} + +BEFORE: bookstack|Up 4 hours (healthy) +AFTER: bookstack|Up 25 seconds (healthy) <- a real outage was taken +``` + +And the copy the button claims to restore from is **not empty** — it holds the app's entire data: + +``` +16K …/secondary/bookstack/recovery-unit/compose +60K …/secondary/bookstack/recovery-unit/db-dumps +156M …/secondary/bookstack/recovery-unit/volume-dumps <- BookStack's actual data +4.0K …/secondary/bookstack/recovery-unit/manifest.json + +ls …/secondary/bookstack/ -> recovery-unit (no hdd/, no userdata/) +``` + +**Why it can never succeed.** `RestoreTier2Files` merges exactly two subtrees — +`destBase/hdd` → `liveNsRoot` and `destBase/userdata` → `liveNsRoot/userdata` +(`internal/backup/tier2_restore.go:101-104`) — and `os.Stat`-skips each when absent +(`:114-116`). `recovery-unit/` is never read by this path. For an app with no bind legs, +`tier2CaptureSet` correctly yields **zero legs** (`internal/backup/tier2_capture.go:42-91`), so +neither subtree is ever written. The loop therefore restores 0 files **on every click, forever**, +and `backupTier2RestoreHandler` maps `n == 0` to the reassuring message +(`internal/web/handlers.go:1313-1316`). + +**Expectation.** Either the button is not offered when the app's Tier-2 copy contains no restorable +file leg, or the message distinguishes "nothing was missing" from "this copy holds no restorable +files for this app". The customer must not be told their files are all in place by a code path that +never looked at where their files live. + +**Severity: HIGH.** This is the customer's own recovery action. The failure mode is the project's +own documented trap class twice over: the message is a **presence-not-success** claim (0 files +copied is read as "nothing was missing"), and it is **unfalsifiable** — identical output whether the +app is pristine or has lost everything. It is also the exact shape of the six shipped-invariant +comments in the workspace `CLAUDE.md`: `tier2_restore.go:14-24` scopes the helper to "class-C data — +HDD bind-mount user files", which is true of the *helper* and silently false of the *button* the UI +renders on top of it. No test pins the UI-level claim. + +**DEFECT.** Not an artifact: no cadence was compressed and no state was manufactured — BookStack +was in its normal, healthy, production configuration. + +--- + +## 4. What came off the never-proven list + +| item | status | +|---|---| +| Tier-2 cross-drive restore of **ordinary app data** (was: only R-7b share definitions, 2026-07-18) | **PROVEN** (A1) | +| Tier-2 „existing files are not modified or deleted" promise | **PROVEN** (A1, both halves) | +| Tier-2 restore after **total loss** of an app data directory | **PROVEN** (A3) | +| Restored data **usable by the app** rather than present on disk | **PROVEN** (A1/A3, via paperless's own API) | +| Tier-1 recovery-unit restore is a **distinct** path | **ESTABLISHED**; path exercised end-to-end (A2) | +| Tier-1 **content recovery** after real loss | **NOT PROVEN** — A2 ran against an intact app; A3 used Tier-2 | + +--- + +## 5. Phase B — do the six recent fixes interact? + +### Setup, and what is therefore an ARTIFACT + +| knob | real | campaign | box | +|---|---|---|---| +| local (vzdump) cadence | `0` ⇒ 24 h | 900 s | both | +| PBS tier cadence | 604800 s (7 d) | 900 s | both | +| restore-test cadence | 302400 s (84 h) | 120 s (demo-hp) / 3600 s (demo-felhom) | both | +| `backup_window_start` | 02:30 / unset | 14:00 ⇒ gate [16:00, 20:00) | both | +| PBS reachability | up | `iptables -j REJECT` on `10.77.0.1:8007` | both | + +**Everything about FREQUENCY below is an artifact of that compression.** What is not an artifact is +the ORDER of events, which mechanism fired, and what the operator was told. + +### B3 — both tiers due, one succeeds, one fails — **COHERENT** + +This arose on the very first cycle after the gate opened (`B3-mixed-outcome-cycle1.txt`): + +``` +15:42:58 [quiesce] backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx] +15:43:05 [quiesce] tier local: backup job …006689159 started — polling +15:44:25 [quiesce] tier local: backup job …006689159 done — next tier may start (app still quiesced) +15:44:25 [quiesce] tier felhom-pbs: backup job …020207536 started — polling +15:44:35 [quiesce] unquiescing (backup failed): restarting 1 stack(s) +15:44:50 [quiesce] tier felhom-pbs: backup job …020207536 failed +15:44:50 [quiesce] tier felhom-pbs has now failed 1 time(s) in a row — deferring its next quiesce + by 15m0s (cap 4h0m0s) so the apps are not stopped again for a backup that cannot succeed +15:44:50 Event pushed: whole_guest_backup_failed (error) — … (next attempt in 15m0s) +``` + +- **R-82's one-quiesce rule holds under mixed outcomes**: two due tiers, apps stopped **once**, both + tiers attempted inside the single window. +- **The breaker is per-target as designed**: `local` succeeded and was untouched; only `felhom-pbs` + armed. A broken offsite tier did not suppress a healthy local one. +- **Exactly ONE event**, edge-triggered at the first failure, naming the tier, the reason and the + retry interval. The hub received exactly one row (`id 1915`) — no duplicates. +- App downtime was ~97 s and paperless came back healthy. + +**Operator picture: coherent.** One event, one cause, one next step. + +### F-CRIT-2's size filter, confirmed live against REAL damage — **not manufactured** + +Unprompted, before any Phase B injection, the demo-hp agent logged: + +``` +17:37:59 WARN backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a +successful backup target=felhom-pbs vmid=9201 volid=felhom-pbs:backup/ct/9201/2026-07-28T05:31:14Z +size_bytes=1 reason="size 1 B is below the 1048576 B plausibility floor" +``` + +Independently confirmed on ep0's filesystem — the snapshot is a genuine aborted archive: + +``` +/mnt/pbs-datastore/ns/demo-hp/ct/9201/2026-07-28T05:31:14Z/ + catalog.pcat1.tmp_didx 0 bytes + root.pxar.tmp_didx 0 bytes <- no manifest, no index.json.blob + pct.conf.blob 425 bytes +``` + +This is F-CRIT-2 doing exactly its job on a real leftover phantom, in production, without being +asked. **PROVEN, not manufactured.** + +### Incidental proof: the local whole-guest archive restores and boots + +The restore-test that ran at 17:47:32 selected the local vzdump written by the B3 cycle minutes +earlier, restored it into scratch guest `990000`, booted it, and tore it down: + +``` +17:49:01 restore-test: scratch guest torn down vmid=990000 +17:49:01 backup: scheduled restore-test passed archive=local:backup/vzdump-lxc-9201-2026_07_28-17_43_05.tar.zst duration_s=89.19 +``` + +`pct list` afterwards shows no `990000` guest — **F-LEAK's fenced destroy is holding.** + +### B1 — contention, then the breaker — **COHERENT; the contention path does NOT swallow the transition** + +Forced by tightening the restore-test cadence to 120 s against its ~89 s duration, so the R-85 gate +is held most of the time. Both sides of the seam were captured in the same second: + +Agent (`agent-demo-hp.log`): + +``` +18:03:09 local-api: backup refused — a heavy operation is already in flight vmid=9201 requested_target=local busy=restore-test +18:03:09 local-api: backup refused — a heavy operation is already in flight vmid=9201 requested_target=felhom-pbs busy=restore-test +``` + +Controller (`ctrl-demo-hp.log`): + +``` +16:02:58 [quiesce] backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx] +16:03:09 [quiesce] tier local is BUSY — the agent refused the backup because a concurrent heavy + operation holds it. This is contention, NOT a failure: the tier stays due and retries in 15m0s +16:03:09 [quiesce] tier felhom-pbs is BUSY — … (contended for 0s) +16:03:09 [quiesce] unquiescing (last tier is busy — deferring to a later cycle): restarting 1 stack(s) +``` + +- **No breaker armed and NO event pushed for the contention** — F-A1's whole point, confirmed. +- The app was **restarted immediately** rather than left down for the restore-test's duration — + the second trap F-A1's comment names ("app thrash") is avoided. +- Note `felhom-pbs` was contended in a cycle where it would certainly have *failed* (its path is + blocked). The 409 precedes the attempt, so contention wins that cycle and the tier stays due — + correct, and it does **not** reset the breaker: `noteTierContention` never touches breaker state, + and the earlier armed failure count survives. + +**Operator picture: coherent, and correctly silent.** Contention produced no operator noise at all, +while the genuine failure minutes earlier produced exactly one event. + +### The breaker's retry discipline, over four polls + +``` +15:44:50 tier felhom-pbs has now failed 1 time(s) in a row — deferring … by 15m0s (cap 4h0m0s) +15:47:59 [DEBUG] tier felhom-pbs is in backoff after 1 consecutive failure(s) — not quiescing until 2026-07-28T15:59:50Z +15:52:58 [DEBUG] … not quiescing until 2026-07-28T15:59:50Z +15:58:00 [DEBUG] … not quiescing until 2026-07-28T15:59:50Z +``` + +Three skipped polls, **one** WARN when the breaker armed, **DEBUG** for each skip, **one** event. +The apps were not stopped once during the backoff. This is the exact behaviour R-88 was built for, +and the pre-R-88 bug (a stop/start cycle every 5 minutes forever) did not recur. + +### Both boxes independently, same result + +demo-felhom armed its own breaker at 16:00:00 with its own single event +(`whole_guest_backup_failed`, hub id 1916), against demo-hp's id 1915. Per-box and per-target +isolation both hold. + +### B4 / B5 — and what they turned up + +The demo-felhom rig (docmost made unstartable, PBS blocked, cadences compressed) produced a finding +rather than a clean pass — see **F-2** below. R-100's success anchor was confirmed correct along the +way: after the induced offsite failure on demo-hp, `last_run` advanced to the failing run while +`last_success` **stayed** at the last genuine success: + +``` +"last_run": "2026-07-28T16:01:56Z", +"last_status": "error", +"last_success": "2026-07-28T16:00:18Z", +``` + +That is precisely R-100's contract — a failure no longer refreshes the staleness clock. + +### Not reached + +B2's *combined* form (a phantom created during a quiesce that ALSO fails to restart an app) and B4's +full form (a stale anchor while the breaker is backed off, observed as one operator picture) were not +reached. Both need a longer run than this session had. + +--- + +## 6. Findings + +### F-2 — an app in a crash loop never alarms, on any channel — **DEFECT, HIGH** + +This is F-CRIT-1's exact shape in a state F-CRIT-1's fix does not cover. Campaign 8 closed +"an app that fails to restart after a quiesce never alarms" for `StateStopped`. A **crash-looping** +app is not `StateStopped`, and nothing sees it. + +**Action.** On demo-felhom, `docmost/docmost:0.95.0` was retagged to an image that exits immediately, +and the stack was started. (The first attempt — editing the compose image tag — was reverted by the +catalog sync within 3 minutes, which is how O-1 below was found.) + +**Observation.** docmost crash-looped continuously. Sampled every 10 s through the controller's own +`/api/stacks` — i.e. exactly the value the alarm logic reads, not `docker ps`: + +``` +16:05:53 docmost_state=restarting restartcount=14 +16:06:24 docmost_state=restarting restartcount=15 +16:07:04 docmost_state=restarting restartcount=15 +16:08:05 docmost_state=restarting restartcount=16 +16:08:25 docmost_state=restarting restartcount=16 … unbroken for the whole observation +``` + +The restart policy makes this **permanent**, not transient: + +``` +docker inspect docmost -> unless-stopped maxretry=0 restartcount=16 +``` + +And across that entire period: + +``` +dashboard page mentions of "docmost": 0 +dashboard warning words (Hiba|Figyel|nem fut): 0 +app_start_failed events pushed: 0 +hub events for demo-felhom: only whole_guest_backup_failed (the tier, not the app) +``` + +**The decisive evidence is the detector's own heartbeat.** F-OBS (v0.180.0) exists so that a stalled +dead-app check cannot hide behind silence — it prints a POSITIVE observable every ~10 minutes. Here +it proves the opposite of what it was built to reassure about: the detector is **alive, running, and +blind**. + +``` +15:59:17 [deadapp] check alive: 160 scans since boot, 4 deployed app(s) evaluated, 0 currently down +16:09:17 [deadapp] check alive: 180 scans since boot, 4 deployed app(s) evaluated, 0 currently down + ^^^^^^^^^^^^^^^^^ +docmost at 16:09:17: status=restarting restartcount=18 policy=unless-stopped +``` + +Twenty scans ran between those two lines. Every one of them looked at a permanently crash-looping +app and counted **0 down**. This is not an absent signal being read as health — it is the health +signal itself, positively asserting that nothing is wrong. + +**The causal chain, pinned to the code.** + +1. `aggregateState` (`internal/stacks/manager.go:753-756`) — *"Any restarting → stack is restarting"*. + This test runs **before** the partial/degraded logic, so one crash-looping container plus two + healthy ones yields `StateRestarting`, never `StateDegraded`. +2. `IsDownState` (`internal/stacks/manager.go:54-56`) is + `StateStopped || StateExited || StateDegraded` — **`StateRestarting` is absent**. +3. `classifyRunStates` (`cmd/controller/main.go:1292-1294`) computes + `down := stacks.IsDownState(st.State) && !userStopped && !quiesced[…]` → **false**. + +So the app never enters `dead` (no dashboard banner) and never enters `states` as `Down` (no +`app_start_failed`, hence no customer email and no hub event). There is **no crash-loop detector +anywhere in the controller** — `RestartCount` is read in exactly one unrelated place +(`internal/web/intermediary.go:105,343`), and `StateRestarting` appears in no alarm path. + +**Why this is not covered by the existing suppression logic.** R-97b's grace window is irrelevant +here: it expired at 16:01:56 and the app was still silent minutes later. The failure is not a +suppression that fails to expire — it is a state that is never considered "down" in the first place. + +**Expectation.** A container that Docker is restarting indefinitely is a dead app. Either +`StateRestarting` belongs in the down set (with a debounce so a normal restart does not alarm), or +the restart count must be watched. As written, `docker restart: unless-stopped` — the catalog's +standard policy — converts every crash into permanent silence. + +**Severity HIGH.** It is the same class the project has now shipped twice (F-CRIT-1, and the +`classifyRunStates` I1 comment in the workspace `CLAUDE.md` table), it affects the most common +real-world app failure mode, and it is *indefinitely* silent rather than delayed. + +**DEFECT.** The trigger (a retagged image) was manufactured; **crash loops are not** — a bad image, +a failed migration, a corrupt config or a repeatedly OOM-killed process all produce exactly this +state, and no cadence was compressed to make it happen. + +--- + +### O-1 — recovery-unit capture does not validate, but the catalog sync bounds the damage — **OBSERVATION, not a defect** + +**This was very nearly filed as a MEDIUM defect. The live system corrected me, and the correction is +the more useful result.** + +What was observed first. The docmost compose was edited to an unpullable image tag; the controller's +periodic refresh captured the recovery unit **7 minutes later** with no validation, so the live app +and its only restore point disagreed — and the restore point was the broken one: + +``` +15:50:07 [INFO] [backup] Recovery unit captured for docmost → …/backups/primary/docmost (images=3, …) +live: docmost|docmost/docmost:0.95.0|Up 4 hours (healthy) +manifest: ['docmost/docmost:0.95.0-campaign9-nonexistent', …] +``` + +That looked serious, because `ListRestorePoints` returns **exactly one** keep-side restore point per +app (`restore_points.go:14-18`, confirmed live) — an unvalidated refresh replaces the only one there +is. `CaptureRecoveryUnit` (`recovery_unit.go:78-130`) does indeed gate its write on a config +*checksum change* alone: no run-state check, no health check, no test that the pinned images resolve. + +**What actually happened next.** Three minutes later the catalog git-sync reverted the drift on its +own, and the recovery unit followed within the same pass: + +``` +15:53:17 [INFO] [sync] Updated docmost/docker-compose.yml +15:53:17 [INFO] [sync] Periodic sync: Sablonok frissítve — frissítve: docmost +15:53:17 [INFO] [backup] Recovery unit captured for docmost → …/backups/primary/docmost (images=3, …) +``` + +Verified afterwards on all three surfaces: + +``` +live compose image: image: docmost/docmost:0.95.0 +Tier-1 recovery unit pins: ['docmost/docmost:0.95.0', 'postgres:16-alpine', 'redis:7-alpine'] +Tier-2 copy's unit pins: ['docmost/docmost:0.95.0', 'postgres:16-alpine', 'redis:7-alpine'] +``` + +**Conclusion.** Compose drift is self-healing and the restore point follows it, bounded by the sync +interval — the poisoned window here was **~3 minutes**. The system does not leave you in the state I +manufactured, so **reporting this as a defect would have been reporting an artifact.** The absence +of validation in `CaptureRecoveryUnit` is real, but it is compensated, and the compensating control +was proven live rather than assumed. + +**The one residue worth an operator's attention** — narrow, and NOT demonstrated here: the sync makes +the *catalog* the source of truth, so a bad image tag **in the catalog** would propagate to every +box's only restore point unhealed, because sync would be restoring it rather than reverting it. That +is a catalog-quality concern owned by the catalog gates, not a controller defect. + +### F-3 — an interrupted offsite run leaves a lock the self-heal cannot reach, reported as "unknown" — **DEFECT, MEDIUM** + +**This answers Phase C item 8** ("is the repo usable after a killed run, or does it need an +unlock?"). It was found by accident — the campaign severed its own in-flight run — which makes it a +genuine interruption rather than a constructed one. + +**Sequence.** An offsite run started at 15:58:12 and took the repo's exclusive lock at 15:59:00. At +15:59:33 its transport was severed. The run died holding the lock. Every later run then failed: + +``` +16:15:48 [offbox] pre-push dump leg completed in 20.056s — snapshot pair is coherent +16:15:50 [ERROR] [offbox] backup failed after 23s: offbox repo unreachable: exit status 1: + unable to create lock in backend: repository is already locked exclusively by PID 14328 on demo-hp by root + lock was created at 2026-07-28 15:59:00 (16m49s ago) + the `unlock` command can be used to remove stale locks +``` + +**The remedy exists in the code and was never reached.** `resticStep` +(`internal/backup/offbox.go:634-648`) detects `repository is already locked`, escalates to +`unlock --remove-all`, and retries once — its comment states plainly that this is required because +"the recreated container has a new hostname, so restic can't verify the dead PID and won't treat the +lock as stale for ~30 min (the overnight-campaign C2 finding)". + +But the failure happens **before** any `resticStep` call. `ensureOffboxRepo` +(`offbox.go:676-696`) probes the repo first; that probe hits the lock, `classifyResticProbe` +(`offbox.go:77-93`) has no case for a lock error so it returns `"other"`, and the default branch +returns `offbox repo unreachable: …` — a fail-fast with **no unlock escalation**. The self-heal sits +behind a gate the failure never opens. + +**And the operator is told the wrong thing.** `ClassifyOffsiteFailure` likewise has no lock case, so +it falls through to `OffsiteFailUnknown`. Hub event 1919: + +``` +"A távoli mentés ismeretlen okból nem sikerült (23s): offbox repo unreachable: … repository is +already locked exclusively by PID 14328 … the `unlock` command can be used to remove stale locks" +``` + +"The remote backup failed **for an unknown reason**" — for a condition that is precisely known, +named in the attached detail, and self-healable by code already in the binary. + +**Impact.** The offsite tier stays **dead until a human intervenes**. Verified: it took a manual +`restic unlock --remove-all` to restore it (`evidence/phaseC/C8-unlock.txt`). + +**Reachability is ordinary, not exotic.** Any interruption of an in-flight run leaves this lock — a +controller container restart, an OOM, a network drop, or **a host reboot mid-backup, which is +Phase C item 6's scenario**. The code's own comment describes exactly this ("the C2 crash lock"). + +**Severity MEDIUM, argued rather than asserted.** Against HIGH: an error event *is* raised +immediately on the operator channel, its detail literally contains the remedy, and R-100's 48 h +`offsite_stale` is a second backstop, so this is loud-but-mislabelled rather than silent — unlike +F-1 and F-2. Against LOW: a backup tier is down until manual intervention, and the headline actively +misdirects triage. + +**DEFECT.** No cadence compression was involved in the mechanism; only the interruption was +deliberate. + +--- + +### O-2 — a hand-run `docker compose up -d` starts a Felhom app WITHOUT its secrets — **OPERATIONAL, worth a runbook line** + +Encountered during recovery, by me. Restoring docmost with a plain +`cd /opt/docker/stacks/docmost && docker compose up -d` produced a container that crash-looped on: + +``` +{"minLength":"APP_SECRET must be longer than or equal to 32 characters","isNotEmpty":"APP_SECRET should not be empty"} +``` + +Secrets are injected by the controller's `stackEnv` at start time (`StartStack`, +`internal/stacks/manager.go:912-948`); they are **not** in a `.env` file beside the compose. A manual +`up -d` therefore recreates the container with empty secrets — turning a healthy app into a broken +one, and looking exactly like the fault you were trying to fix. Recovery must go through the +controller (`POST /api/stacks//restart`), which is what finally fixed it. + +--- + +## 7. Phase C / D / E — reached, and not reached + +| # | item | status | +|---|---|---| +| C6 | host reboot mid-backup | **NOT REACHED.** Approved by the operator; not attempted. Its most likely consequence was nonetheless observed by another route — see F-3, which is exactly the state a reboot mid-backup would leave. | +| C7 | three-way concurrency (backup + restore-test + GC) | **PARTIALLY.** Backup-vs-restore-test contention was proven (B1); GC was not added as a third party. | +| C8 | restic transport interrupted mid-run — is the repo usable after? | **ANSWERED: no, it needs an unlock.** See **F-3**. | +| D9 | Scenario C — never-succeeded Tier-2 on a throwaway app | **NOT REACHED.** No throwaway app was created. | +| D10 | F-DIAG's failure classes + no credential/repo-URL/raw-error passthrough | **PROVEN for the surfaces that leave the box** — see below. | +| D11 | R-100's threshold — see `offsite_stale` actually fire | **NOT REACHED**, and the approved method turned out not to exist — see below. | +| E12 | F-HUB `SQLITE_BUSY` reproduction | **NOT REACHED** (explicitly lowest priority). | + +### D10 — the sanitiser, proven on the exact case that defeated its first version + +Six distinct causes map to six distinct Hungarian messages (`offbox.go:201-208`). The transport case +was induced live by blocking port 23 from inside the guest, and the resulting hub event's +`details_json` was: + +``` +"A távoli tárhely nem érhető el (hálózat vagy bejelentkezés) (51s): offbox repo unreachable: + exit status 1: subprocess ssh: ssh: connect to host port 23: Network is unreachable + Fatal: unable to open repository at unable to start the sftp session, error: EOF" +``` + +- the correct distinct class, not a generic string; +- the **bare hostname** — the precise shape the sanitiser's first version leaked on, per its own + comment (`offbox.go:170-181`) — is replaced with ``; +- the full `sftp:user@host:/path` repo URL is replaced with ``. + +**Nothing raw leaves the box.** The hub *report* carries no error text at all: `OffboxReportStatus` +(`offbox.go:1149-1174`) has no error field by construction. + +**One precise caveat, not a finding.** The `last_error` persisted in the guest's `settings.json` is +raw, and the customer's own `/backups/remote` page renders it verbatim under „Utolsó hiba:", +including the bare hostname and the `sftp:` URL. That is local-only and discloses nothing to that +viewer: the same host, user and repo path are displayed two rows above as *configuration* on the +same page. Sanitisation is applied on the paths that leave the box, which is where the contract puts +it. + +### D11 — the approved compression does not exist as a knob + +Phase 0 approved compressing the hub's `staleAfter` for this item. It is **not configurable**: +`cmd/hub/main.go:552` calls `monitor.NewOffsiteChecker(dataStore, 0, …)` and `0` selects the +compile-time `defaultOffsiteStaleAfter = 48 * time.Hour` (`internal/monitor/offsite.go:41,60-62`). +Compressing it would have required editing, building and deploying hub code — which this campaign is +forbidden to do ("no production code ships", "no inline fixes"). **Reported rather than worked +around**, per the standing rule that a recommendation not followed gets a line saying why. + +The alternative that needs no code change — ageing the controller's reported `last_success` anchor +past 48 h and letting the hub judge it at its real threshold — was identified but not executed. +That is the recommended method for a future run. + +Along the way R-100's producer half was confirmed correct twice, at real settings: after each +induced failure `last_run` advanced while `last_success` held. + +``` +"last_run": "2026-07-28T16:15:50Z", +"last_status": "error", +"last_success": "2026-07-28T16:00:18Z", +``` + +--- + +## 8. Phase F — recovery + +Full log: `evidence/PHASE-F-recovery.log`. Every revert was verified with a positive observable. + +| reverted | verification | +|---|---| +| agent cadences, both hosts | `cadences restored -> 0 302400 [604800]`; `systemctl is-active` = `active` | +| backup windows, both boxes | set back to `02:30` through the same endpoint that changed them | +| PBS path block, both hosts | `pvesm status` shows `felhom-pbs … active` on both | +| offsite transport block (guest) | ssh reaches the storage box again (host key exchanged) | +| docmost broken image | real `0.95.0` re-pulled; **restarted via the controller** after O-2 | +| stale restic lock | `unlock --remove-all`; `list locks` empty; 8 snapshots readable | +| campaign scratch files | `0` remaining `c9-*` on both guests; host `/tmp` cleared | +| collectors | all 7 stopped; `0` remaining remote tails | + +**The offsite tier was not merely unblocked but proven working again** — a full run through the real +endpoint after the unlock: + +``` +"last_run": "2026-07-28T16:20:57Z", +"last_status": "ok", +"last_success": "2026-07-28T16:20:57Z", +"last_duration": "1m35s", +"snapshot_count": 8 +``` + +**Final fleet state — both boxes fully healthy:** + +``` +demo-felhom : docmost + docmost-postgres + docmost-redis, immich (×4), calibre-web, + bookstack (×2), filebrowser, samba, cloudflared, traefik — ALL healthy +demo-hp : paperless (×3), filebrowser, samba, cloudflared, traefik — ALL healthy +agents : active on both hosts +iptables : 0 campaign9 rules on either host and in either guest +scratch VMs : none in the 990000 band on either host +ep0 : /srv/pbs-felhom rollback copy intact (13G); datastore 14% used +peti-felhom : never touched +``` + +**One benign residue, stated for completeness.** The in-memory R-88 breaker on each box still holds +a `felhom-pbs` consecutive-failure count from the induced failures. Its `until` timestamps are long +past, so it blocks nothing; the only effect is that a *next* failure would back off 30 m instead of +15 m. It clears on the next successful whole-guest backup, and by design it also clears on any +controller restart (`breaker.go` — deliberately not persisted). Nothing was done about it because +clearing it would have meant taking another app outage for no benefit. + +--- + +## 9. Where this stopped, and why + +**Stopped at the end of Phase B, with Phase D item 10 additionally completed**, then Phase F. + +Phase A — the campaign's stated headline and its whole justification — is complete and proven. Phase +B produced two of its five combinations cleanly, plus a HIGH finding. Phase C item 8 was answered by +an accident that turned out to be a genuine defect. Phase D item 10 is proven; D11's approved method +did not exist. + +Phase C item 6 (host reboot mid-backup) was deliberately **not started**. It is a large new fault +class against two boxes that are remote until ~08-02, and beginning it would have meant either +rushing it or leaving the fleet in an unknown state. The campaign's own rule — *stop cleanly at a +phase boundary; a broken test node is fine, an unreported one is not* — made the choice: finish the +recovery properly rather than start a fault I could not supervise to completion. F-3 partly covers +its most likely consequence anyway, since an interrupted run is exactly what a reboot mid-backup +produces. + +--- + +## 10. Findings, ranked + +| # | finding | severity | class | +|---|---|---|---| +| **F-1** | the Tier-2 restore button reports success for apps it can never restore (BookStack, Docmost — half the fleet's Tier-2 apps); takes a real outage; message is indistinguishable from a genuine "nothing was missing" | **HIGH** | DEFECT | +| **F-2** | an app in a crash loop never alarms on any channel — `StateRestarting` is in no down-set, and the dead-app heartbeat positively reports "0 currently down" while it loops | **HIGH** | DEFECT | +| **F-3** | an interrupted offsite run leaves an exclusive lock the existing self-heal cannot reach; tier dead until manual `unlock`; reported to the operator as "unknown reason" | **MEDIUM** | DEFECT | +| O-1 | recovery-unit capture validates nothing, but the catalog sync reverts drift within ~3 min and the unit follows | — | OBSERVATION (nearly mis-filed as a defect) | +| O-2 | a hand-run `docker compose up -d` starts a Felhom app without its secrets | — | OPERATIONAL | + +**Mechanisms confirmed working, on live hardware:** R-82's one-quiesce rule under mixed outcomes; +R-88's breaker (per-target, edge-triggered, one event, no app thrash); F-A1's contention deferral +(no breaker, no event, prompt restart); F-CRIT-2's size filter (against a real 1-byte phantom, not a +manufactured one); R-100's success anchor (twice); F-DIAG's sanitiser (on the exact bare-hostname +case that defeated its first version); F-OBS's positive observable (which is what made F-2 +provable); F-LEAK's fenced destroy (no leaked scratch guests across ~10 restore-tests). + +--- + +## 11. What came off the never-proven list — and the honest residue + +**Off the list:** + +- Tier-2 cross-drive restore of **ordinary app data** — the customer's actual button (A1) +- Tier-2's "existing files are NOT modified or deleted" promise, both halves (A1) +- Tier-2 restore after **total loss** of an app's data directory (A3) +- Restored data **usable by the app**, proven through the app's own API (A1/A3) +- Tier-1 recovery-unit restore is a **distinct** path, and it executes end-to-end (A2) +- The six recent fixes **do** compose: R-82 + R-88 + F-A1 + F-CRIT-2 + R-100 produced one coherent + operator story in every combination reached +- Whether an interrupted restic run leaves a usable repo (C8 — it does not) + +**Still not proven — the residue:** + +- **Tier-1 content recovery after real loss.** A2 ran against an intact app; A3 used Tier-2. Nobody + has yet destroyed an app's data and recovered it *from the recovery unit*. This is now the single + most valuable unproven item. +- **Host reboot mid-backup** (C6) — untested, and F-REBOOT proved the guest-reboot sibling was real. +- **Three-way concurrency** with GC as the third party (C7). +- **Scenario C** — a never-succeeded Tier-2, live (D9). +- **`offsite_stale` actually firing** (D11) — never observed. Needs the anchor-ageing method above, + not the `staleAfter` compression that was approved but does not exist. +- **F-HUB** `SQLITE_BUSY` characterisation (E12). +- **Whether F-1 and F-2 have siblings**: both are "a surface reports health it did not establish". + Nothing in this campaign searched systematically for others. + diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index e1ad89b..bc512c3 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -28,6 +28,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **F-A1** | ~~A restore-test in flight made a healthy backup report as FAILED (HTTP 409 read as a tier failure): breaker armed + operator emailed, on both boxes~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → **0** operator emails, real failure → **1** | — | | **R-100** | ~~A restic offsite tier that fails every night never goes stale on the hub — `isStale` counted from `LastRun`, which the controller writes unconditionally on failure~~ | **SHIPPED + PROVEN-LIVE** (controller v0.181.0 + hub v0.80.0, 2026-07-28) | — | Anchored on a new `last_success`. **Severity corrected during Phase 0: this was NOT a silence** — `backup_failed` does fire nightly and reaches the operator (live DB: 5 sends). The real defect is **defeated defence in depth**: the hub-side *pull* net was anchored on a field the failing controller keeps refreshing, so it could not compensate for a lost *push* (cf. F-HUB). Live on demo-hp: induced failure → `last_run` advanced 11:25:48Z, `last_success` **held** 11:24:20Z; demo-felhom healthy → anchor advanced. Legacy degrade logged once per customer, live | — | | **R-101** | ~~Tier-2 `LastRun` is written on failure and rendered to the customer as „Legutóbbi másolat" — including in the restore confirm dialog~~ | **SHIPPED + PROVEN-LIVE** (controller v0.182.0, 2026-07-28) | — | `CrossDriveBackup.LastSuccess` + `SuccessTracked`; the dialog names the last **successful** copy and discloses a failed newest attempt. Legacy rows migrate truthfully on first touch (an `ok` row adopts its time; an `error` row seeds nothing) — without the marker all 7 fleet rows would have flipped to „Még nincs sikeres másolat" on deploy. **Part 2:** the three `record*` sites rebuilt the whole struct; replaced by `tier2Update` (copy-and-overlay, safe by construction) — the naive fix would have had `recordTier2Failure` CLEAR the anchor. Live on demo-hp, rendered dialog read in both states | — | +| **C9-F1** | Tier-2 „Fájlok visszaállítása" is offered for apps whose Tier-2 copy has **no restorable file leg** (BookStack, Docmost = 2 of the 4 Tier-2 apps on demo-felhom). Clicking it stops the app, restores 0 files, and reports **„Nincs hiányzó fájl — minden fájl megvan a helyén."** — indistinguishable from a genuine "nothing was missing", on every click, forever, while 156 MB of that app's data sits unread in `recovery-unit/` in the same copy | **READY (HIGH)** | — | Either don't render the button when the copy has no `hdd/`+`userdata/` leg, or make the message distinguish "nothing missing" from "nothing restorable here". `tier2_restore.go:101-104` reads only those two subtrees; `tier2_restore_test.go:219` pins 0-files-as-success and explicitly accepts the outage; `tier2_honesty_test.go` Scenario D asserts the button MUST render for exactly BookStack's shape | CC | +| **C9-F2** | An app in a **Docker crash loop never alarms on any channel** — no dashboard banner, no `app_start_failed`, no customer email, no hub event. `aggregateState` returns `StateRestarting` (`manager.go:753-756`, checked before the degraded branch) and `IsDownState` (`manager.go:54-56`) excludes it, so `classifyRunStates` computes `down=false`. With `restart: unless-stopped` this is **permanent** silence. F-CRIT-1's fix covers `StateStopped` only | **READY (HIGH)** | — | Add `StateRestarting` to the down set with a debounce, or watch `RestartCount`. Proven live on demo-felhom: 9+ min crash loop, restartcount 18, while the F-OBS heartbeat printed **"180 scans since boot, 4 deployed app(s) evaluated, 0 currently down"** | CC | +| **C9-F3** | An **interrupted offsite run leaves an exclusive restic lock the self-heal cannot reach**: `resticStep` (`offbox.go:634-648`) has `unlock --remove-all`, but `ensureOffboxRepo`'s probe fails first, `classifyResticProbe` (`offbox.go:77-93`) has no lock case → `"other"` → fail-fast. Tier dead until a human unlocks; `ClassifyOffsiteFailure` likewise has no lock case so the operator is told **„A távoli mentés ismeretlen okból nem sikerült"** for a precisely-known, self-healable condition | **READY (MEDIUM)** | — | Add a lock case to both classifiers and let the probe path escalate to `unlock --remove-all`. Answers Phase C item 8: the repo is NOT usable after a killed run. Cleared manually this run; tier proven working again (`ok`, 1m35s). Reachable by any interruption — container restart, OOM, **host reboot mid-backup** | CC | | **F-DIAG** | ~~Four distinct offsite failure causes collapse into two operator-visible strings~~ | **SHIPPED** (controller v0.182.0, 2026-07-28) | — | `ClassifyOffsiteFailure` → quota / orphaned / no_repo / no_units / transport / **unknown**, each with its own Hungarian message. Unclassifiable says so rather than being folded into a neighbour. **Secrets:** the old message was a raw `err.Error()` passthrough carrying `sftp:@:`; redaction is now by the target's **actual** host/user/path (a first regex-only attempt leaked on a bare hostname and its own test caught it). Unit-proven; **not** yet exercised by a live offsite failure of each class | — | | **F-OPS** | ~~A manual `pct restore` inherits the source guest's bind mounts — during a real DR, on a different host, under pressure~~ | **DOCUMENTED** (2026-07-28) | — | `documentation/runbooks/RUNBOOK-manual-guest-restore.md`: which `mpN` are volumes vs host binds, the `mp9` source-VMID trap (it can bind **another guest's bootstrap credentials**), strip-and-re-add before first boot, and a positive pre-start verification. Docs only by design — the agent already neutralises binds on its own restore paths, and a second implementation would drift | — | | **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — |