felhom.eu/documentation/architecture/capability-map-detail/C.md

Capability map — detail, section C. Protection & recovery (the product promise)

What this file is. The long text of every row of section C of 00-capability-map.md, moved here verbatim on 2026-10-12 (R-288, R-290 folded in; 09 §3 decisions 214 and 229) from the map at commit 1b45bb4d. Each entry is one former table row, its cells laid out one per paragraph; the cell text itself was not edited. The map row carries the capability's ONE status word and its dated evidence pointer; this file keeps the argument, the corrections and the history. Where this text and the map row disagree on the status, the map row is current (see the map header for the words that were normalised).

A new correction goes under the entry it corrects, dated — never into the map row.

C1 — Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded DEGRADED rather than silently normal (installer Case A/B)

Map row: C1 · former map line 145.

Scenario (as written): Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded DEGRADED rather than silently normal (installer Case A/B)

Components (as written): agent v0.113, host-install v1.22.0

Status (as written): PROVEN-LIVE

Evidence (as written): E2D-fresh-vm-2026-07-29 C1 (real 1.22.0 install, rc=0, Day-0 provision SUCCESS) + C2 (both DEGRADED lines verbatim, local_backup_target=local, install did not abort)

Gap / roadmap (as written): Case A (a second drive already present at install) has never fired naturally — only Case B has

---

C2 — Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes

Map row: C2 · former map line 146.

Scenario (as written): Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes

Components (as written): controller v0.118

Status (as written): PROVEN-LIVE

Evidence (as written): CAMPAIGN-2 T-BAK-FULL (pg+mariadb autodiscovered); atomicity CAMPAIGN-6B P4 + CAMPAIGN-6E B1/B2 (SIGKILL mid-write → only .tar.tmp touched, last-good byte-unchanged); DB restore CAMPAIGN-6D P-FAB

Gap / roadmap (as written): (Cited CAMPAIGN-3 F7 is the finding of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 DB replay route → 07-backup-architecture.md §8 row 3

---

C3 — Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary

Map row: C3 · former map line 147.

Scenario (as written): Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary

Components (as written): controller v0.135

Status (as written): PROVEN-LIVE

Evidence (as written): CAMPAIGN-6E-2026-07-15 (P-TIER2 deep-4 PASS), CAMPAIGN-6C

Gap / roadmap (as written): Route + RTO → 07-backup-architecture.md §8 rows 1, 2, 3b, 4, 5. R-403 (controller v0.230.0, 2026-08-31) changes NO status on this row and that is stated rather than left ambiguous: the nightly copy now refuses to replace a complete unit package with an empty one (07 §8.2), which removes a way the route could be DESTROYED between uses — it does not change what the route can be relied on for, and every leg it covers is the same one it covered yesterday. Proven live on demo-hp: the same state that deleted 120 082 104 B on v0.229.0 preserved all 7 files on v0.230.0. R-102 CLOSED 2026-08-31 (controller v0.229.0): the copy's recovery-unit/ mirror is now restorable — „Teljes visszaállítás a másolatból" / POST /backup/tier2/unit-restore — and was proven live on demo-hp with the primary unit moved aside (audits/DRILL-r102-tier2-unit-2026-08-31/, §8 row 3b, 28.65 s). R-103 closed with it: the refusal that used to name a button on another page now offers the action on the row itself

---

C4 — Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping

Map row: C4 · former map line 148.

Scenario (as written): Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping

Components (as written): controller v0.134, agent, hub

Status (as written): PROVEN-LIVE

Evidence (as written): CAMPAIGN-6D-2026-07-15 (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); VALIDATION-offbox-storagebox-2026-07-09 (byte-perfect round-trip)

Gap / roadmap (as written): Raw-data quota (SP-1) + retention regrouping (SP-2) are SPIKE-restic-snapshot-shape dry-run verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. Reinstall-continuity (controller v0.142.0, 2026-07-17): a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (wrong password or no key found) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. The live leg FIRED on its own during the 2026-07-18 rehearsal (tests/VALIDATION-n100-rehearsal-2026-07-18.md, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard classified it, pushed offbox_repo_orphaned, skipped the run and showed the card (16:58:14) rather than nightly-spamming a raw restic error; the operator-confirmed reset then moved the repo aside (never deleted) to .orphaned-20260718 and re-initialised (16:59:26→16:59:32), and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — the finding is that it had to fire at all (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). DIAGNOSE-offbox-repo-orphaned-2026-07-17 Route + RTO → 07-backup-architecture.md §8 rows 4, 10, 12, 15 (incl. the R-95 delete exposure and the R-104 stale-lock defect). 2026-08-04 (R-193/R-197, audits/SPIKE-offsite-credential-recovery-2026-08-04.md) — the row is NOT overclaiming and the guard is not the gap; the CADENCE is. This row already recorded that a recreated data volume orphans the repo, and the spike confirms the mechanism at source: WriteOffboxSecrets (offbox.go:392) mints a fresh 256-bit repo password whenever <DataDir>/offbox/repo_password is absent, and no automatic path ever consults the escrowed one — InjectOffboxPassword has exactly one caller, a web form a human pastes into. What was NOT recorded is that this now fires on an ordinary, planned, unattended guest rebuild, on every box: measured on BOTH demo boxes 2026-08-03/04 by comparing host_escrow.restic_pw_sha256 against host_escrow_superseded.restic_pw_sha256 (demo-hp 8e03eddf…→8a9e33aa…, demo-felhom 48741892…→c60c8bc7…), orphaning 15 snapshots / 40.9 MB and 36 snapshots / 1.14 GB respectively. demo-felhom is the important half: it kept its DELIVERY (a stale staged secret restored the target in 76 s) and lost its REPOSITORY anyway, with nothing marking the escrow stale for 13 h — escrow_stale is wired to ReissueCredentials, the one path that does NOT change the repo password (R-196), and absent from the rebuild path that does. Status unchanged: the classify-and-move-aside guard remains PROVEN-LIVE and correct, and is predicted (not yet measured) to refuse the 2026-08-05 run on both boxes rather than start a silent fresh history

---

C5 — Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live

Map row: C5 · former map line 149.

Scenario (as written): Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live

Components (as written): controller v0.134/134.1/135

Status (as written): PROVEN-LIVE (2026-07-20)

Evidence (as written): CAMPAIGN-6D accept legs (immich end-to-end from offsite alone)

Gap / roadmap (as written): 2026-07-19: audits/DIAG-immich-restore-2026-07-19.md finds no offsite path loads a DB dump — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "immich end-to-end from offsite alone" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. RULED 2026-07-19 (Viktor): 6D's destruction hit the FILE TREE ONLY — the database survived in its named volume (immich_postgres_data is a named volume in both the v2 and v3 template eras), so "immich end-to-end from offsite alone" overclaimed scope: the file half was proven, the DB half was never destroyed and therefore never restored. Row downgraded PROVEN-LIVE → PARTIAL, scope-corrected. Evidence: audits/DIAG-immich-restore-2026-07-19.md (no offsite path could replay a DB at all) + the P-FAB destructive re-import (the proven-replay evidence, on the LOCAL path). 2026-07-19, controller v0.148.0: the DB half now exists in code (R-43 + R-44) and its replay reached a live box — but round 2 found it aborts against a running app (audits/DIAG-immich-restore-round2-2026-07-19.md, H4: the replay races immich's own schema repair; clip_index recreated by the app 2 s before the dump's CREATE INDEX). 2026-07-20, controller v0.153.0: H4 IS CLOSED (R-47) — both restore paths now replay into a DB-ONLY window (StartStackServices brings up the database service alone; the app starts only after the replay exits 0), with a fail-closed refusal when a dump has no identifiable DB service. (The earlier note here said "closes in v0.149" — that was wrong: v0.149.0 was the F3 dashboard BackupStatus fix. R-47 shipped in v0.153.0.) 2026-07-20: the clean run HAPPENED — endpoint-level supervised reconstitute of immich from snapshot 49e7cb46 (the very snapshot that aborted in round 2): stop → DB-service-only start → replay rc-0 → full start, no already exists, operation reported SUCCESS, immich's own DatabaseService logged No schema drift detected twice, 11 assets active, 4/4 containers healthy, 231 public indexes. Operator confirmed the immich timeline renders correctly after the reconstitute (screenshot held, 2026-07-20). Evidence: felhom-controller/REPORT.md §4b. 2026-07-20, LATER THE SAME DAY — the destructive drill RAN and the row now earns PROVEN-LIVE. The operator deleted the photos in immich own UI and emptied the trash (the step whose absence makes a drill prove nothing — the round-1 lesson), then restored through the customer-facing UI. 40 file(s) placed against the 6 of the earlier non-destructive run — the files were really gone and really came back — plus 1 DB dump replayed rc-0, 11 assets active, No schema drift detected, timeline confirmed by the operator. This is the destroy-then-recover proof the 6D downgrade asked for, and it was taken through the customer own buttons, not endpoint shortcuts. Evidence: felhom-controller/REPORT.md 4e Route + RTO → 07-backup-architecture.md §8 rows 3, 4 — the matrix also records that no offsite action unpacks the named-volume tars it captures (→ R-107)

---

C6 — Manual .fab export/import: class-scoped capture, browser up/download, tunnel-proof chunking

Map row: C6 · former map line 150.

Scenario (as written): Manual .fab export/import: class-scoped capture, browser up/download, tunnel-proof chunking

Components (as written): controller v0.125/128/130/136

Status (as written): PROVEN-LIVE

Evidence (as written): CAMPAIGN-6D P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking CAMPAIGN-6B P2 (100 MiB via real CF edge, 120 MiB→413)

Gap / roadmap (as written): Chunking proven at the real CF edge via curl --resolve; the rendered browser file-picker upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B finding; fix verified in 6D

---

C7 — Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification

Map row: C7 · former map line 151.

Scenario (as written): Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification

Components (as written): agent v0.75/0.76, PBS

Status (as written): PROVEN-LIVE

Evidence (as written): CAMPAIGN-2 T-P9-DESTROY-RESTORE (whole-guest pct restore of 9201 → running+healthy) + T-PBS-VERIFY (verify_state: ok, 13 snapshots); DRILL-GL6-2026-07-08 Phase 0d (restore-test mount_parity: ok)

Gap / roadmap (as written): (Cited VALIDATION-newbox-restore is offbox restic file-restore, wrong tier — corrected.) Real offsite guest-loss round-trip still R1-blocked → S5 DR drill Route + RTO → 07-backup-architecture.md §8 rows 6, 8, 9 — measured 84–112 s local / 1101 s PBS into a scratch guest; a restore to a DIFFERENT host is unmeasured

---

C8 — PBS-DR secret self-heal on reused-peer re-provision

Map row: C8 · former map line 152.

Scenario (as written): PBS-DR secret self-heal on reused-peer re-provision

Components (as written): hub v0.56

Status (as written): IMPLEMENTED

Evidence (as written): hub v0.56.0 (pbsdrheal/reconciler.go, RestageHostPBSSecret, all §10 red-proofs); SPIKE-pbsdr-selfheal-2026-07-15 (root cause)

Gap / roadmap (as written): Reconciler is scoped to one host (PBSDRHEAL_ONLY_HOST), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending

---

C9 — Box survives an unattended app or guest-network failure (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired

Map row: C9 · former map line 153.

Scenario (as written): Box survives an unattended app or guest-network failure (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired

Components (as written): controller v0.156.0→v0.190.0, agent v0.92.1

Status (as written): PROVEN-LIVE (2026-07-21; boot-orphan leg rebuilt and re-proven 2026-08-02 — 6 of 6 hard resets, repeat count cited per N.5)

Evidence (as written): All three legs exercised on the live demo box, operator-present, in one session — felhom-controller/REPORT.md + felhom-agent/REPORT.md (2026-07-21). Dead primary: docker stop immich-server 12:50:40 CEST → degraded 13 s later → exactly one app_start_failed + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). Boot orphan: pct reboot 9201 → [bootrecon] 1 boot-orphaned app(s) found: [bookstack] → started in 1 attempt of 2, zero alerts (success inside the boot grace is silent); StartedAt proves Docker's unless-stopped did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. Dead DHCP client: deliberate replay of the incident — kill -9 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; the tunnel never dropped (cloudflared Up 29 hours), i.e. the outage was prevented rather than merely observed

Gap / roadmap (as written): The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21). For a drive-backed app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own existing-Exited vs absent predicate: a UI Stop is compose down and removes them. So "a stopped app stays stopped" now holds for drive-backed apps too — PROVEN LIVE 2026-07-21 (TASK-F Part 3, operator-present). immich was stopped through the real UI endpoint (compose down → 0 containers), calibre-web and bookstack left running, then pct reboot 9201: the gate recreated calibre-web and logged 1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose; immich came back stopped, where the identical fixture had brought it back running hours earlier. Zero alerts, ~15 s to steady state. The static-guest half of the network leg stays deliberately out of scope → R-50. 2026-08-02 — the boot-orphan leg now rests on a RECORDED signal, not an inference (R-166, controller v0.189.0). Both the R-52 sweep and the R-55 gate above decided "the customer stopped this" from zero containers, which is also what a power cut mid-compose and an interrupted deploy leave behind — so two real faults were read as deliberate stops and stranded silently (R-157 mechanism B). The customer's intent is now written to app.yaml (desired_state) by their own action and read directly. PROVEN-LIVE on 9201 in three flows: a UI Stop persisted stopped and survived a controller restart with the app still down and NOT listed as a candidate; an app recorded running whose containers were removed out-of-band was recovered by name ([bootrecon] 1 boot-orphaned app(s) found: [calibre-web] → started in 1 attempt) — the case that was invisible before; and a legacy app.yaml with no field was skipped exactly as before and was never inferred to be stopped. Interrupted app-data operations are covered separately and are NOT proven-live — backup.AppStopGuard restarts apps left stopped by a killed volume dump / offsite reconstitute / .fab export, and that leg is unit-proven + red-proofed only (killing the controller mid-backup on a live box was not exercised): IMPLEMENTED, not PROVEN-LIVE. R-170 and R-171 closed the same day (controller v0.190.0). R-157 mechanism A — the sweep observed ONCE at T+5 s, while docker was still restoring, and never re-checked (3 of 6 hard resets). It is now a settle-then-sweep window: sample every 5 s, settled after 3 identical samples, ONE sweep at the end, terminating on settled or a 50 s budget (sized so settle+budget+one retry stays inside the 90 s dead-app grace; a test rejected 60 s at 95 s). Repeat count, per this map's own rule: 6 of 6 hard resets on the shipped build brought every app back, and an app the customer had stopped stayed down in all 6 (window settle times 10/40/10/10/15/15 s — i.e. it routinely waited 2–8× longer than the old fixed 5 s). A same-app before/after on one box is the sharpest evidence: the pre-fix window logged no boot-orphaned apps for calibre-web at 18:08:35; the fixed one found and recovered it at 18:18:50. R-170 — shouldRecreateOnBoot now reads intent too, so the two boot gates agree; proven live in one reboot (calibre-web running+zero containers recreated, immich stopped left alone). R-171 — a regression v0.189.0 introduced, found by reading the diff and CONFIRMED on hardware before any fix was written: the sweep started an app whose drive was absent, burned both attempts and raised a false dead-app alarm. The write hazard was blocked only by an ACCIDENTAL filesystem permission (host-root-owned mountpoint + unprivileged guest) that no code owns and no test pinned — which is why it was fixed rather than noted. New fail-safe bootrecon.StartGate (cannot determine ⇒ do not start), also covering quiesce and in-flight app-data operations. One defect in the fix itself, found by live validation and not by review: the window sampled the Manager's 10 s-refreshed cache, so "settled" could mean "the cache did not update"; sampleBootFleet now refreshes first. Evidence: audits/DIAG-bootrecon-drive-absent-2026-08-02.md, felhom-controller/REPORT.md

---

C10 — Crash/power-loss mid-backup/mid-migration → self-heal on next run

Map row: C10 · former map line 154.

Scenario (as written): Crash/power-loss mid-backup/mid-migration → self-heal on next run

Components (as written): controller, agent

Status (as written): PROVEN-LIVE

Evidence (as written): CAMPAIGN-6D P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); CAMPAIGN-6E B1-B3

Gap / roadmap (as written): (Cited CAMPAIGN-2 T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-migration crash→self-heal is the weakest sub-claim (P5-REST is mid-backup)

---

C11 — An app can be withdrawn from the catalog without orphaning the customers running it (available / hidden / abandoned)

Map row: C11 · former map line 155.

Scenario (as written): An app can be withdrawn from the catalog without orphaning the customers running it (available / hidden / abandoned)

Components (as written): controller v0.158.1, catalog metadata

Status (as written): PROVEN-LIVE (2026-07-21)

Evidence (as written): TASK-F Part 1. Verified on 9201 through the real endpoints: lifecycle: abandoned arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct POST /api/stacks/plant-it/deploy → HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."; the app page carries the permanent notice and offers no Telepítés button. felhom-controller/REPORT.md (2026-07-21)

Gap / roadmap (as written): Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read Elavult and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57

---

C12 — Box survives a site/network change (relocation, different subnet, DHCP re-lease) with the control plane intact

Map row: C12 · former map line 156.

Scenario (as written): Box survives a site/network change (relocation, different subnet, DHCP re-lease) with the control plane intact

Components (as written): agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap

Status (as written): PROVEN-LIVE (2026-07-25)

Evidence (as written): R-50 SHIPPED and deployed to the whole fleet. The control plane now rides a host-internal, portless island bridge (vmbr9, 169.254.253.1/30↔.2/30) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original bind: cannot assign requested address daemon-death) + cold-reboot survival (SPIKE-island-bridge-2026-07-25.md), the migration runbook run verbatim (RUNBOOK-island-migration.md), a fresh provision auto-attaching the island net1 (A4), and the live migration of both demo boxes (demo-hp + demo-felhom, 2026-07-25) — island /storage HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), apps served throughout (0 container restarts), hub reporting 0.96.0. Origin: audits/AUDIT-vacation-remote-ops-2026-07-20.md — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box

Gap / roadmap (as written): Fleet: DONE. Remaining: R-74 — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped

---

C13 — The customer is warned BEFORE a filesystem fills — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered

Map row: C13 · former map line 157.

Scenario (as written): The customer is warned BEFORE a filesystem fills — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered

Components (as written): controller v0.191.0/.1/.2, hub v0.89.0 (R-167, decision D-c)

Status (as written): PROVEN-LIVE (2026-08-02)

Evidence (as written): audits/SPIKE-r165-mp1-merge-2026-08-02.md (context) + felhom-controller/REPORT.md. Exercised on guest 9201 against a REAL filesystem (/mnt/sys_drive filled with fallocate): disk_warning at 90% used / 4.7 GB free → hub notification_log customer | disk_warning | sent with the dynamic Hungarian rendered; grown to 1.7 GB free → disk_critical → customer | sent; file removed → critical → ok … cleared silently, re-armed and the persisted state emptied. Exactly two events across three boots — the boot in between produced none, which is the edge trigger holding

Gap / roadmap (as written): Nothing warned before this. The only prior signal was the healthcheck's generic health_degraded at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. The two event types already existed with NO PRODUCER (disk_warning/disk_critical: allowlisted, copy'd, in DefaultEnabledEvents, checkbox'd) — the sixth built-but-never-wired instance here; this ships their producer rather than a seventh near-duplicate type. Two threshold terms, whichever trips first, and the live proof vindicated the design: the critical crossing fired on the FREE-BYTE term (1.7 GB) at only 91% used — a percentage-only rule would have missed it. The hub's generic customerMessages entries were REMOVED, because FormatCustomerEmail prefers the entry over the message and would discard the label and figures. Known gap → R-177: there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait

---

C14 — A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed

Map row: C14 · former map line 158.

Scenario (as written): A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed

Components (as written): controller v0.194.0, hub v0.90.1 (R-158 → R-167 → R-182)

Status (as written): PROVEN-LIVE (2026-08-03)

Evidence (as written): felhom-controller/REPORT.md. Two real capture failures on guest 9201 (mkdir …/backups: permission denied) → both accepted and stored by the hub, operator | recovery_unit_capture_failed | sent, and the positive observable customer | recovery_unit_capture_failed | skipped | operator_only read from the hub's notification_log. One event per app, loop continuing

Gap / roadmap (as written): Before this the failure was a [WARN] line and nothing else — the manager carried three notify seams and none for the unit capture, so /backups/apps, the page you open to ask whether ONE app is backed up, was the one page that never said. Deliberately NOT backup_failed: that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. D-c routes it to the operator and overrides the proposal. Operator-only is enforced by notify.operatorOnlyEvents, NOT by the absence of a customerMessages entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it. ROW REWRITTEN 2026-08-03 (R-182) — the 2026-08-02 claim was TRUE OF ONE APP AND FALSE OF THE REST, and it is worth saying which. The signal existed and worked; what it did not do was scale past the first failing app. Measured: nine per-app events reached the hub in one day and two operator mails went out, because the cooldown key is customerID:eventType(+tier) and this type carries app but no tier — so the first refused app took the hour and the rest were dropped before LogNotification, leaving no row on any channel. The old row said "One event per app, loop continuing", which was true of what the CONTROLLER emitted and not of what the operator received. Now: the per-app event is the RECORD (hub recordOnlyEvents: stored + logged every time, never mailed) and backup_run_failures is the NOTIFICATION — one mail per run listing every failed app, its leg and its reason, with the counts and free space. A suppressed operator event of ANY type now leaves a suppressed row naming its key. Proven live on demo-hp 2026-08-03 by a real 64.6 GiB fill (thin pool held 30.78 → 30.78): notification_log shows recovery_unit_capture_failed | operator | recorded ×2, backup_run_failures | operator | sent ×1 naming BOTH apps, and backup_run_failures | customer | skipped | operator_only. A second run in the same hour produced a second digest; after freeing space a run completed with 2 volume dumps and no digest. The suppression row proved itself on an unplanned event — disk_critical | suppressed | key=demo-hp:disk_critical — a collapse that yesterday would have left nothing at all. The digest's silence is safe only because the hub's deadline check raises expected_backup_missed from report freshness independently of any mail (monitor/deadline.go:396,417); that check is load-bearing for this row ⚠ CHECKED 2026-08-08 (R-258) AND THIS ROW WAS NOT CLAIMING SOMETHING UNTRUE — but the adjacent thing WAS false and the row's own narrative invites the wrong reading. This row is about the OPERATOR notification path (notification_log, the per-run digest) and it claims nothing about what /backups/apps DRAWS. Independently of it, the page's per-app tier-1 tick was wrong until controller v0.210.0: it went green on the mere presence of a restore point and turned red only when the box's most recent DB dump — whichever app it belonged to — had failed. So an app whose own backup failed could show a tick while this row's operator mail correctly reported the failure; the two halves disagreed, and only the operator half was true. The tick now reads THIS app's own dump result and shows no icon when there is no result for it. Reading this row as evidence that the customer's page answers "is ONE app backed up" would have been wrong for five days.

---

C15 — A local backup is bounded by the box's FREE SPACE, not by a partition set at build time — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime

Map row: C15 · former map line 159.

Scenario (as written): A local backup is bounded by the box's FREE SPACE, not by a partition set at build time — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime

Components (as written): golden build-golden.sh v3.0.0, agent v0.120.0, controller v0.193.1 (R-165 / D-a / B2, completed by R-181)

Status (as written): PROVEN-LIVE (2026-08-03) — BOTH halves

Evidence (as written): REPORT.md (R-178 reinstalls) + audits/SPIKE-r165-phase0-2026-08-03.md (P1/P2/P3) + the bake transcript. The golden bake is real evidence and is cited as such: build-golden.sh v3.0.0 produced including mount point mp0 ('/var/lib/felhom') with no mp1 line at all, and its own guards printed /var/lib/docker is a real mount, /mnt/sys_drive is a real mount and both paths are ONE filesystem. Archive published (registry HTTP 200, sha 54e2a4c4…). The B2 floor is unit-proven with 3 red-proofs and live on 9201

Gap / roadmap (as written): The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately. Proven (R-178, 2026-08-03): "a local backup is bounded by the box's FREE SPACE, not by a partition set at build time" — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp --golden <local volid>; demo-felhom the normal manifest route with verified sha256 54e2a4c431daf580… matches the hub manifest), each showing mp0 at /var/lib/felhom with no mp1, both consumer paths real mounts on ONE filesystem (stat -c %d = 64519 on all three), 3/3 reboots each, and claim → deploy → backup → restore with a planted marker returning byte-identical. Space available to a recovery unit measured at 65 GiB / 233 GiB, against the 19 GiB / 45 GiB those boxes' mp1 slices offered. NOT proven — and measured FALSE in part: "a capture that would exhaust it is refused per app rather than allowed to stop the container runtime". The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — but it is checked only in captureAllRecoveryUnits, while runVolumeDumps writes the bulk with no floor check at all, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → R-181, CLOSED THE SAME DAY (controller v0.193.0 + v0.193.1) and the second half is now PROVEN-LIVE TOO. The reserve became a per-app, per-run ADMISSION decision taken before the app's FIRST write and covering all three legs (DB dump, volume dump, capture) — they write under one per-app root, which is what lets one verdict cover them honestly — and it gained a size term, so an app is no longer admitted at 96% and then allowed to write 2 GB. Re-proven by filling demo-hp deliberately, once for EACH term, using the method that found the defect. Headroom @ 08:59:46 (906 MB free / 99%): both apps refused, the whole backups/primary tree byte-identical — TREE_SHA 111d1760c18d3440f700634ab325f8b8 before and after, opengist's tar still at its original 182,272 B; no Stopping <app> for safe volume dump line at all, which is the positive-by-absence observable that matters because that line IS present in the 08:58 baseline run; 0 volume dumps; one alert per app, HTTP 200. Space freed, re-run @ 09:01:33 → both captured normally. Size @ 09:03:00, reproducing the original sequence with a real 2 GiB file in opengist's volume (previous tar 2,147,666,432 B, the exact figure the defect was measured at) and the filesystem at 91% used / 2.9 GB free — both headroom terms deliberately clear: opengist refused (size) while privatebin was ADMITTED and dumped normally, proving the term is per-app rather than a global halt. The refusal's wording was NOT weakened to fit — the behaviour moved so the wording became true, and it is verified by tree fingerprint rather than by reading the log line, which is what lied. The fallocate instrument was re-proven on the rebuilt box before use (5 GiB step moved guest df while thin-pool data_percent held 36.83 → 36.83), and teardown returned the pool to 29.43%, below its own baseline. The golden is now VOUCHED (2026-08-03, hub Artifact manifest set: … golden=0.192.0), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired -sysdata-grow into the single grow so an older felhom-host-install.sh still provisions the same total capacity

---

C16 — Soft-quota: usage bar, pre-push enlargement block, customer notification

Map row: C16 · former map line 160.

Scenario (as written): Soft-quota: usage bar, pre-push enlargement block, customer notification

Components (as written): controller v0.109/134, hub v0.41/55

Status (as written): PROVEN-LIVE

Evidence (as written): 6D/6E; hub OffsiteChecker

Gap / roadmap (as written):

---

C17 — A customer (not the operator) performs a restore via UI alone

Map row: C17 · former map line 161.

Scenario (as written): A customer (not the operator) performs a restore via UI alone

Components (as written): all

Status (as written): MISSING (as evidence)

Evidence (as written): —

Gap / roadmap (as written): Alpha will produce this; script it into R-3. 2026-07-19: the C6 evidence attempt ran and found a product gap instead of evidence — audits/DIAG-immich-restore-2026-07-19.md. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING by finding, not by absence of attempt — the rehearsal system working, not failing. 2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0 (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. 2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped (audits/DIAG-immich-restore-round2-2026-07-19.md): a deliberate run from snapshot 49e7cb46 did recover all 11 assets (status=active, files resolve), but the operation reported failure and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. 2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED. (The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.) The C6 drill ran end-to-end through the UI: photos deleted, trash emptied, the full files+database restore pressed on /backups/restore, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: felhom-controller/REPORT.md 4e. Residual: the run was performed by the OPERATOR, not by a customer — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means trash, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash and verify the app's content, not the file count Lane split → 07-backup-architecture.md §3: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise

---