felhom.eu/documentation/architecture/capability-map-detail/C.md

# Capability map — detail, section C. Protection & recovery (the product promise)

> **What this file is.** The long text of every row of section C of
> [`00-capability-map.md`](../00-capability-map.md), moved here **verbatim** on 2026-10-12 (R-288,
> R-290 folded in; `09` §3 decisions 214 and 229) from the map at commit `1b45bb4d`. Each entry is one
> former table row, its cells laid out one per paragraph; the cell text itself was not edited. The
> map row carries the capability's ONE status word and its dated evidence pointer; this file keeps
> the argument, the corrections and the history. **Where this text and the map row disagree on the
> status, the map row is current** (see the map header for the words that were normalised).
>
> A new correction goes **under the entry it corrects, dated** — never into the map row.

<a id="c1"></a>
## C1 — Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded DEGRADED rather than silently normal (installer Case A/B)

*Map row: [C1](../00-capability-map.md#c1) · former map line 145.*

**Scenario (as written):** Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded **DEGRADED** rather than silently normal (installer Case A/B)

**Components (as written):** agent v0.113, host-install v1.22.0

**Status (as written):** **PROVEN-LIVE**

**Evidence (as written):** `E2D-fresh-vm-2026-07-29` C1 (real 1.22.0 install, rc=0, `Day-0 provision SUCCESS`) + C2 (both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort)

**Gap / roadmap (as written):** Case A (a second drive already present at install) has never fired naturally — only Case B has

---

<a id="c2"></a>
## C2 — Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes

*Map row: [C2](../00-capability-map.md#c2) · former map line 146.*

**Scenario (as written):** Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes

**Components (as written):** controller v0.118

**Status (as written):** **PROVEN-LIVE**

**Evidence (as written):** `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB

**Gap / roadmap (as written):** (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 **DB replay route → `07-backup-architecture.md` §8 row 3**

---

<a id="c3"></a>
## C3 — Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary

*Map row: [C3](../00-capability-map.md#c3) · former map line 147.*

**Scenario (as written):** Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary

**Components (as written):** controller v0.135

**Status (as written):** **PROVEN-LIVE**

**Evidence (as written):** `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C`

**Gap / roadmap (as written):** **Route + RTO → `07-backup-architecture.md` §8 rows 1, 2, 3b, 4, 5.** **R-403 (controller v0.230.0, 2026-08-31) changes NO status on this row and that is stated rather than left ambiguous:** the nightly copy now refuses to replace a complete unit package with an empty one (`07` §8.2), which removes a way the route could be DESTROYED between uses — it does not change what the route can be relied on for, and every leg it covers is the same one it covered yesterday. Proven live on demo-hp: the same state that deleted 120 082 104 B on v0.229.0 preserved all 7 files on v0.230.0. **R-102 CLOSED 2026-08-31 (controller v0.229.0):** the copy's `recovery-unit/` mirror is now restorable — „Teljes visszaállítás a másolatból" / `POST /backup/tier2/unit-restore` — and was proven live on `demo-hp` with the primary unit moved aside (`audits/DRILL-r102-tier2-unit-2026-08-31/`, §8 row 3b, 28.65 s). R-103 closed with it: the refusal that used to name a button on another page now offers the action on the row itself

---

<a id="c4"></a>
## C4 — Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping

*Map row: [C4](../00-capability-map.md#c4) · former map line 148.*

**Scenario (as written):** Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping

**Components (as written):** controller v0.134, agent, hub

**Status (as written):** **PROVEN-LIVE**

**Evidence (as written):** `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip)

**Gap / roadmap (as written):** Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` **Route + RTO → `07-backup-architecture.md` §8 rows 4, 10, 12, 15** (incl. the R-95 delete exposure and the R-104 stale-lock defect). **2026-08-04 (R-193/R-197, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`) — the row is NOT overclaiming and the guard is not the gap; the CADENCE is.** This row already recorded that a recreated data volume orphans the repo, and the spike confirms the mechanism at source: `WriteOffboxSecrets` (`offbox.go:392`) mints a fresh 256-bit repo password whenever `<DataDir>/offbox/repo_password` is absent, and **no automatic path ever consults the escrowed one** — `InjectOffboxPassword` has exactly one caller, a web form a human pastes into. What was NOT recorded is that this now fires on an **ordinary, planned, unattended guest rebuild**, on every box: measured on BOTH demo boxes 2026-08-03/04 by comparing `host_escrow.restic_pw_sha256` against `host_escrow_superseded.restic_pw_sha256` (demo-hp `8e03eddf…`→`8a9e33aa…`, demo-felhom `48741892…`→`c60c8bc7…`), orphaning **15 snapshots / 40.9 MB** and **36 snapshots / 1.14 GB** respectively. **demo-felhom is the important half:** it kept its DELIVERY (a stale staged secret restored the target in 76 s) and lost its REPOSITORY anyway, with **nothing marking the escrow stale for 13 h** — `escrow_stale` is wired to `ReissueCredentials`, the one path that does NOT change the repo password (**R-196**), and absent from the rebuild path that does. Status unchanged: the classify-and-move-aside guard remains PROVEN-LIVE and correct, and is predicted (not yet measured) to refuse the 2026-08-05 run on both boxes rather than start a silent fresh history

---

<a id="c5"></a>
## C5 — Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live

*Map row: [C5](../00-capability-map.md#c5) · former map line 149.*

**Scenario (as written):** Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live

**Components (as written):** controller v0.134/134.1/135

**Status (as written):** **PROVEN-LIVE** (2026-07-20)

**Evidence (as written):** `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone)

**Gap / roadmap (as written):** **2026-07-19:** `audits/DIAG-immich-restore-2026-07-19.md` finds **no offsite path loads a DB dump** — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "**immich end-to-end from offsite alone**" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. **RULED 2026-07-19 (Viktor): 6D's destruction hit the FILE TREE ONLY — the database survived in its named volume** (`immich_postgres_data` is a named volume in both the v2 and v3 template eras), so "immich end-to-end from offsite alone" **overclaimed scope**: the file half was proven, the DB half was never destroyed and therefore never restored. Row downgraded PROVEN-LIVE → **PARTIAL**, scope-corrected. Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (no offsite path could replay a DB at all) + the P-FAB destructive re-import (the proven-replay evidence, on the LOCAL path). **2026-07-19, controller v0.148.0:** the DB half now exists in code (R-43 + R-44) and its replay reached a live box — but **round 2 found it aborts against a running app** (`audits/DIAG-immich-restore-round2-2026-07-19.md`, H4: the replay races immich's own schema repair; `clip_index` recreated by the app 2 s before the dump's CREATE INDEX). **2026-07-20, controller v0.153.0: H4 IS CLOSED (R-47)** — both restore paths now replay into a DB-ONLY window (`StartStackServices` brings up the database service alone; the app starts only after the replay exits 0), with a fail-closed refusal when a dump has no identifiable DB service. *(The earlier note here said "closes in v0.149" — that was wrong: v0.149.0 was the F3 dashboard `BackupStatus` fix. R-47 shipped in v0.153.0.)* **2026-07-20: the clean run HAPPENED** — endpoint-level supervised reconstitute of immich from snapshot `49e7cb46` (the very snapshot that aborted in round 2): stop → DB-service-only start → replay rc-0 → full start, no `already exists`, operation reported SUCCESS, immich's own DatabaseService logged `No schema drift detected` twice, 11 assets `active`, 4/4 containers healthy, 231 `public` indexes. **Operator confirmed the immich timeline renders correctly after the reconstitute** (screenshot held, 2026-07-20). Evidence: `felhom-controller/REPORT.md` §4b. **2026-07-20, LATER THE SAME DAY — the destructive drill RAN and the row now earns PROVEN-LIVE.** The operator deleted the photos in immich own UI **and emptied the trash** (the step whose absence makes a drill prove nothing — the round-1 lesson), then restored through the customer-facing UI. **`40 file(s) placed`** against the 6 of the earlier non-destructive run — the files were really gone and really came back — plus 1 DB dump replayed rc-0, 11 assets `active`, `No schema drift detected`, timeline confirmed by the operator. This is the destroy-then-recover proof the 6D downgrade asked for, and it was taken through the customer own buttons, not endpoint shortcuts. Evidence: `felhom-controller/REPORT.md` 4e **Route + RTO → `07-backup-architecture.md` §8 rows 3, 4** — the matrix also records that no offsite action unpacks the named-volume tars it captures (→ R-107)

---

<a id="c6"></a>
## C6 — Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking

*Map row: [C6](../00-capability-map.md#c6) · former map line 150.*

**Scenario (as written):** Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking

**Components (as written):** controller v0.125/128/130/136

**Status (as written):** **PROVEN-LIVE**

**Evidence (as written):** `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413)

**Gap / roadmap (as written):** Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D

---

<a id="c7"></a>
## C7 — Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification

*Map row: [C7](../00-capability-map.md#c7) · former map line 151.*

**Scenario (as written):** Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification

**Components (as written):** agent v0.75/0.76, PBS

**Status (as written):** **PROVEN-LIVE**

**Evidence (as written):** `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`)

**Gap / roadmap (as written):** (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill **Route + RTO → `07-backup-architecture.md` §8 rows 6, 8, 9** — measured 84–112 s local / 1101 s PBS into a scratch guest; a restore to a DIFFERENT host is unmeasured

---

<a id="c8"></a>
## C8 — PBS-DR secret self-heal on reused-peer re-provision

*Map row: [C8](../00-capability-map.md#c8) · former map line 152.*

**Scenario (as written):** PBS-DR secret self-heal on reused-peer re-provision

**Components (as written):** hub v0.56

**Status (as written):** **IMPLEMENTED**

**Evidence (as written):** hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause)

**Gap / roadmap (as written):** Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending

---

<a id="c9"></a>
## C9 — Box survives an unattended app or guest-network failure (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired

*Map row: [C9](../00-capability-map.md#c9) · former map line 153.*

**Scenario (as written):** Box survives an **unattended app or guest-network failure** (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired

**Components (as written):** controller v0.156.0→**v0.190.0**, agent v0.92.1

**Status (as written):** **PROVEN-LIVE** (2026-07-21; boot-orphan leg rebuilt and re-proven 2026-08-02 — **6 of 6 hard resets**, repeat count cited per N.5)

**Evidence (as written):** All three legs exercised on the live demo box, operator-present, in one session — `felhom-controller/REPORT.md` + `felhom-agent/REPORT.md` (2026-07-21). **Dead primary:** `docker stop immich-server` 12:50:40 CEST → `degraded` 13 s later → **exactly one** `app_start_failed` + dashboard banner → restart → banner self-cleared (the 2026-07-20 shape that was silent for 18 h). **Boot orphan:** `pct reboot 9201` → `[bootrecon] 1 boot-orphaned app(s) found: [bookstack]` → started in 1 attempt of 2, **zero alerts** (success inside the boot grace is silent); `StartedAt` proves Docker's `unless-stopped` did NOT resurrect it — only the sweep did, which also answers P1 and confirms the F5 hypothesis. **Dead DHCP client:** deliberate replay of the incident — `kill -9` 12:43:18 → detected on process liveness 57 s later while the lease was still live → healed 12:45:18 with the incident's verbatim invocation; **the tunnel never dropped (`cloudflared Up 29 hours`)**, i.e. the outage was prevented rather than merely observed

**Gap / roadmap (as written):** **The gap this validation surfaced → R-55, now FIXED (controller v0.157.0, 2026-07-21).** For a *drive-backed* app a customer's deliberate Stop did NOT survive a reboot — the boot bind gate recreated and started every deployed drive-backed app unconditionally. Pre-existing, not introduced by R-52 (whose own gate was observed correct). The gate now also requires the app to still HAVE containers, which is R-52's own `existing-Exited vs absent` predicate: a UI Stop is `compose down` and removes them. So **"a stopped app stays stopped" now holds for drive-backed apps too — PROVEN LIVE 2026-07-21** (TASK-F Part 3, operator-present). immich was stopped through the real UI endpoint (`compose down` → 0 containers), calibre-web and bookstack left running, then `pct reboot 9201`: the gate recreated calibre-web and logged `1 drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose`; immich came back **stopped**, where the identical fixture had brought it back running hours earlier. Zero alerts, ~15 s to steady state. The **static-guest** half of the network leg stays deliberately out of scope → **R-50**. **2026-08-02 — the boot-orphan leg now rests on a RECORDED signal, not an inference (R-166, controller v0.189.0).** Both the R-52 sweep and the R-55 gate above decided "the customer stopped this" from *zero containers*, which is also what a power cut mid-compose and an interrupted deploy leave behind — so two real faults were read as deliberate stops and stranded silently (R-157 mechanism B). The customer's intent is now written to `app.yaml` (`desired_state`) by their own action and read directly. **PROVEN-LIVE on 9201** in three flows: a UI Stop persisted `stopped` and survived a controller restart with the app still down and NOT listed as a candidate; an app recorded `running` whose containers were removed out-of-band was recovered by name (`[bootrecon] 1 boot-orphaned app(s) found: [calibre-web]` → started in 1 attempt) — **the case that was invisible before**; and a legacy `app.yaml` with no field was skipped exactly as before and was never inferred to be stopped. **Interrupted app-data operations are covered separately and are NOT proven-live** — `backup.AppStopGuard` restarts apps left stopped by a killed volume dump / offsite reconstitute / `.fab` export, and that leg is unit-proven + red-proofed only (killing the controller mid-backup on a live box was not exercised): **IMPLEMENTED, not PROVEN-LIVE.** **R-170 and R-171 closed the same day (controller v0.190.0).** **R-157 mechanism A** — the sweep observed ONCE at T+5 s, while docker was still restoring, and never re-checked (3 of 6 hard resets). It is now a settle-then-sweep window: sample every 5 s, settled after 3 identical samples, ONE sweep at the end, terminating on settled or a 50 s budget (sized so settle+budget+one retry stays inside the 90 s dead-app grace; a test rejected 60 s at 95 s). **Repeat count, per this map's own rule: 6 of 6 hard resets on the shipped build brought every app back, and an app the customer had stopped stayed down in all 6** (window settle times 10/40/10/10/15/15 s — i.e. it routinely waited 2–8× longer than the old fixed 5 s). A same-app before/after on one box is the sharpest evidence: the pre-fix window logged `no boot-orphaned apps` for calibre-web at 18:08:35; the fixed one found and recovered it at 18:18:50. **R-170** — `shouldRecreateOnBoot` now reads intent too, so the two boot gates agree; proven live in one reboot (calibre-web `running`+zero containers recreated, immich `stopped` left alone). **R-171 — a regression v0.189.0 introduced, found by reading the diff and CONFIRMED on hardware before any fix was written:** the sweep started an app whose drive was absent, burned both attempts and raised a false dead-app alarm. The *write* hazard was blocked only by an ACCIDENTAL filesystem permission (host-root-owned mountpoint + unprivileged guest) that no code owns and no test pinned — which is why it was fixed rather than noted. New fail-safe `bootrecon.StartGate` (cannot determine ⇒ do not start), also covering quiesce and in-flight app-data operations. **One defect in the fix itself, found by live validation and not by review:** the window sampled the Manager's 10 s-refreshed cache, so "settled" could mean "the cache did not update"; `sampleBootFleet` now refreshes first. Evidence: `audits/DIAG-bootrecon-drive-absent-2026-08-02.md`, `felhom-controller/REPORT.md`

---

<a id="c10"></a>
## C10 — Crash/power-loss mid-backup/mid-migration → self-heal on next run

*Map row: [C10](../00-capability-map.md#c10) · former map line 154.*

**Scenario (as written):** Crash/power-loss mid-backup/mid-migration → self-heal on next run

**Components (as written):** controller, agent

**Status (as written):** **PROVEN-LIVE**

**Evidence (as written):** `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3

**Gap / roadmap (as written):** (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup)

---

<a id="c11"></a>
## C11 — An app can be withdrawn from the catalog without orphaning the customers running it (available / hidden / abandoned)

*Map row: [C11](../00-capability-map.md#c11) · former map line 155.*

**Scenario (as written):** An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned)

**Components (as written):** controller v0.158.1, catalog metadata

**Status (as written):** **PROVEN-LIVE** (2026-07-21)

**Evidence (as written):** TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21)

**Gap / roadmap (as written):** Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57

---

<a id="c12"></a>
## C12 — Box survives a site/network change (relocation, different subnet, DHCP re-lease) with the control plane intact

*Map row: [C12](../00-capability-map.md#c12) · former map line 156.*

**Scenario (as written):** Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact

**Components (as written):** agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap

**Status (as written):** **PROVEN-LIVE (2026-07-25)**

**Evidence (as written):** **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box

**Gap / roadmap (as written):** **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped

---

<a id="c13"></a>
## C13 — The customer is warned BEFORE a filesystem fills — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered

*Map row: [C13](../00-capability-map.md#c13) · former map line 157.*

**Scenario (as written):** **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered

**Components (as written):** controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c)

**Status (as written):** **PROVEN-LIVE (2026-08-02)**

**Evidence (as written):** `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding

**Gap / roadmap (as written):** **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait

---

<a id="c14"></a>
## C14 — A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed

*Map row: [C14](../00-capability-map.md#c14) · former map line 158.*

**Scenario (as written):** **A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed**

**Components (as written):** controller **v0.194.0**, hub **v0.90.1** (R-158 → R-167 → **R-182**)

**Status (as written):** **PROVEN-LIVE (2026-08-03)**

**Evidence (as written):** `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing

**Gap / roadmap (as written):** **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it. **ROW REWRITTEN 2026-08-03 (R-182) — the 2026-08-02 claim was TRUE OF ONE APP AND FALSE OF THE REST, and it is worth saying which.** The signal existed and worked; what it did not do was scale past the first failing app. Measured: nine per-app events reached the hub in one day and **two** operator mails went out, because the cooldown key is `customerID:eventType(+tier)` and this type carries `app` but no `tier` — so the first refused app took the hour and the rest were dropped **before `LogNotification`**, leaving no row on any channel. The old row said "One event per app, loop continuing", which was true of what the CONTROLLER emitted and not of what the operator received. **Now:** the per-app event is the RECORD (hub `recordOnlyEvents`: stored + logged every time, never mailed) and `backup_run_failures` is the NOTIFICATION — one mail per run listing every failed app, its leg and its reason, with the counts and free space. A suppressed operator event of ANY type now leaves a `suppressed` row naming its key. **Proven live on demo-hp 2026-08-03** by a real 64.6 GiB fill (thin pool held 30.78 → 30.78): `notification_log` shows `recovery_unit_capture_failed | operator | recorded` ×2, `backup_run_failures | operator | sent` ×1 naming BOTH apps, and `backup_run_failures | customer | skipped | operator_only`. A second run in the same hour produced a second digest; after freeing space a run completed with 2 volume dumps and **no** digest. The suppression row proved itself on an unplanned event — `disk_critical | suppressed | key=demo-hp:disk_critical` — a collapse that yesterday would have left nothing at all. **The digest's silence is safe only because** the hub's deadline check raises `expected_backup_missed` from report freshness independently of any mail (`monitor/deadline.go:396,417`); that check is load-bearing for this row  **⚠ CHECKED 2026-08-08 (R-258) AND THIS ROW WAS NOT CLAIMING SOMETHING UNTRUE — but the adjacent thing WAS false and the row's own narrative invites the wrong reading.** This row is about the OPERATOR notification path (`notification_log`, the per-run digest) and it claims nothing about what `/backups/apps` DRAWS. Independently of it, the page's per-app tier-1 tick was wrong until controller v0.210.0: it went green on the mere presence of a restore point and turned red only when the box's most recent DB dump — **whichever app it belonged to** — had failed. So an app whose own backup failed could show a tick while this row's operator mail correctly reported the failure; the two halves disagreed, and only the operator half was true. The tick now reads THIS app's own dump result and shows **no icon** when there is no result for it. Reading this row as evidence that the customer's page answers "is ONE app backed up" would have been wrong for five days.

---

<a id="c15"></a>
## C15 — A local backup is bounded by the box's FREE SPACE, not by a partition set at build time — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime

*Map row: [C15](../00-capability-map.md#c15) · former map line 159.*

**Scenario (as written):** **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime

**Components (as written):** golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.193.1** (R-165 / D-a / B2, completed by R-181)

**Status (as written):** **PROVEN-LIVE (2026-08-03) — BOTH halves**

**Evidence (as written):** `REPORT.md` (R-178 reinstalls) + `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201

**Gap / roadmap (as written):** **The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately.** **Proven (R-178, 2026-08-03):** *"a local backup is bounded by the box's FREE SPACE, not by a partition set at build time"* — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp `--golden <local volid>`; demo-felhom the normal manifest route with **`verified sha256 54e2a4c431daf580… matches the hub manifest`**), each showing `mp0` at `/var/lib/felhom` with **no `mp1`**, both consumer paths real mounts on ONE filesystem (`stat -c %d` = `64519` on all three), 3/3 reboots each, and claim → deploy → backup → **restore** with a planted marker returning byte-identical. Space available to a recovery unit measured at **65 GiB / 233 GiB**, against the **19 GiB / 45 GiB** those boxes' `mp1` slices offered. **NOT proven — and measured FALSE in part:** *"a capture that would exhaust it is refused per app rather than allowed to stop the container runtime"*. The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — **but it is checked only in `captureAllRecoveryUnits`, while `runVolumeDumps` writes the bulk with no floor check at all**, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → **R-181, CLOSED THE SAME DAY (controller v0.193.0 + v0.193.1) and the second half is now PROVEN-LIVE TOO.** The reserve became a **per-app, per-run ADMISSION decision** taken before the app's FIRST write and covering all three legs (DB dump, volume dump, capture) — they write under one per-app root, which is what lets one verdict cover them honestly — and it gained a **size term**, so an app is no longer admitted at 96% and then allowed to write 2 GB. **Re-proven by filling demo-hp deliberately, once for EACH term, using the method that found the defect.** *Headroom @ 08:59:46* (906 MB free / 99%): both apps refused, **the whole `backups/primary` tree byte-identical — `TREE_SHA` 111d1760c18d3440f700634ab325f8b8 before and after**, opengist's tar still at its original 182,272 B; **no `Stopping <app> for safe volume dump` line at all**, which is the positive-by-absence observable that matters because that line IS present in the 08:58 baseline run; 0 volume dumps; one alert per app, HTTP 200. Space freed, re-run @ 09:01:33 → both captured normally. *Size @ 09:03:00*, reproducing the original sequence with a real 2 GiB file in opengist's volume (previous tar **2,147,666,432 B**, the exact figure the defect was measured at) and the filesystem at **91% used / 2.9 GB free — both headroom terms deliberately clear**: opengist refused `(size)` while **privatebin was ADMITTED and dumped normally**, proving the term is per-app rather than a global halt. **The refusal's wording was NOT weakened to fit** — the behaviour moved so the wording became true, and it is verified by tree fingerprint rather than by reading the log line, which is what lied. The `fallocate` instrument was re-proven on the rebuilt box before use (5 GiB step moved guest `df` while thin-pool `data_percent` held **36.83 → 36.83**), and teardown returned the pool to **29.43%**, below its own baseline. The golden **is now VOUCHED** (2026-08-03, hub `Artifact manifest set: … golden=0.192.0`), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity

---

<a id="c16"></a>
## C16 — Soft-quota: usage bar, pre-push enlargement block, customer notification

*Map row: [C16](../00-capability-map.md#c16) · former map line 160.*

**Scenario (as written):** Soft-quota: usage bar, pre-push enlargement block, customer notification

**Components (as written):** controller v0.109/134, hub v0.41/55

**Status (as written):** **PROVEN-LIVE**

**Evidence (as written):** 6D/6E; hub OffsiteChecker

**Gap / roadmap (as written):** 

---

<a id="c17"></a>
## C17 — A customer (not the operator) performs a restore via UI alone

*Map row: [C17](../00-capability-map.md#c17) · former map line 161.*

**Scenario (as written):** **A customer (not the operator) performs a restore via UI alone**

**Components (as written):** all

**Status (as written):** **MISSING** (as evidence)

**Evidence (as written):** —

**Gap / roadmap (as written):** Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise

---