R-178 CLOSED: both demo boxes reinstalled from the merged golden and proven

Two boxes, two DIFFERENT supply paths, so the session proved the disk shape and
the delivery route rather than one of them twice.

demo-hp (layout proof, --golden <local volid>): mp0 at /var/lib/felhom,
backup=1, 70G, no mp1; /var/lib/docker and /mnt/sys_drive both real mounts of
its subdirectories via fstab; one df figure and one device id (64519) on all
three paths; reboots 3/3 with the binds surviving each.

demo-felhom (pipeline proof, --force-gitea-golden): fetch_verify succeeding
against the vouched manifest for BOTH artifacts -- 'verified sha256
54e2a4c431daf580... matches the hub manifest' for the golden, a7763d31... for
the agent. 250G single volume, grep -c '^mp1:' = 0, reboots 3/3.

Journey proven on both, endpoint-level: claim -> deploy -> back up -> restore,
with a planted marker returning byte-identical on each box. Ceiling measured
gone: 65 GiB and 233 GiB available to a recovery unit, against 19 and 45.

R-165 -> IMPLEMENTED, not PROVEN-LIVE, on the operator's ruling. B2, which that
row records as the bulkhead's replacement, fired live for the first time and
does refuse per app, delete nothing and alert -- but it is checked only in
captureAllRecoveryUnits while runVolumeDumps writes the bulk unguarded, and its
'the previous unit is untouched' claim was measured false (182,272 B dump
replaced by 2,147,666,432 B under a manifest still dated 06:34:26). -> R-181.

New: R-179 (uninstall leaves NAS network-storage units), R-180 (--archive-storage
not cross-checked against the ACL grant; 403 at step 8/8 after root@pam is
rotated), R-181. Third instance of R-115 recorded (agent 0.120.0 unpublished).

No code written, no version bumps -- this was a runbook.

documentation/architecture/00-capability-map.md

@@ -86,7 +86,7 @@
 | Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped |
 | **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait |
 | **A failed per-app Tier-1 backup reaches the OPERATOR** (app, error, and the target filesystem's used/free bytes at the moment of failure) | controller **v0.191.0**, hub **v0.89.0** (R-158, closed by R-167) | **PROVEN-LIVE (2026-08-02)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it |
-| **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.192.0** (R-165 / D-a / B2) | **IMPLEMENTED — NOT PROVEN-LIVE** | `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **NOT PROVEN-LIVE, and the missing leg is named: no box has been reinstalled from this golden (R-178).** Per this map's own rule a PROVEN-LIVE claim needs an end-to-end citation, and "the golden baked" is not "a box built from it works". **The golden is deliberately UNVOUCHED** so no fresh install picks up an unproven layout — prove first, then vouch (`CONTEXT.md` S-14). Every box in the field is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity |
+| **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.192.0** (R-165 / D-a / B2) | **IMPLEMENTED — the LAYOUT half is PROVEN-LIVE (2026-08-03); the REFUSAL half is not (R-181)** | `REPORT.md` (R-178 reinstalls) + `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately.** **Proven (R-178, 2026-08-03):** *"a local backup is bounded by the box's FREE SPACE, not by a partition set at build time"* — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp `--golden <local volid>`; demo-felhom the normal manifest route with **`verified sha256 54e2a4c431daf580… matches the hub manifest`**), each showing `mp0` at `/var/lib/felhom` with **no `mp1`**, both consumer paths real mounts on ONE filesystem (`stat -c %d` = `64519` on all three), 3/3 reboots each, and claim → deploy → backup → **restore** with a planted marker returning byte-identical. Space available to a recovery unit measured at **65 GiB / 233 GiB**, against the **19 GiB / 45 GiB** those boxes' `mp1` slices offered. **NOT proven — and measured FALSE in part:** *"a capture that would exhaust it is refused per app rather than allowed to stop the container runtime"*. The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — **but it is checked only in `captureAllRecoveryUnits`, while `runVolumeDumps` writes the bulk with no floor check at all**, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → **R-181**. The golden **is now VOUCHED** (2026-08-03, hub `Artifact manifest set: … golden=0.192.0`), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity |
 | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
 | **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise |