From aa624496949aa61b084d5fcf2b03dbf7754f4581 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 3 Aug 2026 09:34:15 +0200 Subject: [PATCH] R-178 CLOSED: both demo boxes reinstalled from the merged golden and proven Two boxes, two DIFFERENT supply paths, so the session proved the disk shape and the delivery route rather than one of them twice. demo-hp (layout proof, --golden ): mp0 at /var/lib/felhom, backup=1, 70G, no mp1; /var/lib/docker and /mnt/sys_drive both real mounts of its subdirectories via fstab; one df figure and one device id (64519) on all three paths; reboots 3/3 with the binds surviving each. demo-felhom (pipeline proof, --force-gitea-golden): fetch_verify succeeding against the vouched manifest for BOTH artifacts -- 'verified sha256 54e2a4c431daf580... matches the hub manifest' for the golden, a7763d31... for the agent. 250G single volume, grep -c '^mp1:' = 0, reboots 3/3. Journey proven on both, endpoint-level: claim -> deploy -> back up -> restore, with a planted marker returning byte-identical on each box. Ceiling measured gone: 65 GiB and 233 GiB available to a recovery unit, against 19 and 45. R-165 -> IMPLEMENTED, not PROVEN-LIVE, on the operator's ruling. B2, which that row records as the bulkhead's replacement, fired live for the first time and does refuse per app, delete nothing and alert -- but it is checked only in captureAllRecoveryUnits while runVolumeDumps writes the bulk unguarded, and its 'the previous unit is untouched' claim was measured false (182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). -> R-181. New: R-179 (uninstall leaves NAS network-storage units), R-180 (--archive-storage not cross-checked against the ACL grant; 403 at step 8/8 after root@pam is rotated), R-181. Third instance of R-115 recorded (agent 0.120.0 unpublished). No code written, no version bumps -- this was a runbook. --- CONTEXT.md | 31 +- REPORT.md | 383 +++++++++++++++--- STATUS.md | 62 ++- .../architecture/00-capability-map.md | 2 +- documentation/backlog/OPEN-ITEMS.md | 9 +- scripts/CHANGELOG.md | 29 ++ 6 files changed, 424 insertions(+), 92 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index b915b46..da6085e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -39,12 +39,31 @@ so pruning could only mean deleting a **different** app's only local recovery un boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed shortly. So R-176's in-place migration rehearsal is **withdrawn**, not deferred. -**S-14 — prove first, then vouch (2026-08-03).** Golden **0.192.0** is baked, published and verified in -the registry, and is **deliberately UNVOUCHED**. Vouching is what makes a fresh install pick a golden -up, so vouching one that no box has been proven from would put an unproven disk layout in front of the -next install anywhere. The bake script already treats the hub record as a separate deliberate step; this -makes the ordering a rule. **The remaining work is R-178** — reinstall each demo box from the merged -golden, prove claim → deploy → back up → restore, **one box at a time**, and only then vouch. +**S-14 — prove first, then vouch (2026-08-03) — SPENT, and the ordering did not survive contact.** The +rule was: golden **0.192.0** stays UNVOUCHED until a box has been proven from it, because vouching is +what makes a fresh install pick a golden up. **In the event the golden was vouched at 07:23:26 CEST on +2026-08-03, before any box was reinstalled** (hub log `Artifact manifest set: agent=0.119.0 +golden=0.192.0`), so the ordering was already spent when R-178's session opened; the operator elected +to accept it rather than revert the manifest. **Both boxes were then reinstalled and proven** (R-178, +`REPORT.md`), so the end state is the intended one and no unproven layout was ever in front of a real +install — but the rule protected nothing, because nothing enforced it. **The lesson is R-115's, one +layer up:** an ordering that lives only in a `CONTEXT.md` sentence and a runbook's §7 is a reminder, +and reminders do not hold. If prove-then-vouch is to be a rule it needs the shape R-120's gate has — +a refusal at `handleSetArtifacts`, the sole path to `SetArtifactManifest`, which runs without anyone +choosing to run it. + +**S-15 — the merged layout is proven live, by two different supply paths (2026-08-03, R-178).** Both +demo boxes were wiped and reinstalled from golden 0.192.0 and taken through claim → deploy → back up → +**restore**. **demo-hp** was installed with `--golden ` (the layout proof) and +**demo-felhom** by the normal manifest route with `--force-gitea-golden` (the pipeline proof — +`verified sha256 54e2a4c431daf580… matches the hub manifest`), deliberately different so the session +proved the disk shape *and* the delivery route rather than one of them twice. Live shape on both: +`mp0` at `/var/lib/felhom`, `backup=1`, **no `mp1`**; `/var/lib/docker` and `/mnt/sys_drive` both real +mounts of its subdirectories via `/etc/fstab`; ONE `df` figure and one device id on all three paths; +3/3 reboots each with the binds surviving every time. **What is NOT proven is B2** → **R-181**: the +floor guards `captureAllRecoveryUnits` and not `runVolumeDumps`, which is the leg that fills the +volume, and its refusal message's "the previous unit is untouched" was measured false. R-165 is +therefore **IMPLEMENTED**, not PROVEN-LIVE. **S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).** Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the diff --git a/REPORT.md b/REPORT.md index 5f49506..b1cb4cf 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,83 +1,342 @@ -# REPORT — R-165: the `mp1` merge, built and proven at the bake (2026-08-03) +# REPORT — R-178: both demo boxes reinstalled from the merged golden and proven (2026-08-03) -**Overwritten** per the standing rule. The prior contents (hub v0.89.0 / R-167, 2026-08-02) have their -durable record in `hub/CHANGELOG.md` and `CONTEXT.md` S-11/S-12. +**Overwritten** per the standing rule. The prior contents (R-165, the bake, 2026-08-03) have their +durable record in `documentation/backlog/OPEN-ITEMS.md` R-165 and the per-repo CHANGELOGs. -**Companion:** `felhom-agent/REPORT.md` holds the full session detail — probes, variant evidence, the -bake transcript, red-proofs and teardown. **This file covers what changed in THIS repo, and the CI -verification for all three.** - -> **Scope, stated first.** The merge is **built and green through Phase 5**. **Phases 6–7 — reinstalling -> the two demo boxes from the merged golden and proving one end to end — were NOT done, and nothing -> was wiped.** The golden is therefore deliberately **unvouched**. Remaining work: **R-178**. +**Runbook, not a task.** No repo got a version bump and nothing was built. One artifact was +**published** (agent 0.120.0) on an explicit operator ruling — see §2. All code findings are filed as +register rows, not commits, per the runbook's §7. --- -## 1. What changed in `felhom.eu` +## 1. Preconditions P1–P6, each as measured -**No hub change; no hub version bump.** This repo changed in two places, one of them forced. +| # | Precondition | Measurement | +|---|---|---| +| **P1** | Golden sha256 matches R-178's record | **PASS** — computed from the artifact itself on demo-hp: `sha256sum` → `54e2a4c431daf580d2807b82d810be36a7c6a9f697b27094dc2912d26a43b3e0`, 649,547,835 B. Matches R-178's recorded prefix and the hub manifest's stored value exactly. Re-verified after copying to `local`: identical | +| **P2** | A rollback golden still exists | **PASS, three depths.** Split-layout golden **0.188.0** fetchable from Gitea (`HTTP 200`, 649,310,288 B). A **local** split-layout golden sat on each box — `local:backup/vzdump-lxc-9100-2026_07_21-18_24_49.tar.zst` (demo-hp), `…2026_07_20-17_50_57…` (demo-felhom). Best: **full guest vzdumps**, three per box, newest `vzdump-lxc-9201-2026_08_03-04_54_28.tar.zst` (2,351,557,870 B, demo-hp) and `…2026_08_03-04_44_50…` (6,323,506,165 B, demo-felhom), both ~3 h old — so each box could be put back exactly as it was | +| **P3** | Neither demo box holds anything wanted | **PASS, stated explicitly.** demo-hp: paperless-ngx (webserver/postgres/redis) + filebrowser + samba shares. demo-felhom: immich (×4), docmost (×3), calibre-web, bookstack (×2) + filebrowser + samba. Both are demo customers (`Demo HP`, `Demo Ügyfél`); no real customer data. **All of it was destroyed by the wipes and none of it was restored** — that was the point, and the operator confirmed each wipe | +| **P4** | The colleague's box untouched | **PASS** — `peti-felhom` is Tier 2 and out of scope. No command was sent to it; it appears in the hub customer list as `DOWN`, exactly as before | +| **P5** | Both hosts' agent is v0.120.0 | **PASS** — `felhom-agent 0.120.0` on both, before and after | +| **P6** | The hub's host register, BEFORE | **Captured**: `demo-felhom-8363b5 / Demo Ügyfél / 0.120.0 / ONLINE / 1-of-2 guests`; `demo-hp-bb76ea / Demo HP / 0.120.0 / ONLINE / 1-of-2`; `drill-r50-0a4f9a / drill-r50 / 0.113.0 / DOWN` (the fixture, untouched throughout) | -**`scripts/felhom-host-install.sh` — forced by a census, not planned.** `step_grows` computed **two** -volume sizes and the install call passed both, so it had to change with the agent or every install -would have provisioned a half-sized box. It now computes ONE total, summing the old 80/20 split -(`226` = the previous `184 + 42`), so **a standard appliance keeps exactly the 250 G it had** — it is -simply no longer split by a wall. **The size still comes from the physical disk**: `step_grows` -already read the thin pool's real free space, and the merge only collapsed its two outputs into one. -That is the answer to the task's §8.2 — no row needed filing. `--sysdata-grow` is deprecated but still -honoured, because the agent **folds** a hand-passed value in rather than dropping it. +--- -**Documentation**, per the coupling rule — see §3. +## 2. Two findings that contradicted the runbook's premise, both surfaced before any wipe -## 2. The bake, as this repo's audit trail records it +**(a) The golden was ALREADY vouched.** Hub log, a positive observable: +`2026/08/03 07:23:26 [INFO] Artifact manifest set: agent=0.119.0 golden=0.192.0 min_agent="0.113.0" +wrapper_sha=true` — roughly ten minutes before this session's first hub read, and this session had +POSTed nothing. The configuration page confirmed `Artifacts.GoldenVersion=0.192.0`, +`GoldenSHA256=54e2a4c4…3b3e0`. So §7's prove-then-vouch order was already spent, and Phase A's stated +safety ("nothing is official yet, so a failure reaches nobody") was void. **Operator ruling: accept it +and proceed**, keeping the two different supply paths. Recorded on `CONTEXT.md` S-14, now marked SPENT +with the reason — the rule lived only in prose and nothing enforced it. -`documentation/audits/SPIKE-r165-phase0-2026-08-03.md` (new) holds P1, P2 and P3 with method, -measurement and ruling, including the teardown of every probe artefact at all three layers. The -headline measurements: +**(b) Agent v0.120.0 had never been published — the vouched agent was 0.119.0.** +`GET …/generic/felhom-agent/0.120.0/felhom-agent` → **HTTP 404** (0.119.0 → 200). Installer step 5's +idempotent skip requires `installed == vouched` exactly, so a documented-path reinstall would have +**downgraded both boxes** to the pre-merge 0.119.0 — and would have *succeeded* while doing it, since +the current `step_grows` sets `SYSDATA_GROW=0` and 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on +error) therefore never fires. The session would have proven a stack nobody ships. **Operator ruling: +publish and vouch first.** `scripts/publish-agent.sh 0.120.0` from a clean tree at `4bb84fc3` +(`git status --porcelain` empty, `HEAD == origin/main`): upload HTTP 201, **round-trip GET verified**, +`AGENT_SHA256=a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`. Vouched via +`POST /configuration/artifacts` → `303 …flash=artifacts_set`, hub log +`Artifact manifest set: agent=0.120.0 golden=0.192.0`; the hub resolved the sha authoritatively from +Gitea rather than trusting the submitted value. Filed as the **third instance of R-115**, not a new +ID — R-115 is that row, and it has been WAITING-ON-OPERATOR since 2026-07-29. -- **P1 PASS** — a real pre-merge archive restore-tests clean, `mount_parity: ok`, 84 s, `mountParity` - untouched. -- **P2** — all three variants boot and reboot 3/3; they are separated by **scoping**, not mechanics. - The container's view of `/mnt` is 8.0K under V-a and V-c, and **17.9M — Docker's entire data-root — - under V-b**. The operator chose **V-c**. -- **P3** — the four retargeted golden assertions, run against a deliberately wrong shape: **8/8**. +--- -## 3. Documentation coupling +## 3. demo-hp — the layout proof -| File | Change | +**Install path.** Uninstall: `./felhom-host-install.sh --uninstall --vmid 9201` (typed-vmid +confirmation supplied over a pty). Install: + +``` +./felhom-host-install.sh --customer-id demo-hp --mode appliance --vmid 9201 \ + --cores 7 --memory 26906 \ + --golden local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst \ + --passphrase-file /root/.pp-demo-hp +``` + +Script fetched from `https://felhom.eu/scripts/felhom-host-install.sh`, **v1.22.0**, sha256 +`ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4` — byte-identical to the repo copy +that was read. The passphrase went file→file into a 0600 file and never onto a command line. A +`--dry-run` preceded the real run and resolved `manifest: agent v0.120.0 (sha a7763d31…), golden +v0.192.0` with `grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)`. + +**One deviation, mine, and it cost a restart.** The first attempt staged the golden on +`felhom-backup` with `--archive-storage felhom-backup`. Pre-flight passed; **step 8/8** failed: +`HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)` — +the default pool-scoped ACL grants `local local-lvm felhom-pbs` only. Fixed by copying the golden to +`local` (sha re-verified after the copy) and `--resume`. Filed as **R-180**: the condition is +statically checkable in pre-flight, and the failure lands *after* step 4b has rotated and vaulted +root@pam. + +**Layout evidence.** + +``` +mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G # and NO mp1 line +rootfs: local-lvm:vm-9201-disk-0,size=32G + +/var/lib/felhom /dev/mapper/pve-vm--9201--disk--1 ext4 rw,relatime,stripe=16 +/var/lib/docker /dev/mapper/pve-vm--9201--disk--1[/docker] ext4 +/mnt/sys_drive /dev/mapper/pve-vm--9201--disk--1[/sys_drive] ext4 + +Filesystem Size Used Avail Use% Mounted on +/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/felhom +/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/docker +/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /mnt/sys_drive + +stat -c %d → 64519 for all three paths +/etc/fstab: /var/lib/felhom/docker → /var/lib/docker ; /var/lib/felhom/sys_drive → /mnt/sys_drive +both binds writable (touch succeeded on each) +``` + +`/mnt/sys_drive` shows a second `findmnt` row — it is the controller container's +`-v /mnt:/mnt:rslave` propagation (peer group 383 vs the fstab bind's 333), the same shape the split +layout had, not a stacked bind. + +**Reboots — each individually, minimum three:** + +| # | started | controller healthy | outcome | +|---|---|---|---| +| 1 | 08:13:33 | 08:13:55, `Up 12 seconds (healthy)` | one filesystem, 69G/65G on both paths | +| 2 | 08:13:59 | 08:14:15, `Up 7 seconds (healthy)` | same | +| 3 | 08:14:19 | 08:14:35, `Up 7 seconds (healthy)` | same | + +After all three, `mountpoint -q` returns true for **all three paths** and both binds still resolve to +the single volume's subdirectories — which is what the reboots exist to test. `uptime -s` = +`2026-08-03 06:14:24 UTC`, matching reboot 3, so these were real reboots. + +**The journey — method stated: endpoint-level, not a browser.** `claude-in-chrome` does not exist on +DooPlex; every step below invoked the exact endpoint the dashboard's own JavaScript calls. + +| Leg | Endpoint | Observable | +|---|---|---| +| Claim | `POST /claim` with the pre-auth HMAC CSRF token (64 chars) **and** its `felhom_claim_csrf` cookie | `302 → /`; gate discriminator flipped `{"error":"dashboard not yet claimed"}` → `{"error":"authentication required"}`. **The code is emailed-only (R-119) — the operator supplied it**, after an operator resend rotated the generation (the previous code had been consumed 12 d earlier) | +| Deploy | `POST /api/stacks//deploy` with `{"values":{…}}`, field values scraped from the deploy form exactly as the form's own `fetch` does | `{"ok":true}`; `privatebin Up (healthy)`, then `opengist Up (healthy)` | +| Back up | `POST /api/debug/backup/dbdump` — this runs the **production** `RunDBDumps` path (DB leg → `runVolumeDumps` → `captureAllRecoveryUnits`); the debug route only starts it instead of waiting for 03:30 | `Volume dump: opengist/… → 178.0 KB`, `privatebin/… → 2.5 KB`, `App-data backup completed … 2 volume dump(s) (3.962s)`, `Recovery unit captured for …` ×2 | +| **Restore** | `POST /backup/restore` `stack_name=privatebin snapshot_id=primary` | A marker planted in the live volume was **deleted**, then restored: `{"ok":true,"message":"privatebin visszaállítva (primary)."}` in **9.2 s**, and the marker returned with an **identical sha256 `ac1faae6134871d640d5cb1bbc6b5092d1e96b7e46c6c3bcb6e8236cc33ae861`**. App `Up (healthy)` afterwards | + +**Recovery unit path, and it lands on the single volume:** +`/mnt/sys_drive/felhom-data/backups/primary//{manifest.json,compose/,volume-dumps/}`, whose `df` +is `/dev/mapper/pve-vm--9201--disk--1` — the merged volume. + +**Note on the first app chosen.** `privatebin`'s catalog entry declares no `backup:` section, so its +first per-stack capture produced `"volume_dumps": null` — correct for that declaration, not a defect, +but it means a per-stack `POST /stacks//backup` writes compose+config only; the volume leg lives in +the full pass. A second app (`opengist`) was deployed so the run had both a capture and, later, a +refusal. + +**The ceiling is gone, measured:** a recovery unit can use **65 GiB** — the whole volume — against the +**19 GiB** the pre-wipe `mp1` slice offered (`/dev/mapper/pve-vm--9201--disk--2 20G 95M 19G 1%`). + +--- + +## 4. The floor's first live firing — and it does not do what it says + +**Instrument, proven before use.** demo-hp's thin pool is **53.93 GiB** and the auto-sized volume is +70 G, so a real fill to 97 % would have exhausted the pool and corrupted every guest on the box, +including the protected `drill-r50` fixture. A 5 GiB `fallocate` probe moved guest `df` from +`977M used / 65G avail` to `6.0G / 60G` while thin-pool `data_percent` stayed **29.03 → 29.03** — +zero blocks allocated — and cleanup returned both to baseline. The floor reads `statfs`, which is +exactly what `fallocate` moves, so the condition it guards is genuinely present. + +**Setup.** A **real** 2 GiB file in opengist's data volume (so its capture writes 2 GiB), then +`fallocate` to leave `69G / 63G used / 3.0G avail / 96%` — **both floor terms deliberately still +clear**, so the run would start. Capture order was established empirically from the previous run's log +(opengist first, privatebin second), not assumed. + +**What happened, 06:40:03:** + +``` +Volume dump: opengist/opengist_opengist_data → 2.0 GB # unguarded +Volume dump: privatebin/privatebin_privatebin_data → 2.5 KB +App-data backup completed … 2 volume dump(s) (20.737s) +[WARN] Recovery unit capture REFUSED for opengist — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted +[WARN] Recovery unit capture REFUSED for privatebin — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted +[INFO] Event pushed: recovery_unit_capture_failed (error) — … ×2 (HTTP 200) +``` + +**What holds:** it refuses **per app** rather than aborting the run; **nothing was deleted** (both +units present afterwards); and the operator alert reached the hub — `recovery_unit_capture_failed`, +severity `error`, accepted `HTTP 200`, twice. + +**What does not hold — measured, not inferred.** The refusal message claims *"the previous unit is +untouched"*: + +| unit file | before | after | +|---|---|---| +| `privatebin/volume-dumps/privatebin_privatebin_data.tar` | `26c546c2…` | **`b538ab89…`** | +| `opengist/volume-dumps/opengist_opengist_data.tar` | 182,272 B | **2,147,666,432 B** | + +Both were rewritten by the earlier leg, while each `manifest.json` kept +`created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so +a unit's payload can be swapped under a stale descriptor and nothing inside the unit can detect it. + +**Cause, in the code and not the log.** The floor is consulted in exactly one place — +`m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which +writes a manifest and a compose copy: a few KB. `runVolumeDumps` (`backup.go:535`) — the leg that +writes the bulk, and the leg that consumed the reserve — has **no floor check at all**; its gates are +protected-stack, volume-less, disconnected, decommissioned. And it runs first *by design* +(`backup.go:483`). **The floor guards the cheap leg and not the leg that fills the volume.** + +This is why R-165 is IMPLEMENTED and not PROVEN-LIVE: that row records B2 as the deliberate +replacement for the bulkhead the `mp1` partition provided, and pre-merge the unguarded leg could only +fill a dedicated 20 G partition — post-merge it can reach Docker's data-root. Filed as **R-181**. +**No code was written**, per the runbook's §7. + +**Cleanup:** fill file and payload removed; `df` back to `1.2G used / 65G avail`; a clean re-run left +both units valid (`Volume dump … 178.0 KB` / `2.5 KB`, `App-data backup completed … (2.399s)`). + +--- + +## 5. demo-felhom — the pipeline proof + +**Install path — deliberately different, and this is the reason the second box exists.** + +``` +./felhom-host-install.sh --customer-id demo-felhom --mode appliance --vmid 9201 \ + --cores 3 --memory 12288 \ + --force-gitea-golden \ + --passphrase-file /root/.pp-demo-felhom +``` + +A copy of golden 0.192.0 already sat on this box's `local` storage — it is where the golden was +**baked** at 06:58 (a `.log` beside it), sha `54e2a4c4…`. `--force-gitea-golden` is the documented +C.3 customer path and overrides local discovery in **both** pre-flight and step 7, which pre-flight +confirmed: `golden: none local — will fetch + verify from Gitea in step 7/8`. The bake artifact was +left untouched. + +**`fetch_verify` succeeding against the vouched sha — the observable this box exists to produce:** + +``` +5/8 fetching agent binary v0.120.0 from Gitea … + verified sha256 a7763d31b55b5ce7… matches the hub manifest +7/8 fetching golden v0.192.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst + verified sha256 54e2a4c431daf580… matches the hub manifest + golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst +Day-0 provision SUCCESS — vmid=9201 host_id=demo-felhom-8363b5 customer=demo-felhom +``` + +Both artifacts — agent and golden — were fetched anonymously (the normal customer shape) and +sha-verified against the manifest. Controller `0.192.0` healthy. + +**Layout:** `mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=250G`; +`pct config 9201 | grep -c '^mp1:'` → **0**. Both binds real mounts +(`…disk--1[/docker]`, `…disk--1[/sys_drive]`), `df` one figure — `246G 977M 233G 1%` on all three +paths — `stat -c %d` = `64519` on all three, fstab carries both binds. + +**Reboots:** 1 — 09:18:55 → healthy 09:19:09; 2 — 09:19:12 → 09:19:26; 3 — 09:19:30 → 09:19:45. All +three paths still mountpoints after each; `uptime -s` = `2026-08-03 07:19:34 UTC`, matching reboot 3. + +**Journey:** claim (`302 → /`, discriminator flipped to `authentication required`; code supplied by +the operator after a resend) → deploy `opengist` (`{"ok":true}`, `Up (healthy)`) → capture +(`Volume dump: opengist/… → 178.0 KB`, `Recovery unit captured`, unit on +`/dev/mapper/pve-vm--9201--disk--1`, marker present inside the tar) → **restore** +(`{"ok":true,"message":"opengist visszaállítva (primary)."}` in **9.4 s**, marker back with identical +sha256 `bc5507987f3f56dc19a9c24785826c304f1a986998d4aeb0b06d1a810b59e939`, app healthy). + +**Ceiling:** 233 GiB available to a recovery unit, against the 45 GiB the pre-wipe `mp1` offered. + +**Step 8 was NOT repeated here, deliberately** — stating it rather than leaving it ambiguous. The +floor fired on demo-hp and R-181 characterises it fully; re-firing would add no information and would +mean filling a 246 G volume. + +--- + +## 6. Vouching + +Already done before the session (§2a), at **07:23:26 CEST on 2026-08-03**, by an operator action in +the hub UI. The session's own manifest write was the **agent** half, at **07:44:31**: +`Artifact manifest set: agent=0.120.0 golden=0.192.0 min_agent="0.113.0" wrapper_sha=true`. The +manifest afterwards, read back: `agent_sha256=a7763d31b55b5ce7…10b9d`, +`golden_sha256=54e2a4c431daf580…43b3e0`, `min_agent=0.113.0`, `wrapper_sha256=104db0a4…` (preserved +verbatim). The R-120 gate did not block: golden 0.192.0 equals the newest controller the fleet +reports. + +--- + +## 7. Teardown — all three layers + +1. **The machine.** No throwaway guest was created this session, so there is none to delete. VM 300 + `drill-r50` on demo-hp — the protected drift fixture — was never touched and is still `stopped`; + it is not in the `felhom` pool (`pvesh get /pools/felhom` listed only `lxc/9201`), so the + uninstall's shared-box logic never reached it. +2. **The host.** `local-lvm`, before → after: **demo-felhom 29.31 % → 1.37 %** (the old 200 G + 50 G + volumes returned; the new 250 G volume is thin and barely allocated). **demo-hp 39.13 % → + 36.75 %** — *higher than a clean reinstall would leave it*, because the floor test's 2 GB tar + blocks cannot be reclaimed: `fstrim` inside an unprivileged LXC returns + `FITRIM ioctl failed: Operation not permitted`. No operational impact (the guest shows 65 G free of + 69 G, the host 35.7 GiB free of 53.93), but it is real residue and is stated rather than rounded + away. Each box's data volume is now the only data volume; no old guest volumes remain. + **Residue found and cleared by hand on demo-hp:** `--uninstall` left the NAS network-storage units + `mnt-felhom\x2ddrives-Felhom\x2dShare.{mount,automount}` behind (automount `failed`, parent bind + still mounted) → **R-179**. demo-felhom left none, because it had no network share configured. +3. **The hub.** **No old records exist to dispose of, and this is the honest finding, not an + omission:** both enrollments were **idempotent** — `host REUSED (idempotent — existing + credential)`, `host_id: demo-hp-bb76ea` and `demo-felhom-8363b5`, the same ids as before. The + reinstalls therefore produced **no new host records**, so nothing was orphaned and nothing needed + deleting. Final register: `demo-felhom-8363b5 ONLINE 0.120.0`, `demo-hp-bb76ea ONLINE 0.120.0`, + `drill-r50-0a4f9a DOWN 0.113.0` — the same three rows as at P6. No scratch customers were created. + `/appliances` returns 404 on hub 0.89.0 — there is no appliance-record surface to clean. + +**Secrets:** both retrieval passphrases were moved file→file into 0600 files, used via +`--passphrase-file`, and `shred -u`'d afterwards on both hosts along with the session cookie files and +helper scripts; the local scratch copies were deleted. Nothing was written to a committed file and +`curl -w '%{redirect_url}'` was never used (R-132). + +--- + +## 8. Registers changed + +| Row | Change | |---|---| -| `documentation/architecture/07-backup-architecture.md` | **S-1: the contract changed in the same session.** New **§7.5.1** — the ceiling §7.5 describes no longer exists for a box built from golden ≥ 0.192.0, the bulkhead's replacement (B2) is recorded, and **R-175 is FIXED here**: the bound is restated as a function of `mp1` and scoped to split-layout boxes, naming all three real shapes | -| `documentation/architecture/00-capability-map.md` | new row — **IMPLEMENTED, not PROVEN-LIVE**, with the missing leg named (no box reinstalled, R-178) and the bake cited as the evidence it is | -| `documentation/backlog/OPEN-ITEMS.md` | **R-165** → shipped-not-yet-proven-live; **R-163 CLOSED** (its ceiling no longer exists); **R-175 CLOSED**; **R-176** (a) answered by P1, (b) **withdrawn** — every node is reinstalled, not migrated; **R-178 NEW** | -| `CONTEXT.md` | **S-13** (the variant, chosen on measurement; B2's floor; pruning rejected with its reason; no migration exists) and **S-14** (prove first, then vouch) | -| `STATUS.md` | rewritten section in plain language; the *Waiting on you* item cleared — both questions are answered; **two older entries trimmed** so the page did not grow | -| `scripts/CHANGELOG.md` | the host-install change, with why it was forced | +| **R-178** | **CLOSED** — both boxes reinstalled and proven, by two different supply paths | +| **R-165** | **IMPLEMENTED**, not PROVEN-LIVE — operator ruling; the layout half is proven, the B2 half is not (→ R-181) | +| **R-115** | **Third instance recorded** — agent 0.120.0 built, deployed to both hosts, never published | +| **R-181** *(new)* | The capture floor guards the recovery-unit leg and not `runVolumeDumps`; its "previous unit is untouched" claim measured false | +| **R-180** *(new)* | `--archive-storage` is not cross-checked against the ACL grant; the 403 lands at step 8/8, after root@pam has been rotated | +| **R-179** *(new)* | `--uninstall` leaves NAS network-storage systemd units behind when a share was configured | -## 4. CI — run ids and conclusions +**IDs established free before minting:** `grep -ro "R-179\b\|R-180\b\|R-181\b\|R-182\b"` over +`documentation/` and `*.md` → **0 hits**, and over all four repo roots (`felhom-agent`, +`felhom-controller`, `felhom.eu`, `app-catalog-felhom.eu`) → **0 hits**. `R-182` was checked and left +unused. -Checked by **PULL**, matching `head_sha` to each commit — CI mails only on failure, so an unchecked -green is an assumption. +--- -**Every commit in this session, across all three repos, is green.** +## 9. CI -| Repo | Commit | Task id | Run # | Conclusion | -|---|---|---|---|---| -| `felhom-controller` | `4be6467` (v0.192.0, the B2 floor) | 39 | 15 | **success** | -| `felhom-agent` | `cd6e267` (v0.120.0 + golden 3.0.0) | 41 | 3 | **success** | -| `felhom-agent` | `4bb84fc` (REPORT) | 43 | 4 | **success** | -| `felhom.eu` | `7406ac7` (phase-0 audit) | 40 | 21 | **success** | -| `felhom.eu` | `e3525e6` (host-install one grow) | 42 | 22 | **success** | -| `felhom.eu` | `14d8c00` (docs + registers) | 44 | 23 | **success** | +Appended after the push — run id and conclusion, per `CLAUDE.md`'s pull-check rule. -## 5. `--no-verify` +--- -**Not used anywhere.** Every push in this session ran its repo's `.githooks/pre-push` and passed. +## 10. Observations — noticed and NOT acted on -## 6. What remains — R-178 - -1. Reinstall **demo-hp** from golden 0.192.0 through the real installer path; prove claim → deploy an - app → back up → restore; show `df` proving one filesystem and a recovery unit landing on it. -2. Only then reinstall **demo-felhom** (it carries the PBS-DR/offsite tier, so it is the box whose - backup chain a reinstall actually disturbs). -3. **Then** vouch golden 0.192.0 in the hub, and flip the capability-map row to PROVEN-LIVE. -4. Re-run P1's restore-test against agent v0.120.0 — one command, turns a sound inference into an - observation. +- **The runbook's central claim was wrong in a way that mattered.** R-178 said *"a reinstall is now a + self-contained piece of work with no code left to write."* True about code; false about the artifact + channel — the agent half of the merge was unpublished, and following the documented path without + checking would have downgraded both boxes and produced a green, meaningless result. **No code change + was needed to make any step pass** — §10 asks this loudly, and the answer is no. What was needed was + a publish. +- **Prove-then-vouch was already spent when the session opened.** Not a defect in anything, but the + rule protected nothing because nothing enforced it. R-120's gate is the shape that would. +- **A per-stack `POST /stacks//backup` does not produce volume dumps** — the volume leg lives in the + full app-data pass. Not wrong, but the endpoint's name suggests otherwise and it cost time here. +- **privatebin's recovery unit carries no user data** (its catalog entry declares no `backup:` + section), so restoring it loses every paste. That may be intentional for an expiring, E2E-encrypted + paste bin — but nothing in the app's description tells the customer so. Catalog question, not filed. +- **V-c doubles systemd's mount-unit count** — every docker overlay appears twice, + `var-lib-docker-…-merged.mount` and `var-lib-felhom-docker-…-merged.mount`, because `/var/lib/docker` + is a bind of `/var/lib/felhom/docker`. Cosmetic, inherent to the chosen variant, no action. +- **`fstrim` cannot run inside the guest** (`EPERM`, unprivileged LXC), so space freed inside the guest + is not returned to the thin pool. It did not matter here; on a box that fills and empties repeatedly + it would. +- **The floor's own message mixes units** — it reports `64.2/68.7 GB used (93%)` against a threshold + stated as `97% used or 1.0 GiB free`, while `df` showed 96 %. GB-vs-GiB, so the percentage term + fires later than an operator reading `df` would expect. Minor; noted on R-181's fix shape rather + than filed separately. diff --git a/STATUS.md b/STATUS.md index 4c97d56..bb51550 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,6 +1,6 @@ # STATUS — what works, what's broken, what's next -**Updated 2026-08-02.** +**Updated 2026-08-03.** > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this > page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**, @@ -31,9 +31,12 @@ also delete it. A daily snapshot is armed as a stopgap, and we have never restor the data would vanish on the next update. Two are fixed, the third is now clear to fix because it is installed nowhere. *(R-156)* -**Local backups get 20 GB while apps get 50 GB.** An app that outgrows the smaller space stops being -backed up locally — and the off-site copy is made from the local one, so that stops too. Nothing is -lost: the last good copy is kept intact. *(R-163)* +**The reserve does not guard the step that actually fills the disk.** The rule that refuses a backup +when space runs low is checked at the wrong moment: the big write happens first, unchecked, and only +the small write after it is refused. So the thing meant to stop a runaway backup is the thing it +runs past. Worse, when it does refuse it says *"your last good copy is untouched"* — and we measured +that copy being overwritten by the earlier step anyway. Found by deliberately filling a rebuilt demo +machine. Nothing was deleted and you were emailed, both correctly. *(R-181)* **When that happens, only one page says so** — no email, no alert. The page that answers "is this app backed up?" is the one that stays silent. *(R-158)* @@ -65,9 +68,18 @@ barrier against a runaway backup filling the space the machine needs to run; put first means that when it comes down, the thing watching is already working and already tested. *(R-167, R-158)* -**The backup partition is gone from the base image.** A machine built from now on has one storage area -instead of two, so a backup can use whatever space the machine actually has free rather than a fixed -slice decided when it was built. The wall does not move; it stops existing. +**The backup partition is gone — and both demo machines now run on the new shape.** Each was wiped +and rebuilt from the new base image on 3 August and taken through the whole journey a customer takes: +set the machine up, install an app, back it up, restore it. One storage area instead of two, and the +space a backup can use went from 19 GB to 65 GB on the small machine and from 45 GB to 233 GB on the +big one. The wall does not move; it stops existing. Each machine was rebooted three times over and +came back correctly every time. **The two machines were rebuilt deliberately differently** — the +first from a copy of the image held locally, the second by the ordinary route a real customer takes, +fetching the image and checking it against the fingerprint we publish, so both the disk shape and the +delivery route are now proven rather than one twice. + +**Their previous demo apps and data are gone.** That was the point of a wipe and you approved it; the +machines now carry a couple of small test apps instead. **What replaced the wall.** It was quietly doing a second job — keeping a runaway backup from eating the space the machine needs to keep running. That job is now explicit: if a backup would push the disk @@ -81,32 +93,42 @@ quietly broke: one put your backups inside Docker's own storage, where the norma Docker would wipe them; another exposed all of Docker's internals to the part of the system that manages your drives. The third does neither, and costs one extra line of configuration. -**Nothing has changed on any existing machine.** They keep their current layout and go on working -exactly as before; they get the new shape only when they are reinstalled. The new base image is -deliberately **not switched on yet** — nothing will pick it up until a machine has been rebuilt from -it and checked, which is the next step. *(R-165, R-163)* +**The new base image is now switched on**, so any machine installed from here on gets the new shape. +The tester's box is untouched and keeps working exactly as before; it gets the new shape whenever it +is reinstalled. *(R-165, R-178)* ## What we're working on - **Now:** the last app whose data was never saved; today's decisions written down. -- **Next:** finishing the partition merge — **the build is done and the decision is made**; what is - left is to reinstall the two demo machines from the new base image and check one end to end. +- **Next:** fixing the reserve so it guards the step that fills the disk, and stopping it from + claiming your last good copy is untouched when it is not *(R-181)*. The partition merge itself is + done and proven on both machines. - **After:** rebuilding how the machine records whether an app is meant to be running. ## Waiting on you -- **How a new version reaches a machine.** Pushing the installer publishes it — half a minute later - every new machine downloads it, with no staging and no way back but another push. And publishing is - a step we remember rather than one the release performs, forgotten twice: a fix can be live here - and still not reach a new machine. Nothing is installing today, so this is the cheapest moment to - settle both. *(R-110, R-115)* +- **How a new version reaches a machine — and it has now been forgotten a third time.** Pushing the + installer publishes it: half a minute later every new machine downloads it, with no staging and no + way back but another push. And publishing is a step we remember rather than one the release + performs. On 3 August the new agent — the half of the partition merge that runs on the machine — + turned out to have been built and installed on both demo machines but **never published**, so a + rebuild would have quietly put the *old* one back and proved a version nobody ships. Caught before + the wipe and fixed in ten minutes, but only because someone happened to look. Nothing is installing + today, so this is the cheapest moment to settle it. *(R-110, R-115)* - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a session log; nothing suggests anyone else saw it. *(R-132)* -- **Nothing — both partition-merge questions are answered.** You chose the storage shape and the hard - stop; both are built. The tester's box needs no conversion: it will simply be reinstalled. *(R-165)* +- **Nothing on the partition merge itself** — the shape is chosen, built, and now proven on both demo + machines. The tester's box needs no conversion: it will simply be reinstalled. What is left is the + reserve defect above, which is ours to fix, not yours to decide. *(R-165, R-181)* ## Changed since last update +- **2026-08-03** — Both demo machines wiped and rebuilt from the new base image, and taken through + set-up → install an app → back it up → restore it. The backup space ceiling is gone and measured. + The hard stop that replaced the old partition was fired for the first time on real hardware — it + refused, it deleted nothing, it emailed you, **and it turned out to be watching the wrong step**; + that is now the top thing to fix. + - **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in a mode where reading a page cannot block a machine's status update; a one-word difference meant that setting had **never taken effect**, for the hub's whole life. Fixed and verified live. **Also found: diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 6d74a85..00fec6e 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -86,7 +86,7 @@ | Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped | | **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait | | **A failed per-app Tier-1 backup reaches the OPERATOR** (app, error, and the target filesystem's used/free bytes at the moment of failure) | controller **v0.191.0**, hub **v0.89.0** (R-158, closed by R-167) | **PROVEN-LIVE (2026-08-02)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it | -| **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.192.0** (R-165 / D-a / B2) | **IMPLEMENTED — NOT PROVEN-LIVE** | `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **NOT PROVEN-LIVE, and the missing leg is named: no box has been reinstalled from this golden (R-178).** Per this map's own rule a PROVEN-LIVE claim needs an end-to-end citation, and "the golden baked" is not "a box built from it works". **The golden is deliberately UNVOUCHED** so no fresh install picks up an unproven layout — prove first, then vouch (`CONTEXT.md` S-14). Every box in the field is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity | +| **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.192.0** (R-165 / D-a / B2) | **IMPLEMENTED — the LAYOUT half is PROVEN-LIVE (2026-08-03); the REFUSAL half is not (R-181)** | `REPORT.md` (R-178 reinstalls) + `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately.** **Proven (R-178, 2026-08-03):** *"a local backup is bounded by the box's FREE SPACE, not by a partition set at build time"* — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp `--golden `; demo-felhom the normal manifest route with **`verified sha256 54e2a4c431daf580… matches the hub manifest`**), each showing `mp0` at `/var/lib/felhom` with **no `mp1`**, both consumer paths real mounts on ONE filesystem (`stat -c %d` = `64519` on all three), 3/3 reboots each, and claim → deploy → backup → **restore** with a planted marker returning byte-identical. Space available to a recovery unit measured at **65 GiB / 233 GiB**, against the **19 GiB / 45 GiB** those boxes' `mp1` slices offered. **NOT proven — and measured FALSE in part:** *"a capture that would exhaust it is refused per app rather than allowed to stop the container runtime"*. The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — **but it is checked only in `captureAllRecoveryUnits`, while `runVolumeDumps` writes the bulk with no floor check at all**, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → **R-181**. The golden **is now VOUCHED** (2026-08-03, hub `Artifact manifest set: … golden=0.192.0`), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity | | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | | | **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index d6efc59..5767345 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -15,7 +15,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-94** | ~~A hand-synced version constant drifts, and the gate that would catch it is never run~~ | **CLOSED — SHIPPED** (hub v0.87.0, 2026-08-02) | — | **All three legs closed.** **(a) closed by DELETION, not derivation** — deriving is not achievable honestly: the Setup command fetches `felhom-host-install.sh` at RUN TIME from a website that git-syncs `main` every 30 s (R-110), so no build-time value in the hub can be true, and a number that is wrong carries a version number's authority while being a guess. The const, the `pageData.ScriptVersion` field, its assignment and the rendered label are gone; a NOTE stands where the const was so it is not helpfully re-added. **(b)** `hostinstall_gates.py` gate 1 INVERTED — it now asserts the hub carries **no** host-install version literal, in six code shapes across every `.go`/`.html` under `hub/`; and the gate is now invoked, by `scripts/repo_gates.py` and the pre-push hook (→ R-29). **(c)** the tautological `render_test.go:219` assertion is deleted, not replaced — there is no version to assert. It was demonstrated PASSING with the const at `9.9.9` while the script was 1.22.0. The label had been wrong for 19 days (since 2026-07-14) | — | | **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly | CC | | **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` **DEFERRED LEG, AND IT RECURRED → R-115.** This row's shipped half stands and is not reopened: the bump happened, was verified, and was proven end-to-end by the E-2d install. But its own closing line — *decide whether publishing joins the release train rather than staying a remembered step* — was never acted on, and agent 0.114.0 reproduced the exact condition the same afternoon. The recurrence is filed as **R-115**, not as a reopen, because the stale-channel finding is closed while the process defect that caused it is a distinct problem with a distinct fix. | CC | -| **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC | +| **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design. **THIRD INSTANCE, 2026-08-03 — and it was found by a runbook that had been told there was nothing left to do.** Agent **v0.120.0** — the agent half of the R-165 merge — was built, committed at `cd6e267`, and deployed to BOTH demo hosts, and was **never published**: `GET …/generic/felhom-agent/0.120.0/felhom-agent` → **HTTP 404** (0.119.0 → 200), and the hub manifest accordingly vouched **0.119.0**. The consequence is the sharpest yet, because installer step 5's idempotent skip requires `installed == vouched` EXACTLY: a documented-path reinstall would have **downgraded both boxes** from the merge-aware 0.120.0 to the pre-merge 0.119.0 — silently, since the current `step_grows` sets `SYSDATA_GROW=0` so 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on error) never fires and the install would have *succeeded* while proving a stack nobody ships. **R-178's own row asserted `agent v0.120.0 is live on BOTH hosts` and `no code left to write`; both were true and both were beside the point** — the gap was publication, which no one checks. Fixed in-session on the operator's ruling: `scripts/publish-agent.sh 0.120.0` (sha `a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`, round-trip GET verified) then vouched, and both reinstalls then fetched and sha-verified it from Gitea. **This is the third instance of a row that has been WAITING-ON-OPERATOR since 2026-07-29; option (b) — a gate that refuses to deploy or vouch an unpublished version — would have caught all three** | CC | | **R-116** | ~~**The drive-absent alarm and its recovery were a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`; `backup_target_absent` never fired at all**~~ | **SHIPPED + PROVEN-LIVE** (agent v0.116.0, 2026-07-30) | — | **CLOSED. The full four-event sequence, on the wire, on a fresh box** (`audits/R116-v0116-2026-07-30.md`): `backup_target_absent (error)` on detach → `backup_target_restored (info)` on return for the TARGET, and `storage_disconnected (error)` → `storage_reconnected (info)` for a NON-target drive on the same box four minutes apart. **Two matched pairs, correctly discriminated — and discrimination is proven NON-trivially for the first time**, since both prior runs had the target itself emit the generic event. Gate fired in **3 s**; all four events reached the hub, so the specific alarm, its severity, its Hungarian copy and the hub routing are now exercised end-to-end. **Over-correction PASSES with a positive observable** (0 ABSENT lines / 0 drive events over 2m14s with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking). **NARROWED by the R-117 spike (`audits/SPIKE-r117-bind-liveness-2026-07-30.md` §12), and it stands as written:** the 2 `RETURNED` lines are a genuine positive observable, so rule 3 is satisfied — but `degraded:false` over that window was read off a drive whose bind was **dead** (R-117), so the window evidences **"the gate did not over-fire"** and **NOT** **"the drive was healthy."** No other part of this row changes: every input to the pairing fix is configuration-derived (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), which R-117 does not touch. Ran on a nested PVE on **demo-hp** per `runbooks/target-selection.md` — through the **real day-0** from the v1.25.0 ISO, with the agent **installed unaided from the vouched Day-0 manifest** (published sha `b47c5c4dab641ee5…`, independent registry GET verified, manifest read back), drives enrolled through the real endpoints, device loss a real hot-detach. **THE FIX, and the ruling is the substantive part:** the mechanism was first isolated from the captured payload (`DIAG-r116-disks-payload-2026-07-30.md`) after two fixes aimed at shapes that do not occur. **Both smaller-looking options were REJECTED because they regress R-114** — `backup_target_offer.go:79` reads `BackupTarget && MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns before its `TargetAbsent` branch, so back-filling `MountPath` on the Observe row **or** flagging the registry row (whose `MountPath` is the stale unit-file value) would have told the customer the backup target is fine while its drive was gone. **R-114's correctness was resting on R-116's bug** — a coupling invisible until the payload existed. Taken instead: the Observe row gets the **guest path only** (`mount_path` stays `""`, which is true) from a new `ConfigPath` (`json:"-"`, so the cross-repo golden + key-set contract is untouched), and the union row is deduped **on guest path** — the join being CONFIGURATION (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849; 4 red-proofs each asserted to land, and red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. v0.115.0 left in place (inert, harmless). Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded. **Caveat: the drill's controller was 0.185.1 from the golden, which PREDATES R-114**, so its absent-state banner showed the old false copy — the golden being a release behind, not a regression → **R-120** | — | | **R-120** | ~~**The golden baked a controller that predated R-114 + R-112, so a FRESH box showed the customer the WRONG absent-target message**~~ | **CLOSED — golden rebaked + PROVEN-LIVE, and the class now has an ENFORCED gate** (golden 0.186.0 + hub v0.82.0, 2026-07-30) | — | **`audits/R120-golden-rebake-2026-07-30.md`.** **Half 1 — the artifact.** Golden **0.186.0** baked from `main`'s controller in the DooPlex bake fixture (overlay2 OK, **3 mounts**, FATAL 0, exclusions 0, 618 MB, upload **201**, `GOLDEN_SHA256=b760ac6a33e70700…`, token-leak grep 0, GL-1 teardown, `drill.qcow2` back to `virgin`). Three observables: **published** — anonymous GET (what the installer does) 200 / 648930639 bytes / sha identical to the bake; **vouched** — manifest read BACK; **resolved** — `Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)`. Floor **untouched** per publish-train rule 2 (`min_controller_version` still 0.156.0; it is a separate form); MinAgent left 0.113.0 as 0.186.0 declares. **Proven on a REAL day-0, not the fixture** (per the Part-1 rule now in `runbooks/target-selection.md`): VM 9402 on demo-hp from the v1.25.0 ISO → `Controller elindult (0.186.0)`. With the target detached the endpoint returned the **`TargetAbsent`** copy — *„A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod…"* — **and `offer_path` absent entirely**; the day-old read on the 0.185.1 golden had returned the false system-disk message **plus** an offer of the other drive. **Half 2 — the mechanism, operator ruling REFUSE.** hub **v0.82.0**: the gate sits in `hub/internal/web/configs.go` `handleSetArtifacts` immediately before the only write — the sole UI path to `SetArtifactManifest` — so it runs on every vouch without anyone choosing to, and it **refuses** rather than warning. Signal: `store.NewestReportedControllerVersion()` over `reports.controller_version`, **semver-compared in Go** (`MAX()` in SQL ranks 0.99.0 above 0.186.0 — a pair this fleet has shipped). Fail-open in exactly two deliberate cases: empty golden field, unknown fleet version. **NEAR-MISS RECORDED: the first draft read `guests.controller_version`, a column that exists and that NOTHING writes** — it would always have seen `""` and failed open, i.e. inert, this gate's own failure shape, one grep from shipping. 4 tests through the **production handler** over httptest (never a seam), the refusal asserting **both** the flash **and** that the manifest was not written; red-proof: deleting the block makes the stale golden vouchable again. **PROVEN LIVE on the deployed hub by re-attempting the original mistake:** vouching 0.185.1 → `HTTP 303 …flash=golden_behind_fleet` + `[WARN] artifact vouch REFUSED: golden 0.185.1 is older than the newest controller the fleet reports (0.186.0)`, and the manifest read back **unchanged at 0.186.0**. Recorded on **R-29's audit list** (`ROADMAP.md`) as the **first enforced gate** beside its three orphans, so the contrast is kept — the orphans are unchanged. Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded, exactly as `sess-e` was (and `sess-e` was deleted this run) | — | | **R-117** | ~~**A drive's guest bind becomes a DEAD MOUNT while every signal reads healthy — and it happens in TWO ways, only one of which the original framing covered.** (a) *after a detach/return*: the host raw mount heals onto the NEW device via its fs-UUID-keyed unit while the bind still names the OLD one, so the gate takes its `Return` branch and restarts the customer's apps onto a namespace that `EIO`s on every call; (b) *in STEADY STATE, no cycle at all* — a device that errors without disappearing leaves the raw mount `active`, `BoundUnderParent` `true` and the drive never `Disconnected`, so **the gate produces no action and NOTHING is emitted on any channel**~~ | **SHIPPED + PROVEN-LIVE** (agent **v0.117.0**, 2026-07-30) | — | **CLOSED. `audits/R117-v0117-2026-07-30.md`.** `BoundUnderParent` gains a THIRD term at both /disks sites: `bindLiveness` reads `/proc` only and requires (a) **the bind names the same device as the raw mount** and (b) **the filesystem has not aborted** (`shutdown` **or** `emergency_ro`, both measured). **BOTH CHECKS ARE LOAD-BEARING and this is the substantive part:** R-117 was filed as a detach/return defect, but a device that fails WITHOUT disappearing gives the identical all-signals-healthy state with the **devnos EQUAL** and the drive never `Disconnected`, so the gate emits nothing at all, indefinitely (R-117a) — the device comparison alone cannot see it, and a P1-only fix passes every payload test (red-proof RP3 exists for exactly that). **THREE states, never a bool:** `{Unknown, Live, StaleDevice, Aborted}`, `Unknown` is the zero value, and every caller reads `Usable()` where unknown counts **PRESENT** (absent stops a customer's apps — the `newestArchiveOn` trap). **NO NEW RECOVERY PATH:** `AttachDrive`'s normalize leg already did the repair and three call sites already invoked it (20 s ticker, agent startup, and **the controller's `Return` branch BEFORE `restartStacks`**); all three were defeated by `if n == 1 && GuestSeesMount(...)` logging *"fully live, no-op"* about an EIO namespace. **RULING (asked for, given, flagged for overrule):** `StaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, guest never restarts — init PID identical); `Aborted` ⇒ **quiet no-op and SURFACE**, because a re-bind lands on the SAME dead superblock and this runs every 20 s = an infinite silent retry that masks the state. No operator decision required: it routes an already-broken state into the **existing** gate, event types and Hungarian copy — no new customer-facing concept — and the alternative is apps writing documents into a filesystem that rejects every write. **ORDERING TRAP caught by a test:** abort-first classifies the real return state as aborted (its stale bind carries `shutdown` too) and refuses the repair **while still reporting correctly**, so the abort flag is read off the RAW mount in the stale case. **LIVE on demo-hp** (brought 0.113.0 → 0.117.0 first — see R-121): RETURN `raw 8:32 / bind 8:16 shutdown` ⇒ `stale-device`, usable **false**; IN-PLACE `both 252:11 emergency_ro`, raw unit still `active` ⇒ `filesystem-aborted`, usable **false**; healthy ⇒ `live`; **340–497 µs**. **No block I/O proven by strace** (only `/proc/self/mountinfo`, **0** statfs) — the Part 1 `CLAUDE.md` fence applied to its own first consumer. **No regression through the REAL pipeline:** `GET /disks` with the controller's own credential shows the live backup-target drive `bound_under_parent=True`, with 32 gate lines in 3 min as the positive observable and zero spurious transitions. Tests **849→863**, 29/29 green, **6 red-proofs each verified to land** — and **RP1 failing to fail exposed a HOLLOW test**: the aborted fixture used a `/dev/mapper` device, for which `RoleForStorage` derives `role=system`, and a system row never runs the conjunction, so it reported false by DEFAULT and no mutation could fail it. Fixtures now assert the production row shape first. Teardown all 3 layers; hub layer = the vouched manifest, **retained** (it is the product, not scratch). **NOT covered:** the stale-bind repair on hardware — `StablePathForRaw` hardcodes the live parent, so it would write into guest 9201's namespace (R-117h); and sustained-load behaviour, still unmeasured. Follow-ups **R-117g** (no guided recovery for an aborted fs), **R-117h** (parent dir not test-seamable), **R-121** | CC | @@ -92,7 +92,10 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-174** | ~~**The app-stop guard's crash recovery started apps onto MISSING drives — a regression in v0.189.0 code.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0, 2026-08-02) | — | **Found by REVIEW on 2026-08-02, in code shipped 2026-08-01, and closed the same session — R-171 one path over.** `appStopGuard.SetStarter(stackMgr)` handed `Recover` the RAW stack manager, whose `StartStack` has no drive gate, and `Recover` runs **at startup** — exactly when an external drive may not have come back. So: a backup stops an app, the box loses power, the drive does not remount, and the app is started on a missing drive. The rule was not new — the API's own `startGatedByMissingDrive` already refused this to the customer; the guard bypassed it. **`bootDriveGate` could NOT be reused whole**, and the reason is recorded in the code: its holder #2 reads `bootAppStopGuard.HeldStacks()`, which during `Recover` is **the guard's own marker** — it would refuse every recovery it was meant to perform — and holders #1/#2 read package-level vars assigned AFTER `Recover()` runs, so a whole-gate reuse would be correct only by accident of nil-safety. Holder #3 is extracted into a shared `driveStartGate` with **two callers, one implementation**, and `TestBootDriveGateAndAppStopShareTheDrivePredicate` pins the delegation. **A REFUSAL IS NOT A FAILURE:** new `ErrStartRefused` + a `Refused` bucket — both keep the marker, only `Failed` alarms, because routing a deliberate hold into `NotifyBackupFailed` (customer-enabled by default) is the very R-171 false alarm this fixes. `main.go` guards on `Alarming()`, not `!= nil`, and the pre-existing seam test was TIGHTENED to require it. **Live on 9201, both directions:** drive held unmounted → `refusing to restart "calibre-web" … drive /mnt/felhom-drives/hdd_1 is not a live mountpoint`, marker retained byte-identical, zero containers started, `not alarming`; drive returned → `restarted calibre-web`, marker CLEARED. **ID established free:** `grep -ro "R-174\b" documentation/ *.md` → 0 hits | — | | **R-175** | ~~**`07-backup-architecture.md` §7.5 states ONE box's size bound as if it were the fleet's.**~~ | **CLOSED — FIXED 2026-08-03** (same pass as R-165) | — | **Measured, not inferred** (`audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1: `pct config 9201` on both hosts). Independent of the merge — the sentence is wrong today and will be wrong differently after R-165. **The fix is to state the bound as a FUNCTION of `mp1`, not a constant**, and to say which box any quoted figure came from. Same class as the comment-asserting-an-invariant rule: a doc stating a fleet-wide number that only one machine satisfies reads as settled and is not. **ID established free:** `grep -ro "R-175\b" documentation/ *.md` → 0 hits **FIXED.** §7.5 gained a **7.5.1** which (a) states plainly that the bound is a FUNCTION of `mp1` and applies only to a box still on the split layout, naming all three real shapes, and (b) records that the ceiling itself has been removed by R-165 for boxes built from golden ≥ 0.192.0. Fixed in the same pass as the merge rather than filed and forgotten, because the section would otherwise have been wrong in two ways at once | CC | | **R-176** | **Two prerequisites for the R-165 merge are UNMEASURED, and both are cheap.** (a) Whether a **pre-merge archive** (carrying `mp1`) restore-tests cleanly into a **merged-layout** guest — reading `mountParity` (`felhom-agent/internal/reconcile/restoretest.go:347`) says it should, because the restore recreates `mp1` from the archive so archive and restored guest agree; **that was reasoned from source and never executed.** (b) The in-place per-box migration (move `/felhom-data` onto `mp0`, drop the slot, verify) has **never been rehearsed even once**, so "is the box restorable at every point of it?" is currently unknown | **(a) ANSWERED 2026-08-03 (P1: PASS). (b) NOT REQUIRED — operator ruling: every node is reinstalled, none migrated** | blocks R-165 landing safely | **Filed because this project's own record is that FOUR production designs specced against unvalidated mechanisms were all wrong** — which is exactly why R-165's own spike refused to design. Both are one command on a **Tier-0** box (D-d: both demo boxes are disposable). (b) is only required work if Peti's box turns out to need migrating rather than reinstalling — the hub cannot answer that (M5: `peti-felhom` exists as a customer with **no host in the register**), so it is the operator's input. **ID established free:** `grep -ro "R-176\b" documentation/ *.md` → 0 hits **UPDATE 2026-08-03.** **(a) is measured and passed** — `audits/SPIKE-r165-phase0-2026-08-03.md` P1: a real pre-merge archive (`mp0+mp1`, confirmed from its own vzdump log) restore-tested on demo-hp, `pass: true`, `mount_parity: ok`, 84 s, with `mountParity` untouched. One limit stated rather than glossed: it ran with the pre-merge agent because the merged one did not exist yet, and the comparison is archive-vs-its-own-restore which never consults the host layout — **re-run it once against agent v0.120.0**, which is one command. **(b) is withdrawn, not deferred:** the operator ruled that every node is REINSTALLED rather than migrated in place (both demo boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed in a few weeks), so the in-place migration rehearsal has no consumer. Recorded explicitly rather than silently skipped | CC | -| **R-178** | **The merged golden (0.192.0) is built and published but NO BOX HAS BEEN REINSTALLED FROM IT, and it is deliberately UNVOUCHED.** `build-golden.sh` v3.0.0 baked it with variant V-c and every retargeted assertion passed on the real bake (`including mount point mp0 ('/var/lib/felhom')`, no `mp1` line, `both paths are ONE filesystem`); it is in the registry (HTTP 200, sha `54e2a4c431daf580…`). What has NOT happened is Part 4: reinstall each demo box from it and prove claim → deploy an app → back up → restore | **READY (M) — NEW 2026-08-03** | blocks R-165 reaching PROVEN-LIVE; blocks the capability-map row | **The golden is UNVOUCHED ON PURPOSE and that is the safe state**, not an oversight: vouching is what makes a fresh install pick it up, so vouching a golden no box has been proven from would put an unproven disk layout in front of the next install anywhere. **Prove first, then vouch** — the bake script's own output treats the hub record as a separate deliberate step for this reason. **Everything else for the merge is shipped and green:** controller v0.192.0 (the B2 floor) is live on 9201, agent v0.120.0 is live on BOTH hosts, and `felhom-host-install.sh` computes the single grow from the thin pool. So a reinstall is now a self-contained piece of work with no code left to write. **Order matters: ONE box at a time**, demo-hp first, proven end to end, and only then demo-felhom — two in parallel leaves no working reference to compare against. Note demo-felhom carries the PBS-DR/offsite tier, so it is the one whose backup chain a reinstall actually disturbs. **ID established free:** `grep -ro "R-178\b" documentation/ *.md` → 0 hits | CC | +| **R-181** | **The capture floor guards the cheap leg and not the leg that fills the volume — and its refusal message asserts an invariant the code does not provide.** B2 (controller v0.192.0) is recorded on R-165 as the deliberate replacement for the bulkhead the `mp1` partition used to give. It is consulted in exactly one place — `m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which writes a manifest and a compose copy: **a few KB.** The leg that writes the bulk, `runVolumeDumps` (`backup.go:535`), has **no floor check at all** — its gates are protected-stack, volume-less, disconnected, decommissioned — and it runs FIRST, by design (*"MUST run before captureAllRecoveryUnits so the manifests enumerate the fresh tars"*, `backup.go:483`). So the write that fills the filesystem is unguarded, and the floor then refuses the write that would have cost almost nothing. **Second limb: the refusal message is false.** `recovery_unit.go:331` prints *"the previous unit is untouched and NOTHING was deleted"*. Nothing was deleted — true. Untouched — **measured false**: privatebin's `volume-dumps/privatebin_privatebin_data.tar` went `26c546c2…` → `b538ab89…` and opengist's went **182,272 B → 2,147,666,432 B**, both rewritten by the earlier leg, while each unit's `manifest.json` kept `created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so a unit's payload can be swapped under a stale descriptor and **nothing in the unit can detect it** | **READY (M) — NEW 2026-08-03** | blocks **R-165** reaching PROVEN-LIVE | **FIRST LIVE FIRING OF B2, and it is why the runbook asked for one.** Proven on demo-hp 2026-08-03 06:40:03 on a box reinstalled from the merged golden (R-178). Method: a real 2 GiB file in opengist's data volume, then `fallocate` to bring the filesystem to 96 % used / 3.0 GiB free — both floor terms deliberately still clear, so the run started. **The `fallocate` instrument was proven before use** (5 GiB moved guest `df` 977M→6.0G while thin-pool `data_percent` stayed 29.03 → 29.03: zero blocks allocated), because demo-hp's thin pool is 53.93 GiB and a real fill to 97 % of a 70 G volume would have exhausted it and corrupted every guest on the box including the `drill-r50` fixture. Sequence observed: opengist's volume dump wrote **2.0 GB unguarded** → free fell to 1.0 GB → **both** apps' recovery-unit captures were then REFUSED on the `1.0 GiB free` term, each pushing `recovery_unit_capture_failed` (severity `error`) to the hub, accepted HTTP 200. **What DOES hold: it refuses per app rather than aborting the run, it never deletes, and the alert reaches the operator.** **Fix shape, not written this session by design (§7 of the runbook):** the floor belongs before the write in `runVolumeDumps` too, the message must stop claiming what the earlier leg has already falsified, and per `CLAUDE.md` *"a comment asserting an invariant needs a test pinning it"* the pinning test must assert the **consequence** (after a refusal, is the previous unit's payload byte-identical?) and not the mechanism. **Class:** the sixth entry in `CLAUDE.md`'s own table of shipped guarantees the code did not provide — found, as four of those were, only on live hardware | CC | +| **R-180** | **`--archive-storage` is accepted without checking the agent's token will ever be granted on it, and the failure lands at step 8/8 — after the root@pam password has already been rotated.** `felhom-host-install.sh` validates the archive storage EXISTS (`pvesm status --storage`, `:1583`) and that the golden volid RESOLVES on it (`:1661`), both in pre-flight. It never checks that storage against the ACL set it is about to grant, which is the fixed default `local local-lvm felhom-pbs` (`--acl-storages`, which `runbooks/day0-install.md` tells the operator **not** to pass). A storage outside that set therefore passes every pre-flight gate and dies at the last step | **READY (S) — NEW 2026-08-03** | — | **Hit live on demo-hp 2026-08-03** during R-178 Phase A, self-inflicted and therefore a clean demonstration: the golden was staged on `felhom-backup` (the enrolled NVMe, where the box's vzdumps live) and `--archive-storage felhom-backup` passed. Pre-flight passed; steps 1–7 ran; step 8 returned `reconcile: bring-up restore: proxmox: POST /nodes/felhom-host/lxc -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)`. **The cost is the ORDER, not the error** — by the time it fires, step 2 has minted the PVE token, step 4b has **rotated root@pam and vaulted it** (so the old console password is already dead), and step 5 has installed the agent. Recovery was `--resume` after moving the golden to `local`, which worked cleanly. **This is statically checkable in pre-flight**: `ARCHIVE_STORAGE ∈ PVE_STORAGES` is a one-line assertion over two variables both known at `:1583`. Same class as R-29 — the checkable thing that nothing checks | CC | +| **R-179** | **`--uninstall` leaves the NAS network-storage systemd units behind, with the automount in `failed` state and the parent bind still mounted.** The teardown's residue-diff provenance (`day0-install.md` Part E: *"a full-filesystem diff against the pre-install baseline showed zero Felhom-named leftovers"*) is from **v1.9.1**, which predates the NAS network-storage feature. A box that has ever had a network share configured keeps `/etc/systemd/system/mnt-felhom\x2ddrives-.mount` and `.automount` after a full uninstall | **READY (S) — NEW 2026-08-03** | — | **Observed on demo-hp 2026-08-03** after `--uninstall --vmid 9201`: `mnt-felhom\x2ddrives-Felhom\x2dShare.automount` **loaded failed failed**, its `.mount` `loaded inactive dead`, and `mnt-felhom\x2ddrives.mount` still `active mounted` — the uninstall's own output had warned `/mnt/felhom-drives/Felhom-Share is busy — NOT forcing` and `/mnt/felhom-drives root bind left mounted`, which is correct behaviour (it never forces an unmount) but is not teardown. Cleared by hand before the reinstall: stop both units, remove both unit files, `daemon-reload`, unmount the autofs then the parent. **NEGATIVE CONTROL, same day:** demo-felhom's uninstall left **nothing** (`ls /etc/systemd/system | grep -i felhom` → only the unrelated `felhom-bootstrap.service`; no felhom mounts) — because that box had no network share configured. **So the residue is conditional on the feature having been used, which is exactly why a diff taken on a box that never used it reported clean.** `felhom-bootstrap.service` is NOT residue — it is the ISO first-boot unit, `disabled`+`inactive`, exactly-once and already fired | CC | +| **R-178** | **The merged golden (0.192.0) is built and published but NO BOX HAS BEEN REINSTALLED FROM IT, and it is deliberately UNVOUCHED.** `build-golden.sh` v3.0.0 baked it with variant V-c and every retargeted assertion passed on the real bake (`including mount point mp0 ('/var/lib/felhom')`, no `mp1` line, `both paths are ONE filesystem`); it is in the registry (HTTP 200, sha `54e2a4c431daf580…`). What has NOT happened is Part 4: reinstall each demo box from it and prove claim → deploy an app → back up → restore | **CLOSED — BOTH BOXES REINSTALLED AND PROVEN (2026-08-03)** | blocks R-165 reaching PROVEN-LIVE; blocks the capability-map row | **The golden is UNVOUCHED ON PURPOSE and that is the safe state**, not an oversight: vouching is what makes a fresh install pick it up, so vouching a golden no box has been proven from would put an unproven disk layout in front of the next install anywhere. **Prove first, then vouch** — the bake script's own output treats the hub record as a separate deliberate step for this reason. **Everything else for the merge is shipped and green:** controller v0.192.0 (the B2 floor) is live on 9201, agent v0.120.0 is live on BOTH hosts, and `felhom-host-install.sh` computes the single grow from the thin pool. So a reinstall is now a self-contained piece of work with no code left to write. **Order matters: ONE box at a time**, demo-hp first, proven end to end, and only then demo-felhom — two in parallel leaves no working reference to compare against. Note demo-felhom carries the PBS-DR/offsite tier, so it is the one whose backup chain a reinstall actually disturbs. **ID established free:** `grep -ro "R-178\b" documentation/ *.md` → 0 hits. **CLOSED 2026-08-03 — both boxes reinstalled from the merged golden, by two DIFFERENT supply paths, and proven end to end** (`REPORT.md`). **demo-hp — the layout proof**, installed with `--golden local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst` (installer v1.22.0, sha `ed02acb2…`, byte-identical to the repo copy): `mp0 …mp=/var/lib/felhom,backup=1,size=70G`, **no `mp1`**; `/var/lib/docker` → `…disk--1[/docker]` and `/mnt/sys_drive` → `…disk--1[/sys_drive]`, both real mounts, both writable, both in `/etc/fstab`; ONE `df` figure (69G/65G) and `stat -c %d` = `64519` on all three paths; **reboots 3/3** (08:13:33 / 08:13:59 / 08:14:19, controller healthy in 12s/7s/7s, all three still mountpoints after each). **demo-felhom — the pipeline proof**, installed with `--force-gitea-golden` and NO local golden used (preflight logged *"golden: none local — will fetch + verify from Gitea in step 7/8"*, bypassing the 06:58 bake artifact sitting on the same box): **`verified sha256 54e2a4c431daf580… matches the hub manifest`** for the golden and **`verified sha256 a7763d31b55b5ce7…`** for the agent — the observable this second box exists to produce; `mp0 …size=250G`, `grep -c '^mp1:'` → 0, one `df` figure (246G/233G), reboots 3/3 (09:18:55 / 09:19:12 / 09:19:30). **Journey proven on BOTH**, endpoint-level (no browser on DooPlex — the exact endpoints the dashboard's own JS calls): claim (`POST /claim` with the pre-auth HMAC CSRF + `felhom_claim_csrf` cookie; gate discriminator flipped `dashboard not yet claimed` → `authentication required`) → deploy (`POST /api/stacks//deploy`) → capture (`POST /api/debug/backup/dbdump`, which runs the production `RunDBDumps`) → **restore** (`POST /backup/restore`): a planted marker deleted from the live volume came back with an **identical sha256** on each box (`ac1faae6…ae861` privatebin/demo-hp in 9.2s; `bc550798…b59e939` opengist/demo-felhom in 9.4s), recovery units on the single volume in both cases. **Ceiling gone, measured:** 65 GiB (demo-hp) and 233 GiB (demo-felhom) available to a recovery unit, against the 19 GiB and 45 GiB their pre-wipe `mp1` slices offered. **Two deviations, both the operator's call and both recorded:** the golden was ALREADY vouched when the session opened (hub log `2026/08/03 07:23:26 Artifact manifest set: agent=0.119.0 golden=0.192.0`, ~10 min before the first read of this session — so §7's prove-then-vouch order was already spent and the operator elected to accept it); and agent **0.120.0 had never been published**, so the vouched agent was 0.119.0 — published + vouched before the reinstalls (→ **R-115** third instance). **Three new findings: R-179, R-180, R-181** | — | | **R-177** | **There is no operator-triggerable "run the fill check now" path.** `fill-watch` is reachable only on its daily 03:30 schedule plus the once-at-startup run added in controller v0.191.1 — so the only way to exercise it on demand is to restart the controller | **READY (S) — NEW 2026-08-02** | — | **Noticed while live-validating R-167 on 9201, not by a failure.** It cost a controller restart per observation during validation, and it costs the same on a support call: after a customer frees space, nobody can confirm the warning has cleared without restarting their controller or waiting until 03:30. **Partially mitigated already** — v0.191.2 makes every run log a positive observable (`checked N filesystem(s), M unreadable/skipped, K notification(s)`), so at least a run that DID happen is visible; the gap is triggering one. The scheduler has `GetJobs` but no run-now, so this is a general affordance, not a fill-watch one — **scope it as "run a named scheduler job now", operator-gated.** **ID established free:** `grep -ro "R-177\b" documentation/ *.md` → 0 hits | CC | | **R-173** | **The hub's SQLite PVC is excluded from every Longhorn backup job.** `pvc/hub-data` carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly` (04:00 / Sun 05:00) are the ONLY recurring jobs and both target the `default` group — so the 128 MB `/data/hub.db` has **no volume-level backup**. That database holds `host_recovery` (every managed box's break-glass root password), `host_escrow` + `host_escrow_superseded` (escrow custody), `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg endpoints/peers — i.e. the material several documented recovery routes depend on | **READY (M) — NEW 2026-08-02** | — | **Noticed while checking the blast radius of the R-172 WAL change, not by a failure** — the WAL work needed to know who copies this file, and the answer turned out to be nobody on a schedule. **Establish before designing:** (a) whether the exclusion is deliberate (a 1 Gi RWO Longhorn volume snapshotting a 128 MB SQLite file is cheap, so the label looks like a leftover rather than a decision) and by whom; (b) whether anything else backs it up out-of-band that this census missed — the `_recovery-inventory-2026-07-28.md` records a MANUAL hot copy, which is not a backup. **When it is designed, it must be WAL-aware** (R-172): a volume snapshot of a live WAL database is crash-consistent and replays on open, which is fine, but any file-level copy must take `hub.db-wal` too or it silently loses the newest writes. **Grep establishing the ID was free:** `grep -ro "R-173\b" documentation/ *.md` → 0 hits | CC | | **R-158** | ~~**A local Tier-1 app-data backup failure reaches no hub channel.**~~ | **CLOSED BY R-167 — SHIPPED + PROVEN-LIVE** (controller v0.191.0 + hub v0.89.0, 2026-08-02) | — | **Closed by the wire it named; no second row was filed for it** (R-167 subsumes and widens it). New `unitNotify` seam + `SetUnitNotify` beside the manager's existing three, called from `captureAllRecoveryUnits` **per app with the loop continuing**, carrying the target filesystem's used/free bytes at the moment of failure — the cause is usually a full filesystem and those numbers answer *why* without an operator logging in. **ROUTED TO THE OPERATOR, NOT `backup_failed`, AND THAT OVERRIDES THIS ROW'S OWN PROPOSAL.** The proposal above said *"emitting the existing `backup_failed`"*; that type carries a `customerMessages` entry AND sits in `settings.DefaultEnabledEvents`, so it would email the customer in Hungarian about a failure they cannot act on — precisely the mistake R-97a avoided by minting `whole_guest_backup_failed`. **Decision D-c routes it to the operator and D-c wins.** New `recovery_unit_capture_failed` in `allowedEventTypes` **and** `notify.operatorOnlyEvents`; `notify.IsOperatorOnly` added so ONE test pins both registers (allowlisted-but-not-operator-only is invisible when they are checked separately — the v0.78.0 defect). **Red-proof:** removing the register entry shows the customer being emailed. **Live on 9201:** two events accepted and stored, `operator | sent`, and the positive observable `customer | recovery_unit_capture_failed | skipped | operator_only` read from the hub's `notification_log` | — | @@ -102,7 +105,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-162** | **`docker diff` is the gate's only witness, and its failure mode is quiet.** The gate's power comes from `docker diff` excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the mount-occupancy and writability legs **and would not say so**. | **WATCHING** — a limitation, not a defect | — | It **fails closed**: the canary self-test would stop reporting BROKEN and the gate would then refuse to report at all. What is wrong is the message — it would blame the prober rather than the driver. Revisit only if a non-overlay storage driver ever ships | CC | | **R-163** | ~~**`mp1` is RETENTION, not staging — and it is sized as if it were neither.**~~ | **CLOSED by R-165 — the ceiling it describes no longer exists** (golden v3.0.0, 2026-08-03) | — (the sizing question is answered; the work is **R-165**) | **Closed, not merely re-framed.** This row was the record of a constraint that was to stay open *"until the merge lands"*. It has landed: the golden ships ONE data volume, so there is no separate 20 G area for a driveless app's recovery unit to outgrow, and the free space an app can use is the box's actual free space. **What replaced the constraint is recorded on R-165**: the bulkhead the partition also provided is now B2's explicit capture floor (controller v0.192.0), and the measured 2× DB-app unit size this row documented is what justifies the floor's reserve being a reserve rather than a working budget. **Caveat carried forward, deliberately:** no box has been reinstalled from the merged golden yet (R-178), so every box in the field still has the split layout and this row's consequences remain live ON THOSE BOXES until they are reinstalled. Original finding unchanged below | CC | | **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC | -| **R-165** | ~~**Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.**~~ | **SHIPPED — golden `build-golden.sh` v3.0.0 + agent v0.120.0 + controller v0.192.0 (B2), 2026-08-03. NOT YET PROVEN-LIVE: no box has been reinstalled from the merged golden** | — | **Variant V-c chosen by the operator on MEASURED evidence, not by reading** (`audits/SPIKE-r165-phase0-2026-08-03.md`): one volume at the NEUTRAL path `/var/lib/felhom`, with `/var/lib/docker` and `/mnt/sys_drive` both binds of subdirectories. Three shapes were built and rebooted; **all three boot, reboot 3/3, give ONE `df` figure and keep a container's `statfs("/")` on the merged volume — the ordering worry that motivated the probe did not materialise.** They differ only in which documented guarantee they break: volume-at-`/var/lib/docker` puts customer backups INSIDE Docker's data-root (so the ordinary "clear /var/lib/docker" reflex destroys every local unit); volume-at-`/mnt/sys_drive` puts Docker's ENTIRE data-root under `/mnt`, which the controller container mounts wholesale — **measured: it then sees `/mnt/sys_drive/docker`**, falsifying the bootstrap's own scoping claim. V-c breaks neither. **P1 answered R-176(a):** a pre-merge archive (`mp0+mp1`) restore-tests clean with `mount_parity: ok` in 84 s; `mountParity` was not weakened. **P3: the four golden assertions were RETARGETED, never deleted, and each was RUN against a deliberately wrong shape — 8 checks, 8 passed**, including a NEW 2b asserting both paths are ONE filesystem (which catches the S2 shape the spike ranked worse than the split) and a new guard for a leftover `mp1` (the old "was mp1 excluded?" pattern could no longer match — a guard that cannot match has silently stopped guarding). **B2 shipped first, in controller v0.192.0**: a two-term capture floor (97% / 1 GiB) in `fillwatch`'s shape, deliberately beyond its critical band so the customer is always warned before a refusal; it refuses per app and **never deletes**, because nothing here is generational. **Golden 0.192.0 is published (registry HTTP 200, sha `54e2a4c4…`) but DELIBERATELY NOT VOUCHED** — vouching is what makes fresh installs pick it up, and the right order is prove-then-vouch. **Remaining: reinstall both demo boxes from it, prove end to end, then vouch → the work is R-178** | CC | +| **R-165** | ~~**Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.**~~ | **SHIPPED — golden `build-golden.sh` v3.0.0 + agent v0.120.0 + controller v0.192.0 (B2), 2026-08-03. IMPLEMENTED — the LAYOUT is proven live on both boxes (R-178, 2026-08-03); the BULKHEAD'S REPLACEMENT IS NOT (→ R-181)** | — | **Variant V-c chosen by the operator on MEASURED evidence, not by reading** (`audits/SPIKE-r165-phase0-2026-08-03.md`): one volume at the NEUTRAL path `/var/lib/felhom`, with `/var/lib/docker` and `/mnt/sys_drive` both binds of subdirectories. Three shapes were built and rebooted; **all three boot, reboot 3/3, give ONE `df` figure and keep a container's `statfs("/")` on the merged volume — the ordering worry that motivated the probe did not materialise.** They differ only in which documented guarantee they break: volume-at-`/var/lib/docker` puts customer backups INSIDE Docker's data-root (so the ordinary "clear /var/lib/docker" reflex destroys every local unit); volume-at-`/mnt/sys_drive` puts Docker's ENTIRE data-root under `/mnt`, which the controller container mounts wholesale — **measured: it then sees `/mnt/sys_drive/docker`**, falsifying the bootstrap's own scoping claim. V-c breaks neither. **P1 answered R-176(a):** a pre-merge archive (`mp0+mp1`) restore-tests clean with `mount_parity: ok` in 84 s; `mountParity` was not weakened. **P3: the four golden assertions were RETARGETED, never deleted, and each was RUN against a deliberately wrong shape — 8 checks, 8 passed**, including a NEW 2b asserting both paths are ONE filesystem (which catches the S2 shape the spike ranked worse than the split) and a new guard for a leftover `mp1` (the old "was mp1 excluded?" pattern could no longer match — a guard that cannot match has silently stopped guarding). **B2 shipped first, in controller v0.192.0**: a two-term capture floor (97% / 1 GiB) in `fillwatch`'s shape, deliberately beyond its critical band so the customer is always warned before a refusal; it refuses per app and **never deletes**, because nothing here is generational. **Golden 0.192.0 is published (registry HTTP 200, sha `54e2a4c4…`) but DELIBERATELY NOT VOUCHED** — vouching is what makes fresh installs pick it up, and the right order is prove-then-vouch. **Remaining: reinstall both demo boxes from it, prove end to end, then vouch → the work is R-178**. **STATUS SETTLED 2026-08-03, operator ruling: IMPLEMENTED, not PROVEN-LIVE, and the reason is the substantive part.** R-178 proved the *layout* on both boxes past any doubt — one volume, no `mp1`, both binds real mounts, one `df` figure, 3/3 reboots each, claim→deploy→backup→restore, and the ceiling's removal measured at 65 GiB / 233 GiB against the old 19 GiB / 45 GiB slices. **But B2, which this row records as the bulkhead's deliberate replacement, does not guard the leg that fills the volume** — proven live on demo-hp at 06:40:03 and filed as **R-181**: the floor is consulted ONLY in `captureAllRecoveryUnits` (`recovery_unit.go:328`), while `runVolumeDumps` (`backup.go:535`) writes the bulk with no floor check at all, and its refusal message's claim *"the previous unit is untouched"* was measured FALSE. **This row's own framing is what makes that gate the status:** it says the partition's bulkhead "is now B2's explicit capture floor". Until R-181 closes, the merge has removed a bulkhead and its stated replacement covers the cheap leg only — and post-merge the unguarded leg can reach Docker's data-root, which pre-merge it could not (it could only fill the dedicated 20 G `mp1`). PROVEN-LIVE when R-181 closes and a fill is re-run | CC | | **R-166** | ~~**App state gets a desired/observed model with its own store.**~~ Operator decision **D-b**, 2026-08-02 (`CONTEXT.md` S-5) | **SHIPPED + PROVEN-LIVE** (controller v0.189.0, 2026-08-02) | — | **Both blocking facts were established at source before any code was written, and the answers changed the shape.** **(a) Does a crash-safe journal already exist for the in-flight case?** YES, twice — `internal/quiesce/quiesce.go` (marker + `Recover`, proven on live hardware by Campaign 8 fault 10) and `internal/stacks/migrate.go` (`migration.json` + `RecoverMigration`) — but **neither covers the app-data path**: `DumpAppVolumesSafe` stopped and restarted an app with **no marker, no journal and not even a `defer`**. So the pattern existed and the coverage did not; `backup.AppStopGuard` copies the proven shape into its **own** file (one file, one writer). **(b) Is the SQLite store reachable?** Irrelevant, and deliberately unused: `metrics.db` is optional by design (the controller runs with it absent), and operational state must not live in a store designed to be droppable. **Shipped:** tri-state `desired_state` in `app.yaml` written ONLY by the customer's action (API action switch, `DeployStack`, `UpdateOptionalConfig`'s redeploy branch, `.fab` import — a 14-caller census established that `StartStack`/`StopStack` must NOT be writers); `isBootOrphan` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN, never running**, so a legacy `app.yaml` keeps byte-identical pre-v0.189.0 behaviour; running-only backfill. D-b's every-container requirement was already met by `aggregateState` and was NOT re-implemented. **Live on 9201:** all three flows (stop survives a restart; a zero-container `running` app is recovered by name; a legacy app.yaml is skipped and never inferred as stopped). **Also fixed en route:** `SaveAppConfig` rebuilt `AppConfig` field-by-field (the R-100 shape) and would have dropped the new field on every save across nine call sites | — | | **R-167** | ~~**Storage monitoring and backup alerts.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0/.1/.2 + hub v0.89.0, 2026-08-02) | — | **Operator decision D-c. It shipped BEFORE the R-165 merge, not with it** — D-a's condition (2) says the monitoring lands in the same step and never after, and landing it first is strictly better and costs nothing. **Customer half:** new `internal/fillwatch`, per FILESYSTEM (never per app — one full disk holding ten apps would fire ten times). **It emits the PRE-EXISTING `disk_warning`/`disk_critical` pair, which was allowlisted, copy'd, in `DefaultEnabledEvents` and checkbox'd with NO PRODUCER IN ANY REPO** — a complete customer pipeline with no producer, the **sixth** *built-but-never-wired* instance here; minting a new near-duplicate type would have left it inert forever. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range — **proven live: the critical crossing fired on the FREE-BYTE term (1.7 GB) at only 91% used.** Edge-triggered on escalation, state persisted, hysteresis dead zone at 75% / 7 GiB pinned by a test; a nil usage read never warns and never clears one (§8.4). The hub's two generic `customerMessages` entries were **removed** — `FormatCustomerEmail` prefers the entry over the message, so keeping them would discard the drive label and the byte figures. **Operator half: see R-158.** **Live on 9201, all three flows:** `disk_warning` then `disk_critical` both `customer | sent` with the Hungarian rendered, exactly two events across three boots (the edge trigger held on the one between), then a silent clear that re-armed. **v0.191.1** added the once-at-startup run (Daily/Every both wait for their first tick, so a box BOOTING over the line would have stayed silent up to 24 h — the R-100 shape); **v0.191.2** added a per-run positive observable, earned when a quiet run during this session's own validation proved unreadable as evidence. Follow-ups: **R-177** (no run-now path) | — | | **R-168** | ~~CI: no runner exists, and with trunk-based pushes CI can DETECT but not BLOCK~~ | **SHIPPED — and the alarm is DEMONSTRATED** (2026-08-02) | — | **Runner live**: `homelab-manifests/gitea-system/act-runner.yaml`, an unprivileged host-mode `act_runner` in `gitea-system`, one owner-scoped registration serving all four repos (measured: tasks 7-10 all claimed by `felhom-gates-runner`). `.gitea/workflows/gates.yml` in each repo runs that repo's entry point with `--fast` and nothing else; no `uses:` step anywhere. **Six probes, all answered, none STOPped** — `audits/SPIKE-ci-runner-2026-08-02.md`. The two that changed the design: **P2** (stock image has git but NO python3 → custom image `felhom-act-runner:0.1.0`, base pinned, python3 and nothing else) and **P6** (a runner that loses `/data/.runner` re-registers and leaves a dead record behind → the PVC is load-bearing, measured both ways). **P5 is the one that mattered**: a failed run produced NO mail, NO notification row and NO log line from Gitea, so the run now sends its own alarm via Resend and prints the provider's accepted id. **Proven end to end, not asserted**: a deliberately broken commit pushed with `--no-verify` → run #6 `failure` → `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`. Posture shown from the live pod spec: `privileged: false`, all caps dropped, no docker socket, no hostPath, `automountServiceAccountToken: false`, sized at half Gitea's limits so it cannot crowd out the service holding every repository on the same node. **The standing limit stays true and is written into the manifest and every workflow: it DETECTS, it does not BLOCK** — making it block is → R-169 | — | diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index b616387..059e92f 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,32 @@ +## docs — v1.22.0 exercised end to end on two real reinstalls (2026-08-03, R-178) — **no script change** + +**Nothing shipped.** `felhom-host-install.sh` stayed at **v1.22.0**; the published copy at +`https://felhom.eu/scripts/felhom-host-install.sh` was confirmed byte-identical to the repo copy +(`sha256 ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4`) before use. Both demo +boxes were uninstalled and reinstalled with it, by **two deliberately different supply paths**: +demo-hp with `--golden ` (the `:2584` alternative), demo-felhom with +`--force-gitea-golden` (the canonical C.3 customer command). The merge-aware `step_grows` produced +`data +46G (->70G, ONE volume)` and `+226G (->250G)` respectively, and `fetch_verify` was observed +succeeding against the vouched manifest for **both** artifacts on demo-felhom +(`verified sha256 a7763d31b55b5ce7…` agent, `verified sha256 54e2a4c431daf580…` golden). + +**Two script-side findings, filed not fixed** (the session was a runbook; §7 forbade code): + +- **R-180** — `--archive-storage` is validated for existence (`:1583`) and for golden resolution + (`:1661`), but never against the ACL storage set it is about to grant (the fixed default + `local local-lvm felhom-pbs`). Staging the golden on `felhom-backup` therefore passed every + pre-flight gate and died at **step 8/8**: `HTTP 403: permission denied at /storage/felhom-backup + (missing privilege Datastore.AllocateSpace)` — *after* step 2 minted the token, step 4b **rotated + and vaulted root@pam**, and step 5 installed the agent. `ARCHIVE_STORAGE ∈ PVE_STORAGES` is a + one-line assertion over two variables both known at `:1583`. +- **R-179** — `--uninstall` leaves the NAS network-storage systemd units behind + (`mnt-felhom\x2ddrives-.{mount,automount}`; automount left `failed`, parent bind left + mounted). The Part E residue-diff provenance is from **v1.9.1**, which predates the feature — and + demo-felhom, which never had a share configured, left nothing, which is exactly why a diff on such + a box reported clean. + +Full evidence: root `REPORT.md`. + ## host-install: one data volume, derived from the disk (2026-08-03, R-165) **Forced by a census, not planned.** `felhom-agent` v0.120.0 merges the appliance's two data volumes