diff --git a/CONTEXT.md b/CONTEXT.md index b915b46..da6085e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -39,12 +39,31 @@ so pruning could only mean deleting a **different** app's only local recovery un boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed shortly. So R-176's in-place migration rehearsal is **withdrawn**, not deferred. -**S-14 — prove first, then vouch (2026-08-03).** Golden **0.192.0** is baked, published and verified in -the registry, and is **deliberately UNVOUCHED**. Vouching is what makes a fresh install pick a golden -up, so vouching one that no box has been proven from would put an unproven disk layout in front of the -next install anywhere. The bake script already treats the hub record as a separate deliberate step; this -makes the ordering a rule. **The remaining work is R-178** — reinstall each demo box from the merged -golden, prove claim → deploy → back up → restore, **one box at a time**, and only then vouch. +**S-14 — prove first, then vouch (2026-08-03) — SPENT, and the ordering did not survive contact.** The +rule was: golden **0.192.0** stays UNVOUCHED until a box has been proven from it, because vouching is +what makes a fresh install pick a golden up. **In the event the golden was vouched at 07:23:26 CEST on +2026-08-03, before any box was reinstalled** (hub log `Artifact manifest set: agent=0.119.0 +golden=0.192.0`), so the ordering was already spent when R-178's session opened; the operator elected +to accept it rather than revert the manifest. **Both boxes were then reinstalled and proven** (R-178, +`REPORT.md`), so the end state is the intended one and no unproven layout was ever in front of a real +install — but the rule protected nothing, because nothing enforced it. **The lesson is R-115's, one +layer up:** an ordering that lives only in a `CONTEXT.md` sentence and a runbook's §7 is a reminder, +and reminders do not hold. If prove-then-vouch is to be a rule it needs the shape R-120's gate has — +a refusal at `handleSetArtifacts`, the sole path to `SetArtifactManifest`, which runs without anyone +choosing to run it. + +**S-15 — the merged layout is proven live, by two different supply paths (2026-08-03, R-178).** Both +demo boxes were wiped and reinstalled from golden 0.192.0 and taken through claim → deploy → back up → +**restore**. **demo-hp** was installed with `--golden ` (the layout proof) and +**demo-felhom** by the normal manifest route with `--force-gitea-golden` (the pipeline proof — +`verified sha256 54e2a4c431daf580… matches the hub manifest`), deliberately different so the session +proved the disk shape *and* the delivery route rather than one of them twice. Live shape on both: +`mp0` at `/var/lib/felhom`, `backup=1`, **no `mp1`**; `/var/lib/docker` and `/mnt/sys_drive` both real +mounts of its subdirectories via `/etc/fstab`; ONE `df` figure and one device id on all three paths; +3/3 reboots each with the binds surviving every time. **What is NOT proven is B2** → **R-181**: the +floor guards `captureAllRecoveryUnits` and not `runVolumeDumps`, which is the leg that fills the +volume, and its refusal message's "the previous unit is untouched" was measured false. R-165 is +therefore **IMPLEMENTED**, not PROVEN-LIVE. **S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).** Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the diff --git a/REPORT.md b/REPORT.md index 5f49506..b1cb4cf 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,83 +1,342 @@ -# REPORT — R-165: the `mp1` merge, built and proven at the bake (2026-08-03) +# REPORT — R-178: both demo boxes reinstalled from the merged golden and proven (2026-08-03) -**Overwritten** per the standing rule. The prior contents (hub v0.89.0 / R-167, 2026-08-02) have their -durable record in `hub/CHANGELOG.md` and `CONTEXT.md` S-11/S-12. +**Overwritten** per the standing rule. The prior contents (R-165, the bake, 2026-08-03) have their +durable record in `documentation/backlog/OPEN-ITEMS.md` R-165 and the per-repo CHANGELOGs. -**Companion:** `felhom-agent/REPORT.md` holds the full session detail — probes, variant evidence, the -bake transcript, red-proofs and teardown. **This file covers what changed in THIS repo, and the CI -verification for all three.** - -> **Scope, stated first.** The merge is **built and green through Phase 5**. **Phases 6–7 — reinstalling -> the two demo boxes from the merged golden and proving one end to end — were NOT done, and nothing -> was wiped.** The golden is therefore deliberately **unvouched**. Remaining work: **R-178**. +**Runbook, not a task.** No repo got a version bump and nothing was built. One artifact was +**published** (agent 0.120.0) on an explicit operator ruling — see §2. All code findings are filed as +register rows, not commits, per the runbook's §7. --- -## 1. What changed in `felhom.eu` +## 1. Preconditions P1–P6, each as measured -**No hub change; no hub version bump.** This repo changed in two places, one of them forced. +| # | Precondition | Measurement | +|---|---|---| +| **P1** | Golden sha256 matches R-178's record | **PASS** — computed from the artifact itself on demo-hp: `sha256sum` → `54e2a4c431daf580d2807b82d810be36a7c6a9f697b27094dc2912d26a43b3e0`, 649,547,835 B. Matches R-178's recorded prefix and the hub manifest's stored value exactly. Re-verified after copying to `local`: identical | +| **P2** | A rollback golden still exists | **PASS, three depths.** Split-layout golden **0.188.0** fetchable from Gitea (`HTTP 200`, 649,310,288 B). A **local** split-layout golden sat on each box — `local:backup/vzdump-lxc-9100-2026_07_21-18_24_49.tar.zst` (demo-hp), `…2026_07_20-17_50_57…` (demo-felhom). Best: **full guest vzdumps**, three per box, newest `vzdump-lxc-9201-2026_08_03-04_54_28.tar.zst` (2,351,557,870 B, demo-hp) and `…2026_08_03-04_44_50…` (6,323,506,165 B, demo-felhom), both ~3 h old — so each box could be put back exactly as it was | +| **P3** | Neither demo box holds anything wanted | **PASS, stated explicitly.** demo-hp: paperless-ngx (webserver/postgres/redis) + filebrowser + samba shares. demo-felhom: immich (×4), docmost (×3), calibre-web, bookstack (×2) + filebrowser + samba. Both are demo customers (`Demo HP`, `Demo Ügyfél`); no real customer data. **All of it was destroyed by the wipes and none of it was restored** — that was the point, and the operator confirmed each wipe | +| **P4** | The colleague's box untouched | **PASS** — `peti-felhom` is Tier 2 and out of scope. No command was sent to it; it appears in the hub customer list as `DOWN`, exactly as before | +| **P5** | Both hosts' agent is v0.120.0 | **PASS** — `felhom-agent 0.120.0` on both, before and after | +| **P6** | The hub's host register, BEFORE | **Captured**: `demo-felhom-8363b5 / Demo Ügyfél / 0.120.0 / ONLINE / 1-of-2 guests`; `demo-hp-bb76ea / Demo HP / 0.120.0 / ONLINE / 1-of-2`; `drill-r50-0a4f9a / drill-r50 / 0.113.0 / DOWN` (the fixture, untouched throughout) | -**`scripts/felhom-host-install.sh` — forced by a census, not planned.** `step_grows` computed **two** -volume sizes and the install call passed both, so it had to change with the agent or every install -would have provisioned a half-sized box. It now computes ONE total, summing the old 80/20 split -(`226` = the previous `184 + 42`), so **a standard appliance keeps exactly the 250 G it had** — it is -simply no longer split by a wall. **The size still comes from the physical disk**: `step_grows` -already read the thin pool's real free space, and the merge only collapsed its two outputs into one. -That is the answer to the task's §8.2 — no row needed filing. `--sysdata-grow` is deprecated but still -honoured, because the agent **folds** a hand-passed value in rather than dropping it. +--- -**Documentation**, per the coupling rule — see §3. +## 2. Two findings that contradicted the runbook's premise, both surfaced before any wipe -## 2. The bake, as this repo's audit trail records it +**(a) The golden was ALREADY vouched.** Hub log, a positive observable: +`2026/08/03 07:23:26 [INFO] Artifact manifest set: agent=0.119.0 golden=0.192.0 min_agent="0.113.0" +wrapper_sha=true` — roughly ten minutes before this session's first hub read, and this session had +POSTed nothing. The configuration page confirmed `Artifacts.GoldenVersion=0.192.0`, +`GoldenSHA256=54e2a4c4…3b3e0`. So §7's prove-then-vouch order was already spent, and Phase A's stated +safety ("nothing is official yet, so a failure reaches nobody") was void. **Operator ruling: accept it +and proceed**, keeping the two different supply paths. Recorded on `CONTEXT.md` S-14, now marked SPENT +with the reason — the rule lived only in prose and nothing enforced it. -`documentation/audits/SPIKE-r165-phase0-2026-08-03.md` (new) holds P1, P2 and P3 with method, -measurement and ruling, including the teardown of every probe artefact at all three layers. The -headline measurements: +**(b) Agent v0.120.0 had never been published — the vouched agent was 0.119.0.** +`GET …/generic/felhom-agent/0.120.0/felhom-agent` → **HTTP 404** (0.119.0 → 200). Installer step 5's +idempotent skip requires `installed == vouched` exactly, so a documented-path reinstall would have +**downgraded both boxes** to the pre-merge 0.119.0 — and would have *succeeded* while doing it, since +the current `step_grows` sets `SYSDATA_GROW=0` and 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on +error) therefore never fires. The session would have proven a stack nobody ships. **Operator ruling: +publish and vouch first.** `scripts/publish-agent.sh 0.120.0` from a clean tree at `4bb84fc3` +(`git status --porcelain` empty, `HEAD == origin/main`): upload HTTP 201, **round-trip GET verified**, +`AGENT_SHA256=a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`. Vouched via +`POST /configuration/artifacts` → `303 …flash=artifacts_set`, hub log +`Artifact manifest set: agent=0.120.0 golden=0.192.0`; the hub resolved the sha authoritatively from +Gitea rather than trusting the submitted value. Filed as the **third instance of R-115**, not a new +ID — R-115 is that row, and it has been WAITING-ON-OPERATOR since 2026-07-29. -- **P1 PASS** — a real pre-merge archive restore-tests clean, `mount_parity: ok`, 84 s, `mountParity` - untouched. -- **P2** — all three variants boot and reboot 3/3; they are separated by **scoping**, not mechanics. - The container's view of `/mnt` is 8.0K under V-a and V-c, and **17.9M — Docker's entire data-root — - under V-b**. The operator chose **V-c**. -- **P3** — the four retargeted golden assertions, run against a deliberately wrong shape: **8/8**. +--- -## 3. Documentation coupling +## 3. demo-hp — the layout proof -| File | Change | +**Install path.** Uninstall: `./felhom-host-install.sh --uninstall --vmid 9201` (typed-vmid +confirmation supplied over a pty). Install: + +``` +./felhom-host-install.sh --customer-id demo-hp --mode appliance --vmid 9201 \ + --cores 7 --memory 26906 \ + --golden local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst \ + --passphrase-file /root/.pp-demo-hp +``` + +Script fetched from `https://felhom.eu/scripts/felhom-host-install.sh`, **v1.22.0**, sha256 +`ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4` — byte-identical to the repo copy +that was read. The passphrase went file→file into a 0600 file and never onto a command line. A +`--dry-run` preceded the real run and resolved `manifest: agent v0.120.0 (sha a7763d31…), golden +v0.192.0` with `grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)`. + +**One deviation, mine, and it cost a restart.** The first attempt staged the golden on +`felhom-backup` with `--archive-storage felhom-backup`. Pre-flight passed; **step 8/8** failed: +`HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)` — +the default pool-scoped ACL grants `local local-lvm felhom-pbs` only. Fixed by copying the golden to +`local` (sha re-verified after the copy) and `--resume`. Filed as **R-180**: the condition is +statically checkable in pre-flight, and the failure lands *after* step 4b has rotated and vaulted +root@pam. + +**Layout evidence.** + +``` +mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G # and NO mp1 line +rootfs: local-lvm:vm-9201-disk-0,size=32G + +/var/lib/felhom /dev/mapper/pve-vm--9201--disk--1 ext4 rw,relatime,stripe=16 +/var/lib/docker /dev/mapper/pve-vm--9201--disk--1[/docker] ext4 +/mnt/sys_drive /dev/mapper/pve-vm--9201--disk--1[/sys_drive] ext4 + +Filesystem Size Used Avail Use% Mounted on +/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/felhom +/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/docker +/dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /mnt/sys_drive + +stat -c %d → 64519 for all three paths +/etc/fstab: /var/lib/felhom/docker → /var/lib/docker ; /var/lib/felhom/sys_drive → /mnt/sys_drive +both binds writable (touch succeeded on each) +``` + +`/mnt/sys_drive` shows a second `findmnt` row — it is the controller container's +`-v /mnt:/mnt:rslave` propagation (peer group 383 vs the fstab bind's 333), the same shape the split +layout had, not a stacked bind. + +**Reboots — each individually, minimum three:** + +| # | started | controller healthy | outcome | +|---|---|---|---| +| 1 | 08:13:33 | 08:13:55, `Up 12 seconds (healthy)` | one filesystem, 69G/65G on both paths | +| 2 | 08:13:59 | 08:14:15, `Up 7 seconds (healthy)` | same | +| 3 | 08:14:19 | 08:14:35, `Up 7 seconds (healthy)` | same | + +After all three, `mountpoint -q` returns true for **all three paths** and both binds still resolve to +the single volume's subdirectories — which is what the reboots exist to test. `uptime -s` = +`2026-08-03 06:14:24 UTC`, matching reboot 3, so these were real reboots. + +**The journey — method stated: endpoint-level, not a browser.** `claude-in-chrome` does not exist on +DooPlex; every step below invoked the exact endpoint the dashboard's own JavaScript calls. + +| Leg | Endpoint | Observable | +|---|---|---| +| Claim | `POST /claim` with the pre-auth HMAC CSRF token (64 chars) **and** its `felhom_claim_csrf` cookie | `302 → /`; gate discriminator flipped `{"error":"dashboard not yet claimed"}` → `{"error":"authentication required"}`. **The code is emailed-only (R-119) — the operator supplied it**, after an operator resend rotated the generation (the previous code had been consumed 12 d earlier) | +| Deploy | `POST /api/stacks//deploy` with `{"values":{…}}`, field values scraped from the deploy form exactly as the form's own `fetch` does | `{"ok":true}`; `privatebin Up (healthy)`, then `opengist Up (healthy)` | +| Back up | `POST /api/debug/backup/dbdump` — this runs the **production** `RunDBDumps` path (DB leg → `runVolumeDumps` → `captureAllRecoveryUnits`); the debug route only starts it instead of waiting for 03:30 | `Volume dump: opengist/… → 178.0 KB`, `privatebin/… → 2.5 KB`, `App-data backup completed … 2 volume dump(s) (3.962s)`, `Recovery unit captured for …` ×2 | +| **Restore** | `POST /backup/restore` `stack_name=privatebin snapshot_id=primary` | A marker planted in the live volume was **deleted**, then restored: `{"ok":true,"message":"privatebin visszaállítva (primary)."}` in **9.2 s**, and the marker returned with an **identical sha256 `ac1faae6134871d640d5cb1bbc6b5092d1e96b7e46c6c3bcb6e8236cc33ae861`**. App `Up (healthy)` afterwards | + +**Recovery unit path, and it lands on the single volume:** +`/mnt/sys_drive/felhom-data/backups/primary//{manifest.json,compose/,volume-dumps/}`, whose `df` +is `/dev/mapper/pve-vm--9201--disk--1` — the merged volume. + +**Note on the first app chosen.** `privatebin`'s catalog entry declares no `backup:` section, so its +first per-stack capture produced `"volume_dumps": null` — correct for that declaration, not a defect, +but it means a per-stack `POST /stacks//backup` writes compose+config only; the volume leg lives in +the full pass. A second app (`opengist`) was deployed so the run had both a capture and, later, a +refusal. + +**The ceiling is gone, measured:** a recovery unit can use **65 GiB** — the whole volume — against the +**19 GiB** the pre-wipe `mp1` slice offered (`/dev/mapper/pve-vm--9201--disk--2 20G 95M 19G 1%`). + +--- + +## 4. The floor's first live firing — and it does not do what it says + +**Instrument, proven before use.** demo-hp's thin pool is **53.93 GiB** and the auto-sized volume is +70 G, so a real fill to 97 % would have exhausted the pool and corrupted every guest on the box, +including the protected `drill-r50` fixture. A 5 GiB `fallocate` probe moved guest `df` from +`977M used / 65G avail` to `6.0G / 60G` while thin-pool `data_percent` stayed **29.03 → 29.03** — +zero blocks allocated — and cleanup returned both to baseline. The floor reads `statfs`, which is +exactly what `fallocate` moves, so the condition it guards is genuinely present. + +**Setup.** A **real** 2 GiB file in opengist's data volume (so its capture writes 2 GiB), then +`fallocate` to leave `69G / 63G used / 3.0G avail / 96%` — **both floor terms deliberately still +clear**, so the run would start. Capture order was established empirically from the previous run's log +(opengist first, privatebin second), not assumed. + +**What happened, 06:40:03:** + +``` +Volume dump: opengist/opengist_opengist_data → 2.0 GB # unguarded +Volume dump: privatebin/privatebin_privatebin_data → 2.5 KB +App-data backup completed … 2 volume dump(s) (20.737s) +[WARN] Recovery unit capture REFUSED for opengist — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted +[WARN] Recovery unit capture REFUSED for privatebin — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted +[INFO] Event pushed: recovery_unit_capture_failed (error) — … ×2 (HTTP 200) +``` + +**What holds:** it refuses **per app** rather than aborting the run; **nothing was deleted** (both +units present afterwards); and the operator alert reached the hub — `recovery_unit_capture_failed`, +severity `error`, accepted `HTTP 200`, twice. + +**What does not hold — measured, not inferred.** The refusal message claims *"the previous unit is +untouched"*: + +| unit file | before | after | +|---|---|---| +| `privatebin/volume-dumps/privatebin_privatebin_data.tar` | `26c546c2…` | **`b538ab89…`** | +| `opengist/volume-dumps/opengist_opengist_data.tar` | 182,272 B | **2,147,666,432 B** | + +Both were rewritten by the earlier leg, while each `manifest.json` kept +`created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so +a unit's payload can be swapped under a stale descriptor and nothing inside the unit can detect it. + +**Cause, in the code and not the log.** The floor is consulted in exactly one place — +`m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which +writes a manifest and a compose copy: a few KB. `runVolumeDumps` (`backup.go:535`) — the leg that +writes the bulk, and the leg that consumed the reserve — has **no floor check at all**; its gates are +protected-stack, volume-less, disconnected, decommissioned. And it runs first *by design* +(`backup.go:483`). **The floor guards the cheap leg and not the leg that fills the volume.** + +This is why R-165 is IMPLEMENTED and not PROVEN-LIVE: that row records B2 as the deliberate +replacement for the bulkhead the `mp1` partition provided, and pre-merge the unguarded leg could only +fill a dedicated 20 G partition — post-merge it can reach Docker's data-root. Filed as **R-181**. +**No code was written**, per the runbook's §7. + +**Cleanup:** fill file and payload removed; `df` back to `1.2G used / 65G avail`; a clean re-run left +both units valid (`Volume dump … 178.0 KB` / `2.5 KB`, `App-data backup completed … (2.399s)`). + +--- + +## 5. demo-felhom — the pipeline proof + +**Install path — deliberately different, and this is the reason the second box exists.** + +``` +./felhom-host-install.sh --customer-id demo-felhom --mode appliance --vmid 9201 \ + --cores 3 --memory 12288 \ + --force-gitea-golden \ + --passphrase-file /root/.pp-demo-felhom +``` + +A copy of golden 0.192.0 already sat on this box's `local` storage — it is where the golden was +**baked** at 06:58 (a `.log` beside it), sha `54e2a4c4…`. `--force-gitea-golden` is the documented +C.3 customer path and overrides local discovery in **both** pre-flight and step 7, which pre-flight +confirmed: `golden: none local — will fetch + verify from Gitea in step 7/8`. The bake artifact was +left untouched. + +**`fetch_verify` succeeding against the vouched sha — the observable this box exists to produce:** + +``` +5/8 fetching agent binary v0.120.0 from Gitea … + verified sha256 a7763d31b55b5ce7… matches the hub manifest +7/8 fetching golden v0.192.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst + verified sha256 54e2a4c431daf580… matches the hub manifest + golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst +Day-0 provision SUCCESS — vmid=9201 host_id=demo-felhom-8363b5 customer=demo-felhom +``` + +Both artifacts — agent and golden — were fetched anonymously (the normal customer shape) and +sha-verified against the manifest. Controller `0.192.0` healthy. + +**Layout:** `mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=250G`; +`pct config 9201 | grep -c '^mp1:'` → **0**. Both binds real mounts +(`…disk--1[/docker]`, `…disk--1[/sys_drive]`), `df` one figure — `246G 977M 233G 1%` on all three +paths — `stat -c %d` = `64519` on all three, fstab carries both binds. + +**Reboots:** 1 — 09:18:55 → healthy 09:19:09; 2 — 09:19:12 → 09:19:26; 3 — 09:19:30 → 09:19:45. All +three paths still mountpoints after each; `uptime -s` = `2026-08-03 07:19:34 UTC`, matching reboot 3. + +**Journey:** claim (`302 → /`, discriminator flipped to `authentication required`; code supplied by +the operator after a resend) → deploy `opengist` (`{"ok":true}`, `Up (healthy)`) → capture +(`Volume dump: opengist/… → 178.0 KB`, `Recovery unit captured`, unit on +`/dev/mapper/pve-vm--9201--disk--1`, marker present inside the tar) → **restore** +(`{"ok":true,"message":"opengist visszaállítva (primary)."}` in **9.4 s**, marker back with identical +sha256 `bc5507987f3f56dc19a9c24785826c304f1a986998d4aeb0b06d1a810b59e939`, app healthy). + +**Ceiling:** 233 GiB available to a recovery unit, against the 45 GiB the pre-wipe `mp1` offered. + +**Step 8 was NOT repeated here, deliberately** — stating it rather than leaving it ambiguous. The +floor fired on demo-hp and R-181 characterises it fully; re-firing would add no information and would +mean filling a 246 G volume. + +--- + +## 6. Vouching + +Already done before the session (§2a), at **07:23:26 CEST on 2026-08-03**, by an operator action in +the hub UI. The session's own manifest write was the **agent** half, at **07:44:31**: +`Artifact manifest set: agent=0.120.0 golden=0.192.0 min_agent="0.113.0" wrapper_sha=true`. The +manifest afterwards, read back: `agent_sha256=a7763d31b55b5ce7…10b9d`, +`golden_sha256=54e2a4c431daf580…43b3e0`, `min_agent=0.113.0`, `wrapper_sha256=104db0a4…` (preserved +verbatim). The R-120 gate did not block: golden 0.192.0 equals the newest controller the fleet +reports. + +--- + +## 7. Teardown — all three layers + +1. **The machine.** No throwaway guest was created this session, so there is none to delete. VM 300 + `drill-r50` on demo-hp — the protected drift fixture — was never touched and is still `stopped`; + it is not in the `felhom` pool (`pvesh get /pools/felhom` listed only `lxc/9201`), so the + uninstall's shared-box logic never reached it. +2. **The host.** `local-lvm`, before → after: **demo-felhom 29.31 % → 1.37 %** (the old 200 G + 50 G + volumes returned; the new 250 G volume is thin and barely allocated). **demo-hp 39.13 % → + 36.75 %** — *higher than a clean reinstall would leave it*, because the floor test's 2 GB tar + blocks cannot be reclaimed: `fstrim` inside an unprivileged LXC returns + `FITRIM ioctl failed: Operation not permitted`. No operational impact (the guest shows 65 G free of + 69 G, the host 35.7 GiB free of 53.93), but it is real residue and is stated rather than rounded + away. Each box's data volume is now the only data volume; no old guest volumes remain. + **Residue found and cleared by hand on demo-hp:** `--uninstall` left the NAS network-storage units + `mnt-felhom\x2ddrives-Felhom\x2dShare.{mount,automount}` behind (automount `failed`, parent bind + still mounted) → **R-179**. demo-felhom left none, because it had no network share configured. +3. **The hub.** **No old records exist to dispose of, and this is the honest finding, not an + omission:** both enrollments were **idempotent** — `host REUSED (idempotent — existing + credential)`, `host_id: demo-hp-bb76ea` and `demo-felhom-8363b5`, the same ids as before. The + reinstalls therefore produced **no new host records**, so nothing was orphaned and nothing needed + deleting. Final register: `demo-felhom-8363b5 ONLINE 0.120.0`, `demo-hp-bb76ea ONLINE 0.120.0`, + `drill-r50-0a4f9a DOWN 0.113.0` — the same three rows as at P6. No scratch customers were created. + `/appliances` returns 404 on hub 0.89.0 — there is no appliance-record surface to clean. + +**Secrets:** both retrieval passphrases were moved file→file into 0600 files, used via +`--passphrase-file`, and `shred -u`'d afterwards on both hosts along with the session cookie files and +helper scripts; the local scratch copies were deleted. Nothing was written to a committed file and +`curl -w '%{redirect_url}'` was never used (R-132). + +--- + +## 8. Registers changed + +| Row | Change | |---|---| -| `documentation/architecture/07-backup-architecture.md` | **S-1: the contract changed in the same session.** New **§7.5.1** — the ceiling §7.5 describes no longer exists for a box built from golden ≥ 0.192.0, the bulkhead's replacement (B2) is recorded, and **R-175 is FIXED here**: the bound is restated as a function of `mp1` and scoped to split-layout boxes, naming all three real shapes | -| `documentation/architecture/00-capability-map.md` | new row — **IMPLEMENTED, not PROVEN-LIVE**, with the missing leg named (no box reinstalled, R-178) and the bake cited as the evidence it is | -| `documentation/backlog/OPEN-ITEMS.md` | **R-165** → shipped-not-yet-proven-live; **R-163 CLOSED** (its ceiling no longer exists); **R-175 CLOSED**; **R-176** (a) answered by P1, (b) **withdrawn** — every node is reinstalled, not migrated; **R-178 NEW** | -| `CONTEXT.md` | **S-13** (the variant, chosen on measurement; B2's floor; pruning rejected with its reason; no migration exists) and **S-14** (prove first, then vouch) | -| `STATUS.md` | rewritten section in plain language; the *Waiting on you* item cleared — both questions are answered; **two older entries trimmed** so the page did not grow | -| `scripts/CHANGELOG.md` | the host-install change, with why it was forced | +| **R-178** | **CLOSED** — both boxes reinstalled and proven, by two different supply paths | +| **R-165** | **IMPLEMENTED**, not PROVEN-LIVE — operator ruling; the layout half is proven, the B2 half is not (→ R-181) | +| **R-115** | **Third instance recorded** — agent 0.120.0 built, deployed to both hosts, never published | +| **R-181** *(new)* | The capture floor guards the recovery-unit leg and not `runVolumeDumps`; its "previous unit is untouched" claim measured false | +| **R-180** *(new)* | `--archive-storage` is not cross-checked against the ACL grant; the 403 lands at step 8/8, after root@pam has been rotated | +| **R-179** *(new)* | `--uninstall` leaves NAS network-storage systemd units behind when a share was configured | -## 4. CI — run ids and conclusions +**IDs established free before minting:** `grep -ro "R-179\b\|R-180\b\|R-181\b\|R-182\b"` over +`documentation/` and `*.md` → **0 hits**, and over all four repo roots (`felhom-agent`, +`felhom-controller`, `felhom.eu`, `app-catalog-felhom.eu`) → **0 hits**. `R-182` was checked and left +unused. -Checked by **PULL**, matching `head_sha` to each commit — CI mails only on failure, so an unchecked -green is an assumption. +--- -**Every commit in this session, across all three repos, is green.** +## 9. CI -| Repo | Commit | Task id | Run # | Conclusion | -|---|---|---|---|---| -| `felhom-controller` | `4be6467` (v0.192.0, the B2 floor) | 39 | 15 | **success** | -| `felhom-agent` | `cd6e267` (v0.120.0 + golden 3.0.0) | 41 | 3 | **success** | -| `felhom-agent` | `4bb84fc` (REPORT) | 43 | 4 | **success** | -| `felhom.eu` | `7406ac7` (phase-0 audit) | 40 | 21 | **success** | -| `felhom.eu` | `e3525e6` (host-install one grow) | 42 | 22 | **success** | -| `felhom.eu` | `14d8c00` (docs + registers) | 44 | 23 | **success** | +Appended after the push — run id and conclusion, per `CLAUDE.md`'s pull-check rule. -## 5. `--no-verify` +--- -**Not used anywhere.** Every push in this session ran its repo's `.githooks/pre-push` and passed. +## 10. Observations — noticed and NOT acted on -## 6. What remains — R-178 - -1. Reinstall **demo-hp** from golden 0.192.0 through the real installer path; prove claim → deploy an - app → back up → restore; show `df` proving one filesystem and a recovery unit landing on it. -2. Only then reinstall **demo-felhom** (it carries the PBS-DR/offsite tier, so it is the box whose - backup chain a reinstall actually disturbs). -3. **Then** vouch golden 0.192.0 in the hub, and flip the capability-map row to PROVEN-LIVE. -4. Re-run P1's restore-test against agent v0.120.0 — one command, turns a sound inference into an - observation. +- **The runbook's central claim was wrong in a way that mattered.** R-178 said *"a reinstall is now a + self-contained piece of work with no code left to write."* True about code; false about the artifact + channel — the agent half of the merge was unpublished, and following the documented path without + checking would have downgraded both boxes and produced a green, meaningless result. **No code change + was needed to make any step pass** — §10 asks this loudly, and the answer is no. What was needed was + a publish. +- **Prove-then-vouch was already spent when the session opened.** Not a defect in anything, but the + rule protected nothing because nothing enforced it. R-120's gate is the shape that would. +- **A per-stack `POST /stacks//backup` does not produce volume dumps** — the volume leg lives in the + full app-data pass. Not wrong, but the endpoint's name suggests otherwise and it cost time here. +- **privatebin's recovery unit carries no user data** (its catalog entry declares no `backup:` + section), so restoring it loses every paste. That may be intentional for an expiring, E2E-encrypted + paste bin — but nothing in the app's description tells the customer so. Catalog question, not filed. +- **V-c doubles systemd's mount-unit count** — every docker overlay appears twice, + `var-lib-docker-…-merged.mount` and `var-lib-felhom-docker-…-merged.mount`, because `/var/lib/docker` + is a bind of `/var/lib/felhom/docker`. Cosmetic, inherent to the chosen variant, no action. +- **`fstrim` cannot run inside the guest** (`EPERM`, unprivileged LXC), so space freed inside the guest + is not returned to the thin pool. It did not matter here; on a box that fills and empties repeatedly + it would. +- **The floor's own message mixes units** — it reports `64.2/68.7 GB used (93%)` against a threshold + stated as `97% used or 1.0 GiB free`, while `df` showed 96 %. GB-vs-GiB, so the percentage term + fires later than an operator reading `df` would expect. Minor; noted on R-181's fix shape rather + than filed separately. diff --git a/STATUS.md b/STATUS.md index 4c97d56..bb51550 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,6 +1,6 @@ # STATUS — what works, what's broken, what's next -**Updated 2026-08-02.** +**Updated 2026-08-03.** > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this > page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**, @@ -31,9 +31,12 @@ also delete it. A daily snapshot is armed as a stopgap, and we have never restor the data would vanish on the next update. Two are fixed, the third is now clear to fix because it is installed nowhere. *(R-156)* -**Local backups get 20 GB while apps get 50 GB.** An app that outgrows the smaller space stops being -backed up locally — and the off-site copy is made from the local one, so that stops too. Nothing is -lost: the last good copy is kept intact. *(R-163)* +**The reserve does not guard the step that actually fills the disk.** The rule that refuses a backup +when space runs low is checked at the wrong moment: the big write happens first, unchecked, and only +the small write after it is refused. So the thing meant to stop a runaway backup is the thing it +runs past. Worse, when it does refuse it says *"your last good copy is untouched"* — and we measured +that copy being overwritten by the earlier step anyway. Found by deliberately filling a rebuilt demo +machine. Nothing was deleted and you were emailed, both correctly. *(R-181)* **When that happens, only one page says so** — no email, no alert. The page that answers "is this app backed up?" is the one that stays silent. *(R-158)* @@ -65,9 +68,18 @@ barrier against a runaway backup filling the space the machine needs to run; put first means that when it comes down, the thing watching is already working and already tested. *(R-167, R-158)* -**The backup partition is gone from the base image.** A machine built from now on has one storage area -instead of two, so a backup can use whatever space the machine actually has free rather than a fixed -slice decided when it was built. The wall does not move; it stops existing. +**The backup partition is gone — and both demo machines now run on the new shape.** Each was wiped +and rebuilt from the new base image on 3 August and taken through the whole journey a customer takes: +set the machine up, install an app, back it up, restore it. One storage area instead of two, and the +space a backup can use went from 19 GB to 65 GB on the small machine and from 45 GB to 233 GB on the +big one. The wall does not move; it stops existing. Each machine was rebooted three times over and +came back correctly every time. **The two machines were rebuilt deliberately differently** — the +first from a copy of the image held locally, the second by the ordinary route a real customer takes, +fetching the image and checking it against the fingerprint we publish, so both the disk shape and the +delivery route are now proven rather than one twice. + +**Their previous demo apps and data are gone.** That was the point of a wipe and you approved it; the +machines now carry a couple of small test apps instead. **What replaced the wall.** It was quietly doing a second job — keeping a runaway backup from eating the space the machine needs to keep running. That job is now explicit: if a backup would push the disk @@ -81,32 +93,42 @@ quietly broke: one put your backups inside Docker's own storage, where the norma Docker would wipe them; another exposed all of Docker's internals to the part of the system that manages your drives. The third does neither, and costs one extra line of configuration. -**Nothing has changed on any existing machine.** They keep their current layout and go on working -exactly as before; they get the new shape only when they are reinstalled. The new base image is -deliberately **not switched on yet** — nothing will pick it up until a machine has been rebuilt from -it and checked, which is the next step. *(R-165, R-163)* +**The new base image is now switched on**, so any machine installed from here on gets the new shape. +The tester's box is untouched and keeps working exactly as before; it gets the new shape whenever it +is reinstalled. *(R-165, R-178)* ## What we're working on - **Now:** the last app whose data was never saved; today's decisions written down. -- **Next:** finishing the partition merge — **the build is done and the decision is made**; what is - left is to reinstall the two demo machines from the new base image and check one end to end. +- **Next:** fixing the reserve so it guards the step that fills the disk, and stopping it from + claiming your last good copy is untouched when it is not *(R-181)*. The partition merge itself is + done and proven on both machines. - **After:** rebuilding how the machine records whether an app is meant to be running. ## Waiting on you -- **How a new version reaches a machine.** Pushing the installer publishes it — half a minute later - every new machine downloads it, with no staging and no way back but another push. And publishing is - a step we remember rather than one the release performs, forgotten twice: a fix can be live here - and still not reach a new machine. Nothing is installing today, so this is the cheapest moment to - settle both. *(R-110, R-115)* +- **How a new version reaches a machine — and it has now been forgotten a third time.** Pushing the + installer publishes it: half a minute later every new machine downloads it, with no staging and no + way back but another push. And publishing is a step we remember rather than one the release + performs. On 3 August the new agent — the half of the partition merge that runs on the machine — + turned out to have been built and installed on both demo machines but **never published**, so a + rebuild would have quietly put the *old* one back and proved a version nobody ships. Caught before + the wipe and fixed in ten minutes, but only because someone happened to look. Nothing is installing + today, so this is the cheapest moment to settle it. *(R-110, R-115)* - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a session log; nothing suggests anyone else saw it. *(R-132)* -- **Nothing — both partition-merge questions are answered.** You chose the storage shape and the hard - stop; both are built. The tester's box needs no conversion: it will simply be reinstalled. *(R-165)* +- **Nothing on the partition merge itself** — the shape is chosen, built, and now proven on both demo + machines. The tester's box needs no conversion: it will simply be reinstalled. What is left is the + reserve defect above, which is ours to fix, not yours to decide. *(R-165, R-181)* ## Changed since last update +- **2026-08-03** — Both demo machines wiped and rebuilt from the new base image, and taken through + set-up → install an app → back it up → restore it. The backup space ceiling is gone and measured. + The hard stop that replaced the old partition was fired for the first time on real hardware — it + refused, it deleted nothing, it emailed you, **and it turned out to be watching the wrong step**; + that is now the top thing to fix. + - **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in a mode where reading a page cannot block a machine's status update; a one-word difference meant that setting had **never taken effect**, for the hub's whole life. Fixed and verified live. **Also found: diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 6d74a85..00fec6e 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -86,7 +86,7 @@ | Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped | | **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait | | **A failed per-app Tier-1 backup reaches the OPERATOR** (app, error, and the target filesystem's used/free bytes at the moment of failure) | controller **v0.191.0**, hub **v0.89.0** (R-158, closed by R-167) | **PROVEN-LIVE (2026-08-02)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it | -| **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.192.0** (R-165 / D-a / B2) | **IMPLEMENTED — NOT PROVEN-LIVE** | `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **NOT PROVEN-LIVE, and the missing leg is named: no box has been reinstalled from this golden (R-178).** Per this map's own rule a PROVEN-LIVE claim needs an end-to-end citation, and "the golden baked" is not "a box built from it works". **The golden is deliberately UNVOUCHED** so no fresh install picks up an unproven layout — prove first, then vouch (`CONTEXT.md` S-14). Every box in the field is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity | +| **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.192.0** (R-165 / D-a / B2) | **IMPLEMENTED — the LAYOUT half is PROVEN-LIVE (2026-08-03); the REFUSAL half is not (R-181)** | `REPORT.md` (R-178 reinstalls) + `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately.** **Proven (R-178, 2026-08-03):** *"a local backup is bounded by the box's FREE SPACE, not by a partition set at build time"* — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp `--golden `; demo-felhom the normal manifest route with **`verified sha256 54e2a4c431daf580… matches the hub manifest`**), each showing `mp0` at `/var/lib/felhom` with **no `mp1`**, both consumer paths real mounts on ONE filesystem (`stat -c %d` = `64519` on all three), 3/3 reboots each, and claim → deploy → backup → **restore** with a planted marker returning byte-identical. Space available to a recovery unit measured at **65 GiB / 233 GiB**, against the **19 GiB / 45 GiB** those boxes' `mp1` slices offered. **NOT proven — and measured FALSE in part:** *"a capture that would exhaust it is refused per app rather than allowed to stop the container runtime"*. The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — **but it is checked only in `captureAllRecoveryUnits`, while `runVolumeDumps` writes the bulk with no floor check at all**, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → **R-181**. The golden **is now VOUCHED** (2026-08-03, hub `Artifact manifest set: … golden=0.192.0`), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity | | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | | | **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index d6efc59..5767345 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -15,7 +15,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-94** | ~~A hand-synced version constant drifts, and the gate that would catch it is never run~~ | **CLOSED — SHIPPED** (hub v0.87.0, 2026-08-02) | — | **All three legs closed.** **(a) closed by DELETION, not derivation** — deriving is not achievable honestly: the Setup command fetches `felhom-host-install.sh` at RUN TIME from a website that git-syncs `main` every 30 s (R-110), so no build-time value in the hub can be true, and a number that is wrong carries a version number's authority while being a guess. The const, the `pageData.ScriptVersion` field, its assignment and the rendered label are gone; a NOTE stands where the const was so it is not helpfully re-added. **(b)** `hostinstall_gates.py` gate 1 INVERTED — it now asserts the hub carries **no** host-install version literal, in six code shapes across every `.go`/`.html` under `hub/`; and the gate is now invoked, by `scripts/repo_gates.py` and the pre-push hook (→ R-29). **(c)** the tautological `render_test.go:219` assertion is deleted, not replaced — there is no version to assert. It was demonstrated PASSING with the const at `9.9.9` while the script was 1.22.0. The label had been wrong for 19 days (since 2026-07-14) | — | | **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly | CC | | **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` **DEFERRED LEG, AND IT RECURRED → R-115.** This row's shipped half stands and is not reopened: the bump happened, was verified, and was proven end-to-end by the E-2d install. But its own closing line — *decide whether publishing joins the release train rather than staying a remembered step* — was never acted on, and agent 0.114.0 reproduced the exact condition the same afternoon. The recurrence is filed as **R-115**, not as a reopen, because the stale-channel finding is closed while the process defect that caused it is a distinct problem with a distinct fix. | CC | -| **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC | +| **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design. **THIRD INSTANCE, 2026-08-03 — and it was found by a runbook that had been told there was nothing left to do.** Agent **v0.120.0** — the agent half of the R-165 merge — was built, committed at `cd6e267`, and deployed to BOTH demo hosts, and was **never published**: `GET …/generic/felhom-agent/0.120.0/felhom-agent` → **HTTP 404** (0.119.0 → 200), and the hub manifest accordingly vouched **0.119.0**. The consequence is the sharpest yet, because installer step 5's idempotent skip requires `installed == vouched` EXACTLY: a documented-path reinstall would have **downgraded both boxes** from the merge-aware 0.120.0 to the pre-merge 0.119.0 — silently, since the current `step_grows` sets `SYSDATA_GROW=0` so 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on error) never fires and the install would have *succeeded* while proving a stack nobody ships. **R-178's own row asserted `agent v0.120.0 is live on BOTH hosts` and `no code left to write`; both were true and both were beside the point** — the gap was publication, which no one checks. Fixed in-session on the operator's ruling: `scripts/publish-agent.sh 0.120.0` (sha `a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`, round-trip GET verified) then vouched, and both reinstalls then fetched and sha-verified it from Gitea. **This is the third instance of a row that has been WAITING-ON-OPERATOR since 2026-07-29; option (b) — a gate that refuses to deploy or vouch an unpublished version — would have caught all three** | CC | | **R-116** | ~~**The drive-absent alarm and its recovery were a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`; `backup_target_absent` never fired at all**~~ | **SHIPPED + PROVEN-LIVE** (agent v0.116.0, 2026-07-30) | — | **CLOSED. The full four-event sequence, on the wire, on a fresh box** (`audits/R116-v0116-2026-07-30.md`): `backup_target_absent (error)` on detach → `backup_target_restored (info)` on return for the TARGET, and `storage_disconnected (error)` → `storage_reconnected (info)` for a NON-target drive on the same box four minutes apart. **Two matched pairs, correctly discriminated — and discrimination is proven NON-trivially for the first time**, since both prior runs had the target itself emit the generic event. Gate fired in **3 s**; all four events reached the hub, so the specific alarm, its severity, its Hungarian copy and the hub routing are now exercised end-to-end. **Over-correction PASSES with a positive observable** (0 ABSENT lines / 0 drive events over 2m14s with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking). **NARROWED by the R-117 spike (`audits/SPIKE-r117-bind-liveness-2026-07-30.md` §12), and it stands as written:** the 2 `RETURNED` lines are a genuine positive observable, so rule 3 is satisfied — but `degraded:false` over that window was read off a drive whose bind was **dead** (R-117), so the window evidences **"the gate did not over-fire"** and **NOT** **"the drive was healthy."** No other part of this row changes: every input to the pairing fix is configuration-derived (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), which R-117 does not touch. Ran on a nested PVE on **demo-hp** per `runbooks/target-selection.md` — through the **real day-0** from the v1.25.0 ISO, with the agent **installed unaided from the vouched Day-0 manifest** (published sha `b47c5c4dab641ee5…`, independent registry GET verified, manifest read back), drives enrolled through the real endpoints, device loss a real hot-detach. **THE FIX, and the ruling is the substantive part:** the mechanism was first isolated from the captured payload (`DIAG-r116-disks-payload-2026-07-30.md`) after two fixes aimed at shapes that do not occur. **Both smaller-looking options were REJECTED because they regress R-114** — `backup_target_offer.go:79` reads `BackupTarget && MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns before its `TargetAbsent` branch, so back-filling `MountPath` on the Observe row **or** flagging the registry row (whose `MountPath` is the stale unit-file value) would have told the customer the backup target is fine while its drive was gone. **R-114's correctness was resting on R-116's bug** — a coupling invisible until the payload existed. Taken instead: the Observe row gets the **guest path only** (`mount_path` stays `""`, which is true) from a new `ConfigPath` (`json:"-"`, so the cross-repo golden + key-set contract is untouched), and the union row is deduped **on guest path** — the join being CONFIGURATION (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849; 4 red-proofs each asserted to land, and red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. v0.115.0 left in place (inert, harmless). Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded. **Caveat: the drill's controller was 0.185.1 from the golden, which PREDATES R-114**, so its absent-state banner showed the old false copy — the golden being a release behind, not a regression → **R-120** | — | | **R-120** | ~~**The golden baked a controller that predated R-114 + R-112, so a FRESH box showed the customer the WRONG absent-target message**~~ | **CLOSED — golden rebaked + PROVEN-LIVE, and the class now has an ENFORCED gate** (golden 0.186.0 + hub v0.82.0, 2026-07-30) | — | **`audits/R120-golden-rebake-2026-07-30.md`.** **Half 1 — the artifact.** Golden **0.186.0** baked from `main`'s controller in the DooPlex bake fixture (overlay2 OK, **3 mounts**, FATAL 0, exclusions 0, 618 MB, upload **201**, `GOLDEN_SHA256=b760ac6a33e70700…`, token-leak grep 0, GL-1 teardown, `drill.qcow2` back to `virgin`). Three observables: **published** — anonymous GET (what the installer does) 200 / 648930639 bytes / sha identical to the bake; **vouched** — manifest read BACK; **resolved** — `Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)`. Floor **untouched** per publish-train rule 2 (`min_controller_version` still 0.156.0; it is a separate form); MinAgent left 0.113.0 as 0.186.0 declares. **Proven on a REAL day-0, not the fixture** (per the Part-1 rule now in `runbooks/target-selection.md`): VM 9402 on demo-hp from the v1.25.0 ISO → `Controller elindult (0.186.0)`. With the target detached the endpoint returned the **`TargetAbsent`** copy — *„A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod…"* — **and `offer_path` absent entirely**; the day-old read on the 0.185.1 golden had returned the false system-disk message **plus** an offer of the other drive. **Half 2 — the mechanism, operator ruling REFUSE.** hub **v0.82.0**: the gate sits in `hub/internal/web/configs.go` `handleSetArtifacts` immediately before the only write — the sole UI path to `SetArtifactManifest` — so it runs on every vouch without anyone choosing to, and it **refuses** rather than warning. Signal: `store.NewestReportedControllerVersion()` over `reports.controller_version`, **semver-compared in Go** (`MAX()` in SQL ranks 0.99.0 above 0.186.0 — a pair this fleet has shipped). Fail-open in exactly two deliberate cases: empty golden field, unknown fleet version. **NEAR-MISS RECORDED: the first draft read `guests.controller_version`, a column that exists and that NOTHING writes** — it would always have seen `""` and failed open, i.e. inert, this gate's own failure shape, one grep from shipping. 4 tests through the **production handler** over httptest (never a seam), the refusal asserting **both** the flash **and** that the manifest was not written; red-proof: deleting the block makes the stale golden vouchable again. **PROVEN LIVE on the deployed hub by re-attempting the original mistake:** vouching 0.185.1 → `HTTP 303 …flash=golden_behind_fleet` + `[WARN] artifact vouch REFUSED: golden 0.185.1 is older than the newest controller the fleet reports (0.186.0)`, and the manifest read back **unchanged at 0.186.0**. Recorded on **R-29's audit list** (`ROADMAP.md`) as the **first enforced gate** beside its three orphans, so the contrast is kept — the orphans are unchanged. Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded, exactly as `sess-e` was (and `sess-e` was deleted this run) | — | | **R-117** | ~~**A drive's guest bind becomes a DEAD MOUNT while every signal reads healthy — and it happens in TWO ways, only one of which the original framing covered.** (a) *after a detach/return*: the host raw mount heals onto the NEW device via its fs-UUID-keyed unit while the bind still names the OLD one, so the gate takes its `Return` branch and restarts the customer's apps onto a namespace that `EIO`s on every call; (b) *in STEADY STATE, no cycle at all* — a device that errors without disappearing leaves the raw mount `active`, `BoundUnderParent` `true` and the drive never `Disconnected`, so **the gate produces no action and NOTHING is emitted on any channel**~~ | **SHIPPED + PROVEN-LIVE** (agent **v0.117.0**, 2026-07-30) | — | **CLOSED. `audits/R117-v0117-2026-07-30.md`.** `BoundUnderParent` gains a THIRD term at both /disks sites: `bindLiveness` reads `/proc` only and requires (a) **the bind names the same device as the raw mount** and (b) **the filesystem has not aborted** (`shutdown` **or** `emergency_ro`, both measured). **BOTH CHECKS ARE LOAD-BEARING and this is the substantive part:** R-117 was filed as a detach/return defect, but a device that fails WITHOUT disappearing gives the identical all-signals-healthy state with the **devnos EQUAL** and the drive never `Disconnected`, so the gate emits nothing at all, indefinitely (R-117a) — the device comparison alone cannot see it, and a P1-only fix passes every payload test (red-proof RP3 exists for exactly that). **THREE states, never a bool:** `{Unknown, Live, StaleDevice, Aborted}`, `Unknown` is the zero value, and every caller reads `Usable()` where unknown counts **PRESENT** (absent stops a customer's apps — the `newestArchiveOn` trap). **NO NEW RECOVERY PATH:** `AttachDrive`'s normalize leg already did the repair and three call sites already invoked it (20 s ticker, agent startup, and **the controller's `Return` branch BEFORE `restartStacks`**); all three were defeated by `if n == 1 && GuestSeesMount(...)` logging *"fully live, no-op"* about an EIO namespace. **RULING (asked for, given, flagged for overrule):** `StaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, guest never restarts — init PID identical); `Aborted` ⇒ **quiet no-op and SURFACE**, because a re-bind lands on the SAME dead superblock and this runs every 20 s = an infinite silent retry that masks the state. No operator decision required: it routes an already-broken state into the **existing** gate, event types and Hungarian copy — no new customer-facing concept — and the alternative is apps writing documents into a filesystem that rejects every write. **ORDERING TRAP caught by a test:** abort-first classifies the real return state as aborted (its stale bind carries `shutdown` too) and refuses the repair **while still reporting correctly**, so the abort flag is read off the RAW mount in the stale case. **LIVE on demo-hp** (brought 0.113.0 → 0.117.0 first — see R-121): RETURN `raw 8:32 / bind 8:16 shutdown` ⇒ `stale-device`, usable **false**; IN-PLACE `both 252:11 emergency_ro`, raw unit still `active` ⇒ `filesystem-aborted`, usable **false**; healthy ⇒ `live`; **340–497 µs**. **No block I/O proven by strace** (only `/proc/self/mountinfo`, **0** statfs) — the Part 1 `CLAUDE.md` fence applied to its own first consumer. **No regression through the REAL pipeline:** `GET /disks` with the controller's own credential shows the live backup-target drive `bound_under_parent=True`, with 32 gate lines in 3 min as the positive observable and zero spurious transitions. Tests **849→863**, 29/29 green, **6 red-proofs each verified to land** — and **RP1 failing to fail exposed a HOLLOW test**: the aborted fixture used a `/dev/mapper` device, for which `RoleForStorage` derives `role=system`, and a system row never runs the conjunction, so it reported false by DEFAULT and no mutation could fail it. Fixtures now assert the production row shape first. Teardown all 3 layers; hub layer = the vouched manifest, **retained** (it is the product, not scratch). **NOT covered:** the stale-bind repair on hardware — `StablePathForRaw` hardcodes the live parent, so it would write into guest 9201's namespace (R-117h); and sustained-load behaviour, still unmeasured. Follow-ups **R-117g** (no guided recovery for an aborted fs), **R-117h** (parent dir not test-seamable), **R-121** | CC | @@ -92,7 +92,10 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-174** | ~~**The app-stop guard's crash recovery started apps onto MISSING drives — a regression in v0.189.0 code.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0, 2026-08-02) | — | **Found by REVIEW on 2026-08-02, in code shipped 2026-08-01, and closed the same session — R-171 one path over.** `appStopGuard.SetStarter(stackMgr)` handed `Recover` the RAW stack manager, whose `StartStack` has no drive gate, and `Recover` runs **at startup** — exactly when an external drive may not have come back. So: a backup stops an app, the box loses power, the drive does not remount, and the app is started on a missing drive. The rule was not new — the API's own `startGatedByMissingDrive` already refused this to the customer; the guard bypassed it. **`bootDriveGate` could NOT be reused whole**, and the reason is recorded in the code: its holder #2 reads `bootAppStopGuard.HeldStacks()`, which during `Recover` is **the guard's own marker** — it would refuse every recovery it was meant to perform — and holders #1/#2 read package-level vars assigned AFTER `Recover()` runs, so a whole-gate reuse would be correct only by accident of nil-safety. Holder #3 is extracted into a shared `driveStartGate` with **two callers, one implementation**, and `TestBootDriveGateAndAppStopShareTheDrivePredicate` pins the delegation. **A REFUSAL IS NOT A FAILURE:** new `ErrStartRefused` + a `Refused` bucket — both keep the marker, only `Failed` alarms, because routing a deliberate hold into `NotifyBackupFailed` (customer-enabled by default) is the very R-171 false alarm this fixes. `main.go` guards on `Alarming()`, not `!= nil`, and the pre-existing seam test was TIGHTENED to require it. **Live on 9201, both directions:** drive held unmounted → `refusing to restart "calibre-web" … drive /mnt/felhom-drives/hdd_1 is not a live mountpoint`, marker retained byte-identical, zero containers started, `not alarming`; drive returned → `restarted calibre-web`, marker CLEARED. **ID established free:** `grep -ro "R-174\b" documentation/ *.md` → 0 hits | — | | **R-175** | ~~**`07-backup-architecture.md` §7.5 states ONE box's size bound as if it were the fleet's.**~~ | **CLOSED — FIXED 2026-08-03** (same pass as R-165) | — | **Measured, not inferred** (`audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1: `pct config 9201` on both hosts). Independent of the merge — the sentence is wrong today and will be wrong differently after R-165. **The fix is to state the bound as a FUNCTION of `mp1`, not a constant**, and to say which box any quoted figure came from. Same class as the comment-asserting-an-invariant rule: a doc stating a fleet-wide number that only one machine satisfies reads as settled and is not. **ID established free:** `grep -ro "R-175\b" documentation/ *.md` → 0 hits **FIXED.** §7.5 gained a **7.5.1** which (a) states plainly that the bound is a FUNCTION of `mp1` and applies only to a box still on the split layout, naming all three real shapes, and (b) records that the ceiling itself has been removed by R-165 for boxes built from golden ≥ 0.192.0. Fixed in the same pass as the merge rather than filed and forgotten, because the section would otherwise have been wrong in two ways at once | CC | | **R-176** | **Two prerequisites for the R-165 merge are UNMEASURED, and both are cheap.** (a) Whether a **pre-merge archive** (carrying `mp1`) restore-tests cleanly into a **merged-layout** guest — reading `mountParity` (`felhom-agent/internal/reconcile/restoretest.go:347`) says it should, because the restore recreates `mp1` from the archive so archive and restored guest agree; **that was reasoned from source and never executed.** (b) The in-place per-box migration (move `/felhom-data` onto `mp0`, drop the slot, verify) has **never been rehearsed even once**, so "is the box restorable at every point of it?" is currently unknown | **(a) ANSWERED 2026-08-03 (P1: PASS). (b) NOT REQUIRED — operator ruling: every node is reinstalled, none migrated** | blocks R-165 landing safely | **Filed because this project's own record is that FOUR production designs specced against unvalidated mechanisms were all wrong** — which is exactly why R-165's own spike refused to design. Both are one command on a **Tier-0** box (D-d: both demo boxes are disposable). (b) is only required work if Peti's box turns out to need migrating rather than reinstalling — the hub cannot answer that (M5: `peti-felhom` exists as a customer with **no host in the register**), so it is the operator's input. **ID established free:** `grep -ro "R-176\b" documentation/ *.md` → 0 hits **UPDATE 2026-08-03.** **(a) is measured and passed** — `audits/SPIKE-r165-phase0-2026-08-03.md` P1: a real pre-merge archive (`mp0+mp1`, confirmed from its own vzdump log) restore-tested on demo-hp, `pass: true`, `mount_parity: ok`, 84 s, with `mountParity` untouched. One limit stated rather than glossed: it ran with the pre-merge agent because the merged one did not exist yet, and the comparison is archive-vs-its-own-restore which never consults the host layout — **re-run it once against agent v0.120.0**, which is one command. **(b) is withdrawn, not deferred:** the operator ruled that every node is REINSTALLED rather than migrated in place (both demo boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed in a few weeks), so the in-place migration rehearsal has no consumer. Recorded explicitly rather than silently skipped | CC | -| **R-178** | **The merged golden (0.192.0) is built and published but NO BOX HAS BEEN REINSTALLED FROM IT, and it is deliberately UNVOUCHED.** `build-golden.sh` v3.0.0 baked it with variant V-c and every retargeted assertion passed on the real bake (`including mount point mp0 ('/var/lib/felhom')`, no `mp1` line, `both paths are ONE filesystem`); it is in the registry (HTTP 200, sha `54e2a4c431daf580…`). What has NOT happened is Part 4: reinstall each demo box from it and prove claim → deploy an app → back up → restore | **READY (M) — NEW 2026-08-03** | blocks R-165 reaching PROVEN-LIVE; blocks the capability-map row | **The golden is UNVOUCHED ON PURPOSE and that is the safe state**, not an oversight: vouching is what makes a fresh install pick it up, so vouching a golden no box has been proven from would put an unproven disk layout in front of the next install anywhere. **Prove first, then vouch** — the bake script's own output treats the hub record as a separate deliberate step for this reason. **Everything else for the merge is shipped and green:** controller v0.192.0 (the B2 floor) is live on 9201, agent v0.120.0 is live on BOTH hosts, and `felhom-host-install.sh` computes the single grow from the thin pool. So a reinstall is now a self-contained piece of work with no code left to write. **Order matters: ONE box at a time**, demo-hp first, proven end to end, and only then demo-felhom — two in parallel leaves no working reference to compare against. Note demo-felhom carries the PBS-DR/offsite tier, so it is the one whose backup chain a reinstall actually disturbs. **ID established free:** `grep -ro "R-178\b" documentation/ *.md` → 0 hits | CC | +| **R-181** | **The capture floor guards the cheap leg and not the leg that fills the volume — and its refusal message asserts an invariant the code does not provide.** B2 (controller v0.192.0) is recorded on R-165 as the deliberate replacement for the bulkhead the `mp1` partition used to give. It is consulted in exactly one place — `m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which writes a manifest and a compose copy: **a few KB.** The leg that writes the bulk, `runVolumeDumps` (`backup.go:535`), has **no floor check at all** — its gates are protected-stack, volume-less, disconnected, decommissioned — and it runs FIRST, by design (*"MUST run before captureAllRecoveryUnits so the manifests enumerate the fresh tars"*, `backup.go:483`). So the write that fills the filesystem is unguarded, and the floor then refuses the write that would have cost almost nothing. **Second limb: the refusal message is false.** `recovery_unit.go:331` prints *"the previous unit is untouched and NOTHING was deleted"*. Nothing was deleted — true. Untouched — **measured false**: privatebin's `volume-dumps/privatebin_privatebin_data.tar` went `26c546c2…` → `b538ab89…` and opengist's went **182,272 B → 2,147,666,432 B**, both rewritten by the earlier leg, while each unit's `manifest.json` kept `created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so a unit's payload can be swapped under a stale descriptor and **nothing in the unit can detect it** | **READY (M) — NEW 2026-08-03** | blocks **R-165** reaching PROVEN-LIVE | **FIRST LIVE FIRING OF B2, and it is why the runbook asked for one.** Proven on demo-hp 2026-08-03 06:40:03 on a box reinstalled from the merged golden (R-178). Method: a real 2 GiB file in opengist's data volume, then `fallocate` to bring the filesystem to 96 % used / 3.0 GiB free — both floor terms deliberately still clear, so the run started. **The `fallocate` instrument was proven before use** (5 GiB moved guest `df` 977M→6.0G while thin-pool `data_percent` stayed 29.03 → 29.03: zero blocks allocated), because demo-hp's thin pool is 53.93 GiB and a real fill to 97 % of a 70 G volume would have exhausted it and corrupted every guest on the box including the `drill-r50` fixture. Sequence observed: opengist's volume dump wrote **2.0 GB unguarded** → free fell to 1.0 GB → **both** apps' recovery-unit captures were then REFUSED on the `1.0 GiB free` term, each pushing `recovery_unit_capture_failed` (severity `error`) to the hub, accepted HTTP 200. **What DOES hold: it refuses per app rather than aborting the run, it never deletes, and the alert reaches the operator.** **Fix shape, not written this session by design (§7 of the runbook):** the floor belongs before the write in `runVolumeDumps` too, the message must stop claiming what the earlier leg has already falsified, and per `CLAUDE.md` *"a comment asserting an invariant needs a test pinning it"* the pinning test must assert the **consequence** (after a refusal, is the previous unit's payload byte-identical?) and not the mechanism. **Class:** the sixth entry in `CLAUDE.md`'s own table of shipped guarantees the code did not provide — found, as four of those were, only on live hardware | CC | +| **R-180** | **`--archive-storage` is accepted without checking the agent's token will ever be granted on it, and the failure lands at step 8/8 — after the root@pam password has already been rotated.** `felhom-host-install.sh` validates the archive storage EXISTS (`pvesm status --storage`, `:1583`) and that the golden volid RESOLVES on it (`:1661`), both in pre-flight. It never checks that storage against the ACL set it is about to grant, which is the fixed default `local local-lvm felhom-pbs` (`--acl-storages`, which `runbooks/day0-install.md` tells the operator **not** to pass). A storage outside that set therefore passes every pre-flight gate and dies at the last step | **READY (S) — NEW 2026-08-03** | — | **Hit live on demo-hp 2026-08-03** during R-178 Phase A, self-inflicted and therefore a clean demonstration: the golden was staged on `felhom-backup` (the enrolled NVMe, where the box's vzdumps live) and `--archive-storage felhom-backup` passed. Pre-flight passed; steps 1–7 ran; step 8 returned `reconcile: bring-up restore: proxmox: POST /nodes/felhom-host/lxc -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)`. **The cost is the ORDER, not the error** — by the time it fires, step 2 has minted the PVE token, step 4b has **rotated root@pam and vaulted it** (so the old console password is already dead), and step 5 has installed the agent. Recovery was `--resume` after moving the golden to `local`, which worked cleanly. **This is statically checkable in pre-flight**: `ARCHIVE_STORAGE ∈ PVE_STORAGES` is a one-line assertion over two variables both known at `:1583`. Same class as R-29 — the checkable thing that nothing checks | CC | +| **R-179** | **`--uninstall` leaves the NAS network-storage systemd units behind, with the automount in `failed` state and the parent bind still mounted.** The teardown's residue-diff provenance (`day0-install.md` Part E: *"a full-filesystem diff against the pre-install baseline showed zero Felhom-named leftovers"*) is from **v1.9.1**, which predates the NAS network-storage feature. A box that has ever had a network share configured keeps `/etc/systemd/system/mnt-felhom\x2ddrives-.mount` and `.automount` after a full uninstall | **READY (S) — NEW 2026-08-03** | — | **Observed on demo-hp 2026-08-03** after `--uninstall --vmid 9201`: `mnt-felhom\x2ddrives-Felhom\x2dShare.automount` **loaded failed failed**, its `.mount` `loaded inactive dead`, and `mnt-felhom\x2ddrives.mount` still `active mounted` — the uninstall's own output had warned `/mnt/felhom-drives/Felhom-Share is busy — NOT forcing` and `/mnt/felhom-drives root bind left mounted`, which is correct behaviour (it never forces an unmount) but is not teardown. Cleared by hand before the reinstall: stop both units, remove both unit files, `daemon-reload`, unmount the autofs then the parent. **NEGATIVE CONTROL, same day:** demo-felhom's uninstall left **nothing** (`ls /etc/systemd/system | grep -i felhom` → only the unrelated `felhom-bootstrap.service`; no felhom mounts) — because that box had no network share configured. **So the residue is conditional on the feature having been used, which is exactly why a diff taken on a box that never used it reported clean.** `felhom-bootstrap.service` is NOT residue — it is the ISO first-boot unit, `disabled`+`inactive`, exactly-once and already fired | CC | +| **R-178** | **The merged golden (0.192.0) is built and published but NO BOX HAS BEEN REINSTALLED FROM IT, and it is deliberately UNVOUCHED.** `build-golden.sh` v3.0.0 baked it with variant V-c and every retargeted assertion passed on the real bake (`including mount point mp0 ('/var/lib/felhom')`, no `mp1` line, `both paths are ONE filesystem`); it is in the registry (HTTP 200, sha `54e2a4c431daf580…`). What has NOT happened is Part 4: reinstall each demo box from it and prove claim → deploy an app → back up → restore | **CLOSED — BOTH BOXES REINSTALLED AND PROVEN (2026-08-03)** | blocks R-165 reaching PROVEN-LIVE; blocks the capability-map row | **The golden is UNVOUCHED ON PURPOSE and that is the safe state**, not an oversight: vouching is what makes a fresh install pick it up, so vouching a golden no box has been proven from would put an unproven disk layout in front of the next install anywhere. **Prove first, then vouch** — the bake script's own output treats the hub record as a separate deliberate step for this reason. **Everything else for the merge is shipped and green:** controller v0.192.0 (the B2 floor) is live on 9201, agent v0.120.0 is live on BOTH hosts, and `felhom-host-install.sh` computes the single grow from the thin pool. So a reinstall is now a self-contained piece of work with no code left to write. **Order matters: ONE box at a time**, demo-hp first, proven end to end, and only then demo-felhom — two in parallel leaves no working reference to compare against. Note demo-felhom carries the PBS-DR/offsite tier, so it is the one whose backup chain a reinstall actually disturbs. **ID established free:** `grep -ro "R-178\b" documentation/ *.md` → 0 hits. **CLOSED 2026-08-03 — both boxes reinstalled from the merged golden, by two DIFFERENT supply paths, and proven end to end** (`REPORT.md`). **demo-hp — the layout proof**, installed with `--golden local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst` (installer v1.22.0, sha `ed02acb2…`, byte-identical to the repo copy): `mp0 …mp=/var/lib/felhom,backup=1,size=70G`, **no `mp1`**; `/var/lib/docker` → `…disk--1[/docker]` and `/mnt/sys_drive` → `…disk--1[/sys_drive]`, both real mounts, both writable, both in `/etc/fstab`; ONE `df` figure (69G/65G) and `stat -c %d` = `64519` on all three paths; **reboots 3/3** (08:13:33 / 08:13:59 / 08:14:19, controller healthy in 12s/7s/7s, all three still mountpoints after each). **demo-felhom — the pipeline proof**, installed with `--force-gitea-golden` and NO local golden used (preflight logged *"golden: none local — will fetch + verify from Gitea in step 7/8"*, bypassing the 06:58 bake artifact sitting on the same box): **`verified sha256 54e2a4c431daf580… matches the hub manifest`** for the golden and **`verified sha256 a7763d31b55b5ce7…`** for the agent — the observable this second box exists to produce; `mp0 …size=250G`, `grep -c '^mp1:'` → 0, one `df` figure (246G/233G), reboots 3/3 (09:18:55 / 09:19:12 / 09:19:30). **Journey proven on BOTH**, endpoint-level (no browser on DooPlex — the exact endpoints the dashboard's own JS calls): claim (`POST /claim` with the pre-auth HMAC CSRF + `felhom_claim_csrf` cookie; gate discriminator flipped `dashboard not yet claimed` → `authentication required`) → deploy (`POST /api/stacks//deploy`) → capture (`POST /api/debug/backup/dbdump`, which runs the production `RunDBDumps`) → **restore** (`POST /backup/restore`): a planted marker deleted from the live volume came back with an **identical sha256** on each box (`ac1faae6…ae861` privatebin/demo-hp in 9.2s; `bc550798…b59e939` opengist/demo-felhom in 9.4s), recovery units on the single volume in both cases. **Ceiling gone, measured:** 65 GiB (demo-hp) and 233 GiB (demo-felhom) available to a recovery unit, against the 19 GiB and 45 GiB their pre-wipe `mp1` slices offered. **Two deviations, both the operator's call and both recorded:** the golden was ALREADY vouched when the session opened (hub log `2026/08/03 07:23:26 Artifact manifest set: agent=0.119.0 golden=0.192.0`, ~10 min before the first read of this session — so §7's prove-then-vouch order was already spent and the operator elected to accept it); and agent **0.120.0 had never been published**, so the vouched agent was 0.119.0 — published + vouched before the reinstalls (→ **R-115** third instance). **Three new findings: R-179, R-180, R-181** | — | | **R-177** | **There is no operator-triggerable "run the fill check now" path.** `fill-watch` is reachable only on its daily 03:30 schedule plus the once-at-startup run added in controller v0.191.1 — so the only way to exercise it on demand is to restart the controller | **READY (S) — NEW 2026-08-02** | — | **Noticed while live-validating R-167 on 9201, not by a failure.** It cost a controller restart per observation during validation, and it costs the same on a support call: after a customer frees space, nobody can confirm the warning has cleared without restarting their controller or waiting until 03:30. **Partially mitigated already** — v0.191.2 makes every run log a positive observable (`checked N filesystem(s), M unreadable/skipped, K notification(s)`), so at least a run that DID happen is visible; the gap is triggering one. The scheduler has `GetJobs` but no run-now, so this is a general affordance, not a fill-watch one — **scope it as "run a named scheduler job now", operator-gated.** **ID established free:** `grep -ro "R-177\b" documentation/ *.md` → 0 hits | CC | | **R-173** | **The hub's SQLite PVC is excluded from every Longhorn backup job.** `pvc/hub-data` carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly` (04:00 / Sun 05:00) are the ONLY recurring jobs and both target the `default` group — so the 128 MB `/data/hub.db` has **no volume-level backup**. That database holds `host_recovery` (every managed box's break-glass root password), `host_escrow` + `host_escrow_superseded` (escrow custody), `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg endpoints/peers — i.e. the material several documented recovery routes depend on | **READY (M) — NEW 2026-08-02** | — | **Noticed while checking the blast radius of the R-172 WAL change, not by a failure** — the WAL work needed to know who copies this file, and the answer turned out to be nobody on a schedule. **Establish before designing:** (a) whether the exclusion is deliberate (a 1 Gi RWO Longhorn volume snapshotting a 128 MB SQLite file is cheap, so the label looks like a leftover rather than a decision) and by whom; (b) whether anything else backs it up out-of-band that this census missed — the `_recovery-inventory-2026-07-28.md` records a MANUAL hot copy, which is not a backup. **When it is designed, it must be WAL-aware** (R-172): a volume snapshot of a live WAL database is crash-consistent and replays on open, which is fine, but any file-level copy must take `hub.db-wal` too or it silently loses the newest writes. **Grep establishing the ID was free:** `grep -ro "R-173\b" documentation/ *.md` → 0 hits | CC | | **R-158** | ~~**A local Tier-1 app-data backup failure reaches no hub channel.**~~ | **CLOSED BY R-167 — SHIPPED + PROVEN-LIVE** (controller v0.191.0 + hub v0.89.0, 2026-08-02) | — | **Closed by the wire it named; no second row was filed for it** (R-167 subsumes and widens it). New `unitNotify` seam + `SetUnitNotify` beside the manager's existing three, called from `captureAllRecoveryUnits` **per app with the loop continuing**, carrying the target filesystem's used/free bytes at the moment of failure — the cause is usually a full filesystem and those numbers answer *why* without an operator logging in. **ROUTED TO THE OPERATOR, NOT `backup_failed`, AND THAT OVERRIDES THIS ROW'S OWN PROPOSAL.** The proposal above said *"emitting the existing `backup_failed`"*; that type carries a `customerMessages` entry AND sits in `settings.DefaultEnabledEvents`, so it would email the customer in Hungarian about a failure they cannot act on — precisely the mistake R-97a avoided by minting `whole_guest_backup_failed`. **Decision D-c routes it to the operator and D-c wins.** New `recovery_unit_capture_failed` in `allowedEventTypes` **and** `notify.operatorOnlyEvents`; `notify.IsOperatorOnly` added so ONE test pins both registers (allowlisted-but-not-operator-only is invisible when they are checked separately — the v0.78.0 defect). **Red-proof:** removing the register entry shows the customer being emailed. **Live on 9201:** two events accepted and stored, `operator | sent`, and the positive observable `customer | recovery_unit_capture_failed | skipped | operator_only` read from the hub's `notification_log` | — | @@ -102,7 +105,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-162** | **`docker diff` is the gate's only witness, and its failure mode is quiet.** The gate's power comes from `docker diff` excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the mount-occupancy and writability legs **and would not say so**. | **WATCHING** — a limitation, not a defect | — | It **fails closed**: the canary self-test would stop reporting BROKEN and the gate would then refuse to report at all. What is wrong is the message — it would blame the prober rather than the driver. Revisit only if a non-overlay storage driver ever ships | CC | | **R-163** | ~~**`mp1` is RETENTION, not staging — and it is sized as if it were neither.**~~ | **CLOSED by R-165 — the ceiling it describes no longer exists** (golden v3.0.0, 2026-08-03) | — (the sizing question is answered; the work is **R-165**) | **Closed, not merely re-framed.** This row was the record of a constraint that was to stay open *"until the merge lands"*. It has landed: the golden ships ONE data volume, so there is no separate 20 G area for a driveless app's recovery unit to outgrow, and the free space an app can use is the box's actual free space. **What replaced the constraint is recorded on R-165**: the bulkhead the partition also provided is now B2's explicit capture floor (controller v0.192.0), and the measured 2× DB-app unit size this row documented is what justifies the floor's reserve being a reserve rather than a working budget. **Caveat carried forward, deliberately:** no box has been reinstalled from the merged golden yet (R-178), so every box in the field still has the split layout and this row's consequences remain live ON THOSE BOXES until they are reinstalled. Original finding unchanged below | CC | | **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC | -| **R-165** | ~~**Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.**~~ | **SHIPPED — golden `build-golden.sh` v3.0.0 + agent v0.120.0 + controller v0.192.0 (B2), 2026-08-03. NOT YET PROVEN-LIVE: no box has been reinstalled from the merged golden** | — | **Variant V-c chosen by the operator on MEASURED evidence, not by reading** (`audits/SPIKE-r165-phase0-2026-08-03.md`): one volume at the NEUTRAL path `/var/lib/felhom`, with `/var/lib/docker` and `/mnt/sys_drive` both binds of subdirectories. Three shapes were built and rebooted; **all three boot, reboot 3/3, give ONE `df` figure and keep a container's `statfs("/")` on the merged volume — the ordering worry that motivated the probe did not materialise.** They differ only in which documented guarantee they break: volume-at-`/var/lib/docker` puts customer backups INSIDE Docker's data-root (so the ordinary "clear /var/lib/docker" reflex destroys every local unit); volume-at-`/mnt/sys_drive` puts Docker's ENTIRE data-root under `/mnt`, which the controller container mounts wholesale — **measured: it then sees `/mnt/sys_drive/docker`**, falsifying the bootstrap's own scoping claim. V-c breaks neither. **P1 answered R-176(a):** a pre-merge archive (`mp0+mp1`) restore-tests clean with `mount_parity: ok` in 84 s; `mountParity` was not weakened. **P3: the four golden assertions were RETARGETED, never deleted, and each was RUN against a deliberately wrong shape — 8 checks, 8 passed**, including a NEW 2b asserting both paths are ONE filesystem (which catches the S2 shape the spike ranked worse than the split) and a new guard for a leftover `mp1` (the old "was mp1 excluded?" pattern could no longer match — a guard that cannot match has silently stopped guarding). **B2 shipped first, in controller v0.192.0**: a two-term capture floor (97% / 1 GiB) in `fillwatch`'s shape, deliberately beyond its critical band so the customer is always warned before a refusal; it refuses per app and **never deletes**, because nothing here is generational. **Golden 0.192.0 is published (registry HTTP 200, sha `54e2a4c4…`) but DELIBERATELY NOT VOUCHED** — vouching is what makes fresh installs pick it up, and the right order is prove-then-vouch. **Remaining: reinstall both demo boxes from it, prove end to end, then vouch → the work is R-178** | CC | +| **R-165** | ~~**Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.**~~ | **SHIPPED — golden `build-golden.sh` v3.0.0 + agent v0.120.0 + controller v0.192.0 (B2), 2026-08-03. IMPLEMENTED — the LAYOUT is proven live on both boxes (R-178, 2026-08-03); the BULKHEAD'S REPLACEMENT IS NOT (→ R-181)** | — | **Variant V-c chosen by the operator on MEASURED evidence, not by reading** (`audits/SPIKE-r165-phase0-2026-08-03.md`): one volume at the NEUTRAL path `/var/lib/felhom`, with `/var/lib/docker` and `/mnt/sys_drive` both binds of subdirectories. Three shapes were built and rebooted; **all three boot, reboot 3/3, give ONE `df` figure and keep a container's `statfs("/")` on the merged volume — the ordering worry that motivated the probe did not materialise.** They differ only in which documented guarantee they break: volume-at-`/var/lib/docker` puts customer backups INSIDE Docker's data-root (so the ordinary "clear /var/lib/docker" reflex destroys every local unit); volume-at-`/mnt/sys_drive` puts Docker's ENTIRE data-root under `/mnt`, which the controller container mounts wholesale — **measured: it then sees `/mnt/sys_drive/docker`**, falsifying the bootstrap's own scoping claim. V-c breaks neither. **P1 answered R-176(a):** a pre-merge archive (`mp0+mp1`) restore-tests clean with `mount_parity: ok` in 84 s; `mountParity` was not weakened. **P3: the four golden assertions were RETARGETED, never deleted, and each was RUN against a deliberately wrong shape — 8 checks, 8 passed**, including a NEW 2b asserting both paths are ONE filesystem (which catches the S2 shape the spike ranked worse than the split) and a new guard for a leftover `mp1` (the old "was mp1 excluded?" pattern could no longer match — a guard that cannot match has silently stopped guarding). **B2 shipped first, in controller v0.192.0**: a two-term capture floor (97% / 1 GiB) in `fillwatch`'s shape, deliberately beyond its critical band so the customer is always warned before a refusal; it refuses per app and **never deletes**, because nothing here is generational. **Golden 0.192.0 is published (registry HTTP 200, sha `54e2a4c4…`) but DELIBERATELY NOT VOUCHED** — vouching is what makes fresh installs pick it up, and the right order is prove-then-vouch. **Remaining: reinstall both demo boxes from it, prove end to end, then vouch → the work is R-178**. **STATUS SETTLED 2026-08-03, operator ruling: IMPLEMENTED, not PROVEN-LIVE, and the reason is the substantive part.** R-178 proved the *layout* on both boxes past any doubt — one volume, no `mp1`, both binds real mounts, one `df` figure, 3/3 reboots each, claim→deploy→backup→restore, and the ceiling's removal measured at 65 GiB / 233 GiB against the old 19 GiB / 45 GiB slices. **But B2, which this row records as the bulkhead's deliberate replacement, does not guard the leg that fills the volume** — proven live on demo-hp at 06:40:03 and filed as **R-181**: the floor is consulted ONLY in `captureAllRecoveryUnits` (`recovery_unit.go:328`), while `runVolumeDumps` (`backup.go:535`) writes the bulk with no floor check at all, and its refusal message's claim *"the previous unit is untouched"* was measured FALSE. **This row's own framing is what makes that gate the status:** it says the partition's bulkhead "is now B2's explicit capture floor". Until R-181 closes, the merge has removed a bulkhead and its stated replacement covers the cheap leg only — and post-merge the unguarded leg can reach Docker's data-root, which pre-merge it could not (it could only fill the dedicated 20 G `mp1`). PROVEN-LIVE when R-181 closes and a fill is re-run | CC | | **R-166** | ~~**App state gets a desired/observed model with its own store.**~~ Operator decision **D-b**, 2026-08-02 (`CONTEXT.md` S-5) | **SHIPPED + PROVEN-LIVE** (controller v0.189.0, 2026-08-02) | — | **Both blocking facts were established at source before any code was written, and the answers changed the shape.** **(a) Does a crash-safe journal already exist for the in-flight case?** YES, twice — `internal/quiesce/quiesce.go` (marker + `Recover`, proven on live hardware by Campaign 8 fault 10) and `internal/stacks/migrate.go` (`migration.json` + `RecoverMigration`) — but **neither covers the app-data path**: `DumpAppVolumesSafe` stopped and restarted an app with **no marker, no journal and not even a `defer`**. So the pattern existed and the coverage did not; `backup.AppStopGuard` copies the proven shape into its **own** file (one file, one writer). **(b) Is the SQLite store reachable?** Irrelevant, and deliberately unused: `metrics.db` is optional by design (the controller runs with it absent), and operational state must not live in a store designed to be droppable. **Shipped:** tri-state `desired_state` in `app.yaml` written ONLY by the customer's action (API action switch, `DeployStack`, `UpdateOptionalConfig`'s redeploy branch, `.fab` import — a 14-caller census established that `StartStack`/`StopStack` must NOT be writers); `isBootOrphan` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN, never running**, so a legacy `app.yaml` keeps byte-identical pre-v0.189.0 behaviour; running-only backfill. D-b's every-container requirement was already met by `aggregateState` and was NOT re-implemented. **Live on 9201:** all three flows (stop survives a restart; a zero-container `running` app is recovered by name; a legacy app.yaml is skipped and never inferred as stopped). **Also fixed en route:** `SaveAppConfig` rebuilt `AppConfig` field-by-field (the R-100 shape) and would have dropped the new field on every save across nine call sites | — | | **R-167** | ~~**Storage monitoring and backup alerts.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0/.1/.2 + hub v0.89.0, 2026-08-02) | — | **Operator decision D-c. It shipped BEFORE the R-165 merge, not with it** — D-a's condition (2) says the monitoring lands in the same step and never after, and landing it first is strictly better and costs nothing. **Customer half:** new `internal/fillwatch`, per FILESYSTEM (never per app — one full disk holding ten apps would fire ten times). **It emits the PRE-EXISTING `disk_warning`/`disk_critical` pair, which was allowlisted, copy'd, in `DefaultEnabledEvents` and checkbox'd with NO PRODUCER IN ANY REPO** — a complete customer pipeline with no producer, the **sixth** *built-but-never-wired* instance here; minting a new near-duplicate type would have left it inert forever. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range — **proven live: the critical crossing fired on the FREE-BYTE term (1.7 GB) at only 91% used.** Edge-triggered on escalation, state persisted, hysteresis dead zone at 75% / 7 GiB pinned by a test; a nil usage read never warns and never clears one (§8.4). The hub's two generic `customerMessages` entries were **removed** — `FormatCustomerEmail` prefers the entry over the message, so keeping them would discard the drive label and the byte figures. **Operator half: see R-158.** **Live on 9201, all three flows:** `disk_warning` then `disk_critical` both `customer | sent` with the Hungarian rendered, exactly two events across three boots (the edge trigger held on the one between), then a silent clear that re-armed. **v0.191.1** added the once-at-startup run (Daily/Every both wait for their first tick, so a box BOOTING over the line would have stayed silent up to 24 h — the R-100 shape); **v0.191.2** added a per-run positive observable, earned when a quiet run during this session's own validation proved unreadable as evidence. Follow-ups: **R-177** (no run-now path) | — | | **R-168** | ~~CI: no runner exists, and with trunk-based pushes CI can DETECT but not BLOCK~~ | **SHIPPED — and the alarm is DEMONSTRATED** (2026-08-02) | — | **Runner live**: `homelab-manifests/gitea-system/act-runner.yaml`, an unprivileged host-mode `act_runner` in `gitea-system`, one owner-scoped registration serving all four repos (measured: tasks 7-10 all claimed by `felhom-gates-runner`). `.gitea/workflows/gates.yml` in each repo runs that repo's entry point with `--fast` and nothing else; no `uses:` step anywhere. **Six probes, all answered, none STOPped** — `audits/SPIKE-ci-runner-2026-08-02.md`. The two that changed the design: **P2** (stock image has git but NO python3 → custom image `felhom-act-runner:0.1.0`, base pinned, python3 and nothing else) and **P6** (a runner that loses `/data/.runner` re-registers and leaves a dead record behind → the PVC is load-bearing, measured both ways). **P5 is the one that mattered**: a failed run produced NO mail, NO notification row and NO log line from Gitea, so the run now sends its own alarm via Resend and prints the provider's accepted id. **Proven end to end, not asserted**: a deliberately broken commit pushed with `--no-verify` → run #6 `failure` → `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`. Posture shown from the live pod spec: `privileged: false`, all caps dropped, no docker socket, no hostPath, `automountServiceAccountToken: false`, sized at half Gitea's limits so it cannot crowd out the service holding every repository on the same node. **The standing limit stays true and is written into the manifest and every workflow: it DETECTS, it does not BLOCK** — making it block is → R-169 | — | diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index b616387..059e92f 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,32 @@ +## docs — v1.22.0 exercised end to end on two real reinstalls (2026-08-03, R-178) — **no script change** + +**Nothing shipped.** `felhom-host-install.sh` stayed at **v1.22.0**; the published copy at +`https://felhom.eu/scripts/felhom-host-install.sh` was confirmed byte-identical to the repo copy +(`sha256 ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4`) before use. Both demo +boxes were uninstalled and reinstalled with it, by **two deliberately different supply paths**: +demo-hp with `--golden ` (the `:2584` alternative), demo-felhom with +`--force-gitea-golden` (the canonical C.3 customer command). The merge-aware `step_grows` produced +`data +46G (->70G, ONE volume)` and `+226G (->250G)` respectively, and `fetch_verify` was observed +succeeding against the vouched manifest for **both** artifacts on demo-felhom +(`verified sha256 a7763d31b55b5ce7…` agent, `verified sha256 54e2a4c431daf580…` golden). + +**Two script-side findings, filed not fixed** (the session was a runbook; §7 forbade code): + +- **R-180** — `--archive-storage` is validated for existence (`:1583`) and for golden resolution + (`:1661`), but never against the ACL storage set it is about to grant (the fixed default + `local local-lvm felhom-pbs`). Staging the golden on `felhom-backup` therefore passed every + pre-flight gate and died at **step 8/8**: `HTTP 403: permission denied at /storage/felhom-backup + (missing privilege Datastore.AllocateSpace)` — *after* step 2 minted the token, step 4b **rotated + and vaulted root@pam**, and step 5 installed the agent. `ARCHIVE_STORAGE ∈ PVE_STORAGES` is a + one-line assertion over two variables both known at `:1583`. +- **R-179** — `--uninstall` leaves the NAS network-storage systemd units behind + (`mnt-felhom\x2ddrives-.{mount,automount}`; automount left `failed`, parent bind left + mounted). The Part E residue-diff provenance is from **v1.9.1**, which predates the feature — and + demo-felhom, which never had a share configured, left nothing, which is exactly why a diff on such + a box reported clean. + +Full evidence: root `REPORT.md`. + ## host-install: one data volume, derived from the disk (2026-08-03, R-165) **Forced by a census, not planned.** `felhom-agent` v0.120.0 merges the appliance's two data volumes