# REPORT — R-178: both demo boxes reinstalled from the merged golden and proven (2026-08-03) **Overwritten** per the standing rule. The prior contents (R-165, the bake, 2026-08-03) have their durable record in `documentation/backlog/OPEN-ITEMS.md` R-165 and the per-repo CHANGELOGs. **Runbook, not a task.** No repo got a version bump and nothing was built. One artifact was **published** (agent 0.120.0) on an explicit operator ruling — see §2. All code findings are filed as register rows, not commits, per the runbook's §7. --- ## 1. Preconditions P1–P6, each as measured | # | Precondition | Measurement | |---|---|---| | **P1** | Golden sha256 matches R-178's record | **PASS** — computed from the artifact itself on demo-hp: `sha256sum` → `54e2a4c431daf580d2807b82d810be36a7c6a9f697b27094dc2912d26a43b3e0`, 649,547,835 B. Matches R-178's recorded prefix and the hub manifest's stored value exactly. Re-verified after copying to `local`: identical | | **P2** | A rollback golden still exists | **PASS, three depths.** Split-layout golden **0.188.0** fetchable from Gitea (`HTTP 200`, 649,310,288 B). A **local** split-layout golden sat on each box — `local:backup/vzdump-lxc-9100-2026_07_21-18_24_49.tar.zst` (demo-hp), `…2026_07_20-17_50_57…` (demo-felhom). Best: **full guest vzdumps**, three per box, newest `vzdump-lxc-9201-2026_08_03-04_54_28.tar.zst` (2,351,557,870 B, demo-hp) and `…2026_08_03-04_44_50…` (6,323,506,165 B, demo-felhom), both ~3 h old — so each box could be put back exactly as it was | | **P3** | Neither demo box holds anything wanted | **PASS, stated explicitly.** demo-hp: paperless-ngx (webserver/postgres/redis) + filebrowser + samba shares. demo-felhom: immich (×4), docmost (×3), calibre-web, bookstack (×2) + filebrowser + samba. Both are demo customers (`Demo HP`, `Demo Ügyfél`); no real customer data. **All of it was destroyed by the wipes and none of it was restored** — that was the point, and the operator confirmed each wipe | | **P4** | The colleague's box untouched | **PASS** — `peti-felhom` is Tier 2 and out of scope. No command was sent to it; it appears in the hub customer list as `DOWN`, exactly as before | | **P5** | Both hosts' agent is v0.120.0 | **PASS** — `felhom-agent 0.120.0` on both, before and after | | **P6** | The hub's host register, BEFORE | **Captured**: `demo-felhom-8363b5 / Demo Ügyfél / 0.120.0 / ONLINE / 1-of-2 guests`; `demo-hp-bb76ea / Demo HP / 0.120.0 / ONLINE / 1-of-2`; `drill-r50-0a4f9a / drill-r50 / 0.113.0 / DOWN` (the fixture, untouched throughout) | --- ## 2. Two findings that contradicted the runbook's premise, both surfaced before any wipe **(a) The golden was ALREADY vouched.** Hub log, a positive observable: `2026/08/03 07:23:26 [INFO] Artifact manifest set: agent=0.119.0 golden=0.192.0 min_agent="0.113.0" wrapper_sha=true` — roughly ten minutes before this session's first hub read, and this session had POSTed nothing. The configuration page confirmed `Artifacts.GoldenVersion=0.192.0`, `GoldenSHA256=54e2a4c4…3b3e0`. So §7's prove-then-vouch order was already spent, and Phase A's stated safety ("nothing is official yet, so a failure reaches nobody") was void. **Operator ruling: accept it and proceed**, keeping the two different supply paths. Recorded on `CONTEXT.md` S-14, now marked SPENT with the reason — the rule lived only in prose and nothing enforced it. **(b) Agent v0.120.0 had never been published — the vouched agent was 0.119.0.** `GET …/generic/felhom-agent/0.120.0/felhom-agent` → **HTTP 404** (0.119.0 → 200). Installer step 5's idempotent skip requires `installed == vouched` exactly, so a documented-path reinstall would have **downgraded both boxes** to the pre-merge 0.119.0 — and would have *succeeded* while doing it, since the current `step_grows` sets `SYSDATA_GROW=0` and 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on error) therefore never fires. The session would have proven a stack nobody ships. **Operator ruling: publish and vouch first.** `scripts/publish-agent.sh 0.120.0` from a clean tree at `4bb84fc3` (`git status --porcelain` empty, `HEAD == origin/main`): upload HTTP 201, **round-trip GET verified**, `AGENT_SHA256=a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`. Vouched via `POST /configuration/artifacts` → `303 …flash=artifacts_set`, hub log `Artifact manifest set: agent=0.120.0 golden=0.192.0`; the hub resolved the sha authoritatively from Gitea rather than trusting the submitted value. Filed as the **third instance of R-115**, not a new ID — R-115 is that row, and it has been WAITING-ON-OPERATOR since 2026-07-29. --- ## 3. demo-hp — the layout proof **Install path.** Uninstall: `./felhom-host-install.sh --uninstall --vmid 9201` (typed-vmid confirmation supplied over a pty). Install: ``` ./felhom-host-install.sh --customer-id demo-hp --mode appliance --vmid 9201 \ --cores 7 --memory 26906 \ --golden local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst \ --passphrase-file /root/.pp-demo-hp ``` Script fetched from `https://felhom.eu/scripts/felhom-host-install.sh`, **v1.22.0**, sha256 `ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4` — byte-identical to the repo copy that was read. The passphrase went file→file into a 0600 file and never onto a command line. A `--dry-run` preceded the real run and resolved `manifest: agent v0.120.0 (sha a7763d31…), golden v0.192.0` with `grows: rootfs +0G (->32G), data +46G (->70G, ONE volume)`. **One deviation, mine, and it cost a restart.** The first attempt staged the golden on `felhom-backup` with `--archive-storage felhom-backup`. Pre-flight passed; **step 8/8** failed: `HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)` — the default pool-scoped ACL grants `local local-lvm felhom-pbs` only. Fixed by copying the golden to `local` (sha re-verified after the copy) and `--resume`. Filed as **R-180**: the condition is statically checkable in pre-flight, and the failure lands *after* step 4b has rotated and vaulted root@pam. **Layout evidence.** ``` mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G # and NO mp1 line rootfs: local-lvm:vm-9201-disk-0,size=32G /var/lib/felhom /dev/mapper/pve-vm--9201--disk--1 ext4 rw,relatime,stripe=16 /var/lib/docker /dev/mapper/pve-vm--9201--disk--1[/docker] ext4 /mnt/sys_drive /dev/mapper/pve-vm--9201--disk--1[/sys_drive] ext4 Filesystem Size Used Avail Use% Mounted on /dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/felhom /dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /var/lib/docker /dev/mapper/pve-vm--9201--disk--1 69G 977M 65G 2% /mnt/sys_drive stat -c %d → 64519 for all three paths /etc/fstab: /var/lib/felhom/docker → /var/lib/docker ; /var/lib/felhom/sys_drive → /mnt/sys_drive both binds writable (touch succeeded on each) ``` `/mnt/sys_drive` shows a second `findmnt` row — it is the controller container's `-v /mnt:/mnt:rslave` propagation (peer group 383 vs the fstab bind's 333), the same shape the split layout had, not a stacked bind. **Reboots — each individually, minimum three:** | # | started | controller healthy | outcome | |---|---|---|---| | 1 | 08:13:33 | 08:13:55, `Up 12 seconds (healthy)` | one filesystem, 69G/65G on both paths | | 2 | 08:13:59 | 08:14:15, `Up 7 seconds (healthy)` | same | | 3 | 08:14:19 | 08:14:35, `Up 7 seconds (healthy)` | same | After all three, `mountpoint -q` returns true for **all three paths** and both binds still resolve to the single volume's subdirectories — which is what the reboots exist to test. `uptime -s` = `2026-08-03 06:14:24 UTC`, matching reboot 3, so these were real reboots. **The journey — method stated: endpoint-level, not a browser.** `claude-in-chrome` does not exist on DooPlex; every step below invoked the exact endpoint the dashboard's own JavaScript calls. | Leg | Endpoint | Observable | |---|---|---| | Claim | `POST /claim` with the pre-auth HMAC CSRF token (64 chars) **and** its `felhom_claim_csrf` cookie | `302 → /`; gate discriminator flipped `{"error":"dashboard not yet claimed"}` → `{"error":"authentication required"}`. **The code is emailed-only (R-119) — the operator supplied it**, after an operator resend rotated the generation (the previous code had been consumed 12 d earlier) | | Deploy | `POST /api/stacks//deploy` with `{"values":{…}}`, field values scraped from the deploy form exactly as the form's own `fetch` does | `{"ok":true}`; `privatebin Up (healthy)`, then `opengist Up (healthy)` | | Back up | `POST /api/debug/backup/dbdump` — this runs the **production** `RunDBDumps` path (DB leg → `runVolumeDumps` → `captureAllRecoveryUnits`); the debug route only starts it instead of waiting for 03:30 | `Volume dump: opengist/… → 178.0 KB`, `privatebin/… → 2.5 KB`, `App-data backup completed … 2 volume dump(s) (3.962s)`, `Recovery unit captured for …` ×2 | | **Restore** | `POST /backup/restore` `stack_name=privatebin snapshot_id=primary` | A marker planted in the live volume was **deleted**, then restored: `{"ok":true,"message":"privatebin visszaállítva (primary)."}` in **9.2 s**, and the marker returned with an **identical sha256 `ac1faae6134871d640d5cb1bbc6b5092d1e96b7e46c6c3bcb6e8236cc33ae861`**. App `Up (healthy)` afterwards | **Recovery unit path, and it lands on the single volume:** `/mnt/sys_drive/felhom-data/backups/primary//{manifest.json,compose/,volume-dumps/}`, whose `df` is `/dev/mapper/pve-vm--9201--disk--1` — the merged volume. **Note on the first app chosen.** `privatebin`'s catalog entry declares no `backup:` section, so its first per-stack capture produced `"volume_dumps": null` — correct for that declaration, not a defect, but it means a per-stack `POST /stacks//backup` writes compose+config only; the volume leg lives in the full pass. A second app (`opengist`) was deployed so the run had both a capture and, later, a refusal. **The ceiling is gone, measured:** a recovery unit can use **65 GiB** — the whole volume — against the **19 GiB** the pre-wipe `mp1` slice offered (`/dev/mapper/pve-vm--9201--disk--2 20G 95M 19G 1%`). --- ## 4. The floor's first live firing — and it does not do what it says **Instrument, proven before use.** demo-hp's thin pool is **53.93 GiB** and the auto-sized volume is 70 G, so a real fill to 97 % would have exhausted the pool and corrupted every guest on the box, including the protected `drill-r50` fixture. A 5 GiB `fallocate` probe moved guest `df` from `977M used / 65G avail` to `6.0G / 60G` while thin-pool `data_percent` stayed **29.03 → 29.03** — zero blocks allocated — and cleanup returned both to baseline. The floor reads `statfs`, which is exactly what `fallocate` moves, so the condition it guards is genuinely present. **Setup.** A **real** 2 GiB file in opengist's data volume (so its capture writes 2 GiB), then `fallocate` to leave `69G / 63G used / 3.0G avail / 96%` — **both floor terms deliberately still clear**, so the run would start. Capture order was established empirically from the previous run's log (opengist first, privatebin second), not assumed. **What happened, 06:40:03:** ``` Volume dump: opengist/opengist_opengist_data → 2.0 GB # unguarded Volume dump: privatebin/privatebin_privatebin_data → 2.5 KB App-data backup completed … 2 volume dump(s) (20.737s) [WARN] Recovery unit capture REFUSED for opengist — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted [WARN] Recovery unit capture REFUSED for privatebin — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted [INFO] Event pushed: recovery_unit_capture_failed (error) — … ×2 (HTTP 200) ``` **What holds:** it refuses **per app** rather than aborting the run; **nothing was deleted** (both units present afterwards); and the operator alert reached the hub — `recovery_unit_capture_failed`, severity `error`, accepted `HTTP 200`, twice. **What does not hold — measured, not inferred.** The refusal message claims *"the previous unit is untouched"*: | unit file | before | after | |---|---|---| | `privatebin/volume-dumps/privatebin_privatebin_data.tar` | `26c546c2…` | **`b538ab89…`** | | `opengist/volume-dumps/opengist_opengist_data.tar` | 182,272 B | **2,147,666,432 B** | Both were rewritten by the earlier leg, while each `manifest.json` kept `created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so a unit's payload can be swapped under a stale descriptor and nothing inside the unit can detect it. **Cause, in the code and not the log.** The floor is consulted in exactly one place — `m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which writes a manifest and a compose copy: a few KB. `runVolumeDumps` (`backup.go:535`) — the leg that writes the bulk, and the leg that consumed the reserve — has **no floor check at all**; its gates are protected-stack, volume-less, disconnected, decommissioned. And it runs first *by design* (`backup.go:483`). **The floor guards the cheap leg and not the leg that fills the volume.** This is why R-165 is IMPLEMENTED and not PROVEN-LIVE: that row records B2 as the deliberate replacement for the bulkhead the `mp1` partition provided, and pre-merge the unguarded leg could only fill a dedicated 20 G partition — post-merge it can reach Docker's data-root. Filed as **R-181**. **No code was written**, per the runbook's §7. **Cleanup:** fill file and payload removed; `df` back to `1.2G used / 65G avail`; a clean re-run left both units valid (`Volume dump … 178.0 KB` / `2.5 KB`, `App-data backup completed … (2.399s)`). --- ## 5. demo-felhom — the pipeline proof **Install path — deliberately different, and this is the reason the second box exists.** ``` ./felhom-host-install.sh --customer-id demo-felhom --mode appliance --vmid 9201 \ --cores 3 --memory 12288 \ --force-gitea-golden \ --passphrase-file /root/.pp-demo-felhom ``` A copy of golden 0.192.0 already sat on this box's `local` storage — it is where the golden was **baked** at 06:58 (a `.log` beside it), sha `54e2a4c4…`. `--force-gitea-golden` is the documented C.3 customer path and overrides local discovery in **both** pre-flight and step 7, which pre-flight confirmed: `golden: none local — will fetch + verify from Gitea in step 7/8`. The bake artifact was left untouched. **`fetch_verify` succeeding against the vouched sha — the observable this box exists to produce:** ``` 5/8 fetching agent binary v0.120.0 from Gitea … verified sha256 a7763d31b55b5ce7… matches the hub manifest 7/8 fetching golden v0.192.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst verified sha256 54e2a4c431daf580… matches the hub manifest golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst Day-0 provision SUCCESS — vmid=9201 host_id=demo-felhom-8363b5 customer=demo-felhom ``` Both artifacts — agent and golden — were fetched anonymously (the normal customer shape) and sha-verified against the manifest. Controller `0.192.0` healthy. **Layout:** `mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=250G`; `pct config 9201 | grep -c '^mp1:'` → **0**. Both binds real mounts (`…disk--1[/docker]`, `…disk--1[/sys_drive]`), `df` one figure — `246G 977M 233G 1%` on all three paths — `stat -c %d` = `64519` on all three, fstab carries both binds. **Reboots:** 1 — 09:18:55 → healthy 09:19:09; 2 — 09:19:12 → 09:19:26; 3 — 09:19:30 → 09:19:45. All three paths still mountpoints after each; `uptime -s` = `2026-08-03 07:19:34 UTC`, matching reboot 3. **Journey:** claim (`302 → /`, discriminator flipped to `authentication required`; code supplied by the operator after a resend) → deploy `opengist` (`{"ok":true}`, `Up (healthy)`) → capture (`Volume dump: opengist/… → 178.0 KB`, `Recovery unit captured`, unit on `/dev/mapper/pve-vm--9201--disk--1`, marker present inside the tar) → **restore** (`{"ok":true,"message":"opengist visszaállítva (primary)."}` in **9.4 s**, marker back with identical sha256 `bc5507987f3f56dc19a9c24785826c304f1a986998d4aeb0b06d1a810b59e939`, app healthy). **Ceiling:** 233 GiB available to a recovery unit, against the 45 GiB the pre-wipe `mp1` offered. **Step 8 was NOT repeated here, deliberately** — stating it rather than leaving it ambiguous. The floor fired on demo-hp and R-181 characterises it fully; re-firing would add no information and would mean filling a 246 G volume. --- ## 6. Vouching Already done before the session (§2a), at **07:23:26 CEST on 2026-08-03**, by an operator action in the hub UI. The session's own manifest write was the **agent** half, at **07:44:31**: `Artifact manifest set: agent=0.120.0 golden=0.192.0 min_agent="0.113.0" wrapper_sha=true`. The manifest afterwards, read back: `agent_sha256=a7763d31b55b5ce7…10b9d`, `golden_sha256=54e2a4c431daf580…43b3e0`, `min_agent=0.113.0`, `wrapper_sha256=104db0a4…` (preserved verbatim). The R-120 gate did not block: golden 0.192.0 equals the newest controller the fleet reports. --- ## 7. Teardown — all three layers 1. **The machine.** No throwaway guest was created this session, so there is none to delete. VM 300 `drill-r50` on demo-hp — the protected drift fixture — was never touched and is still `stopped`; it is not in the `felhom` pool (`pvesh get /pools/felhom` listed only `lxc/9201`), so the uninstall's shared-box logic never reached it. 2. **The host.** `local-lvm`, before → after: **demo-felhom 29.31 % → 1.37 %** (the old 200 G + 50 G volumes returned; the new 250 G volume is thin and barely allocated). **demo-hp 39.13 % → 36.75 %** — *higher than a clean reinstall would leave it*, because the floor test's 2 GB tar blocks cannot be reclaimed: `fstrim` inside an unprivileged LXC returns `FITRIM ioctl failed: Operation not permitted`. No operational impact (the guest shows 65 G free of 69 G, the host 35.7 GiB free of 53.93), but it is real residue and is stated rather than rounded away. Each box's data volume is now the only data volume; no old guest volumes remain. **Residue found and cleared by hand on demo-hp:** `--uninstall` left the NAS network-storage units `mnt-felhom\x2ddrives-Felhom\x2dShare.{mount,automount}` behind (automount `failed`, parent bind still mounted) → **R-179**. demo-felhom left none, because it had no network share configured. 3. **The hub.** **No old records exist to dispose of, and this is the honest finding, not an omission:** both enrollments were **idempotent** — `host REUSED (idempotent — existing credential)`, `host_id: demo-hp-bb76ea` and `demo-felhom-8363b5`, the same ids as before. The reinstalls therefore produced **no new host records**, so nothing was orphaned and nothing needed deleting. Final register: `demo-felhom-8363b5 ONLINE 0.120.0`, `demo-hp-bb76ea ONLINE 0.120.0`, `drill-r50-0a4f9a DOWN 0.113.0` — the same three rows as at P6. No scratch customers were created. `/appliances` returns 404 on hub 0.89.0 — there is no appliance-record surface to clean. **Secrets:** both retrieval passphrases were moved file→file into 0600 files, used via `--passphrase-file`, and `shred -u`'d afterwards on both hosts along with the session cookie files and helper scripts; the local scratch copies were deleted. Nothing was written to a committed file and `curl -w '%{redirect_url}'` was never used (R-132). --- ## 8. Registers changed | Row | Change | |---|---| | **R-178** | **CLOSED** — both boxes reinstalled and proven, by two different supply paths | | **R-165** | **IMPLEMENTED**, not PROVEN-LIVE — operator ruling; the layout half is proven, the B2 half is not (→ R-181) | | **R-115** | **Third instance recorded** — agent 0.120.0 built, deployed to both hosts, never published | | **R-181** *(new)* | The capture floor guards the recovery-unit leg and not `runVolumeDumps`; its "previous unit is untouched" claim measured false | | **R-180** *(new)* | `--archive-storage` is not cross-checked against the ACL grant; the 403 lands at step 8/8, after root@pam has been rotated | | **R-179** *(new)* | `--uninstall` leaves NAS network-storage systemd units behind when a share was configured | **IDs established free before minting:** `grep -ro "R-179\b\|R-180\b\|R-181\b\|R-182\b"` over `documentation/` and `*.md` → **0 hits**, and over all four repo roots (`felhom-agent`, `felhom-controller`, `felhom.eu`, `app-catalog-felhom.eu`) → **0 hits**. `R-182` was checked and left unused. --- ## 9. CI — checked by run id, not assumed Two commits, both docs-only. | repo | commit | CI run | result | |---|---|---|---| | `felhom-agent` | `9dfd89c` — *docs: agent 0.120.0 published + vouched…* | **run 46**, `head_sha 9dfd89cb` | **success** (`gates`) | | `felhom.eu` | `aa62449` — *R-178 CLOSED: both demo boxes reinstalled…* | **run 47**, `head_sha aa624496` | **success** (`gates`) | Queried with `curl -s "https://gitea.dooplex.hu/api/v1/repos/admin//actions/tasks?limit=3"` and matched on `head_sha`, per `CLAUDE.md`'s pull-check rule — CI mails on failure, which is a push signal; this is the pull check that catches a lost or unread mail. **`--no-verify` was NOT used**, anywhere. This clone is armed (`core.hooksPath = .githooks`). Both gate entry points were also run by hand before committing: `felhom.eu/scripts/repo_gates.py` → **all five OK** (site, hostinstall, hub-confirm, manifest-bearer, reuse-refs), and `felhom-agent/scripts/agent_gates.py` → **OK** (reuse-refs). No test suite was run because no code changed in either repo. --- ## 10. Observations — noticed and NOT acted on - **The runbook's central claim was wrong in a way that mattered.** R-178 said *"a reinstall is now a self-contained piece of work with no code left to write."* True about code; false about the artifact channel — the agent half of the merge was unpublished, and following the documented path without checking would have downgraded both boxes and produced a green, meaningless result. **No code change was needed to make any step pass** — §10 asks this loudly, and the answer is no. What was needed was a publish. - **Prove-then-vouch was already spent when the session opened.** Not a defect in anything, but the rule protected nothing because nothing enforced it. R-120's gate is the shape that would. - **A per-stack `POST /stacks//backup` does not produce volume dumps** — the volume leg lives in the full app-data pass. Not wrong, but the endpoint's name suggests otherwise and it cost time here. - **privatebin's recovery unit carries no user data** (its catalog entry declares no `backup:` section), so restoring it loses every paste. That may be intentional for an expiring, E2E-encrypted paste bin — but nothing in the app's description tells the customer so. Catalog question, not filed. - **V-c doubles systemd's mount-unit count** — every docker overlay appears twice, `var-lib-docker-…-merged.mount` and `var-lib-felhom-docker-…-merged.mount`, because `/var/lib/docker` is a bind of `/var/lib/felhom/docker`. Cosmetic, inherent to the chosen variant, no action. - **`fstrim` cannot run inside the guest** (`EPERM`, unprivileged LXC), so space freed inside the guest is not returned to the thin pool. It did not matter here; on a box that fills and empties repeatedly it would. - **The floor's own message mixes units** — it reports `64.2/68.7 GB used (93%)` against a threshold stated as `97% used or 1.0 GiB free`, while `df` showed 96 %. GB-vs-GiB, so the percentage term fires later than an operator reading `df` would expect. Minor; noted on R-181's fix shape rather than filed separately.