felhom.eu/REPORT.md

REPORT — R-178: both demo boxes reinstalled from the merged golden and proven (2026-08-03)

Overwritten per the standing rule. The prior contents (R-165, the bake, 2026-08-03) have their durable record in documentation/backlog/OPEN-ITEMS.md R-165 and the per-repo CHANGELOGs.

Runbook, not a task. No repo got a version bump and nothing was built. One artifact was published (agent 0.120.0) on an explicit operator ruling — see §2. All code findings are filed as register rows, not commits, per the runbook's §7.

---

1. Preconditions P1–P6, each as measured

#	Precondition	Measurement
P1	Golden sha256 matches R-178's record	PASS — computed from the artifact itself on demo-hp: sha256sum → 54e2a4c431daf580d2807b82d810be36a7c6a9f697b27094dc2912d26a43b3e0, 649,547,835 B. Matches R-178's recorded prefix and the hub manifest's stored value exactly. Re-verified after copying to local: identical
P2	A rollback golden still exists	PASS, three depths. Split-layout golden 0.188.0 fetchable from Gitea (HTTP 200, 649,310,288 B). A local split-layout golden sat on each box — local:backup/vzdump-lxc-9100-2026_07_21-18_24_49.tar.zst (demo-hp), …2026_07_20-17_50_57… (demo-felhom). Best: full guest vzdumps, three per box, newest vzdump-lxc-9201-2026_08_03-04_54_28.tar.zst (2,351,557,870 B, demo-hp) and …2026_08_03-04_44_50… (6,323,506,165 B, demo-felhom), both ~3 h old — so each box could be put back exactly as it was
P3	Neither demo box holds anything wanted	PASS, stated explicitly. demo-hp: paperless-ngx (webserver/postgres/redis) + filebrowser + samba shares. demo-felhom: immich (×4), docmost (×3), calibre-web, bookstack (×2) + filebrowser + samba. Both are demo customers (Demo HP, Demo Ügyfél); no real customer data. All of it was destroyed by the wipes and none of it was restored — that was the point, and the operator confirmed each wipe
P4	The colleague's box untouched	PASS — peti-felhom is Tier 2 and out of scope. No command was sent to it; it appears in the hub customer list as DOWN, exactly as before
P5	Both hosts' agent is v0.120.0	PASS — felhom-agent 0.120.0 on both, before and after
P6	The hub's host register, BEFORE	Captured: demo-felhom-8363b5 / Demo Ügyfél / 0.120.0 / ONLINE / 1-of-2 guests; demo-hp-bb76ea / Demo HP / 0.120.0 / ONLINE / 1-of-2; drill-r50-0a4f9a / drill-r50 / 0.113.0 / DOWN (the fixture, untouched throughout)

---

2. Two findings that contradicted the runbook's premise, both surfaced before any wipe

(a) The golden was ALREADY vouched. Hub log, a positive observable: 2026/08/03 07:23:26 [INFO] Artifact manifest set: agent=0.119.0 golden=0.192.0 min_agent="0.113.0" wrapper_sha=true — roughly ten minutes before this session's first hub read, and this session had POSTed nothing. The configuration page confirmed Artifacts.GoldenVersion=0.192.0, GoldenSHA256=54e2a4c4…3b3e0. So §7's prove-then-vouch order was already spent, and Phase A's stated safety ("nothing is official yet, so a failure reaches nobody") was void. Operator ruling: accept it and proceed, keeping the two different supply paths. Recorded on CONTEXT.md S-14, now marked SPENT with the reason — the rule lived only in prose and nothing enforced it.

(b) Agent v0.120.0 had never been published — the vouched agent was 0.119.0. GET …/generic/felhom-agent/0.120.0/felhom-agent → HTTP 404 (0.119.0 → 200). Installer step 5's idempotent skip requires installed == vouched exactly, so a documented-path reinstall would have downgraded both boxes to the pre-merge 0.119.0 — and would have succeeded while doing it, since the current step_grows sets SYSDATA_GROW=0 and 0.119.0's mp1 resize (bringup.go 4c, fatal on error) therefore never fires. The session would have proven a stack nobody ships. Operator ruling: publish and vouch first. scripts/publish-agent.sh 0.120.0 from a clean tree at 4bb84fc3 (git status --porcelain empty, HEAD == origin/main): upload HTTP 201, round-trip GET verified, AGENT_SHA256=a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d. Vouched via POST /configuration/artifacts → 303 …flash=artifacts_set, hub log Artifact manifest set: agent=0.120.0 golden=0.192.0; the hub resolved the sha authoritatively from Gitea rather than trusting the submitted value. Filed as the third instance of R-115, not a new ID — R-115 is that row, and it has been WAITING-ON-OPERATOR since 2026-07-29.

---

3. demo-hp — the layout proof

Install path. Uninstall: ./felhom-host-install.sh --uninstall --vmid 9201 (typed-vmid confirmation supplied over a pty). Install:

./felhom-host-install.sh --customer-id demo-hp --mode appliance --vmid 9201 \
    --cores 7 --memory 26906 \
    --golden local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst \
    --passphrase-file /root/.pp-demo-hp

Script fetched from https://felhom.eu/scripts/felhom-host-install.sh, v1.22.0, sha256 ed02acb2da46c8d2b5c486ce99d5b9a2747e8786c6eb03652cf755ed1abdd9f4 — byte-identical to the repo copy that was read. The passphrase went file→file into a 0600 file and never onto a command line. A --dry-run preceded the real run and resolved manifest: agent v0.120.0 (sha a7763d31…), golden v0.192.0 with grows: rootfs +0G (->32G), data +46G (->70G, ONE volume).

One deviation, mine, and it cost a restart. The first attempt staged the golden on felhom-backup with --archive-storage felhom-backup. Pre-flight passed; step 8/8 failed: HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace) — the default pool-scoped ACL grants local local-lvm felhom-pbs only. Fixed by copying the golden to local (sha re-verified after the copy) and --resume. Filed as R-180: the condition is statically checkable in pre-flight, and the failure lands after step 4b has rotated and vaulted root@pam.

Layout evidence.

mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G      # and NO mp1 line
rootfs: local-lvm:vm-9201-disk-0,size=32G

/var/lib/felhom  /dev/mapper/pve-vm--9201--disk--1             ext4  rw,relatime,stripe=16
/var/lib/docker  /dev/mapper/pve-vm--9201--disk--1[/docker]    ext4
/mnt/sys_drive   /dev/mapper/pve-vm--9201--disk--1[/sys_drive] ext4

Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/pve-vm--9201--disk--1   69G  977M   65G   2% /var/lib/felhom
/dev/mapper/pve-vm--9201--disk--1   69G  977M   65G   2% /var/lib/docker
/dev/mapper/pve-vm--9201--disk--1   69G  977M   65G   2% /mnt/sys_drive

stat -c %d → 64519 for all three paths
/etc/fstab: /var/lib/felhom/docker → /var/lib/docker ; /var/lib/felhom/sys_drive → /mnt/sys_drive
both binds writable (touch succeeded on each)

/mnt/sys_drive shows a second findmnt row — it is the controller container's -v /mnt:/mnt:rslave propagation (peer group 383 vs the fstab bind's 333), the same shape the split layout had, not a stacked bind.

Reboots — each individually, minimum three:

#	started	controller healthy	outcome
1	08:13:33	08:13:55, Up 12 seconds (healthy)	one filesystem, 69G/65G on both paths
2	08:13:59	08:14:15, Up 7 seconds (healthy)	same
3	08:14:19	08:14:35, Up 7 seconds (healthy)	same

After all three, mountpoint -q returns true for all three paths and both binds still resolve to the single volume's subdirectories — which is what the reboots exist to test. uptime -s = 2026-08-03 06:14:24 UTC, matching reboot 3, so these were real reboots.

The journey — method stated: endpoint-level, not a browser. claude-in-chrome does not exist on DooPlex; every step below invoked the exact endpoint the dashboard's own JavaScript calls.

Leg	Endpoint	Observable
Claim	POST /claim with the pre-auth HMAC CSRF token (64 chars) and its felhom_claim_csrf cookie	302 → /; gate discriminator flipped {"error":"dashboard not yet claimed"} → {"error":"authentication required"}. The code is emailed-only (R-119) — the operator supplied it, after an operator resend rotated the generation (the previous code had been consumed 12 d earlier)
Deploy	POST /api/stacks/<n>/deploy with {"values":{…}}, field values scraped from the deploy form exactly as the form's own fetch does	{"ok":true}; privatebin Up (healthy), then opengist Up (healthy)
Back up	POST /api/debug/backup/dbdump — this runs the production RunDBDumps path (DB leg → runVolumeDumps → captureAllRecoveryUnits); the debug route only starts it instead of waiting for 03:30	Volume dump: opengist/… → 178.0 KB, privatebin/… → 2.5 KB, App-data backup completed … 2 volume dump(s) (3.962s), Recovery unit captured for … ×2
Restore	POST /backup/restore stack_name=privatebin snapshot_id=primary	A marker planted in the live volume was deleted, then restored: {"ok":true,"message":"privatebin visszaállítva (primary)."} in 9.2 s, and the marker returned with an identical sha256 ac1faae6134871d640d5cb1bbc6b5092d1e96b7e46c6c3bcb6e8236cc33ae861. App Up (healthy) afterwards

Recovery unit path, and it lands on the single volume: /mnt/sys_drive/felhom-data/backups/primary/<app>/{manifest.json,compose/,volume-dumps/}, whose df is /dev/mapper/pve-vm--9201--disk--1 — the merged volume.

Note on the first app chosen. privatebin's catalog entry declares no backup: section, so its first per-stack capture produced "volume_dumps": null — correct for that declaration, not a defect, but it means a per-stack POST /stacks/<n>/backup writes compose+config only; the volume leg lives in the full pass. A second app (opengist) was deployed so the run had both a capture and, later, a refusal.

The ceiling is gone, measured: a recovery unit can use 65 GiB — the whole volume — against the 19 GiB the pre-wipe mp1 slice offered (/dev/mapper/pve-vm--9201--disk--2 20G 95M 19G 1%).

---

4. The floor's first live firing — and it does not do what it says

Instrument, proven before use. demo-hp's thin pool is 53.93 GiB and the auto-sized volume is 70 G, so a real fill to 97 % would have exhausted the pool and corrupted every guest on the box, including the protected drill-r50 fixture. A 5 GiB fallocate probe moved guest df from 977M used / 65G avail to 6.0G / 60G while thin-pool data_percent stayed 29.03 → 29.03 — zero blocks allocated — and cleanup returned both to baseline. The floor reads statfs, which is exactly what fallocate moves, so the condition it guards is genuinely present.

Setup. A real 2 GiB file in opengist's data volume (so its capture writes 2 GiB), then fallocate to leave 69G / 63G used / 3.0G avail / 96% — both floor terms deliberately still clear, so the run would start. Capture order was established empirically from the previous run's log (opengist first, privatebin second), not assumed.

What happened, 06:40:03:

Volume dump: opengist/opengist_opengist_data → 2.0 GB        # unguarded
Volume dump: privatebin/privatebin_privatebin_data → 2.5 KB
App-data backup completed … 2 volume dump(s) (20.737s)
[WARN] Recovery unit capture REFUSED for opengist   — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted
[WARN] Recovery unit capture REFUSED for privatebin — … 1.0 GB free; the previous unit is untouched and NOTHING was deleted
[INFO] Event pushed: recovery_unit_capture_failed (error) — … ×2   (HTTP 200)

What holds: it refuses per app rather than aborting the run; nothing was deleted (both units present afterwards); and the operator alert reached the hub — recovery_unit_capture_failed, severity error, accepted HTTP 200, twice.

What does not hold — measured, not inferred. The refusal message claims "the previous unit is untouched":

unit file	before	after
privatebin/volume-dumps/privatebin_privatebin_data.tar	26c546c2…	b538ab89…
opengist/volume-dumps/opengist_opengist_data.tar	182,272 B	2,147,666,432 B

Both were rewritten by the earlier leg, while each manifest.json kept created_at: 2026-08-03T06:34:26Z and its checksums block covers only the three compose files — so a unit's payload can be swapped under a stale descriptor and nothing inside the unit can detect it.

Cause, in the code and not the log. The floor is consulted in exactly one place — m.unitFloorBlocked(stack.Name) at recovery_unit.go:328, inside captureAllRecoveryUnits, which writes a manifest and a compose copy: a few KB. runVolumeDumps (backup.go:535) — the leg that writes the bulk, and the leg that consumed the reserve — has no floor check at all; its gates are protected-stack, volume-less, disconnected, decommissioned. And it runs first by design (backup.go:483). The floor guards the cheap leg and not the leg that fills the volume.

This is why R-165 is IMPLEMENTED and not PROVEN-LIVE: that row records B2 as the deliberate replacement for the bulkhead the mp1 partition provided, and pre-merge the unguarded leg could only fill a dedicated 20 G partition — post-merge it can reach Docker's data-root. Filed as R-181. No code was written, per the runbook's §7.

Cleanup: fill file and payload removed; df back to 1.2G used / 65G avail; a clean re-run left both units valid (Volume dump … 178.0 KB / 2.5 KB, App-data backup completed … (2.399s)).

---

5. demo-felhom — the pipeline proof

Install path — deliberately different, and this is the reason the second box exists.

./felhom-host-install.sh --customer-id demo-felhom --mode appliance --vmid 9201 \
    --cores 3 --memory 12288 \
    --force-gitea-golden \
    --passphrase-file /root/.pp-demo-felhom

A copy of golden 0.192.0 already sat on this box's local storage — it is where the golden was baked at 06:58 (a .log beside it), sha 54e2a4c4…. --force-gitea-golden is the documented C.3 customer path and overrides local discovery in both pre-flight and step 7, which pre-flight confirmed: golden: none local — will fetch + verify from Gitea in step 7/8. The bake artifact was left untouched.

fetch_verify succeeding against the vouched sha — the observable this box exists to produce:

5/8  fetching agent binary v0.120.0 from Gitea …
     verified sha256 a7763d31b55b5ce7… matches the hub manifest
7/8  fetching golden v0.192.0 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst
     verified sha256 54e2a4c431daf580… matches the hub manifest
     golden imported + verified: local:backup/vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst
Day-0 provision SUCCESS — vmid=9201 host_id=demo-felhom-8363b5 customer=demo-felhom

Both artifacts — agent and golden — were fetched anonymously (the normal customer shape) and sha-verified against the manifest. Controller 0.192.0 healthy.

Layout: mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=250G; pct config 9201 | grep -c '^mp1:' → 0. Both binds real mounts (…disk--1[/docker], …disk--1[/sys_drive]), df one figure — 246G 977M 233G 1% on all three paths — stat -c %d = 64519 on all three, fstab carries both binds.

Reboots: 1 — 09:18:55 → healthy 09:19:09; 2 — 09:19:12 → 09:19:26; 3 — 09:19:30 → 09:19:45. All three paths still mountpoints after each; uptime -s = 2026-08-03 07:19:34 UTC, matching reboot 3.

Journey: claim (302 → /, discriminator flipped to authentication required; code supplied by the operator after a resend) → deploy opengist ({"ok":true}, Up (healthy)) → capture (Volume dump: opengist/… → 178.0 KB, Recovery unit captured, unit on /dev/mapper/pve-vm--9201--disk--1, marker present inside the tar) → restore ({"ok":true,"message":"opengist visszaállítva (primary)."} in 9.4 s, marker back with identical sha256 bc5507987f3f56dc19a9c24785826c304f1a986998d4aeb0b06d1a810b59e939, app healthy).

Ceiling: 233 GiB available to a recovery unit, against the 45 GiB the pre-wipe mp1 offered.

Step 8 was NOT repeated here, deliberately — stating it rather than leaving it ambiguous. The floor fired on demo-hp and R-181 characterises it fully; re-firing would add no information and would mean filling a 246 G volume.

---

6. Vouching

Already done before the session (§2a), at 07:23:26 CEST on 2026-08-03, by an operator action in the hub UI. The session's own manifest write was the agent half, at 07:44:31: Artifact manifest set: agent=0.120.0 golden=0.192.0 min_agent="0.113.0" wrapper_sha=true. The manifest afterwards, read back: agent_sha256=a7763d31b55b5ce7…10b9d, golden_sha256=54e2a4c431daf580…43b3e0, min_agent=0.113.0, wrapper_sha256=104db0a4… (preserved verbatim). The R-120 gate did not block: golden 0.192.0 equals the newest controller the fleet reports.

---

7. Teardown — all three layers

1. The machine. No throwaway guest was created this session, so there is none to delete. VM 300 drill-r50 on demo-hp — the protected drift fixture — was never touched and is still stopped; it is not in the felhom pool (pvesh get /pools/felhom listed only lxc/9201), so the uninstall's shared-box logic never reached it.
2. The host. local-lvm, before → after: demo-felhom 29.31 % → 1.37 % (the old 200 G + 50 G volumes returned; the new 250 G volume is thin and barely allocated). demo-hp 39.13 % → 36.75 % — higher than a clean reinstall would leave it, because the floor test's 2 GB tar blocks cannot be reclaimed: fstrim inside an unprivileged LXC returns FITRIM ioctl failed: Operation not permitted. No operational impact (the guest shows 65 G free of 69 G, the host 35.7 GiB free of 53.93), but it is real residue and is stated rather than rounded away. Each box's data volume is now the only data volume; no old guest volumes remain. Residue found and cleared by hand on demo-hp: --uninstall left the NAS network-storage units mnt-felhom\x2ddrives-Felhom\x2dShare.{mount,automount} behind (automount failed, parent bind still mounted) → R-179. demo-felhom left none, because it had no network share configured.
3. The hub. No old records exist to dispose of, and this is the honest finding, not an omission: both enrollments were idempotent — host REUSED (idempotent — existing credential), host_id: demo-hp-bb76ea and demo-felhom-8363b5, the same ids as before. The reinstalls therefore produced no new host records, so nothing was orphaned and nothing needed deleting. Final register: demo-felhom-8363b5 ONLINE 0.120.0, demo-hp-bb76ea ONLINE 0.120.0, drill-r50-0a4f9a DOWN 0.113.0 — the same three rows as at P6. No scratch customers were created. /appliances returns 404 on hub 0.89.0 — there is no appliance-record surface to clean.

Secrets: both retrieval passphrases were moved file→file into 0600 files, used via --passphrase-file, and shred -u'd afterwards on both hosts along with the session cookie files and helper scripts; the local scratch copies were deleted. Nothing was written to a committed file and curl -w '%{redirect_url}' was never used (R-132).

---

8. Registers changed

Row	Change
R-178	CLOSED — both boxes reinstalled and proven, by two different supply paths
R-165	IMPLEMENTED, not PROVEN-LIVE — operator ruling; the layout half is proven, the B2 half is not (→ R-181)
R-115	Third instance recorded — agent 0.120.0 built, deployed to both hosts, never published
R-181 (new)	The capture floor guards the recovery-unit leg and not runVolumeDumps; its "previous unit is untouched" claim measured false
R-180 (new)	--archive-storage is not cross-checked against the ACL grant; the 403 lands at step 8/8, after root@pam has been rotated
R-179 (new)	--uninstall leaves NAS network-storage systemd units behind when a share was configured

IDs established free before minting: grep -ro "R-179\b\|R-180\b\|R-181\b\|R-182\b" over documentation/ and *.md → 0 hits, and over all four repo roots (felhom-agent, felhom-controller, felhom.eu, app-catalog-felhom.eu) → 0 hits. R-182 was checked and left unused.

---

9. CI — checked by run id, not assumed

Two commits, both docs-only.

repo	commit	CI run	result
felhom-agent	9dfd89c — docs: agent 0.120.0 published + vouched…	run 46, head_sha 9dfd89cb	success (gates)
felhom.eu	aa62449 — R-178 CLOSED: both demo boxes reinstalled…	run 47, head_sha aa624496	success (gates)

Queried with curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/<repo>/actions/tasks?limit=3" and matched on head_sha, per CLAUDE.md's pull-check rule — CI mails on failure, which is a push signal; this is the pull check that catches a lost or unread mail.

--no-verify was NOT used, anywhere. This clone is armed (core.hooksPath = .githooks). Both gate entry points were also run by hand before committing: felhom.eu/scripts/repo_gates.py → all five OK (site, hostinstall, hub-confirm, manifest-bearer, reuse-refs), and felhom-agent/scripts/agent_gates.py → OK (reuse-refs). No test suite was run because no code changed in either repo.

---

10. Observations — noticed and NOT acted on

• The runbook's central claim was wrong in a way that mattered. R-178 said "a reinstall is now a self-contained piece of work with no code left to write." True about code; false about the artifact channel — the agent half of the merge was unpublished, and following the documented path without checking would have downgraded both boxes and produced a green, meaningless result. No code change was needed to make any step pass — §10 asks this loudly, and the answer is no. What was needed was a publish.
• Prove-then-vouch was already spent when the session opened. Not a defect in anything, but the rule protected nothing because nothing enforced it. R-120's gate is the shape that would.
• A per-stack POST /stacks/<n>/backup does not produce volume dumps — the volume leg lives in the full app-data pass. Not wrong, but the endpoint's name suggests otherwise and it cost time here.
• privatebin's recovery unit carries no user data (its catalog entry declares no backup: section), so restoring it loses every paste. That may be intentional for an expiring, E2E-encrypted paste bin — but nothing in the app's description tells the customer so. Catalog question, not filed.
• V-c doubles systemd's mount-unit count — every docker overlay appears twice, var-lib-docker-…-merged.mount and var-lib-felhom-docker-…-merged.mount, because /var/lib/docker is a bind of /var/lib/felhom/docker. Cosmetic, inherent to the chosen variant, no action.
• fstrim cannot run inside the guest (EPERM, unprivileged LXC), so space freed inside the guest is not returned to the thin pool. It did not matter here; on a box that fills and empties repeatedly it would.
• The floor's own message mixes units — it reports 64.2/68.7 GB used (93%) against a threshold stated as 97% used or 1.0 GiB free, while df showed 96 %. GB-vs-GiB, so the percentage term fires later than an operator reading df would expect. Minor; noted on R-181's fix shape rather than filed separately.