felhom.eu/documentation/audits/SPIKE-r165-phase0-2026-08-03.md

SPIKE R-165 Phase 0 — the two probes the merge spike left unmeasured

Date: 2026-08-03 · Author: Claude Code · Status: MEASURED — no layout changed

SPIKE-r165-mp1-merge-2026-08-02.md named two things as unmeasured and both are load-bearing. This document measures them. It changes nothing: no golden rebuilt, no box reinstalled, no guest config edited outside the throwaway probe guests, which are destroyed at the end.

Host: felhom-pve (N100) — Tier 0. demo-hp would have been the default per the 2026-07-25 ruling, but target-selection.md records its local-lvm as an over-subscribed thin pool backing live guest 9201 (~144 GiB allocated over ~54 GiB) where filling it corrupts every guest, and it holds no container template. felhom-pve has the template and 258 GiB free at 29% pool usage. Both are Tier 0; this picked the one where a probe cannot damage a live guest. Neither DooPlex, ep0, nor the colleague's box was contacted at any point.

---

P1 — does a PRE-merge archive restore cleanly into the merged world?

Verdict: PASS.

Method

The spike reasoned from mountParity that it should pass and said plainly that this had never been executed. It is now executed, on real hardware, with the real restore-test path — not a hand-assembled restore.

• Archive: local:backup/vzdump-lxc-9201-2026_07_28-17_43_05.tar.zst from demo-hp, 1.68 GB. Confirmed pre-merge by its own vzdump log rather than by assumption:

  including mount point rootfs ('/') in backup
  including mount point mp0 ('/var/lib/docker') in backup
  including mount point mp1 ('/mnt/sys_drive') in backup
  

• Command: felhom-agent --selftest=restore-test -archive <volid> on demo-hp.

Measured result

"pass": true,
"verified": "boot+running",
"mount_parity": "ok",
"duration_seconds": 84.2,
"mount_inventory": [
  "mp0=/var/lib/docker (50G)",
  "mp1=/mnt/sys_drive (20G)",
  "mp8=/mnt/felhom-drives (throwaway for the archived bind)",
  "mp9=/etc/felhom-bootstrap (throwaway for the archived bind)"
]
=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m24s) ===

mountParity was not relaxed, weakened or touched in any way.

The limit of this result, stated rather than glossed

It was run with the CURRENT agent (v0.119.0), because the merged agent does not exist yet — Part 2 sits after this session's STOP. What it proves is that mountParity compares the archive against its own restore, so a pre-merge archive recreates its own mp0 + mp1 in the scratch guest and the two agree. That comparison never consults the host's golden layout, which is why the merge cannot invalidate it — provided Part 2 honours its own constraint not to touch mountParity or the restore path (§5, §12 of the task). It is an 84-second command and should be re-run once the merged agent exists, which is cheap and turns a sound inference into an observation.

---

P2 — which S1 variant actually works on this platform?

Verdict: all three probed variants are mechanically clean. They are separated by SCOPING, not by mechanics — and the deciding fact was not in the task's table.

Method

A throwaway unprivileged LXC per variant (nesting=1,keyctl=1, rootfs 8 G + one 10 G volume, backup=1), Docker installed from the same repo with the same daemon.json the golden bakes (containerd-snapshotter: false, overlay2, log caps). Per variant, measured at first boot and after each of three reboots:

1. both /var/lib/docker and /mnt/sys_drive present and writable (write → read back → delete, a positive observable rather than an ls);
2. one filesystem — same source device and the same free-space figure for both paths;
3. dockerd active and docker run hello-world succeeding;
4. then once: the real bootstrap propagation sequence (mount --rbind /mnt /mnt, mount --make-rshared /mnt, docker run -v /mnt:/mnt:rslave);
5. and: what a container mounting /mnt:rslave actually sees — the scoping check.

No pipe hides a non-zero; every check reports its own rc and the script aborts with MEASURED-FAIL.

The variants

	volume mounted at	then
V-a	/var/lib/docker	/mnt/sys_drive = bind of /var/lib/docker/sys_drive
V-b	/mnt/sys_drive	/var/lib/docker = bind of /mnt/sys_drive/docker
V-c	/var/lib/felhom (neutral)	both consumer paths are binds of subdirectories

V-c was not in the task's table. It was probed because the measurements below showed V-a and V-b each violate a different documented invariant, and V-c is the shape that violates neither. It is offered as a measured option for the operator at the STOP, not adopted — the task is explicit that a variant is not chosen mid-session.

Measured results

check	V-a	V-b	V-c
both paths present + writable	yes	yes	yes
ONE filesystem, ONE free-space figure	yes	yes	yes
dockerd active + docker run — initial	yes	yes	yes
dockerd active + docker run — reboots 1/2/3	3/3	3/3	3/3
/mnt propagation shared; container sees /mnt	yes	yes	yes
both paths still real mountpoints (findmnt non-empty — the form the golden's assertions use)	yes	yes	yes
container statfs("/") reports the merged volume	yes (10218772 KiB)	yes	yes
what a container mounting /mnt:rslave SEES	sys_drive only — 8.0K	sys_drive + sys_drive/docker — 17.9M	sys_drive only — 8.0K
customer data inside Docker's data-root	YES	no	no

The mechanical worry was misplaced. The task flagged V-b's ordering risk — /var/lib/docker must be bound before dockerd starts. An /etc/fstab bind is ordered by local-fs.target, which precedes basic.target and therefore docker.service, and it held 3 reboots out of 3. Ordering is not what separates these variants.

What actually separates them

V-b breaks a documented scoping invariant, and the measurement is the proof. The controller container is started with -v /mnt:/mnt:rslave, and the bootstrap script's own comment states the scope it relies on:

"scoped to /mnt, which (Model A) holds only Felhom's felhom-data-namespace mounts, never the customer's other on-drive data"

Under V-b the container sees /mnt/sys_drive/docker — Docker's entire data-root, 17.9 MB on an empty probe box and growing with every image and every app volume. That sentence becomes false. The controller already holds the Docker socket, so this is not a capability escalation — but it puts Docker's internal tree inside the one path the controller's own scanners, the FileBrowser surface and the data-migration engine (which works "in-process over the controller's /mnt:/mnt:rslave RW mount") treat as Felhom-only.

V-a breaks the other one: customer backups live inside Docker's data-root, so du on the data-root stops meaning what it says, and the ordinary operator reflex for a sick Docker — clear /var/lib/docker — destroys every local recovery unit on the box.

V-c breaks neither, at the cost of one new mount path and two fstab lines instead of one.

---

P3 — the golden's four assertions

Not yet run — it belongs to Part 1, which is after the STOP. Recorded here so it is not lost: build-golden.sh fails closed on the split in four places (:126, :130 separate-mount asserts; :315, :319 vzdump-exclusion guards), and each retargeted guard must be shown to abort against a deliberately wrong shape before the golden is trusted.

One measurement already de-risks it: under all three variants both /var/lib/docker and /mnt/sys_drive remain real mountpoints, so findmnt -no SOURCE,FSTYPE <path> | grep -q . — the exact form the two existing assertions use — still returns non-empty. The assertions can be retargeted with a changed message and an added guard for the single volume, rather than rewritten from scratch. They must not be deleted.

---

Teardown

layer	action	evidence
the machine	probe guests 9401, 9402, 9403 destroyed; P1's scratch 990000 was torn down by the restore-test itself	pct list shows only 9201
the host	local-lvm 112398205 → 107204406 KiB used (30.73% → 29.31%) — 5.19 GB actually returned, not merely deallocated	pvesm status before/after
the hub	none created, and verified rather than assumed. No probe claimed a box, minted a customer or registered an appliance — none ever ran a controller. The registers hold the same 5 customers (david, demo-felhom, demo-hp, peti-felhom, sess-f) and same 4 hosts (demo-felhom-8363b5, demo-hp-bb76ea, drill-r50-0a4f9a, sess-f-2670b5) as before Phase 0	hub / + /hosts

Scratch scripts and logs removed from felhom-pve (/root/p2-probe.sh, /tmp/p2-*.log, /tmp/p1-restoretest.log). pct list on both demo hosts shows only their own 9201.

---

What this changes about the merge

1. P1 removes the restore risk from the decision. A pre-merge archive restores clean with parity ok; it should be re-confirmed against the merged agent, which is one command.
2. The variant question is not "will it boot" — it is "which invariant do we break". Both named variants work perfectly and each violates one documented guarantee. That is the operator's call and is the subject of this session's STOP.
3. V-c exists and is measured. It costs one extra mount path and one extra fstab line, and is the only probed shape that keeps both guarantees.