# SPIKE R-165 Phase 0 — the two probes the merge spike left unmeasured **Date:** 2026-08-03 · **Author:** Claude Code · **Status:** MEASURED — **no layout changed** `SPIKE-r165-mp1-merge-2026-08-02.md` named two things as unmeasured and both are load-bearing. This document measures them. **It changes nothing**: no golden rebuilt, no box reinstalled, no guest config edited outside the throwaway probe guests, which are destroyed at the end. **Host: `felhom-pve` (N100) — Tier 0.** demo-hp would have been the default per the 2026-07-25 ruling, but `target-selection.md` records its `local-lvm` as an **over-subscribed thin pool backing live guest 9201** (~144 GiB allocated over ~54 GiB) where filling it corrupts every guest, and it holds **no container template**. felhom-pve has the template and 258 GiB free at 29% pool usage. Both are Tier 0; this picked the one where a probe cannot damage a live guest. **Neither DooPlex, `ep0`, nor the colleague's box was contacted at any point.** --- ## P1 — does a PRE-merge archive restore cleanly into the merged world? **Verdict: PASS.** ### Method The spike reasoned from `mountParity` that it should pass and said plainly that this had never been executed. It is now executed, on real hardware, with the real restore-test path — not a hand-assembled restore. - **Archive:** `local:backup/vzdump-lxc-9201-2026_07_28-17_43_05.tar.zst` from **demo-hp**, 1.68 GB. Confirmed pre-merge by its own vzdump log rather than by assumption: ``` including mount point rootfs ('/') in backup including mount point mp0 ('/var/lib/docker') in backup including mount point mp1 ('/mnt/sys_drive') in backup ``` - **Command:** `felhom-agent --selftest=restore-test -archive ` on demo-hp. ### Measured result ``` "pass": true, "verified": "boot+running", "mount_parity": "ok", "duration_seconds": 84.2, "mount_inventory": [ "mp0=/var/lib/docker (50G)", "mp1=/mnt/sys_drive (20G)", "mp8=/mnt/felhom-drives (throwaway for the archived bind)", "mp9=/etc/felhom-bootstrap (throwaway for the archived bind)" ] === selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m24s) === ``` `mountParity` was **not** relaxed, weakened or touched in any way. ### The limit of this result, stated rather than glossed **It was run with the CURRENT agent (v0.119.0), because the merged agent does not exist yet** — Part 2 sits after this session's STOP. What it proves is that `mountParity` compares the **archive** against **its own restore**, so a pre-merge archive recreates its own `mp0 + mp1` in the scratch guest and the two agree. That comparison never consults the *host's* golden layout, which is why the merge cannot invalidate it — **provided Part 2 honours its own constraint not to touch `mountParity` or the restore path** (§5, §12 of the task). It is an 84-second command and **should be re-run once the merged agent exists**, which is cheap and turns a sound inference into an observation. --- ## P2 — which S1 variant actually works on this platform? **Verdict: all three probed variants are mechanically clean. They are separated by SCOPING, not by mechanics — and the deciding fact was not in the task's table.** ### Method A throwaway unprivileged LXC per variant (`nesting=1,keyctl=1`, rootfs 8 G + one 10 G volume, `backup=1`), Docker installed from the same repo with the **same `daemon.json` the golden bakes** (`containerd-snapshotter: false`, overlay2, log caps). Per variant, measured at first boot and after **each of three reboots**: 1. both `/var/lib/docker` and `/mnt/sys_drive` present and **writable** (write → read back → delete, a positive observable rather than an `ls`); 2. **one** filesystem — same source device **and** the same free-space figure for both paths; 3. `dockerd` active and `docker run hello-world` succeeding; 4. then once: the **real bootstrap propagation sequence** (`mount --rbind /mnt /mnt`, `mount --make-rshared /mnt`, `docker run -v /mnt:/mnt:rslave`); 5. and: what a container mounting `/mnt:rslave` **actually sees** — the scoping check. No pipe hides a non-zero; every check reports its own rc and the script aborts with `MEASURED-FAIL`. ### The variants | | volume mounted at | then | |---|---|---| | **V-a** | `/var/lib/docker` | `/mnt/sys_drive` = bind of `/var/lib/docker/sys_drive` | | **V-b** | `/mnt/sys_drive` | `/var/lib/docker` = bind of `/mnt/sys_drive/docker` | | **V-c** | `/var/lib/felhom` *(neutral)* | **both** consumer paths are binds of subdirectories | **V-c was not in the task's table.** It was probed *because* the measurements below showed V-a and V-b each violate a different documented invariant, and V-c is the shape that violates neither. It is offered as a **measured option for the operator at the STOP**, not adopted — the task is explicit that a variant is not chosen mid-session. ### Measured results | check | V-a | V-b | V-c | |---|---|---|---| | both paths present + writable | **yes** | **yes** | **yes** | | ONE filesystem, ONE free-space figure | **yes** | **yes** | **yes** | | dockerd active + `docker run` — initial | **yes** | **yes** | **yes** | | dockerd active + `docker run` — reboots **1/2/3** | **3/3** | **3/3** | **3/3** | | `/mnt` propagation `shared`; container sees `/mnt` | **yes** | **yes** | **yes** | | both paths still real mountpoints (`findmnt` non-empty — the form the golden's assertions use) | **yes** | **yes** | **yes** | | container `statfs("/")` reports the merged volume | **yes** (10218772 KiB) | **yes** | **yes** | | **what a container mounting `/mnt:rslave` SEES** | `sys_drive` only — **8.0K** | `sys_drive` **+ `sys_drive/docker`** — **17.9M** | `sys_drive` only — **8.0K** | | customer data inside Docker's data-root | **YES** | no | no | **The mechanical worry was misplaced.** The task flagged V-b's ordering risk — `/var/lib/docker` must be bound before dockerd starts. An `/etc/fstab` bind is ordered by `local-fs.target`, which precedes `basic.target` and therefore `docker.service`, and it held **3 reboots out of 3**. Ordering is not what separates these variants. ### What actually separates them **V-b breaks a documented scoping invariant, and the measurement is the proof.** The controller container is started with `-v /mnt:/mnt:rslave`, and the bootstrap script's own comment states the scope it relies on: > *"scoped to /mnt, which (Model A) holds only Felhom's felhom-data-namespace mounts, never the > customer's other on-drive data"* Under V-b the container sees `/mnt/sys_drive/docker` — **Docker's entire data-root**, 17.9 MB on an empty probe box and growing with every image and every app volume. That sentence becomes false. The controller already holds the Docker socket, so this is **not a capability escalation** — but it puts Docker's internal tree inside the one path the controller's own scanners, the FileBrowser surface and the data-migration engine (which works *"in-process over the controller's `/mnt:/mnt:rslave` RW mount"*) treat as Felhom-only. **V-a breaks the other one:** customer backups live inside Docker's data-root, so `du` on the data-root stops meaning what it says, and the ordinary operator reflex for a sick Docker — clear `/var/lib/docker` — destroys every local recovery unit on the box. **V-c breaks neither**, at the cost of one new mount path and two fstab lines instead of one. --- ## P3 — the golden's four assertions **Not yet run — it belongs to Part 1, which is after the STOP.** Recorded here so it is not lost: `build-golden.sh` fails closed on the split in **four** places (`:126`, `:130` separate-mount asserts; `:315`, `:319` vzdump-exclusion guards), and each retargeted guard must be shown to **abort** against a deliberately wrong shape before the golden is trusted. **One measurement already de-risks it:** under all three variants both `/var/lib/docker` and `/mnt/sys_drive` remain **real mountpoints**, so `findmnt -no SOURCE,FSTYPE | grep -q .` — the exact form the two existing assertions use — still returns non-empty. The assertions can be **retargeted with a changed message and an added guard for the single volume**, rather than rewritten from scratch. They must not be deleted. --- ## Teardown | layer | action | evidence | |---|---|---| | **the machine** | probe guests `9401`, `9402`, `9403` destroyed; P1's scratch `990000` was torn down by the restore-test itself | `pct list` shows only `9201` | | **the host** | `local-lvm` **112398205 → 107204406 KiB** used (30.73% → 29.31%) — **5.19 GB actually returned**, not merely deallocated | `pvesm status` before/after | | **the hub** | **none created, and verified rather than assumed.** No probe claimed a box, minted a customer or registered an appliance — none ever ran a controller. The registers hold the same **5 customers** (`david`, `demo-felhom`, `demo-hp`, `peti-felhom`, `sess-f`) and same **4 hosts** (`demo-felhom-8363b5`, `demo-hp-bb76ea`, `drill-r50-0a4f9a`, `sess-f-2670b5`) as before Phase 0 | hub `/` + `/hosts` | Scratch scripts and logs removed from `felhom-pve` (`/root/p2-probe.sh`, `/tmp/p2-*.log`, `/tmp/p1-restoretest.log`). `pct list` on both demo hosts shows only their own `9201`. --- ## What this changes about the merge 1. **P1 removes the restore risk from the decision.** A pre-merge archive restores clean with parity ok; it should be re-confirmed against the merged agent, which is one command. 2. **The variant question is not "will it boot" — it is "which invariant do we break".** Both named variants work perfectly and each violates one documented guarantee. That is the operator's call and is the subject of this session's STOP. 3. **V-c exists and is measured.** It costs one extra mount path and one extra fstab line, and is the only probed shape that keeps both guarantees.