diff --git a/documentation/audits/SPIKE-r165-phase0-2026-08-03.md b/documentation/audits/SPIKE-r165-phase0-2026-08-03.md new file mode 100644 index 0000000..b1a224f --- /dev/null +++ b/documentation/audits/SPIKE-r165-phase0-2026-08-03.md @@ -0,0 +1,183 @@ +# SPIKE R-165 Phase 0 — the two probes the merge spike left unmeasured + +**Date:** 2026-08-03 · **Author:** Claude Code · **Status:** MEASURED — **no layout changed** + +`SPIKE-r165-mp1-merge-2026-08-02.md` named two things as unmeasured and both are load-bearing. This +document measures them. **It changes nothing**: no golden rebuilt, no box reinstalled, no guest config +edited outside the throwaway probe guests, which are destroyed at the end. + +**Host: `felhom-pve` (N100) — Tier 0.** demo-hp would have been the default per the 2026-07-25 ruling, +but `target-selection.md` records its `local-lvm` as an **over-subscribed thin pool backing live guest +9201** (~144 GiB allocated over ~54 GiB) where filling it corrupts every guest, and it holds **no +container template**. felhom-pve has the template and 258 GiB free at 29% pool usage. Both are Tier 0; +this picked the one where a probe cannot damage a live guest. **Neither DooPlex, `ep0`, nor the +colleague's box was contacted at any point.** + +--- + +## P1 — does a PRE-merge archive restore cleanly into the merged world? + +**Verdict: PASS.** + +### Method + +The spike reasoned from `mountParity` that it should pass and said plainly that this had never been +executed. It is now executed, on real hardware, with the real restore-test path — not a hand-assembled +restore. + +- **Archive:** `local:backup/vzdump-lxc-9201-2026_07_28-17_43_05.tar.zst` from **demo-hp**, 1.68 GB. + Confirmed pre-merge by its own vzdump log rather than by assumption: + + ``` + including mount point rootfs ('/') in backup + including mount point mp0 ('/var/lib/docker') in backup + including mount point mp1 ('/mnt/sys_drive') in backup + ``` + +- **Command:** `felhom-agent --selftest=restore-test -archive ` on demo-hp. + +### Measured result + +``` +"pass": true, +"verified": "boot+running", +"mount_parity": "ok", +"duration_seconds": 84.2, +"mount_inventory": [ + "mp0=/var/lib/docker (50G)", + "mp1=/mnt/sys_drive (20G)", + "mp8=/mnt/felhom-drives (throwaway for the archived bind)", + "mp9=/etc/felhom-bootstrap (throwaway for the archived bind)" +] +=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m24s) === +``` + +`mountParity` was **not** relaxed, weakened or touched in any way. + +### The limit of this result, stated rather than glossed + +**It was run with the CURRENT agent (v0.119.0), because the merged agent does not exist yet** — Part 2 +sits after this session's STOP. What it proves is that `mountParity` compares the **archive** against +**its own restore**, so a pre-merge archive recreates its own `mp0 + mp1` in the scratch guest and the +two agree. That comparison never consults the *host's* golden layout, which is why the merge cannot +invalidate it — **provided Part 2 honours its own constraint not to touch `mountParity` or the restore +path** (§5, §12 of the task). It is an 84-second command and **should be re-run once the merged agent +exists**, which is cheap and turns a sound inference into an observation. + +--- + +## P2 — which S1 variant actually works on this platform? + +**Verdict: all three probed variants are mechanically clean. They are separated by SCOPING, not by +mechanics — and the deciding fact was not in the task's table.** + +### Method + +A throwaway unprivileged LXC per variant (`nesting=1,keyctl=1`, rootfs 8 G + one 10 G volume, +`backup=1`), Docker installed from the same repo with the **same `daemon.json` the golden bakes** +(`containerd-snapshotter: false`, overlay2, log caps). Per variant, measured at first boot and after +**each of three reboots**: + +1. both `/var/lib/docker` and `/mnt/sys_drive` present and **writable** (write → read back → delete, + a positive observable rather than an `ls`); +2. **one** filesystem — same source device **and** the same free-space figure for both paths; +3. `dockerd` active and `docker run hello-world` succeeding; +4. then once: the **real bootstrap propagation sequence** (`mount --rbind /mnt /mnt`, + `mount --make-rshared /mnt`, `docker run -v /mnt:/mnt:rslave`); +5. and: what a container mounting `/mnt:rslave` **actually sees** — the scoping check. + +No pipe hides a non-zero; every check reports its own rc and the script aborts with `MEASURED-FAIL`. + +### The variants + +| | volume mounted at | then | +|---|---|---| +| **V-a** | `/var/lib/docker` | `/mnt/sys_drive` = bind of `/var/lib/docker/sys_drive` | +| **V-b** | `/mnt/sys_drive` | `/var/lib/docker` = bind of `/mnt/sys_drive/docker` | +| **V-c** | `/var/lib/felhom` *(neutral)* | **both** consumer paths are binds of subdirectories | + +**V-c was not in the task's table.** It was probed *because* the measurements below showed V-a and V-b +each violate a different documented invariant, and V-c is the shape that violates neither. It is +offered as a **measured option for the operator at the STOP**, not adopted — the task is explicit that +a variant is not chosen mid-session. + +### Measured results + +| check | V-a | V-b | V-c | +|---|---|---|---| +| both paths present + writable | **yes** | **yes** | **yes** | +| ONE filesystem, ONE free-space figure | **yes** | **yes** | **yes** | +| dockerd active + `docker run` — initial | **yes** | **yes** | **yes** | +| dockerd active + `docker run` — reboots **1/2/3** | **3/3** | **3/3** | **3/3** | +| `/mnt` propagation `shared`; container sees `/mnt` | **yes** | **yes** | **yes** | +| both paths still real mountpoints (`findmnt` non-empty — the form the golden's assertions use) | **yes** | **yes** | **yes** | +| container `statfs("/")` reports the merged volume | **yes** (10218772 KiB) | **yes** | **yes** | +| **what a container mounting `/mnt:rslave` SEES** | `sys_drive` only — **8.0K** | `sys_drive` **+ `sys_drive/docker`** — **17.9M** | `sys_drive` only — **8.0K** | +| customer data inside Docker's data-root | **YES** | no | no | + +**The mechanical worry was misplaced.** The task flagged V-b's ordering risk — `/var/lib/docker` must +be bound before dockerd starts. An `/etc/fstab` bind is ordered by `local-fs.target`, which precedes +`basic.target` and therefore `docker.service`, and it held **3 reboots out of 3**. Ordering is not what +separates these variants. + +### What actually separates them + +**V-b breaks a documented scoping invariant, and the measurement is the proof.** The controller +container is started with `-v /mnt:/mnt:rslave`, and the bootstrap script's own comment states the +scope it relies on: + +> *"scoped to /mnt, which (Model A) holds only Felhom's felhom-data-namespace mounts, never the +> customer's other on-drive data"* + +Under V-b the container sees `/mnt/sys_drive/docker` — **Docker's entire data-root**, 17.9 MB on an +empty probe box and growing with every image and every app volume. That sentence becomes false. The +controller already holds the Docker socket, so this is **not a capability escalation** — but it puts +Docker's internal tree inside the one path the controller's own scanners, the FileBrowser surface and +the data-migration engine (which works *"in-process over the controller's `/mnt:/mnt:rslave` RW +mount"*) treat as Felhom-only. + +**V-a breaks the other one:** customer backups live inside Docker's data-root, so `du` on the data-root +stops meaning what it says, and the ordinary operator reflex for a sick Docker — clear `/var/lib/docker` +— destroys every local recovery unit on the box. + +**V-c breaks neither**, at the cost of one new mount path and two fstab lines instead of one. + +--- + +## P3 — the golden's four assertions + +**Not yet run — it belongs to Part 1, which is after the STOP.** Recorded here so it is not lost: +`build-golden.sh` fails closed on the split in **four** places (`:126`, `:130` separate-mount asserts; +`:315`, `:319` vzdump-exclusion guards), and each retargeted guard must be shown to **abort** against a +deliberately wrong shape before the golden is trusted. + +**One measurement already de-risks it:** under all three variants both `/var/lib/docker` and +`/mnt/sys_drive` remain **real mountpoints**, so `findmnt -no SOURCE,FSTYPE | grep -q .` — the +exact form the two existing assertions use — still returns non-empty. The assertions can be +**retargeted with a changed message and an added guard for the single volume**, rather than rewritten +from scratch. They must not be deleted. + +--- + +## Teardown + +| layer | action | evidence | +|---|---|---| +| **the machine** | probe guests `9401`, `9402`, `9403` destroyed; P1's scratch `990000` was torn down by the restore-test itself | `pct list` shows only `9201` | +| **the host** | `local-lvm` **112398205 → 107204406 KiB** used (30.73% → 29.31%) — **5.19 GB actually returned**, not merely deallocated | `pvesm status` before/after | +| **the hub** | **none created, and verified rather than assumed.** No probe claimed a box, minted a customer or registered an appliance — none ever ran a controller. The registers hold the same **5 customers** (`david`, `demo-felhom`, `demo-hp`, `peti-felhom`, `sess-f`) and same **4 hosts** (`demo-felhom-8363b5`, `demo-hp-bb76ea`, `drill-r50-0a4f9a`, `sess-f-2670b5`) as before Phase 0 | hub `/` + `/hosts` | + +Scratch scripts and logs removed from `felhom-pve` (`/root/p2-probe.sh`, `/tmp/p2-*.log`, +`/tmp/p1-restoretest.log`). `pct list` on both demo hosts shows only their own `9201`. + +--- + +## What this changes about the merge + +1. **P1 removes the restore risk from the decision.** A pre-merge archive restores clean with parity + ok; it should be re-confirmed against the merged agent, which is one command. +2. **The variant question is not "will it boot" — it is "which invariant do we break".** Both named + variants work perfectly and each violates one documented guarantee. That is the operator's call and + is the subject of this session's STOP. +3. **V-c exists and is measured.** It costs one extra mount path and one extra fstab line, and is the + only probed shape that keeps both guarantees.