P1 PASS: a pre-merge archive (mp0+mp1, confirmed from its own vzdump log)
restore-tests clean on demo-hp with mount_parity ok in 84s. mountParity was
not touched. Limit stated: run with the current agent because the merged one
does not exist until after the STOP; the comparison is archive-vs-its-own-
restore and never consults the host layout, so it carries provided Part 2
honours its constraint not to touch the restore path. Re-run after Part 2.
P2: all three probed variants are mechanically clean — both paths writable,
ONE df figure, dockerd 3/3 reboots, /mnt propagation, and a container's
statfs(/) reporting the merged volume. The task's flagged ordering risk for
V-b did not materialise. They are separated by SCOPING instead:
V-a container sees /mnt = 8.0K but customer data sits inside Docker's
data-root, so clearing /var/lib/docker destroys every local unit
V-b container sees Docker's ENTIRE data-root under /mnt (17.9M on an
empty box), making the bootstrap's own scoping comment false
V-c neutral mount at /var/lib/felhom, both paths binds — breaks neither
V-c was probed because the measurements showed each named variant violates
a different documented invariant. It is offered as a measured option for the
operator, NOT adopted.
Teardown all three layers: 9401/9402/9403 destroyed, 5.19 GB returned, and
the hub registers verified unchanged (5 customers, 4 hosts).
This commit is contained in:
@@ -0,0 +1,183 @@
|
||||
# SPIKE R-165 Phase 0 — the two probes the merge spike left unmeasured
|
||||
|
||||
**Date:** 2026-08-03 · **Author:** Claude Code · **Status:** MEASURED — **no layout changed**
|
||||
|
||||
`SPIKE-r165-mp1-merge-2026-08-02.md` named two things as unmeasured and both are load-bearing. This
|
||||
document measures them. **It changes nothing**: no golden rebuilt, no box reinstalled, no guest config
|
||||
edited outside the throwaway probe guests, which are destroyed at the end.
|
||||
|
||||
**Host: `felhom-pve` (N100) — Tier 0.** demo-hp would have been the default per the 2026-07-25 ruling,
|
||||
but `target-selection.md` records its `local-lvm` as an **over-subscribed thin pool backing live guest
|
||||
9201** (~144 GiB allocated over ~54 GiB) where filling it corrupts every guest, and it holds **no
|
||||
container template**. felhom-pve has the template and 258 GiB free at 29% pool usage. Both are Tier 0;
|
||||
this picked the one where a probe cannot damage a live guest. **Neither DooPlex, `ep0`, nor the
|
||||
colleague's box was contacted at any point.**
|
||||
|
||||
---
|
||||
|
||||
## P1 — does a PRE-merge archive restore cleanly into the merged world?
|
||||
|
||||
**Verdict: PASS.**
|
||||
|
||||
### Method
|
||||
|
||||
The spike reasoned from `mountParity` that it should pass and said plainly that this had never been
|
||||
executed. It is now executed, on real hardware, with the real restore-test path — not a hand-assembled
|
||||
restore.
|
||||
|
||||
- **Archive:** `local:backup/vzdump-lxc-9201-2026_07_28-17_43_05.tar.zst` from **demo-hp**, 1.68 GB.
|
||||
Confirmed pre-merge by its own vzdump log rather than by assumption:
|
||||
|
||||
```
|
||||
including mount point rootfs ('/') in backup
|
||||
including mount point mp0 ('/var/lib/docker') in backup
|
||||
including mount point mp1 ('/mnt/sys_drive') in backup
|
||||
```
|
||||
|
||||
- **Command:** `felhom-agent --selftest=restore-test -archive <volid>` on demo-hp.
|
||||
|
||||
### Measured result
|
||||
|
||||
```
|
||||
"pass": true,
|
||||
"verified": "boot+running",
|
||||
"mount_parity": "ok",
|
||||
"duration_seconds": 84.2,
|
||||
"mount_inventory": [
|
||||
"mp0=/var/lib/docker (50G)",
|
||||
"mp1=/mnt/sys_drive (20G)",
|
||||
"mp8=/mnt/felhom-drives (throwaway for the archived bind)",
|
||||
"mp9=/etc/felhom-bootstrap (throwaway for the archived bind)"
|
||||
]
|
||||
=== selftest=restore-test OK (scratch 990000 restored+booted+verified+torn-down in 1m24s) ===
|
||||
```
|
||||
|
||||
`mountParity` was **not** relaxed, weakened or touched in any way.
|
||||
|
||||
### The limit of this result, stated rather than glossed
|
||||
|
||||
**It was run with the CURRENT agent (v0.119.0), because the merged agent does not exist yet** — Part 2
|
||||
sits after this session's STOP. What it proves is that `mountParity` compares the **archive** against
|
||||
**its own restore**, so a pre-merge archive recreates its own `mp0 + mp1` in the scratch guest and the
|
||||
two agree. That comparison never consults the *host's* golden layout, which is why the merge cannot
|
||||
invalidate it — **provided Part 2 honours its own constraint not to touch `mountParity` or the restore
|
||||
path** (§5, §12 of the task). It is an 84-second command and **should be re-run once the merged agent
|
||||
exists**, which is cheap and turns a sound inference into an observation.
|
||||
|
||||
---
|
||||
|
||||
## P2 — which S1 variant actually works on this platform?
|
||||
|
||||
**Verdict: all three probed variants are mechanically clean. They are separated by SCOPING, not by
|
||||
mechanics — and the deciding fact was not in the task's table.**
|
||||
|
||||
### Method
|
||||
|
||||
A throwaway unprivileged LXC per variant (`nesting=1,keyctl=1`, rootfs 8 G + one 10 G volume,
|
||||
`backup=1`), Docker installed from the same repo with the **same `daemon.json` the golden bakes**
|
||||
(`containerd-snapshotter: false`, overlay2, log caps). Per variant, measured at first boot and after
|
||||
**each of three reboots**:
|
||||
|
||||
1. both `/var/lib/docker` and `/mnt/sys_drive` present and **writable** (write → read back → delete,
|
||||
a positive observable rather than an `ls`);
|
||||
2. **one** filesystem — same source device **and** the same free-space figure for both paths;
|
||||
3. `dockerd` active and `docker run hello-world` succeeding;
|
||||
4. then once: the **real bootstrap propagation sequence** (`mount --rbind /mnt /mnt`,
|
||||
`mount --make-rshared /mnt`, `docker run -v /mnt:/mnt:rslave`);
|
||||
5. and: what a container mounting `/mnt:rslave` **actually sees** — the scoping check.
|
||||
|
||||
No pipe hides a non-zero; every check reports its own rc and the script aborts with `MEASURED-FAIL`.
|
||||
|
||||
### The variants
|
||||
|
||||
| | volume mounted at | then |
|
||||
|---|---|---|
|
||||
| **V-a** | `/var/lib/docker` | `/mnt/sys_drive` = bind of `/var/lib/docker/sys_drive` |
|
||||
| **V-b** | `/mnt/sys_drive` | `/var/lib/docker` = bind of `/mnt/sys_drive/docker` |
|
||||
| **V-c** | `/var/lib/felhom` *(neutral)* | **both** consumer paths are binds of subdirectories |
|
||||
|
||||
**V-c was not in the task's table.** It was probed *because* the measurements below showed V-a and V-b
|
||||
each violate a different documented invariant, and V-c is the shape that violates neither. It is
|
||||
offered as a **measured option for the operator at the STOP**, not adopted — the task is explicit that
|
||||
a variant is not chosen mid-session.
|
||||
|
||||
### Measured results
|
||||
|
||||
| check | V-a | V-b | V-c |
|
||||
|---|---|---|---|
|
||||
| both paths present + writable | **yes** | **yes** | **yes** |
|
||||
| ONE filesystem, ONE free-space figure | **yes** | **yes** | **yes** |
|
||||
| dockerd active + `docker run` — initial | **yes** | **yes** | **yes** |
|
||||
| dockerd active + `docker run` — reboots **1/2/3** | **3/3** | **3/3** | **3/3** |
|
||||
| `/mnt` propagation `shared`; container sees `/mnt` | **yes** | **yes** | **yes** |
|
||||
| both paths still real mountpoints (`findmnt` non-empty — the form the golden's assertions use) | **yes** | **yes** | **yes** |
|
||||
| container `statfs("/")` reports the merged volume | **yes** (10218772 KiB) | **yes** | **yes** |
|
||||
| **what a container mounting `/mnt:rslave` SEES** | `sys_drive` only — **8.0K** | `sys_drive` **+ `sys_drive/docker`** — **17.9M** | `sys_drive` only — **8.0K** |
|
||||
| customer data inside Docker's data-root | **YES** | no | no |
|
||||
|
||||
**The mechanical worry was misplaced.** The task flagged V-b's ordering risk — `/var/lib/docker` must
|
||||
be bound before dockerd starts. An `/etc/fstab` bind is ordered by `local-fs.target`, which precedes
|
||||
`basic.target` and therefore `docker.service`, and it held **3 reboots out of 3**. Ordering is not what
|
||||
separates these variants.
|
||||
|
||||
### What actually separates them
|
||||
|
||||
**V-b breaks a documented scoping invariant, and the measurement is the proof.** The controller
|
||||
container is started with `-v /mnt:/mnt:rslave`, and the bootstrap script's own comment states the
|
||||
scope it relies on:
|
||||
|
||||
> *"scoped to /mnt, which (Model A) holds only Felhom's felhom-data-namespace mounts, never the
|
||||
> customer's other on-drive data"*
|
||||
|
||||
Under V-b the container sees `/mnt/sys_drive/docker` — **Docker's entire data-root**, 17.9 MB on an
|
||||
empty probe box and growing with every image and every app volume. That sentence becomes false. The
|
||||
controller already holds the Docker socket, so this is **not a capability escalation** — but it puts
|
||||
Docker's internal tree inside the one path the controller's own scanners, the FileBrowser surface and
|
||||
the data-migration engine (which works *"in-process over the controller's `/mnt:/mnt:rslave` RW
|
||||
mount"*) treat as Felhom-only.
|
||||
|
||||
**V-a breaks the other one:** customer backups live inside Docker's data-root, so `du` on the data-root
|
||||
stops meaning what it says, and the ordinary operator reflex for a sick Docker — clear `/var/lib/docker`
|
||||
— destroys every local recovery unit on the box.
|
||||
|
||||
**V-c breaks neither**, at the cost of one new mount path and two fstab lines instead of one.
|
||||
|
||||
---
|
||||
|
||||
## P3 — the golden's four assertions
|
||||
|
||||
**Not yet run — it belongs to Part 1, which is after the STOP.** Recorded here so it is not lost:
|
||||
`build-golden.sh` fails closed on the split in **four** places (`:126`, `:130` separate-mount asserts;
|
||||
`:315`, `:319` vzdump-exclusion guards), and each retargeted guard must be shown to **abort** against a
|
||||
deliberately wrong shape before the golden is trusted.
|
||||
|
||||
**One measurement already de-risks it:** under all three variants both `/var/lib/docker` and
|
||||
`/mnt/sys_drive` remain **real mountpoints**, so `findmnt -no SOURCE,FSTYPE <path> | grep -q .` — the
|
||||
exact form the two existing assertions use — still returns non-empty. The assertions can be
|
||||
**retargeted with a changed message and an added guard for the single volume**, rather than rewritten
|
||||
from scratch. They must not be deleted.
|
||||
|
||||
---
|
||||
|
||||
## Teardown
|
||||
|
||||
| layer | action | evidence |
|
||||
|---|---|---|
|
||||
| **the machine** | probe guests `9401`, `9402`, `9403` destroyed; P1's scratch `990000` was torn down by the restore-test itself | `pct list` shows only `9201` |
|
||||
| **the host** | `local-lvm` **112398205 → 107204406 KiB** used (30.73% → 29.31%) — **5.19 GB actually returned**, not merely deallocated | `pvesm status` before/after |
|
||||
| **the hub** | **none created, and verified rather than assumed.** No probe claimed a box, minted a customer or registered an appliance — none ever ran a controller. The registers hold the same **5 customers** (`david`, `demo-felhom`, `demo-hp`, `peti-felhom`, `sess-f`) and same **4 hosts** (`demo-felhom-8363b5`, `demo-hp-bb76ea`, `drill-r50-0a4f9a`, `sess-f-2670b5`) as before Phase 0 | hub `/` + `/hosts` |
|
||||
|
||||
Scratch scripts and logs removed from `felhom-pve` (`/root/p2-probe.sh`, `/tmp/p2-*.log`,
|
||||
`/tmp/p1-restoretest.log`). `pct list` on both demo hosts shows only their own `9201`.
|
||||
|
||||
---
|
||||
|
||||
## What this changes about the merge
|
||||
|
||||
1. **P1 removes the restore risk from the decision.** A pre-merge archive restores clean with parity
|
||||
ok; it should be re-confirmed against the merged agent, which is one command.
|
||||
2. **The variant question is not "will it boot" — it is "which invariant do we break".** Both named
|
||||
variants work perfectly and each violates one documented guarantee. That is the operator's call and
|
||||
is the subject of this session's STOP.
|
||||
3. **V-c exists and is measured.** It costs one extra mount path and one extra fstab line, and is the
|
||||
only probed shape that keeps both guarantees.
|
||||
Reference in New Issue
Block a user