# SPIKE — measuring the `mp1` → `mp0` merge (R-165, decision D-a) **Date:** 2026-08-02 · **Author:** Claude Code · **Status:** MEASUREMENT ONLY — **no layout changed** > **Decision D-a is already taken** (`CONTEXT.md` S-5): the dedicated backup partition is merged away > rather than resized. This document measures **how**, never **whether**. It ends in a question for the > operator and a **STOP** — the merge itself is next session's supervised work. ## 0. What was and was not touched Nothing was created, resized, moved or deleted. No golden was rebuilt, no guest config edited, no partition altered. Every figure below is a read of live state or of committed source. `ep0` and Peti's box were not contacted at all (`runbooks/target-selection.md`, decision D-d). The monitoring D-a requires (**R-167**) shipped **before** this measurement, in controller v0.191.x and hub v0.89.0, and was proven on guest 9201. That ordering is the point: D-a's condition (2) says the monitoring lands with the merge and never after, and landing it first is strictly better. --- ## M1 — what is actually there **`pct config 9201`, both demo hosts, read 2026-08-02.** | | demo-felhom (N100) | demo-hp (t740) | golden default | |---|---|---|---| | `rootfs` | 32 G | 32 G | `OS_SIZE_GB=32` | | `mp0` `/var/lib/docker` | **200 G**, `backup=1` | **50 G**, `backup=1` | `GOLDEN_DOCKER_GB=16` | | `mp1` `/mnt/sys_drive` | **50 G**, `backup=1` | **20 G**, `backup=1` | `GOLDEN_SYSDATA_GB=8` | | `mp8` `/mnt/felhom-drives` | bind | bind | — | | `mp9` bootstrap | bind, `ro=1` | bind, `ro=1` | — | **Occupancy (`df`, in-guest):** | filesystem | demo-felhom | demo-hp | |---|---|---| | `/` | 945 M / 32 G (4%) | 942 M / 32 G (4%) | | `/var/lib/docker` (`mp0`) | 13 G / 197 G (7%) | 5.4 G / 50 G (12%) | | `/mnt/sys_drive` (`mp1`) | **2.0 G / 50 G (5%)** | **92 M / 20 G (1%)** | | data drive | 21 G / 916 G (3%) — `hdd_1` | 6.7 G / 938 G (1%) — `nvme-1tb` | **THE FIRST FINDING IS THAT "THE LAYOUT" IS NOT ONE THING, AND THE SPREAD IS WIDER THAN §7.5 RECORDS.** `architecture/07-backup-architecture.md` §7.5 states the default appliance as `mp0 50G / mp1 20G` — that is demo-hp exactly, and it is **not** demo-felhom, which ships **200 G / 50 G**, four and two and a half times larger. Both differ again from the golden's own `16 G / 8 G`, because `provision` grows the volumes after restore. **Any merge plan expressed as a fixed pair of numbers is already wrong for one of the two boxes that exist.** **The corollary matters more than the numbers.** §7.5's headline bound — *"≈ 19 GB of app data for a file-only app, ≈ 10 GB for a DB-backed one"* — is derived from `mp1 = 20 G`. On demo-felhom, where `mp1` is 50 G, the real bound is ~49 GB / ~24 GB. **The architecture doc states one bound as if it were the fleet's, and it is one box's.** That is a documentation defect independent of the merge and is filed as its own row. --- ## M2 — what lives on `mp1` (it is not only backups) **`du`, in-guest, read 2026-08-02.** ``` demo-felhom /mnt/sys_drive/felhom-data 2.0 G ├── backups/primary 269 M Tier-1 units of DRIVELESS apps (~30 apps) ├── backups/secondary 1.7 G Tier-2 mirrors └── userdata/import 12 K the canonical drop-zone (R-75) demo-hp /mnt/sys_drive/felhom-data 92 M ├── backups/primary uptime-kuma, paperless ├── backups/secondary paperless-ngx └── userdata/import paperless ``` **Four distinct things would move, not one**, confirming the task's warning that a plan accounting only for the units is wrong: 1. **`backups/primary/`** — the RETAINED Tier-1 recovery unit of every app with no data drive. Thirty apps on demo-felhom. 2. **`backups/secondary/`** — Tier-2 cross-drive mirrors, including the `_shares` pseudo-stack. On demo-felhom this is **1.7 G of the 2.0 G — the majority is Tier 2, not Tier 1.** 3. **`userdata/import`** — the canonical, app-INDEPENDENT drop zone (`GetImportRoot`, R-75). It lives on the system drive **by contract**, not by convenience. 4. **`userdata/`** more generally — the system-data userdata namespace for driveless apps. **Observed occupancy is far below capacity on both boxes** (5% and 1%). The measured pressure today is zero; the constraint R-163 records is a *ceiling* problem, not a *current fill* problem. That is worth stating plainly because it bears on urgency, not on correctness. --- ## M3 — which merge shapes exist, and what each breaks Three interacting assertions exist today, and a merge touches all three. **All three were read at source; none had been measured before.** **(a) The golden build ASSERTS the split and ABORTS if it is absent.** `felhom-agent/configs/build-golden.sh:130`: ``` findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . || { echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; } ``` There is a sibling assertion for `mp0` at :126, and **two more** at :315-324 that abort if the vzdump log shows `excluding volume mount point mp0` or `mp1`. So the golden build fails closed on the split in **four** places, not one. **(b) The whole-guest archive's scope is `rootfs + mp0 + mp1`,** and only because both carry `backup=1`. `build-golden.sh:69` records why: *"backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra mountpoints default backup=0)"*. **(c) `mount_parity` compares the ARCHIVE's `mpN` set against the RESTORED guest's** (`felhom-agent/internal/reconcile/restoretest.go:271-283`, `mountParity` at :347). Per slot it requires the same mount path and a restored size **not smaller** than the archive's, and it checks the reverse direction too. A mismatch fails the restore-test outright. | shape | (a) golden assertion | (b) archive scope | (c) `mount_parity` | verdict | |---|---|---|---|---| | **S1 — one volume, two directories.** `mp1` stops existing; `/mnt/sys_drive` becomes a directory on the `mp0` filesystem | **BREAKS** — :130 aborts the build. Must be deleted, and :319's vzdump guard for `mp1` with it | **HOLDS** — scope becomes `rootfs + mp0`, still complete, because the data moved onto `mp0` | **HOLDS for new archives** (no `mp1` in archive ⇒ none required in restore). **A pre-merge archive restored into a merged guest is a different question — see below** | the shape D-a describes | | **S2 — two mounts, one backing pool.** `mp0` and `mp1` remain separate `mpN` slots on the same storage | **HOLDS** — both are still separate mounts | **HOLDS** unchanged | **HOLDS** unchanged | **does NOT remove the ceiling** — two filesystems still have two independent `df`s. This is thin-provisioning, not a merge, and it converts a clean per-app refusal into a shared-pool exhaustion that neither volume can see coming | | **S3 — grow `mp1`, keep the split** (**the shape D-a REJECTED — measured here as the baseline**) | **HOLDS** | **HOLDS** | **HOLDS** | zero structural risk, one `--sysdata-grow` value. It is *"the same wall further away"* — D-a's own words — and does not close R-163 | **The one genuinely unmeasured item in M3, stated as unmeasured:** whether a **pre-merge archive** (carrying `mp1`) restores cleanly into the merged world. Reading `mountParity` says it should — the restore recreates `mp1` from the archive, so archive and restored guest agree, and parity passes. But **that was reasoned from source and not executed**, and this project's own record is that four production designs specced against unvalidated mechanisms were all wrong. **It is a one-command restore-test on a Tier-0 box and should be run before the merge, not after.** --- ## M4 — the bulkhead question (the important one) **Today `mp1` is not only a ceiling; it is a BULKHEAD.** An app whose recovery unit outgrows the space is refused **per app**, its last good unit is preserved **byte-identical** (R-158's measurement), and — critically — **the overflow cannot reach `/var/lib/docker`**, because it is a different filesystem. The container runtime keeps running. After S1 the same overflow lands on the filesystem Docker itself runs on. A runaway recovery-unit capture would fill `/var/lib/docker`, and a full Docker data-root is not a degraded state, it is a stopped one. **This is the one place where "the merge is cheap" stops being true**, and it is why the warnings shipped first rather than alongside. Four candidate replacements. **No choice is made here — this is the operator's ruling.** | # | replacement | what it costs | what it leaves open | |---|---|---|---| | **B1 — a reserved block percentage on the merged filesystem.** `tune2fs -m` reserving N% for root; the controller runs unprivileged, so a capture cannot consume the reserve while root-owned Docker can | one `tune2fs` at build time; no code | the reserve protects *root*, not *Docker's runtime need* specifically; sizing it is a guess without a measured worst case | | **B2 — a refusal threshold in the capture path.** `captureAllRecoveryUnits` refuses when free space would drop below a floor, per app, and emits the alert R-158 just wired | small, local, testable; **reuses the alert that now exists** and restores the per-app-refusal semantics the bulkhead gave for free | the floor is a number needing justification; it protects the capture path only — a customer filling `mp0` through app data is untouched by it | | **B3 — a filesystem quota on the backup directory.** XFS project quota / ext4 project quota on `/sys_drive/backups` | enforces at the filesystem, so **every** writer is bounded, not only the capture path | recreates a fixed ceiling — i.e. it is R-163 again inside one volume, and D-a's objection to *"a bigger number is the same wall further away"* applies to it word for word | | **B4 — R-167's warnings are deemed sufficient.** No hard stop; the customer is warned at 85% / 5 GiB and critically at 95% / 2 GiB | zero — it is already shipped and proven live | **a warning is not a bulkhead.** It depends on a human acting within the window, and the failure it fails to prevent is "Docker's data-root is full", which is the worst failure on the box | **The measured input to that ruling:** on demo-felhom `mp1` holds 2.0 G against a 50 G ceiling, and `mp0` has 175 G free. The overflow scenario is not close today on either box. **B2 is the only option that preserves the property the bulkhead actually provided** — a per-app refusal with the last good unit intact — and it is the one that reuses what R-158 just built. --- ## M5 — existing boxes, and what a migration costs **Read from the hub's own registers, 2026-08-02. No box was contacted.** The hub's `/hosts` register holds **four** hosts, of which **two are ONLINE**: | host | customer | agent | status | |---|---|---|---| | `demo-felhom-8363b5` | Demo Ügyfél | 0.119.0 | **ONLINE** | | `demo-hp-bb76ea` | Demo HP | 0.119.0 | **ONLINE** | | `drill-r50-0a4f9a` | drill-r50 | 0.113.0 | DOWN | | `sess-f-2670b5` | R-120 golden 0.186.0 proof | 0.116.0 | DOWN | **The customer register lists five customers** — `david`, `demo-felhom`, `demo-hp`, `peti-felhom`, `sess-f`. **THE FINDING THAT CHANGES THE COST: `peti-felhom` EXISTS AS A CUSTOMER BUT HAS NO HOST IN THE REGISTER.** Consistent with the long-standing *"guest not on agent node"* stop in the Peti-return runbook. So: - **The two demo boxes are the entire measurable migrated population, and both are Tier 0 — disposable, per decision D-d.** For them a "migration" is not required at all: they can be **reinstalled** from a merged golden, which is cheaper and lower-risk than migrating in place, and D-d explicitly permits it. - **D-a's condition (1) — "it must land before any external install" — is currently SATISFIED.** No external box appears in the hub's host register. **This is the cheapest this decision will ever be, and the window is open now.** **UNMEASURED, and reported as unmeasured rather than omitted:** Peti's box's actual disk layout. It does not report to the hub, so the hub holds no `pct config` for it, and it is protected by D-d and `target-selection.md` — so it was not contacted. **Whether a merged golden implies an in-place migration for that box, and what that costs, is not established by this spike.** Nor is whether the box is currently restorable at every point of such a migration. Both are inputs the operator has and this session does not. **Also unmeasured:** the per-box in-place migration procedure itself (move `/felhom-data` onto `mp0`, drop the `mp1` slot, verify) has not been executed even once on a throwaway guest. If an in-place migration is ever needed, **that rehearsal is the first thing to do, and its "is the box restorable at every point?" answer is currently unknown.** --- ## Ranked options, and a recommendation **Ranked by what the measurements support, not by preference.** 1. **S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), shipped as a FRESH-INSTALL shape, with the demo boxes REINSTALLED rather than migrated.** It is what D-a describes; it genuinely removes the ceiling rather than moving it; the four golden assertions and the two vzdump guards are a bounded, greppable edit; the archive scope stays complete; and B2 restores the per-app refusal that is the bulkhead's real value, reusing the alert R-158 just wired. The migration cost for the measurable population is **zero**, because both boxes are Tier 0 and reinstallable. 2. **S1 + B4 (warnings only).** Cheapest, and everything it needs is already shipped and proven live. Rejected as the *recommendation* only because the failure it declines to prevent — a full Docker data-root — is the worst one on the box, and it depends on a human acting inside the window. 3. **S3 (grow `mp1`, keep the split).** Zero structural risk, one number. It is the measured baseline and D-a rejected it; recorded so the decision is compared against something. 4. **S2 (two mounts, one pool).** Not recommended at all. It keeps both assertions satisfied while delivering none of the benefit, and converts a clean per-app refusal into a shared-pool exhaustion neither `df` can see coming — strictly worse than today. **Two things to do BEFORE the merge session, both cheap, both currently unmeasured:** - **Run one restore-test of a PRE-MERGE archive into a merged-layout guest on a Tier-0 box.** M3 reasons it passes; nothing has executed it. This project's own record is that reasoning from source about an unvalidated mechanism has been wrong four times. - **Rehearse the in-place migration once on a throwaway guest**, and record whether the guest is restorable at every point of it. Only needed if Peti's box turns out to require migrating rather than reinstalling — which is the operator's information, not the hub's. --- ## THE QUESTION FOR THE OPERATOR **Which shape, and what replaces the bulkhead?** The recommendation is **S1 + B2**, fresh-install shape, demo boxes reinstalled. The two open inputs only the operator has are: 1. **Does Peti's box need an in-place migration, or can it be reinstalled?** The hub cannot answer this — the host is not in its register. It decides whether the migration rehearsal is required work or optional insurance. 2. **Is B2's per-app refusal the right replacement for the bulkhead, or is B4 (warnings only) acceptable?** B2 costs a small amount of code and a justified floor; B4 costs nothing and is already live, but accepts that a full Docker data-root is reachable. **STOP.** The merge is next session's work, and it is a supervised one.