felhom.eu/documentation/audits/SPIKE-r165-mp1-merge-2026-08-02.md

SPIKE — measuring the mp1 → mp0 merge (R-165, decision D-a)

Date: 2026-08-02 · Author: Claude Code · Status: MEASUREMENT ONLY — no layout changed

Decision D-a is already taken (CONTEXT.md S-5): the dedicated backup partition is merged away rather than resized. This document measures how, never whether. It ends in a question for the operator and a STOP — the merge itself is next session's supervised work.

0. What was and was not touched

Nothing was created, resized, moved or deleted. No golden was rebuilt, no guest config edited, no partition altered. Every figure below is a read of live state or of committed source. ep0 and Peti's box were not contacted at all (runbooks/target-selection.md, decision D-d).

The monitoring D-a requires (R-167) shipped before this measurement, in controller v0.191.x and hub v0.89.0, and was proven on guest 9201. That ordering is the point: D-a's condition (2) says the monitoring lands with the merge and never after, and landing it first is strictly better.

---

M1 — what is actually there

pct config 9201, both demo hosts, read 2026-08-02.

	demo-felhom (N100)	demo-hp (t740)	golden default
rootfs	32 G	32 G	OS_SIZE_GB=32
mp0 /var/lib/docker	200 G, backup=1	50 G, backup=1	GOLDEN_DOCKER_GB=16
mp1 /mnt/sys_drive	50 G, backup=1	20 G, backup=1	GOLDEN_SYSDATA_GB=8
mp8 /mnt/felhom-drives	bind	bind	—
mp9 bootstrap	bind, ro=1	bind, ro=1	—

Occupancy (df, in-guest):

filesystem	demo-felhom	demo-hp
/	945 M / 32 G (4%)	942 M / 32 G (4%)
/var/lib/docker (mp0)	13 G / 197 G (7%)	5.4 G / 50 G (12%)
/mnt/sys_drive (mp1)	2.0 G / 50 G (5%)	92 M / 20 G (1%)
data drive	21 G / 916 G (3%) — hdd_1	6.7 G / 938 G (1%) — nvme-1tb

THE FIRST FINDING IS THAT "THE LAYOUT" IS NOT ONE THING, AND THE SPREAD IS WIDER THAN §7.5 RECORDS. architecture/07-backup-architecture.md §7.5 states the default appliance as mp0 50G / mp1 20G — that is demo-hp exactly, and it is not demo-felhom, which ships 200 G / 50 G, four and two and a half times larger. Both differ again from the golden's own 16 G / 8 G, because provision grows the volumes after restore. Any merge plan expressed as a fixed pair of numbers is already wrong for one of the two boxes that exist.

The corollary matters more than the numbers. §7.5's headline bound — "≈ 19 GB of app data for a file-only app, ≈ 10 GB for a DB-backed one" — is derived from mp1 = 20 G. On demo-felhom, where mp1 is 50 G, the real bound is ~49 GB / ~24 GB. The architecture doc states one bound as if it were the fleet's, and it is one box's. That is a documentation defect independent of the merge and is filed as its own row.

---

M2 — what lives on mp1 (it is not only backups)

du, in-guest, read 2026-08-02.

demo-felhom  /mnt/sys_drive/felhom-data          2.0 G
               ├── backups/primary               269 M   Tier-1 units of DRIVELESS apps (~30 apps)
               ├── backups/secondary             1.7 G   Tier-2 mirrors
               └── userdata/import                12 K   the canonical drop-zone (R-75)

demo-hp      /mnt/sys_drive/felhom-data           92 M
               ├── backups/primary                       uptime-kuma, paperless
               ├── backups/secondary                     paperless-ngx
               └── userdata/import                       paperless

Four distinct things would move, not one, confirming the task's warning that a plan accounting only for the units is wrong:

1. backups/primary/<app> — the RETAINED Tier-1 recovery unit of every app with no data drive. Thirty apps on demo-felhom.
2. backups/secondary/<app> — Tier-2 cross-drive mirrors, including the _shares pseudo-stack. On demo-felhom this is 1.7 G of the 2.0 G — the majority is Tier 2, not Tier 1.
3. userdata/import — the canonical, app-INDEPENDENT drop zone (GetImportRoot, R-75). It lives on the system drive by contract, not by convenience.
4. userdata/ more generally — the system-data userdata namespace for driveless apps.

Observed occupancy is far below capacity on both boxes (5% and 1%). The measured pressure today is zero; the constraint R-163 records is a ceiling problem, not a current fill problem. That is worth stating plainly because it bears on urgency, not on correctness.

---

M3 — which merge shapes exist, and what each breaks

Three interacting assertions exist today, and a merge touches all three. All three were read at source; none had been measured before.

(a) The golden build ASSERTS the split and ABORTS if it is absent. felhom-agent/configs/build-golden.sh:130:

findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . || {
  echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; }

There is a sibling assertion for mp0 at :126, and two more at :315-324 that abort if the vzdump log shows excluding volume mount point mp0 or mp1. So the golden build fails closed on the split in four places, not one.

(b) The whole-guest archive's scope is rootfs + mp0 + mp1, and only because both carry backup=1. build-golden.sh:69 records why: "backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra mountpoints default backup=0)".

(c) mount_parity compares the ARCHIVE's mpN set against the RESTORED guest's (felhom-agent/internal/reconcile/restoretest.go:271-283, mountParity at :347). Per slot it requires the same mount path and a restored size not smaller than the archive's, and it checks the reverse direction too. A mismatch fails the restore-test outright.

shape	(a) golden assertion	(b) archive scope	(c) mount_parity	verdict
S1 — one volume, two directories. mp1 stops existing; /mnt/sys_drive becomes a directory on the mp0 filesystem	BREAKS — :130 aborts the build. Must be deleted, and :319's vzdump guard for mp1 with it	HOLDS — scope becomes rootfs + mp0, still complete, because the data moved onto mp0	HOLDS for new archives (no mp1 in archive ⇒ none required in restore). A pre-merge archive restored into a merged guest is a different question — see below	the shape D-a describes
S2 — two mounts, one backing pool. mp0 and mp1 remain separate mpN slots on the same storage	HOLDS — both are still separate mounts	HOLDS unchanged	HOLDS unchanged	does NOT remove the ceiling — two filesystems still have two independent dfs. This is thin-provisioning, not a merge, and it converts a clean per-app refusal into a shared-pool exhaustion that neither volume can see coming
S3 — grow mp1, keep the split (the shape D-a REJECTED — measured here as the baseline)	HOLDS	HOLDS	HOLDS	zero structural risk, one --sysdata-grow value. It is "the same wall further away" — D-a's own words — and does not close R-163

The one genuinely unmeasured item in M3, stated as unmeasured: whether a pre-merge archive (carrying mp1) restores cleanly into the merged world. Reading mountParity says it should — the restore recreates mp1 from the archive, so archive and restored guest agree, and parity passes. But that was reasoned from source and not executed, and this project's own record is that four production designs specced against unvalidated mechanisms were all wrong. It is a one-command restore-test on a Tier-0 box and should be run before the merge, not after.

---

M4 — the bulkhead question (the important one)

Today mp1 is not only a ceiling; it is a BULKHEAD. An app whose recovery unit outgrows the space is refused per app, its last good unit is preserved byte-identical (R-158's measurement), and — critically — the overflow cannot reach /var/lib/docker, because it is a different filesystem. The container runtime keeps running.

After S1 the same overflow lands on the filesystem Docker itself runs on. A runaway recovery-unit capture would fill /var/lib/docker, and a full Docker data-root is not a degraded state, it is a stopped one.

This is the one place where "the merge is cheap" stops being true, and it is why the warnings shipped first rather than alongside.

Four candidate replacements. No choice is made here — this is the operator's ruling.

#	replacement	what it costs	what it leaves open
B1 — a reserved block percentage on the merged filesystem. tune2fs -m reserving N% for root; the controller runs unprivileged, so a capture cannot consume the reserve while root-owned Docker can	one tune2fs at build time; no code	the reserve protects root, not Docker's runtime need specifically; sizing it is a guess without a measured worst case
B2 — a refusal threshold in the capture path. captureAllRecoveryUnits refuses when free space would drop below a floor, per app, and emits the alert R-158 just wired	small, local, testable; reuses the alert that now exists and restores the per-app-refusal semantics the bulkhead gave for free	the floor is a number needing justification; it protects the capture path only — a customer filling mp0 through app data is untouched by it
B3 — a filesystem quota on the backup directory. XFS project quota / ext4 project quota on <mp0>/sys_drive/backups	enforces at the filesystem, so every writer is bounded, not only the capture path	recreates a fixed ceiling — i.e. it is R-163 again inside one volume, and D-a's objection to "a bigger number is the same wall further away" applies to it word for word
B4 — R-167's warnings are deemed sufficient. No hard stop; the customer is warned at 85% / 5 GiB and critically at 95% / 2 GiB	zero — it is already shipped and proven live	a warning is not a bulkhead. It depends on a human acting within the window, and the failure it fails to prevent is "Docker's data-root is full", which is the worst failure on the box

The measured input to that ruling: on demo-felhom mp1 holds 2.0 G against a 50 G ceiling, and mp0 has 175 G free. The overflow scenario is not close today on either box. B2 is the only option that preserves the property the bulkhead actually provided — a per-app refusal with the last good unit intact — and it is the one that reuses what R-158 just built.

---

M5 — existing boxes, and what a migration costs

Read from the hub's own registers, 2026-08-02. No box was contacted.

The hub's /hosts register holds four hosts, of which two are ONLINE:

host	customer	agent	status
demo-felhom-8363b5	Demo Ügyfél	0.119.0	ONLINE
demo-hp-bb76ea	Demo HP	0.119.0	ONLINE
drill-r50-0a4f9a	drill-r50	0.113.0	DOWN
sess-f-2670b5	R-120 golden 0.186.0 proof	0.116.0	DOWN

The customer register lists five customers — david, demo-felhom, demo-hp, peti-felhom, sess-f.

THE FINDING THAT CHANGES THE COST: peti-felhom EXISTS AS A CUSTOMER BUT HAS NO HOST IN THE REGISTER. Consistent with the long-standing "guest not on agent node" stop in the Peti-return runbook. So:

• The two demo boxes are the entire measurable migrated population, and both are Tier 0 — disposable, per decision D-d. For them a "migration" is not required at all: they can be reinstalled from a merged golden, which is cheaper and lower-risk than migrating in place, and D-d explicitly permits it.
• D-a's condition (1) — "it must land before any external install" — is currently SATISFIED. No external box appears in the hub's host register. This is the cheapest this decision will ever be, and the window is open now.

UNMEASURED, and reported as unmeasured rather than omitted: Peti's box's actual disk layout. It does not report to the hub, so the hub holds no pct config for it, and it is protected by D-d and target-selection.md — so it was not contacted. Whether a merged golden implies an in-place migration for that box, and what that costs, is not established by this spike. Nor is whether the box is currently restorable at every point of such a migration. Both are inputs the operator has and this session does not.

Also unmeasured: the per-box in-place migration procedure itself (move <mp1>/felhom-data onto mp0, drop the mp1 slot, verify) has not been executed even once on a throwaway guest. If an in-place migration is ever needed, that rehearsal is the first thing to do, and its "is the box restorable at every point?" answer is currently unknown.

---

Ranked options, and a recommendation

Ranked by what the measurements support, not by preference.

1. S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), shipped as a FRESH-INSTALL shape, with the demo boxes REINSTALLED rather than migrated. It is what D-a describes; it genuinely removes the ceiling rather than moving it; the four golden assertions and the two vzdump guards are a bounded, greppable edit; the archive scope stays complete; and B2 restores the per-app refusal that is the bulkhead's real value, reusing the alert R-158 just wired. The migration cost for the measurable population is zero, because both boxes are Tier 0 and reinstallable.
2. S1 + B4 (warnings only). Cheapest, and everything it needs is already shipped and proven live. Rejected as the recommendation only because the failure it declines to prevent — a full Docker data-root — is the worst one on the box, and it depends on a human acting inside the window.
3. S3 (grow mp1, keep the split). Zero structural risk, one number. It is the measured baseline and D-a rejected it; recorded so the decision is compared against something.
4. S2 (two mounts, one pool). Not recommended at all. It keeps both assertions satisfied while delivering none of the benefit, and converts a clean per-app refusal into a shared-pool exhaustion neither df can see coming — strictly worse than today.

Two things to do BEFORE the merge session, both cheap, both currently unmeasured:

• Run one restore-test of a PRE-MERGE archive into a merged-layout guest on a Tier-0 box. M3 reasons it passes; nothing has executed it. This project's own record is that reasoning from source about an unvalidated mechanism has been wrong four times.
• Rehearse the in-place migration once on a throwaway guest, and record whether the guest is restorable at every point of it. Only needed if Peti's box turns out to require migrating rather than reinstalling — which is the operator's information, not the hub's.

---

THE QUESTION FOR THE OPERATOR

Which shape, and what replaces the bulkhead?

The recommendation is S1 + B2, fresh-install shape, demo boxes reinstalled. The two open inputs only the operator has are:

1. Does Peti's box need an in-place migration, or can it be reinstalled? The hub cannot answer this — the host is not in its register. It decides whether the migration rehearsal is required work or optional insurance.
2. Is B2's per-app refusal the right replacement for the bulkhead, or is B4 (warnings only) acceptable? B2 costs a small amount of code and a justified floor; B4 costs nothing and is already live, but accepts that a full Docker data-root is reachable.

STOP. The merge is next session's work, and it is a supervised one.