felhom.eu/documentation/audits/SPIKE-r165-mp1-merge-2026-08-02.md

# SPIKE — measuring the `mp1` → `mp0` merge (R-165, decision D-a)

**Date:** 2026-08-02 · **Author:** Claude Code · **Status:** MEASUREMENT ONLY — **no layout changed**

> **Decision D-a is already taken** (`CONTEXT.md` S-5): the dedicated backup partition is merged away
> rather than resized. This document measures **how**, never **whether**. It ends in a question for the
> operator and a **STOP** — the merge itself is next session's supervised work.

## 0. What was and was not touched

Nothing was created, resized, moved or deleted. No golden was rebuilt, no guest config edited, no
partition altered. Every figure below is a read of live state or of committed source. `ep0` and Peti's
box were not contacted at all (`runbooks/target-selection.md`, decision D-d).

The monitoring D-a requires (**R-167**) shipped **before** this measurement, in controller v0.191.x and
hub v0.89.0, and was proven on guest 9201. That ordering is the point: D-a's condition (2) says the
monitoring lands with the merge and never after, and landing it first is strictly better.

---

## M1 — what is actually there

**`pct config 9201`, both demo hosts, read 2026-08-02.**

| | demo-felhom (N100) | demo-hp (t740) | golden default |
|---|---|---|---|
| `rootfs` | 32 G | 32 G | `OS_SIZE_GB=32` |
| `mp0` `/var/lib/docker` | **200 G**, `backup=1` | **50 G**, `backup=1` | `GOLDEN_DOCKER_GB=16` |
| `mp1` `/mnt/sys_drive` | **50 G**, `backup=1` | **20 G**, `backup=1` | `GOLDEN_SYSDATA_GB=8` |
| `mp8` `/mnt/felhom-drives` | bind | bind | — |
| `mp9` bootstrap | bind, `ro=1` | bind, `ro=1` | — |

**Occupancy (`df`, in-guest):**

| filesystem | demo-felhom | demo-hp |
|---|---|---|
| `/` | 945 M / 32 G (4%) | 942 M / 32 G (4%) |
| `/var/lib/docker` (`mp0`) | 13 G / 197 G (7%) | 5.4 G / 50 G (12%) |
| `/mnt/sys_drive` (`mp1`) | **2.0 G / 50 G (5%)** | **92 M / 20 G (1%)** |
| data drive | 21 G / 916 G (3%) — `hdd_1` | 6.7 G / 938 G (1%) — `nvme-1tb` |

**THE FIRST FINDING IS THAT "THE LAYOUT" IS NOT ONE THING, AND THE SPREAD IS WIDER THAN §7.5 RECORDS.**
`architecture/07-backup-architecture.md` §7.5 states the default appliance as `mp0 50G / mp1 20G` —
that is demo-hp exactly, and it is **not** demo-felhom, which ships **200 G / 50 G**, four and two and
a half times larger. Both differ again from the golden's own `16 G / 8 G`, because `provision` grows
the volumes after restore. **Any merge plan expressed as a fixed pair of numbers is already wrong for
one of the two boxes that exist.**

**The corollary matters more than the numbers.** §7.5's headline bound — *"≈ 19 GB of app data for a
file-only app, ≈ 10 GB for a DB-backed one"* — is derived from `mp1 = 20 G`. On demo-felhom, where
`mp1` is 50 G, the real bound is ~49 GB / ~24 GB. **The architecture doc states one bound as if it
were the fleet's, and it is one box's.** That is a documentation defect independent of the merge and
is filed as its own row.

---

## M2 — what lives on `mp1` (it is not only backups)

**`du`, in-guest, read 2026-08-02.**

```
demo-felhom  /mnt/sys_drive/felhom-data          2.0 G
               ├── backups/primary               269 M   Tier-1 units of DRIVELESS apps (~30 apps)
               ├── backups/secondary             1.7 G   Tier-2 mirrors
               └── userdata/import                12 K   the canonical drop-zone (R-75)

demo-hp      /mnt/sys_drive/felhom-data           92 M
               ├── backups/primary                       uptime-kuma, paperless
               ├── backups/secondary                     paperless-ngx
               └── userdata/import                       paperless
```

**Four distinct things would move, not one**, confirming the task's warning that a plan accounting
only for the units is wrong:

1. **`backups/primary/<app>`** — the RETAINED Tier-1 recovery unit of every app with no data drive.
   Thirty apps on demo-felhom.
2. **`backups/secondary/<app>`** — Tier-2 cross-drive mirrors, including the `_shares` pseudo-stack.
   On demo-felhom this is **1.7 G of the 2.0 G — the majority is Tier 2, not Tier 1.**
3. **`userdata/import`** — the canonical, app-INDEPENDENT drop zone (`GetImportRoot`, R-75). It lives
   on the system drive **by contract**, not by convenience.
4. **`userdata/`** more generally — the system-data userdata namespace for driveless apps.

**Observed occupancy is far below capacity on both boxes** (5% and 1%). The measured pressure today is
zero; the constraint R-163 records is a *ceiling* problem, not a *current fill* problem. That is worth
stating plainly because it bears on urgency, not on correctness.

---

## M3 — which merge shapes exist, and what each breaks

Three interacting assertions exist today, and a merge touches all three. **All three were read at
source; none had been measured before.**

**(a) The golden build ASSERTS the split and ABORTS if it is absent.**
`felhom-agent/configs/build-golden.sh:130`:

```
findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . || {
  echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; }
```

There is a sibling assertion for `mp0` at :126, and **two more** at :315-324 that abort if the vzdump
log shows `excluding volume mount point mp0` or `mp1`. So the golden build fails closed on the split
in **four** places, not one.

**(b) The whole-guest archive's scope is `rootfs + mp0 + mp1`,** and only because both carry
`backup=1`. `build-golden.sh:69` records why: *"backup=1 is MANDATORY: without it vzdump EXCLUDES the
volume (extra mountpoints default backup=0)"*.

**(c) `mount_parity` compares the ARCHIVE's `mpN` set against the RESTORED guest's**
(`felhom-agent/internal/reconcile/restoretest.go:271-283`, `mountParity` at :347). Per slot it requires
the same mount path and a restored size **not smaller** than the archive's, and it checks the reverse
direction too. A mismatch fails the restore-test outright.

| shape | (a) golden assertion | (b) archive scope | (c) `mount_parity` | verdict |
|---|---|---|---|---|
| **S1 — one volume, two directories.** `mp1` stops existing; `/mnt/sys_drive` becomes a directory on the `mp0` filesystem | **BREAKS** — :130 aborts the build. Must be deleted, and :319's vzdump guard for `mp1` with it | **HOLDS** — scope becomes `rootfs + mp0`, still complete, because the data moved onto `mp0` | **HOLDS for new archives** (no `mp1` in archive ⇒ none required in restore). **A pre-merge archive restored into a merged guest is a different question — see below** | the shape D-a describes |
| **S2 — two mounts, one backing pool.** `mp0` and `mp1` remain separate `mpN` slots on the same storage | **HOLDS** — both are still separate mounts | **HOLDS** unchanged | **HOLDS** unchanged | **does NOT remove the ceiling** — two filesystems still have two independent `df`s. This is thin-provisioning, not a merge, and it converts a clean per-app refusal into a shared-pool exhaustion that neither volume can see coming |
| **S3 — grow `mp1`, keep the split** (**the shape D-a REJECTED — measured here as the baseline**) | **HOLDS** | **HOLDS** | **HOLDS** | zero structural risk, one `--sysdata-grow` value. It is *"the same wall further away"* — D-a's own words — and does not close R-163 |

**The one genuinely unmeasured item in M3, stated as unmeasured:** whether a **pre-merge archive**
(carrying `mp1`) restores cleanly into the merged world. Reading `mountParity` says it should — the
restore recreates `mp1` from the archive, so archive and restored guest agree, and parity passes. But
**that was reasoned from source and not executed**, and this project's own record is that four
production designs specced against unvalidated mechanisms were all wrong. **It is a one-command
restore-test on a Tier-0 box and should be run before the merge, not after.**

---

## M4 — the bulkhead question (the important one)

**Today `mp1` is not only a ceiling; it is a BULKHEAD.** An app whose recovery unit outgrows the space
is refused **per app**, its last good unit is preserved **byte-identical** (R-158's measurement), and
— critically — **the overflow cannot reach `/var/lib/docker`**, because it is a different filesystem.
The container runtime keeps running.

After S1 the same overflow lands on the filesystem Docker itself runs on. A runaway recovery-unit
capture would fill `/var/lib/docker`, and a full Docker data-root is not a degraded state, it is a
stopped one.

**This is the one place where "the merge is cheap" stops being true**, and it is why the warnings
shipped first rather than alongside.

Four candidate replacements. **No choice is made here — this is the operator's ruling.**

| # | replacement | what it costs | what it leaves open |
|---|---|---|---|
| **B1 — a reserved block percentage on the merged filesystem.** `tune2fs -m` reserving N% for root; the controller runs unprivileged, so a capture cannot consume the reserve while root-owned Docker can | one `tune2fs` at build time; no code | the reserve protects *root*, not *Docker's runtime need* specifically; sizing it is a guess without a measured worst case |
| **B2 — a refusal threshold in the capture path.** `captureAllRecoveryUnits` refuses when free space would drop below a floor, per app, and emits the alert R-158 just wired | small, local, testable; **reuses the alert that now exists** and restores the per-app-refusal semantics the bulkhead gave for free | the floor is a number needing justification; it protects the capture path only — a customer filling `mp0` through app data is untouched by it |
| **B3 — a filesystem quota on the backup directory.** XFS project quota / ext4 project quota on `<mp0>/sys_drive/backups` | enforces at the filesystem, so **every** writer is bounded, not only the capture path | recreates a fixed ceiling — i.e. it is R-163 again inside one volume, and D-a's objection to *"a bigger number is the same wall further away"* applies to it word for word |
| **B4 — R-167's warnings are deemed sufficient.** No hard stop; the customer is warned at 85% / 5 GiB and critically at 95% / 2 GiB | zero — it is already shipped and proven live | **a warning is not a bulkhead.** It depends on a human acting within the window, and the failure it fails to prevent is "Docker's data-root is full", which is the worst failure on the box |

**The measured input to that ruling:** on demo-felhom `mp1` holds 2.0 G against a 50 G ceiling, and
`mp0` has 175 G free. The overflow scenario is not close today on either box. **B2 is the only option
that preserves the property the bulkhead actually provided** — a per-app refusal with the last good
unit intact — and it is the one that reuses what R-158 just built.

---

## M5 — existing boxes, and what a migration costs

**Read from the hub's own registers, 2026-08-02. No box was contacted.**

The hub's `/hosts` register holds **four** hosts, of which **two are ONLINE**:

| host | customer | agent | status |
|---|---|---|---|
| `demo-felhom-8363b5` | Demo Ügyfél | 0.119.0 | **ONLINE** |
| `demo-hp-bb76ea` | Demo HP | 0.119.0 | **ONLINE** |
| `drill-r50-0a4f9a` | drill-r50 | 0.113.0 | DOWN |
| `sess-f-2670b5` | R-120 golden 0.186.0 proof | 0.116.0 | DOWN |

**The customer register lists five customers** — `david`, `demo-felhom`, `demo-hp`, `peti-felhom`,
`sess-f`.

**THE FINDING THAT CHANGES THE COST: `peti-felhom` EXISTS AS A CUSTOMER BUT HAS NO HOST IN THE
REGISTER.** Consistent with the long-standing *"guest not on agent node"* stop in the Peti-return
runbook. So:

- **The two demo boxes are the entire measurable migrated population, and both are Tier 0 —
  disposable, per decision D-d.** For them a "migration" is not required at all: they can be
  **reinstalled** from a merged golden, which is cheaper and lower-risk than migrating in place, and
  D-d explicitly permits it.
- **D-a's condition (1) — "it must land before any external install" — is currently SATISFIED.** No
  external box appears in the hub's host register. **This is the cheapest this decision will ever be,
  and the window is open now.**

**UNMEASURED, and reported as unmeasured rather than omitted:** Peti's box's actual disk layout. It
does not report to the hub, so the hub holds no `pct config` for it, and it is protected by D-d and
`target-selection.md` — so it was not contacted. **Whether a merged golden implies an in-place
migration for that box, and what that costs, is not established by this spike.** Nor is whether the
box is currently restorable at every point of such a migration. Both are inputs the operator has and
this session does not.

**Also unmeasured:** the per-box in-place migration procedure itself (move `<mp1>/felhom-data` onto
`mp0`, drop the `mp1` slot, verify) has not been executed even once on a throwaway guest. If an
in-place migration is ever needed, **that rehearsal is the first thing to do, and its "is the box
restorable at every point?" answer is currently unknown.**

---

## Ranked options, and a recommendation

**Ranked by what the measurements support, not by preference.**

1. **S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), shipped as a
   FRESH-INSTALL shape, with the demo boxes REINSTALLED rather than migrated.**
   It is what D-a describes; it genuinely removes the ceiling rather than moving it; the four golden
   assertions and the two vzdump guards are a bounded, greppable edit; the archive scope stays
   complete; and B2 restores the per-app refusal that is the bulkhead's real value, reusing the alert
   R-158 just wired. The migration cost for the measurable population is **zero**, because both boxes
   are Tier 0 and reinstallable.
2. **S1 + B4 (warnings only).** Cheapest, and everything it needs is already shipped and proven live.
   Rejected as the *recommendation* only because the failure it declines to prevent — a full Docker
   data-root — is the worst one on the box, and it depends on a human acting inside the window.
3. **S3 (grow `mp1`, keep the split).** Zero structural risk, one number. It is the measured baseline
   and D-a rejected it; recorded so the decision is compared against something.
4. **S2 (two mounts, one pool).** Not recommended at all. It keeps both assertions satisfied while
   delivering none of the benefit, and converts a clean per-app refusal into a shared-pool exhaustion
   neither `df` can see coming — strictly worse than today.

**Two things to do BEFORE the merge session, both cheap, both currently unmeasured:**

- **Run one restore-test of a PRE-MERGE archive into a merged-layout guest on a Tier-0 box.** M3
  reasons it passes; nothing has executed it. This project's own record is that reasoning from source
  about an unvalidated mechanism has been wrong four times.
- **Rehearse the in-place migration once on a throwaway guest**, and record whether the guest is
  restorable at every point of it. Only needed if Peti's box turns out to require migrating rather
  than reinstalling — which is the operator's information, not the hub's.

---

## THE QUESTION FOR THE OPERATOR

**Which shape, and what replaces the bulkhead?**

The recommendation is **S1 + B2**, fresh-install shape, demo boxes reinstalled. The two open inputs
only the operator has are:

1. **Does Peti's box need an in-place migration, or can it be reinstalled?** The hub cannot answer
   this — the host is not in its register. It decides whether the migration rehearsal is required work
   or optional insurance.
2. **Is B2's per-app refusal the right replacement for the bulkhead, or is B4 (warnings only)
   acceptable?** B2 costs a small amount of code and a justified floor; B4 costs nothing and is
   already live, but accepts that a full Docker data-root is reachable.

**STOP.** The merge is next session's work, and it is a supervised one.