felhom-agent/REPORT.md

# REPORT — v0.120.0 + golden 3.0.0: one data volume (R-165 · D-a · V-c), 2026-08-03

**Overwritten** per the standing rule. The prior contents (R-168, CI) have their durable record in
`CHANGELOG.md`.

**Outcome: the merge is BUILT, PROVEN AT THE BAKE, and NOT YET PROVEN ON A BOX.** Everything through
Phase 5 is shipped and green. **Phases 6–7 — reinstalling the two demo boxes from the merged golden
and proving one end to end — were NOT done**, and the golden is deliberately left **unvouched** as a
result. That is a scope reduction against what was approved at the STOP and it is stated here first
rather than at the end. Remaining work: **R-178**.

---

## 1. Baselines — all three matched §1, re-read on arrival

| Repo | §1 | found | match |
|---|---|---|---|
| `felhom-agent` | `587dbb43fe97`, v0.119.0, script 2.1.0 | same | **yes** |
| `felhom-controller` | `d5be67b91363`, v0.191.2 | same | **yes** |
| `felhom.eu` | `1806dfa8e9f6`, hub v0.89.0 | same | **yes** |

Highest register ID **R-177**; `R-178` grepped free before minting.

## 2. Phase 0 — the two probes (full detail: `felhom.eu/documentation/audits/SPIKE-r165-phase0-2026-08-03.md`)

**P1 — PASS.** A real pre-merge archive (`mp0+mp1`, confirmed from its own vzdump log, not assumed)
restore-tested on demo-hp: `pass: true`, `mount_parity: "ok"`, 84 s, inventory listing `mp0=50G` and
`mp1=20G`. **`mountParity` was not relaxed.** Limit stated: it ran with the pre-merge agent because the
merged one did not exist yet; the comparison is archive-vs-its-own-restore and never consults the host
layout, so it carries — and it is one command to re-run against v0.120.0.

**P2 — all three variants mechanically clean; the deciding fact was not in the task's table.** Each
built as a throwaway LXC with the golden's own `daemon.json`, then measured at first boot and after
**each of three reboots**.

| | V-a | V-b | **V-c (chosen)** |
|---|---|---|---|
| dockerd + `docker run`, reboots | 3/3 | 3/3 | **3/3** |
| ONE filesystem, ONE free figure | yes | yes | **yes** |
| container `statfs("/")` = merged volume | yes | yes | **yes** |
| container's view of `/mnt` | 8.0K | **17.9M — all of Docker** | **8.0K** |
| customer data inside Docker's data-root | **YES** | no | **no** |

**The ordering risk that motivated the probe did not materialise** — an `/etc/fstab` bind is ordered
under `local-fs.target`, before `docker.service`. What separates them is which documented guarantee
they break. **V-c was probed because both named variants broke one**, and it is offered/chosen as the
shape that breaks neither. **The operator ruled V-c at the STOP.**

**P3 — 8 checks, 8 passed.** Every retargeted golden assertion was run against a **deliberately wrong
shape** (a real split-layout guest and a real archive of it) and shown to reject it, and against a
right shape and shown to accept it. Guest and archive torn down.

## 3. Files changed and commits

| Repo | Commit | Subject |
|---|---|---|
| `felhom-controller` | `4be6467` | v0.192.0 — the capture floor replaces the bulkhead (B2) |
| `felhom-agent` | `cd6e267` | v0.120.0 — one data volume (V-c), golden 3.0.0 |
| `felhom.eu` | `7406ac7` | audits: R-165 Phase 0 |
| `felhom.eu` | `e3525e6` | host-install: one data volume, derived from the disk |

Agent files: `configs/build-golden.sh` (2.1.0 → **3.0.0**), `internal/reconcile/bringup.go`,
`internal/reconcile/bringup_test.go`, `cmd/felhom-agent/main.go`, `CHANGELOG.md`.

## 4. The variant, and the golden bake as evidence

```
[golden] build-golden.sh v3.0.0 — baking controller …felhom-controller:0.192.0
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds
  docker OK (overlay2; data-root /var/lib/docker)
  /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
  /mnt/sys_drive is a real mount:  /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
  both paths are ONE filesystem:   /dev/mapper/pve-vm--9100--disk--1  23317576
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/var/lib/felhom') in backup      <-- and NO mp1 line at all
INFO: archive file size: 619MB
```

Published: registry **HTTP 200**, `felhom-golden/0.192.0`, sha `54e2a4c431daf580d2807b82d810be36…`.
Build guest 9100 destroyed; scratch scripts and credential file removed from the host.

**Baked on `felhom-pve`, not DooPlex.** The historical bake fixture is DooPlex's nested drill VM, and
this task forbids DooPlex outright — so it ran on a Tier-0 demo host, which the task names as the
target. demo-hp was avoided for the probes and bake because `target-selection.md` records its
`local-lvm` as an over-subscribed thin pool backing a live guest, and it carries no template.

## 5. Provisioning — and a census that changed the answer

**`--sysdata-grow` is FOLDED IN, not removed.** The task's rule was to remove it unless something
outside the repo passes it. **My first census said nothing did — and it was wrong, because I truncated
it with `head -10` and the output was exactly 10 lines.** That is §9.11's own trap
("an instrument that can silently drop results is not a measurement") and this project's own recorded
lesson. Re-run without truncation: **`felhom.eu/scripts/felhom-host-install.sh:2621` passes it.**

So: the flags stay accepted (removing them would fail every install on an unknown flag), and their GiB
are **added** to the single volume's grow rather than dropped — dropping them would silently shrink
every appliance an older installer builds by 42 of 250 GiB. The existing test was **retargeted** to pin
the fold, and it caught the behaviour change before I did.

**§8.2 — the single volume's size IS derived from the physical disk, and already was.** The same
census found `felhom-host-install.sh:1776 step_grows` reading the thin pool's real free space; the
merge only collapsed its 80/20 split into one total (`226` = the previous `184+42`), so **a standard
appliance keeps exactly its 250 G**. An unflagged install does **not** get the golden's 24 G. No row
needed to be filed for this.

## 6. The floor's constants, and the proof it sits below the warning band

**97% used or 1 GiB free**, whichever binds first — `fillwatch`'s two-term shape, which proved live on
2026-08-02 when the critical alert fired on the free-byte term at 91% used. **Deliberately beyond
fillwatch's critical band (95% / 2 GiB)**, so the chain is warn (85/5) → critical (95/2) → refuse
(97/1) and the customer is always warned before a refusal. `TestFloorSitsBelowTheCriticalWarningBand`
pins the ordering on **both** terms; a red-proof setting the floor equal to the critical band fails it.

1 GiB is a **reserve, not a working budget**: §7.5 measures a DB-backed unit at up to ~2× its data, so
no fixed number guarantees a capture fits. What it guarantees is the bulkhead's actual job — that a
capture cannot consume the last of the space the container runtime needs.

## 7. Tests and red-proofs

Controller **1184 → 1191**; agent suite green; all gates green in all three repos.

| # | mutated | result |
|---|---|---|
| C | the **whole** floor block removed | 3 tests FAIL |
| D | headroom predicate → per-unit cap (`UsedGB > 20`) | 2 tests FAIL |
| E | floor set **equal** to fillwatch's critical band | 1 test FAILS |
| P3 | 4 retargeted shell assertions vs a wrong shape | **8/8 rejected/accepted correctly** |

**One fixture was hollow and was strengthened mid-red-proof.** `TestFloor_TheOld20GCeilingIsGone` sat
at exactly 20 GB, so it **survived** a literal `UsedGB > 20` cap — it passed the very shape it forbids.
Raised to 120 GB; the mutation now fails it. Reported rather than quietly fixed, per §9.10's rule that
a mutation must remove every guard a test covers.

## 8. Deployed versions

| what | where | version |
|---|---|---|
| controller | guest 9201 (felhom-pve) | **0.192.0**, healthy |
| agent | felhom-pve | **0.120.0**, `active` |
| agent | demo-hp | **0.120.0**, `active` |
| golden script | repo | **3.0.0** |
| golden archive | registry | **0.192.0**, HTTP 200 — **UNVOUCHED on purpose** |

Prior agent binaries backed up in place as `/root/felhom-agent.pre-0.120.0.bak` on both hosts.

## 9. WHAT WAS NOT DONE — Part 4, and why

**Neither demo box was reinstalled, and nothing was wiped.** I stopped before the destructive phase
because the remaining session budget could not confidently cover a wipe **plus** its end-to-end proof
**plus** the second box, and a half-reinstalled box is worse than an untouched one. Both demo boxes are
running normally on the split layout.

**Consequently the golden is deliberately UNVOUCHED.** Vouching is what makes a fresh install pick a
golden up; vouching one no box has been proven from would put an unproven disk layout in front of the
next install anywhere. **Prove first, then vouch** (`CONTEXT.md` S-14). The golden being published but
unvouched is a safe, reversible state — nothing consumes it.

**Nothing in the field is affected.** No box has the merged layout, nothing assumes it at runtime
(§8.5), and agent v0.120.0 folds the retired grow so an older installer still produces the same
capacity. **The work is filed as R-178** with the ordering (demo-hp first, proven, then demo-felhom).

## 10. Teardown — all three layers

| layer | evidence |
|---|---|
| **machine** | probe guests 9401/9402/9403 destroyed; P3 guest 9410 destroyed; golden build guest 9100 destroyed; P1's scratch 990000 torn down by the restore-test itself. `pct list` on both hosts shows only their own `9201` |
| **host** | felhom-pve `local-lvm` 112398205 → 107204406 KiB (**5.19 GB returned**) after the probes; P3's archive freed with `pvesm free`; `/root/{build-golden.sh,.regcred.env,p2-probe.sh,p3-assertions.sh}` and `/tmp/p*.log` removed |
| **hub** | **none created, verified not assumed** — no probe ran a controller or claimed a box. Registers hold the same 5 customers and 4 hosts as before. The reinstalls that would have produced NEW host records did not happen, so no old records needed disposing |

## 11. Register rows

**Opened:** R-178. **Closed:** R-163 (the ceiling it records no longer exists), R-175 (§7.5 fixed in
the same pass). **Updated:** R-165 → SHIPPED-not-yet-proven-live; R-176 → (a) answered by P1, (b)
**withdrawn** because every node is reinstalled rather than migrated.

## 12. CI and `--no-verify`

**`--no-verify` NOT used** — every push ran `.githooks/pre-push` and passed. CI run ids are recorded
in `felhom.eu/REPORT.md`.

## 13. Observations — noticed, NOT acted on

- **I gated a wait on `DONE. golden archive volid`, which the bake prints BEFORE publishing**, and
  briefly reported the publish as failed. It had not failed; a 649 MB upload was still running. No harm
  (the archive is on disk and the guest destroy did not touch it), but it is the same class as the
  truncated census: **a marker that is not a completion marker.** Worth a `[golden] publish OK` line at
  the true end of the script — not added here, out of scope.
- The `felhom-golden` package listing endpoint returns a shape my query did not expect; the direct
  artifact URL is the reliable check.