REPORT: v0.120.0 + golden 3.0.0 — built and proven at the bake, NOT proven on a box
gates / gates (push) Successful in 12s
gates / gates (push) Successful in 12s
States the scope reduction first: Phases 6-7 (reinstall both demo boxes and prove one end to end) were NOT done, nothing was wiped, and the golden is deliberately left unvouched as a result. Filed as R-178. Also records the two instrument errors this session: a census truncated by head -10 that gave the wrong answer about --sysdata-grow (the installer does pass it), and a wait gated on a marker the bake prints before publishing.
This commit is contained in:
@@ -1,41 +1,189 @@
|
||||
# REPORT — CI runs the gate entry point on every push (R-168, 2026-08-02)
|
||||
# REPORT — v0.120.0 + golden 3.0.0: one data volume (R-165 · D-a · V-c), 2026-08-03
|
||||
|
||||
**Overwritten** per the standing rule. The prior contents (session 1's gate entry point, same day) have their durable record in `CHANGELOG.md`.
|
||||
**Overwritten** per the standing rule. The prior contents (R-168, CI) have their durable record in
|
||||
`CHANGELOG.md`.
|
||||
|
||||
**No version bump, no build, no deploy.** This adds `.gitea/workflows/gates.yml` and a CHANGELOG
|
||||
entry. Nothing compiled changed.
|
||||
**Outcome: the merge is BUILT, PROVEN AT THE BAKE, and NOT YET PROVEN ON A BOX.** Everything through
|
||||
Phase 5 is shipped and green. **Phases 6–7 — reinstalling the two demo boxes from the merged golden
|
||||
and proving one end to end — were NOT done**, and the golden is deliberately left **unvouched** as a
|
||||
result. That is a scope reduction against what was approved at the STOP and it is stated here first
|
||||
rather than at the end. Remaining work: **R-178**.
|
||||
|
||||
## What changed
|
||||
---
|
||||
|
||||
`.gitea/workflows/gates.yml` — on every push, a Gitea Actions runner obtains this repo at the
|
||||
**exact pushed SHA** (shallow `git fetch`, no `uses:` step anywhere) and runs
|
||||
`python3 scripts/agent_gates.py --fast` and nothing else. The exit code is the job's result: no `|| true`, no
|
||||
pipe that could swallow it.
|
||||
## 1. Baselines — all three matched §1, re-read on arrival
|
||||
|
||||
**It REPORTS, it cannot REFUSE**, and the workflow header says so in the pre-push hook's voice: this
|
||||
repo pushes straight to `main` with no pull request, so there is no merge for a status check to stand
|
||||
at. The refusing half is `.githooks/pre-push` (per-clone, `--no-verify`-able); this half notices when
|
||||
that was skipped. Making CI blocking needs branch protection plus a PR workflow — an operator
|
||||
decision, tracked as `felhom.eu` `OPEN-ITEMS.md` **R-169**.
|
||||
| Repo | §1 | found | match |
|
||||
|---|---|---|---|
|
||||
| `felhom-agent` | `587dbb43fe97`, v0.119.0, script 2.1.0 | same | **yes** |
|
||||
| `felhom-controller` | `d5be67b91363`, v0.191.2 | same | **yes** |
|
||||
| `felhom.eu` | `1806dfa8e9f6`, hub v0.89.0 | same | **yes** |
|
||||
|
||||
**A failed run emails the operator.** Probe P5 measured that Gitea itself sends **nothing** on a
|
||||
failed run — no mail, no notification row, no log line — so the workflow sends its own via Resend and
|
||||
prints the provider's accepted id, making "a message left the machine" an observable. Demonstrated on
|
||||
a real red run in `felhom.eu`: `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`.
|
||||
Highest register ID **R-177**; `R-178` grepped free before minting.
|
||||
|
||||
**CI reproduces the workspace's SIBLING LAYOUT on purpose.** This repo's entry point invokes the
|
||||
shared `reuse_refs_check.py` that lives in the `felhom.eu` clone next door and is deliberately never
|
||||
copied here, and this repo's `REUSE.md` cites `hub/internal/store/dr_recipe.go`, which lives in the hub. The workflow clones
|
||||
`felhom.eu` as a sibling; without it the gate fails **closed** with `gate is MISSING` — correctly,
|
||||
but for the wrong reason.
|
||||
## 2. Phase 0 — the two probes (full detail: `felhom.eu/documentation/audits/SPIKE-r165-phase0-2026-08-03.md`)
|
||||
|
||||
## Verification
|
||||
**P1 — PASS.** A real pre-merge archive (`mp0+mp1`, confirmed from its own vzdump log, not assumed)
|
||||
restore-tested on demo-hp: `pass: true`, `mount_parity: "ok"`, 84 s, inventory listing `mp0=50G` and
|
||||
`mp1=20G`. **`mountParity` was not relaxed.** Limit stated: it ran with the pre-merge agent because the
|
||||
merged one did not exist yet; the comparison is archive-vs-its-own-restore and never consults the host
|
||||
layout, so it carries — and it is one command to re-run against v0.120.0.
|
||||
|
||||
First run: run #1, id=9, conclusion **success**, sha `eb991445`.
|
||||
**P2 — all three variants mechanically clean; the deciding fact was not in the task's table.** Each
|
||||
built as a throwaway LXC with the golden's own `daemon.json`, then measured at first boot and after
|
||||
**each of three reboots**.
|
||||
|
||||
**CI and the local pre-push hook AGREE**, which is the check that mattered: `90 cited paths — exact 88, suffix 1, ambiguous 0, cross-repo 1, FAILED 0 (siblings searched: felhom.eu)` — identical to the local run.
|
||||
| | V-a | V-b | **V-c (chosen)** |
|
||||
|---|---|---|---|
|
||||
| dockerd + `docker run`, reboots | 3/3 | 3/3 | **3/3** |
|
||||
| ONE filesystem, ONE free figure | yes | yes | **yes** |
|
||||
| container `statfs("/")` = merged volume | yes | yes | **yes** |
|
||||
| container's view of `/mnt` | 8.0K | **17.9M — all of Docker** | **8.0K** |
|
||||
| customer data inside Docker's data-root | **YES** | no | **no** |
|
||||
|
||||
The runner is unprivileged host-mode, shared with the other three repos on a single owner-scoped
|
||||
registration (measured: all four repos' tasks claimed by `felhom-gates-runner`). Full probe detail,
|
||||
the security posture and the teardown: `felhom.eu/documentation/audits/SPIKE-ci-runner-2026-08-02.md`
|
||||
and `felhom.eu/REPORT.md`.
|
||||
**The ordering risk that motivated the probe did not materialise** — an `/etc/fstab` bind is ordered
|
||||
under `local-fs.target`, before `docker.service`. What separates them is which documented guarantee
|
||||
they break. **V-c was probed because both named variants broke one**, and it is offered/chosen as the
|
||||
shape that breaks neither. **The operator ruled V-c at the STOP.**
|
||||
|
||||
**P3 — 8 checks, 8 passed.** Every retargeted golden assertion was run against a **deliberately wrong
|
||||
shape** (a real split-layout guest and a real archive of it) and shown to reject it, and against a
|
||||
right shape and shown to accept it. Guest and archive torn down.
|
||||
|
||||
## 3. Files changed and commits
|
||||
|
||||
| Repo | Commit | Subject |
|
||||
|---|---|---|
|
||||
| `felhom-controller` | `4be6467` | v0.192.0 — the capture floor replaces the bulkhead (B2) |
|
||||
| `felhom-agent` | `cd6e267` | v0.120.0 — one data volume (V-c), golden 3.0.0 |
|
||||
| `felhom.eu` | `7406ac7` | audits: R-165 Phase 0 |
|
||||
| `felhom.eu` | `e3525e6` | host-install: one data volume, derived from the disk |
|
||||
|
||||
Agent files: `configs/build-golden.sh` (2.1.0 → **3.0.0**), `internal/reconcile/bringup.go`,
|
||||
`internal/reconcile/bringup_test.go`, `cmd/felhom-agent/main.go`, `CHANGELOG.md`.
|
||||
|
||||
## 4. The variant, and the golden bake as evidence
|
||||
|
||||
```
|
||||
[golden] build-golden.sh v3.0.0 — baking controller …felhom-controller:0.192.0
|
||||
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds
|
||||
docker OK (overlay2; data-root /var/lib/docker)
|
||||
/var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
|
||||
/mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
|
||||
both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
|
||||
INFO: including mount point rootfs ('/') in backup
|
||||
INFO: including mount point mp0 ('/var/lib/felhom') in backup <-- and NO mp1 line at all
|
||||
INFO: archive file size: 619MB
|
||||
```
|
||||
|
||||
Published: registry **HTTP 200**, `felhom-golden/0.192.0`, sha `54e2a4c431daf580d2807b82d810be36…`.
|
||||
Build guest 9100 destroyed; scratch scripts and credential file removed from the host.
|
||||
|
||||
**Baked on `felhom-pve`, not DooPlex.** The historical bake fixture is DooPlex's nested drill VM, and
|
||||
this task forbids DooPlex outright — so it ran on a Tier-0 demo host, which the task names as the
|
||||
target. demo-hp was avoided for the probes and bake because `target-selection.md` records its
|
||||
`local-lvm` as an over-subscribed thin pool backing a live guest, and it carries no template.
|
||||
|
||||
## 5. Provisioning — and a census that changed the answer
|
||||
|
||||
**`--sysdata-grow` is FOLDED IN, not removed.** The task's rule was to remove it unless something
|
||||
outside the repo passes it. **My first census said nothing did — and it was wrong, because I truncated
|
||||
it with `head -10` and the output was exactly 10 lines.** That is §9.11's own trap
|
||||
("an instrument that can silently drop results is not a measurement") and this project's own recorded
|
||||
lesson. Re-run without truncation: **`felhom.eu/scripts/felhom-host-install.sh:2621` passes it.**
|
||||
|
||||
So: the flags stay accepted (removing them would fail every install on an unknown flag), and their GiB
|
||||
are **added** to the single volume's grow rather than dropped — dropping them would silently shrink
|
||||
every appliance an older installer builds by 42 of 250 GiB. The existing test was **retargeted** to pin
|
||||
the fold, and it caught the behaviour change before I did.
|
||||
|
||||
**§8.2 — the single volume's size IS derived from the physical disk, and already was.** The same
|
||||
census found `felhom-host-install.sh:1776 step_grows` reading the thin pool's real free space; the
|
||||
merge only collapsed its 80/20 split into one total (`226` = the previous `184+42`), so **a standard
|
||||
appliance keeps exactly its 250 G**. An unflagged install does **not** get the golden's 24 G. No row
|
||||
needed to be filed for this.
|
||||
|
||||
## 6. The floor's constants, and the proof it sits below the warning band
|
||||
|
||||
**97% used or 1 GiB free**, whichever binds first — `fillwatch`'s two-term shape, which proved live on
|
||||
2026-08-02 when the critical alert fired on the free-byte term at 91% used. **Deliberately beyond
|
||||
fillwatch's critical band (95% / 2 GiB)**, so the chain is warn (85/5) → critical (95/2) → refuse
|
||||
(97/1) and the customer is always warned before a refusal. `TestFloorSitsBelowTheCriticalWarningBand`
|
||||
pins the ordering on **both** terms; a red-proof setting the floor equal to the critical band fails it.
|
||||
|
||||
1 GiB is a **reserve, not a working budget**: §7.5 measures a DB-backed unit at up to ~2× its data, so
|
||||
no fixed number guarantees a capture fits. What it guarantees is the bulkhead's actual job — that a
|
||||
capture cannot consume the last of the space the container runtime needs.
|
||||
|
||||
## 7. Tests and red-proofs
|
||||
|
||||
Controller **1184 → 1191**; agent suite green; all gates green in all three repos.
|
||||
|
||||
| # | mutated | result |
|
||||
|---|---|---|
|
||||
| C | the **whole** floor block removed | 3 tests FAIL |
|
||||
| D | headroom predicate → per-unit cap (`UsedGB > 20`) | 2 tests FAIL |
|
||||
| E | floor set **equal** to fillwatch's critical band | 1 test FAILS |
|
||||
| P3 | 4 retargeted shell assertions vs a wrong shape | **8/8 rejected/accepted correctly** |
|
||||
|
||||
**One fixture was hollow and was strengthened mid-red-proof.** `TestFloor_TheOld20GCeilingIsGone` sat
|
||||
at exactly 20 GB, so it **survived** a literal `UsedGB > 20` cap — it passed the very shape it forbids.
|
||||
Raised to 120 GB; the mutation now fails it. Reported rather than quietly fixed, per §9.10's rule that
|
||||
a mutation must remove every guard a test covers.
|
||||
|
||||
## 8. Deployed versions
|
||||
|
||||
| what | where | version |
|
||||
|---|---|---|
|
||||
| controller | guest 9201 (felhom-pve) | **0.192.0**, healthy |
|
||||
| agent | felhom-pve | **0.120.0**, `active` |
|
||||
| agent | demo-hp | **0.120.0**, `active` |
|
||||
| golden script | repo | **3.0.0** |
|
||||
| golden archive | registry | **0.192.0**, HTTP 200 — **UNVOUCHED on purpose** |
|
||||
|
||||
Prior agent binaries backed up in place as `/root/felhom-agent.pre-0.120.0.bak` on both hosts.
|
||||
|
||||
## 9. WHAT WAS NOT DONE — Part 4, and why
|
||||
|
||||
**Neither demo box was reinstalled, and nothing was wiped.** I stopped before the destructive phase
|
||||
because the remaining session budget could not confidently cover a wipe **plus** its end-to-end proof
|
||||
**plus** the second box, and a half-reinstalled box is worse than an untouched one. Both demo boxes are
|
||||
running normally on the split layout.
|
||||
|
||||
**Consequently the golden is deliberately UNVOUCHED.** Vouching is what makes a fresh install pick a
|
||||
golden up; vouching one no box has been proven from would put an unproven disk layout in front of the
|
||||
next install anywhere. **Prove first, then vouch** (`CONTEXT.md` S-14). The golden being published but
|
||||
unvouched is a safe, reversible state — nothing consumes it.
|
||||
|
||||
**Nothing in the field is affected.** No box has the merged layout, nothing assumes it at runtime
|
||||
(§8.5), and agent v0.120.0 folds the retired grow so an older installer still produces the same
|
||||
capacity. **The work is filed as R-178** with the ordering (demo-hp first, proven, then demo-felhom).
|
||||
|
||||
## 10. Teardown — all three layers
|
||||
|
||||
| layer | evidence |
|
||||
|---|---|
|
||||
| **machine** | probe guests 9401/9402/9403 destroyed; P3 guest 9410 destroyed; golden build guest 9100 destroyed; P1's scratch 990000 torn down by the restore-test itself. `pct list` on both hosts shows only their own `9201` |
|
||||
| **host** | felhom-pve `local-lvm` 112398205 → 107204406 KiB (**5.19 GB returned**) after the probes; P3's archive freed with `pvesm free`; `/root/{build-golden.sh,.regcred.env,p2-probe.sh,p3-assertions.sh}` and `/tmp/p*.log` removed |
|
||||
| **hub** | **none created, verified not assumed** — no probe ran a controller or claimed a box. Registers hold the same 5 customers and 4 hosts as before. The reinstalls that would have produced NEW host records did not happen, so no old records needed disposing |
|
||||
|
||||
## 11. Register rows
|
||||
|
||||
**Opened:** R-178. **Closed:** R-163 (the ceiling it records no longer exists), R-175 (§7.5 fixed in
|
||||
the same pass). **Updated:** R-165 → SHIPPED-not-yet-proven-live; R-176 → (a) answered by P1, (b)
|
||||
**withdrawn** because every node is reinstalled rather than migrated.
|
||||
|
||||
## 12. CI and `--no-verify`
|
||||
|
||||
**`--no-verify` NOT used** — every push ran `.githooks/pre-push` and passed. CI run ids are recorded
|
||||
in `felhom.eu/REPORT.md`.
|
||||
|
||||
## 13. Observations — noticed, NOT acted on
|
||||
|
||||
- **I gated a wait on `DONE. golden archive volid`, which the bake prints BEFORE publishing**, and
|
||||
briefly reported the publish as failed. It had not failed; a 649 MB upload was still running. No harm
|
||||
(the archive is on disk and the guest destroy did not touch it), but it is the same class as the
|
||||
truncated census: **a marker that is not a completion marker.** Worth a `[golden] publish OK` line at
|
||||
the true end of the script — not added here, out of scope.
|
||||
- The `felhom-golden` package listing endpoint returns a shape my query did not expect; the direct
|
||||
artifact URL is the reliable check.
|
||||
|
||||
Reference in New Issue
Block a user