diff --git a/REPORT.md b/REPORT.md index 0f449dd..9589fd3 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,189 +1,81 @@ -# REPORT — v0.120.0 + golden 3.0.0: one data volume (R-165 · D-a · V-c), 2026-08-03 +# REPORT — releasing publishes, and an unreleasable version fails CI (R-115, R-183) -**Overwritten** per the standing rule. The prior contents (R-168, CI) have their durable record in -`CHANGELOG.md`. +**Date:** 2026-08-03 · **Repo:** `felhom-agent` · **NO VERSION BUMP** — the agent stays **v0.120.0**, +no Go code changed, nothing was built or deployed. -**Outcome: the merge is BUILT, PROVEN AT THE BAKE, and NOT YET PROVEN ON A BOX.** Everything through -Phase 5 is shipped and green. **Phases 6–7 — reinstalling the two demo boxes from the merged golden -and proving one end to end — were NOT done**, and the golden is deliberately left **unvouched** as a -result. That is a scope reduction against what was approved at the STOP and it is stated here first -rather than at the end. Remaining work: **R-178**. +## What changed ---- - -## 1. Baselines — all three matched §1, re-read on arrival - -| Repo | §1 | found | match | -|---|---|---|---| -| `felhom-agent` | `587dbb43fe97`, v0.119.0, script 2.1.0 | same | **yes** | -| `felhom-controller` | `d5be67b91363`, v0.191.2 | same | **yes** | -| `felhom.eu` | `1806dfa8e9f6`, hub v0.89.0 | same | **yes** | - -Highest register ID **R-177**; `R-178` grepped free before minting. - -## 2. Phase 0 — the two probes (full detail: `felhom.eu/documentation/audits/SPIKE-r165-phase0-2026-08-03.md`) - -**P1 — PASS.** A real pre-merge archive (`mp0+mp1`, confirmed from its own vzdump log, not assumed) -restore-tested on demo-hp: `pass: true`, `mount_parity: "ok"`, 84 s, inventory listing `mp0=50G` and -`mp1=20G`. **`mountParity` was not relaxed.** Limit stated: it ran with the pre-merge agent because the -merged one did not exist yet; the comparison is archive-vs-its-own-restore and never consults the host -layout, so it carries — and it is one command to re-run against v0.120.0. - -**P2 — all three variants mechanically clean; the deciding fact was not in the task's table.** Each -built as a throwaway LXC with the golden's own `daemon.json`, then measured at first boot and after -**each of three reboots**. - -| | V-a | V-b | **V-c (chosen)** | -|---|---|---|---| -| dockerd + `docker run`, reboots | 3/3 | 3/3 | **3/3** | -| ONE filesystem, ONE free figure | yes | yes | **yes** | -| container `statfs("/")` = merged volume | yes | yes | **yes** | -| container's view of `/mnt` | 8.0K | **17.9M — all of Docker** | **8.0K** | -| customer data inside Docker's data-root | **YES** | no | **no** | - -**The ordering risk that motivated the probe did not materialise** — an `/etc/fstab` bind is ordered -under `local-fs.target`, before `docker.service`. What separates them is which documented guarantee -they break. **V-c was probed because both named variants broke one**, and it is offered/chosen as the -shape that breaks neither. **The operator ruled V-c at the STOP.** - -**P3 — 8 checks, 8 passed.** Every retargeted golden assertion was run against a **deliberately wrong -shape** (a real split-layout guest and a real archive of it) and shown to reject it, and against a -right shape and shown to accept it. Guest and archive torn down. - -## 3. Files changed and commits - -| Repo | Commit | Subject | -|---|---|---| -| `felhom-controller` | `4be6467` | v0.192.0 — the capture floor replaces the bulkhead (B2) | -| `felhom-agent` | `cd6e267` | v0.120.0 — one data volume (V-c), golden 3.0.0 | -| `felhom.eu` | `7406ac7` | audits: R-165 Phase 0 | -| `felhom.eu` | `e3525e6` | host-install: one data volume, derived from the disk | - -Agent files: `configs/build-golden.sh` (2.1.0 → **3.0.0**), `internal/reconcile/bringup.go`, -`internal/reconcile/bringup_test.go`, `cmd/felhom-agent/main.go`, `CHANGELOG.md`. - -## 4. The variant, and the golden bake as evidence - -``` -[golden] build-golden.sh v3.0.0 — baking controller …felhom-controller:0.192.0 -[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds - docker OK (overlay2; data-root /var/lib/docker) - /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4 - /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4 - both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576 -INFO: including mount point rootfs ('/') in backup -INFO: including mount point mp0 ('/var/lib/felhom') in backup <-- and NO mp1 line at all -INFO: archive file size: 619MB -``` - -Published: registry **HTTP 200**, `felhom-golden/0.192.0`, sha `54e2a4c431daf580d2807b82d810be36…`. -Build guest 9100 destroyed; scratch scripts and credential file removed from the host. - -**Baked on `felhom-pve`, not DooPlex.** The historical bake fixture is DooPlex's nested drill VM, and -this task forbids DooPlex outright — so it ran on a Tier-0 demo host, which the task names as the -target. demo-hp was avoided for the probes and bake because `target-selection.md` records its -`local-lvm` as an over-subscribed thin pool backing a live guest, and it carries no template. - -## 5. Provisioning — and a census that changed the answer - -**`--sysdata-grow` is FOLDED IN, not removed.** The task's rule was to remove it unless something -outside the repo passes it. **My first census said nothing did — and it was wrong, because I truncated -it with `head -10` and the output was exactly 10 lines.** That is §9.11's own trap -("an instrument that can silently drop results is not a measurement") and this project's own recorded -lesson. Re-run without truncation: **`felhom.eu/scripts/felhom-host-install.sh:2621` passes it.** - -So: the flags stay accepted (removing them would fail every install on an unknown flag), and their GiB -are **added** to the single volume's grow rather than dropped — dropping them would silently shrink -every appliance an older installer builds by 42 of 250 GiB. The existing test was **retargeted** to pin -the fold, and it caught the behaviour change before I did. - -**§8.2 — the single volume's size IS derived from the physical disk, and already was.** The same -census found `felhom-host-install.sh:1776 step_grows` reading the thin pool's real free space; the -merge only collapsed its 80/20 split into one total (`226` = the previous `184+42`), so **a standard -appliance keeps exactly its 250 G**. An unflagged install does **not** get the golden's 24 G. No row -needed to be filed for this. - -## 6. The floor's constants, and the proof it sits below the warning band - -**97% used or 1 GiB free**, whichever binds first — `fillwatch`'s two-term shape, which proved live on -2026-08-02 when the critical alert fired on the free-byte term at 91% used. **Deliberately beyond -fillwatch's critical band (95% / 2 GiB)**, so the chain is warn (85/5) → critical (95/2) → refuse -(97/1) and the customer is always warned before a refusal. `TestFloorSitsBelowTheCriticalWarningBand` -pins the ordering on **both** terms; a red-proof setting the floor equal to the critical band fails it. - -1 GiB is a **reserve, not a working budget**: §7.5 measures a DB-backed unit at up to ~2× its data, so -no fixed number guarantees a capture fits. What it guarantees is the bulkhead's actual job — that a -capture cannot consume the last of the space the container runtime needs. - -## 7. Tests and red-proofs - -Controller **1184 → 1191**; agent suite green; all gates green in all three repos. - -| # | mutated | result | -|---|---|---| -| C | the **whole** floor block removed | 3 tests FAIL | -| D | headroom predicate → per-unit cap (`UsedGB > 20`) | 2 tests FAIL | -| E | floor set **equal** to fillwatch's critical band | 1 test FAILS | -| P3 | 4 retargeted shell assertions vs a wrong shape | **8/8 rejected/accepted correctly** | - -**One fixture was hollow and was strengthened mid-red-proof.** `TestFloor_TheOld20GCeilingIsGone` sat -at exactly 20 GB, so it **survived** a literal `UsedGB > 20` cap — it passed the very shape it forbids. -Raised to 120 GB; the mutation now fails it. Reported rather than quietly fixed, per §9.10's rule that -a mutation must remove every guard a test covers. - -## 8. Deployed versions - -| what | where | version | -|---|---|---| -| controller | guest 9201 (felhom-pve) | **0.192.0**, healthy | -| agent | felhom-pve | **0.120.0**, `active` | -| agent | demo-hp | **0.120.0**, `active` | -| golden script | repo | **3.0.0** | -| golden archive | registry | **0.192.0**, HTTP 200 — **UNVOUCHED on purpose** | - -Prior agent binaries backed up in place as `/root/felhom-agent.pre-0.120.0.bak` on both hosts. - -## 9. WHAT WAS NOT DONE — Part 4, and why - -**Neither demo box was reinstalled, and nothing was wiped.** I stopped before the destructive phase -because the remaining session budget could not confidently cover a wipe **plus** its end-to-end proof -**plus** the second box, and a half-reinstalled box is worse than an untouched one. Both demo boxes are -running normally on the split layout. - -**Consequently the golden is deliberately UNVOUCHED.** Vouching is what makes a fresh install pick a -golden up; vouching one no box has been proven from would put an unproven disk layout in front of the -next install anywhere. **Prove first, then vouch** (`CONTEXT.md` S-14). The golden being published but -unvouched is a safe, reversible state — nothing consumes it. - -**Nothing in the field is affected.** No box has the merged layout, nothing assumes it at runtime -(§8.5), and agent v0.120.0 folds the retired grow so an older installer still produces the same -capacity. **The work is filed as R-178** with the ordering (demo-hp first, proven, then demo-felhom). - -## 10. Teardown — all three layers - -| layer | evidence | +| File | | |---|---| -| **machine** | probe guests 9401/9402/9403 destroyed; P3 guest 9410 destroyed; golden build guest 9100 destroyed; P1's scratch 990000 torn down by the restore-test itself. `pct list` on both hosts shows only their own `9201` | -| **host** | felhom-pve `local-lvm` 112398205 → 107204406 KiB (**5.19 GB returned**) after the probes; P3's archive freed with `pvesm free`; `/root/{build-golden.sh,.regcred.env,p2-probe.sh,p3-assertions.sh}` and `/tmp/p*.log` removed | -| **hub** | **none created, verified not assumed** — no probe ran a controller or claimed a box. Registers hold the same 5 customers and 4 hosts as before. The reinstalls that would have produced NEW host records did not happen, so no old records needed disposing | +| `scripts/release-agent.sh` | **new** — THE release path: build → tag → publish → verify by independent download | +| `scripts/check-published-versions.py` | **new** — the R-115 gate | +| `scripts/agent_gates.py` | registers the gate as **not `--fast`** (it needs network) | +| `.gitea/workflows/gates.yml` | CI now runs the **full** gate set, not `--fast` | +| `CLAUDE.md` | the raw `go build` line is replaced by the release script; a **Vouch** row replaces the old Publish row | -## 11. Register rows +## Why -**Opened:** R-178. **Closed:** R-163 (the ceiling it records no longer exists), R-175 (§7.5 fixed in -the same pass). **Updated:** R-165 → SHIPPED-not-yet-proven-live; R-176 → (a) answered by P1, (b) -**withdrawn** because every node is reinstalled rather than migrated. +Publishing was a step someone had to remember and was **forgotten three times in five days** — +R-111's seventeen stranded releases, 0.114.0, and 0.120.0, which sat deployed on both demo hosts and +undownloadable, so a documented-path reinstall would have silently downgraded them to the pre-merge +agent **while reporting success**. R-111's own closing line named this leg and closed SHIPPED without +it; it recurred the same afternoon. A note is not a mechanism. -## 12. CI and `--no-verify` +The script also **tags**, because `felhom-host-install.sh` now fetches the agent's sixteen config +files from `raw/tag/v/` (R-183). A released version with no tag 404s a box mid-install, as +root, on a virgin machine. Tag and package are two halves of one release. -**`--no-verify` NOT used** — every push ran `.githooks/pre-push` and passed. CI run ids are recorded -in `felhom.eu/REPORT.md`. +It **verifies by downloading what it just published** and comparing the sha to what it built. The +publish step's own success is a report on its own write; a fetch returning the right bytes is a +different claim, and it is the one that matters. -## 13. Observations — noticed, NOT acted on +It **does not vouch** — that points machines at a version and stays the operator's act. -- **I gated a wait on `DONE. golden archive volid`, which the bake prints BEFORE publishing**, and - briefly reported the publish as failed. It had not failed; a 649 MB upload was still running. No harm - (the archive is on disk and the guest destroy did not touch it), but it is the same class as the - truncated census: **a marker that is not a completion marker.** Worth a `[golden] publish OK` line at - the true end of the script — not added here, out of scope. -- The `felhom-golden` package listing endpoint returns a shape my query did not expect; the direct - artifact URL is the reliable check. +## The gate's invariant — not the one specified, and the reason was measured + +The task's §8.4 asked for *"the version the hub tells machines to install must be downloadable"*. +**CI cannot see that**, measured rather than assumed (P-C): + +| Endpoint | Anonymous | +|---|---| +| Gitea package **download** | **200** (and **404** for a fake version — it discriminates) | +| Gitea **tags** api | **200** | +| Gitea package **listing** api | **401** — token required | +| Hub `/api/v1/artifacts/` | **401** — per-customer passphrase required | + +So a credential-free gate can ask *"is this version installable"* but not *"which version is +vouched"*. Adding an operator credential to CI to close that is the operator's call, not a gate +author's. The implemented invariant — **every `v` tag must have a downloadable package and a +tag tree that serves the agent's configs** — needs no credential and **catches all three recorded +instances**, because the release script creates the tag and publishes in one act. + +**What it does not catch, stated rather than assumed away:** the hub vouching a version that was +never released at all. Nothing here can see that; it belongs at vouch time in the hub. → **R-184**. + +## Proof + +| Check | Result | +|---|---| +| `go build ./... && go vet ./...` | OK | +| `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) | +| `agent_gates.py --fast` | `published` correctly **SKIPPED** — the pre-push hook must not fail because Gitea blinked | +| `agent_gates.py` (full) | `reuse-refs` OK, `published` OK | +| release script: re-release guard | `ERROR: tag v0.120.0 already exists — releasing over it would make one version name two binaries`, rc=1 | +| release script: clean-tree guard | `ERROR: working tree is dirty — commit and push first`, rc=1 | + +### Red-proof F — both directions + +- **A tagged-but-unpublished version** (`v9.9.9` created for the purpose): gate **rc=1**, + `binary NOT downloadable (HTTP 404 …)`. This is the R-115 shape exactly. +- **The gate deregistered from the entry point**, same bad state: `agent_gates.py` → **rc=0, "all + agent gates OK"**. Restored → **rc=1, CONVICTED: published**. The guard is what catches it, not + something else. + +`v9.9.9` was deleted afterwards; `git ls-remote --tags` shows only `v0.120.0`. + +## Tag convention + +`v`, at the commit the binary was built from. `v0.120.0` was created retroactively at +`cd6e267` — the commit that produced the published binary (sha `a7763d31b55b5ce7…`). `configs/` is +byte-identical between that commit and `main`, so nothing about the sixteen fetched files depends on +the choice; `cd6e267` is tagged because it is the honest one.