REPORT: release path + published-versions gate (no version bump)
gates / gates (push) Failing after 7s
gates / gates (push) Failing after 7s
This commit is contained in:
@@ -1,189 +1,81 @@
|
||||
# REPORT — v0.120.0 + golden 3.0.0: one data volume (R-165 · D-a · V-c), 2026-08-03
|
||||
# REPORT — releasing publishes, and an unreleasable version fails CI (R-115, R-183)
|
||||
|
||||
**Overwritten** per the standing rule. The prior contents (R-168, CI) have their durable record in
|
||||
`CHANGELOG.md`.
|
||||
**Date:** 2026-08-03 · **Repo:** `felhom-agent` · **NO VERSION BUMP** — the agent stays **v0.120.0**,
|
||||
no Go code changed, nothing was built or deployed.
|
||||
|
||||
**Outcome: the merge is BUILT, PROVEN AT THE BAKE, and NOT YET PROVEN ON A BOX.** Everything through
|
||||
Phase 5 is shipped and green. **Phases 6–7 — reinstalling the two demo boxes from the merged golden
|
||||
and proving one end to end — were NOT done**, and the golden is deliberately left **unvouched** as a
|
||||
result. That is a scope reduction against what was approved at the STOP and it is stated here first
|
||||
rather than at the end. Remaining work: **R-178**.
|
||||
## What changed
|
||||
|
||||
---
|
||||
|
||||
## 1. Baselines — all three matched §1, re-read on arrival
|
||||
|
||||
| Repo | §1 | found | match |
|
||||
|---|---|---|---|
|
||||
| `felhom-agent` | `587dbb43fe97`, v0.119.0, script 2.1.0 | same | **yes** |
|
||||
| `felhom-controller` | `d5be67b91363`, v0.191.2 | same | **yes** |
|
||||
| `felhom.eu` | `1806dfa8e9f6`, hub v0.89.0 | same | **yes** |
|
||||
|
||||
Highest register ID **R-177**; `R-178` grepped free before minting.
|
||||
|
||||
## 2. Phase 0 — the two probes (full detail: `felhom.eu/documentation/audits/SPIKE-r165-phase0-2026-08-03.md`)
|
||||
|
||||
**P1 — PASS.** A real pre-merge archive (`mp0+mp1`, confirmed from its own vzdump log, not assumed)
|
||||
restore-tested on demo-hp: `pass: true`, `mount_parity: "ok"`, 84 s, inventory listing `mp0=50G` and
|
||||
`mp1=20G`. **`mountParity` was not relaxed.** Limit stated: it ran with the pre-merge agent because the
|
||||
merged one did not exist yet; the comparison is archive-vs-its-own-restore and never consults the host
|
||||
layout, so it carries — and it is one command to re-run against v0.120.0.
|
||||
|
||||
**P2 — all three variants mechanically clean; the deciding fact was not in the task's table.** Each
|
||||
built as a throwaway LXC with the golden's own `daemon.json`, then measured at first boot and after
|
||||
**each of three reboots**.
|
||||
|
||||
| | V-a | V-b | **V-c (chosen)** |
|
||||
|---|---|---|---|
|
||||
| dockerd + `docker run`, reboots | 3/3 | 3/3 | **3/3** |
|
||||
| ONE filesystem, ONE free figure | yes | yes | **yes** |
|
||||
| container `statfs("/")` = merged volume | yes | yes | **yes** |
|
||||
| container's view of `/mnt` | 8.0K | **17.9M — all of Docker** | **8.0K** |
|
||||
| customer data inside Docker's data-root | **YES** | no | **no** |
|
||||
|
||||
**The ordering risk that motivated the probe did not materialise** — an `/etc/fstab` bind is ordered
|
||||
under `local-fs.target`, before `docker.service`. What separates them is which documented guarantee
|
||||
they break. **V-c was probed because both named variants broke one**, and it is offered/chosen as the
|
||||
shape that breaks neither. **The operator ruled V-c at the STOP.**
|
||||
|
||||
**P3 — 8 checks, 8 passed.** Every retargeted golden assertion was run against a **deliberately wrong
|
||||
shape** (a real split-layout guest and a real archive of it) and shown to reject it, and against a
|
||||
right shape and shown to accept it. Guest and archive torn down.
|
||||
|
||||
## 3. Files changed and commits
|
||||
|
||||
| Repo | Commit | Subject |
|
||||
|---|---|---|
|
||||
| `felhom-controller` | `4be6467` | v0.192.0 — the capture floor replaces the bulkhead (B2) |
|
||||
| `felhom-agent` | `cd6e267` | v0.120.0 — one data volume (V-c), golden 3.0.0 |
|
||||
| `felhom.eu` | `7406ac7` | audits: R-165 Phase 0 |
|
||||
| `felhom.eu` | `e3525e6` | host-install: one data volume, derived from the disk |
|
||||
|
||||
Agent files: `configs/build-golden.sh` (2.1.0 → **3.0.0**), `internal/reconcile/bringup.go`,
|
||||
`internal/reconcile/bringup_test.go`, `cmd/felhom-agent/main.go`, `CHANGELOG.md`.
|
||||
|
||||
## 4. The variant, and the golden bake as evidence
|
||||
|
||||
```
|
||||
[golden] build-golden.sh v3.0.0 — baking controller …felhom-controller:0.192.0
|
||||
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds
|
||||
docker OK (overlay2; data-root /var/lib/docker)
|
||||
/var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
|
||||
/mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
|
||||
both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
|
||||
INFO: including mount point rootfs ('/') in backup
|
||||
INFO: including mount point mp0 ('/var/lib/felhom') in backup <-- and NO mp1 line at all
|
||||
INFO: archive file size: 619MB
|
||||
```
|
||||
|
||||
Published: registry **HTTP 200**, `felhom-golden/0.192.0`, sha `54e2a4c431daf580d2807b82d810be36…`.
|
||||
Build guest 9100 destroyed; scratch scripts and credential file removed from the host.
|
||||
|
||||
**Baked on `felhom-pve`, not DooPlex.** The historical bake fixture is DooPlex's nested drill VM, and
|
||||
this task forbids DooPlex outright — so it ran on a Tier-0 demo host, which the task names as the
|
||||
target. demo-hp was avoided for the probes and bake because `target-selection.md` records its
|
||||
`local-lvm` as an over-subscribed thin pool backing a live guest, and it carries no template.
|
||||
|
||||
## 5. Provisioning — and a census that changed the answer
|
||||
|
||||
**`--sysdata-grow` is FOLDED IN, not removed.** The task's rule was to remove it unless something
|
||||
outside the repo passes it. **My first census said nothing did — and it was wrong, because I truncated
|
||||
it with `head -10` and the output was exactly 10 lines.** That is §9.11's own trap
|
||||
("an instrument that can silently drop results is not a measurement") and this project's own recorded
|
||||
lesson. Re-run without truncation: **`felhom.eu/scripts/felhom-host-install.sh:2621` passes it.**
|
||||
|
||||
So: the flags stay accepted (removing them would fail every install on an unknown flag), and their GiB
|
||||
are **added** to the single volume's grow rather than dropped — dropping them would silently shrink
|
||||
every appliance an older installer builds by 42 of 250 GiB. The existing test was **retargeted** to pin
|
||||
the fold, and it caught the behaviour change before I did.
|
||||
|
||||
**§8.2 — the single volume's size IS derived from the physical disk, and already was.** The same
|
||||
census found `felhom-host-install.sh:1776 step_grows` reading the thin pool's real free space; the
|
||||
merge only collapsed its 80/20 split into one total (`226` = the previous `184+42`), so **a standard
|
||||
appliance keeps exactly its 250 G**. An unflagged install does **not** get the golden's 24 G. No row
|
||||
needed to be filed for this.
|
||||
|
||||
## 6. The floor's constants, and the proof it sits below the warning band
|
||||
|
||||
**97% used or 1 GiB free**, whichever binds first — `fillwatch`'s two-term shape, which proved live on
|
||||
2026-08-02 when the critical alert fired on the free-byte term at 91% used. **Deliberately beyond
|
||||
fillwatch's critical band (95% / 2 GiB)**, so the chain is warn (85/5) → critical (95/2) → refuse
|
||||
(97/1) and the customer is always warned before a refusal. `TestFloorSitsBelowTheCriticalWarningBand`
|
||||
pins the ordering on **both** terms; a red-proof setting the floor equal to the critical band fails it.
|
||||
|
||||
1 GiB is a **reserve, not a working budget**: §7.5 measures a DB-backed unit at up to ~2× its data, so
|
||||
no fixed number guarantees a capture fits. What it guarantees is the bulkhead's actual job — that a
|
||||
capture cannot consume the last of the space the container runtime needs.
|
||||
|
||||
## 7. Tests and red-proofs
|
||||
|
||||
Controller **1184 → 1191**; agent suite green; all gates green in all three repos.
|
||||
|
||||
| # | mutated | result |
|
||||
|---|---|---|
|
||||
| C | the **whole** floor block removed | 3 tests FAIL |
|
||||
| D | headroom predicate → per-unit cap (`UsedGB > 20`) | 2 tests FAIL |
|
||||
| E | floor set **equal** to fillwatch's critical band | 1 test FAILS |
|
||||
| P3 | 4 retargeted shell assertions vs a wrong shape | **8/8 rejected/accepted correctly** |
|
||||
|
||||
**One fixture was hollow and was strengthened mid-red-proof.** `TestFloor_TheOld20GCeilingIsGone` sat
|
||||
at exactly 20 GB, so it **survived** a literal `UsedGB > 20` cap — it passed the very shape it forbids.
|
||||
Raised to 120 GB; the mutation now fails it. Reported rather than quietly fixed, per §9.10's rule that
|
||||
a mutation must remove every guard a test covers.
|
||||
|
||||
## 8. Deployed versions
|
||||
|
||||
| what | where | version |
|
||||
|---|---|---|
|
||||
| controller | guest 9201 (felhom-pve) | **0.192.0**, healthy |
|
||||
| agent | felhom-pve | **0.120.0**, `active` |
|
||||
| agent | demo-hp | **0.120.0**, `active` |
|
||||
| golden script | repo | **3.0.0** |
|
||||
| golden archive | registry | **0.192.0**, HTTP 200 — **UNVOUCHED on purpose** |
|
||||
|
||||
Prior agent binaries backed up in place as `/root/felhom-agent.pre-0.120.0.bak` on both hosts.
|
||||
|
||||
## 9. WHAT WAS NOT DONE — Part 4, and why
|
||||
|
||||
**Neither demo box was reinstalled, and nothing was wiped.** I stopped before the destructive phase
|
||||
because the remaining session budget could not confidently cover a wipe **plus** its end-to-end proof
|
||||
**plus** the second box, and a half-reinstalled box is worse than an untouched one. Both demo boxes are
|
||||
running normally on the split layout.
|
||||
|
||||
**Consequently the golden is deliberately UNVOUCHED.** Vouching is what makes a fresh install pick a
|
||||
golden up; vouching one no box has been proven from would put an unproven disk layout in front of the
|
||||
next install anywhere. **Prove first, then vouch** (`CONTEXT.md` S-14). The golden being published but
|
||||
unvouched is a safe, reversible state — nothing consumes it.
|
||||
|
||||
**Nothing in the field is affected.** No box has the merged layout, nothing assumes it at runtime
|
||||
(§8.5), and agent v0.120.0 folds the retired grow so an older installer still produces the same
|
||||
capacity. **The work is filed as R-178** with the ordering (demo-hp first, proven, then demo-felhom).
|
||||
|
||||
## 10. Teardown — all three layers
|
||||
|
||||
| layer | evidence |
|
||||
| File | |
|
||||
|---|---|
|
||||
| **machine** | probe guests 9401/9402/9403 destroyed; P3 guest 9410 destroyed; golden build guest 9100 destroyed; P1's scratch 990000 torn down by the restore-test itself. `pct list` on both hosts shows only their own `9201` |
|
||||
| **host** | felhom-pve `local-lvm` 112398205 → 107204406 KiB (**5.19 GB returned**) after the probes; P3's archive freed with `pvesm free`; `/root/{build-golden.sh,.regcred.env,p2-probe.sh,p3-assertions.sh}` and `/tmp/p*.log` removed |
|
||||
| **hub** | **none created, verified not assumed** — no probe ran a controller or claimed a box. Registers hold the same 5 customers and 4 hosts as before. The reinstalls that would have produced NEW host records did not happen, so no old records needed disposing |
|
||||
| `scripts/release-agent.sh` | **new** — THE release path: build → tag → publish → verify by independent download |
|
||||
| `scripts/check-published-versions.py` | **new** — the R-115 gate |
|
||||
| `scripts/agent_gates.py` | registers the gate as **not `--fast`** (it needs network) |
|
||||
| `.gitea/workflows/gates.yml` | CI now runs the **full** gate set, not `--fast` |
|
||||
| `CLAUDE.md` | the raw `go build` line is replaced by the release script; a **Vouch** row replaces the old Publish row |
|
||||
|
||||
## 11. Register rows
|
||||
## Why
|
||||
|
||||
**Opened:** R-178. **Closed:** R-163 (the ceiling it records no longer exists), R-175 (§7.5 fixed in
|
||||
the same pass). **Updated:** R-165 → SHIPPED-not-yet-proven-live; R-176 → (a) answered by P1, (b)
|
||||
**withdrawn** because every node is reinstalled rather than migrated.
|
||||
Publishing was a step someone had to remember and was **forgotten three times in five days** —
|
||||
R-111's seventeen stranded releases, 0.114.0, and 0.120.0, which sat deployed on both demo hosts and
|
||||
undownloadable, so a documented-path reinstall would have silently downgraded them to the pre-merge
|
||||
agent **while reporting success**. R-111's own closing line named this leg and closed SHIPPED without
|
||||
it; it recurred the same afternoon. A note is not a mechanism.
|
||||
|
||||
## 12. CI and `--no-verify`
|
||||
The script also **tags**, because `felhom-host-install.sh` now fetches the agent's sixteen config
|
||||
files from `raw/tag/v<version>/` (R-183). A released version with no tag 404s a box mid-install, as
|
||||
root, on a virgin machine. Tag and package are two halves of one release.
|
||||
|
||||
**`--no-verify` NOT used** — every push ran `.githooks/pre-push` and passed. CI run ids are recorded
|
||||
in `felhom.eu/REPORT.md`.
|
||||
It **verifies by downloading what it just published** and comparing the sha to what it built. The
|
||||
publish step's own success is a report on its own write; a fetch returning the right bytes is a
|
||||
different claim, and it is the one that matters.
|
||||
|
||||
## 13. Observations — noticed, NOT acted on
|
||||
It **does not vouch** — that points machines at a version and stays the operator's act.
|
||||
|
||||
- **I gated a wait on `DONE. golden archive volid`, which the bake prints BEFORE publishing**, and
|
||||
briefly reported the publish as failed. It had not failed; a 649 MB upload was still running. No harm
|
||||
(the archive is on disk and the guest destroy did not touch it), but it is the same class as the
|
||||
truncated census: **a marker that is not a completion marker.** Worth a `[golden] publish OK` line at
|
||||
the true end of the script — not added here, out of scope.
|
||||
- The `felhom-golden` package listing endpoint returns a shape my query did not expect; the direct
|
||||
artifact URL is the reliable check.
|
||||
## The gate's invariant — not the one specified, and the reason was measured
|
||||
|
||||
The task's §8.4 asked for *"the version the hub tells machines to install must be downloadable"*.
|
||||
**CI cannot see that**, measured rather than assumed (P-C):
|
||||
|
||||
| Endpoint | Anonymous |
|
||||
|---|---|
|
||||
| Gitea package **download** | **200** (and **404** for a fake version — it discriminates) |
|
||||
| Gitea **tags** api | **200** |
|
||||
| Gitea package **listing** api | **401** — token required |
|
||||
| Hub `/api/v1/artifacts/<customer>` | **401** — per-customer passphrase required |
|
||||
|
||||
So a credential-free gate can ask *"is this version installable"* but not *"which version is
|
||||
vouched"*. Adding an operator credential to CI to close that is the operator's call, not a gate
|
||||
author's. The implemented invariant — **every `v<semver>` tag must have a downloadable package and a
|
||||
tag tree that serves the agent's configs** — needs no credential and **catches all three recorded
|
||||
instances**, because the release script creates the tag and publishes in one act.
|
||||
|
||||
**What it does not catch, stated rather than assumed away:** the hub vouching a version that was
|
||||
never released at all. Nothing here can see that; it belongs at vouch time in the hub. → **R-184**.
|
||||
|
||||
## Proof
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `go build ./... && go vet ./...` | OK |
|
||||
| `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
|
||||
| `agent_gates.py --fast` | `published` correctly **SKIPPED** — the pre-push hook must not fail because Gitea blinked |
|
||||
| `agent_gates.py` (full) | `reuse-refs` OK, `published` OK |
|
||||
| release script: re-release guard | `ERROR: tag v0.120.0 already exists — releasing over it would make one version name two binaries`, rc=1 |
|
||||
| release script: clean-tree guard | `ERROR: working tree is dirty — commit and push first`, rc=1 |
|
||||
|
||||
### Red-proof F — both directions
|
||||
|
||||
- **A tagged-but-unpublished version** (`v9.9.9` created for the purpose): gate **rc=1**,
|
||||
`binary NOT downloadable (HTTP 404 …)`. This is the R-115 shape exactly.
|
||||
- **The gate deregistered from the entry point**, same bad state: `agent_gates.py` → **rc=0, "all
|
||||
agent gates OK"**. Restored → **rc=1, CONVICTED: published**. The guard is what catches it, not
|
||||
something else.
|
||||
|
||||
`v9.9.9` was deleted afterwards; `git ls-remote --tags` shows only `v0.120.0`.
|
||||
|
||||
## Tag convention
|
||||
|
||||
`v<semver>`, at the commit the binary was built from. `v0.120.0` was created retroactively at
|
||||
`cd6e267` — the commit that produced the published binary (sha `a7763d31b55b5ce7…`). `configs/` is
|
||||
byte-identical between that commit and `main`, so nothing about the sixteen fetched files depends on
|
||||
the choice; `cd6e267` is tagged because it is the honest one.
|
||||
|
||||
Reference in New Issue
Block a user