docs(report): golden bakes every infra image (build tooling v2.1.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
This commit is contained in:
2026-07-19 10:01:11 +02:00
parent 25c30ba625
commit 8fc44d8d9e
+68 -91
View File
@@ -1,116 +1,93 @@
# REPORT — felhom-agent v0.90.1 (R-39 hotfix: PBS reconcile must not pass `--server`)
# REPORT — the golden bakes EVERY infra image, asked from the controller (2026-07-19)
**Date:** 2026-07-18 · **Baseline:** v0.90.0 → **v0.90.1** · **Scope:** config-only.
`configs/felhom-pbs-apply` (one argv line + its WHY) and one red-proof test. **The Go binary is
unchanged** — this ships with the next agent deploy as a config artifact.
**Green:** `go build ./... && go vet ./... && go test ./...` all pass.
**Scope: `configs/build-golden.sh` only (v2.0.0 → v2.1.0). No agent version bump — this is build
tooling, not the binary. Effective at the NEXT golden build; the current golden was deliberately NOT
rebuilt for this.**
## What was wrong
## The bug
`reconcile` built its argv as `args=(--server "$server" --fingerprint "$fp")`. PVE treats a PBS
storage's `server` as a **create-only** parameter and rejects the whole `pvesm set` call —
`can't change value of fixed parameter 'server'` — **even when the value is byte-identical to the
stored one**. `reconcile` could therefore never succeed against an existing entry.
Enabling Megosztás on a fresh box pulled `felhom-samba` from the registry with **zero feedback**
minutes of silent nothing. Observed live, twice.
That is severe rather than cosmetic because **the agent consumes the hub's one-time PBS token secret
before invoking the wrapper**. Every failed reconcile burned a credential:
The cause was not the pull. It was that the golden should already have carried the image, like the
other infra stacks, and did not. `build-golden.sh` held its own hand-maintained array of three image
tags, with a comment instructing the reader to keep it in sync with the controller's
`internal/infra` constants:
```
hub "Re-issue PBS credentials" → fresh one-time secret minted
agent → secret CONSUMED (single-use, now spent)
wrapper reconcile → exit 255, "fixed parameter 'server'"
storage entry → still pinned to the REVOKED secret → 401 forever
```bash
INFRA_IMAGES=(
"traefik:v3.6.7"
"cloudflare/cloudflared:2026.6.0"
"gtstef/filebrowser:1.3.3-stable"
)
```
## How it was found and proven
That comment is the whole failure mode. When `felhom-samba` was added as the fourth infra stack
(R-7 slice 1), the controller learned about it and this list did not. The golden baked **3 of 4**.
Live-diagnosed on the N100 demo host during the 2026-07-18 rehearsal wrap
(`felhom.eu/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md` finding F2, ROADMAP
**R-39**). The box had been reporting `pbsdr: converged state=applied` while `pvesm status` returned
`401 Unauthorized` / `inactive`.
## The fix — structural, not a fourth copy
Proven directly on the live entry before any code was written:
The list now comes from the controller image the bake has just pulled:
| Probe | Result |
|---|---|
| `pvesm set felhom-pbs --server <same> --fingerprint <same>` | **rejected**`can't change value of fixed parameter 'server'` |
| `pvesm set felhom-pbs --fingerprint <same>` | **rc 0 — accepted** |
```bash
docker run --rm --entrypoint /usr/local/bin/felhom-controller "$CONTROLLER_IMAGE" --print-infra-images
```
The encryption key `K` (`<id>.enc`) and the `.pw` store were verified byte-untouched after the
rejected call: PVE rejects atomically, so the set-only law held throughout.
backed by `infra.Images()`, which derives from the pins themselves. The golden therefore bakes
exactly what **that** controller version will request, and the two cannot disagree by construction.
Adding a fifth infra stack is now two edits in one file (`internal/infra/infra.go`) and zero here.
## The fix
On the controller side a test parses the `const` block **out of the source** with `go/ast` and fails
if a `*Image` const exists that `Images()` does not return — a hand-written expected list would need
the same edit and would rot the same way. Red-proofed: removing `SambaImage` from `Images()` fails
it with the exact drift message.
Drop `--server` from the reconcile argv. The server address is immutable by construction —
relocating a PBS endpoint requires a fresh `create` — so there was never anything for `reconcile` to
reconcile there. `--fingerprint` (and `--password` when a secret is fed on stdin) remain, which is
the mutable identity the verb exists to push.
## The ordering bug this exposed
## Red-proof
`docker logout` + `rm -f /root/.docker/config.json` ran **immediately after the controller pull**,
before the infra loop. That was fine while all three infra images were public Docker Hub images.
`felhom-samba` lives on `gitea.dooplex.hu` — the **same private registry as the controller** — so the
new loop would have `401`ed on it.
`TestReconcileNeverPassesServerToPvesmSet` (`internal/pbsdr/manager_test.go`) isolates the
`reconcile)` block from the shipped wrapper and asserts no `--server` reaches `pvesm set`, plus that
`--fingerprint` is still pushed so the verb cannot be hollowed out. **Verified RED against the
unfixed wrapper and GREEN after the fix.**
The logout moved to **after** the loop, and gained a hard assertion that no credential remains in the
guest before it is archived:
Two traps the proof handles explicitly, both of which would have made it pass vacuously — and both
of which actually fired during authoring, which is the argument for running a proof red first:
- **Line endings.** This repo is cloned on Windows; the working copy carries CRLF, so an `\n`-only
pattern matches nothing and the guard passes silently. The pattern is `\r?\n` throughout. The
first run failed with "could not locate the reconcile) block" — the vacuous-pass failure mode,
caught only because the test was run against the broken wrapper first.
- **Comments.** The WHY note above the fix necessarily quotes `--server`, the very flag the test
forbids, so the suite went red *after* the fix was applied. Comment lines are now stripped before
matching.
```bash
pct exec "$VMID" -- bash -c '[ ! -s /root/.docker/config.json ]' \
|| { echo "[golden] FATAL: registry credential still present in the guest — refusing to archive"; exit 1; }
```
## Deliberately NOT fixed here — each still open
The credential is still never baked; the guarantee is now checked rather than assumed.
1. **R-39's primary half.** The agent re-applies on a change of the **descriptor hash**
(`internal/pbsdr/manager.go` ~L235:
`if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { return }`).
A hub credential re-issue leaves the descriptor **byte-identical** — same `token_id`, same
`fingerprint`; only the side-table secret rotates — and bumps only `desired_generation`. So a
converged agent short-circuits and never consumes the fresh secret. This wrapper fix makes the
apply *succeed* once the agent is made to re-apply; **it does not make it re-apply.**
2. **The verify-loop read.** `pbs: cannot read token secret … permission denied` — the non-root
agent reads `/etc/pve/priv/storage/<id>.pw` **directly**, a path it can only ever *write* through
the root wrapper. `/etc/pve/priv` is `0700 root:www-data` and sudoers exposes only
`create|reconcile|grant`**there is no read verb**. The loop is permanently blind to exactly
the failure it exists to catch.
## Fallback, loudly
Both ride the spec'd R-39 agent train. Recorded, not improvised.
A controller older than v0.147.0 has no `--print-infra-images`. The bake falls back to the historical
three-image list and prints three WARN lines saying felhom-samba will not be baked and Megosztás will
pull at runtime. The fallback **is** the drift-prone thing this change removes, so it announces
itself rather than passing silently.
## The demo box — HEALED and proven (2026-07-18)
## Verification
The wrapper is hotfixed on the N100 (`/usr/local/sbin/felhom-pbs-apply`, backup kept at
`.bak-20260718-preR39`, `bash -n` clean) so the repo and the box agree.
- `bash -n configs/build-golden.sh` — clean.
- The seam was exercised against the **real published image** on the live demo guest:
Diagnosis had consumed the one pending secret against the still-unfixed wrapper; the agent parked
correctly in `consumed-failed.json` with `NOT retrying silently`**no burn loop**, the fail-safe
worked. Viktor then clicked **"Re-issue PBS credentials"** (the one password-gated action CC cannot
perform) and the whole chain closed itself in **9 seconds**:
```
$ docker run --rm --entrypoint /usr/local/bin/felhom-controller \
gitea.dooplex.hu/admin/felhom-controller:0.147.0 --print-infra-images
traefik:v3.6.7
cloudflare/cloudflared:2026.6.0
gtstef/filebrowser:1.3.3-stable
gitea.dooplex.hu/admin/felhom-samba:1.0.0
```
| Time (CEST) | Event |
|---|---|
| 20:28:44 | hub: `tenantsync: reissue ok``pbsdr credentials re-issued … fresh consume-once secret stored` |
| 20:28:51 | agent: `one-time token secret consumed`, wrapper `reconcile` invoked — **now the patched one** |
| 20:28:53 | agent: **`pbsdr: converged state=applied`** — no CONSUMED-BUT-FAILED this time |
All four, from the image the golden would bake. The flag is config-free by design (no
`controller.yaml`, no data dir, no settings) precisely so a bare `docker run` can ask it.
- **Not run: a full golden bake.** The task scoped this to "effective at the next golden build, do
not rebuild the golden for this alone", so the loop itself has not executed end-to-end on a real
bake. The parts that could be verified without one were.
`consumed-failed.json` was cleared automatically by `finishConverged` and `marker.json` rewritten.
The agent picked the re-issue up unaided because `marker.json` was absent, so the L235 hash
short-circuit did not apply.
## Follow-up
**Proof the tier is genuinely alive, not merely reported alive:**
| Check | Before | After |
|---|---|---|
| `pvesm status` | `401 Unauthorized` / `inactive` | **`active`** |
| Direct token probe, `/api2/json/version` | `401` | **`200`** |
| Real backup | none possible | **`felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z`, 9 744 319 312 B, finished in 13m36s** |
The backup is encrypted under key fingerprint `7e:a6:af:f7:ea:6d:3e:d9` — the **escrowed** key, i.e.
the one customer zero holds the recovery code for. This is the DR tier's **first real backup on the
reborn box**.
Nothing was destroyed at any point: `.pw`, `.enc` (K), and the `storage.cfg` entry were verified
intact after the rejected `pvesm set` (PVE rejects atomically, so the set-only law held).
`documentation/backlog/ROADMAP.md` pre-invite checklist records that **golden ≥ 0.147.x** carries all
four infra images. Until the next bake, a fresh box still pulls felhom-samba at enable time — which
controller v0.147.0's progress card now at least explains rather than leaving silent.