docs(report): golden bakes every infra image (build tooling v2.1.0)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
This commit is contained in:
@@ -1,116 +1,93 @@
|
|||||||
# REPORT — felhom-agent v0.90.1 (R-39 hotfix: PBS reconcile must not pass `--server`)
|
# REPORT — the golden bakes EVERY infra image, asked from the controller (2026-07-19)
|
||||||
|
|
||||||
**Date:** 2026-07-18 · **Baseline:** v0.90.0 → **v0.90.1** · **Scope:** config-only.
|
**Scope: `configs/build-golden.sh` only (v2.0.0 → v2.1.0). No agent version bump — this is build
|
||||||
`configs/felhom-pbs-apply` (one argv line + its WHY) and one red-proof test. **The Go binary is
|
tooling, not the binary. Effective at the NEXT golden build; the current golden was deliberately NOT
|
||||||
unchanged** — this ships with the next agent deploy as a config artifact.
|
rebuilt for this.**
|
||||||
**Green:** `go build ./... && go vet ./... && go test ./...` all pass.
|
|
||||||
|
|
||||||
## What was wrong
|
## The bug
|
||||||
|
|
||||||
`reconcile` built its argv as `args=(--server "$server" --fingerprint "$fp")`. PVE treats a PBS
|
Enabling Megosztás on a fresh box pulled `felhom-samba` from the registry with **zero feedback** —
|
||||||
storage's `server` as a **create-only** parameter and rejects the whole `pvesm set` call —
|
minutes of silent nothing. Observed live, twice.
|
||||||
`can't change value of fixed parameter 'server'` — **even when the value is byte-identical to the
|
|
||||||
stored one**. `reconcile` could therefore never succeed against an existing entry.
|
|
||||||
|
|
||||||
That is severe rather than cosmetic because **the agent consumes the hub's one-time PBS token secret
|
The cause was not the pull. It was that the golden should already have carried the image, like the
|
||||||
before invoking the wrapper**. Every failed reconcile burned a credential:
|
other infra stacks, and did not. `build-golden.sh` held its own hand-maintained array of three image
|
||||||
|
tags, with a comment instructing the reader to keep it in sync with the controller's
|
||||||
|
`internal/infra` constants:
|
||||||
|
|
||||||
```
|
```bash
|
||||||
hub "Re-issue PBS credentials" → fresh one-time secret minted
|
INFRA_IMAGES=(
|
||||||
agent → secret CONSUMED (single-use, now spent)
|
"traefik:v3.6.7"
|
||||||
wrapper reconcile → exit 255, "fixed parameter 'server'"
|
"cloudflare/cloudflared:2026.6.0"
|
||||||
storage entry → still pinned to the REVOKED secret → 401 forever
|
"gtstef/filebrowser:1.3.3-stable"
|
||||||
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
## How it was found and proven
|
That comment is the whole failure mode. When `felhom-samba` was added as the fourth infra stack
|
||||||
|
(R-7 slice 1), the controller learned about it and this list did not. The golden baked **3 of 4**.
|
||||||
|
|
||||||
Live-diagnosed on the N100 demo host during the 2026-07-18 rehearsal wrap
|
## The fix — structural, not a fourth copy
|
||||||
(`felhom.eu/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md` finding F2, ROADMAP
|
|
||||||
**R-39**). The box had been reporting `pbsdr: converged state=applied` while `pvesm status` returned
|
|
||||||
`401 Unauthorized` / `inactive`.
|
|
||||||
|
|
||||||
Proven directly on the live entry before any code was written:
|
The list now comes from the controller image the bake has just pulled:
|
||||||
|
|
||||||
| Probe | Result |
|
```bash
|
||||||
|---|---|
|
docker run --rm --entrypoint /usr/local/bin/felhom-controller "$CONTROLLER_IMAGE" --print-infra-images
|
||||||
| `pvesm set felhom-pbs --server <same> --fingerprint <same>` | **rejected** — `can't change value of fixed parameter 'server'` |
|
```
|
||||||
| `pvesm set felhom-pbs --fingerprint <same>` | **rc 0 — accepted** |
|
|
||||||
|
|
||||||
The encryption key `K` (`<id>.enc`) and the `.pw` store were verified byte-untouched after the
|
backed by `infra.Images()`, which derives from the pins themselves. The golden therefore bakes
|
||||||
rejected call: PVE rejects atomically, so the set-only law held throughout.
|
exactly what **that** controller version will request, and the two cannot disagree by construction.
|
||||||
|
Adding a fifth infra stack is now two edits in one file (`internal/infra/infra.go`) and zero here.
|
||||||
|
|
||||||
## The fix
|
On the controller side a test parses the `const` block **out of the source** with `go/ast` and fails
|
||||||
|
if a `*Image` const exists that `Images()` does not return — a hand-written expected list would need
|
||||||
|
the same edit and would rot the same way. Red-proofed: removing `SambaImage` from `Images()` fails
|
||||||
|
it with the exact drift message.
|
||||||
|
|
||||||
Drop `--server` from the reconcile argv. The server address is immutable by construction —
|
## The ordering bug this exposed
|
||||||
relocating a PBS endpoint requires a fresh `create` — so there was never anything for `reconcile` to
|
|
||||||
reconcile there. `--fingerprint` (and `--password` when a secret is fed on stdin) remain, which is
|
|
||||||
the mutable identity the verb exists to push.
|
|
||||||
|
|
||||||
## Red-proof
|
`docker logout` + `rm -f /root/.docker/config.json` ran **immediately after the controller pull**,
|
||||||
|
before the infra loop. That was fine while all three infra images were public Docker Hub images.
|
||||||
|
`felhom-samba` lives on `gitea.dooplex.hu` — the **same private registry as the controller** — so the
|
||||||
|
new loop would have `401`ed on it.
|
||||||
|
|
||||||
`TestReconcileNeverPassesServerToPvesmSet` (`internal/pbsdr/manager_test.go`) isolates the
|
The logout moved to **after** the loop, and gained a hard assertion that no credential remains in the
|
||||||
`reconcile)` block from the shipped wrapper and asserts no `--server` reaches `pvesm set`, plus that
|
guest before it is archived:
|
||||||
`--fingerprint` is still pushed so the verb cannot be hollowed out. **Verified RED against the
|
|
||||||
unfixed wrapper and GREEN after the fix.**
|
|
||||||
|
|
||||||
Two traps the proof handles explicitly, both of which would have made it pass vacuously — and both
|
```bash
|
||||||
of which actually fired during authoring, which is the argument for running a proof red first:
|
pct exec "$VMID" -- bash -c '[ ! -s /root/.docker/config.json ]' \
|
||||||
- **Line endings.** This repo is cloned on Windows; the working copy carries CRLF, so an `\n`-only
|
|| { echo "[golden] FATAL: registry credential still present in the guest — refusing to archive"; exit 1; }
|
||||||
pattern matches nothing and the guard passes silently. The pattern is `\r?\n` throughout. The
|
```
|
||||||
first run failed with "could not locate the reconcile) block" — the vacuous-pass failure mode,
|
|
||||||
caught only because the test was run against the broken wrapper first.
|
|
||||||
- **Comments.** The WHY note above the fix necessarily quotes `--server`, the very flag the test
|
|
||||||
forbids, so the suite went red *after* the fix was applied. Comment lines are now stripped before
|
|
||||||
matching.
|
|
||||||
|
|
||||||
## Deliberately NOT fixed here — each still open
|
The credential is still never baked; the guarantee is now checked rather than assumed.
|
||||||
|
|
||||||
1. **R-39's primary half.** The agent re-applies on a change of the **descriptor hash**
|
## Fallback, loudly
|
||||||
(`internal/pbsdr/manager.go` ~L235:
|
|
||||||
`if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { return }`).
|
|
||||||
A hub credential re-issue leaves the descriptor **byte-identical** — same `token_id`, same
|
|
||||||
`fingerprint`; only the side-table secret rotates — and bumps only `desired_generation`. So a
|
|
||||||
converged agent short-circuits and never consumes the fresh secret. This wrapper fix makes the
|
|
||||||
apply *succeed* once the agent is made to re-apply; **it does not make it re-apply.**
|
|
||||||
2. **The verify-loop read.** `pbs: cannot read token secret … permission denied` — the non-root
|
|
||||||
agent reads `/etc/pve/priv/storage/<id>.pw` **directly**, a path it can only ever *write* through
|
|
||||||
the root wrapper. `/etc/pve/priv` is `0700 root:www-data` and sudoers exposes only
|
|
||||||
`create|reconcile|grant` — **there is no read verb**. The loop is permanently blind to exactly
|
|
||||||
the failure it exists to catch.
|
|
||||||
|
|
||||||
Both ride the spec'd R-39 agent train. Recorded, not improvised.
|
A controller older than v0.147.0 has no `--print-infra-images`. The bake falls back to the historical
|
||||||
|
three-image list and prints three WARN lines saying felhom-samba will not be baked and Megosztás will
|
||||||
|
pull at runtime. The fallback **is** the drift-prone thing this change removes, so it announces
|
||||||
|
itself rather than passing silently.
|
||||||
|
|
||||||
## The demo box — HEALED and proven (2026-07-18)
|
## Verification
|
||||||
|
|
||||||
The wrapper is hotfixed on the N100 (`/usr/local/sbin/felhom-pbs-apply`, backup kept at
|
- `bash -n configs/build-golden.sh` — clean.
|
||||||
`.bak-20260718-preR39`, `bash -n` clean) so the repo and the box agree.
|
- The seam was exercised against the **real published image** on the live demo guest:
|
||||||
|
|
||||||
Diagnosis had consumed the one pending secret against the still-unfixed wrapper; the agent parked
|
```
|
||||||
correctly in `consumed-failed.json` with `NOT retrying silently` — **no burn loop**, the fail-safe
|
$ docker run --rm --entrypoint /usr/local/bin/felhom-controller \
|
||||||
worked. Viktor then clicked **"Re-issue PBS credentials"** (the one password-gated action CC cannot
|
gitea.dooplex.hu/admin/felhom-controller:0.147.0 --print-infra-images
|
||||||
perform) and the whole chain closed itself in **9 seconds**:
|
traefik:v3.6.7
|
||||||
|
cloudflare/cloudflared:2026.6.0
|
||||||
|
gtstef/filebrowser:1.3.3-stable
|
||||||
|
gitea.dooplex.hu/admin/felhom-samba:1.0.0
|
||||||
|
```
|
||||||
|
|
||||||
| Time (CEST) | Event |
|
All four, from the image the golden would bake. The flag is config-free by design (no
|
||||||
|---|---|
|
`controller.yaml`, no data dir, no settings) precisely so a bare `docker run` can ask it.
|
||||||
| 20:28:44 | hub: `tenantsync: reissue ok` → `pbsdr credentials re-issued … fresh consume-once secret stored` |
|
- **Not run: a full golden bake.** The task scoped this to "effective at the next golden build, do
|
||||||
| 20:28:51 | agent: `one-time token secret consumed`, wrapper `reconcile` invoked — **now the patched one** |
|
not rebuild the golden for this alone", so the loop itself has not executed end-to-end on a real
|
||||||
| 20:28:53 | agent: **`pbsdr: converged state=applied`** — no CONSUMED-BUT-FAILED this time |
|
bake. The parts that could be verified without one were.
|
||||||
|
|
||||||
`consumed-failed.json` was cleared automatically by `finishConverged` and `marker.json` rewritten.
|
## Follow-up
|
||||||
The agent picked the re-issue up unaided because `marker.json` was absent, so the L235 hash
|
|
||||||
short-circuit did not apply.
|
|
||||||
|
|
||||||
**Proof the tier is genuinely alive, not merely reported alive:**
|
`documentation/backlog/ROADMAP.md` pre-invite checklist records that **golden ≥ 0.147.x** carries all
|
||||||
|
four infra images. Until the next bake, a fresh box still pulls felhom-samba at enable time — which
|
||||||
| Check | Before | After |
|
controller v0.147.0's progress card now at least explains rather than leaving silent.
|
||||||
|---|---|---|
|
|
||||||
| `pvesm status` | `401 Unauthorized` / `inactive` | **`active`** |
|
|
||||||
| Direct token probe, `/api2/json/version` | `401` | **`200`** |
|
|
||||||
| Real backup | none possible | **`felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z`, 9 744 319 312 B, finished in 13m36s** |
|
|
||||||
|
|
||||||
The backup is encrypted under key fingerprint `7e:a6:af:f7:ea:6d:3e:d9` — the **escrowed** key, i.e.
|
|
||||||
the one customer zero holds the recovery code for. This is the DR tier's **first real backup on the
|
|
||||||
reborn box**.
|
|
||||||
|
|
||||||
Nothing was destroyed at any point: `.pw`, `.enc` (K), and the `storage.cfg` entry were verified
|
|
||||||
intact after the rejected `pvesm set` (PVE rejects atomically, so the set-only law held).
|
|
||||||
|
|||||||
Reference in New Issue
Block a user