From edac02ec9f33b225d661d361dacf6436bc39661f Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 21 Jul 2026 10:23:05 +0200 Subject: [PATCH] =?UTF-8?q?docs(report):=20TASK-B=20agent=20half=20?= =?UTF-8?q?=E2=80=94=20v0.91.2=20shipped=20+=20deployed;=20STOP-2/3=20need?= =?UTF-8?q?=20the=20operator?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- REPORT.md | 204 ++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 137 insertions(+), 67 deletions(-) diff --git a/REPORT.md b/REPORT.md index 8e46272..b180041 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,93 +1,163 @@ -# REPORT — the golden bakes EVERY infra image, asked from the controller (2026-07-19) +# REPORT — TASK-B: R-39 fleet fix + R-50b(a) · felhom-agent v0.90.0 → **v0.91.2** -**Scope: `configs/build-golden.sh` only (v2.0.0 → v2.1.0). No agent version bump — this is build -tooling, not the binary. Effective at the NEXT golden build; the current golden was deliberately NOT -rebuilt for this.** +**Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `8c55ac7` (clean, == `origin/main`). +Companion: hub **v0.68.0** (`felhom.eu`). -## The bug +## 0. Status -Enabling Megosztás on a fresh box pulled `felhom-samba` from the registry with **zero feedback** — -minutes of silent nothing. Observed live, twice. +| Leg | Status | +|---|---| +| Part 1 — hub v0.68.0 | **SHIPPED + DEPLOYED** (GitOps sync, Synced/Healthy) | +| Part 2 — agent v0.91.2 | **SHIPPED + PUBLISHED + DEPLOYED** to felhom-pve | +| **STOP-1** — felhom-pve update | **DONE + VERIFIED** (§5) | +| **STOP-2** — live re-issue | **BLOCKED: needs the operator.** I cannot authenticate to the hub UI — the operator password is a bcrypt hash I do not hold (verified: login → HTTP 401). | +| **STOP-3** — manifest save | **BLOCKED: same reason.** Values to record are in §7. | -The cause was not the pull. It was that the golden should already have carried the image, like the -other infra stacks, and did not. `build-golden.sh` held its own hand-maintained array of three image -tags, with a comment instructing the reader to keep it in sync with the controller's -`internal/infra` constants: +## 1. Probes -```bash -INFRA_IMAGES=( - "traefik:v3.6.7" - "cloudflare/cloudflared:2026.6.0" - "gtstef/filebrowser:1.3.3-stable" -) -``` +- **P1 — version stamping: PASS.** ldflags `-X main.version`; `publish-agent.sh` refuses to publish + when the binary's `--version` disagrees, so the stale `0.89.0` default cannot leak. +- **P2 — hub deploy path: PASS.** GitOps only: ArgoCD app `felhom`, auto-sync **off**, manifest bump + + deliberate sync. The `kubectl set image` line (in the old controller CLAUDE.md *and* printed by + `build.sh` itself) was **not** used. +- **P3 — felhom-pve before-state: CAPTURED** (§5 table). Notably the DR tier was **healthy**: a live + `GET /api2/json/version` with the stored credential returned **200**. This is a fix for a latent + defect, not a broken box. +- **P4 — fleet: ONE enrolled host** (`demo-felhom-8363b5`, agent 0.90.0). **Peti is not enrolled**, so + the MinAgent raise at STOP-3 cannot strand anyone. (Read from a temporary copy of the hub DB, which + carries live credentials — shredded immediately after each read.) -That comment is the whole failure mode. When `felhom-samba` was added as the fourth infra stack -(R-7 slice 1), the controller learned about it and this list did not. The golden baked **3 of 4**. +## 2. Two spec/reality mismatches, both resolved -## The fix — structural, not a fourth copy +**(a) The generation source does not exist as specified.** The brief said to return "the new row's id +(int64) … no schema change". `host_pbs_secrets` is `host_id PRIMARY KEY`, UPSERTed last-write-wins — +**there is no row id and no new row**, and `created_at` collides for two mints in one second. Added an +additive `generation` counter column using the repo's existing idempotent `ALTER TABLE` idiom. Same +semantics the brief wanted; different mechanism. -The list now comes from the controller image the bake has just pulled: +**(b) A second load-bearing fact the brief did not flag — checked, not trusted.** `Apply` bails out +entirely if the storage *status probe* errors, and converges via `adopt` **without consuming the +secret** if the storage reads active. So the whole fix depended on what PVE returns for a 401'd PBS +storage. Read PVE's own source: `storage_info` wraps `activate_storage` and `$plugin->status` in +`eval{}`, warns, and leaves the pre-initialised `active => 0`. **A 401 therefore returns HTTP 200 with +`active: 0`, never an API error** — `StorageActive` gives `(false, nil)` and `Apply` falls through to +verify → consume → reconcile. Scenario A's chain is sound *by proof*. The re-arm test encodes this +fixture shape explicitly so the assumption is visible. -```bash -docker run --rm --entrypoint /usr/local/bin/felhom-controller "$CONTROLLER_IMAGE" --print-infra-images -``` +## 3. What shipped (agent) -backed by `infra.Images()`, which derives from the pins themselves. The golden therefore bakes -exactly what **that** controller version will request, and the two cannot disagree by construction. -Adding a fifth infra stack is now two edits in one file (`internal/infra/infra.go`) and zero here. +`WirePBSDR.SecretGeneration` (field-exact with the hub) · wrapper `read` verb + **exactly one** +sudoers line + a `pbsdr-read` capability row · `pbs.ProbeAuth` + `ErrUnauthorized` · the probe on the +**15-minute collect path** feeding a loud `auth_failed` · `readPBSSecret` (direct read, wrapper +fallback) · wrapper sha256 in the report (R-50b(a)). -On the controller side a test parses the `const` block **out of the source** with `go/ast` and fails -if a `*Image` const exists that `Images()` does not return — a hand-written expected list would need -the same edit and would rot the same way. Red-proofed: removing `SambaImage` from `Images()` fails -it with the exact drift message. +**403 is deliberately NOT treated as unauthorized** — re-keying a too-narrow token would mint +credentials forever without fixing anything. **A transport error is UNKNOWN, never a rejection** — +otherwise every network blip burns a credential. -## The ordering bug this exposed +## 4. Tests + the four red-proofs -`docker logout` + `rm -f /root/.docker/config.json` ran **immediately after the controller pull**, -before the infra loop. That was fine while all three infra images were public Docker Hub images. -`felhom-samba` lives on `gitea.dooplex.hu` — the **same private registry as the controller** — so the -new loop would have `401`ed on it. +Full gate green in both repos at every commit. All red-proofs run at the **assertion** level. -The logout moved to **after** the loop, and gained a hard assertion that no credential remains in the -guest before it is archived: +| # | Red-proof | Result | +|---|---|---| +| A | Remove `SecretGeneration` from the agent struct | `consume calls=1, want 2` — the converged short-circuit fires; the July-18 defect exactly | +| B | Stop stamping the generation in `ReissuePBSDR` | descriptor **byte-identical** across a re-issue, block printed | +| D | Remove the id charset guard **and** the prefix assertion | the out-of-tree secret is **printed** (`rc=0 stdout="NOT-FOR-THE-AGENT-…"`) | +| E | Swallow the probe result in `NoteAuthResult` | `State:applied AuthFailed:false` — the July-18 shape | -```bash -pct exec "$VMID" -- bash -c '[ ! -s /root/.docker/config.json ]' \ - || { echo "[golden] FATAL: registry credential still present in the guest — refusing to archive"; exit 1; } -``` +**A finding from red-proof D worth keeping:** deleting the charset guard *alone* does **not** open a +hole — `readlink -f` plus the prefix assertion still catch it, so the naive single-guard red-proof +**passed vacuously**. The layering is real; the isolating proof had to remove both. A red-proof that +passes is not automatically reassuring. -The credential is still never baked; the guarantee is now checked rather than assumed. +The wrapper is tested as the artifact it is — executed under real `bash`. The two cases needing a +writable allowlisted dir run a **copy with only the prefix constant relocated** (and re-assert that +traversal is still refused in the copy), rather than `t.Skip` — a skipped test proves nothing. -## Fallback, loudly +## 5. STOP-1 — felhom-pve, before → after -A controller older than v0.147.0 has no `--print-infra-images`. The bake falls back to the historical -three-image list and prints three WARN lines saying felhom-samba will not be baked and Megosztás will -pull at runtime. The fallback **is** the drift-prone thing this change removes, so it announces -itself rather than passing silently. +| | before (P3) | after | +|---|---|---| +| agent | 0.90.0 · `18c48272…` | **0.91.2** · `34d309be…` | +| wrapper | `2888f2ea…` (the in-place July-18 hotfix) | `104db0a4…` (repo) | +| sudoers | `174125ef…` | new, +1 line | +| capabilities | 63 | **64/64 ok, 0 degraded, 0 inactive** | +| guest 9201 | running | running (untouched) | +| `wg-felhom` | handshake `…673` | fresh `…793` | -## Verification +Gates run before install: `bash -n` on the wrapper, `visudo -cf` on the staged sudoers, `visudo -c` +on the whole tree **after** install, and the staged binary's own `--version`. -- `bash -n configs/build-golden.sh` — clean. -- The seam was exercised against the **real published image** on the live demo guest: +**Rollback set retained on the host:** +`/usr/local/sbin/felhom-pbs-apply.bak-20260721-101635` (and the older `.bak-20260718-preR39`), +`/usr/local/bin/felhom-agent.bak-2026072{1-101635,1-101905,…}`, +`/root/felhom-agent.sudoers.bak-20260721-101635`. + +**The two live proofs that matter:** ``` -$ docker run --rm --entrypoint /usr/local/bin/felhom-controller \ - gitea.dooplex.hu/admin/felhom-controller:0.147.0 --print-infra-images -traefik:v3.6.7 -cloudflare/cloudflared:2026.6.0 -gtstef/filebrowser:1.3.3-stable -gitea.dooplex.hu/admin/felhom-samba:1.0.0 +# the NON-ROOT agent can finally read its own credential +sudo -u felhom-agent sudo -n felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage + rc=0 secret_len=37 stderr=(empty) + +# and the probe actually runs (debug enabled briefly, then restored to info) +level=DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite ``` - All four, from the image the golden would bake. The flag is config-free by design (no - `controller.yaml`, no data dir, no settings) precisely so a bare `docker run` can ask it. -- **Not run: a full golden bake.** The task scoped this to "effective at the next golden build, do - not rebuild the golden for this alone", so the loop itself has not executed end-to-end on a real - bake. The parts that could be verified without one were. +Report verified hub-side: `agent_version 0.91.1→0.91.2`, `wrapper_sha256 +104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3`. -## Follow-up +## 6. A defect I shipped and caught mid-STOP-1 -`documentation/backlog/ROADMAP.md` pre-invite checklist records that **golden ≥ 0.147.x** carries all -four infra images. Until the next bake, a fresh box still pulls felhom-samba at enable time — which -controller v0.147.0's progress card now at least explains rather than leaving silent. +**v0.91.0 shipped the auth-honesty leg INERT.** I built the `AuthSink` seam and the `NoteAuthResult` +consumer, and `main.go` never called `SetAuthSink`. The reporter deliberately skips probing with no +sink attached, so there was no probe, no `auth_failed`, no self-heal — and **nothing failed**, because +every unit test injected the sink directly. + +Caught during STOP-1 by checking the wiring instead of trusting it. **This is the same class as the +controller v0.154.0 defect the day before: a table test over a seam proves the seam, not the caller.** +Fixed in v0.91.1; the published 0.91.0 artifact was **superseded, not overwritten** (a published +version stays immutable). + +**v0.91.2** then made a healthy probe observable at Debug — it previously logged only on failure, so +"no `auth_failed`" was indistinguishable from "never probed". That silence is exactly how v0.91.0 +shipped inert unnoticed. + +## 7. For the operator — STOP-2 and STOP-3 + +**STOP-2 (one click):** press **Re-issue PBS credentials** for the demo customer. Expected chain: +fresh secret row → `secret_generation` **0 → 1** in the descriptor (the current live descriptor has no +such key at all) → poke → agent re-applies with **no short-circuit** → fresh secret consumed +(`consumed_at` stamped) → reconcile rc-0 → probe 200 → tier `active`. **The July-18 negative — the +same click doing nothing — is the historical red-proof this run is measured against.** + +**STOP-3 (manifest save):** + +| field | value | +|---|---| +| Agent version | `0.91.2` | +| Agent sha256 | `34d309be429473f3f0ab34e3185e17b22463a341b30bf46e162306bff4aec22a` | +| **PBS wrapper sha256** (new field) | `104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3` | +| MinAgent | `0.91.2` — safe now: **P4 shows one enrolled host and it already runs 0.91.2** | + +Superseded, do not vouch: **0.91.0** (inert leg) and **0.91.1**. `0.90.1` correctly remains **404**. + +## 8. Residual + +- **R-50b(b)/(c) NOT solved** — the wrapper is still fetched from `raw/branch/main`, unversioned and + unpinned. This release makes drift *visible*; it does not fix the channel. +- The **0440 sudoers file is not agent-readable**, so its drift stays invisible — recorded, not solved. +- R-39's own residual: the verify loop reads via the wrapper now, but a host whose wrapper predates + v0.91.0 has no `read` verb and degrades to the old skip (loudly, via the `pbsdr-read` capability). + +## 9. Observations + +- **The hub's build script prints `kubectl set image` as its deploy hint**, which contradicts the + GitOps rule in `felhom.eu/CLAUDE.md`. It is a trap for exactly the kind of session that trusts tool + output over the runbook. Worth fixing in `build.sh`. +- **The July-18 hotfixed wrapper hashed `2888f2ea…`, matching no commit anyone could name.** That is + R-50b in one line, and it is why the manifest field earns its place. +- `pvesm config` does not exist (it is `pvesm status` / `/etc/pve/storage.cfg`); PVE's per-storage + status endpoint returns `active: 0` rather than erroring — both worth knowing before designing + against them.