admin/felhom-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(report): TASK-B agent half — v0.91.2 shipped + deployed; STOP-2/3 need the operator

Browse Source
This commit is contained in:
authentik Default Admin
2026-07-21 10:23:05 +02:00
parent d4a7a5bad3
commit edac02ec9f
1 changed files with 137 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+137 -67
View File
@@ -1,93 +1,163 @@
# REPORT — the golden bakes EVERY infra image, asked from the controller (2026-07-19)
# REPORT — TASK-B: R-39 fleet fix + R-50b(a) · felhom-agent v0.90.0 → **v0.91.2**
**Scope: `configs/build-golden.sh` only (v2.0.0 → v2.1.0). No agent version bump — this is build
tooling, not the binary. Effective at the NEXT golden build; the current golden was deliberately NOT
rebuilt for this.**
**Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `8c55ac7` (clean, == `origin/main`).
Companion: hub **v0.68.0** (`felhom.eu`).
## The bug
## 0. Status
Enabling Megosztás on a fresh box pulled `felhom-samba` from the registry with **zero feedback** —
minutes of silent nothing. Observed live, twice.
| Leg | Status |
|---|---|
| Part 1 — hub v0.68.0 | **SHIPPED + DEPLOYED** (GitOps sync, Synced/Healthy) |
| Part 2 — agent v0.91.2 | **SHIPPED + PUBLISHED + DEPLOYED** to felhom-pve |
| **STOP-1** — felhom-pve update | **DONE + VERIFIED** (§5) |
| **STOP-2** — live re-issue | **BLOCKED: needs the operator.** I cannot authenticate to the hub UI — the operator password is a bcrypt hash I do not hold (verified: login → HTTP 401). |
| **STOP-3** — manifest save | **BLOCKED: same reason.** Values to record are in §7. |
The cause was not the pull. It was that the golden should already have carried the image, like the
other infra stacks, and did not. `build-golden.sh` held its own hand-maintained array of three image
tags, with a comment instructing the reader to keep it in sync with the controller's
`internal/infra` constants:
## 1. Probes
```bash
INFRA_IMAGES=(
"traefik:v3.6.7"
"cloudflare/cloudflared:2026.6.0"
"gtstef/filebrowser:1.3.3-stable"
)
```
- **P1 — version stamping: PASS.** ldflags `-X main.version`; `publish-agent.sh` refuses to publish
when the binary's `--version` disagrees, so the stale `0.89.0` default cannot leak.
- **P2 — hub deploy path: PASS.** GitOps only: ArgoCD app `felhom`, auto-sync **off**, manifest bump
+ deliberate sync. The `kubectl set image` line (in the old controller CLAUDE.md *and* printed by
`build.sh` itself) was **not** used.
- **P3 — felhom-pve before-state: CAPTURED** (§5 table). Notably the DR tier was **healthy**: a live
`GET /api2/json/version` with the stored credential returned **200**. This is a fix for a latent
defect, not a broken box.
- **P4 — fleet: ONE enrolled host** (`demo-felhom-8363b5`, agent 0.90.0). **Peti is not enrolled**, so
the MinAgent raise at STOP-3 cannot strand anyone. (Read from a temporary copy of the hub DB, which
carries live credentials — shredded immediately after each read.)
That comment is the whole failure mode. When `felhom-samba` was added as the fourth infra stack
(R-7 slice 1), the controller learned about it and this list did not. The golden baked **3 of 4**.
## 2. Two spec/reality mismatches, both resolved
## The fix — structural, not a fourth copy
**(a) The generation source does not exist as specified.** The brief said to return "the new row's id
(int64) … no schema change". `host_pbs_secrets` is `host_id PRIMARY KEY`, UPSERTed last-write-wins —
**there is no row id and no new row**, and `created_at` collides for two mints in one second. Added an
additive `generation` counter column using the repo's existing idempotent `ALTER TABLE` idiom. Same
semantics the brief wanted; different mechanism.
The list now comes from the controller image the bake has just pulled:
**(b) A second load-bearing fact the brief did not flag — checked, not trusted.** `Apply` bails out
entirely if the storage *status probe* errors, and converges via `adopt` **without consuming the
secret** if the storage reads active. So the whole fix depended on what PVE returns for a 401'd PBS
storage. Read PVE's own source: `storage_info` wraps `activate_storage` and `$plugin->status` in
`eval{}`, warns, and leaves the pre-initialised `active => 0`. **A 401 therefore returns HTTP 200 with
`active: 0`, never an API error** — `StorageActive` gives `(false, nil)` and `Apply` falls through to
verify → consume → reconcile. Scenario A's chain is sound *by proof*. The re-arm test encodes this
fixture shape explicitly so the assumption is visible.
```bash
docker run --rm --entrypoint /usr/local/bin/felhom-controller "$CONTROLLER_IMAGE" --print-infra-images
```
## 3. What shipped (agent)
backed by `infra.Images()`, which derives from the pins themselves. The golden therefore bakes
exactly what **that** controller version will request, and the two cannot disagree by construction.
Adding a fifth infra stack is now two edits in one file (`internal/infra/infra.go`) and zero here.
`WirePBSDR.SecretGeneration` (field-exact with the hub) · wrapper `read` verb + **exactly one**
sudoers line + a `pbsdr-read` capability row · `pbs.ProbeAuth` + `ErrUnauthorized` · the probe on the
**15-minute collect path** feeding a loud `auth_failed` · `readPBSSecret` (direct read, wrapper
fallback) · wrapper sha256 in the report (R-50b(a)).
On the controller side a test parses the `const` block **out of the source** with `go/ast` and fails
if a `*Image` const exists that `Images()` does not return — a hand-written expected list would need
the same edit and would rot the same way. Red-proofed: removing `SambaImage` from `Images()` fails
it with the exact drift message.
**403 is deliberately NOT treated as unauthorized** — re-keying a too-narrow token would mint
credentials forever without fixing anything. **A transport error is UNKNOWN, never a rejection** —
otherwise every network blip burns a credential.
## The ordering bug this exposed
## 4. Tests + the four red-proofs
`docker logout` + `rm -f /root/.docker/config.json` ran **immediately after the controller pull**,
before the infra loop. That was fine while all three infra images were public Docker Hub images.
`felhom-samba` lives on `gitea.dooplex.hu` — the **same private registry as the controller** — so the
new loop would have `401`ed on it.
Full gate green in both repos at every commit. All red-proofs run at the **assertion** level.
The logout moved to **after** the loop, and gained a hard assertion that no credential remains in the
guest before it is archived:
| # | Red-proof | Result |
|---|---|---|
| A | Remove `SecretGeneration` from the agent struct | `consume calls=1, want 2` — the converged short-circuit fires; the July-18 defect exactly |
| B | Stop stamping the generation in `ReissuePBSDR` | descriptor **byte-identical** across a re-issue, block printed |
| D | Remove the id charset guard **and** the prefix assertion | the out-of-tree secret is **printed** (`rc=0 stdout="NOT-FOR-THE-AGENT-…"`) |
| E | Swallow the probe result in `NoteAuthResult` | `State:applied AuthFailed:false` — the July-18 shape |
```bash
pct exec "$VMID" -- bash -c '[ ! -s /root/.docker/config.json ]' \
|| { echo "[golden] FATAL: registry credential still present in the guest — refusing to archive"; exit 1; }
```
**A finding from red-proof D worth keeping:** deleting the charset guard *alone* does **not** open a
hole — `readlink -f` plus the prefix assertion still catch it, so the naive single-guard red-proof
**passed vacuously**. The layering is real; the isolating proof had to remove both. A red-proof that
passes is not automatically reassuring.
The credential is still never baked; the guarantee is now checked rather than assumed.
The wrapper is tested as the artifact it is — executed under real `bash`. The two cases needing a
writable allowlisted dir run a **copy with only the prefix constant relocated** (and re-assert that
traversal is still refused in the copy), rather than `t.Skip` — a skipped test proves nothing.
## Fallback, loudly
## 5. STOP-1 — felhom-pve, before → after
A controller older than v0.147.0 has no `--print-infra-images`. The bake falls back to the historical
three-image list and prints three WARN lines saying felhom-samba will not be baked and Megosztás will
pull at runtime. The fallback **is** the drift-prone thing this change removes, so it announces
itself rather than passing silently.
| | before (P3) | after |
|---|---|---|
| agent | 0.90.0 · `18c48272…` | **0.91.2** · `34d309be…` |
| wrapper | `2888f2ea…` (the in-place July-18 hotfix) | `104db0a4…` (repo) |
| sudoers | `174125ef…` | new, +1 line |
| capabilities | 63 | **64/64 ok, 0 degraded, 0 inactive** |
| guest 9201 | running | running (untouched) |
| `wg-felhom` | handshake `…673` | fresh `…793` |
## Verification
Gates run before install: `bash -n` on the wrapper, `visudo -cf` on the staged sudoers, `visudo -c`
on the whole tree **after** install, and the staged binary's own `--version`.
- `bash -n configs/build-golden.sh` — clean.
- The seam was exercised against the **real published image** on the live demo guest:
**Rollback set retained on the host:**
`/usr/local/sbin/felhom-pbs-apply.bak-20260721-101635` (and the older `.bak-20260718-preR39`),
`/usr/local/bin/felhom-agent.bak-2026072{1-101635,1-101905,…}`,
`/root/felhom-agent.sudoers.bak-20260721-101635`.
**The two live proofs that matter:**
```
$ docker run --rm --entrypoint /usr/local/bin/felhom-controller \
gitea.dooplex.hu/admin/felhom-controller:0.147.0 --print-infra-images
traefik:v3.6.7
cloudflare/cloudflared:2026.6.0
gtstef/filebrowser:1.3.3-stable
gitea.dooplex.hu/admin/felhom-samba:1.0.0
# the NON-ROOT agent can finally read its own credential
sudo -u felhom-agent sudo -n felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
rc=0 secret_len=37 stderr=(empty)
# and the probe actually runs (debug enabled briefly, then restored to info)
level=DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
```
All four, from the image the golden would bake. The flag is config-free by design (no
`controller.yaml`, no data dir, no settings) precisely so a bare `docker run` can ask it.
- **Not run: a full golden bake.** The task scoped this to "effective at the next golden build, do
not rebuild the golden for this alone", so the loop itself has not executed end-to-end on a real
bake. The parts that could be verified without one were.
Report verified hub-side: `agent_version 0.91.1→0.91.2`, `wrapper_sha256
104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3`.
## Follow-up
## 6. A defect I shipped and caught mid-STOP-1
`documentation/backlog/ROADMAP.md` pre-invite checklist records that **golden ≥ 0.147.x** carries all
four infra images. Until the next bake, a fresh box still pulls felhom-samba at enable time — which
controller v0.147.0's progress card now at least explains rather than leaving silent.
**v0.91.0 shipped the auth-honesty leg INERT.** I built the `AuthSink` seam and the `NoteAuthResult`
consumer, and `main.go` never called `SetAuthSink`. The reporter deliberately skips probing with no
sink attached, so there was no probe, no `auth_failed`, no self-heal — and **nothing failed**, because
every unit test injected the sink directly.
Caught during STOP-1 by checking the wiring instead of trusting it. **This is the same class as the
controller v0.154.0 defect the day before: a table test over a seam proves the seam, not the caller.**
Fixed in v0.91.1; the published 0.91.0 artifact was **superseded, not overwritten** (a published
version stays immutable).
**v0.91.2** then made a healthy probe observable at Debug — it previously logged only on failure, so
"no `auth_failed`" was indistinguishable from "never probed". That silence is exactly how v0.91.0
shipped inert unnoticed.
## 7. For the operator — STOP-2 and STOP-3
**STOP-2 (one click):** press **Re-issue PBS credentials** for the demo customer. Expected chain:
fresh secret row → `secret_generation` **0 → 1** in the descriptor (the current live descriptor has no
such key at all) → poke → agent re-applies with **no short-circuit** → fresh secret consumed
(`consumed_at` stamped) → reconcile rc-0 → probe 200 → tier `active`. **The July-18 negative — the
same click doing nothing — is the historical red-proof this run is measured against.**
**STOP-3 (manifest save):**
| field | value |
|---|---|
| Agent version | `0.91.2` |
| Agent sha256 | `34d309be429473f3f0ab34e3185e17b22463a341b30bf46e162306bff4aec22a` |
| **PBS wrapper sha256** (new field) | `104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3` |
| MinAgent | `0.91.2` — safe now: **P4 shows one enrolled host and it already runs 0.91.2** |
Superseded, do not vouch: **0.91.0** (inert leg) and **0.91.1**. `0.90.1` correctly remains **404**.
## 8. Residual
- **R-50b(b)/(c) NOT solved** — the wrapper is still fetched from `raw/branch/main`, unversioned and
unpinned. This release makes drift *visible*; it does not fix the channel.
- The **0440 sudoers file is not agent-readable**, so its drift stays invisible — recorded, not solved.
- R-39's own residual: the verify loop reads via the wrapper now, but a host whose wrapper predates
v0.91.0 has no `read` verb and degrades to the old skip (loudly, via the `pbsdr-read` capability).
## 9. Observations
- **The hub's build script prints `kubectl set image` as its deploy hint**, which contradicts the
GitOps rule in `felhom.eu/CLAUDE.md`. It is a trap for exactly the kind of session that trusts tool
output over the runbook. Worth fixing in `build.sh`.
- **The July-18 hotfixed wrapper hashed `2888f2ea…`, matching no commit anyone could name.** That is
R-50b in one line, and it is why the manifest field earns its place.
- `pvesm config` does not exist (it is `pvesm status` / `/etc/pve/storage.cfg`); PVE's per-storage
status endpoint returns `active: 0` rather than erroring — both worth knowing before designing
against them.