felhom-agent/REPORT.md

REPORT — TASK-B: R-39 fleet fix + R-50b(a) · felhom-agent v0.90.0 → v0.91.2

Date: 2026-07-21 · Trunk, pushed to main. Baseline: 8c55ac7 (clean, == origin/main). Companion: hub v0.68.0 (felhom.eu).

0. Status

Leg	Status
Part 1 — hub v0.68.0	SHIPPED + DEPLOYED (GitOps sync, Synced/Healthy)
Part 2 — agent v0.91.2	SHIPPED + PUBLISHED + DEPLOYED to felhom-pve
STOP-1 — felhom-pve update	DONE + VERIFIED (§5)
STOP-3 — manifest save	DONE by the operator 2026-07-21 08:31:12Z — agent 0.91.2 + sha + wrapper sha 104db0a4… + MinAgent 0.91.2, all six fields persisted. The agent reports the same wrapper hash, so the R-50b drift gauge reads ok.
STOP-2 — live re-issue	DONE + PROVEN 2026-07-21 — the chain closed in 13 s. 08:39:31Z hub mint gen 0→1 + secret_generation: 1 (token_id/fingerprint byte-identical) → 10:39:34 wrapper read → 10:39:38 REJECTED … applied and DEAD previous_state=applied → 10:39:45 secret consumed (no short-circuit) → 10:39:45 set-only reconcile → 10:39:47 converged state=applied. Marker hash moved to afbb3b41…; secret mtime 2026-07-18→2026-07-21 10:39:45; new credential probes 200; reports trace applied → auth_failed → applied; zero self-heal escalations, one mint, one consume. First attempt hit the offsite re-issue by mistake (two distinct actions) — harmless to PBS-DR but it made the escrow stale, ceremony re-run.

1. Probes

• P1 — version stamping: PASS. ldflags -X main.version; publish-agent.sh refuses to publish when the binary's --version disagrees, so the stale 0.89.0 default cannot leak.
• P2 — hub deploy path: PASS. GitOps only: ArgoCD app felhom, auto-sync off, manifest bump
  • deliberate sync. The kubectl set image line (in the old controller CLAUDE.md and printed by build.sh itself) was not used.
• P3 — felhom-pve before-state: CAPTURED (§5 table). Notably the DR tier was healthy: a live GET /api2/json/version with the stored credential returned 200. This is a fix for a latent defect, not a broken box.
• P4 — fleet: ONE enrolled host (demo-felhom-8363b5, agent 0.90.0). Peti is not enrolled, so the MinAgent raise at STOP-3 cannot strand anyone. (Read from a temporary copy of the hub DB, which carries live credentials — shredded immediately after each read.)

2. Two spec/reality mismatches, both resolved

(a) The generation source does not exist as specified. The brief said to return "the new row's id (int64) … no schema change". host_pbs_secrets is host_id PRIMARY KEY, UPSERTed last-write-wins — there is no row id and no new row, and created_at collides for two mints in one second. Added an additive generation counter column using the repo's existing idempotent ALTER TABLE idiom. Same semantics the brief wanted; different mechanism.

(b) A second load-bearing fact the brief did not flag — checked, not trusted. Apply bails out entirely if the storage status probe errors, and converges via adopt without consuming the secret if the storage reads active. So the whole fix depended on what PVE returns for a 401'd PBS storage. Read PVE's own source: storage_info wraps activate_storage and $plugin->status in eval{}, warns, and leaves the pre-initialised active => 0. A 401 therefore returns HTTP 200 with active: 0, never an API error — StorageActive gives (false, nil) and Apply falls through to verify → consume → reconcile. Scenario A's chain is sound by proof. The re-arm test encodes this fixture shape explicitly so the assumption is visible.

3. What shipped (agent)

WirePBSDR.SecretGeneration (field-exact with the hub) · wrapper read verb + exactly one sudoers line + a pbsdr-read capability row · pbs.ProbeAuth + ErrUnauthorized · the probe on the 15-minute collect path feeding a loud auth_failed · readPBSSecret (direct read, wrapper fallback) · wrapper sha256 in the report (R-50b(a)).

403 is deliberately NOT treated as unauthorized — re-keying a too-narrow token would mint credentials forever without fixing anything. A transport error is UNKNOWN, never a rejection — otherwise every network blip burns a credential.

4. Tests + the four red-proofs

Full gate green in both repos at every commit. All red-proofs run at the assertion level.

#	Red-proof	Result
A	Remove SecretGeneration from the agent struct	consume calls=1, want 2 — the converged short-circuit fires; the July-18 defect exactly
B	Stop stamping the generation in ReissuePBSDR	descriptor byte-identical across a re-issue, block printed
D	Remove the id charset guard and the prefix assertion	the out-of-tree secret is printed (rc=0 stdout="NOT-FOR-THE-AGENT-…")
E	Swallow the probe result in NoteAuthResult	State:applied AuthFailed:false — the July-18 shape

A finding from red-proof D worth keeping: deleting the charset guard alone does not open a hole — readlink -f plus the prefix assertion still catch it, so the naive single-guard red-proof passed vacuously. The layering is real; the isolating proof had to remove both. A red-proof that passes is not automatically reassuring.

The wrapper is tested as the artifact it is — executed under real bash. The two cases needing a writable allowlisted dir run a copy with only the prefix constant relocated (and re-assert that traversal is still refused in the copy), rather than t.Skip — a skipped test proves nothing.

5. STOP-1 — felhom-pve, before → after

	before (P3)	after
agent	0.90.0 · 18c48272…	0.91.2 · 34d309be…
wrapper	2888f2ea… (the in-place July-18 hotfix)	104db0a4… (repo)
sudoers	174125ef…	new, +1 line
capabilities	63	64/64 ok, 0 degraded, 0 inactive
guest 9201	running	running (untouched)
wg-felhom	handshake …673	fresh …793

Gates run before install: bash -n on the wrapper, visudo -cf on the staged sudoers, visudo -c on the whole tree after install, and the staged binary's own --version.

Rollback set retained on the host: /usr/local/sbin/felhom-pbs-apply.bak-20260721-101635 (and the older .bak-20260718-preR39), /usr/local/bin/felhom-agent.bak-2026072{1-101635,1-101905,…}, /root/felhom-agent.sudoers.bak-20260721-101635.

The two live proofs that matter:

# the NON-ROOT agent can finally read its own credential
sudo -u felhom-agent sudo -n felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
  rc=0  secret_len=37  stderr=(empty)

# and the probe actually runs (debug enabled briefly, then restored to info)
level=DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite

Report verified hub-side: agent_version 0.91.1→0.91.2, wrapper_sha256 104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3.

6. A defect I shipped and caught mid-STOP-1

v0.91.0 shipped the auth-honesty leg INERT. I built the AuthSink seam and the NoteAuthResult consumer, and main.go never called SetAuthSink. The reporter deliberately skips probing with no sink attached, so there was no probe, no auth_failed, no self-heal — and nothing failed, because every unit test injected the sink directly.

Caught during STOP-1 by checking the wiring instead of trusting it. This is the same class as the controller v0.154.0 defect the day before: a table test over a seam proves the seam, not the caller. Fixed in v0.91.1; the published 0.91.0 artifact was superseded, not overwritten (a published version stays immutable).

v0.91.2 then made a healthy probe observable at Debug — it previously logged only on failure, so "no auth_failed" was indistinguishable from "never probed". That silence is exactly how v0.91.0 shipped inert unnoticed.

7. For the operator — STOP-2 and STOP-3

STOP-2 (one click): press Re-issue PBS credentials for the demo customer. Expected chain: fresh secret row → secret_generation 0 → 1 in the descriptor (the current live descriptor has no such key at all) → poke → agent re-applies with no short-circuit → fresh secret consumed (consumed_at stamped) → reconcile rc-0 → probe 200 → tier active. The July-18 negative — the same click doing nothing — is the historical red-proof this run is measured against.

STOP-3 (manifest save):

field	value
Agent version	0.91.2
Agent sha256	34d309be429473f3f0ab34e3185e17b22463a341b30bf46e162306bff4aec22a
PBS wrapper sha256 (new field)	104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3
MinAgent	0.91.2 — safe now: P4 shows one enrolled host and it already runs 0.91.2

Superseded, do not vouch: 0.91.0 (inert leg) and 0.91.1. 0.90.1 correctly remains 404.

8. Residual

• R-50b(b)/(c) NOT solved — the wrapper is still fetched from raw/branch/main, unversioned and unpinned. This release makes drift visible; it does not fix the channel.
• The 0440 sudoers file is not agent-readable, so its drift stays invisible — recorded, not solved.
• R-39's own residual: the verify loop reads via the wrapper now, but a host whose wrapper predates v0.91.0 has no read verb and degrades to the old skip (loudly, via the pbsdr-read capability).

9. Observations

• The hub's build script prints kubectl set image as its deploy hint, which contradicts the GitOps rule in felhom.eu/CLAUDE.md. It is a trap for exactly the kind of session that trusts tool output over the runbook. Worth fixing in build.sh.
• The July-18 hotfixed wrapper hashed 2888f2ea…, matching no commit anyone could name. That is R-50b in one line, and it is why the manifest field earns its place.
• pvesm config does not exist (it is pvesm status / /etc/pve/storage.cfg); PVE's per-storage status endpoint returns active: 0 rather than erroring — both worth knowing before designing against them.