felhom-agent/REPORT.md

# REPORT — TASK-B: R-39 fleet fix + R-50b(a) · felhom-agent v0.90.0 → **v0.91.2**

**Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `8c55ac7` (clean, == `origin/main`).
Companion: hub **v0.68.0** (`felhom.eu`).

## 0. Status

| Leg | Status |
|---|---|
| Part 1 — hub v0.68.0 | **SHIPPED + DEPLOYED** (GitOps sync, Synced/Healthy) |
| Part 2 — agent v0.91.2 | **SHIPPED + PUBLISHED + DEPLOYED** to felhom-pve |
| **STOP-1** — felhom-pve update | **DONE + VERIFIED** (§5) |
| **STOP-3** — manifest save | **DONE by the operator 2026-07-21 08:31:12Z** — agent 0.91.2 + sha + **wrapper sha `104db0a4…`** + MinAgent 0.91.2, all six fields persisted. The agent reports the same wrapper hash, so the R-50b drift gauge reads **ok**. |
| **STOP-2** — live re-issue | **DONE + PROVEN 2026-07-21 — the chain closed in 13 s.** `08:39:31Z` hub mint gen 0→1 + `secret_generation: 1` (token_id/fingerprint byte-identical) → `10:39:34` wrapper read → `10:39:38` `REJECTED … applied and DEAD` `previous_state=applied` → `10:39:45` **secret consumed (no short-circuit)** → `10:39:45` set-only reconcile → `10:39:47` `converged state=applied`. Marker hash moved to `afbb3b41…`; secret mtime `2026-07-18`→`2026-07-21 10:39:45`; new credential probes 200; reports trace `applied → auth_failed → applied`; **zero** self-heal escalations, one mint, one consume. First attempt hit the **offsite** re-issue by mistake (two distinct actions) — harmless to PBS-DR but it made the escrow stale, ceremony re-run. |

## 1. Probes

- **P1 — version stamping: PASS.** ldflags `-X main.version`; `publish-agent.sh` refuses to publish
  when the binary's `--version` disagrees, so the stale `0.89.0` default cannot leak.
- **P2 — hub deploy path: PASS.** GitOps only: ArgoCD app `felhom`, auto-sync **off**, manifest bump
  + deliberate sync. The `kubectl set image` line (in the old controller CLAUDE.md *and* printed by
  `build.sh` itself) was **not** used.
- **P3 — felhom-pve before-state: CAPTURED** (§5 table). Notably the DR tier was **healthy**: a live
  `GET /api2/json/version` with the stored credential returned **200**. This is a fix for a latent
  defect, not a broken box.
- **P4 — fleet: ONE enrolled host** (`demo-felhom-8363b5`, agent 0.90.0). **Peti is not enrolled**, so
  the MinAgent raise at STOP-3 cannot strand anyone. (Read from a temporary copy of the hub DB, which
  carries live credentials — shredded immediately after each read.)

## 2. Two spec/reality mismatches, both resolved

**(a) The generation source does not exist as specified.** The brief said to return "the new row's id
(int64) … no schema change". `host_pbs_secrets` is `host_id PRIMARY KEY`, UPSERTed last-write-wins —
**there is no row id and no new row**, and `created_at` collides for two mints in one second. Added an
additive `generation` counter column using the repo's existing idempotent `ALTER TABLE` idiom. Same
semantics the brief wanted; different mechanism.

**(b) A second load-bearing fact the brief did not flag — checked, not trusted.** `Apply` bails out
entirely if the storage *status probe* errors, and converges via `adopt` **without consuming the
secret** if the storage reads active. So the whole fix depended on what PVE returns for a 401'd PBS
storage. Read PVE's own source: `storage_info` wraps `activate_storage` and `$plugin->status` in
`eval{}`, warns, and leaves the pre-initialised `active => 0`. **A 401 therefore returns HTTP 200 with
`active: 0`, never an API error** — `StorageActive` gives `(false, nil)` and `Apply` falls through to
verify → consume → reconcile. Scenario A's chain is sound *by proof*. The re-arm test encodes this
fixture shape explicitly so the assumption is visible.

## 3. What shipped (agent)

`WirePBSDR.SecretGeneration` (field-exact with the hub) · wrapper `read` verb + **exactly one**
sudoers line + a `pbsdr-read` capability row · `pbs.ProbeAuth` + `ErrUnauthorized` · the probe on the
**15-minute collect path** feeding a loud `auth_failed` · `readPBSSecret` (direct read, wrapper
fallback) · wrapper sha256 in the report (R-50b(a)).

**403 is deliberately NOT treated as unauthorized** — re-keying a too-narrow token would mint
credentials forever without fixing anything. **A transport error is UNKNOWN, never a rejection** —
otherwise every network blip burns a credential.

## 4. Tests + the four red-proofs

Full gate green in both repos at every commit. All red-proofs run at the **assertion** level.

| # | Red-proof | Result |
|---|---|---|
| A | Remove `SecretGeneration` from the agent struct | `consume calls=1, want 2` — the converged short-circuit fires; the July-18 defect exactly |
| B | Stop stamping the generation in `ReissuePBSDR` | descriptor **byte-identical** across a re-issue, block printed |
| D | Remove the id charset guard **and** the prefix assertion | the out-of-tree secret is **printed** (`rc=0 stdout="NOT-FOR-THE-AGENT-…"`) |
| E | Swallow the probe result in `NoteAuthResult` | `State:applied AuthFailed:false` — the July-18 shape |

**A finding from red-proof D worth keeping:** deleting the charset guard *alone* does **not** open a
hole — `readlink -f` plus the prefix assertion still catch it, so the naive single-guard red-proof
**passed vacuously**. The layering is real; the isolating proof had to remove both. A red-proof that
passes is not automatically reassuring.

The wrapper is tested as the artifact it is — executed under real `bash`. The two cases needing a
writable allowlisted dir run a **copy with only the prefix constant relocated** (and re-assert that
traversal is still refused in the copy), rather than `t.Skip` — a skipped test proves nothing.

## 5. STOP-1 — felhom-pve, before → after

| | before (P3) | after |
|---|---|---|
| agent | 0.90.0 · `18c48272…` | **0.91.2** · `34d309be…` |
| wrapper | `2888f2ea…` (the in-place July-18 hotfix) | `104db0a4…` (repo) |
| sudoers | `174125ef…` | new, +1 line |
| capabilities | 63 | **64/64 ok, 0 degraded, 0 inactive** |
| guest 9201 | running | running (untouched) |
| `wg-felhom` | handshake `…673` | fresh `…793` |

Gates run before install: `bash -n` on the wrapper, `visudo -cf` on the staged sudoers, `visudo -c`
on the whole tree **after** install, and the staged binary's own `--version`.

**Rollback set retained on the host:**
`/usr/local/sbin/felhom-pbs-apply.bak-20260721-101635` (and the older `.bak-20260718-preR39`),
`/usr/local/bin/felhom-agent.bak-2026072{1-101635,1-101905,…}`,
`/root/felhom-agent.sudoers.bak-20260721-101635`.

**The two live proofs that matter:**

```
# the NON-ROOT agent can finally read its own credential
sudo -u felhom-agent sudo -n felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
  rc=0  secret_len=37  stderr=(empty)

# and the probe actually runs (debug enabled briefly, then restored to info)
level=DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
```

Report verified hub-side: `agent_version 0.91.1→0.91.2`, `wrapper_sha256
104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3`.

## 6. A defect I shipped and caught mid-STOP-1

**v0.91.0 shipped the auth-honesty leg INERT.** I built the `AuthSink` seam and the `NoteAuthResult`
consumer, and `main.go` never called `SetAuthSink`. The reporter deliberately skips probing with no
sink attached, so there was no probe, no `auth_failed`, no self-heal — and **nothing failed**, because
every unit test injected the sink directly.

Caught during STOP-1 by checking the wiring instead of trusting it. **This is the same class as the
controller v0.154.0 defect the day before: a table test over a seam proves the seam, not the caller.**
Fixed in v0.91.1; the published 0.91.0 artifact was **superseded, not overwritten** (a published
version stays immutable).

**v0.91.2** then made a healthy probe observable at Debug — it previously logged only on failure, so
"no `auth_failed`" was indistinguishable from "never probed". That silence is exactly how v0.91.0
shipped inert unnoticed.

## 7. For the operator — STOP-2 and STOP-3

**STOP-2 (one click):** press **Re-issue PBS credentials** for the demo customer. Expected chain:
fresh secret row → `secret_generation` **0 → 1** in the descriptor (the current live descriptor has no
such key at all) → poke → agent re-applies with **no short-circuit** → fresh secret consumed
(`consumed_at` stamped) → reconcile rc-0 → probe 200 → tier `active`. **The July-18 negative — the
same click doing nothing — is the historical red-proof this run is measured against.**

**STOP-3 (manifest save):**

| field | value |
|---|---|
| Agent version | `0.91.2` |
| Agent sha256 | `34d309be429473f3f0ab34e3185e17b22463a341b30bf46e162306bff4aec22a` |
| **PBS wrapper sha256** (new field) | `104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3` |
| MinAgent | `0.91.2` — safe now: **P4 shows one enrolled host and it already runs 0.91.2** |

Superseded, do not vouch: **0.91.0** (inert leg) and **0.91.1**. `0.90.1` correctly remains **404**.

## 8. Residual

- **R-50b(b)/(c) NOT solved** — the wrapper is still fetched from `raw/branch/main`, unversioned and
  unpinned. This release makes drift *visible*; it does not fix the channel.
- The **0440 sudoers file is not agent-readable**, so its drift stays invisible — recorded, not solved.
- R-39's own residual: the verify loop reads via the wrapper now, but a host whose wrapper predates
  v0.91.0 has no `read` verb and degrades to the old skip (loudly, via the `pbsdr-read` capability).

## 9. Observations

- **The hub's build script prints `kubectl set image` as its deploy hint**, which contradicts the
  GitOps rule in `felhom.eu/CLAUDE.md`. It is a trap for exactly the kind of session that trusts tool
  output over the runbook. Worth fixing in `build.sh`.
- **The July-18 hotfixed wrapper hashed `2888f2ea…`, matching no commit anyone could name.** That is
  R-50b in one line, and it is why the manifest field earns its place.
- `pvesm config` does not exist (it is `pvesm status` / `/etc/pve/storage.cfg`); PVE's per-storage
  status endpoint returns `active: 0` rather than erroring — both worth knowing before designing
  against them.