felhom-agent/REPORT.md

REPORT — felhom-agent v0.90.1 (R-39 hotfix: PBS reconcile must not pass --server)

Date: 2026-07-18 · Baseline: v0.90.0 → v0.90.1 · Scope: config-only. configs/felhom-pbs-apply (one argv line + its WHY) and one red-proof test. The Go binary is unchanged — this ships with the next agent deploy as a config artifact. Green: go build ./... && go vet ./... && go test ./... all pass.

What was wrong

reconcile built its argv as args=(--server "$server" --fingerprint "$fp"). PVE treats a PBS storage's server as a create-only parameter and rejects the whole pvesm set call — can't change value of fixed parameter 'server' — even when the value is byte-identical to the stored one. reconcile could therefore never succeed against an existing entry.

That is severe rather than cosmetic because the agent consumes the hub's one-time PBS token secret before invoking the wrapper. Every failed reconcile burned a credential:

hub "Re-issue PBS credentials"  →  fresh one-time secret minted
agent                           →  secret CONSUMED (single-use, now spent)
wrapper reconcile               →  exit 255, "fixed parameter 'server'"
storage entry                   →  still pinned to the REVOKED secret  →  401 forever

How it was found and proven

Live-diagnosed on the N100 demo host during the 2026-07-18 rehearsal wrap (felhom.eu/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md finding F2, ROADMAP R-39). The box had been reporting pbsdr: converged state=applied while pvesm status returned 401 Unauthorized / inactive.

Proven directly on the live entry before any code was written:

Probe	Result
pvesm set felhom-pbs --server <same> --fingerprint <same>	rejected — can't change value of fixed parameter 'server'
pvesm set felhom-pbs --fingerprint <same>	rc 0 — accepted

The encryption key K (<id>.enc) and the .pw store were verified byte-untouched after the rejected call: PVE rejects atomically, so the set-only law held throughout.

The fix

Drop --server from the reconcile argv. The server address is immutable by construction — relocating a PBS endpoint requires a fresh create — so there was never anything for reconcile to reconcile there. --fingerprint (and --password when a secret is fed on stdin) remain, which is the mutable identity the verb exists to push.

Red-proof

TestReconcileNeverPassesServerToPvesmSet (internal/pbsdr/manager_test.go) isolates the reconcile) block from the shipped wrapper and asserts no --server reaches pvesm set, plus that --fingerprint is still pushed so the verb cannot be hollowed out. Verified RED against the unfixed wrapper and GREEN after the fix.

Two traps the proof handles explicitly, both of which would have made it pass vacuously — and both of which actually fired during authoring, which is the argument for running a proof red first:

• Line endings. This repo is cloned on Windows; the working copy carries CRLF, so an \n-only pattern matches nothing and the guard passes silently. The pattern is \r?\n throughout. The first run failed with "could not locate the reconcile) block" — the vacuous-pass failure mode, caught only because the test was run against the broken wrapper first.
• Comments. The WHY note above the fix necessarily quotes --server, the very flag the test forbids, so the suite went red after the fix was applied. Comment lines are now stripped before matching.

Deliberately NOT fixed here — each still open

1. R-39's primary half. The agent re-applies on a change of the descriptor hash (internal/pbsdr/manager.go ~L235: if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { return }). A hub credential re-issue leaves the descriptor byte-identical — same token_id, same fingerprint; only the side-table secret rotates — and bumps only desired_generation. So a converged agent short-circuits and never consumes the fresh secret. This wrapper fix makes the apply succeed once the agent is made to re-apply; it does not make it re-apply.
2. The verify-loop read. pbs: cannot read token secret … permission denied — the non-root agent reads /etc/pve/priv/storage/<id>.pw directly, a path it can only ever write through the root wrapper. /etc/pve/priv is 0700 root:www-data and sudoers exposes only create|reconcile|grant — there is no read verb. The loop is permanently blind to exactly the failure it exists to catch.

Both ride the spec'd R-39 agent train. Recorded, not improvised.

The demo box — HEALED and proven (2026-07-18)

The wrapper is hotfixed on the N100 (/usr/local/sbin/felhom-pbs-apply, backup kept at .bak-20260718-preR39, bash -n clean) so the repo and the box agree.

Diagnosis had consumed the one pending secret against the still-unfixed wrapper; the agent parked correctly in consumed-failed.json with NOT retrying silently — no burn loop, the fail-safe worked. Viktor then clicked "Re-issue PBS credentials" (the one password-gated action CC cannot perform) and the whole chain closed itself in 9 seconds:

Time (CEST)	Event
20:28:44	hub: tenantsync: reissue ok → pbsdr credentials re-issued … fresh consume-once secret stored
20:28:51	agent: one-time token secret consumed, wrapper reconcile invoked — now the patched one
20:28:53	agent: pbsdr: converged state=applied — no CONSUMED-BUT-FAILED this time

consumed-failed.json was cleared automatically by finishConverged and marker.json rewritten. The agent picked the re-issue up unaided because marker.json was absent, so the L235 hash short-circuit did not apply.

Proof the tier is genuinely alive, not merely reported alive:

Check	Before	After
pvesm status	401 Unauthorized / inactive	active
Direct token probe, /api2/json/version	401	200
Real backup	none possible	felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z, 9 744 319 312 B, finished in 13m36s

The backup is encrypted under key fingerprint 7e:a6:af:f7:ea:6d:3e:d9 — the escrowed key, i.e. the one customer zero holds the recovery code for. This is the DR tier's first real backup on the reborn box.

Nothing was destroyed at any point: .pw, .enc (K), and the storage.cfg entry were verified intact after the rejected pvesm set (PVE rejects atomically, so the set-only law held).