Config-only (wrapper + red-proof); the Go binary is unchanged, so this ships
with the next agent deploy as a config artifact.
The reconcile verb built `args=(--server "$server" --fingerprint "$fp")`. PVE
treats a PBS storage's `server` as a CREATE-ONLY parameter and rejects the
ENTIRE `pvesm set` call — "can't change value of fixed parameter 'server'" —
even when the value passed is byte-identical to the stored one. So reconcile
could never succeed against an existing entry; it exited 255 every time.
That is severe rather than cosmetic because the agent consumes the hub's
ONE-TIME PBS token secret BEFORE invoking the wrapper. Each hub "Re-issue PBS
credentials" therefore minted a secret, the agent burned it, the wrapper
rejected the apply, and the entry stayed pinned to the revoked credential —
a PBS DR tier authenticating 401 indefinitely while the agent reported
`pbsdr: converged state=applied`.
Live-diagnosed on the N100 during the rehearsal wrap (felhom.eu
tests/VALIDATION-n100-rehearsal-2026-07-18.md F2, ROADMAP R-39). Proven on the
live entry before writing code: `pvesm set <id> --server <same> --fingerprint
<same>` -> rejected; the same call without --server -> rc 0. K (<id>.enc) and
the .pw store verified byte-untouched after the rejected call — PVE rejects
atomically, so the set-only law held.
Fix: drop --server. The server address is immutable by construction (relocating
a PBS endpoint needs a fresh create), so there was never anything to reconcile
there. --fingerprint (+ --password when a secret is fed) remain.
Red-proof TestReconcileNeverPassesServerToPvesmSet: isolates the reconcile)
block from the shipped wrapper, asserts no --server reaches `pvesm set` and
that --fingerprint is still pushed. Verified RED on the unfixed wrapper, GREEN
after. Handles two vacuous-pass traps that both fired while authoring it: the
pattern is line-ending tolerant (\r?\n — this repo is cloned on Windows, and an
\n-only pattern matches nothing and passes silently), and comment lines are
stripped before matching (the WHY note quotes the very flag under test).
NOT fixed here, both still open and riding the spec'd R-39 agent train:
1. R-39's primary half — the agent re-applies on a change of the DESCRIPTOR
HASH (manager.go ~L235), but a credential re-issue leaves the descriptor
byte-identical (same token_id/fingerprint; only the side-table secret
rotates) and bumps only the generation, so a converged agent still ignores
a fresh secret. This makes the apply succeed once it re-applies; it does
not make it re-apply.
2. The verify loop reads /etc/pve/priv/storage/<id>.pw directly as non-root —
a path it can only ever WRITE through the root wrapper (0700 root:www-data;
sudoers exposes create|reconcile|grant, no read verb), so it is permanently
blind to the failure it exists to catch.
Demo box: wrapper hotfixed in place (.bak-20260718-preR39 kept). NOT yet healed
— diagnosis consumed the pending secret against the unfixed wrapper; the agent
parked correctly in consumed-failed (no burn loop). Healing needs Viktor to
click "Re-issue PBS credentials"; the agent will then pick it up unaided
(marker.json absent, so the L235 short-circuit does not apply).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
5.8 KiB
REPORT — felhom-agent v0.90.1 (R-39 hotfix: PBS reconcile must not pass --server)
Date: 2026-07-18 · Baseline: v0.90.0 → v0.90.1 · Scope: config-only.
configs/felhom-pbs-apply (one argv line + its WHY) and one red-proof test. The Go binary is
unchanged — this ships with the next agent deploy as a config artifact.
Green: go build ./... && go vet ./... && go test ./... all pass.
What was wrong
reconcile built its argv as args=(--server "$server" --fingerprint "$fp"). PVE treats a PBS
storage's server as a create-only parameter and rejects the whole pvesm set call —
can't change value of fixed parameter 'server' — even when the value is byte-identical to the
stored one. reconcile could therefore never succeed against an existing entry.
That is severe rather than cosmetic because the agent consumes the hub's one-time PBS token secret before invoking the wrapper. Every failed reconcile burned a credential:
hub "Re-issue PBS credentials" → fresh one-time secret minted
agent → secret CONSUMED (single-use, now spent)
wrapper reconcile → exit 255, "fixed parameter 'server'"
storage entry → still pinned to the REVOKED secret → 401 forever
How it was found and proven
Live-diagnosed on the N100 demo host during the 2026-07-18 rehearsal wrap
(felhom.eu/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md finding F2, ROADMAP
R-39). The box had been reporting pbsdr: converged state=applied while pvesm status returned
401 Unauthorized / inactive.
Proven directly on the live entry before any code was written:
| Probe | Result |
|---|---|
pvesm set felhom-pbs --server <same> --fingerprint <same> |
rejected — can't change value of fixed parameter 'server' |
pvesm set felhom-pbs --fingerprint <same> |
rc 0 — accepted |
The encryption key K (<id>.enc) and the .pw store were verified byte-untouched after the
rejected call: PVE rejects atomically, so the set-only law held throughout.
The fix
Drop --server from the reconcile argv. The server address is immutable by construction —
relocating a PBS endpoint requires a fresh create — so there was never anything for reconcile to
reconcile there. --fingerprint (and --password when a secret is fed on stdin) remain, which is
the mutable identity the verb exists to push.
Red-proof
TestReconcileNeverPassesServerToPvesmSet (internal/pbsdr/manager_test.go) isolates the
reconcile) block from the shipped wrapper and asserts no --server reaches pvesm set, plus that
--fingerprint is still pushed so the verb cannot be hollowed out. Verified RED against the
unfixed wrapper and GREEN after the fix.
Two traps the proof handles explicitly, both of which would have made it pass vacuously — and both of which actually fired during authoring, which is the argument for running a proof red first:
- Line endings. This repo is cloned on Windows; the working copy carries CRLF, so an
\n-only pattern matches nothing and the guard passes silently. The pattern is\r?\nthroughout. The first run failed with "could not locate the reconcile) block" — the vacuous-pass failure mode, caught only because the test was run against the broken wrapper first. - Comments. The WHY note above the fix necessarily quotes
--server, the very flag the test forbids, so the suite went red after the fix was applied. Comment lines are now stripped before matching.
Deliberately NOT fixed here — each still open
- R-39's primary half. The agent re-applies on a change of the descriptor hash
(
internal/pbsdr/manager.go~L235:if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { return }). A hub credential re-issue leaves the descriptor byte-identical — sametoken_id, samefingerprint; only the side-table secret rotates — and bumps onlydesired_generation. So a converged agent short-circuits and never consumes the fresh secret. This wrapper fix makes the apply succeed once the agent is made to re-apply; it does not make it re-apply. - The verify-loop read.
pbs: cannot read token secret … permission denied— the non-root agent reads/etc/pve/priv/storage/<id>.pwdirectly, a path it can only ever write through the root wrapper./etc/pve/privis0700 root:www-dataand sudoers exposes onlycreate|reconcile|grant— there is no read verb. The loop is permanently blind to exactly the failure it exists to catch.
Both ride the spec'd R-39 agent train. Recorded, not improvised.
Live state of the demo box at hand-off
The wrapper is hotfixed on the N100 (/usr/local/sbin/felhom-pbs-apply, backup kept at
.bak-20260718-preR39, bash -n clean) so the repo and the box agree.
The box is not yet healed, and cannot be by CC. During diagnosis the agent was made to re-apply
(marker moved aside — never deleted, the R-22 precedent), which proved the mechanism but consumed
the one pending secret against the still-unfixed wrapper. The agent parked correctly in
consumed-failed.json with NOT retrying silently — no burn loop, the fail-safe worked.
Healing needs one password-gated operator action CC cannot perform:
Viktor: click "Re-issue PBS credentials" for
demo-felhomon the hub.
The agent will then pick it up unaided — marker.json is absent, so the L235 short-circuit does not
apply — consume the fresh secret, and the patched wrapper will now apply it. Verification to run
afterwards: pvesm status shows felhom-pbs active, then one on-demand PBS backup of guest 9201
confirmed in the PBS-side listing (the tier's first real backup on the reborn box).
Nothing was destroyed: .pw, .enc (K), and the storage.cfg entry are all intact and verified.