Files
felhom-agent/REPORT.md
T
admin 9596d5a8d0 v0.90.1 — R-39 hotfix: PBS reconcile must not pass --server to pvesm set
Config-only (wrapper + red-proof); the Go binary is unchanged, so this ships
with the next agent deploy as a config artifact.

The reconcile verb built `args=(--server "$server" --fingerprint "$fp")`. PVE
treats a PBS storage's `server` as a CREATE-ONLY parameter and rejects the
ENTIRE `pvesm set` call — "can't change value of fixed parameter 'server'" —
even when the value passed is byte-identical to the stored one. So reconcile
could never succeed against an existing entry; it exited 255 every time.

That is severe rather than cosmetic because the agent consumes the hub's
ONE-TIME PBS token secret BEFORE invoking the wrapper. Each hub "Re-issue PBS
credentials" therefore minted a secret, the agent burned it, the wrapper
rejected the apply, and the entry stayed pinned to the revoked credential —
a PBS DR tier authenticating 401 indefinitely while the agent reported
`pbsdr: converged state=applied`.

Live-diagnosed on the N100 during the rehearsal wrap (felhom.eu
tests/VALIDATION-n100-rehearsal-2026-07-18.md F2, ROADMAP R-39). Proven on the
live entry before writing code: `pvesm set <id> --server <same> --fingerprint
<same>` -> rejected; the same call without --server -> rc 0. K (<id>.enc) and
the .pw store verified byte-untouched after the rejected call — PVE rejects
atomically, so the set-only law held.

Fix: drop --server. The server address is immutable by construction (relocating
a PBS endpoint needs a fresh create), so there was never anything to reconcile
there. --fingerprint (+ --password when a secret is fed) remain.

Red-proof TestReconcileNeverPassesServerToPvesmSet: isolates the reconcile)
block from the shipped wrapper, asserts no --server reaches `pvesm set` and
that --fingerprint is still pushed. Verified RED on the unfixed wrapper, GREEN
after. Handles two vacuous-pass traps that both fired while authoring it: the
pattern is line-ending tolerant (\r?\n — this repo is cloned on Windows, and an
\n-only pattern matches nothing and passes silently), and comment lines are
stripped before matching (the WHY note quotes the very flag under test).

NOT fixed here, both still open and riding the spec'd R-39 agent train:
 1. R-39's primary half — the agent re-applies on a change of the DESCRIPTOR
    HASH (manager.go ~L235), but a credential re-issue leaves the descriptor
    byte-identical (same token_id/fingerprint; only the side-table secret
    rotates) and bumps only the generation, so a converged agent still ignores
    a fresh secret. This makes the apply succeed once it re-applies; it does
    not make it re-apply.
 2. The verify loop reads /etc/pve/priv/storage/<id>.pw directly as non-root —
    a path it can only ever WRITE through the root wrapper (0700 root:www-data;
    sudoers exposes create|reconcile|grant, no read verb), so it is permanently
    blind to the failure it exists to catch.

Demo box: wrapper hotfixed in place (.bak-20260718-preR39 kept). NOT yet healed
— diagnosis consumed the pending secret against the unfixed wrapper; the agent
parked correctly in consumed-failed (no burn loop). Healing needs Viktor to
click "Re-issue PBS credentials"; the agent will then pick it up unaided
(marker.json absent, so the L235 short-circuit does not apply).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
2026-07-18 20:27:14 +02:00

5.8 KiB

REPORT — felhom-agent v0.90.1 (R-39 hotfix: PBS reconcile must not pass --server)

Date: 2026-07-18 · Baseline: v0.90.0 → v0.90.1 · Scope: config-only. configs/felhom-pbs-apply (one argv line + its WHY) and one red-proof test. The Go binary is unchanged — this ships with the next agent deploy as a config artifact. Green: go build ./... && go vet ./... && go test ./... all pass.

What was wrong

reconcile built its argv as args=(--server "$server" --fingerprint "$fp"). PVE treats a PBS storage's server as a create-only parameter and rejects the whole pvesm set call — can't change value of fixed parameter 'server'even when the value is byte-identical to the stored one. reconcile could therefore never succeed against an existing entry.

That is severe rather than cosmetic because the agent consumes the hub's one-time PBS token secret before invoking the wrapper. Every failed reconcile burned a credential:

hub "Re-issue PBS credentials"  →  fresh one-time secret minted
agent                           →  secret CONSUMED (single-use, now spent)
wrapper reconcile               →  exit 255, "fixed parameter 'server'"
storage entry                   →  still pinned to the REVOKED secret  →  401 forever

How it was found and proven

Live-diagnosed on the N100 demo host during the 2026-07-18 rehearsal wrap (felhom.eu/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md finding F2, ROADMAP R-39). The box had been reporting pbsdr: converged state=applied while pvesm status returned 401 Unauthorized / inactive.

Proven directly on the live entry before any code was written:

Probe Result
pvesm set felhom-pbs --server <same> --fingerprint <same> rejectedcan't change value of fixed parameter 'server'
pvesm set felhom-pbs --fingerprint <same> rc 0 — accepted

The encryption key K (<id>.enc) and the .pw store were verified byte-untouched after the rejected call: PVE rejects atomically, so the set-only law held throughout.

The fix

Drop --server from the reconcile argv. The server address is immutable by construction — relocating a PBS endpoint requires a fresh create — so there was never anything for reconcile to reconcile there. --fingerprint (and --password when a secret is fed on stdin) remain, which is the mutable identity the verb exists to push.

Red-proof

TestReconcileNeverPassesServerToPvesmSet (internal/pbsdr/manager_test.go) isolates the reconcile) block from the shipped wrapper and asserts no --server reaches pvesm set, plus that --fingerprint is still pushed so the verb cannot be hollowed out. Verified RED against the unfixed wrapper and GREEN after the fix.

Two traps the proof handles explicitly, both of which would have made it pass vacuously — and both of which actually fired during authoring, which is the argument for running a proof red first:

  • Line endings. This repo is cloned on Windows; the working copy carries CRLF, so an \n-only pattern matches nothing and the guard passes silently. The pattern is \r?\n throughout. The first run failed with "could not locate the reconcile) block" — the vacuous-pass failure mode, caught only because the test was run against the broken wrapper first.
  • Comments. The WHY note above the fix necessarily quotes --server, the very flag the test forbids, so the suite went red after the fix was applied. Comment lines are now stripped before matching.

Deliberately NOT fixed here — each still open

  1. R-39's primary half. The agent re-applies on a change of the descriptor hash (internal/pbsdr/manager.go ~L235: if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { return }). A hub credential re-issue leaves the descriptor byte-identical — same token_id, same fingerprint; only the side-table secret rotates — and bumps only desired_generation. So a converged agent short-circuits and never consumes the fresh secret. This wrapper fix makes the apply succeed once the agent is made to re-apply; it does not make it re-apply.
  2. The verify-loop read. pbs: cannot read token secret … permission denied — the non-root agent reads /etc/pve/priv/storage/<id>.pw directly, a path it can only ever write through the root wrapper. /etc/pve/priv is 0700 root:www-data and sudoers exposes only create|reconcile|grantthere is no read verb. The loop is permanently blind to exactly the failure it exists to catch.

Both ride the spec'd R-39 agent train. Recorded, not improvised.

Live state of the demo box at hand-off

The wrapper is hotfixed on the N100 (/usr/local/sbin/felhom-pbs-apply, backup kept at .bak-20260718-preR39, bash -n clean) so the repo and the box agree.

The box is not yet healed, and cannot be by CC. During diagnosis the agent was made to re-apply (marker moved aside — never deleted, the R-22 precedent), which proved the mechanism but consumed the one pending secret against the still-unfixed wrapper. The agent parked correctly in consumed-failed.json with NOT retrying silentlyno burn loop, the fail-safe worked.

Healing needs one password-gated operator action CC cannot perform:

Viktor: click "Re-issue PBS credentials" for demo-felhom on the hub.

The agent will then pick it up unaided — marker.json is absent, so the L235 short-circuit does not apply — consume the fresh secret, and the patched wrapper will now apply it. Verification to run afterwards: pvesm status shows felhom-pbs active, then one on-demand PBS backup of guest 9201 confirmed in the PBS-side listing (the tier's first real backup on the reborn box).

Nothing was destroyed: .pw, .enc (K), and the storage.cfg entry are all intact and verified.