admin
9596d5a8d0
Config-only (wrapper + red-proof); the Go binary is unchanged, so this ships
with the next agent deploy as a config artifact.
The reconcile verb built `args=(--server "$server" --fingerprint "$fp")`. PVE
treats a PBS storage's `server` as a CREATE-ONLY parameter and rejects the
ENTIRE `pvesm set` call — "can't change value of fixed parameter 'server'" —
even when the value passed is byte-identical to the stored one. So reconcile
could never succeed against an existing entry; it exited 255 every time.
That is severe rather than cosmetic because the agent consumes the hub's
ONE-TIME PBS token secret BEFORE invoking the wrapper. Each hub "Re-issue PBS
credentials" therefore minted a secret, the agent burned it, the wrapper
rejected the apply, and the entry stayed pinned to the revoked credential —
a PBS DR tier authenticating 401 indefinitely while the agent reported
`pbsdr: converged state=applied`.
Live-diagnosed on the N100 during the rehearsal wrap (felhom.eu
tests/VALIDATION-n100-rehearsal-2026-07-18.md F2, ROADMAP R-39). Proven on the
live entry before writing code: `pvesm set <id> --server <same> --fingerprint
<same>` -> rejected; the same call without --server -> rc 0. K (<id>.enc) and
the .pw store verified byte-untouched after the rejected call — PVE rejects
atomically, so the set-only law held.
Fix: drop --server. The server address is immutable by construction (relocating
a PBS endpoint needs a fresh create), so there was never anything to reconcile
there. --fingerprint (+ --password when a secret is fed) remain.
Red-proof TestReconcileNeverPassesServerToPvesmSet: isolates the reconcile)
block from the shipped wrapper, asserts no --server reaches `pvesm set` and
that --fingerprint is still pushed. Verified RED on the unfixed wrapper, GREEN
after. Handles two vacuous-pass traps that both fired while authoring it: the
pattern is line-ending tolerant (\r?\n — this repo is cloned on Windows, and an
\n-only pattern matches nothing and passes silently), and comment lines are
stripped before matching (the WHY note quotes the very flag under test).
NOT fixed here, both still open and riding the spec'd R-39 agent train:
1. R-39's primary half — the agent re-applies on a change of the DESCRIPTOR
HASH (manager.go ~L235), but a credential re-issue leaves the descriptor
byte-identical (same token_id/fingerprint; only the side-table secret
rotates) and bumps only the generation, so a converged agent still ignores
a fresh secret. This makes the apply succeed once it re-applies; it does
not make it re-apply.
2. The verify loop reads /etc/pve/priv/storage/<id>.pw directly as non-root —
a path it can only ever WRITE through the root wrapper (0700 root:www-data;
sudoers exposes create|reconcile|grant, no read verb), so it is permanently
blind to the failure it exists to catch.
Demo box: wrapper hotfixed in place (.bak-20260718-preR39 kept). NOT yet healed
— diagnosis consumed the pending secret against the unfixed wrapper; the agent
parked correctly in consumed-failed (no burn loop). Healing needs Viktor to
click "Re-issue PBS credentials"; the agent will then pick it up unaided
(marker.json absent, so the L235 short-circuit does not apply).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
103 lines
5.8 KiB
Markdown
103 lines
5.8 KiB
Markdown
# REPORT — felhom-agent v0.90.1 (R-39 hotfix: PBS reconcile must not pass `--server`)
|
|
|
|
**Date:** 2026-07-18 · **Baseline:** v0.90.0 → **v0.90.1** · **Scope:** config-only.
|
|
`configs/felhom-pbs-apply` (one argv line + its WHY) and one red-proof test. **The Go binary is
|
|
unchanged** — this ships with the next agent deploy as a config artifact.
|
|
**Green:** `go build ./... && go vet ./... && go test ./...` all pass.
|
|
|
|
## What was wrong
|
|
|
|
`reconcile` built its argv as `args=(--server "$server" --fingerprint "$fp")`. PVE treats a PBS
|
|
storage's `server` as a **create-only** parameter and rejects the whole `pvesm set` call —
|
|
`can't change value of fixed parameter 'server'` — **even when the value is byte-identical to the
|
|
stored one**. `reconcile` could therefore never succeed against an existing entry.
|
|
|
|
That is severe rather than cosmetic because **the agent consumes the hub's one-time PBS token secret
|
|
before invoking the wrapper**. Every failed reconcile burned a credential:
|
|
|
|
```
|
|
hub "Re-issue PBS credentials" → fresh one-time secret minted
|
|
agent → secret CONSUMED (single-use, now spent)
|
|
wrapper reconcile → exit 255, "fixed parameter 'server'"
|
|
storage entry → still pinned to the REVOKED secret → 401 forever
|
|
```
|
|
|
|
## How it was found and proven
|
|
|
|
Live-diagnosed on the N100 demo host during the 2026-07-18 rehearsal wrap
|
|
(`felhom.eu/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md` finding F2, ROADMAP
|
|
**R-39**). The box had been reporting `pbsdr: converged state=applied` while `pvesm status` returned
|
|
`401 Unauthorized` / `inactive`.
|
|
|
|
Proven directly on the live entry before any code was written:
|
|
|
|
| Probe | Result |
|
|
|---|---|
|
|
| `pvesm set felhom-pbs --server <same> --fingerprint <same>` | **rejected** — `can't change value of fixed parameter 'server'` |
|
|
| `pvesm set felhom-pbs --fingerprint <same>` | **rc 0 — accepted** |
|
|
|
|
The encryption key `K` (`<id>.enc`) and the `.pw` store were verified byte-untouched after the
|
|
rejected call: PVE rejects atomically, so the set-only law held throughout.
|
|
|
|
## The fix
|
|
|
|
Drop `--server` from the reconcile argv. The server address is immutable by construction —
|
|
relocating a PBS endpoint requires a fresh `create` — so there was never anything for `reconcile` to
|
|
reconcile there. `--fingerprint` (and `--password` when a secret is fed on stdin) remain, which is
|
|
the mutable identity the verb exists to push.
|
|
|
|
## Red-proof
|
|
|
|
`TestReconcileNeverPassesServerToPvesmSet` (`internal/pbsdr/manager_test.go`) isolates the
|
|
`reconcile)` block from the shipped wrapper and asserts no `--server` reaches `pvesm set`, plus that
|
|
`--fingerprint` is still pushed so the verb cannot be hollowed out. **Verified RED against the
|
|
unfixed wrapper and GREEN after the fix.**
|
|
|
|
Two traps the proof handles explicitly, both of which would have made it pass vacuously — and both
|
|
of which actually fired during authoring, which is the argument for running a proof red first:
|
|
- **Line endings.** This repo is cloned on Windows; the working copy carries CRLF, so an `\n`-only
|
|
pattern matches nothing and the guard passes silently. The pattern is `\r?\n` throughout. The
|
|
first run failed with "could not locate the reconcile) block" — the vacuous-pass failure mode,
|
|
caught only because the test was run against the broken wrapper first.
|
|
- **Comments.** The WHY note above the fix necessarily quotes `--server`, the very flag the test
|
|
forbids, so the suite went red *after* the fix was applied. Comment lines are now stripped before
|
|
matching.
|
|
|
|
## Deliberately NOT fixed here — each still open
|
|
|
|
1. **R-39's primary half.** The agent re-applies on a change of the **descriptor hash**
|
|
(`internal/pbsdr/manager.go` ~L235:
|
|
`if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { return }`).
|
|
A hub credential re-issue leaves the descriptor **byte-identical** — same `token_id`, same
|
|
`fingerprint`; only the side-table secret rotates — and bumps only `desired_generation`. So a
|
|
converged agent short-circuits and never consumes the fresh secret. This wrapper fix makes the
|
|
apply *succeed* once the agent is made to re-apply; **it does not make it re-apply.**
|
|
2. **The verify-loop read.** `pbs: cannot read token secret … permission denied` — the non-root
|
|
agent reads `/etc/pve/priv/storage/<id>.pw` **directly**, a path it can only ever *write* through
|
|
the root wrapper. `/etc/pve/priv` is `0700 root:www-data` and sudoers exposes only
|
|
`create|reconcile|grant` — **there is no read verb**. The loop is permanently blind to exactly
|
|
the failure it exists to catch.
|
|
|
|
Both ride the spec'd R-39 agent train. Recorded, not improvised.
|
|
|
|
## Live state of the demo box at hand-off
|
|
|
|
The wrapper is **hotfixed on the N100** (`/usr/local/sbin/felhom-pbs-apply`, backup kept at
|
|
`.bak-20260718-preR39`, `bash -n` clean) so the repo and the box agree.
|
|
|
|
**The box is not yet healed, and cannot be by CC.** During diagnosis the agent was made to re-apply
|
|
(marker moved aside — never deleted, the R-22 precedent), which proved the mechanism but consumed
|
|
the one pending secret against the still-unfixed wrapper. The agent parked correctly in
|
|
`consumed-failed.json` with `NOT retrying silently` — **no burn loop**, the fail-safe worked.
|
|
|
|
Healing needs one password-gated operator action CC cannot perform:
|
|
|
|
> **Viktor: click "Re-issue PBS credentials" for `demo-felhom` on the hub.**
|
|
|
|
The agent will then pick it up unaided — `marker.json` is absent, so the L235 short-circuit does not
|
|
apply — consume the fresh secret, and the patched wrapper will now apply it. Verification to run
|
|
afterwards: `pvesm status` shows `felhom-pbs` **active**, then one on-demand PBS backup of guest 9201
|
|
confirmed in the PBS-side listing (the tier's first real backup on the reborn box).
|
|
|
|
Nothing was destroyed: `.pw`, `.enc` (K), and the `storage.cfg` entry are all intact and verified.
|