admin
f22f70ce58
Supersedes the "not yet healed" hand-off section. Viktor clicked "Re-issue PBS
credentials" and the chain closed in 9 seconds: hub re-issued 20:28:44 -> agent
consumed 20:28:51 -> `pbsdr: converged state=applied` 20:28:53, with the PATCHED
wrapper. consumed-failed.json cleared automatically, marker.json rewritten. The
agent picked it up unaided because marker.json was absent, so the L235 hash
short-circuit did not apply.
Proof the tier is alive rather than merely reporting alive:
pvesm status 401/inactive -> ACTIVE
token /version 401 -> 200
real backup none -> felhom-pbs:backup/ct/9201/
2026-07-18T18:31:06Z, 9744319312 B, 13m36s
Encrypted under fingerprint 7e:a6:af:f7:ea:6d:3e:d9 — the ESCROWED key, the one
customer zero holds the recovery code for. First real DR-tier backup on the
reborn box.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
117 lines
6.5 KiB
Markdown
117 lines
6.5 KiB
Markdown
# REPORT — felhom-agent v0.90.1 (R-39 hotfix: PBS reconcile must not pass `--server`)
|
|
|
|
**Date:** 2026-07-18 · **Baseline:** v0.90.0 → **v0.90.1** · **Scope:** config-only.
|
|
`configs/felhom-pbs-apply` (one argv line + its WHY) and one red-proof test. **The Go binary is
|
|
unchanged** — this ships with the next agent deploy as a config artifact.
|
|
**Green:** `go build ./... && go vet ./... && go test ./...` all pass.
|
|
|
|
## What was wrong
|
|
|
|
`reconcile` built its argv as `args=(--server "$server" --fingerprint "$fp")`. PVE treats a PBS
|
|
storage's `server` as a **create-only** parameter and rejects the whole `pvesm set` call —
|
|
`can't change value of fixed parameter 'server'` — **even when the value is byte-identical to the
|
|
stored one**. `reconcile` could therefore never succeed against an existing entry.
|
|
|
|
That is severe rather than cosmetic because **the agent consumes the hub's one-time PBS token secret
|
|
before invoking the wrapper**. Every failed reconcile burned a credential:
|
|
|
|
```
|
|
hub "Re-issue PBS credentials" → fresh one-time secret minted
|
|
agent → secret CONSUMED (single-use, now spent)
|
|
wrapper reconcile → exit 255, "fixed parameter 'server'"
|
|
storage entry → still pinned to the REVOKED secret → 401 forever
|
|
```
|
|
|
|
## How it was found and proven
|
|
|
|
Live-diagnosed on the N100 demo host during the 2026-07-18 rehearsal wrap
|
|
(`felhom.eu/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md` finding F2, ROADMAP
|
|
**R-39**). The box had been reporting `pbsdr: converged state=applied` while `pvesm status` returned
|
|
`401 Unauthorized` / `inactive`.
|
|
|
|
Proven directly on the live entry before any code was written:
|
|
|
|
| Probe | Result |
|
|
|---|---|
|
|
| `pvesm set felhom-pbs --server <same> --fingerprint <same>` | **rejected** — `can't change value of fixed parameter 'server'` |
|
|
| `pvesm set felhom-pbs --fingerprint <same>` | **rc 0 — accepted** |
|
|
|
|
The encryption key `K` (`<id>.enc`) and the `.pw` store were verified byte-untouched after the
|
|
rejected call: PVE rejects atomically, so the set-only law held throughout.
|
|
|
|
## The fix
|
|
|
|
Drop `--server` from the reconcile argv. The server address is immutable by construction —
|
|
relocating a PBS endpoint requires a fresh `create` — so there was never anything for `reconcile` to
|
|
reconcile there. `--fingerprint` (and `--password` when a secret is fed on stdin) remain, which is
|
|
the mutable identity the verb exists to push.
|
|
|
|
## Red-proof
|
|
|
|
`TestReconcileNeverPassesServerToPvesmSet` (`internal/pbsdr/manager_test.go`) isolates the
|
|
`reconcile)` block from the shipped wrapper and asserts no `--server` reaches `pvesm set`, plus that
|
|
`--fingerprint` is still pushed so the verb cannot be hollowed out. **Verified RED against the
|
|
unfixed wrapper and GREEN after the fix.**
|
|
|
|
Two traps the proof handles explicitly, both of which would have made it pass vacuously — and both
|
|
of which actually fired during authoring, which is the argument for running a proof red first:
|
|
- **Line endings.** This repo is cloned on Windows; the working copy carries CRLF, so an `\n`-only
|
|
pattern matches nothing and the guard passes silently. The pattern is `\r?\n` throughout. The
|
|
first run failed with "could not locate the reconcile) block" — the vacuous-pass failure mode,
|
|
caught only because the test was run against the broken wrapper first.
|
|
- **Comments.** The WHY note above the fix necessarily quotes `--server`, the very flag the test
|
|
forbids, so the suite went red *after* the fix was applied. Comment lines are now stripped before
|
|
matching.
|
|
|
|
## Deliberately NOT fixed here — each still open
|
|
|
|
1. **R-39's primary half.** The agent re-applies on a change of the **descriptor hash**
|
|
(`internal/pbsdr/manager.go` ~L235:
|
|
`if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { return }`).
|
|
A hub credential re-issue leaves the descriptor **byte-identical** — same `token_id`, same
|
|
`fingerprint`; only the side-table secret rotates — and bumps only `desired_generation`. So a
|
|
converged agent short-circuits and never consumes the fresh secret. This wrapper fix makes the
|
|
apply *succeed* once the agent is made to re-apply; **it does not make it re-apply.**
|
|
2. **The verify-loop read.** `pbs: cannot read token secret … permission denied` — the non-root
|
|
agent reads `/etc/pve/priv/storage/<id>.pw` **directly**, a path it can only ever *write* through
|
|
the root wrapper. `/etc/pve/priv` is `0700 root:www-data` and sudoers exposes only
|
|
`create|reconcile|grant` — **there is no read verb**. The loop is permanently blind to exactly
|
|
the failure it exists to catch.
|
|
|
|
Both ride the spec'd R-39 agent train. Recorded, not improvised.
|
|
|
|
## The demo box — HEALED and proven (2026-07-18)
|
|
|
|
The wrapper is hotfixed on the N100 (`/usr/local/sbin/felhom-pbs-apply`, backup kept at
|
|
`.bak-20260718-preR39`, `bash -n` clean) so the repo and the box agree.
|
|
|
|
Diagnosis had consumed the one pending secret against the still-unfixed wrapper; the agent parked
|
|
correctly in `consumed-failed.json` with `NOT retrying silently` — **no burn loop**, the fail-safe
|
|
worked. Viktor then clicked **"Re-issue PBS credentials"** (the one password-gated action CC cannot
|
|
perform) and the whole chain closed itself in **9 seconds**:
|
|
|
|
| Time (CEST) | Event |
|
|
|---|---|
|
|
| 20:28:44 | hub: `tenantsync: reissue ok` → `pbsdr credentials re-issued … fresh consume-once secret stored` |
|
|
| 20:28:51 | agent: `one-time token secret consumed`, wrapper `reconcile` invoked — **now the patched one** |
|
|
| 20:28:53 | agent: **`pbsdr: converged state=applied`** — no CONSUMED-BUT-FAILED this time |
|
|
|
|
`consumed-failed.json` was cleared automatically by `finishConverged` and `marker.json` rewritten.
|
|
The agent picked the re-issue up unaided because `marker.json` was absent, so the L235 hash
|
|
short-circuit did not apply.
|
|
|
|
**Proof the tier is genuinely alive, not merely reported alive:**
|
|
|
|
| Check | Before | After |
|
|
|---|---|---|
|
|
| `pvesm status` | `401 Unauthorized` / `inactive` | **`active`** |
|
|
| Direct token probe, `/api2/json/version` | `401` | **`200`** |
|
|
| Real backup | none possible | **`felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z`, 9 744 319 312 B, finished in 13m36s** |
|
|
|
|
The backup is encrypted under key fingerprint `7e:a6:af:f7:ea:6d:3e:d9` — the **escrowed** key, i.e.
|
|
the one customer zero holds the recovery code for. This is the DR tier's **first real backup on the
|
|
reborn box**.
|
|
|
|
Nothing was destroyed at any point: `.pw`, `.enc` (K), and the `storage.cfg` entry were verified
|
|
intact after the rejected `pvesm set` (PVE rejects atomically, so the set-only law held).
|