agent v0.49.0: reboot-during-backup stale-lock recovery (F2-b) + shared-parent script redeploy fix (F2-a)

F2-b: at startup, recover a guest left with a stale vzdump lock by a
reboot-during-backup — pct unlock -> delete dangling vzdump snapshot ->
start iff onboot, guarded by a no-vzdump-running invariant (fail-safe).
New internal/localapi/stalelock.go; proxmox GuestConfig.Lock()/OnBoot(),
ListSnapshots, ListRunningTasks, Snapshot type. New narrow sudoers grant
FELHOM_STALELOCK (pct unlock) + Critical capability stalelock-unlock.

F2-a: EnsureSharedParent only redeployed the boot script when the UNIT
differed, so the v0.36.6 make-private fix never reached hosts whose unit
was current -> /mnt/felhom-drives stayed in root's shared:1 and doubled
every drive bind. New sharedParentInstallStale compares BOTH script and
unit. Boot-time-only; never churns the live mount.

Both root causes confirmed live on felhom-pve before fixing. Green gate
(build/vet/test) all pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
This commit is contained in:
2026-06-30 09:00:32 +02:00
parent 81954c3d2a
commit 6e38e2f921
12 changed files with 743 additions and 32 deletions
+55 -25
View File
@@ -1,31 +1,61 @@
# REPORT — agent reports served leaf fingerprint (hub re-key detection, Part A), v0.48.0
# REPORT — felhom-agent v0.49.0
**Task:** the last self-health leg — the agent reports its served local-API **leaf fingerprint** so the
hub can detect an agent re-key fleet-wide (`host_leaf_changed`, hub v0.22.x). This repo = **Part A**.
Closes the two host-reboot findings (F2-a, F2-b) from
`felhom.eu/documentation/audits/TESTRUN-fullstack-2026-06-29.md`. Diagnose-before-fix on both; both root
causes confirmed live on `felhom-pve` before any code change.
**Baseline:** agent `main` @ `3844df7` (v0.47.0) → **v0.48.0** @ `bf8e3be`, sha `a3dad050…`.
## F2-b — reboot-during-backup stale-lock recovery (Part B)
## Change
- **`internal/hub/report.go`:** new `HostReport.LeafFingerprint string` (`json:"leaf_fingerprint"`) —
the SHA-256 of the leaf the agent currently serves. Empty when the local API is disabled (no leaf) →
the hub treats "" as unknown, never an alert. Not a secret (the fp is public; the token is never
reported).
- **`internal/hub/collect.go` + `cmd/felhom-agent/main.go`:** `Collector.SetLeafFingerprint(fp)` threads
the `fp` from `EnsureLeaf` (the SAME value the LOADED/REGENERATED log reports) into every report, next
to `Capabilities`.
**Diagnosis (B.0, confirmed live).** An interrupted vzdump leaves the guest config with a backup lock
(`lock: snapshot-delete` after a snapshot-mode backup's cleanup, or `lock: backup`) + a dangling `vzdump`
snapshot. `onboot:1` then fails: *"Starting CT … failed: CT is locked (snapshot-delete)"* — the box stays
DOWN. **Invariant:** at agent startup the agent's own backup loop has not run, so a backup lock present then
is stale by definition — *unless* a vzdump is genuinely in-flight (external, or one that outlived a bare
agent restart). That is the one case clearing would corrupt, so the recovery confirms no vzdump is running
and **fails safe** (leaves the lock) if it can't.
## Tests (green: `go build/vet/test ./...`)
`TestCollect_LeafFingerprint`: the report carries the fp when set, and `""` when unset (local API
disabled) — companion proving the threading is what populates it. The golden + contract + field-names
tests updated; the cross-repo golden mirrors `leaf_fingerprint` (byte-identical with the hub copy).
**Fix.** `internal/localapi/stalelock.go` (NEW) + `Server.RecoverStaleLockedGuests`, run at startup next to
`ReassertGuestBinds`/`RecoverFormatJob`. Per guest with a backup lock and no in-flight vzdump: `pct unlock`
→ delete the dangling `vzdump` snapshot (API + WaitTask, only when one exists) → start **iff** `onboot` and
not already running. Reads (`GuestConfig.Lock()`/`OnBoot()`, `ListSnapshots`, `ListRunningTasks`),
snapshot-delete and start go through the API token; only `pct unlock` shells out (no API equivalent).
## Live validation (felhom-pve) — PASS
Deployed 0.48.0 (`leaf LOADED` fp `60b5974d…`, capabilities 45/45). The hub received the report with
`leaf_fingerprint: 60b5974d…` (confirmed in the hub DB), then — during the end-to-end re-key test — a
regenerated leaf (`82078fab…`) and its restore (`60b5974d…`) both rode the report stream and the hub
raised `host_leaf_changed` on the change (see felhom.eu/REPORT.md). Final state: leaf == pin
`60b5974d…`, capabilities 45/45, channel recovered.
**Grants.** New narrow `FELHOM_STALELOCK = /usr/sbin/pct unlock [0-9]*` + Critical capability
`stalelock-unlock`. `visudo -cf` clean; the manifest↔sudoers build gate covers it.
## NOT changed
The pin/transport, the token store, the sudoers/capability surface, the swap, the channel-health logic.
Report contract + collector threading only.
**Tests.** Sequence + companions: no-lock → nothing called; `migrate` lock → left alone; onboot=0 →
unlocked-but-not-started; delsnapshot only when a snapshot exists; **invariant guard** (live backup → not
cleared; unconfirmable → fail-safe); already-running → not restarted; nil controller → no-op.
**Live (B.3).** Simulated stale state on a scratch guest (lock + dangling `vzdump` snapshot) → agent
restart auto-recovered (unlock + delsnapshot + start). The real reboot-during-backup is the §D supervised
step.
## F2-a — shared-parent boot script never redeployed (Part C)
**Diagnosis (C.0, confirmed live).** `/mnt/felhom-drives` was in root's `shared:1` peer group, so each
drive bind appeared twice. Reproduced exactly in a scratch dir at `/mnt`: the **stale** sequence (`bind +
make-shared`, no `make-private`) leaves the self-bind in `shared:1` → sub-bind count 2 (doubling); the
**correct** sequence (`bind → make-private → make-shared`) gives an own group → count 1. The live boot
script was the pre-v0.36.6 body (no `make-private`). Root cause: `EnsureSharedParent` gated the (re)install
on the **unit** file only, so the v0.36.6 *script-only* fix never deployed to a host whose unit was current.
**Fix (provably safe, boot-time-only).** New `sharedParentInstallStale` compares **both** the script and
unit (missing/differing → reinstall). It rewrites the on-disk script; it does **not** touch the live mount
(the live bind/make-private/make-shared stays guarded on `!isHostMountpoint`), so it can't orphan the
guest's slave bind. The doubling persists live until the next boot, when the corrected script gives the
parent its own peer group.
**Tests.** Stale-script/current-unit → reinstall (the F2-a regression); both-current → no-op; missing →
stale; content guard that the shipped script keeps `make-private`.
**Red-proof (C.1).** Requires a reboot — the §D supervised step: parent must come up as its own group
(not `shared:1`) AND the guest must still see both drives (`bound_under_parent=true`).
## Green gate
`go build ./... && go vet ./... && go test ./...` — all green (full suite, 0 failures). Version
`0.48.0 → 0.49.0`.
Secrets-safe: no tokens/keys logged or written. Leaf fingerprints are not secret; the operator email is not
a secret.