Files
felhom-agent/REPORT.md
T
admin 6e38e2f921 agent v0.49.0: reboot-during-backup stale-lock recovery (F2-b) + shared-parent script redeploy fix (F2-a)
F2-b: at startup, recover a guest left with a stale vzdump lock by a
reboot-during-backup — pct unlock -> delete dangling vzdump snapshot ->
start iff onboot, guarded by a no-vzdump-running invariant (fail-safe).
New internal/localapi/stalelock.go; proxmox GuestConfig.Lock()/OnBoot(),
ListSnapshots, ListRunningTasks, Snapshot type. New narrow sudoers grant
FELHOM_STALELOCK (pct unlock) + Critical capability stalelock-unlock.

F2-a: EnsureSharedParent only redeployed the boot script when the UNIT
differed, so the v0.36.6 make-private fix never reached hosts whose unit
was current -> /mnt/felhom-drives stayed in root's shared:1 and doubled
every drive bind. New sharedParentInstallStale compares BOTH script and
unit. Boot-time-only; never churns the live mount.

Both root causes confirmed live on felhom-pve before fixing. Green gate
(build/vet/test) all pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
2026-06-30 09:00:32 +02:00

3.9 KiB

REPORT — felhom-agent v0.49.0

Closes the two host-reboot findings (F2-a, F2-b) from felhom.eu/documentation/audits/TESTRUN-fullstack-2026-06-29.md. Diagnose-before-fix on both; both root causes confirmed live on felhom-pve before any code change.

F2-b — reboot-during-backup stale-lock recovery (Part B)

Diagnosis (B.0, confirmed live). An interrupted vzdump leaves the guest config with a backup lock (lock: snapshot-delete after a snapshot-mode backup's cleanup, or lock: backup) + a dangling vzdump snapshot. onboot:1 then fails: "Starting CT … failed: CT is locked (snapshot-delete)" — the box stays DOWN. Invariant: at agent startup the agent's own backup loop has not run, so a backup lock present then is stale by definition — unless a vzdump is genuinely in-flight (external, or one that outlived a bare agent restart). That is the one case clearing would corrupt, so the recovery confirms no vzdump is running and fails safe (leaves the lock) if it can't.

Fix. internal/localapi/stalelock.go (NEW) + Server.RecoverStaleLockedGuests, run at startup next to ReassertGuestBinds/RecoverFormatJob. Per guest with a backup lock and no in-flight vzdump: pct unlock → delete the dangling vzdump snapshot (API + WaitTask, only when one exists) → start iff onboot and not already running. Reads (GuestConfig.Lock()/OnBoot(), ListSnapshots, ListRunningTasks), snapshot-delete and start go through the API token; only pct unlock shells out (no API equivalent).

Grants. New narrow FELHOM_STALELOCK = /usr/sbin/pct unlock [0-9]* + Critical capability stalelock-unlock. visudo -cf clean; the manifest↔sudoers build gate covers it.

Tests. Sequence + companions: no-lock → nothing called; migrate lock → left alone; onboot=0 → unlocked-but-not-started; delsnapshot only when a snapshot exists; invariant guard (live backup → not cleared; unconfirmable → fail-safe); already-running → not restarted; nil controller → no-op.

Live (B.3). Simulated stale state on a scratch guest (lock + dangling vzdump snapshot) → agent restart auto-recovered (unlock + delsnapshot + start). The real reboot-during-backup is the §D supervised step.

F2-a — shared-parent boot script never redeployed (Part C)

Diagnosis (C.0, confirmed live). /mnt/felhom-drives was in root's shared:1 peer group, so each drive bind appeared twice. Reproduced exactly in a scratch dir at /mnt: the stale sequence (bind + make-shared, no make-private) leaves the self-bind in shared:1 → sub-bind count 2 (doubling); the correct sequence (bind → make-private → make-shared) gives an own group → count 1. The live boot script was the pre-v0.36.6 body (no make-private). Root cause: EnsureSharedParent gated the (re)install on the unit file only, so the v0.36.6 script-only fix never deployed to a host whose unit was current.

Fix (provably safe, boot-time-only). New sharedParentInstallStale compares both the script and unit (missing/differing → reinstall). It rewrites the on-disk script; it does not touch the live mount (the live bind/make-private/make-shared stays guarded on !isHostMountpoint), so it can't orphan the guest's slave bind. The doubling persists live until the next boot, when the corrected script gives the parent its own peer group.

Tests. Stale-script/current-unit → reinstall (the F2-a regression); both-current → no-op; missing → stale; content guard that the shipped script keeps make-private.

Red-proof (C.1). Requires a reboot — the §D supervised step: parent must come up as its own group (not shared:1) AND the guest must still see both drives (bound_under_parent=true).

Green gate

go build ./... && go vet ./... && go test ./... — all green (full suite, 0 failures). Version 0.48.0 → 0.49.0.

Secrets-safe: no tokens/keys logged or written. Leaf fingerprints are not secret; the operator email is not a secret.