felhom-agent/REPORT.md

REPORT — felhom-agent v0.49.0

Closes the two host-reboot findings (F2-a, F2-b) from felhom.eu/documentation/audits/TESTRUN-fullstack-2026-06-29.md. Diagnose-before-fix on both; both root causes confirmed live on felhom-pve before any code change.

F2-b — reboot-during-backup stale-lock recovery (Part B)

Diagnosis (B.0, confirmed live). An interrupted vzdump leaves the guest config with a backup lock (lock: snapshot-delete after a snapshot-mode backup's cleanup, or lock: backup) + a dangling vzdump snapshot. onboot:1 then fails: "Starting CT … failed: CT is locked (snapshot-delete)" — the box stays DOWN. Invariant: at agent startup the agent's own backup loop has not run, so a backup lock present then is stale by definition — unless a vzdump is genuinely in-flight (external, or one that outlived a bare agent restart). That is the one case clearing would corrupt, so the recovery confirms no vzdump is running and fails safe (leaves the lock) if it can't.

Fix. internal/localapi/stalelock.go (NEW) + Server.RecoverStaleLockedGuests, run at startup next to ReassertGuestBinds/RecoverFormatJob. Per guest with a backup lock and no in-flight vzdump: pct unlock → delete the dangling vzdump snapshot (API + WaitTask, only when one exists) → start iff onboot and not already running. Reads (GuestConfig.Lock()/OnBoot(), ListSnapshots, ListRunningTasks), snapshot-delete and start go through the API token; only pct unlock shells out (no API equivalent).

Grants. New narrow FELHOM_STALELOCK = /usr/sbin/pct unlock [0-9]* + Critical capability stalelock-unlock. visudo -cf clean; the manifest↔sudoers build gate covers it.

Tests. Sequence + companions: no-lock → nothing called; migrate lock → left alone; onboot=0 → unlocked-but-not-started; delsnapshot only when a snapshot exists; invariant guard (live backup → not cleared; unconfirmable → fail-safe); already-running → not restarted; nil controller → no-op.

Live (B.3 + §D). Simulated stale state on scratch 9999 → agent restart auto-recovered (unlock + start); the invariant guard was proven against a REAL in-flight 9201 backup (left its lock alone). Then the §D supervised reboot validated the genuine path: a reboot with the EXACT F2-b state (induced snapshot-delete lock + a real dangling vzdump snapshot) reproduced the stuck symptom (pve-guests "CT is locked (snapshot-delete)" → "Starting CT 9201 failed"), then the agent auto-recovered — clearing a stale backup lock → removed dangling vzdump snapshot (the real delsnapshot path) → started CT. 9201 running, lock cleared, snapshot deleted, no error. Two live-caught bugs fixed en route: PVE 9.x rejects ?running=1 (→ ?source=active); the LXC-start WARNINGS: 1 nesting advisory false-failed the start (→ AllowWarnings).

F2-a — shared-parent boot script never redeployed (Part C)

Diagnosis (C.0, confirmed live). /mnt/felhom-drives was in root's shared:1 peer group, so each drive bind appeared twice. Reproduced exactly in a scratch dir at /mnt: the stale sequence (bind + make-shared, no make-private) leaves the self-bind in shared:1 → sub-bind count 2 (doubling); the correct sequence (bind → make-private → make-shared) gives an own group → count 1. The live boot script was the pre-v0.36.6 body (no make-private). Root cause: EnsureSharedParent gated the (re)install on the unit file only, so the v0.36.6 script-only fix never deployed to a host whose unit was current.

Fix (provably safe, boot-time-only). New sharedParentInstallStale compares both the script and unit (missing/differing → reinstall). It rewrites the on-disk script; it does not touch the live mount (the live bind/make-private/make-shared stays guarded on !isHostMountpoint), so it can't orphan the guest's slave bind. The doubling persists live until the next boot, when the corrected script gives the parent its own peer group.

Tests. Stale-script/current-unit → reinstall (the F2-a regression); both-current → no-op; missing → stale; content guard that the shipped script keeps make-private.

Red-proof (C.1) — PASSED (§D reboot). /mnt/felhom-drives came up as its own peer group shared:94 (NOT root's shared:1) with exactly 1 host bind per drive (doubling gone, was 2); the guest still sees both drives (bound_under_parent=true; apps healthy). Consistent across two reboots. (Separate pre-existing, harmless observation: the guest shows 2 mounts per drive — two parent mounts, same /felhom-data source, correctly master:-slaved; not a regression of this host-side fix.)

Green gate

go build ./... && go vet ./... && go test ./... — all green (full suite, 0 failures). Version 0.48.0 → 0.49.0.

Secrets-safe: no tokens/keys logged or written. Leaf fingerprints are not secret; the operator email is not a secret.