admin
7aeb7caefe
F2-a red-proof PASS: parent came up own peer group (shared:94, not shared:1), no doubling, guest sees both drives. F2-b PASS: reboot with the exact snapshot-delete stuck state → agent auto-recovered (unlock → removed real dangling vzdump snapshot → started CT). Zero spurious pages. Two live-caught fixes noted (?source=active, AllowWarnings). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
71 lines
4.9 KiB
Markdown
71 lines
4.9 KiB
Markdown
# REPORT — felhom-agent v0.49.0
|
|
|
|
Closes the two host-reboot findings (F2-a, F2-b) from
|
|
`felhom.eu/documentation/audits/TESTRUN-fullstack-2026-06-29.md`. Diagnose-before-fix on both; both root
|
|
causes confirmed live on `felhom-pve` before any code change.
|
|
|
|
## F2-b — reboot-during-backup stale-lock recovery (Part B)
|
|
|
|
**Diagnosis (B.0, confirmed live).** An interrupted vzdump leaves the guest config with a backup lock
|
|
(`lock: snapshot-delete` after a snapshot-mode backup's cleanup, or `lock: backup`) + a dangling `vzdump`
|
|
snapshot. `onboot:1` then fails: *"Starting CT … failed: CT is locked (snapshot-delete)"* — the box stays
|
|
DOWN. **Invariant:** at agent startup the agent's own backup loop has not run, so a backup lock present then
|
|
is stale by definition — *unless* a vzdump is genuinely in-flight (external, or one that outlived a bare
|
|
agent restart). That is the one case clearing would corrupt, so the recovery confirms no vzdump is running
|
|
and **fails safe** (leaves the lock) if it can't.
|
|
|
|
**Fix.** `internal/localapi/stalelock.go` (NEW) + `Server.RecoverStaleLockedGuests`, run at startup next to
|
|
`ReassertGuestBinds`/`RecoverFormatJob`. Per guest with a backup lock and no in-flight vzdump: `pct unlock`
|
|
→ delete the dangling `vzdump` snapshot (API + WaitTask, only when one exists) → start **iff** `onboot` and
|
|
not already running. Reads (`GuestConfig.Lock()`/`OnBoot()`, `ListSnapshots`, `ListRunningTasks`),
|
|
snapshot-delete and start go through the API token; only `pct unlock` shells out (no API equivalent).
|
|
|
|
**Grants.** New narrow `FELHOM_STALELOCK = /usr/sbin/pct unlock [0-9]*` + Critical capability
|
|
`stalelock-unlock`. `visudo -cf` clean; the manifest↔sudoers build gate covers it.
|
|
|
|
**Tests.** Sequence + companions: no-lock → nothing called; `migrate` lock → left alone; onboot=0 →
|
|
unlocked-but-not-started; delsnapshot only when a snapshot exists; **invariant guard** (live backup → not
|
|
cleared; unconfirmable → fail-safe); already-running → not restarted; nil controller → no-op.
|
|
|
|
**Live (B.3 + §D).** Simulated stale state on scratch 9999 → agent restart auto-recovered (unlock +
|
|
start); the **invariant guard was proven against a REAL in-flight 9201 backup** (left its lock alone). Then
|
|
the **§D supervised reboot** validated the genuine path: a reboot with the EXACT F2-b state (induced
|
|
`snapshot-delete` lock + a real dangling `vzdump` snapshot) reproduced the stuck symptom (pve-guests *"CT
|
|
is locked (snapshot-delete)"* → *"Starting CT 9201 failed"*), then the agent auto-recovered — `clearing a
|
|
stale backup lock` → **`removed dangling vzdump snapshot`** (the real delsnapshot path) → `started CT`. 9201
|
|
running, lock cleared, snapshot deleted, no error. Two live-caught bugs fixed en route: PVE 9.x rejects
|
|
`?running=1` (→ `?source=active`); the LXC-start `WARNINGS: 1` nesting advisory false-failed the start
|
|
(→ `AllowWarnings`).
|
|
|
|
## F2-a — shared-parent boot script never redeployed (Part C)
|
|
|
|
**Diagnosis (C.0, confirmed live).** `/mnt/felhom-drives` was in root's `shared:1` peer group, so each
|
|
drive bind appeared twice. Reproduced exactly in a scratch dir at `/mnt`: the **stale** sequence (`bind +
|
|
make-shared`, no `make-private`) leaves the self-bind in `shared:1` → sub-bind count 2 (doubling); the
|
|
**correct** sequence (`bind → make-private → make-shared`) gives an own group → count 1. The live boot
|
|
script was the pre-v0.36.6 body (no `make-private`). Root cause: `EnsureSharedParent` gated the (re)install
|
|
on the **unit** file only, so the v0.36.6 *script-only* fix never deployed to a host whose unit was current.
|
|
|
|
**Fix (provably safe, boot-time-only).** New `sharedParentInstallStale` compares **both** the script and
|
|
unit (missing/differing → reinstall). It rewrites the on-disk script; it does **not** touch the live mount
|
|
(the live bind/make-private/make-shared stays guarded on `!isHostMountpoint`), so it can't orphan the
|
|
guest's slave bind. The doubling persists live until the next boot, when the corrected script gives the
|
|
parent its own peer group.
|
|
|
|
**Tests.** Stale-script/current-unit → reinstall (the F2-a regression); both-current → no-op; missing →
|
|
stale; content guard that the shipped script keeps `make-private`.
|
|
|
|
**Red-proof (C.1) — PASSED (§D reboot).** `/mnt/felhom-drives` came up as its **own peer group
|
|
`shared:94`** (NOT root's `shared:1`) with **exactly 1 host bind per drive** (doubling gone, was 2); the
|
|
guest still sees both drives (`bound_under_parent=true`; apps healthy). Consistent across two reboots.
|
|
(Separate pre-existing, harmless observation: the guest shows 2 mounts per drive — two parent mounts, same
|
|
`/felhom-data` source, correctly `master:`-slaved; not a regression of this host-side fix.)
|
|
|
|
## Green gate
|
|
|
|
`go build ./... && go vet ./... && go test ./...` — all green (full suite, 0 failures). Version
|
|
`0.48.0 → 0.49.0`.
|
|
|
|
Secrets-safe: no tokens/keys logged or written. Leaf fingerprints are not secret; the operator email is not
|
|
a secret.
|