Campaign 8: F-REBOOT/F-LEAK/F-OBS closed; R-100 filed (offsite staleness reset by failures)

F-LEAK's first fix (pool adoption) was refuted live and is recorded as such. R-100 is F-CRIT-2's defect class on the hub for the restic tier: isStale reads only LastRun, which the controller writes unconditionally on failure. Investigated, not fixed.
2026-07-28 11:17:59 +02:00
parent b4c528801a
commit 8c793955b1
2 changed files with 86 additions and 0 deletions
@@ -371,10 +371,92 @@ no re-backup thrash (with 91 scheduler ticks proving the loop was alive). Detail
 two real snapshots PLUS the phantom — so the feared retention/data-loss scenario does not occur. It
 never removes them either; filed as R-99.

+### F-REBOOT — FIXED (agent v0.107.0, 2026-07-28)
+A periodic guest-power watchdog (`internal/localapi/guestpower.go`, 60 s) starts a guest that is
+`onboot:1`, stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
+`RecoverStaleLockedGuests` miss fault 11: that recovery acts only on a guest holding a **stale vzdump
+lock** (fault 11's guest was unlocked) and runs **once at agent startup** (fault 11's guest went down
+while the agent was already up).
+
+`onboot` is the deliberate-stop discriminator, and it is not invented here — it is already what the
+stale-lock path uses for this decision, it is 0 on scratch/golden guests, and it is what `pve-guests`
+itself consults at host boot, so the agent agrees with the platform rather than keeping a second
+private definition of "should be running". Retry is bounded (3 attempts, 1m/2m/4m) and then escalates
+once; an unbounded silent retry loop would be the over-correction.
+
+Replayed live on demo-hp: a guest stopped out from under the agent came back **unattended in 120 s**,
+against the **587 s** the finding's original incident needed a human. The lock-deferral guard fired
+live during the replay. Scenario B proven on the same box — an `onboot:0` guest was left stopped
+across the whole window.
+
+### F-LEAK — FIXED (host-install v1.21.0, 2026-07-28) — and the first fix was WRONG
+`FelhomAgentGuest` is now granted at each `/vms/990000`…`/vms/990009` — the restore-test's scratch
+band. The cause is structural rather than a missing privilege in the role: the role is granted at
+`/pool/felhom`, and **a guest only joins that pool when its restore completes**, so a *failed*
+restore-test leaves a guest that exists, is in no pool, and is outside the token's reach.
+
+**The first attempt was refuted live and is recorded because it looked right.** Agent v0.107.0 shipped
+a teardown fallback that adopted the stranded guest into the pool and retried. It fired exactly as
+designed and PVE refused it: `PUT /pools/felhom -> HTTP 500: permission denied at /vms/990000 (missing
+privilege ...)`. `PUT /pools/{pool}` **also** requires `VM.Allocate` on the VM being added — pool
+membership cannot bootstrap its own authority. That code was removed in agent v0.108.0.
+
+**Why the grant is not a widening**, proven live at the same seam the defect lives in — and note that
+PVE checks **permission before existence**, so these 403s are genuine refusals, not artifacts:
+
+| `DELETE /nodes/<node>/lxc/<id>` with the agent's own token | result |
+|---|---|
+| `990000` (stranded, pool-less, restored scratch) | **destroyed** — the defect's exact case |
+| `990010` (one past the band) | **403** `Permission check failed (/vms/990010, VM.Allocate)` |
+| `100` (arbitrary non-pool guest) | **403** same |
+
+Granting at `/vms` was rejected: it would authorise destroying every guest on the box, including a
+co-tenant's.
+
+### F-OBS — FIXED (controller v0.180.0, 2026-07-28)
+`deadapp-check` now emits a summary line at **INFO** every 20th scan (10 min at its 30 s cadence),
+carrying scans-since-boot, stacks evaluated, and how many are currently down. The original silence
+came from its per-cycle line going through `Scheduler.dbg()`, which is gated on `logging.level==debug`
+and so was never *produced* on a default box — meaning "no alarms" was indistinguishable from "the
+detector never ran", the exact fallacy this project has a standing rule against. A line per run was
+not an option either (2880/day is what made silence attractive), so the cadence is asserted by test in
+both directions: it must not flood, and it must stay frequent enough to expose a stalled detector
+inside the 180 s alarm grace it feeds.
+
 *(Full write-ups for every finding, with complete evidence, are in `~/campaign8/evidence/phaseB/`.)*

 ---

+## 6b. Follow-up investigation, 2026-07-28 — R-100 (filed, deliberately NOT fixed)
+
+### A restic offsite tier that fails every night never goes stale on the hub
+
+This is **F-CRIT-2's defect class, one layer up and on the other tier** — a *failed* run resetting the
+freshness clock — and it was found by asking whether the F-CRIT-2 shape existed anywhere else.
+
+Two halves, each verified in the source rather than inferred:
+
+- **Controller.** `o.LastRun = time.Now()` is set **unconditionally** in
+  `controller/internal/backup/offbox.go:716`, outside the `runErr` branch. The failure is recorded
+  faithfully — but in a *different* field: `o.LastStatus = "error"` at :725.
+- **Hub.** `isStale()` reads **only** `off.LastRun`
+  (`hub/internal/monitor/offsite.go:120`, `:127`, `:131`). It never consults `LastStatus`.
+
+So a nightly restic run that fails every night keeps `LastRun` fresh, `isStale` is permanently false,
+and the staleness alarm never fires — while no successful offsite backup has occurred at all.
+
+**Scope of the silence, stated precisely.** `LastStatus` *does* reach the hub: it is parsed into the
+report struct and **only logged** (`offsite.go:270`). It drives no checker and no notification. The
+controller's own guest UI does surface `LastStatus="error"`, so the failure is *visible to someone who
+looks*; what is missing is the push — the operator's fleet-wide alarm plane is silent, which is the
+plane that matters for an unattended appliance.
+
+**Not fixed**, per this task's investigation-only scope. Filed as **R-100**; the fix direction is to
+gate staleness on the last *successful* run rather than the last attempt, which is precisely what
+F-CRIT-2's `NewestArchiveTime` fix did for the PBS tier.
+
+---
+
 ## 7. What can come off the "never validated" list

 **Retired — proven live tonight:**