Campaign 8: F-REBOOT/F-LEAK/F-OBS closed; R-100 filed (offsite staleness reset by failures)
F-LEAK's first fix (pool adoption) was refuted live and is recorded as such. R-100 is F-CRIT-2's defect class on the hub for the restic tier: isStale reads only LastRun, which the controller writes unconditionally on failure. Investigated, not fixed.
This commit is contained in:
@@ -371,10 +371,92 @@ no re-backup thrash (with 91 scheduler ticks proving the loop was alive). Detail
|
||||
two real snapshots PLUS the phantom — so the feared retention/data-loss scenario does not occur. It
|
||||
never removes them either; filed as R-99.
|
||||
|
||||
### F-REBOOT — FIXED (agent v0.107.0, 2026-07-28)
|
||||
A periodic guest-power watchdog (`internal/localapi/guestpower.go`, 60 s) starts a guest that is
|
||||
`onboot:1`, stopped, unlocked, and has no vzdump in flight. It closes the two narrow gaps that let
|
||||
`RecoverStaleLockedGuests` miss fault 11: that recovery acts only on a guest holding a **stale vzdump
|
||||
lock** (fault 11's guest was unlocked) and runs **once at agent startup** (fault 11's guest went down
|
||||
while the agent was already up).
|
||||
|
||||
`onboot` is the deliberate-stop discriminator, and it is not invented here — it is already what the
|
||||
stale-lock path uses for this decision, it is 0 on scratch/golden guests, and it is what `pve-guests`
|
||||
itself consults at host boot, so the agent agrees with the platform rather than keeping a second
|
||||
private definition of "should be running". Retry is bounded (3 attempts, 1m/2m/4m) and then escalates
|
||||
once; an unbounded silent retry loop would be the over-correction.
|
||||
|
||||
Replayed live on demo-hp: a guest stopped out from under the agent came back **unattended in 120 s**,
|
||||
against the **587 s** the finding's original incident needed a human. The lock-deferral guard fired
|
||||
live during the replay. Scenario B proven on the same box — an `onboot:0` guest was left stopped
|
||||
across the whole window.
|
||||
|
||||
### F-LEAK — FIXED (host-install v1.21.0, 2026-07-28) — and the first fix was WRONG
|
||||
`FelhomAgentGuest` is now granted at each `/vms/990000`…`/vms/990009` — the restore-test's scratch
|
||||
band. The cause is structural rather than a missing privilege in the role: the role is granted at
|
||||
`/pool/felhom`, and **a guest only joins that pool when its restore completes**, so a *failed*
|
||||
restore-test leaves a guest that exists, is in no pool, and is outside the token's reach.
|
||||
|
||||
**The first attempt was refuted live and is recorded because it looked right.** Agent v0.107.0 shipped
|
||||
a teardown fallback that adopted the stranded guest into the pool and retried. It fired exactly as
|
||||
designed and PVE refused it: `PUT /pools/felhom -> HTTP 500: permission denied at /vms/990000 (missing
|
||||
privilege ...)`. `PUT /pools/{pool}` **also** requires `VM.Allocate` on the VM being added — pool
|
||||
membership cannot bootstrap its own authority. That code was removed in agent v0.108.0.
|
||||
|
||||
**Why the grant is not a widening**, proven live at the same seam the defect lives in — and note that
|
||||
PVE checks **permission before existence**, so these 403s are genuine refusals, not artifacts:
|
||||
|
||||
| `DELETE /nodes/<node>/lxc/<id>` with the agent's own token | result |
|
||||
|---|---|
|
||||
| `990000` (stranded, pool-less, restored scratch) | **destroyed** — the defect's exact case |
|
||||
| `990010` (one past the band) | **403** `Permission check failed (/vms/990010, VM.Allocate)` |
|
||||
| `100` (arbitrary non-pool guest) | **403** same |
|
||||
|
||||
Granting at `/vms` was rejected: it would authorise destroying every guest on the box, including a
|
||||
co-tenant's.
|
||||
|
||||
### F-OBS — FIXED (controller v0.180.0, 2026-07-28)
|
||||
`deadapp-check` now emits a summary line at **INFO** every 20th scan (10 min at its 30 s cadence),
|
||||
carrying scans-since-boot, stacks evaluated, and how many are currently down. The original silence
|
||||
came from its per-cycle line going through `Scheduler.dbg()`, which is gated on `logging.level==debug`
|
||||
and so was never *produced* on a default box — meaning "no alarms" was indistinguishable from "the
|
||||
detector never ran", the exact fallacy this project has a standing rule against. A line per run was
|
||||
not an option either (2880/day is what made silence attractive), so the cadence is asserted by test in
|
||||
both directions: it must not flood, and it must stay frequent enough to expose a stalled detector
|
||||
inside the 180 s alarm grace it feeds.
|
||||
|
||||
*(Full write-ups for every finding, with complete evidence, are in `~/campaign8/evidence/phaseB/`.)*
|
||||
|
||||
---
|
||||
|
||||
## 6b. Follow-up investigation, 2026-07-28 — R-100 (filed, deliberately NOT fixed)
|
||||
|
||||
### A restic offsite tier that fails every night never goes stale on the hub
|
||||
|
||||
This is **F-CRIT-2's defect class, one layer up and on the other tier** — a *failed* run resetting the
|
||||
freshness clock — and it was found by asking whether the F-CRIT-2 shape existed anywhere else.
|
||||
|
||||
Two halves, each verified in the source rather than inferred:
|
||||
|
||||
- **Controller.** `o.LastRun = time.Now()` is set **unconditionally** in
|
||||
`controller/internal/backup/offbox.go:716`, outside the `runErr` branch. The failure is recorded
|
||||
faithfully — but in a *different* field: `o.LastStatus = "error"` at :725.
|
||||
- **Hub.** `isStale()` reads **only** `off.LastRun`
|
||||
(`hub/internal/monitor/offsite.go:120`, `:127`, `:131`). It never consults `LastStatus`.
|
||||
|
||||
So a nightly restic run that fails every night keeps `LastRun` fresh, `isStale` is permanently false,
|
||||
and the staleness alarm never fires — while no successful offsite backup has occurred at all.
|
||||
|
||||
**Scope of the silence, stated precisely.** `LastStatus` *does* reach the hub: it is parsed into the
|
||||
report struct and **only logged** (`offsite.go:270`). It drives no checker and no notification. The
|
||||
controller's own guest UI does surface `LastStatus="error"`, so the failure is *visible to someone who
|
||||
looks*; what is missing is the push — the operator's fleet-wide alarm plane is silent, which is the
|
||||
plane that matters for an unattended appliance.
|
||||
|
||||
**Not fixed**, per this task's investigation-only scope. Filed as **R-100**; the fix direction is to
|
||||
gate staleness on the last *successful* run rather than the last attempt, which is precisely what
|
||||
F-CRIT-2's `NewestArchiveTime` fix did for the PBS tier.
|
||||
|
||||
---
|
||||
|
||||
## 7. What can come off the "never validated" list
|
||||
|
||||
**Retired — proven live tonight:**
|
||||
|
||||
@@ -26,6 +26,10 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-99** | Server-side prune **never removes** a phantom snapshot. Confirmed it does NOT count them toward `keep-last` (dry-run kept 2 real + the phantom) so there is **no retention/data-loss bug** — but one accumulates per aborted upload, forever | READY (S) | — | Decide a cleanup path. Deletion on a **customer** datastore is a separate ruling — detection shipped, removal deliberately not automated | CC |
|
||||
| **F-CRIT-1** | ~~An app that **fails to restart** after a quiesce never alarms on any channel — `restartAll` discarded the error AND `StateStopped` was whitelisted on invariant I1, which the quiesce path had made false~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | Both causes fixed. Live on demo-hp: alarmed 9s after grace expiry, banner shows `(stopped)`; a deliberate user stop stayed silent through 9 dead-app scans | — |
|
||||
| **F-A1** | ~~A restore-test in flight made a healthy backup report as FAILED (HTTP 409 read as a tier failure): breaker armed + operator emailed, on both boxes~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → **0** operator emails, real failure → **1** | — |
|
||||
| **R-100** | **A restic offsite tier that fails every night never goes stale on the hub.** `isStale()` reads only `LastRun` (`hub/internal/monitor/offsite.go:120,127,131`) and the controller writes `LastRun` **unconditionally**, outside the error branch (`controller/internal/backup/offbox.go:716`, with `LastStatus="error"` set at :725). So a tier failing nightly keeps a fresh clock, `isStale` is permanently false, and the operator's fleet-wide alarm plane is silent. `LastStatus` reaches the hub and is **only logged** (`offsite.go:270`) — it drives no checker and no notification | READY (S) | — | **This is F-CRIT-2's defect class on the hub, for the restic tier**: a failed run resetting the freshness clock. Gate staleness on the last *successful* run, not the last attempt. Found by Part 4 investigation 2026-07-28; NOT fixed (investigation-only scope) | CC |
|
||||
| **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — |
|
||||
| **F-LEAK** | ~~A failed restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); the 10-slot VMID band shrinks silently~~ | **SHIPPED + PROVEN-LIVE** (host-install v1.21.0, 2026-07-28) | — | `FelhomAgentGuest` granted per-path at `/vms/990000..990009`. Cause was structural: the role is granted at `/pool/felhom` and a guest joins the pool only when its restore **completes**. Live A/B on demo-hp, same guest minutes apart: grant removed → `403 (/vms/990000, VM.Allocate)`; granted → `200 UPID:...vzdestroy`. `/vms/100` and `/vms/990010` still **403** (PVE checks permission before existence, so these are real refusals) | — |
|
||||
| **F-OBS** | ~~`deadapp-check` leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran"~~ | **SHIPPED + PROVEN-LIVE** (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. **Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0** — it logged only at startup and when it acted, so its health could be read only from absence | — |
|
||||
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
|
||||
| **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
|
||||
| **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
|
||||
|
||||
Reference in New Issue
Block a user