REPORT + OPEN-ITEMS: R-100 closed and proven live; R-101 filed
This commit is contained in:
@@ -26,7 +26,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-99** | Server-side prune **never removes** a phantom snapshot. Confirmed it does NOT count them toward `keep-last` (dry-run kept 2 real + the phantom) so there is **no retention/data-loss bug** — but one accumulates per aborted upload, forever | READY (S) | — | Decide a cleanup path. Deletion on a **customer** datastore is a separate ruling — detection shipped, removal deliberately not automated | CC |
|
||||
| **F-CRIT-1** | ~~An app that **fails to restart** after a quiesce never alarms on any channel — `restartAll` discarded the error AND `StateStopped` was whitelisted on invariant I1, which the quiesce path had made false~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | Both causes fixed. Live on demo-hp: alarmed 9s after grace expiry, banner shows `(stopped)`; a deliberate user stop stayed silent through 9 dead-app scans | — |
|
||||
| **F-A1** | ~~A restore-test in flight made a healthy backup report as FAILED (HTTP 409 read as a tier failure): breaker armed + operator emailed, on both boxes~~ | **SHIPPED + PROVEN-LIVE** (controller v0.179.0, 2026-07-28) | — | 409 → contention: tier stays DUE, dropped before anything stops (15m), and BLOCKED alarm if contention outlives the agent's 120m ceiling (3h). Hub DB: 409 → **0** operator emails, real failure → **1** | — |
|
||||
| **R-100** | **A restic offsite tier that fails every night never goes stale on the hub.** `isStale()` reads only `LastRun` (`hub/internal/monitor/offsite.go:120,127,131`) and the controller writes `LastRun` **unconditionally**, outside the error branch (`controller/internal/backup/offbox.go:716`, with `LastStatus="error"` set at :725). So a tier failing nightly keeps a fresh clock, `isStale` is permanently false, and the operator's fleet-wide alarm plane is silent. `LastStatus` reaches the hub and is **only logged** (`offsite.go:270`) — it drives no checker and no notification | READY (S) | — | **This is F-CRIT-2's defect class on the hub, for the restic tier**: a failed run resetting the freshness clock. Gate staleness on the last *successful* run, not the last attempt. Found by Part 4 investigation 2026-07-28; NOT fixed (investigation-only scope) | CC |
|
||||
| **R-100** | ~~A restic offsite tier that fails every night never goes stale on the hub — `isStale` counted from `LastRun`, which the controller writes unconditionally on failure~~ | **SHIPPED + PROVEN-LIVE** (controller v0.181.0 + hub v0.80.0, 2026-07-28) | — | Anchored on a new `last_success`. **Severity corrected during Phase 0: this was NOT a silence** — `backup_failed` does fire nightly and reaches the operator (live DB: 5 sends). The real defect is **defeated defence in depth**: the hub-side *pull* net was anchored on a field the failing controller keeps refreshing, so it could not compensate for a lost *push* (cf. F-HUB). Live on demo-hp: induced failure → `last_run` advanced 11:25:48Z, `last_success` **held** 11:24:20Z; demo-felhom healthy → anchor advanced. Legacy degrade logged once per customer, live | — |
|
||||
| **R-101** | **Tier-2 (cross-drive) `LastRun` is also written on failure** (`recordTier2Failure`), and three customer surfaces render it **without** a status: the `Tier2DestDisconnected` and `Tier2DestInactive` branches of `backups_apps.html`, and the restore-confirm dialog (`Legutóbbi másolat: {{.Tier2LastRun}}`) — which presents a possibly-failed run's timestamp at the moment the customer decides whether to restore | READY (S) | — | Found by R-100's P0.3/P0.4 sweep; **filed not fixed** (out of that task's path). No hub verdict reads it, so this is a UI-honesty issue, not an alarm one. The two degraded branches do carry a `tag-warn`; the restore-confirm is the sharpest instance. Fix: pair the timestamp with the status everywhere, as the main Tier-2 branch already does | CC |
|
||||
| **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — |
|
||||
| **F-LEAK** | ~~A failed restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); the 10-slot VMID band shrinks silently~~ | **SHIPPED + PROVEN-LIVE** (agent v0.110.0 + host-install v1.21.0, 2026-07-28) | — | **Three attempts, two refuted live.** (1) Pool adoption: `PUT /pools/{pool}` also needs `VM.Allocate` on the VM — membership cannot bootstrap its own authority. (2) Per-path `/vms/990000..990009` ACLs: work, but PVE's destroy calls `remove_vm_access` (`LXC.pm:906`) which deletes every ACL at `/vms/<vmid>` — **consumed by the op it authorises**, one use per slot. (3) SHIPPED: 4th root-fenced exception, band enforced in sudoers **literally** (`pct destroy 99000[0-9] --purge`) + in code + at the caller; API destroy still tried first. Live: band PERMITTED, `9201`/`9100`/`9999`/`990010`/`1` REFUSED, and `pct start 990000` REFUSED too | — |
|
||||
| **F-OBS** | ~~`deadapp-check` leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran"~~ | **SHIPPED + PROVEN-LIVE** (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. **Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0** — it logged only at startup and when it acted, so its health could be read only from absence | — |
|
||||
|
||||
Reference in New Issue
Block a user