diff --git a/REPORT.md b/REPORT.md index 6178767..5772ece 100644 --- a/REPORT.md +++ b/REPORT.md @@ -109,3 +109,75 @@ mock, because the failure class is "nothing arrives". two identically-exposed boxes, so demo-felhom was rolled to the same proven artifacts. - `StorageTarget.Role` carries a documented vocabulary including `vzdump-target` and is always `""`. Checked — it is a deliberate unpopulated field ("hub-owned, slice 10"), **not** a sixth dead seam. + +--- + +# E-2 increment 2 (same day): the wrapper, the installer, the offer and the banner + +**Live fleet:** hub **0.81.0**, agent **0.113.0**, controller **0.185.1**, host-install **1.22.0** +(script; no reinstall performed). Both demo boxes. `peti-felhom` untouched. + +## E-2a — the guarded wrapper + +`configs/felhom-backup-target-apply` behind a literal `FELHOM_BACKUPTARGET` sudoers alias. **The +agent's PVE role was not widened** — it still holds neither `Datastore.Allocate` at `/storage` nor +`Permissions.Modify`. + +All five laws proven live as root on demo-hp, with **0 stray storages created**: + +| Probe | Result | +|---|---| +| F-1 subdirectory (`/mnt/nvme-1tb/dump`) | REFUSED — "must be the drive's OWN mountpoint" | +| F-2 unmounted path | REFUSED — "would silently retarget onto the system drive" | +| root device (`/`) | REFUSED | +| idempotent re-apply on the real target | no-op, exit 0 | +| repointing an existing id | REFUSED | + +`POST /backup/target` drives it **create → grant → config**, and deliberately does **not** restart the +agent: it returns `restart_required` so the restart happens behind the caller's own immediate +in-flight check. That is E-1's spurious-tier-failure mistake encoded as a contract, and a test asserts +the handler never restarts. + +## Part 2 — the installer (v1.22.0) + +Case A creates the target through the same wrapper; Case B records DEGRADED in two warning lines and +**still succeeds** — a single-drive appliance is valid. Scenario F returns early with **0 wrapper +calls**; the red-proof (guard removed) shows **2**, i.e. it would have "corrected" a correct box. + +**Installer-logic-tested, NOT install-tested** — the harness runs the extracted functions with +stubbed `pvesm`/wrapper. No reinstall was performed. + +## Parts 3+4 — the offer and the banner + +`POST /api/storage/backup-target/assign` is the **only** writer of the role. `GET +/api/storage/backup-target` drives the banner. Hungarian copy is FACT → CONSEQUENCE → REMEDY. + +**Scenario E proven live on both boxes** — healthy renders nothing: + +``` +demo-hp {"degraded":false,"known":true,"label":"NVME 1TB","target":"felhom-backup"} +demo-felhom {"degraded":false,"known":true,"label":"USB HDD 1TB","target":"felhom-backup"} +``` + +No `message` key at all. Red-proofed: adding a reassurance to the healthy branch fails Scenario E. + +## Three defects I introduced and caught + +1. **The offer endpoints were unreachable.** Registered as `/api/backup-target` inside + `ServeStorageAPI`, which `main.go` mounts only at `/api/storage/`. Every unit test passed because + they called the handlers directly; the **first live call** returned `endpoint not found`. Fixed in + v0.185.1 with a test that asserts the dispatcher's own source contains both routes. +2. **A hollow test.** `TestUnknownStateRendersNothing` used `{Known:false}` with `Degraded` left + false, so it passed even with the `!Known` guard deleted. Its own red-proof exposed it; the + fixture is now `{Known:false, Degraded:true}`. +3. **A gofmt-realignment no-op** (again): a `sed`-style replace silently matched nothing because the + formatter had changed the spacing. Every mutation is now verified to have landed before its result + is trusted. + +## Still not live-proven + +- The **DEGRADED banner** and the **offer acceptance** — both boxes are healthy, so neither state + occurs naturally. Unit- and red-proofed only. +- **`backup_target_absent` end-to-end** — needs a live enrolled drive to go absent. +- **Case A/B on a real install** — installer-logic-tested only. +- **Full drive-loss recovery** — physical access. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index d55c3a1..926e98e 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -39,8 +39,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — | | **F-LEAK** | ~~A failed restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); the 10-slot VMID band shrinks silently~~ | **SHIPPED + PROVEN-LIVE** (agent v0.110.0 + host-install v1.21.0, 2026-07-28) | — | **Three attempts, two refuted live.** (1) Pool adoption: `PUT /pools/{pool}` also needs `VM.Allocate` on the VM — membership cannot bootstrap its own authority. (2) Per-path `/vms/990000..990009` ACLs: work, but PVE's destroy calls `remove_vm_access` (`LXC.pm:906`) which deletes every ACL at `/vms/` — **consumed by the op it authorises**, one use per slot. (3) SHIPPED: 4th root-fenced exception, band enforced in sudoers **literally** (`pct destroy 99000[0-9] --purge`) + in code + at the caller; API destroy still tried first. Live: band PERMITTED, `9201`/`9100`/`9999`/`990010`/`1` REFUSED, and `pct start 990000` REFUSED too | — | | **F-OBS** | ~~`deadapp-check` leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran"~~ | **SHIPPED + PROVEN-LIVE** (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. **Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0** — it logged only at startup and when it acted, so its health could be read only from absence | — | -| **E-2** | **IN PROGRESS 2026-07-29.** SHIPPED: hub v0.81.0 (`backup_target_absent`/`_restored` in allowedEventTypes + Hungarian customerMessages, 3 tests, red-proofed) and controller Part 1 (`StoragePath.BackupTarget` role + `SetBackupTarget`/`ClearBackupTarget`, 5 invariant tests, red-proof C). **REMAINING: Part 2 installer Case A/B, Part 3 the offer + the agent-side move, Part 4 the degraded banner, Part 5 the controller half of the signal, red-proofs E/F, live validation.** | **READY (M) — partially shipped** | — | **Phase 0 inverted the emphasis: the installer has NO drive-enrollment step (`/mnt/felhom-drives` is agent-installed at runtime), so at install time the common case is system-drive-only and Part 3 (drive added later) is the PRIMARY path, not Case A.** Scope + rationale: `runbooks/RUNBOOK-vzdump-target-move-2026-07-29.md` §7. Flips matrix row 4 | CC | -| **E-2a** | **The target move needs a root-fenced wrapper — the agent cannot do it.** Creating a PVE `dir` storage needs `Datastore.Allocate` at `/storage` and the ACL grant needs `Permissions.Modify`; the agent holds neither by design (blast-radius containment), and its sudoers is a literal-argument `Cmnd_Alias` model with **no `pvesm` and no `pveum`**. The established pattern is a guarded wrapper (`felhom-mkfs-guarded`, `felhom-pbs-apply` — the latter already has the exact `grant ` dual-grant verb E-2 needs) | READY (M) | blocks E-2 Part 3 | Add `felhom-backup-target-apply` (create-dir-storage with `is_mountpoint 1` + grant) + its literal sudoers rule + installer install step, then the agent endpoint. **Do NOT widen the agent's PVE role instead** | CC | +| **E-2** | ~~Drive-role machinery around the moved vzdump target~~ | **SHIPPED** (hub 0.81.0, agent 0.113.0, controller 0.185.1, host-install 1.22.0 — 2026-07-29) | — | Parts 1–5 complete. Role model + offer-only assignment + Hungarian degraded banner + absent-target signal + installer Case A/B. **NOT yet live-proven:** the DEGRADED banner and the offer acceptance (both demo boxes are healthy, so neither state occurs naturally) and `backup_target_absent` end-to-end. Installer is **installer-logic-tested, not install-tested** — no reinstall was performed | CC | +| **E-2a** | ~~The target move needs a root-fenced wrapper — the agent cannot do it~~ | **SHIPPED + PROVEN-LIVE** (agent v0.113.0 + host-install v1.22.0, 2026-07-29) | — | `felhom-backup-target-apply` behind a literal `FELHOM_BACKUPTARGET` sudoers alias; the agent's PVE role was NOT widened. Enforces F-1 (`mountpoint -q`) and F-2 (`is_mountpoint 1` hardcoded), refuses a root-device target, has NO storage-removal path (grep-assertable), is idempotent and refuses to repoint. All five laws proven live as root on demo-hp with 0 stray storages | — | | **E-2b** | ~~`NotifyStorageDisconnected`/`Reconnected` defined and called NOWHERE — a drive going absent emitted no event on any channel~~ | **SHIPPED + PROVEN-LIVE** (controller v0.184.1 + agent v0.112.0 + hub v0.81.0, 2026-07-29) | — | Seam wired in `ReconcileDriveGates`; a target drive raises the specific `backup_target_absent` instead. **A keying bug was caught before deploy:** `a.Path` is the registered GUEST path, not the agent's host `MountPath`, so the target branch was unreachable — every absent drive, target included, fell through to the generic event (v0.184.1). Tests observe the WIRE (httptest hub), not a mock | — | | **E-2c** | ~~E-1 put the whole-guest backups on a drive `POST /disks/eject` would eject~~ | **SHIPPED + PROVEN-LIVE** (agent v0.112.0, 2026-07-29) | — | Eject + decommission refuse 409 on the backup-target mount, naming the storage and the remedy. **Live on BOTH boxes:** demo-hp `/mnt/nvme-1tb` and demo-felhom `/mnt/hdd_1` both refused, drives unmoved. NOT a role reclassification — `RoleForStorage` untouched, because on both boxes that drive is ALSO the enrolled user-data drive; `TestEjectStillAllowedOnANonTargetDrive` pins the non-over-correction and `/var/lib/vz` is still refused by the PRE-EXISTING role gate, not this one | — | | **PETI** | **`peti-felhom` deliberately NOT migrated.** Its whole-guest backup still shares a device with its guest, so a drive failure there is **offsite-only recovery** | **ACCEPTED RISK — parked** | operator's next visit (tester reinstalling from scratch) | **Accepted until the reinstall; re-evaluate if that slips past ~2026-09-01.** Do not migrate, do not touch | operator |