docs(REPORT): slice 6 Phase A — backup + self-restore-test, live-validated on demo-felhom

Live: backup of 9001 (crash-consistent, 934MiB), restore-test (restore→boot→verify→
teardown, gate-benign, no leak), AND a real crash-recovery proof — a DestroyLXC DELETE
bug leaked scratch 990000, the fix landed, and Recover reaped the leaked guest on the
next run before a clean test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

REPORT.md

@@ -1,106 +1,78 @@
-# REPORT — Slice 5 live storage validation on `demo-felhom` (felhom-agent v0.5.1) (2026-06-09)
+# REPORT — Slice 6 Phase A: backup + the self-restore-test (v0.6.0-rc1) (2026-06-09)
 
 > Overwrite-latest report (most recent significant work only). Cumulative history lives in [CHANGELOG.md](CHANGELOG.md).
 
 ## Outcome
 
-**Slice 5 (storage observe/report + watchdog + the host-root surface) is validated live** on the
-demo Proxmox host `demo-felhom` against a real **1 TB USB HDD** (`sdb1`, ext4, label `hdd_1`,
-fs-UUID `277a2179-…`, rotational). Agent **v0.5.1** (the validation-prep fixes — unmounted-dir
-durable_id mis-id + watchdog UUID memory). **No live destructive op** — the destructive
-detach/wipe gate refusal is proven in the unit suite, never exercised on the real drive.
+**Slice 6 Phase A is implemented, live-validated on `demo-felhom`, and pushed as
+`v0.6.0-rc1`** — the guest-level backup layer + the journaled **self-restore-test** that
+closes "a backup you haven't restored isn't a backup" (doc 03 §8). Everything is **benign**
+(backup, restore-to-NEW, scratch teardown): it reuses the slice-4 classifier/gate/journal —
+no new destructive class, no new crypto. **Local target only**; PBS is Phase B. Restore is
+to a **NEW** guest only (no overwrite). Backups are **crash-consistent** (marked so). This is
+the hard checkpoint: awaiting validation before Phase B.
 
-Two HUMAN steps (physical unplug/replug, H1/H2) were **software-simulated** by CC via a
-reversible USB `authorized` toggle (W3) — same agent code paths; see "Not done" for the real
-physical confirmation, which is Viktor's to run.
+## What landed
 
-## Pre-flight (P1–P3)
+- **proxmox**: `DestroyLXC` (the scratch-teardown primitive), `VzdumpOptions.Notes` →
+  `notes-template`, `LatestBackupVolID` (resolve a produced archive — the task status carries
+  no result volid).
+- **The self-restore-test** (`reconcile.Engine.RunRestoreTest`): pick a free scratch VMID
+  (configured band, excludes 9999; full band → skip) → **journal a `Scratch`-owned entry
+  BEFORE any mutation** → restore-to-new → benign net **link-down** SetConfig (so the clone
+  can't conflict with a running source's MAC/IP — test-safety, NOT slice-7 identity reset) →
+  boot → verify **reaches `running`** → **defer teardown that ALWAYS runs** (benign
+  `ClassGuestDestroy` + agent-tagged-scratch provenance, through the gate).
+- **Crash-safe recovery** (`Recover`): a `Scratch` entry is resolved by **teardown**, not by
+  re-checking the restore sub-task's UPID — special-cased before the generic path (else a
+  leaked guest would be marked "succeeded"). Idempotent: already-gone → clean; list-unreadable
+  → left in-flight.
+- **`internal/backup`**: `BackupRunner` (vzdump + archive/size resolve + the **bulk-volume
+  gap** — a mountpoint is uncovered unless it carries an explicit `backup=1`, so an unset
+  `backup=` counts as uncovered, the safe DR direction), `PickRestoreCandidate`, an in-memory
+  `Store` (latest backup-per-target + latest restore-test) feeding the hub reporter seams, and
+  a cadence `Scheduler` (default 24h; the fourth daemon goroutine; disabled cleanly when
+  off/misconfigured).
+- **hub report**: `Backup` + `RestoreTest` filled (`PBSSnapshot` stays a Phase-B stub);
+  collector seams; cross-repo golden byte-identical + bidirectional key-set tests in both
+  repos; hub `handler.go` parses + persists them and **logs a FAILED restore-test
+  prominently** (the loudest DR signal).
+- **config** `BackupConfig` (band 990000–990009 default); **`--selftest=backup`** and
+  **`--selftest=restore-test`** (Recover-then restore→boot→verify→teardown).
 
-- **P1** — staged the v0.5.1 binary at `/usr/local/bin/felhom-agent` on `felhom-pve`; wrote
-  `/root/.config/felhom-agent/agent.json` (mode 600). The earlier-slice token secret was
-  **stale (HTTP 401 — host rebuilt 2026-06-07)**, so CC **regenerated** the `felhom-agent@pve!agent`
-  token (privsep) and re-granted the token ACL (a `pveum user token remove` purges the token's
-  ACL — re-added `FelhomAgent` on the token). The `FelhomAgent` role (16 privs) + the user ACL
-  were already correct. Token stored out-of-band (config, 0600) — never committed. The TLS pin
-  is the **served leaf cert** `BA:7C:99:7D:45:D0:…` (note: `pvenode cert info` reports a
-  *different* cert — `B2:02:2E…` — so always pin the value from `openssl s_client … :8006`).
-  Runs as **root@pam** here, so `privileged.mode = "direct"` (no sudo prefix; identical HostOps
-  code + arg-validation). `smartctl`/`lvs`/`systemctl`/`install` all present.
-  **`--selftest=read` → fully green** (version, nodes, node-status, 2 guests, 2 stores).
-- **P2** — the 1 TB USB HDD is present as `sdb`/`sdb1` (ext4, UUID `277a2179-…`, ROTA=1,
-  TRAN=usb, removable=0). It was **not** previously a Proxmox storage, so CC set it up
-  (non-destructive: mounted the already-formatted partition by-UUID at `/mnt/felhom-usb`).
-- **P3 — `is_mountpoint 1`**: defined `pvesm add dir felhom-usb --path /mnt/felhom-usb
-  --content backup --is_mountpoint 1`. Before: no USB storage existed; after: defined **with**
-  `is_mountpoint 1`, so when the drive drops PVE marks the storage inactive instead of writing
-  to the bare mountpoint dir on root.
+## Tests
 
-## Static observe (S1–S2)
+`go test ./...` green; **`go test -race ./...` green on the build server** (the scheduler
+goroutine). Notable unit tests: vzdump async UPID→WaitTask + bulk-gap (`backup!=1` incl.
+unset); restore-test pass + **teardown-on-failed-verify** + restore-failure-still-tears-down +
+full-band-skip; the DELETE wire contract (no body + force=1); and the headline **recover-
+leaked-scratch** trio (destroyed / already-gone-clean / list-unreadable-left-in-flight).
 
-**S1 — `--selftest=storage`** reported `felhom-usb`:
-- `type=local-dir` (removable flag is 0 on this USB HDD, so it classifies as local-dir, not
-  usb — both are mount-backed and treated identically; noted), `state=attached`, `reachable=true`;
-- **`durable_id = uuid:277a2179-a764-4758-b840-9ea741517914`** — matches `blkid` exactly (the
-  DR re-attach key is correct);
-- `class_hint = slow` (rotational HDD ✓);
-- usage `8.0/915.8 GiB`, `mount=/mnt/felhom-usb`, `dev=/dev/sdb1`;
-- **`smart: health=PASSED temp=30 poh=2419 realloc=0 pending=0 offline_unc=0`** — SMART works
-  through the USB-SATA bridge (SATA attribute set parsed; NOT degraded to UNKNOWN);
-- bonus: `local-lvm` **thin-pool data=1.9% metadata=0.5%** — the Phase-B `lvs` metadata fill
-  is populated live; `local` → `durable=path:/var/lib/vz` (correctly not a fs-UUID).
+## Live validation on demo-felhom
 
-**S2 — durable_id stability**: re-ran twice; `durable_id` identical both times.
+- **`--selftest=backup -vmid 9001`** → vzdump of the stopped spike-lxc to `local`: success,
+  **crash-consistent**, archive `local:backup/vzdump-lxc-9001-…tar.zst` (~934 MiB) resolved,
+  26 s. Bulk gap empty (9001 has no extra mountpoints).
+- **`--selftest=restore-test`** → restored that backup into scratch **990000** on local-lvm,
+  net link-down, booted, verified `running`, torn down (gate-benign) — **pass, 15 s, no leak**.
+- **Crash-recovery proven for real**: the first teardown hit a `DestroyLXC` bug (DELETE with
+  a form body → PVE HTTP 501; a booted guest also needs `force=1`), which **leaked scratch
+  990000**. The fix moved the flags to the query string + `force=1`, **no body** (regression-
+  tested). On the next run, `Recover` found the journaled in-flight `Scratch` entry, confirmed
+  990000 existed, and **destroyed the leaked guest** (`ScratchDestroyed:1`) through the benign
+  gate — then ran a clean test. `pct list` after: **no 99000x guests**. The accidental leak +
+  reap is the best possible live proof of the slice's load-bearing crash-safety path.
 
-## Watchdog cycle — software-simulated (W1–W3)
+## Notes / cleanup
 
-Watchdog config for the test: 5 s poll, 15 s debounce, 20 s known-refresh.
+- The demo config (`/root/.config/felhom-agent/agent.json`) gained a `backup` block with the
+  cadence **disabled** (`restore_test_cadence_seconds: -1`); the validation used the selftest
+  harness. The example config ships cadence default-ON (24h) with the 990000-band + local-lvm.
+- One test backup of 9001 remains on `local` (a valid artifact; ~934 MiB of ~88 GiB free).
+- The binary on felhom-pve is v0.6.0-rc1; token/TLS pin per memory `demo-felhom-live-agent`.
 
-- **W2 — mount-drop** (`umount` at T=0):
-  - **+2.8 s** → `attached → disconnected` detected + out-of-band report fired + **re-mount
-    dispatched** (device still present, just unmounted);
-  - **+3.7 s** → **re-mounted by-UUID** via a freshly-written systemd unit
-    `mnt-felhom\x2dusb.mount` (`active`); `disconnected → attached` detected at +8 s; debounced
-    trailing report at +17 s. Detect latency **seconds, not the ~15-min cycle**.
-- **W3 — device soft-remove** (reversible USB `authorized` 0→1, simulating unplug/replug):
-  - deauthorize → block device + `/dev/disk/by-uuid/277a…` **fully gone**;
-  - **+2.8 s** → `attached → disconnected` detected + report — **and crucially NO re-mount**
-    (device absent → nothing to mount, correct); only on **reauthorize** (+6 s) did the
-    watchdog dispatch the re-mount and **mount by the stable by-UUID unit**, so a different
-    `/dev/sdX` enumeration would still mount the right filesystem (the H2 goal, simulated).
+## Deferred → Phase B
 
-This exercised both the v0.5.1 fixes live: an unmounted target never reported a root-fs UUID,
-and the re-mount used the remembered UUID across the known-set cache refresh.
-
-## Captured timings
-
-| event | latency |
-|---|---|
-| umount → `disconnected` detected (W2) | ~2.8 s |
-| umount → re-mounted by-UUID (W2) | ~3.7 s |
-| deauthorize → `disconnected` detected, **no re-mount** (W3) | ~2.8 s |
-| reauthorize → re-mounted by-UUID (W3) | ~6 s |
-
-## Not done (flagged)
-
-- **H1/H2 (physical unplug/replug) — HUMAN (Viktor).** Software-simulated by W3 (reversible
-  `authorized` toggle), which drives the identical agent code (device-gone detect, no-re-mount-
-  while-gone, re-mount-by-UUID on return). A real cable pull is the final confirmation; the
-  agent logic is already exercised.
-- **End-to-end hub reflection** (the out-of-band report landing on `hub.felhom.eu`): the
-  `--selftest=storage` harness is standalone (no hub), so the trigger **logged** “would send
-  out-of-band host-report now” at the right moments rather than POSTing. Verifying the hub
-  dashboard shows disconnected→attached needs the **full daemon with a minted per-host key**
-  (enrolling `demo-felhom` on the live operator hub — additive but outward-facing state). The
-  watchdog→report-trigger path is proven here; the hub-side ingest of `storage_targets` was
-  proven during the hub v0.7.2 deploy. **Offered, not done unprompted.**
-
-## Cleanup / end-state
-
-- Drive **mounted/attached**, PVE storage `felhom-usb` **active**, systemd unit
-  `mnt-felhom\x2dusb.mount` **active + enabled** (survives reboot), `is_mountpoint 1` **kept** —
-  the intended managed end-state.
-- No `--watch`/daemon process left running; `/tmp/watch*.log` removed. The sudoers allowlist is
-  installed at `/etc/sudoers.d/felhom-agent` (inert here — `sudo` isn't installed on this
-  minimal PVE, and the agent runs as root; it documents the eventual non-root deployment).
-- Nothing was wiped/detached; the regenerated Proxmox token lives only in the agent config
-  (0600) on `felhom-pve`, never committed.
+PBS: the datastore on the USB, zero-knowledge key custody, restore-from-PBS, and the PBS
+integrity-verify as the lighter frequent check. Then slice 7 (provisioning + identity-reset +
+golden base, §9).