# REPORT — Slice 6 Phase A: backup + the self-restore-test (v0.6.0-rc1) (2026-06-09)

> Overwrite-latest report (most recent significant work only). Cumulative history lives in [CHANGELOG.md](CHANGELOG.md).

## Outcome

**Slice 6 Phase A is implemented, live-validated on `demo-felhom`, and pushed as
`v0.6.0-rc1`** — the guest-level backup layer + the journaled **self-restore-test** that
closes "a backup you haven't restored isn't a backup" (doc 03 §8). Everything is **benign**
(backup, restore-to-NEW, scratch teardown): it reuses the slice-4 classifier/gate/journal —
no new destructive class, no new crypto. **Local target only**; PBS is Phase B. Restore is
to a **NEW** guest only (no overwrite). Backups are **crash-consistent** (marked so). This is
the hard checkpoint: awaiting validation before Phase B.

## What landed

- **proxmox**: `DestroyLXC` (the scratch-teardown primitive), `VzdumpOptions.Notes` →
  `notes-template`, `LatestBackupVolID` (resolve a produced archive — the task status carries
  no result volid).
- **The self-restore-test** (`reconcile.Engine.RunRestoreTest`): pick a free scratch VMID
  (configured band, excludes 9999; full band → skip) → **journal a `Scratch`-owned entry
  BEFORE any mutation** → restore-to-new → benign net **link-down** SetConfig (so the clone
  can't conflict with a running source's MAC/IP — test-safety, NOT slice-7 identity reset) →
  boot → verify **reaches `running`** → **defer teardown that ALWAYS runs** (benign
  `ClassGuestDestroy` + agent-tagged-scratch provenance, through the gate).
- **Crash-safe recovery** (`Recover`): a `Scratch` entry is resolved by **teardown**, not by
  re-checking the restore sub-task's UPID — special-cased before the generic path (else a
  leaked guest would be marked "succeeded"). Idempotent: already-gone → clean; list-unreadable
  → left in-flight.
- **`internal/backup`**: `BackupRunner` (vzdump + archive/size resolve + the **bulk-volume
  gap** — a mountpoint is uncovered unless it carries an explicit `backup=1`, so an unset
  `backup=` counts as uncovered, the safe DR direction), `PickRestoreCandidate`, an in-memory
  `Store` (latest backup-per-target + latest restore-test) feeding the hub reporter seams, and
  a cadence `Scheduler` (default 24h; the fourth daemon goroutine; disabled cleanly when
  off/misconfigured).
- **hub report**: `Backup` + `RestoreTest` filled (`PBSSnapshot` stays a Phase-B stub);
  collector seams; cross-repo golden byte-identical + bidirectional key-set tests in both
  repos; hub `handler.go` parses + persists them and **logs a FAILED restore-test
  prominently** (the loudest DR signal).
- **config** `BackupConfig` (band 990000–990009 default); **`--selftest=backup`** and
  **`--selftest=restore-test`** (Recover-then restore→boot→verify→teardown).

## Tests

`go test ./...` green; **`go test -race ./...` green on the build server** (the scheduler
goroutine). Notable unit tests: vzdump async UPID→WaitTask + bulk-gap (`backup!=1` incl.
unset); restore-test pass + **teardown-on-failed-verify** + restore-failure-still-tears-down +
full-band-skip; the DELETE wire contract (no body + force=1); and the headline **recover-
leaked-scratch** trio (destroyed / already-gone-clean / list-unreadable-left-in-flight).

## Live validation on demo-felhom

- **`--selftest=backup -vmid 9001`** → vzdump of the stopped spike-lxc to `local`: success,
  **crash-consistent**, archive `local:backup/vzdump-lxc-9001-…tar.zst` (~934 MiB) resolved,
  26 s. Bulk gap empty (9001 has no extra mountpoints).
- **`--selftest=restore-test`** → restored that backup into scratch **990000** on local-lvm,
  net link-down, booted, verified `running`, torn down (gate-benign) — **pass, 15 s, no leak**.
- **Crash-recovery proven for real**: the first teardown hit a `DestroyLXC` bug (DELETE with
  a form body → PVE HTTP 501; a booted guest also needs `force=1`), which **leaked scratch
  990000**. The fix moved the flags to the query string + `force=1`, **no body** (regression-
  tested). On the next run, `Recover` found the journaled in-flight `Scratch` entry, confirmed
  990000 existed, and **destroyed the leaked guest** (`ScratchDestroyed:1`) through the benign
  gate — then ran a clean test. `pct list` after: **no 99000x guests**. The accidental leak +
  reap is the best possible live proof of the slice's load-bearing crash-safety path.

## Notes / cleanup

- The demo config (`/root/.config/felhom-agent/agent.json`) gained a `backup` block with the
  cadence **disabled** (`restore_test_cadence_seconds: -1`); the validation used the selftest
  harness. The example config ships cadence default-ON (24h) with the 990000-band + local-lvm.
- One test backup of 9001 remains on `local` (a valid artifact; ~934 MiB of ~88 GiB free).
- The binary on felhom-pve is v0.6.0-rc1; token/TLS pin per memory `demo-felhom-live-agent`.

## Deferred → Phase B

PBS: the datastore on the USB, zero-knowledge key custody, restore-from-PBS, and the PBS
integrity-verify as the lighter frequent check. Then slice 7 (provisioning + identity-reset +
golden base, §9).