docs: REPORT.md for v0.28.0 (PBS re-target + decommission + restore-test fix)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

REPORT.md

@@ -1,79 +1,32 @@
-# REPORT — slice 10: external user-data drive passthrough (P1 spike + P2)
+# REPORT — felhom-agent v0.28.0
 
-**Agent v0.25.0** (+ controller v0.48.0, golden rebaked). P1 spike PASSED (gate); P2 BUILT + validated
-live on guest 9201. P3 (self-heal) + P4 (dual-role) are the next phases.
+Backup re-target → offsite PBS (real DR) + operator-signed decommission + restore-test bind-mount fix.
+Implemented from the "whole-guest backup → felhom-pbs" CC SPEC (PHASE 0 gate → PHASE 1/3 + a validation-discovered fix). Live-validated on demo-felhom (PVE 9.2.2, guest 9201). Secrets (PBS token/fingerprint, encryption key) redacted / stored out-of-band.
 
-## PHASE 1 — SPIKE (GATE PASSED), four proofs on 9201
-- **1A host→guest bind:** `pct set 9201 -mp0 /mnt/felhom-usb,mp=/mnt/felhom-usb` — **bind form** (host
-  path), never `storage:size`. Propagation `shared:49` host↔guest automatic.
-- **1B write — chown, not idmap:** idmap not clean (mixed host ownership 1000+0, container-wide,
-  restart, subuid). Decision (refined with Viktor): chown only a fresh **`<drive>/felhom-data`**
-  namespace to `100000:100000`; the customer's existing data is never touched. Guest-root r+w confirmed.
-- **1C guest→controller-container:** `-v /mnt:/mnt:rslave` + `/mnt` rshared in the guest → newly-mounted
-  drives propagate into the running container (proven). The de-priv `/mnt` gap, reopened scoped.
-- **1D app-container:** busybox bind read+write; bytes land on `/dev/sdb1`.
+## PHASE 0 — GATE (proven live before any re-target code)
+- **Backup → PBS, snapshot mode** fires the `create storage snapshot 'vzdump'` marker → 8B.2 early-resume/quiesce survives a PBS target (marker is mode-driven, not target-driven).
+- **Restore-test enumerates PBS** through the SAME generic `StorageContent` (`/nodes/<node>/storage/felhom-pbs/content` returns `content:"backup"` + ctime/vmid/volid) — `PickRestoreCandidate`/`latestArchive` need NO PBS-client change.
+- **pct restore from a PBS volid** round-trips (storage.cfg encryption key applied transparently). PBS gotchas (`ignore-verified`, node-from-UPID, privsep) touch only the verify-API path.
 
-## PHASE 2 — passthrough (Model A: the felhom-data namespace is the in-guest mount)
-- **P2A agent — `POST /disks/guest-attach`** (`internal/localapi`, `GuestBinder`): self-scoped; creates
-  `<drive>/felhom-data`, chowns it to the guest base (not -R), and `pct set <vmid> -mpN
-  <drive>/felhom-data,mp=/mnt/<name>` (RW bind). Idempotent; lowest free `mpN`; `where` validated. Only
-  Felhom's namespace crosses into the guest — the customer's other on-drive data never does. Tests:
-  `TestGuestAttach_*` (slot select, idempotency, bad-path, not-configured).
-- **P2B golden — `configs/build-golden.sh`:** controller `docker run` gains `-v /mnt:/mnt:rslave`; the
-  bootstrap makes `/mnt` rshared first. Scoped to `/mnt` (only felhom-data-namespace mounts).
-- **P2C controller (v0.48.0):** `agentapi.GuestAttach`; `runStorageInit`/`runStorageAttach`/
-  `handleStorageRegister` call `attachIntoGuest` after register (best-effort; P3 heals a miss).
+## PHASE 1 — backup re-target → felhom-pbs
+- `config.BackupConfig.BackupTarget()` returns `backup.local_backup_target` or, when empty, the new default `felhom-pbs` (separate hardware = real DR). Configurable; no call site hardcodes it. All `NewBackupRunner` sites + restore-test `SourceTier` route through it.
+- Live: demo `agent.json` `local_backup_target` "local" → **felhom-pbs**; `--selftest=backup -vmid 9201` → `target=felhom-pbs`, snapshot, crash-consistent ✓. Daemon `/backup/status` now reports `target_id=felhom-pbs`.
+- **PBS prune default** set on the storage: `prune-backups keep-daily=7,keep-weekly=4,keep-monthly=3` (was keep-all at 73% full).
 
-## Live validation (9201)
-After `guest-attach` + a guest restart to activate mp0:
-- mp0 = `/mnt/felhom-usb SOURCE /dev/sdb1[/felhom-data]` (Model A; the `[/felhom-data]` suffix the
-  controller's mount strip already handles).
-- Controller container mountinfo has `/felhom-data /mnt/felhom-usb … /dev/sdb1`.
-- An app (busybox bind) writes `proof.txt` → present on the **host** `/mnt/felhom-usb/felhom-data/...`,
-  `df` device `/dev/sdb1` (NOT the rootfs).
-- Banner cleared: **`[PASS] Storage paths: 1 connected, 0 disconnected`**.
-- `go test ./...` green (both repos).
+## Restore-test bind-mount fix (validation-discovered; scope-approved)
+- A slice-10 enrolled guest's data drive is a host **bind-mount** `mp0`; vzrestore refuses it under the privsep token ("restoring 'mpN' to bind mount is only possible for root") — so the restore-test failed for EVERY enrolled guest, independent of backup tier.
+- Fix: the restore-test reads the SOURCE guest config (vmid parsed from the archive volid) and passes `RestoreLXCOptions.MountOverrides` converting each bind-mount `mpN` → a throwaway 1G volume on the restore storage, plus a `rootfs` override sized from the source (PVE requires rootfs when mp params are present). Boot-verify doesn't need the data. Storage-backed mounts restore normally; best-effort if the source config is unreadable.
+- Live: daemon **scheduled restore-test from felhom-pbs → `pass:true, verified:"boot+running", source_tier:"pbs"`** (44s, scratch torn down clean).
 
-## LIVE-ACTIVATION — investigated, mechanism chosen (evidence-based)
-A drive enrolled into a **running** unprivileged guest cannot be activated live, and the host-side
-bind-inject is **blocked** (proven on 9201): `pct set` doesn't hot-apply a mountpoint to a running
-guest; `/proc/<pid>/root/...` bind → `mount: bad superblock` (unprivileged mount-locking); `nsenter -m`
-into the guest ns loses the host source path. The bind activates at the **next guest boot** (validated:
-reboot → mp0 active `/dev/sdb1[/felhom-data]`, controller sees it, banner clears). Fresh guests from the
-rebaked golden are unaffected (mp activates at first boot, before the controller starts).
+## PHASE 3 — operator-signed decommission (reachable now)
+- The previously-unreachable `IntentDecommissioned` is now reached ONLY via a gate-VERIFIED operator signature (classified destructive). New `signedjobs.DecommissionExecutor` (op `decommission`) → `IntentStore.SetDecommissioned`, keyed by the drive's **storage** durable-id (the watchdog's key, e.g. `uuid:<fs-uuid>` — NOT the device-level `byid:/byuuid:` of `storage_wipe`), so the intent actually gates remounts. New `ExecutorChain` serves both wipe + decommission; runner wiring moved below the intent-store open. `felhom-opsign` builds decommission params from `-durable-id`. No customer/controller UI — operator path is hub jobs-queue → signed-jobs runner. Distinct from a customer-confirmable safe eject.
 
-**Decision (Viktor):** enroll persists via `pct set` with **NO forced reboot**; the UI shows a
-"pending activation" state + a user-triggered **"Újraindítás most (~30s)"** button that batches all
-pending drives; P3 self-heal flags "restart to reconnect" for recovered drives. The staging-mp
-live-propagation alternative is deferred to its own spike. (Build remaining — see below.)
+## Tests
+- `internal/signedjobs`: decommission happy-path / foreign-op / unbound-refusal / no-store; ExecutorChain dispatch.
+- `internal/reconcile`: `archiveVMID` (PBS ct/vm + vzdump forms), `bindMountOverrides`, `rootfsSizeGB`/`sizeToGB`.
+- Full `go build ./...` + package tests green.
 
-## Activation-UX — DONE (agent v0.26.0 + controller v0.49.0)
-- **Agent `POST /guest/reboot`** (self-scoped, detached `pct reboot`, 202). Controller
-  `pendingActivationDrives()` flags registered drives the agent shows attached but not live-mounted in
-  the container; settings banner + batched **"Újraindítás most (~30 mp)"** button → `/api/storage/
-  activate` → `agentapi.GuestReboot`. Live-validated on 9201: activate → guest reboots → drive active,
-  app writes to `/dev/sdb1`, banner absent. (Pending-banner display couldn't be forced live — LXC mp
-  mounts resist manual unmount + no 2nd drive — but the logic is unit-confident.)
-- **Minor known artifact:** the bootstrap's `mount --rbind /mnt /mnt` duplicates the mp submount, so
-  `findmnt /mnt/<name>` shows two (identical) rows — functional (IsMountPoint true, apps write fine);
-  inherent to rbind (a non-recursive bind would hide the submounts from the container). Tidy later.
-
-## P3 self-heal reconcile — DONE (agent v0.27.0)
-- **`IntentStore`** (`internal/storage/intent.go`): durable, durable-id-keyed 4-state model
-  (new/enrolled/ejected/decommissioned), atomic writes, `OnAbsent` replug rule. Intent recorded ONLY
-  via the official enroll (`/disks/guest-attach`, incl. the idempotent path) + eject (`/disks/eject`,
-  before unmount) paths.
-- **Watchdog**: re-mount candidate now intent-gated (only `enrolled` self-heals) + a present→absent
-  hook clears `ejected`; exponential-backoff flapping guard (alert@4, hard stop@8, failure = "still
-  not present a full backoff window after dispatch" so a slow async re-mount isn't miscounted).
-- **Wired in main.go**: one `IntentStore` at `<state-dir>/drive-intents.json`, shared by the watchdog
-  + the local API; open failure degrades to ungated legacy remount.
-- **Tests**: intent states/persistence/replug-rule/gate + the watchdog scenarios (colleague-unmount→
-  reconciled, ejected/new/decommissioned→left alone, ejected→absent→replug→auto-mount, flapping→caps).
-- **Live-validated on 9201**: out-of-band `umount` → watchdog "self-heal — re-mounting enrolled drive
-  (no eject intent)" → host **HEALED** within a tick; official eject → intent `ejected` → **NOT
-  reconciled** (stays unmounted). (No golden change — P3 is host-side.)
-
-## Not done
-- **P4** dual-role eligibility + backup-aware wipe warning (controller). Cross-drive backup ENGINE stays
-  out of scope (needs a 2nd physical drive).
+## Live deploy
+- `felhom-agent 0.28.0` active on demo-felhom (prior binary kept as `felhom-agent.bak-0.27.0`); config `local_backup_target=felhom-pbs`, restore-test cadence restored to 86400s.
+- **Golden prune** (housekeeping): kept the newest 2 golden (VMID 9100) archives on `local`, freed the older 11.
+- **Note (not actioned):** the drive's legacy bare-metal `felhom_data` (underscore) dir is orphaned under the new `felhom-data` (hyphen) namespace on `/mnt/felhom-usb`; flagged for a real-customer migration path (out of scope here).