diff --git a/REPORT.md b/REPORT.md index e11bdac..17008ec 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,79 +1,32 @@ -# REPORT — slice 10: external user-data drive passthrough (P1 spike + P2) +# REPORT — felhom-agent v0.28.0 -**Agent v0.25.0** (+ controller v0.48.0, golden rebaked). P1 spike PASSED (gate); P2 BUILT + validated -live on guest 9201. P3 (self-heal) + P4 (dual-role) are the next phases. +Backup re-target → offsite PBS (real DR) + operator-signed decommission + restore-test bind-mount fix. +Implemented from the "whole-guest backup → felhom-pbs" CC SPEC (PHASE 0 gate → PHASE 1/3 + a validation-discovered fix). Live-validated on demo-felhom (PVE 9.2.2, guest 9201). Secrets (PBS token/fingerprint, encryption key) redacted / stored out-of-band. -## PHASE 1 — SPIKE (GATE PASSED), four proofs on 9201 -- **1A host→guest bind:** `pct set 9201 -mp0 /mnt/felhom-usb,mp=/mnt/felhom-usb` — **bind form** (host - path), never `storage:size`. Propagation `shared:49` host↔guest automatic. -- **1B write — chown, not idmap:** idmap not clean (mixed host ownership 1000+0, container-wide, - restart, subuid). Decision (refined with Viktor): chown only a fresh **`/felhom-data`** - namespace to `100000:100000`; the customer's existing data is never touched. Guest-root r+w confirmed. -- **1C guest→controller-container:** `-v /mnt:/mnt:rslave` + `/mnt` rshared in the guest → newly-mounted - drives propagate into the running container (proven). The de-priv `/mnt` gap, reopened scoped. -- **1D app-container:** busybox bind read+write; bytes land on `/dev/sdb1`. +## PHASE 0 — GATE (proven live before any re-target code) +- **Backup → PBS, snapshot mode** fires the `create storage snapshot 'vzdump'` marker → 8B.2 early-resume/quiesce survives a PBS target (marker is mode-driven, not target-driven). +- **Restore-test enumerates PBS** through the SAME generic `StorageContent` (`/nodes//storage/felhom-pbs/content` returns `content:"backup"` + ctime/vmid/volid) — `PickRestoreCandidate`/`latestArchive` need NO PBS-client change. +- **pct restore from a PBS volid** round-trips (storage.cfg encryption key applied transparently). PBS gotchas (`ignore-verified`, node-from-UPID, privsep) touch only the verify-API path. -## PHASE 2 — passthrough (Model A: the felhom-data namespace is the in-guest mount) -- **P2A agent — `POST /disks/guest-attach`** (`internal/localapi`, `GuestBinder`): self-scoped; creates - `/felhom-data`, chowns it to the guest base (not -R), and `pct set -mpN - /felhom-data,mp=/mnt/` (RW bind). Idempotent; lowest free `mpN`; `where` validated. Only - Felhom's namespace crosses into the guest — the customer's other on-drive data never does. Tests: - `TestGuestAttach_*` (slot select, idempotency, bad-path, not-configured). -- **P2B golden — `configs/build-golden.sh`:** controller `docker run` gains `-v /mnt:/mnt:rslave`; the - bootstrap makes `/mnt` rshared first. Scoped to `/mnt` (only felhom-data-namespace mounts). -- **P2C controller (v0.48.0):** `agentapi.GuestAttach`; `runStorageInit`/`runStorageAttach`/ - `handleStorageRegister` call `attachIntoGuest` after register (best-effort; P3 heals a miss). +## PHASE 1 — backup re-target → felhom-pbs +- `config.BackupConfig.BackupTarget()` returns `backup.local_backup_target` or, when empty, the new default `felhom-pbs` (separate hardware = real DR). Configurable; no call site hardcodes it. All `NewBackupRunner` sites + restore-test `SourceTier` route through it. +- Live: demo `agent.json` `local_backup_target` "local" → **felhom-pbs**; `--selftest=backup -vmid 9201` → `target=felhom-pbs`, snapshot, crash-consistent ✓. Daemon `/backup/status` now reports `target_id=felhom-pbs`. +- **PBS prune default** set on the storage: `prune-backups keep-daily=7,keep-weekly=4,keep-monthly=3` (was keep-all at 73% full). -## Live validation (9201) -After `guest-attach` + a guest restart to activate mp0: -- mp0 = `/mnt/felhom-usb SOURCE /dev/sdb1[/felhom-data]` (Model A; the `[/felhom-data]` suffix the - controller's mount strip already handles). -- Controller container mountinfo has `/felhom-data /mnt/felhom-usb … /dev/sdb1`. -- An app (busybox bind) writes `proof.txt` → present on the **host** `/mnt/felhom-usb/felhom-data/...`, - `df` device `/dev/sdb1` (NOT the rootfs). -- Banner cleared: **`[PASS] Storage paths: 1 connected, 0 disconnected`**. -- `go test ./...` green (both repos). +## Restore-test bind-mount fix (validation-discovered; scope-approved) +- A slice-10 enrolled guest's data drive is a host **bind-mount** `mp0`; vzrestore refuses it under the privsep token ("restoring 'mpN' to bind mount is only possible for root") — so the restore-test failed for EVERY enrolled guest, independent of backup tier. +- Fix: the restore-test reads the SOURCE guest config (vmid parsed from the archive volid) and passes `RestoreLXCOptions.MountOverrides` converting each bind-mount `mpN` → a throwaway 1G volume on the restore storage, plus a `rootfs` override sized from the source (PVE requires rootfs when mp params are present). Boot-verify doesn't need the data. Storage-backed mounts restore normally; best-effort if the source config is unreadable. +- Live: daemon **scheduled restore-test from felhom-pbs → `pass:true, verified:"boot+running", source_tier:"pbs"`** (44s, scratch torn down clean). -## LIVE-ACTIVATION — investigated, mechanism chosen (evidence-based) -A drive enrolled into a **running** unprivileged guest cannot be activated live, and the host-side -bind-inject is **blocked** (proven on 9201): `pct set` doesn't hot-apply a mountpoint to a running -guest; `/proc//root/...` bind → `mount: bad superblock` (unprivileged mount-locking); `nsenter -m` -into the guest ns loses the host source path. The bind activates at the **next guest boot** (validated: -reboot → mp0 active `/dev/sdb1[/felhom-data]`, controller sees it, banner clears). Fresh guests from the -rebaked golden are unaffected (mp activates at first boot, before the controller starts). +## PHASE 3 — operator-signed decommission (reachable now) +- The previously-unreachable `IntentDecommissioned` is now reached ONLY via a gate-VERIFIED operator signature (classified destructive). New `signedjobs.DecommissionExecutor` (op `decommission`) → `IntentStore.SetDecommissioned`, keyed by the drive's **storage** durable-id (the watchdog's key, e.g. `uuid:` — NOT the device-level `byid:/byuuid:` of `storage_wipe`), so the intent actually gates remounts. New `ExecutorChain` serves both wipe + decommission; runner wiring moved below the intent-store open. `felhom-opsign` builds decommission params from `-durable-id`. No customer/controller UI — operator path is hub jobs-queue → signed-jobs runner. Distinct from a customer-confirmable safe eject. -**Decision (Viktor):** enroll persists via `pct set` with **NO forced reboot**; the UI shows a -"pending activation" state + a user-triggered **"Újraindítás most (~30s)"** button that batches all -pending drives; P3 self-heal flags "restart to reconnect" for recovered drives. The staging-mp -live-propagation alternative is deferred to its own spike. (Build remaining — see below.) +## Tests +- `internal/signedjobs`: decommission happy-path / foreign-op / unbound-refusal / no-store; ExecutorChain dispatch. +- `internal/reconcile`: `archiveVMID` (PBS ct/vm + vzdump forms), `bindMountOverrides`, `rootfsSizeGB`/`sizeToGB`. +- Full `go build ./...` + package tests green. -## Activation-UX — DONE (agent v0.26.0 + controller v0.49.0) -- **Agent `POST /guest/reboot`** (self-scoped, detached `pct reboot`, 202). Controller - `pendingActivationDrives()` flags registered drives the agent shows attached but not live-mounted in - the container; settings banner + batched **"Újraindítás most (~30 mp)"** button → `/api/storage/ - activate` → `agentapi.GuestReboot`. Live-validated on 9201: activate → guest reboots → drive active, - app writes to `/dev/sdb1`, banner absent. (Pending-banner display couldn't be forced live — LXC mp - mounts resist manual unmount + no 2nd drive — but the logic is unit-confident.) -- **Minor known artifact:** the bootstrap's `mount --rbind /mnt /mnt` duplicates the mp submount, so - `findmnt /mnt/` shows two (identical) rows — functional (IsMountPoint true, apps write fine); - inherent to rbind (a non-recursive bind would hide the submounts from the container). Tidy later. - -## P3 self-heal reconcile — DONE (agent v0.27.0) -- **`IntentStore`** (`internal/storage/intent.go`): durable, durable-id-keyed 4-state model - (new/enrolled/ejected/decommissioned), atomic writes, `OnAbsent` replug rule. Intent recorded ONLY - via the official enroll (`/disks/guest-attach`, incl. the idempotent path) + eject (`/disks/eject`, - before unmount) paths. -- **Watchdog**: re-mount candidate now intent-gated (only `enrolled` self-heals) + a present→absent - hook clears `ejected`; exponential-backoff flapping guard (alert@4, hard stop@8, failure = "still - not present a full backoff window after dispatch" so a slow async re-mount isn't miscounted). -- **Wired in main.go**: one `IntentStore` at `/drive-intents.json`, shared by the watchdog - + the local API; open failure degrades to ungated legacy remount. -- **Tests**: intent states/persistence/replug-rule/gate + the watchdog scenarios (colleague-unmount→ - reconciled, ejected/new/decommissioned→left alone, ejected→absent→replug→auto-mount, flapping→caps). -- **Live-validated on 9201**: out-of-band `umount` → watchdog "self-heal — re-mounting enrolled drive - (no eject intent)" → host **HEALED** within a tick; official eject → intent `ejected` → **NOT - reconciled** (stays unmounted). (No golden change — P3 is host-side.) - -## Not done -- **P4** dual-role eligibility + backup-aware wipe warning (controller). Cross-drive backup ENGINE stays - out of scope (needs a 2nd physical drive). +## Live deploy +- `felhom-agent 0.28.0` active on demo-felhom (prior binary kept as `felhom-agent.bak-0.27.0`); config `local_backup_target=felhom-pbs`, restore-test cadence restored to 86400s. +- **Golden prune** (housekeeping): kept the newest 2 golden (VMID 9100) archives on `local`, freed the older 11. +- **Note (not actioned):** the drive's legacy bare-metal `felhom_data` (underscore) dir is orphaned under the new `felhom-data` (hyphen) namespace on `/mnt/felhom-usb`; flagged for a real-customer migration path (out of scope here).