diff --git a/REPORT.md b/REPORT.md index 17008ec..84aadd2 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,32 +1,58 @@ -# REPORT — felhom-agent v0.28.0 +# REPORT — felhom-agent v0.29.0 (OS / Docker-data storage split: golden + provision) -Backup re-target → offsite PBS (real DR) + operator-signed decommission + restore-test bind-mount fix. -Implemented from the "whole-guest backup → felhom-pbs" CC SPEC (PHASE 0 gate → PHASE 1/3 + a validation-discovered fix). Live-validated on demo-felhom (PVE 9.2.2, guest 9201). Secrets (PBS token/fingerprint, encryption key) redacted / stored out-of-band. +Phase 1 of the storage-split slice (Phase 2 = felhom-controller v0.58.0 prevention layer). The +controller guest's OS rootfs and Docker data are carved onto separate `local-lvm` volumes for +**resilience** — an isolated OS rootfs stays bootable + agent-recoverable if the Docker volume fills. +Built, the golden re-baked, and **live-validated by destroying + re-provisioning guest 9201**. -## PHASE 0 — GATE (proven live before any re-target code) -- **Backup → PBS, snapshot mode** fires the `create storage snapshot 'vzdump'` marker → 8B.2 early-resume/quiesce survives a PBS target (marker is mode-driven, not target-driven). -- **Restore-test enumerates PBS** through the SAME generic `StorageContent` (`/nodes//storage/felhom-pbs/content` returns `content:"backup"` + ctime/vmid/volid) — `PickRestoreCandidate`/`latestArchive` need NO PBS-client change. -- **pct restore from a PBS volid** round-trips (storage.cfg encryption key applied transparently). PBS gotchas (`ignore-verified`, node-from-UPID, privsep) touch only the verify-API path. +## What shipped (v0.29.0) +- **`configs/build-golden.sh`** — split baked in: `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` (default + **32**, was hardcoded 8) **plus** `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1` + (default 16). `daemon.json` bakes the **classic overlay2 driver** (`features.containerd-snapshotter: + false`) + log rotation (`max-size 10m`, `max-file 3`). Guards: aborts unless `/var/lib/docker` is a + separate mount, the driver is overlay2, and vzdump **includes** mp0 (the B3 trap). +- **`internal/reconcile/bringup.go`** — `GuestMount.Backup` emits `,backup=1` (closes the spike-B3/B5 + silent-DB-loss trap at the mount builder). `BringUpSpec.{DataVolGrowGB,DataVolMount}` grow the + golden-carried Docker-data volume (default `mp0`) online to the per-customer target rather than + attaching a fresh empty volume that would shadow the baked images. Plus `RootfsGrowGB`. +- **CLI:** `--selftest=bring-up|provision` gain `-rootfs-grow` / `-datavol-grow` / `-datavol-mount`. +- **`RUNBOOK-provisioning-storage.md`** — provisioning procedure + fresh-PVE-install thin-pool carving + (`hdsize`/`maxroot`/`maxvz`) + the per-customer sizing seam (flags now; hub storage manifest later). +- Tests: `buildBringUpConfig` backup=1 emission; bring-up issues rootfs + data-volume resizes. -## PHASE 1 — backup re-target → felhom-pbs -- `config.BackupConfig.BackupTarget()` returns `backup.local_backup_target` or, when empty, the new default `felhom-pbs` (separate hardware = real DR). Configurable; no call site hardcodes it. All `NewBackupRunner` sites + restore-test `SourceTier` route through it. -- Live: demo `agent.json` `local_backup_target` "local" → **felhom-pbs**; `--selftest=backup -vmid 9201` → `target=felhom-pbs`, snapshot, crash-consistent ✓. Daemon `/backup/status` now reports `target_id=felhom-pbs`. -- **PBS prune default** set on the storage: `prune-backups keep-daily=7,keep-weekly=4,keep-monthly=3` (was keep-all at 73% full). +## The critical fix validation caught (overlay2) +Docker 29's **default containerd-snapshotter keeps the image store at `/var/lib/containerd`** (on the OS +rootfs), so mounting the data volume at `/var/lib/docker` only moved named volumes — **1.2 GB of images +stayed on the rootfs** (validated live), defeating the split and breaking the controller's `statfs("/")` +guard. Fix: the classic **overlay2** driver stores everything (images + overlay + volumes) under +data-root = the data volume. A start-vs-restart trap (docker-ce auto-starts on install) was also fixed +(daemon.json needs a `restart`). After the fix, a provisioned guest showed images on the data volume +(`/var/lib/docker/overlay2`), `/var/lib/containerd` idle, and a lean OS rootfs. -## Restore-test bind-mount fix (validation-discovered; scope-approved) -- A slice-10 enrolled guest's data drive is a host **bind-mount** `mp0`; vzrestore refuses it under the privsep token ("restoring 'mpN' to bind mount is only possible for root") — so the restore-test failed for EVERY enrolled guest, independent of backup tier. -- Fix: the restore-test reads the SOURCE guest config (vmid parsed from the archive volid) and passes `RestoreLXCOptions.MountOverrides` converting each bind-mount `mpN` → a throwaway 1G volume on the restore storage, plus a `rootfs` override sized from the source (PVE requires rootfs when mp params are present). Boot-verify doesn't need the data. Storage-backed mounts restore normally; best-effort if the source config is unreadable. -- Live: daemon **scheduled restore-test from felhom-pbs → `pass:true, verified:"boot+running", source_tier:"pbs"`** (44s, scratch torn down clean). +## Live validation +1. **Re-baked the golden** with the split (overlay2): docker OK on overlay2, `/var/lib/docker` a separate + ext4 mount, vzdump "including mount point mp0 ('/var/lib/docker')", 580 MB archive carrying the images. +2. **Throwaway provision (9301)** from the golden: booted in 39 s; `pct config` mp0=`/var/lib/docker,backup=1,size=256G`, + rootfs 32G; overlay2; baked images on the data volume; **OS isolation** — filled the data volume to + 100%, the OS rootfs stayed at 4% + writable; log rotation effective. +3. **Destroyed + re-provisioned the live 9201** from the same golden via `--selftest=provision` + (customer `demo-felhom`, retrieval passphrase reused from its old bootstrap; `-datavol-grow 240` → + 256 GB) + a reboot: bring-up booted in 39 s, the back-half minted a token + attached the bootstrap + mp9, and after reboot the controller-bootstrap deployed the baked controller, which pulled its config + from the hub. All four infra containers came up **from baked images with no pull**. Split + overlay2 + + lean rootfs confirmed on the real guest; the controller's deploy gate refused (507) on the full data + volume. (Phase-2 details in the controller REPORT.) -## PHASE 3 — operator-signed decommission (reachable now) -- The previously-unreachable `IntentDecommissioned` is now reached ONLY via a gate-VERIFIED operator signature (classified destructive). New `signedjobs.DecommissionExecutor` (op `decommission`) → `IntentStore.SetDecommissioned`, keyed by the drive's **storage** durable-id (the watchdog's key, e.g. `uuid:` — NOT the device-level `byid:/byuuid:` of `storage_wipe`), so the intent actually gates remounts. New `ExecutorChain` serves both wipe + decommission; runner wiring moved below the intent-store open. `felhom-opsign` builds decommission params from `-durable-id`. No customer/controller UI — operator path is hub jobs-queue → signed-jobs runner. Distinct from a customer-confirmable safe eject. +## Provisioning facts confirmed (for the spec/runbook) +- The restore carries all mountpoints (golden mp0 travels in); `freeMountSlot` picks the lowest free slot + for a later USB enroll, so **mp0 = Docker-data, mp9 = bootstrap, USB = mp1+** — no collision. +- Attaching/flag-changing a mountpoint on a running unprivileged guest is **pending until reboot** — the + provision flow's post-provision reboot activates the bootstrap mp9 (and would activate a USB bind). +- `backup=1` on the Docker-data mp is mandatory twice over: keeps named-volume DBs in PBS, AND captures + the volume (with baked images) into the golden archive. -## Tests -- `internal/signedjobs`: decommission happy-path / foreign-op / unbound-refusal / no-store; ExecutorChain dispatch. -- `internal/reconcile`: `archiveVMID` (PBS ct/vm + vzdump forms), `bindMountOverrides`, `rootfsSizeGB`/`sizeToGB`. -- Full `go build ./...` + package tests green. - -## Live deploy -- `felhom-agent 0.28.0` active on demo-felhom (prior binary kept as `felhom-agent.bak-0.27.0`); config `local_backup_target=felhom-pbs`, restore-test cadence restored to 86400s. -- **Golden prune** (housekeeping): kept the newest 2 golden (VMID 9100) archives on `local`, freed the older 11. -- **Note (not actioned):** the drive's legacy bare-metal `felhom_data` (underscore) dir is orphaned under the new `felhom-data` (hyphen) namespace on `/mnt/felhom-usb`; flagged for a real-customer migration path (out of scope here). +## Outstanding (demo restoration, not slice validation) +RomM (the HDD app) + USB re-enroll: its data is safe on the host USB (`/mnt/felhom-usb/felhom-data`, +untouched). Restoring it is the slice-10 enroll flow (assign → guest-attach → reboot → register storage +→ deploy); with the split it binds to mp1+ (mp0 is the Docker-data volume). Documented as the final +restore step.