docs(REPORT): slice 5 live storage validation on demo-felhom (1TB USB HDD)

Validated v0.5.1 live: observe (durable_id=fs-UUID, class_hint slow, SMART PASSED via
USB-SATA bridge, lvs thin-pool metadata), durable_id stability, and the watchdog cycle —
mount-drop (W2) and reversible device soft-remove/return (W3): detect in ~3s, re-mount
by stable by-UUID systemd unit, and NO re-mount while the device is truly gone. Both
v0.5.1 fixes confirmed live. Physical H1/H2 + end-to-end hub reflection flagged as not done.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

REPORT.md

@@ -1,79 +1,106 @@
-# REPORT — Slice 5 Phase B: the host-root surface (v0.5.0) (2026-06-09)
+# REPORT — Slice 5 live storage validation on `demo-felhom` (felhom-agent v0.5.1) (2026-06-09)
 
 > Overwrite-latest report (most recent significant work only). Cumulative history lives in [CHANGELOG.md](CHANGELOG.md).
 
 ## Outcome
 
-**Slice 5 is complete and pushed as `v0.5.0`.** Phase B adds the privileged **write surface** —
+**Slice 5 (storage observe/report + watchdog + the host-root surface) is validated live** on the
-the agent's first step outside its Proxmox API token into OS-root — isolated behind a narrow,
+demo Proxmox host `demo-felhom` against a real **1 TB USB HDD** (`sdb1`, ext4, label `hdd_1`,
-argument-validated, adversarially-tested seam, the same discipline as the slice-4 reversibility
+fs-UUID `277a2179-…`, rotational). Agent **v0.5.1** (the validation-prep fixes — unmounted-dir
-gate. Phase A (read-only observe/report/watchdog, `v0.5.0-rc1`) is reused unchanged.
+durable_id mis-id + watchdog UUID memory). **No live destructive op** — the destructive
+detach/wipe gate refusal is proven in the unit suite, never exercised on the real drive.
 
-## What landed
+Two HUMAN steps (physical unplug/replug, H1/H2) were **software-simulated** by CC via a
+reversible USB `authorized` toggle (W3) — same agent code paths; see "Not done" for the real
+physical confirmation, which is Viktor's to run.
 
-- **`HostOps` seam + `SudoHostOps`** — the one privileged surface: persistent mounts via
+## Pre-flight (P1–P3)
-  **systemd `.mount` units keyed by fs-UUID** (enabled, survive reboot), detach (stop+disable),
-  SMART, thin-pool metadata. Shells out via the fenced Runner (`sudo -n`, fixed arg vectors,
-  **no shell**). Tests use a fake — **no real root in the suite**. `NoopHostOps` is the
-  degrade-cleanly fallback.
-- **The argument validator (the security boundary)** — `ValidateUUID` / `ValidateMountPath`
-  (absolute, no traversal, no metacharacters) / `ValidateSMARTDevice` (raw-disk whitelist) /
-  `ValidateLVMName`, plus an in-process `systemd-escape`. **Every argument is validated before a
-  command is constructed.** The headline test is an adversarial matrix (shell metacharacters,
-  `../`, malformed) that must be refused with **zero exec** — proven both at the validators and
-  at `SudoHostOps` (a recording runner asserts no command was built).
-- **SMART** — `smartctl -a -j` parsed into `StorageTarget.smart` for **SATA and NVMe** attribute
-  sets, degrading to `UNKNOWN` on devices with no SMART. **`lvs`** fills the lvmthin thin-pool
-  **metadata** fill. Enrichment runs in `Observe` only (not the watchdog's fast `Known` path).
-- **Watchdog re-mount response** — a new `DevicePresent` probe lets the watchdog detect a known
-  mount-backed target whose **device returned but is unmounted**, and **dispatch a benign
-  by-UUID re-mount off the poll path** (goroutine, never under the lock), rate-limited to the
-  debounce window. Routed through the gate as benign (`gateRemounter` in `main.go`, keeping
-  `storage` decoupled from `reconcile`).
-- **Disk-grow executor** — `proxmox.ResizeLXC` (API, async) + a benign `ActionResize` planned
-  **grow-only**; a shrink is refused (never silently grown) at the planner and re-guarded at the
-  executor. Built + fixture-tested; unfed live.
-- **Destructive storage ops through the slice-4 gate** — `IntentForStorageMount` (benign) /
-  `IntentForStorageDestructive` (`ClassStorageWipe`/`ClassDecommission`), host/target-scoped
-  (binds on the storage **target identity** in `target.guest_id`). Reuses the existing
-  verifier/role-scoping/binding/audit. Adversarial cases: unsigned → `pending_signature`,
-  wrong-target → `binding_mismatch`, valid → accepted. Inert live.
-- **`--selftest=storage` [`-watch <dur>`]** — the live USB-runbook harness (observe pass + a
-  bounded watchdog window with re-mount live), standalone on the Proxmox host (no hub).
-- **`configs/felhom-agent.sudoers`** — the documented narrow allowlist + the agent-side fine
-  validation. New `privileged.*` config knobs for the binary paths / dirs.
 
-## Tests
+- **P1** — staged the v0.5.1 binary at `/usr/local/bin/felhom-agent` on `felhom-pve`; wrote
+  `/root/.config/felhom-agent/agent.json` (mode 600). The earlier-slice token secret was
+  **stale (HTTP 401 — host rebuilt 2026-06-07)**, so CC **regenerated** the `felhom-agent@pve!agent`
+  token (privsep) and re-granted the token ACL (a `pveum user token remove` purges the token's
+  ACL — re-added `FelhomAgent` on the token). The `FelhomAgent` role (16 privs) + the user ACL
+  were already correct. Token stored out-of-band (config, 0600) — never committed. The TLS pin
+  is the **served leaf cert** `BA:7C:99:7D:45:D0:…` (note: `pvenode cert info` reports a
+  *different* cert — `B2:02:2E…` — so always pin the value from `openssl s_client … :8006`).
+  Runs as **root@pam** here, so `privileged.mode = "direct"` (no sudo prefix; identical HostOps
+  code + arg-validation). `smartctl`/`lvs`/`systemctl`/`install` all present.
+  **`--selftest=read` → fully green** (version, nodes, node-status, 2 guests, 2 stores).
+- **P2** — the 1 TB USB HDD is present as `sdb`/`sdb1` (ext4, UUID `277a2179-…`, ROTA=1,
+  TRAN=usb, removable=0). It was **not** previously a Proxmox storage, so CC set it up
+  (non-destructive: mounted the already-formatted partition by-UUID at `/mnt/felhom-usb`).
+- **P3 — `is_mountpoint 1`**: defined `pvesm add dir felhom-usb --path /mnt/felhom-usb
+  --content backup --is_mountpoint 1`. Before: no USB storage existed; after: defined **with**
+  `is_mountpoint 1`, so when the drive drops PVE marks the storage inactive instead of writing
+  to the bare mountpoint dir on root.
 
-`go test ./...` green; **`go test -race ./...` green on the build server** (the watchdog
+## Static observe (S1–S2)
-re-mount dispatch runs off the poll path concurrently). New tests: the **validator adversarial
-matrix** (headline) + `SudoHostOps` "no-exec-on-hostile-arg"; HostOps mount/re-mount lifecycle
-against a fake; SMART parsing SATA + NVMe + unsupported; thin-pool metadata parse; the grow
-executor (grow applies, non-grow refused at the executor) + plan grow-only/shrink-refused;
-the destructive-storage gate cases (unsigned / wrong-target / valid) reusing the slice-4
-real-verifier harness; and the watchdog → re-mount path (device-return triggers, rate-limited,
-re-armed after a successful mount).
 
-## Design decisions (flagged for the reviewer)
+**S1 — `--selftest=storage`** reported `felhom-usb`:
+- `type=local-dir` (removable flag is 0 on this USB HDD, so it classifies as local-dir, not
+  usb — both are mount-backed and treated identically; noted), `state=attached`, `reachable=true`;
+- **`durable_id = uuid:277a2179-a764-4758-b840-9ea741517914`** — matches `blkid` exactly (the
+  DR re-attach key is correct);
+- `class_hint = slow` (rotational HDD ✓);
+- usage `8.0/915.8 GiB`, `mount=/mnt/felhom-usb`, `dev=/dev/sdb1`;
+- **`smart: health=PASSED temp=30 poh=2419 realloc=0 pending=0 offline_unc=0`** — SMART works
+  through the USB-SATA bridge (SATA attribute set parsed; NOT degraded to UNKNOWN);
+- bonus: `local-lvm` **thin-pool data=1.9% metadata=0.5%** — the Phase-B `lvs` metadata fill
+  is populated live; `local` → `durable=path:/var/lib/vz` (correctly not a fs-UUID).
 
-1. **Unit-file write mechanism.** The agent stages the `.mount` unit to an agent-owned dir
+**S2 — durable_id stability**: re-ran twice; `durable_id` identical both times.
-   (`privileged.stage_dir`, default `/var/lib/felhom-agent/units`) then `sudo install -o root
-   -g root -m 0644 -- <stage> /etc/systemd/system/<unit>`. This avoids a stdin-to-root channel
-   and keeps fixed arg vectors. The sudoers entry uses path-prefix wildcards; the agent's strict
-   validation (unit name derived from a validated mountpoint; no traversal) is the fine gate, so
-   the wildcard can't be abused. Swappable behind the seam if you prefer another mechanism.
-2. **Storage-op gate scoping.** Host/target-scoped destructive ops carry the storage **target
-   name** (the operator handle / hub manifest key) in `target.guest_id`, with VMID 0. So "wipe A"
-   cannot authorize "wipe B" (binding_mismatch), exactly like the wrong-guest case.
-3. **Shrink handling.** A desired disk < actual is refused by omission at the planner (no
-   action) and re-guarded at the executor (size must start with `+`); a deliberate shrink would
-   be a signed destructive op (slice 10), never a benign reconcile grow.
 
-## Live validation (for project Claude / the runbook)
+## Watchdog cycle — software-simulated (W1–W3)
 
-Not run from here (the observer reads the *local* host's `/proc/mounts` + privileged tools, so a
+Watchdog config for the test: 5 s poll, 15 s debounce, 20 s known-refresh.
-meaningful run must be **on the demo Proxmox host**, not the build server). The
+
-`--selftest=storage -watch` harness is the intended live USB-cycle test:
+- **W2 — mount-drop** (`umount` at T=0):
-`felhom-agent --selftest=storage -watch 3m` on `felhom-pve`, then physically unplug/replug a USB
+  - **+2.8 s** → `attached → disconnected` detected + out-of-band report fired + **re-mount
-target and watch detect → out-of-band report → by-UUID re-mount in the logs. SMART/lvs need the
+    dispatched** (device still present, just unmounted);
-sudoers drop-in installed (`configs/felhom-agent.sudoers`).
+  - **+3.7 s** → **re-mounted by-UUID** via a freshly-written systemd unit
+    `mnt-felhom\x2dusb.mount` (`active`); `disconnected → attached` detected at +8 s; debounced
+    trailing report at +17 s. Detect latency **seconds, not the ~15-min cycle**.
+- **W3 — device soft-remove** (reversible USB `authorized` 0→1, simulating unplug/replug):
+  - deauthorize → block device + `/dev/disk/by-uuid/277a…` **fully gone**;
+  - **+2.8 s** → `attached → disconnected` detected + report — **and crucially NO re-mount**
+    (device absent → nothing to mount, correct); only on **reauthorize** (+6 s) did the
+    watchdog dispatch the re-mount and **mount by the stable by-UUID unit**, so a different
+    `/dev/sdX` enumeration would still mount the right filesystem (the H2 goal, simulated).
+
+This exercised both the v0.5.1 fixes live: an unmounted target never reported a root-fs UUID,
+and the re-mount used the remembered UUID across the known-set cache refresh.
+
+## Captured timings
+
+| event | latency |
+|---|---|
+| umount → `disconnected` detected (W2) | ~2.8 s |
+| umount → re-mounted by-UUID (W2) | ~3.7 s |
+| deauthorize → `disconnected` detected, **no re-mount** (W3) | ~2.8 s |
+| reauthorize → re-mounted by-UUID (W3) | ~6 s |
+
+## Not done (flagged)
+
+- **H1/H2 (physical unplug/replug) — HUMAN (Viktor).** Software-simulated by W3 (reversible
+  `authorized` toggle), which drives the identical agent code (device-gone detect, no-re-mount-
+  while-gone, re-mount-by-UUID on return). A real cable pull is the final confirmation; the
+  agent logic is already exercised.
+- **End-to-end hub reflection** (the out-of-band report landing on `hub.felhom.eu`): the
+  `--selftest=storage` harness is standalone (no hub), so the trigger **logged** “would send
+  out-of-band host-report now” at the right moments rather than POSTing. Verifying the hub
+  dashboard shows disconnected→attached needs the **full daemon with a minted per-host key**
+  (enrolling `demo-felhom` on the live operator hub — additive but outward-facing state). The
+  watchdog→report-trigger path is proven here; the hub-side ingest of `storage_targets` was
+  proven during the hub v0.7.2 deploy. **Offered, not done unprompted.**
+
+## Cleanup / end-state
+
+- Drive **mounted/attached**, PVE storage `felhom-usb` **active**, systemd unit
+  `mnt-felhom\x2dusb.mount` **active + enabled** (survives reboot), `is_mountpoint 1` **kept** —
+  the intended managed end-state.
+- No `--watch`/daemon process left running; `/tmp/watch*.log` removed. The sudoers allowlist is
+  installed at `/etc/sudoers.d/felhom-agent` (inert here — `sudo` isn't installed on this
+  minimal PVE, and the agent runs as root; it documents the eventual non-root deployment).
+- Nothing was wiped/detached; the regenerated Proxmox token lives only in the agent config
+  (0600) on `felhom-pve`, never committed.