felhom-agent/REPORT.md

REPORT — Slice 5 live storage validation on demo-felhom (felhom-agent v0.5.1) (2026-06-09)

Overwrite-latest report (most recent significant work only). Cumulative history lives in CHANGELOG.md.

Outcome

Slice 5 (storage observe/report + watchdog + the host-root surface) is validated live on the demo Proxmox host demo-felhom against a real 1 TB USB HDD (sdb1, ext4, label hdd_1, fs-UUID 277a2179-…, rotational). Agent v0.5.1 (the validation-prep fixes — unmounted-dir durable_id mis-id + watchdog UUID memory). No live destructive op — the destructive detach/wipe gate refusal is proven in the unit suite, never exercised on the real drive.

Two HUMAN steps (physical unplug/replug, H1/H2) were software-simulated by CC via a reversible USB authorized toggle (W3) — same agent code paths; see "Not done" for the real physical confirmation, which is Viktor's to run.

Pre-flight (P1–P3)

• P1 — staged the v0.5.1 binary at /usr/local/bin/felhom-agent on felhom-pve; wrote /root/.config/felhom-agent/agent.json (mode 600). The earlier-slice token secret was stale (HTTP 401 — host rebuilt 2026-06-07), so CC regenerated the felhom-agent@pve!agent token (privsep) and re-granted the token ACL (a pveum user token remove purges the token's ACL — re-added FelhomAgent on the token). The FelhomAgent role (16 privs) + the user ACL were already correct. Token stored out-of-band (config, 0600) — never committed. The TLS pin is the served leaf cert BA:7C:99:7D:45:D0:… (note: pvenode cert info reports a different cert — B2:02:2E… — so always pin the value from openssl s_client … :8006). Runs as root@pam here, so privileged.mode = "direct" (no sudo prefix; identical HostOps code + arg-validation). smartctl/lvs/systemctl/install all present. --selftest=read → fully green (version, nodes, node-status, 2 guests, 2 stores).
• P2 — the 1 TB USB HDD is present as sdb/sdb1 (ext4, UUID 277a2179-…, ROTA=1, TRAN=usb, removable=0). It was not previously a Proxmox storage, so CC set it up (non-destructive: mounted the already-formatted partition by-UUID at /mnt/felhom-usb).
• P3 — is_mountpoint 1: defined pvesm add dir felhom-usb --path /mnt/felhom-usb --content backup --is_mountpoint 1. Before: no USB storage existed; after: defined with is_mountpoint 1, so when the drive drops PVE marks the storage inactive instead of writing to the bare mountpoint dir on root.

Static observe (S1–S2)

S1 — --selftest=storage reported felhom-usb:

• type=local-dir (removable flag is 0 on this USB HDD, so it classifies as local-dir, not usb — both are mount-backed and treated identically; noted), state=attached, reachable=true;
• durable_id = uuid:277a2179-a764-4758-b840-9ea741517914 — matches blkid exactly (the DR re-attach key is correct);
• class_hint = slow (rotational HDD ✓);
• usage 8.0/915.8 GiB, mount=/mnt/felhom-usb, dev=/dev/sdb1;
• smart: health=PASSED temp=30 poh=2419 realloc=0 pending=0 offline_unc=0 — SMART works through the USB-SATA bridge (SATA attribute set parsed; NOT degraded to UNKNOWN);
• bonus: local-lvm thin-pool data=1.9% metadata=0.5% — the Phase-B lvs metadata fill is populated live; local → durable=path:/var/lib/vz (correctly not a fs-UUID).

S2 — durable_id stability: re-ran twice; durable_id identical both times.

Watchdog cycle — software-simulated (W1–W3)

Watchdog config for the test: 5 s poll, 15 s debounce, 20 s known-refresh.

• W2 — mount-drop (umount at T=0):
  • +2.8 s → attached → disconnected detected + out-of-band report fired + re-mount dispatched (device still present, just unmounted);
  • +3.7 s → re-mounted by-UUID via a freshly-written systemd unit mnt-felhom\x2dusb.mount (active); disconnected → attached detected at +8 s; debounced trailing report at +17 s. Detect latency seconds, not the ~15-min cycle.
• W3 — device soft-remove (reversible USB authorized 0→1, simulating unplug/replug):
  • deauthorize → block device + /dev/disk/by-uuid/277a… fully gone;
  • +2.8 s → attached → disconnected detected + report — and crucially NO re-mount (device absent → nothing to mount, correct); only on reauthorize (+6 s) did the watchdog dispatch the re-mount and mount by the stable by-UUID unit, so a different /dev/sdX enumeration would still mount the right filesystem (the H2 goal, simulated).

This exercised both the v0.5.1 fixes live: an unmounted target never reported a root-fs UUID, and the re-mount used the remembered UUID across the known-set cache refresh.

Captured timings

event	latency
umount → disconnected detected (W2)	~2.8 s
umount → re-mounted by-UUID (W2)	~3.7 s
deauthorize → disconnected detected, no re-mount (W3)	~2.8 s
reauthorize → re-mounted by-UUID (W3)	~6 s

Not done (flagged)

• H1/H2 (physical unplug/replug) — HUMAN (Viktor). Software-simulated by W3 (reversible authorized toggle), which drives the identical agent code (device-gone detect, no-re-mount- while-gone, re-mount-by-UUID on return). A real cable pull is the final confirmation; the agent logic is already exercised.
• End-to-end hub reflection (the out-of-band report landing on hub.felhom.eu): the --selftest=storage harness is standalone (no hub), so the trigger logged “would send out-of-band host-report now” at the right moments rather than POSTing. Verifying the hub dashboard shows disconnected→attached needs the full daemon with a minted per-host key (enrolling demo-felhom on the live operator hub — additive but outward-facing state). The watchdog→report-trigger path is proven here; the hub-side ingest of storage_targets was proven during the hub v0.7.2 deploy. Offered, not done unprompted.

Cleanup / end-state

• Drive mounted/attached, PVE storage felhom-usb active, systemd unit mnt-felhom\x2dusb.mount active + enabled (survives reboot), is_mountpoint 1 kept — the intended managed end-state.
• No --watch/daemon process left running; /tmp/watch*.log removed. The sudoers allowlist is installed at /etc/sudoers.d/felhom-agent (inert here — sudo isn't installed on this minimal PVE, and the agent runs as root; it documents the eventual non-root deployment).
• Nothing was wiped/detached; the regenerated Proxmox token lives only in the agent config (0600) on felhom-pve, never committed.