admin
66bd015ee5
Validated v0.5.1 live: observe (durable_id=fs-UUID, class_hint slow, SMART PASSED via USB-SATA bridge, lvs thin-pool metadata), durable_id stability, and the watchdog cycle — mount-drop (W2) and reversible device soft-remove/return (W3): detect in ~3s, re-mount by stable by-UUID systemd unit, and NO re-mount while the device is truly gone. Both v0.5.1 fixes confirmed live. Physical H1/H2 + end-to-end hub reflection flagged as not done. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
107 lines
6.6 KiB
Markdown
107 lines
6.6 KiB
Markdown
# REPORT — Slice 5 live storage validation on `demo-felhom` (felhom-agent v0.5.1) (2026-06-09)
|
||
|
||
> Overwrite-latest report (most recent significant work only). Cumulative history lives in [CHANGELOG.md](CHANGELOG.md).
|
||
|
||
## Outcome
|
||
|
||
**Slice 5 (storage observe/report + watchdog + the host-root surface) is validated live** on the
|
||
demo Proxmox host `demo-felhom` against a real **1 TB USB HDD** (`sdb1`, ext4, label `hdd_1`,
|
||
fs-UUID `277a2179-…`, rotational). Agent **v0.5.1** (the validation-prep fixes — unmounted-dir
|
||
durable_id mis-id + watchdog UUID memory). **No live destructive op** — the destructive
|
||
detach/wipe gate refusal is proven in the unit suite, never exercised on the real drive.
|
||
|
||
Two HUMAN steps (physical unplug/replug, H1/H2) were **software-simulated** by CC via a
|
||
reversible USB `authorized` toggle (W3) — same agent code paths; see "Not done" for the real
|
||
physical confirmation, which is Viktor's to run.
|
||
|
||
## Pre-flight (P1–P3)
|
||
|
||
- **P1** — staged the v0.5.1 binary at `/usr/local/bin/felhom-agent` on `felhom-pve`; wrote
|
||
`/root/.config/felhom-agent/agent.json` (mode 600). The earlier-slice token secret was
|
||
**stale (HTTP 401 — host rebuilt 2026-06-07)**, so CC **regenerated** the `felhom-agent@pve!agent`
|
||
token (privsep) and re-granted the token ACL (a `pveum user token remove` purges the token's
|
||
ACL — re-added `FelhomAgent` on the token). The `FelhomAgent` role (16 privs) + the user ACL
|
||
were already correct. Token stored out-of-band (config, 0600) — never committed. The TLS pin
|
||
is the **served leaf cert** `BA:7C:99:7D:45:D0:…` (note: `pvenode cert info` reports a
|
||
*different* cert — `B2:02:2E…` — so always pin the value from `openssl s_client … :8006`).
|
||
Runs as **root@pam** here, so `privileged.mode = "direct"` (no sudo prefix; identical HostOps
|
||
code + arg-validation). `smartctl`/`lvs`/`systemctl`/`install` all present.
|
||
**`--selftest=read` → fully green** (version, nodes, node-status, 2 guests, 2 stores).
|
||
- **P2** — the 1 TB USB HDD is present as `sdb`/`sdb1` (ext4, UUID `277a2179-…`, ROTA=1,
|
||
TRAN=usb, removable=0). It was **not** previously a Proxmox storage, so CC set it up
|
||
(non-destructive: mounted the already-formatted partition by-UUID at `/mnt/felhom-usb`).
|
||
- **P3 — `is_mountpoint 1`**: defined `pvesm add dir felhom-usb --path /mnt/felhom-usb
|
||
--content backup --is_mountpoint 1`. Before: no USB storage existed; after: defined **with**
|
||
`is_mountpoint 1`, so when the drive drops PVE marks the storage inactive instead of writing
|
||
to the bare mountpoint dir on root.
|
||
|
||
## Static observe (S1–S2)
|
||
|
||
**S1 — `--selftest=storage`** reported `felhom-usb`:
|
||
- `type=local-dir` (removable flag is 0 on this USB HDD, so it classifies as local-dir, not
|
||
usb — both are mount-backed and treated identically; noted), `state=attached`, `reachable=true`;
|
||
- **`durable_id = uuid:277a2179-a764-4758-b840-9ea741517914`** — matches `blkid` exactly (the
|
||
DR re-attach key is correct);
|
||
- `class_hint = slow` (rotational HDD ✓);
|
||
- usage `8.0/915.8 GiB`, `mount=/mnt/felhom-usb`, `dev=/dev/sdb1`;
|
||
- **`smart: health=PASSED temp=30 poh=2419 realloc=0 pending=0 offline_unc=0`** — SMART works
|
||
through the USB-SATA bridge (SATA attribute set parsed; NOT degraded to UNKNOWN);
|
||
- bonus: `local-lvm` **thin-pool data=1.9% metadata=0.5%** — the Phase-B `lvs` metadata fill
|
||
is populated live; `local` → `durable=path:/var/lib/vz` (correctly not a fs-UUID).
|
||
|
||
**S2 — durable_id stability**: re-ran twice; `durable_id` identical both times.
|
||
|
||
## Watchdog cycle — software-simulated (W1–W3)
|
||
|
||
Watchdog config for the test: 5 s poll, 15 s debounce, 20 s known-refresh.
|
||
|
||
- **W2 — mount-drop** (`umount` at T=0):
|
||
- **+2.8 s** → `attached → disconnected` detected + out-of-band report fired + **re-mount
|
||
dispatched** (device still present, just unmounted);
|
||
- **+3.7 s** → **re-mounted by-UUID** via a freshly-written systemd unit
|
||
`mnt-felhom\x2dusb.mount` (`active`); `disconnected → attached` detected at +8 s; debounced
|
||
trailing report at +17 s. Detect latency **seconds, not the ~15-min cycle**.
|
||
- **W3 — device soft-remove** (reversible USB `authorized` 0→1, simulating unplug/replug):
|
||
- deauthorize → block device + `/dev/disk/by-uuid/277a…` **fully gone**;
|
||
- **+2.8 s** → `attached → disconnected` detected + report — **and crucially NO re-mount**
|
||
(device absent → nothing to mount, correct); only on **reauthorize** (+6 s) did the
|
||
watchdog dispatch the re-mount and **mount by the stable by-UUID unit**, so a different
|
||
`/dev/sdX` enumeration would still mount the right filesystem (the H2 goal, simulated).
|
||
|
||
This exercised both the v0.5.1 fixes live: an unmounted target never reported a root-fs UUID,
|
||
and the re-mount used the remembered UUID across the known-set cache refresh.
|
||
|
||
## Captured timings
|
||
|
||
| event | latency |
|
||
|---|---|
|
||
| umount → `disconnected` detected (W2) | ~2.8 s |
|
||
| umount → re-mounted by-UUID (W2) | ~3.7 s |
|
||
| deauthorize → `disconnected` detected, **no re-mount** (W3) | ~2.8 s |
|
||
| reauthorize → re-mounted by-UUID (W3) | ~6 s |
|
||
|
||
## Not done (flagged)
|
||
|
||
- **H1/H2 (physical unplug/replug) — HUMAN (Viktor).** Software-simulated by W3 (reversible
|
||
`authorized` toggle), which drives the identical agent code (device-gone detect, no-re-mount-
|
||
while-gone, re-mount-by-UUID on return). A real cable pull is the final confirmation; the
|
||
agent logic is already exercised.
|
||
- **End-to-end hub reflection** (the out-of-band report landing on `hub.felhom.eu`): the
|
||
`--selftest=storage` harness is standalone (no hub), so the trigger **logged** “would send
|
||
out-of-band host-report now” at the right moments rather than POSTing. Verifying the hub
|
||
dashboard shows disconnected→attached needs the **full daemon with a minted per-host key**
|
||
(enrolling `demo-felhom` on the live operator hub — additive but outward-facing state). The
|
||
watchdog→report-trigger path is proven here; the hub-side ingest of `storage_targets` was
|
||
proven during the hub v0.7.2 deploy. **Offered, not done unprompted.**
|
||
|
||
## Cleanup / end-state
|
||
|
||
- Drive **mounted/attached**, PVE storage `felhom-usb` **active**, systemd unit
|
||
`mnt-felhom\x2dusb.mount` **active + enabled** (survives reboot), `is_mountpoint 1` **kept** —
|
||
the intended managed end-state.
|
||
- No `--watch`/daemon process left running; `/tmp/watch*.log` removed. The sudoers allowlist is
|
||
installed at `/etc/sudoers.d/felhom-agent` (inert here — `sudo` isn't installed on this
|
||
minimal PVE, and the agent runs as root; it documents the eventual non-root deployment).
|
||
- Nothing was wiped/detached; the regenerated Proxmox token lives only in the agent config
|
||
(0600) on `felhom-pve`, never committed.
|