From 79eb0a848615c08db0b003f61137d5e4d3d27801 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 30 Jun 2026 11:40:08 +0200 Subject: [PATCH] docs: REPORT for v0.50.0 (NAS Part A1, sim-validated A-D) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs --- REPORT.md | 168 ++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 113 insertions(+), 55 deletions(-) diff --git a/REPORT.md b/REPORT.md index 2121783..a820654 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,70 +1,128 @@ -# REPORT — felhom-agent v0.49.0 +# REPORT — agent v0.50.0: NAS network storage Part A1 (NFS/SMB automount foundation) -Closes the two host-reboot findings (F2-a, F2-b) from -`felhom.eu/documentation/audits/TESTRUN-fullstack-2026-06-29.md`. Diagnose-before-fix on both; both root -causes confirmed live on `felhom-pve` before any code change. +**Date:** 2026-06-30 +**Task:** `TASK.md` — NAS network storage Part A1 (agent foundation: automount + bind + liveness). +**Class:** Risky / supervised (creates real network mounts on the Proxmox host). Implemented on `main`, +sim-validated live, sim torn down. -## F2-b — reboot-during-backup stale-lock recovery (Part B) +## Baseline (confirmed live) -**Diagnosis (B.0, confirmed live).** An interrupted vzdump leaves the guest config with a backup lock -(`lock: snapshot-delete` after a snapshot-mode backup's cleanup, or `lock: backup`) + a dangling `vzdump` -snapshot. `onboot:1` then fails: *"Starting CT … failed: CT is locked (snapshot-delete)"* — the box stays -DOWN. **Invariant:** at agent startup the agent's own backup loop has not run, so a backup lock present then -is stale by definition — *unless* a vzdump is genuinely in-flight (external, or one that outlived a bare -agent restart). That is the one case clearing would corrupt, so the recovery confirms no vzdump is running -and **fails safe** (leaves the lock) if it can't. +| | value | +|---|---| +| Agent `main` before | v0.49.0 (commit `7aeb7ca`), live on felhom-pve | +| Agent after | **v0.50.0** (commit `63aa63d`) | +| Deployed binary sha256 (felhom-pve) | `6ddb6ff2efe6b10377e00ff7612f738ff437dfbcbcc35e3fc566e40d139aedf1` | +| Host client tooling | `mount.nfs` (nfs-common) + `mount.cifs` (cifs-utils) present | -**Fix.** `internal/localapi/stalelock.go` (NEW) + `Server.RecoverStaleLockedGuests`, run at startup next to -`ReassertGuestBinds`/`RecoverFormatJob`. Per guest with a backup lock and no in-flight vzdump: `pct unlock` -→ delete the dangling `vzdump` snapshot (API + WaitTask, only when one exists) → start **iff** `onboot` and -not already running. Reads (`GuestConfig.Lock()`/`OnBoot()`, `ListSnapshots`, `ListRunningTasks`), -snapshot-delete and start go through the API token; only `pct unlock` shells out (no API equivalent). +## What was built -**Grants.** New narrow `FELHOM_STALELOCK = /usr/sbin/pct unlock [0-9]*` + Critical capability -`stalelock-unlock`. `visudo -cf` clean; the manifest↔sudoers build gate covers it. +The agent foundation of `SPIKE-nas-storage-2026-06-29.md` (verdict READY) — a customer NAS can serve +**bulk media** to a media app. The agent mounts a NAS share **host-side** under `/mnt/felhom-drives/` +via a systemd `.automount` (+ `.mount`) pair; it propagates into guest 9201 for free through the existing +shared `mp8` bind. A NAS is a **distinct storage class** — no durable-id, never in the drive +enroll/eject/decommission/wipe/SMART/watchdog machinery. **Bulk-media class only.** -**Tests.** Sequence + companions: no-lock → nothing called; `migrate` lock → left alone; onboot=0 → -unlocked-but-not-started; delsnapshot only when a snapshot exists; **invariant guard** (live backup → not -cleared; unconfirmable → fail-safe); already-running → not restarted; nil controller → no-op. +### Files changed (commit `63aa63d`, +1351/-5) +- `internal/storage/netmount.go` (NEW, 511) — `NetworkMountSpec`, the locked NFS/SMB option sets, the + `+100000` uid recipe, `.mount`/`.automount` rendering, validation, per-share liveness, and the + `SudoHostOps` `EnsureNetworkMount`/`RemoveNetworkMount`/`ListNetworkMounts` methods. +- `internal/storage/mountunit.go` — drive-machinery guard: `parseFelhomMountUnit` explicitly refuses any + unit carrying the network marker (Scenario D). +- `internal/localapi/netstorage.go` (NEW, 204) — self-scoped `POST /netstorage/add`, `GET /netstorage`, + `POST /netstorage/remove`; role gate; out-of-band 0600 SMB creds file. +- `internal/localapi/server.go` — wire `NetStorage` + `SmbCredsDir` into Options + routes. +- `cmd/felhom-agent/main.go` — `newHostOps` returns `*storage.SudoHostOps`; wire `NetStorage`; v0.50.0. +- `internal/config/config.go` — `privileged.smb_creds_dir` (default `/var/lib/felhom-agent/smb-creds`). +- `configs/felhom-agent.sudoers` — new narrow `FELHOM_NETMOUNT` alias (`.automount` install/enable/ + disable/stop + felhom mount-unit removal); `visudo -cf` clean on the live host. -**Live (B.3 + §D).** Simulated stale state on scratch 9999 → agent restart auto-recovered (unlock + -start); the **invariant guard was proven against a REAL in-flight 9201 backup** (left its lock alone). Then -the **§D supervised reboot** validated the genuine path: a reboot with the EXACT F2-b state (induced -`snapshot-delete` lock + a real dangling `vzdump` snapshot) reproduced the stuck symptom (pve-guests *"CT -is locked (snapshot-delete)"* → *"Starting CT 9201 failed"*), then the agent auto-recovered — `clearing a -stale backup lock` → **`removed dangling vzdump snapshot`** (the real delsnapshot path) → `started CT`. 9201 -running, lock cleared, snapshot deleted, no error. Two live-caught bugs fixed en route: PVE 9.x rejects -`?running=1` (→ `?source=active`); the LXC-start `WARNINGS: 1` nesting advisory false-failed the start -(→ `AllowWarnings`). +## Tests — `go build ./... && go vet ./... && go test ./...` GREEN (build server, Linux) -## F2-a — shared-parent boot script never redeployed (Part C) +Unit tests with **companion red-proofs**: +- **Exact option-set string-asserts** — NFS `vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev` (no `hard`, + no client uid); SMB `vers=3.0,credentials=…,uid=101000,gid=101000,forceuid,forcegid,file_mode=0664, + dir_mode=0775,_netdev`. +- **+100000 companion** — container uid 1000 → the SMB unit renders `uid=101000`; a `+0` impl (`uid=1000`) + FAILS the test (the documented non-writable trap). `dir_mode` must be plain `0775`, never setgid `2775`. +- **Validation matrix** — bad name/traversal/slash/space, bad protocol, server metachar, NFS relative/ + traversal export, uid/gid range, SMB-without-creds, SMB bad share name. +- **Role gate** — under `/mnt/felhom-drives` → user-data; anything else → system (refused). +- **Drive-machinery guard (Scenario D) + companion** — `parseFelhomMountUnit` refuses a network unit even + when contrived with a by-uuid `What=`; the **same content with the drive marker DOES parse** — proving + the guard (not luck) is the discriminator. +- **Unit round-trip + health + isNetworkMounted**; **Ensure/Remove command-sequence** (mkdir → install ×2 + → daemon-reload → enable `--now` the `.automount`, never the `.mount`; remove = stop/disable automount → + stop mount → rm ×2 → daemon-reload); **bad spec → ZERO commands**. +- localapi: add NFS/SMB happy-path; SMB writes a **0600** creds file + no secret in the response; SMB + missing creds → 400; **role-gate refusal → 403, surface never touched**; list; remove (creds cleaned); + not-configured → 503; auth required → 401. -**Diagnosis (C.0, confirmed live).** `/mnt/felhom-drives` was in root's `shared:1` peer group, so each -drive bind appeared twice. Reproduced exactly in a scratch dir at `/mnt`: the **stale** sequence (`bind + -make-shared`, no `make-private`) leaves the self-bind in `shared:1` → sub-bind count 2 (doubling); the -**correct** sequence (`bind → make-private → make-shared`) gives an own group → count 1. The live boot -script was the pre-v0.36.6 body (no `make-private`). Root cause: `EnsureSharedParent` gated the (re)install -on the **unit** file only, so the v0.36.6 *script-only* fix never deployed to a host whose unit was current. +(The two `SudoHostOps` command-sequence tests skip on Windows — the systemd-escaped unit filename embeds a +backslash — and run on the Linux build server, where the full suite is green.) -**Fix (provably safe, boot-time-only).** New `sharedParentInstallStale` compares **both** the script and -unit (missing/differing → reinstall). It rewrites the on-disk script; it does **not** touch the live mount -(the live bind/make-private/make-shared stays guarded on `!isHostMountpoint`), so it can't orphan the -guest's slave bind. The doubling persists live until the next boot, when the corrected script gives the -parent its own peer group. +## Live sim validation (Scenarios A–D) -**Tests.** Stale-script/current-unit → reinstall (the F2-a regression); both-current → no-op; missing → -stale; content guard that the shipped script keeps `make-private`. +**Isolated sim NAS:** a throwaway **privileged LXC (VMID 9300, `nas-sim`, 192.168.0.114)** on felhom-pve +running `nfs-kernel-server` + `samba` with **its own config** — it did NOT modify any production host's +`/etc/exports` or `smb.conf` (the spike's near-miss avoided). Export squashed to `anonuid/anongid=101000`; +SMB `force user/group = nasguest` (uid/gid 101000). Throwaway creds, out-of-band. **Sim destroyed after.** +Endpoints driven exactly as A2 will (the controller's own local-api token + the real agent server pipeline). -**Red-proof (C.1) — PASSED (§D reboot).** `/mnt/felhom-drives` came up as its **own peer group -`shared:94`** (NOT root's `shared:1`) with **exactly 1 host bind per drive** (doubling gone, was 2); the -guest still sees both drives (`bound_under_parent=true`; apps healthy). Consistent across two reboots. -(Separate pre-existing, harmless observation: the guest shows 2 mounts per drive — two parent mounts, same -`/felhom-data` source, correctly `master:`-slaved; not a regression of this host-side fix.) +### A — NFS host-side mount → propagates → container read+write (the +100000 recipe) ✅ +- `POST /netstorage/add` (nfs) → host automount installed; first access mounted `nfs4` with effective opts + `vers=4.1,soft,timeo=50,retrans=2,noatime` (exact recipe). +- **Guest 9201 saw the `nfs4` mount with NO restart** (shared-bind propagation); guest view ownership + `1000:1000` (the +100000 idmap). +- A **uid-1000 Docker container** READ `readme.txt` AND WROTE `ctest.txt` (`WRITE_OK`); the new file landed + on the NAS as **`nasguest:nasguest` (101000:101000)**. -## Green gate +### B — SMB fallback read+write ✅ +- `POST /netstorage/add` (smb) → creds staged **0600** (`felhom-agent`-owned, out-of-band); mounted `cifs` + with `vers=3.0,uid=101000,forceuid,gid=101000,forcegid,file_mode=0664,dir_mode=0775` (exact recipe). +- uid-1000 container READ + WROTE (`SMB_WRITE_OK`); landed on the NAS as `nasguest` (101000). No secret in + any response, log, or committed file. -`go build ./... && go vet ./... && go test ./...` — all green (full suite, 0 failures). Version -`0.48.0 → 0.49.0`. +### C — failure isolation (the risk) ✅ +- NAS black-holed from felhom-pve (`iptables DROP`, both directions — a dead-NAS black hole). +- **Clean fail-soft:** uncached `stat` → error **`No such device` in 15s**; `cat` data read → error in + **16s** (≈ the spike's ~16s). It **errors, does not hang**. +- **`df /` did NOT hang** (box-wide health intact); **guest 9201 responsive + `felhom-controller` + healthy** throughout — blast radius contained to the process touching the mount. +- **Per-share liveness reported `unreachable`** for the affected shares only (never box-wide); the endpoint + TCP-probe never touched the wedged mount. +- **Automatic recovery** on NAS return: a fresh read succeeded with **no remount**; liveness returned to + `ok`/`idle`. +- *Note:* a first C run hit a 30s ceiling — it coincided with a concurrent scheduled `vzdump` of 9201 + (load 4.7); the clean re-run after the backup matched the spike (~15–16s). The box-wide isolation held + **even under that backup load.** -Secrets-safe: no tokens/keys logged or written. Leaf fingerprints are not secret; the operator email is not -a secret. +### D — the drive machinery ignores the NAS mount ✅ +- `GET /disks` listed only real drives (`felhom-pbs`, `felhom-usb`, `local`, `felhom-flash`, `local-lvm`) + — **never `media`/`vids`**; the NAS shares carry **no durable-id**. +- The 20 s drive reconcile/`ReassertEnrolledMounts` tick re-bound only the enrolled drives + (felhom-flash/felhom-usb) and **never touched** the NAS mounts; they remained present and unmodified. + +### Lifecycle + teardown +- `POST /netstorage/remove` (both) → units gone, creds file gone, mounts gone, list empty. +- Sim LXC 9300 destroyed; all iptables DROP rules removed; helper scripts removed; `/mnt/felhom-drives` + back to only `felhom-flash` + `felhom-usb`; agent **v0.50.0 active**; guest 9201 + controller healthy. + +## Deploy + clean restart +sudoers (`FELHOM_NETMOUNT`, `visudo -cf` clean) and the v0.50.0 binary installed on felhom-pve (prior +binary backed up `.bak-0.49.0`). Restart: `capabilities self-check ok=46 total=46 degraded=0`, +`local-api server listening 192.168.0.162:8443`, no errors — `ReassertEnrolledMounts` ran without touching +any network mount. + +## Operational note (pre-existing, not this feature) +During validation `df /` on felhom-pve showed the **root fs at ~100%** (vzdump output under `/var/lib/vz/ +dump` on `pve-root`). Unrelated to network storage (the NAS data lives on the sim, the agent state is +tiny) — flagging it as a host backup-retention/disk-pressure item for follow-up. + +## STOP — not yet built (per the task) +- **A2 (controller "network storage" registry kind + UI + per-share health surface)** — NOT built; A2 will + drive these A1 endpoints. +- **B (restic-over-SFTP NAS backup target, class 2)** — NOT built (separate task). +- **No real customer media app wired to a NAS path** — awaiting A2 + a real app. +- Real-Synology/QNAP (virtual-dsm) GA-fidelity confirmation — out of scope (later pass). + +SMB/NFS test credentials were throwaway and out-of-band only. No secrets in any committed file.