From 1502ca644a0f6bb4daf013abc56bd000db5f4657 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 30 Jun 2026 20:03:42 +0200 Subject: [PATCH] docs: REPORT for agent v0.51.0 (vzdump retention, live-validated keep-last=3) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs --- REPORT.md | 170 +++++++++++++++++------------------------------------- 1 file changed, 54 insertions(+), 116 deletions(-) diff --git a/REPORT.md b/REPORT.md index a820654..3fc3364 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,128 +1,66 @@ -# REPORT — agent v0.50.0: NAS network storage Part A1 (NFS/SMB automount foundation) +# REPORT — agent v0.51.0: local vzdump retention default (`--prune-backups keep-last=3`) **Date:** 2026-06-30 -**Task:** `TASK.md` — NAS network storage Part A1 (agent foundation: automount + bind + liveness). -**Class:** Risky / supervised (creates real network mounts on the Proxmox host). Implemented on `main`, -sim-validated live, sim torn down. - -## Baseline (confirmed live) +**Task:** `TASK.md` — local vzdump retention default. The preventive counterpart to the hub's host_disk +(v0.23.0) + storage_fill (v0.25.0) detectors — completing the disk-pressure arc. +## Baseline | | value | |---|---| -| Agent `main` before | v0.49.0 (commit `7aeb7ca`), live on felhom-pve | -| Agent after | **v0.50.0** (commit `63aa63d`) | -| Deployed binary sha256 (felhom-pve) | `6ddb6ff2efe6b10377e00ff7612f738ff437dfbcbcc35e3fc566e40d139aedf1` | -| Host client tooling | `mount.nfs` (nfs-common) + `mount.cifs` (cifs-utils) present | +| felhom-agent | v0.50.0 → **v0.51.0** | +| Commits | `06e0bc9` (agent) · felhom.eu `fdcbc04` (host-install seed) | +| Deployed | felhom-agent 0.51.0 on felhom-pve (prior binary backed up `.bak-0.50.0`); clean restart 46/46 caps | +| VzdumpOptions prune passthrough | **added** — `VzdumpOptions.PruneBackups` → PVE `--prune-backups` on the vzdump POST | -## What was built +## What changed +The agent's periodic local whole-guest vzdump now prunes its own old archives, so a box can't refill its +own root via its own backups (the felhom-pve incident's root cause — that vzdump carried no retention). +- **`internal/proxmox/mutate.go`:** `VzdumpOptions.PruneBackups` → `--prune-backups` (vmid+storage scoped, + so PVE prunes only THIS guest's archives on THIS storage). +- **`internal/backup/runner.go`:** `NewBackupRunner` gains `retention`; `localPruneSpec` applies it ONLY + when the target is a **non-PBS** storage (resolved via `ListStorage`). **Fail-safe:** an unconfirmable + target type (lookup error / not found) SKIPS pruning rather than risk pruning PBS. Only the periodic + local-API runner sets retention; restore-test/selftest runners pass "". +- **`internal/config/config.go`:** `backup.local_backup_retention` with `KeepLast()` **clamped ≥1** + (0/unset/negative → default 3) + `PruneBackupsSpec()` → `keep-last=N`. Wired into the local-API runner. +- **Seeding:** `felhom.eu scripts/felhom-host-install.sh` seeds `local_backup_retention: 3`; the code + default also protects any box where it's unset (KeepLast → 3) from day 0. +- F2-b stale-vzdump-lock recovery untouched. -The agent foundation of `SPIKE-nas-storage-2026-06-29.md` (verdict READY) — a customer NAS can serve -**bulk media** to a media app. The agent mounts a NAS share **host-side** under `/mnt/felhom-drives/` -via a systemd `.automount` (+ `.mount`) pair; it propagates into guest 9201 for free through the existing -shared `mp8` bind. A NAS is a **distinct storage class** — no durable-id, never in the drive -enroll/eject/decommission/wipe/SMART/watchdog machinery. **Bulk-media class only.** +## Tests — `go build ./... && go vet ./... && go test ./...` GREEN (Linux build server) +- **flag-present** — a local-target backup carries `--prune-backups keep-last=3`; **companion:** a + no-retention runner emits NO prune option (dumps would accumulate). +- **PBS never pruned** (scope) — a PBS target gets no prune; **companion:** the same retention on a local + target IS applied (the gate keys on storage type, not luck). Plus fail-safe-on-unknown-target. +- **clamp** — `LocalBackupRetention` 0/negative/unset → `KeepLast()` ≥1 (default 3); a no-clamp impl + emitting `keep-last=0` would prune the fresh backup → FAILS the ≥1 assertion. -### Files changed (commit `63aa63d`, +1351/-5) -- `internal/storage/netmount.go` (NEW, 511) — `NetworkMountSpec`, the locked NFS/SMB option sets, the - `+100000` uid recipe, `.mount`/`.automount` rendering, validation, per-share liveness, and the - `SudoHostOps` `EnsureNetworkMount`/`RemoveNetworkMount`/`ListNetworkMounts` methods. -- `internal/storage/mountunit.go` — drive-machinery guard: `parseFelhomMountUnit` explicitly refuses any - unit carrying the network marker (Scenario D). -- `internal/localapi/netstorage.go` (NEW, 204) — self-scoped `POST /netstorage/add`, `GET /netstorage`, - `POST /netstorage/remove`; role gate; out-of-band 0600 SMB creds file. -- `internal/localapi/server.go` — wire `NetStorage` + `SmbCredsDir` into Options + routes. -- `cmd/felhom-agent/main.go` — `newHostOps` returns `*storage.SudoHostOps`; wire `NetStorage`; v0.50.0. -- `internal/config/config.go` — `privileged.smb_creds_dir` (default `/var/lib/felhom-agent/smb-creds`). -- `configs/felhom-agent.sudoers` — new narrow `FELHOM_NETMOUNT` alias (`.automount` install/enable/ - disable/stop + felhom mount-unit removal); `visudo -cf` clean on the live host. +## Live validation (felhom-pve, real backups — demo dumps expendable) +Triggered the agent's local whole-guest backup of guest 9201 via the local-API `POST /backup` (the exact +path the controller's periodic backup uses; the retention-enabled runner). Pre-state: local held 2 archives +for 9201 (the host_disk task had already pruned the original ~13 to 2), PBS held 8 snapshots, root 24% used. -## Tests — `go build ./... && go vet ./... && go test ./...` GREEN (build server, Linux) +- Backup #1 → local count **2 → 3** (no deletion yet; 3 ≤ keep-last=3 — the prune flag is exercised but + retains all 3). +- Each subsequent backup → **prune fired**, local count held at **3** while the retention window slid + (the oldest rotates out as a new one lands): + - after #1: `…10_07_19`, `…11_22_20`, `…19_45_54` + - after #2: `…11_22_20`, `…19_45_54`, `…19_51_07` (`…10_07_19` pruned) + - after a further: `…19_45_54`, `…19_51_07`, `…19_57_03` (`…11_22_20` pruned) + Count never exceeded 3 despite repeated backups (the old no-retention behaviour grew to ~13/18). +- **The real PVE command carried the flag** (from the vzdump task log): + `vzdump 9201 --mode snapshot --storage local --compress zstd --prune-backups 'keep-last=3' --notes-template 'felhom local-api'` +- **PBS untouched:** felhom-pbs still **8** snapshots for 9201 (the per-run flag never touched the offsite + repo). Root stayed bounded (24% → 31%, not growing unbounded). -Unit tests with **companion red-proofs**: -- **Exact option-set string-asserts** — NFS `vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev` (no `hard`, - no client uid); SMB `vers=3.0,credentials=…,uid=101000,gid=101000,forceuid,forcegid,file_mode=0664, - dir_mode=0775,_netdev`. -- **+100000 companion** — container uid 1000 → the SMB unit renders `uid=101000`; a `+0` impl (`uid=1000`) - FAILS the test (the documented non-writable trap). `dir_mode` must be plain `0775`, never setgid `2775`. -- **Validation matrix** — bad name/traversal/slash/space, bad protocol, server metachar, NFS relative/ - traversal export, uid/gid range, SMB-without-creds, SMB bad share name. -- **Role gate** — under `/mnt/felhom-drives` → user-data; anything else → system (refused). -- **Drive-machinery guard (Scenario D) + companion** — `parseFelhomMountUnit` refuses a network unit even - when contrived with a by-uuid `What=`; the **same content with the drive marker DOES parse** — proving - the guard (not luck) is the discriminator. -- **Unit round-trip + health + isNetworkMounted**; **Ensure/Remove command-sequence** (mkdir → install ×2 - → daemon-reload → enable `--now` the `.automount`, never the `.mount`; remove = stop/disable automount → - stop mount → rm ×2 → daemon-reload); **bad spec → ZERO commands**. -- localapi: add NFS/SMB happy-path; SMB writes a **0600** creds file + no secret in the response; SMB - missing creds → 400; **role-gate refusal → 403, surface never touched**; list; remove (creds cleaned); - not-configured → 503; auth required → 401. +This demonstrates the retention working on real data: repeated local backups stay bounded at keep-last=3 +(oldest rotates out) instead of accumulating, and PBS is never pruned. -(The two `SudoHostOps` command-sequence tests skip on Windows — the systemd-escaped unit filename embeds a -backslash — and run on the Linux build server, where the full suite is green.) +## Observations +- The disk-pressure arc is now complete: **detectors** (hub host_disk v0.23.0 + storage_fill v0.25.0) + + **preventive** (this agent-side local-vzdump retention). A box can no longer silently refill its own root + via its own backups, and if any storage still fills, the operator is paged. +- Remaining disk follow-ons (not blockers): thin-pool *metadata* exhaustion alerting and per-storage-type + thresholds (both noted in the storage_fill report). -## Live sim validation (Scenarios A–D) - -**Isolated sim NAS:** a throwaway **privileged LXC (VMID 9300, `nas-sim`, 192.168.0.114)** on felhom-pve -running `nfs-kernel-server` + `samba` with **its own config** — it did NOT modify any production host's -`/etc/exports` or `smb.conf` (the spike's near-miss avoided). Export squashed to `anonuid/anongid=101000`; -SMB `force user/group = nasguest` (uid/gid 101000). Throwaway creds, out-of-band. **Sim destroyed after.** -Endpoints driven exactly as A2 will (the controller's own local-api token + the real agent server pipeline). - -### A — NFS host-side mount → propagates → container read+write (the +100000 recipe) ✅ -- `POST /netstorage/add` (nfs) → host automount installed; first access mounted `nfs4` with effective opts - `vers=4.1,soft,timeo=50,retrans=2,noatime` (exact recipe). -- **Guest 9201 saw the `nfs4` mount with NO restart** (shared-bind propagation); guest view ownership - `1000:1000` (the +100000 idmap). -- A **uid-1000 Docker container** READ `readme.txt` AND WROTE `ctest.txt` (`WRITE_OK`); the new file landed - on the NAS as **`nasguest:nasguest` (101000:101000)**. - -### B — SMB fallback read+write ✅ -- `POST /netstorage/add` (smb) → creds staged **0600** (`felhom-agent`-owned, out-of-band); mounted `cifs` - with `vers=3.0,uid=101000,forceuid,gid=101000,forcegid,file_mode=0664,dir_mode=0775` (exact recipe). -- uid-1000 container READ + WROTE (`SMB_WRITE_OK`); landed on the NAS as `nasguest` (101000). No secret in - any response, log, or committed file. - -### C — failure isolation (the risk) ✅ -- NAS black-holed from felhom-pve (`iptables DROP`, both directions — a dead-NAS black hole). -- **Clean fail-soft:** uncached `stat` → error **`No such device` in 15s**; `cat` data read → error in - **16s** (≈ the spike's ~16s). It **errors, does not hang**. -- **`df /` did NOT hang** (box-wide health intact); **guest 9201 responsive + `felhom-controller` - healthy** throughout — blast radius contained to the process touching the mount. -- **Per-share liveness reported `unreachable`** for the affected shares only (never box-wide); the endpoint - TCP-probe never touched the wedged mount. -- **Automatic recovery** on NAS return: a fresh read succeeded with **no remount**; liveness returned to - `ok`/`idle`. -- *Note:* a first C run hit a 30s ceiling — it coincided with a concurrent scheduled `vzdump` of 9201 - (load 4.7); the clean re-run after the backup matched the spike (~15–16s). The box-wide isolation held - **even under that backup load.** - -### D — the drive machinery ignores the NAS mount ✅ -- `GET /disks` listed only real drives (`felhom-pbs`, `felhom-usb`, `local`, `felhom-flash`, `local-lvm`) - — **never `media`/`vids`**; the NAS shares carry **no durable-id**. -- The 20 s drive reconcile/`ReassertEnrolledMounts` tick re-bound only the enrolled drives - (felhom-flash/felhom-usb) and **never touched** the NAS mounts; they remained present and unmodified. - -### Lifecycle + teardown -- `POST /netstorage/remove` (both) → units gone, creds file gone, mounts gone, list empty. -- Sim LXC 9300 destroyed; all iptables DROP rules removed; helper scripts removed; `/mnt/felhom-drives` - back to only `felhom-flash` + `felhom-usb`; agent **v0.50.0 active**; guest 9201 + controller healthy. - -## Deploy + clean restart -sudoers (`FELHOM_NETMOUNT`, `visudo -cf` clean) and the v0.50.0 binary installed on felhom-pve (prior -binary backed up `.bak-0.49.0`). Restart: `capabilities self-check ok=46 total=46 degraded=0`, -`local-api server listening 192.168.0.162:8443`, no errors — `ReassertEnrolledMounts` ran without touching -any network mount. - -## Operational note (pre-existing, not this feature) -During validation `df /` on felhom-pve showed the **root fs at ~100%** (vzdump output under `/var/lib/vz/ -dump` on `pve-root`). Unrelated to network storage (the NAS data lives on the sim, the agent state is -tiny) — flagging it as a host backup-retention/disk-pressure item for follow-up. - -## STOP — not yet built (per the task) -- **A2 (controller "network storage" registry kind + UI + per-share health surface)** — NOT built; A2 will - drive these A1 endpoints. -- **B (restic-over-SFTP NAS backup target, class 2)** — NOT built (separate task). -- **No real customer media app wired to a NAS path** — awaiting A2 + a real app. -- Real-Synology/QNAP (virtual-dsm) GA-fidelity confirmation — out of scope (later pass). - -SMB/NFS test credentials were throwaway and out-of-band only. No secrets in any committed file. +No secrets. Demo backup archives are expendable; PBS offsite retention is a separate lifecycle (untouched).