docs: REPORT for agent v0.51.0 (vzdump retention, live-validated keep-last=3)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
This commit is contained in:
@@ -1,128 +1,66 @@
|
||||
# REPORT — agent v0.50.0: NAS network storage Part A1 (NFS/SMB automount foundation)
|
||||
# REPORT — agent v0.51.0: local vzdump retention default (`--prune-backups keep-last=3`)
|
||||
|
||||
**Date:** 2026-06-30
|
||||
**Task:** `TASK.md` — NAS network storage Part A1 (agent foundation: automount + bind + liveness).
|
||||
**Class:** Risky / supervised (creates real network mounts on the Proxmox host). Implemented on `main`,
|
||||
sim-validated live, sim torn down.
|
||||
|
||||
## Baseline (confirmed live)
|
||||
**Task:** `TASK.md` — local vzdump retention default. The preventive counterpart to the hub's host_disk
|
||||
(v0.23.0) + storage_fill (v0.25.0) detectors — completing the disk-pressure arc.
|
||||
|
||||
## Baseline
|
||||
| | value |
|
||||
|---|---|
|
||||
| Agent `main` before | v0.49.0 (commit `7aeb7ca`), live on felhom-pve |
|
||||
| Agent after | **v0.50.0** (commit `63aa63d`) |
|
||||
| Deployed binary sha256 (felhom-pve) | `6ddb6ff2efe6b10377e00ff7612f738ff437dfbcbcc35e3fc566e40d139aedf1` |
|
||||
| Host client tooling | `mount.nfs` (nfs-common) + `mount.cifs` (cifs-utils) present |
|
||||
| felhom-agent | v0.50.0 → **v0.51.0** |
|
||||
| Commits | `06e0bc9` (agent) · felhom.eu `fdcbc04` (host-install seed) |
|
||||
| Deployed | felhom-agent 0.51.0 on felhom-pve (prior binary backed up `.bak-0.50.0`); clean restart 46/46 caps |
|
||||
| VzdumpOptions prune passthrough | **added** — `VzdumpOptions.PruneBackups` → PVE `--prune-backups` on the vzdump POST |
|
||||
|
||||
## What was built
|
||||
## What changed
|
||||
The agent's periodic local whole-guest vzdump now prunes its own old archives, so a box can't refill its
|
||||
own root via its own backups (the felhom-pve incident's root cause — that vzdump carried no retention).
|
||||
- **`internal/proxmox/mutate.go`:** `VzdumpOptions.PruneBackups` → `--prune-backups` (vmid+storage scoped,
|
||||
so PVE prunes only THIS guest's archives on THIS storage).
|
||||
- **`internal/backup/runner.go`:** `NewBackupRunner` gains `retention`; `localPruneSpec` applies it ONLY
|
||||
when the target is a **non-PBS** storage (resolved via `ListStorage`). **Fail-safe:** an unconfirmable
|
||||
target type (lookup error / not found) SKIPS pruning rather than risk pruning PBS. Only the periodic
|
||||
local-API runner sets retention; restore-test/selftest runners pass "".
|
||||
- **`internal/config/config.go`:** `backup.local_backup_retention` with `KeepLast()` **clamped ≥1**
|
||||
(0/unset/negative → default 3) + `PruneBackupsSpec()` → `keep-last=N`. Wired into the local-API runner.
|
||||
- **Seeding:** `felhom.eu scripts/felhom-host-install.sh` seeds `local_backup_retention: 3`; the code
|
||||
default also protects any box where it's unset (KeepLast → 3) from day 0.
|
||||
- F2-b stale-vzdump-lock recovery untouched.
|
||||
|
||||
The agent foundation of `SPIKE-nas-storage-2026-06-29.md` (verdict READY) — a customer NAS can serve
|
||||
**bulk media** to a media app. The agent mounts a NAS share **host-side** under `/mnt/felhom-drives/<name>`
|
||||
via a systemd `.automount` (+ `.mount`) pair; it propagates into guest 9201 for free through the existing
|
||||
shared `mp8` bind. A NAS is a **distinct storage class** — no durable-id, never in the drive
|
||||
enroll/eject/decommission/wipe/SMART/watchdog machinery. **Bulk-media class only.**
|
||||
## Tests — `go build ./... && go vet ./... && go test ./...` GREEN (Linux build server)
|
||||
- **flag-present** — a local-target backup carries `--prune-backups keep-last=3`; **companion:** a
|
||||
no-retention runner emits NO prune option (dumps would accumulate).
|
||||
- **PBS never pruned** (scope) — a PBS target gets no prune; **companion:** the same retention on a local
|
||||
target IS applied (the gate keys on storage type, not luck). Plus fail-safe-on-unknown-target.
|
||||
- **clamp** — `LocalBackupRetention` 0/negative/unset → `KeepLast()` ≥1 (default 3); a no-clamp impl
|
||||
emitting `keep-last=0` would prune the fresh backup → FAILS the ≥1 assertion.
|
||||
|
||||
### Files changed (commit `63aa63d`, +1351/-5)
|
||||
- `internal/storage/netmount.go` (NEW, 511) — `NetworkMountSpec`, the locked NFS/SMB option sets, the
|
||||
`+100000` uid recipe, `.mount`/`.automount` rendering, validation, per-share liveness, and the
|
||||
`SudoHostOps` `EnsureNetworkMount`/`RemoveNetworkMount`/`ListNetworkMounts` methods.
|
||||
- `internal/storage/mountunit.go` — drive-machinery guard: `parseFelhomMountUnit` explicitly refuses any
|
||||
unit carrying the network marker (Scenario D).
|
||||
- `internal/localapi/netstorage.go` (NEW, 204) — self-scoped `POST /netstorage/add`, `GET /netstorage`,
|
||||
`POST /netstorage/remove`; role gate; out-of-band 0600 SMB creds file.
|
||||
- `internal/localapi/server.go` — wire `NetStorage` + `SmbCredsDir` into Options + routes.
|
||||
- `cmd/felhom-agent/main.go` — `newHostOps` returns `*storage.SudoHostOps`; wire `NetStorage`; v0.50.0.
|
||||
- `internal/config/config.go` — `privileged.smb_creds_dir` (default `/var/lib/felhom-agent/smb-creds`).
|
||||
- `configs/felhom-agent.sudoers` — new narrow `FELHOM_NETMOUNT` alias (`.automount` install/enable/
|
||||
disable/stop + felhom mount-unit removal); `visudo -cf` clean on the live host.
|
||||
## Live validation (felhom-pve, real backups — demo dumps expendable)
|
||||
Triggered the agent's local whole-guest backup of guest 9201 via the local-API `POST /backup` (the exact
|
||||
path the controller's periodic backup uses; the retention-enabled runner). Pre-state: local held 2 archives
|
||||
for 9201 (the host_disk task had already pruned the original ~13 to 2), PBS held 8 snapshots, root 24% used.
|
||||
|
||||
## Tests — `go build ./... && go vet ./... && go test ./...` GREEN (build server, Linux)
|
||||
- Backup #1 → local count **2 → 3** (no deletion yet; 3 ≤ keep-last=3 — the prune flag is exercised but
|
||||
retains all 3).
|
||||
- Each subsequent backup → **prune fired**, local count held at **3** while the retention window slid
|
||||
(the oldest rotates out as a new one lands):
|
||||
- after #1: `…10_07_19`, `…11_22_20`, `…19_45_54`
|
||||
- after #2: `…11_22_20`, `…19_45_54`, `…19_51_07` (`…10_07_19` pruned)
|
||||
- after a further: `…19_45_54`, `…19_51_07`, `…19_57_03` (`…11_22_20` pruned)
|
||||
Count never exceeded 3 despite repeated backups (the old no-retention behaviour grew to ~13/18).
|
||||
- **The real PVE command carried the flag** (from the vzdump task log):
|
||||
`vzdump 9201 --mode snapshot --storage local --compress zstd --prune-backups 'keep-last=3' --notes-template 'felhom local-api'`
|
||||
- **PBS untouched:** felhom-pbs still **8** snapshots for 9201 (the per-run flag never touched the offsite
|
||||
repo). Root stayed bounded (24% → 31%, not growing unbounded).
|
||||
|
||||
Unit tests with **companion red-proofs**:
|
||||
- **Exact option-set string-asserts** — NFS `vers=4.1,soft,timeo=50,retrans=2,noatime,_netdev` (no `hard`,
|
||||
no client uid); SMB `vers=3.0,credentials=…,uid=101000,gid=101000,forceuid,forcegid,file_mode=0664,
|
||||
dir_mode=0775,_netdev`.
|
||||
- **+100000 companion** — container uid 1000 → the SMB unit renders `uid=101000`; a `+0` impl (`uid=1000`)
|
||||
FAILS the test (the documented non-writable trap). `dir_mode` must be plain `0775`, never setgid `2775`.
|
||||
- **Validation matrix** — bad name/traversal/slash/space, bad protocol, server metachar, NFS relative/
|
||||
traversal export, uid/gid range, SMB-without-creds, SMB bad share name.
|
||||
- **Role gate** — under `/mnt/felhom-drives` → user-data; anything else → system (refused).
|
||||
- **Drive-machinery guard (Scenario D) + companion** — `parseFelhomMountUnit` refuses a network unit even
|
||||
when contrived with a by-uuid `What=`; the **same content with the drive marker DOES parse** — proving
|
||||
the guard (not luck) is the discriminator.
|
||||
- **Unit round-trip + health + isNetworkMounted**; **Ensure/Remove command-sequence** (mkdir → install ×2
|
||||
→ daemon-reload → enable `--now` the `.automount`, never the `.mount`; remove = stop/disable automount →
|
||||
stop mount → rm ×2 → daemon-reload); **bad spec → ZERO commands**.
|
||||
- localapi: add NFS/SMB happy-path; SMB writes a **0600** creds file + no secret in the response; SMB
|
||||
missing creds → 400; **role-gate refusal → 403, surface never touched**; list; remove (creds cleaned);
|
||||
not-configured → 503; auth required → 401.
|
||||
This demonstrates the retention working on real data: repeated local backups stay bounded at keep-last=3
|
||||
(oldest rotates out) instead of accumulating, and PBS is never pruned.
|
||||
|
||||
(The two `SudoHostOps` command-sequence tests skip on Windows — the systemd-escaped unit filename embeds a
|
||||
backslash — and run on the Linux build server, where the full suite is green.)
|
||||
## Observations
|
||||
- The disk-pressure arc is now complete: **detectors** (hub host_disk v0.23.0 + storage_fill v0.25.0) +
|
||||
**preventive** (this agent-side local-vzdump retention). A box can no longer silently refill its own root
|
||||
via its own backups, and if any storage still fills, the operator is paged.
|
||||
- Remaining disk follow-ons (not blockers): thin-pool *metadata* exhaustion alerting and per-storage-type
|
||||
thresholds (both noted in the storage_fill report).
|
||||
|
||||
## Live sim validation (Scenarios A–D)
|
||||
|
||||
**Isolated sim NAS:** a throwaway **privileged LXC (VMID 9300, `nas-sim`, 192.168.0.114)** on felhom-pve
|
||||
running `nfs-kernel-server` + `samba` with **its own config** — it did NOT modify any production host's
|
||||
`/etc/exports` or `smb.conf` (the spike's near-miss avoided). Export squashed to `anonuid/anongid=101000`;
|
||||
SMB `force user/group = nasguest` (uid/gid 101000). Throwaway creds, out-of-band. **Sim destroyed after.**
|
||||
Endpoints driven exactly as A2 will (the controller's own local-api token + the real agent server pipeline).
|
||||
|
||||
### A — NFS host-side mount → propagates → container read+write (the +100000 recipe) ✅
|
||||
- `POST /netstorage/add` (nfs) → host automount installed; first access mounted `nfs4` with effective opts
|
||||
`vers=4.1,soft,timeo=50,retrans=2,noatime` (exact recipe).
|
||||
- **Guest 9201 saw the `nfs4` mount with NO restart** (shared-bind propagation); guest view ownership
|
||||
`1000:1000` (the +100000 idmap).
|
||||
- A **uid-1000 Docker container** READ `readme.txt` AND WROTE `ctest.txt` (`WRITE_OK`); the new file landed
|
||||
on the NAS as **`nasguest:nasguest` (101000:101000)**.
|
||||
|
||||
### B — SMB fallback read+write ✅
|
||||
- `POST /netstorage/add` (smb) → creds staged **0600** (`felhom-agent`-owned, out-of-band); mounted `cifs`
|
||||
with `vers=3.0,uid=101000,forceuid,gid=101000,forcegid,file_mode=0664,dir_mode=0775` (exact recipe).
|
||||
- uid-1000 container READ + WROTE (`SMB_WRITE_OK`); landed on the NAS as `nasguest` (101000). No secret in
|
||||
any response, log, or committed file.
|
||||
|
||||
### C — failure isolation (the risk) ✅
|
||||
- NAS black-holed from felhom-pve (`iptables DROP`, both directions — a dead-NAS black hole).
|
||||
- **Clean fail-soft:** uncached `stat` → error **`No such device` in 15s**; `cat` data read → error in
|
||||
**16s** (≈ the spike's ~16s). It **errors, does not hang**.
|
||||
- **`df /` did NOT hang** (box-wide health intact); **guest 9201 responsive + `felhom-controller`
|
||||
healthy** throughout — blast radius contained to the process touching the mount.
|
||||
- **Per-share liveness reported `unreachable`** for the affected shares only (never box-wide); the endpoint
|
||||
TCP-probe never touched the wedged mount.
|
||||
- **Automatic recovery** on NAS return: a fresh read succeeded with **no remount**; liveness returned to
|
||||
`ok`/`idle`.
|
||||
- *Note:* a first C run hit a 30s ceiling — it coincided with a concurrent scheduled `vzdump` of 9201
|
||||
(load 4.7); the clean re-run after the backup matched the spike (~15–16s). The box-wide isolation held
|
||||
**even under that backup load.**
|
||||
|
||||
### D — the drive machinery ignores the NAS mount ✅
|
||||
- `GET /disks` listed only real drives (`felhom-pbs`, `felhom-usb`, `local`, `felhom-flash`, `local-lvm`)
|
||||
— **never `media`/`vids`**; the NAS shares carry **no durable-id**.
|
||||
- The 20 s drive reconcile/`ReassertEnrolledMounts` tick re-bound only the enrolled drives
|
||||
(felhom-flash/felhom-usb) and **never touched** the NAS mounts; they remained present and unmodified.
|
||||
|
||||
### Lifecycle + teardown
|
||||
- `POST /netstorage/remove` (both) → units gone, creds file gone, mounts gone, list empty.
|
||||
- Sim LXC 9300 destroyed; all iptables DROP rules removed; helper scripts removed; `/mnt/felhom-drives`
|
||||
back to only `felhom-flash` + `felhom-usb`; agent **v0.50.0 active**; guest 9201 + controller healthy.
|
||||
|
||||
## Deploy + clean restart
|
||||
sudoers (`FELHOM_NETMOUNT`, `visudo -cf` clean) and the v0.50.0 binary installed on felhom-pve (prior
|
||||
binary backed up `.bak-0.49.0`). Restart: `capabilities self-check ok=46 total=46 degraded=0`,
|
||||
`local-api server listening 192.168.0.162:8443`, no errors — `ReassertEnrolledMounts` ran without touching
|
||||
any network mount.
|
||||
|
||||
## Operational note (pre-existing, not this feature)
|
||||
During validation `df /` on felhom-pve showed the **root fs at ~100%** (vzdump output under `/var/lib/vz/
|
||||
dump` on `pve-root`). Unrelated to network storage (the NAS data lives on the sim, the agent state is
|
||||
tiny) — flagging it as a host backup-retention/disk-pressure item for follow-up.
|
||||
|
||||
## STOP — not yet built (per the task)
|
||||
- **A2 (controller "network storage" registry kind + UI + per-share health surface)** — NOT built; A2 will
|
||||
drive these A1 endpoints.
|
||||
- **B (restic-over-SFTP NAS backup target, class 2)** — NOT built (separate task).
|
||||
- **No real customer media app wired to a NAS path** — awaiting A2 + a real app.
|
||||
- Real-Synology/QNAP (virtual-dsm) GA-fidelity confirmation — out of scope (later pass).
|
||||
|
||||
SMB/NFS test credentials were throwaway and out-of-band only. No secrets in any committed file.
|
||||
No secrets. Demo backup archives are expendable; PBS offsite retention is a separate lifecycle (untouched).
|
||||
|
||||
Reference in New Issue
Block a user