diff --git a/REPORT.md b/REPORT.md index c44fa31..6a7e0fc 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,53 +1,64 @@ -# REPORT — agent v0.83.0: observability pass (capture ring + /debug/logs + heartbeat log-pull + sweep) +# REPORT — agent v0.84.0: ReassertNetworkMounts — NAS automount survives guest reboots (RCA fix 1) -**Date:** 2026-07-11 · **Commit:** `cb692f8` · **Deployed:** felhom-pve (backup `.bak-0.82.0`) · -**NOT published** (Peti stays 0.81.0; reaches him with the next publish train) +**Date:** 2026-07-11 · **Commit:** `474b858` · **Deployed:** felhom-pve (backup `.bak-0.83.0`) · +**NOT published** (Peti stays 0.81.0; publish-train queue now carries 0.84.0) +**Source:** `felhom.eu/documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md` (fix 1); pairs with +controller v0.117.0 (fixes 2+4). + +## Baseline (fresh-verified at session start) + +Agent 0.83.0 live on felhom-pve; controller 0.116.1 on 9201; nas-media active-mounted; CWA healthy. ## What shipped -- **Capture layer** (`internal/log`): `applog.New` → `(logger, *Ring)`. slog fan-out: stderr/journald - at the configured level (unchanged), a 1000-entry ring FIXED at LevelDebug. `Ring.Lines(maxBytes)` - renders newest-kept plain text (the heartbeat tail source). -- **`GET /debug/logs`** on the local API (token-authed, self-scoped like every sibling; `?raw=1` - plain text; 503 unwired) + request-level DEBUG middleware (method/path/status/duration, never bodies). -- **Heartbeat log-pull**: `ControlEnvelope.log_tail_requested` (additive) → the NEXT heartbeat - carries `log_tail {collected_at, lines[]}` (128 KB cap, newest kept). Consume-once mirrored from - report logtail.go: local pending drains onto the carrying push; a failed push is re-armed by the - next envelope (retry proven in tests). Serving a pull logs `operator log pull served` (INFO). -- **Gap-fill sweep**: netverify (job start / trigger outcome / **/proc/mounts verdict** / journal - byte-count / classification code / rollback outcome / durations), netstorage add (pre-probe PASS - verdict, creds staged/removed — path only), netmount (unit install/enable/remove-step results), - signedjobs (jobs fetched ids+duration; op received class/host/expiry — never signatures), - selfupdate (invariants passed, download sha-match+duration), disks (assign/eject/decommission - outcome INFO), ReassertGuestBinds pass summary, controller-swap (pre-pull verify, negative health - verdict), desired syncer + hub loop per-exchange DEBUG. +- **`internal/storage/netreassert.go`** `ReassertNetworkAutomounts` — the §8 decision table: real + nfs/nfs4/cifs mounted → skip (fresh namespaces inherit real mounts); `autofs` trigger → + `systemctl stop -- .automount` + `enable --now -- .automount` (**existing + FELHOM_NETMOUNT verbs — the §3 sudoers fallback was NOT needed**, see equivalence below); + neither → skip (orphan states owned by add/remove). Idempotent; per-share errors never stop the + pass. `networkUnitEntries()` factored out of `ListNetworkMounts` (behavior unchanged). +- **`internal/localapi/netreassert.go`** `Server.ReassertNetworkMounts` — daemon leg: host-global + pass once, then per-RUNNING-guest `GuestSeesMount` verify (WARN-only). Type-asserted capability + (lean `NetworkStorageOps` + fakes untouched). Wired at startup after `ReassertGuestBinds`; + deliberately NOT in the 20 s ticker (idle trigger = healthy, never churned). +- **`internal/guesthook/netreassert.go`** + `PhasePostStart` — hook leg: PVE runs the hookscript as + root → direct systemctl; `GuestSeesPath` verify (hook-process mirror of GuestSeesMount); + non-fatal by contract (stderr → PVE task log, exit 0 always, 30 s bound). The installed wrapper + already forwards all phases — **no snippet re-install was needed**. -## Red-proofs (all demonstrated, then restored) +## Tests + red-proofs -| Scenario | Reverted shape | Failure seen | +| Test | Result | Red-proof | |---|---|---| -| S1 capture-at-info | ring handler gated at the emit level | `ring holds 1 entries, want 2` | -| S2 consume-once | `logTailPending = false` drain removed | `report 3 carries a tail again — consume-once broken` | -| S7 log sequence | /proc/mounts verdict Debug dropped | `phase line "netverify: /proc/mounts verdict" missing` | +| §8 decision table (`TestNetReassertAction_Table`) | PASS | always-rearm shape → FAIL `netReassertAction("nfs") = "rearmed", want "skip-active"` (the live-mount churn) | +| rearm emits EXACTLY stop+enable-now on the right unit | PASS (linux) | covered by the table red-proof (same predicate) | +| active mount → ZERO systemctl calls | PASS (linux) | the always-rearm shape fails it with 2 recorded calls | +| idempotent double-pass; zero-units no-op | PASS (linux) | — | +| hook wiring (`TestRunGuestHook_PostStartInvokesNetworkReassert`) | PASS | PhasePostStart case removed → FAIL `got []` | +| hook core verifies rearmed+active rows only; verify-failure non-fatal | PASS | — | +| daemon leg verifies RUNNING guests only; invisible share = WARN-only | PASS | — | -## Live validation (felhom-pve, 2026-07-11 ~17:00 CEST) +Full linux suite on 180: clean. Windows suite: clean (unit-name tests skip by design). -- `felhom-agent 0.83.0` live; journal clean (ReassertGuestBinds bound felhom-usb, local-api - listening, no capability degradation). -- A real NAS add from controller 0.116.1 at `logging.level=info`: the agent ring (via the - controller's Ügynök tab) shows the request middleware lines including - `POST /netstorage/add status=502 duration_ms=2001` — the refusal flow is reconstructable remotely. -- Heartbeat log-pull live round-trip: **pending the operator's hub button click** (hub UI is - password-gated; CC cannot log in). The channel is fully proven by hub API tests incl. the S6 - stays-pending case. +## Deploy + live validation -## Green gate +- Installed on felhom-pve 21:16; `felhom-agent 0.84.0`; journal clean. +- **Equivalence (spike gate §3, live):** the startup sweep found nas-media idle → re-armed via + stop+enable-now → guest visibility retained (no WARN) → **stop+enable-now ≡ restart proven; no + sudoers widening**. +- **Hook leg, real entry:** `felhom-agent guest-hook 9201 post-start` as root → exit 0, + `network share nas-media visible in guest (skip-active)`. +- **🛑 Q1c reboot (operator GO, 21:20 `pct reboot 9201`):** the start-task transcript itself carried + `netreassert: automount trigger re-armed` + `guest-hook: vmid 9201 post-start — network share + nas-media visible in guest (rearmed)`. Post-boot, NO manual action: guest had trigger + nfs4; + **calibre-web auto-started healthy on the NAS library** (the RCA's 4-hour silent death is gone); + uid-1000 write-through `WRITE-OK`; controller health `status=ok` first cycle; no storage badge. + **Q1c GREEN.** -`go build ./... && go vet ./... && go test ./...` — all green (no flakes this run). +## Observations / follow-ups -## Observations - -- Pre-existing (not from this change): `lanresolver: cannot list provisioned guests — permission - denied on /var/lib/felhom-agent/guests` WARN at startup — worth a follow-up look. -- The 20 s reconcile tick adds one DEBUG ring line per pass (`guest-bind re-assert pass`) — accepted - ring churn (~180/h; the ring holds ~1000). +- Sudoers fallback not needed (record per §3: stop+enable-now re-propagates identically). +- RCA fixes 3 (boot-failure alerting), 5 (label mojibake), 6 (ring spam) remain queued as separate + tasks. +- Peti publish-train queue: agent 0.84.0 + controller 0.117.0 + journal-group one-liner + + temp Git Sync creds deletion.