docs: REPORT for v0.84.0 (deployed felhom-pve; Q1c reboot GREEN; equivalence proven, no sudoers widening)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
2026-07-11 21:25:10 +02:00
parent 474b858c0b
commit a8b773f892
+52 -41
View File
@@ -1,53 +1,64 @@
# REPORT — agent v0.83.0: observability pass (capture ring + /debug/logs + heartbeat log-pull + sweep)
# REPORT — agent v0.84.0: ReassertNetworkMounts — NAS automount survives guest reboots (RCA fix 1)
**Date:** 2026-07-11 · **Commit:** `cb692f8` · **Deployed:** felhom-pve (backup `.bak-0.82.0`) ·
**NOT published** (Peti stays 0.81.0; reaches him with the next publish train)
**Date:** 2026-07-11 · **Commit:** `474b858` · **Deployed:** felhom-pve (backup `.bak-0.83.0`) ·
**NOT published** (Peti stays 0.81.0; publish-train queue now carries 0.84.0)
**Source:** `felhom.eu/documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md` (fix 1); pairs with
controller v0.117.0 (fixes 2+4).
## Baseline (fresh-verified at session start)
Agent 0.83.0 live on felhom-pve; controller 0.116.1 on 9201; nas-media active-mounted; CWA healthy.
## What shipped
- **Capture layer** (`internal/log`): `applog.New``(logger, *Ring)`. slog fan-out: stderr/journald
at the configured level (unchanged), a 1000-entry ring FIXED at LevelDebug. `Ring.Lines(maxBytes)`
renders newest-kept plain text (the heartbeat tail source).
- **`GET /debug/logs`** on the local API (token-authed, self-scoped like every sibling; `?raw=1`
plain text; 503 unwired) + request-level DEBUG middleware (method/path/status/duration, never bodies).
- **Heartbeat log-pull**: `ControlEnvelope.log_tail_requested` (additive) → the NEXT heartbeat
carries `log_tail {collected_at, lines[]}` (128 KB cap, newest kept). Consume-once mirrored from
report logtail.go: local pending drains onto the carrying push; a failed push is re-armed by the
next envelope (retry proven in tests). Serving a pull logs `operator log pull served` (INFO).
- **Gap-fill sweep**: netverify (job start / trigger outcome / **/proc/mounts verdict** / journal
byte-count / classification code / rollback outcome / durations), netstorage add (pre-probe PASS
verdict, creds staged/removed — path only), netmount (unit install/enable/remove-step results),
signedjobs (jobs fetched ids+duration; op received class/host/expiry — never signatures),
selfupdate (invariants passed, download sha-match+duration), disks (assign/eject/decommission
outcome INFO), ReassertGuestBinds pass summary, controller-swap (pre-pull verify, negative health
verdict), desired syncer + hub loop per-exchange DEBUG.
- **`internal/storage/netreassert.go`** `ReassertNetworkAutomounts` — the §8 decision table: real
nfs/nfs4/cifs mounted → skip (fresh namespaces inherit real mounts); `autofs` trigger →
`systemctl stop -- <unit>.automount` + `enable --now -- <unit>.automount` (**existing
FELHOM_NETMOUNT verbs — the §3 sudoers fallback was NOT needed**, see equivalence below);
neither → skip (orphan states owned by add/remove). Idempotent; per-share errors never stop the
pass. `networkUnitEntries()` factored out of `ListNetworkMounts` (behavior unchanged).
- **`internal/localapi/netreassert.go`** `Server.ReassertNetworkMounts` — daemon leg: host-global
pass once, then per-RUNNING-guest `GuestSeesMount` verify (WARN-only). Type-asserted capability
(lean `NetworkStorageOps` + fakes untouched). Wired at startup after `ReassertGuestBinds`;
deliberately NOT in the 20 s ticker (idle trigger = healthy, never churned).
- **`internal/guesthook/netreassert.go`** + `PhasePostStart` — hook leg: PVE runs the hookscript as
root → direct systemctl; `GuestSeesPath` verify (hook-process mirror of GuestSeesMount);
non-fatal by contract (stderr → PVE task log, exit 0 always, 30 s bound). The installed wrapper
already forwards all phases — **no snippet re-install was needed**.
## Red-proofs (all demonstrated, then restored)
## Tests + red-proofs
| Scenario | Reverted shape | Failure seen |
| Test | Result | Red-proof |
|---|---|---|
| S1 capture-at-info | ring handler gated at the emit level | `ring holds 1 entries, want 2` |
| S2 consume-once | `logTailPending = false` drain removed | `report 3 carries a tail again — consume-once broken` |
| S7 log sequence | /proc/mounts verdict Debug dropped | `phase line "netverify: /proc/mounts verdict" missing` |
| §8 decision table (`TestNetReassertAction_Table`) | PASS | always-rearm shape → FAIL `netReassertAction("nfs") = "rearmed", want "skip-active"` (the live-mount churn) |
| rearm emits EXACTLY stop+enable-now on the right unit | PASS (linux) | covered by the table red-proof (same predicate) |
| active mount → ZERO systemctl calls | PASS (linux) | the always-rearm shape fails it with 2 recorded calls |
| idempotent double-pass; zero-units no-op | PASS (linux) | — |
| hook wiring (`TestRunGuestHook_PostStartInvokesNetworkReassert`) | PASS | PhasePostStart case removed → FAIL `got []` |
| hook core verifies rearmed+active rows only; verify-failure non-fatal | PASS | — |
| daemon leg verifies RUNNING guests only; invisible share = WARN-only | PASS | — |
## Live validation (felhom-pve, 2026-07-11 ~17:00 CEST)
Full linux suite on 180: clean. Windows suite: clean (unit-name tests skip by design).
- `felhom-agent 0.83.0` live; journal clean (ReassertGuestBinds bound felhom-usb, local-api
listening, no capability degradation).
- A real NAS add from controller 0.116.1 at `logging.level=info`: the agent ring (via the
controller's Ügynök tab) shows the request middleware lines including
`POST /netstorage/add status=502 duration_ms=2001` — the refusal flow is reconstructable remotely.
- Heartbeat log-pull live round-trip: **pending the operator's hub button click** (hub UI is
password-gated; CC cannot log in). The channel is fully proven by hub API tests incl. the S6
stays-pending case.
## Deploy + live validation
## Green gate
- Installed on felhom-pve 21:16; `felhom-agent 0.84.0`; journal clean.
- **Equivalence (spike gate §3, live):** the startup sweep found nas-media idle → re-armed via
stop+enable-now → guest visibility retained (no WARN) → **stop+enable-now ≡ restart proven; no
sudoers widening**.
- **Hook leg, real entry:** `felhom-agent guest-hook 9201 post-start` as root → exit 0,
`network share nas-media visible in guest (skip-active)`.
- **🛑 Q1c reboot (operator GO, 21:20 `pct reboot 9201`):** the start-task transcript itself carried
`netreassert: automount trigger re-armed` + `guest-hook: vmid 9201 post-start — network share
nas-media visible in guest (rearmed)`. Post-boot, NO manual action: guest had trigger + nfs4;
**calibre-web auto-started healthy on the NAS library** (the RCA's 4-hour silent death is gone);
uid-1000 write-through `WRITE-OK`; controller health `status=ok` first cycle; no storage badge.
**Q1c GREEN.**
`go build ./... && go vet ./... && go test ./...` — all green (no flakes this run).
## Observations / follow-ups
## Observations
- Pre-existing (not from this change): `lanresolver: cannot list provisioned guests — permission
denied on /var/lib/felhom-agent/guests` WARN at startup — worth a follow-up look.
- The 20 s reconcile tick adds one DEBUG ring line per pass (`guest-bind re-assert pass`) — accepted
ring churn (~180/h; the ring holds ~1000).
- Sudoers fallback not needed (record per §3: stop+enable-now re-propagates identically).
- RCA fixes 3 (boot-failure alerting), 5 (label mojibake), 6 (ring spam) remain queued as separate
tasks.
- Peti publish-train queue: agent 0.84.0 + controller 0.117.0 + journal-group one-liner +
temp Git Sync creds deletion.