From bcb8dad2aa2ab3f893259da6b7f48ecd6b192b03 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 12 Jul 2026 08:26:14 +0200 Subject: [PATCH] =?UTF-8?q?agent=20v0.85.0:=20CHANGELOG=20+=20REPORT=20?= =?UTF-8?q?=E2=80=94=20boot/recovery=20plane=20shipped=20+=20live=20matrix?= =?UTF-8?q?=20(F12=20dead,=20self-heal=20proven)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf --- REPORT.md | 133 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 74 insertions(+), 59 deletions(-) diff --git a/REPORT.md b/REPORT.md index 5099797..31cd7e6 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,70 +1,85 @@ -# REPORT — agent v0.84.0: ReassertNetworkMounts — NAS automount survives guest reboots (RCA fix 1) +# REPORT — agent v0.85.0: the boot/recovery plane + appliance self-heal (CAMPAIGN-3 Task A) -**Date:** 2026-07-11 · **Commit:** `474b858` · **Deployed:** felhom-pve (backup `.bak-0.83.0`) · -**PUBLISHED to Gitea (same evening, operator-requested):** `publish-agent.sh 0.84.0` with the LIVE -felhom-pve bytes (sha identical felhom-pve↔180) — pre-GET 404 → PUT 201 → round-trip match → -**anonymous GET 200, 13 582 910 bytes, sha == AGENT_SHA256 -`2e31b9a8f9355d674c8f81523015421bf1ecb590023d7fdf959a2bfbaf4c42d3`**; hop copies removed -(180 + local + felhom-pve /tmp), token-leak in transcript: none (creds decoded remotely only). -**OPERATOR follow-up: hub Day-0 manifest vouch (Configs → Day-0 artifacts: agent 0.84.0 / sha -above) — password-gated UI. Peti still on 0.81.0 until the next train.** -**Source:** `felhom.eu/documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md` (fix 1); pairs with -controller v0.117.0 (fixes 2+4). - -## Baseline (fresh-verified at session start) - -Agent 0.83.0 live on felhom-pve; controller 0.116.1 on 9201; nas-media active-mounted; CWA healthy. +**Date:** 2026-07-12 · **Baselines:** agent v0.84.0 (`bec4bac`) → **v0.85.0**; felhom.eu post-`f97f7803`. +**Source of truth:** `felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md`. +**Scope:** F12 (CRITICAL), F11/F10/F9 (HIGH/CRITICAL), F2/F1 (LOW), the guest-hook rc255 belt, and the +operator-mandated appliance-mode node self-heal. Controller untouched (MinAgent unchanged). Green gate +(`go build ./... && go vet ./... && go test ./...`) PASS on the build server. ## What shipped -- **`internal/storage/netreassert.go`** `ReassertNetworkAutomounts` — the §8 decision table: real - nfs/nfs4/cifs mounted → skip (fresh namespaces inherit real mounts); `autofs` trigger → - `systemctl stop -- .automount` + `enable --now -- .automount` (**existing - FELHOM_NETMOUNT verbs — the §3 sudoers fallback was NOT needed**, see equivalence below); - neither → skip (orphan states owned by add/remove). Idempotent; per-share errors never stop the - pass. `networkUnitEntries()` factored out of `ListNetworkMounts` (behavior unchanged). -- **`internal/localapi/netreassert.go`** `Server.ReassertNetworkMounts` — daemon leg: host-global - pass once, then per-RUNNING-guest `GuestSeesMount` verify (WARN-only). Type-asserted capability - (lean `NetworkStorageOps` + fakes untouched). Wired at startup after `ReassertGuestBinds`; - deliberately NOT in the 20 s ticker (idle trigger = healthy, never churned). -- **`internal/guesthook/netreassert.go`** + `PhasePostStart` — hook leg: PVE runs the hookscript as - root → direct systemctl; `GuestSeesPath` verify (hook-process mirror of GuestSeesMount); - non-fatal by contract (stderr → PVE task log, exit 0 always, 30 s bound). The installed wrapper - already forwards all phases — **no snippet re-install was needed**. +- **F12 (CRITICAL) — boot ordering cycle.** Both rendered units drop `After=/Wants=network-online.target` + (`.mount` keeps `_netdev`; `.automount` gets no network relation). `MigrateNetworkUnits` — a general + template-drift reconcile (SHA-256 content compare vs a fresh render of the reconstructed spec; + rewrite + one batched `daemon-reload`; idempotent) — repairs ALREADY-INSTALLED units at startup and + at the head of `EnsureNetworkMount`. +- **F11 — read the right unit** (`netReassertClassify`): decision by host `/proc/mounts` fstype only, + never the automount's own state. **Matrix-corrected:** a rebooted guest inherits neither an idle + trigger NOR an active mount, so the callers re-arm reactively when a running guest is found blind + (`RearmNetworkAutomount` after a failed visibility check). +- **F10 — re-arm for real:** a failed/start-limited unit is `reset-failed` before `enable --now` + (unprivileged `systemctl is-failed` seam + new sudoers verb). +- **F9 — say what you did:** enumerate by marker-owned unit files; one INFO verdict per share. +- **F2/F1 — zero residue:** `RemoveNetworkMount` (and every verify-fail rollback via it) reset-failed's + the pair and `rmdir`s the mountpoint (rmdir-only fail-safe; never `rm -rf`). +- **Hook can never take a guest down:** `runHookPhase` — per-phase recover + hard timeout, always + returns; the installed wrapper snippet no longer `exec`s (`… || true; exit 0`). +- **Appliance self-heal (`internal/selfheal`):** one heal — host networking recovery, gated on + `deployment_mode:"appliance"`; byo runs the check + WARNs, remedy unreachable. Config + `deployment_mode` (+`FELHOM_AGENT_DEPLOYMENT_MODE`, `IsAppliance()` fail-safe-to-byo). -## Tests + red-proofs +## Sudoers — three grants added (LOUD, per the no-widening rule) -| Test | Result | Red-proof | -|---|---|---| -| §8 decision table (`TestNetReassertAction_Table`) | PASS | always-rearm shape → FAIL `netReassertAction("nfs") = "rearmed", want "skip-active"` (the live-mount churn) | -| rearm emits EXACTLY stop+enable-now on the right unit | PASS (linux) | covered by the table red-proof (same predicate) | -| active mount → ZERO systemctl calls | PASS (linux) | the always-rearm shape fails it with 2 recorded calls | -| idempotent double-pass; zero-units no-op | PASS (linux) | — | -| hook wiring (`TestRunGuestHook_PostStartInvokesNetworkReassert`) | PASS | PhasePostStart case removed → FAIL `got []` | -| hook core verifies rearmed+active rows only; verify-failure non-fatal | PASS | — | -| daemon leg verifies RUNNING guests only; invisible share = WARN-only | PASS | — | +Shipped with the binary; capability manifest updated (probed live: 62/62, 0 degraded): +1. **`systemctl reset-failed -- mnt-felhom*`** (FELHOM_NETMOUNT) — F10: no platform path could re-arm a + start-limited automount. reset-failed only clears a failed latch; it cannot start/stop anything. +2. **`rmdir /mnt/felhom-drives/*`** (FELHOM_NETMOUNT) — F1: removed-share mountpoint cleanup. rmdir + refuses a non-empty dir (fail-safe; unexpected data is preserved, never `rm -rf`'d). +3. **`systemctl start networking.service`** (new FELHOM_SELFHEAL alias) — F12-class appliance self-heal. + Fixed unit, no glob; the grant alone cannot harm; the remedy is ALSO code-gated on appliance mode. -Full linux suite on 180: clean. Windows suite: clean (unit-name tests skip by design). +## Tests + red-proofs (all green) -## Deploy + live validation +- Render: no network-online in either unit, `_netdev` present (companion: re-adding the line fails). +- Reconcile: drifted (legacy) unit rewritten once + batched daemon-reload; idempotent (0 rewrites 2nd + pass); spec round-trip exact; foreign units ignored. +- Reassert: fstype table with the automount-state-ignored red-proof; reset-failed+rearm verdict; + per-unit verdict count (F9); re-arm-on-guest-blind (F11 matrix-correction); byo-blind non-fatal. +- Hook rc-0 under panic + timeout (Part 4); zero-residue (reset-failed + rmdir, never rm -rf). +- Self-heal state machine; **byo-never-invokes red-proof** (byo + unhealthy → zero privileged + invocations); absent field ⇒ byo. -- Installed on felhom-pve 21:16; `felhom-agent 0.84.0`; journal clean. -- **Equivalence (spike gate §3, live):** the startup sweep found nas-media idle → re-armed via - stop+enable-now → guest visibility retained (no WARN) → **stop+enable-now ≡ restart proven; no - sudoers widening**. -- **Hook leg, real entry:** `felhom-agent guest-hook 9201 post-start` as root → exit 0, - `network share nas-media visible in guest (skip-active)`. -- **🛑 Q1c reboot (operator GO, 21:20 `pct reboot 9201`):** the start-task transcript itself carried - `netreassert: automount trigger re-armed` + `guest-hook: vmid 9201 post-start — network share - nas-media visible in guest (rearmed)`. Post-boot, NO manual action: guest had trigger + nfs4; - **calibre-web auto-started healthy on the NAS library** (the RCA's 4-hour silent death is gone); - uid-1000 write-through `WRITE-OK`; controller health `status=ok` first cycle; no storage badge. - **Q1c GREEN.** +## Live MATRIX (demo felhom-pve, agent 0.85.0 + deployment_mode=appliance) -## Observations / follow-ups +Deploy verified: startup migrated the installed nas-media units (0 network-online lines); no interim +drop-in existed to remove (the campaign recovery used reset-failed+restart, not a drop-in). -- Sudoers fallback not needed (record per §3: stop+enable-now re-propagates identically). -- RCA fixes 3 (boot-failure alerting), 5 (label mojibake), 6 (ring spam) remain queued as separate - tasks. -- Peti publish-train queue: agent 0.84.0 + controller 0.117.0 + journal-group one-liner + - temp Git Sync creds deletion. +| # | Row | Result | +|---|-----|--------| +| 1 | guest reboot × idle share ×2 | **PASS** — zero-touch: reassert re-armed, guest sees NAS, write-through OK, 4 NAS apps auto-started | +| 2 | guest reboot × active share | **F13 (new finding)** — active nfs4 under mp8 can fail PVE's rbind (rc255, idmap); NAS-down-at-boot → apps `Created` (deferred fix-3). Independent of the felhom hook (rc0). | +| 3 | F10 manufacture → agent restart | **PASS** — sweep `reset-failed` both units (loud WARN) + re-armed (verdict `reset-failed+rearmed`); the campaign's silent no-op is gone | +| 4 | F10 state → guest reboot (hook path) | **PASS** — guest STARTS (rc0); F11 matrix-correction healed the blind guest live ("re-arming … visible after re-arm (healed)") | +| 5 | 🛑 host reboot ×2 | **PASS** — `grep -c "ordering cycle"` == **0 on BOTH boots**, networking up, automount armed, guest + 8 apps healthy. **F12 coin flip is dead.** | +| 6 | 🛑 stop networking (appliance) | **PASS** — watchdog WARNed `active=false`, fired the privileged start, recovered on attempt 1 (~11 s). BYO live variant NOT run (red-proof only). | +| 7 | refused add + remove cycles ×3 | **PASS** — zero residue (0 units/dirs/failed/creds; campaign left 10 dirs); registry clean | + +## NOT live-validated / standing items + +- **Row 6 byo variant** — not run live (would leave the box dark); byo is red-proof-tested only. +- **F13 (NEW, HIGH)** — active-nfs-under-mp8 rc255 + NAS-down-at-boot apps-`Created`. Deferred (needs a + pre-start idle-unmount design for multi-guest hosts, or an idmapped nfs mount + fix-3 alerting). +- **Peti** runs agent 0.81 — unaffected by this train until his rollout; if his box has a NAS share he + carries the F12 coin flip until then (interim drop-in recipe below). +- Task B/C/D findings (F7/F6/F5, F8/F4, ring/alerting) remain queued. +- **Publish train:** 0.85 is customer-critical (F12 affects any customer with a NAS share) — prioritize. + +## Box state at wrap + +agent 0.85.0 / deployment_mode=appliance; nas-media clean (no ordering cycle) + active; 0 failed felhom +units; 0 leftover mountpoint dirs (19 pre-fix campaign leftovers rmdir'd); guest 9201 + all 10 +containers healthy with NAS data intact. + +### Interim F12 mitigation for a not-yet-upgraded host +A systemd drop-in on each installed `mnt-felhom…-.automount` with empty `After=`/`Wants=` +overrides + `daemon-reload` strips the ordering until the 0.85 agent migrates the unit at startup.