felhom-agent/REPORT.md

REPORT — agent v0.85.0: the boot/recovery plane + appliance self-heal (CAMPAIGN-3 Task A)

Date: 2026-07-12 · Baselines: agent v0.84.0 (bec4bac) → v0.85.0; felhom.eu post-f97f7803. Source of truth: felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md. Scope: F12 (CRITICAL), F11/F10/F9 (HIGH/CRITICAL), F2/F1 (LOW), the guest-hook rc255 belt, and the operator-mandated appliance-mode node self-heal. Controller untouched (MinAgent unchanged). Green gate (go build ./... && go vet ./... && go test ./...) PASS on the build server.

What shipped

• F12 (CRITICAL) — boot ordering cycle. Both rendered units drop After=/Wants=network-online.target (.mount keeps _netdev; .automount gets no network relation). MigrateNetworkUnits — a general template-drift reconcile (SHA-256 content compare vs a fresh render of the reconstructed spec; rewrite + one batched daemon-reload; idempotent) — repairs ALREADY-INSTALLED units at startup and at the head of EnsureNetworkMount.
• F11 — read the right unit (netReassertClassify): decision by host /proc/mounts fstype only, never the automount's own state. Matrix-corrected: a rebooted guest inherits neither an idle trigger NOR an active mount, so the callers re-arm reactively when a running guest is found blind (RearmNetworkAutomount after a failed visibility check).
• F10 — re-arm for real: a failed/start-limited unit is reset-failed before enable --now (unprivileged systemctl is-failed seam + new sudoers verb).
• F9 — say what you did: enumerate by marker-owned unit files; one INFO verdict per share.
• F2/F1 — zero residue: RemoveNetworkMount (and every verify-fail rollback via it) reset-failed's the pair and rmdirs the mountpoint (rmdir-only fail-safe; never rm -rf).
• Hook can never take a guest down: runHookPhase — per-phase recover + hard timeout, always returns; the installed wrapper snippet no longer execs (… || true; exit 0).
• Appliance self-heal (internal/selfheal): one heal — host networking recovery, gated on deployment_mode:"appliance"; byo runs the check + WARNs, remedy unreachable. Config deployment_mode (+FELHOM_AGENT_DEPLOYMENT_MODE, IsAppliance() fail-safe-to-byo).

Sudoers — three grants added (LOUD, per the no-widening rule)

Shipped with the binary; capability manifest updated (probed live: 62/62, 0 degraded):

1. systemctl reset-failed -- mnt-felhom* (FELHOM_NETMOUNT) — F10: no platform path could re-arm a start-limited automount. reset-failed only clears a failed latch; it cannot start/stop anything.
2. rmdir /mnt/felhom-drives/* (FELHOM_NETMOUNT) — F1: removed-share mountpoint cleanup. rmdir refuses a non-empty dir (fail-safe; unexpected data is preserved, never rm -rf'd).
3. systemctl start networking.service (new FELHOM_SELFHEAL alias) — F12-class appliance self-heal. Fixed unit, no glob; the grant alone cannot harm; the remedy is ALSO code-gated on appliance mode.

Tests + red-proofs (all green)

• Render: no network-online in either unit, _netdev present (companion: re-adding the line fails).
• Reconcile: drifted (legacy) unit rewritten once + batched daemon-reload; idempotent (0 rewrites 2nd pass); spec round-trip exact; foreign units ignored.
• Reassert: fstype table with the automount-state-ignored red-proof; reset-failed+rearm verdict; per-unit verdict count (F9); re-arm-on-guest-blind (F11 matrix-correction); byo-blind non-fatal.
• Hook rc-0 under panic + timeout (Part 4); zero-residue (reset-failed + rmdir, never rm -rf).
• Self-heal state machine; byo-never-invokes red-proof (byo + unhealthy → zero privileged invocations); absent field ⇒ byo.

Live MATRIX (demo felhom-pve, agent 0.85.0 + deployment_mode=appliance)

Deploy verified: startup migrated the installed nas-media units (0 network-online lines); no interim drop-in existed to remove (the campaign recovery used reset-failed+restart, not a drop-in).

#	Row	Result
1	guest reboot × idle share ×2	PASS — zero-touch: reassert re-armed, guest sees NAS, write-through OK, 4 NAS apps auto-started
2	guest reboot × active share	F13 (new finding) — active nfs4 under mp8 can fail PVE's rbind (rc255, idmap); NAS-down-at-boot → apps Created (deferred fix-3). Independent of the felhom hook (rc0).
3	F10 manufacture → agent restart	PASS — sweep reset-failed both units (loud WARN) + re-armed (verdict reset-failed+rearmed); the campaign's silent no-op is gone
4	F10 state → guest reboot (hook path)	PASS — guest STARTS (rc0); F11 matrix-correction healed the blind guest live ("re-arming … visible after re-arm (healed)")
5	🛑 host reboot ×2	PASS — grep -c "ordering cycle" == 0 on BOTH boots, networking up, automount armed, guest + 8 apps healthy. F12 coin flip is dead.
6	🛑 stop networking (appliance)	PASS — watchdog WARNed active=false, fired the privileged start, recovered on attempt 1 (~11 s). BYO live variant NOT run (red-proof only).
7	refused add + remove cycles ×3	PASS — zero residue (0 units/dirs/failed/creds; campaign left 10 dirs); registry clean

NOT live-validated / standing items

• Row 6 byo variant — not run live (would leave the box dark); byo is red-proof-tested only.
• F13 (NEW, HIGH) — active-nfs-under-mp8 rc255 + NAS-down-at-boot apps-Created. Deferred (needs a pre-start idle-unmount design for multi-guest hosts, or an idmapped nfs mount + fix-3 alerting).
• Peti runs agent 0.81 — unaffected by this train until his rollout; if his box has a NAS share he carries the F12 coin flip until then (interim drop-in recipe below).
• Task B/C/D findings (F7/F6/F5, F8/F4, ring/alerting) remain queued.
• Publish train: 0.85 is customer-critical (F12 affects any customer with a NAS share) — prioritize.

Box state at wrap

agent 0.85.0 / deployment_mode=appliance; nas-media clean (no ordering cycle) + active; 0 failed felhom units; 0 leftover mountpoint dirs (19 pre-fix campaign leftovers rmdir'd); guest 9201 + all 10 containers healthy with NAS data intact.

Interim F12 mitigation for a not-yet-upgraded host

A systemd drop-in on each installed mnt-felhom…-<name>.automount with empty After=/Wants= overrides + daemon-reload strips the ordering until the 0.85 agent migrates the unit at startup.