diff --git a/CONTEXT.md b/CONTEXT.md index 6482399..23e373e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,19 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-12 — CAMPAIGN-3 Task A SHIPPED: agent v0.85.0 boot/recovery plane + appliance self-heal.** + Fixes F12 (CRITICAL boot ordering cycle — templates drop network-online, `MigrateNetworkUnits` + repairs installed units), F11/F10/F9 (reassert: fstype-driven classify, reset-failed+rearm, per-unit + verdict, + re-arm-on-guest-blind matrix-correction), F2/F1 (zero residue: reset-failed + rmdir), the + guest-hook rc255 belt (recover+timeout, wrapper no longer execs), and the appliance node self-heal + (`internal/selfheal`: host networking recovery, gated on `deployment_mode` — host-install `--mode` + templates it; byo remedy unreachable). 3 sudoers grants added (reset-failed, rmdir, start networking). + **Live matrix on demo felhom-pve: host reboot ×2 = 0 ordering-cycle lines both boots (F12 dead); + appliance self-heal recovered networking on attempt 1; F10 reset-failed+rearm sweep loud; zero + residue; guest-reboot heals.** **NEW finding F13 (HIGH, deferred):** active nfs4 under mp8 can fail + PVE rbind (rc255); NAS-down-at-boot → apps `Created` (fix-3). Peti's box (agent 0.81) unaffected + until his rollout — carries the F12 coin flip if it has a NAS share (interim drop-in in the agent + REPORT). host-install v1.14.0; scripts CHANGELOG. Details: `documentation/controller/network-storage-nas.md`. - **2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; host recovered, RCA closed.** Unattended chaos campaign 22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/ restore byte-identical/EIO instant/stub badge + deploy-409 live-validated). The NAS automount diff --git a/documentation/controller/network-storage-nas.md b/documentation/controller/network-storage-nas.md index bb12e13..cf36bd4 100644 --- a/documentation/controller/network-storage-nas.md +++ b/documentation/controller/network-storage-nas.md @@ -158,6 +158,72 @@ the NAS library, uid-1000 write-through OK, health `status=ok` on the first cycl (The backlog spec `documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md` is implemented by this.) +## Boot ordering — the F12 fix (agent v0.85.0, CAMPAIGN-3) + +**The bug (CRITICAL):** the `.mount` and `.automount` templates ordered `After=`/`Wants=network-online.target`. +An automount is implicitly `Before=local-fs.target`, and `networking.service` runs after `local-fs`, +so the units closed the cycle `networking → local-fs → automount → network-online → networking`. +systemd breaks such a cycle by deleting an **arbitrary** job in it — on one host boot it deleted +`networking.service` (host up, **no network, for 7 h**), on the next it deleted the automount (network +up, NAS dead). **Every boot of a host with an enrolled network share was a coin flip.** + +**The fix:** neither rendered unit carries any network-online ordering. The `.mount` keeps `_netdev` +in `Options` — that is the correct + sufficient network ordering for the REAL mount (systemd classes +a `_netdev` mount under remote-fs.target and orders it after the network). The `.automount` gets NO +network relation at all — a trigger needs none, and it must stay orderable before local-fs without +dragging the network into the transaction. + +**Installed-unit migration:** `SudoHostOps.MigrateNetworkUnits` is a general template-drift reconcile +(SHA-256 content compare of each marker-owned unit against a fresh render of its reconstructed spec; +rewrite + one batched `daemon-reload`; idempotent). It runs at agent startup (before the reassert +sweep) and at the head of `EnsureNetworkMount`, so pre-0.85 units carrying the cycle are repaired, +not just future adds. **Live-proven 2026-07-12:** `felhom-pve` rebooted ×2 after the 0.85 deploy — +`journalctl -b | grep -c "ordering cycle"` == **0 on both boots**, networking up, automount armed, +guest + apps healthy. The coin flip is dead. + +### Reassert hardening (same release, matrix-corrected) + +- **F11 — read the right unit:** the re-arm decision is driven ONLY by the host `/proc/mounts` fstype + at the mountpoint; the `.automount` unit's own state is never consulted (an armed trigger always + reports "active"). **Matrix-corrected live finding (2026-07-12):** a freshly-rebooted guest inherits + neither an idle trigger NOR an *active* host mount — only a FRESH trigger event propagates in. So + skip-active is proactively safe (don't churn shares guests already see) but the caller re-arms + *reactively* when a specific running guest is found blind (`RearmNetworkAutomount` after a failed + `GuestSeesMount`/`GuestSeesPath`). +- **F10 — re-arm for real:** a `.mount`/`.automount` left `failed`/start-limit-hit (unexport → + idle-timeout → repeated access) is `reset-failed` FIRST (new sudoers verb) — without it the + `enable --now` is refused by the start limit and the share stays dead across every boot. +- **F9 — say what you did:** the sweep enumerates by marker-owned unit files and logs an INFO verdict + per share (reasserted / reset-failed+rearmed / skip-active / skip-foreign / error). +- **The hook can never take a guest down:** every guest-hook phase runs recover-wrapped under a hard + timeout and returns cleanly; the installed wrapper snippet no longer `exec`s (`… || true; exit 0`). + +### Known limitation — F13 (active NFS under the mp8 bind at guest start) + +An **actively-mounted** nfs4 at `/mnt/felhom-drives/` at the moment a guest starts can make +PVE's rbind of `mp8` fail with `rc255` (an unprivileged-container idmap incompatibility), independent +of the felhom hook (which returns 0). The realistic steady state is IDLE (the share idle-unmounts +after `TimeoutIdleSec=60`), where the guest starts clean and the post-start reassert propagates it — +but a share in active use at the reboot moment is a hazard. Deferred to a follow-up (candidate fixes: +a pre-start conditional idle-unmount for single-guest hosts, or an idmapped nfs mount). Related: apps +whose HDD_PATH is on a NAS that is down at guest boot land in docker `Created` state (the deferred +fix-3, boot-time app-start-failure alerting). + +## Node self-heal (appliance) — F12-class defense in depth (agent v0.85.0) + +`internal/selfheal` is a minimal check/remedy registry gated on `deployment_mode`. One heal ships: +**host networking recovery.** Healthy ⇔ `networking.service` active AND a default route exists; the +remedy — `systemctl start networking.service` (new `FELHOM_SELFHEAL` sudoers verb, ≤3 attempts, +10/30/60 s backoff, terminal give-up logged) — runs ONLY on `deployment_mode:"appliance"`. A **byo** +host runs the check + WARNs but the remedy is structurally unreachable (the Manager gates before any +exec; unit-tested: byo + unhealthy → zero privileged invocations). Absent/unknown mode → byo +(fail-safe). The F12 template fix is the CURE for the ordering instance; this watchdog is the belt for +the CLASS (any boot that leaves networking down, for any reason). **Live-proven 2026-07-12:** +`systemctl stop networking.service` on the appliance → the watchdog WARNed +(`networking.service active=false`), fired the privileged start, and recovered on attempt 1. +`deployment_mode` is set by host-install `--mode appliance|byo` (default byo; the byo config assert +refuses `deployment_mode=appliance`). + ## Open items - QNAP appliance fidelity pass (no emulator) before GA — Synology is now validated. - The demo's `nas-media` share predates `retry=0`; re-adding re-creates the unit with the current diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 0a77f18..d147ba2 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,5 +1,15 @@ # Felhom scripts — Changelog +## felhom-host-install v1.14.0 — deployment_mode from --mode (appliance self-heal gate) (2026-07-12) + +- The agent-config writer now templates the top-level **`deployment_mode`** field from `--mode` + (`appliance` | `byo`), authoritative even over a `--preserve-from` config. It gates the agent's + node self-heal (host networking recovery at boot — F12-class defense in depth, agent v0.85.0): + only `deployment_mode:"appliance"` unlocks the remedy; anything else, including a stale preserved + value or a typo, is byo (the check runs + WARNs, the remedy is unreachable). The byo config assert + additionally refuses `deployment_mode=appliance` (a byo box must never carry a host-service + self-heal). Companion to CAMPAIGN-3 Task A (`documentation/audits/CAMPAIGN-3-2026-07-11.md`). + ## felhom-host-install v1.13.0 — systemd-journal group for the agent user (NAS verify) (2026-07-11) - The agent-user setup now adds `felhom-agent` to the **`systemd-journal` group** (idempotent diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh index 2144693..0e714eb 100644 --- a/scripts/felhom-host-install.sh +++ b/scripts/felhom-host-install.sh @@ -1,6 +1,6 @@ #!/bin/bash #=============================================================================== -# felhom-host-install.sh v1.13.0 +# felhom-host-install.sh v1.14.0 # Day-0 host-bootstrap for a Felhom Proxmox host (operator-deploy model). # # Run by the operator on a FRESHLY-PVE-INSTALLED box (after a manual PVE install @@ -19,6 +19,11 @@ # new credential. The checksum trust root is the HUB, not Gitea. This removes the # old prerequisite "install the agent binary + unit manually". # +# v1.14.0 (CAMPAIGN-3 Part 6): --mode now also templates the agent's top-level `deployment_mode` +# ("appliance" | "byo") — authoritative from --mode, written even over a --preserve-from config, so a +# byo box can never inherit an appliance host-service self-heal. Gates the agent's node self-heal +# (host networking recovery at boot, F12-class; agent v0.85.0). The byo config assert now also refuses +# deployment_mode=appliance. absent/typo → byo (fail-safe). # v1.10.0 (GL-2, go-live G2/G4/G5): explicit --mode appliance|byo install profile — the flag is now # REQUIRED for a fresh install (no default; build-golden v2.0.0 precedent: defaults rot). byo = # BYO-host hardening for a Proxmox host the operator does NOT own: break-glass (step 4b) gated OFF @@ -2002,6 +2007,11 @@ if pf and os.path.exists(pf): except Exception: base = {} # fresh-host defaults for any section not preserved base.setdefault('log_level','info') +# CAMPAIGN-3 Part 6: deployment_mode gates the agent's node self-heal (host networking recovery at +# boot, F12-class). Authoritative from --mode: "appliance" unlocks the remedy; "byo" (or anything +# else, including a stale preserved value) leaves it CHECK-only. Written explicitly so a byo box can +# never inherit an appliance self-heal from a --preserve-from config. +base['deployment_mode'] = 'appliance' if os.environ.get('INSTALL_MODE') == 'appliance' else 'byo' # privileged.mode = "sudo": the canonical unit runs the agent as the NON-root felhom-agent user, so # every host-root op goes through `sudo -n` against /etc/sudoers.d/felhom-agent. ("direct" was the old # dev/CI shortcut for a root agent.) Force the mode authoritative (a stale preserved "direct" config @@ -2070,18 +2080,19 @@ PY # interpreter pattern as the write above) and refuse to start the daemon on any true value — # a --preserve-from carrying lan_resolver.enable=true is exactly what this catches. if [[ "$MODE" == "byo" ]]; then - AGENT_CONFIG="$AGENT_CONFIG" python3 <<'PY' || die "byo config assert FAILED — see the keys above; a byo box must keep lan_resolver/wg_tunnel/oob off (fix the preserved config and re-run with --resume)" + AGENT_CONFIG="$AGENT_CONFIG" python3 <<'PY' || die "byo config assert FAILED — see the keys above; a byo box must keep lan_resolver/wg_tunnel/oob off and deployment_mode=byo (fix the preserved config and re-run with --resume)" import json, os, sys d = json.load(open(os.environ['AGENT_CONFIG'])) bad = [] if d.get('lan_resolver', {}).get('enable'): bad.append('lan_resolver.enable (takes over host DNS on :53)') if d.get('wg_tunnel', {}).get('enabled'): bad.append('wg_tunnel.enabled (opens an outbound WG tunnel)') if d.get('oob', {}).get('enabled'): bad.append('oob.enabled (starts a second operator sshd)') +if d.get('deployment_mode') == 'appliance': bad.append('deployment_mode=appliance (unlocks host-service self-heal on a host we do not own)') if bad: print('byo-forbidden config keys are TRUE: ' + '; '.join(bad), file=sys.stderr) sys.exit(1) PY - log_success " byo config asserts passed (lan_resolver.enable / wg_tunnel.enabled / oob.enabled all off)" + log_success " byo config asserts passed (lan_resolver.enable / wg_tunnel.enabled / oob.enabled off; deployment_mode=byo)" fi # health: read-only selftest (proxmox) must pass before provisioning