CAMPAIGN-3 Task A: agent v0.85.0 boot/recovery plane docs + host-install deployment_mode (--mode gates node self-heal)

network-storage-nas.md gains Boot-ordering (F12), reassert-hardening (F11/F10/F9),
F13 limitation, and Node self-heal (appliance) sections. host-install v1.14.0
templates deployment_mode from --mode + byo assert. Live matrix: host reboot x2 =
0 ordering-cycle lines (F12 dead); appliance self-heal recovered attempt 1.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf

CONTEXT.md

@@ -3,6 +3,19 @@
 > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
 > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
 
+- **2026-07-12 — CAMPAIGN-3 Task A SHIPPED: agent v0.85.0 boot/recovery plane + appliance self-heal.**
+  Fixes F12 (CRITICAL boot ordering cycle — templates drop network-online, `MigrateNetworkUnits`
+  repairs installed units), F11/F10/F9 (reassert: fstype-driven classify, reset-failed+rearm, per-unit
+  verdict, + re-arm-on-guest-blind matrix-correction), F2/F1 (zero residue: reset-failed + rmdir), the
+  guest-hook rc255 belt (recover+timeout, wrapper no longer execs), and the appliance node self-heal
+  (`internal/selfheal`: host networking recovery, gated on `deployment_mode` — host-install `--mode`
+  templates it; byo remedy unreachable). 3 sudoers grants added (reset-failed, rmdir, start networking).
+  **Live matrix on demo felhom-pve: host reboot ×2 = 0 ordering-cycle lines both boots (F12 dead);
+  appliance self-heal recovered networking on attempt 1; F10 reset-failed+rearm sweep loud; zero
+  residue; guest-reboot heals.** **NEW finding F13 (HIGH, deferred):** active nfs4 under mp8 can fail
+  PVE rbind (rc255); NAS-down-at-boot → apps `Created` (fix-3). Peti's box (agent 0.81) unaffected
+  until his rollout — carries the F12 coin flip if it has a NAS share (interim drop-in in the agent
+  REPORT). host-install v1.14.0; scripts CHANGELOG. Details: `documentation/controller/network-storage-nas.md`.
 - **2026-07-12 — CAMPAIGN-3 NIGHT RUN DONE; host recovered, RCA closed.** Unattended chaos campaign
   22:09→04:27 vs demo box (ctrl 0.117.0/agent 0.84.0): data plane solid (refusals/verify/rollback/
   restore byte-identical/EIO instant/stub badge + deploy-409 live-validated). The NAS automount

documentation/controller/network-storage-nas.md

@@ -158,6 +158,72 @@ the NAS library, uid-1000 write-through OK, health `status=ok` on the first cycl
 (The backlog spec `documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md` is
 implemented by this.)
 
+## Boot ordering — the F12 fix (agent v0.85.0, CAMPAIGN-3)
+
+**The bug (CRITICAL):** the `.mount` and `.automount` templates ordered `After=`/`Wants=network-online.target`.
+An automount is implicitly `Before=local-fs.target`, and `networking.service` runs after `local-fs`,
+so the units closed the cycle `networking → local-fs → automount → network-online → networking`.
+systemd breaks such a cycle by deleting an **arbitrary** job in it — on one host boot it deleted
+`networking.service` (host up, **no network, for 7 h**), on the next it deleted the automount (network
+up, NAS dead). **Every boot of a host with an enrolled network share was a coin flip.**
+
+**The fix:** neither rendered unit carries any network-online ordering. The `.mount` keeps `_netdev`
+in `Options` — that is the correct + sufficient network ordering for the REAL mount (systemd classes
+a `_netdev` mount under remote-fs.target and orders it after the network). The `.automount` gets NO
+network relation at all — a trigger needs none, and it must stay orderable before local-fs without
+dragging the network into the transaction.
+
+**Installed-unit migration:** `SudoHostOps.MigrateNetworkUnits` is a general template-drift reconcile
+(SHA-256 content compare of each marker-owned unit against a fresh render of its reconstructed spec;
+rewrite + one batched `daemon-reload`; idempotent). It runs at agent startup (before the reassert
+sweep) and at the head of `EnsureNetworkMount`, so pre-0.85 units carrying the cycle are repaired,
+not just future adds. **Live-proven 2026-07-12:** `felhom-pve` rebooted ×2 after the 0.85 deploy —
+`journalctl -b | grep -c "ordering cycle"` == **0 on both boots**, networking up, automount armed,
+guest + apps healthy. The coin flip is dead.
+
+### Reassert hardening (same release, matrix-corrected)
+
+- **F11 — read the right unit:** the re-arm decision is driven ONLY by the host `/proc/mounts` fstype
+  at the mountpoint; the `.automount` unit's own state is never consulted (an armed trigger always
+  reports "active"). **Matrix-corrected live finding (2026-07-12):** a freshly-rebooted guest inherits
+  neither an idle trigger NOR an *active* host mount — only a FRESH trigger event propagates in. So
+  skip-active is proactively safe (don't churn shares guests already see) but the caller re-arms
+  *reactively* when a specific running guest is found blind (`RearmNetworkAutomount` after a failed
+  `GuestSeesMount`/`GuestSeesPath`).
+- **F10 — re-arm for real:** a `.mount`/`.automount` left `failed`/start-limit-hit (unexport →
+  idle-timeout → repeated access) is `reset-failed` FIRST (new sudoers verb) — without it the
+  `enable --now` is refused by the start limit and the share stays dead across every boot.
+- **F9 — say what you did:** the sweep enumerates by marker-owned unit files and logs an INFO verdict
+  per share (reasserted / reset-failed+rearmed / skip-active / skip-foreign / error).
+- **The hook can never take a guest down:** every guest-hook phase runs recover-wrapped under a hard
+  timeout and returns cleanly; the installed wrapper snippet no longer `exec`s (`… || true; exit 0`).
+
+### Known limitation — F13 (active NFS under the mp8 bind at guest start)
+
+An **actively-mounted** nfs4 at `/mnt/felhom-drives/<name>` at the moment a guest starts can make
+PVE's rbind of `mp8` fail with `rc255` (an unprivileged-container idmap incompatibility), independent
+of the felhom hook (which returns 0). The realistic steady state is IDLE (the share idle-unmounts
+after `TimeoutIdleSec=60`), where the guest starts clean and the post-start reassert propagates it —
+but a share in active use at the reboot moment is a hazard. Deferred to a follow-up (candidate fixes:
+a pre-start conditional idle-unmount for single-guest hosts, or an idmapped nfs mount). Related: apps
+whose HDD_PATH is on a NAS that is down at guest boot land in docker `Created` state (the deferred
+fix-3, boot-time app-start-failure alerting).
+
+## Node self-heal (appliance) — F12-class defense in depth (agent v0.85.0)
+
+`internal/selfheal` is a minimal check/remedy registry gated on `deployment_mode`. One heal ships:
+**host networking recovery.** Healthy ⇔ `networking.service` active AND a default route exists; the
+remedy — `systemctl start networking.service` (new `FELHOM_SELFHEAL` sudoers verb, ≤3 attempts,
+10/30/60 s backoff, terminal give-up logged) — runs ONLY on `deployment_mode:"appliance"`. A **byo**
+host runs the check + WARNs but the remedy is structurally unreachable (the Manager gates before any
+exec; unit-tested: byo + unhealthy → zero privileged invocations). Absent/unknown mode → byo
+(fail-safe). The F12 template fix is the CURE for the ordering instance; this watchdog is the belt for
+the CLASS (any boot that leaves networking down, for any reason). **Live-proven 2026-07-12:**
+`systemctl stop networking.service` on the appliance → the watchdog WARNed
+(`networking.service active=false`), fired the privileged start, and recovered on attempt 1.
+`deployment_mode` is set by host-install `--mode appliance|byo` (default byo; the byo config assert
+refuses `deployment_mode=appliance`).
+
 ## Open items
 - QNAP appliance fidelity pass (no emulator) before GA — Synology is now validated.
 - The demo's `nas-media` share predates `retry=0`; re-adding re-creates the unit with the current

scripts/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Felhom scripts — Changelog
 
+## felhom-host-install v1.14.0 — deployment_mode from --mode (appliance self-heal gate) (2026-07-12)
+
+- The agent-config writer now templates the top-level **`deployment_mode`** field from `--mode`
+  (`appliance` | `byo`), authoritative even over a `--preserve-from` config. It gates the agent's
+  node self-heal (host networking recovery at boot — F12-class defense in depth, agent v0.85.0):
+  only `deployment_mode:"appliance"` unlocks the remedy; anything else, including a stale preserved
+  value or a typo, is byo (the check runs + WARNs, the remedy is unreachable). The byo config assert
+  additionally refuses `deployment_mode=appliance` (a byo box must never carry a host-service
+  self-heal). Companion to CAMPAIGN-3 Task A (`documentation/audits/CAMPAIGN-3-2026-07-11.md`).
+
 ## felhom-host-install v1.13.0 — systemd-journal group for the agent user (NAS verify) (2026-07-11)
 
 - The agent-user setup now adds `felhom-agent` to the **`systemd-journal` group** (idempotent

scripts/felhom-host-install.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 #===============================================================================
-# felhom-host-install.sh v1.13.0
+# felhom-host-install.sh v1.14.0
 # Day-0 host-bootstrap for a Felhom Proxmox host (operator-deploy model).
 #
 # Run by the operator on a FRESHLY-PVE-INSTALLED box (after a manual PVE install
@@ -19,6 +19,11 @@
 # new credential. The checksum trust root is the HUB, not Gitea. This removes the
 # old prerequisite "install the agent binary + unit manually".
 #
+# v1.14.0 (CAMPAIGN-3 Part 6): --mode now also templates the agent's top-level `deployment_mode`
+# ("appliance" | "byo") — authoritative from --mode, written even over a --preserve-from config, so a
+# byo box can never inherit an appliance host-service self-heal. Gates the agent's node self-heal
+# (host networking recovery at boot, F12-class; agent v0.85.0). The byo config assert now also refuses
+# deployment_mode=appliance. absent/typo → byo (fail-safe).
 # v1.10.0 (GL-2, go-live G2/G4/G5): explicit --mode appliance|byo install profile — the flag is now
 # REQUIRED for a fresh install (no default; build-golden v2.0.0 precedent: defaults rot). byo =
 # BYO-host hardening for a Proxmox host the operator does NOT own: break-glass (step 4b) gated OFF
@@ -2002,6 +2007,11 @@ if pf and os.path.exists(pf):
     except Exception: base = {}
 # fresh-host defaults for any section not preserved
 base.setdefault('log_level','info')
+# CAMPAIGN-3 Part 6: deployment_mode gates the agent's node self-heal (host networking recovery at
+# boot, F12-class). Authoritative from --mode: "appliance" unlocks the remedy; "byo" (or anything
+# else, including a stale preserved value) leaves it CHECK-only. Written explicitly so a byo box can
+# never inherit an appliance self-heal from a --preserve-from config.
+base['deployment_mode'] = 'appliance' if os.environ.get('INSTALL_MODE') == 'appliance' else 'byo'
 # privileged.mode = "sudo": the canonical unit runs the agent as the NON-root felhom-agent user, so
 # every host-root op goes through `sudo -n` against /etc/sudoers.d/felhom-agent. ("direct" was the old
 # dev/CI shortcut for a root agent.) Force the mode authoritative (a stale preserved "direct" config
@@ -2070,18 +2080,19 @@ PY
     # interpreter pattern as the write above) and refuse to start the daemon on any true value —
     # a --preserve-from carrying lan_resolver.enable=true is exactly what this catches.
     if [[ "$MODE" == "byo" ]]; then
-        AGENT_CONFIG="$AGENT_CONFIG" python3 <<'PY' || die "byo config assert FAILED — see the keys above; a byo box must keep lan_resolver/wg_tunnel/oob off (fix the preserved config and re-run with --resume)"
+        AGENT_CONFIG="$AGENT_CONFIG" python3 <<'PY' || die "byo config assert FAILED — see the keys above; a byo box must keep lan_resolver/wg_tunnel/oob off and deployment_mode=byo (fix the preserved config and re-run with --resume)"
 import json, os, sys
 d = json.load(open(os.environ['AGENT_CONFIG']))
 bad = []
 if d.get('lan_resolver', {}).get('enable'):  bad.append('lan_resolver.enable (takes over host DNS on :53)')
 if d.get('wg_tunnel', {}).get('enabled'):    bad.append('wg_tunnel.enabled (opens an outbound WG tunnel)')
 if d.get('oob', {}).get('enabled'):          bad.append('oob.enabled (starts a second operator sshd)')
+if d.get('deployment_mode') == 'appliance':  bad.append('deployment_mode=appliance (unlocks host-service self-heal on a host we do not own)')
 if bad:
     print('byo-forbidden config keys are TRUE: ' + '; '.join(bad), file=sys.stderr)
     sys.exit(1)
 PY
-        log_success "  byo config asserts passed (lan_resolver.enable / wg_tunnel.enabled / oob.enabled all off)"
+        log_success "  byo config asserts passed (lan_resolver.enable / wg_tunnel.enabled / oob.enabled off; deployment_mode=byo)"
     fi
 
     # health: read-only selftest (proxmox) must pass before provisioning