CAMPAIGN-3 Task A: agent v0.85.0 boot/recovery plane docs + host-install deployment_mode (--mode gates node self-heal)

network-storage-nas.md gains Boot-ordering (F12), reassert-hardening (F11/F10/F9),
F13 limitation, and Node self-heal (appliance) sections. host-install v1.14.0
templates deployment_mode from --mode + byo assert. Live matrix: host reboot x2 =
0 ordering-cycle lines (F12 dead); appliance self-heal recovered attempt 1.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
This commit is contained in:
2026-07-12 08:26:55 +02:00
parent f97f7803c7
commit 0d73ca579b
4 changed files with 103 additions and 3 deletions
@@ -158,6 +158,72 @@ the NAS library, uid-1000 write-through OK, health `status=ok` on the first cycl
(The backlog spec `documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md` is
implemented by this.)
## Boot ordering — the F12 fix (agent v0.85.0, CAMPAIGN-3)
**The bug (CRITICAL):** the `.mount` and `.automount` templates ordered `After=`/`Wants=network-online.target`.
An automount is implicitly `Before=local-fs.target`, and `networking.service` runs after `local-fs`,
so the units closed the cycle `networking → local-fs → automount → network-online → networking`.
systemd breaks such a cycle by deleting an **arbitrary** job in it — on one host boot it deleted
`networking.service` (host up, **no network, for 7 h**), on the next it deleted the automount (network
up, NAS dead). **Every boot of a host with an enrolled network share was a coin flip.**
**The fix:** neither rendered unit carries any network-online ordering. The `.mount` keeps `_netdev`
in `Options` — that is the correct + sufficient network ordering for the REAL mount (systemd classes
a `_netdev` mount under remote-fs.target and orders it after the network). The `.automount` gets NO
network relation at all — a trigger needs none, and it must stay orderable before local-fs without
dragging the network into the transaction.
**Installed-unit migration:** `SudoHostOps.MigrateNetworkUnits` is a general template-drift reconcile
(SHA-256 content compare of each marker-owned unit against a fresh render of its reconstructed spec;
rewrite + one batched `daemon-reload`; idempotent). It runs at agent startup (before the reassert
sweep) and at the head of `EnsureNetworkMount`, so pre-0.85 units carrying the cycle are repaired,
not just future adds. **Live-proven 2026-07-12:** `felhom-pve` rebooted ×2 after the 0.85 deploy —
`journalctl -b | grep -c "ordering cycle"` == **0 on both boots**, networking up, automount armed,
guest + apps healthy. The coin flip is dead.
### Reassert hardening (same release, matrix-corrected)
- **F11 — read the right unit:** the re-arm decision is driven ONLY by the host `/proc/mounts` fstype
at the mountpoint; the `.automount` unit's own state is never consulted (an armed trigger always
reports "active"). **Matrix-corrected live finding (2026-07-12):** a freshly-rebooted guest inherits
neither an idle trigger NOR an *active* host mount — only a FRESH trigger event propagates in. So
skip-active is proactively safe (don't churn shares guests already see) but the caller re-arms
*reactively* when a specific running guest is found blind (`RearmNetworkAutomount` after a failed
`GuestSeesMount`/`GuestSeesPath`).
- **F10 — re-arm for real:** a `.mount`/`.automount` left `failed`/start-limit-hit (unexport →
idle-timeout → repeated access) is `reset-failed` FIRST (new sudoers verb) — without it the
`enable --now` is refused by the start limit and the share stays dead across every boot.
- **F9 — say what you did:** the sweep enumerates by marker-owned unit files and logs an INFO verdict
per share (reasserted / reset-failed+rearmed / skip-active / skip-foreign / error).
- **The hook can never take a guest down:** every guest-hook phase runs recover-wrapped under a hard
timeout and returns cleanly; the installed wrapper snippet no longer `exec`s (`… || true; exit 0`).
### Known limitation — F13 (active NFS under the mp8 bind at guest start)
An **actively-mounted** nfs4 at `/mnt/felhom-drives/<name>` at the moment a guest starts can make
PVE's rbind of `mp8` fail with `rc255` (an unprivileged-container idmap incompatibility), independent
of the felhom hook (which returns 0). The realistic steady state is IDLE (the share idle-unmounts
after `TimeoutIdleSec=60`), where the guest starts clean and the post-start reassert propagates it —
but a share in active use at the reboot moment is a hazard. Deferred to a follow-up (candidate fixes:
a pre-start conditional idle-unmount for single-guest hosts, or an idmapped nfs mount). Related: apps
whose HDD_PATH is on a NAS that is down at guest boot land in docker `Created` state (the deferred
fix-3, boot-time app-start-failure alerting).
## Node self-heal (appliance) — F12-class defense in depth (agent v0.85.0)
`internal/selfheal` is a minimal check/remedy registry gated on `deployment_mode`. One heal ships:
**host networking recovery.** Healthy ⇔ `networking.service` active AND a default route exists; the
remedy — `systemctl start networking.service` (new `FELHOM_SELFHEAL` sudoers verb, ≤3 attempts,
10/30/60 s backoff, terminal give-up logged) — runs ONLY on `deployment_mode:"appliance"`. A **byo**
host runs the check + WARNs but the remedy is structurally unreachable (the Manager gates before any
exec; unit-tested: byo + unhealthy → zero privileged invocations). Absent/unknown mode → byo
(fail-safe). The F12 template fix is the CURE for the ordering instance; this watchdog is the belt for
the CLASS (any boot that leaves networking down, for any reason). **Live-proven 2026-07-12:**
`systemctl stop networking.service` on the appliance → the watchdog WARNed
(`networking.service active=false`), fired the privileged start, and recovered on attempt 1.
`deployment_mode` is set by host-install `--mode appliance|byo` (default byo; the byo config assert
refuses `deployment_mode=appliance`).
## Open items
- QNAP appliance fidelity pass (no emulator) before GA — Synology is now validated.
- The demo's `nas-media` share predates `retry=0`; re-adding re-creates the unit with the current