CAMPAIGN-3 Task A: agent v0.85.0 boot/recovery plane docs + host-install deployment_mode (--mode gates node self-heal)
network-storage-nas.md gains Boot-ordering (F12), reassert-hardening (F11/F10/F9), F13 limitation, and Node self-heal (appliance) sections. host-install v1.14.0 templates deployment_mode from --mode + byo assert. Live matrix: host reboot x2 = 0 ordering-cycle lines (F12 dead); appliance self-heal recovered attempt 1. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
This commit is contained in:
@@ -158,6 +158,72 @@ the NAS library, uid-1000 write-through OK, health `status=ok` on the first cycl
|
||||
(The backlog spec `documentation/backlog/FOLLOWUP-nas-automount-guest-reboot-reassert.md` is
|
||||
implemented by this.)
|
||||
|
||||
## Boot ordering — the F12 fix (agent v0.85.0, CAMPAIGN-3)
|
||||
|
||||
**The bug (CRITICAL):** the `.mount` and `.automount` templates ordered `After=`/`Wants=network-online.target`.
|
||||
An automount is implicitly `Before=local-fs.target`, and `networking.service` runs after `local-fs`,
|
||||
so the units closed the cycle `networking → local-fs → automount → network-online → networking`.
|
||||
systemd breaks such a cycle by deleting an **arbitrary** job in it — on one host boot it deleted
|
||||
`networking.service` (host up, **no network, for 7 h**), on the next it deleted the automount (network
|
||||
up, NAS dead). **Every boot of a host with an enrolled network share was a coin flip.**
|
||||
|
||||
**The fix:** neither rendered unit carries any network-online ordering. The `.mount` keeps `_netdev`
|
||||
in `Options` — that is the correct + sufficient network ordering for the REAL mount (systemd classes
|
||||
a `_netdev` mount under remote-fs.target and orders it after the network). The `.automount` gets NO
|
||||
network relation at all — a trigger needs none, and it must stay orderable before local-fs without
|
||||
dragging the network into the transaction.
|
||||
|
||||
**Installed-unit migration:** `SudoHostOps.MigrateNetworkUnits` is a general template-drift reconcile
|
||||
(SHA-256 content compare of each marker-owned unit against a fresh render of its reconstructed spec;
|
||||
rewrite + one batched `daemon-reload`; idempotent). It runs at agent startup (before the reassert
|
||||
sweep) and at the head of `EnsureNetworkMount`, so pre-0.85 units carrying the cycle are repaired,
|
||||
not just future adds. **Live-proven 2026-07-12:** `felhom-pve` rebooted ×2 after the 0.85 deploy —
|
||||
`journalctl -b | grep -c "ordering cycle"` == **0 on both boots**, networking up, automount armed,
|
||||
guest + apps healthy. The coin flip is dead.
|
||||
|
||||
### Reassert hardening (same release, matrix-corrected)
|
||||
|
||||
- **F11 — read the right unit:** the re-arm decision is driven ONLY by the host `/proc/mounts` fstype
|
||||
at the mountpoint; the `.automount` unit's own state is never consulted (an armed trigger always
|
||||
reports "active"). **Matrix-corrected live finding (2026-07-12):** a freshly-rebooted guest inherits
|
||||
neither an idle trigger NOR an *active* host mount — only a FRESH trigger event propagates in. So
|
||||
skip-active is proactively safe (don't churn shares guests already see) but the caller re-arms
|
||||
*reactively* when a specific running guest is found blind (`RearmNetworkAutomount` after a failed
|
||||
`GuestSeesMount`/`GuestSeesPath`).
|
||||
- **F10 — re-arm for real:** a `.mount`/`.automount` left `failed`/start-limit-hit (unexport →
|
||||
idle-timeout → repeated access) is `reset-failed` FIRST (new sudoers verb) — without it the
|
||||
`enable --now` is refused by the start limit and the share stays dead across every boot.
|
||||
- **F9 — say what you did:** the sweep enumerates by marker-owned unit files and logs an INFO verdict
|
||||
per share (reasserted / reset-failed+rearmed / skip-active / skip-foreign / error).
|
||||
- **The hook can never take a guest down:** every guest-hook phase runs recover-wrapped under a hard
|
||||
timeout and returns cleanly; the installed wrapper snippet no longer `exec`s (`… || true; exit 0`).
|
||||
|
||||
### Known limitation — F13 (active NFS under the mp8 bind at guest start)
|
||||
|
||||
An **actively-mounted** nfs4 at `/mnt/felhom-drives/<name>` at the moment a guest starts can make
|
||||
PVE's rbind of `mp8` fail with `rc255` (an unprivileged-container idmap incompatibility), independent
|
||||
of the felhom hook (which returns 0). The realistic steady state is IDLE (the share idle-unmounts
|
||||
after `TimeoutIdleSec=60`), where the guest starts clean and the post-start reassert propagates it —
|
||||
but a share in active use at the reboot moment is a hazard. Deferred to a follow-up (candidate fixes:
|
||||
a pre-start conditional idle-unmount for single-guest hosts, or an idmapped nfs mount). Related: apps
|
||||
whose HDD_PATH is on a NAS that is down at guest boot land in docker `Created` state (the deferred
|
||||
fix-3, boot-time app-start-failure alerting).
|
||||
|
||||
## Node self-heal (appliance) — F12-class defense in depth (agent v0.85.0)
|
||||
|
||||
`internal/selfheal` is a minimal check/remedy registry gated on `deployment_mode`. One heal ships:
|
||||
**host networking recovery.** Healthy ⇔ `networking.service` active AND a default route exists; the
|
||||
remedy — `systemctl start networking.service` (new `FELHOM_SELFHEAL` sudoers verb, ≤3 attempts,
|
||||
10/30/60 s backoff, terminal give-up logged) — runs ONLY on `deployment_mode:"appliance"`. A **byo**
|
||||
host runs the check + WARNs but the remedy is structurally unreachable (the Manager gates before any
|
||||
exec; unit-tested: byo + unhealthy → zero privileged invocations). Absent/unknown mode → byo
|
||||
(fail-safe). The F12 template fix is the CURE for the ordering instance; this watchdog is the belt for
|
||||
the CLASS (any boot that leaves networking down, for any reason). **Live-proven 2026-07-12:**
|
||||
`systemctl stop networking.service` on the appliance → the watchdog WARNed
|
||||
(`networking.service active=false`), fired the privileged start, and recovered on attempt 1.
|
||||
`deployment_mode` is set by host-install `--mode appliance|byo` (default byo; the byo config assert
|
||||
refuses `deployment_mode=appliance`).
|
||||
|
||||
## Open items
|
||||
- QNAP appliance fidelity pass (no emulator) before GA — Synology is now validated.
|
||||
- The demo's `nas-media` share predates `retry=0`; re-adding re-creates the unit with the current
|
||||
|
||||
Reference in New Issue
Block a user