feat(mgmtplane): break-glass privsep-dir watchdog + mgmt_plane health (TASK G1) — v0.71.0
Prerequisite for felhom-sshd (H1). Closes the SPIKE-felhom-sshd §8 lockout: a second sshd's RuntimeDirectory=sshd removed the SHARED /run/sshd privsep dir and took stock sshd on :22 down (sessions reset after KEXINIT). Host artifacts (configs/, installed by felhom-host-install): - felhom-privsep.tmpfiles: layer 1, boot-persistent /run/sshd owned by no unit - felhom-mgmt-watchdog.sh/.service/.timer: layer 2, AGENT-INDEPENDENT ~60s heal (stat-first recreate + reset-failed sshd only if failed + heal-marker); never RuntimeDirectory=, never restarts stock sshd, never touches a healthy dir. Go (internal/mgmtplane): read-only Reporter → additive omitempty mgmt_plane heartbeat stanza (privsep_dir_ok/sshd_reachable/healed_recently/privsep_healed_at), wired via Collector.SetMgmtPlaneReporter. Non-hollow tests + red-proofs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -0,0 +1,19 @@
|
||||
# felhom-mgmt-watchdog.service (TASK G1) — install as /etc/systemd/system/felhom-mgmt-watchdog.service.
|
||||
#
|
||||
# LAYER 2 (auto-heal without login): a DUMB oneshot that runs /usr/local/sbin/felhom-mgmt-watchdog,
|
||||
# triggered every ~60s by felhom-mgmt-watchdog.timer. It recreates a missing /run/sshd and clears a
|
||||
# start-limited stock sshd — so a management-plane lockout self-corrects in ≤1 tick with NOBODY logged
|
||||
# in and WITHOUT the felhom-agent binary running (the agent only OBSERVES/reports; the heal must not
|
||||
# depend on it — SPIKE-felhom-sshd-2026-07-05 finding #9 / TASK G1 trap 1).
|
||||
#
|
||||
# CRITICAL: this unit MUST NOT declare `RuntimeDirectory=` — that directive (with value `sshd`) is the
|
||||
# incident it exists to fix. It writes only to /run (dir + heal marker) as root; no state dir needed.
|
||||
[Unit]
|
||||
Description=Felhom management-plane watchdog (privsep-dir auto-heal; agent-independent)
|
||||
# Ordering only — the timer drives cadence; no hard dependency so it runs even in degraded boots.
|
||||
After=systemd-tmpfiles-setup.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/felhom-mgmt-watchdog
|
||||
# Deliberately NO RuntimeDirectory=, NO PrivateTmp, NO sandboxing that would hide /run/sshd from it.
|
||||
@@ -0,0 +1,68 @@
|
||||
#!/bin/sh
|
||||
# felhom-mgmt-watchdog — the DUMB, agent-INDEPENDENT management-plane healer (TASK G1).
|
||||
#
|
||||
# WHY THIS EXISTS: /run/sshd is OpenSSH's compiled-in privilege-separation directory, SHARED by
|
||||
# every sshd on the host. If it goes missing, sshd's listener stays up but every new session RESETS
|
||||
# right after SSH2_MSG_KEXINIT — a management-plane lockout on an otherwise-healthy box
|
||||
# (SPIKE-felhom-sshd-2026-07-05 §8, caused live by a second unit's `RuntimeDirectory=sshd` cleanup).
|
||||
# The tmpfiles.d entry (felhom-privsep.tmpfiles) PREVENTS the known cause; this script AUTO-HEALS any
|
||||
# novel cause with ZERO login and ZERO dependency on the felhom-agent binary (so it self-corrects even
|
||||
# when the agent is down — the whole point). It is run by felhom-mgmt-watchdog.timer every ~60s.
|
||||
#
|
||||
# CONTRACT (all four are load-bearing):
|
||||
# 1. STAT-FIRST / IDEMPOTENT — a HEALTHY /run/sshd is NEVER touched (no mkdir/chmod/chown call), so
|
||||
# steady state is zero mutation and the dir's mtime is preserved. Only a MISSING or
|
||||
# wrong-mode/owner dir is corrected.
|
||||
# 2. reset-failed the STOCK sshd ONLY when it is in the `failed` state (a start-limit lockout) —
|
||||
# never otherwise, and NEVER `restart` it (mkdir alone restores it; PID stays, spike-proven).
|
||||
# 3. HEAL MARKER — writes an RFC3339 UTC timestamp to $MARKER only on a real heal, so the agent can
|
||||
# REPORT the condition to the hub (a recurring clobber must surface BEFORE it becomes a lockout).
|
||||
# 4. NO `RuntimeDirectory=` anywhere in this feature (that IS the incident cause) — the unit that
|
||||
# runs this script must not declare one either.
|
||||
#
|
||||
# Touches host /run + the stock sshd unit ONLY — no guests (pool-scoping is moot here). Fail-safe:
|
||||
# any single step failing is logged and does not abort the others (`|| true`); the next tick retries.
|
||||
|
||||
set -u
|
||||
|
||||
PRIVSEP=/run/sshd
|
||||
MARKER=/run/felhom-mgmt-watchdog.healed
|
||||
SSHD_UNIT=ssh.service # Debian/PVE stock sshd unit name (sshd.service is an alias)
|
||||
healed=0
|
||||
|
||||
log() { logger -t felhom-mgmt-watchdog "$*" 2>/dev/null || true; }
|
||||
|
||||
# 1. Privsep dir — stat-first: create only when missing; correct mode/owner only when wrong.
|
||||
if [ ! -d "$PRIVSEP" ]; then
|
||||
if mkdir -p "$PRIVSEP" && chown root:root "$PRIVSEP" && chmod 0755 "$PRIVSEP"; then
|
||||
healed=1
|
||||
log "recreated missing privsep dir $PRIVSEP (0755 root:root)"
|
||||
else
|
||||
log "ERROR: failed to recreate $PRIVSEP"
|
||||
fi
|
||||
else
|
||||
mode=$(stat -c %a "$PRIVSEP" 2>/dev/null || echo "")
|
||||
owner=$(stat -c %U:%G "$PRIVSEP" 2>/dev/null || echo "")
|
||||
if [ "$mode" != "755" ]; then
|
||||
chmod 0755 "$PRIVSEP" && healed=1 && log "corrected $PRIVSEP mode ($mode -> 755)"
|
||||
fi
|
||||
if [ "$owner" != "root:root" ]; then
|
||||
chown root:root "$PRIVSEP" && healed=1 && log "corrected $PRIVSEP owner ($owner -> root:root)"
|
||||
fi
|
||||
fi
|
||||
|
||||
# 2. Stock sshd — clear a start-limit lockout ONLY when the unit is genuinely `failed`. Never restart
|
||||
# it (unnecessary — a recreated privsep dir is picked up by the next fork; the listener never died).
|
||||
if systemctl is-failed --quiet "$SSHD_UNIT" 2>/dev/null; then
|
||||
if systemctl reset-failed "$SSHD_UNIT" 2>/dev/null; then
|
||||
healed=1
|
||||
log "reset-failed $SSHD_UNIT (was in the failed state)"
|
||||
fi
|
||||
fi
|
||||
|
||||
# 3. Heal marker — only on a real heal, so the agent reports the condition (hub raises a warning).
|
||||
if [ "$healed" = "1" ]; then
|
||||
date -u +%Y-%m-%dT%H:%M:%SZ > "$MARKER" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
exit 0
|
||||
@@ -0,0 +1,18 @@
|
||||
# felhom-mgmt-watchdog.timer (TASK G1) — install as /etc/systemd/system/felhom-mgmt-watchdog.timer.
|
||||
#
|
||||
# Drives felhom-mgmt-watchdog.service on a ~60s cadence (the auto-heal tick). OnBootSec fires shortly
|
||||
# after boot (belt-and-suspenders with the tmpfiles layer); OnUnitActiveSec=60s gives the ≤1-tick
|
||||
# heal budget the G1 acceptance drill measures. Persistent=true runs a missed tick immediately after a
|
||||
# resume/late boot. The oneshot is idempotent, so a fast cadence never churns a healthy host.
|
||||
[Unit]
|
||||
Description=Felhom management-plane watchdog timer (~60s privsep-dir auto-heal tick)
|
||||
|
||||
[Timer]
|
||||
OnBootSec=30s
|
||||
OnUnitActiveSec=60s
|
||||
AccuracySec=5s
|
||||
Persistent=true
|
||||
Unit=felhom-mgmt-watchdog.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,15 @@
|
||||
# felhom privsep-dir persistence (TASK G1) — install as /etc/tmpfiles.d/felhom-privsep.conf.
|
||||
#
|
||||
# LAYER 1 (prevent): make OpenSSH's shared privilege-separation directory /run/sshd boot-persistent
|
||||
# and owned by NO systemd unit's lifecycle. This closes the exact incident cause from
|
||||
# SPIKE-felhom-sshd-2026-07-05 §8: a second sshd unit declaring `RuntimeDirectory=sshd` had systemd
|
||||
# REMOVE the shared /run/sshd on that unit's stop/failure, taking the stock sshd on :22 down with it
|
||||
# (sessions reset right after SSH2_MSG_KEXINIT). A tmpfiles.d entry recreates the dir at every boot
|
||||
# independently of any unit, so no unit's RuntimeDirectory cleanup can be the sole owner.
|
||||
#
|
||||
# systemd-tmpfiles is idempotent: `systemd-tmpfiles --create` re-run is a no-op on an existing,
|
||||
# correct dir (it only creates/fixes, never churns). Complemented at runtime by the
|
||||
# felhom-mgmt-watchdog timer (layer 2), which re-heals a dir removed AFTER boot.
|
||||
#
|
||||
# Type d = create the directory if absent (leaves an existing one, only fixing mode/owner).
|
||||
d /run/sshd 0755 root root -
|
||||
Reference in New Issue
Block a user