felhom.eu/documentation/runbooks/break-glass.md

RUNBOOK — Break-glass: management-plane recovery (privsep watchdog + vaulted console credential)

What this covers: how a Felhom host stays reachable when its SSH management plane breaks, and how the operator recovers if it does anyway. Delivered by TASK G1 (felhom-agent v0.71.0 + hub v0.34.0 + host-install). Design provenance: documentation/audits/SPIKE-felhom-sshd-2026-07-05.md §8 (the incident) + #9 (the prerequisite). This is the blocker the felhom-sshd feature (H1) depends on — H1 may assume /run/sshd is guaranteed present and must never declare RuntimeDirectory=.

1. The failure (know the fingerprint)

/run/sshd is OpenSSH's compiled-in privilege-separation directory, SHARED by every sshd on the host. If it goes missing, sshd's listener stays up (TCP connects, the version banner exchanges) but every new session RESETS right after SSH2_MSG_KEXINIT. The box is healthy underneath (guests, PVE web, agent all fine) but remotely unreachable over SSH.

Diagnostic (from any client):

ssh -vv root@<host> 2>&1 | grep -A1 KEXINIT
# ... "SSH2_MSG_KEXINIT sent" immediately followed by "Connection reset by <host> port 22" = /run/sshd missing

The spike caused this live via a second sshd's RuntimeDirectory=sshd cleanup removing the shared dir.

2. The three layers (defense in depth)

Layer	Artifact	What it does	Failure domain it survives
1 — prevent	/etc/tmpfiles.d/felhom-privsep.conf (d /run/sshd 0755 root root -)	/run/sshd is boot-persistent, owned by no unit's lifecycle → the known cause can't recur	reboots, unit-cleanup edges
2 — auto-heal (no login)	felhom-mgmt-watchdog.timer → .service → /usr/local/sbin/felhom-mgmt-watchdog (~60s)	recreates a missing /run/sshd, reset-faileds a start-limited sshd, writes a heal-marker. Agent-INDEPENDENT — heals with felhom-agent stopped	a novel runtime cause; the agent being down
3 — break-glass (human)	root@pam password, hub-vaulted at day-0	operator reaches the PVE web console (pveproxy :8006 — a daemon/port/codebase distinct from sshd) and runs the one-line fix	both the sshd path AND the auto-heal failing

Visibility: the agent's heartbeat carries a mgmt_plane stanza (privsep_dir_ok, sshd_reachable, healed_recently, privsep_healed_at); the hub raises a mgmt_plane_healed warning on each new auto-heal so a recurring clobber is investigated before it becomes a lockout.

3. Recovery — do this in order

3.0 First: is it even broken? (it usually self-heals)

Layer 2 heals within ~60s with nobody logged in. If you just saw the KEXINIT-reset, wait one minute and retry SSH. If it's back, check the hub for a mgmt_plane_healed warning and go to §4 (find the cause).

3.1 If SSH is still dead — reach the PVE web console (break-glass)

1. Retrieve the vaulted root@pam password (operator/global key):

   curl -s -H "Authorization: Bearer <GLOBAL-KEY out-of-band>" \
     https://hub.felhom.eu/api/v1/admin/hosts/<host_id>/recovery-credential
   # → {"host_id":"…","username":"root@pam","password":"…","set_at":"…"}
   

   (A per-host key is refused here by design — only the operator key reads a console password back.)
2. Open the PVE web UI https://<host-ip>:8006, log in root@pam + that password, realm Linux PAM standard authentication.
3. Node → Shell (xterm.js root console). Or navigate directly: https://<host-ip>:8006/?console=shell&xtermjs=1&node=<node>&resize=off
4. Run the one-line fix:

   mkdir -p /run/sshd && chmod 0755 /run/sshd
   

   SSH on :22 is restored instantly — no sshd restart needed (the listener never died; PID is unchanged). Verify: ls -ld /run/sshd and a fresh ssh root@<host>.
5. If the stock sshd is stuck failed from a start-limit: systemctl reset-failed ssh && systemctl start ssh.

3.2 If you have ANY root shell on the box

Just run the §3.1 step 4 one-liner. That's the whole fix. The watchdog will also keep /run/sshd present from then on (layer 2), and the tmpfiles entry restores it on the next boot (layer 1).

4. After recovery — find the recurring cause

A single auto-heal is noise; a repeating mgmt_plane_healed warning means something keeps removing /run/sshd. Look for:

• a systemd unit declaring RuntimeDirectory=sshd (the incident cause) — grep -rl RuntimeDirectory /etc/systemd/system /run/systemd; a second sshd unit must NEVER declare it (H1's constraint).
• a cleanup/upgrade job that rms /run/sshd. Fix the cause; the three layers are the safety net, not the cure.

5. Operator hygiene / standing notes

• Keep the build-server PVE token fresh. In the spike incident the build-server-cached PVE token (kisfenyo@192.168.0.180:~/felhom-agent-demo.json) was stale (401) and left no non-SSH lever before the web-console path was used. A working break-glass path independent of sshd must always exist.
• Rotate the console credential with felhom-host-install … --rotate-recovery (regenerates + re-vaults; a plain re-run does NOT rotate, to avoid stranding your saved copy). Full auto-rotation is a future item.
• Operator-set FIXED credential (e.g. demo-felhom-01, pinned 2026-07-12). A host's root@pam may be deliberately set to a known operator-chosen password instead of the random day-0 one. Do it the same way step 4b does, just with your password: set it on the box (printf 'root:<pw>\n' | chpasswd), then re-vault with PUT /api/v1/hosts/<host_id>/recovery-credential (host api_key from /etc/felhom-agent/agent.json; expect HTTP 200), and ensure break_glass is in /var/lib/felhom-install/state.json's completed[]. This survives plain installer re-runs (step 4b skips on the marker). Two caveats: (a) never --rotate-recovery such a host — it replaces your known copy with a random one; (b) a full from-scratch reinstall (uninstall→install) wipes state.json, so step 4b re-randomizes — re-run the set-and-vault afterward. Background: on each full reinstall this is exactly why root@pam "kept changing" until it was pinned.
• The password is never logged or filed — it exists only in the hub vault (retrieve as in §3.1).
• Least-privilege console user instead of root@pam was probed as a non-blocking observation: PVE has historically gated the node Shell to root@pam, so G1 ships the proven root@pam path. Revisit if PVE later allows a scoped realm user to open a node Shell.

6. Cross-references

• documentation/architecture/06-offsite-connectivity.md (the OOB transport this complements).
• documentation/audits/SPIKE-felhom-sshd-2026-07-05.md §8/#9 (the incident + prerequisite).
• felhom-agent internal/mgmtplane/ + configs/felhom-mgmt-watchdog.* + configs/felhom-privsep.tmpfiles.
• hub internal/monitor/host_mgmtplane.go + internal/store/host_recovery.go.