felhom.eu/documentation/runbooks/break-glass.md

RUNBOOK — Break-glass: management-plane recovery (privsep watchdog + vaulted console credential)

What this covers: how a Felhom host stays reachable when its SSH management plane breaks, and how the operator recovers if it does anyway. Delivered by TASK G1 (felhom-agent v0.71.0 + hub v0.34.0 + host-install). Design provenance: documentation/audits/SPIKE-felhom-sshd-2026-07-05.md §8 (the incident) + #9 (the prerequisite). This is the blocker the felhom-sshd feature (H1) depends on — H1 may assume /run/sshd is guaranteed present and must never declare RuntimeDirectory=.

1. The failure (know the fingerprint)

/run/sshd is OpenSSH's compiled-in privilege-separation directory, SHARED by every sshd on the host. If it goes missing, sshd's listener stays up (TCP connects, the version banner exchanges) but every new session RESETS right after SSH2_MSG_KEXINIT. The box is healthy underneath (guests, PVE web, agent all fine) but remotely unreachable over SSH.

Diagnostic (from any client):

ssh -vv root@<host> 2>&1 | grep -A1 KEXINIT
# ... "SSH2_MSG_KEXINIT sent" immediately followed by "Connection reset by <host> port 22" = /run/sshd missing

The spike caused this live via a second sshd's RuntimeDirectory=sshd cleanup removing the shared dir.

2. The three layers (defense in depth)

Layer	Artifact	What it does	Failure domain it survives
1 — prevent	/etc/tmpfiles.d/felhom-privsep.conf (d /run/sshd 0755 root root -)	/run/sshd is boot-persistent, owned by no unit's lifecycle → the known cause can't recur	reboots, unit-cleanup edges
2 — auto-heal (no login)	felhom-mgmt-watchdog.timer → .service → /usr/local/sbin/felhom-mgmt-watchdog (~60s)	recreates a missing /run/sshd, reset-faileds a start-limited sshd, writes a heal-marker. Agent-INDEPENDENT — heals with felhom-agent stopped	a novel runtime cause; the agent being down
3 — break-glass (human)	root@pam password, hub-vaulted at day-0	operator reaches the PVE web console (pveproxy :8006 — a daemon/port/codebase distinct from sshd) and runs the one-line fix	both the sshd path AND the auto-heal failing

Visibility: the agent's heartbeat carries a mgmt_plane stanza (privsep_dir_ok, sshd_reachable, healed_recently, privsep_healed_at); the hub raises a mgmt_plane_healed warning on each new auto-heal so a recurring clobber is investigated before it becomes a lockout.

3. Recovery — do this in order

3.0 First: is it even broken? (it usually self-heals)

Layer 2 heals within ~60s with nobody logged in. If you just saw the KEXINIT-reset, wait one minute and retry SSH. If it's back, check the hub for a mgmt_plane_healed warning and go to §4 (find the cause).

3.1 If SSH is still dead — reach the PVE web console (break-glass)

1. Retrieve the vaulted root@pam password.

   Primary route — the hub UI (hub v0.84.0): hub → Hosts → the host → the Console access card → Reveal. The password shows for 60 s with a Copy button. The card also states which user it is for and when it was last vaulted, so a stale copy is visible before you try it. A host with nothing vaulted says so and says why, instead of showing a blank. Needs only the hub login password — no out-of-band key.

   Fallback — the global-key curl. Use this when the hub UI itself is unavailable (that is the whole reason it still exists, and why it deliberately has no dependency on the hub's session layer):

   curl -s -H "Authorization: Bearer <GLOBAL-KEY out-of-band>" \
     https://hub.felhom.eu/api/v1/admin/hosts/<host_id>/recovery-credential
   # → {"host_id":"…","username":"root@pam","password":"…","set_at":"…"}
   

   (A per-host key is refused here by design — only the operator key reads a console password back.)

   Either route is an operator action against a customer's box, and the UI route records it: see §5.

2. Open the PVE web UI https://<host-ip>:8006, log in root@pam + that password, realm Linux PAM standard authentication.

3. Node → Shell (xterm.js root console). Or navigate directly: https://<host-ip>:8006/?console=shell&xtermjs=1&node=<node>&resize=off

4. Run the one-line fix:

   mkdir -p /run/sshd && chmod 0755 /run/sshd
   

   SSH on :22 is restored instantly — no sshd restart needed (the listener never died; PID is unchanged). Verify: ls -ld /run/sshd and a fresh ssh root@<host>.

5. If the stock sshd is stuck failed from a start-limit: systemctl reset-failed ssh && systemctl start ssh.

3.2 If you have ANY root shell on the box

Just run the §3.1 step 4 one-liner. That's the whole fix. The watchdog will also keep /run/sshd present from then on (layer 2), and the tmpfiles entry restores it on the next boot (layer 1).

4. After recovery — find the recurring cause

A single auto-heal is noise; a repeating mgmt_plane_healed warning means something keeps removing /run/sshd. Look for:

• a systemd unit declaring RuntimeDirectory=sshd (the incident cause) — grep -rl RuntimeDirectory /etc/systemd/system /run/systemd; a second sshd unit must NEVER declare it (H1's constraint).
• a cleanup/upgrade job that rms /run/sshd. Fix the cause; the three layers are the safety net, not the cure.

5. Operator hygiene / standing notes

• Keep the build-server PVE token fresh. In the spike incident the build-server-cached PVE token (kisfenyo@192.168.0.180:~/felhom-agent-demo.json) was stale (401) and left no non-SSH lever before the web-console path was used. A working break-glass path independent of sshd must always exist.
• Rotate the console credential ALONE with sudo bash felhom-host-install.sh --rotate-recovery-only (R-1141; try it first with --dry-run). It runs step 4b and nothing else: a new random root@pam password, VAULTED at the hub FIRST and only then set with chpasswd, then checked with a PVE login (R-1145, 2026-10-18 — a refused vault changes nothing), with the host key from the agent's own config (/etc/felhom-agent/agent.json). It does NOT reinstall or change the agent, its config or the root files, needs no passphrase or customer id, checks that the hub answers (/healthz 200) before the password changes, and refuses a byo host (also an agent config whose deployment_mode is not appliance). Proof after a rotation: the new vaulted value gets a PVE ticket (200), the old one gets 401. The flag ships with the first installer-v… tag after installer 1.33.0 — until then the served installer does not know it. The older --rotate-recovery flag only works inside a FULL installer re-run, which also reinstalls the vouched agent and the root files (on a box ahead of the vouched agent it moves the agent back) — do not use it for a password change. A plain re-run does NOT rotate, to avoid stranding your saved copy. Full auto-rotation is a future item.
• Operator-set FIXED credential (e.g. demo-felhom-01, pinned 2026-07-12). A host's root@pam may be deliberately set to a known operator-chosen password instead of the random day-0 one. Do it the same way step 4b does, just with your password: set it on the box (printf 'root:<pw>\n' | chpasswd), then re-vault with PUT /api/v1/hosts/<host_id>/recovery-credential (host api_key from /etc/felhom-agent/agent.json; expect HTTP 200), and ensure break_glass is in /var/lib/felhom-install/state.json's completed[]. This survives plain installer re-runs (step 4b skips on the marker). Two caveats: (a) never --rotate-recovery or --rotate-recovery-only such a host — it replaces your known copy with a random one; (b) a full from-scratch reinstall (uninstall→install) wipes state.json, so step 4b re-randomizes — re-run the set-and-vault afterward. Background: on each full reinstall this is exactly why root@pam "kept changing" until it was pinned.
• The password is never logged or filed — it exists only in the hub vault (retrieve as in §3.1).
• Revealing it through the hub UI writes a customer-visible event line (hub v0.84.0): recovery_credential_revealed, info severity, on that host's customer timeline, in Hungarian — „Az üzemeltető lekérte a géped konzolos hozzáférési jelszavát (távoli hibaelhárítás)." Nobody is emailed (no dispatcher call), and the register records accesses, so two reveals write two lines. Deliberate, for a product sold on data sovereignty; it mirrors how a remote log pull is already recorded. An unbound host has no customer to tell — there the hub [INFO] log line is the only record. The global-key curl path writes no event.
• The hub login password now unlocks console root on every managed box. Before v0.84.0, retrieval also required the global API key. Accepted for a single-operator, HU-geo-fenced hub that already holds these passwords in plaintext at rest — a convenience trade, and a real one. The plaintext-at-rest half (every hub DB backup is a fleet-wide console-credential dump) is R-133.
• Least-privilege console user instead of root@pam was probed as a non-blocking observation: PVE has historically gated the node Shell to root@pam, so G1 ships the proven root@pam path. Revisit if PVE later allows a scoped realm user to open a node Shell.

6. Cross-references

• documentation/architecture/06-offsite-connectivity.md (the OOB transport this complements).
• documentation/audits/SPIKE-felhom-sshd-2026-07-05.md §8/#9 (the incident + prerequisite).
• felhom-agent internal/mgmtplane/ + configs/felhom-mgmt-watchdog.* + configs/felhom-privsep.tmpfiles.
• hub internal/monitor/host_mgmtplane.go + internal/store/host_recovery.go.