felhom.eu/documentation/runbooks/break-glass.md

# RUNBOOK — Break-glass: management-plane recovery (privsep watchdog + vaulted console credential)

> **What this covers:** how a Felhom host stays reachable when its SSH management plane breaks, and
> how the operator recovers if it does anyway. Delivered by **TASK G1** (felhom-agent v0.71.0 +
> hub v0.34.0 + host-install). Design provenance: `documentation/audits/SPIKE-felhom-sshd-2026-07-05.md`
> §8 (the incident) + #9 (the prerequisite). This is the **blocker** the felhom-sshd feature (H1)
> depends on — H1 may assume `/run/sshd` is guaranteed present and must never declare `RuntimeDirectory=`.

## 1. The failure (know the fingerprint)

`/run/sshd` is OpenSSH's compiled-in **privilege-separation directory**, SHARED by every sshd on the
host. If it goes missing, sshd's **listener stays up** (TCP connects, the version banner exchanges) but
every new session **RESETS right after `SSH2_MSG_KEXINIT`**. The box is healthy underneath (guests,
PVE web, agent all fine) but **remotely unreachable over SSH**.

**Diagnostic (from any client):**
```
ssh -vv root@<host> 2>&1 | grep -A1 KEXINIT
# ... "SSH2_MSG_KEXINIT sent" immediately followed by "Connection reset by <host> port 22" = /run/sshd missing
```
The spike caused this live via a second sshd's `RuntimeDirectory=sshd` cleanup removing the shared dir.

## 2. The three layers (defense in depth)

| Layer | Artifact | What it does | Failure domain it survives |
|---|---|---|---|
| 1 — **prevent** | `/etc/tmpfiles.d/felhom-privsep.conf` (`d /run/sshd 0755 root root -`) | `/run/sshd` is boot-persistent, owned by no unit's lifecycle → the known cause can't recur | reboots, unit-cleanup edges |
| 2 — **auto-heal (no login)** | `felhom-mgmt-watchdog.timer` → `.service` → `/usr/local/sbin/felhom-mgmt-watchdog` (~60s) | recreates a missing `/run/sshd`, `reset-failed`s a start-limited sshd, writes a heal-marker. **Agent-INDEPENDENT** — heals with felhom-agent stopped | a novel runtime cause; the agent being down |
| 3 — **break-glass (human)** | root@pam password, hub-vaulted at day-0 | operator reaches the **PVE web console** (pveproxy :8006 — a daemon/port/codebase distinct from sshd) and runs the one-line fix | both the sshd path AND the auto-heal failing |

Visibility: the agent's heartbeat carries a `mgmt_plane` stanza (`privsep_dir_ok`, `sshd_reachable`,
`healed_recently`, `privsep_healed_at`); the hub raises a **`mgmt_plane_healed` warning** on each new
auto-heal so a **recurring** clobber is investigated before it becomes a lockout.

## 3. Recovery — do this in order

### 3.0 First: is it even broken? (it usually self-heals)
Layer 2 heals within ~60s with nobody logged in. If you just saw the KEXINIT-reset, wait one minute and
retry SSH. If it's back, check the hub for a `mgmt_plane_healed` warning and go to §4 (find the cause).

### 3.1 If SSH is still dead — reach the PVE web console (break-glass)
1. **Retrieve the vaulted root@pam password** (operator/global key):
   ```
   curl -s -H "Authorization: Bearer <GLOBAL-KEY out-of-band>" \
     https://hub.felhom.eu/api/v1/admin/hosts/<host_id>/recovery-credential
   # → {"host_id":"…","username":"root@pam","password":"…","set_at":"…"}
   ```
   (A per-host key is refused here by design — only the operator key reads a console password back.)
2. **Open the PVE web UI** `https://<host-ip>:8006`, log in `root@pam` + that password, realm
   *Linux PAM standard authentication*.
3. **Node → Shell** (xterm.js root console). Or navigate directly:
   `https://<host-ip>:8006/?console=shell&xtermjs=1&node=<node>&resize=off`
4. **Run the one-line fix:**
   ```
   mkdir -p /run/sshd && chmod 0755 /run/sshd
   ```
   SSH on :22 is restored **instantly** — no sshd restart needed (the listener never died; PID is
   unchanged). Verify: `ls -ld /run/sshd` and a fresh `ssh root@<host>`.
5. If the stock sshd is stuck `failed` from a start-limit: `systemctl reset-failed ssh && systemctl start ssh`.

### 3.2 If you have ANY root shell on the box
Just run the §3.1 step 4 one-liner. That's the whole fix. The watchdog will also keep `/run/sshd`
present from then on (layer 2), and the tmpfiles entry restores it on the next boot (layer 1).

## 4. After recovery — find the recurring cause
A single auto-heal is noise; a **repeating** `mgmt_plane_healed` warning means something keeps removing
`/run/sshd`. Look for:
- a systemd unit declaring **`RuntimeDirectory=sshd`** (the incident cause) — `grep -rl RuntimeDirectory /etc/systemd/system /run/systemd`; a second sshd unit must NEVER declare it (H1's constraint).
- a cleanup/upgrade job that `rm`s `/run/sshd`.
Fix the cause; the three layers are the safety net, not the cure.

## 5. Operator hygiene / standing notes
- **Keep the build-server PVE token fresh.** In the spike incident the build-server-cached PVE token
  (`kisfenyo@192.168.0.180:~/felhom-agent-demo.json`) was **stale (401)** and left no non-SSH lever
  before the web-console path was used. A working break-glass path independent of sshd must always exist.
- **Rotate the console credential** with `felhom-host-install … --rotate-recovery` (regenerates +
  re-vaults; a plain re-run does NOT rotate, to avoid stranding your saved copy). Full auto-rotation is
  a future item.
- **The password is never logged or filed** — it exists only in the hub vault (retrieve as in §3.1).
- **Least-privilege console user** instead of root@pam was probed as a non-blocking observation: PVE
  has historically gated the node Shell to `root@pam`, so G1 ships the proven root@pam path. Revisit if
  PVE later allows a scoped realm user to open a node Shell.

## 6. Cross-references
- `documentation/architecture/06-offsite-connectivity.md` (the OOB transport this complements).
- `documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8/#9 (the incident + prerequisite).
- felhom-agent `internal/mgmtplane/` + `configs/felhom-mgmt-watchdog.*` + `configs/felhom-privsep.tmpfiles`.
- hub `internal/monitor/host_mgmtplane.go` + `internal/store/host_recovery.go`.