# RUNBOOK — Break-glass: management-plane recovery (privsep watchdog + vaulted console credential)

> **What this covers:** how a Felhom host stays reachable when its SSH management plane breaks, and
> how the operator recovers if it does anyway. Delivered by **TASK G1** (felhom-agent v0.71.0 +
> hub v0.34.0 + host-install). Design provenance: `documentation/audits/SPIKE-felhom-sshd-2026-07-05.md`
> §8 (the incident) + #9 (the prerequisite). This is the **blocker** the felhom-sshd feature (H1)
> depends on — H1 may assume `/run/sshd` is guaranteed present and must never declare `RuntimeDirectory=`.

## 1. The failure (know the fingerprint)

`/run/sshd` is OpenSSH's compiled-in **privilege-separation directory**, SHARED by every sshd on the
host. If it goes missing, sshd's **listener stays up** (TCP connects, the version banner exchanges) but
every new session **RESETS right after `SSH2_MSG_KEXINIT`**. The box is healthy underneath (guests,
PVE web, agent all fine) but **remotely unreachable over SSH**.

**Diagnostic (from any client):**
```
ssh -vv root@<host> 2>&1 | grep -A1 KEXINIT
# ... "SSH2_MSG_KEXINIT sent" immediately followed by "Connection reset by <host> port 22" = /run/sshd missing
```
The spike caused this live via a second sshd's `RuntimeDirectory=sshd` cleanup removing the shared dir.

## 2. The three layers (defense in depth)

| Layer | Artifact | What it does | Failure domain it survives |
|---|---|---|---|
| 1 — **prevent** | `/etc/tmpfiles.d/felhom-privsep.conf` (`d /run/sshd 0755 root root -`) | `/run/sshd` is boot-persistent, owned by no unit's lifecycle → the known cause can't recur | reboots, unit-cleanup edges |
| 2 — **auto-heal (no login)** | `felhom-mgmt-watchdog.timer` → `.service` → `/usr/local/sbin/felhom-mgmt-watchdog` (~60s) | recreates a missing `/run/sshd`, `reset-failed`s a start-limited sshd, writes a heal-marker. **Agent-INDEPENDENT** — heals with felhom-agent stopped | a novel runtime cause; the agent being down |
| 3 — **break-glass (human)** | root@pam password, hub-vaulted at day-0 | operator reaches the **PVE web console** (pveproxy :8006 — a daemon/port/codebase distinct from sshd) and runs the one-line fix | both the sshd path AND the auto-heal failing |

Visibility: the agent's heartbeat carries a `mgmt_plane` stanza (`privsep_dir_ok`, `sshd_reachable`,
`healed_recently`, `privsep_healed_at`); the hub raises a **`mgmt_plane_healed` warning** on each new
auto-heal so a **recurring** clobber is investigated before it becomes a lockout.

## 3. Recovery — do this in order

### 3.0 First: is it even broken? (it usually self-heals)
Layer 2 heals within ~60s with nobody logged in. If you just saw the KEXINIT-reset, wait one minute and
retry SSH. If it's back, check the hub for a `mgmt_plane_healed` warning and go to §4 (find the cause).

### 3.1 If SSH is still dead — reach the PVE web console (break-glass)
1. **Retrieve the vaulted root@pam password.**

   **Primary route — the hub UI (hub v0.84.0):** hub → **Hosts** → the host → the **Console access**
   card → **Reveal**. The password shows for 60 s with a Copy button. The card also states which user
   it is for and when it was last vaulted, so a stale copy is visible before you try it. A host with
   nothing vaulted says so and says why, instead of showing a blank. Needs only the hub login
   password — no out-of-band key.

   **Fallback — the global-key curl. Use this when the hub UI itself is unavailable** (that is the
   whole reason it still exists, and why it deliberately has no dependency on the hub's session
   layer):
   ```
   curl -s -H "Authorization: Bearer <GLOBAL-KEY out-of-band>" \
     https://hub.felhom.eu/api/v1/admin/hosts/<host_id>/recovery-credential
   # → {"host_id":"…","username":"root@pam","password":"…","set_at":"…"}
   ```
   (A per-host key is refused here by design — only the operator key reads a console password back.)

   Either route is an **operator action against a customer's box**, and the UI route records it: see
   §5.
2. **Open the PVE web UI** `https://<host-ip>:8006`, log in `root@pam` + that password, realm
   *Linux PAM standard authentication*.
3. **Node → Shell** (xterm.js root console). Or navigate directly:
   `https://<host-ip>:8006/?console=shell&xtermjs=1&node=<node>&resize=off`
4. **Run the one-line fix:**
   ```
   mkdir -p /run/sshd && chmod 0755 /run/sshd
   ```
   SSH on :22 is restored **instantly** — no sshd restart needed (the listener never died; PID is
   unchanged). Verify: `ls -ld /run/sshd` and a fresh `ssh root@<host>`.
5. If the stock sshd is stuck `failed` from a start-limit: `systemctl reset-failed ssh && systemctl start ssh`.

### 3.2 If you have ANY root shell on the box
Just run the §3.1 step 4 one-liner. That's the whole fix. The watchdog will also keep `/run/sshd`
present from then on (layer 2), and the tmpfiles entry restores it on the next boot (layer 1).

## 4. After recovery — find the recurring cause
A single auto-heal is noise; a **repeating** `mgmt_plane_healed` warning means something keeps removing
`/run/sshd`. Look for:
- a systemd unit declaring **`RuntimeDirectory=sshd`** (the incident cause) — `grep -rl RuntimeDirectory /etc/systemd/system /run/systemd`; a second sshd unit must NEVER declare it (H1's constraint).
- a cleanup/upgrade job that `rm`s `/run/sshd`.
Fix the cause; the three layers are the safety net, not the cure.

## 5. Operator hygiene / standing notes
- **Keep the build-server PVE token fresh.** In the spike incident the build-server-cached PVE token
  (`kisfenyo@192.168.0.180:~/felhom-agent-demo.json`) was **stale (401)** and left no non-SSH lever
  before the web-console path was used. A working break-glass path independent of sshd must always exist.
- **Rotate the console credential** with `felhom-host-install … --rotate-recovery` (regenerates +
  re-vaults; a plain re-run does NOT rotate, to avoid stranding your saved copy). Full auto-rotation is
  a future item.
- **Operator-set FIXED credential (e.g. demo-felhom-01, pinned 2026-07-12).** A host's `root@pam` may
  be deliberately set to a *known operator-chosen* password instead of the random day-0 one. Do it the
  same way step 4b does, just with your password: set it on the box (`printf 'root:<pw>\n' | chpasswd`),
  then re-vault with `PUT /api/v1/hosts/<host_id>/recovery-credential` (host `api_key` from
  `/etc/felhom-agent/agent.json`; expect HTTP 200), and ensure `break_glass` is in
  `/var/lib/felhom-install/state.json`'s `completed[]`. This **survives plain installer re-runs**
  (step 4b skips on the marker). Two caveats: (a) **never `--rotate-recovery` such a host** — it
  replaces your known copy with a random one; (b) a **full from-scratch reinstall** (uninstall→install)
  wipes `state.json`, so step 4b re-randomizes — re-run the set-and-vault afterward. Background: on each
  full reinstall this is exactly why `root@pam` "kept changing" until it was pinned.
- **The password is never logged or filed** — it exists only in the hub vault (retrieve as in §3.1).
- **Revealing it through the hub UI writes a customer-visible event line** (hub v0.84.0):
  `recovery_credential_revealed`, info severity, on that host's customer timeline, in Hungarian —
  *„Az üzemeltető lekérte a géped konzolos hozzáférési jelszavát (távoli hibaelhárítás)."* Nobody is
  emailed (no dispatcher call), and the register records **accesses**, so two reveals write two
  lines. Deliberate, for a product sold on data sovereignty; it mirrors how a remote log pull is
  already recorded. An **unbound** host has no customer to tell — there the hub `[INFO]` log line is
  the only record. The global-key curl path writes no event.
- **The hub login password now unlocks console root on every managed box.** Before v0.84.0,
  retrieval also required the global API key. Accepted for a single-operator, HU-geo-fenced hub that
  already holds these passwords in plaintext at rest — a convenience trade, and a real one. The
  plaintext-at-rest half (every hub DB backup is a fleet-wide console-credential dump) is **R-133**.
- **Least-privilege console user** instead of root@pam was probed as a non-blocking observation: PVE
  has historically gated the node Shell to `root@pam`, so G1 ships the proven root@pam path. Revisit if
  PVE later allows a scoped realm user to open a node Shell.

## 6. Cross-references
- `documentation/architecture/06-offsite-connectivity.md` (the OOB transport this complements).
- `documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8/#9 (the incident + prerequisite).
- felhom-agent `internal/mgmtplane/` + `configs/felhom-mgmt-watchdog.*` + `configs/felhom-privsep.tmpfiles`.
- hub `internal/monitor/host_mgmtplane.go` + `internal/store/host_recovery.go`.