admin
1956e5d390
The credential existed and was not reachable when it was wanted. Every box has
had a strong random root@pam password since TASK G1, vaulted in the hub at day 0
and used for real during the sshd incident — but the only way to read it back was
a hand-written curl carrying the global operator key, a secret kept out-of-band.
In practice the PVE web console on a demo box felt locked.
The host page grows a Console access card: presence + username + set_at by
default, Reveal fetches the plaintext on demand for 60 s with a Copy button.
Masking clears the JS variable, and also fires on a second click and on
visibilitychange. A host with nothing vaulted says so, and says why.
The secret is NEVER rendered into the page, and that constraint shapes the
change. The render path uses a new store.GetHostRecoveryMeta whose struct and
SELECT both omit the secret column, so it is structurally incapable of carrying
one. The plaintext crosses the wire only in the response to POST
/hosts/{id}/reveal-recovery-credential (Cache-Control: no-store, CSRF-gated at
the ServeHTTP level; POST precisely so that gate applies and so no secret is
retrievable by URL alone). Deliberately NOT the customer page's data-secret
widget, which embeds the plaintext on every load.
A delivered reveal writes one recovery_credential_revealed event on the host's
customer timeline (info, source hub, Hungarian) via SaveEvent alone — no
dispatcher, nobody emailed, the log_tail_requested shape. Two reveals write two
events: the register records accesses, not states. A 404 is not an access. An
unbound host reveals fine and writes no event; the [INFO] hub line, carrying the
username and a length only, is then the record.
The global-key API path is untouched by design — it is the route for when the
hub UI itself is broken, and coupling it to the session layer would delete the
independence that makes it a fallback.
Recorded as a real trade: the hub session password alone now unlocks console root
fleet-wide, where retrieval previously also needed the global key. Accepted for a
single-operator, HU-geo-fenced hub that already stores these passwords in
plaintext at rest (CONTEXT.md ruling S-4). The plaintext-at-rest half is filed as
R-133 — every hub DB backup is a fleet-wide console-credential dump.
Tests 550 -> 559; four red-proofs (page leak, audit event, CSRF gate, route
order) each run, observed failing, and reverted. The route-order proof is a seam
test driving ServeHTTP: a handler-level test cannot see that defect, because the
handler is correct and simply never runs.
123 lines
8.6 KiB
Markdown
123 lines
8.6 KiB
Markdown
# RUNBOOK — Break-glass: management-plane recovery (privsep watchdog + vaulted console credential)
|
|
|
|
> **What this covers:** how a Felhom host stays reachable when its SSH management plane breaks, and
|
|
> how the operator recovers if it does anyway. Delivered by **TASK G1** (felhom-agent v0.71.0 +
|
|
> hub v0.34.0 + host-install). Design provenance: `documentation/audits/SPIKE-felhom-sshd-2026-07-05.md`
|
|
> §8 (the incident) + #9 (the prerequisite). This is the **blocker** the felhom-sshd feature (H1)
|
|
> depends on — H1 may assume `/run/sshd` is guaranteed present and must never declare `RuntimeDirectory=`.
|
|
|
|
## 1. The failure (know the fingerprint)
|
|
|
|
`/run/sshd` is OpenSSH's compiled-in **privilege-separation directory**, SHARED by every sshd on the
|
|
host. If it goes missing, sshd's **listener stays up** (TCP connects, the version banner exchanges) but
|
|
every new session **RESETS right after `SSH2_MSG_KEXINIT`**. The box is healthy underneath (guests,
|
|
PVE web, agent all fine) but **remotely unreachable over SSH**.
|
|
|
|
**Diagnostic (from any client):**
|
|
```
|
|
ssh -vv root@<host> 2>&1 | grep -A1 KEXINIT
|
|
# ... "SSH2_MSG_KEXINIT sent" immediately followed by "Connection reset by <host> port 22" = /run/sshd missing
|
|
```
|
|
The spike caused this live via a second sshd's `RuntimeDirectory=sshd` cleanup removing the shared dir.
|
|
|
|
## 2. The three layers (defense in depth)
|
|
|
|
| Layer | Artifact | What it does | Failure domain it survives |
|
|
|---|---|---|---|
|
|
| 1 — **prevent** | `/etc/tmpfiles.d/felhom-privsep.conf` (`d /run/sshd 0755 root root -`) | `/run/sshd` is boot-persistent, owned by no unit's lifecycle → the known cause can't recur | reboots, unit-cleanup edges |
|
|
| 2 — **auto-heal (no login)** | `felhom-mgmt-watchdog.timer` → `.service` → `/usr/local/sbin/felhom-mgmt-watchdog` (~60s) | recreates a missing `/run/sshd`, `reset-failed`s a start-limited sshd, writes a heal-marker. **Agent-INDEPENDENT** — heals with felhom-agent stopped | a novel runtime cause; the agent being down |
|
|
| 3 — **break-glass (human)** | root@pam password, hub-vaulted at day-0 | operator reaches the **PVE web console** (pveproxy :8006 — a daemon/port/codebase distinct from sshd) and runs the one-line fix | both the sshd path AND the auto-heal failing |
|
|
|
|
Visibility: the agent's heartbeat carries a `mgmt_plane` stanza (`privsep_dir_ok`, `sshd_reachable`,
|
|
`healed_recently`, `privsep_healed_at`); the hub raises a **`mgmt_plane_healed` warning** on each new
|
|
auto-heal so a **recurring** clobber is investigated before it becomes a lockout.
|
|
|
|
## 3. Recovery — do this in order
|
|
|
|
### 3.0 First: is it even broken? (it usually self-heals)
|
|
Layer 2 heals within ~60s with nobody logged in. If you just saw the KEXINIT-reset, wait one minute and
|
|
retry SSH. If it's back, check the hub for a `mgmt_plane_healed` warning and go to §4 (find the cause).
|
|
|
|
### 3.1 If SSH is still dead — reach the PVE web console (break-glass)
|
|
1. **Retrieve the vaulted root@pam password.**
|
|
|
|
**Primary route — the hub UI (hub v0.84.0):** hub → **Hosts** → the host → the **Console access**
|
|
card → **Reveal**. The password shows for 60 s with a Copy button. The card also states which user
|
|
it is for and when it was last vaulted, so a stale copy is visible before you try it. A host with
|
|
nothing vaulted says so and says why, instead of showing a blank. Needs only the hub login
|
|
password — no out-of-band key.
|
|
|
|
**Fallback — the global-key curl. Use this when the hub UI itself is unavailable** (that is the
|
|
whole reason it still exists, and why it deliberately has no dependency on the hub's session
|
|
layer):
|
|
```
|
|
curl -s -H "Authorization: Bearer <GLOBAL-KEY out-of-band>" \
|
|
https://hub.felhom.eu/api/v1/admin/hosts/<host_id>/recovery-credential
|
|
# → {"host_id":"…","username":"root@pam","password":"…","set_at":"…"}
|
|
```
|
|
(A per-host key is refused here by design — only the operator key reads a console password back.)
|
|
|
|
Either route is an **operator action against a customer's box**, and the UI route records it: see
|
|
§5.
|
|
2. **Open the PVE web UI** `https://<host-ip>:8006`, log in `root@pam` + that password, realm
|
|
*Linux PAM standard authentication*.
|
|
3. **Node → Shell** (xterm.js root console). Or navigate directly:
|
|
`https://<host-ip>:8006/?console=shell&xtermjs=1&node=<node>&resize=off`
|
|
4. **Run the one-line fix:**
|
|
```
|
|
mkdir -p /run/sshd && chmod 0755 /run/sshd
|
|
```
|
|
SSH on :22 is restored **instantly** — no sshd restart needed (the listener never died; PID is
|
|
unchanged). Verify: `ls -ld /run/sshd` and a fresh `ssh root@<host>`.
|
|
5. If the stock sshd is stuck `failed` from a start-limit: `systemctl reset-failed ssh && systemctl start ssh`.
|
|
|
|
### 3.2 If you have ANY root shell on the box
|
|
Just run the §3.1 step 4 one-liner. That's the whole fix. The watchdog will also keep `/run/sshd`
|
|
present from then on (layer 2), and the tmpfiles entry restores it on the next boot (layer 1).
|
|
|
|
## 4. After recovery — find the recurring cause
|
|
A single auto-heal is noise; a **repeating** `mgmt_plane_healed` warning means something keeps removing
|
|
`/run/sshd`. Look for:
|
|
- a systemd unit declaring **`RuntimeDirectory=sshd`** (the incident cause) — `grep -rl RuntimeDirectory /etc/systemd/system /run/systemd`; a second sshd unit must NEVER declare it (H1's constraint).
|
|
- a cleanup/upgrade job that `rm`s `/run/sshd`.
|
|
Fix the cause; the three layers are the safety net, not the cure.
|
|
|
|
## 5. Operator hygiene / standing notes
|
|
- **Keep the build-server PVE token fresh.** In the spike incident the build-server-cached PVE token
|
|
(`kisfenyo@192.168.0.180:~/felhom-agent-demo.json`) was **stale (401)** and left no non-SSH lever
|
|
before the web-console path was used. A working break-glass path independent of sshd must always exist.
|
|
- **Rotate the console credential** with `felhom-host-install … --rotate-recovery` (regenerates +
|
|
re-vaults; a plain re-run does NOT rotate, to avoid stranding your saved copy). Full auto-rotation is
|
|
a future item.
|
|
- **Operator-set FIXED credential (e.g. demo-felhom-01, pinned 2026-07-12).** A host's `root@pam` may
|
|
be deliberately set to a *known operator-chosen* password instead of the random day-0 one. Do it the
|
|
same way step 4b does, just with your password: set it on the box (`printf 'root:<pw>\n' | chpasswd`),
|
|
then re-vault with `PUT /api/v1/hosts/<host_id>/recovery-credential` (host `api_key` from
|
|
`/etc/felhom-agent/agent.json`; expect HTTP 200), and ensure `break_glass` is in
|
|
`/var/lib/felhom-install/state.json`'s `completed[]`. This **survives plain installer re-runs**
|
|
(step 4b skips on the marker). Two caveats: (a) **never `--rotate-recovery` such a host** — it
|
|
replaces your known copy with a random one; (b) a **full from-scratch reinstall** (uninstall→install)
|
|
wipes `state.json`, so step 4b re-randomizes — re-run the set-and-vault afterward. Background: on each
|
|
full reinstall this is exactly why `root@pam` "kept changing" until it was pinned.
|
|
- **The password is never logged or filed** — it exists only in the hub vault (retrieve as in §3.1).
|
|
- **Revealing it through the hub UI writes a customer-visible event line** (hub v0.84.0):
|
|
`recovery_credential_revealed`, info severity, on that host's customer timeline, in Hungarian —
|
|
*„Az üzemeltető lekérte a géped konzolos hozzáférési jelszavát (távoli hibaelhárítás)."* Nobody is
|
|
emailed (no dispatcher call), and the register records **accesses**, so two reveals write two
|
|
lines. Deliberate, for a product sold on data sovereignty; it mirrors how a remote log pull is
|
|
already recorded. An **unbound** host has no customer to tell — there the hub `[INFO]` log line is
|
|
the only record. The global-key curl path writes no event.
|
|
- **The hub login password now unlocks console root on every managed box.** Before v0.84.0,
|
|
retrieval also required the global API key. Accepted for a single-operator, HU-geo-fenced hub that
|
|
already holds these passwords in plaintext at rest — a convenience trade, and a real one. The
|
|
plaintext-at-rest half (every hub DB backup is a fleet-wide console-credential dump) is **R-133**.
|
|
- **Least-privilege console user** instead of root@pam was probed as a non-blocking observation: PVE
|
|
has historically gated the node Shell to `root@pam`, so G1 ships the proven root@pam path. Revisit if
|
|
PVE later allows a scoped realm user to open a node Shell.
|
|
|
|
## 6. Cross-references
|
|
- `documentation/architecture/06-offsite-connectivity.md` (the OOB transport this complements).
|
|
- `documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8/#9 (the incident + prerequisite).
|
|
- felhom-agent `internal/mgmtplane/` + `configs/felhom-mgmt-watchdog.*` + `configs/felhom-privsep.tmpfiles`.
|
|
- hub `internal/monitor/host_mgmtplane.go` + `internal/store/host_recovery.go`.
|