Files

T

admin 1956e5d390 hub v0.84.0 — break-glass console credential on the host page

The credential existed and was not reachable when it was wanted. Every box has
had a strong random root@pam password since TASK G1, vaulted in the hub at day 0
and used for real during the sshd incident — but the only way to read it back was
a hand-written curl carrying the global operator key, a secret kept out-of-band.
In practice the PVE web console on a demo box felt locked.

The host page grows a Console access card: presence + username + set_at by
default, Reveal fetches the plaintext on demand for 60 s with a Copy button.
Masking clears the JS variable, and also fires on a second click and on
visibilitychange. A host with nothing vaulted says so, and says why.

The secret is NEVER rendered into the page, and that constraint shapes the
change. The render path uses a new store.GetHostRecoveryMeta whose struct and
SELECT both omit the secret column, so it is structurally incapable of carrying
one. The plaintext crosses the wire only in the response to POST
/hosts/{id}/reveal-recovery-credential (Cache-Control: no-store, CSRF-gated at
the ServeHTTP level; POST precisely so that gate applies and so no secret is
retrievable by URL alone). Deliberately NOT the customer page's data-secret
widget, which embeds the plaintext on every load.

A delivered reveal writes one recovery_credential_revealed event on the host's
customer timeline (info, source hub, Hungarian) via SaveEvent alone — no
dispatcher, nobody emailed, the log_tail_requested shape. Two reveals write two
events: the register records accesses, not states. A 404 is not an access. An
unbound host reveals fine and writes no event; the [INFO] hub line, carrying the
username and a length only, is then the record.

The global-key API path is untouched by design — it is the route for when the
hub UI itself is broken, and coupling it to the session layer would delete the
independence that makes it a fallback.

Recorded as a real trade: the hub session password alone now unlocks console root
fleet-wide, where retrieval previously also needed the global key. Accepted for a
single-operator, HU-geo-fenced hub that already stores these passwords in
plaintext at rest (CONTEXT.md ruling S-4). The plaintext-at-rest half is filed as
R-133 — every hub DB backup is a fleet-wide console-credential dump.

Tests 550 -> 559; four red-proofs (page leak, audit event, CSRF gate, route
order) each run, observed failing, and reverted. The route-order proof is a seam
test driving ServeHTTP: a handler-level test cannot see that defect, because the
handler is correct and simply never runs.

2026-07-31 08:19:36 +02:00

8.6 KiB

Raw Permalink Blame History

RUNBOOK — Break-glass: management-plane recovery (privsep watchdog + vaulted console credential)

What this covers: how a Felhom host stays reachable when its SSH management plane breaks, and how the operator recovers if it does anyway. Delivered by TASK G1 (felhom-agent v0.71.0 + hub v0.34.0 + host-install). Design provenance: documentation/audits/SPIKE-felhom-sshd-2026-07-05.md §8 (the incident) + #9 (the prerequisite). This is the blocker the felhom-sshd feature (H1) depends on — H1 may assume /run/sshd is guaranteed present and must never declare RuntimeDirectory=.

1. The failure (know the fingerprint)

/run/sshd is OpenSSH's compiled-in privilege-separation directory, SHARED by every sshd on the host. If it goes missing, sshd's listener stays up (TCP connects, the version banner exchanges) but every new session RESETS right after SSH2_MSG_KEXINIT. The box is healthy underneath (guests, PVE web, agent all fine) but remotely unreachable over SSH.

Diagnostic (from any client):

ssh -vv root@<host> 2>&1 | grep -A1 KEXINIT
# ... "SSH2_MSG_KEXINIT sent" immediately followed by "Connection reset by <host> port 22" = /run/sshd missing

The spike caused this live via a second sshd's RuntimeDirectory=sshd cleanup removing the shared dir.

2. The three layers (defense in depth)

Layer	Artifact	What it does	Failure domain it survives
1 — prevent	`/etc/tmpfiles.d/felhom-privsep.conf` (`d /run/sshd 0755 root root -`)	`/run/sshd` is boot-persistent, owned by no unit's lifecycle → the known cause can't recur	reboots, unit-cleanup edges
2 — auto-heal (no login)	`felhom-mgmt-watchdog.timer` → `.service` → `/usr/local/sbin/felhom-mgmt-watchdog` (~60s)	recreates a missing `/run/sshd`, `reset-failed`s a start-limited sshd, writes a heal-marker. Agent-INDEPENDENT — heals with felhom-agent stopped	a novel runtime cause; the agent being down
3 — break-glass (human)	root@pam password, hub-vaulted at day-0	operator reaches the PVE web console (pveproxy :8006 — a daemon/port/codebase distinct from sshd) and runs the one-line fix	both the sshd path AND the auto-heal failing

Visibility: the agent's heartbeat carries a mgmt_plane stanza (privsep_dir_ok, sshd_reachable, healed_recently, privsep_healed_at); the hub raises a mgmt_plane_healed warning on each new auto-heal so a recurring clobber is investigated before it becomes a lockout.

3. Recovery — do this in order

3.0 First: is it even broken? (it usually self-heals)

Layer 2 heals within ~60s with nobody logged in. If you just saw the KEXINIT-reset, wait one minute and retry SSH. If it's back, check the hub for a mgmt_plane_healed warning and go to §4 (find the cause).

3.1 If SSH is still dead — reach the PVE web console (break-glass)

Retrieve the vaulted root@pam password.

Primary route — the hub UI (hub v0.84.0): hub → Hosts → the host → the Console access card → Reveal. The password shows for 60 s with a Copy button. The card also states which user it is for and when it was last vaulted, so a stale copy is visible before you try it. A host with nothing vaulted says so and says why, instead of showing a blank. Needs only the hub login password — no out-of-band key.

Fallback — the global-key curl. Use this when the hub UI itself is unavailable (that is the whole reason it still exists, and why it deliberately has no dependency on the hub's session layer):
```
curl -s -H "Authorization: Bearer <GLOBAL-KEY out-of-band>" \
  https://hub.felhom.eu/api/v1/admin/hosts/<host_id>/recovery-credential
# → {"host_id":"…","username":"root@pam","password":"…","set_at":"…"}
```
(A per-host key is refused here by design — only the operator key reads a console password back.)

Either route is an operator action against a customer's box, and the UI route records it: see §5.
Open the PVE web UI https://<host-ip>:8006, log in root@pam + that password, realm Linux PAM standard authentication.
Node → Shell (xterm.js root console). Or navigate directly: https://<host-ip>:8006/?console=shell&xtermjs=1&node=<node>&resize=off
Run the one-line fix:
```
mkdir -p /run/sshd && chmod 0755 /run/sshd
```
SSH on :22 is restored instantly — no sshd restart needed (the listener never died; PID is unchanged). Verify: ls -ld /run/sshd and a fresh ssh root@<host>.
If the stock sshd is stuck failed from a start-limit: systemctl reset-failed ssh && systemctl start ssh.

3.2 If you have ANY root shell on the box

Just run the §3.1 step 4 one-liner. That's the whole fix. The watchdog will also keep /run/sshd present from then on (layer 2), and the tmpfiles entry restores it on the next boot (layer 1).

4. After recovery — find the recurring cause

A single auto-heal is noise; a repeating mgmt_plane_healed warning means something keeps removing /run/sshd. Look for:

a systemd unit declaring RuntimeDirectory=sshd (the incident cause) — grep -rl RuntimeDirectory /etc/systemd/system /run/systemd; a second sshd unit must NEVER declare it (H1's constraint).
a cleanup/upgrade job that rms /run/sshd. Fix the cause; the three layers are the safety net, not the cure.

5. Operator hygiene / standing notes

Keep the build-server PVE token fresh. In the spike incident the build-server-cached PVE token (kisfenyo@192.168.0.180:~/felhom-agent-demo.json) was stale (401) and left no non-SSH lever before the web-console path was used. A working break-glass path independent of sshd must always exist.
Rotate the console credential with felhom-host-install … --rotate-recovery (regenerates + re-vaults; a plain re-run does NOT rotate, to avoid stranding your saved copy). Full auto-rotation is a future item.
Operator-set FIXED credential (e.g. demo-felhom-01, pinned 2026-07-12). A host's root@pam may be deliberately set to a known operator-chosen password instead of the random day-0 one. Do it the same way step 4b does, just with your password: set it on the box (printf 'root:<pw>\n' | chpasswd), then re-vault with PUT /api/v1/hosts/<host_id>/recovery-credential (host api_key from /etc/felhom-agent/agent.json; expect HTTP 200), and ensure break_glass is in /var/lib/felhom-install/state.json's completed[]. This survives plain installer re-runs (step 4b skips on the marker). Two caveats: (a) never --rotate-recovery such a host — it replaces your known copy with a random one; (b) a full from-scratch reinstall (uninstall→install) wipes state.json, so step 4b re-randomizes — re-run the set-and-vault afterward. Background: on each full reinstall this is exactly why root@pam "kept changing" until it was pinned.
The password is never logged or filed — it exists only in the hub vault (retrieve as in §3.1).
Revealing it through the hub UI writes a customer-visible event line (hub v0.84.0): recovery_credential_revealed, info severity, on that host's customer timeline, in Hungarian — „Az üzemeltető lekérte a géped konzolos hozzáférési jelszavát (távoli hibaelhárítás)." Nobody is emailed (no dispatcher call), and the register records accesses, so two reveals write two lines. Deliberate, for a product sold on data sovereignty; it mirrors how a remote log pull is already recorded. An unbound host has no customer to tell — there the hub [INFO] log line is the only record. The global-key curl path writes no event.
The hub login password now unlocks console root on every managed box. Before v0.84.0, retrieval also required the global API key. Accepted for a single-operator, HU-geo-fenced hub that already holds these passwords in plaintext at rest — a convenience trade, and a real one. The plaintext-at-rest half (every hub DB backup is a fleet-wide console-credential dump) is R-133.
Least-privilege console user instead of root@pam was probed as a non-blocking observation: PVE has historically gated the node Shell to root@pam, so G1 ships the proven root@pam path. Revisit if PVE later allows a scoped realm user to open a node Shell.

6. Cross-references

documentation/architecture/06-offsite-connectivity.md (the OOB transport this complements).
documentation/audits/SPIKE-felhom-sshd-2026-07-05.md §8/#9 (the incident + prerequisite).
felhom-agent internal/mgmtplane/ + configs/felhom-mgmt-watchdog.* + configs/felhom-privsep.tmpfiles.
hub internal/monitor/host_mgmtplane.go + internal/store/host_recovery.go.

8.6 KiB Raw Permalink Blame History