admin
8675bff3da
Auto-heal drill with the agent STOPPED healed /run/sshd in 30.0s (measured); mgmt_plane stanza + hub mgmt_plane_healed warning proven end-to-end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
79 lines
4.9 KiB
Markdown
79 lines
4.9 KiB
Markdown
# REPORT — TASK G1: management-plane break-glass (felhom-agent half) — v0.71.0
|
|
|
|
**Baseline:** felhom-agent `main` @ `1c75a45` (v0.70.0) → **v0.71.0** (commit `fd4e177`). felhom.eu @
|
|
`2f97ce3`. Trunk-based, direct to `main`.
|
|
|
|
## What shipped (this repo)
|
|
|
|
The **prevent + auto-heal + observe** thirds of the break-glass system (the hub vault + surfacing are
|
|
the felhom.eu half, hub v0.34.1). Closes the lockout from
|
|
`felhom.eu/documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8: a second sshd's
|
|
`RuntimeDirectory=sshd` removed the SHARED `/run/sshd` privsep dir → the stock sshd on :22 reset every
|
|
session after `SSH2_MSG_KEXINIT`.
|
|
|
|
- **Host artifacts** (`configs/`, installed by felhom-host-install):
|
|
- `felhom-privsep.tmpfiles` — `d /run/sshd 0755 root root -` (layer 1: boot-persistent, owned by no
|
|
unit's lifecycle).
|
|
- `felhom-mgmt-watchdog.sh` — the heal action: stat-first recreate `/run/sshd`, `reset-failed` the
|
|
stock sshd ONLY when `failed`, write an RFC3339 heal-marker; never restarts the stock sshd, never
|
|
touches a healthy dir. shellcheck-clean.
|
|
- `felhom-mgmt-watchdog.service` (oneshot) + `.timer` (OnUnitActiveSec=60s, Persistent). **No unit
|
|
declares `RuntimeDirectory=`** (the incident cause).
|
|
- **Go** (`internal/mgmtplane/`): a read-only `Reporter` (os.Stat `/run/sshd`, read the heal-marker,
|
|
TCP-dial sshd:22) → the additive `omitempty` `mgmt_plane` heartbeat stanza
|
|
(`privsep_dir_ok, sshd_reachable, healed_recently, privsep_healed_at`). Wired always-on via
|
|
`Collector.SetMgmtPlaneReporter` (report.go additive fields, the SelfUpdatePending precedent — no
|
|
hub-schema change; the golden contract test uses a nil reporter so the golden stays byte-identical).
|
|
|
|
## Tests + red-proofs
|
|
|
|
`go build ./... && go vet ./... && go test ./...` on the build server — **all green**.
|
|
- `internal/mgmtplane`: healthy (dir ok, no marker → healed_recently=false); missing dir detected
|
|
while sshd_reachable stays true (the "TCP up, sessions broken" trap); marker present → healed_recently
|
|
+ timestamp; empty/whitespace marker treated as absent (red-proof for a hub warning with no
|
|
timestamp); sshd-unreachable reported.
|
|
- `internal/hub` collector: nil reporter → `mgmt_plane` omitted; wired → carried through.
|
|
|
|
## Live validation (felhom-pve) — the acceptance
|
|
|
|
Deployed **manually** (agent self-update needs an operator-signed op + a pinned signer; the box has
|
|
**no signer pinned**, so D1's self-update path was not usable — stated honestly). Build on 180
|
|
(`-X main.version=0.71.0`, sha `b3b616fe…`) → scp via local → `install -m0755` (backup `.bak-0.70.0`)
|
|
→ restart. Host artifacts installed + timer enabled; `systemd-analyze verify` + `systemd-tmpfiles
|
|
--create` + shellcheck all clean; RuntimeDirectory guard verified (no directive, only the warning
|
|
comment).
|
|
|
|
**Auto-heal drill (agent STOPPED — the load-bearing proof):** `systemctl stop felhom-agent` →
|
|
`rm -rf /run/sshd` → a new `:22` session reproduced the incident (`Connection reset` right after the
|
|
banner) → the agent-independent watchdog **recreated `/run/sshd` and a new `:22` session succeeded
|
|
again in 30.0 s (measured), with the agent still `inactive`.** Marker written. This proves the heal is
|
|
login-free AND agent-independent.
|
|
|
|
**End-to-end visibility chain (hub v0.34.1):** the v0.71.0 report carried
|
|
`mgmt_plane:{privsep_dir_ok:true, sshd_reachable:true, healed_recently:true,
|
|
privsep_healed_at:"2026-07-05T17:15:32Z"}`; the prior healthy report had `healed_recently:false`; the
|
|
old v0.70.0 report had no stanza (correctly absent). The hub monitor raised the
|
|
**`mgmt_plane_healed` warning** at 17:16:21 ("privsep dir … was missing and was AUTO-HEALED …").
|
|
|
|
**KEXINIT-reset reproduction:** confirmed live in the drill (`Connection reset by 127.0.0.1 port 22`
|
|
immediately after the banner exchange, `/run/sshd` absent).
|
|
|
|
## Cross-repo state
|
|
|
|
- Agent **v0.71.0** live on felhom-pve; three break-glass layers active + enabled (they STAY — this
|
|
is the shipped feature). Rollback: `.bak-0.70.0`.
|
|
- Hub **v0.34.1** live; break-glass vault + `mgmt_plane_healed` monitor. felhom.eu commits `05d8181`
|
|
(hub+installer), `28ac5f5` (runbook), `012e5f3` (monitor fix + v0.34.1).
|
|
- **H1 (felhom-sshd) can now assume `/run/sshd` is guaranteed present** (tmpfiles + watchdog) and must
|
|
never declare `RuntimeDirectory=` — the installer refuses any unit that does.
|
|
|
|
## Notes / observations
|
|
|
|
- **Secret discipline verified:** the root@pam recovery password appears in NO log/commit (the vault
|
|
handler logs username + length only; red-proofed hub-side).
|
|
- **Least-privilege console user** (vs root@pam): not probed live (PVE gates the node Shell to
|
|
root@pam historically); G1 ships the proven root@pam path — a non-blocking future item.
|
|
- **Auto-rotation** of the recovery credential is a noted future item; `--rotate-recovery` re-vaults on
|
|
demand.
|
|
- Heal cadence: OnUnitActiveSec=60s (measured 30 s this run, mid-cycle); worst case ~60 s.
|