felhom-agent/REPORT.md

REPORT — TASK G1: management-plane break-glass (felhom-agent half) — v0.71.0

Baseline: felhom-agent main @ 1c75a45 (v0.70.0) → v0.71.0 (commit fd4e177). felhom.eu @ 2f97ce3. Trunk-based, direct to main.

What shipped (this repo)

The prevent + auto-heal + observe thirds of the break-glass system (the hub vault + surfacing are the felhom.eu half, hub v0.34.1). Closes the lockout from felhom.eu/documentation/audits/SPIKE-felhom-sshd-2026-07-05.md §8: a second sshd's RuntimeDirectory=sshd removed the SHARED /run/sshd privsep dir → the stock sshd on :22 reset every session after SSH2_MSG_KEXINIT.

• Host artifacts (configs/, installed by felhom-host-install):
  • felhom-privsep.tmpfiles — d /run/sshd 0755 root root - (layer 1: boot-persistent, owned by no unit's lifecycle).
  • felhom-mgmt-watchdog.sh — the heal action: stat-first recreate /run/sshd, reset-failed the stock sshd ONLY when failed, write an RFC3339 heal-marker; never restarts the stock sshd, never touches a healthy dir. shellcheck-clean.
  • felhom-mgmt-watchdog.service (oneshot) + .timer (OnUnitActiveSec=60s, Persistent). No unit declares RuntimeDirectory= (the incident cause).
• Go (internal/mgmtplane/): a read-only Reporter (os.Stat /run/sshd, read the heal-marker, TCP-dial sshd:22) → the additive omitempty mgmt_plane heartbeat stanza (privsep_dir_ok, sshd_reachable, healed_recently, privsep_healed_at). Wired always-on via Collector.SetMgmtPlaneReporter (report.go additive fields, the SelfUpdatePending precedent — no hub-schema change; the golden contract test uses a nil reporter so the golden stays byte-identical).

Tests + red-proofs

go build ./... && go vet ./... && go test ./... on the build server — all green.

• internal/mgmtplane: healthy (dir ok, no marker → healed_recently=false); missing dir detected while sshd_reachable stays true (the "TCP up, sessions broken" trap); marker present → healed_recently
  • timestamp; empty/whitespace marker treated as absent (red-proof for a hub warning with no timestamp); sshd-unreachable reported.
• internal/hub collector: nil reporter → mgmt_plane omitted; wired → carried through.

Live validation (felhom-pve) — the acceptance

Deployed manually (agent self-update needs an operator-signed op + a pinned signer; the box has no signer pinned, so D1's self-update path was not usable — stated honestly). Build on 180 (-X main.version=0.71.0, sha b3b616fe…) → scp via local → install -m0755 (backup .bak-0.70.0) → restart. Host artifacts installed + timer enabled; systemd-analyze verify + systemd-tmpfiles --create + shellcheck all clean; RuntimeDirectory guard verified (no directive, only the warning comment).

Auto-heal drill (agent STOPPED — the load-bearing proof): systemctl stop felhom-agent → rm -rf /run/sshd → a new :22 session reproduced the incident (Connection reset right after the banner) → the agent-independent watchdog recreated /run/sshd and a new :22 session succeeded again in 30.0 s (measured), with the agent still inactive. Marker written. This proves the heal is login-free AND agent-independent.

End-to-end visibility chain (hub v0.34.1): the v0.71.0 report carried mgmt_plane:{privsep_dir_ok:true, sshd_reachable:true, healed_recently:true, privsep_healed_at:"2026-07-05T17:15:32Z"}; the prior healthy report had healed_recently:false; the old v0.70.0 report had no stanza (correctly absent). The hub monitor raised the mgmt_plane_healed warning at 17:16:21 ("privsep dir … was missing and was AUTO-HEALED …").

KEXINIT-reset reproduction: confirmed live in the drill (Connection reset by 127.0.0.1 port 22 immediately after the banner exchange, /run/sshd absent).

Cross-repo state

• Agent v0.71.0 live on felhom-pve; three break-glass layers active + enabled (they STAY — this is the shipped feature). Rollback: .bak-0.70.0.
• Hub v0.34.1 live; break-glass vault + mgmt_plane_healed monitor. felhom.eu commits 05d8181 (hub+installer), 28ac5f5 (runbook), 012e5f3 (monitor fix + v0.34.1).
• H1 (felhom-sshd) can now assume /run/sshd is guaranteed present (tmpfiles + watchdog) and must never declare RuntimeDirectory= — the installer refuses any unit that does.

Notes / observations

• Secret discipline verified: the root@pam recovery password appears in NO log/commit (the vault handler logs username + length only; red-proofed hub-side).
• Least-privilege console user (vs root@pam): not probed live (PVE gates the node Shell to root@pam historically); G1 ships the proven root@pam path — a non-blocking future item.
• Auto-rotation of the recovery credential is a noted future item; --rotate-recovery re-vaults on demand.
• Heal cadence: OnUnitActiveSec=60s (measured 30 s this run, mid-cycle); worst case ~60 s.