felhom.eu/REPORT.md

felhom.eu — task reports

Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.

TASK G1 — management-plane break-glass (hub + installer half) — hub v0.34.1 (2026-07-05)

Baseline: felhom.eu @ 2f97ce3 → 012e5f3. Hub 0.33.0 → 0.34.1 (live via ArgoCD). Agent half = felhom-agent v0.71.0. Prerequisite for the felhom-sshd OOB feature (H1). Provenance: documentation/audits/SPIKE-felhom-sshd-2026-07-05.md §8/#9.

Shipped

• Break-glass credential vault (store.host_recovery + internal/store/host_recovery.go): a per-host root@pam console password, stored at rest, operator-retrievable — the human fallback for reaching the PVE web console (pveproxy :8006, a failure domain distinct from sshd) when both the sshd path and the agent-independent auto-heal have failed. PUT /hosts/{id}/recovery-credential (SELF-scoped host key — day-0 vaults it) + GET /admin/hosts/{id}/recovery-credential (GLOBAL key only). Secret never logged (username + length only).
• mgmt_plane surfacing (internal/monitor/host_mgmtplane.go, 60s sweep): parses the agent's additive mgmt_plane stanza and raises mgmt_plane_healed WARNING on a new privsep_healed_at (a recurring /run/sshd clobber surfaces before it becomes a lockout; complements host_staleness). v0.34.1 fix: a heal is an EVENT — construction seeds pre-existing markers (startup false-alarm guard) but a newly-observed marker alerts, so the FIRST auto-heal surfaces.
• host-install (scripts/felhom-host-install.sh): step_break_glass generates a strong root@pam password (openssl rand, never logged/filed — stdin→chpasswd + stdin→curl), sets it, and vaults it via the host key; idempotent unless --rotate-recovery. Also installs the G1 host artifacts (tmpfiles + agent-independent watchdog timer), RuntimeDirectory-guarded (refuses any unit that declares it); uninstall removes all of them.

Tests + red-proofs (all green: go build/vet/test ./...)

• store: recovery-credential round-trip + upsert + absent→nil; GetHostMgmtPlaneStates parses the marker + old-agent report degrades to empty.
• api: vault self-scoped (own 200, cross-host 403, unauth 401); operator read global-only (host key 401, absent 404); password-never-logged (buffer-logger red-proof).
• monitor: first-heal-after-healthy alerts once; recurring heals each alert; pre-existing marker seeded silently; no-heal never alerts. Red-proofed: neutering the emit fails the alert test.

Live validation (felhom-pve + hub)

• Auto-heal drill (agent stopped): /run/sshd removed → agent-independent watchdog healed it in 30.0 s, new :22 session restored with the agent still down.
• Chain: agent report mgmt_plane (healed_recently + timestamp) → hub raised mgmt_plane_healed warning (17:16:21).
• Break-glass drill: day-0 vault via the host key (200) → operator retrieval via the global key → the vaulted root@pam password authenticated to PVE (POST /access/ticket → 200 = opens the web console); a host key on the admin read path → 401 (operator-only). Secret never printed/logged.

Notes

• felhom-pve's root@pam password is now the G1-vaulted strong value (the intended day-0 outcome); retrieve it via GET /admin/hosts/demo-felhom-01/recovery-credential with the operator key. CC's key-based SSH is unaffected.
• Keep the build-server PVE token fresh (the incident's secondary lesson); least-privilege console user
  • credential auto-rotation are noted future items.