docs(H1): doc06 §4.5/§4.6 amendment + endpoint runbook §9 + scripts CHANGELOG + REPORT + CONTEXT

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
2026-07-05 23:03:33 +02:00
parent b70f2d0763
commit 61f4898d30
5 changed files with 113 additions and 49 deletions
+35 -39
View File
@@ -2,50 +2,46 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## TASK G1 — management-plane break-glass (hub + installer half) — hub v0.34.1 (2026-07-05)
## TASK H1 — OOB operator access (hub + installer + endpoint half) — hub v0.35.0 (2026-07-05)
**Baseline:** felhom.eu @ `2f97ce3``012e5f3`. Hub `0.33.0`**`0.34.1`** (live via ArgoCD). Agent
half = felhom-agent v0.71.0. Prerequisite for the felhom-sshd OOB feature (H1). Provenance:
`documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8/#9.
**Baseline:** felhom.eu @ `a3ee93e` → pushed. Hub `0.34.1`**`0.35.0`** (live via ArgoCD). Agent half
= felhom-agent v0.72.0. The merged E1+H1 operator-SSH-access feature. Provenance: both
`SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05`.
### Shipped
- **Break-glass credential vault** (`store.host_recovery` + `internal/store/host_recovery.go`): a
per-host root@pam console password, stored at rest, operator-retrievable — the human fallback for
reaching the PVE web console (pveproxy :8006, a failure domain distinct from sshd) when both the
sshd path and the agent-independent auto-heal have failed. `PUT /hosts/{id}/recovery-credential`
(SELF-scoped host key — day-0 vaults it) + `GET /admin/hosts/{id}/recovery-credential` (GLOBAL key
only). Secret never logged (username + length only).
- **mgmt_plane surfacing** (`internal/monitor/host_mgmtplane.go`, 60s sweep): parses the agent's
additive `mgmt_plane` stanza and raises `mgmt_plane_healed` WARNING on a new `privsep_healed_at`
(a recurring `/run/sshd` clobber surfaces before it becomes a lockout; complements host_staleness).
v0.34.1 fix: a heal is an EVENT — construction seeds pre-existing markers (startup false-alarm
guard) but a newly-observed marker alerts, so the FIRST auto-heal surfaces.
- **host-install** (`scripts/felhom-host-install.sh`): `step_break_glass` generates a strong root@pam
password (`openssl rand`, never logged/filed — stdin→chpasswd + stdin→curl), sets it, and vaults it
via the host key; idempotent unless `--rotate-recovery`. Also installs the G1 host artifacts
(tmpfiles + agent-independent watchdog timer), **RuntimeDirectory-guarded** (refuses any unit that
declares it); uninstall removes all of them.
- **Operator OOB peer** (`store/wg_operator.go`): the fleet operator peer as an UNBOUND `wg_peers` row
(host_id '', note operator-oob) at an EXPLICIT /32; validated; last-write-wins rotation; rides
ListWGPeers → peersync pushes it to the endpoint. `PUT/GET /admin/wg/operator-peer` (global key);
PUT also takes `ssh_pubkey` (hub_settings) + bumps every host's generation.
- **Desired-state** (`api/wg.go` mergeWireguard): the served wireguard block gains `oob_peer_ip`
(rendered into the box's AllowedIPs — survives self-heal [OF-1]) + `oob_operator_ssh_key` (agent
writes felhom-sshd's authorized_keys). Absent operator peer → byte-identical pass-through.
- **OOB health** (`monitor/host_oob.go`): ingests the agent `oob` heartbeat stanza; transition-based
`oob_degraded`/`oob_recovered` warning (felhom-sshd down while the operator peer is configured, OR
config invalid). Unconfigured OOB never alerts.
- **host-install** (`scripts`): `--enable-oob` + `install_oob` install the static felhom-sshd + belt +
`felhom-op` user (RuntimeDirectory-guarded); uninstall removes them.
- **Doc 06 §4.5/§4.6 amended**: forwarding ON but per-pair allow-listed, box↔box drop now explicit;
the `oob` health stanza + `oob_degraded` alert. Endpoint runbook §9 added (forward chain + operator
peer registration).
### Tests + red-proofs (all green: `go build/vet/test ./...`)
- store: recovery-credential round-trip + upsert + absent→nil; `GetHostMgmtPlaneStates` parses the
marker + old-agent report degrades to empty.
- api: vault self-scoped (own 200, cross-host 403, unauth 401); operator read global-only (host key
401, absent 404); **password-never-logged** (buffer-logger red-proof).
- monitor: first-heal-after-healthy alerts once; recurring heals each alert; pre-existing marker seeded
silently; no-heal never alerts. Red-proofed: neutering the emit fails the alert test.
- store: operator-peer round-trip/rotate/unbound + validation (reserved/taken/out-of-subnet); merge
includes oob_peer_ip when configured (absent = byte-identical).
- api: operator-peer PUT self-scoped-global-only; ssh_pubkey validation.
- monitor: degraded/recovered transitions; config-invalid alerts; unconfigured-never-alerts;
no-stanza-ignored. Red-proof: neuter the emit → the alert test fails.
### Live validation (felhom-pve + hub)
- Auto-heal drill (agent stopped): `/run/sshd` removed → agent-independent watchdog healed it in
**30.0 s**, new `:22` session restored with the agent still down.
- Chain: agent report `mgmt_plane` (healed_recently + timestamp) → hub raised `mgmt_plane_healed`
warning (17:16:21).
- **Break-glass drill:** day-0 vault via the host key (200) → operator retrieval via the global key →
the vaulted root@pam password authenticated to PVE (`POST /access/ticket` → 200 = opens the web
console); a host key on the admin read path → 401 (operator-only). Secret never printed/logged.
### Live validation (felhom-pve + dev endpoint)
Endpoint set up first (ip_forward=1 + per-pair forward chain + operator peer registered → pushed to
wg0). Both spikes' key probes re-run as acceptance — all pass: operator→box SSH as felhom-op with
scoped sudo; the OF-1 self-heal /32 survival; coexistence (stock :22 PID 922 unchanged, distinct host
keys, felhom-op denied on :22); the belt (LAN→port dropped, :22 untouched); box↔box drop (counter) +
peersync survival; PBS unaffected throughout; the healthy `oob` stanza reaches the hub +
`oob_degraded`/`oob_recovered` fired around a real felhom-sshd downtime. Full detail + the 5
live-found-and-fixed agent bugs: felhom-agent `REPORT.md`.
### Notes
- **felhom-pve's root@pam password is now the G1-vaulted strong value** (the intended day-0 outcome);
retrieve it via `GET /admin/hosts/demo-felhom-01/recovery-credential` with the operator key. CC's
key-based SSH is unaffected.
- Keep the build-server PVE token fresh (the incident's secondary lesson); least-privilege console user
+ credential auto-rotation are noted future items.
- CGNAT still unproven; IPv6/AAAA out of scope; operator-key auto-rotation is a manual re-PUT; the
"customer network fully down" case is explicitly OUT OF SCOPE / accepted risk. Operator (global) key
= hub `report_api_key`. felhom-pve's operator peer + felhom-sshd + endpoint forwarding stay live.