diff --git a/CONTEXT.md b/CONTEXT.md index c3041f8..24bd107 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,14 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent + v0.72.0)** — merged E1+H1. Hub: operator OOB peer (`store/wg_operator.go`, `PUT/GET + /admin/wg/operator-peer` global key) + `oob_peer_ip`/`oob_operator_ssh_key` in the desired-state + wireguard block + `monitor/host_oob` (oob_degraded/recovered). Installer: `--enable-oob`/`install_oob` + (static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box + drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as + felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt + drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key. - **2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent v0.71.0)** — prereq for felhom-sshd (H1). Hub: `store.host_recovery` vault (per-host root@pam console password; `PUT /hosts/{id}/recovery-credential` self-scoped + `GET /admin/hosts/{id}/ diff --git a/REPORT.md b/REPORT.md index ae71740..6db6013 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,50 +2,46 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## TASK G1 — management-plane break-glass (hub + installer half) — hub v0.34.1 (2026-07-05) +## TASK H1 — OOB operator access (hub + installer + endpoint half) — hub v0.35.0 (2026-07-05) -**Baseline:** felhom.eu @ `2f97ce3` → `012e5f3`. Hub `0.33.0` → **`0.34.1`** (live via ArgoCD). Agent -half = felhom-agent v0.71.0. Prerequisite for the felhom-sshd OOB feature (H1). Provenance: -`documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8/#9. +**Baseline:** felhom.eu @ `a3ee93e` → pushed. Hub `0.34.1` → **`0.35.0`** (live via ArgoCD). Agent half += felhom-agent v0.72.0. The merged E1+H1 operator-SSH-access feature. Provenance: both +`SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05`. ### Shipped -- **Break-glass credential vault** (`store.host_recovery` + `internal/store/host_recovery.go`): a - per-host root@pam console password, stored at rest, operator-retrievable — the human fallback for - reaching the PVE web console (pveproxy :8006, a failure domain distinct from sshd) when both the - sshd path and the agent-independent auto-heal have failed. `PUT /hosts/{id}/recovery-credential` - (SELF-scoped host key — day-0 vaults it) + `GET /admin/hosts/{id}/recovery-credential` (GLOBAL key - only). Secret never logged (username + length only). -- **mgmt_plane surfacing** (`internal/monitor/host_mgmtplane.go`, 60s sweep): parses the agent's - additive `mgmt_plane` stanza and raises `mgmt_plane_healed` WARNING on a new `privsep_healed_at` - (a recurring `/run/sshd` clobber surfaces before it becomes a lockout; complements host_staleness). - v0.34.1 fix: a heal is an EVENT — construction seeds pre-existing markers (startup false-alarm - guard) but a newly-observed marker alerts, so the FIRST auto-heal surfaces. -- **host-install** (`scripts/felhom-host-install.sh`): `step_break_glass` generates a strong root@pam - password (`openssl rand`, never logged/filed — stdin→chpasswd + stdin→curl), sets it, and vaults it - via the host key; idempotent unless `--rotate-recovery`. Also installs the G1 host artifacts - (tmpfiles + agent-independent watchdog timer), **RuntimeDirectory-guarded** (refuses any unit that - declares it); uninstall removes all of them. +- **Operator OOB peer** (`store/wg_operator.go`): the fleet operator peer as an UNBOUND `wg_peers` row + (host_id '', note operator-oob) at an EXPLICIT /32; validated; last-write-wins rotation; rides + ListWGPeers → peersync pushes it to the endpoint. `PUT/GET /admin/wg/operator-peer` (global key); + PUT also takes `ssh_pubkey` (hub_settings) + bumps every host's generation. +- **Desired-state** (`api/wg.go` mergeWireguard): the served wireguard block gains `oob_peer_ip` + (rendered into the box's AllowedIPs — survives self-heal [OF-1]) + `oob_operator_ssh_key` (agent + writes felhom-sshd's authorized_keys). Absent operator peer → byte-identical pass-through. +- **OOB health** (`monitor/host_oob.go`): ingests the agent `oob` heartbeat stanza; transition-based + `oob_degraded`/`oob_recovered` warning (felhom-sshd down while the operator peer is configured, OR + config invalid). Unconfigured OOB never alerts. +- **host-install** (`scripts`): `--enable-oob` + `install_oob` install the static felhom-sshd + belt + + `felhom-op` user (RuntimeDirectory-guarded); uninstall removes them. +- **Doc 06 §4.5/§4.6 amended**: forwarding ON but per-pair allow-listed, box↔box drop now explicit; + the `oob` health stanza + `oob_degraded` alert. Endpoint runbook §9 added (forward chain + operator + peer registration). ### Tests + red-proofs (all green: `go build/vet/test ./...`) -- store: recovery-credential round-trip + upsert + absent→nil; `GetHostMgmtPlaneStates` parses the - marker + old-agent report degrades to empty. -- api: vault self-scoped (own 200, cross-host 403, unauth 401); operator read global-only (host key - 401, absent 404); **password-never-logged** (buffer-logger red-proof). -- monitor: first-heal-after-healthy alerts once; recurring heals each alert; pre-existing marker seeded - silently; no-heal never alerts. Red-proofed: neutering the emit fails the alert test. +- store: operator-peer round-trip/rotate/unbound + validation (reserved/taken/out-of-subnet); merge + includes oob_peer_ip when configured (absent = byte-identical). +- api: operator-peer PUT self-scoped-global-only; ssh_pubkey validation. +- monitor: degraded/recovered transitions; config-invalid alerts; unconfigured-never-alerts; + no-stanza-ignored. Red-proof: neuter the emit → the alert test fails. -### Live validation (felhom-pve + hub) -- Auto-heal drill (agent stopped): `/run/sshd` removed → agent-independent watchdog healed it in - **30.0 s**, new `:22` session restored with the agent still down. -- Chain: agent report `mgmt_plane` (healed_recently + timestamp) → hub raised `mgmt_plane_healed` - warning (17:16:21). -- **Break-glass drill:** day-0 vault via the host key (200) → operator retrieval via the global key → - the vaulted root@pam password authenticated to PVE (`POST /access/ticket` → 200 = opens the web - console); a host key on the admin read path → 401 (operator-only). Secret never printed/logged. +### Live validation (felhom-pve + dev endpoint) +Endpoint set up first (ip_forward=1 + per-pair forward chain + operator peer registered → pushed to +wg0). Both spikes' key probes re-run as acceptance — all pass: operator→box SSH as felhom-op with +scoped sudo; the OF-1 self-heal /32 survival; coexistence (stock :22 PID 922 unchanged, distinct host +keys, felhom-op denied on :22); the belt (LAN→port dropped, :22 untouched); box↔box drop (counter) + +peersync survival; PBS unaffected throughout; the healthy `oob` stanza reaches the hub + +`oob_degraded`/`oob_recovered` fired around a real felhom-sshd downtime. Full detail + the 5 +live-found-and-fixed agent bugs: felhom-agent `REPORT.md`. ### Notes -- **felhom-pve's root@pam password is now the G1-vaulted strong value** (the intended day-0 outcome); - retrieve it via `GET /admin/hosts/demo-felhom-01/recovery-credential` with the operator key. CC's - key-based SSH is unaffected. -- Keep the build-server PVE token fresh (the incident's secondary lesson); least-privilege console user - + credential auto-rotation are noted future items. +- CGNAT still unproven; IPv6/AAAA out of scope; operator-key auto-rotation is a manual re-PUT; the + "customer network fully down" case is explicitly OUT OF SCOPE / accepted risk. Operator (global) key + = hub `report_api_key`. felhom-pve's operator peer + felhom-sshd + endpoint forwarding stay live. diff --git a/documentation/architecture/06-offsite-connectivity.md b/documentation/architecture/06-offsite-connectivity.md index 010f00f..d4ff44c 100644 --- a/documentation/architecture/06-offsite-connectivity.md +++ b/documentation/architecture/06-offsite-connectivity.md @@ -224,16 +224,30 @@ production endpoint exists. 11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic; **further proven through a live mobile-carrier NAT for a 32-minute fully-idle soak, zero stalls** (2026-07-04 CGNAT smoke test, §7). -- **4.5 Isolation.** Per-peer `/32` `AllowedIPs`; IP forwarding stays **off** on the endpoint; its - firewall admits, from the WG interface, only the PBS port — so a box can reach the PBS API and - nothing else, and boxes cannot see each other **by topology** (spike P6). Public surface: SSH - (operator) + the WG UDP port, nothing more. PBS tenancy on top: namespace + per-customer token + - per-customer key (D5). -- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one — the - storage-manifest model (01 §8: agent "continuously checks presence/reachability, and reports - per-target status; a disconnected target → actionable notification") gains a tunnel-health - input: no handshake within ~3 keepalive periods → the offsite target reports unreachable → the - existing alerting path carries it. No new alarm channel. +- **4.5 Isolation.** Per-peer `/32` `AllowedIPs`; boxes cannot see each other (spike P6). PBS tenancy + on top: namespace + per-customer token + per-customer key (D5). + **AMENDED 2026-07-05 (TASK H1 — OOB operator access).** Forwarding is no longer blanket-**off**; it + is **ON but per-pair allow-listed**. The endpoint runs `net.ipv4.ip_forward=1` (sysctl.d) and a + static forward posture: `ct established,related accept`; **per (operator, box) pair** `ip saddr + ip daddr accept`; and **box↔box `iifname wg0 oifname wg0` DROP is now an + EXPLICIT rule** (previously implicit under the absent capability), backed by the base-chain `policy + drop`. These rules live in the endpoint's static nftables (NOT in `felhom-peersync`, which still + manages only the peer *list*). Net effect: the operator peer reaches a box's `felhom-sshd`; boxes + still cannot reach each other or the operator (only conntrack replies flow). The box side adds a + second layer independent of the endpoint: a dedicated `felhom-sshd` on a claimed non-22 port, gated + by the host-local `inet felhom_oob` belt (reachable only from the operator `/32` over `wg-felhom`; + the customer's stock sshd on :22 is never touched). Live-proven: operator→box SSH works; a dummy + tunnel peer is dropped box↔box (counter); the operator `/32` is **rendered** into the box's + `wg-felhom` AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected. +- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one: no handshake + within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path + carries it. No new alarm channel. + **EXTENDED 2026-07-05 (TASK H1).** The agent's heartbeat now also carries an **`oob` stanza** + (`felhom_sshd_active`, `felhom_sshd_port`, `reachable`, `config_invalid`, `operator_peer_configured`, + `operator_key_configured`, `wg_handshake_age_s`) — the operator's "can I get into this box right + now, and if not, why" signal. It reaches the hub over HTTPS even when `felhom-sshd` or the tunnel is + DOWN (channel independence). The hub raises a transition-based **`oob_degraded`/`oob_recovered`** + warning (felhom-sshd down while the operator peer is configured, OR config invalid). --- diff --git a/documentation/runbooks/offsite-endpoint.md b/documentation/runbooks/offsite-endpoint.md index 4d5c4d3..b32f569 100644 --- a/documentation/runbooks/offsite-endpoint.md +++ b/documentation/runbooks/offsite-endpoint.md @@ -311,3 +311,38 @@ The rebuild is **steps 1–7 on a fresh VM**. What is lost vs regenerable: (customers still hold local backups + a re-seedable offsite). - The hub's SSH credential + pinned host key must be **rotated on rebuild** (new box = new host key): repeat step 6 (`kubectl delete secret wg-endpoint-ssh` first), roll the hub. + +## 9. OOB operator forwarding (TASK H1 — 2026-07-05) + +The endpoint gains a **per-pair operator→box forward** posture so the operator peer can reach each +box's `felhom-sshd` (doc 06 §4.5 amended: forwarding ON but per-pair allow-listed; box↔box drop is +now explicit). Peersync is UNCHANGED — it still manages only the peer *list*; these forward rules are +STATIC endpoint config. + +```sh +# 1. permanent forwarding +echo 'net.ipv4.ip_forward = 1' > /etc/sysctl.d/99-felhom-oob.conf +sysctl -w net.ipv4.ip_forward=1 + +# 2. forward posture in the STATIC nftables filter forward chain (add to /etc/nftables.conf's +# `chain forward` — which keeps `policy drop`). ONE accept rule per (operator, box) pair; the +# box↔box drop is explicit. Reload path so replies + the PBS path are unaffected (INPUT hook). +# = GET /api/v1/admin/wg/operator-peer ; = each host's assigned_ip. + ct state established,related accept + iifname "wg0" oifname "wg0" ip saddr ip daddr counter accept + iifname "wg0" oifname "wg0" counter drop +``` + +**Register the operator peer** (hub, global key) — it becomes an UNBOUND `wg_peers` row (peersync +pushes it to wg0) and its `/32` flows to every box as `oob_peer_ip`: +```sh +curl -s -X PUT https://hub.felhom.eu/api/v1/admin/wg/operator-peer \ + -H "Authorization: Bearer " \ + -d '{"pubkey":"","assigned_ip":"10.77.0.250","ssh_pubkey":"ssh-ed25519 AAAA… operator"}' +``` + +**Box side:** `felhom-host-install … --enable-oob` (static felhom-sshd + belt) + `oob.enabled=true` +in `agent.json`. The agent renders the config, claims a port, writes `felhom-op`'s authorized_keys +from `oob_operator_ssh_key`, and fills the belt sets. Verify: from the operator peer, +`ssh -p felhom-op@`; the host belt drops any non-operator tunnel source. +Re-verify PBS (`pvesm status --storage felhom-offsite` on the box) after the forward change. diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 8d3dbd5..cbf0647 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,5 +1,16 @@ # Felhom scripts — Changelog +## felhom-host-install — --enable-oob installs the dedicated felhom-sshd OOB instance + belt (TASK H1) (2026-07-05) + +- `--enable-oob` + `install_oob` (in step 5): lays down the STATIC OOB parts — `/etc/felhom-sshd` tree + + a dedicated ed25519 host key + the `felhom-op` login user + its scoped `/etc/sudoers.d/felhom-op` + (visudo-validated) + the `felhom-sshd.service` unit (**RuntimeDirectory-guarded** — refuses the G1 + incident cause) enabled-not-started (the agent renders the config) + the static `inet felhom_oob` + nft table (`nft -c`-checked) + its boot loader `felhom-oob-nft.service`. The agent reconciles the + DYNAMIC parts (port claim, config, operator authorized_keys, belt set elements) once `oob.enabled`. +- Uninstall (step 4b3): stops/disables felhom-sshd + the belt loader, removes both units + the config + tree + belt table + `felhom-op` user/sudoers; NEVER touches the stock sshd/:22/`/etc/ssh`. + ## felhom-host-install — break-glass credential + management-plane watchdog install (TASK G1) (2026-07-05) - `step_break_glass` (new step 4b): generates a strong root@pam console password (`openssl rand`,