docs(H1): doc06 §4.5/§4.6 amendment + endpoint runbook §9 + scripts CHANGELOG + REPORT + CONTEXT

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

CONTEXT.md

@@ -3,6 +3,14 @@
 > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
 > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
 
+- **2026-07-05 — TASK H1 SHIPPED: OOB operator access (hub v0.35.0 + installer + endpoint; agent
+  v0.72.0)** — merged E1+H1. Hub: operator OOB peer (`store/wg_operator.go`, `PUT/GET
+  /admin/wg/operator-peer` global key) + `oob_peer_ip`/`oob_operator_ssh_key` in the desired-state
+  wireguard block + `monitor/host_oob` (oob_degraded/recovered). Installer: `--enable-oob`/`install_oob`
+  (static felhom-sshd + belt). Doc 06 §4.5/§4.6 amended (forwarding per-pair allow-listed, box↔box
+  drop explicit, oob health stanza); endpoint runbook §9. Live-validated: operator→box SSH as
+  felhom-op with scoped sudo; OF-1 /32 self-heal survival; coexistence (:22 PID unchanged); belt
+  drops LAN + box↔box; PBS unaffected. Operator key = hub report_api_key.
 - **2026-07-05 — TASK G1 SHIPPED: management-plane break-glass (hub v0.34.1 + installer; agent
   v0.71.0)** — prereq for felhom-sshd (H1). Hub: `store.host_recovery` vault (per-host root@pam
   console password; `PUT /hosts/{id}/recovery-credential` self-scoped + `GET /admin/hosts/{id}/

REPORT.md

@@ -2,50 +2,46 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
 
-## TASK G1 — management-plane break-glass (hub + installer half) — hub v0.34.1 (2026-07-05)
+## TASK H1 — OOB operator access (hub + installer + endpoint half) — hub v0.35.0 (2026-07-05)
 
-**Baseline:** felhom.eu @ `2f97ce3` → `012e5f3`. Hub `0.33.0` → **`0.34.1`** (live via ArgoCD). Agent
-half = felhom-agent v0.71.0. Prerequisite for the felhom-sshd OOB feature (H1). Provenance:
-`documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8/#9.
+**Baseline:** felhom.eu @ `a3ee93e` → pushed. Hub `0.34.1` → **`0.35.0`** (live via ArgoCD). Agent half
+= felhom-agent v0.72.0. The merged E1+H1 operator-SSH-access feature. Provenance: both
+`SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05`.
 
 ### Shipped
-- **Break-glass credential vault** (`store.host_recovery` + `internal/store/host_recovery.go`): a
-  per-host root@pam console password, stored at rest, operator-retrievable — the human fallback for
-  reaching the PVE web console (pveproxy :8006, a failure domain distinct from sshd) when both the
-  sshd path and the agent-independent auto-heal have failed. `PUT /hosts/{id}/recovery-credential`
-  (SELF-scoped host key — day-0 vaults it) + `GET /admin/hosts/{id}/recovery-credential` (GLOBAL key
-  only). Secret never logged (username + length only).
-- **mgmt_plane surfacing** (`internal/monitor/host_mgmtplane.go`, 60s sweep): parses the agent's
-  additive `mgmt_plane` stanza and raises `mgmt_plane_healed` WARNING on a new `privsep_healed_at`
-  (a recurring `/run/sshd` clobber surfaces before it becomes a lockout; complements host_staleness).
-  v0.34.1 fix: a heal is an EVENT — construction seeds pre-existing markers (startup false-alarm
-  guard) but a newly-observed marker alerts, so the FIRST auto-heal surfaces.
-- **host-install** (`scripts/felhom-host-install.sh`): `step_break_glass` generates a strong root@pam
-  password (`openssl rand`, never logged/filed — stdin→chpasswd + stdin→curl), sets it, and vaults it
-  via the host key; idempotent unless `--rotate-recovery`. Also installs the G1 host artifacts
-  (tmpfiles + agent-independent watchdog timer), **RuntimeDirectory-guarded** (refuses any unit that
-  declares it); uninstall removes all of them.
+- **Operator OOB peer** (`store/wg_operator.go`): the fleet operator peer as an UNBOUND `wg_peers` row
+  (host_id '', note operator-oob) at an EXPLICIT /32; validated; last-write-wins rotation; rides
+  ListWGPeers → peersync pushes it to the endpoint. `PUT/GET /admin/wg/operator-peer` (global key);
+  PUT also takes `ssh_pubkey` (hub_settings) + bumps every host's generation.
+- **Desired-state** (`api/wg.go` mergeWireguard): the served wireguard block gains `oob_peer_ip`
+  (rendered into the box's AllowedIPs — survives self-heal [OF-1]) + `oob_operator_ssh_key` (agent
+  writes felhom-sshd's authorized_keys). Absent operator peer → byte-identical pass-through.
+- **OOB health** (`monitor/host_oob.go`): ingests the agent `oob` heartbeat stanza; transition-based
+  `oob_degraded`/`oob_recovered` warning (felhom-sshd down while the operator peer is configured, OR
+  config invalid). Unconfigured OOB never alerts.
+- **host-install** (`scripts`): `--enable-oob` + `install_oob` install the static felhom-sshd + belt +
+  `felhom-op` user (RuntimeDirectory-guarded); uninstall removes them.
+- **Doc 06 §4.5/§4.6 amended**: forwarding ON but per-pair allow-listed, box↔box drop now explicit;
+  the `oob` health stanza + `oob_degraded` alert. Endpoint runbook §9 added (forward chain + operator
+  peer registration).
 
 ### Tests + red-proofs (all green: `go build/vet/test ./...`)
-- store: recovery-credential round-trip + upsert + absent→nil; `GetHostMgmtPlaneStates` parses the
-  marker + old-agent report degrades to empty.
-- api: vault self-scoped (own 200, cross-host 403, unauth 401); operator read global-only (host key
-  401, absent 404); **password-never-logged** (buffer-logger red-proof).
-- monitor: first-heal-after-healthy alerts once; recurring heals each alert; pre-existing marker seeded
-  silently; no-heal never alerts. Red-proofed: neutering the emit fails the alert test.
+- store: operator-peer round-trip/rotate/unbound + validation (reserved/taken/out-of-subnet); merge
+  includes oob_peer_ip when configured (absent = byte-identical).
+- api: operator-peer PUT self-scoped-global-only; ssh_pubkey validation.
+- monitor: degraded/recovered transitions; config-invalid alerts; unconfigured-never-alerts;
+  no-stanza-ignored. Red-proof: neuter the emit → the alert test fails.
 
-### Live validation (felhom-pve + hub)
-- Auto-heal drill (agent stopped): `/run/sshd` removed → agent-independent watchdog healed it in
-  **30.0 s**, new `:22` session restored with the agent still down.
-- Chain: agent report `mgmt_plane` (healed_recently + timestamp) → hub raised `mgmt_plane_healed`
-  warning (17:16:21).
-- **Break-glass drill:** day-0 vault via the host key (200) → operator retrieval via the global key →
-  the vaulted root@pam password authenticated to PVE (`POST /access/ticket` → 200 = opens the web
-  console); a host key on the admin read path → 401 (operator-only). Secret never printed/logged.
+### Live validation (felhom-pve + dev endpoint)
+Endpoint set up first (ip_forward=1 + per-pair forward chain + operator peer registered → pushed to
+wg0). Both spikes' key probes re-run as acceptance — all pass: operator→box SSH as felhom-op with
+scoped sudo; the OF-1 self-heal /32 survival; coexistence (stock :22 PID 922 unchanged, distinct host
+keys, felhom-op denied on :22); the belt (LAN→port dropped, :22 untouched); box↔box drop (counter) +
+peersync survival; PBS unaffected throughout; the healthy `oob` stanza reaches the hub +
+`oob_degraded`/`oob_recovered` fired around a real felhom-sshd downtime. Full detail + the 5
+live-found-and-fixed agent bugs: felhom-agent `REPORT.md`.
 
 ### Notes
-- **felhom-pve's root@pam password is now the G1-vaulted strong value** (the intended day-0 outcome);
-  retrieve it via `GET /admin/hosts/demo-felhom-01/recovery-credential` with the operator key. CC's
-  key-based SSH is unaffected.
-- Keep the build-server PVE token fresh (the incident's secondary lesson); least-privilege console user
-  + credential auto-rotation are noted future items.
+- CGNAT still unproven; IPv6/AAAA out of scope; operator-key auto-rotation is a manual re-PUT; the
+  "customer network fully down" case is explicitly OUT OF SCOPE / accepted risk. Operator (global) key
+  = hub `report_api_key`. felhom-pve's operator peer + felhom-sshd + endpoint forwarding stay live.

documentation/architecture/06-offsite-connectivity.md

@@ -224,16 +224,30 @@ production endpoint exists.
   11.4-minute fully-idle window (P2) at ~150 B/s of overhead traffic; **further proven through a
   live mobile-carrier NAT for a 32-minute fully-idle soak, zero stalls** (2026-07-04 CGNAT smoke
   test, §7).
-- **4.5 Isolation.** Per-peer `/32` `AllowedIPs`; IP forwarding stays **off** on the endpoint; its
-  firewall admits, from the WG interface, only the PBS port — so a box can reach the PBS API and
-  nothing else, and boxes cannot see each other **by topology** (spike P6). Public surface: SSH
-  (operator) + the WG UDP port, nothing more. PBS tenancy on top: namespace + per-customer token +
-  per-customer key (D5).
-- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one — the
-  storage-manifest model (01 §8: agent "continuously checks presence/reachability, and reports
-  per-target status; a disconnected target → actionable notification") gains a tunnel-health
-  input: no handshake within ~3 keepalive periods → the offsite target reports unreachable → the
-  existing alerting path carries it. No new alarm channel.
+- **4.5 Isolation.** Per-peer `/32` `AllowedIPs`; boxes cannot see each other (spike P6). PBS tenancy
+  on top: namespace + per-customer token + per-customer key (D5).
+  **AMENDED 2026-07-05 (TASK H1 — OOB operator access).** Forwarding is no longer blanket-**off**; it
+  is **ON but per-pair allow-listed**. The endpoint runs `net.ipv4.ip_forward=1` (sysctl.d) and a
+  static forward posture: `ct established,related accept`; **per (operator, box) pair** `ip saddr
+  <operator/32> ip daddr <box/32> accept`; and **box↔box `iifname wg0 oifname wg0` DROP is now an
+  EXPLICIT rule** (previously implicit under the absent capability), backed by the base-chain `policy
+  drop`. These rules live in the endpoint's static nftables (NOT in `felhom-peersync`, which still
+  manages only the peer *list*). Net effect: the operator peer reaches a box's `felhom-sshd`; boxes
+  still cannot reach each other or the operator (only conntrack replies flow). The box side adds a
+  second layer independent of the endpoint: a dedicated `felhom-sshd` on a claimed non-22 port, gated
+  by the host-local `inet felhom_oob` belt (reachable only from the operator `/32` over `wg-felhom`;
+  the customer's stock sshd on :22 is never touched). Live-proven: operator→box SSH works; a dummy
+  tunnel peer is dropped box↔box (counter); the operator `/32` is **rendered** into the box's
+  `wg-felhom` AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected.
+- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one: no handshake
+  within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path
+  carries it. No new alarm channel.
+  **EXTENDED 2026-07-05 (TASK H1).** The agent's heartbeat now also carries an **`oob` stanza**
+  (`felhom_sshd_active`, `felhom_sshd_port`, `reachable`, `config_invalid`, `operator_peer_configured`,
+  `operator_key_configured`, `wg_handshake_age_s`) — the operator's "can I get into this box right
+  now, and if not, why" signal. It reaches the hub over HTTPS even when `felhom-sshd` or the tunnel is
+  DOWN (channel independence). The hub raises a transition-based **`oob_degraded`/`oob_recovered`**
+  warning (felhom-sshd down while the operator peer is configured, OR config invalid).
 
 ---
 

documentation/runbooks/offsite-endpoint.md

@@ -311,3 +311,38 @@ The rebuild is **steps 1–7 on a fresh VM**. What is lost vs regenerable:
   (customers still hold local backups + a re-seedable offsite).
 - The hub's SSH credential + pinned host key must be **rotated on rebuild** (new box = new
   host key): repeat step 6 (`kubectl delete secret wg-endpoint-ssh` first), roll the hub.
+
+## 9. OOB operator forwarding (TASK H1 — 2026-07-05)
+
+The endpoint gains a **per-pair operator→box forward** posture so the operator peer can reach each
+box's `felhom-sshd` (doc 06 §4.5 amended: forwarding ON but per-pair allow-listed; box↔box drop is
+now explicit). Peersync is UNCHANGED — it still manages only the peer *list*; these forward rules are
+STATIC endpoint config.
+
+```sh
+# 1. permanent forwarding
+echo 'net.ipv4.ip_forward = 1' > /etc/sysctl.d/99-felhom-oob.conf
+sysctl -w net.ipv4.ip_forward=1
+
+# 2. forward posture in the STATIC nftables filter forward chain (add to /etc/nftables.conf's
+#    `chain forward` — which keeps `policy drop`). ONE accept rule per (operator, box) pair; the
+#    box↔box drop is explicit. Reload path so replies + the PBS path are unaffected (INPUT hook).
+#    <operator/32> = GET /api/v1/admin/wg/operator-peer ; <box/32> = each host's assigned_ip.
+    ct state established,related accept
+    iifname "wg0" oifname "wg0" ip saddr <operator/32> ip daddr <box/32> counter accept
+    iifname "wg0" oifname "wg0" counter drop
+```
+
+**Register the operator peer** (hub, global key) — it becomes an UNBOUND `wg_peers` row (peersync
+pushes it to wg0) and its `/32` flows to every box as `oob_peer_ip`:
+```sh
+curl -s -X PUT https://hub.felhom.eu/api/v1/admin/wg/operator-peer \
+  -H "Authorization: Bearer <GLOBAL-KEY>" \
+  -d '{"pubkey":"<operator WG pubkey>","assigned_ip":"10.77.0.250","ssh_pubkey":"ssh-ed25519 AAAA… operator"}'
+```
+
+**Box side:** `felhom-host-install … --enable-oob` (static felhom-sshd + belt) + `oob.enabled=true`
+in `agent.json`. The agent renders the config, claims a port, writes `felhom-op`'s authorized_keys
+from `oob_operator_ssh_key`, and fills the belt sets. Verify: from the operator peer,
+`ssh -p <claimed-port> felhom-op@<box tunnel IP>`; the host belt drops any non-operator tunnel source.
+Re-verify PBS (`pvesm status --storage felhom-offsite` on the box) after the forward change.

scripts/CHANGELOG.md

@@ -1,5 +1,16 @@
 # Felhom scripts — Changelog
 
+## felhom-host-install — --enable-oob installs the dedicated felhom-sshd OOB instance + belt (TASK H1) (2026-07-05)
+
+- `--enable-oob` + `install_oob` (in step 5): lays down the STATIC OOB parts — `/etc/felhom-sshd` tree
+  + a dedicated ed25519 host key + the `felhom-op` login user + its scoped `/etc/sudoers.d/felhom-op`
+  (visudo-validated) + the `felhom-sshd.service` unit (**RuntimeDirectory-guarded** — refuses the G1
+  incident cause) enabled-not-started (the agent renders the config) + the static `inet felhom_oob`
+  nft table (`nft -c`-checked) + its boot loader `felhom-oob-nft.service`. The agent reconciles the
+  DYNAMIC parts (port claim, config, operator authorized_keys, belt set elements) once `oob.enabled`.
+- Uninstall (step 4b3): stops/disables felhom-sshd + the belt loader, removes both units + the config
+  tree + belt table + `felhom-op` user/sudoers; NEVER touches the stock sshd/:22/`/etc/ssh`.
+
 ## felhom-host-install — break-glass credential + management-plane watchdog install (TASK G1) (2026-07-05)
 
 - `step_break_glass` (new step 4b): generates a strong root@pam console password (`openssl rand`,