docs(H1): v0.72.0 CHANGELOG + REPORT + CONTEXT (OOB operator access, live-validated)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

CHANGELOG.md

@@ -1,3 +1,33 @@
+## v0.72.0 — OOB operator access: rendered operator /32 + dedicated felhom-sshd + port-adaptive belt + oob health (TASK H1) (2026-07-05)
+
+The agent half of the merged E1+H1 operator-SSH-access feature (hub half = felhom-hub v0.35.0).
+Provenance: `felhom.eu/documentation/audits/SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05.md`.
+Live-validated on felhom-pve + the dev endpoint (both spikes' key probes re-run as acceptance).
+
+- **Operator /32 rendered into wg-felhom** (`internal/wgtunnel/manager.go` `renderConf`/`allowedIPsLine`):
+  `oob_peer_ip` from the desired-state block is appended to AllowedIPs, deterministically SORTED
+  (byte-stable conf-hash — no per-tick flap). RENDERED, not a runtime `wg set`, so it survives the
+  agent's self-heal ([OF-1]; live-proven: tunnel stopped → self-heal → operator SSH still works).
+- **Dedicated felhom-sshd** (`internal/felhomsshd/`): a SECOND sshd on a claimed non-22 port
+  (`[8822,2222,8022,62222]`, LOUD-fail on exhaustion [SF-4]), own config/host-key/AuthorizedKeysFile
+  (`/etc/felhom-sshd/authorized_keys/%u`, outside ~/.ssh [SF-3])/unit — COEXISTS with the customer's
+  :22 (never touched). Config: render→`sshd -t`→**reload** (never restart-on-change [SF-2]); operator
+  authorized_keys from the hub block; `reset-failed`-then-restart heal with a 10-min cooldown, NEVER
+  restarting onto an invalid config. `configs/felhom-sshd.service` SAFE — **no `RuntimeDirectory=`**
+  [SF-1].
+- **Port-adaptive belt** (`internal/felhomsshd/belt.go` + `configs/felhom-oob.nft`): a STATIC
+  `inet felhom_oob` table; the agent mutates ONLY its SETS — `@operator_ips` + `@ssh_port` [trap 4] —
+  so felhom-sshd's port is reachable ONLY from the operator `/32` over `wg-felhom` (off-tunnel +
+  box↔box dropped at the host; :22 untouched). Idempotent; a nil/unfetched block never empties it
+  (no operator lockout).
+- **OOB health** (`internal/felhomsshd/health.go`): the additive `oob` heartbeat stanza
+  (`felhom_sshd_active/port/reachable/config_invalid/operator_peer_configured/operator_key_configured/
+  wg_handshake_age_s`) — reaches the hub over HTTPS even with felhom-sshd/tunnel down. `reachable` =
+  a listener check (the belt blocks a dial); `operator_*_configured` from persistent state
+  (belt/authorized_keys), accurate immediately after a restart.
+- **Sudoers**: `FELHOM_SSHD` (config/authkeys install + `sshd -t/-T` + scoped systemctl) + `FELHOM_OOB`
+  (nft SET-element ops only — never a rule grant). `oob.enabled` config DEFAULT FALSE.
+
 ## v0.71.0 — management-plane break-glass: privsep-dir watchdog + mgmt_plane health (TASK G1) (2026-07-05)
 
 Prerequisite for the felhom-sshd OOB feature (H1). Closes the lockout from

CONTEXT.md

@@ -5,6 +5,15 @@
 
 ## Current
 
+- **v0.72.0** (2026-07-05) — **OOB operator access (merged E1+H1)** — TASK H1, provenance both
+  `SPIKE-{felhom-sshd,oob-wg-operator-peer}-2026-07-05`. Operator `/32` RENDERED into wg-felhom
+  AllowedIPs (survives self-heal, [OF-1]); dedicated `internal/felhomsshd` (port claim + config
+  render→sshd -t→reload + operator authorized_keys + heal + oob heartbeat stanza); static
+  `inet felhom_oob` belt (agent mutates SET ELEMENTS ONLY); `configs/felhom-sshd.service` (NO
+  RuntimeDirectory [SF-1]) + `felhom-oob.nft` + `felhom-op.sudoers`; `FELHOM_SSHD`+`FELHOM_OOB`
+  grants; `oob.enabled` DEFAULT FALSE. Live on felhom-pve (8822, belt filled, operator SSH as
+  felhom-op with scoped sudo); hub v0.35.0. Rollback `.bak-0.71.0`. 5 live-found bugs fixed (port
+  path, self-listen flip-flop, nil-block lockout, reachable-via-dial, operator-configured source).
 - **v0.71.0** (2026-07-05) — **management-plane break-glass: privsep-dir watchdog + mgmt_plane
   health** — TASK G1 (prereq for felhom-sshd/H1), provenance `SPIKE-felhom-sshd-2026-07-05` §8.
   Host artifacts (`configs/felhom-privsep.tmpfiles` + `felhom-mgmt-watchdog.{sh,service,timer}`) make

REPORT.md

@@ -1,78 +1,82 @@
-# REPORT — TASK G1: management-plane break-glass (felhom-agent half) — v0.71.0
+# REPORT — TASK H1: OOB operator access (felhom-agent half) — v0.72.0
 
-**Baseline:** felhom-agent `main` @ `1c75a45` (v0.70.0) → **v0.71.0** (commit `fd4e177`). felhom.eu @
-`2f97ce3`. Trunk-based, direct to `main`.
+**Baseline:** felhom-agent `main` @ `4b7c5bf` (v0.71.0) → **v0.72.0** (final binary sha
+`f72c2fed…`). felhom.eu @ `a3ee93e` → hub v0.35.0. Trunk-based, direct to `main`. Provenance: BOTH
+`SPIKE-felhom-sshd-2026-07-05` [SF-n] and `SPIKE-oob-wg-operator-peer-2026-07-05` [OF-n].
 
 ## What shipped (this repo)
 
-The **prevent + auto-heal + observe** thirds of the break-glass system (the hub vault + surfacing are
-the felhom.eu half, hub v0.34.1). Closes the lockout from
-`felhom.eu/documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8: a second sshd's
-`RuntimeDirectory=sshd` removed the SHARED `/run/sshd` privsep dir → the stock sshd on :22 reset every
-session after `SSH2_MSG_KEXINIT`.
+- **Operator `/32` rendered into `wg-felhom` AllowedIPs** (`internal/wgtunnel`): `oob_peer_ip` appended
+  sorted + byte-stable; RENDERED so it survives self-heal ([OF-1]).
+- **Dedicated `felhom-sshd`** (`internal/felhomsshd`): claim (loud-fail exhaustion), render→`sshd -t`→
+  reload (never restart-on-change [SF-2]), operator authorized_keys outside ~/.ssh [SF-3], heal with
+  cooldown + never-onto-invalid-config, `configs/felhom-sshd.service` with **no `RuntimeDirectory=`**
+  [SF-1].
+- **Port-adaptive belt** (`internal/felhomsshd/belt.go` + `configs/felhom-oob.nft`): static table, agent
+  mutates SET ELEMENTS ONLY (`@operator_ips`/`@ssh_port`) [trap 4]; a nil block never empties it.
+- **OOB heartbeat stanza** + `FELHOM_SSHD`/`FELHOM_OOB` sudoers. `oob.enabled` DEFAULT FALSE.
 
-- **Host artifacts** (`configs/`, installed by felhom-host-install):
-  - `felhom-privsep.tmpfiles` — `d /run/sshd 0755 root root -` (layer 1: boot-persistent, owned by no
-    unit's lifecycle).
-  - `felhom-mgmt-watchdog.sh` — the heal action: stat-first recreate `/run/sshd`, `reset-failed` the
-    stock sshd ONLY when `failed`, write an RFC3339 heal-marker; never restarts the stock sshd, never
-    touches a healthy dir. shellcheck-clean.
-  - `felhom-mgmt-watchdog.service` (oneshot) + `.timer` (OnUnitActiveSec=60s, Persistent). **No unit
-    declares `RuntimeDirectory=`** (the incident cause).
-- **Go** (`internal/mgmtplane/`): a read-only `Reporter` (os.Stat `/run/sshd`, read the heal-marker,
-  TCP-dial sshd:22) → the additive `omitempty` `mgmt_plane` heartbeat stanza
-  (`privsep_dir_ok, sshd_reachable, healed_recently, privsep_healed_at`). Wired always-on via
-  `Collector.SetMgmtPlaneReporter` (report.go additive fields, the SelfUpdatePending precedent — no
-  hub-schema change; the golden contract test uses a nil reporter so the golden stays byte-identical).
+## Tests + red-proofs (all green — `go build/vet/test ./...`, 23 packages ok)
 
-## Tests + red-proofs
+- renderConf: OOB /32 appended sorted + byte-stable; lower-IP-sorts-first; absent = byte-identical
+  golden; invalid/v6 rejected. Companion: byte-stability is the anti-flap red-proof.
+- claim: clean/contention/idempotent/exhaustion (LOUD-fail is the key negative — non-zero, never :22);
+  persisted-port kept unconditionally (the self-listen flip-flop fix).
+- config: SAFE template, byte-stable, refuses :22, never `RuntimeDirectory`.
+- belt: mutate-then-idempotent (zero nft ops on the 2nd sync) + **never touches rules** (trap 4
+  asserted); empty operator empties the set.
+- heal: NO restart on an invalid config (red-proof) + 10-min cooldown; status reflects the block.
+- hub collector + OOB monitor: stanza ingest + transition-based `oob_degraded`/`oob_recovered` (+ the
+  unconfigured-not-alerted and no-stanza-ignored negatives). Red-proof: neutering the emit fails it.
 
-`go build ./... && go vet ./... && go test ./...` on the build server — **all green**.
-- `internal/mgmtplane`: healthy (dir ok, no marker → healed_recently=false); missing dir detected
-  while sshd_reachable stays true (the "TCP up, sessions broken" trap); marker present → healed_recently
-  + timestamp; empty/whitespace marker treated as absent (red-proof for a hub warning with no
-  timestamp); sshd-unreachable reported.
-- `internal/hub` collector: nil reporter → `mgmt_plane` omitted; wired → carried through.
+## Live validation (felhom-pve + dev endpoint) — the spikes re-run
 
-## Live validation (felhom-pve) — the acceptance
+Deployed **manually** (agent self-update needs a pinned operator signer; the box has none — D1's
+self-update path was not usable, stated honestly). Endpoint set up first (ip_forward=1 +
+per-pair forward chain + operator peer via the hub); then agent v0.72.0 + `--enable-oob` artifacts +
+`oob.enabled`.
 
-Deployed **manually** (agent self-update needs an operator-signed op + a pinned signer; the box has
-**no signer pinned**, so D1's self-update path was not usable — stated honestly). Build on 180
-(`-X main.version=0.71.0`, sha `b3b616fe…`) → scp via local → `install -m0755` (backup `.bak-0.70.0`)
-→ restart. Host artifacts installed + timer enabled; `systemd-analyze verify` + `systemd-tmpfiles
---create` + shellcheck all clean; RuntimeDirectory guard verified (no directive, only the warning
-comment).
+- **Core** — operator→box SSH as **`felhom-op`** over the tunnel, with a scoped `sudo pct list`. ✓
+- **A [OF-1]** — stopped `wg-quick@wg-felhom` → agent self-healed (+48s) → box AllowedIPs STILL
+  `{10.77.0.1/32, 10.77.0.250/32}` (the /32 is rendered) → operator SSH survived. Conf-hash stable
+  (no per-tick flap). ✓
+- **B** — coexistence: stock :22 **PID 922 unchanged** across all felhom-sshd churn; both listening;
+  distinct host-key fingerprints (stock `8Tea…` vs felhom `EzuJ…`); **felhom-op DENIED on :22**
+  (`Permission denied (publickey)` — the AuthorizedKeysFile isolation); pveproxy active. ✓
+- **C** — port claim: felhom-sshd on 8822, belt `@ssh_port={8822}` follows; clean/contention/
+  idempotent/exhaustion unit-proven (the flip-flop bug was found + fixed live: once claimed the port is
+  kept unconditionally, since felhom-sshd itself holds it). ✓
+- **D** — belt: LAN→8822 **dropped** (off-tunnel); LAN :22 untouched (safety line); PBS unaffected. ✓
+- **E** — heal: stopped felhom-sshd → agent restored it in ~15s. Config-invalid-no-restart unit-proven
+  (the agent always renders a valid config, so it self-corrects). ✓
+- **F** — endpoint: operator peer survives peersync; a dummy tunnel peer is **dropped box↔box**
+  (§4.5 drop counter=4); PBS healthy. ✓
+- **G** — hub: the healthy `oob` stanza `{active:true, port:8822, reachable:true, config_invalid:false,
+  operator_peer_configured:true, operator_key_configured:true, wg_handshake_age_s:24}` reaches the hub;
+  `oob_degraded` fired during the real felhom-sshd downtime (E) and `oob_recovered` on recovery. ✓
 
-**Auto-heal drill (agent STOPPED — the load-bearing proof):** `systemctl stop felhom-agent` →
-`rm -rf /run/sshd` → a new `:22` session reproduced the incident (`Connection reset` right after the
-banner) → the agent-independent watchdog **recreated `/run/sshd` and a new `:22` session succeeded
-again in 30.0 s (measured), with the agent still `inactive`.** Marker written. This proves the heal is
-login-free AND agent-independent.
-
-**End-to-end visibility chain (hub v0.34.1):** the v0.71.0 report carried
-`mgmt_plane:{privsep_dir_ok:true, sshd_reachable:true, healed_recently:true,
-privsep_healed_at:"2026-07-05T17:15:32Z"}`; the prior healthy report had `healed_recently:false`; the
-old v0.70.0 report had no stanza (correctly absent). The hub monitor raised the
-**`mgmt_plane_healed` warning** at 17:16:21 ("privsep dir … was missing and was AUTO-HEALED …").
-
-**KEXINIT-reset reproduction:** confirmed live in the drill (`Connection reset by 127.0.0.1 port 22`
-immediately after the banner exchange, `/run/sshd` absent).
+**Bugs found + fixed during live validation** (each committed): (1) port persisted under root-owned
+`/etc/felhom-sshd` → moved to the agent StateDir; (2) self-listen flip-flop (isFree(persisted) sees our
+own daemon) → keep the persisted port unconditionally; (3) nil-block emptied the belt/authkeys
+(operator lockout on restart) → fetched-gate like wgtunnel; (4) `reachable` via a dial always failed
+(belt blocks localhost) → listener check; (5) `operator_*_configured` from persistent state (belt +
+authorized_keys), not only the in-memory block.
 
 ## Cross-repo state
 
-- Agent **v0.71.0** live on felhom-pve; three break-glass layers active + enabled (they STAY — this
-  is the shipped feature). Rollback: `.bak-0.70.0`.
-- Hub **v0.34.1** live; break-glass vault + `mgmt_plane_healed` monitor. felhom.eu commits `05d8181`
-  (hub+installer), `28ac5f5` (runbook), `012e5f3` (monitor fix + v0.34.1).
-- **H1 (felhom-sshd) can now assume `/run/sshd` is guaranteed present** (tmpfiles + watchdog) and must
-  never declare `RuntimeDirectory=` — the installer refuses any unit that does.
+- Agent **v0.72.0** live on felhom-pve (sha `f72c2fed`); felhom-sshd active on 8822, belt filled,
+  operator `/32` rendered, oob stanza healthy. Rollback `.bak-0.71.0`. Endpoint + operator peer +
+  felhom-sshd all STAY (shipped). Stock :22 PID 922 unchanged throughout.
+- Hub **v0.35.0** live. felhom.eu commits pushed; doc 06 §4.5/§4.6 amended.
 
-## Notes / observations
+## Observations not acted on (honest ledger)
 
-- **Secret discipline verified:** the root@pam recovery password appears in NO log/commit (the vault
-  handler logs username + length only; red-proofed hub-side).
-- **Least-privilege console user** (vs root@pam): not probed live (PVE gates the node Shell to
-  root@pam historically); G1 ships the proven root@pam path — a non-blocking future item.
-- **Auto-rotation** of the recovery credential is a noted future item; `--rotate-recovery` re-vaults on
-  demand.
-- Heal cadence: OnUnitActiveSec=60s (measured 30 s this run, mid-cycle); worst case ~60 s.
+- **CGNAT** OOB traversal still unproven (the box is single-NAT public-v4, as in both spikes).
+- **IPv6/AAAA** out of scope (arc v4-pinned); the standing ep0-AAAA item persists.
+- **Operator-key auto-rotation** not implemented (re-`PUT /admin/wg/operator-peer` rotates manually).
+- **"Customer network fully down (powered)"** — explicitly OUT OF SCOPE / accepted risk (operator
+  decision 2026-07-05); the smart-plug is the only mitigation and is not part of this task.
+- **`operator_peer_configured` post-restart lag** eliminated by reading the persistent belt (not the
+  0600 root-owned wg conf the non-root agent can't read).
+- Agent binary **not yet published** to Gitea (manual deploy for validation); a fleet rollout should
+  `scripts/publish-agent.sh 0.72.0 <bin>` + Day-0 vouch.