felhom-agent/REPORT.md

# REPORT — TASK H1: OOB operator access (felhom-agent half) — v0.72.0

**Baseline:** felhom-agent `main` @ `4b7c5bf` (v0.71.0) → **v0.72.0** (final binary sha
`f72c2fed…`). felhom.eu @ `a3ee93e` → hub v0.35.0. Trunk-based, direct to `main`. Provenance: BOTH
`SPIKE-felhom-sshd-2026-07-05` [SF-n] and `SPIKE-oob-wg-operator-peer-2026-07-05` [OF-n].

## What shipped (this repo)

- **Operator `/32` rendered into `wg-felhom` AllowedIPs** (`internal/wgtunnel`): `oob_peer_ip` appended
  sorted + byte-stable; RENDERED so it survives self-heal ([OF-1]).
- **Dedicated `felhom-sshd`** (`internal/felhomsshd`): claim (loud-fail exhaustion), render→`sshd -t`→
  reload (never restart-on-change [SF-2]), operator authorized_keys outside ~/.ssh [SF-3], heal with
  cooldown + never-onto-invalid-config, `configs/felhom-sshd.service` with **no `RuntimeDirectory=`**
  [SF-1].
- **Port-adaptive belt** (`internal/felhomsshd/belt.go` + `configs/felhom-oob.nft`): static table, agent
  mutates SET ELEMENTS ONLY (`@operator_ips`/`@ssh_port`) [trap 4]; a nil block never empties it.
- **OOB heartbeat stanza** + `FELHOM_SSHD`/`FELHOM_OOB` sudoers. `oob.enabled` DEFAULT FALSE.

## Tests + red-proofs (all green — `go build/vet/test ./...`, 23 packages ok)

- renderConf: OOB /32 appended sorted + byte-stable; lower-IP-sorts-first; absent = byte-identical
  golden; invalid/v6 rejected. Companion: byte-stability is the anti-flap red-proof.
- claim: clean/contention/idempotent/exhaustion (LOUD-fail is the key negative — non-zero, never :22);
  persisted-port kept unconditionally (the self-listen flip-flop fix).
- config: SAFE template, byte-stable, refuses :22, never `RuntimeDirectory`.
- belt: mutate-then-idempotent (zero nft ops on the 2nd sync) + **never touches rules** (trap 4
  asserted); empty operator empties the set.
- heal: NO restart on an invalid config (red-proof) + 10-min cooldown; status reflects the block.
- hub collector + OOB monitor: stanza ingest + transition-based `oob_degraded`/`oob_recovered` (+ the
  unconfigured-not-alerted and no-stanza-ignored negatives). Red-proof: neutering the emit fails it.

## Live validation (felhom-pve + dev endpoint) — the spikes re-run

Deployed **manually** (agent self-update needs a pinned operator signer; the box has none — D1's
self-update path was not usable, stated honestly). Endpoint set up first (ip_forward=1 +
per-pair forward chain + operator peer via the hub); then agent v0.72.0 + `--enable-oob` artifacts +
`oob.enabled`.

- **Core** — operator→box SSH as **`felhom-op`** over the tunnel, with a scoped `sudo pct list`. ✓
- **A [OF-1]** — stopped `wg-quick@wg-felhom` → agent self-healed (+48s) → box AllowedIPs STILL
  `{10.77.0.1/32, 10.77.0.250/32}` (the /32 is rendered) → operator SSH survived. Conf-hash stable
  (no per-tick flap). ✓
- **B** — coexistence: stock :22 **PID 922 unchanged** across all felhom-sshd churn; both listening;
  distinct host-key fingerprints (stock `8Tea…` vs felhom `EzuJ…`); **felhom-op DENIED on :22**
  (`Permission denied (publickey)` — the AuthorizedKeysFile isolation); pveproxy active. ✓
- **C** — port claim: felhom-sshd on 8822, belt `@ssh_port={8822}` follows; clean/contention/
  idempotent/exhaustion unit-proven (the flip-flop bug was found + fixed live: once claimed the port is
  kept unconditionally, since felhom-sshd itself holds it). ✓
- **D** — belt: LAN→8822 **dropped** (off-tunnel); LAN :22 untouched (safety line); PBS unaffected. ✓
- **E** — heal: stopped felhom-sshd → agent restored it in ~15s. Config-invalid-no-restart unit-proven
  (the agent always renders a valid config, so it self-corrects). ✓
- **F** — endpoint: operator peer survives peersync; a dummy tunnel peer is **dropped box↔box**
  (§4.5 drop counter=4); PBS healthy. ✓
- **G** — hub: the healthy `oob` stanza `{active:true, port:8822, reachable:true, config_invalid:false,
  operator_peer_configured:true, operator_key_configured:true, wg_handshake_age_s:24}` reaches the hub;
  `oob_degraded` fired during the real felhom-sshd downtime (E) and `oob_recovered` on recovery. ✓

**Bugs found + fixed during live validation** (each committed): (1) port persisted under root-owned
`/etc/felhom-sshd` → moved to the agent StateDir; (2) self-listen flip-flop (isFree(persisted) sees our
own daemon) → keep the persisted port unconditionally; (3) nil-block emptied the belt/authkeys
(operator lockout on restart) → fetched-gate like wgtunnel; (4) `reachable` via a dial always failed
(belt blocks localhost) → listener check; (5) `operator_*_configured` from persistent state (belt +
authorized_keys), not only the in-memory block.

## Cross-repo state

- Agent **v0.72.0** live on felhom-pve (sha `f72c2fed`); felhom-sshd active on 8822, belt filled,
  operator `/32` rendered, oob stanza healthy. Rollback `.bak-0.71.0`. Endpoint + operator peer +
  felhom-sshd all STAY (shipped). Stock :22 PID 922 unchanged throughout.
- Hub **v0.35.0** live. felhom.eu commits pushed; doc 06 §4.5/§4.6 amended.

## Observations not acted on (honest ledger)

- **CGNAT** OOB traversal still unproven (the box is single-NAT public-v4, as in both spikes).
- **IPv6/AAAA** out of scope (arc v4-pinned); the standing ep0-AAAA item persists.
- **Operator-key auto-rotation** not implemented (re-`PUT /admin/wg/operator-peer` rotates manually).
- **"Customer network fully down (powered)"** — explicitly OUT OF SCOPE / accepted risk (operator
  decision 2026-07-05); the smart-plug is the only mitigation and is not part of this task.
- **`operator_peer_configured` post-restart lag** eliminated by reading the persistent belt (not the
  0600 root-owned wg conf the non-root agent can't read).
- Agent binary **not yet published** to Gitea (manual deploy for validation); a fleet rollout should
  `scripts/publish-agent.sh 0.72.0 <bin>` + Day-0 vouch.