felhom-agent/REPORT.md

REPORT — TASK H1: OOB operator access (felhom-agent half) — v0.72.0

Baseline: felhom-agent main @ 4b7c5bf (v0.71.0) → v0.72.0 (final binary sha f72c2fed…). felhom.eu @ a3ee93e → hub v0.35.0. Trunk-based, direct to main. Provenance: BOTH SPIKE-felhom-sshd-2026-07-05 [SF-n] and SPIKE-oob-wg-operator-peer-2026-07-05 [OF-n].

What shipped (this repo)

• Operator /32 rendered into wg-felhom AllowedIPs (internal/wgtunnel): oob_peer_ip appended sorted + byte-stable; RENDERED so it survives self-heal ([OF-1]).
• Dedicated felhom-sshd (internal/felhomsshd): claim (loud-fail exhaustion), render→sshd -t→ reload (never restart-on-change [SF-2]), operator authorized_keys outside ~/.ssh [SF-3], heal with cooldown + never-onto-invalid-config, configs/felhom-sshd.service with no RuntimeDirectory= [SF-1].
• Port-adaptive belt (internal/felhomsshd/belt.go + configs/felhom-oob.nft): static table, agent mutates SET ELEMENTS ONLY (@operator_ips/@ssh_port) [trap 4]; a nil block never empties it.
• OOB heartbeat stanza + FELHOM_SSHD/FELHOM_OOB sudoers. oob.enabled DEFAULT FALSE.

Tests + red-proofs (all green — go build/vet/test ./..., 23 packages ok)

• renderConf: OOB /32 appended sorted + byte-stable; lower-IP-sorts-first; absent = byte-identical golden; invalid/v6 rejected. Companion: byte-stability is the anti-flap red-proof.
• claim: clean/contention/idempotent/exhaustion (LOUD-fail is the key negative — non-zero, never :22); persisted-port kept unconditionally (the self-listen flip-flop fix).
• config: SAFE template, byte-stable, refuses :22, never RuntimeDirectory.
• belt: mutate-then-idempotent (zero nft ops on the 2nd sync) + never touches rules (trap 4 asserted); empty operator empties the set.
• heal: NO restart on an invalid config (red-proof) + 10-min cooldown; status reflects the block.
• hub collector + OOB monitor: stanza ingest + transition-based oob_degraded/oob_recovered (+ the unconfigured-not-alerted and no-stanza-ignored negatives). Red-proof: neutering the emit fails it.

Live validation (felhom-pve + dev endpoint) — the spikes re-run

Deployed manually (agent self-update needs a pinned operator signer; the box has none — D1's self-update path was not usable, stated honestly). Endpoint set up first (ip_forward=1 + per-pair forward chain + operator peer via the hub); then agent v0.72.0 + --enable-oob artifacts + oob.enabled.

• Core — operator→box SSH as felhom-op over the tunnel, with a scoped sudo pct list. ✓
• A [OF-1] — stopped wg-quick@wg-felhom → agent self-healed (+48s) → box AllowedIPs STILL {10.77.0.1/32, 10.77.0.250/32} (the /32 is rendered) → operator SSH survived. Conf-hash stable (no per-tick flap). ✓
• B — coexistence: stock :22 PID 922 unchanged across all felhom-sshd churn; both listening; distinct host-key fingerprints (stock 8Tea… vs felhom EzuJ…); felhom-op DENIED on :22 (Permission denied (publickey) — the AuthorizedKeysFile isolation); pveproxy active. ✓
• C — port claim: felhom-sshd on 8822, belt @ssh_port={8822} follows; clean/contention/ idempotent/exhaustion unit-proven (the flip-flop bug was found + fixed live: once claimed the port is kept unconditionally, since felhom-sshd itself holds it). ✓
• D — belt: LAN→8822 dropped (off-tunnel); LAN :22 untouched (safety line); PBS unaffected. ✓
• E — heal: stopped felhom-sshd → agent restored it in ~15s. Config-invalid-no-restart unit-proven (the agent always renders a valid config, so it self-corrects). ✓
• F — endpoint: operator peer survives peersync; a dummy tunnel peer is dropped box↔box (§4.5 drop counter=4); PBS healthy. ✓
• G — hub: the healthy oob stanza {active:true, port:8822, reachable:true, config_invalid:false, operator_peer_configured:true, operator_key_configured:true, wg_handshake_age_s:24} reaches the hub; oob_degraded fired during the real felhom-sshd downtime (E) and oob_recovered on recovery. ✓

Bugs found + fixed during live validation (each committed): (1) port persisted under root-owned /etc/felhom-sshd → moved to the agent StateDir; (2) self-listen flip-flop (isFree(persisted) sees our own daemon) → keep the persisted port unconditionally; (3) nil-block emptied the belt/authkeys (operator lockout on restart) → fetched-gate like wgtunnel; (4) reachable via a dial always failed (belt blocks localhost) → listener check; (5) operator_*_configured from persistent state (belt + authorized_keys), not only the in-memory block.

Cross-repo state

• Agent v0.72.0 live on felhom-pve (sha f72c2fed); felhom-sshd active on 8822, belt filled, operator /32 rendered, oob stanza healthy. Rollback .bak-0.71.0. Endpoint + operator peer + felhom-sshd all STAY (shipped). Stock :22 PID 922 unchanged throughout.
• Hub v0.35.0 live. felhom.eu commits pushed; doc 06 §4.5/§4.6 amended.

Observations not acted on (honest ledger)

• CGNAT OOB traversal still unproven (the box is single-NAT public-v4, as in both spikes).
• IPv6/AAAA out of scope (arc v4-pinned); the standing ep0-AAAA item persists.
• Operator-key auto-rotation not implemented (re-PUT /admin/wg/operator-peer rotates manually).
• "Customer network fully down (powered)" — explicitly OUT OF SCOPE / accepted risk (operator decision 2026-07-05); the smart-plug is the only mitigation and is not part of this task.
• operator_peer_configured post-restart lag eliminated by reading the persistent belt (not the 0600 root-owned wg conf the non-root agent can't read).
• Agent binary not yet published to Gitea (manual deploy for validation); a fleet rollout should scripts/publish-agent.sh 0.72.0 <bin> + Day-0 vouch.