# SPIKE — OOB management over the existing WG arc (operator peer, isolation, mutual repair) — 2026-07-05

> **STATUS: COMPLETE — verdict below.** Empirically validates the unproven mechanisms of the OOB
> (out-of-band operator access) design BEFORE the production spec. No production code shipped; no
> agent binary change; every config change on the live demo arc reverted at cleanup (§13, verified).
> The design under test: an **operator peer on the existing hub-and-spoke WG arc** (doc 06),
> endpoint-forwarded **operator→box only**, box sshd reachable solely from the operator tunnel IP,
> while §4.5 box↔box isolation stays intact — and the hub desired-state channel as the tunnel's
> **repair path**.

**Class:** SPIKE (empirical; no product code). **Repos:** felhom.eu (this doc only); felhom-agent
read-only for grounding (`internal/wgtunnel/{manager,loop}.go`, `internal/desired/syncer.go`,
`configs/felhom-agent.sudoers`); felhom-hub read-only (`internal/wgsync/reconciler.go`,
`internal/store/wg.go`).

**Probe ends (the LIVE demo arc, ground-truthed in P0):**
- **Endpoint** = `felhom-hetzner` (Hetzner CX23, Debian 13, `167.233.158.164`, DNS `ep0.felhom.eu`),
  the **dev** offsite endpoint: WG server `wg0` on **443/UDP**, tunnel `10.77.0.0/24`, endpoint
  `10.77.0.1`, PBS `felhom-offsite`. Peer list is hub-driven (`internal/wgsync` SSH push →
  `felhom-peersync`). CC-SSH-reachable as root ✓ (the P0 access gate passed).
- **Box** = `felhom-pve` / node `demo-felhom` (`192.168.0.162`), agent **v0.70.0**, `wg-felhom`
  client `10.77.0.2/32`, tunnel to `ep0:443`. CC's LAN SSH to it was the safety line throughout.

**Verdict (one line):** **GO — an operator peer on the existing arc reaches a box's sshd through the
endpoint with a single added `/32`, at ~24 ms interactive latency, WITHOUT breaking §4.5 isolation
(both negatives held with packet-level evidence); mutual repair is REAL and already built (the
agent self-healed a stopped tunnel in ~15 s with zero hub/LAN involvement, and OOB works with the
agent dead); the one load-bearing gap the TASK must close is that the operator `/32` must become a
RENDERED conf field — a runtime-only `wg set` is wiped by the agent's own self-heal/restart.**

---

## 0. What "production ep0" this demo could and could not represent

The demo endpoint (`felhom-hetzner`) **is** a real public dual-stack cloud VM with a public IPv4,
443/UDP WG, hub-driven peersync, and a real PBS — architecturally identical to the intended
production `ep0`. So forwarding, isolation, and the SSH-through-endpoint path are represented
faithfully. What it does **not** represent: (a) a **CGNAT** customer line — the box here is
`felhom-pve` on the operator's One-Hungary fixed-cable line (public IPv4 `37.191.56.193`,
single-NAT), same caveat as the prior spike §5; the OOB path rides the same outbound-mapping
mechanism, so this is low-risk but unproven on true `100.64/10`. (b) A **second real customer box**
— the "dummy customer" was a throwaway netns peer at the endpoint, which is sufficient for the
topology-isolation negative but is not a full second agent. (c) The production `ep0` **AAAA** is
still mis-set per the standing OPERATOR follow-up (memory: "OPERATOR fix ep0 AAAA"); this spike
used v4 throughout, consistent with the agent's v4-pin (doc 06 §4.2).

---

## 1. P0 — ground-truth discovery (no changes) — **GATE PASSED**

**Endpoint `wg0` (redacted; `wg show` prints only the public key — `dump` is BANNED, S1 rule):**
```
interface: wg0   listening port: 443
peer: yV5hFFZF548r6kq2Nl/CiSyOtFb0xYnvEAjw1ofrTCg=   (= the box)
  endpoint: 37.191.56.193:54900   allowed ips: 10.77.0.2/32
  latest handshake: ~1.5 min ago   transfer: 5.51 GiB rx / 16.43 GiB tx
wg0.conf: [Interface] Address 10.77.0.1/24, ListenPort 443, MTU 1420 ; [Peer] box only
sysctl: net.ipv4.ip_forward = 0   net.ipv6.conf.all.forwarding = 0
nft: table inet filter { input: drop-policy, allows 22, 443/udp, 8007 iif wg0 only, icmp ;
                         forward: policy DROP (empty) }
PBS: proxmox-backup + proxmox-backup-proxy active
```
**Box `wg-felhom` (redacted):**
```
wg-felhom.conf (agent-rendered, "DO NOT EDIT"): Address 10.77.0.2/32, MTU 1280,
  Peer f3d1ZI7…=  Endpoint 167.233.158.164:443  AllowedIPs 10.77.0.1/32  PersistentKeepalive 25
unit wg-quick@wg-felhom: enabled, active (exited)   handshake fresh
agent marker /var/lib/felhom-agent/wg/registered.json:
  {"pubkey":"yV5hFFZF548r6kq2Nl/CiSyOtFb0xYnvEAjw1ofrTCg=","assigned_ip":"10.77.0.2/32","generation":5}
sshd: port 22, ListenAddress 0.0.0.0:22 + [::]:22 (NOT interface-scoped today)
pve-firewall: disabled ;  nft list tables: EMPTY (no host nft tables at baseline)
```
**Reconcile model (source + journal):**
- **Box→tunnel reconcile is CONTINUOUS, not generation-gated.** `wgtunnel.Loop.Run`
  (`internal/wgtunnel/loop.go`) calls `Manager.Apply` **on every tick** — `wg_tunnel.interval_seconds
  = 60` (agent.json, confirmed) — AND on each desired-state nudge. The desired-state *fetch* is
  generation-gated (`desired/syncer.go` `OnEnvelope`, hub poll `poll_seconds = 900`), but the
  wireguard block is cached and re-applied every 60 s regardless. **Consequence for probes:** any
  hand-edit to `wg-felhom.conf` or a runtime `wg set` that diverges from the agent's rendered conf
  is at risk of being reconciled within ≤60 s. This is why the AllowedIPs widening (P2) used
  runtime-only `wg set` (which the agent does NOT touch — it only rewrites the *conf* and restarts
  on a conf-hash change) and why the self-heal in P5b wiped it (§P5).
- **Endpoint peer-list reconcile** (`hub/internal/wgsync/reconciler.go`): declarative FULL-list push
  every **5 min** (or on a mutation Trigger); drift (a manually-added/removed peer) is erased on the
  next push. Proven live at cleanup (§13).

**Gate:** endpoint is CC-SSH-accessible as root ✓ → proceed.

---

## 2. P1 — operator peer, endpoint-side — **GO**

Throwaway operator + dummy-customer keypairs generated on the endpoint (`wg genkey`, 0600,
private keys never printed). Registered BOTH into the **hub** `wg_peers` registry (not hand-added
to `wg0`) so they survive the 5-min reconciler — via `sqlite3 /data/hub.db` in the hub pod
(stdin-piped SQL to avoid the base64 `=`/`+` quoting trap):
```
operator  cdmN4U+fjR18zBk+SKoceJQyz9HgA9+hN8/FiKF1u0o=  10.77.0.250
dummy     tQ6eJC9y8pSKtam44DaSrrtYBZYUvZG2z1stS6PsaD4=  10.77.0.251
```
The hub reconciler pushed both to `wg0` in **~150 s** (next 5-min tick fired early on a warm loop).

**Endpoint forwarding, enabled NARROWLY** — `ip_forward=1` (was 0) + a named, one-flush nft chain
`felhom_spike_oob` hung off `forward`, ALL rules scoped `iifname wg0 oifname wg0`:
```
table inet filter { chain forward { … ; jump felhom_spike_oob } }
chain felhom_spike_oob {
  iifname "wg0" oifname "wg0" ct state established,related accept          # conntrack replies
  iifname "wg0" oifname "wg0" ip saddr 10.77.0.250 ip daddr 10.77.0.2 counter accept   # operator→box
  iifname "wg0" oifname "wg0" ip daddr 10.77.0.250 counter drop            # box→operator NEW: drop
  iifname "wg0" oifname "wg0" counter drop                                 # box↔box (any other): drop
}
```
The operator "laptop" is a **netns client** on the endpoint (`spike-op` netns, veth to a
`192.168.99.0/30` link, `wg-op` interface, MTU 1280, `AllowedIPs 10.77.0.0/24`) — an acceptable
stand-in; the crypto+routing path through `wg0` is identical to a remote laptop. **GO:** operator
peer handshakes with `wg0` and pings the endpoint `10.77.0.1` at **0.2 ms** (on-box netns).

---

## 3. P2 — box-side AllowedIPs widening + SSH-over-tunnel + PBS — **GO**

**The minimal box change, runtime-only** (nothing persists past a wg-quick restart — see P5):
```
wg set wg-felhom peer f3d1ZI7…= allowed-ips 10.77.0.1/32,10.77.0.250/32
ip route add 10.77.0.250/32 dev wg-felhom
```
Exactly one added `/32` (+ the return route). `wg-felhom.conf` mtime UNCHANGED (2026-07-04) — the
agent's rendered conf was not touched.

From the operator netns, through the endpoint's forwarding:
```
ping 10.77.0.2  → 5/5, avg 24.7 ms, TTL 63  (TTL 63 = exactly one forwarded hop, i.e. via ep0)
ssh root@10.77.0.2 "hostname; pct list; uptime"  → interactive session in 1.205 s:
    demo-felhom ;  9201 running demo-felhom (+9001/9100/9999) ;  load avg 0.43
```
Interactive SSH over the tunnel works end-to-end.

**PBS-path check (mandatory, immediately after the box change):**
```
wg show wg-felhom latest-handshakes → age 73 s (fresh, advancing)
pvesm status --storage felhom-offsite → active   (PVE reaches PBS at 10.77.0.1:8007 over the tunnel)
openssl s_client -connect 10.77.0.1:8007 → CONNECTED, Certificate chain present
```
**PBS unaffected = GO condition met.** Adding the operator reach did not perturb the backup path.

---

## 4. P3 — isolation negative probes (§4.5 must survive) — **GO (both negatives, counter-proven)**

**Negative 1 — dummy customer → box: DROPPED.** The dummy peer handshakes with the endpoint (0.2 ms
to `10.77.0.1` — it IS a registered peer) but:
```
spike-cust → ping 10.77.0.2  → 3/3 packets LOST (100%)
spike-cust → tcp/22 10.77.0.2 → REFUSED/TIMEOUT
felhom_spike_oob box↔box drop counter: 0 → 8 packets / 552 bytes   (the drops are real, not a route miss)
```
**Negative 2 — box → operator: NEW connections DROPPED, replies flow.** With an operator→box SSH
session OPEN (so ESTABLISHED replies were flowing), from the box:
```
felhom-pve → ping 10.77.0.250        → 3/3 LOST
felhom-pve → tcp/22 & tcp/80 10.77.0.250 → REFUSED/TIMEOUT
felhom_spike_oob box→operator drop counter: 0 → 13 packets / 852 bytes
  (while the operator→box accept counter incremented normally — conntrack replies unaffected)
```
Both hold with **packet-level evidence** (nft counters), not merely "command timed out". §4.5
isolation survives the operator-peer addition: a box still cannot reach another box, and a box
cannot initiate to the operator.

---

## 5. P4 — sshd scoping on the box (belt rule) — **GO (all three)**

A named, one-flush nft table on `felhom-pve` (baseline had NO host nft tables), independent hook
so LAN and other interfaces are untouched:
```
table inet felhom_spike_oob {
  chain input { type filter hook input priority -10; policy accept;
    iifname "wg-felhom" tcp dport 22 ip saddr 10.77.0.250 counter accept   # operator only
    iifname "wg-felhom" tcp dport 22 counter drop                          # any other tunnel src
  } }
```
Verified:
1. **LAN SSH from CC — unaffected** (the safety line): the rule matches `iifname wg-felhom` only;
   LAN commands kept running throughout. ✓
2. **Operator SSH over the tunnel — still works** (accept counter 27 pkts). ✓
3. **Defense-in-depth — dummy refused at the HOST even if the endpoint belt failed:** I temporarily
   simulated an endpoint-belt failure (added an endpoint forward-accept for dummy→box AND widened
   the box AllowedIPs to include `10.77.0.251/32`). Result: dummy→box **ping now passes** (endpoint
   forwards it, 27 ms) but dummy→box **SSH is dropped at the host** (host belt drop counter 5) —
   `Connection timed out`. The two layers are independent; the host belt holds when the endpoint
   layer is bypassed. The temporary widening was reverted immediately after. ✓

---

## 6. P5 — mutual repair (the crown probe)

### 6a. Agent dead, wg alive — **GO**
`systemctl stop felhom-agent` (via LAN, simulating no other way in). Then from the operator peer
**over the tunnel**: SSH into the box (agent DOWN → OOB does not depend on the agent), and
`systemctl reset-failed felhom-agent; systemctl start felhom-agent` from that session →
`active`, `felhom-agent 0.70.0`. **OOB works with the agent down AND repairs it.**

### 6b. wg dead, agent alive — **GO, ~15 s, and this is THE central TASK finding**
`systemctl stop wg-quick@wg-felhom` (interface gone). Operator SSH over the tunnel now
`Connection timed out` (OOB down, confirmed). Then, watching the endpoint's view of the box peer
handshake with **zero LAN intervention**:
```
box re-handshake observed at +15 s   (agent log: "wgtunnel: service inactive with current conf —
                                       re-enabling (self-heal)")
```
The agent's reconcile loop **already covers a stopped unit** — `Manager.ensureTunnelLocked`'s
self-heal branch (`internal/wgtunnel/manager.go:394-402`): conf hash current + `isActive()` false →
`systemctl enable --now`. No `wg_repair`-shaped desired action is needed; the "wg dead, agent alive"
repair path exists today and fires within one 60 s tick (here 15 s). **This closes O-B (mutual
repair via the desired-state channel) — with a caveat that becomes the TASK's main work:**

> **THE GAP.** The self-heal rebuilds `wg-felhom` **purely from the agent-rendered conf**, whose
> `AllowedIPs` is derived solely from `pbs_tunnel_ip/32` (`renderConf`, `manager.go:239`:
> `AllowedIPs = <pbsIP>/32`). After the self-heal, the box's live AllowedIPs was back to **only
> `10.77.0.1/32`** — the operator `/32` I added at runtime was **wiped**, and OOB SSH failed until I
> re-added it. So **OOB survival across the agent's own self-heal/restart/reboot REQUIRES the
> operator peer's `/32` to be part of what the agent RENDERS into the conf** — a runtime `wg set` is
> not durable. (Re-adding `10.77.0.250/32` at runtime immediately restored OOB, confirming the
> mechanism; it is only the *durability* that needs the rendered-conf change.)

### 6c. Failure-domain sweep with an open OOB session — **GO**
Over a single open operator→box tunnel session: attempted `cloudflared` stop (it is NOT a guest
systemd unit — it runs as a **Docker container** in the controller stack, so the unit stop no-op'd;
noted), then `pct stop 9201` (whole guest down). **The session stayed alive** (`ALIVE_2`, handshake
fresh) — the tunnel terminates **host-side** on `felhom-pve`, so guest death cannot touch it — and
`pct start 9201` from the SAME session brought the guest back. Post-check: `felhom-controller Up
(healthy)`, `cloudflared`/`nextcloud` containers `Up (healthy)`. OOB survives the failure domains it
exists for (guest down; and by construction cloudflared/controller down, since OOB is host-side and
independent of the guest's own edge).

---

## 7. P6 — usability + environment (record-only)

- **Interactive feel over MTU 1280:** `top -bn1`, `pct exec 9201 -- …` (into the guest), and a
  20 MB `dd | wc -c` bulk pull all ran cleanly over the tunnel at ~24 ms RTT — no black-holing, no
  stalls (MTU 1280 is the doc 06 §4.3 floor; interactive OOB is well within it).
- **Endpoint load** during the probes: `load average 0.10` — WG forwarding + a shell session is
  negligible on a CX23.
- **Active port: 443/UDP** (`wg show wg0 listen-port → 443`). Relevant to CGNAT: this is the
  friendly-port path already; the box's NAT source was `37.191.56.193:47922` (single-NAT public v4,
  not CGNAT — the standing caveat).
- **Keepalive:** `PersistentKeepalive 25` on the box; handshakes stayed fresh (<125 s) throughout.

---

## 8. Findings that must shape the TASK

1. **Render the operator `/32` into the agent conf — the one required agent change (§6b).** OOB
   reach must survive the agent's self-heal/restart/reboot, so the operator peer's tunnel `/32`
   must be a field the agent puts in `wg-felhom.conf` `AllowedIPs` (today hard-derived from
   `pbs_tunnel_ip` alone). Shape options for the TASK to pick: (a) extend the hub `wireguard`
   desired-state block with an optional `oob_peer_ip` (or a small `extra_allowed_ips` list) and have
   `renderConf` append it; (b) a separate `[Peer]`-less allowed-ips widening is NOT enough (WG ties
   AllowedIPs to the *server* peer here, and all tunnel traffic transits the one endpoint peer, so
   the operator `/32` just joins the existing peer's AllowedIPs). Keep the derivation deterministic
   and validated (the existing `netip.ParsePrefix`/`/32` checks). **Nothing else in the agent needs
   to change for O-A/O-B** — reconcile continuity + self-heal already do the repair.

2. **Mutual repair (O-B) is already built — do NOT add a `wg_repair` action.** The 60 s reconcile
   tick + `ensureTunnelLocked` self-heal restored a stopped tunnel in 15 s unaided (§6b). The TASK
   should *rely on* this, and at most add the tunnel-health→alert wiring already slated for S6 (doc
   06 §4.6 stanza exists). Endpoint-side drift repair likewise already works (5-min declarative
   full-list push erased the spike peers from `wg0.conf` at cleanup, §13).

3. **The exact nft rulesets that worked (copy-paste ready).** Endpoint forward chain and the
   host belt are in §2 and §5 verbatim. Both are named, one-flush tables. Production shape:
   - **Endpoint:** `ip_forward=1` + a `felhom_oob` forward chain: `ct established,related accept`;
     `saddr <operator/32> daddr <box/32> accept` **per (operator,box) pair**; a catch-all
     `iifname wg0 oifname wg0 … drop` to preserve §4.5 for every non-OOB pair. Note this makes the
     endpoint's forward posture **per-pair allow-list** instead of doc 06 §4.5's blanket
     "forwarding OFF" — the TASK must state that the OOB feature deliberately turns forwarding ON
     but gates it to operator→box pairs only, and that the box↔box drop is now an explicit rule, not
     an absent capability. Peersync (`felhom-peersync.sh`) would need to learn these forward rules,
     OR they live in the endpoint's static nftables and only the peer *list* stays hub-driven
     (simpler; recommended — the operator set is small and stable).
   - **Box (belt, defense-in-depth):** a `felhom_oob` inet table, `input` hook, `iifname wg-felhom
     tcp dport 22 ip saddr <operator/32> accept` then `iifname wg-felhom tcp dport 22 drop`. Scope
     strictly to `wg-felhom`; never touch LAN sshd (the safety line). This belongs in the agent's
     managed surface if the box is to enforce it (a new narrow grant), OR ship as host-install
     static config; the spike proved the *rule* works, the TASK picks the ownership.

4. **AllowedIPs method: runtime `wg set` is NOT durable (finding 1); the durable path is the
   rendered conf.** For the endpoint side, the operator peer is a normal hub `wg_peers` row
   (survives the reconciler) — that part needs no new mechanism, only that the operator peer is
   *not* a customer host (host_id empty, as the spike used) and is excluded from any customer-facing
   accounting.

5. **sshd scoping shape.** Today box sshd listens on `0.0.0.0:22 + [::]:22` (P0). The spike scoped
   access with an **nft belt on `iifname wg-felhom`**, NOT by changing `ListenAddress` — deliberate:
   an interface-bound `ListenAddress` would fight the LAN safety line and the tunnel's late bring-up.
   The TASK should keep sshd listening broadly and gate at the packet layer (belt rule), exactly as
   §5. The endpoint forward rule is the primary gate; the box belt is defense-in-depth (both proven
   independent in §5).

6. **Measured latencies (production budget input).** Operator→box forwarded RTT **~24 ms** (One
   Hungary ↔ Hetzner-DE); interactive SSH incl. `pct list` **1.2 s** wall; self-heal repair **~15 s**;
   endpoint peer-push convergence **~150 s** (≤5 min bound). All comfortably interactive.

7. **Reconcile continuity constraint (from P0, load-bearing for any future OOB probe/op).** The box
   applies its conf every 60 s. Any OOB mechanism that mutates box wg state MUST go through the
   agent's rendered conf (finding 1), never a side-channel `wg set`/hand-edit — those are silently
   reverted within a tick. This is a *feature* (drift-repair) to build with, not around.

8. **Endpoint-access + production prerequisites discovered.** (a) The demo endpoint is a real public
   VM, so O-A is faithfully represented; production `ep0` needs the **AAAA fixed** (standing
   OPERATOR item) though the v4-pin means OOB rides v4 regardless. (b) `cloudflared` in the guest is
   a **Docker container, not a systemd unit** — any runbook step that "stops cloudflared" must
   `docker stop` it, not `systemctl`. (c) The hub `wg_peers` insert path for a NON-host operator
   peer exists in the store (`AddWGPeer` with empty host_id — the S1 unbound-admin-row shape); a
   production OOB feature can reuse it or add an explicit operator-peer admin endpoint.

---

## 9. What this spike did NOT prove (honest ledger)

- **True CGNAT** OOB traversal — the box is single-NAT public-v4 (same caveat as the prior spike
  §5). OOB rides the same outbound WG mapping as the backup path, so low-risk, but unproven on
  `100.64/10`.
- **A second real agent box** — the dummy customer was a netns peer (sufficient for the topology
  isolation negative; not a full agent). The box↔box drop is proven by topology + counters, which
  is the load-bearing property.
- **The rendered-conf operator-`/32` change itself** — finding 1 is the *diagnosis* (runtime `wg
  set` wiped by self-heal, re-add restored OOB); the agent code change to render it is TASK work,
  not shipped here.
- **Endpoint peersync owning the forward rules** — the spike put forward rules in ad-hoc nft; the
  TASK decides static-vs-hub-driven (finding 3 recommends static forward rules + hub-driven peer
  list).
- **Long-horizon hold** of an idle OOB session (minutes-scale only here; the backup-path 32-min
  soak is in doc 06 §7).

---

## 10. Cleanup assertion (§13 of the plan) — VERIFIED

**Hub:** both spike `wg_peers` rows deleted (`WHERE note LIKE 'spike-oob-%'`); registry back to the
single real peer `demo-felhom-01 / 10.77.0.2`.
**Endpoint `felhom-hetzner`:** operator+dummy peers removed from live `wg0`; the 5-min reconciler
then rewrote `wg0.conf` to the box-only peer (drift-repair confirmed live — the persisted conf
self-healed); `felhom_spike_oob` forward chain + jump rule deleted; `ip_forward` restored to **0**
(the P0 value); `spike-op`/`spike-cust` netns + veths deleted; throwaway WG + SSH keys `shred -u`'d,
`/root/spike-oob` removed. Final: `nft` forward chain is the empty drop-policy chain (P0), `wg0`
lists only the box, `ip_forward=0`.
**Box `felhom-pve`:** runtime AllowedIPs back to `10.77.0.1/32` (operator `/32` removed) + return
route deleted; `felhom_spike_oob` host table deleted (`nft list tables` empty, as P0); the
`spike-oob-operator` line removed from `/root/.ssh/authorized_keys` (grep count 0); `/tmp/wgdown_t0`
removed. Final baseline re-verified: `felhom-agent` active, `wg-quick@wg-felhom` active + handshake
fresh (age 92 s), `pvesm status felhom-offsite` active, guest 9201 running + controller healthy, no
host nft tables, LAN SSH fine.
**No repo/hub/agent/manifest production change** — this commit is docs-only.

---

## 11. Method bar honored

`wg show <if> dump` NEVER run (private-key leak ban, S1); only `latest-handshakes` + redacted conf
reads. Every mutation was a named one-flush nft table or a runtime-only `wg set`, inventoried as
made. PBS-path re-checked after the box wg change (P2). Negative probes carry nft-counter evidence,
not just timeouts. Surprises recorded as findings (the self-heal wipe of the runtime `/32` — finding
1 — is the headline example).
