docs: SPIKE — offsite-backup connectivity: plain WireGuard wins the ladder (transport decision made)
P0-P7 probe ladder on real ends (demo-felhom PVE host <-> throwaway Hetzner). Verdict: GO at H1 — plain host-side WG, box-dials-out to a small public endpoint; Headscale is a separable fleet layer, not a traversal necessity. Line is plain-NAT (not CGNAT) with zero IPv6 — recorded honestly; CGNAT vantage deferred to Peti VM 110. Real 2 GiB worst-case PBS backup over the tunnel at the full home uplink, pin intact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
+15
@@ -3,6 +3,21 @@
|
|||||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||||
|
|
||||||
|
- **2026-07-03 — OFFSITE-BACKUP TRANSPORT DECIDED (connectivity spike, GO at the lowest rung)** —
|
||||||
|
plain WireGuard, host-side, box-dials-out to a small public endpoint; **no Headscale/DERP needed
|
||||||
|
for backup** (separable fleet layer, adoptable later). Live-proven on demo-felhom → throwaway
|
||||||
|
Hetzner: 11.4-min idle hold, PBS API through the tunnel with the exact-cert pin intact
|
||||||
|
(positive + negative), real 2 GiB worst-case backup at 4.26 MiB/s (= the home uplink; tunnel
|
||||||
|
overhead ~5%), UDP 51820 *and* 443 both pass. P0 honesty: the operator's line is plain-NAT
|
||||||
|
(fixed public IP, One Hungary), NOT CGNAT, and has zero IPv6 — CGNAT confirmation deferred to
|
||||||
|
Peti's VM 110 (second ISP). P7 surprise: kernel WG *works* inside the unprivileged guest (no TUN
|
||||||
|
needed) — host placement stands on architecture (backups originate host-side; cloudflared
|
||||||
|
precedent), not infeasibility. Evidence + architecture recommendation (endpoint VM shapes, key
|
||||||
|
custody via IdentityBundle escrow, per-peer /32 isolation):
|
||||||
|
`documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`. **This unblocks the
|
||||||
|
DR-completeness work: next task = production connectivity spec** (endpoint VM, agent-managed WG
|
||||||
|
service, hub peer registry, escrow join). Throwaway Hetzner box parked powered-off (secrets
|
||||||
|
shredded); demo-felhom verified residue-free.
|
||||||
- **2026-07-03 — 4th skill: `felhom-app-catalog`** (catalog authoring workflow; points at
|
- **2026-07-03 — 4th skill: `felhom-app-catalog`** (catalog authoring workflow; points at
|
||||||
app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by `install_skills.py`, 4/4
|
app-catalog REUSE.md §1–2, no duplicated tables; auto-discovered by `install_skills.py`, 4/4
|
||||||
found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected
|
found fresh-session). Proven by finalizing SparkyFitness live on demo (image-inspected
|
||||||
|
|||||||
@@ -4,76 +4,69 @@
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# REPORT — `felhom-app-catalog` skill (4th) + SparkyFitness finalized as its worked example (2026-07-03)
|
# REPORT — SPIKE: offsite-backup connectivity (WireGuard, home-NAT traversal) (2026-07-03)
|
||||||
|
|
||||||
Companion half in `app-catalog-felhom.eu/REPORT.md` (the template changes + live-deploy evidence).
|
Docs-only run; evidence doc: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`.
|
||||||
|
No production code, no Headscale on the real hub, no agent change, no persistent felhom-pve config.
|
||||||
|
|
||||||
## Baselines
|
## Verdict
|
||||||
|
|
||||||
| Repo | Base → head |
|
**GO at the lowest rung (H1): plain WireGuard, host-side, box-dials-out to a small public
|
||||||
|---|---|
|
endpoint.** No coordinator, no relay, no mesh. Headscale (P5: 0.29.2 stood up throwaway) is a
|
||||||
| felhom.eu | `7ad111c` → this push (`skills/felhom-app-catalog/SKILL.md` NEW + docs) |
|
*separable fleet-management layer* — TLS control plane + DB + tailscaled-per-box + Tailscale-hosted
|
||||||
| app-catalog-felhom.eu | `15cdf7b` → `c7490f7` (sparkyfitness finalized + REUSE.md probe-naming row) |
|
DERP by default — not a traversal necessity for hub-and-spoke backup.
|
||||||
| felhom-controller | READ-ONLY (sync.go + healthprobe.go consulted to state consumer behavior accurately) |
|
|
||||||
|
|
||||||
## The skill
|
## Probe ladder results (both ends real: demo-felhom PVE host ⟷ throwaway Hetzner CX23)
|
||||||
|
|
||||||
- **Workflow, not reference:** research → inspect-the-image healthcheck choice → compose →
|
- **P0** — the operator's line is **plain single-NAT with a fixed public IPv4**
|
||||||
`.felhom.yml` → live UI deploy on demo → verify healthy → reconcile count → finish. Every field
|
(`37.191.56.193`, catv.fixed.one.hu, AS21334), *not* CGNAT (no 100.64/10 hop; source IP at the
|
||||||
table is a POINTER to app-catalog `REUSE.md` §1–2 / `README.md` §format — the shipped SKILL.md
|
server = egress IP). **Zero IPv6** (no address, no egress, 15-min RA capture: 0 packets).
|
||||||
contains no field tables at all (checked); the one new fact it teaches (probe-container naming)
|
- **P1 (H0 v6-direct)** — N/A on this line; keep the production endpoint dual-stack anyway.
|
||||||
was placed in REUSE.md and is referenced from the skill.
|
- **P2 (H1)** — plain WG v4 up on the **first try**; RTT ~30–40 ms; **11.4-min fully-idle hold**
|
||||||
- **Installer:** no change needed — `install_skills.py` auto-discovers `skills/*/SKILL.md` dirs
|
(keepalive-only, NAT mapping unmoved, instant ping after); PBS 4.2.2 API through the tunnel with
|
||||||
(line 69). Ran it: `felhom-app-catalog junction -> E:\git\felhom.eu\skills\felhom-app-catalog`.
|
the **exact-cert pin intact** (`pin.go` model) + negative wrong-pin proof; `:8007` tunnel-only
|
||||||
- **Fresh-session discovery:** `claude -p "List your custom skills…"` → all 4 listed
|
by firewall (public probe: unreachable).
|
||||||
(`felhom-app-catalog`, `felhom-build-deploy`, `felhom-testing`, `felhom-ui-design`).
|
- **P3** — uplink is the bottleneck: direct 42 Mbit/s up / 200–230 down; tunnel upload ~39 Mbit/s
|
||||||
|
(~5% overhead); PBS TLS benchmark through tunnel **4.94 MB/s = the full uplink**; **real 2 GiB
|
||||||
|
worst-case (urandom) backup: 4.26 MiB/s sustained, 8 min, zero stalls**. First backup 50–100 GB
|
||||||
|
≈ 3–6 h (overnight window). Oddity recorded: tunnel *download* (565 Mbit/s, 0 retrans) beat
|
||||||
|
direct TCP (230, 285 retrans) — UDP encapsulation dodges whatever shapes inbound TCP.
|
||||||
|
- **P4 (H2)** — outbound UDP fine on **51820 and 443**; no TCP/443 fallback machinery needed.
|
||||||
|
- **P6** — plain-WG isolation is topological: per-peer `/32` AllowedIPs + forwarding off at the
|
||||||
|
endpoint; only the PBS port admitted from the WG interface.
|
||||||
|
- **P7** — hypothesis **refuted with evidence**: kernel WG link creation **succeeds** inside the
|
||||||
|
unprivileged guest (no `/dev/net/tun` needed — that's confirmed absent and only blocks
|
||||||
|
tailscaled). Host placement therefore stands on architecture: backups originate host-side, the
|
||||||
|
tunnel must survive guest death (cloudflared precedent), the guest keeps zero infra creds.
|
||||||
|
|
||||||
## Two verified facts the exercise added to the knowledge base
|
## Architecture recommendation (input to the next task — the production spec)
|
||||||
|
|
||||||
1. **Probe-container naming (REUSE.md row, new):** the controller-side `healthcheck.checks[]`
|
Host-side agent-managed WG service (cloudflared pattern); per-box keypair minted at install →
|
||||||
probe dials the container named EXACTLY like the stack dir; fallback = first running
|
pubkey to hub, private key 0600 in agent state dir + **joins the escrowed IdentityBundle** for DR;
|
||||||
prefix-match — in a multi-container stack that can be the DB. Convention: exposed service's
|
endpoint = small public dual-stack VM (€4–8/mo class), two shapes deferred to the spec: PBS-on-VM
|
||||||
`container_name` = stack name (vaultwarden/rallly already comply; sparkyfitness now does).
|
vs **VM-as-rendezvous-relay with DooPlex PBS dialing out too** (both ends outbound-only,
|
||||||
2. **BusyBox `ash` `command -v` takes ONE argument** — `command -v wget curl node python3`
|
ciphertext-only transit — also solves DooPlex's no-public-UDP). Trust model preserved: box only
|
||||||
reported "wget only" in both SparkyFitness images while per-tool runs showed curl (and node in
|
dials out; operator-inbound (B) and game-ingress (C) explicitly out of scope.
|
||||||
the server) present. The skill's inspect command is a per-tool loop for this reason.
|
|
||||||
|
|
||||||
## SparkyFitness — markers resolved by inspection (no more VERIFY-BEFORE-FINALIZE)
|
## Honest caveats
|
||||||
|
|
||||||
| Image (both tags RESOLVE) | wget | curl | node | python3 | Family chosen |
|
CGNAT traversal is argued from mechanism, **not proven on this line** (it's plain-NAT); second-ISP
|
||||||
|---|---|---|---|---|---|
|
confirmation deferred to Peti's VM 110 (currently unreachable). Single ISP/evening; no multi-day
|
||||||
| `codewithcj/sparkyfitness:v0.17.2` (nginx SPA) | /usr/bin/wget | /usr/bin/curl | no | no | BusyBox-wget `--spider :80/` (kept — most common family) |
|
hold; no home-IP-rotation event observed; Headscale assessed server-side only (no tailscaled on
|
||||||
| `codewithcj/sparkyfitness_server:v0.17.2` (Node) | /usr/bin/wget | /usr/bin/curl | /usr/local/bin/node v24.17.0 | no | node-exec HTTP GET `:3010/api/health` status<400 (real status check, kept) |
|
the production host); no restore-direction PBS run.
|
||||||
|
|
||||||
`/api/health` existence proven live: `{"status":"UP"}`. `mem_limit` arithmetic verified:
|
## Cleanup (verified)
|
||||||
512 (db) + 1024 (server) + 256 (frontend) = **1792M** = the `.felhom.yml` value (comment added).
|
|
||||||
|
|
||||||
## Live validation (method stated per claim)
|
demo-felhom: interface/keys/config/dataset gone (keys **shredded**), iperf3 purged, WG module
|
||||||
|
unloaded, `/etc/wireguard` untouched-empty, guest probe link deleted — residue-free. Hetzner
|
||||||
|
throwaway: WG + headscale + PBS torn down, secrets shredded, datastore deleted, box **powered
|
||||||
|
off** (parked for the Peti follow-up or destruction; jarrs.eu box never touched).
|
||||||
|
|
||||||
SparkyFitness was ALREADY deployed on demo from the draft (4 h prior, three containers healthy) —
|
## Repo changes (docs-only)
|
||||||
so the live action was applying the FINALIZED template: "Sablonok frissítése" then the card's
|
|
||||||
"Frissítés", both clicked in the real dashboard via the browser bridge (claude-in-chrome).
|
|
||||||
Compose recreated only the renamed frontend (`sparkyfitness`, healthy in 21 s); server + db
|
|
||||||
untouched (Up 4 h) → the two `data_key` secrets were NOT regenerated, as required. Verifications:
|
|
||||||
git-sync file content, `docker ps`, `/api/health`, traefik Host-header curl (200), and
|
|
||||||
`GET /api/stacks/sparkyfitness` (the endpoint the UI polls) — controller probe verdict
|
|
||||||
`healthy: true, http :80/ → 200 (6ms)` against the renamed container. Kept deployed (7.3 option A;
|
|
||||||
expendable demo box, real catalog app).
|
|
||||||
|
|
||||||
## App-count reconciliation (recorded, NOT fixed)
|
- NEW `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (P0–P7 verbatim + twelve
|
||||||
|
§7 answers + recommendation).
|
||||||
Real catalog: **53** templates. Website `gyik.html` claims "több mint 45" (true but lagging);
|
- `documentation/runbooks/day0-install.md` — the "LAN-only until Headscale/WireGuard lands"
|
||||||
`alkalmazasok.html` per-category `app-count` spans sum **49** ≠ 53. Website fixes are
|
backlog line now points at the spike (decision made; spec is the remaining gate).
|
||||||
`felhom-ui-design` + `site_gates.py` territory — separate task.
|
- `CONTEXT.md` — transport decision recorded; next task = production connectivity spec.
|
||||||
|
- `scripts/CHANGELOG.md` — docs entry.
|
||||||
## Observations (not acted on)
|
|
||||||
|
|
||||||
- The controller UI card and `docker ps` agreed throughout; the probe's DEBUG-level per-stack log
|
|
||||||
lines are invisible at INFO — the `GET /api/stacks/<name>` endpoint is the reliable verdict
|
|
||||||
source for validation.
|
|
||||||
- `paperless-ngx`'s containers (`paperless-webserver`…) match NEITHER the exact-name nor the
|
|
||||||
prefix fallback of its stack name `paperless-ngx` — its controller-side probe likely never runs
|
|
||||||
(silently "skipped no container"). Candidate follow-up: rename or add a probe-container hint;
|
|
||||||
NOT touched here.
|
|
||||||
- The pre-rename fallback DID probe the right container on demo (probe was green before the
|
|
||||||
update) — the rename converts order-dependent luck into the deterministic convention.
|
|
||||||
|
|||||||
@@ -0,0 +1,385 @@
|
|||||||
|
# SPIKE — offsite-backup connectivity: WireGuard vs the home-NAT reality (2026-07-03)
|
||||||
|
|
||||||
|
> **STATUS: COMPLETE — verdict below.** This spike validates the offsite-backup *transport* before
|
||||||
|
> any production connectivity spec is written. No production code shipped; no Headscale on the real
|
||||||
|
> hub; no agent binary change; no persistent config on felhom-pve. All probes ran on a throwaway
|
||||||
|
> Hetzner box (`felhom-hetzner`, destroyed-or-parked after — see §6) and as additive, fully-removed
|
||||||
|
> interfaces/services on the demo-felhom host. The real DooPlex PBS and its customer snapshots were
|
||||||
|
> **never** a probe target.
|
||||||
|
|
||||||
|
**Class:** SPIKE (empirical validation; no product code). **Repos:** felhom.eu (this doc only);
|
||||||
|
felhom-agent read-only for grounding (`internal/pbs/{client,pin}.go`, `configs/build-golden.sh`,
|
||||||
|
`internal/hub/cloudflared.go`, `internal/escrow/identity.go`).
|
||||||
|
**Probe ends:** `felhom-hetzner` = Hetzner CX23, Debian 13.4, public IPv4 `167.233.158.164`,
|
||||||
|
global IPv6 `2a01:4f8:...::/64` (throwaway — NOT the live jarrs.eu box) ⟷ **demo-felhom** =
|
||||||
|
the real PVE 9.2.2 host on the operator's home line (One Hungary fixed cable, Budapest), driven
|
||||||
|
over the existing LAN SSH path; the tunnel itself always dialed **out**.
|
||||||
|
|
||||||
|
**Verdict (one line):** **GO at the lowest rung — H1: plain WireGuard, box-dials-out to a small
|
||||||
|
public endpoint, holds through the home NAT with zero coordinator/relay, carries the PBS API with
|
||||||
|
the TLS pin intact at ~wire-speed of the home uplink; Headscale is a separable fleet-management
|
||||||
|
option, not a traversal necessity — with the honest caveat that the operator's line turned out to
|
||||||
|
be plain-NAT (public IP), not CGNAT, so the CGNAT-traversal claim rests on mechanism + a deferred
|
||||||
|
second-ISP confirmation (Peti's VM 110), not on this line's evidence.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Why this spike exists (the gap, grounded)
|
||||||
|
|
||||||
|
Offsite backup — the paid DR substrate (`architecture/01-topology-and-trust.md` §8) — is
|
||||||
|
structurally unreachable for a real customer today:
|
||||||
|
|
||||||
|
- Backup is outbound push to PBS (§7/§8), and the only PBS lives at a **LAN address**,
|
||||||
|
`192.168.0.180:8007` (`felhom-agent/internal/pbs/client.go`).
|
||||||
|
- The public edge is Cloudflare-Tunnel → `nginx-internal` — HTTP(S) only, **no public inbound UDP
|
||||||
|
anywhere**; DooPlex has no public IP.
|
||||||
|
- So offsite backup works only because the demo box shares a LAN with PBS. A real customer's box
|
||||||
|
cannot reach `192.168.0.180` at all. The DR crypto/identity side is done and drilled
|
||||||
|
(`SPIKE-dr-recipe-2026-06-16.md`); **transport is the only missing piece**.
|
||||||
|
|
||||||
|
**Architectural corollary (recorded as a finding):** production offsite backup requires a
|
||||||
|
**publicly-reachable endpoint that today's infrastructure does not provide**. The throwaway Hetzner
|
||||||
|
box in this spike is a preview of that production endpoint, not just a test rig (§4.4).
|
||||||
|
|
||||||
|
The hypothesis ladder tested, cheapest-first (the roadmap's "Headscale + DERP" sketch assumes the
|
||||||
|
hard problem — *peer-to-peer* traversal between two NAT'd hosts — but backup is **hub-and-spoke to
|
||||||
|
a public endpoint**, which any NAT'd box can reach outbound):
|
||||||
|
|
||||||
|
| Rung | Hypothesis | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| H0 | both ends global IPv6 → direct, no NAT | **N/A — the home side has no IPv6 at all** (P0/P1) |
|
||||||
|
| H1 | plain WG over IPv4, box→public endpoint, keepalive | **CONFIRMED** on this line (P2/P3) |
|
||||||
|
| H2 | outbound UDP blocked → TCP/443 fallback needed | **not needed** — UDP 51820 *and* 443 both pass (P4) |
|
||||||
|
| H3 | fleet needs → Headscale | **separable** — fleet layer, not traversal (P5) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Probe results (verbatim, redacted)
|
||||||
|
|
||||||
|
### P0 — NAT classification of the demo-felhom line — **plain single-NAT with a public IPv4, no IPv6**
|
||||||
|
|
||||||
|
- Egress IP seen by `ifconfig.me` from the host: `37.191.56.193`.
|
||||||
|
- Source IP observed **at the Hetzner box** for an inbound TCP connection from demo-felhom
|
||||||
|
(HTTP listener log): `37.191.56.193` — same address, i.e. one NAT, stable mapping.
|
||||||
|
- `ipinfo.io`: `37.191.56.193` = `catv-37-191-56-193.catv.fixed.one.hu`, AS21334 One Hungary Ltd.
|
||||||
|
— a *fixed* (subscriber-routable) cable address, not shared-pool space.
|
||||||
|
- Traceroute from the host (first hops):
|
||||||
|
|
||||||
|
```
|
||||||
|
1 192.168.0.1 0.239 ms ← home router
|
||||||
|
2 89.133.164.70 12.485 ms ← already PUBLIC ISP space (catv...one.hu, same AS)
|
||||||
|
3 89.135.220.78 22.316 ms
|
||||||
|
```
|
||||||
|
|
||||||
|
**No `100.64.0.0/10` (or private) hop after the home router** — no CGNAT tier on the path.
|
||||||
|
- IPv6: `ip -6 addr` shows link-local only; `curl -6` fails (`NO-IPV6-EGRESS`); a **15-minute
|
||||||
|
passive `tcpdump` capture for router advertisements on `vmbr0` caught zero RAs**. (Caveat noted:
|
||||||
|
`accept_ra=1` + `forwarding=1` means the kernel would ignore RAs anyway, which is why the
|
||||||
|
wire-level capture, not the address absence, is the evidence.) No IPv6 exists on this LAN.
|
||||||
|
|
||||||
|
**Consequences:** the operator's line is a *softer* premise than the task assumed — H1 success here
|
||||||
|
does **not** prove CGNAT traversal (see §5). H0 (IPv6-direct) is dead on arrival for this line and
|
||||||
|
cannot be the plan's foundation: production must assume v4-only homes.
|
||||||
|
|
||||||
|
### P1 — H0 IPv6-direct — **N/A** (recorded gap)
|
||||||
|
|
||||||
|
Not testable: the home side has no global IPv6 (P0). The Hetzner side has native v6, so the
|
||||||
|
server-side of a dual-stack endpoint is free — worth keeping the endpoint dual-stack so v6-capable
|
||||||
|
customer lines use it, but it solves nothing this spike needed solving.
|
||||||
|
|
||||||
|
### P2 — H1: plain WG over IPv4, outbound-initiated — **CONFIRMED, first try, no coordinator/relay**
|
||||||
|
|
||||||
|
Setup: WG keypair per side (private keys generated on-box, 0600, never transited or logged);
|
||||||
|
Hetzner `wgspike` = `10.99.0.1/24`, `ListenPort 51820`; demo-felhom `wgspike` = `10.99.0.2/24`,
|
||||||
|
`Endpoint 167.233.158.164:51820`, `PersistentKeepalive = 25`, config under `/tmp` (nothing in
|
||||||
|
`/etc/wireguard`). PVE 9.2 needed nothing beyond `modprobe wireguard` (module ships in the kernel)
|
||||||
|
+ the `wireguard-tools` package.
|
||||||
|
|
||||||
|
```
|
||||||
|
demo-felhom# wg-quick up /tmp/wgspike.conf && ping -c 3 10.99.0.1
|
||||||
|
64 bytes from 10.99.0.1: icmp_seq=1 ttl=64 time=51.2 ms
|
||||||
|
3 packets transmitted, 3 received, 0% packet loss
|
||||||
|
rtt min/avg/max/mdev = 33.411/40.659/51.204/7.628 ms
|
||||||
|
```
|
||||||
|
|
||||||
|
Server-side view: peer endpoint `37.191.56.193:32873` (the NAT-translated source port), i.e. a
|
||||||
|
classic outbound UDP mapping — exactly the mechanism that also works through CGNAT.
|
||||||
|
|
||||||
|
**Keepalive hold (idle test):** the link carried **zero data traffic for 11.4 minutes**
|
||||||
|
(keepalives only — total transfer counters after the window: 1 888 B rx / 3 744 B tx). Result:
|
||||||
|
handshake stayed fresh throughout (age 120 s at check — keepalive-driven rekeys continuing on
|
||||||
|
schedule), the NAT mapping never moved (`37.191.56.193:32873` unchanged), and the first ping after
|
||||||
|
the idle window went straight through:
|
||||||
|
|
||||||
|
```
|
||||||
|
3 packets transmitted, 3 received, 0% packet loss
|
||||||
|
rtt min/avg/max/mdev = 30.119/30.812/31.279/0.499 ms
|
||||||
|
```
|
||||||
|
|
||||||
|
The tunnel **holds through idle with no help** — the single mechanism CGNAT UDP timeouts would
|
||||||
|
threaten, working as designed.
|
||||||
|
|
||||||
|
### P2b — PBS `:8007` through the tunnel, TLS pin intact — **CONFIRMED (positive + negative)**
|
||||||
|
|
||||||
|
A real PBS 4.2.2 was installed on the throwaway box (datastore `scratch` at `/srv/pbs-scratch`,
|
||||||
|
API token `root@pam!spike`, secret 36 chars — never logged; box firewalled to SSH + WG/iperf ports
|
||||||
|
only, so `:8007` is **not publicly reachable — tunnel-only by construction**). From demo-felhom,
|
||||||
|
with `PBS_FINGERPRINT` = the scratch PBS leaf-cert SHA-256 (the exact pin model of
|
||||||
|
`felhom-agent/internal/pbs/pin.go`):
|
||||||
|
|
||||||
|
```
|
||||||
|
=== public :8007 reachability (must FAIL, firewall proof):
|
||||||
|
000 / UNREACHABLE (as intended)
|
||||||
|
=== tunnel :8007 API version (pin intact):
|
||||||
|
client version: 4.2.0
|
||||||
|
server version: 4.2.2
|
||||||
|
=== NEGATIVE pin test (wrong fingerprint, must FAIL):
|
||||||
|
WARNING: certificate fingerprint does not match expected fingerprint!
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact-cert pin validates **through** the tunnel (WG is below TLS; the pinned leaf is
|
||||||
|
unchanged), and a wrong pin is detected — the red-proof for the pin surviving the transport
|
||||||
|
change. (Deploy gotcha found + fixed on the scratch box: a datastore under `/root` fails with
|
||||||
|
`Permission denied` — the PBS `backup` user can't traverse `/root`; use a world-traversable
|
||||||
|
parent like `/srv`.)
|
||||||
|
|
||||||
|
### P3 — throughput — home uplink is the bottleneck; tunnel overhead ≈ 5% on the backup direction
|
||||||
|
|
||||||
|
Direct (no tunnel) `iperf3` demo-felhom ⟷ Hetzner (two runs each):
|
||||||
|
|
||||||
|
```
|
||||||
|
UPLOAD (home→Hetzner): 42.1 / 42.0 Mbit/s sender (receiver 40.7 both runs)
|
||||||
|
DOWNLOAD (Hetzner→home): 203 / 230 Mbit/s (retransmits: 3 / 285)
|
||||||
|
```
|
||||||
|
|
||||||
|
Through the WG tunnel:
|
||||||
|
|
||||||
|
```
|
||||||
|
UPLOAD (tunnel): 39.1 Mbit/s sender / 38.4 receiver ← ~5% under direct
|
||||||
|
DOWNLOAD (tunnel): 565 Mbit/s, 0 retransmits ← FASTER than direct TCP
|
||||||
|
```
|
||||||
|
|
||||||
|
The download anomaly is real and repeatable-in-session: plain TCP from Hetzner→home tops out at
|
||||||
|
~200–230 Mbit/s with hundreds of retransmits, while the same TCP stream **inside** the WG/UDP
|
||||||
|
encapsulation ran at 565 Mbit/s with zero retransmits — whatever middlebox/shaping hurts inbound
|
||||||
|
TCP on this cable line does not see the UDP tunnel. Download is the restore direction; a pleasant
|
||||||
|
surprise, not a design input. (Single-session observation; not load-tested further.)
|
||||||
|
|
||||||
|
**PBS-native measurements over the tunnel** (the actual workload):
|
||||||
|
|
||||||
|
```
|
||||||
|
proxmox-backup-client benchmark: TLS (maximal backup upload speed): 4.94 MB/s
|
||||||
|
AES256-GCM 3 904 MB/s, SHA256 2 152 MB/s (CPU is never the limit)
|
||||||
|
|
||||||
|
REAL backup, 2 GiB of /dev/urandom (worst case — incompressible, non-dedupable):
|
||||||
|
spike.pxar: had to backup 2 GiB of 2 GiB in 480.75 s (average 4.26 MiB/s)
|
||||||
|
Duration: 481.26s — sustained, zero stalls, saturating the uplink for 8 minutes straight
|
||||||
|
```
|
||||||
|
|
||||||
|
4.94 MB/s TLS-in-tunnel ≈ 41 Mbit/s = the full measured uplink: **the tunnel + TLS stack costs
|
||||||
|
effectively nothing against the line's own ceiling.**
|
||||||
|
|
||||||
|
**First-backup window math (at the measured ~40 Mbit/s ≈ 5 MB/s effective uplink):** 10 GB ≈ 35 min,
|
||||||
|
50 GB ≈ 2.8 h, 100 GB ≈ 5.6 h — a realistic first backup fits an overnight window; incrementals
|
||||||
|
(PBS dedup) are far smaller. The *download* direction (restore) is ~5× faster on this line.
|
||||||
|
|
||||||
|
### P4 — H2: outbound UDP reality — **51820 AND 443/UDP both pass**
|
||||||
|
|
||||||
|
UDP 51820 egress is proven by P2 itself. The same tunnel re-pointed at **`:443/UDP`**
|
||||||
|
(`wg set … listen-port 443` server-side, endpoint updated client-side) handshook immediately and
|
||||||
|
carried pings at the same RTT:
|
||||||
|
|
||||||
|
```
|
||||||
|
3 packets transmitted, 3 received, 0% packet loss
|
||||||
|
rtt min/avg/max/mdev = 30.300/32.612/35.044/1.938 ms
|
||||||
|
endpoint: 167.233.158.164:443
|
||||||
|
```
|
||||||
|
|
||||||
|
No TCP/443 fallback (DERP-over-HTTPS etc.) is needed on this ISP; and 443/UDP working means even a
|
||||||
|
hypothetical 51820-blocking ISP has a first, zero-cost escalation (same protocol, friendlier port)
|
||||||
|
before any TCP fallback machinery is justified.
|
||||||
|
|
||||||
|
### P5 — H3: what Headscale actually buys — **a fleet layer, not traversal**
|
||||||
|
|
||||||
|
Headscale 0.29.2 stood up on the throwaway box (systemd service + `/etc/headscale/config.yaml` +
|
||||||
|
sqlite at `/var/lib/headscale/db.sqlite`): created users `customer-a`/`customer-b`, minted a
|
||||||
|
preauth key, inspected the config surface. Observations:
|
||||||
|
|
||||||
|
- **Operational surface:** a public control-plane service (TLS **required** — the embedded DERP
|
||||||
|
server refuses a non-HTTPS `server_url`), a database to back up, user + preauthkey + node
|
||||||
|
lifecycle, an ACL policy file, plus a **client-side daemon (tailscaled) on every box**. The
|
||||||
|
default DERP config points at **Tailscale's public derpmap**
|
||||||
|
(`controlplane.tailscale.com/derpmap/default`) — a third-party dependency unless you run your
|
||||||
|
own DERP.
|
||||||
|
- **For hub-and-spoke backup it is redundant:** traversal is already solved by rung H1 (the spoke
|
||||||
|
dials a public endpoint). Headscale's real value is fleet mechanics — key rotation, node expiry,
|
||||||
|
per-customer users/ACLs, automatic endpoint re-resolution, and DERP *fallback* for UDP-hostile
|
||||||
|
lines (H2 — which this ISP does not need).
|
||||||
|
- **Recorded gap:** no client was joined to the throwaway headscale (joining = installing
|
||||||
|
tailscaled on the production PVE host; declined). Surface inspection + docs only.
|
||||||
|
|
||||||
|
### P6 — per-customer isolation
|
||||||
|
|
||||||
|
- **Plain WG (recommended rung):** isolation is inherent in hub-and-spoke: the server holds one
|
||||||
|
peer entry per box with `AllowedIPs = <that box's /32>` — cross-peer traffic requires the
|
||||||
|
*server* to forward between WG peers, which the endpoint's firewall simply does not allow
|
||||||
|
(forwarding stays off; only `INPUT` to the PBS port is open on the WG interface). A box can
|
||||||
|
reach the PBS API and nothing else; boxes cannot see each other. PBS-side tenancy = the existing
|
||||||
|
per-customer namespace + token model (unchanged).
|
||||||
|
- **Headscale (if ever adopted):** the same property must be *written* as ACL policy
|
||||||
|
(per-customer users + a policy file allowing only spoke→PBS) — more expressive, but a policy to
|
||||||
|
maintain and test rather than a topology that cannot express the attack.
|
||||||
|
|
||||||
|
### P7 — guest-side WG refutation — **hypothesis REFUTED: kernel WG works in the unprivileged guest; host placement stands on architecture, not infeasibility**
|
||||||
|
|
||||||
|
Against the real guest 9201 (`features: nesting=1,keyctl=1`, `unprivileged: 1`):
|
||||||
|
|
||||||
|
```
|
||||||
|
# pct exec 9201 -- ls -la /dev/net/
|
||||||
|
ls: cannot access '/dev/net/': No such file or directory ← no TUN, as expected
|
||||||
|
# pct exec 9201 -- ip link add wgtest type wireguard
|
||||||
|
rc=0 ← SUCCEEDS
|
||||||
|
wgtest DOWN <POINTOPOINT,NOARP>
|
||||||
|
# pct exec 9201 -- ip link del wgtest ← removed immediately
|
||||||
|
```
|
||||||
|
|
||||||
|
Kernel WireGuard is **not** a TUN consumer — an unprivileged LXC with the stock golden features can
|
||||||
|
create a WG interface as long as the host kernel has the module. So the "no TUN → guest can't"
|
||||||
|
argument is dead. The **host-placement recommendation stands on architecture instead** (§4.2):
|
||||||
|
backup traffic *originates on the host* (the agent drives vzdump/PBS push with host-side
|
||||||
|
credentials), the tunnel must survive guest death/rebuild by construction (the `cloudflared`
|
||||||
|
precedent, 01 §7), and the guest must keep holding **zero** infra credentials (01 §3). A guest-side
|
||||||
|
WG key would violate the trust model, not the kernel. (`/dev/net/tun` absence *does* still block a
|
||||||
|
guest-side **tailscaled**, relevant only if the Headscale rung were ever chosen.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. The twelve §7 answers
|
||||||
|
|
||||||
|
1. **CGNAT or plain-NAT?** Plain single-NAT with a fixed public IPv4 (`37.191.56.193`, One Hungary
|
||||||
|
catv-fixed); **no IPv6 at all** (no address, no egress, no RAs on the wire). The CGNAT premise
|
||||||
|
did not hold *on this line*.
|
||||||
|
2. **H0 (v6-direct)?** N/A — no home IPv6. Keep the production endpoint dual-stack, but v4 is the
|
||||||
|
planning baseline.
|
||||||
|
3. **H1 (plain WG through the NAT)?** Yes — outbound-initiated, `PersistentKeepalive=25`, zero
|
||||||
|
coordinator/relay, held through an 11.4-minute fully-idle window with a stable NAT mapping,
|
||||||
|
and carried the PBS API with the exact-cert pin validating through the tunnel (positive +
|
||||||
|
negative pin proof, P2b).
|
||||||
|
4. **Throughput?** Tunnel costs ~5% on the ~40 Mbit/s uplink bottleneck; PBS-native TLS speed
|
||||||
|
through the tunnel = the full uplink (4.94 MB/s); a real worst-case 2 GiB backup sustained
|
||||||
|
4.26 MiB/s for 8 min with zero stalls. First backup of 50–100 GB ≈ 3–6 h (overnight window);
|
||||||
|
incrementals trivial.
|
||||||
|
5. **H2 (UDP blocked)?** No — UDP 51820 **and** 443 both pass on this ISP. TCP/443 fallback is a
|
||||||
|
*contingency for other ISPs*, not a launch requirement; note it as the escalation if a customer
|
||||||
|
line ever proves UDP-hostile.
|
||||||
|
6. **H3 (Headscale needed)?** Not for backup. It is a separable fleet-management layer (key
|
||||||
|
lifecycle, ACLs, endpoint re-resolution, DERP fallback) purchasable later without re-architecting
|
||||||
|
— the spoke config is the same WG primitive either way. Its cost: a public TLS control plane,
|
||||||
|
a DB, tailscaled on every box, and (by default) a Tailscale-operated DERP dependency.
|
||||||
|
7. **Isolation?** Plain WG hub-and-spoke: per-peer `/32` `AllowedIPs` + no forwarding on the
|
||||||
|
endpoint = no lateral path by topology (P6).
|
||||||
|
8. **Placement?** Host — same placement as `cloudflared` (agent-managed systemd service). The P7
|
||||||
|
probe *refuted* the infeasibility argument (kernel WG works in the guest), so the decision rests
|
||||||
|
on: backups originate host-side, tunnel survives guest death, guest keeps zero infra creds.
|
||||||
|
9. **Key custody + DR:** mint the box's WG keypair at install into the agent state dir (0600,
|
||||||
|
`felhom-agent` user), register the pubkey with the hub, and add the private key to the
|
||||||
|
escrowed `IdentityBundle` (`felhom-agent/internal/escrow/identity.go`) so a re-provisioned box
|
||||||
|
re-establishes its tunnel from the same recovery flow that already restores PBS identity. The
|
||||||
|
endpoint's own pubkey+address are non-secret config in the hub record.
|
||||||
|
10. **Where does the production endpoint live?** A small public relay VM (this spike's CX23-class
|
||||||
|
box ≈ €4–8/mo) — see §4.4 for the two variants (PBS-on-VM vs VM-as-rendezvous-relay with
|
||||||
|
DooPlex PBS dialing out too). DooPlex itself cannot be the endpoint (no public IP/UDP).
|
||||||
|
11. **Trust model preserved?** Yes — the box only ever dials **out** (UDP to the endpoint); no new
|
||||||
|
inbound listener, no new inbound attack surface on the box; the endpoint exposes exactly one
|
||||||
|
UDP port publicly. Operator-inbound access (B) and game-server ingress (C — public players are
|
||||||
|
not tunnel members) are **explicitly separate problems, out of scope here**, and nothing in
|
||||||
|
this design presumes them.
|
||||||
|
12. **Go/no-go:** **GO** on rung **H1** — plain WireGuard, host-side, box-dials-out, to a small
|
||||||
|
public endpoint carrying PBS. Headscale/DERP only if (a) a real customer ISP proves
|
||||||
|
UDP-hostile (H2) or (b) fleet size makes static peer config painful (H3). No pivot to an
|
||||||
|
outbound-mTLS PBS proxy needed — the simplest thing worked.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Architecture recommendation (input to the production spec — NOT the spec)
|
||||||
|
|
||||||
|
### 4.1 Transport rung
|
||||||
|
Plain kernel WireGuard, one tunnel per box, box-initiated, `PersistentKeepalive=25`, endpoint =
|
||||||
|
a public relay VM. No coordinator, no relay, no mesh.
|
||||||
|
|
||||||
|
### 4.2 Placement
|
||||||
|
On the Proxmox **host**, as an **agent-managed systemd service** — the exact `cloudflared`
|
||||||
|
precedent (01 §7, `internal/hub/cloudflared.go` prober pattern). Interface + config under the
|
||||||
|
agent's ownership; guest untouched.
|
||||||
|
|
||||||
|
### 4.3 Identity & custody
|
||||||
|
Per-box keypair minted at install; private key 0600 in the agent state dir; pubkey → hub record;
|
||||||
|
private key → escrowed `IdentityBundle`. Server pubkey/endpoint = plain config from the hub.
|
||||||
|
Revocation = delete the peer entry at the endpoint (hub-driven, auditable).
|
||||||
|
|
||||||
|
### 4.4 The production public endpoint (the §1 corollary)
|
||||||
|
Two viable shapes, decision deferred to the production spec:
|
||||||
|
- **(a) PBS on the public VM** — simplest ops; storage cost scales on the VM; client-side
|
||||||
|
encryption means at-rest exposure is ciphertext-only anyway.
|
||||||
|
- **(b) VM as dumb rendezvous relay** — DooPlex PBS *also* dials out a WG spoke to the VM; the VM
|
||||||
|
forwards spoke↔PBS traffic and stores nothing. Keeps bulk storage on DooPlex's existing disks;
|
||||||
|
the VM needs only bandwidth; **both** sides remain outbound-only (this also solves DooPlex's own
|
||||||
|
no-public-UDP problem). Ciphertext-only transits the VM.
|
||||||
|
Either way the endpoint is small, dual-stack, firewalled to WG-UDP-only, and *is* the thing the
|
||||||
|
subscription's offsite promise runs through — it becomes managed infrastructure, not a lab box.
|
||||||
|
|
||||||
|
### 4.5 Isolation
|
||||||
|
Per-peer `/32` AllowedIPs; endpoint forwarding off; WG-interface firewall admits only the PBS port.
|
||||||
|
Per-customer PBS namespaces/tokens unchanged on top.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Scope caveats + NOT-validated list
|
||||||
|
|
||||||
|
- **The CGNAT claim is NOT proven by this spike** — the operator's line turned out to be
|
||||||
|
plain-NAT. What *is* proven: outbound-initiated WG through a NAT with keepalive, which is the
|
||||||
|
same mechanism CGNAT permits (CGNAT breaks *inbound* and *p2p*, not outbound-to-public). The
|
||||||
|
honest residual risk is an ISP that breaks long-lived UDP mappings aggressively — exactly what
|
||||||
|
the deferred second-ISP vantage (**Peti's Debian VM 110**, currently unreachable — "No route to
|
||||||
|
host") should confirm once fixed. **Follow-up recorded.**
|
||||||
|
- Single ISP (One Hungary catv), single evening, single geography. No long-horizon (days) hold
|
||||||
|
data; no home-IP-rotation event observed (WG roaming handles it by design — untested here).
|
||||||
|
- No Headscale client join (tailscaled kept off the production host); Headscale assessed on
|
||||||
|
server-side surface + docs only.
|
||||||
|
- No real vzdump→PBS run over the tunnel (synthetic `proxmox-backup-client` traffic only — real
|
||||||
|
guest snapshots stay on the real PBS).
|
||||||
|
- IPv6 absence is a One-Hungary-line fact, not a market fact; the endpoint should stay dual-stack.
|
||||||
|
- The tunnel-download anomaly (UDP encapsulation outrunning direct TCP) is a single-session
|
||||||
|
observation — noted, not relied on.
|
||||||
|
- Restore-direction (PBS → box) bulk transfer was measured only as iperf3, not as a
|
||||||
|
`proxmox-backup-client restore` run.
|
||||||
|
- Relayed-mode (DERP) throughput was NOT measured — moot, since no relay is recommended.
|
||||||
|
|
||||||
|
## 6. Cleanup assertion
|
||||||
|
|
||||||
|
**demo-felhom host (all verified after removal):** `wgspike` interface deleted (`wg-quick down`,
|
||||||
|
"Device does not exist" confirmed); WG private key + PBS token secret **shredded**; `/tmp` spike
|
||||||
|
files (config, RA capture log, 2 GiB dataset) removed; the `iperf3` package I installed purged;
|
||||||
|
`wireguard` kernel module unloaded (`lsmod` count 0); `/etc/wireguard/` empty and untouched
|
||||||
|
(mtime Apr 2025); guest 9201's probe link `wgtest` deleted seconds after creation ("Device does
|
||||||
|
not exist" confirmed); **no persistent config, unit, or package change remains from this spike**
|
||||||
|
(`wireguard-tools` was already installed before the spike and was left as found).
|
||||||
|
|
||||||
|
**Throwaway Hetzner box (`felhom-hetzner`, 167.233.158.164):** WG interface down + config/private
|
||||||
|
key/PBS token **shredded**; headscale + PBS services stopped and disabled; headscale DB
|
||||||
|
(`/var/lib/headscale`) and the scratch datastore (`/srv/pbs-scratch`, random-data chunks only)
|
||||||
|
deleted; then the box was **powered off** (`shutdown -h`). Disposition: **kept powered-off** for
|
||||||
|
the deferred second-ISP follow-up (Peti's VM 110) — the operator can start it from the Hetzner
|
||||||
|
console for that probe or destroy it outright; nothing secret survives on it either way. The live
|
||||||
|
jarrs.eu Hetzner server was never touched.
|
||||||
|
|
||||||
|
**No repo/hub/agent/manifest production change** — this commit is docs-only.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Backlog resolution
|
||||||
|
|
||||||
|
`runbooks/day0-install.md` "PBS host is LAN-only until the Headscale/WireGuard work lands" → now
|
||||||
|
points at this spike: the transport decision is **made** (plain WG to a public endpoint); what
|
||||||
|
remains is the production connectivity spec + implementation (endpoint VM, agent-managed WG
|
||||||
|
service, escrow join, hub peer registry).
|
||||||
@@ -118,10 +118,12 @@ Notes:
|
|||||||
- vmid picking: the agent's token is pool-scoped, so the hub's guest inventory only shows
|
- vmid picking: the agent's token is pool-scoped, so the hub's guest inventory only shows
|
||||||
Felhom-pool guests — other guests on a shared box are invisible there. Always pick from
|
Felhom-pool guests — other guests on a shared box are invisible there. Always pick from
|
||||||
`pct list` + `qm list` on the box (`documentation/runbooks/provisioning.md`).
|
`pct list` + `qm list` on the box (`documentation/runbooks/provisioning.md`).
|
||||||
- Offsite backup (`felhom-pbs`) is NOT part of a go-live today: the PBS host is LAN-only until the
|
- Offsite backup (`felhom-pbs`) is NOT part of a go-live today: the PBS host is LAN-only. The
|
||||||
Headscale/WireGuard work lands. Go-live standard = **local backups only** (that is what the
|
transport decision is now MADE — plain WireGuard, host-side, box-dials-out to a small public
|
||||||
Part C command configures); offsite DR is retrofitted later via `--rescope-acl` /
|
endpoint (spike-proven: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md`);
|
||||||
`--acl-storages` without reinstalling.
|
what remains is the production connectivity spec + endpoint VM. Go-live standard = **local
|
||||||
|
backups only** (that is what the Part C command configures); offsite DR is retrofitted later
|
||||||
|
via `--rescope-acl` / `--acl-storages` without reinstalling.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,20 @@
|
|||||||
# Felhom scripts — Changelog
|
# Felhom scripts — Changelog
|
||||||
|
|
||||||
|
## docs — SPIKE: offsite-backup connectivity — plain WireGuard WINS the ladder; Headscale = separable fleet layer (2026-07-03)
|
||||||
|
|
||||||
|
`documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` — the offsite-backup transport
|
||||||
|
decision, empirically grounded on both real ends (demo-felhom PVE host ⟷ throwaway Hetzner box,
|
||||||
|
end state: powered off, secrets shredded). Headline results: the operator's line is **plain-NAT
|
||||||
|
with a fixed public IP, not CGNAT, and has zero IPv6** (P0 honesty — CGNAT confirmation deferred
|
||||||
|
to Peti's VM 110); plain outbound WG held an 11.4-min idle window and carried a **real 2 GiB
|
||||||
|
worst-case PBS backup at 4.26 MiB/s = the full home uplink** (~5% tunnel overhead), TLS pin
|
||||||
|
intact through the tunnel (positive + negative proof); UDP 51820 **and** 443 both pass; kernel WG
|
||||||
|
surprisingly *works* inside the unprivileged guest (P7 — host placement stands on architecture,
|
||||||
|
not infeasibility). Recommendation: host-side agent-managed WG (cloudflared pattern), key in the
|
||||||
|
escrowed IdentityBundle, small public endpoint VM (PBS-on-VM vs rendezvous-relay deferred to the
|
||||||
|
spec). `runbooks/day0-install.md` backlog line resolved to point here; `CONTEXT.md` notes the
|
||||||
|
DR-completeness task is unblocked (next: the production connectivity spec).
|
||||||
|
|
||||||
## skills — NEW: felhom-app-catalog (4th skill) + SparkyFitness as its worked example (2026-07-03)
|
## skills — NEW: felhom-app-catalog (4th skill) + SparkyFitness as its worked example (2026-07-03)
|
||||||
|
|
||||||
`skills/felhom-app-catalog/SKILL.md` — the catalog **authoring workflow** (research → inspect the
|
`skills/felhom-app-catalog/SKILL.md` — the catalog **authoring workflow** (research → inspect the
|
||||||
|
|||||||
Reference in New Issue
Block a user