From 83c47eaf23d6f4a140b1d1f4c874b74094f77baa Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 19 Jul 2026 13:30:39 +0200 Subject: [PATCH] docs(ops): Tailscale makes the N100 (felhom-pve) location-independent felhom-pve joins the tailnet as a host package (100.70.170.35, expiry disabled); DooPlex already advertised 192.168.0.0/24 via its GitOps k3s tailscale pod (100.107.87.53). ssh felhom-pve now targets the tailnet IP (direct-over-LAN at home, tunnel when away); felhom-pve-lan = LAN fallback. --accept-dns=false on the host. Host converted static->DHCP (reservation keeps .162 at home). Measured: --accept-routes hairpins the local subnet at home -> travel-only opt-in. PBS is offsite (own WireGuard tunnel), needs none of this. Recorded not fixed: controller<->agent plane hard-pinned to 192.168.0.162 (agent listen_addr + guest bootstrap.json) -> not yet portable off-LAN. No code changed. Full doc: documentation/operations/tailscale.md Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01XMForrE4c1wZxd9LukxYVt --- CLAUDE.md | 6 +- CONTEXT.md | 13 ++ REPORT.md | 268 ++++++++++---------------- documentation/operations/tailscale.md | 104 ++++++++++ skills/felhom-build-deploy/SKILL.md | 2 +- 5 files changed, 222 insertions(+), 171 deletions(-) create mode 100644 documentation/operations/tailscale.md diff --git a/CLAUDE.md b/CLAUDE.md index 26f5bf3..fdda5ca 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -106,7 +106,11 @@ image build/push are local commands; felhom-pve is one SSH hop. | Host | Access | Role | |------|--------|------| | **DooPlex (this host)** | local — `/mnt/5_hdd/felhom.eu/{git,build}/` | Build + push images, `sudo kubectl` | -| Demo Proxmox host | `ssh felhom-pve` (root@192.168.0.162) | pveum/pct + live Proxmox validation | +| Demo Proxmox host (N100) | `ssh felhom-pve` — via Tailscale `100.70.170.35` (location-independent); `felhom-pve-lan` = LAN `192.168.0.162` fallback | pveum/pct + live Proxmox validation | + +The `felhom-pve` transport is Tailscale (the N100 is travel-portable) — topology, the accept-dns +rule, the accept-routes spike result, rollback, and the vacation-day checklist live in +`documentation/operations/tailscale.md`. > **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11 with repos in `E:\git\`, > and every remote command needed `SSH=/c/Windows/System32/OpenSSH/ssh.exe` (Git Bash's ssh fails diff --git a/CONTEXT.md b/CONTEXT.md index 3ba2c93..ba04946 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,19 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-19 — N100 MADE LOCATION-INDEPENDENT via Tailscale.** `felhom-pve` (the N100) joined the + tailnet as a host package (`100.70.170.35`, expiry disabled); DooPlex already advertised + `192.168.0.0/24` via its GitOps k3s `admin-system/tailscale` pod (`100.107.87.53`). `ssh felhom-pve` + now targets the tailnet IP (direct-over-LAN at home, tunnel when away); `felhom-pve-lan` = + `192.168.0.162` fallback. `--accept-dns=false` on the host (pi-hole split-horizon preserved). The + host was converted **static→DHCP** (reservation MAC `68:1d:ef:5d:a6:64`→`.162` keeps home behaviour). + **Measured spike:** `--accept-routes` on the N100 *while on `192.168.0.0/24`* hairpins the local + subnet (table 52 overrides `main`) → kept OFF, it's a travel-only opt-in. **PBS is offsite** (its own + WireGuard tunnel to Hetzner, not the home LAN) so it needs none of this. **Recorded, NOT fixed:** the + controller↔agent plane is hard-pinned to `192.168.0.162` (`felhom-agent` `listen_addr` + + guest `bootstrap.json`), so the box is reachable + backs up off-LAN but its local operator plane is + not yet portable. Full doc: `documentation/operations/tailscale.md`. No code changed. + - **2026-07-19 — CC WORKFLOW MIGRATED TO DOOPLEX.** Claude Code now runs natively on DooPlex (192.168.0.180, Debian 13, `kisfenyo`) in a tmux session, working directly in `/mnt/5_hdd/felhom.eu/git/`. **DooPlex is now the canonical CC environment; the Windows diff --git a/REPORT.md b/REPORT.md index 6887868..b427b02 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,193 +2,123 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## CC workflow migrated from the Windows workstation to DooPlex — 2026-07-19 +## Tailscale on DooPlex + felhom-pve — make the N100 location-independent — 2026-07-19 -Claude Code now runs natively on **DooPlex (192.168.0.180, Debian 13, `kisfenyo`)**, working directly -in `/mnt/5_hdd/felhom.eu/git/`. Builds are local commands; felhom-pve is one `ssh` hop; -`kubectl` is local. The Windows workstation can stay powered off. Docs/skills adapted in all four -repos, non-versioned assets migrated, environment wired and verified end-to-end. +**Not a code change.** Network/ops on two production hosts (DooPlex, felhom-pve). Run supervised, +normal permission prompts, one host at a time, verify-after-every-mutation. Baseline `felhom.eu` +`a771cda`. -**Not a code change** — no Go source, no `build.sh`, no ArgoCD/GitOps rule was touched. Only *where -the commands are typed from*. +### 1. Part 0 audit (verbatim highlights) + dependency classification ---- +``` +# felhom-pve host — STATIC, not DHCP (contradicted the operator's expectation) +iface vmbr0 inet static / address 192.168.0.162/24 / gateway 192.168.0.1 +resolv.conf: search local / nameserver 192.168.0.250 (pi-hole) -### 1. Part 0 inventory +# PBS is OFFSITE, not on 192.168.0.180 (contradicted the task premise) +storage.cfg: pbs felhom-pbs server 10.77.0.1 datastore felhom-offsite namespace demo-felhom +ip route get 10.77.0.1 -> dev wg-felhom src 10.77.0.2 +wg-felhom endpoint = 167.233.158.164:443 (Hetzner, public); handshake fresh, 3.75 GiB sent -| Asset | Class | Disposition | -|---|---|---| -| Repo docs (`CLAUDE.md`, `CONTEXT.md`, `REUSE.md`, runbooks, `README.md`) ×4 repos | versioned | adapted in place, pushed (§2) | -| `felhom.eu/skills/*` (4 skills — build-deploy, ui-design, testing, app-catalog) | versioned | adapted; installed on DooPlex as symlinks | -| `felhom.eu/scripts/install_skills.py` | versioned | made cross-platform (POSIX symlink / Windows junction) | -| `E:\git\CLAUDE.md` (workspace root, in no repo) | portable | ported → `/mnt/5_hdd/felhom.eu/git/CLAUDE.md` **and** versioned copy at `documentation/runbooks/workspace-CLAUDE.md` (verified byte-identical) | -| `~/.claude/settings.json` | portable | adapted → DooPlex (§5) | -| `E:\git\.claude\settings.local.json` (workspace-scoped) | portable | **folded into** the user settings; every entry was a Windows `$SSH`/path one-off with no Linux meaning | -| **Project memory — 119 files, 672 KB** (`~/.claude/projects/E--git/memory/`) | portable | **migrated** → `/mnt/5_hdd/felhom.eu/git/.claude-memory/`, surfaced via `~/.claude/projects/-mnt-5-hdd-felhom-eu-git/memory` symlink. *Not anticipated by the spec* — flagged per its "migrate by the same rules" clause | -| `~/.claude/skills/*` (Windows junctions) | versioned source | recreated on DooPlex as POSIX symlinks by the fixed installer | -| Per-repo `.claude/settings.local.json` | — | **none exist** (checked all repos) | -| `~/.claude/CLAUDE.md` | — | **does not exist** on the Windows side; nothing to merge | -| Session history, `~/.claude.json`, `history.jsonl`, `shell-snapshots/`, `file-history/` | not portable | path-keyed / machine-local — **not copied**, accepted loss | -| Stray workspace files (`felhom-agent-0.2x.0.bin` ×4, `felhom-opsign.exe`, `longhorn-faulted-diagnosis-*.md`, `prompt_template.txt`) | not portable | build leftovers/scratch — **not copied**; left on Windows | -| Non-felhom repos (`jarr`, `recipe-importer`, `revfulop-calendar`, `sparkyfitness-export`, `misc-scripts`) | out of scope | untouched | +# guest 9201 — already DHCP; grep 192.168.0.x found the pinned control-plane endpoint +pct config 9201 net0: ...,ip=dhcp +9201:/etc/felhom-bootstrap/bootstrap.json "endpoint": "192.168.0.162:8443" +felhom-agent listen_addr = 192.168.0.162:8443 (LISTEN 192.168.0.162:8443 users:(("felhom-agent"))) -### 2. Commits pushed +# other +grep 192.168.0.x in /etc/pve /etc/cron /etc/systemd (excl .162): (none) +pve-firewall: disabled/running (no Tailscale allow-rule needed) +DooPlex ipv4 ip_forward = 1 (k3s), ipv6 = 0 ; both hosts Debian 13 trixie +``` -Windows batch (the last Windows pushes): - -| Repo | Commit | Content | -|---|---|---| -| felhom-controller | `ac7323dc9a` | docs: migrate workflow to DooPlex-local execution | -| felhom-agent | `d81fddc7b1` | docs: migrate workflow to DooPlex-local execution | -| felhom.eu | `ee48a48288` | docs: migrate workflow to DooPlex-local execution | -| felhom-controller | `29eda5d86e` | docs(runbook): e2e live-drive commands are DooPlex-local | -| felhom.eu | `29b2b8ad7c` | docs: DooPlex-local commands in PROMPT-TEMPLATE + manual-build runbook | -| app-catalog-felhom.eu | `34d50a33ac` | docs: migrate workflow to DooPlex-local execution | - -Pushed **from DooPlex** (the new-home proof): - -| Repo | Commit | Content | -|---|---|---| -| felhom-controller | `2fcae041ae` | chore: mark `build-samba-image.sh` executable (mode-only) | -| felhom.eu | `a868e3585a` | docs: DooPlex is the canonical CC environment (CONTEXT.md) + iso scripts executable | -| felhom.eu | `fa8c0d5972` | docs(workspace): record the migrated project-memory location | - -**Final HEADs, all four trees clean on DooPlex:** felhom-controller `29eda5d86e`, felhom-agent -`d81fddc7b1`, felhom.eu `29b2b8ad7c`, app-catalog-felhom.eu `34d50a33ac`. - -### 3. Grep-hit counts (env-specific references in live docs) - -Scope: `*.md` + `*.py`, excluding `CHANGELOG.md`, `REPORT*.md`, and the historical -`documentation/{audits,tests,pilot}/` (history is history). - -| Repo | Before | After | Remaining — all intentional | +| Dep | Where | Address | Survives the move? | |---|---|---|---| -| felhom-controller | 47 | 36 | `CLAUDE.md` 4 + `RUNBOOK-e2e-live-drive.md` 2 = the new "Legacy: Windows workstation" notes; `docs/vscode-ssh-fix.md` 27 = the legacy doc, now carrying a LEGACY banner; **`TASK.md` 3 = see §6** | -| felhom-agent | 4 | 4 | `CLAUDE.md` 3 = new legacy note; `PLAN.md` 1 = historical plan note | -| felhom.eu | 27 | 20 | `CLAUDE.md` 3 + `skills/felhom-build-deploy` 2 + `workspace-CLAUDE.md` 8 = legacy sections; `CONTEXT.md` 2 = the migration note itself; `install_skills.py` 3 = the deliberate Windows branch; `architecture/04` 1 (`sk-ssh-ed25519@openssh.com`) + `break-glass.md` 1 (`/run/sshd`) = **protocol/package names, never environment** | -| app-catalog-felhom.eu | 1 | 0 | — | +| PBS backup | host storage.cfg | `10.77.0.1` via `wg-felhom`→Hetzner:443 | **Yes** — offsite tunnel, internet-only | +| Default gateway | host | `192.168.0.1` | Yes — DHCP-derived once host is DHCP | +| DNS (pi-hole) | host + guest | `192.168.0.250` | Yes for general DNS; `gitea.dooplex.hu` split-horizon won't resolve off-LAN → **finding** | +| **Agent listen + guest endpoint** | agent.json + guest bootstrap.json | **`192.168.0.162:8443`** | **NO** — hard-pinned to host LAN IP; L2-bridged → subnet routes don't help → **finding, record-not-fix** | +| `*.demo-felhom.eu` LAN forward | home pi-hole → box | inbound | LAN-local demo breaks off-LAN; Cloudflare Tunnel external access unaffected → note only | -**No unadapted live instruction remains.** The one deliberate exception is `felhom-controller/TASK.md` -(§6). Genuinely-platform mentions were left alone by design: `_linux.go`/`_other.go` dev stubs -(`REUSE.md`), `\\FELHOM` shares in Windows Explorer (`controller/README.md`), the `openssh-client` -package in the restic image, and the Windows-grep-multibyte rationale in the gate scripts. +### 2. Tailscale nodes -### 4. Verification (all from DooPlex, verbatim) +| Node | Tailnet IPv4 | Route advertised | Key expiry | How installed | +|---|---|---|---|---| +| `dooplex` | `100.107.87.53` | `192.168.0.0/24` (approved) | disabled | **pre-existing** k3s pod `admin-system/tailscale` (hostNetwork, GitOps) — *not* installed by this task | +| `felhom-pve` | `100.70.170.35` | — | disabled (operator-confirmed) | host apt package (trixie repo), `--accept-dns=false` | + +`tailscale status` from the N100: both nodes listed/online. resolv.conf on felhom-pve **byte-identical** +after `tailscale up` (`CorpDNS:false`). + +**Course correction:** the task assumed Tailscale had to be *installed* on DooPlex and that PBS lived +on `.180`. Both were wrong — DooPlex already ran Tailscale as a GitOps k3s pod (the operator was +right; my `which tailscale`/systemd probe was too narrow and missed it), and PBS is offsite. The +redundant host-level Tailscale I briefly installed on DooPlex was **fully purged** (package + repo + +keyring + sysctl file), verified: k3s `tailscale0` and the `dooplex` node stayed up, `ip_forward` +still 1. + +### 3. accept-routes spike (§3.1) — CONFIRMED degradation, rolled back + +Enabling `--accept-routes` on felhom-pve *while on `192.168.0.0/24`*: ``` -=== felhom-controller (controller/) === BUILD+VET OK -=== felhom-agent (root) === BUILD+VET OK -=== felhom.eu (hub/) === BUILD+VET OK - -=== kubectl (local) === -NAME READY STATUS RESTARTS AGE -contact-mailer-5bb869b85b-pqtjt 1/1 Running 0 20d -felhom-webpage-69cdbfc47-jm7lv 2/2 Running 2 (9d ago) 18d -filebrowser-59ff87cd88-662kz 1/1 Running 0 38d -hub-6d859c4dff-r5vrx 1/1 Running 0 14h -umami-7cd7f95cd8-89wrn 1/1 Running 1 (38d ago) 42d -umami-db-5fd98f59c5-xhwfr 1/1 Running 0 38d - -=== pve hop === -pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) - -=== guest 9201 (through the pve hop) === -gitea.dooplex.hu/admin/felhom-controller:0.147.3 Up 2 hours (healthy) - -=== claude + skills === -2.1.205 (Claude Code) -felhom-app-catalog felhom-build-deploy felhom-testing felhom-ui-design - -=== tmux === tmux 3.5a - -=== disk headroom === -/dev/sda1 9.1T 2.1T 6.6T 24% /mnt/5_hdd -/dev/sdb1 445G 284G 139G 68% / - -=== repo HEADs / cleanliness === -felhom-controller 29eda5d86e dirty=[] -felhom-agent d81fddc7b1 dirty=[] -felhom.eu 29b2b8ad7c dirty=[] -app-catalog-felhom.eu 34d50a33ac dirty=[] - -=== live workspace CLAUDE.md vs versioned copy === IDENTICAL +PRE : ip route get 192.168.0.180 -> dev vmbr0 src 192.168.0.162 ; ping .180 avg 0.3 ms +POST: ip route get 192.168.0.180 -> dev tailscale0 table 52 src 100.70.170.35 ; ping .180 avg 68 ms (max 136) + table 52 contains: 192.168.0.0/24 dev tailscale0 ; ip rule 5270: from all lookup 52 (outranks main) + -> local subnet hijacked onto the tunnel; inbound SSH went asymmetric and STALLED + PBS (10.77.0.1 via wg-felhom) UNAFFECTED +ROLLBACK (via the tailnet path 100.70.170.35, which was immune): tailscale set --accept-routes=false + -> route to .180 back on vmbr0 direct ``` -`install_skills.py` was exercised on DooPlex twice: first run created 4 symlinks, second run reported -"already installed" for all 4 with the repo's `skills/*/SKILL.md` intact and the tree clean — -confirming idempotency **and** that the rmtree hazard (§5) does not fire. +Verdict: keep `--accept-routes=false` at home; it is a **travel-only opt-in** (safe on a foreign +subnet). Documented with the reasoning in `documentation/operations/tailscale.md`. -**Explicitly NOT validated:** a full image **build → push → deploy** cycle from DooPlex. Nothing -here required a build, and a docs-only change must not push an image. The first real task run on -DooPlex should **treat its build step as a validation step — watch it, do not assume.** The pieces it -depends on are individually proven (local Go toolchain, registry creds already on the host, `ssh -felhom-pve` → `pct exec 9201`), but the composed path has not run end-to-end. +### 4. SSH chain over the tailnet (§3.2) -### 5. Notable fixes made along the way +`~/.ssh/config` on DooPlex: `Host felhom-pve` → `HostName 100.70.170.35`; added `Host felhom-pve-lan` +→ `192.168.0.162`. Proof: -- **`install_skills.py` had a latent data-loss bug on POSIX.** `remove_existing()` used - `os.rmdir()`, which fails on a symlink-to-directory and fell through to - `shutil.rmtree()` — and rmtree following the symlink would have **deleted the repo's own - `skills/` sources**. Now symlinks are unlinked explicitly before any rmdir/rmtree path is reached, - and existence checks use `os.path.lexists()` so a broken link is replaced rather than ignored. - (Windows never hit this: junctions satisfy `os.rmdir`.) -- **`skipDangerousModePermissionPrompt: true` was deliberately DROPPED** from the DooPlex settings — - it directly contradicts the new production-host guardrail ("do not run CC with permission prompts - disabled on this host"). Everything else portable was kept: `model: opus[1m]`, `effortLevel: high`, - `tui: fullscreen`, `agentPushNotifEnabled`, the three plugins. -- **Permission allowlist rebuilt** (30 rules): Windows one-offs dropped (`$SSH …`, `cygpath`, - `SCP=/c/Windows/…`, `/e/git/…` paths, `dir:*`, `Read(//c/Users/…)`), Linux equivalents added - (`sudo kubectl:*`, `ssh felhom-pve:*`, `python3:*`, `go test:*`, `df:*`). **Also dropped:** the - recipe-site WebFetch domains (`streetkitchen.hu`, `sobors.hu`, `kiskegyed.hu`, `gastrohobbi.hu`), - the jarr/pnpm entries, and assorted malformed rules (`Bash(1:*)`, `Bash(done)`, `Bash(settings")`) - — all belonged to non-felhom projects that are not migrating. -- **`PROMPT-TEMPLATE.md` §13 contradicted the GitOps rule**: it instructed `kubectl set image` for - hub deploys, which the hub docs explicitly forbid (auto-sync is OFF; imperative changes are - reverted on the next sync). Corrected to the manifest-bump + deliberate-sync method while the - block was being de-SSH'd. -- **Demo PVE host key had changed** on DooPlex (the known N100-reprovision gotcha). The new key was - **verified identical to the one the Windows workstation already trusted** - (`SHA256:bcSDbVzxoLQhOURUDzRGf0zJheU3wyXR4EmQ4NEPuTo`) before being accepted — not blindly - trust-on-first-use. `ssh-keyscan` was avoided (it hangs, per prior findings). +``` +ssh felhom-pve -> pve-manager/9.2.2 ; pct exec 9201 docker ps -> felhom-controller:0.148.0 Up (healthy) +tailscale ping 100.70.170.35 -> pong via 192.168.0.162:41641 in 1ms (DIRECT over LAN, not DERP) +ssh felhom-pve-lan -> OK (LAN fallback works) +``` -### 6. Pending — Viktor +### 5. Part 4 — host static→DHCP: **DONE** (operator at console, reservation set) -1. **`claude` login (OAuth) — the only blocking step.** Everything else is ready. On DooPlex: - ``` - ssh kisfenyo@192.168.0.180 - tmux new -A -s cc - cd /mnt/5_hdd/felhom.eu/git && claude - ``` - (`~/.local/bin` was appended to `~/.bashrc`; a fresh login shell picks it up.) -2. **Push token: NOT needed.** DooPlex already had a working `credential.helper=store` + - `~/.git-credentials`; four pushes from DooPlex succeeded during this task. Nothing to issue, - nothing written anywhere — the credential stays out-of-band in `~/.git-credentials`. -3. **`felhom-controller/TASK.md` is a stale spec** — it targets **v0.28.0** and the retired - `/opt/docker` compose deploy mechanism, and per the artifact taxonomy a `TASK.md` at a repo root - reads as "implement this". It was left untouched deliberately (not mine to delete). Recommend - deleting or moving it to `documentation/`; it is the last live-looking Windows-command block in - the four repos. -4. **Remote-access preflight**, before relying on this from outside the LAN: verify the SSH path to - DooPlex from that location. Gitea's **web UI is geo-fenced (HU/DE/US)** — CC on DooPlex is - unaffected (it pushes over the LAN), but browser access to Gitea from elsewhere is not. -5. **Windows side is untouched** and remains a stale fallback. Nothing was deleted there. +Reservation MAC `68:1d:ef:5d:a6:64`→`192.168.0.162` set by operator first (so the LAN-pinned agent +keeps working at home). Applied detached (`systemd-run … ifreload -a`); result: -### 7. Observations (out of scope) +``` +ifreload rc=0 ; vmbr0 inet dhcp -> got 192.168.0.162 back (reservation) +default via 192.168.0.1 ; felhom-agent active, still bound 192.168.0.162:8443 (no restart) +guest 9201: 12 containers ; Tailscale still direct 1ms +``` -- **R-43/R-44 landed mid-task.** `felhom-controller` had ~721 lines of uncommitted, build-clean - offsite-reconstitute work when this task started. Rather than commit unreleased data-restore code - under a docs commit, it was surfaced and left alone; you shipped it as **v0.148.0 (`062357f`)** - during the run. The Part 3 gate then compiled that code on DooPlex — `go build ./... && go vet - ./...` **clean** — so v0.148.0 has a green toolchain gate on the new host as a side effect. -- **Windows `grep -i` combined with `-F -f ` silently returns zero matches** in this Git - Bash build — it produced a false "0 remaining references" during verification and was only caught - by cross-checking on Linux. This is the same false-negative class the repos already warn about for - multibyte emoji; the counts in §3 are all from Linux grep. One more reason the Python gates exist. -- **Four DooPlex script files were mode-only dirty** (`100644 → 100755`) — the exec bit set when they - were run locally, which Windows commits cannot carry. Committed as mode-only changes so the new - clean-tree gate starts from a genuinely clean state; expect the same for any script Windows added. -- **The skill listing says three skills; there are four.** `felhom-app-catalog` was missing from the - workspace CLAUDE.md's Skills section — corrected in the ported file. -- `~/.claude/projects/` on DooPlex already contained a `-mnt-5-hdd-felhom-eu` entry, which is what - confirmed the path-key mangling scheme (`_` and `.` both → `-`) and let the memory symlink be - placed on the correct key rather than guessed. +Exact revert staged on the box: `/root/interfaces.static-revert-20260719`. resolv.conf is now +DHCP-managed (`.250` + `.1`) instead of the old static `search local` — expected (Tailscale didn't +touch it; accept-dns rule intact). + +### 6. Off-LAN legs NOT validated at home → vacation-day checklist + +Cannot be tested until the box is on a foreign LAN: PBS from a foreign subnet, DERP-relay fallback, +off-LAN `gitea.dooplex.hu` resolution. The step-by-step is in `documentation/operations/tailscale.md` +(§ *Vacation-day checklist*) — notably `ssh felhom-pve "pvesm status | grep -i pbs"` is the real +off-LAN PBS test. + +### 7. Findings recorded, NOT fixed + +1. **Control plane pinned to `192.168.0.162`** (HIGH for travel): `felhom-agent` `listen_addr` and + guest `bootstrap.json` `endpoint` both = `192.168.0.162:8443`. Off-LAN the agent can't bind → + agent down → controller can't reach it. Subnet routes don't help (L2 bridge). Needs a design + change (bind `0.0.0.0` + stable guest→host path). **The N100 is reachable and backs up off-LAN, + but its local operator plane is not yet portable.** +2. **Off-LAN gitea DNS**: `gitea.dooplex.hu` (pi-hole split-horizon) won't resolve on a foreign LAN; + image pulls while travelling need the travel `--accept-routes` toggle + a temporary `/etc/hosts` + entry. Contingency ("Plan B") documented, not armed. + +### 8. Commit + +Docs commit: `__COMMIT__` (felhom.eu). Files: `documentation/operations/tailscale.md` (new), +`CLAUDE.md` + `skills/felhom-build-deploy/SKILL.md` env-table notes, `CONTEXT.md` decision entry, +this `REPORT.md`. No secrets committed (tunnel keys/tokens/PBS fingerprint referenced out-of-band). diff --git a/documentation/operations/tailscale.md b/documentation/operations/tailscale.md new file mode 100644 index 0000000..eb4f144 --- /dev/null +++ b/documentation/operations/tailscale.md @@ -0,0 +1,104 @@ +# Tailscale — location-independent access to felhom-pve (the N100) + +> Added 2026-07-19. Goal: the demo Proxmox host (N100, `felhom-pve`) can be taken to a different +> LAN (e.g. a vacation home) and stay reachable from DooPlex, while its own dependencies keep +> working. The `ssh felhom-pve` alias remains the interface — only its transport changed. + +## Topology (who is what) + +| Node | Tailnet name | Tailnet IPv4 | Transport it provides | Managed as | +|---|---|---|---|---| +| DooPlex (192.168.0.180) | `dooplex` | `100.107.87.53` | **subnet router** advertising `192.168.0.0/24`; expiry disabled | **k3s pod** `admin-system/tailscale` (`hostNetwork`, GitOps/ArgoCD) — 157 d old, predates this task | +| N100 (192.168.0.162) | `felhom-pve` | `100.70.170.35` | its own node; expiry disabled | **host package** `tailscale` (Debian trixie apt repo), `tailscaled.service` | + +Tailnet: `nagyfenyvesi.viktor@gmail.com` (Tailscale Free). Both nodes run v1.98.x. + +`ssh felhom-pve` now resolves to `100.70.170.35` (see `~/.ssh/config` on DooPlex). Tailscale takes +the **direct LAN path** when both boxes are home (measured: `pong … via 192.168.0.162:41641 in 1ms`) +and falls back to the tunnel / a DERP relay when the N100 is remote. `felhom-pve-lan` → +`192.168.0.162` is the explicit LAN-only fallback. + +## Hard rule: `--accept-dns=false` on the host node + +The N100's `tailscaled` runs with **`--accept-dns=false`** (`CorpDNS:false`). MagicDNS must never +rewrite `/etc/resolv.conf`: the box relies on pi-hole (`192.168.0.250`) for split-horizon +(`gitea.dooplex.hu` → `192.168.0.180`). Verified byte-identical after `tailscale up`. (DooPlex's +containerised node runs `TS_ACCEPT_DNS=true`, but because it is a pod with its own mount namespace +that only rewrites the *container's* resolv.conf — the DooPlex host resolv.conf is untouched.) + +## The accept-routes spike (measured 2026-07-19) — leave it OFF at home + +`--accept-routes` is **OFF** on `felhom-pve` and should stay off **while the box is on +`192.168.0.0/24`**. Enabling it there was tested and **degraded local traffic**: + +- Tailscale installed `192.168.0.0/24 dev tailscale0` into **routing table 52** with an + `ip rule 5270: from all lookup 52` that outranks `main`, so the *whole local subnet* hijacked onto + the tunnel: `ip route get 192.168.0.180` and `…0.250` flipped from `vmbr0` to `tailscale0`, + ping RTT to `.180` went `0.3 ms → ~68 ms`, and inbound SSH went asymmetric and stalled. +- **PBS was unaffected** (`10.77.0.1` stays on `wg-felhom`), because PBS does not use this subnet. +- Rollback (`tailscale set --accept-routes=false`) restored direct `vmbr0` immediately. + +**When travelling**, the N100 is on a *different* subnet, so accepting `192.168.0.0/24` no longer +overlaps the local net and is safe — it lets the box reach `192.168.0.180`/gitea over the tunnel. +Treat it as an **opt-in, travel-only** toggle you turn on only if you need a home-LAN service (e.g. a +new controller-image pull), and **turn back OFF before the box returns home**. Management SSH over +`100.x` is immune to the hairpin either way (that path was used to perform the rollback). + +## PBS does NOT depend on the home LAN + +Contrary to the original assumption, PBS is **offsite**: `storage.cfg` targets `10.77.0.1` over the +**WireGuard** interface `wg-felhom` whose endpoint is a public Hetzner address (`…:443`). It works +from any location with internet and needs neither the subnet router nor `--accept-routes`. This is +why the subnet router is *nice-to-have* (gitea image pulls while away), not load-bearing. + +## Key expiry + +Disabled on **both** nodes in the admin console (a node whose key expires mid-trip drops off the +tailnet). Confirm on the Machines page: each row shows an *Expiry disabled* badge. + +## Rollback / uninstall (host node) + +```bash +ssh felhom-pve "tailscale set --accept-routes=false" # undo the travel toggle +ssh felhom-pve "tailscale down" # leave the tailnet (reversible: tailscale up) +ssh felhom-pve "systemctl disable --now tailscaled && apt-get purge -y tailscale" # full removal +# then repoint ~/.ssh/config Host felhom-pve back to 192.168.0.162 (or use felhom-pve-lan) +``` + +## Portability of the N100 itself + +- **Host network is DHCP** (converted 2026-07-19; `iface vmbr0 inet dhcp`). At home a DHCP + **reservation** pins MAC `68:1d:ef:5d:a6:64` → `192.168.0.162` so home behaviour is unchanged. + Revert on the box: `cp /root/interfaces.static-revert-20260719 /etc/network/interfaces && ifreload -a`. +- **Guest 9201** is already DHCP. + +### ⚠️ Known blocker — the controller↔agent plane is pinned to `192.168.0.162` (NOT fixed) + +`felhom-agent` binds `listen_addr: 192.168.0.162:8443` (in `/etc/felhom-agent/agent.json`) and guest +9201's `/etc/felhom-bootstrap/bootstrap.json` points its controller at `192.168.0.162:8443`. On a +foreign LAN the host will not *have* `192.168.0.162`, so the agent cannot bind it (agent down) and +the guest cannot reach it. **Tailscale and host-DHCP do not fix this** (guests bridge at L2). Making +the box fully functional off-LAN needs a design change — bind `0.0.0.0:8443` and give the guest a +stable way to reach the host (e.g. a host-only bridge with a fixed private IP). **Until then the N100 +is reachable and backs up off-LAN, but its local operator plane is not portable.** + +### Off-LAN DNS note + +At a foreign LAN, DHCP repoints the host's resolv.conf at the local resolver, so `gitea.dooplex.hu` +(pi-hole split-horizon) will not resolve. To pull a controller image while away, either turn on the +travel `--accept-routes` toggle *and* add a temporary `192.168.0.180 gitea.dooplex.hu` line to +`/etc/hosts`, or point DNS at `192.168.0.250` over the subnet route. External customer access is via +Cloudflare Tunnel and is unaffected. + +## Vacation-day checklist (for the operator) + +1. At the destination, connect the N100 by **wired ethernet**, power on, wait ~2 min. +2. From DooPlex: `tailscale`-side — check the pod: `sudo kubectl -n admin-system exec deploy/tailscale -- tailscale status` shows `felhom-pve` **online** (likely *relayed*/DERP at first — fine). +3. `ssh felhom-pve pveversion` — management reachable over the tailnet. +4. `ssh felhom-pve "pvesm status | grep -i pbs"` — **the real off-LAN PBS test** (could not be + validated at home; PBS rides its own WireGuard tunnel, so it should be green). +5. Demo URL via Cloudflare Tunnel loads (customer-facing path, independent of all the above). +6. If you need a home-LAN service (gitea image pull): `ssh felhom-pve "tailscale set --accept-routes=true"` + + `/etc/hosts` entry as above — then set it back to `false` before the box comes home. +7. Expect the controller↔agent plane to be **down** off-LAN until the pinned-`.162` blocker above is + fixed. diff --git a/skills/felhom-build-deploy/SKILL.md b/skills/felhom-build-deploy/SKILL.md index 0ee3093..135fedb 100644 --- a/skills/felhom-build-deploy/SKILL.md +++ b/skills/felhom-build-deploy/SKILL.md @@ -25,7 +25,7 @@ FELHOM_ROOT=/mnt/5_hdd/felhom.eu # working root — ALL felhom repos/build/dri | Host | Access | Role | |---|---|---| | **DooPlex (this host, k3s)** | local | build+push images/binaries (`$FELHOM_ROOT/build/felhom-{controller,hub,agent}`, repos under `$FELHOM_ROOT/git/`), `sudo kubectl` | -| Demo Proxmox host | `ssh felhom-pve` (root@192.168.0.162) | agent deploy, `pct` into guests | +| Demo Proxmox host (N100) | `ssh felhom-pve` — via Tailscale `100.70.170.35` (location-independent; `felhom-pve-lan` = `192.168.0.162` LAN fallback). See `documentation/operations/tailscale.md` | agent deploy, `pct` into guests | | Demo guest 9201 | via `ssh felhom-pve "pct exec 9201 -- bash -c '...'"` | the live controller | | felhotest (legacy) | `ssh -p 33022 kisfenyo@router.abonet.hu` | OLD /opt/docker compose mechanism — not the 9201 flow |