felhom.eu/documentation/operations/tailscale.md

# Tailscale — location-independent access to the demo nodes (N100 + HP t740)

> Added 2026-07-19. Goal: the demo Proxmox host (N100, `felhom-pve`) can be taken to a different
> LAN (e.g. a vacation home) and stay reachable from DooPlex, while its own dependencies keep
> working. The `ssh felhom-pve` alias remains the interface — only its transport changed.

## Topology (who is what)

| Node | Tailnet name | Tailnet IPv4 | Transport it provides | Managed as |
|---|---|---|---|---|
| DooPlex (192.168.0.180) | `dooplex` | `100.107.87.53` | **subnet router** advertising `192.168.0.0/24`; expiry disabled | **k3s pod** `admin-system/tailscale` (`hostNetwork`, GitOps/ArgoCD) — 157 d old, predates this task |
| N100 (192.168.0.162) | `felhom-pve` | `100.70.170.35` | its own node; expiry disabled | **host package** `tailscale` (Debian trixie apt repo), `tailscaled.service` |
| HP t740 (`demo-hp-bb76ea`) | `demo-hp` | `100.76.96.79` | its own node; **expiry NOT yet disabled — see below** | **host package** `tailscale` 1.98.9 (Debian trixie apt repo), `tailscaled.service`, added 2026-07-21 |

Tailnet: `nagyfenyvesi.viktor@gmail.com` (Tailscale Free). Both nodes run v1.98.x.

`ssh felhom-pve` now resolves to `100.70.170.35` (see `~/.ssh/config` on DooPlex). Tailscale takes
the **direct LAN path** when both boxes are home (measured: `pong … via 192.168.0.162:41641 in 1ms`)
and falls back to the tunnel / a DERP relay when the N100 is remote. `felhom-pve-lan` →
`192.168.0.162` is the explicit LAN-only fallback.

## This is an OPERATOR-LAB EXCEPTION, not product shape

**Real customer boxes never get tailscale.** Customer operator-access is the WireGuard tunnel plus
the H1 OOB path, full stop. Tailscale exists on `felhom-pve` and `demo-hp` only because those two are
lab machines that live on LANs the operator does not control.

`demo-hp` is the confusing one: it is **customer-shaped** — a normal appliance install, a real
customer record, a real guest, a real day-0 — with tailscale bolted on afterwards by hand. A future
product-shape audit that finds it there must not conclude the product ships it. See
`operations/nodes.md`.

## demo-hp joined 2026-07-21

`tailscale up --authkey=file:… --hostname=demo-hp --accept-routes=false`, then
`tailscale set --accept-dns=false` (see the hard rule below — it was needed, and why is worth
reading). Auth was a pre-auth key, passed as `--authkey=file:<path>` and shredded immediately so it
never appeared in the box's process list. Verified from DooPlex: `ping` ~40 ms, direct SSH with no
ProxyJump, and the path is **direct** (`37.191.56.193:45127`), not a DERP relay.

**OPEN: key expiry is still enabled on `demo-hp`** (expires `2027-01-17`), unlike the other two
nodes which have it disabled. Disabling it is a per-device **admin-console toggle** (Machines → 
demo-hp → Disable key expiry) or an API call with a `tskey-api-…` token; a `tskey-auth-…` pre-auth
key cannot do it (verified: the API returns 401). Until it is toggled the node will drop off the
tailnet on that date and need re-authentication.

## Hard rule: `--accept-dns=false` on the host node

The N100's `tailscaled` runs with **`--accept-dns=false`** (`CorpDNS:false`). MagicDNS must never
rewrite `/etc/resolv.conf`: the box relies on pi-hole (`192.168.0.250`) for split-horizon
(`gitea.dooplex.hu` → `192.168.0.180`). Verified byte-identical after `tailscale up`.

**On `demo-hp` this rule was broken and then fixed, which is the useful part of the story.** The join
omitted `--accept-dns=false`, and MagicDNS promptly rewrote `/etc/resolv.conf` to
`nameserver 100.100.100.100` — exactly what this rule forbids. Nothing broke *at the vacation site*,
because there is no pi-hole there and both `gitea.dooplex.hu` and `hub.felhom.eu` resolve publicly
anyway; the damage would have appeared silently **when the box comes home**, where split-horizon is
what makes `gitea.dooplex.hu` → `192.168.0.180` local. `tailscale set --accept-dns=false` restored
`nameserver 192.168.0.1` immediately, with the tailnet and the agent unaffected. **Pass
`--accept-dns=false` at join time on any host node; do not rely on noticing afterwards.** (DooPlex's
containerised node runs `TS_ACCEPT_DNS=true`, but because it is a pod with its own mount namespace
that only rewrites the *container's* resolv.conf — the DooPlex host resolv.conf is untouched.)

## The accept-routes spike (measured 2026-07-19) — leave it OFF at home

`--accept-routes` is **OFF** on `felhom-pve` and should stay off **while the box is on
`192.168.0.0/24`**. Enabling it there was tested and **degraded local traffic**:

- Tailscale installed `192.168.0.0/24 dev tailscale0` into **routing table 52** with an
  `ip rule 5270: from all lookup 52` that outranks `main`, so the *whole local subnet* hijacked onto
  the tunnel: `ip route get 192.168.0.180` and `…0.250` flipped from `vmbr0` to `tailscale0`,
  ping RTT to `.180` went `0.3 ms → ~68 ms`, and inbound SSH went asymmetric and stalled.
- **PBS was unaffected** (`10.77.0.1` stays on `wg-felhom`), because PBS does not use this subnet.
- Rollback (`tailscale set --accept-routes=false`) restored direct `vmbr0` immediately.

**When travelling**, the N100 is on a *different* subnet, so accepting `192.168.0.0/24` no longer
overlaps the local net and is safe — it lets the box reach `192.168.0.180`/gitea over the tunnel.
Treat it as an **opt-in, travel-only** toggle you turn on only if you need a home-LAN service (e.g. a
new controller-image pull), and **turn back OFF before the box returns home**. Management SSH over
`100.x` is immune to the hairpin either way (that path was used to perform the rollback).

## PBS does NOT depend on the home LAN

Contrary to the original assumption, PBS is **offsite**: `storage.cfg` targets `10.77.0.1` over the
**WireGuard** interface `wg-felhom` whose endpoint is a public Hetzner address (`…:443`). It works
from any location with internet and needs neither the subnet router nor `--accept-routes`. This is
why the subnet router is *nice-to-have* (gitea image pulls while away), not load-bearing.

## Key expiry

Disabled on **both** nodes in the admin console (a node whose key expires mid-trip drops off the
tailnet). Confirm on the Machines page: each row shows an *Expiry disabled* badge.

## Rollback / uninstall (host node)

```bash
ssh felhom-pve "tailscale set --accept-routes=false"     # undo the travel toggle
ssh felhom-pve "tailscale down"                           # leave the tailnet (reversible: tailscale up)
ssh felhom-pve "systemctl disable --now tailscaled && apt-get purge -y tailscale"   # full removal
# then repoint ~/.ssh/config Host felhom-pve back to 192.168.0.162 (or use felhom-pve-lan)
```

## Portability of the N100 itself

- **Host network is DHCP** (converted 2026-07-19; `iface vmbr0 inet dhcp`). At home a DHCP
  **reservation** pins MAC `68:1d:ef:5d:a6:64` → `192.168.0.162` so home behaviour is unchanged.
  Revert on the box: `cp /root/interfaces.static-revert-20260719 /etc/network/interfaces && ifreload -a`.
- **Guest 9201** is already DHCP.

### ⚠️ Known blocker — the controller↔agent plane is pinned to `192.168.0.162` (NOT fixed)

`felhom-agent` binds `listen_addr: 192.168.0.162:8443` (in `/etc/felhom-agent/agent.json`) and guest
9201's `/etc/felhom-bootstrap/bootstrap.json` points its controller at `192.168.0.162:8443`. On a
foreign LAN the host will not *have* `192.168.0.162`, so the agent cannot bind it (agent down) and
the guest cannot reach it. **Tailscale and host-DHCP do not fix this** (guests bridge at L2). Making
the box fully functional off-LAN needs a design change — bind `0.0.0.0:8443` and give the guest a
stable way to reach the host (e.g. a host-only bridge with a fixed private IP). **Until then the N100
is reachable and backs up off-LAN, but its local operator plane is not portable.**

### Off-LAN DNS note

At a foreign LAN, DHCP repoints the host's resolv.conf at the local resolver, so `gitea.dooplex.hu`
(pi-hole split-horizon) will not resolve. To pull a controller image while away, either turn on the
travel `--accept-routes` toggle *and* add a temporary `192.168.0.180 gitea.dooplex.hu` line to
`/etc/hosts`, or point DNS at `192.168.0.250` over the subnet route. External customer access is via
Cloudflare Tunnel and is unaffected.

## Vacation-day checklist (for the operator)

1. At the destination, connect the N100 by **wired ethernet**, power on, wait ~2 min.
2. From DooPlex: `tailscale`-side — check the pod: `sudo kubectl -n admin-system exec deploy/tailscale -- tailscale status` shows `felhom-pve` **online** (likely *relayed*/DERP at first — fine).
3. `ssh felhom-pve pveversion` — management reachable over the tailnet.
4. `ssh felhom-pve "pvesm status | grep -i pbs"` — **the real off-LAN PBS test** (could not be
   validated at home; PBS rides its own WireGuard tunnel, so it should be green).
5. Demo URL via Cloudflare Tunnel loads (customer-facing path, independent of all the above).
6. If you need a home-LAN service (gitea image pull): `ssh felhom-pve "tailscale set --accept-routes=true"`
   + `/etc/hosts` entry as above — then set it back to `false` before the box comes home.
7. Expect the controller↔agent plane to be **down** off-LAN until the pinned-`.162` blocker above is
   fixed.