# INCIDENT — guest 9201 lost its LAN address; the Cloudflare tunnel went down (2026-07-20) **Severity:** HIGH — the demo box was off the internet for ~1h15m: tunnel down, all public URLs dead, hub reporting dead, catalog sync dead, agent unreachable from the guest. **Cause:** self-inflicted. A "cleanup" step in the v0.150.0 session killed what it believed was an orphaned dhclient **on the host**. It was guest 9201's dhclient, visible in the host's PID namespace. **Status:** RESOLVED — connectivity and tunnel restored 10:04 UTC. One durable risk left open (§5). Times are UTC unless marked CEST (the host runs CEST; guests and containers run UTC). --- ## 1. What the operator saw Cloudflare dashboard: tunnel `demo-minipc` **Down**, zero connectors, uptime `--`. The mini-PC was powered on with NIC LEDs blinking, so the box itself was clearly alive. ## 2. Timeline | Time (UTC) | Event | |---|---| | 05:25:30 | Guest 9201 boots. `systemd-networkd` = guest pid 216, **`dhclient` = guest pid 222**, started in the same second. | | 06:18:03 | `DHCPACK of 192.168.0.104`, renewal in 2610 s. | | 07:01:34 | `DHCPACK of 192.168.0.104`, **renewal in 3063 s** → next renewal due ~07:52:37. | | ~07:2x–07:5x | **The v0.150.0 session kills the process it identified as an orphaned host dhclient.** It was guest pid 222. | | 07:52:37 | The renewal that should have happened. **Nothing in the guest journal — dhclient is gone.** | | ~08:45 | Lease expires. `eth0` loses its address; the default route goes with it. | | 08:47:26 | Controller v0.151.0 deploy — the last outbound work that succeeded, on the tail of the expiring lease. | | 09:58–10:02 | cloudflared retry storm: `failed to dial to edge with quic: timeout`, then `Failed to refresh DNS local resolver: lookup region1.v2.argotunnel.com: i/o timeout`. | | 10:02:26 | `[quiesce] … dial tcp 192.168.0.162:8443: connect: network is unreachable`; catalog sync `Could not resolve host: gitea.dooplex.hu`; hub push fails. | | **10:04:3x** | **Fix applied** — dhclient restarted in the guest. `192.168.0.104` back, default route back, DNS back. | | 10:04:41 | cloudflared registers 4 connections (bud01 ×2, vie05, +1). Tunnel **UP**. | ## 3. Diagnosis — how the layers split The decisive observation was that the **host was completely healthy while the guest was not**, with both using the same nameserver: ``` HOST getent hosts cloudflare.com → 2606:4700::6810:85e5 rc=0 ping 192.168.0.1 0% loss · ping 1.1.1.1 0% loss · vmbr0 192.168.0.162/24 GUEST getent hosts gitea.dooplex.hu → rc=2 (everything failed) ping 192.168.0.1 → connect: Network is unreachable ``` That is not a DNS fault and not an ISP fault. `ip -4 addr show eth0` in the guest printed **nothing**, and `ip route` had no default — only the four docker bridges: ``` 172.17.0.0/16 dev docker0 … 172.18.0.0/16 dev br-0dc01a0f13cb … 172.19.0.0/16 dev br-5c561a9b0f80 … 172.20.0.0/16 dev br-6f074c0973c4 … ``` `eth0` was `UP` with `LOWER_UP` — the link was fine; the box simply had no IPv4 address. So: a DHCP client problem, not a cable, switch or router problem. The guest journal then showed dhclient's log stopping dead after 07:01:34 with no renewal at 07:52, and `ps` showed no dhclient running at all. ## 4. Root cause — the host PID namespace shows guest processes The v0.150.0 session recorded this (REPORT §4, "the orphaned `dhclient`"): ``` BEFORE: 1922 /sbin/dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 PPID 1652, started Mon Jul 20 07:25:30 (boot) interfaces: lo enp1s0 wlp2s0 tailscale0 vmbr0 wg-felhom veth9201i0 ← no eth0 vmbr0: iface vmbr0 inet static / address 192.168.0.162/24 ← no dhclient should run at all neither /run/dhclient.eth0.pid nor /var/lib/dhcp/dhclient.eth0.leases existed ``` Every one of those observations was **true and misleading**, because each was made in the host's namespaces about a process living in the guest's: * **"no eth0"** — correct for the host. `eth0` is the guest's interface; from the host the same link is `veth9201i0@if2`. * **"vmbr0 is static, so no dhclient should run"** — correct for the host, and irrelevant: this dhclient was never configuring vmbr0. * **"neither the pidfile nor the leases file existed"** — correct on the host's filesystem. Both exist in the guest: `/var/lib/dhcp/dhclient.eth0.leases`, mtime `Jul 20 07:01`, the exact minute of the last DHCPACK. * **"started 07:25:30 (boot)"** — host-local CEST = **05:25:30 UTC**, which is the second guest pid 216 (`systemd-networkd`) started. Its sibling pid 222 was the dhclient. The identification is exact. **The general fact:** guest 9201 is an unprivileged LXC container, and LXC containers share the host's PID namespace view — `ps`/`pgrep` on a Proxmox host list every guest process. A process whose interface, pidfile and lease file are all absent *on the host* is therefore **not evidence of an orphan**; it is the normal appearance of a container process seen from outside. The kill was also verified afterwards in a way that reinforced the wrong conclusion: "re-checked immediately and again ~25 min later: no respawn". Nothing respawned it because ifupdown had started it once at guest boot and nothing supervises it — and the guest kept working for another ~80 minutes on its unexpired lease, so the damage was invisible for the rest of that session. ### The check that settles it in one command ``` $ cat /proc//cgroup 0::/lxc/9201/ns/.lxc ← the process belongs to guest 9201, not the host $ readlink /proc//ns/net net:[4026532482] $ readlink /proc/1/ns/net net:[4026531833] ← different netns: not a host process ``` **Rule: on a Proxmox host, never kill a process by name/appearance without reading `/proc//cgroup` first.** A `/lxc//` prefix means it is a guest's process and the host's interface list, config files and filesystem say nothing about whether it is orphaned. ## 5. Fix applied, and the risk left open Restored with the original invocation, in the guest: ``` pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 → inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0 → default via 192.168.0.1 dev eth0 ``` Verified after the fix: | Check | Result | |---|---| | cloudflared | 4 × `Registered tunnel connection` (bud01, bud01, vie05) at 10:04:41 | | `https://felhom.demo-felhom.eu/` | **302** (0.22 s) — the login redirect, i.e. the controller answered through the tunnel | | `https://photos.demo-felhom.eu/` | **200** (0.22 s) | | hub from inside the controller container | `hub:302` | | gitea from inside the controller container | `gitea:200` | | agent local-API `192.168.0.162:8443` from the guest | **OPEN** (was `network is unreachable`) | | dhclient | running, host pid 336708, cgroup `/lxc/9201/ns/.lxc` | **OPEN RISK — the guest's dhclient is unsupervised.** ifupdown starts it once at boot and nothing restarts it, so any future death of that process repeats this outage silently ~1–2 h later, when the lease expires. The blast radius is everything the box does outbound. Options, for a ruling rather than a drive-by change: 1. **A DHCP reservation for `bc:24:11:de:1b:e7` at the site router** — keeps DHCP but makes the address predictable; does not fix the unsupervised-client problem. 2. **A static address in the guest** (`iface eth0 inet static`) — removes the dependency entirely, but hard-codes a site-specific address into a guest that is supposed to be portable, which is precisely the coupling R-50 exists to remove. 3. **Supervise it** — a systemd unit with `Restart=always` for the DHCP client, or hand `eth0` to `systemd-networkd`, which is already running in the guest but currently manages nothing. Note that **the guest and the host now differ**: the host was flipped to a static `192.168.0.162` this morning (F1 mitigation), while the guest is still DHCP. Whatever is chosen should be chosen for both, and belongs with **R-50 (island-bridge)** rather than as another point fix. ## 6. Lessons 1. **A host `ps` on Proxmox is not a host-only view.** Guest processes appear there; the host's own interface and file layout cannot tell you whether one is orphaned. `/proc//cgroup` can. 2. **"No respawn after 25 minutes" is not proof a kill was harmless** when the killed thing is a lease renewer. The failure mode is timed: it appears when the lease expires, which here was ~80 minutes later — after the session had ended and reported success. 3. **Cleanup steps deserve the same evidence bar as changes.** This was the only action in that session with no test behind it, and it is the only one that caused an outage. 4. When a box is remote and the symptom is "everything outbound is dead", **compare host and guest at the same layer first**. Host-healthy/guest-dead isolated this to the guest's own addressing in two commands, before any cloudflared log was worth reading.