From a7d983797680c8fe2fe56b171ca18d6eebf5e48a Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 20 Jul 2026 12:09:01 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20INCIDENT=20=E2=80=94=20guest=209201=20d?= =?UTF-8?q?hclient=20killed=20as=20a=20host=20orphan;=20tunnel=20down=20~1?= =?UTF-8?q?h15m?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v0.150.0 'orphaned dhclient' cleanup killed guest 9201's DHCP client, not a host one: LXC guests share the host PID namespace, so the missing eth0, pidfile and lease file were all true ON THE HOST and all irrelevant. Lease expired ~80 min later, guest lost its address, tunnel/hub/catalog/agent all went dark. Restored; records the one-command ownership check (/proc//cgroup) and leaves the unsupervised-dhclient risk open for a ruling with R-50. --- ...CIDENT-guest-dhclient-killed-2026-07-20.md | 157 ++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100644 documentation/audits/INCIDENT-guest-dhclient-killed-2026-07-20.md diff --git a/documentation/audits/INCIDENT-guest-dhclient-killed-2026-07-20.md b/documentation/audits/INCIDENT-guest-dhclient-killed-2026-07-20.md new file mode 100644 index 0000000..24a7d29 --- /dev/null +++ b/documentation/audits/INCIDENT-guest-dhclient-killed-2026-07-20.md @@ -0,0 +1,157 @@ +# INCIDENT — guest 9201 lost its LAN address; the Cloudflare tunnel went down (2026-07-20) + +**Severity:** HIGH — the demo box was off the internet for ~1h15m: tunnel down, all public URLs +dead, hub reporting dead, catalog sync dead, agent unreachable from the guest. +**Cause:** self-inflicted. A "cleanup" step in the v0.150.0 session killed what it believed was an +orphaned dhclient **on the host**. It was guest 9201's dhclient, visible in the host's PID namespace. +**Status:** RESOLVED — connectivity and tunnel restored 10:04 UTC. One durable risk left open (§5). + +Times are UTC unless marked CEST (the host runs CEST; guests and containers run UTC). + +--- + +## 1. What the operator saw + +Cloudflare dashboard: tunnel `demo-minipc` **Down**, zero connectors, uptime `--`. The mini-PC was +powered on with NIC LEDs blinking, so the box itself was clearly alive. + +## 2. Timeline + +| Time (UTC) | Event | +|---|---| +| 05:25:30 | Guest 9201 boots. `systemd-networkd` = guest pid 216, **`dhclient` = guest pid 222**, started in the same second. | +| 06:18:03 | `DHCPACK of 192.168.0.104`, renewal in 2610 s. | +| 07:01:34 | `DHCPACK of 192.168.0.104`, **renewal in 3063 s** → next renewal due ~07:52:37. | +| ~07:2x–07:5x | **The v0.150.0 session kills the process it identified as an orphaned host dhclient.** It was guest pid 222. | +| 07:52:37 | The renewal that should have happened. **Nothing in the guest journal — dhclient is gone.** | +| ~08:45 | Lease expires. `eth0` loses its address; the default route goes with it. | +| 08:47:26 | Controller v0.151.0 deploy — the last outbound work that succeeded, on the tail of the expiring lease. | +| 09:58–10:02 | cloudflared retry storm: `failed to dial to edge with quic: timeout`, then `Failed to refresh DNS local resolver: lookup region1.v2.argotunnel.com: i/o timeout`. | +| 10:02:26 | `[quiesce] … dial tcp 192.168.0.162:8443: connect: network is unreachable`; catalog sync `Could not resolve host: gitea.dooplex.hu`; hub push fails. | +| **10:04:3x** | **Fix applied** — dhclient restarted in the guest. `192.168.0.104` back, default route back, DNS back. | +| 10:04:41 | cloudflared registers 4 connections (bud01 ×2, vie05, +1). Tunnel **UP**. | + +## 3. Diagnosis — how the layers split + +The decisive observation was that the **host was completely healthy while the guest was not**, with +both using the same nameserver: + +``` +HOST getent hosts cloudflare.com → 2606:4700::6810:85e5 rc=0 + ping 192.168.0.1 0% loss · ping 1.1.1.1 0% loss · vmbr0 192.168.0.162/24 +GUEST getent hosts gitea.dooplex.hu → rc=2 (everything failed) + ping 192.168.0.1 → connect: Network is unreachable +``` + +That is not a DNS fault and not an ISP fault. `ip -4 addr show eth0` in the guest printed **nothing**, +and `ip route` had no default — only the four docker bridges: + +``` +172.17.0.0/16 dev docker0 … 172.18.0.0/16 dev br-0dc01a0f13cb … +172.19.0.0/16 dev br-5c561a9b0f80 … 172.20.0.0/16 dev br-6f074c0973c4 … +``` + +`eth0` was `UP` with `LOWER_UP` — the link was fine; the box simply had no IPv4 address. So: a DHCP +client problem, not a cable, switch or router problem. The guest journal then showed dhclient's log +stopping dead after 07:01:34 with no renewal at 07:52, and `ps` showed no dhclient running at all. + +## 4. Root cause — the host PID namespace shows guest processes + +The v0.150.0 session recorded this (REPORT §4, "the orphaned `dhclient`"): + +``` +BEFORE: 1922 /sbin/dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 + PPID 1652, started Mon Jul 20 07:25:30 (boot) +interfaces: lo enp1s0 wlp2s0 tailscale0 vmbr0 wg-felhom veth9201i0 ← no eth0 +vmbr0: iface vmbr0 inet static / address 192.168.0.162/24 ← no dhclient should run at all +neither /run/dhclient.eth0.pid nor /var/lib/dhcp/dhclient.eth0.leases existed +``` + +Every one of those observations was **true and misleading**, because each was made in the host's +namespaces about a process living in the guest's: + +* **"no eth0"** — correct for the host. `eth0` is the guest's interface; from the host the same link + is `veth9201i0@if2`. +* **"vmbr0 is static, so no dhclient should run"** — correct for the host, and irrelevant: this + dhclient was never configuring vmbr0. +* **"neither the pidfile nor the leases file existed"** — correct on the host's filesystem. Both + exist in the guest: `/var/lib/dhcp/dhclient.eth0.leases`, mtime `Jul 20 07:01`, the exact minute of + the last DHCPACK. +* **"started 07:25:30 (boot)"** — host-local CEST = **05:25:30 UTC**, which is the second guest pid + 216 (`systemd-networkd`) started. Its sibling pid 222 was the dhclient. The identification is exact. + +**The general fact:** guest 9201 is an unprivileged LXC container, and LXC containers share the +host's PID namespace view — `ps`/`pgrep` on a Proxmox host list every guest process. A process whose +interface, pidfile and lease file are all absent *on the host* is therefore **not evidence of an +orphan**; it is the normal appearance of a container process seen from outside. + +The kill was also verified afterwards in a way that reinforced the wrong conclusion: "re-checked +immediately and again ~25 min later: no respawn". Nothing respawned it because ifupdown had started +it once at guest boot and nothing supervises it — and the guest kept working for another ~80 minutes +on its unexpired lease, so the damage was invisible for the rest of that session. + +### The check that settles it in one command + +``` +$ cat /proc//cgroup +0::/lxc/9201/ns/.lxc ← the process belongs to guest 9201, not the host + +$ readlink /proc//ns/net net:[4026532482] +$ readlink /proc/1/ns/net net:[4026531833] ← different netns: not a host process +``` + +**Rule: on a Proxmox host, never kill a process by name/appearance without reading +`/proc//cgroup` first.** A `/lxc//` prefix means it is a guest's process and the host's +interface list, config files and filesystem say nothing about whether it is orphaned. + +## 5. Fix applied, and the risk left open + +Restored with the original invocation, in the guest: + +``` +pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 +→ inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0 +→ default via 192.168.0.1 dev eth0 +``` + +Verified after the fix: + +| Check | Result | +|---|---| +| cloudflared | 4 × `Registered tunnel connection` (bud01, bud01, vie05) at 10:04:41 | +| `https://felhom.demo-felhom.eu/` | **302** (0.22 s) — the login redirect, i.e. the controller answered through the tunnel | +| `https://photos.demo-felhom.eu/` | **200** (0.22 s) | +| hub from inside the controller container | `hub:302` | +| gitea from inside the controller container | `gitea:200` | +| agent local-API `192.168.0.162:8443` from the guest | **OPEN** (was `network is unreachable`) | +| dhclient | running, host pid 336708, cgroup `/lxc/9201/ns/.lxc` | + +**OPEN RISK — the guest's dhclient is unsupervised.** ifupdown starts it once at boot and nothing +restarts it, so any future death of that process repeats this outage silently ~1–2 h later, when the +lease expires. The blast radius is everything the box does outbound. Options, for a ruling rather +than a drive-by change: + +1. **A DHCP reservation for `bc:24:11:de:1b:e7` at the site router** — keeps DHCP but makes the + address predictable; does not fix the unsupervised-client problem. +2. **A static address in the guest** (`iface eth0 inet static`) — removes the dependency entirely, + but hard-codes a site-specific address into a guest that is supposed to be portable, which is + precisely the coupling R-50 exists to remove. +3. **Supervise it** — a systemd unit with `Restart=always` for the DHCP client, or hand `eth0` to + `systemd-networkd`, which is already running in the guest but currently manages nothing. + +Note that **the guest and the host now differ**: the host was flipped to a static `192.168.0.162` +this morning (F1 mitigation), while the guest is still DHCP. Whatever is chosen should be chosen for +both, and belongs with **R-50 (island-bridge)** rather than as another point fix. + +## 6. Lessons + +1. **A host `ps` on Proxmox is not a host-only view.** Guest processes appear there; the host's own + interface and file layout cannot tell you whether one is orphaned. `/proc//cgroup` can. +2. **"No respawn after 25 minutes" is not proof a kill was harmless** when the killed thing is a + lease renewer. The failure mode is timed: it appears when the lease expires, which here was ~80 + minutes later — after the session had ended and reported success. +3. **Cleanup steps deserve the same evidence bar as changes.** This was the only action in that + session with no test behind it, and it is the only one that caused an outage. +4. When a box is remote and the symptom is "everything outbound is dead", **compare host and guest + at the same layer first**. Host-healthy/guest-dead isolated this to the guest's own addressing in + two commands, before any cloudflared log was worth reading.