felhom.eu/documentation/audits/INCIDENT-guest-dhclient-killed-2026-07-20.md

INCIDENT — guest 9201 lost its LAN address; the Cloudflare tunnel went down (2026-07-20)

Severity: HIGH — the demo box was off the internet for ~1h15m: tunnel down, all public URLs dead, hub reporting dead, catalog sync dead, agent unreachable from the guest. Cause: self-inflicted. A "cleanup" step in the v0.150.0 session killed what it believed was an orphaned dhclient on the host. It was guest 9201's dhclient, visible in the host's PID namespace. Status: RESOLVED — connectivity and tunnel restored 10:04 UTC. One durable risk left open (§5).

Times are UTC unless marked CEST (the host runs CEST; guests and containers run UTC).

---

1. What the operator saw

Cloudflare dashboard: tunnel demo-minipc Down, zero connectors, uptime --. The mini-PC was powered on with NIC LEDs blinking, so the box itself was clearly alive.

2. Timeline

Time (UTC)	Event
05:25:30	Guest 9201 boots. systemd-networkd = guest pid 216, dhclient = guest pid 222, started in the same second.
06:18:03	DHCPACK of 192.168.0.104, renewal in 2610 s.
07:01:34	DHCPACK of 192.168.0.104, renewal in 3063 s → next renewal due ~07:52:37.
~07:2x–07:5x	The v0.150.0 session kills the process it identified as an orphaned host dhclient. It was guest pid 222.
07:52:37	The renewal that should have happened. Nothing in the guest journal — dhclient is gone.
~08:45	Lease expires. eth0 loses its address; the default route goes with it.
08:47:26	Controller v0.151.0 deploy — the last outbound work that succeeded, on the tail of the expiring lease.
09:58–10:02	cloudflared retry storm: failed to dial to edge with quic: timeout, then Failed to refresh DNS local resolver: lookup region1.v2.argotunnel.com: i/o timeout.
10:02:26	[quiesce] … dial tcp 192.168.0.162:8443: connect: network is unreachable; catalog sync Could not resolve host: gitea.dooplex.hu; hub push fails.
10:04:3x	Fix applied — dhclient restarted in the guest. 192.168.0.104 back, default route back, DNS back.
10:04:41	cloudflared registers 4 connections (bud01 ×2, vie05, +1). Tunnel UP.

3. Diagnosis — how the layers split

The decisive observation was that the host was completely healthy while the guest was not, with both using the same nameserver:

HOST   getent hosts cloudflare.com  → 2606:4700::6810:85e5    rc=0
       ping 192.168.0.1  0% loss · ping 1.1.1.1  0% loss · vmbr0 192.168.0.162/24
GUEST  getent hosts gitea.dooplex.hu → rc=2   (everything failed)
       ping 192.168.0.1 → connect: Network is unreachable

That is not a DNS fault and not an ISP fault. ip -4 addr show eth0 in the guest printed nothing, and ip route had no default — only the four docker bridges:

172.17.0.0/16 dev docker0 …      172.18.0.0/16 dev br-0dc01a0f13cb …
172.19.0.0/16 dev br-5c561a9b0f80 …   172.20.0.0/16 dev br-6f074c0973c4 …

eth0 was UP with LOWER_UP — the link was fine; the box simply had no IPv4 address. So: a DHCP client problem, not a cable, switch or router problem. The guest journal then showed dhclient's log stopping dead after 07:01:34 with no renewal at 07:52, and ps showed no dhclient running at all.

4. Root cause — the host PID namespace shows guest processes

The v0.150.0 session recorded this (REPORT §4, "the orphaned dhclient"):

BEFORE: 1922 /sbin/dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
        PPID 1652, started Mon Jul 20 07:25:30 (boot)
interfaces: lo enp1s0 wlp2s0 tailscale0 vmbr0 wg-felhom veth9201i0   ← no eth0
vmbr0: iface vmbr0 inet static / address 192.168.0.162/24            ← no dhclient should run at all
neither /run/dhclient.eth0.pid nor /var/lib/dhcp/dhclient.eth0.leases existed

Every one of those observations was true and misleading, because each was made in the host's namespaces about a process living in the guest's:

• "no eth0" — correct for the host. eth0 is the guest's interface; from the host the same link is veth9201i0@if2.
• "vmbr0 is static, so no dhclient should run" — correct for the host, and irrelevant: this dhclient was never configuring vmbr0.
• "neither the pidfile nor the leases file existed" — correct on the host's filesystem. Both exist in the guest: /var/lib/dhcp/dhclient.eth0.leases, mtime Jul 20 07:01, the exact minute of the last DHCPACK.
• "started 07:25:30 (boot)" — host-local CEST = 05:25:30 UTC, which is the second guest pid 216 (systemd-networkd) started. Its sibling pid 222 was the dhclient. The identification is exact.

The general fact: guest 9201 is an unprivileged LXC container, and LXC containers share the host's PID namespace view — ps/pgrep on a Proxmox host list every guest process. A process whose interface, pidfile and lease file are all absent on the host is therefore not evidence of an orphan; it is the normal appearance of a container process seen from outside.

The kill was also verified afterwards in a way that reinforced the wrong conclusion: "re-checked immediately and again ~25 min later: no respawn". Nothing respawned it because ifupdown had started it once at guest boot and nothing supervises it — and the guest kept working for another ~80 minutes on its unexpired lease, so the damage was invisible for the rest of that session.

The check that settles it in one command

$ cat /proc/<pid>/cgroup
0::/lxc/9201/ns/.lxc                    ← the process belongs to guest 9201, not the host

$ readlink /proc/<pid>/ns/net           net:[4026532482]
$ readlink /proc/1/ns/net               net:[4026531833]   ← different netns: not a host process

Rule: on a Proxmox host, never kill a process by name/appearance without reading /proc/<pid>/cgroup first. A /lxc/<vmid>/ prefix means it is a guest's process and the host's interface list, config files and filesystem say nothing about whether it is orphaned.

5. Fix applied, and the risk left open

Restored with the original invocation, in the guest:

pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
→ inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0
→ default via 192.168.0.1 dev eth0

Verified after the fix:

Check	Result
cloudflared	4 × Registered tunnel connection (bud01, bud01, vie05) at 10:04:41
https://felhom.demo-felhom.eu/	302 (0.22 s) — the login redirect, i.e. the controller answered through the tunnel
https://photos.demo-felhom.eu/	200 (0.22 s)
hub from inside the controller container	hub:302
gitea from inside the controller container	gitea:200
agent local-API 192.168.0.162:8443 from the guest	OPEN (was network is unreachable)
dhclient	running, host pid 336708, cgroup /lxc/9201/ns/.lxc

OPEN RISK — the guest's dhclient is unsupervised. ifupdown starts it once at boot and nothing restarts it, so any future death of that process repeats this outage silently ~1–2 h later, when the lease expires. The blast radius is everything the box does outbound. Options, for a ruling rather than a drive-by change:

1. A DHCP reservation for bc:24:11:de:1b:e7 at the site router — keeps DHCP but makes the address predictable; does not fix the unsupervised-client problem.
2. A static address in the guest (iface eth0 inet static) — removes the dependency entirely, but hard-codes a site-specific address into a guest that is supposed to be portable, which is precisely the coupling R-50 exists to remove.
3. Supervise it — a systemd unit with Restart=always for the DHCP client, or hand eth0 to systemd-networkd, which is already running in the guest but currently manages nothing.

Note that the guest and the host now differ: the host was flipped to a static 192.168.0.162 this morning (F1 mitigation), while the guest is still DHCP. Whatever is chosen should be chosen for both, and belongs with R-50 (island-bridge) rather than as another point fix.

6. Lessons

1. A host ps on Proxmox is not a host-only view. Guest processes appear there; the host's own interface and file layout cannot tell you whether one is orphaned. /proc/<pid>/cgroup can.
2. "No respawn after 25 minutes" is not proof a kill was harmless when the killed thing is a lease renewer. The failure mode is timed: it appears when the lease expires, which here was ~80 minutes later — after the session had ended and reported success.
3. Cleanup steps deserve the same evidence bar as changes. This was the only action in that session with no test behind it, and it is the only one that caused an outage.
4. When a box is remote and the symptom is "everything outbound is dead", compare host and guest at the same layer first. Host-healthy/guest-dead isolated this to the guest's own addressing in two commands, before any cloudflared log was worth reading.