admin
a7d9837976
The v0.150.0 'orphaned dhclient' cleanup killed guest 9201's DHCP client, not a host one: LXC guests share the host PID namespace, so the missing eth0, pidfile and lease file were all true ON THE HOST and all irrelevant. Lease expired ~80 min later, guest lost its address, tunnel/hub/catalog/agent all went dark. Restored; records the one-command ownership check (/proc/<pid>/cgroup) and leaves the unsupervised-dhclient risk open for a ruling with R-50.
158 lines
8.9 KiB
Markdown
158 lines
8.9 KiB
Markdown
# INCIDENT — guest 9201 lost its LAN address; the Cloudflare tunnel went down (2026-07-20)
|
||
|
||
**Severity:** HIGH — the demo box was off the internet for ~1h15m: tunnel down, all public URLs
|
||
dead, hub reporting dead, catalog sync dead, agent unreachable from the guest.
|
||
**Cause:** self-inflicted. A "cleanup" step in the v0.150.0 session killed what it believed was an
|
||
orphaned dhclient **on the host**. It was guest 9201's dhclient, visible in the host's PID namespace.
|
||
**Status:** RESOLVED — connectivity and tunnel restored 10:04 UTC. One durable risk left open (§5).
|
||
|
||
Times are UTC unless marked CEST (the host runs CEST; guests and containers run UTC).
|
||
|
||
---
|
||
|
||
## 1. What the operator saw
|
||
|
||
Cloudflare dashboard: tunnel `demo-minipc` **Down**, zero connectors, uptime `--`. The mini-PC was
|
||
powered on with NIC LEDs blinking, so the box itself was clearly alive.
|
||
|
||
## 2. Timeline
|
||
|
||
| Time (UTC) | Event |
|
||
|---|---|
|
||
| 05:25:30 | Guest 9201 boots. `systemd-networkd` = guest pid 216, **`dhclient` = guest pid 222**, started in the same second. |
|
||
| 06:18:03 | `DHCPACK of 192.168.0.104`, renewal in 2610 s. |
|
||
| 07:01:34 | `DHCPACK of 192.168.0.104`, **renewal in 3063 s** → next renewal due ~07:52:37. |
|
||
| ~07:2x–07:5x | **The v0.150.0 session kills the process it identified as an orphaned host dhclient.** It was guest pid 222. |
|
||
| 07:52:37 | The renewal that should have happened. **Nothing in the guest journal — dhclient is gone.** |
|
||
| ~08:45 | Lease expires. `eth0` loses its address; the default route goes with it. |
|
||
| 08:47:26 | Controller v0.151.0 deploy — the last outbound work that succeeded, on the tail of the expiring lease. |
|
||
| 09:58–10:02 | cloudflared retry storm: `failed to dial to edge with quic: timeout`, then `Failed to refresh DNS local resolver: lookup region1.v2.argotunnel.com: i/o timeout`. |
|
||
| 10:02:26 | `[quiesce] … dial tcp 192.168.0.162:8443: connect: network is unreachable`; catalog sync `Could not resolve host: gitea.dooplex.hu`; hub push fails. |
|
||
| **10:04:3x** | **Fix applied** — dhclient restarted in the guest. `192.168.0.104` back, default route back, DNS back. |
|
||
| 10:04:41 | cloudflared registers 4 connections (bud01 ×2, vie05, +1). Tunnel **UP**. |
|
||
|
||
## 3. Diagnosis — how the layers split
|
||
|
||
The decisive observation was that the **host was completely healthy while the guest was not**, with
|
||
both using the same nameserver:
|
||
|
||
```
|
||
HOST getent hosts cloudflare.com → 2606:4700::6810:85e5 rc=0
|
||
ping 192.168.0.1 0% loss · ping 1.1.1.1 0% loss · vmbr0 192.168.0.162/24
|
||
GUEST getent hosts gitea.dooplex.hu → rc=2 (everything failed)
|
||
ping 192.168.0.1 → connect: Network is unreachable
|
||
```
|
||
|
||
That is not a DNS fault and not an ISP fault. `ip -4 addr show eth0` in the guest printed **nothing**,
|
||
and `ip route` had no default — only the four docker bridges:
|
||
|
||
```
|
||
172.17.0.0/16 dev docker0 … 172.18.0.0/16 dev br-0dc01a0f13cb …
|
||
172.19.0.0/16 dev br-5c561a9b0f80 … 172.20.0.0/16 dev br-6f074c0973c4 …
|
||
```
|
||
|
||
`eth0` was `UP` with `LOWER_UP` — the link was fine; the box simply had no IPv4 address. So: a DHCP
|
||
client problem, not a cable, switch or router problem. The guest journal then showed dhclient's log
|
||
stopping dead after 07:01:34 with no renewal at 07:52, and `ps` showed no dhclient running at all.
|
||
|
||
## 4. Root cause — the host PID namespace shows guest processes
|
||
|
||
The v0.150.0 session recorded this (REPORT §4, "the orphaned `dhclient`"):
|
||
|
||
```
|
||
BEFORE: 1922 /sbin/dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
|
||
PPID 1652, started Mon Jul 20 07:25:30 (boot)
|
||
interfaces: lo enp1s0 wlp2s0 tailscale0 vmbr0 wg-felhom veth9201i0 ← no eth0
|
||
vmbr0: iface vmbr0 inet static / address 192.168.0.162/24 ← no dhclient should run at all
|
||
neither /run/dhclient.eth0.pid nor /var/lib/dhcp/dhclient.eth0.leases existed
|
||
```
|
||
|
||
Every one of those observations was **true and misleading**, because each was made in the host's
|
||
namespaces about a process living in the guest's:
|
||
|
||
* **"no eth0"** — correct for the host. `eth0` is the guest's interface; from the host the same link
|
||
is `veth9201i0@if2`.
|
||
* **"vmbr0 is static, so no dhclient should run"** — correct for the host, and irrelevant: this
|
||
dhclient was never configuring vmbr0.
|
||
* **"neither the pidfile nor the leases file existed"** — correct on the host's filesystem. Both
|
||
exist in the guest: `/var/lib/dhcp/dhclient.eth0.leases`, mtime `Jul 20 07:01`, the exact minute of
|
||
the last DHCPACK.
|
||
* **"started 07:25:30 (boot)"** — host-local CEST = **05:25:30 UTC**, which is the second guest pid
|
||
216 (`systemd-networkd`) started. Its sibling pid 222 was the dhclient. The identification is exact.
|
||
|
||
**The general fact:** guest 9201 is an unprivileged LXC container, and LXC containers share the
|
||
host's PID namespace view — `ps`/`pgrep` on a Proxmox host list every guest process. A process whose
|
||
interface, pidfile and lease file are all absent *on the host* is therefore **not evidence of an
|
||
orphan**; it is the normal appearance of a container process seen from outside.
|
||
|
||
The kill was also verified afterwards in a way that reinforced the wrong conclusion: "re-checked
|
||
immediately and again ~25 min later: no respawn". Nothing respawned it because ifupdown had started
|
||
it once at guest boot and nothing supervises it — and the guest kept working for another ~80 minutes
|
||
on its unexpired lease, so the damage was invisible for the rest of that session.
|
||
|
||
### The check that settles it in one command
|
||
|
||
```
|
||
$ cat /proc/<pid>/cgroup
|
||
0::/lxc/9201/ns/.lxc ← the process belongs to guest 9201, not the host
|
||
|
||
$ readlink /proc/<pid>/ns/net net:[4026532482]
|
||
$ readlink /proc/1/ns/net net:[4026531833] ← different netns: not a host process
|
||
```
|
||
|
||
**Rule: on a Proxmox host, never kill a process by name/appearance without reading
|
||
`/proc/<pid>/cgroup` first.** A `/lxc/<vmid>/` prefix means it is a guest's process and the host's
|
||
interface list, config files and filesystem say nothing about whether it is orphaned.
|
||
|
||
## 5. Fix applied, and the risk left open
|
||
|
||
Restored with the original invocation, in the guest:
|
||
|
||
```
|
||
pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
|
||
→ inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0
|
||
→ default via 192.168.0.1 dev eth0
|
||
```
|
||
|
||
Verified after the fix:
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| cloudflared | 4 × `Registered tunnel connection` (bud01, bud01, vie05) at 10:04:41 |
|
||
| `https://felhom.demo-felhom.eu/` | **302** (0.22 s) — the login redirect, i.e. the controller answered through the tunnel |
|
||
| `https://photos.demo-felhom.eu/` | **200** (0.22 s) |
|
||
| hub from inside the controller container | `hub:302` |
|
||
| gitea from inside the controller container | `gitea:200` |
|
||
| agent local-API `192.168.0.162:8443` from the guest | **OPEN** (was `network is unreachable`) |
|
||
| dhclient | running, host pid 336708, cgroup `/lxc/9201/ns/.lxc` |
|
||
|
||
**OPEN RISK — the guest's dhclient is unsupervised.** ifupdown starts it once at boot and nothing
|
||
restarts it, so any future death of that process repeats this outage silently ~1–2 h later, when the
|
||
lease expires. The blast radius is everything the box does outbound. Options, for a ruling rather
|
||
than a drive-by change:
|
||
|
||
1. **A DHCP reservation for `bc:24:11:de:1b:e7` at the site router** — keeps DHCP but makes the
|
||
address predictable; does not fix the unsupervised-client problem.
|
||
2. **A static address in the guest** (`iface eth0 inet static`) — removes the dependency entirely,
|
||
but hard-codes a site-specific address into a guest that is supposed to be portable, which is
|
||
precisely the coupling R-50 exists to remove.
|
||
3. **Supervise it** — a systemd unit with `Restart=always` for the DHCP client, or hand `eth0` to
|
||
`systemd-networkd`, which is already running in the guest but currently manages nothing.
|
||
|
||
Note that **the guest and the host now differ**: the host was flipped to a static `192.168.0.162`
|
||
this morning (F1 mitigation), while the guest is still DHCP. Whatever is chosen should be chosen for
|
||
both, and belongs with **R-50 (island-bridge)** rather than as another point fix.
|
||
|
||
## 6. Lessons
|
||
|
||
1. **A host `ps` on Proxmox is not a host-only view.** Guest processes appear there; the host's own
|
||
interface and file layout cannot tell you whether one is orphaned. `/proc/<pid>/cgroup` can.
|
||
2. **"No respawn after 25 minutes" is not proof a kill was harmless** when the killed thing is a
|
||
lease renewer. The failure mode is timed: it appears when the lease expires, which here was ~80
|
||
minutes later — after the session had ended and reported success.
|
||
3. **Cleanup steps deserve the same evidence bar as changes.** This was the only action in that
|
||
session with no test behind it, and it is the only one that caused an outage.
|
||
4. When a box is remote and the symptom is "everything outbound is dead", **compare host and guest
|
||
at the same layer first**. Host-healthy/guest-dead isolated this to the guest's own addressing in
|
||
two commands, before any cloudflared log was worth reading.
|