admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: INCIDENT — guest 9201 dhclient killed as a host orphan; tunnel down ~1h15m

Browse Source 
The v0.150.0 'orphaned dhclient' cleanup killed guest 9201's DHCP client, not a
host one: LXC guests share the host PID namespace, so the missing eth0, pidfile
and lease file were all true ON THE HOST and all irrelevant. Lease expired ~80
min later, guest lost its address, tunnel/hub/catalog/agent all went dark.

Restored; records the one-command ownership check (/proc/<pid>/cgroup) and
leaves the unsupervised-dhclient risk open for a ruling with R-50.
This commit is contained in:
authentik Default Admin
2026-07-20 12:09:01 +02:00
parent eaa217373f
commit a7d9837976
1 changed files with 157 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/INCIDENT-guest-dhclient-killed-2026-07-20.md
+157
View File
@@ -0,0 +1,157 @@
# INCIDENT — guest 9201 lost its LAN address; the Cloudflare tunnel went down (2026-07-20)
**Severity:** HIGH — the demo box was off the internet for ~1h15m: tunnel down, all public URLs
dead, hub reporting dead, catalog sync dead, agent unreachable from the guest.
**Cause:** self-inflicted. A "cleanup" step in the v0.150.0 session killed what it believed was an
orphaned dhclient **on the host**. It was guest 9201's dhclient, visible in the host's PID namespace.
**Status:** RESOLVED — connectivity and tunnel restored 10:04 UTC. One durable risk left open (§5).
Times are UTC unless marked CEST (the host runs CEST; guests and containers run UTC).
---
## 1. What the operator saw
Cloudflare dashboard: tunnel `demo-minipc` **Down**, zero connectors, uptime `--`. The mini-PC was
powered on with NIC LEDs blinking, so the box itself was clearly alive.
## 2. Timeline
| Time (UTC) | Event |
|---|---|
| 05:25:30 | Guest 9201 boots. `systemd-networkd` = guest pid 216, **`dhclient` = guest pid 222**, started in the same second. |
| 06:18:03 | `DHCPACK of 192.168.0.104`, renewal in 2610 s. |
| 07:01:34 | `DHCPACK of 192.168.0.104`, **renewal in 3063 s** → next renewal due ~07:52:37. |
| ~07:2x–07:5x | **The v0.150.0 session kills the process it identified as an orphaned host dhclient.** It was guest pid 222. |
| 07:52:37 | The renewal that should have happened. **Nothing in the guest journal — dhclient is gone.** |
| ~08:45 | Lease expires. `eth0` loses its address; the default route goes with it. |
| 08:47:26 | Controller v0.151.0 deploy — the last outbound work that succeeded, on the tail of the expiring lease. |
| 09:58–10:02 | cloudflared retry storm: `failed to dial to edge with quic: timeout`, then `Failed to refresh DNS local resolver: lookup region1.v2.argotunnel.com: i/o timeout`. |
| 10:02:26 | `[quiesce] … dial tcp 192.168.0.162:8443: connect: network is unreachable`; catalog sync `Could not resolve host: gitea.dooplex.hu`; hub push fails. |
| **10:04:3x** | **Fix applied** — dhclient restarted in the guest. `192.168.0.104` back, default route back, DNS back. |
| 10:04:41 | cloudflared registers 4 connections (bud01 ×2, vie05, +1). Tunnel **UP**. |
## 3. Diagnosis — how the layers split
The decisive observation was that the **host was completely healthy while the guest was not**, with
both using the same nameserver:
```
HOST getent hosts cloudflare.com → 2606:4700::6810:85e5 rc=0
ping 192.168.0.1 0% loss · ping 1.1.1.1 0% loss · vmbr0 192.168.0.162/24
GUEST getent hosts gitea.dooplex.hu → rc=2 (everything failed)
ping 192.168.0.1 → connect: Network is unreachable
```
That is not a DNS fault and not an ISP fault. `ip -4 addr show eth0` in the guest printed **nothing**,
and `ip route` had no default — only the four docker bridges:
```
172.17.0.0/16 dev docker0 … 172.18.0.0/16 dev br-0dc01a0f13cb …
172.19.0.0/16 dev br-5c561a9b0f80 … 172.20.0.0/16 dev br-6f074c0973c4 …
```
`eth0` was `UP` with `LOWER_UP` — the link was fine; the box simply had no IPv4 address. So: a DHCP
client problem, not a cable, switch or router problem. The guest journal then showed dhclient's log
stopping dead after 07:01:34 with no renewal at 07:52, and `ps` showed no dhclient running at all.
## 4. Root cause — the host PID namespace shows guest processes
The v0.150.0 session recorded this (REPORT §4, "the orphaned `dhclient`"):
```
BEFORE: 1922 /sbin/dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
PPID 1652, started Mon Jul 20 07:25:30 (boot)
interfaces: lo enp1s0 wlp2s0 tailscale0 vmbr0 wg-felhom veth9201i0 ← no eth0
vmbr0: iface vmbr0 inet static / address 192.168.0.162/24 ← no dhclient should run at all
neither /run/dhclient.eth0.pid nor /var/lib/dhcp/dhclient.eth0.leases existed
```
Every one of those observations was **true and misleading**, because each was made in the host's
namespaces about a process living in the guest's:
* **"no eth0"** — correct for the host. `eth0` is the guest's interface; from the host the same link
is `veth9201i0@if2`.
* **"vmbr0 is static, so no dhclient should run"** — correct for the host, and irrelevant: this
dhclient was never configuring vmbr0.
* **"neither the pidfile nor the leases file existed"** — correct on the host's filesystem. Both
exist in the guest: `/var/lib/dhcp/dhclient.eth0.leases`, mtime `Jul 20 07:01`, the exact minute of
the last DHCPACK.
* **"started 07:25:30 (boot)"** — host-local CEST = **05:25:30 UTC**, which is the second guest pid
216 (`systemd-networkd`) started. Its sibling pid 222 was the dhclient. The identification is exact.
**The general fact:** guest 9201 is an unprivileged LXC container, and LXC containers share the
host's PID namespace view — `ps`/`pgrep` on a Proxmox host list every guest process. A process whose
interface, pidfile and lease file are all absent *on the host* is therefore **not evidence of an
orphan**; it is the normal appearance of a container process seen from outside.
The kill was also verified afterwards in a way that reinforced the wrong conclusion: "re-checked
immediately and again ~25 min later: no respawn". Nothing respawned it because ifupdown had started
it once at guest boot and nothing supervises it — and the guest kept working for another ~80 minutes
on its unexpired lease, so the damage was invisible for the rest of that session.
### The check that settles it in one command
```
$ cat /proc/<pid>/cgroup
0::/lxc/9201/ns/.lxc ← the process belongs to guest 9201, not the host
$ readlink /proc/<pid>/ns/net net:[4026532482]
$ readlink /proc/1/ns/net net:[4026531833] ← different netns: not a host process
```
**Rule: on a Proxmox host, never kill a process by name/appearance without reading
`/proc/<pid>/cgroup` first.** A `/lxc/<vmid>/` prefix means it is a guest's process and the host's
interface list, config files and filesystem say nothing about whether it is orphaned.
## 5. Fix applied, and the risk left open
Restored with the original invocation, in the guest:
```
pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
→ inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0
→ default via 192.168.0.1 dev eth0
```
Verified after the fix:
| Check | Result |
|---|---|
| cloudflared | 4 × `Registered tunnel connection` (bud01, bud01, vie05) at 10:04:41 |
| `https://felhom.demo-felhom.eu/` | **302** (0.22 s) — the login redirect, i.e. the controller answered through the tunnel |
| `https://photos.demo-felhom.eu/` | **200** (0.22 s) |
| hub from inside the controller container | `hub:302` |
| gitea from inside the controller container | `gitea:200` |
| agent local-API `192.168.0.162:8443` from the guest | **OPEN** (was `network is unreachable`) |
| dhclient | running, host pid 336708, cgroup `/lxc/9201/ns/.lxc` |
**OPEN RISK — the guest's dhclient is unsupervised.** ifupdown starts it once at boot and nothing
restarts it, so any future death of that process repeats this outage silently ~1–2 h later, when the
lease expires. The blast radius is everything the box does outbound. Options, for a ruling rather
than a drive-by change:
1. **A DHCP reservation for `bc:24:11:de:1b:e7` at the site router** — keeps DHCP but makes the
address predictable; does not fix the unsupervised-client problem.
2. **A static address in the guest** (`iface eth0 inet static`) — removes the dependency entirely,
but hard-codes a site-specific address into a guest that is supposed to be portable, which is
precisely the coupling R-50 exists to remove.
3. **Supervise it** — a systemd unit with `Restart=always` for the DHCP client, or hand `eth0` to
`systemd-networkd`, which is already running in the guest but currently manages nothing.
Note that **the guest and the host now differ**: the host was flipped to a static `192.168.0.162`
this morning (F1 mitigation), while the guest is still DHCP. Whatever is chosen should be chosen for
both, and belongs with **R-50 (island-bridge)** rather than as another point fix.
## 6. Lessons
1. **A host `ps` on Proxmox is not a host-only view.** Guest processes appear there; the host's own
interface and file layout cannot tell you whether one is orphaned. `/proc/<pid>/cgroup` can.
2. **"No respawn after 25 minutes" is not proof a kill was harmless** when the killed thing is a
lease renewer. The failure mode is timed: it appears when the lease expires, which here was ~80
minutes later — after the session had ended and reported success.
3. **Cleanup steps deserve the same evidence bar as changes.** This was the only action in that
session with no test behind it, and it is the only one that caused an outage.
4. When a box is remote and the symptom is "everything outbound is dead", **compare host and guest
at the same layer first**. Host-healthy/guest-dead isolated this to the guest's own addressing in
two commands, before any cloudflared log was worth reading.