# INCIDENT — guest 9201 lost its LAN address; the Cloudflare tunnel went down (2026-07-20)

**Severity:** HIGH — the demo box was off the internet for ~1h15m: tunnel down, all public URLs
dead, hub reporting dead, catalog sync dead, agent unreachable from the guest.
**Cause:** self-inflicted. A "cleanup" step in the v0.150.0 session killed what it believed was an
orphaned dhclient **on the host**. It was guest 9201's dhclient, visible in the host's PID namespace.
**Status:** RESOLVED — connectivity and tunnel restored 10:04 UTC. One durable risk left open (§5).

Times are UTC unless marked CEST (the host runs CEST; guests and containers run UTC).

---

## 1. What the operator saw

Cloudflare dashboard: tunnel `demo-minipc` **Down**, zero connectors, uptime `--`. The mini-PC was
powered on with NIC LEDs blinking, so the box itself was clearly alive.

## 2. Timeline

| Time (UTC) | Event |
|---|---|
| 05:25:30 | Guest 9201 boots. `systemd-networkd` = guest pid 216, **`dhclient` = guest pid 222**, started in the same second. |
| 06:18:03 | `DHCPACK of 192.168.0.104`, renewal in 2610 s. |
| 07:01:34 | `DHCPACK of 192.168.0.104`, **renewal in 3063 s** → next renewal due ~07:52:37. |
| ~07:2x–07:5x | **The v0.150.0 session kills the process it identified as an orphaned host dhclient.** It was guest pid 222. |
| 07:52:37 | The renewal that should have happened. **Nothing in the guest journal — dhclient is gone.** |
| ~08:45 | Lease expires. `eth0` loses its address; the default route goes with it. |
| 08:47:26 | Controller v0.151.0 deploy — the last outbound work that succeeded, on the tail of the expiring lease. |
| 09:58–10:02 | cloudflared retry storm: `failed to dial to edge with quic: timeout`, then `Failed to refresh DNS local resolver: lookup region1.v2.argotunnel.com: i/o timeout`. |
| 10:02:26 | `[quiesce] … dial tcp 192.168.0.162:8443: connect: network is unreachable`; catalog sync `Could not resolve host: gitea.dooplex.hu`; hub push fails. |
| **10:04:3x** | **Fix applied** — dhclient restarted in the guest. `192.168.0.104` back, default route back, DNS back. |
| 10:04:41 | cloudflared registers 4 connections (bud01 ×2, vie05, +1). Tunnel **UP**. |

## 3. Diagnosis — how the layers split

The decisive observation was that the **host was completely healthy while the guest was not**, with
both using the same nameserver:

```
HOST   getent hosts cloudflare.com  → 2606:4700::6810:85e5    rc=0
       ping 192.168.0.1  0% loss · ping 1.1.1.1  0% loss · vmbr0 192.168.0.162/24
GUEST  getent hosts gitea.dooplex.hu → rc=2   (everything failed)
       ping 192.168.0.1 → connect: Network is unreachable
```

That is not a DNS fault and not an ISP fault. `ip -4 addr show eth0` in the guest printed **nothing**,
and `ip route` had no default — only the four docker bridges:

```
172.17.0.0/16 dev docker0 …      172.18.0.0/16 dev br-0dc01a0f13cb …
172.19.0.0/16 dev br-5c561a9b0f80 …   172.20.0.0/16 dev br-6f074c0973c4 …
```

`eth0` was `UP` with `LOWER_UP` — the link was fine; the box simply had no IPv4 address. So: a DHCP
client problem, not a cable, switch or router problem. The guest journal then showed dhclient's log
stopping dead after 07:01:34 with no renewal at 07:52, and `ps` showed no dhclient running at all.

## 4. Root cause — the host PID namespace shows guest processes

The v0.150.0 session recorded this (REPORT §4, "the orphaned `dhclient`"):

```
BEFORE: 1922 /sbin/dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
        PPID 1652, started Mon Jul 20 07:25:30 (boot)
interfaces: lo enp1s0 wlp2s0 tailscale0 vmbr0 wg-felhom veth9201i0   ← no eth0
vmbr0: iface vmbr0 inet static / address 192.168.0.162/24            ← no dhclient should run at all
neither /run/dhclient.eth0.pid nor /var/lib/dhcp/dhclient.eth0.leases existed
```

Every one of those observations was **true and misleading**, because each was made in the host's
namespaces about a process living in the guest's:

* **"no eth0"** — correct for the host. `eth0` is the guest's interface; from the host the same link
  is `veth9201i0@if2`.
* **"vmbr0 is static, so no dhclient should run"** — correct for the host, and irrelevant: this
  dhclient was never configuring vmbr0.
* **"neither the pidfile nor the leases file existed"** — correct on the host's filesystem. Both
  exist in the guest: `/var/lib/dhcp/dhclient.eth0.leases`, mtime `Jul 20 07:01`, the exact minute of
  the last DHCPACK.
* **"started 07:25:30 (boot)"** — host-local CEST = **05:25:30 UTC**, which is the second guest pid
  216 (`systemd-networkd`) started. Its sibling pid 222 was the dhclient. The identification is exact.

**The general fact:** guest 9201 is an unprivileged LXC container, and LXC containers share the
host's PID namespace view — `ps`/`pgrep` on a Proxmox host list every guest process. A process whose
interface, pidfile and lease file are all absent *on the host* is therefore **not evidence of an
orphan**; it is the normal appearance of a container process seen from outside.

The kill was also verified afterwards in a way that reinforced the wrong conclusion: "re-checked
immediately and again ~25 min later: no respawn". Nothing respawned it because ifupdown had started
it once at guest boot and nothing supervises it — and the guest kept working for another ~80 minutes
on its unexpired lease, so the damage was invisible for the rest of that session.

### The check that settles it in one command

```
$ cat /proc/<pid>/cgroup
0::/lxc/9201/ns/.lxc                    ← the process belongs to guest 9201, not the host

$ readlink /proc/<pid>/ns/net           net:[4026532482]
$ readlink /proc/1/ns/net               net:[4026531833]   ← different netns: not a host process
```

**Rule: on a Proxmox host, never kill a process by name/appearance without reading
`/proc/<pid>/cgroup` first.** A `/lxc/<vmid>/` prefix means it is a guest's process and the host's
interface list, config files and filesystem say nothing about whether it is orphaned.

## 5. Fix applied, and the risk left open

Restored with the original invocation, in the guest:

```
pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
→ inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0
→ default via 192.168.0.1 dev eth0
```

Verified after the fix:

| Check | Result |
|---|---|
| cloudflared | 4 × `Registered tunnel connection` (bud01, bud01, vie05) at 10:04:41 |
| `https://felhom.demo-felhom.eu/` | **302** (0.22 s) — the login redirect, i.e. the controller answered through the tunnel |
| `https://photos.demo-felhom.eu/` | **200** (0.22 s) |
| hub from inside the controller container | `hub:302` |
| gitea from inside the controller container | `gitea:200` |
| agent local-API `192.168.0.162:8443` from the guest | **OPEN** (was `network is unreachable`) |
| dhclient | running, host pid 336708, cgroup `/lxc/9201/ns/.lxc` |

**OPEN RISK — the guest's dhclient is unsupervised.** ifupdown starts it once at boot and nothing
restarts it, so any future death of that process repeats this outage silently ~1–2 h later, when the
lease expires. The blast radius is everything the box does outbound. Options, for a ruling rather
than a drive-by change:

1. **A DHCP reservation for `bc:24:11:de:1b:e7` at the site router** — keeps DHCP but makes the
   address predictable; does not fix the unsupervised-client problem.
2. **A static address in the guest** (`iface eth0 inet static`) — removes the dependency entirely,
   but hard-codes a site-specific address into a guest that is supposed to be portable, which is
   precisely the coupling R-50 exists to remove.
3. **Supervise it** — a systemd unit with `Restart=always` for the DHCP client, or hand `eth0` to
   `systemd-networkd`, which is already running in the guest but currently manages nothing.

Note that **the guest and the host now differ**: the host was flipped to a static `192.168.0.162`
this morning (F1 mitigation), while the guest is still DHCP. Whatever is chosen should be chosen for
both, and belongs with **R-50 (island-bridge)** rather than as another point fix.

## 6. Lessons

1. **A host `ps` on Proxmox is not a host-only view.** Guest processes appear there; the host's own
   interface and file layout cannot tell you whether one is orphaned. `/proc/<pid>/cgroup` can.
2. **"No respawn after 25 minutes" is not proof a kill was harmless** when the killed thing is a
   lease renewer. The failure mode is timed: it appears when the lease expires, which here was ~80
   minutes later — after the session had ended and reported success.
3. **Cleanup steps deserve the same evidence bar as changes.** This was the only action in that
   session with no test behind it, and it is the only one that caused an outage.
4. When a box is remote and the symptom is "everything outbound is dead", **compare host and guest
   at the same layer first**. Host-healthy/guest-dead isolated this to the guest's own addressing in
   two commands, before any cloudflared log was worth reading.
