feat(v0.92.0): guest-network watchdog (R-54) — supervise the guest's DHCP client

Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).

So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.

internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.

guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.

Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.

Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.

Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
This commit is contained in:
2026-07-21 12:29:37 +02:00
parent 08b55a1015
commit c0966d753d
12 changed files with 1575 additions and 1 deletions
+63
View File
@@ -1,3 +1,66 @@
## v0.92.0 — the guest network gets a watchdog (R-54) (2026-07-21)
**Host-tier only — no controller coupling, no wire change the hub must understand today** (the
`guest_net` stanza is additive and stored opaquely, exactly like `pbs_dr` and `wireguard`).
Closes the OPEN RISK left by `INCIDENT-guest-dhclient-killed-2026-07-20` §5: **the guest's DHCP
client is started once by ifupdown at boot and nothing supervises it.** When it died on 2026-07-20
the guest kept working for another ~80 minutes on its unexpired lease; only when the lease expired
did the address and the default route vanish, taking the Cloudflare tunnel, the hub reports, the
catalog sync and the controller→agent channel with them — a 1h15m outage in which every observable
signal said healthy for the first 80 minutes.
**The design consequence, and the point of the whole package: liveness of the DHCP client is itself
a probe.** Waiting for the address to disappear is waiting out precisely that silent window. The
watchdog therefore flags a DHCP guest unhealthy on `pgrep -x dhclient` alone, while the lease is
still live and everything else still looks perfect.
`internal/guestnet`, built on the wg-tunnel/storage watchdog loop shape:
- **Probes** (four fixed-shape `pct exec` argvs, no shell anywhere, no guest data interpolated):
address, default route, `/etc/network/interfaces` mode, dhclient liveness. Parsers are pinned to
output captured live from guest 9201 on 2026-07-21 — including the literal backslash `ip -o`
emits and the docker-bridge routes that must not read as a default route.
- **Heals** with the incident's restored invocation, verbatim, logged at INFO before it runs:
`pct exec <vmid> -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0`
- **Dampers**, because this runs a privileged command inside a customer's container: two CONSECUTIVE
bad probes before any heal (one blip is not a diagnosis), ≥10 min between heals per guest, ≤3
heals/hour, and an observe-only window while the guest (or the agent) has been up under 3 minutes.
- **Refuses to act** on a static guest (dhclient must never fight a static config — a static guest
missing its address is reported loudly and left to R-50), on an unknown interface mode, on a guest
it cannot probe, and when the guest list cannot be ownership-proven. The guest source is the
pool-verified one (`ListLXC` ∩ the felhom pool, audit A1) — never a bare `ListLXC`, which under a
broad token would run dhclient inside a co-tenant's container.
- **A failed PROBE never reads as a dead client.** `pgrep` exits 1 with empty stderr when there is
no match; anything on stderr means the probe itself failed, which reads as unknown. Otherwise a
missing `pgrep` would heal forever.
- **Healthy cycles log a Debug line** (v0.91.2's lesson, one day old): if the quiet path is silent,
"no alarms" and "never probed" are the same evidence, and an inert watchdog is indistinguishable
from a working one.
- **Deliberately NOT in the `errc` fan-out** — a watchdog over customer guests must never be able to
terminate the agent. A test asserts that, because joining the fan-out would also make the shutdown
drain bound off by one.
**Config `guest_net` is this repo's first DEFAULT-ON feature gate, and the inversion is deliberate.**
Every other gate defaults to false because those features reach outward (an offsite endpoint, an OOB
tunnel) and enrolling a box by an update would be wrong. This one looks only INWARD at guests the
agent already owns, and the failure it prevents exists on every box today. A watchdog that must be
remembered per box is a watchdog that is missing on the box that needed it. Opting out is the
explicit act: `"guest_net": {"disable": true}`.
**Naming deviation from the spec, deliberate:** TASK-D called the report block `WireGuestNet`. In
this repo `Wire*` is the DOWN direction (`WireDesiredState` / `WirePBSDR` — what the hub sends), and
UP-direction report stanzas are `*Status`. It ships as `GuestNetStatus` so it is not the one report
block named against the convention.
- Wiring is asserted from `package main` by an AST walk (construct + `SetGuestNetReporter` + the
started goroutine) — the v0.91.0 defect was exactly a seam whose caller was never written, with
every unit test green. Red-proof: un-wiring both lines fails the test with both reasons named.
- Red-proof for the detection itself: reverting `classify` to IP-presence-only makes the July-20
fixture report **"healthy"** and records **zero** heals — the 80-minute silent window, reproduced.
- `var version` in main.go was stale at `0.89.0` (three releases behind); builds set it via ldflags,
but `go run` and any forgotten `-X` reported a version that had not existed for days.
## v0.91.2 — a healthy credential probe is observable (2026-07-21)
The probe logged only on failure, so a healthy one was silent — which makes "no `auth_failed`"