feat(v0.92.0): guest-network watchdog (R-54) — supervise the guest's DHCP client
Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).
So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.
internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.
guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.
Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.
Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.
Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
This commit is contained in:
@@ -1,3 +1,66 @@
|
||||
## v0.92.0 — the guest network gets a watchdog (R-54) (2026-07-21)
|
||||
|
||||
**Host-tier only — no controller coupling, no wire change the hub must understand today** (the
|
||||
`guest_net` stanza is additive and stored opaquely, exactly like `pbs_dr` and `wireguard`).
|
||||
|
||||
Closes the OPEN RISK left by `INCIDENT-guest-dhclient-killed-2026-07-20` §5: **the guest's DHCP
|
||||
client is started once by ifupdown at boot and nothing supervises it.** When it died on 2026-07-20
|
||||
the guest kept working for another ~80 minutes on its unexpired lease; only when the lease expired
|
||||
did the address and the default route vanish, taking the Cloudflare tunnel, the hub reports, the
|
||||
catalog sync and the controller→agent channel with them — a 1h15m outage in which every observable
|
||||
signal said healthy for the first 80 minutes.
|
||||
|
||||
**The design consequence, and the point of the whole package: liveness of the DHCP client is itself
|
||||
a probe.** Waiting for the address to disappear is waiting out precisely that silent window. The
|
||||
watchdog therefore flags a DHCP guest unhealthy on `pgrep -x dhclient` alone, while the lease is
|
||||
still live and everything else still looks perfect.
|
||||
|
||||
`internal/guestnet`, built on the wg-tunnel/storage watchdog loop shape:
|
||||
|
||||
- **Probes** (four fixed-shape `pct exec` argvs, no shell anywhere, no guest data interpolated):
|
||||
address, default route, `/etc/network/interfaces` mode, dhclient liveness. Parsers are pinned to
|
||||
output captured live from guest 9201 on 2026-07-21 — including the literal backslash `ip -o`
|
||||
emits and the docker-bridge routes that must not read as a default route.
|
||||
- **Heals** with the incident's restored invocation, verbatim, logged at INFO before it runs:
|
||||
`pct exec <vmid> -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0`
|
||||
- **Dampers**, because this runs a privileged command inside a customer's container: two CONSECUTIVE
|
||||
bad probes before any heal (one blip is not a diagnosis), ≥10 min between heals per guest, ≤3
|
||||
heals/hour, and an observe-only window while the guest (or the agent) has been up under 3 minutes.
|
||||
- **Refuses to act** on a static guest (dhclient must never fight a static config — a static guest
|
||||
missing its address is reported loudly and left to R-50), on an unknown interface mode, on a guest
|
||||
it cannot probe, and when the guest list cannot be ownership-proven. The guest source is the
|
||||
pool-verified one (`ListLXC` ∩ the felhom pool, audit A1) — never a bare `ListLXC`, which under a
|
||||
broad token would run dhclient inside a co-tenant's container.
|
||||
- **A failed PROBE never reads as a dead client.** `pgrep` exits 1 with empty stderr when there is
|
||||
no match; anything on stderr means the probe itself failed, which reads as unknown. Otherwise a
|
||||
missing `pgrep` would heal forever.
|
||||
- **Healthy cycles log a Debug line** (v0.91.2's lesson, one day old): if the quiet path is silent,
|
||||
"no alarms" and "never probed" are the same evidence, and an inert watchdog is indistinguishable
|
||||
from a working one.
|
||||
- **Deliberately NOT in the `errc` fan-out** — a watchdog over customer guests must never be able to
|
||||
terminate the agent. A test asserts that, because joining the fan-out would also make the shutdown
|
||||
drain bound off by one.
|
||||
|
||||
**Config `guest_net` is this repo's first DEFAULT-ON feature gate, and the inversion is deliberate.**
|
||||
Every other gate defaults to false because those features reach outward (an offsite endpoint, an OOB
|
||||
tunnel) and enrolling a box by an update would be wrong. This one looks only INWARD at guests the
|
||||
agent already owns, and the failure it prevents exists on every box today. A watchdog that must be
|
||||
remembered per box is a watchdog that is missing on the box that needed it. Opting out is the
|
||||
explicit act: `"guest_net": {"disable": true}`.
|
||||
|
||||
**Naming deviation from the spec, deliberate:** TASK-D called the report block `WireGuestNet`. In
|
||||
this repo `Wire*` is the DOWN direction (`WireDesiredState` / `WirePBSDR` — what the hub sends), and
|
||||
UP-direction report stanzas are `*Status`. It ships as `GuestNetStatus` so it is not the one report
|
||||
block named against the convention.
|
||||
|
||||
- Wiring is asserted from `package main` by an AST walk (construct + `SetGuestNetReporter` + the
|
||||
started goroutine) — the v0.91.0 defect was exactly a seam whose caller was never written, with
|
||||
every unit test green. Red-proof: un-wiring both lines fails the test with both reasons named.
|
||||
- Red-proof for the detection itself: reverting `classify` to IP-presence-only makes the July-20
|
||||
fixture report **"healthy"** and records **zero** heals — the 80-minute silent window, reproduced.
|
||||
- `var version` in main.go was stale at `0.89.0` (three releases behind); builds set it via ldflags,
|
||||
but `go run` and any forgotten `-X` reported a version that had not existed for days.
|
||||
|
||||
## v0.91.2 — a healthy credential probe is observable (2026-07-21)
|
||||
|
||||
The probe logged only on failure, so a healthy one was silent — which makes "no `auth_failed`"
|
||||
|
||||
Reference in New Issue
Block a user