feat(v0.92.0): guest-network watchdog (R-54) — supervise the guest's DHCP client

Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).

So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.

internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.

guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.

Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.

Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.

Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
This commit is contained in:
2026-07-21 12:29:37 +02:00
parent 08b55a1015
commit c0966d753d
12 changed files with 1575 additions and 1 deletions
+18
View File
@@ -65,6 +65,12 @@ type PBSDRReporter interface {
PBSDRStatus(ctx context.Context) *PBSDRStatus
}
// GuestNetReporter is the R-54 seam the guestnet watchdog plugs into (same consumer-side pattern —
// hub does not import guestnet). nil (feature not wired) → no guest_net stanza.
type GuestNetReporter interface {
GuestNetStatus(ctx context.Context) *GuestNetStatus
}
// Collector builds a HostReport from read-only sources. All deps are behind narrow
// interfaces for unit testing.
type Collector struct {
@@ -79,6 +85,7 @@ type Collector struct {
leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled)
wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted)
pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
@@ -145,6 +152,13 @@ func (c *Collector) SetPBSDRReporter(p PBSDRReporter) *Collector {
return c
}
// SetGuestNetReporter wires the R-54 guest-network watchdog as a report source (nil-safe → stanza
// omitted). Returns the collector for chaining.
func (c *Collector) SetGuestNetReporter(g GuestNetReporter) *Collector {
c.guestNet = g
return c
}
// SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side
// pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report.
type SelfUpdateReporter interface {
@@ -227,6 +241,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.pbsdr != nil {
report.PBSDR = c.pbsdr.PBSDRStatus(ctx)
}
// R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted).
if c.guestNet != nil {
report.GuestNet = c.guestNet.GuestNetStatus(ctx)
}
// D1: agent self-update pending status (nil reporter → pending=false, the steady state).
if c.selfUpdate != nil {
report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending()