feat(v0.92.0): guest-network watchdog (R-54) — supervise the guest's DHCP client

Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).

So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.

internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.

guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.

Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.

Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.

Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
This commit is contained in:
2026-07-21 12:29:37 +02:00
parent 08b55a1015
commit c0966d753d
12 changed files with 1575 additions and 1 deletions
+41
View File
@@ -32,6 +32,7 @@ type Config struct {
LocalAPI LocalAPIConfig `json:"local_api"`
LANResolver LANResolverConfig `json:"lan_resolver"`
WGTunnel WGTunnelConfig `json:"wg_tunnel"`
GuestNet GuestNetConfig `json:"guest_net"`
OOB OOBConfig `json:"oob"`
SelfUpdate SelfUpdateConfig `json:"selfupdate"`
LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
@@ -138,6 +139,46 @@ func (w WGTunnelConfig) WithDefaults() WGTunnelConfig {
return w
}
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
//
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
// gate here is `Enabled bool` defaulting to false, because those features reach outward (an offsite
// endpoint, an OOB tunnel) and enrolling a box into one by an update would be wrong. This one only
// looks INWARD at guests the agent already owns, and the failure it prevents — an unsupervised DHCP
// client dying and taking the box off the internet 1-2 hours later, invisibly
// (INCIDENT-guest-dhclient-killed-2026-07-20) — is one every box has today. A watchdog that must be
// remembered per box is a watchdog that is missing on the box that needed it. Opting out is
// therefore the explicit act: `"guest_net": {"disable": true}`.
type GuestNetConfig struct {
Disable bool `json:"disable"` // explicit opt-OUT; default is enabled
IntervalSeconds int `json:"interval_seconds"` // probe cadence; default 60
MinHealIntervalSeconds int `json:"min_heal_interval_seconds"` // per-guest cool-off; default 600
MaxHealsPerHour int `json:"max_heals_per_hour"` // per-guest hourly cap; default 3
SettleSeconds int `json:"settle_seconds"` // boot-race guard (guest AND agent uptime); default 180
}
// Enabled reports whether the guest-network watchdog should run.
func (g GuestNetConfig) Enabled() bool { return !g.Disable }
// WithDefaults fills the cadence and the three dampers. A NEGATIVE value is honoured as-is by the
// watchdog constructor's own guards, so an operator can set 0 to mean "package default" without
// having to know the number.
func (g GuestNetConfig) WithDefaults() GuestNetConfig {
if g.IntervalSeconds == 0 {
g.IntervalSeconds = 60
}
if g.MinHealIntervalSeconds == 0 {
g.MinHealIntervalSeconds = 600
}
if g.MaxHealsPerHour == 0 {
g.MaxHealsPerHour = 3
}
if g.SettleSeconds == 0 {
g.SettleSeconds = 180
}
return g
}
// LANResolverConfig configures the host-level split-horizon DNS resolver (internal/lanresolver): a
// dnsmasq the agent manages so LAN clients reach their guest DIRECTLY at the same hostname + real cert.
// Disabled unless Enable is set. HostIP defaults to the local-API bridge IP (the host LAN anchor);