feat(v0.92.0): guest-network watchdog (R-54) — supervise the guest's DHCP client
Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).
So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.
internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.
guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.
Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.
Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.
Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
This commit is contained in:
@@ -32,6 +32,7 @@ type Config struct {
|
||||
LocalAPI LocalAPIConfig `json:"local_api"`
|
||||
LANResolver LANResolverConfig `json:"lan_resolver"`
|
||||
WGTunnel WGTunnelConfig `json:"wg_tunnel"`
|
||||
GuestNet GuestNetConfig `json:"guest_net"`
|
||||
OOB OOBConfig `json:"oob"`
|
||||
SelfUpdate SelfUpdateConfig `json:"selfupdate"`
|
||||
LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
|
||||
@@ -138,6 +139,46 @@ func (w WGTunnelConfig) WithDefaults() WGTunnelConfig {
|
||||
return w
|
||||
}
|
||||
|
||||
// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet).
|
||||
//
|
||||
// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other
|
||||
// gate here is `Enabled bool` defaulting to false, because those features reach outward (an offsite
|
||||
// endpoint, an OOB tunnel) and enrolling a box into one by an update would be wrong. This one only
|
||||
// looks INWARD at guests the agent already owns, and the failure it prevents — an unsupervised DHCP
|
||||
// client dying and taking the box off the internet 1-2 hours later, invisibly
|
||||
// (INCIDENT-guest-dhclient-killed-2026-07-20) — is one every box has today. A watchdog that must be
|
||||
// remembered per box is a watchdog that is missing on the box that needed it. Opting out is
|
||||
// therefore the explicit act: `"guest_net": {"disable": true}`.
|
||||
type GuestNetConfig struct {
|
||||
Disable bool `json:"disable"` // explicit opt-OUT; default is enabled
|
||||
IntervalSeconds int `json:"interval_seconds"` // probe cadence; default 60
|
||||
MinHealIntervalSeconds int `json:"min_heal_interval_seconds"` // per-guest cool-off; default 600
|
||||
MaxHealsPerHour int `json:"max_heals_per_hour"` // per-guest hourly cap; default 3
|
||||
SettleSeconds int `json:"settle_seconds"` // boot-race guard (guest AND agent uptime); default 180
|
||||
}
|
||||
|
||||
// Enabled reports whether the guest-network watchdog should run.
|
||||
func (g GuestNetConfig) Enabled() bool { return !g.Disable }
|
||||
|
||||
// WithDefaults fills the cadence and the three dampers. A NEGATIVE value is honoured as-is by the
|
||||
// watchdog constructor's own guards, so an operator can set 0 to mean "package default" without
|
||||
// having to know the number.
|
||||
func (g GuestNetConfig) WithDefaults() GuestNetConfig {
|
||||
if g.IntervalSeconds == 0 {
|
||||
g.IntervalSeconds = 60
|
||||
}
|
||||
if g.MinHealIntervalSeconds == 0 {
|
||||
g.MinHealIntervalSeconds = 600
|
||||
}
|
||||
if g.MaxHealsPerHour == 0 {
|
||||
g.MaxHealsPerHour = 3
|
||||
}
|
||||
if g.SettleSeconds == 0 {
|
||||
g.SettleSeconds = 180
|
||||
}
|
||||
return g
|
||||
}
|
||||
|
||||
// LANResolverConfig configures the host-level split-horizon DNS resolver (internal/lanresolver): a
|
||||
// dnsmasq the agent manages so LAN clients reach their guest DIRECTLY at the same hostname + real cert.
|
||||
// Disabled unless Enable is set. HostIP defaults to the local-API bridge IP (the host LAN anchor);
|
||||
|
||||
Reference in New Issue
Block a user