feat(v0.92.0): guest-network watchdog (R-54) — supervise the guest's DHCP client
Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).
So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.
internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.
guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.
Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.
Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.
Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
This commit is contained in:
@@ -0,0 +1,38 @@
|
||||
package guestnet
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// GuestNetStatus implements hub.GuestNetReporter: the heartbeat stanza built from the last sweep.
|
||||
// Pure read of already-collected state — it never probes, so a hub report can never trigger a pct
|
||||
// exec storm.
|
||||
//
|
||||
// It returns a stanza even when no guest has been probed yet (empty guests + a checked_at), because
|
||||
// "the watchdog is running and has nothing to say" must be distinguishable on the hub from "the
|
||||
// watchdog is not wired", which is the shape the v0.91.0 inert seam hid behind.
|
||||
func (w *Watchdog) GuestNetStatus(context.Context) *hub.GuestNetStatus {
|
||||
snap := w.Snapshot()
|
||||
out := &hub.GuestNetStatus{CheckedAt: w.now().UTC().Format(time.RFC3339)}
|
||||
for _, g := range snap {
|
||||
out.Guests = append(out.Guests, hub.GuestNetGuest{
|
||||
VMID: g.VMID,
|
||||
State: g.State,
|
||||
Mode: g.Mode,
|
||||
IP: g.IP,
|
||||
HasRoute: g.HasRoute,
|
||||
DHClientAlive: g.DHClientAlive,
|
||||
CheckedAt: g.CheckedAt,
|
||||
Healed: g.Healed,
|
||||
HealSucceeded: g.HealSucceeded,
|
||||
LastHealAt: g.LastHealAt,
|
||||
HealsLastHour: g.HealsLastHour,
|
||||
Damped: g.Damped,
|
||||
Message: g.Message,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
Reference in New Issue
Block a user