feat(v0.92.0): guest-network watchdog (R-54) — supervise the guest's DHCP client
Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).
So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.
internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.
guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.
Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.
Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.
Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
This commit is contained in:
@@ -85,6 +85,17 @@ type HostReport struct {
|
||||
// Carries NO secret.
|
||||
PBSDR *PBSDRStatus `json:"pbs_dr,omitempty"`
|
||||
|
||||
// GuestNet is the per-guest network-watchdog stanza (R-54). Present only when the guestnet
|
||||
// watchdog is wired. Same additive/opaque contract as PBSDR and Wireguard above — no
|
||||
// hub-schema change, absent when the reporter is not wired. Carries NO secret: addresses,
|
||||
// route/liveness booleans, heal timestamps and counters only.
|
||||
//
|
||||
// NAMING NOTE (deliberate deviation from TASK-D, which called this block `WireGuestNet`):
|
||||
// in this repo `Wire*` types are the DOWN direction (WireDesiredState/WirePBSDR — what the
|
||||
// hub sends the agent), while UP-direction report stanzas are `*Status`. A `WireGuestNet`
|
||||
// on HostReport would have been the only report block named against that convention.
|
||||
GuestNet *GuestNetStatus `json:"guest_net,omitempty"`
|
||||
|
||||
// LogTail is the agent's on-demand debug-ring tail (v0.83.0 observability) — the agent
|
||||
// mirror of the controller's report log_tails channel. Present ONLY on the heartbeat
|
||||
// right after the control envelope requested it (log_tail_requested); consume-once on
|
||||
@@ -109,6 +120,33 @@ type HostReport struct {
|
||||
// "verify_failed" (fingerprint/reachability pre-consume check failing — retrying, NOTHING
|
||||
// consumed), "consumed_failed" (LOUD: secret burned, apply failed, no auto-retry — operator
|
||||
// re-issue required), "disabled" (descriptor enabled:false). Carries no secret.
|
||||
// GuestNetStatus is the R-54 guest-network watchdog stanza. `guests` carries one entry per owned
|
||||
// RUNNING guest that has been probed at least once; an empty list with a fresh `checked_at` means
|
||||
// the watchdog ran and found nothing to report, which is deliberately distinguishable from the
|
||||
// stanza being absent (= the watchdog is not wired at all).
|
||||
type GuestNetStatus struct {
|
||||
CheckedAt string `json:"checked_at"` // RFC3339, the sweep this snapshot came from
|
||||
Guests []GuestNetGuest `json:"guests,omitempty"`
|
||||
}
|
||||
|
||||
// GuestNetGuest mirrors guestnet.GuestReport on the wire. The two structs are deliberately separate:
|
||||
// internal/hub owns the wire contract and imports no feature package (the consumer-side seam rule).
|
||||
type GuestNetGuest struct {
|
||||
VMID int `json:"vmid"`
|
||||
State string `json:"state"` // healthy | unhealthy | static_fault | unknown
|
||||
Mode string `json:"mode"` // dhcp | static | unknown
|
||||
IP string `json:"ip,omitempty"`
|
||||
HasRoute bool `json:"has_route"`
|
||||
DHClientAlive bool `json:"dhclient_alive"`
|
||||
CheckedAt string `json:"checked_at,omitempty"`
|
||||
Healed bool `json:"healed,omitempty"`
|
||||
HealSucceeded bool `json:"heal_succeeded,omitempty"`
|
||||
LastHealAt string `json:"last_heal_at,omitempty"`
|
||||
HealsLastHour int `json:"heals_last_hour,omitempty"`
|
||||
Damped bool `json:"damped,omitempty"`
|
||||
Message string `json:"message,omitempty"`
|
||||
}
|
||||
|
||||
type PBSDRStatus struct {
|
||||
State string `json:"state"`
|
||||
StorageID string `json:"storage_id,omitempty"`
|
||||
|
||||
Reference in New Issue
Block a user