admin
c0966d753d
Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient
is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20
the guest ran another ~80 minutes on its unexpired lease, then lost its address and default
route and took the tunnel, hub reports, catalog sync and the controller->agent channel with
it (1h15m outage, healthy-looking for the first 80 minutes).
So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment
`pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address
to vanish is waiting out the silent window.
internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces
mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's
heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min
between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on
a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source
is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as
unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never
probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able
to kill the agent.
guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only
inward at guests we already own, and the failure exists on every box today.
Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN
direction in this repo, report stanzas are *Status.
Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy"
with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test.
Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not).
39 lines
1.2 KiB
Go
39 lines
1.2 KiB
Go
package guestnet
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
|
|
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
|
)
|
|
|
|
// GuestNetStatus implements hub.GuestNetReporter: the heartbeat stanza built from the last sweep.
|
|
// Pure read of already-collected state — it never probes, so a hub report can never trigger a pct
|
|
// exec storm.
|
|
//
|
|
// It returns a stanza even when no guest has been probed yet (empty guests + a checked_at), because
|
|
// "the watchdog is running and has nothing to say" must be distinguishable on the hub from "the
|
|
// watchdog is not wired", which is the shape the v0.91.0 inert seam hid behind.
|
|
func (w *Watchdog) GuestNetStatus(context.Context) *hub.GuestNetStatus {
|
|
snap := w.Snapshot()
|
|
out := &hub.GuestNetStatus{CheckedAt: w.now().UTC().Format(time.RFC3339)}
|
|
for _, g := range snap {
|
|
out.Guests = append(out.Guests, hub.GuestNetGuest{
|
|
VMID: g.VMID,
|
|
State: g.State,
|
|
Mode: g.Mode,
|
|
IP: g.IP,
|
|
HasRoute: g.HasRoute,
|
|
DHClientAlive: g.DHClientAlive,
|
|
CheckedAt: g.CheckedAt,
|
|
Healed: g.Healed,
|
|
HealSucceeded: g.HealSucceeded,
|
|
LastHealAt: g.LastHealAt,
|
|
HealsLastHour: g.HealsLastHour,
|
|
Damped: g.Damped,
|
|
Message: g.Message,
|
|
})
|
|
}
|
|
return out
|
|
}
|