From c0966d753dcfbda185f91ec001c09a1239fd0ffd Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 21 Jul 2026 12:29:37 +0200 Subject: [PATCH] =?UTF-8?q?feat(v0.92.0):=20guest-network=20watchdog=20(R-?= =?UTF-8?q?54)=20=E2=80=94=20supervise=20the=20guest's=20DHCP=20client?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the OPEN RISK in INCIDENT-guest-dhclient-killed-2026-07-20 §5. The guest's dhclient is started once by ifupdown at boot and nothing supervises it; when it died on 2026-07-20 the guest ran another ~80 minutes on its unexpired lease, then lost its address and default route and took the tunnel, hub reports, catalog sync and the controller->agent channel with it (1h15m outage, healthy-looking for the first 80 minutes). So liveness of the DHCP client is itself a probe: a DHCP guest is unhealthy the moment `pgrep -x dhclient` comes back empty, while the lease is still live. Waiting for the address to vanish is waiting out the silent window. internal/guestnet: four fixed-shape pct exec probes (address, default route, interfaces mode, dhclient liveness — parsers pinned to output captured live from 9201), the incident's heal invocation verbatim, and dampers throughout: two consecutive bad probes, >=10 min between heals, <=3/hour, observe-only while guest or agent uptime < 3 min. Refuses to act on a static guest, an unknown mode, an unprobeable guest, or an unproven guest list (the source is the pool-verified ListLXC ∩ felhom pool, never a bare ListLXC). A failed probe reads as unknown, never as a dead client. Healthy cycles log a Debug line so "no alarms" and "never probed" stay distinguishable. Not in the errc fan-out — a guest watchdog must never be able to kill the agent. guest_net is the repo's first default-ON gate (opt-out is `{"disable": true}`): it looks only inward at guests we already own, and the failure exists on every box today. Report block ships as GuestNetStatus, not the spec's WireGuestNet: Wire* is the DOWN direction in this repo, report stanzas are *Status. Red-proofs: classify reverted to IP-presence-only -> the July-20 fixture reports "healthy" with zero heals; un-wiring the reporter and the goroutine fails the AST wiring test. Also: `var version` was stale at 0.89.0 (ldflags hid it; `go run` did not). --- CHANGELOG.md | 63 +++ REUSE.md | 3 + cmd/felhom-agent/guestnet_wiring_test.go | 100 +++++ cmd/felhom-agent/main.go | 29 +- internal/config/config.go | 41 ++ internal/guestnet/probe.go | 239 ++++++++++ internal/guestnet/report.go | 38 ++ internal/guestnet/watchdog.go | 377 ++++++++++++++++ internal/guestnet/watchdog_test.go | 550 +++++++++++++++++++++++ internal/hub/collect.go | 18 + internal/hub/collect_guestnet_test.go | 80 ++++ internal/hub/report.go | 38 ++ 12 files changed, 1575 insertions(+), 1 deletion(-) create mode 100644 cmd/felhom-agent/guestnet_wiring_test.go create mode 100644 internal/guestnet/probe.go create mode 100644 internal/guestnet/report.go create mode 100644 internal/guestnet/watchdog.go create mode 100644 internal/guestnet/watchdog_test.go create mode 100644 internal/hub/collect_guestnet_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index f8d2244..1d168e1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,66 @@ +## v0.92.0 — the guest network gets a watchdog (R-54) (2026-07-21) + +**Host-tier only — no controller coupling, no wire change the hub must understand today** (the +`guest_net` stanza is additive and stored opaquely, exactly like `pbs_dr` and `wireguard`). + +Closes the OPEN RISK left by `INCIDENT-guest-dhclient-killed-2026-07-20` §5: **the guest's DHCP +client is started once by ifupdown at boot and nothing supervises it.** When it died on 2026-07-20 +the guest kept working for another ~80 minutes on its unexpired lease; only when the lease expired +did the address and the default route vanish, taking the Cloudflare tunnel, the hub reports, the +catalog sync and the controller→agent channel with them — a 1h15m outage in which every observable +signal said healthy for the first 80 minutes. + +**The design consequence, and the point of the whole package: liveness of the DHCP client is itself +a probe.** Waiting for the address to disappear is waiting out precisely that silent window. The +watchdog therefore flags a DHCP guest unhealthy on `pgrep -x dhclient` alone, while the lease is +still live and everything else still looks perfect. + +`internal/guestnet`, built on the wg-tunnel/storage watchdog loop shape: + +- **Probes** (four fixed-shape `pct exec` argvs, no shell anywhere, no guest data interpolated): + address, default route, `/etc/network/interfaces` mode, dhclient liveness. Parsers are pinned to + output captured live from guest 9201 on 2026-07-21 — including the literal backslash `ip -o` + emits and the docker-bridge routes that must not read as a default route. +- **Heals** with the incident's restored invocation, verbatim, logged at INFO before it runs: + `pct exec -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0` +- **Dampers**, because this runs a privileged command inside a customer's container: two CONSECUTIVE + bad probes before any heal (one blip is not a diagnosis), ≥10 min between heals per guest, ≤3 + heals/hour, and an observe-only window while the guest (or the agent) has been up under 3 minutes. +- **Refuses to act** on a static guest (dhclient must never fight a static config — a static guest + missing its address is reported loudly and left to R-50), on an unknown interface mode, on a guest + it cannot probe, and when the guest list cannot be ownership-proven. The guest source is the + pool-verified one (`ListLXC` ∩ the felhom pool, audit A1) — never a bare `ListLXC`, which under a + broad token would run dhclient inside a co-tenant's container. +- **A failed PROBE never reads as a dead client.** `pgrep` exits 1 with empty stderr when there is + no match; anything on stderr means the probe itself failed, which reads as unknown. Otherwise a + missing `pgrep` would heal forever. +- **Healthy cycles log a Debug line** (v0.91.2's lesson, one day old): if the quiet path is silent, + "no alarms" and "never probed" are the same evidence, and an inert watchdog is indistinguishable + from a working one. +- **Deliberately NOT in the `errc` fan-out** — a watchdog over customer guests must never be able to + terminate the agent. A test asserts that, because joining the fan-out would also make the shutdown + drain bound off by one. + +**Config `guest_net` is this repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** +Every other gate defaults to false because those features reach outward (an offsite endpoint, an OOB +tunnel) and enrolling a box by an update would be wrong. This one looks only INWARD at guests the +agent already owns, and the failure it prevents exists on every box today. A watchdog that must be +remembered per box is a watchdog that is missing on the box that needed it. Opting out is the +explicit act: `"guest_net": {"disable": true}`. + +**Naming deviation from the spec, deliberate:** TASK-D called the report block `WireGuestNet`. In +this repo `Wire*` is the DOWN direction (`WireDesiredState` / `WirePBSDR` — what the hub sends), and +UP-direction report stanzas are `*Status`. It ships as `GuestNetStatus` so it is not the one report +block named against the convention. + +- Wiring is asserted from `package main` by an AST walk (construct + `SetGuestNetReporter` + the + started goroutine) — the v0.91.0 defect was exactly a seam whose caller was never written, with + every unit test green. Red-proof: un-wiring both lines fails the test with both reasons named. +- Red-proof for the detection itself: reverting `classify` to IP-presence-only makes the July-20 + fixture report **"healthy"** and records **zero** heals — the 80-minute silent window, reproduced. +- `var version` in main.go was stale at `0.89.0` (three releases behind); builds set it via ldflags, + but `go run` and any forgotten `-X` reported a version that had not existed for days. + ## v0.91.2 — a healthy credential probe is observable (2026-07-21) The probe logged only on failure, so a healthy one was silent — which makes "no `auth_failed`" diff --git a/REUSE.md b/REUSE.md index 0ae2176..bab1a75 100644 --- a/REUSE.md +++ b/REUSE.md @@ -144,6 +144,9 @@ | `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go | | `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go | | `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go | +| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls | +| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment | +| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind | | `reconcile.OpVerifier` | internal/reconcile/gate.go | `*authz.Verifier` | fake verifier in internal/reconcile gate tests | | `signedjobs.WipeOps` / `Executor` (`ExecutorChain`) | internal/signedjobs/wipe.go + runner.go | `*storage.SudoHostOps`; `WipeExecutor`+`DecommissionExecutor` | internal/signedjobs wipe/runner/decommission tests | | `hub.reporter` / `collectorIface` / `EnvelopeObserver` | internal/hub/loop.go | `*hub.Client`, `*hub.Collector`; `desired.Syncer` + `signedjobs.Runner` | `fakeReporter`/`fakeCollector` internal/hub/loop_test.go; `recordingReporter` loop_logtail_test.go | diff --git a/cmd/felhom-agent/guestnet_wiring_test.go b/cmd/felhom-agent/guestnet_wiring_test.go new file mode 100644 index 0000000..ddb81a5 --- /dev/null +++ b/cmd/felhom-agent/guestnet_wiring_test.go @@ -0,0 +1,100 @@ +package main + +import ( + "go/ast" + "go/parser" + "go/token" + "testing" +) + +// R-54 §9 rule 6 — the seam-discipline test, and the one this repo has the most reason to write: +// v0.91.0 shipped the PBS auth-probe seam with `main.go` never calling `SetAuthSink`, every unit +// test green because they all injected the seam directly. The guestnet watchdog has the identical +// shape (a component + a reporter seam + a goroutine), so its wiring is asserted here rather than +// trusted. +// +// This walks the AST rather than grepping the source: a commented-out call still satisfies a +// substring match (found while red-proofing the controller's twin of this test), and a comment is +// not a caller. +func TestMainWiresGuestNetWatchdog(t *testing.T) { + fset := token.NewFileSet() + f, err := parser.ParseFile(fset, "main.go", nil, 0) + if err != nil { + t.Fatalf("parse main.go: %v", err) + } + + var constructed, reporterWired, started bool + ast.Inspect(f, func(n ast.Node) bool { + switch node := n.(type) { + case *ast.CallExpr: + switch fn := node.Fun.(type) { + case *ast.SelectorExpr: + switch fn.Sel.Name { + case "New": + // guestnet.New(...) + if pkg, ok := fn.X.(*ast.Ident); ok && pkg.Name == "guestnet" { + constructed = true + } + case "SetGuestNetReporter": + reporterWired = true + } + } + case *ast.GoStmt: + if sel, ok := node.Call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Watch" { + if id, ok := sel.X.(*ast.Ident); ok && id.Name == "gnWatchdog" { + started = true + } + } + } + return true + }) + + if !constructed { + t.Error("main.go never calls guestnet.New — the watchdog does not exist at runtime") + } + if !reporterWired { + t.Error("main.go never calls collector.SetGuestNetReporter — the guest_net stanza would " + + "never reach the hub (the exact v0.91.0 inert-seam defect)") + } + if !started { + t.Error("main.go never starts the watchdog with `go gnWatchdog.Watch(ctx)` — it would be " + + "constructed, reported on, and never probe anything") + } +} + +// The watchdog must NOT join the errc fan-out: a guest-network watchdog that can terminate the +// agent turns a customer's DHCP problem into an operator-plane outage. If it is ever changed to +// `errc <- ...`, the drain bound at the bottom of main() also has to change — this catches the +// first half of that mistake. +func TestGuestNetWatchdogIsNotInTheErrcFanout(t *testing.T) { + fset := token.NewFileSet() + f, err := parser.ParseFile(fset, "main.go", nil, 0) + if err != nil { + t.Fatalf("parse main.go: %v", err) + } + bad := false + ast.Inspect(f, func(n ast.Node) bool { + send, ok := n.(*ast.SendStmt) + if !ok { + return true + } + if ch, ok := send.Chan.(*ast.Ident); !ok || ch.Name != "errc" { + return true + } + call, ok := send.Value.(*ast.CallExpr) + if !ok { + return true + } + if sel, ok := call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Watch" { + if id, ok := sel.X.(*ast.Ident); ok && id.Name == "gnWatchdog" { + bad = true + } + } + return true + }) + if bad { + t.Fatal("the guestnet watchdog was added to the errc fan-out — a watchdog over customer " + + "guests must never be able to bring the agent down (and the drain bound in main() " + + "would now be off by one, hanging shutdown)") + } +} diff --git a/cmd/felhom-agent/main.go b/cmd/felhom-agent/main.go index b66913d..c9c8fdd 100644 --- a/cmd/felhom-agent/main.go +++ b/cmd/felhom-agent/main.go @@ -37,6 +37,7 @@ import ( "gitea.dooplex.hu/admin/felhom-agent/internal/fasttick" "gitea.dooplex.hu/admin/felhom-agent/internal/felhomsshd" "gitea.dooplex.hu/admin/felhom-agent/internal/guesthook" + "gitea.dooplex.hu/admin/felhom-agent/internal/guestnet" "gitea.dooplex.hu/admin/felhom-agent/internal/hub" "gitea.dooplex.hu/admin/felhom-agent/internal/lanresolver" "gitea.dooplex.hu/admin/felhom-agent/internal/localapi" @@ -57,7 +58,7 @@ import ( // version is the agent version. Overridable at build time with // -ldflags "-X main.version="; defaults to the in-repo CHANGELOG version. -var version = "0.89.0" +var version = "0.92.0" // runGuestHook is the PVE hook body (`felhom-agent guest-hook `). On pre-start it // creates placeholder dirs for any absent bind-mount source so the guest always boots (the C1 net); @@ -1020,6 +1021,32 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int logger.Info("selfheal: node watchdog starting", "mode", mode, "interval_s", 60) go shMgr.Watch(ctx) } + // R-54: guest-network watchdog. Closes the OPEN RISK left by + // INCIDENT-guest-dhclient-killed-2026-07-20 §5 — the guest's DHCP client is started once by + // ifupdown at boot and nothing supervises it, so its death takes the box off the internet + // ~1-2 h later, when the lease expires. Host-tier by necessity: a guest with no default + // route cannot repair its own default route. Deliberately NOT part of the errc fan-out — a + // watchdog over customer guests must never be able to bring the agent down. + if gn := cfg.GuestNet.WithDefaults(); gn.Enabled() { + gnMode := proxmox.RunnerMode(cfg.Privileged.Mode) + if gnMode == "" { + gnMode = proxmox.RunnerSudo + } + gnRunner := &proxmox.ExecRunner{Mode: gnMode, SudoPath: cfg.Privileged.SudoPath} + // The guest source is the POOL-VERIFIED one (ListLXC ∩ felhom pool, audit A1) — never a + // bare ListLXC, which under a broad token would run dhclient inside a co-tenant's guest. + gnGuests := localapi.NewStaleLockController(px, gnRunner, reconcile.DefaultPool, logger) + gnWatchdog := guestnet.New(gnRunner, gnGuests, time.Duration(gn.IntervalSeconds)*time.Second, logger) + gnWatchdog.SetDampers( + time.Duration(gn.MinHealIntervalSeconds)*time.Second, + gn.MaxHealsPerHour, + time.Duration(gn.SettleSeconds)*time.Second, + ) + collector.SetGuestNetReporter(gnWatchdog) + go gnWatchdog.Watch(ctx) + } else { + logger.Info("guestnet: watchdog disabled by config (guest_net.disable)") + } // F20-BUG3: complete a disk format that an agent restart interrupted (durable-id-bound, re-resolved). localSrv.RecoverFormatJob(ctx) // F2-b: recover any guest left with a stale vzdump lock by a reboot-during-backup (unlock → delete diff --git a/internal/config/config.go b/internal/config/config.go index 80f9b53..45095cb 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -32,6 +32,7 @@ type Config struct { LocalAPI LocalAPIConfig `json:"local_api"` LANResolver LANResolverConfig `json:"lan_resolver"` WGTunnel WGTunnelConfig `json:"wg_tunnel"` + GuestNet GuestNetConfig `json:"guest_net"` OOB OOBConfig `json:"oob"` SelfUpdate SelfUpdateConfig `json:"selfupdate"` LogLevel string `json:"log_level"` // debug|info|warn|error (default info) @@ -138,6 +139,46 @@ func (w WGTunnelConfig) WithDefaults() WGTunnelConfig { return w } +// GuestNetConfig configures the R-54 guest-network watchdog (internal/guestnet). +// +// **This is the repo's first DEFAULT-ON feature gate, and the inversion is deliberate.** Every other +// gate here is `Enabled bool` defaulting to false, because those features reach outward (an offsite +// endpoint, an OOB tunnel) and enrolling a box into one by an update would be wrong. This one only +// looks INWARD at guests the agent already owns, and the failure it prevents — an unsupervised DHCP +// client dying and taking the box off the internet 1-2 hours later, invisibly +// (INCIDENT-guest-dhclient-killed-2026-07-20) — is one every box has today. A watchdog that must be +// remembered per box is a watchdog that is missing on the box that needed it. Opting out is +// therefore the explicit act: `"guest_net": {"disable": true}`. +type GuestNetConfig struct { + Disable bool `json:"disable"` // explicit opt-OUT; default is enabled + IntervalSeconds int `json:"interval_seconds"` // probe cadence; default 60 + MinHealIntervalSeconds int `json:"min_heal_interval_seconds"` // per-guest cool-off; default 600 + MaxHealsPerHour int `json:"max_heals_per_hour"` // per-guest hourly cap; default 3 + SettleSeconds int `json:"settle_seconds"` // boot-race guard (guest AND agent uptime); default 180 +} + +// Enabled reports whether the guest-network watchdog should run. +func (g GuestNetConfig) Enabled() bool { return !g.Disable } + +// WithDefaults fills the cadence and the three dampers. A NEGATIVE value is honoured as-is by the +// watchdog constructor's own guards, so an operator can set 0 to mean "package default" without +// having to know the number. +func (g GuestNetConfig) WithDefaults() GuestNetConfig { + if g.IntervalSeconds == 0 { + g.IntervalSeconds = 60 + } + if g.MinHealIntervalSeconds == 0 { + g.MinHealIntervalSeconds = 600 + } + if g.MaxHealsPerHour == 0 { + g.MaxHealsPerHour = 3 + } + if g.SettleSeconds == 0 { + g.SettleSeconds = 180 + } + return g +} + // LANResolverConfig configures the host-level split-horizon DNS resolver (internal/lanresolver): a // dnsmasq the agent manages so LAN clients reach their guest DIRECTLY at the same hostname + real cert. // Disabled unless Enable is set. HostIP defaults to the local-API bridge IP (the host LAN anchor); diff --git a/internal/guestnet/probe.go b/internal/guestnet/probe.go new file mode 100644 index 0000000..b795a73 --- /dev/null +++ b/internal/guestnet/probe.go @@ -0,0 +1,239 @@ +// Package guestnet implements R-54: the host-tier watchdog for each customer guest's own network. +// +// Origin — INCIDENT-guest-dhclient-killed-2026-07-20 §5 "OPEN RISK". The guest's DHCP client is +// started once by ifupdown at boot and NOTHING supervises it. When it was killed on 2026-07-20 the +// guest kept working for another ~80 minutes on its unexpired lease; only when the lease expired did +// the address and default route vanish, taking the Cloudflare tunnel, the hub reports, the catalog +// sync and the controller→agent channel with them. Total outage ~1h15m, and for the first 80 minutes +// every observable signal said healthy. +// +// The design consequence is the whole point of this package: **liveness of the DHCP client process +// is itself a probe**, not a detail. Waiting for the IP to disappear is waiting out the exact silent +// window the incident proved exists. See TestProbe_DeadDHClientWithLiveLeaseIsUnhealthy. +// +// The agent is the right tier for this: it lives on the host, keeps its own line to the hub, and can +// still see and repair a guest that has gone completely mute. The controller cannot fix its own +// missing default route. +package guestnet + +import ( + "context" + "strings" +) + +// Runner is the privileged-exec seam (satisfied by *proxmox.ExecRunner). Declared consumer-side so +// tests inject a scripted runner and no unit test goes near pct. +type Runner interface { + Run(ctx context.Context, name string, args ...string) (stdout, stderr []byte, err error) +} + +// Mode is how the guest is configured to get its address. +type Mode string + +const ( + ModeDHCP Mode = "dhcp" + ModeStatic Mode = "static" + ModeUnknown Mode = "unknown" +) + +// State is a guest's verdict for one cycle. +type State string + +const ( + StateHealthy State = "healthy" + // StateUnhealthy: DHCP-configured and something is wrong that dhclient can fix. + StateUnhealthy State = "unhealthy" + // StateStaticFault: a static guest missing its address/route. Reported loudly, NEVER healed — + // re-running dhclient on a statically-configured guest would fight its own configuration, and + // the durable answer is R-50 (island bridge), not a point fix here. + StateStaticFault State = "static_fault" + // StateUnknown: the guest could not be probed at all (pct exec failed, an interface file we + // cannot read, a probe tool missing). Never healed — acting blind is how the incident happened. + StateUnknown State = "unknown" +) + +// Probe is one guest's observed network facts. +type Probe struct { + VMID int + Mode Mode + IP string // empty when absent + HasRoute bool + DHCPAlive bool + Reachable bool // pct exec worked at all + Detail string // human-readable reason, operator-tier English +} + +// eth0 is the guest interface every felhom guest uses (the LXC veth peer inside the guest). +const eth0 = "eth0" + +// probe runs the four fixed-shape reads. Every argv is a constant plus the vmid — no guest-supplied +// data is ever interpolated into a command, and there is no shell anywhere in this path. +func (w *Watchdog) probe(ctx context.Context, vmid int) Probe { + p := Probe{VMID: vmid, Mode: ModeUnknown} + id := itoa(vmid) + + // 1. Address. This also settles reachability: if pct exec cannot run here, nothing else is + // worth attempting. + out, errOut, err := w.runner.Run(ctx, "pct", "exec", id, "--", "ip", "-4", "-o", "addr", "show", "dev", eth0) + if err != nil { + p.Detail = "address probe failed: " + firstLine(string(errOut)) + return p // Reachable stays false → StateUnknown + } + p.Reachable = true + p.IP = parseInet(string(out)) + + // 2. Default route. + out, _, err = w.runner.Run(ctx, "pct", "exec", id, "--", "ip", "route", "show", "default") + if err == nil { + p.HasRoute = hasDefaultRoute(string(out)) + } + + // 3. Configured mode. An unreadable interfaces file leaves ModeUnknown, which never heals. + out, _, err = w.runner.Run(ctx, "pct", "exec", id, "--", "cat", "/etc/network/interfaces") + if err == nil { + p.Mode = parseMode(string(out), eth0) + } + + // 4. DHCP client liveness — the probe the incident was invisible to. + // pgrep exits 1 with EMPTY stderr when there is no match; anything on stderr means the probe + // itself failed (pgrep absent, guest wedged), which must read as unknown rather than as a + // dead client, or a missing tool would trigger heals forever. + out, errOut, err = w.runner.Run(ctx, "pct", "exec", id, "--", "pgrep", "-x", "dhclient") + switch { + case err == nil && strings.TrimSpace(string(out)) != "": + p.DHCPAlive = true + case err != nil && strings.TrimSpace(string(errOut)) != "": + p.Reachable = false + p.Detail = "dhclient liveness probe failed: " + firstLine(string(errOut)) + default: + p.DHCPAlive = false + } + return p +} + +// classify turns observed facts into the verdict. Pure — table-tested. +func classify(p Probe) (State, string) { + if !p.Reachable { + d := p.Detail + if d == "" { + d = "guest not reachable via pct exec" + } + return StateUnknown, d + } + switch p.Mode { + case ModeStatic: + if p.IP != "" && p.HasRoute { + return StateHealthy, "static address and default route present" + } + return StateStaticFault, "statically configured guest is missing its address or default route — reported only; dhclient must never be run against a static configuration (R-50 owns the durable fix)" + case ModeDHCP: + switch { + case p.IP == "": + return StateUnhealthy, "no IPv4 address on " + eth0 + case !p.HasRoute: + return StateUnhealthy, "no default route" + case !p.DHCPAlive: + // THE incident state: address and route still present on an unexpired lease, with + // nothing left to renew them. Damage is ~1-2 h in the future and invisible today. + return StateUnhealthy, "dhclient is not running — the lease will not be renewed (the 2026-07-20 failure mode; address still present, renewal already dead)" + default: + return StateHealthy, "address, default route and dhclient all present" + } + default: + return StateUnknown, "interface configuration mode could not be determined — not healing" + } +} + +// --- parsing (fixtures captured live from guest 9201 on 2026-07-21, probe P3) ------------------- + +// parseInet extracts the address from `ip -4 -o addr show dev eth0` output, e.g. +// +// 2: eth0 inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0\ valid_lft ... +// +// Returns "" when there is no inet line at all (the post-lease-expiry state: the command succeeds +// and prints NOTHING). +func parseInet(out string) string { + for _, line := range strings.Split(out, "\n") { + fields := strings.Fields(line) + for i, f := range fields { + if f == "inet" && i+1 < len(fields) { + addr := fields[i+1] + if idx := strings.IndexByte(addr, '/'); idx > 0 { + addr = addr[:idx] + } + return addr + } + } + } + return "" +} + +// hasDefaultRoute parses `ip route show default`, e.g. "default via 192.168.0.1 dev eth0 ". +// Empty output = no default route (the incident state). +func hasDefaultRoute(out string) bool { + for _, line := range strings.Split(out, "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "default ") { + return true + } + } + return false +} + +// parseMode reads the iface stanza for dev out of /etc/network/interfaces: +// +// auto eth0 +// iface eth0 inet dhcp +// +// Anything else (no stanza, a manual/loopback mode, a file we could not read) is ModeUnknown, and +// unknown never heals. +func parseMode(out, dev string) Mode { + for _, line := range strings.Split(out, "\n") { + f := strings.Fields(strings.TrimSpace(line)) + // iface inet + if len(f) >= 4 && f[0] == "iface" && f[1] == dev && f[2] == "inet" { + switch f[3] { + case "dhcp": + return ModeDHCP + case "static": + return ModeStatic + default: + return ModeUnknown + } + } + } + return ModeUnknown +} + +func firstLine(s string) string { + s = strings.TrimSpace(s) + if i := strings.IndexByte(s, '\n'); i >= 0 { + s = s[:i] + } + if len(s) > 200 { + s = s[:200] + } + return s +} + +// itoa avoids pulling strconv into every call site's readability. +func itoa(i int) string { + if i == 0 { + return "0" + } + neg := i < 0 + if neg { + i = -i + } + var b [20]byte + pos := len(b) + for i > 0 { + pos-- + b[pos] = byte('0' + i%10) + i /= 10 + } + if neg { + pos-- + b[pos] = '-' + } + return string(b[pos:]) +} diff --git a/internal/guestnet/report.go b/internal/guestnet/report.go new file mode 100644 index 0000000..5e15adc --- /dev/null +++ b/internal/guestnet/report.go @@ -0,0 +1,38 @@ +package guestnet + +import ( + "context" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/hub" +) + +// GuestNetStatus implements hub.GuestNetReporter: the heartbeat stanza built from the last sweep. +// Pure read of already-collected state — it never probes, so a hub report can never trigger a pct +// exec storm. +// +// It returns a stanza even when no guest has been probed yet (empty guests + a checked_at), because +// "the watchdog is running and has nothing to say" must be distinguishable on the hub from "the +// watchdog is not wired", which is the shape the v0.91.0 inert seam hid behind. +func (w *Watchdog) GuestNetStatus(context.Context) *hub.GuestNetStatus { + snap := w.Snapshot() + out := &hub.GuestNetStatus{CheckedAt: w.now().UTC().Format(time.RFC3339)} + for _, g := range snap { + out.Guests = append(out.Guests, hub.GuestNetGuest{ + VMID: g.VMID, + State: g.State, + Mode: g.Mode, + IP: g.IP, + HasRoute: g.HasRoute, + DHClientAlive: g.DHClientAlive, + CheckedAt: g.CheckedAt, + Healed: g.Healed, + HealSucceeded: g.HealSucceeded, + LastHealAt: g.LastHealAt, + HealsLastHour: g.HealsLastHour, + Damped: g.Damped, + Message: g.Message, + }) + } + return out +} diff --git a/internal/guestnet/watchdog.go b/internal/guestnet/watchdog.go new file mode 100644 index 0000000..f4460e3 --- /dev/null +++ b/internal/guestnet/watchdog.go @@ -0,0 +1,377 @@ +package guestnet + +import ( + "context" + "log/slog" + "sort" + "sync" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" +) + +// GuestSource yields the guests this agent OWNS. Production passes the pool-verified source +// (ListLXC ∩ pool membership, audit A1) — never a bare ListLXC, which under a broad token would let +// the watchdog run dhclient inside a co-tenant's container. +type GuestSource interface { + Guests(ctx context.Context) ([]proxmox.Guest, error) +} + +// Defaults. Every one of these is a damper: this watchdog runs a privileged command inside a +// customer's container, so it is designed to under-act. +const ( + DefaultInterval = 60 * time.Second + // DefaultMinHealInterval is the per-guest cool-off between heals. + DefaultMinHealInterval = 10 * time.Minute + // DefaultMaxHealsPerHour caps a guest's heals; beyond it the watchdog only reports, because a + // guest needing a fourth heal in an hour has a problem dhclient cannot fix. + DefaultMaxHealsPerHour = 3 + // DefaultSettle is the boot-race guard, applied to BOTH the guest's uptime and the agent's own. + // A guest that booted 40 s ago legitimately has no lease yet. + DefaultSettle = 3 * time.Minute + // requiredBadProbes: two CONSECUTIVE bad cycles before any heal. One blip is not a diagnosis. + requiredBadProbes = 2 +) + +// Watchdog probes each owned, running guest's network every interval and heals a DHCP guest whose +// client has died. It never returns an error: a guest-network fault is a reported fact, not an agent +// failure. +type Watchdog struct { + runner Runner + guests GuestSource + logger *slog.Logger + + interval time.Duration + minHealInterval time.Duration + maxHealsPerHour int + settle time.Duration + + now func() time.Time + startedAt time.Time + + mu sync.Mutex + state map[int]*guestState +} + +type guestState struct { + badProbes int + heals []time.Time // heal timestamps, pruned to the last hour + lastHealAt time.Time + lastState State + report GuestReport +} + +// New builds a Watchdog with the shipped dampers. interval <= 0 uses DefaultInterval. +func New(runner Runner, guests GuestSource, interval time.Duration, logger *slog.Logger) *Watchdog { + if interval <= 0 { + interval = DefaultInterval + } + if logger == nil { + logger = slog.Default() + } + w := &Watchdog{ + runner: runner, + guests: guests, + logger: logger, + interval: interval, + minHealInterval: DefaultMinHealInterval, + maxHealsPerHour: DefaultMaxHealsPerHour, + settle: DefaultSettle, + now: time.Now, + state: map[int]*guestState{}, + } + w.startedAt = w.now() + return w +} + +// SetDampers overrides the three rate limits from config. Non-positive values keep the default, +// the same "0 = package default" convention the storage watchdog and wg loop use. +func (w *Watchdog) SetDampers(minHealInterval time.Duration, maxHealsPerHour int, settle time.Duration) { + w.mu.Lock() + defer w.mu.Unlock() + if minHealInterval > 0 { + w.minHealInterval = minHealInterval + } + if maxHealsPerHour > 0 { + w.maxHealsPerHour = maxHealsPerHour + } + if settle > 0 { + w.settle = settle + } +} + +// Watch runs until ctx is cancelled. Started with `go wd.Watch(ctx)` — deliberately not part of the +// errc fan-out, because a guest-network watchdog must never be able to bring the agent down. +func (w *Watchdog) Watch(ctx context.Context) { + w.logger.Info("guestnet: watchdog starting", + "interval", w.interval, "min_heal_interval", w.minHealInterval, + "max_heals_per_hour", w.maxHealsPerHour, "settle", w.settle) + t := time.NewTicker(w.interval) + defer t.Stop() + w.Tick(ctx) // immediate baseline + for { + select { + case <-ctx.Done(): + w.logger.Info("guestnet: watchdog shutting down", "reason", ctx.Err()) + return + case <-t.C: + w.Tick(ctx) + } + } +} + +// Tick performs one full sweep. Exported so the wiring test and the live STOP leg can drive exactly +// one cycle instead of waiting on a ticker. +func (w *Watchdog) Tick(ctx context.Context) { + guests, err := w.guests.Guests(ctx) + if err != nil { + // Unknown ownership ⇒ do nothing. Never fall back to an unfiltered guest list. + w.logger.Warn("guestnet: guest list unavailable — skipping sweep (ownership unproven)", "err", err) + return + } + for _, g := range guests { + if ctx.Err() != nil { + return + } + if g.Status != "running" { + w.forget(g.VMID) + continue + } + w.checkGuest(ctx, g) + } +} + +// forget drops state for a guest that is no longer running, so a stopped-and-restarted guest starts +// from a clean slate rather than inheriting a stale bad-probe count. +func (w *Watchdog) forget(vmid int) { + w.mu.Lock() + delete(w.state, vmid) + w.mu.Unlock() +} + +func (w *Watchdog) checkGuest(ctx context.Context, g proxmox.Guest) { + now := w.now() + p := w.probe(ctx, g.VMID) + state, detail := classify(p) + + w.mu.Lock() + st := w.state[g.VMID] + if st == nil { + st = &guestState{} + w.state[g.VMID] = st + } + prev := st.lastState + st.lastState = state + st.pruneHeals(now) + rep := GuestReport{ + VMID: g.VMID, Mode: string(p.Mode), IP: p.IP, HasRoute: p.HasRoute, + DHClientAlive: p.DHCPAlive, State: string(state), Message: detail, + HealsLastHour: len(st.heals), CheckedAt: now.UTC().Format(time.RFC3339), + } + if !st.lastHealAt.IsZero() { + rep.LastHealAt = st.lastHealAt.UTC().Format(time.RFC3339) + } + + switch state { + case StateHealthy: + st.badProbes = 0 + w.mu.Unlock() + // The healthy path MUST be observable. v0.91.2's lesson, learned the hard way one day + // earlier: if a healthy cycle logs nothing, "no alarms" and "never probed" are the same + // line of evidence, and an inert watchdog is indistinguishable from a working one. + w.logger.Debug("guestnet: guest network healthy", "vmid", g.VMID, "mode", string(p.Mode), + "has_route", p.HasRoute, "dhclient_alive", p.DHCPAlive) + if prev != "" && prev != StateHealthy { + w.logger.Info("guestnet: guest network recovered", "vmid", g.VMID, "previous_state", string(prev)) + } + w.record(g.VMID, rep) + return + + case StateUnknown, StateStaticFault: + st.badProbes = 0 // neither is a dhclient fault; don't accumulate toward a heal + w.mu.Unlock() + if prev != state { // loud once per transition, not once per minute + w.logger.Warn("guestnet: guest network not actionable — reporting only", + "vmid", g.VMID, "state", string(state), "mode", string(p.Mode), + "has_ip", p.IP != "", "has_route", p.HasRoute, "detail", detail) + } + w.record(g.VMID, rep) + return + } + + // --- StateUnhealthy: a DHCP guest with something dhclient can fix --------------------------- + + st.badProbes++ + bad := st.badProbes + lastHeal := st.lastHealAt + healsInHour := len(st.heals) + w.mu.Unlock() + + if reason, ok := w.observeOnly(g, now); !ok { + rep.Damped = true + rep.Message = detail + " — observing only: " + reason + w.logger.Info("guestnet: guest network unhealthy but not acting", "vmid", g.VMID, + "reason", reason, "detail", detail) + w.record(g.VMID, rep) + return + } + if bad < requiredBadProbes { + rep.Message = detail + " — awaiting a second consecutive bad probe before healing" + w.logger.Info("guestnet: guest network unhealthy (first bad probe — not acting yet)", + "vmid", g.VMID, "detail", detail, "bad_probes", bad, "required", requiredBadProbes) + w.record(g.VMID, rep) + return + } + if damped, reason := w.damped(lastHeal, healsInHour, now); damped { + rep.Damped = true + rep.Message = detail + " — heal damped: " + reason + w.logger.Warn("guestnet: guest network unhealthy but healing is DAMPED — reporting only", + "vmid", g.VMID, "reason", reason, "heals_last_hour", healsInHour, "detail", detail) + w.record(g.VMID, rep) + return + } + + // --- heal ---------------------------------------------------------------------------------- + w.logger.Warn("guestnet: guest network unhealthy — healing", + "vmid", g.VMID, "detail", detail, "bad_probes", bad) + healed, healErr := w.heal(ctx, g.VMID) + + w.mu.Lock() + st = w.state[g.VMID] + if st != nil { + st.heals = append(st.heals, now) + st.lastHealAt = now + st.badProbes = 0 // the post-heal probe below is the new evidence + healsInHour = len(st.heals) + } + w.mu.Unlock() + + after := w.probe(ctx, g.VMID) + afterState, afterDetail := classify(after) + + rep = GuestReport{ + VMID: g.VMID, Mode: string(after.Mode), IP: after.IP, HasRoute: after.HasRoute, + DHClientAlive: after.DHCPAlive, State: string(afterState), Message: afterDetail, + Healed: true, HealSucceeded: afterState == StateHealthy, + LastHealAt: now.UTC().Format(time.RFC3339), HealsLastHour: healsInHour, + CheckedAt: w.now().UTC().Format(time.RFC3339), + } + if healErr != nil { + rep.Message = "heal command failed: " + healErr.Error() + "; " + afterDetail + } + + if afterState == StateHealthy { + w.logger.Info("guestnet: guest network healed", "vmid", g.VMID, "ip", after.IP, + "has_route", after.HasRoute, "dhclient_alive", after.DHCPAlive, "heals_last_hour", healsInHour) + } else { + w.logger.Error("guestnet: heal did not restore the guest network", "vmid", g.VMID, + "state", string(afterState), "detail", afterDetail, "heal_ran", healed, "err", healErr) + } + + w.mu.Lock() + if st = w.state[g.VMID]; st != nil { + st.lastState = afterState + } + w.mu.Unlock() + w.record(g.VMID, rep) +} + +// heal runs the incident's restored invocation, VERBATIM (INCIDENT-guest-dhclient-killed-2026-07-20 +// §5) — the same argv that brought guest 9201 back at 10:04:3x UTC. Fixed shape, no shell, no guest +// data interpolated. Logged at INFO before it runs so the operator sees the exact command. +func (w *Watchdog) heal(ctx context.Context, vmid int) (bool, error) { + args := []string{"exec", itoa(vmid), "--", "dhclient", + "-pf", "/run/dhclient." + eth0 + ".pid", + "-lf", "/var/lib/dhcp/dhclient." + eth0 + ".leases", eth0} + w.logger.Info("guestnet: running heal command", "vmid", vmid, "cmd", "pct "+joinArgs(args)) + _, errOut, err := w.runner.Run(ctx, "pct", args...) + if err != nil { + w.logger.Error("guestnet: heal command failed", "vmid", vmid, "stderr", firstLine(string(errOut)), "err", err) + return false, err + } + return true, nil +} + +// observeOnly reports whether a boot race means this cycle must look and not touch. +func (w *Watchdog) observeOnly(g proxmox.Guest, now time.Time) (string, bool) { + if now.Sub(w.startedAt) < w.settle { + return "agent started less than " + w.settle.String() + " ago", false + } + if g.Uptime > 0 && time.Duration(g.Uptime)*time.Second < w.settle { + return "guest has been up for less than " + w.settle.String(), false + } + return "", true +} + +// damped applies the two rate limits. +func (w *Watchdog) damped(lastHeal time.Time, healsInHour int, now time.Time) (bool, string) { + if !lastHeal.IsZero() && now.Sub(lastHeal) < w.minHealInterval { + return true, "last heal was less than " + w.minHealInterval.String() + " ago" + } + if healsInHour >= w.maxHealsPerHour { + return true, "heal budget for the hour is spent (a guest needing more than this has a fault dhclient cannot fix)" + } + return false, "" +} + +func (s *guestState) pruneHeals(now time.Time) { + kept := s.heals[:0] + for _, t := range s.heals { + if now.Sub(t) < time.Hour { + kept = append(kept, t) + } + } + s.heals = kept +} + +func (w *Watchdog) record(vmid int, rep GuestReport) { + w.mu.Lock() + defer w.mu.Unlock() + if st := w.state[vmid]; st != nil { + st.report = rep + } +} + +func joinArgs(args []string) string { + out := "" + for i, a := range args { + if i > 0 { + out += " " + } + out += a + } + return out +} + +// --- the hub report block ------------------------------------------------------------------------ + +// GuestReport is one guest's last observed network state, mirrored to the hub. +type GuestReport struct { + VMID int `json:"vmid"` + State string `json:"state"` // healthy | unhealthy | static_fault | unknown + Mode string `json:"mode"` // dhcp | static | unknown + IP string `json:"ip,omitempty"` + HasRoute bool `json:"has_route"` + DHClientAlive bool `json:"dhclient_alive"` + CheckedAt string `json:"checked_at,omitempty"` + Healed bool `json:"healed,omitempty"` // a heal ran on THIS cycle + HealSucceeded bool `json:"heal_succeeded,omitempty"` // and the re-probe came back healthy + LastHealAt string `json:"last_heal_at,omitempty"` + HealsLastHour int `json:"heals_last_hour,omitempty"` + Damped bool `json:"damped,omitempty"` + Message string `json:"message,omitempty"` +} + +// Snapshot returns the per-guest blocks for the heartbeat, VMID-sorted for a stable wire shape. +func (w *Watchdog) Snapshot() []GuestReport { + w.mu.Lock() + defer w.mu.Unlock() + out := make([]GuestReport, 0, len(w.state)) + for _, st := range w.state { + if st.report.VMID != 0 { + out = append(out, st.report) + } + } + sort.Slice(out, func(i, j int) bool { return out[i].VMID < out[j].VMID }) + return out +} diff --git a/internal/guestnet/watchdog_test.go b/internal/guestnet/watchdog_test.go new file mode 100644 index 0000000..22c1146 --- /dev/null +++ b/internal/guestnet/watchdog_test.go @@ -0,0 +1,550 @@ +package guestnet + +import ( + "bytes" + "context" + "errors" + "io" + "log/slog" + "strings" + "sync" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" +) + +// --- fixtures captured LIVE from guest 9201 on 2026-07-21 (probe P3, via `ssh felhom-pve`) ------- +// +// These are the byte shapes the parser must survive; note the literal backslash `ip -o` emits and +// the trailing space on the route line. + +const ( + fxAddr = "2: eth0 inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0\\ valid_lft 4916sec preferred_lft 4916sec\n" + fxRoute = "default via 192.168.0.1 dev eth0 \n" + fxPgrep = "235839\n" + fxIfacesDHCP = "auto lo\niface lo inet loopback\n\nauto eth0\niface eth0 inet dhcp\n" + fxIfacesStat = "auto lo\niface lo inet loopback\n\nauto eth0\niface eth0 inet static\n\taddress 192.168.0.162/24\n\tgateway 192.168.0.1\n" + // pct exec against a guest that does not exist / is not running (rc=2, message on stderr). + fxNoGuestErr = "Configuration file 'nodes/demo-felhom/lxc/9999.conf' does not exist\n" +) + +// scriptedRunner answers per probe kind and records EVERY argv. The counts are the assertions that +// matter: a watchdog that heals when it must not is worse than one that never heals. +type scriptedRunner struct { + mu sync.Mutex + out map[string]string // kind → stdout + fail map[string]error // kind → error + errs map[string]string // kind → stderr + call [][]string + // healFixes models what a successful dhclient actually does: the client is running again and + // the lease is renewed. Set false to model a guest whose network is broken beyond dhclient. + healFixes bool +} + +func newRunner() *scriptedRunner { + return &scriptedRunner{ + out: map[string]string{ + "addr": fxAddr, "route": fxRoute, "iface": fxIfacesDHCP, "pgrep": fxPgrep, "dhclient": "", + }, + fail: map[string]error{}, + errs: map[string]string{}, + healFixes: true, + } +} + +// kind classifies a `pct exec -- ...` argv. +func kind(args []string) string { + if len(args) < 4 || args[0] != "exec" { + return "other" + } + rest := args[3:] + switch rest[0] { + case "ip": + if len(rest) > 1 && rest[1] == "route" { + return "route" + } + return "addr" + case "cat": + return "iface" + case "pgrep": + return "pgrep" + case "dhclient": + return "dhclient" + } + return "other" +} + +func (r *scriptedRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) { + r.mu.Lock() + defer r.mu.Unlock() + r.call = append(r.call, append([]string{name}, args...)) + k := kind(args) + if k == "dhclient" && r.fail[k] == nil && r.healFixes { + // a real dhclient re-acquires the lease and stays resident + r.out["pgrep"], r.out["addr"], r.out["route"] = fxPgrep, fxAddr, fxRoute + delete(r.fail, "pgrep") + } + return []byte(r.out[k]), []byte(r.errs[k]), r.fail[k] +} + +func (r *scriptedRunner) countOf(k string) int { + r.mu.Lock() + defer r.mu.Unlock() + n := 0 + for _, c := range r.call { + if len(c) > 1 && kind(c[1:]) == k { + n++ + } + } + return n +} + +func (r *scriptedRunner) lastOf(k string) []string { + r.mu.Lock() + defer r.mu.Unlock() + for i := len(r.call) - 1; i >= 0; i-- { + if len(r.call[i]) > 1 && kind(r.call[i][1:]) == k { + return r.call[i] + } + } + return nil +} + +// killDHClient models the exact 2026-07-20 state: lease still valid (address AND route present), +// dhclient gone. This is the fixture the whole feature exists for. +func (r *scriptedRunner) killDHClient() { + r.mu.Lock() + defer r.mu.Unlock() + r.out["pgrep"] = "" + r.fail["pgrep"] = errors.New("exit status 1") // pgrep: no match, EMPTY stderr +} + +func (r *scriptedRunner) reviveDHClient() { + r.mu.Lock() + defer r.mu.Unlock() + r.out["pgrep"] = fxPgrep + delete(r.fail, "pgrep") +} + +// expireLease models the state 80 minutes later: address and route gone too. +func (r *scriptedRunner) expireLease() { + r.mu.Lock() + defer r.mu.Unlock() + r.out["addr"] = "" + r.out["route"] = "" +} + +type fakeGuests struct { + guests []proxmox.Guest + err error +} + +func (f *fakeGuests) Guests(context.Context) ([]proxmox.Guest, error) { return f.guests, f.err } + +func running9201() *fakeGuests { + return &fakeGuests{guests: []proxmox.Guest{ + {VMID: 9201, Name: "felhom-demo", Status: "running", Type: "lxc", Uptime: 7200}, + }} +} + +// newTestWatchdog wires a watchdog with a manual clock the test advances, and captures the log so +// the "healthy cycles are observable" contract can be asserted rather than assumed. +func newTestWatchdog(r Runner, g GuestSource) (*Watchdog, *time.Time, *bytes.Buffer) { + clock := time.Unix(1_784_000_000, 0).UTC() + buf := &bytes.Buffer{} + logger := slog.New(slog.NewTextHandler(buf, &slog.HandlerOptions{Level: slog.LevelDebug})) + w := New(r, g, time.Minute, logger) + w.now = func() time.Time { return clock } + // The agent's own settle window is measured from startedAt, which New stamped with the REAL + // clock; restamp it against the fake one, well in the past. + w.startedAt = clock.Add(-time.Hour) + return w, &clock, buf +} + +// --- Scenario E: the TIMED failure, detected instantly ------------------------------------------ +// +// RED-PROOF (recorded in REPORT.md): reverting classify()'s dhcp arm to IP-presence-only — +// +// case ModeDHCP: +// if p.IP == "" { return StateUnhealthy, ... } +// return StateHealthy, ... +// +// makes TestProbe_DeadDHClientWithLiveLeaseIsUnhealthy report "healthy" for the July-20 fixture, and +// TestWatchdog_HealsTheIncidentState records ZERO heals. That is the 80-minute silent window, exactly +// as it happened. + +func TestProbe_DeadDHClientWithLiveLeaseIsUnhealthy(t *testing.T) { + r := newRunner() + r.killDHClient() + w, _, _ := newTestWatchdog(r, running9201()) + + p := w.probe(context.Background(), 9201) + if !p.Reachable { + t.Fatalf("guest must read as reachable: %+v", p) + } + if p.IP != "192.168.0.104" || !p.HasRoute { + t.Fatalf("the lease is still live in this fixture — IP/route must be present: %+v", p) + } + if p.DHCPAlive { + t.Fatalf("dhclient must read as dead: %+v", p) + } + + state, detail := classify(p) + if state != StateUnhealthy { + t.Fatalf("classify = %q, want %q — waiting for the IP to vanish is the 80-minute silent "+ + "window the incident proved (detail: %s)", state, StateUnhealthy, detail) + } + if !strings.Contains(detail, "dhclient") { + t.Fatalf("the reason must name the dead client, got %q", detail) + } +} + +func TestWatchdog_HealsTheIncidentState(t *testing.T) { + r := newRunner() + r.killDHClient() + w, clock, logBuf := newTestWatchdog(r, running9201()) + ctx := context.Background() + + // Cycle 1: unhealthy, but one bad probe is not a diagnosis. + w.Tick(ctx) + if n := r.countOf("dhclient"); n != 0 { + t.Fatalf("healed after ONE bad probe (%d heals) — a single blip must never trigger a heal", n) + } + + // Cycle 2: second consecutive bad probe → heal. The heal makes the client live again. + *clock = clock.Add(time.Minute) + w.Tick(ctx) + + if n := r.countOf("dhclient"); n != 1 { + t.Fatalf("heal ran %d times, want exactly 1", n) + } + // The invocation must be the incident's, verbatim. + want := []string{"pct", "exec", "9201", "--", "dhclient", + "-pf", "/run/dhclient.eth0.pid", "-lf", "/var/lib/dhcp/dhclient.eth0.leases", "eth0"} + got := r.lastOf("dhclient") + if len(got) != len(want) { + t.Fatalf("heal argv = %v, want %v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("heal argv[%d] = %q, want %q (full: %v)", i, got[i], want[i], got) + } + } + + // The report must show the heal AND the verified-healthy re-probe. + snap := w.Snapshot() + if len(snap) != 1 { + t.Fatalf("snapshot = %+v, want one guest", snap) + } + g := snap[0] + if !g.Healed || !g.HealSucceeded { + t.Fatalf("report must record a successful heal: %+v", g) + } + if g.State != string(StateHealthy) || g.LastHealAt == "" || g.HealsLastHour != 1 { + t.Fatalf("post-heal report is wrong: %+v", g) + } + if !strings.Contains(logBuf.String(), "guestnet: guest network healed") { + t.Fatalf("the heal was not logged: %s", logBuf.String()) + } +} + +// A healthy box must be silent about alarms but NOT silent about having looked. +func TestWatchdog_HealthyCycleProbesAndNeverHeals(t *testing.T) { + r := newRunner() + w, _, logBuf := newTestWatchdog(r, running9201()) + + w.Tick(context.Background()) + + if n := r.countOf("dhclient"); n != 0 { + t.Fatalf("a healthy guest was healed %d times, want 0", n) + } + if r.countOf("pgrep") != 1 || r.countOf("addr") != 1 { + t.Fatalf("the healthy path must still probe: %v", r.call) + } + if !strings.Contains(logBuf.String(), "guest network healthy") { + t.Fatalf("a healthy cycle must be observable — otherwise 'no alarms' and 'never probed' "+ + "are the same evidence (v0.91.2's lesson). Log: %s", logBuf.String()) + } + snap := w.Snapshot() + if len(snap) != 1 || snap[0].State != string(StateHealthy) || !snap[0].DHClientAlive { + t.Fatalf("healthy snapshot wrong: %+v", snap) + } +} + +// --- Scenario F: configuration and damping guards ------------------------------------------------ + +func TestWatchdog_StaticGuestIsNeverHealedWithDHClient(t *testing.T) { + r := newRunner() + r.out["iface"] = fxIfacesStat + r.killDHClient() // on a static guest this is NORMAL + w, clock, _ := newTestWatchdog(r, running9201()) + + for i := 0; i < 5; i++ { + w.Tick(context.Background()) + *clock = clock.Add(time.Minute) + } + if n := r.countOf("dhclient"); n != 0 { + t.Fatalf("a static guest was healed with dhclient %d times, want 0", n) + } + if s := w.Snapshot(); len(s) != 1 || s[0].State != string(StateHealthy) { + t.Fatalf("a static guest with address+route is healthy, got %+v", s) + } +} + +func TestWatchdog_StaticGuestMissingAddressReportsButNeverHeals(t *testing.T) { + r := newRunner() + r.out["iface"] = fxIfacesStat + r.expireLease() // no address, no route on a STATIC guest → R-50 territory, not ours + w, clock, logBuf := newTestWatchdog(r, running9201()) + + for i := 0; i < 5; i++ { + w.Tick(context.Background()) + *clock = clock.Add(time.Minute) + } + if n := r.countOf("dhclient"); n != 0 { + t.Fatalf("healed a static guest %d times, want 0 — dhclient must never fight a static config", n) + } + s := w.Snapshot() + if len(s) != 1 || s[0].State != string(StateStaticFault) { + t.Fatalf("state = %+v, want static_fault", s) + } + if !strings.Contains(logBuf.String(), "not actionable") { + t.Fatalf("a static fault must be reported loudly: %s", logBuf.String()) + } + // Loud ONCE per transition, not once per cycle. + if n := strings.Count(logBuf.String(), "not actionable"); n != 1 { + t.Fatalf("static fault logged %d times over 5 cycles, want 1 (per transition)", n) + } +} + +func TestWatchdog_UnreachableGuestIsUnknownAndNeverHealed(t *testing.T) { + r := newRunner() + r.fail["addr"] = errors.New("exit status 2") + r.errs["addr"] = fxNoGuestErr + w, clock, _ := newTestWatchdog(r, running9201()) + + for i := 0; i < 4; i++ { + w.Tick(context.Background()) + *clock = clock.Add(time.Minute) + } + if n := r.countOf("dhclient"); n != 0 { + t.Fatalf("healed a guest we could not probe %d times, want 0 — acting blind is how the "+ + "incident happened", n) + } + if s := w.Snapshot(); len(s) != 1 || s[0].State != string(StateUnknown) { + t.Fatalf("state = %+v, want unknown", s) + } +} + +// A missing pgrep (or any probe tool) must read as unknown, never as a dead client — otherwise a +// broken probe would heal forever. +func TestWatchdog_ProbeToolFailureIsUnknownNotDead(t *testing.T) { + r := newRunner() + r.fail["pgrep"] = errors.New("exit status 127") + r.errs["pgrep"] = "pgrep: command not found\n" + w, clock, _ := newTestWatchdog(r, running9201()) + + for i := 0; i < 4; i++ { + w.Tick(context.Background()) + *clock = clock.Add(time.Minute) + } + if n := r.countOf("dhclient"); n != 0 { + t.Fatalf("a failed liveness probe caused %d heals, want 0", n) + } + if s := w.Snapshot(); len(s) != 1 || s[0].State != string(StateUnknown) { + t.Fatalf("state = %+v, want unknown", s) + } +} + +func TestWatchdog_DampingCeilings(t *testing.T) { + r := newRunner() + r.killDHClient() + r.healFixes = false // the heal "works" but the client dies again immediately + w, clock, _ := newTestWatchdog(r, running9201()) + ctx := context.Background() + + // 10 hours of one-minute cycles against a permanently broken guest. + for i := 0; i < 600; i++ { + w.Tick(ctx) + *clock = clock.Add(time.Minute) + } + + heals := r.countOf("dhclient") + // Ceiling: 3 per hour AND ≥10 min apart ⇒ at most 3 in any rolling hour. Over 10 h the + // min-interval rule dominates: 6 slots/hour capped to 3/hour ⇒ ≤ 30. + if heals > 30 { + t.Fatalf("heals = %d over 10 h, want ≤ 30 (≤3/hour) — the damper is not holding", heals) + } + if heals == 0 { + t.Fatalf("heals = 0 — the damper has become a mute") + } +} + +func TestWatchdog_MinimumIntervalBetweenHeals(t *testing.T) { + r := newRunner() + r.killDHClient() + r.healFixes = false + w, clock, _ := newTestWatchdog(r, running9201()) + ctx := context.Background() + + w.Tick(ctx) // bad probe 1 + *clock = clock.Add(time.Minute) + w.Tick(ctx) // bad probe 2 → heal #1 + if r.countOf("dhclient") != 1 { + t.Fatalf("expected exactly one heal by now, got %d", r.countOf("dhclient")) + } + // Nine more minutes of failure: still inside the 10-minute cool-off. + for i := 0; i < 9; i++ { + *clock = clock.Add(time.Minute) + w.Tick(ctx) + } + if n := r.countOf("dhclient"); n != 1 { + t.Fatalf("heals = %d within the 10-minute cool-off, want 1", n) + } + if s := w.Snapshot(); len(s) != 1 || !s[0].Damped { + t.Fatalf("a damped cycle must say so in the report: %+v", s) + } + // Past the cool-off, one more heal is allowed. + *clock = clock.Add(2 * time.Minute) + w.Tick(ctx) + if n := r.countOf("dhclient"); n != 2 { + t.Fatalf("heals = %d after the cool-off expired, want 2", n) + } +} + +func TestWatchdog_BootRacesObserveOnly(t *testing.T) { + t.Run("young guest", func(t *testing.T) { + r := newRunner() + r.killDHClient() + g := running9201() + g.guests[0].Uptime = 40 // seconds + w, clock, _ := newTestWatchdog(r, g) + for i := 0; i < 4; i++ { + w.Tick(context.Background()) + *clock = clock.Add(time.Minute) + } + if n := r.countOf("dhclient"); n != 0 { + t.Fatalf("healed a guest that booted 40 s ago %d times, want 0 — it has no lease YET", n) + } + }) + + t.Run("young agent", func(t *testing.T) { + r := newRunner() + r.killDHClient() + w, clock, _ := newTestWatchdog(r, running9201()) + w.startedAt = *clock // the agent just started + for i := 0; i < 2; i++ { + w.Tick(context.Background()) + *clock = clock.Add(time.Minute) + } + if n := r.countOf("dhclient"); n != 0 { + t.Fatalf("healed %d times within the agent's own settle window, want 0", n) + } + }) +} + +func TestWatchdog_StoppedGuestIsNotProbed(t *testing.T) { + r := newRunner() + g := running9201() + g.guests[0].Status = "stopped" + w, _, _ := newTestWatchdog(r, g) + + w.Tick(context.Background()) + + if len(r.call) != 0 { + t.Fatalf("a stopped guest was probed: %v", r.call) + } +} + +func TestWatchdog_GuestListFailureSkipsTheSweep(t *testing.T) { + r := newRunner() + w, _, logBuf := newTestWatchdog(r, &fakeGuests{err: errors.New("pool membership read: 403")}) + + w.Tick(context.Background()) + + if len(r.call) != 0 { + t.Fatalf("acted with unproven ownership: %v", r.call) + } + if !strings.Contains(logBuf.String(), "ownership unproven") { + t.Fatalf("the skip must be logged: %s", logBuf.String()) + } +} + +// A transient bad probe followed by recovery must never heal, and must log the recovery. +func TestWatchdog_SingleBlipNeverHeals(t *testing.T) { + r := newRunner() + w, clock, logBuf := newTestWatchdog(r, running9201()) + ctx := context.Background() + + w.Tick(ctx) // healthy + r.killDHClient() + *clock = clock.Add(time.Minute) + w.Tick(ctx) // bad probe 1 + r.reviveDHClient() + *clock = clock.Add(time.Minute) + w.Tick(ctx) // healthy again + + if n := r.countOf("dhclient"); n != 0 { + t.Fatalf("a single blip caused %d heals, want 0", n) + } + if !strings.Contains(logBuf.String(), "guest network recovered") { + t.Fatalf("the recovery must be visible: %s", logBuf.String()) + } +} + +// --- parsers over the live P3 fixtures ------------------------------------------------------------ + +func TestParsers_OverLiveFixtures(t *testing.T) { + if got := parseInet(fxAddr); got != "192.168.0.104" { + t.Fatalf("parseInet = %q, want 192.168.0.104", got) + } + if got := parseInet(""); got != "" { + t.Fatalf("parseInet(empty) = %q, want empty (the post-expiry state prints nothing)", got) + } + if !hasDefaultRoute(fxRoute) { + t.Fatalf("hasDefaultRoute(%q) = false", fxRoute) + } + if hasDefaultRoute("") || hasDefaultRoute("172.17.0.0/16 dev docker0 proto kernel scope link\n") { + t.Fatal("docker bridge routes must not read as a default route (the incident's exact leftovers)") + } + if got := parseMode(fxIfacesDHCP, "eth0"); got != ModeDHCP { + t.Fatalf("parseMode(dhcp) = %q", got) + } + if got := parseMode(fxIfacesStat, "eth0"); got != ModeStatic { + t.Fatalf("parseMode(static) = %q", got) + } + if got := parseMode("auto lo\niface lo inet loopback\n", "eth0"); got != ModeUnknown { + t.Fatalf("parseMode(no eth0 stanza) = %q, want unknown", got) + } + if got := parseMode("iface eth0 inet manual\n", "eth0"); got != ModeUnknown { + t.Fatalf("parseMode(manual) = %q, want unknown", got) + } +} + +func TestClassify_Table(t *testing.T) { + cases := []struct { + name string + p Probe + want State + }{ + {"dhcp all green", Probe{Reachable: true, Mode: ModeDHCP, IP: "1.2.3.4", HasRoute: true, DHCPAlive: true}, StateHealthy}, + {"dhcp dead client, live lease", Probe{Reachable: true, Mode: ModeDHCP, IP: "1.2.3.4", HasRoute: true}, StateUnhealthy}, + {"dhcp no address", Probe{Reachable: true, Mode: ModeDHCP, DHCPAlive: true}, StateUnhealthy}, + {"dhcp no route", Probe{Reachable: true, Mode: ModeDHCP, IP: "1.2.3.4", DHCPAlive: true}, StateUnhealthy}, + {"static green", Probe{Reachable: true, Mode: ModeStatic, IP: "1.2.3.4", HasRoute: true}, StateHealthy}, + {"static no client is normal", Probe{Reachable: true, Mode: ModeStatic, IP: "1.2.3.4", HasRoute: true}, StateHealthy}, + {"static broken", Probe{Reachable: true, Mode: ModeStatic}, StateStaticFault}, + {"unreachable", Probe{Mode: ModeDHCP}, StateUnknown}, + {"unknown mode", Probe{Reachable: true, Mode: ModeUnknown, IP: "1.2.3.4", HasRoute: true}, StateUnknown}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got, _ := classify(tc.p); got != tc.want { + t.Fatalf("classify = %q, want %q", got, tc.want) + } + }) + } +} + +var _ io.Writer = (*bytes.Buffer)(nil) diff --git a/internal/hub/collect.go b/internal/hub/collect.go index 07f99e7..516bd46 100644 --- a/internal/hub/collect.go +++ b/internal/hub/collect.go @@ -65,6 +65,12 @@ type PBSDRReporter interface { PBSDRStatus(ctx context.Context) *PBSDRStatus } +// GuestNetReporter is the R-54 seam the guestnet watchdog plugs into (same consumer-side pattern — +// hub does not import guestnet). nil (feature not wired) → no guest_net stanza. +type GuestNetReporter interface { + GuestNetStatus(ctx context.Context) *GuestNetStatus +} + // Collector builds a HostReport from read-only sources. All deps are behind narrow // interfaces for unit testing. type Collector struct { @@ -79,6 +85,7 @@ type Collector struct { leafFP string // v0.48.0: served local-API leaf fp (static per process; "" when local API disabled) wg WireguardReporter // S3: offsite-tunnel status (nil → stanza omitted) pbsdr PBSDRReporter // slice 2: PBS DR tier bridge state (nil → stanza omitted) + guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted) selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false) mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted) oob OOBReporter // H1: operator-access health (nil → stanza omitted) @@ -145,6 +152,13 @@ func (c *Collector) SetPBSDRReporter(p PBSDRReporter) *Collector { return c } +// SetGuestNetReporter wires the R-54 guest-network watchdog as a report source (nil-safe → stanza +// omitted). Returns the collector for chaining. +func (c *Collector) SetGuestNetReporter(g GuestNetReporter) *Collector { + c.guestNet = g + return c +} + // SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side // pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report. type SelfUpdateReporter interface { @@ -227,6 +241,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) { if c.pbsdr != nil { report.PBSDR = c.pbsdr.PBSDRStatus(ctx) } + // R-54: guest-network watchdog state (nil reporter = feature not wired → stanza omitted). + if c.guestNet != nil { + report.GuestNet = c.guestNet.GuestNetStatus(ctx) + } // D1: agent self-update pending status (nil reporter → pending=false, the steady state). if c.selfUpdate != nil { report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending() diff --git a/internal/hub/collect_guestnet_test.go b/internal/hub/collect_guestnet_test.go new file mode 100644 index 0000000..0b76de1 --- /dev/null +++ b/internal/hub/collect_guestnet_test.go @@ -0,0 +1,80 @@ +package hub + +import ( + "context" + "encoding/json" + "testing" +) + +// R-54 §9 rule 6: the guest_net stanza must appear in a report built through the PRODUCTION collect +// path, not only in a struct a test constructed. The v0.91.0 defect was exactly this gap — a seam +// with green tests and no caller. + +type fakeGuestNet struct{ st *GuestNetStatus } + +func (f fakeGuestNet) GuestNetStatus(context.Context) *GuestNetStatus { return f.st } + +func TestCollect_GuestNetOmittedWhenReporterNil(t *testing.T) { + px := &fakePx{node: "n", ns: newTestNodeStatus()} + c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger()) + r, err := c.Collect(context.Background()) + if err != nil { + t.Fatalf("Collect: %v", err) + } + if r.GuestNet != nil { + t.Fatalf("no reporter wired → guest_net must be omitted, got %+v", r.GuestNet) + } + // And it must be absent from the WIRE, not merely nil in Go — an always-present empty stanza + // would make "watchdog not wired" indistinguishable from "watchdog found nothing". + b, _ := json.Marshal(r) + var m map[string]any + if err := json.Unmarshal(b, &m); err != nil { + t.Fatal(err) + } + if _, ok := m["guest_net"]; ok { + t.Fatalf("guest_net key present on the wire with no reporter wired: %s", b) + } +} + +func TestCollect_GuestNetPopulatedWhenWired(t *testing.T) { + px := &fakePx{node: "n", ns: newTestNodeStatus()} + c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger()) + c.SetGuestNetReporter(fakeGuestNet{st: &GuestNetStatus{ + CheckedAt: "2026-07-21T10:00:00Z", + Guests: []GuestNetGuest{{ + VMID: 9201, State: "healthy", Mode: "dhcp", IP: "192.168.0.104", + HasRoute: true, DHClientAlive: true, CheckedAt: "2026-07-21T10:00:00Z", + }}, + }}) + r, err := c.Collect(context.Background()) + if err != nil { + t.Fatalf("Collect: %v", err) + } + if r.GuestNet == nil || len(r.GuestNet.Guests) != 1 { + t.Fatalf("guest_net stanza missing from a collected report: %+v", r.GuestNet) + } + + // The wire keys are the contract the hub will read; pin the ones an operator diagnoses with. + b, _ := json.Marshal(r) + var m map[string]any + if err := json.Unmarshal(b, &m); err != nil { + t.Fatal(err) + } + gn, ok := m["guest_net"].(map[string]any) + if !ok { + t.Fatalf("guest_net missing or wrong shape on the wire: %s", b) + } + guests, ok := gn["guests"].([]any) + if !ok || len(guests) != 1 { + t.Fatalf("guest_net.guests wrong on the wire: %v", gn) + } + g := guests[0].(map[string]any) + for _, key := range []string{"vmid", "state", "mode", "ip", "has_route", "dhclient_alive"} { + if _, ok := g[key]; !ok { + t.Fatalf("guest_net.guests[0] is missing the %q key: %v", key, g) + } + } + if g["dhclient_alive"] != true { + t.Fatalf("dhclient_alive must survive the round trip: %v", g) + } +} diff --git a/internal/hub/report.go b/internal/hub/report.go index c46510a..6ed135a 100644 --- a/internal/hub/report.go +++ b/internal/hub/report.go @@ -85,6 +85,17 @@ type HostReport struct { // Carries NO secret. PBSDR *PBSDRStatus `json:"pbs_dr,omitempty"` + // GuestNet is the per-guest network-watchdog stanza (R-54). Present only when the guestnet + // watchdog is wired. Same additive/opaque contract as PBSDR and Wireguard above — no + // hub-schema change, absent when the reporter is not wired. Carries NO secret: addresses, + // route/liveness booleans, heal timestamps and counters only. + // + // NAMING NOTE (deliberate deviation from TASK-D, which called this block `WireGuestNet`): + // in this repo `Wire*` types are the DOWN direction (WireDesiredState/WirePBSDR — what the + // hub sends the agent), while UP-direction report stanzas are `*Status`. A `WireGuestNet` + // on HostReport would have been the only report block named against that convention. + GuestNet *GuestNetStatus `json:"guest_net,omitempty"` + // LogTail is the agent's on-demand debug-ring tail (v0.83.0 observability) — the agent // mirror of the controller's report log_tails channel. Present ONLY on the heartbeat // right after the control envelope requested it (log_tail_requested); consume-once on @@ -109,6 +120,33 @@ type HostReport struct { // "verify_failed" (fingerprint/reachability pre-consume check failing — retrying, NOTHING // consumed), "consumed_failed" (LOUD: secret burned, apply failed, no auto-retry — operator // re-issue required), "disabled" (descriptor enabled:false). Carries no secret. +// GuestNetStatus is the R-54 guest-network watchdog stanza. `guests` carries one entry per owned +// RUNNING guest that has been probed at least once; an empty list with a fresh `checked_at` means +// the watchdog ran and found nothing to report, which is deliberately distinguishable from the +// stanza being absent (= the watchdog is not wired at all). +type GuestNetStatus struct { + CheckedAt string `json:"checked_at"` // RFC3339, the sweep this snapshot came from + Guests []GuestNetGuest `json:"guests,omitempty"` +} + +// GuestNetGuest mirrors guestnet.GuestReport on the wire. The two structs are deliberately separate: +// internal/hub owns the wire contract and imports no feature package (the consumer-side seam rule). +type GuestNetGuest struct { + VMID int `json:"vmid"` + State string `json:"state"` // healthy | unhealthy | static_fault | unknown + Mode string `json:"mode"` // dhcp | static | unknown + IP string `json:"ip,omitempty"` + HasRoute bool `json:"has_route"` + DHClientAlive bool `json:"dhclient_alive"` + CheckedAt string `json:"checked_at,omitempty"` + Healed bool `json:"healed,omitempty"` + HealSucceeded bool `json:"heal_succeeded,omitempty"` + LastHealAt string `json:"last_heal_at,omitempty"` + HealsLastHour int `json:"heals_last_hour,omitempty"` + Damped bool `json:"damped,omitempty"` + Message string `json:"message,omitempty"` +} + type PBSDRStatus struct { State string `json:"state"` StorageID string `json:"storage_id,omitempty"`