felhom-agent/REPORT.md

REPORT — TASK-D Part 3: the guest-network watchdog (R-54) · felhom-agent v0.91.2 → v0.92.1

Date: 2026-07-21 · Trunk, pushed to main. Baseline: 08b55a1 (clean, == origin/main). Deployed, running on felhom-pve, and STOP-2 RAN — the incident was replayed and the watchdog prevented it (§6b). Every claim below was observed.

---

1. What this closes

INCIDENT-guest-dhclient-killed-2026-07-20.md §5, "OPEN RISK": the guest's DHCP client is unsupervised. ifupdown starts it once at boot and nothing restarts it. When it was killed on 2026-07-20 the guest kept working for another ~80 minutes on its unexpired lease; only at expiry did the address and default route vanish, taking the Cloudflare tunnel, hub reports, catalog sync and the controller→agent channel with them — a 1h15m outage in which every observable signal said healthy for the first 80 minutes.

So the design consequence is the whole feature: liveness of the DHCP client is itself a probe. The watchdog flags a DHCP guest unhealthy on pgrep -x dhclient alone, while the address and route are still perfectly present. Waiting for the IP to disappear is waiting out exactly that silent window — and the red-proof reproduces it (§5).

Host tier is not a preference: a guest with no default route cannot repair its own default route.

---

2. Shipped

internal/guestnet (probe.go / watchdog.go / report.go), built on the wg-tunnel + storage watchdog loop shape, started with go wd.Watch(ctx) like selfheal.

• Four fixed-shape pct exec probes, all constant argv + the numeric vmid: address, default route, /etc/network/interfaces mode, pgrep -x dhclient. No shell anywhere; no guest-supplied data is ever interpolated into a command.
• Heal = the incident's restored invocation, verbatim, logged at INFO before it runs: pct exec <vmid> -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 A test pins that argv element by element.
• Dampers (this runs a privileged command inside a customer's container, so it is built to under-act): two CONSECUTIVE bad probes before any heal, ≥10 min between heals per guest, ≤3 heals/hour, and observe-only while the guest — or the agent itself — has been up under 3 minutes.
• Refuses to act on: a static guest (dhclient must never fight a static config; a static guest missing its address is reported loudly and left to R-50), an unknown interface mode, a guest it cannot probe, and an ownership-unproven guest list. The guest source is the pool-verified ListLXC ∩ felhom-pool (audit A1) — never a bare ListLXC, which under a broad token would run dhclient inside a co-tenant's container.
• A failed PROBE is never a dead client. pgrep exits 1 with EMPTY stderr on no-match; anything on stderr means the probe itself failed → unknown. Without that rule a missing pgrep would heal forever.
• Healthy cycles log a Debug line. v0.91.2's lesson, one day old: if the quiet path is silent, "no alarms" and "never probed" are the same evidence.
• Not in the errc fan-out — a watchdog over customer guests must never be able to terminate the agent. A test asserts that, because joining the fan-out would also make the shutdown drain bound off by one.
• Report block: GuestNetStatus on HostReport (guest_net, omitempty), additive and stored opaquely hub-side like pbs_dr / wireguard. No hub code was touched.

Two deliberate deviations from TASK-D, both stated up front:

1. GuestNetStatus, not WireGuestNet. In this repo Wire* is the DOWN direction (WireDesiredState / WirePBSDR — what the hub sends the agent); UP-direction report stanzas are *Status. WireGuestNet on HostReport would have been the only report block named against the convention.
2. A sudoers change was required — see §4. The brief said none was needed.

Config guest_net is this repo's first default-ON gate. Every other gate defaults to false because those features reach outward (an offsite endpoint, an OOB tunnel) and enrolling a box by an update would be wrong. This one looks only inward at guests the agent already owns, and the failure it prevents exists on every box today. A watchdog that must be remembered per box is a watchdog that is missing on the box that needed it. Opt-out is explicit: "guest_net": {"disable": true}.

---

3. Phase-0 probes

P3 (watchdog ground truth) — DONE 2026-07-21, live from guest 9201. These exact bytes are the parser fixtures, including the literal backslash ip -o emits and the trailing space on the route:

ip -4 -o addr show dev eth0 → 2: eth0    inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0\       valid_lft 4916sec preferred_lft 4916sec
ip route show default       → default via 192.168.0.1 dev eth0
pgrep -x dhclient           → 235839                       (rc=0; rc=1 + EMPTY stderr when absent)
ps -o args= -C dhclient     → dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
/etc/network/interfaces     → iface eth0 inet dhcp
pct exec <bad vmid>         → rc=2, stderr "Configuration file 'nodes/demo-felhom/lxc/9999.conf' does not exist"

The live ps line confirms the running client's argv is byte-identical to the incident's restored invocation — i.e. the heal reproduces the guest's own boot-time command, not an approximation. A docker-bridge-only route table is also pinned as a negative (172.17.0.0/16 dev docker0 … must never read as a default route — those were the exact leftovers in the incident).

P4 (config surface + report pattern + loop precedent) — DONE by reading the tree; the wg loop's interval/damping/logging shape and the pbs_dr stanza's collector-seam pattern are what this copies.

---

4. The live finding: three of four probes had no sudoers grant

The agent runs non-root; Privileged.Mode=sudo fails closed with no prompt. The first sweep after deploying v0.92.0 logged:

level=WARN msg="guestnet: guest network not actionable — reporting only" vmid=9201
  state=unknown mode=unknown has_ip=true has_route=false
  detail="dhclient liveness probe failed: sudo: a password is required"

The watchdog behaved exactly as designed — it reported unknown and healed nothing rather than acting blind — but it was blind. The existing allowlist granted only lanresolver's address read (pct exec [0-9]* -- ip -4 -o addr show dev eth0), which is why has_ip=true while route, mode and liveness all failed.

Fixed in v0.92.1: a FELHOM_GUESTNET alias with four fixed vectors (route, interfaces, pgrep, heal). Every argument after the numeric vmid is a literal, so nothing the guest or the hub says can widen the grant. The address read is not duplicated — it stays FELHOM_DNSMASQ's; one command, one grant.

Plus four guestnet-* capability rows, so a host that has not taken the new sudoers file is VISIBLE as degraded instead of silently watchdog-less. Deliberately non-critical: a missing grant must not page an operator for every box on rollout day (the R-50b amber-fleet lesson).

v0.92.0 is superseded, not overwritten — do not vouch it. It was published before this was found, so its binary lacks the capability rows and its release lacks the sudoers file. A published version stays immutable (the v0.91.0 → v0.91.1 precedent).

---

5. Tests and red-proofs

Green gate: go build ./... && go vet ./... && go test ./... — all packages ok except the known flake TestGenerateRecoveryCode_EntropyAndFormat (internal/escrow), which fails when the wordlist yields a hyphenated word (drop-down → 11 tokens instead of 10). Confirmed pre-existing: internal/escrow has not been touched since v0.88.0 and this task changes nothing there; observed 3/8 runs, consistent with the documented ~1/5.

New: internal/guestnet/watchdog_test.go (16 cases), internal/hub/collect_guestnet_test.go (2), cmd/felhom-agent/guestnet_wiring_test.go (2).

#	Red-proof	Mutation	Result
E	detect on process liveness, not address presence	classify's DHCP arm reverted to IP-presence-only	FAIL ×5. The decisive one: classify = "healthy", want "unhealthy" for the July-20 fixture, detail="address, default route and dhclient all present", and heal ran 0 times. That is the 80-minute silent window, reproduced exactly. Restored, green.
W	the watchdog must be wired	SetGuestNetReporter and go gnWatchdog.Watch(ctx) both commented out	FAIL with both reasons named — "the guest_net stanza would never reach the hub (the exact v0.91.0 inert-seam defect)" and "it would be constructed, reported on, and never probe anything". Restored, green.

Every damper is asserted as an exact count, and the load-bearing assertions are the negatives — a static guest, an unprobeable guest, a boot-race guest (young guest AND young agent), a failed probe tool and an ownership-unproven guest list must each record zero heal calls. The ceilings are driven by an injected clock over a scripted 10 hours of permanent failure: ≤30 heals total, and never a second heal inside the 10-minute cool-off.

Seam discipline (§9 rule 6) — three production-path tests: the guest_net stanza is asserted through the real Collect (and asserted ABSENT from the wire when no reporter is wired, so "not wired" and "found nothing" can never look identical); and the main.go wiring is an AST walk for the construction, the reporter call and the started goroutine. The AST form is deliberate — a strings.Contains version of the twin test in felhom-controller passed its own red-proof, because a commented-out call still contains the string.

---

6. Live validation (method: journald + capability self-check on felhom-pve)

Step	Evidence
Publish 0.92.0	AGENT_SHA256=b1302790d412d22e969936ff52e3ee33e3edc111b8426364a21cdb1c5127ca6a, round-trip GET verified — superseded, do not vouch
Publish 0.92.1	AGENT_SHA256=7424bc1c3c533eff9157e15a18d4635c624931f5a479a48126de77a94e6a3d4d, round-trip GET verified
Deploy	visudo -c parsed OK → sudoers installed 0440 root:root (backup /root/felhom-agent.sudoers.bak-preR54) + binary installed (backup felhom-agent.bak-0.91.2-preR54) → felhom-agent --version = 0.92.1, service active
Capability self-check	ok=68 total=68 degraded=0 inactive=0 (was 64/64 before the four guestnet-* rows) — the sudoers grant is proven from the agent's own side, not assumed
Watchdog start	INFO guestnet: watchdog starting interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
Healthy cycle	level=DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true (12:34:15 CEST)
Default-ON proven	/etc/felhom-agent/agent.json has no guest_net key at all — the watchdog runs on defaults, which is the whole point of the inverted gate

Note for the operator: log_level on felhom-pve was temporarily raised to debug to capture that Debug line (backup at /root/agent.json.bak-debuglevel). It is still debug, deliberately, so STOP-2's heal chain is visible in journald. Revert it to info after STOP-2.

---

6b. STOP-2 — the incident replay (operator-present, 2026-07-21)

The 2026-07-20 kill, repeated deliberately. The /proc/<pid>/cgroup check the incident produced was applied before the kill — the script refuses unless the pid's cgroup is guest 9201's, which is the rule that would have prevented the original outage:

GATE OK — pid 336708 belongs to guest 9201 (0::/lxc/9201/ns/.lxc)
KILL at 2026-07-21T10:43:18Z / 12:43:18 CEST
after kill: no dhclient running; address 192.168.0.104/24 STILL PRESENT (valid_lft 4998s); default route STILL PRESENT

That second line is the whole point: the box looked perfectly healthy, with ~83 minutes of lease left before any symptom would appear.

Time (CEST)	Event
12:43:15	DEBUG guest network healthy … dhclient_alive=true — last good cycle
12:43:18	kill -9
12:44:15	detected in 57 s, on process liveness alone — unhealthy (first bad probe — not acting yet) bad_probes=1 required=2, detail "dhclient is not running — the lease will not be renewed (the 2026-07-20 failure mode; address still present, renewal already dead)"
12:45:15	second consecutive bad probe → healing, then running heal command … cmd="pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0"
12:45:18	guest network healed ip=192.168.0.104 has_route=true dhclient_alive=true heals_last_hour=1

Healed 120 seconds after the kill — roughly 80 minutes before the outage would have begun. The strongest evidence is therefore what did not happen, verified from inside the guest:

Check	Result
cloudflared	Up 29 hours — the tunnel never dropped, never reconnected
DNS gitea.dooplex.hu	OK
hub	302 in 0.16 s
https://felhom.demo-felhom.eu/	302 in 0.25 s
new dhclient	pid 2043719, cgroup=0::/lxc/9201/ns/.lxc, argv byte-identical to the original

On 2026-07-20 every one of those was dead for 1h15m. This time the outage was prevented, not detected.

The negative leg — 30 healthy minutes, run in full. From the heal at 12:45:18 to 13:16:15: 30 healthy Debug cycles (exactly one per minute — the cadence is precise), 0 heals, 0 WARN, 0 ERROR. So the damper is not a mute and the quiet path is not silence: "no alarms" stayed continuously distinguishable from "not probing", which is the v0.91.2 lesson holding in production.

An unplanned damper proof, against a REAL transient. STOP-1's guest reboot at 12:53 caught the guest mid-boot:

12:53:16 INFO  guest network unhealthy (first bad probe — not acting yet) detail="no IPv4 address on eth0" bad_probes=1
12:54:15 DEBUG guest network healthy …
12:54:15 INFO  guest network recovered previous_state=unhealthy

One bad probe, no action, then recovery. The two-consecutive-probes rule and the boot-race guard did exactly what they exist for — a booting guest was not injected with a dhclient — and this was a genuine transient, not a scripted one.

6c. The stanza ON THE WIRE — confirmed hub-side after STOP-3

Initially unverifiable here: --selftest=hub builds its own one-shot collector and never wires the guestnet reporter, so it would have printed a misleading absence — worse evidence than none. (That divergence between the selftest collector and the daemon's is worth a small fix; the same caveat is already commented in the code for the pbs reporter.)

Closed instead by a read-only query of the hub's own store (kubectl cp of /data/hub.db, opened mode=ro, copy deleted afterwards). The guest_net stanza is present, complete, and round-trips every field:

"guest_net": { "checked_at": "2026-07-21T11:32:10Z", "guests": [
  { "vmid": 9201, "state": "healthy", "mode": "dhcp", "ip": "192.168.0.104",
    "has_route": true, "dhclient_alive": true, "checked_at": "2026-07-21T11:31:07Z",
    "message": "address, default route and dhclient all present" } ] }

And the report history tells the whole story of this session from the hub's side:

received (UTC)	agent	state	evidence
10:30:27	0.92.0	unknown	the sudoers-blind window — reported honestly as unknown, never as a false healthy and never as a false dead. The fail-safe is visible fleet-side, and this is independent confirmation that 0.92.0 was genuinely blind (so superseding it was right)
10:48:12	0.92.1	healthy	last_heal_at=2026-07-21T10:45:12Z, heals_last_hour=1 — the STOP-2 heal, surfaced to the hub
10:49:14 / 11:04:14	0.92.1	healthy	same heal stamp, counter still 1
11:32:10	0.92.1	healthy	heal history absent — see the limitation below

A limitation this surfaced, worth stating plainly: the damping state is in-memory only. The counters vanished from the 11:32 report because the agent was restarted at 11:17Z (the log_level revert), which resets heals/lastHealAt. So the "≥10 min apart, ≤3 per hour" ceilings hold within one agent lifetime, not across restarts. In practice the exposure is small — a restart also re-arms the 3-minute settle window and the two-consecutive-bad-probes rule — but a crash-looping agent could heal more often than the ceiling implies. Not worth persisting state for today; worth knowing before anyone quotes the ceiling as a hard guarantee.

7. Deliverables

• c0966d7 — v0.92.0: the watchdog, the report block, config, tests.
• 0e8fd81 — the sudoers grant + capability rows (the live finding).
• 98adb72 — v0.92.1: supersede + version bump.
• Published: felhom-agent 0.92.1 / 7424bc1c3c533eff… (0.92.0 superseded).
• Live on felhom-pve: binary 0.92.1 + the new sudoers file.

8. Operator actions outstanding

1. STOP-2 — DONE 2026-07-21, passed (§6b).
2. STOP-3 — DONE 2026-07-21. Manifest Agent → 0.92.1, sha matches the published artifact byte for byte; MinAgent → 0.92.1; PBS wrapper sha unchanged (104db0a4…, correct — the wrapper was not touched); controller floor → 0.156.0. The host page shows all four guestnet-* capability rows ok, which is the fleet-visible proof of the sudoers grant.
3. Revert log_level to info — DONE (13:17 CEST, after the quiet window closed; agent restarted clean, caps 68/68 ok, degraded=0, watchdog back up). The temporary raise is recorded here only so the journald volume change is explainable; /root/agent.json.bak-debuglevel remains as the pre-change copy.