diff --git a/REPORT.md b/REPORT.md index 7e7865f..1bb2e34 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,8 +1,8 @@ # REPORT — TASK-D Part 3: the guest-network watchdog (R-54) · felhom-agent v0.91.2 → **v0.92.1** **Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `08b55a1` (clean, == `origin/main`). -**Deployed and running on felhom-pve.** The healthy-cycle leg is **PROVEN LIVE**; the heal leg -(STOP-2, a deliberate replay of the 2026-07-20 incident) is operator-present and has **not** run. +**Deployed, running on felhom-pve, and STOP-2 RAN — the incident was replayed and the watchdog +prevented it (§6b).** Every claim below was observed. --- @@ -174,6 +174,69 @@ so STOP-2's heal chain is visible in journald. **Revert it to `info` after STOP- --- +## 6b. STOP-2 — the incident replay (operator-present, 2026-07-21) + +The 2026-07-20 kill, repeated deliberately. **The `/proc//cgroup` check the incident produced +was applied before the kill** — the script refuses unless the pid's cgroup is guest 9201's, which is +the rule that would have prevented the original outage: + +``` +GATE OK — pid 336708 belongs to guest 9201 (0::/lxc/9201/ns/.lxc) +KILL at 2026-07-21T10:43:18Z / 12:43:18 CEST +after kill: no dhclient running; address 192.168.0.104/24 STILL PRESENT (valid_lft 4998s); default route STILL PRESENT +``` + +That second line is the whole point: the box looked perfectly healthy, with ~83 minutes of lease +left before any symptom would appear. + +| Time (CEST) | Event | +|---|---| +| 12:43:15 | `DEBUG guest network healthy … dhclient_alive=true` — last good cycle | +| 12:43:18 | **kill -9** | +| **12:44:15** | **detected in 57 s, on process liveness alone** — `unhealthy (first bad probe — not acting yet) bad_probes=1 required=2`, detail *"dhclient is not running — the lease will not be renewed (the 2026-07-20 failure mode; address still present, renewal already dead)"* | +| 12:45:15 | second consecutive bad probe → `healing`, then `running heal command … cmd="pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0"` | +| **12:45:18** | **`guest network healed` ip=192.168.0.104 has_route=true dhclient_alive=true heals_last_hour=1** | + +**Healed 120 seconds after the kill — roughly 80 minutes before the outage would have begun.** The +strongest evidence is therefore what did *not* happen, verified from inside the guest: + +| Check | Result | +|---|---| +| `cloudflared` | **`Up 29 hours`** — the tunnel never dropped, never reconnected | +| DNS `gitea.dooplex.hu` | OK | +| hub | **302 in 0.16 s** | +| `https://felhom.demo-felhom.eu/` | **302 in 0.25 s** | +| new dhclient | pid 2043719, `cgroup=0::/lxc/9201/ns/.lxc`, argv byte-identical to the original | + +On 2026-07-20 every one of those was dead for 1h15m. This time the outage was **prevented**, not +detected. + +**The negative legs, both clean.** Exactly ONE heal (`grep -c` = 1), zero re-heals, zero +guestnet WARN/ERROR in the quiet window that followed, and the healthy Debug line kept appearing +each cycle — so "no alarms" stayed distinguishable from "not probing". + +**An unplanned damper proof, against a REAL transient.** STOP-1's guest reboot at 12:53 caught the +guest mid-boot: + +``` +12:53:16 INFO guest network unhealthy (first bad probe — not acting yet) detail="no IPv4 address on eth0" bad_probes=1 +12:54:15 DEBUG guest network healthy … +12:54:15 INFO guest network recovered previous_state=unhealthy +``` + +One bad probe, **no action**, then recovery. The two-consecutive-probes rule and the boot-race guard +did exactly what they exist for — a booting guest was not injected with a dhclient — and this was a +genuine transient, not a scripted one. + +## 6c. What could NOT be verified here + +The `guest_net` stanza **on the wire**. `--selftest=hub` builds its own one-shot collector and never +wires the guestnet reporter, so it would print a misleading absence — using it as evidence would be +worse than having none. (That divergence between the selftest collector and the daemon's is itself +worth a small fix; the same comment already exists in the code for the pbs reporter.) The stanza is +proven through the production `Collect` path by test, and `SetGuestNetReporter` is proven called by +the AST wiring test, but seeing it in a real hub report needs the operator UI. + ## 7. Deliverables - `c0966d7` — v0.92.0: the watchdog, the report block, config, tests. @@ -184,8 +247,7 @@ so STOP-2's heal chain is visible in journald. **Revert it to `info` after STOP- ## 8. Operator actions outstanding -1. **STOP-2** — the incident replay (kill 9201's dhclient, watch it heal within ≤2 cycles). It is - the deliberate reproduction of a real outage, so it is operator-present by construction. +1. ~~STOP-2~~ — **DONE 2026-07-21, passed** (§6b). 2. **STOP-3** — Day-0 manifest Agent → **0.92.1** / `7424bc1c3c533eff9157e15a18d4635c624931f5a479a48126de77a94e6a3d4d` (wrapper sha unchanged — `configs/felhom-pbs-apply` was not touched). MinAgent is the operator's call; 0.92.1 is safe once felhom-pve has converged, which it has.