docs(report): STOP-2 evidence — the incident replayed and PREVENTED (tunnel never dropped)

REPORT.md

@@ -1,8 +1,8 @@
 # REPORT — TASK-D Part 3: the guest-network watchdog (R-54) · felhom-agent v0.91.2 → **v0.92.1**
 
 **Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `08b55a1` (clean, == `origin/main`).
-**Deployed and running on felhom-pve.** The healthy-cycle leg is **PROVEN LIVE**; the heal leg
+**Deployed, running on felhom-pve, and STOP-2 RAN — the incident was replayed and the watchdog
-(STOP-2, a deliberate replay of the 2026-07-20 incident) is operator-present and has **not** run.
+prevented it (§6b).** Every claim below was observed.
 
 ---
 
@@ -174,6 +174,69 @@ so STOP-2's heal chain is visible in journald. **Revert it to `info` after STOP-
 
 ---
 
+## 6b. STOP-2 — the incident replay (operator-present, 2026-07-21)
+
+The 2026-07-20 kill, repeated deliberately. **The `/proc/<pid>/cgroup` check the incident produced
+was applied before the kill** — the script refuses unless the pid's cgroup is guest 9201's, which is
+the rule that would have prevented the original outage:
+
+```
+GATE OK — pid 336708 belongs to guest 9201 (0::/lxc/9201/ns/.lxc)
+KILL at 2026-07-21T10:43:18Z / 12:43:18 CEST
+after kill: no dhclient running; address 192.168.0.104/24 STILL PRESENT (valid_lft 4998s); default route STILL PRESENT
+```
+
+That second line is the whole point: the box looked perfectly healthy, with ~83 minutes of lease
+left before any symptom would appear.
+
+| Time (CEST) | Event |
+|---|---|
+| 12:43:15 | `DEBUG guest network healthy … dhclient_alive=true` — last good cycle |
+| 12:43:18 | **kill -9** |
+| **12:44:15** | **detected in 57 s, on process liveness alone** — `unhealthy (first bad probe — not acting yet) bad_probes=1 required=2`, detail *"dhclient is not running — the lease will not be renewed (the 2026-07-20 failure mode; address still present, renewal already dead)"* |
+| 12:45:15 | second consecutive bad probe → `healing`, then `running heal command … cmd="pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0"` |
+| **12:45:18** | **`guest network healed` ip=192.168.0.104 has_route=true dhclient_alive=true heals_last_hour=1** |
+
+**Healed 120 seconds after the kill — roughly 80 minutes before the outage would have begun.** The
+strongest evidence is therefore what did *not* happen, verified from inside the guest:
+
+| Check | Result |
+|---|---|
+| `cloudflared` | **`Up 29 hours`** — the tunnel never dropped, never reconnected |
+| DNS `gitea.dooplex.hu` | OK |
+| hub | **302 in 0.16 s** |
+| `https://felhom.demo-felhom.eu/` | **302 in 0.25 s** |
+| new dhclient | pid 2043719, `cgroup=0::/lxc/9201/ns/.lxc`, argv byte-identical to the original |
+
+On 2026-07-20 every one of those was dead for 1h15m. This time the outage was **prevented**, not
+detected.
+
+**The negative legs, both clean.** Exactly ONE heal (`grep -c` = 1), zero re-heals, zero
+guestnet WARN/ERROR in the quiet window that followed, and the healthy Debug line kept appearing
+each cycle — so "no alarms" stayed distinguishable from "not probing".
+
+**An unplanned damper proof, against a REAL transient.** STOP-1's guest reboot at 12:53 caught the
+guest mid-boot:
+
+```
+12:53:16 INFO  guest network unhealthy (first bad probe — not acting yet) detail="no IPv4 address on eth0" bad_probes=1
+12:54:15 DEBUG guest network healthy …
+12:54:15 INFO  guest network recovered previous_state=unhealthy
+```
+
+One bad probe, **no action**, then recovery. The two-consecutive-probes rule and the boot-race guard
+did exactly what they exist for — a booting guest was not injected with a dhclient — and this was a
+genuine transient, not a scripted one.
+
+## 6c. What could NOT be verified here
+
+The `guest_net` stanza **on the wire**. `--selftest=hub` builds its own one-shot collector and never
+wires the guestnet reporter, so it would print a misleading absence — using it as evidence would be
+worse than having none. (That divergence between the selftest collector and the daemon's is itself
+worth a small fix; the same comment already exists in the code for the pbs reporter.) The stanza is
+proven through the production `Collect` path by test, and `SetGuestNetReporter` is proven called by
+the AST wiring test, but seeing it in a real hub report needs the operator UI.
+
 ## 7. Deliverables
 
 - `c0966d7` — v0.92.0: the watchdog, the report block, config, tests.
@@ -184,8 +247,7 @@ so STOP-2's heal chain is visible in journald. **Revert it to `info` after STOP-
 
 ## 8. Operator actions outstanding
 
-1. **STOP-2** — the incident replay (kill 9201's dhclient, watch it heal within ≤2 cycles). It is
+1. ~~STOP-2~~ — **DONE 2026-07-21, passed** (§6b).
-   the deliberate reproduction of a real outage, so it is operator-present by construction.
 2. **STOP-3** — Day-0 manifest Agent → **0.92.1** / `7424bc1c3c533eff9157e15a18d4635c624931f5a479a48126de77a94e6a3d4d`
    (wrapper sha unchanged — `configs/felhom-pbs-apply` was not touched). MinAgent is the operator's
    call; 0.92.1 is safe once felhom-pve has converged, which it has.