admin/felhom-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(report): STOP-2 evidence — the incident replayed and PREVENTED (tunnel never dropped)

Browse Source
This commit is contained in:
authentik Default Admin
2026-07-21 12:58:12 +02:00
parent 71e1777369
commit 3f58121b98
1 changed files with 66 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+66 -4
View File
@@ -1,8 +1,8 @@
# REPORT — TASK-D Part 3: the guest-network watchdog (R-54) · felhom-agent v0.91.2 → **v0.92.1**
**Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `08b55a1` (clean, == `origin/main`).
**Deployed and running on felhom-pve.** The healthy-cycle leg is **PROVEN LIVE**; the heal leg
(STOP-2, a deliberate replay of the 2026-07-20 incident) is operator-present and has **not** run.
**Deployed, running on felhom-pve, and STOP-2 RAN — the incident was replayed and the watchdog
prevented it (§6b).** Every claim below was observed.
---
@@ -174,6 +174,69 @@ so STOP-2's heal chain is visible in journald. **Revert it to `info` after STOP-
---
## 6b. STOP-2 — the incident replay (operator-present, 2026-07-21)
The 2026-07-20 kill, repeated deliberately. **The `/proc/<pid>/cgroup` check the incident produced
was applied before the kill** — the script refuses unless the pid's cgroup is guest 9201's, which is
the rule that would have prevented the original outage:
```
GATE OK — pid 336708 belongs to guest 9201 (0::/lxc/9201/ns/.lxc)
KILL at 2026-07-21T10:43:18Z / 12:43:18 CEST
after kill: no dhclient running; address 192.168.0.104/24 STILL PRESENT (valid_lft 4998s); default route STILL PRESENT
```
That second line is the whole point: the box looked perfectly healthy, with ~83 minutes of lease
left before any symptom would appear.
| Time (CEST) | Event |
|---|---|
| 12:43:15 | `DEBUG guest network healthy … dhclient_alive=true` — last good cycle |
| 12:43:18 | **kill -9** |
| **12:44:15** | **detected in 57 s, on process liveness alone** — `unhealthy (first bad probe — not acting yet) bad_probes=1 required=2`, detail *"dhclient is not running — the lease will not be renewed (the 2026-07-20 failure mode; address still present, renewal already dead)"* |
| 12:45:15 | second consecutive bad probe → `healing`, then `running heal command … cmd="pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0"` |
| **12:45:18** | **`guest network healed` ip=192.168.0.104 has_route=true dhclient_alive=true heals_last_hour=1** |
**Healed 120 seconds after the kill — roughly 80 minutes before the outage would have begun.** The
strongest evidence is therefore what did *not* happen, verified from inside the guest:
| Check | Result |
|---|---|
| `cloudflared` | **`Up 29 hours`** — the tunnel never dropped, never reconnected |
| DNS `gitea.dooplex.hu` | OK |
| hub | **302 in 0.16 s** |
| `https://felhom.demo-felhom.eu/` | **302 in 0.25 s** |
| new dhclient | pid 2043719, `cgroup=0::/lxc/9201/ns/.lxc`, argv byte-identical to the original |
On 2026-07-20 every one of those was dead for 1h15m. This time the outage was **prevented**, not
detected.
**The negative legs, both clean.** Exactly ONE heal (`grep -c` = 1), zero re-heals, zero
guestnet WARN/ERROR in the quiet window that followed, and the healthy Debug line kept appearing
each cycle — so "no alarms" stayed distinguishable from "not probing".
**An unplanned damper proof, against a REAL transient.** STOP-1's guest reboot at 12:53 caught the
guest mid-boot:
```
12:53:16 INFO guest network unhealthy (first bad probe — not acting yet) detail="no IPv4 address on eth0" bad_probes=1
12:54:15 DEBUG guest network healthy …
12:54:15 INFO guest network recovered previous_state=unhealthy
```
One bad probe, **no action**, then recovery. The two-consecutive-probes rule and the boot-race guard
did exactly what they exist for — a booting guest was not injected with a dhclient — and this was a
genuine transient, not a scripted one.
## 6c. What could NOT be verified here
The `guest_net` stanza **on the wire**. `--selftest=hub` builds its own one-shot collector and never
wires the guestnet reporter, so it would print a misleading absence — using it as evidence would be
worse than having none. (That divergence between the selftest collector and the daemon's is itself
worth a small fix; the same comment already exists in the code for the pbs reporter.) The stanza is
proven through the production `Collect` path by test, and `SetGuestNetReporter` is proven called by
the AST wiring test, but seeing it in a real hub report needs the operator UI.
## 7. Deliverables
- `c0966d7` — v0.92.0: the watchdog, the report block, config, tests.
@@ -184,8 +247,7 @@ so STOP-2's heal chain is visible in journald. **Revert it to `info` after STOP-
## 8. Operator actions outstanding
1. **STOP-2** — the incident replay (kill 9201's dhclient, watch it heal within ≤2 cycles). It is
the deliberate reproduction of a real outage, so it is operator-present by construction.
1. ~~STOP-2~~ — **DONE 2026-07-21, passed** (§6b).
2. **STOP-3** — Day-0 manifest Agent → **0.92.1** / `7424bc1c3c533eff9157e15a18d4635c624931f5a479a48126de77a94e6a3d4d`
(wrapper sha unchanged — `configs/felhom-pbs-apply` was not touched). MinAgent is the operator's
call; 0.92.1 is safe once felhom-pve has converged, which it has.