From 71e1777369c0a03c8e597a0e21001e8788750fff Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 21 Jul 2026 12:39:12 +0200 Subject: [PATCH] =?UTF-8?q?docs(report):=20TASK-D=20Part=203=20=E2=80=94?= =?UTF-8?q?=20R-54=20shipped,=20healthy=20cycle=20proven=20live;=20STOP-2?= =?UTF-8?q?=20pending?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- REPORT.md | 287 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 158 insertions(+), 129 deletions(-) diff --git a/REPORT.md b/REPORT.md index 0e3490f..7e7865f 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,163 +1,192 @@ -# REPORT — TASK-B: R-39 fleet fix + R-50b(a) · felhom-agent v0.90.0 → **v0.91.2** +# REPORT — TASK-D Part 3: the guest-network watchdog (R-54) · felhom-agent v0.91.2 → **v0.92.1** -**Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `8c55ac7` (clean, == `origin/main`). -Companion: hub **v0.68.0** (`felhom.eu`). +**Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `08b55a1` (clean, == `origin/main`). +**Deployed and running on felhom-pve.** The healthy-cycle leg is **PROVEN LIVE**; the heal leg +(STOP-2, a deliberate replay of the 2026-07-20 incident) is operator-present and has **not** run. -## 0. Status +--- -| Leg | Status | -|---|---| -| Part 1 — hub v0.68.0 | **SHIPPED + DEPLOYED** (GitOps sync, Synced/Healthy) | -| Part 2 — agent v0.91.2 | **SHIPPED + PUBLISHED + DEPLOYED** to felhom-pve | -| **STOP-1** — felhom-pve update | **DONE + VERIFIED** (§5) | -| **STOP-3** — manifest save | **DONE by the operator 2026-07-21 08:31:12Z** — agent 0.91.2 + sha + **wrapper sha `104db0a4…`** + MinAgent 0.91.2, all six fields persisted. The agent reports the same wrapper hash, so the R-50b drift gauge reads **ok**. | -| **STOP-2** — live re-issue | **DONE + PROVEN 2026-07-21 — the chain closed in 13 s.** `08:39:31Z` hub mint gen 0→1 + `secret_generation: 1` (token_id/fingerprint byte-identical) → `10:39:34` wrapper read → `10:39:38` `REJECTED … applied and DEAD` `previous_state=applied` → `10:39:45` **secret consumed (no short-circuit)** → `10:39:45` set-only reconcile → `10:39:47` `converged state=applied`. Marker hash moved to `afbb3b41…`; secret mtime `2026-07-18`→`2026-07-21 10:39:45`; new credential probes 200; reports trace `applied → auth_failed → applied`; **zero** self-heal escalations, one mint, one consume. First attempt hit the **offsite** re-issue by mistake (two distinct actions) — harmless to PBS-DR but it made the escrow stale, ceremony re-run. | +## 1. What this closes -## 1. Probes +`INCIDENT-guest-dhclient-killed-2026-07-20.md` §5, "OPEN RISK": **the guest's DHCP client is +unsupervised.** ifupdown starts it once at boot and nothing restarts it. When it was killed on +2026-07-20 the guest kept working for another **~80 minutes** on its unexpired lease; only at expiry +did the address and default route vanish, taking the Cloudflare tunnel, hub reports, catalog sync +and the controller→agent channel with them — a 1h15m outage in which every observable signal said +healthy for the first 80 minutes. -- **P1 — version stamping: PASS.** ldflags `-X main.version`; `publish-agent.sh` refuses to publish - when the binary's `--version` disagrees, so the stale `0.89.0` default cannot leak. -- **P2 — hub deploy path: PASS.** GitOps only: ArgoCD app `felhom`, auto-sync **off**, manifest bump - + deliberate sync. The `kubectl set image` line (in the old controller CLAUDE.md *and* printed by - `build.sh` itself) was **not** used. -- **P3 — felhom-pve before-state: CAPTURED** (§5 table). Notably the DR tier was **healthy**: a live - `GET /api2/json/version` with the stored credential returned **200**. This is a fix for a latent - defect, not a broken box. -- **P4 — fleet: ONE enrolled host** (`demo-felhom-8363b5`, agent 0.90.0). **Peti is not enrolled**, so - the MinAgent raise at STOP-3 cannot strand anyone. (Read from a temporary copy of the hub DB, which - carries live credentials — shredded immediately after each read.) +**So the design consequence is the whole feature: liveness of the DHCP client is itself a probe.** +The watchdog flags a DHCP guest unhealthy on `pgrep -x dhclient` alone, while the address and route +are still perfectly present. Waiting for the IP to disappear is waiting out exactly that silent +window — and the red-proof reproduces it (§5). -## 2. Two spec/reality mismatches, both resolved +Host tier is not a preference: a guest with no default route cannot repair its own default route. -**(a) The generation source does not exist as specified.** The brief said to return "the new row's id -(int64) … no schema change". `host_pbs_secrets` is `host_id PRIMARY KEY`, UPSERTed last-write-wins — -**there is no row id and no new row**, and `created_at` collides for two mints in one second. Added an -additive `generation` counter column using the repo's existing idempotent `ALTER TABLE` idiom. Same -semantics the brief wanted; different mechanism. +--- -**(b) A second load-bearing fact the brief did not flag — checked, not trusted.** `Apply` bails out -entirely if the storage *status probe* errors, and converges via `adopt` **without consuming the -secret** if the storage reads active. So the whole fix depended on what PVE returns for a 401'd PBS -storage. Read PVE's own source: `storage_info` wraps `activate_storage` and `$plugin->status` in -`eval{}`, warns, and leaves the pre-initialised `active => 0`. **A 401 therefore returns HTTP 200 with -`active: 0`, never an API error** — `StorageActive` gives `(false, nil)` and `Apply` falls through to -verify → consume → reconcile. Scenario A's chain is sound *by proof*. The re-arm test encodes this -fixture shape explicitly so the assumption is visible. +## 2. Shipped -## 3. What shipped (agent) +`internal/guestnet` (probe.go / watchdog.go / report.go), built on the wg-tunnel + storage watchdog +loop shape, started with `go wd.Watch(ctx)` like `selfheal`. -`WirePBSDR.SecretGeneration` (field-exact with the hub) · wrapper `read` verb + **exactly one** -sudoers line + a `pbsdr-read` capability row · `pbs.ProbeAuth` + `ErrUnauthorized` · the probe on the -**15-minute collect path** feeding a loud `auth_failed` · `readPBSSecret` (direct read, wrapper -fallback) · wrapper sha256 in the report (R-50b(a)). +- **Four fixed-shape `pct exec` probes**, all constant argv + the numeric vmid: address, default + route, `/etc/network/interfaces` mode, `pgrep -x dhclient`. No shell anywhere; no guest-supplied + data is ever interpolated into a command. +- **Heal = the incident's restored invocation, verbatim**, logged at INFO before it runs: + `pct exec -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0` + A test pins that argv element by element. +- **Dampers** (this runs a privileged command inside a customer's container, so it is built to + under-act): two CONSECUTIVE bad probes before any heal, ≥10 min between heals per guest, ≤3 + heals/hour, and observe-only while the guest — or the agent itself — has been up under 3 minutes. +- **Refuses to act** on: a static guest (dhclient must never fight a static config; a static guest + missing its address is reported loudly and left to **R-50**), an unknown interface mode, a guest + it cannot probe, and an ownership-unproven guest list. The guest source is the pool-verified + `ListLXC` ∩ felhom-pool (audit A1) — never a bare `ListLXC`, which under a broad token would run + dhclient inside a co-tenant's container. +- **A failed PROBE is never a dead client.** `pgrep` exits 1 with EMPTY stderr on no-match; anything + on stderr means the probe itself failed → `unknown`. Without that rule a missing `pgrep` would + heal forever. +- **Healthy cycles log a Debug line.** v0.91.2's lesson, one day old: if the quiet path is silent, + "no alarms" and "never probed" are the same evidence. +- **Not in the `errc` fan-out** — a watchdog over customer guests must never be able to terminate + the agent. A test asserts that, because joining the fan-out would also make the shutdown drain + bound off by one. +- **Report block:** `GuestNetStatus` on `HostReport` (`guest_net`, omitempty), additive and stored + opaquely hub-side like `pbs_dr` / `wireguard`. **No hub code was touched.** -**403 is deliberately NOT treated as unauthorized** — re-keying a too-narrow token would mint -credentials forever without fixing anything. **A transport error is UNKNOWN, never a rejection** — -otherwise every network blip burns a credential. +**Two deliberate deviations from TASK-D, both stated up front:** -## 4. Tests + the four red-proofs +1. **`GuestNetStatus`, not `WireGuestNet`.** In this repo `Wire*` is the DOWN direction + (`WireDesiredState` / `WirePBSDR` — what the hub sends the agent); UP-direction report stanzas + are `*Status`. `WireGuestNet` on `HostReport` would have been the only report block named against + the convention. +2. **A sudoers change was required** — see §4. The brief said none was needed. -Full gate green in both repos at every commit. All red-proofs run at the **assertion** level. +**Config `guest_net` is this repo's first default-ON gate.** Every other gate defaults to false +because those features reach outward (an offsite endpoint, an OOB tunnel) and enrolling a box by an +update would be wrong. This one looks only inward at guests the agent already owns, and the failure +it prevents exists on every box today. A watchdog that must be remembered per box is a watchdog +that is missing on the box that needed it. Opt-out is explicit: `"guest_net": {"disable": true}`. -| # | Red-proof | Result | -|---|---|---| -| A | Remove `SecretGeneration` from the agent struct | `consume calls=1, want 2` — the converged short-circuit fires; the July-18 defect exactly | -| B | Stop stamping the generation in `ReissuePBSDR` | descriptor **byte-identical** across a re-issue, block printed | -| D | Remove the id charset guard **and** the prefix assertion | the out-of-tree secret is **printed** (`rc=0 stdout="NOT-FOR-THE-AGENT-…"`) | -| E | Swallow the probe result in `NoteAuthResult` | `State:applied AuthFailed:false` — the July-18 shape | +--- -**A finding from red-proof D worth keeping:** deleting the charset guard *alone* does **not** open a -hole — `readlink -f` plus the prefix assertion still catch it, so the naive single-guard red-proof -**passed vacuously**. The layering is real; the isolating proof had to remove both. A red-proof that -passes is not automatically reassuring. +## 3. Phase-0 probes -The wrapper is tested as the artifact it is — executed under real `bash`. The two cases needing a -writable allowlisted dir run a **copy with only the prefix constant relocated** (and re-assert that -traversal is still refused in the copy), rather than `t.Skip` — a skipped test proves nothing. - -## 5. STOP-1 — felhom-pve, before → after - -| | before (P3) | after | -|---|---|---| -| agent | 0.90.0 · `18c48272…` | **0.91.2** · `34d309be…` | -| wrapper | `2888f2ea…` (the in-place July-18 hotfix) | `104db0a4…` (repo) | -| sudoers | `174125ef…` | new, +1 line | -| capabilities | 63 | **64/64 ok, 0 degraded, 0 inactive** | -| guest 9201 | running | running (untouched) | -| `wg-felhom` | handshake `…673` | fresh `…793` | - -Gates run before install: `bash -n` on the wrapper, `visudo -cf` on the staged sudoers, `visudo -c` -on the whole tree **after** install, and the staged binary's own `--version`. - -**Rollback set retained on the host:** -`/usr/local/sbin/felhom-pbs-apply.bak-20260721-101635` (and the older `.bak-20260718-preR39`), -`/usr/local/bin/felhom-agent.bak-2026072{1-101635,1-101905,…}`, -`/root/felhom-agent.sudoers.bak-20260721-101635`. - -**The two live proofs that matter:** +**P3 (watchdog ground truth) — DONE 2026-07-21, live from guest 9201.** These exact bytes are the +parser fixtures, including the literal backslash `ip -o` emits and the trailing space on the route: ``` -# the NON-ROOT agent can finally read its own credential -sudo -u felhom-agent sudo -n felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage - rc=0 secret_len=37 stderr=(empty) - -# and the probe actually runs (debug enabled briefly, then restored to info) -level=DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite +ip -4 -o addr show dev eth0 → 2: eth0 inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0\ valid_lft 4916sec preferred_lft 4916sec +ip route show default → default via 192.168.0.1 dev eth0 +pgrep -x dhclient → 235839 (rc=0; rc=1 + EMPTY stderr when absent) +ps -o args= -C dhclient → dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0 +/etc/network/interfaces → iface eth0 inet dhcp +pct exec → rc=2, stderr "Configuration file 'nodes/demo-felhom/lxc/9999.conf' does not exist" ``` -Report verified hub-side: `agent_version 0.91.1→0.91.2`, `wrapper_sha256 -104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3`. +The live `ps` line confirms the running client's argv is byte-identical to the incident's restored +invocation — i.e. the heal reproduces the guest's own boot-time command, not an approximation. +A docker-bridge-only route table is also pinned as a negative (`172.17.0.0/16 dev docker0 …` must +never read as a default route — those were the exact leftovers in the incident). -## 6. A defect I shipped and caught mid-STOP-1 +**P4 (config surface + report pattern + loop precedent) — DONE** by reading the tree; the wg loop's +interval/damping/logging shape and the `pbs_dr` stanza's collector-seam pattern are what this copies. -**v0.91.0 shipped the auth-honesty leg INERT.** I built the `AuthSink` seam and the `NoteAuthResult` -consumer, and `main.go` never called `SetAuthSink`. The reporter deliberately skips probing with no -sink attached, so there was no probe, no `auth_failed`, no self-heal — and **nothing failed**, because -every unit test injected the sink directly. +--- -Caught during STOP-1 by checking the wiring instead of trusting it. **This is the same class as the -controller v0.154.0 defect the day before: a table test over a seam proves the seam, not the caller.** -Fixed in v0.91.1; the published 0.91.0 artifact was **superseded, not overwritten** (a published -version stays immutable). +## 4. The live finding: three of four probes had no sudoers grant -**v0.91.2** then made a healthy probe observable at Debug — it previously logged only on failure, so -"no `auth_failed`" was indistinguishable from "never probed". That silence is exactly how v0.91.0 -shipped inert unnoticed. +The agent runs non-root; `Privileged.Mode=sudo` fails closed with no prompt. The **first sweep after +deploying v0.92.0** logged: -## 7. For the operator — STOP-2 and STOP-3 +``` +level=WARN msg="guestnet: guest network not actionable — reporting only" vmid=9201 + state=unknown mode=unknown has_ip=true has_route=false + detail="dhclient liveness probe failed: sudo: a password is required" +``` -**STOP-2 (one click):** press **Re-issue PBS credentials** for the demo customer. Expected chain: -fresh secret row → `secret_generation` **0 → 1** in the descriptor (the current live descriptor has no -such key at all) → poke → agent re-applies with **no short-circuit** → fresh secret consumed -(`consumed_at` stamped) → reconcile rc-0 → probe 200 → tier `active`. **The July-18 negative — the -same click doing nothing — is the historical red-proof this run is measured against.** +The watchdog behaved exactly as designed — it reported `unknown` and healed nothing rather than +acting blind — but it was blind. The existing allowlist granted only lanresolver's address read +(`pct exec [0-9]* -- ip -4 -o addr show dev eth0`), which is why `has_ip=true` while route, mode and +liveness all failed. -**STOP-3 (manifest save):** +Fixed in **v0.92.1**: a `FELHOM_GUESTNET` alias with four fixed vectors (route, interfaces, pgrep, +heal). Every argument after the numeric vmid is a literal, so nothing the guest or the hub says can +widen the grant. The address read is **not** duplicated — it stays FELHOM_DNSMASQ's; one command, +one grant. -| field | value | +Plus **four `guestnet-*` capability rows**, so a host that has not taken the new sudoers file is +VISIBLE as degraded instead of silently watchdog-less. Deliberately **non-critical**: a missing +grant must not page an operator for every box on rollout day (the R-50b amber-fleet lesson). + +**v0.92.0 is superseded, not overwritten — do not vouch it.** It was published before this was +found, so its binary lacks the capability rows and its release lacks the sudoers file. A published +version stays immutable (the v0.91.0 → v0.91.1 precedent). + +--- + +## 5. Tests and red-proofs + +Green gate: `go build ./... && go vet ./... && go test ./...` — all packages ok **except the known +flake** `TestGenerateRecoveryCode_EntropyAndFormat` (`internal/escrow`), which fails when the +wordlist yields a hyphenated word (`drop-down` → 11 tokens instead of 10). Confirmed pre-existing: +`internal/escrow` has not been touched since v0.88.0 and this task changes nothing there; observed +3/8 runs, consistent with the documented ~1/5. + +New: `internal/guestnet/watchdog_test.go` (16 cases), `internal/hub/collect_guestnet_test.go` (2), +`cmd/felhom-agent/guestnet_wiring_test.go` (2). + +| # | Red-proof | Mutation | Result | +|---|---|---|---| +| E | detect on process liveness, not address presence | `classify`'s DHCP arm reverted to IP-presence-only | **FAIL ×5.** The decisive one: `classify = "healthy", want "unhealthy"` for the July-20 fixture, `detail="address, default route and dhclient all present"`, and `heal ran 0 times`. That is the 80-minute silent window, reproduced exactly. Restored, green. | +| W | the watchdog must be wired | `SetGuestNetReporter` and `go gnWatchdog.Watch(ctx)` both commented out | **FAIL** with both reasons named — *"the guest_net stanza would never reach the hub (the exact v0.91.0 inert-seam defect)"* and *"it would be constructed, reported on, and never probe anything"*. Restored, green. | + +**Every damper is asserted as an exact count, and the load-bearing assertions are the negatives** — +a static guest, an unprobeable guest, a boot-race guest (young guest AND young agent), a failed +probe tool and an ownership-unproven guest list must each record **zero** heal calls. The ceilings +are driven by an injected clock over a scripted **10 hours** of permanent failure: ≤30 heals total, +and never a second heal inside the 10-minute cool-off. + +**Seam discipline (§9 rule 6)** — three production-path tests: the `guest_net` stanza is asserted +through the real `Collect` (and asserted ABSENT from the wire when no reporter is wired, so "not +wired" and "found nothing" can never look identical); and the `main.go` wiring is an AST walk for +the construction, the reporter call and the started goroutine. The AST form is deliberate — a +`strings.Contains` version of the twin test in felhom-controller **passed its own red-proof**, +because a commented-out call still contains the string. + +--- + +## 6. Live validation (method: journald + capability self-check on felhom-pve) + +| Step | Evidence | |---|---| -| Agent version | `0.91.2` | -| Agent sha256 | `34d309be429473f3f0ab34e3185e17b22463a341b30bf46e162306bff4aec22a` | -| **PBS wrapper sha256** (new field) | `104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3` | -| MinAgent | `0.91.2` — safe now: **P4 shows one enrolled host and it already runs 0.91.2** | +| Publish 0.92.0 | `AGENT_SHA256=b1302790d412d22e969936ff52e3ee33e3edc111b8426364a21cdb1c5127ca6a`, round-trip GET verified — **superseded, do not vouch** | +| Publish 0.92.1 | `AGENT_SHA256=7424bc1c3c533eff9157e15a18d4635c624931f5a479a48126de77a94e6a3d4d`, round-trip GET verified | +| Deploy | `visudo -c` parsed OK → sudoers installed 0440 root:root (backup `/root/felhom-agent.sudoers.bak-preR54`) + binary installed (backup `felhom-agent.bak-0.91.2-preR54`) → `felhom-agent --version` = **0.92.1**, service `active` | +| Capability self-check | **`ok=68 total=68 degraded=0 inactive=0`** (was 64/64 before the four `guestnet-*` rows) — the sudoers grant is proven from the agent's own side, not assumed | +| Watchdog start | `INFO guestnet: watchdog starting interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s` | +| **Healthy cycle** | **`level=DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true`** (12:34:15 CEST) | +| Default-ON proven | `/etc/felhom-agent/agent.json` has **no** `guest_net` key at all — the watchdog runs on defaults, which is the whole point of the inverted gate | -Superseded, do not vouch: **0.91.0** (inert leg) and **0.91.1**. `0.90.1` correctly remains **404**. +**Note for the operator:** `log_level` on felhom-pve was temporarily raised to `debug` to capture +that Debug line (backup at `/root/agent.json.bak-debuglevel`). It is **still `debug`**, deliberately, +so STOP-2's heal chain is visible in journald. **Revert it to `info` after STOP-2.** -## 8. Residual +--- -- **R-50b(b)/(c) NOT solved** — the wrapper is still fetched from `raw/branch/main`, unversioned and - unpinned. This release makes drift *visible*; it does not fix the channel. -- The **0440 sudoers file is not agent-readable**, so its drift stays invisible — recorded, not solved. -- R-39's own residual: the verify loop reads via the wrapper now, but a host whose wrapper predates - v0.91.0 has no `read` verb and degrades to the old skip (loudly, via the `pbsdr-read` capability). +## 7. Deliverables -## 9. Observations +- `c0966d7` — v0.92.0: the watchdog, the report block, config, tests. +- `0e8fd81` — the sudoers grant + capability rows (the live finding). +- `98adb72` — v0.92.1: supersede + version bump. +- Published: `felhom-agent` **0.92.1** / `7424bc1c3c533eff…` (0.92.0 superseded). +- Live on felhom-pve: binary 0.92.1 + the new sudoers file. -- **The hub's build script prints `kubectl set image` as its deploy hint**, which contradicts the - GitOps rule in `felhom.eu/CLAUDE.md`. It is a trap for exactly the kind of session that trusts tool - output over the runbook. Worth fixing in `build.sh`. -- **The July-18 hotfixed wrapper hashed `2888f2ea…`, matching no commit anyone could name.** That is - R-50b in one line, and it is why the manifest field earns its place. -- `pvesm config` does not exist (it is `pvesm status` / `/etc/pve/storage.cfg`); PVE's per-storage - status endpoint returns `active: 0` rather than erroring — both worth knowing before designing - against them. +## 8. Operator actions outstanding + +1. **STOP-2** — the incident replay (kill 9201's dhclient, watch it heal within ≤2 cycles). It is + the deliberate reproduction of a real outage, so it is operator-present by construction. +2. **STOP-3** — Day-0 manifest Agent → **0.92.1** / `7424bc1c3c533eff9157e15a18d4635c624931f5a479a48126de77a94e6a3d4d` + (wrapper sha unchanged — `configs/felhom-pbs-apply` was not touched). MinAgent is the operator's + call; 0.92.1 is safe once felhom-pve has converged, which it has. +3. Revert `log_level` to `info` after STOP-2.