admin/felhom-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(report): TASK-D Part 3 — R-54 shipped, healthy cycle proven live; STOP-2 pending

Browse Source
This commit is contained in:
authentik Default Admin
2026-07-21 12:39:12 +02:00
parent 98adb72204
commit 71e1777369
1 changed files with 158 additions and 129 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+158 -129
View File
@@ -1,163 +1,192 @@
# REPORT — TASK-B: R-39 fleet fix + R-50b(a) · felhom-agent v0.90.0 → **v0.91.2**
# REPORT — TASK-D Part 3: the guest-network watchdog (R-54) · felhom-agent v0.91.2 → **v0.92.1**
**Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `8c55ac7` (clean, == `origin/main`).
Companion: hub **v0.68.0** (`felhom.eu`).
**Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `08b55a1` (clean, == `origin/main`).
**Deployed and running on felhom-pve.** The healthy-cycle leg is **PROVEN LIVE**; the heal leg
(STOP-2, a deliberate replay of the 2026-07-20 incident) is operator-present and has **not** run.
## 0. Status
---
| Leg | Status |
|---|---|
| Part 1 — hub v0.68.0 | **SHIPPED + DEPLOYED** (GitOps sync, Synced/Healthy) |
| Part 2 — agent v0.91.2 | **SHIPPED + PUBLISHED + DEPLOYED** to felhom-pve |
| **STOP-1** — felhom-pve update | **DONE + VERIFIED** (§5) |
| **STOP-3** — manifest save | **DONE by the operator 2026-07-21 08:31:12Z** — agent 0.91.2 + sha + **wrapper sha `104db0a4…`** + MinAgent 0.91.2, all six fields persisted. The agent reports the same wrapper hash, so the R-50b drift gauge reads **ok**. |
| **STOP-2** — live re-issue | **DONE + PROVEN 2026-07-21 — the chain closed in 13 s.** `08:39:31Z` hub mint gen 0→1 + `secret_generation: 1` (token_id/fingerprint byte-identical) → `10:39:34` wrapper read → `10:39:38` `REJECTED … applied and DEAD` `previous_state=applied` → `10:39:45` **secret consumed (no short-circuit)** → `10:39:45` set-only reconcile → `10:39:47` `converged state=applied`. Marker hash moved to `afbb3b41…`; secret mtime `2026-07-18`→`2026-07-21 10:39:45`; new credential probes 200; reports trace `applied → auth_failed → applied`; **zero** self-heal escalations, one mint, one consume. First attempt hit the **offsite** re-issue by mistake (two distinct actions) — harmless to PBS-DR but it made the escrow stale, ceremony re-run. |
## 1. What this closes
## 1. Probes
`INCIDENT-guest-dhclient-killed-2026-07-20.md` §5, "OPEN RISK": **the guest's DHCP client is
unsupervised.** ifupdown starts it once at boot and nothing restarts it. When it was killed on
2026-07-20 the guest kept working for another **~80 minutes** on its unexpired lease; only at expiry
did the address and default route vanish, taking the Cloudflare tunnel, hub reports, catalog sync
and the controller→agent channel with them — a 1h15m outage in which every observable signal said
healthy for the first 80 minutes.
- **P1 — version stamping: PASS.** ldflags `-X main.version`; `publish-agent.sh` refuses to publish
when the binary's `--version` disagrees, so the stale `0.89.0` default cannot leak.
- **P2 — hub deploy path: PASS.** GitOps only: ArgoCD app `felhom`, auto-sync **off**, manifest bump
+ deliberate sync. The `kubectl set image` line (in the old controller CLAUDE.md *and* printed by
`build.sh` itself) was **not** used.
- **P3 — felhom-pve before-state: CAPTURED** (§5 table). Notably the DR tier was **healthy**: a live
`GET /api2/json/version` with the stored credential returned **200**. This is a fix for a latent
defect, not a broken box.
- **P4 — fleet: ONE enrolled host** (`demo-felhom-8363b5`, agent 0.90.0). **Peti is not enrolled**, so
the MinAgent raise at STOP-3 cannot strand anyone. (Read from a temporary copy of the hub DB, which
carries live credentials — shredded immediately after each read.)
**So the design consequence is the whole feature: liveness of the DHCP client is itself a probe.**
The watchdog flags a DHCP guest unhealthy on `pgrep -x dhclient` alone, while the address and route
are still perfectly present. Waiting for the IP to disappear is waiting out exactly that silent
window — and the red-proof reproduces it (§5).
## 2. Two spec/reality mismatches, both resolved
Host tier is not a preference: a guest with no default route cannot repair its own default route.
**(a) The generation source does not exist as specified.** The brief said to return "the new row's id
(int64) … no schema change". `host_pbs_secrets` is `host_id PRIMARY KEY`, UPSERTed last-write-wins —
**there is no row id and no new row**, and `created_at` collides for two mints in one second. Added an
additive `generation` counter column using the repo's existing idempotent `ALTER TABLE` idiom. Same
semantics the brief wanted; different mechanism.
---
**(b) A second load-bearing fact the brief did not flag — checked, not trusted.** `Apply` bails out
entirely if the storage *status probe* errors, and converges via `adopt` **without consuming the
secret** if the storage reads active. So the whole fix depended on what PVE returns for a 401'd PBS
storage. Read PVE's own source: `storage_info` wraps `activate_storage` and `$plugin->status` in
`eval{}`, warns, and leaves the pre-initialised `active => 0`. **A 401 therefore returns HTTP 200 with
`active: 0`, never an API error** — `StorageActive` gives `(false, nil)` and `Apply` falls through to
verify → consume → reconcile. Scenario A's chain is sound *by proof*. The re-arm test encodes this
fixture shape explicitly so the assumption is visible.
## 2. Shipped
## 3. What shipped (agent)
`internal/guestnet` (probe.go / watchdog.go / report.go), built on the wg-tunnel + storage watchdog
loop shape, started with `go wd.Watch(ctx)` like `selfheal`.
`WirePBSDR.SecretGeneration` (field-exact with the hub) · wrapper `read` verb + **exactly one**
sudoers line + a `pbsdr-read` capability row · `pbs.ProbeAuth` + `ErrUnauthorized` · the probe on the
**15-minute collect path** feeding a loud `auth_failed` · `readPBSSecret` (direct read, wrapper
fallback) · wrapper sha256 in the report (R-50b(a)).
- **Four fixed-shape `pct exec` probes**, all constant argv + the numeric vmid: address, default
route, `/etc/network/interfaces` mode, `pgrep -x dhclient`. No shell anywhere; no guest-supplied
data is ever interpolated into a command.
- **Heal = the incident's restored invocation, verbatim**, logged at INFO before it runs:
`pct exec <vmid> -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0`
A test pins that argv element by element.
- **Dampers** (this runs a privileged command inside a customer's container, so it is built to
under-act): two CONSECUTIVE bad probes before any heal, ≥10 min between heals per guest, ≤3
heals/hour, and observe-only while the guest — or the agent itself — has been up under 3 minutes.
- **Refuses to act** on: a static guest (dhclient must never fight a static config; a static guest
missing its address is reported loudly and left to **R-50**), an unknown interface mode, a guest
it cannot probe, and an ownership-unproven guest list. The guest source is the pool-verified
`ListLXC` ∩ felhom-pool (audit A1) — never a bare `ListLXC`, which under a broad token would run
dhclient inside a co-tenant's container.
- **A failed PROBE is never a dead client.** `pgrep` exits 1 with EMPTY stderr on no-match; anything
on stderr means the probe itself failed → `unknown`. Without that rule a missing `pgrep` would
heal forever.
- **Healthy cycles log a Debug line.** v0.91.2's lesson, one day old: if the quiet path is silent,
"no alarms" and "never probed" are the same evidence.
- **Not in the `errc` fan-out** — a watchdog over customer guests must never be able to terminate
the agent. A test asserts that, because joining the fan-out would also make the shutdown drain
bound off by one.
- **Report block:** `GuestNetStatus` on `HostReport` (`guest_net`, omitempty), additive and stored
opaquely hub-side like `pbs_dr` / `wireguard`. **No hub code was touched.**
**403 is deliberately NOT treated as unauthorized** — re-keying a too-narrow token would mint
credentials forever without fixing anything. **A transport error is UNKNOWN, never a rejection** —
otherwise every network blip burns a credential.
**Two deliberate deviations from TASK-D, both stated up front:**
## 4. Tests + the four red-proofs
1. **`GuestNetStatus`, not `WireGuestNet`.** In this repo `Wire*` is the DOWN direction
(`WireDesiredState` / `WirePBSDR` — what the hub sends the agent); UP-direction report stanzas
are `*Status`. `WireGuestNet` on `HostReport` would have been the only report block named against
the convention.
2. **A sudoers change was required** — see §4. The brief said none was needed.
Full gate green in both repos at every commit. All red-proofs run at the **assertion** level.
**Config `guest_net` is this repo's first default-ON gate.** Every other gate defaults to false
because those features reach outward (an offsite endpoint, an OOB tunnel) and enrolling a box by an
update would be wrong. This one looks only inward at guests the agent already owns, and the failure
it prevents exists on every box today. A watchdog that must be remembered per box is a watchdog
that is missing on the box that needed it. Opt-out is explicit: `"guest_net": {"disable": true}`.
| # | Red-proof | Result |
|---|---|---|
| A | Remove `SecretGeneration` from the agent struct | `consume calls=1, want 2` — the converged short-circuit fires; the July-18 defect exactly |
| B | Stop stamping the generation in `ReissuePBSDR` | descriptor **byte-identical** across a re-issue, block printed |
| D | Remove the id charset guard **and** the prefix assertion | the out-of-tree secret is **printed** (`rc=0 stdout="NOT-FOR-THE-AGENT-…"`) |
| E | Swallow the probe result in `NoteAuthResult` | `State:applied AuthFailed:false` — the July-18 shape |
---
**A finding from red-proof D worth keeping:** deleting the charset guard *alone* does **not** open a
hole — `readlink -f` plus the prefix assertion still catch it, so the naive single-guard red-proof
**passed vacuously**. The layering is real; the isolating proof had to remove both. A red-proof that
passes is not automatically reassuring.
## 3. Phase-0 probes
The wrapper is tested as the artifact it is — executed under real `bash`. The two cases needing a
writable allowlisted dir run a **copy with only the prefix constant relocated** (and re-assert that
traversal is still refused in the copy), rather than `t.Skip` — a skipped test proves nothing.
## 5. STOP-1 — felhom-pve, before → after
| | before (P3) | after |
|---|---|---|
| agent | 0.90.0 · `18c48272…` | **0.91.2** · `34d309be…` |
| wrapper | `2888f2ea…` (the in-place July-18 hotfix) | `104db0a4…` (repo) |
| sudoers | `174125ef…` | new, +1 line |
| capabilities | 63 | **64/64 ok, 0 degraded, 0 inactive** |
| guest 9201 | running | running (untouched) |
| `wg-felhom` | handshake `…673` | fresh `…793` |
Gates run before install: `bash -n` on the wrapper, `visudo -cf` on the staged sudoers, `visudo -c`
on the whole tree **after** install, and the staged binary's own `--version`.
**Rollback set retained on the host:**
`/usr/local/sbin/felhom-pbs-apply.bak-20260721-101635` (and the older `.bak-20260718-preR39`),
`/usr/local/bin/felhom-agent.bak-2026072{1-101635,1-101905,…}`,
`/root/felhom-agent.sudoers.bak-20260721-101635`.
**The two live proofs that matter:**
**P3 (watchdog ground truth) — DONE 2026-07-21, live from guest 9201.** These exact bytes are the
parser fixtures, including the literal backslash `ip -o` emits and the trailing space on the route:
```
# the NON-ROOT agent can finally read its own credential
sudo -u felhom-agent sudo -n felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
rc=0 secret_len=37 stderr=(empty)
# and the probe actually runs (debug enabled briefly, then restored to info)
level=DEBUG msg="pbs: credential probe OK" storage=felhom-pbs datastore=felhom-offsite
ip -4 -o addr show dev eth0 → 2: eth0 inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0\ valid_lft 4916sec preferred_lft 4916sec
ip route show default → default via 192.168.0.1 dev eth0
pgrep -x dhclient → 235839 (rc=0; rc=1 + EMPTY stderr when absent)
ps -o args= -C dhclient → dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
/etc/network/interfaces → iface eth0 inet dhcp
pct exec <bad vmid> → rc=2, stderr "Configuration file 'nodes/demo-felhom/lxc/9999.conf' does not exist"
```
Report verified hub-side: `agent_version 0.91.1→0.91.2`, `wrapper_sha256
104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3`.
The live `ps` line confirms the running client's argv is byte-identical to the incident's restored
invocation — i.e. the heal reproduces the guest's own boot-time command, not an approximation.
A docker-bridge-only route table is also pinned as a negative (`172.17.0.0/16 dev docker0 …` must
never read as a default route — those were the exact leftovers in the incident).
## 6. A defect I shipped and caught mid-STOP-1
**P4 (config surface + report pattern + loop precedent) — DONE** by reading the tree; the wg loop's
interval/damping/logging shape and the `pbs_dr` stanza's collector-seam pattern are what this copies.
**v0.91.0 shipped the auth-honesty leg INERT.** I built the `AuthSink` seam and the `NoteAuthResult`
consumer, and `main.go` never called `SetAuthSink`. The reporter deliberately skips probing with no
sink attached, so there was no probe, no `auth_failed`, no self-heal — and **nothing failed**, because
every unit test injected the sink directly.
---
Caught during STOP-1 by checking the wiring instead of trusting it. **This is the same class as the
controller v0.154.0 defect the day before: a table test over a seam proves the seam, not the caller.**
Fixed in v0.91.1; the published 0.91.0 artifact was **superseded, not overwritten** (a published
version stays immutable).
## 4. The live finding: three of four probes had no sudoers grant
**v0.91.2** then made a healthy probe observable at Debug — it previously logged only on failure, so
"no `auth_failed`" was indistinguishable from "never probed". That silence is exactly how v0.91.0
shipped inert unnoticed.
The agent runs non-root; `Privileged.Mode=sudo` fails closed with no prompt. The **first sweep after
deploying v0.92.0** logged:
## 7. For the operator — STOP-2 and STOP-3
```
level=WARN msg="guestnet: guest network not actionable — reporting only" vmid=9201
state=unknown mode=unknown has_ip=true has_route=false
detail="dhclient liveness probe failed: sudo: a password is required"
```
**STOP-2 (one click):** press **Re-issue PBS credentials** for the demo customer. Expected chain:
fresh secret row → `secret_generation` **0 → 1** in the descriptor (the current live descriptor has no
such key at all) → poke → agent re-applies with **no short-circuit** → fresh secret consumed
(`consumed_at` stamped) → reconcile rc-0 → probe 200 → tier `active`. **The July-18 negative — the
same click doing nothing — is the historical red-proof this run is measured against.**
The watchdog behaved exactly as designed — it reported `unknown` and healed nothing rather than
acting blind — but it was blind. The existing allowlist granted only lanresolver's address read
(`pct exec [0-9]* -- ip -4 -o addr show dev eth0`), which is why `has_ip=true` while route, mode and
liveness all failed.
**STOP-3 (manifest save):**
Fixed in **v0.92.1**: a `FELHOM_GUESTNET` alias with four fixed vectors (route, interfaces, pgrep,
heal). Every argument after the numeric vmid is a literal, so nothing the guest or the hub says can
widen the grant. The address read is **not** duplicated — it stays FELHOM_DNSMASQ's; one command,
one grant.
| field | value |
Plus **four `guestnet-*` capability rows**, so a host that has not taken the new sudoers file is
VISIBLE as degraded instead of silently watchdog-less. Deliberately **non-critical**: a missing
grant must not page an operator for every box on rollout day (the R-50b amber-fleet lesson).
**v0.92.0 is superseded, not overwritten — do not vouch it.** It was published before this was
found, so its binary lacks the capability rows and its release lacks the sudoers file. A published
version stays immutable (the v0.91.0 → v0.91.1 precedent).
---
## 5. Tests and red-proofs
Green gate: `go build ./... && go vet ./... && go test ./...` — all packages ok **except the known
flake** `TestGenerateRecoveryCode_EntropyAndFormat` (`internal/escrow`), which fails when the
wordlist yields a hyphenated word (`drop-down` → 11 tokens instead of 10). Confirmed pre-existing:
`internal/escrow` has not been touched since v0.88.0 and this task changes nothing there; observed
3/8 runs, consistent with the documented ~1/5.
New: `internal/guestnet/watchdog_test.go` (16 cases), `internal/hub/collect_guestnet_test.go` (2),
`cmd/felhom-agent/guestnet_wiring_test.go` (2).
| # | Red-proof | Mutation | Result |
|---|---|---|---|
| E | detect on process liveness, not address presence | `classify`'s DHCP arm reverted to IP-presence-only | **FAIL ×5.** The decisive one: `classify = "healthy", want "unhealthy"` for the July-20 fixture, `detail="address, default route and dhclient all present"`, and `heal ran 0 times`. That is the 80-minute silent window, reproduced exactly. Restored, green. |
| W | the watchdog must be wired | `SetGuestNetReporter` and `go gnWatchdog.Watch(ctx)` both commented out | **FAIL** with both reasons named — *"the guest_net stanza would never reach the hub (the exact v0.91.0 inert-seam defect)"* and *"it would be constructed, reported on, and never probe anything"*. Restored, green. |
**Every damper is asserted as an exact count, and the load-bearing assertions are the negatives** —
a static guest, an unprobeable guest, a boot-race guest (young guest AND young agent), a failed
probe tool and an ownership-unproven guest list must each record **zero** heal calls. The ceilings
are driven by an injected clock over a scripted **10 hours** of permanent failure: ≤30 heals total,
and never a second heal inside the 10-minute cool-off.
**Seam discipline (§9 rule 6)** — three production-path tests: the `guest_net` stanza is asserted
through the real `Collect` (and asserted ABSENT from the wire when no reporter is wired, so "not
wired" and "found nothing" can never look identical); and the `main.go` wiring is an AST walk for
the construction, the reporter call and the started goroutine. The AST form is deliberate — a
`strings.Contains` version of the twin test in felhom-controller **passed its own red-proof**,
because a commented-out call still contains the string.
---
## 6. Live validation (method: journald + capability self-check on felhom-pve)
| Step | Evidence |
|---|---|
| Agent version | `0.91.2` |
| Agent sha256 | `34d309be429473f3f0ab34e3185e17b22463a341b30bf46e162306bff4aec22a` |
| **PBS wrapper sha256** (new field) | `104db0a4401f65bbc476e82bfb1796433bcb36f8f8cce69efb3bb5c40fcb16b3` |
| MinAgent | `0.91.2` — safe now: **P4 shows one enrolled host and it already runs 0.91.2** |
| Publish 0.92.0 | `AGENT_SHA256=b1302790d412d22e969936ff52e3ee33e3edc111b8426364a21cdb1c5127ca6a`, round-trip GET verified — **superseded, do not vouch** |
| Publish 0.92.1 | `AGENT_SHA256=7424bc1c3c533eff9157e15a18d4635c624931f5a479a48126de77a94e6a3d4d`, round-trip GET verified |
| Deploy | `visudo -c` parsed OK → sudoers installed 0440 root:root (backup `/root/felhom-agent.sudoers.bak-preR54`) + binary installed (backup `felhom-agent.bak-0.91.2-preR54`) → `felhom-agent --version` = **0.92.1**, service `active` |
| Capability self-check | **`ok=68 total=68 degraded=0 inactive=0`** (was 64/64 before the four `guestnet-*` rows) — the sudoers grant is proven from the agent's own side, not assumed |
| Watchdog start | `INFO guestnet: watchdog starting interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s` |
| **Healthy cycle** | **`level=DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true`** (12:34:15 CEST) |
| Default-ON proven | `/etc/felhom-agent/agent.json` has **no** `guest_net` key at all — the watchdog runs on defaults, which is the whole point of the inverted gate |
Superseded, do not vouch: **0.91.0** (inert leg) and **0.91.1**. `0.90.1` correctly remains **404**.
**Note for the operator:** `log_level` on felhom-pve was temporarily raised to `debug` to capture
that Debug line (backup at `/root/agent.json.bak-debuglevel`). It is **still `debug`**, deliberately,
so STOP-2's heal chain is visible in journald. **Revert it to `info` after STOP-2.**
## 8. Residual
---
- **R-50b(b)/(c) NOT solved** — the wrapper is still fetched from `raw/branch/main`, unversioned and
unpinned. This release makes drift *visible*; it does not fix the channel.
- The **0440 sudoers file is not agent-readable**, so its drift stays invisible — recorded, not solved.
- R-39's own residual: the verify loop reads via the wrapper now, but a host whose wrapper predates
v0.91.0 has no `read` verb and degrades to the old skip (loudly, via the `pbsdr-read` capability).
## 7. Deliverables
## 9. Observations
- `c0966d7` — v0.92.0: the watchdog, the report block, config, tests.
- `0e8fd81` — the sudoers grant + capability rows (the live finding).
- `98adb72` — v0.92.1: supersede + version bump.
- Published: `felhom-agent` **0.92.1** / `7424bc1c3c533eff…` (0.92.0 superseded).
- Live on felhom-pve: binary 0.92.1 + the new sudoers file.
- **The hub's build script prints `kubectl set image` as its deploy hint**, which contradicts the
GitOps rule in `felhom.eu/CLAUDE.md`. It is a trap for exactly the kind of session that trusts tool
output over the runbook. Worth fixing in `build.sh`.
- **The July-18 hotfixed wrapper hashed `2888f2ea…`, matching no commit anyone could name.** That is
R-50b in one line, and it is why the manifest field earns its place.
- `pvesm config` does not exist (it is `pvesm status` / `/etc/pve/storage.cfg`); PVE's per-storage
status endpoint returns `active: 0` rather than erroring — both worth knowing before designing
against them.
## 8. Operator actions outstanding
1. **STOP-2** — the incident replay (kill 9201's dhclient, watch it heal within ≤2 cycles). It is
the deliberate reproduction of a real outage, so it is operator-present by construction.
2. **STOP-3** — Day-0 manifest Agent → **0.92.1** / `7424bc1c3c533eff9157e15a18d4635c624931f5a479a48126de77a94e6a3d4d`
(wrapper sha unchanged — `configs/felhom-pbs-apply` was not touched). MinAgent is the operator's
call; 0.92.1 is safe once felhom-pve has converged, which it has.
3. Revert `log_level` to `info` after STOP-2.