Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 admin
|
eba040d0be | ||
|
admin
|
a452dc3314 |
@@ -1,3 +1,28 @@
|
||||
## v0.93.0 — a recovery code can no longer contain a hyphenated word (2026-07-21)
|
||||
|
||||
**Generation-only change. Every recovery code already issued remains valid** — R is consumed as a
|
||||
whole passphrase by the PBS scrypt KDF (`Wrap`/`Unwrap`) and is never re-split, so nothing about
|
||||
verification moves. Nothing in the KDF/consume path, the word count, or the joiner changed.
|
||||
|
||||
The EFF large wordlist contains exactly four entries that themselves contain the hyphen we join
|
||||
words with: `drop-down`, `felt-tip`, `t-shirt`, `yo-yo`. Drawing one produced a code that reads as
|
||||
11 words rather than 10 — ambiguous to transcribe in exactly the situation R exists for, a customer
|
||||
reading a code back during a disaster. The generator now draws from the list filtered of those four
|
||||
(`joinSafe`), so a code always segments back into exactly `RecoveryCodeWords`.
|
||||
|
||||
- **Entropy floor holds, with the numbers asserted in the tests:** the draw space goes 7776 → 7772,
|
||||
so a 10-word code goes 129.248 → 129.241 bits. The cost is 0.007 bits against a 128-bit floor.
|
||||
- **The long-standing ~1/5 test flake was this defect, not a flaky test.**
|
||||
`TestGenerateRecoveryCode_EntropyAndFormat` counted words by splitting the joined string, which
|
||||
conflates "how many words were drawn" with "how many segments the code has". It now counts what
|
||||
the generator drew, and asserts the segmentation property separately — the property `joinSafe`
|
||||
actually buys. (`felhom.eu/REPORT.md` §6 item 3 recorded it at 3/8 in one session.)
|
||||
- **Deterministic red-proof, in-tree:** `TestGeneratedCodeSegments_FilteredVsUnfiltered` drives the
|
||||
generator against a fixture list where every word is hyphenated, so the pre-fix defect reproduces
|
||||
with probability 1 instead of ~1/5, and shows the same list through `joinSafe` refuses to generate.
|
||||
- New for audit: `WordlistFilteredOut()`, `RecoveryCodeSep`. `WordlistSize()` now reports the
|
||||
effective (filtered) draw space, 7772.
|
||||
|
||||
## v0.92.1 — ship the guestnet sudoers grant with the binary (supersedes v0.92.0) (2026-07-21)
|
||||
|
||||
**Supersedes v0.92.0; that artifact is materially incomplete — do not vouch it.** It was published
|
||||
|
||||
@@ -1,286 +1,79 @@
|
||||
# REPORT — TASK-D Part 3: the guest-network watchdog (R-54) · felhom-agent v0.91.2 → **v0.92.1**
|
||||
# REPORT — escrow recovery-code wordlist fix (TASK-E Part 1), 2026-07-21
|
||||
|
||||
**Date:** 2026-07-21 · Trunk, pushed to `main`. **Baseline:** `08b55a1` (clean, == `origin/main`).
|
||||
**Deployed, running on felhom-pve, and STOP-2 RAN — the incident was replayed and the watchdog
|
||||
prevented it (§6b).** Every claim below was observed.
|
||||
**Version: v0.93.0. Baseline `935904f` (v0.92.1), clean tree. NOT built or deployed** — no operator
|
||||
action needed today; this rides the next agent publish train.
|
||||
|
||||
---
|
||||
## What was wrong
|
||||
|
||||
## 1. What this closes
|
||||
The EFF large wordlist contains exactly four entries that themselves contain the hyphen the words are
|
||||
joined with: `drop-down` (L2009), `felt-tip` (L2528), `t-shirt` (L6640), `yo-yo` (L7748). Drawing one
|
||||
produced a recovery code that reads as **11 words instead of 10** — ambiguous to transcribe in the one
|
||||
situation R exists for, a customer reading a code back during a disaster.
|
||||
|
||||
`INCIDENT-guest-dhclient-killed-2026-07-20.md` §5, "OPEN RISK": **the guest's DHCP client is
|
||||
unsupervised.** ifupdown starts it once at boot and nothing restarts it. When it was killed on
|
||||
2026-07-20 the guest kept working for another **~80 minutes** on its unexpired lease; only at expiry
|
||||
did the address and default route vanish, taking the Cloudflare tunnel, hub reports, catalog sync
|
||||
and the controller→agent channel with them — a 1h15m outage in which every observable signal said
|
||||
healthy for the first 80 minutes.
|
||||
**The "known flake" was this defect, not a flaky test.** Per code the hit rate is
|
||||
`1 − (7772/7776)^10 ≈ 0.51%`; over the test's 50 draws that is ≈ **23%**, which is exactly the "fails
|
||||
~1/5" that had been documented in the `felhom-testing` skill and re-run past for weeks.
|
||||
`felhom.eu/REPORT.md` §6 had it at 3/8 in one session.
|
||||
|
||||
**So the design consequence is the whole feature: liveness of the DHCP client is itself a probe.**
|
||||
The watchdog flags a DHCP guest unhealthy on `pgrep -x dhclient` alone, while the address and route
|
||||
are still perfectly present. Waiting for the IP to disappear is waiting out exactly that silent
|
||||
window — and the red-proof reproduces it (§5).
|
||||
## The fix
|
||||
|
||||
Host tier is not a preference: a guest with no default route cannot repair its own default route.
|
||||
`joinSafe` filters those four out of the effective wordlist at init. Generation-only.
|
||||
|
||||
---
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| draw space | 7776 | **7772** |
|
||||
| 10-word code entropy | 129.248 bits | **129.241 bits** |
|
||||
| margin over the 128-bit floor | 1.248 | **1.241** |
|
||||
|
||||
## 2. Shipped
|
||||
The cost is **0.007 bits**. Both numbers are asserted in `TestEntropyFloorSurvivesFiltering`, so a
|
||||
future wordlist swap cannot quietly move the floor.
|
||||
|
||||
`internal/guestnet` (probe.go / watchdog.go / report.go), built on the wg-tunnel + storage watchdog
|
||||
loop shape, started with `go wd.Watch(ctx)` like `selfheal`.
|
||||
**Every recovery code already issued remains valid**, and this was verified rather than assumed: R is
|
||||
consumed as a whole passphrase by the PBS scrypt KDF (`Wrap`/`Unwrap`), and `grep` confirms nothing in
|
||||
the consume path ever splits it. The joiner, the word count and the KDF path are untouched.
|
||||
|
||||
- **Four fixed-shape `pct exec` probes**, all constant argv + the numeric vmid: address, default
|
||||
route, `/etc/network/interfaces` mode, `pgrep -x dhclient`. No shell anywhere; no guest-supplied
|
||||
data is ever interpolated into a command.
|
||||
- **Heal = the incident's restored invocation, verbatim**, logged at INFO before it runs:
|
||||
`pct exec <vmid> -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0`
|
||||
A test pins that argv element by element.
|
||||
- **Dampers** (this runs a privileged command inside a customer's container, so it is built to
|
||||
under-act): two CONSECUTIVE bad probes before any heal, ≥10 min between heals per guest, ≤3
|
||||
heals/hour, and observe-only while the guest — or the agent itself — has been up under 3 minutes.
|
||||
- **Refuses to act** on: a static guest (dhclient must never fight a static config; a static guest
|
||||
missing its address is reported loudly and left to **R-50**), an unknown interface mode, a guest
|
||||
it cannot probe, and an ownership-unproven guest list. The guest source is the pool-verified
|
||||
`ListLXC` ∩ felhom-pool (audit A1) — never a bare `ListLXC`, which under a broad token would run
|
||||
dhclient inside a co-tenant's container.
|
||||
- **A failed PROBE is never a dead client.** `pgrep` exits 1 with EMPTY stderr on no-match; anything
|
||||
on stderr means the probe itself failed → `unknown`. Without that rule a missing `pgrep` would
|
||||
heal forever.
|
||||
- **Healthy cycles log a Debug line.** v0.91.2's lesson, one day old: if the quiet path is silent,
|
||||
"no alarms" and "never probed" are the same evidence.
|
||||
- **Not in the `errc` fan-out** — a watchdog over customer guests must never be able to terminate
|
||||
the agent. A test asserts that, because joining the fan-out would also make the shutdown drain
|
||||
bound off by one.
|
||||
- **Report block:** `GuestNetStatus` on `HostReport` (`guest_net`, omitempty), additive and stored
|
||||
opaquely hub-side like `pbs_dr` / `wireguard`. **No hub code was touched.**
|
||||
## Tests
|
||||
|
||||
**Two deliberate deviations from TASK-D, both stated up front:**
|
||||
`TestGenerateRecoveryCode_EntropyAndFormat` now counts words by **generation count**, not by splitting
|
||||
the joined string — conflating those two is what made it flake. It asserts the segmentation property
|
||||
separately, because that is the property `joinSafe` actually buys.
|
||||
|
||||
1. **`GuestNetStatus`, not `WireGuestNet`.** In this repo `Wire*` is the DOWN direction
|
||||
(`WireDesiredState` / `WirePBSDR` — what the hub sends the agent); UP-direction report stanzas
|
||||
are `*Status`. `WireGuestNet` on `HostReport` would have been the only report block named against
|
||||
the convention.
|
||||
2. **A sudoers change was required** — see §4. The brief said none was needed.
|
||||
New in `wordlist_test.go`: the four filtered words are pinned by name (a wordlist swap that changes the
|
||||
set fails loudly); the entropy numbers above; and a production-wiring test that drives the exported
|
||||
`GenerateRecoveryCode` 500×, so the fix is proven at the entry point and not only in the helper.
|
||||
|
||||
**Config `guest_net` is this repo's first default-ON gate.** Every other gate defaults to false
|
||||
because those features reach outward (an offsite endpoint, an OOB tunnel) and enrolling a box by an
|
||||
update would be wrong. This one looks only inward at guests the agent already owns, and the failure
|
||||
it prevents exists on every box today. A watchdog that must be remembered per box is a watchdog
|
||||
that is missing on the box that needed it. Opt-out is explicit: `"guest_net": {"disable": true}`.
|
||||
### Red-proof — RUN, both deterministic and probabilistic halves
|
||||
|
||||
---
|
||||
|
||||
## 3. Phase-0 probes
|
||||
|
||||
**P3 (watchdog ground truth) — DONE 2026-07-21, live from guest 9201.** These exact bytes are the
|
||||
parser fixtures, including the literal backslash `ip -o` emits and the trailing space on the route:
|
||||
Pre-fix shape restored (`var wordlist = parseWordlist(wordlistRaw)`), suite re-run:
|
||||
|
||||
```
|
||||
ip -4 -o addr show dev eth0 → 2: eth0 inet 192.168.0.104/24 brd 192.168.0.255 scope global dynamic eth0\ valid_lft 4916sec preferred_lft 4916sec
|
||||
ip route show default → default via 192.168.0.1 dev eth0
|
||||
pgrep -x dhclient → 235839 (rc=0; rc=1 + EMPTY stderr when absent)
|
||||
ps -o args= -C dhclient → dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
|
||||
/etc/network/interfaces → iface eth0 inet dhcp
|
||||
pct exec <bad vmid> → rc=2, stderr "Configuration file 'nodes/demo-felhom/lxc/9999.conf' does not exist"
|
||||
--- FAIL: TestWordlistLoaded
|
||||
effective wordlist should be 7772 words (7776 EFF - 4 hyphenated), got 7776
|
||||
--- FAIL: TestGenerateRecoveryCode_EntropyAndFormat
|
||||
joined code must segment into 10 words, got 11 (a drawn word contained "-")
|
||||
--- FAIL: TestEntropyFloorSurvivesFiltering
|
||||
filtered entropy moved: 129.248, expected 129.241
|
||||
--- FAIL: TestGenerateRecoveryCode_NeverContainsAmbiguousWord
|
||||
code 193 segmented into 11 parts, want 10
|
||||
```
|
||||
|
||||
The live `ps` line confirms the running client's argv is byte-identical to the incident's restored
|
||||
invocation — i.e. the heal reproduces the guest's own boot-time command, not an approximation.
|
||||
A docker-bridge-only route table is also pinned as a negative (`172.17.0.0/16 dev docker0 …` must
|
||||
never read as a default route — those were the exact leftovers in the incident).
|
||||
Fix restored → green; `git diff` clean.
|
||||
|
||||
**P4 (config surface + report pattern + loop precedent) — DONE** by reading the tree; the wg loop's
|
||||
interval/damping/logging shape and the `pbs_dr` stanza's collector-seam pattern are what this copies.
|
||||
`TestGeneratedCodeSegments_FilteredVsUnfiltered` is the **deterministic** companion: it drives the
|
||||
generator against a fixture list where every word is hyphenated, so the pre-fix defect reproduces with
|
||||
probability 1 rather than ~1/5, and shows the same list through `joinSafe` refuses to generate at all.
|
||||
|
||||
---
|
||||
## Gates
|
||||
|
||||
## 4. The live finding: three of four probes had no sudoers grant
|
||||
`go build ./... && go vet ./... && go test ./...` — **green**. The ex-flaky package was additionally
|
||||
run 10× consecutively: 10/10 pass (it previously failed ~1 in 5).
|
||||
|
||||
The agent runs non-root; `Privileged.Mode=sudo` fails closed with no prompt. The **first sweep after
|
||||
deploying v0.92.0** logged:
|
||||
## Observations
|
||||
|
||||
```
|
||||
level=WARN msg="guestnet: guest network not actionable — reporting only" vmid=9201
|
||||
state=unknown mode=unknown has_ip=true has_route=false
|
||||
detail="dhclient liveness probe failed: sudo: a password is required"
|
||||
```
|
||||
|
||||
The watchdog behaved exactly as designed — it reported `unknown` and healed nothing rather than
|
||||
acting blind — but it was blind. The existing allowlist granted only lanresolver's address read
|
||||
(`pct exec [0-9]* -- ip -4 -o addr show dev eth0`), which is why `has_ip=true` while route, mode and
|
||||
liveness all failed.
|
||||
|
||||
Fixed in **v0.92.1**: a `FELHOM_GUESTNET` alias with four fixed vectors (route, interfaces, pgrep,
|
||||
heal). Every argument after the numeric vmid is a literal, so nothing the guest or the hub says can
|
||||
widen the grant. The address read is **not** duplicated — it stays FELHOM_DNSMASQ's; one command,
|
||||
one grant.
|
||||
|
||||
Plus **four `guestnet-*` capability rows**, so a host that has not taken the new sudoers file is
|
||||
VISIBLE as degraded instead of silently watchdog-less. Deliberately **non-critical**: a missing
|
||||
grant must not page an operator for every box on rollout day (the R-50b amber-fleet lesson).
|
||||
|
||||
**v0.92.0 is superseded, not overwritten — do not vouch it.** It was published before this was
|
||||
found, so its binary lacks the capability rows and its release lacks the sudoers file. A published
|
||||
version stays immutable (the v0.91.0 → v0.91.1 precedent).
|
||||
|
||||
---
|
||||
|
||||
## 5. Tests and red-proofs
|
||||
|
||||
Green gate: `go build ./... && go vet ./... && go test ./...` — all packages ok **except the known
|
||||
flake** `TestGenerateRecoveryCode_EntropyAndFormat` (`internal/escrow`), which fails when the
|
||||
wordlist yields a hyphenated word (`drop-down` → 11 tokens instead of 10). Confirmed pre-existing:
|
||||
`internal/escrow` has not been touched since v0.88.0 and this task changes nothing there; observed
|
||||
3/8 runs, consistent with the documented ~1/5.
|
||||
|
||||
New: `internal/guestnet/watchdog_test.go` (16 cases), `internal/hub/collect_guestnet_test.go` (2),
|
||||
`cmd/felhom-agent/guestnet_wiring_test.go` (2).
|
||||
|
||||
| # | Red-proof | Mutation | Result |
|
||||
|---|---|---|---|
|
||||
| E | detect on process liveness, not address presence | `classify`'s DHCP arm reverted to IP-presence-only | **FAIL ×5.** The decisive one: `classify = "healthy", want "unhealthy"` for the July-20 fixture, `detail="address, default route and dhclient all present"`, and `heal ran 0 times`. That is the 80-minute silent window, reproduced exactly. Restored, green. |
|
||||
| W | the watchdog must be wired | `SetGuestNetReporter` and `go gnWatchdog.Watch(ctx)` both commented out | **FAIL** with both reasons named — *"the guest_net stanza would never reach the hub (the exact v0.91.0 inert-seam defect)"* and *"it would be constructed, reported on, and never probe anything"*. Restored, green. |
|
||||
|
||||
**Every damper is asserted as an exact count, and the load-bearing assertions are the negatives** —
|
||||
a static guest, an unprobeable guest, a boot-race guest (young guest AND young agent), a failed
|
||||
probe tool and an ownership-unproven guest list must each record **zero** heal calls. The ceilings
|
||||
are driven by an injected clock over a scripted **10 hours** of permanent failure: ≤30 heals total,
|
||||
and never a second heal inside the 10-minute cool-off.
|
||||
|
||||
**Seam discipline (§9 rule 6)** — three production-path tests: the `guest_net` stanza is asserted
|
||||
through the real `Collect` (and asserted ABSENT from the wire when no reporter is wired, so "not
|
||||
wired" and "found nothing" can never look identical); and the `main.go` wiring is an AST walk for
|
||||
the construction, the reporter call and the started goroutine. The AST form is deliberate — a
|
||||
`strings.Contains` version of the twin test in felhom-controller **passed its own red-proof**,
|
||||
because a commented-out call still contains the string.
|
||||
|
||||
---
|
||||
|
||||
## 6. Live validation (method: journald + capability self-check on felhom-pve)
|
||||
|
||||
| Step | Evidence |
|
||||
|---|---|
|
||||
| Publish 0.92.0 | `AGENT_SHA256=b1302790d412d22e969936ff52e3ee33e3edc111b8426364a21cdb1c5127ca6a`, round-trip GET verified — **superseded, do not vouch** |
|
||||
| Publish 0.92.1 | `AGENT_SHA256=7424bc1c3c533eff9157e15a18d4635c624931f5a479a48126de77a94e6a3d4d`, round-trip GET verified |
|
||||
| Deploy | `visudo -c` parsed OK → sudoers installed 0440 root:root (backup `/root/felhom-agent.sudoers.bak-preR54`) + binary installed (backup `felhom-agent.bak-0.91.2-preR54`) → `felhom-agent --version` = **0.92.1**, service `active` |
|
||||
| Capability self-check | **`ok=68 total=68 degraded=0 inactive=0`** (was 64/64 before the four `guestnet-*` rows) — the sudoers grant is proven from the agent's own side, not assumed |
|
||||
| Watchdog start | `INFO guestnet: watchdog starting interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s` |
|
||||
| **Healthy cycle** | **`level=DEBUG msg="guestnet: guest network healthy" vmid=9201 mode=dhcp has_route=true dhclient_alive=true`** (12:34:15 CEST) |
|
||||
| Default-ON proven | `/etc/felhom-agent/agent.json` has **no** `guest_net` key at all — the watchdog runs on defaults, which is the whole point of the inverted gate |
|
||||
|
||||
**Note for the operator:** `log_level` on felhom-pve was temporarily raised to `debug` to capture
|
||||
that Debug line (backup at `/root/agent.json.bak-debuglevel`). It is **still `debug`**, deliberately,
|
||||
so STOP-2's heal chain is visible in journald. **Revert it to `info` after STOP-2.**
|
||||
|
||||
---
|
||||
|
||||
## 6b. STOP-2 — the incident replay (operator-present, 2026-07-21)
|
||||
|
||||
The 2026-07-20 kill, repeated deliberately. **The `/proc/<pid>/cgroup` check the incident produced
|
||||
was applied before the kill** — the script refuses unless the pid's cgroup is guest 9201's, which is
|
||||
the rule that would have prevented the original outage:
|
||||
|
||||
```
|
||||
GATE OK — pid 336708 belongs to guest 9201 (0::/lxc/9201/ns/.lxc)
|
||||
KILL at 2026-07-21T10:43:18Z / 12:43:18 CEST
|
||||
after kill: no dhclient running; address 192.168.0.104/24 STILL PRESENT (valid_lft 4998s); default route STILL PRESENT
|
||||
```
|
||||
|
||||
That second line is the whole point: the box looked perfectly healthy, with ~83 minutes of lease
|
||||
left before any symptom would appear.
|
||||
|
||||
| Time (CEST) | Event |
|
||||
|---|---|
|
||||
| 12:43:15 | `DEBUG guest network healthy … dhclient_alive=true` — last good cycle |
|
||||
| 12:43:18 | **kill -9** |
|
||||
| **12:44:15** | **detected in 57 s, on process liveness alone** — `unhealthy (first bad probe — not acting yet) bad_probes=1 required=2`, detail *"dhclient is not running — the lease will not be renewed (the 2026-07-20 failure mode; address still present, renewal already dead)"* |
|
||||
| 12:45:15 | second consecutive bad probe → `healing`, then `running heal command … cmd="pct exec 9201 -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0"` |
|
||||
| **12:45:18** | **`guest network healed` ip=192.168.0.104 has_route=true dhclient_alive=true heals_last_hour=1** |
|
||||
|
||||
**Healed 120 seconds after the kill — roughly 80 minutes before the outage would have begun.** The
|
||||
strongest evidence is therefore what did *not* happen, verified from inside the guest:
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `cloudflared` | **`Up 29 hours`** — the tunnel never dropped, never reconnected |
|
||||
| DNS `gitea.dooplex.hu` | OK |
|
||||
| hub | **302 in 0.16 s** |
|
||||
| `https://felhom.demo-felhom.eu/` | **302 in 0.25 s** |
|
||||
| new dhclient | pid 2043719, `cgroup=0::/lxc/9201/ns/.lxc`, argv byte-identical to the original |
|
||||
|
||||
On 2026-07-20 every one of those was dead for 1h15m. This time the outage was **prevented**, not
|
||||
detected.
|
||||
|
||||
**The negative leg — 30 healthy minutes, run in full.** From the heal at 12:45:18 to 13:16:15:
|
||||
**30 healthy Debug cycles** (exactly one per minute — the cadence is precise), **0 heals**,
|
||||
**0 WARN**, **0 ERROR**. So the damper is not a mute and the quiet path is not silence: "no alarms"
|
||||
stayed continuously distinguishable from "not probing", which is the v0.91.2 lesson holding in
|
||||
production.
|
||||
|
||||
**An unplanned damper proof, against a REAL transient.** STOP-1's guest reboot at 12:53 caught the
|
||||
guest mid-boot:
|
||||
|
||||
```
|
||||
12:53:16 INFO guest network unhealthy (first bad probe — not acting yet) detail="no IPv4 address on eth0" bad_probes=1
|
||||
12:54:15 DEBUG guest network healthy …
|
||||
12:54:15 INFO guest network recovered previous_state=unhealthy
|
||||
```
|
||||
|
||||
One bad probe, **no action**, then recovery. The two-consecutive-probes rule and the boot-race guard
|
||||
did exactly what they exist for — a booting guest was not injected with a dhclient — and this was a
|
||||
genuine transient, not a scripted one.
|
||||
|
||||
## 6c. The stanza ON THE WIRE — confirmed hub-side after STOP-3
|
||||
|
||||
Initially unverifiable here: `--selftest=hub` builds its own one-shot collector and never wires the
|
||||
guestnet reporter, so it would have printed a misleading absence — worse evidence than none. (That
|
||||
divergence between the selftest collector and the daemon's is worth a small fix; the same caveat is
|
||||
already commented in the code for the pbs reporter.)
|
||||
|
||||
Closed instead by a **read-only** query of the hub's own store (`kubectl cp` of `/data/hub.db`,
|
||||
opened `mode=ro`, copy deleted afterwards). The `guest_net` stanza is present, complete, and
|
||||
round-trips every field:
|
||||
|
||||
```json
|
||||
"guest_net": { "checked_at": "2026-07-21T11:32:10Z", "guests": [
|
||||
{ "vmid": 9201, "state": "healthy", "mode": "dhcp", "ip": "192.168.0.104",
|
||||
"has_route": true, "dhclient_alive": true, "checked_at": "2026-07-21T11:31:07Z",
|
||||
"message": "address, default route and dhclient all present" } ] }
|
||||
```
|
||||
|
||||
And the report history tells the whole story of this session from the hub's side:
|
||||
|
||||
| received (UTC) | agent | state | evidence |
|
||||
|---|---|---|---|
|
||||
| 10:30:27 | **0.92.0** | **`unknown`** | the sudoers-blind window — reported honestly as unknown, never as a false healthy and never as a false dead. The fail-safe is visible fleet-side, and this is independent confirmation that 0.92.0 was genuinely blind (so superseding it was right) |
|
||||
| 10:48:12 | 0.92.1 | `healthy` | **`last_heal_at=2026-07-21T10:45:12Z`, `heals_last_hour=1`** — the STOP-2 heal, surfaced to the hub |
|
||||
| 10:49:14 / 11:04:14 | 0.92.1 | `healthy` | same heal stamp, counter still 1 |
|
||||
| 11:32:10 | 0.92.1 | `healthy` | heal history absent — see the limitation below |
|
||||
|
||||
**A limitation this surfaced, worth stating plainly: the damping state is in-memory only.** The
|
||||
counters vanished from the 11:32 report because the agent was restarted at 11:17Z (the `log_level`
|
||||
revert), which resets `heals`/`lastHealAt`. So the "≥10 min apart, ≤3 per hour" ceilings hold within
|
||||
one agent lifetime, not across restarts. In practice the exposure is small — a restart also re-arms
|
||||
the 3-minute settle window and the two-consecutive-bad-probes rule — but a crash-looping agent could
|
||||
heal more often than the ceiling implies. Not worth persisting state for today; worth knowing before
|
||||
anyone quotes the ceiling as a hard guarantee.
|
||||
|
||||
## 7. Deliverables
|
||||
|
||||
- `c0966d7` — v0.92.0: the watchdog, the report block, config, tests.
|
||||
- `0e8fd81` — the sudoers grant + capability rows (the live finding).
|
||||
- `98adb72` — v0.92.1: supersede + version bump.
|
||||
- Published: `felhom-agent` **0.92.1** / `7424bc1c3c533eff…` (0.92.0 superseded).
|
||||
- Live on felhom-pve: binary 0.92.1 + the new sudoers file.
|
||||
|
||||
## 8. Operator actions outstanding
|
||||
|
||||
1. ~~STOP-2~~ — **DONE 2026-07-21, passed** (§6b).
|
||||
2. ~~STOP-3~~ — **DONE 2026-07-21.** Manifest Agent → **0.92.1**, sha matches the published artifact
|
||||
byte for byte; MinAgent → 0.92.1; PBS wrapper sha unchanged (`104db0a4…`, correct — the wrapper
|
||||
was not touched); controller floor → 0.156.0. The host page shows all four `guestnet-*`
|
||||
capability rows **ok**, which is the fleet-visible proof of the sudoers grant.
|
||||
3. ~~Revert `log_level` to `info`~~ — **DONE** (13:17 CEST, after the quiet window closed; agent
|
||||
restarted clean, caps `68/68 ok, degraded=0`, watchdog back up). The temporary raise is recorded
|
||||
here only so the journald volume change is explainable; `/root/agent.json.bak-debuglevel` remains
|
||||
as the pre-change copy.
|
||||
- The `felhom-testing` skill's "Known flake … re-run before diagnosing; it is not a regression" line
|
||||
is now removed. It had been actively harmful: it told every future session to dismiss a true
|
||||
positive. Replaced with the generalised lesson — "known flake, just re-run it" is a diagnosis and
|
||||
needs evidence like any other.
|
||||
- `WordlistSize()` now reports the **effective** (filtered) draw space, 7772. `WordlistFilteredOut()`
|
||||
is new, for audit. Any external consumer expecting a literal 7776 would need updating; there is none
|
||||
outside the tests.
|
||||
|
||||
@@ -92,6 +92,7 @@
|
||||
| `reconcile.Queue.Submit` | internal/reconcile/queue.go | `Submit(vmid, fn) <-chan error` | per-guest serialization of ALL mutations | Same vmid strictly FIFO; lanes parallel across guests |
|
||||
| `Engine.RunSignedJob` | internal/reconcile/job.go | `RunSignedJob(ctx, intent, signed, exec) JobResult` | executing a gated destructive job | Idempotency by nonce; journaled |
|
||||
| `escrow.Create` | internal/escrow/escrow.go | `Create(ctx, CreateOptions) (CreateResult, R, error)` | PBS-key escrow (zero-knowledge) | Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability |
|
||||
| `escrow.GenerateRecoveryCode` / `joinSafe` / `RecoveryCodeSep` | internal/escrow/wordlist.go | `GenerateRecoveryCode() (string, error)` | minting the customer recovery code R | Draws from the EFF large list **filtered of every word containing `RecoveryCodeSep`** (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: **already-issued codes stay valid**, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew |
|
||||
| `escrow.CeremonyBinary` / `CeremonyArgs()` / `CeremonyOutput` | internal/escrow/ceremony.go | the ONE fixed sudo self-invocation argv + the `--output=json` wire object (v1) | controller-driven ceremony (v0.88.0) | SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — `TestEscrowCeremonyArgvPinned` + `TestManifestCoveredBySudoers` lock all three. Never flag-helpers, never `--`→`-` (spike §2.2) |
|
||||
| localapi escrow ceremony job | internal/localapi/escrow_ceremony.go | `POST /escrow/ceremony` + status + ONE-SHOT claim + preflight | the wizard's agent half | R lives ONLY in `Server.escrowR` (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (`unclaimed_void`); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged |
|
||||
| `poke.Listener` + `poke.Port` | internal/poke/poke.go | `NewListener(resolve, trigger, port, logger)`; `poke.Port = 51822` | agent-plane immediate-sync (Direction-2a, v0.89.0) | Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (`wgtunnel.LoadAssignedAddr`), fires the hub-loop out-of-band trigger. **Port 51822 is a SHARED cross-repo contract** — the hub poke sender + the ep0 `felhom-poke` forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (`DebounceWindow`), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when `wg_tunnel.enabled` |
|
||||
|
||||
@@ -11,8 +11,16 @@ import (
|
||||
)
|
||||
|
||||
func TestWordlistLoaded(t *testing.T) {
|
||||
if WordlistSize() != 7776 {
|
||||
t.Fatalf("EFF large wordlist should be 7776 words, got %d", WordlistSize())
|
||||
// The EFF large list is 7776 entries; joinSafe removes the 4 that contain RecoveryCodeSep
|
||||
// (drop-down, felt-tip, t-shirt, yo-yo), leaving 7772 as the effective draw space.
|
||||
if got := WordlistSize(); got != 7772 {
|
||||
t.Fatalf("effective wordlist should be 7772 words (7776 EFF - 4 hyphenated), got %d", got)
|
||||
}
|
||||
if got := WordlistFilteredOut(); got != 4 {
|
||||
t.Fatalf("joinSafe should have removed exactly 4 hyphenated entries, removed %d", got)
|
||||
}
|
||||
if got := WordlistSize() + WordlistFilteredOut(); got != 7776 {
|
||||
t.Fatalf("filtered + removed should reconstitute the 7776-word EFF list, got %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -25,19 +33,27 @@ func TestGenerateRecoveryCode_EntropyAndFormat(t *testing.T) {
|
||||
inList[w] = true
|
||||
}
|
||||
for i := 0; i < 50; i++ {
|
||||
r, err := GenerateRecoveryCode()
|
||||
// Count words by GENERATION count, not by re-splitting the joined string: the two agree
|
||||
// only because joinSafe holds, and conflating them is what made this test flake ~1/5.
|
||||
words, err := generateWords(wordlist)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateRecoveryCode: %v", err)
|
||||
t.Fatalf("generateWords: %v", err)
|
||||
}
|
||||
words := strings.Split(r, "-")
|
||||
if len(words) != RecoveryCodeWords {
|
||||
t.Fatalf("recovery code must be %d words, got %d (%q)", RecoveryCodeWords, len(words), r)
|
||||
t.Fatalf("generator must draw %d words, drew %d", RecoveryCodeWords, len(words))
|
||||
}
|
||||
for _, w := range words {
|
||||
if !inList[w] {
|
||||
t.Errorf("recovery-code word %q is not from the EFF wordlist", w)
|
||||
}
|
||||
}
|
||||
// Separately assert the property joinSafe buys: the joined code segments back to the same
|
||||
// count. Never print r — it is a live-shaped secret.
|
||||
r := strings.Join(words, RecoveryCodeSep)
|
||||
if got := len(strings.Split(r, RecoveryCodeSep)); got != RecoveryCodeWords {
|
||||
t.Fatalf("joined code must segment into %d words, got %d (a drawn word contained %q)",
|
||||
RecoveryCodeWords, got, RecoveryCodeSep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+58
-17
@@ -29,9 +29,20 @@ import (
|
||||
//go:embed eff_large_wordlist.txt
|
||||
var wordlistRaw []byte
|
||||
|
||||
// wordlist is the EFF large wordlist (7776 words, 12.92 bits/word) — the diceware standard for
|
||||
// human-transcribed passphrases. Parsed once at init.
|
||||
var wordlist = parseWordlist(wordlistRaw)
|
||||
// RecoveryCodeSep joins the words of a recovery code R. It is ALSO the reason for the
|
||||
// joinSafe filter below: a word that itself contains the separator makes the joined code
|
||||
// ambiguous to segment by eye, which is unaffordable in the one situation R exists for — a
|
||||
// customer transcribing it during a disaster. Do not change it: R is consumed as a whole
|
||||
// passphrase (see Wrap/Unwrap), so the separator is a transcription aid, not a parsed delimiter.
|
||||
const RecoveryCodeSep = "-"
|
||||
|
||||
// wordlist is the EFF large wordlist (the diceware standard for human-transcribed passphrases),
|
||||
// minus the handful of entries that contain RecoveryCodeSep. Parsed and filtered once at init.
|
||||
// Sizes are asserted in wordlist_test.go so a wordlist swap cannot silently move the entropy floor.
|
||||
var wordlist = joinSafe(parseWordlist(wordlistRaw))
|
||||
|
||||
// wordlistRawSize is the unfiltered parse length, kept for audit (see WordlistFilteredOut).
|
||||
var wordlistRawSize = len(parseWordlist(wordlistRaw))
|
||||
|
||||
func parseWordlist(raw []byte) []string {
|
||||
var w []string
|
||||
@@ -44,28 +55,55 @@ func parseWordlist(raw []byte) []string {
|
||||
return w
|
||||
}
|
||||
|
||||
// RecoveryCodeWords is the number of words in a recovery code R. 10 words from the 7776-word EFF
|
||||
// list ≈ 129.2 bits (≥128) — the chosen entropy floor (slice7-escrow-spike-findings.md §4).
|
||||
// joinSafe drops every word containing RecoveryCodeSep, so that a generated code always segments
|
||||
// back into exactly RecoveryCodeWords words. In the EFF large list this removes exactly 4 entries
|
||||
// (drop-down, felt-tip, t-shirt, yo-yo) of 7776, costing ~0.0007 bits/word — the floor still holds
|
||||
// (asserted in the tests). Generation-time only: codes already issued remain valid, because R is
|
||||
// verified as a whole passphrase and is never re-split.
|
||||
func joinSafe(words []string) []string {
|
||||
out := make([]string, 0, len(words))
|
||||
for _, w := range words {
|
||||
if strings.Contains(w, RecoveryCodeSep) {
|
||||
continue
|
||||
}
|
||||
out = append(out, w)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// RecoveryCodeWords is the number of words in a recovery code R. 10 words from the filtered EFF
|
||||
// list (7772 words) ≈ 129.2 bits (≥128) — the chosen entropy floor (slice7-escrow-spike-findings.md §4).
|
||||
const RecoveryCodeWords = 10
|
||||
|
||||
// GenerateRecoveryCode returns a fresh recovery code R: RecoveryCodeWords words chosen uniformly
|
||||
// (crypto/rand via big.Int — no modulo bias) from the EFF large wordlist, hyphen-joined.
|
||||
//
|
||||
// SECRET: the returned string is R. Surface it to the customer exactly once; never log or persist it.
|
||||
func GenerateRecoveryCode() (string, error) {
|
||||
if len(wordlist) < 2 {
|
||||
return "", fmt.Errorf("escrow: wordlist not loaded (%d words)", len(wordlist))
|
||||
// generateWords draws RecoveryCodeWords words uniformly (crypto/rand via big.Int — no modulo bias)
|
||||
// from list. Split out from GenerateRecoveryCode so tests can drive an unfiltered list and prove
|
||||
// the filter is what keeps a code segmentable.
|
||||
func generateWords(list []string) ([]string, error) {
|
||||
if len(list) < 2 {
|
||||
return nil, fmt.Errorf("escrow: wordlist not loaded (%d words)", len(list))
|
||||
}
|
||||
n := big.NewInt(int64(len(wordlist)))
|
||||
n := big.NewInt(int64(len(list)))
|
||||
words := make([]string, RecoveryCodeWords)
|
||||
for i := range words {
|
||||
idx, err := rand.Int(rand.Reader, n)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("escrow: recovery-code rng: %w", err)
|
||||
return nil, fmt.Errorf("escrow: recovery-code rng: %w", err)
|
||||
}
|
||||
words[i] = wordlist[idx.Int64()]
|
||||
words[i] = list[idx.Int64()]
|
||||
}
|
||||
return strings.Join(words, "-"), nil
|
||||
return words, nil
|
||||
}
|
||||
|
||||
// GenerateRecoveryCode returns a fresh recovery code R: RecoveryCodeWords words chosen uniformly
|
||||
// from the filtered EFF large wordlist, joined with RecoveryCodeSep.
|
||||
//
|
||||
// SECRET: the returned string is R. Surface it to the customer exactly once; never log or persist it.
|
||||
func GenerateRecoveryCode() (string, error) {
|
||||
words, err := generateWords(wordlist)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return strings.Join(words, RecoveryCodeSep), nil
|
||||
}
|
||||
|
||||
// RecoveryCodeEntropyBits is the approximate entropy of a generated code, for display/audit only
|
||||
@@ -77,5 +115,8 @@ func RecoveryCodeEntropyBits() float64 {
|
||||
return float64(RecoveryCodeWords) * math.Log2(float64(len(wordlist)))
|
||||
}
|
||||
|
||||
// WordlistSize is the loaded wordlist length (for audit/tests).
|
||||
// WordlistSize is the effective (filtered) wordlist length — the draw space. For audit/tests.
|
||||
func WordlistSize() int { return len(wordlist) }
|
||||
|
||||
// WordlistFilteredOut is how many parsed entries joinSafe removed. For audit/tests.
|
||||
func WordlistFilteredOut() int { return wordlistRawSize - len(wordlist) }
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
package escrow
|
||||
|
||||
import (
|
||||
"math"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The four EFF large-list entries that contain RecoveryCodeSep. Named here so a wordlist swap that
|
||||
// changes the set fails loudly rather than silently re-opening the ambiguity.
|
||||
var hyphenatedEFFWords = []string{"drop-down", "felt-tip", "t-shirt", "yo-yo"}
|
||||
|
||||
func TestJoinSafe_RemovesExactlyTheHyphenatedEFFWords(t *testing.T) {
|
||||
raw := parseWordlist(wordlistRaw)
|
||||
rawSet := make(map[string]bool, len(raw))
|
||||
for _, w := range raw {
|
||||
rawSet[w] = true
|
||||
}
|
||||
for _, w := range hyphenatedEFFWords {
|
||||
if !rawSet[w] {
|
||||
t.Fatalf("fixture drift: %q is no longer in the embedded EFF list", w)
|
||||
}
|
||||
}
|
||||
|
||||
filtered := joinSafe(raw)
|
||||
if len(raw)-len(filtered) != len(hyphenatedEFFWords) {
|
||||
t.Fatalf("joinSafe removed %d entries, expected exactly %d",
|
||||
len(raw)-len(filtered), len(hyphenatedEFFWords))
|
||||
}
|
||||
got := make(map[string]bool, len(filtered))
|
||||
for _, w := range filtered {
|
||||
if strings.Contains(w, RecoveryCodeSep) {
|
||||
t.Errorf("filtered wordlist still contains a separator-bearing word %q", w)
|
||||
}
|
||||
got[w] = true
|
||||
}
|
||||
for _, w := range hyphenatedEFFWords {
|
||||
if got[w] {
|
||||
t.Errorf("joinSafe kept %q, which contains %q", w, RecoveryCodeSep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestEntropyFloorSurvivesFiltering states the numbers explicitly: dropping 4 of 7776 words costs
|
||||
// ~0.0007 bits/word, so the 10-word code stays above the 128-bit floor with room to spare.
|
||||
func TestEntropyFloorSurvivesFiltering(t *testing.T) {
|
||||
const floorBits = 128.0
|
||||
before := float64(RecoveryCodeWords) * math.Log2(7776)
|
||||
after := RecoveryCodeEntropyBits()
|
||||
|
||||
if after < floorBits {
|
||||
t.Fatalf("filtered entropy %.3f bits is below the %.0f-bit floor", after, floorBits)
|
||||
}
|
||||
if want := float64(RecoveryCodeWords) * math.Log2(float64(WordlistSize())); math.Abs(after-want) > 1e-9 {
|
||||
t.Fatalf("RecoveryCodeEntropyBits() = %.6f, want %.6f (10 * log2(%d))", after, want, WordlistSize())
|
||||
}
|
||||
// Concrete expectations, so a wordlist change that quietly erodes the margin is visible:
|
||||
// 10*log2(7776) = 129.248 bits before, 10*log2(7772) = 129.241 bits after — a 0.007-bit cost.
|
||||
if math.Abs(before-129.248) > 0.001 {
|
||||
t.Fatalf("unfiltered entropy baseline moved: %.3f, expected 129.248", before)
|
||||
}
|
||||
if math.Abs(after-129.241) > 0.001 {
|
||||
t.Fatalf("filtered entropy moved: %.3f, expected 129.241", after)
|
||||
}
|
||||
if cost := before - after; cost > 0.01 {
|
||||
t.Fatalf("filtering cost %.4f bits, expected well under 0.01", cost)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGeneratedCodeSegments_FilteredVsUnfiltered is the deterministic red-proof companion.
|
||||
//
|
||||
// Against a list where EVERY word contains the separator, a 10-word draw MUST segment into more
|
||||
// than 10 parts — that is the pre-fix behaviour, reproduced with probability 1 instead of the ~1/5
|
||||
// flake the real list produced. Against the same list run through joinSafe, generation must refuse
|
||||
// (nothing is left to draw from), proving joinSafe — not luck — is what makes a code segmentable.
|
||||
func TestGeneratedCodeSegments_FilteredVsUnfiltered(t *testing.T) {
|
||||
unfiltered := hyphenatedEFFWords
|
||||
|
||||
words, err := generateWords(unfiltered)
|
||||
if err != nil {
|
||||
t.Fatalf("generateWords(unfiltered): %v", err)
|
||||
}
|
||||
if len(words) != RecoveryCodeWords {
|
||||
t.Fatalf("generator drew %d words, want %d", len(words), RecoveryCodeWords)
|
||||
}
|
||||
joined := strings.Join(words, RecoveryCodeSep)
|
||||
segs := len(strings.Split(joined, RecoveryCodeSep))
|
||||
if segs <= RecoveryCodeWords {
|
||||
t.Fatalf("unfiltered draw segmented into %d parts; the pre-fix defect should yield more than %d",
|
||||
segs, RecoveryCodeWords)
|
||||
}
|
||||
if segs != 2*RecoveryCodeWords {
|
||||
t.Fatalf("every fixture word has exactly one separator, so 10 words must segment into 20 parts, got %d", segs)
|
||||
}
|
||||
|
||||
// Same fixture, filtered: the draw space is empty, so generation must error rather than
|
||||
// silently fall back to something ambiguous.
|
||||
if _, err := generateWords(joinSafe(unfiltered)); err == nil {
|
||||
t.Fatal("generateWords on a fully-filtered list must fail, not return a code")
|
||||
}
|
||||
}
|
||||
|
||||
// TestGenerateRecoveryCode_NeverContainsAmbiguousWord is the production-wiring test: it asserts the
|
||||
// exported entry point (not just the helper) draws from the filtered list.
|
||||
func TestGenerateRecoveryCode_NeverContainsAmbiguousWord(t *testing.T) {
|
||||
inFiltered := make(map[string]bool, len(wordlist))
|
||||
for _, w := range wordlist {
|
||||
inFiltered[w] = true
|
||||
}
|
||||
for i := 0; i < 500; i++ {
|
||||
r, err := GenerateRecoveryCode()
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateRecoveryCode: %v", err)
|
||||
}
|
||||
parts := strings.Split(r, RecoveryCodeSep)
|
||||
if len(parts) != RecoveryCodeWords {
|
||||
// Do not print r: it is a live-shaped secret.
|
||||
t.Fatalf("code %d segmented into %d parts, want %d", i, len(parts), RecoveryCodeWords)
|
||||
}
|
||||
for _, p := range parts {
|
||||
if !inFiltered[p] {
|
||||
t.Fatalf("segment %q is not a filtered-wordlist word", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user