diff --git a/documentation/audits/DIAG-agent-channel-2026-07-26.md b/documentation/audits/DIAG-agent-channel-2026-07-26.md new file mode 100644 index 0000000..a65b2f6 --- /dev/null +++ b/documentation/audits/DIAG-agent-channel-2026-07-26.md @@ -0,0 +1,353 @@ +# DIAG — agent-channel banner + felhom-samba false alarm (2026-07-26) + +**Read-only diagnostic. Nothing was restarted, reloaded, reconfigured or deployed.** No +`systemctl` action, no `ifreload`/`ifup`, no config edit, no runbook leg, no sharing change, no +container created or removed. Every probe was a read, a log tail, or a kernel *route lookup* +(`ip route get`, which sends no packets). Two TCP connect attempts were made from guest shells — +explicitly sanctioned by the brief as non-mutating. + +**Verdict up front, and it is unambiguous:** + +> The island migration rewrote **`bootstrap.json`** but the controller reads **`controller.yaml`**, +> and `ensureLocalAPI` only fills a `local_api.endpoint` that is **missing** — never one that is +> **stale**. Both controllers are still dialling the pre-island LAN address, where the agent no +> longer listens. **H2 SUPPORTED. H1, H3, H4 all REFUTED by direct observation.** + +Symptom B is confirmed, and is a **side-effect of leg 4 of the R-75 task** (sharing enabled on +demo-hp without a household password) — see §5, including the honest note that this diagnostic's own +author caused it. + +--- + +## 1. D0 — access inventory + +| Target | Reachable | Path | User | sudo | +|---|---|---|---|---| +| demo-felhom PVE host | **YES** | `ssh felhom-pve` (Tailscale alias) | `root` (uid 0) | n/a — already root | +| demo-felhom guest 9201 | **YES** | `pct exec 9201` from the host | root in guest | n/a | +| demo-hp PVE host | **YES** | `sshpass … root@demo-hp` (TS `100.76.96.79`) | `root` (uid 0) | n/a | +| demo-hp guest 9201 | **YES** | `pct exec 9201` from the host | root in guest | n/a | + +**How demo-hp was reached, stated plainly:** it has no baked operator key. I used the **documented +G1 break-glass** — the root password vaulted in the hub `host_recovery/demo-hp-bb76ea`, per +`felhom-controller/CLAUDE.md:133` and `operations/nodes.md`. That is the sanctioned and only access +path to that box, not an escalation. The hub DB copy and the plaintext password were `shred -u`'d +immediately after use, and no credential appears in this document. + +**No access gaps.** Everything §D1–§D6 asked for was collectable except one item, recorded in §7. + +**Exit-code discipline.** Most probes ran through `ssh → pct exec → bash -c` pipelines where `$?` is +the last stage's status, so I read **stderr and content**, not rc. Three probes failed *visibly* and +were re-run: two quoting failures (fixed by base64-encoding the probe script — the +`ssh → pct exec → bash -c` escaping chain mangles `(`, `'` and `$`), and one wrong SQLite column +name. `agent.json` reported "unreadable" — recorded in §7 rather than glossed. + +--- + +## 2. D1/D2/D3 — Symptom A evidence table + +| | **demo-felhom** | **demo-hp** | +|---|---|---| +| Reason | `unreachable` | `unreachable` | +| Which branch | `checker.go:94-96` (`connection refused`) — **NOT** the `default` unknown case | same | +| Raw error | `agentapi: GET /storage: Get "https://192.168.0.162:8443/storage": dial tcp 192.168.0.162:8443: connect: connection refused` | `agentapi: GET /storage: Get "https://192.168.0.87:8443/storage": dial tcp 192.168.0.87:8443: connect: connection refused` | +| Transition logged | `up->down:unreachable` | `up->down:unreachable` (preceded by `transient down (unreachable, 1/2)`) | +| **Actually born-down?** | **YES — see §2.1** | **YES** | +| Ever recovered? | **No** `agent_channel_recovered` since 2026-07-20 | **No**, ever | +| First seen (hub, persisted) | **2026-07-25 12:46:42 UTC** | **2026-07-25 12:44:19 UTC** | +| `controller.yaml` endpoint | `192.168.0.162:8443` **(old LAN)** | `192.168.0.87:8443` **(old LAN)** | +| `controller.yaml` mtime | **2026-07-21 08:29:31** | **2026-07-23 09:53:39** | +| `bootstrap.json` endpoint | `169.254.253.1:8443` **(island)** | `169.254.253.1:8443` **(island)** | +| `bootstrap.json` mtime | **2026-07-25 12:44:41** | **2026-07-25 12:42:17** | +| **Agree?** | **NO** | **NO** | +| token / fingerprint | both present; fingerprint 64 chars | both present; fingerprint 64 chars | +| Agent listening | `169.254.253.1:8443` **only** | `169.254.253.1:8443` **only** | +| `felhom-agent` | `active`, since 2026-07-25 14:44:37 CEST | `active`, since 2026-07-25 14:42:13 CEST | +| `vmbr9` | `169.254.253.1/30`, up | `169.254.253.1/30`, up | +| `vmbr0` | `192.168.0.162/24`, normal | `192.168.0.87/24`, normal | +| Persistent vmbr9 stanza | present in `/etc/network/interfaces` | present | +| Guest `net1` | `eth1,bridge=vmbr9,ip=169.254.253.2/30` | same | +| Guest `eth1` | `169.254.253.2/30`, up | `169.254.253.2/30`, up | +| Host uptime | up since **2026-07-22 19:13** (no reboot since migration) | up since **2026-07-23 20:30** (no reboot since migration) | +| TCP **from guest** → island `169.254.253.1:8443` | **OPEN** | **OPEN** | +| TCP **from guest** → old LAN `:8443` | **REFUSED** | **REFUSED** | +| TCP **from container** | **not collectable** — the image has no `nc`/`wget`/`ip`; answered by route simulation instead (§2.2) | same | + +*(Tokens and fingerprints were redacted at the point of collection — the probe script rewrote them to +`` before the value ever left the box.)* + +### 2.1 `up->down` is an ARTIFACT — the channel was never up + +`checker.go` seeds an unseeded state to `"up"` inside the debounce branch: + +```go +if cls.debounce && c.consecutiveDown < debounceThreshold { + if c.state == "" { c.state = "up" } // <-- unseeded assumed up until confirmed +``` + +So the sequence is: unseeded → probe 1 fails → state artificially set `"up"` → probe 2 fails → +logged as `up->down`. **Neither controller has ever reached the agent since it restarted.** This is +not a degradation of a working channel; it is a wrong address read at startup. Do not read the +`up->` in that log line as evidence the channel once worked. + +### 2.2 Container → island: route-simulated, not packet-proven + +The controller runs `NetworkMode=bridge` (`172.17.0.2` on `docker0`, `172.18.0.4` on +`traefik-public`), so the guest-shell result does not automatically transfer. The image ships no +`nc`, `wget` or `ip`, and creating a probe container would have been a mutation, so I asked the +kernel for its forwarding decision instead — a pure lookup: + +``` +ip route get 169.254.253.1 from 172.17.0.2 iif docker0 + -> 169.254.253.1 from 172.17.0.2 dev eth1 cache iif docker0 +ip route get 169.254.253.1 from 172.18.0.4 iif br- + -> 169.254.253.1 from 172.18.0.4 dev eth1 cache iif br- +``` + +Identical on both boxes. Supporting conditions all present: `net.ipv4.ip_forward=1`; `rp_filter=2` +(loose, will not drop the asymmetric-looking reply); `-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j +MASQUERADE` (and the same for `172.18.0.0/16`), so the source becomes `169.254.253.2`; +`DOCKER-FORWARD` ACCEPTs `-i docker0` and `-i br-`. + +**This is strong but not conclusive** — it is a routing-table verdict, not a delivered packet. It +matters because the R-50 spike chose link-local *precisely because* RFC-3927 "structurally does not +route" (`SPIKE-island-bridge-2026-07-25.md:155-161`), and that spike's P5 proved the island reachable +**from the guest**, never from a bridged container. **The fix in §6.1 should be validated by watching +for `agent channel recovered`, not assumed.** + +--- + +## 3. D2 — when it started (the H3 killer) + +`agent_channel_unreachable`, from the **hub's persisted `events` table**, which survives the +container-log truncation the brief warned about: + +``` +2026-07-25 12:31:55 drill-r50 agent_channel_unreachable +2026-07-25 12:44:19 demo-hp agent_channel_unreachable <-- FIRST +2026-07-25 12:46:42 demo-felhom agent_channel_unreachable <-- FIRST +2026-07-26 06:16:07 demo-felhom (re-fire after v0.172.0 restart) +2026-07-26 06:19:24 demo-hp (re-fire) +2026-07-26 06:25:16 demo-felhom (re-fire) +2026-07-26 06:25:41 demo-hp (re-fire) +``` + +Alongside, on demo-hp: `controller_started (0.167.1)` at **2026-07-25 12:42:24** — i.e. the migration +restarted the controller, which re-read the stale `controller.yaml`, two minutes before the first +alarm. `bootstrap.json` mtime is 12:42:17 on that box: **the migration wrote bootstrap.json and +restarted the controller within seven seconds, and the controller ignored the new file.** + +### VERDICT: the banner PRE-DATES today's v0.172.0 deploy by ~17.5 hours. **H3 is dead.** + +Today's 06:16–06:25 events are re-fires: channel state is in-memory (`checker.go`: *"No persistence — +the state is re-derived each run"*), so every controller restart re-alarms. Four controller restarts +today (two per box, from the two rebuilds) explain them exactly. + +--- + +## 4. Hypothesis verdicts + +| | Verdict | The observation that decided it | +|---|---|---| +| **H1 — island fallout** | **REFUTED** | Every island leg is healthy on both hosts: `felhom-agent` `active`, bound `169.254.253.1:8443`; `vmbr9` up at `169.254.253.1/30` with a persistent `/etc/network/interfaces` stanza; guest `net1`/`eth1` up at `169.254.253.2/30`; `vmbr0` untouched. **TCP from the guest to `169.254.253.1:8443` is OPEN on both boxes.** The island works; nothing is using it. | +| **H2 — endpoint in two places** | **SUPPORTED — this is the cause** | `bootstrap.json` = `169.254.253.1:8443` (mtime = migration). `controller.yaml` = the old LAN address (mtime **predates** the migration by 2–4 days). The controller dials exactly what `controller.yaml` says. Mechanism named in §4.1. | +| **H3 — R-75 / v0.172.0** | **REFUTED** | First alarm 2026-07-25 12:44/12:46; first v0.172.0 deploy 2026-07-26 ~06:14. R-75 touched no agent, network or `local_api` surface. It is a coincidence of *noticing*, not of cause. | +| **H4 — genuinely down** | **REFUTED** | `systemctl is-active felhom-agent` → `active` on both hosts, since the migration, listening on the island address. The agent is up; the controller is knocking on the wrong door. | + +### 4.1 The exact mechanism (source-confirmed) + +`internal/bootstrap/bootstrap.go:254`: + +```go +func ensureLocalAPI(configPath string, cfg *config.Config, logger *log.Logger) *config.Config { + if cfg == nil || cfg.LocalAPI.Endpoint != "" { + return cfg // already has the agent path → nothing to do + } +``` + +`MaybeIngest` short-circuits for an already-configured guest (`cfg.Customer.ID != ""`) and delegates +to `ensureLocalAPI`, whose contract is documented as *"merges it in from bootstrap.json **if +missing**"*. It fills an **empty** endpoint; it never reconciles a **stale** one. + +So: **`controller.yaml` is generated from `bootstrap.json` exactly once, at Day-0, and maintained +independently thereafter.** Anything that later rewrites `bootstrap.json` — the island migration — +silently fails to reach the controller. The mtimes are the proof: `bootstrap.json` 2026-07-25, +`controller.yaml` 2026-07-21 / 2026-07-23. + +The stale comment the brief flagged is real and is a contributing signpost — +`config/config.go:75` still documents the field as *"host bridge IP:port, e.g. 192.168.0.162:8443"*, +i.e. the pre-island shape, and that is literally the value now sitting in the file. + +--- + +## 5. Symptom B — `Protected container not running: felhom-samba` + +**Mechanism CONFIRMED.** + +- `docker ps -a --filter name=felhom-samba` on demo-hp → **empty**; the container was never created. +- Settings: `{'enabled': True, 'server_name': 'FELHOM-HP'}` — **no `user_set` key**, i.e. the + household password has never been set. (Read only; nothing changed.) +- `reconcileSambaAt` correctly refuses to deploy: `if !smb.UserSet { … "deploy skipped — household + SMB password not set yet" … return nil }`. +- `EffectiveProtected` (`healthcheck.go:267`) nevertheless adds it: + +```go +for _, name := range cfg.Stacks.Protected { + if name == "cloudflared" && cfg.Infrastructure.CFTunnelToken == "" { continue } // <-- has a predicate + out = append(out, name) +} +if smb.Enabled { // <-- no password / deployed predicate + out = append(out, infra.SambaContainerName) +} +``` + +Live confirmation: +``` +[DEBUG] [monitor] Checking 5 protected containers: [traefik cloudflared felhom-controller filebrowser felhom-samba] +[DEBUG] [monitor] Protected containers missing: [felhom-samba] +[DEBUG] [monitor] Final status: fail (issues=1, warnings=0, info=5) +``` + +**The diagnosis holds:** sharing enabled + no password → samba deliberately not deployed → the health +monitor reports FAIL for an intentional state. + +**A sharper finding than the brief anticipated: the function's own doc comment asserts the invariant +that is broken.** `healthcheck.go:260-261` says *"The bring-up applies the same conditions +(`stacks.EnsureBaseStack` for cloudflared, `ensureSamba`'s `if !smb.Enabled { return }` for samba), so +detection and deployment agree in both directions."* That was true when written. `reconcileSambaAt` +now has a **second** early return on `!smb.UserSet` that the protected set never mirrored, so the two +sides silently diverged and the comment documents a guarantee the code no longer provides. + +**I caused this state.** Sharing was enabled on demo-hp at ~06:20 UTC today by leg 4 of the R-75 task +(explicitly authorised by that brief). The hub data shows the box was `ok` for 124 consecutive +reports and flipped at the first cycle after that change. + +--- + +## 6. D6 — blast radius + +### Symptom A — 4 operator emails, no customer email + +`notification_log`, `event_type LIKE 'agent_channel%'`: + +| when (UTC) | box | channel | status | +|---|---|---|---| +| 2026-07-25 12:44:20 | demo-hp | operator | sent | +| 2026-07-25 12:46:43 | demo-felhom | operator | sent | +| 2026-07-26 06:16:08 | demo-felhom | operator | sent | +| 2026-07-26 06:19:24 | demo-hp | operator | sent | + +Four, not one per cycle — the checker's `alerted` flag suppresses repeats within a down-spell, and +the hub applies its own cooldown (7 hub *events* today produced 2 emails). `warning` severity is +operator-only, so **no customer was emailed**. `drill-r50` also alarmed twice on 2026-07-25, which is +consistent: it is on the same island and was the migration's first subject. + +### Symptom B — zero emails so far, but a customer-facing one is plausibly pending + +- **Zero** notifications mentioning `samba` or `Protected`, ever. +- demo-hp report health: `ok` ×124 through 2026-07-26 06:23:47, then **`fail` at 06:38:42** — one + fail cycle at the time of the snapshot. demo-felhom stayed `ok` throughout (control: sharing there + has a password and samba is running). +- **But the path exists and has fired before.** `health_degraded` / `health_critical` are real + notification types, and `health_critical` has gone to the **customer** channel three times + historically (2026-02-24, 2026-06-15, 2026-07-18: *"Rendszer állapot kritikus (volt: ok)"*). A + sustained `fail` on demo-hp is therefore likely to produce an operator **and possibly a customer** + email on the next transition evaluation. It had not fired as of the last snapshot. **This is a live, + pending consequence, not a closed one.** + +### The English-string defect is WHOLE-SURFACE, not one-off + +Every `report.Issues` and `report.Warnings` string in the codebase is English: + +``` +healthcheck.go:56 "SSD disk usage critical: %.0f%%" +healthcheck.go:82 "HDD disk usage critical: %.0f%%" +healthcheck.go:150 "Docker: %v" +healthcheck.go:169 "Protected container not running: %s" +healthcheck.go:64/87/97/115/133 "SSD/HDD disk usage high", "Memory usage high", "CPU usage high", "Temperature high" +``` + +So this is not a slip in one line — the whole health-issue surface is operator-English and is +rendered on a customer-facing Hungarian page. Nothing was changed. + +--- + +## 7. Not collected, and what it would take + +| Item | Why | What would get it | +|---|---|---| +| Live TCP connect **from inside the controller container** to `169.254.253.1:8443` | the image ships no `nc`/`wget`/`ip`, and creating a probe container would be a mutation this task forbids | one `docker run --rm --network container:felhom-controller alpine nc -z 169.254.253.1 8443` — a throwaway container, no touch to the controller. Answered indirectly by the route simulation in §2.2, which is strong but not a delivered packet | +| `agent.json` `listen_addr` value | `/etc/felhom-agent/agent.json` read as "unreadable" on both hosts even as root — the path is likely different in v0.96.0 | `systemctl cat felhom-agent` for the actual config path, or the agent's own startup log. **`ss` already proves the effective bind** (`169.254.253.1:8443`), so this is corroboration, not a gap in the conclusion | +| Whether a `health_critical` email will fire for demo-hp | only one `fail` cycle had elapsed at snapshot time | re-check `notification_log` in an hour, or read the hub's health-transition checker thresholds | + +--- + +## 8. Proposed fixes — DESCRIBED, NOT APPLIED + +### 8.1 The endpoint (Symptom A) — the actual outage + +**Immediate, per box:** set `local_api.endpoint` in `controller.yaml` to `169.254.253.1:8443` on both +guests and restart the controller. Two boxes, one line each. **Safe-obvious in shape, but validate +rather than assume** — §2.2's link-local routability from a bridged container is route-simulated, not +packet-proven. Success signal: `[INFO] [channel] agent channel recovered` plus an +`agent_channel_recovered` hub event. + +**The real fix — `ensureLocalAPI` (`internal/bootstrap/bootstrap.go:254`) reconciles rather than only +fills.** Its guard `if cfg.LocalAPI.Endpoint != "" { return cfg }` is what made the migration a no-op +for the controller. If `bootstrap.json` carries a `local_api` block that differs from +`controller.yaml`, `bootstrap.json` should win (it is the per-guest source of truth the agent side +writes) and the file should be rewritten. **Needs a small spike, not a one-liner:** it changes which +file is authoritative on every boot, so it must not clobber a legitimately setup-wizard-configured or +hand-repaired guest, and the token/fingerprint must be reconciled atomically with the endpoint or a +half-updated triple is worse than a stale one. Also worth deciding whether a mismatch should be a +loud operator event in its own right — right now the two files can disagree indefinitely in silence. + +**Adjacent, trivial:** `config/config.go:75`'s comment still documents the pre-island shape +(*"host bridge IP:port, e.g. 192.168.0.162:8443"*). Safe-obvious doc fix. + +**Process, not code:** `RUNBOOK-island-migration.md` should gain a post-migration verification step +that asserts `controller.yaml` and `bootstrap.json` agree, and that the channel recovered. The +migration reported success while leaving every controller on the fleet blind to its agent for 18 h. + +### 8.2 `EffectiveProtected` (Symptom B) + +`internal/monitor/healthcheck.go:267` — gate samba on the same predicate the bring-up uses, i.e. +`smb.Enabled && smb.UserSet`, mirroring the cloudflared branch five lines above. **Safe-obvious**, one +condition, and it makes the function's own doc comment true again — that comment at L260-261 must be +updated in the same change, since it currently asserts a parity that does not hold. A red-proof is +easy: enabled-without-password must yield a protected set excluding samba, and enabled-with-password +must include it. + +### 8.3 The English strings on a Hungarian page + +Whole-surface (§6), so this is a **copy/i18n arc, not a one-line fix** — every `report.Issues` and +`report.Warnings` producer, plus a decision about where translation belongs (at the producer, or at +the render/notification boundary where operator-English and customer-Hungarian already diverge). The +`health_critical` customer email suggests the boundary already exists and these strings simply bypass +it. **Needs a spike** to pick the seam; the strings themselves are then mechanical. + +--- + +## 9. Observations — noticed, not acted on + +1. **`drill-r50` is also alarming** (2026-07-25 10:04, 10:11, 10:14, 12:31) and was the migration's + first subject. If that VM still exists it is likely in the same stale-endpoint state. Out of scope + here; worth a look when the fix lands. Its 10:04–10:14 alarms *precede* the demo boxes' 12:4x, so + it may also carry earlier, unrelated spike churn. +2. **`expected_backup_missed` is firing nightly on all three customers** (2026-07-26 03:00, including + a **customer**-channel email to demo-felhom: *"newest backup is 176h0m0s old"* = 7.3 days). That is + a separate, louder problem than either symptom here, and it reached a customer channel. Almost + certainly downstream of the same outage — the agent channel is how PBS backup state is learned — + but I did not verify the causal link. +3. **The channel checker has no persistence by design**, so a restart re-alarms and the operator sees + a "new" alert for an old condition. That is defensible (state re-derivation is the model) but it + is why this looked like it started today. +4. **`rp_filter=2` (loose) on both guests** is what makes the container→island path viable at all; + under strict `rp_filter=1` the asymmetry could bite. Worth knowing before anyone hardens it. +5. **demo-hp has been up 2 d 12 h and demo-felhom 3 d 13 h — neither has rebooted since the + migration.** So the persistent `vmbr9` stanza in `/etc/network/interfaces` is present but has + **never been exercised across a boot** on either box. The island's reboot-survival remains + untested in the field. +6. **Sharing is still ON with no password on demo-hp**, which is what keeps Symptom B live. Left + exactly as found, per the prime directive — but it means the box will keep reporting `fail` until + either the fix lands or sharing is turned off.