# DIAG — agent-channel banner + felhom-samba false alarm (2026-07-26) **Read-only diagnostic. Nothing was restarted, reloaded, reconfigured or deployed.** No `systemctl` action, no `ifreload`/`ifup`, no config edit, no runbook leg, no sharing change, no container created or removed. Every probe was a read, a log tail, or a kernel *route lookup* (`ip route get`, which sends no packets). Two TCP connect attempts were made from guest shells — explicitly sanctioned by the brief as non-mutating. **Verdict up front, and it is unambiguous:** > The island migration rewrote **`bootstrap.json`** but the controller reads **`controller.yaml`**, > and `ensureLocalAPI` only fills a `local_api.endpoint` that is **missing** — never one that is > **stale**. Both controllers are still dialling the pre-island LAN address, where the agent no > longer listens. **H2 SUPPORTED. H1, H3, H4 all REFUTED by direct observation.** Symptom B is confirmed, and is a **side-effect of leg 4 of the R-75 task** (sharing enabled on demo-hp without a household password) — see §5, including the honest note that this diagnostic's own author caused it. --- ## 1. D0 — access inventory | Target | Reachable | Path | User | sudo | |---|---|---|---|---| | demo-felhom PVE host | **YES** | `ssh felhom-pve` (Tailscale alias) | `root` (uid 0) | n/a — already root | | demo-felhom guest 9201 | **YES** | `pct exec 9201` from the host | root in guest | n/a | | demo-hp PVE host | **YES** | `sshpass … root@demo-hp` (TS `100.76.96.79`) | `root` (uid 0) | n/a | | demo-hp guest 9201 | **YES** | `pct exec 9201` from the host | root in guest | n/a | **How demo-hp was reached, stated plainly:** it has no baked operator key. I used the **documented G1 break-glass** — the root password vaulted in the hub `host_recovery/demo-hp-bb76ea`, per `felhom-controller/CLAUDE.md:133` and `operations/nodes.md`. That is the sanctioned and only access path to that box, not an escalation. The hub DB copy and the plaintext password were `shred -u`'d immediately after use, and no credential appears in this document. **No access gaps.** Everything §D1–§D6 asked for was collectable except one item, recorded in §7. **Exit-code discipline.** Most probes ran through `ssh → pct exec → bash -c` pipelines where `$?` is the last stage's status, so I read **stderr and content**, not rc. Three probes failed *visibly* and were re-run: two quoting failures (fixed by base64-encoding the probe script — the `ssh → pct exec → bash -c` escaping chain mangles `(`, `'` and `$`), and one wrong SQLite column name. `agent.json` reported "unreadable" — recorded in §7 rather than glossed. --- ## 2. D1/D2/D3 — Symptom A evidence table | | **demo-felhom** | **demo-hp** | |---|---|---| | Reason | `unreachable` | `unreachable` | | Which branch | `checker.go:94-96` (`connection refused`) — **NOT** the `default` unknown case | same | | Raw error | `agentapi: GET /storage: Get "https://192.168.0.162:8443/storage": dial tcp 192.168.0.162:8443: connect: connection refused` | `agentapi: GET /storage: Get "https://192.168.0.87:8443/storage": dial tcp 192.168.0.87:8443: connect: connection refused` | | Transition logged | `up->down:unreachable` | `up->down:unreachable` (preceded by `transient down (unreachable, 1/2)`) | | **Actually born-down?** | **YES — see §2.1** | **YES** | | Ever recovered? | **No** `agent_channel_recovered` since 2026-07-20 | **No**, ever | | First seen (hub, persisted) | **2026-07-25 12:46:42 UTC** | **2026-07-25 12:44:19 UTC** | | `controller.yaml` endpoint | `192.168.0.162:8443` **(old LAN)** | `192.168.0.87:8443` **(old LAN)** | | `controller.yaml` mtime | **2026-07-21 08:29:31** | **2026-07-23 09:53:39** | | `bootstrap.json` endpoint | `169.254.253.1:8443` **(island)** | `169.254.253.1:8443` **(island)** | | `bootstrap.json` mtime | **2026-07-25 12:44:41** | **2026-07-25 12:42:17** | | **Agree?** | **NO** | **NO** | | token / fingerprint | both present; fingerprint 64 chars | both present; fingerprint 64 chars | | Agent listening | `169.254.253.1:8443` **only** | `169.254.253.1:8443` **only** | | `felhom-agent` | `active`, since 2026-07-25 14:44:37 CEST | `active`, since 2026-07-25 14:42:13 CEST | | `vmbr9` | `169.254.253.1/30`, up | `169.254.253.1/30`, up | | `vmbr0` | `192.168.0.162/24`, normal | `192.168.0.87/24`, normal | | Persistent vmbr9 stanza | present in `/etc/network/interfaces` | present | | Guest `net1` | `eth1,bridge=vmbr9,ip=169.254.253.2/30` | same | | Guest `eth1` | `169.254.253.2/30`, up | `169.254.253.2/30`, up | | Host uptime | up since **2026-07-22 19:13** (no reboot since migration) | up since **2026-07-23 20:30** (no reboot since migration) | | TCP **from guest** → island `169.254.253.1:8443` | **OPEN** | **OPEN** | | TCP **from guest** → old LAN `:8443` | **REFUSED** | **REFUSED** | | TCP **from container** | **not collectable** — the image has no `nc`/`wget`/`ip`; answered by route simulation instead (§2.2) | same | *(Tokens and fingerprints were redacted at the point of collection — the probe script rewrote them to `` before the value ever left the box.)* ### 2.1 `up->down` is an ARTIFACT — the channel was never up `checker.go` seeds an unseeded state to `"up"` inside the debounce branch: ```go if cls.debounce && c.consecutiveDown < debounceThreshold { if c.state == "" { c.state = "up" } // <-- unseeded assumed up until confirmed ``` So the sequence is: unseeded → probe 1 fails → state artificially set `"up"` → probe 2 fails → logged as `up->down`. **Neither controller has ever reached the agent since it restarted.** This is not a degradation of a working channel; it is a wrong address read at startup. Do not read the `up->` in that log line as evidence the channel once worked. ### 2.2 Container → island: route-simulated, not packet-proven The controller runs `NetworkMode=bridge` (`172.17.0.2` on `docker0`, `172.18.0.4` on `traefik-public`), so the guest-shell result does not automatically transfer. The image ships no `nc`, `wget` or `ip`, and creating a probe container would have been a mutation, so I asked the kernel for its forwarding decision instead — a pure lookup: ``` ip route get 169.254.253.1 from 172.17.0.2 iif docker0 -> 169.254.253.1 from 172.17.0.2 dev eth1 cache iif docker0 ip route get 169.254.253.1 from 172.18.0.4 iif br- -> 169.254.253.1 from 172.18.0.4 dev eth1 cache iif br- ``` Identical on both boxes. Supporting conditions all present: `net.ipv4.ip_forward=1`; `rp_filter=2` (loose, will not drop the asymmetric-looking reply); `-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE` (and the same for `172.18.0.0/16`), so the source becomes `169.254.253.2`; `DOCKER-FORWARD` ACCEPTs `-i docker0` and `-i br-`. **This is strong but not conclusive** — it is a routing-table verdict, not a delivered packet. It matters because the R-50 spike chose link-local *precisely because* RFC-3927 "structurally does not route" (`SPIKE-island-bridge-2026-07-25.md:155-161`), and that spike's P5 proved the island reachable **from the guest**, never from a bridged container. **The fix in §6.1 should be validated by watching for `agent channel recovered`, not assumed.** --- ## 3. D2 — when it started (the H3 killer) `agent_channel_unreachable`, from the **hub's persisted `events` table**, which survives the container-log truncation the brief warned about: ``` 2026-07-25 12:31:55 drill-r50 agent_channel_unreachable 2026-07-25 12:44:19 demo-hp agent_channel_unreachable <-- FIRST 2026-07-25 12:46:42 demo-felhom agent_channel_unreachable <-- FIRST 2026-07-26 06:16:07 demo-felhom (re-fire after v0.172.0 restart) 2026-07-26 06:19:24 demo-hp (re-fire) 2026-07-26 06:25:16 demo-felhom (re-fire) 2026-07-26 06:25:41 demo-hp (re-fire) ``` Alongside, on demo-hp: `controller_started (0.167.1)` at **2026-07-25 12:42:24** — i.e. the migration restarted the controller, which re-read the stale `controller.yaml`, two minutes before the first alarm. `bootstrap.json` mtime is 12:42:17 on that box: **the migration wrote bootstrap.json and restarted the controller within seven seconds, and the controller ignored the new file.** ### VERDICT: the banner PRE-DATES today's v0.172.0 deploy by ~17.5 hours. **H3 is dead.** Today's 06:16–06:25 events are re-fires: channel state is in-memory (`checker.go`: *"No persistence — the state is re-derived each run"*), so every controller restart re-alarms. Four controller restarts today (two per box, from the two rebuilds) explain them exactly. --- ## 4. Hypothesis verdicts | | Verdict | The observation that decided it | |---|---|---| | **H1 — island fallout** | **REFUTED** | Every island leg is healthy on both hosts: `felhom-agent` `active`, bound `169.254.253.1:8443`; `vmbr9` up at `169.254.253.1/30` with a persistent `/etc/network/interfaces` stanza; guest `net1`/`eth1` up at `169.254.253.2/30`; `vmbr0` untouched. **TCP from the guest to `169.254.253.1:8443` is OPEN on both boxes.** The island works; nothing is using it. | | **H2 — endpoint in two places** | **SUPPORTED — this is the cause** | `bootstrap.json` = `169.254.253.1:8443` (mtime = migration). `controller.yaml` = the old LAN address (mtime **predates** the migration by 2–4 days). The controller dials exactly what `controller.yaml` says. Mechanism named in §4.1. | | **H3 — R-75 / v0.172.0** | **REFUTED** | First alarm 2026-07-25 12:44/12:46; first v0.172.0 deploy 2026-07-26 ~06:14. R-75 touched no agent, network or `local_api` surface. It is a coincidence of *noticing*, not of cause. | | **H4 — genuinely down** | **REFUTED** | `systemctl is-active felhom-agent` → `active` on both hosts, since the migration, listening on the island address. The agent is up; the controller is knocking on the wrong door. | ### 4.1 The exact mechanism (source-confirmed) `internal/bootstrap/bootstrap.go:254`: ```go func ensureLocalAPI(configPath string, cfg *config.Config, logger *log.Logger) *config.Config { if cfg == nil || cfg.LocalAPI.Endpoint != "" { return cfg // already has the agent path → nothing to do } ``` `MaybeIngest` short-circuits for an already-configured guest (`cfg.Customer.ID != ""`) and delegates to `ensureLocalAPI`, whose contract is documented as *"merges it in from bootstrap.json **if missing**"*. It fills an **empty** endpoint; it never reconciles a **stale** one. So: **`controller.yaml` is generated from `bootstrap.json` exactly once, at Day-0, and maintained independently thereafter.** Anything that later rewrites `bootstrap.json` — the island migration — silently fails to reach the controller. The mtimes are the proof: `bootstrap.json` 2026-07-25, `controller.yaml` 2026-07-21 / 2026-07-23. The stale comment the brief flagged is real and is a contributing signpost — `config/config.go:75` still documents the field as *"host bridge IP:port, e.g. 192.168.0.162:8443"*, i.e. the pre-island shape, and that is literally the value now sitting in the file. --- ## 5. Symptom B — `Protected container not running: felhom-samba` **Mechanism CONFIRMED.** - `docker ps -a --filter name=felhom-samba` on demo-hp → **empty**; the container was never created. - Settings: `{'enabled': True, 'server_name': 'FELHOM-HP'}` — **no `user_set` key**, i.e. the household password has never been set. (Read only; nothing changed.) - `reconcileSambaAt` correctly refuses to deploy: `if !smb.UserSet { … "deploy skipped — household SMB password not set yet" … return nil }`. - `EffectiveProtected` (`healthcheck.go:267`) nevertheless adds it: ```go for _, name := range cfg.Stacks.Protected { if name == "cloudflared" && cfg.Infrastructure.CFTunnelToken == "" { continue } // <-- has a predicate out = append(out, name) } if smb.Enabled { // <-- no password / deployed predicate out = append(out, infra.SambaContainerName) } ``` Live confirmation: ``` [DEBUG] [monitor] Checking 5 protected containers: [traefik cloudflared felhom-controller filebrowser felhom-samba] [DEBUG] [monitor] Protected containers missing: [felhom-samba] [DEBUG] [monitor] Final status: fail (issues=1, warnings=0, info=5) ``` **The diagnosis holds:** sharing enabled + no password → samba deliberately not deployed → the health monitor reports FAIL for an intentional state. **A sharper finding than the brief anticipated: the function's own doc comment asserts the invariant that is broken.** `healthcheck.go:260-261` says *"The bring-up applies the same conditions (`stacks.EnsureBaseStack` for cloudflared, `ensureSamba`'s `if !smb.Enabled { return }` for samba), so detection and deployment agree in both directions."* That was true when written. `reconcileSambaAt` now has a **second** early return on `!smb.UserSet` that the protected set never mirrored, so the two sides silently diverged and the comment documents a guarantee the code no longer provides. **I caused this state.** Sharing was enabled on demo-hp at ~06:20 UTC today by leg 4 of the R-75 task (explicitly authorised by that brief). The hub data shows the box was `ok` for 124 consecutive reports and flipped at the first cycle after that change. --- ## 6. D6 — blast radius ### Symptom A — 4 operator emails, no customer email `notification_log`, `event_type LIKE 'agent_channel%'`: | when (UTC) | box | channel | status | |---|---|---|---| | 2026-07-25 12:44:20 | demo-hp | operator | sent | | 2026-07-25 12:46:43 | demo-felhom | operator | sent | | 2026-07-26 06:16:08 | demo-felhom | operator | sent | | 2026-07-26 06:19:24 | demo-hp | operator | sent | Four, not one per cycle — the checker's `alerted` flag suppresses repeats within a down-spell, and the hub applies its own cooldown (7 hub *events* today produced 2 emails). `warning` severity is operator-only, so **no customer was emailed**. `drill-r50` also alarmed twice on 2026-07-25, which is consistent: it is on the same island and was the migration's first subject. ### Symptom B — zero emails so far, but a customer-facing one is plausibly pending - **Zero** notifications mentioning `samba` or `Protected`, ever. - demo-hp report health: `ok` ×124 through 2026-07-26 06:23:47, then **`fail` at 06:38:42** — one fail cycle at the time of the snapshot. demo-felhom stayed `ok` throughout (control: sharing there has a password and samba is running). - **But the path exists and has fired before.** `health_degraded` / `health_critical` are real notification types, and `health_critical` has gone to the **customer** channel three times historically (2026-02-24, 2026-06-15, 2026-07-18: *"Rendszer állapot kritikus (volt: ok)"*). A sustained `fail` on demo-hp is therefore likely to produce an operator **and possibly a customer** email on the next transition evaluation. It had not fired as of the last snapshot. **This is a live, pending consequence, not a closed one.** ### The English-string defect is WHOLE-SURFACE, not one-off Every `report.Issues` and `report.Warnings` string in the codebase is English: ``` healthcheck.go:56 "SSD disk usage critical: %.0f%%" healthcheck.go:82 "HDD disk usage critical: %.0f%%" healthcheck.go:150 "Docker: %v" healthcheck.go:169 "Protected container not running: %s" healthcheck.go:64/87/97/115/133 "SSD/HDD disk usage high", "Memory usage high", "CPU usage high", "Temperature high" ``` So this is not a slip in one line — the whole health-issue surface is operator-English and is rendered on a customer-facing Hungarian page. Nothing was changed. --- ## 7. Not collected, and what it would take | Item | Why | What would get it | |---|---|---| | Live TCP connect **from inside the controller container** to `169.254.253.1:8443` | the image ships no `nc`/`wget`/`ip`, and creating a probe container would be a mutation this task forbids | one `docker run --rm --network container:felhom-controller alpine nc -z 169.254.253.1 8443` — a throwaway container, no touch to the controller. Answered indirectly by the route simulation in §2.2, which is strong but not a delivered packet | | `agent.json` `listen_addr` value | `/etc/felhom-agent/agent.json` read as "unreadable" on both hosts even as root — the path is likely different in v0.96.0 | `systemctl cat felhom-agent` for the actual config path, or the agent's own startup log. **`ss` already proves the effective bind** (`169.254.253.1:8443`), so this is corroboration, not a gap in the conclusion | | Whether a `health_critical` email will fire for demo-hp | only one `fail` cycle had elapsed at snapshot time | re-check `notification_log` in an hour, or read the hub's health-transition checker thresholds | --- ## 8. Proposed fixes — DESCRIBED, NOT APPLIED ### 8.1 The endpoint (Symptom A) — the actual outage **Immediate, per box:** set `local_api.endpoint` in `controller.yaml` to `169.254.253.1:8443` on both guests and restart the controller. Two boxes, one line each. **Safe-obvious in shape, but validate rather than assume** — §2.2's link-local routability from a bridged container is route-simulated, not packet-proven. Success signal: `[INFO] [channel] agent channel recovered` plus an `agent_channel_recovered` hub event. **The real fix — `ensureLocalAPI` (`internal/bootstrap/bootstrap.go:254`) reconciles rather than only fills.** Its guard `if cfg.LocalAPI.Endpoint != "" { return cfg }` is what made the migration a no-op for the controller. If `bootstrap.json` carries a `local_api` block that differs from `controller.yaml`, `bootstrap.json` should win (it is the per-guest source of truth the agent side writes) and the file should be rewritten. **Needs a small spike, not a one-liner:** it changes which file is authoritative on every boot, so it must not clobber a legitimately setup-wizard-configured or hand-repaired guest, and the token/fingerprint must be reconciled atomically with the endpoint or a half-updated triple is worse than a stale one. Also worth deciding whether a mismatch should be a loud operator event in its own right — right now the two files can disagree indefinitely in silence. **Adjacent, trivial:** `config/config.go:75`'s comment still documents the pre-island shape (*"host bridge IP:port, e.g. 192.168.0.162:8443"*). Safe-obvious doc fix. **Process, not code:** `RUNBOOK-island-migration.md` should gain a post-migration verification step that asserts `controller.yaml` and `bootstrap.json` agree, and that the channel recovered. The migration reported success while leaving every controller on the fleet blind to its agent for 18 h. ### 8.2 `EffectiveProtected` (Symptom B) `internal/monitor/healthcheck.go:267` — gate samba on the same predicate the bring-up uses, i.e. `smb.Enabled && smb.UserSet`, mirroring the cloudflared branch five lines above. **Safe-obvious**, one condition, and it makes the function's own doc comment true again — that comment at L260-261 must be updated in the same change, since it currently asserts a parity that does not hold. A red-proof is easy: enabled-without-password must yield a protected set excluding samba, and enabled-with-password must include it. ### 8.3 The English strings on a Hungarian page Whole-surface (§6), so this is a **copy/i18n arc, not a one-line fix** — every `report.Issues` and `report.Warnings` producer, plus a decision about where translation belongs (at the producer, or at the render/notification boundary where operator-English and customer-Hungarian already diverge). The `health_critical` customer email suggests the boundary already exists and these strings simply bypass it. **Needs a spike** to pick the seam; the strings themselves are then mechanical. --- ## 9. Observations — noticed, not acted on 1. **`drill-r50` is also alarming** (2026-07-25 10:04, 10:11, 10:14, 12:31) and was the migration's first subject. If that VM still exists it is likely in the same stale-endpoint state. Out of scope here; worth a look when the fix lands. Its 10:04–10:14 alarms *precede* the demo boxes' 12:4x, so it may also carry earlier, unrelated spike churn. 2. **`expected_backup_missed` is firing nightly on all three customers** (2026-07-26 03:00, including a **customer**-channel email to demo-felhom: *"newest backup is 176h0m0s old"* = 7.3 days). That is a separate, louder problem than either symptom here, and it reached a customer channel. Almost certainly downstream of the same outage — the agent channel is how PBS backup state is learned — but I did not verify the causal link. 3. **The channel checker has no persistence by design**, so a restart re-alarms and the operator sees a "new" alert for an old condition. That is defensible (state re-derivation is the model) but it is why this looked like it started today. 4. **`rp_filter=2` (loose) on both guests** is what makes the container→island path viable at all; under strict `rp_filter=1` the asymmetry could bite. Worth knowing before anyone hardens it. 5. **demo-hp has been up 2 d 12 h and demo-felhom 3 d 13 h — neither has rebooted since the migration.** So the persistent `vmbr9` stanza in `/etc/network/interfaces` is present but has **never been exercised across a boot** on either box. The island's reboot-survival remains untested in the field. 6. **Sharing is still ON with no password on demo-hp**, which is what keeps Symptom B live. Left exactly as found, per the prime directive — but it means the box will keep reporting `fail` until either the fix lands or sharing is turned off.