diff --git a/REPORT.md b/REPORT.md index b4b163c..5877a2b 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,53 +1,122 @@ -# REPORT — Power-outage recovery audit, vacation site (2026-07-22) +# REPORT — Hub notification train v0.71.0: F11 + F12 + F14-light (2026-07-22) -**Task:** RUNBOOK-driven, strictly read-only forensic + recovery audit after the 14:41–42 CEST -site-wide power cut at the vacation site (felhom-pve N100 + demo-hp t740, ~4.5 h down, manual -power-on ~19:13). No code, no fixes, no restarts — diagnosis only. Deliverable: -`documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md`; raw evidence on DooPlex -`~/outage-20260722/evidence/`. +**Task:** paired recovery notifications, prefs seeding at claim, empty-email no-clobber guard, +priority headers, operator test leg — all in `hub/`; felhom-controller read-only reference. +Origin: `documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md` (F11/F12/F14-light). -## Verdict +## Baselines (live-verified before starting) -- **Cause:** simultaneous external power cut on both boxes — both prev-boot journals end abruptly - mid-agent-routine, `last -x` says `crash`, zero non-agent sudo/sshd entries in the window. The - earlier CC session is exonerated. -- **Recovery:** every layer self-healed unaided in **~3 m 15 s** after power-on — fs journal - replay, agent 0.93.0 (+26 s), wg + PBS-over-wg (+32 s), tailscale, guest 9201 (onboot=1, - +2 m 38 s), all stacks healthy on the controllers' own surfaces, cloudflared 4/4 (+2 m 48 s), - hub `*_recovered` (+3 m 17 s). Edge-pinned public probes 302. Felhom-Share autofs woke + CIFS - mounted on first access (first unplanned re-run of the R-67 dead-NAS proof — passed). -- **Dead-man's-switch live fire:** host/node stale + down fired **exactly on the 30 m/60 m design - schedule** (measured from last received report, not from the outage instant), all 9 notification - dispatches `sent` (8 operator + 1 customer). Recovery events are severity `info` → intentionally - no email (flagged as product question F11). -- **F1 regression:** did NOT recur — static .162 held, demo-hp re-acquired .87, local API bound - cleanly on both. -- **Scars:** only OS-self-healed ones (journald rotate, ESP dirty bit, orphan inodes). Metrics DBs - `quick_check` ok, gap 14:40→19:16. Offbox restic: 14 snapshots, latest pre-outage 04:16, zero - stale locks. No backup window fell inside the outage; catch-up is tonight 03:30/04:15. +| Repo | main @ start | → shipped | +|---|---|---| +| felhom.eu | `5b35023574173e17f9ee968ba09b6ab34616a322`, hub 0.70.1 in `manifests/hub.yaml` | hub **0.71.0** — code `c766c8a`, manifest `f1da171` | +| felhom-controller | `0eba37d5cdc6` v0.160.0 | unchanged (read-only) | -## Findings (F8+ continuing the vacation arc) +## Phase 0a — controller prefs-push fact (read-only, cited) -- **F8 (HIGH, mitigated-on-site):** no auto-power-on — power was back in minutes, boxes sat off - 4.5 h. BIOS AC-power-on now set (attested); roadmap-candidate: provisioning-checklist item. -- **F9 (MEDIUM, needs-ruling):** H1 OOB belt (`felhom-sshd` + `felhom-oob-nft`) installed on - NEITHER fleet box — only the mgmt-watchdog timer; access is stock sshd :22 + break-glass. -- **F10 (MEDIUM, roadmap):** demo-hp has tier-2 only — no offbox target, 0 PBS DR snapshots. -- **F11 (LOW, needs-ruling):** recovery is silent (`info` severity never emails). -- **F12 (LOW, roadmap):** demo-hp has no customer notification prefs row — node_down customer - email impossible there. -- **F13 (LOW, needs-ruling):** PBS DR tier shows no snapshot cadence (jobs.cfg empty; single - 07-18 snapshot on N100, none on demo-hp) — intended? +**Does the startup push fire for a never-configured customer, and with what email?** It does NOT +fire: `felhom-controller/controller/cmd/controller/main.go:821–830` guards the startup sync with +`if prefs.Email != ""` — an unconfigured box pushes nothing. The settings-save leg +(`internal/web/handlers.go:1532–1545`, the 2026-07-15 wipe-guard) refuses empty-email-with-events +outright; only the intentional all-off case (empty email + zero events) pushes an empty email, and +that push is legitimate. **Verdict: the clobber was LATENT on controller 0.160.0, not live** — the +hub-side guard shipped as the belt for older/rogue controllers. The debug re-sync +(`internal/web/handler_debug.go:432`) pushes stored prefs as-is. Test-leg endpoint the UI invokes: +`POST /settings/notifications/test` (`internal/web/server.go:395` → +`settingsNotificationsTestHandler`, `handlers.go:1616`) → `notifier.SendTest()` → hub +`POST /api/v1/event` with eventType `test`. -## Observations +## Phase 0b — Resend custom-headers probe -demo-vm tunnel Down is permanent-correct (nested drill VMs 9310–9312 destroyed pre-outage this -morning; record discard pending). peti-felhom silent since 07-15 08:39 UTC (his site, untouched). -Controllers had been on 0.160.0 for only ~28 min when the cut hit — first dirty shutdown on the -new version came back clean. +One real mail via the Resend API (`from monitoring@felhom.eu`, `to admin@felhom.eu`, subject +"[Felhom] header probe") with `"headers": {"X-Priority":"1","Importance":"high"}` — **HTTP 200**, +mail id `34d3f7f3-849f-432a-ac8e-eb479a1c6695`. Key sourced from the k8s Secret into the shell env +only; never written to any file or log. Mechanism GO. -**Open question for Viktor:** did the ~14:57–15:30 CEST alert emails actually land in the inbox? -Hub-side they are all `sent`; delivery is inbox/Resend-dashboard-side. +## Files modified (commit `c766c8a`, docs+code; manifest bump `f1da171`) -**Not validated (needs-mutation or operator-side):** BIOS setting itself, inbox delivery, -Cloudflare dashboard state, UI click-throughs. +- `hub/internal/store/store.go` — `LastCustomerSentAt` (pairing-evidence query, uses the + `(customer_id, created_at DESC)` index), `SeedNotificationPrefs` (INSERT OR IGNORE, empty-email + no-op); `strings` import. +- `hub/internal/notify/dispatcher.go` — `priorityHeaders`; `recoveredPairedDownTypes`; recovery + branch in `ProcessEvent` (before the severity gate; `severityNotifies` byte-untouched); + `processRecovery` (operator = existing `processOperator`; customer = blocked → prefs-nil/email → + **pairing** → prefs cooldown; INFO logs on every gated skip); `sendEmailFn`/`sendEmail` grew + `headers map[string]string` (payload carries `"headers"` only when non-empty; all callers + updated); `sendTestEmail` operator copy with forced priority headers + **latent nil-prefs panic + fixed** (`GetNotificationPrefs` returns `(nil, nil)` — the old code dereferenced `prefs.Email`). +- `hub/internal/notify/templates.go` — ✅ icon for `*_recovered` (eventType-aware, severity + fallback); `customerMessages["host_recovered"]`. +- `hub/internal/claim/engine.go` — `defaultSeedEvents` (critical-only) + seed call in + `MarkClaimed` (WARN-and-continue on failure; claim never hinges on notification plumbing). +- `hub/internal/api/handler.go` — `handleSavePreferences` empty-email no-clobber guard (INFO log). +- Tests: `notifications_seed_test.go` (store), `dispatcher_recovery_test.go` (notify), + `seed_test.go` (claim), `preferences_guard_test.go` (api); `dispatcher_test.go` seam signatures. +- Docs: `hub/CHANGELOG.md`, `CONTEXT.md` (4 rulings), `REUSE.md` (§Alerting rewritten — serves as + the notification-plane map; no standalone notification feature doc exists, noted below), + `00-capability-map.md` (§F rows), `ROADMAP.md` (R-68 SHIPPED, R-69 minted for F14-full). + +## Tests — 449 → 466 (17 new), full suite green (`go build && go vet && go test ./...` exit 0) + +| Group | Test | Result | +|---|---|---| +| A | `TestRecovery_PairedCustomerMail` (down→recovery: both channels, Hungarian subject, ✅ subject, both log rows) | PASS | +| B | `TestRecovery_UnpairedStaysCustomerSilent` (operator only; enabled_events ignored) | PASS | +| C | `TestRecovery_FlapDamping` (suppressed down ⇒ no 2nd recovery mail; op 1 h cooldown) | PASS | +| D | `TestMarkClaimed_SeedsNotificationPrefs` / `_NeverOverwritesExistingPrefs` / `_EmptyEmail_NoRow_ClaimSucceeds` + store `TestSeedNotificationPrefs_*` (2) + `TestLastCustomerSentAt_PairingQuery` | PASS | +| E | `TestSavePreferences_EmptyEmailCannotClobber` / `_NonEmptyEmailStillUpdates` / `_EmptyEmailNoStoredRow` | PASS | +| F | `TestTestEvent_BothChannels` / `TestTestEvent_NoPrefsRow_NoPanic` | PASS | +| G | `TestPriorityHeaders` (table) / `TestPriorityHeaders_EndToEnd` (headers on error both channels, absent on warning) | PASS | +| guard | `TestRecovery_SeverityStaysInfo` (severityNotifies unwidened; non-recovery info silent) | PASS | + +**Red-proofs (each: mutation applied → FAIL confirmed → reverted → suite green):** +1. Pairing check forced open (`if false && …`) → `TestRecovery_UnpairedStaysCustomerSilent` FAILED + ("unpaired recovery must NOT mail the customer, got 1"). ✔ +2. Seed via `SaveNotificationPrefs` upsert → `TestMarkClaimed_NeverOverwritesExistingPrefs` FAILED + ("claim seed MODIFIED an existing row"). ✔ +3. Clobber guard removed → `TestSavePreferences_EmptyEmailCannotClobber` FAILED ("CLOBBERED the + stored address: email=\"\""). ✔ +4. Headers unconditional → `TestPriorityHeaders_EndToEnd` FAILED ("warning mail … must carry NO + headers") (+ table test). ✔ + +## Deploy verification + +Image built + pushed (`build.sh 0.71.0 --push`), manifest bumped (`f1da171`), ArgoCD hard-refresh + +deliberate sync → **Synced / Healthy**, pod `hub-6ffbf574d8-p2w5p` Running, deployed image +`gitea.dooplex.hu/admin/felhom-hub:0.71.0`, startup log clean (checkers seeded 2 ok, no dispatch +errors). + +## Live test-mail leg (the exact endpoint the UI invokes) + +`POST /settings/notifications/test` on the demo-felhom guest controller (in-guest, session + +CSRF): controller returned „Teszt email elküldve." Hub log: +`Event from demo-felhom: test (info)` → `Test email sent to doodoo21@freemail.hu` → +`Operator test email sent for demo-felhom`. `notification_log` rows (UTC): + +``` +demo-felhom test customer sent Teszt értesítés 2026-07-22 18:59:34 +demo-felhom test operator sent operator test copy 2026-07-22 18:59:34 +``` + +Resend accepted both (no `failed` rows). **HANDED TO VIKTOR:** confirm the operator mail landed in +Primary, the phone buzzed (the new Gmail filter's proof), and the client shows it high-priority. + +## NOT live-validated in this task (verbatim, per spec) + +- The natural `*_recovered` mail on a real staleness cycle — rides the next real event or the + deliberate reboot drill; NOT fabricated by blocking reports (F9-bypass-shaped). +- Seed-at-claim on a real claim — next onboarding; Peti's Friday reinstall is the natural candidate. + +## Observations (not acted on) + +- `sendTestEmail` nil-prefs panic was latent-reachable live: a „Teszt értesítés" click on a + customer with no `customer_notifications` row (demo-hp today) would have panicked the dispatcher + goroutine. Fixed in-scope (the function was already an edit surface); noted here because it + predates this train. +- No standalone notification-plane feature doc exists under `documentation/` — N.4 was satisfied + in `REUSE.md` §Alerting (the de-facto plane map) + the capability map row; if Viktor wants a + dedicated `architecture/notifications.md`, that is a docs-only follow-up. +- `handleNotify` (legacy), mailrelay, `SendClaimEmail`/`SendSelfBindEmail` byte-untouched (claim/ + selfbind sends now pass explicit `nil` headers — no behavior change). +- Website shows no hub version → N.6 n/a. +- The `notification_log` is now pairing evidence (recovery gate reads it) — future retention/prune + work must keep customer-channel stale/down/recovered rows long enough to pair (noted in REUSE).