F10 offsite leg resolved on demo-hp: day-0 update-vs-bridge race diagnosed, designed-path repair + tier round-trip proven; R-70/R-71 minted
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
This commit is contained in:
@@ -1,122 +1,63 @@
|
||||
# REPORT — Hub notification train v0.71.0: F11 + F12 + F14-light (2026-07-22)
|
||||
# REPORT — F10 diagnostic: demo-hp offsite "enabled at the hub, absent on the box" (2026-07-23)
|
||||
|
||||
**Task:** paired recovery notifications, prefs seeding at claim, empty-email no-clobber guard,
|
||||
priority headers, operator test leg — all in `hub/`; felhom-controller read-only reference.
|
||||
Origin: `documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md` (F11/F12/F14-light).
|
||||
**Task:** F10 diagnostic spec (project Claude, 2026-07-23) — diagnose first, repair only via the
|
||||
designed path, prove the tier. **No code changed in any repo.** Full evidence record:
|
||||
`documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`.
|
||||
|
||||
## Baselines (live-verified before starting)
|
||||
## Phase-0 verdict
|
||||
|
||||
| Repo | main @ start | → shipped |
|
||||
|---|---|---|
|
||||
| felhom.eu | `5b35023574173e17f9ee968ba09b6ab34616a322`, hub 0.70.1 in `manifests/hub.yaml` | hub **0.71.0** — code `c766c8a`, manifest `f1da171` |
|
||||
| felhom-controller | `0eba37d5cdc6` v0.160.0 | unchanged (read-only) |
|
||||
Neither of the spec's two candidate shapes. The evidence (hub DB + box state + live logs, every
|
||||
claim cited in the DIAG) proves a third: **the day-0 managed floor-update (0.153.0→0.156.0,
|
||||
07-21 16:28:17Z) killed the offsite apply-bridge ~35 s after it consumed the one-time password**
|
||||
(16:27:42Z), before key-install/persist. Consume-then-persist + retry-only-on-restart
|
||||
(`offsiteapply.go:106–187`) ⇒ the credential was burned, no key was ever installed (so the
|
||||
key-auth-first recovery path could never engage), and every later start logged the consume-404 WARN
|
||||
and gave up. 153 reports over 2 days never carried an offbox object; the hub's "Provisioned…"
|
||||
line is static copy that reads neither `consumed_at` nor the reports.
|
||||
|
||||
## Phase 0a — controller prefs-push fact (read-only, cited)
|
||||
- Shape B ruled out from source: managed offsite is fully automatic; the box's „Távoli mentési cél
|
||||
beállítása" button is the BYO NAS/SFTP form only (`offbox_handlers.go:44–126`).
|
||||
- Strictly this was the spec's "consumed but persist failed → STOP" class; since the mechanism
|
||||
provably held its fail-safe and the source itself designates the recovery ("the password is
|
||||
spent; reset it on the hub to retry" = the offsite Re-issue), the operator ruled in-session:
|
||||
proceed on the Re-issue path.
|
||||
|
||||
**Does the startup push fire for a never-configured customer, and with what email?** It does NOT
|
||||
fire: `felhom-controller/controller/cmd/controller/main.go:821–830` guards the startup sync with
|
||||
`if prefs.Email != ""` — an unconfigured box pushes nothing. The settings-save leg
|
||||
(`internal/web/handlers.go:1532–1545`, the 2026-07-15 wipe-guard) refuses empty-email-with-events
|
||||
outright; only the intentional all-off case (empty email + zero events) pushes an empty email, and
|
||||
that push is legitimate. **Verdict: the clobber was LATENT on controller 0.160.0, not live** — the
|
||||
hub-side guard shipped as the belt for older/rogue controllers. The debug re-sync
|
||||
(`internal/web/handler_debug.go:432`) pushes stored prefs as-is. Test-leg endpoint the UI invokes:
|
||||
`POST /settings/notifications/test` (`internal/web/server.go:395` →
|
||||
`settingsNotificationsTestHandler`, `handlers.go:1616`) → `notifier.SendTest()` → hub
|
||||
`POST /api/v1/event` with eventType `test`.
|
||||
## Repair (designed path only)
|
||||
|
||||
## Phase 0b — Resend custom-headers probe
|
||||
- Operator clicked **Re-issue offsite credentials** ONCE (R-31 click-once discipline; pre-verified
|
||||
side-effect-free: no escrow blob existed, the box never held the old password, sub3 was empty).
|
||||
Click 09:53:37Z → box consumed 09:53:41 → `offsite configured … (pending key escrow)` 09:53:45.
|
||||
**8 seconds click-to-converged.**
|
||||
- Escrow ceremony run by the operator through the real `/backup/escrow` wizard (one-shot R on the
|
||||
operator's screen only): blob stored 10:01:17 (zero_knowledge, pw-hash recorded), hub-verified
|
||||
auto-confirm 10:01:24 → `escrowed; offsite runs enabled`.
|
||||
|
||||
One real mail via the Resend API (`from monitoring@felhom.eu`, `to admin@felhom.eu`, subject
|
||||
"[Felhom] header probe") with `"headers": {"X-Priority":"1","Importance":"high"}` — **HTTP 200**,
|
||||
mail id `34d3f7f3-849f-432a-ac8e-eb479a1c6695`. Key sourced from the k8s Secret into the shell env
|
||||
only; never written to any file or log. Mechanism GO.
|
||||
## Tier proof (F10 closure bar)
|
||||
|
||||
## Files modified (commit `c766c8a`, docs+code; manifest bump `f1da171`)
|
||||
paperless-ngx toggled into offsite scope via the real endpoint (per-app default is OFF). Then, all
|
||||
via real endpoints from inside guest 9201 (endpoint-level method; no browser on DooPlex):
|
||||
probe (md5 `9120e65d6a9f071072d827fc404dc840`) in the mandatory `appdata/paperless/media` →
|
||||
**first offsite run**: repo initialized fresh on sub3, 79.8 MB / 49 files, 1m19s, ok →
|
||||
probe deleted → **`mode=full` restore** (size gate 79.8 MB → confirm): snapshot **`2bf7f2e1`** to
|
||||
staging, staging md5-identical → **place**: `1 file(s) merged (missing-only)`, live md5-identical.
|
||||
Cleanup: probe removed, second run (2m17s ok) leaves the latest snapshot probe-free (retention
|
||||
pruned the probe-bearing one); zero residue on box/repo; break-glass + DB copies shredded.
|
||||
Hub now reports demo-hp `offsite: enabled/escrowed/quota 50`; nightly run scheduled (04:15 UTC).
|
||||
|
||||
- `hub/internal/store/store.go` — `LastCustomerSentAt` (pairing-evidence query, uses the
|
||||
`(customer_id, created_at DESC)` index), `SeedNotificationPrefs` (INSERT OR IGNORE, empty-email
|
||||
no-op); `strings` import.
|
||||
- `hub/internal/notify/dispatcher.go` — `priorityHeaders`; `recoveredPairedDownTypes`; recovery
|
||||
branch in `ProcessEvent` (before the severity gate; `severityNotifies` byte-untouched);
|
||||
`processRecovery` (operator = existing `processOperator`; customer = blocked → prefs-nil/email →
|
||||
**pairing** → prefs cooldown; INFO logs on every gated skip); `sendEmailFn`/`sendEmail` grew
|
||||
`headers map[string]string` (payload carries `"headers"` only when non-empty; all callers
|
||||
updated); `sendTestEmail` operator copy with forced priority headers + **latent nil-prefs panic
|
||||
fixed** (`GetNotificationPrefs` returns `(nil, nil)` — the old code dereferenced `prefs.Email`).
|
||||
- `hub/internal/notify/templates.go` — ✅ icon for `*_recovered` (eventType-aware, severity
|
||||
fallback); `customerMessages["host_recovered"]`.
|
||||
- `hub/internal/claim/engine.go` — `defaultSeedEvents` (critical-only) + seed call in
|
||||
`MarkClaimed` (WARN-and-continue on failure; claim never hinges on notification plumbing).
|
||||
- `hub/internal/api/handler.go` — `handleSavePreferences` empty-email no-clobber guard (INFO log).
|
||||
- Tests: `notifications_seed_test.go` (store), `dispatcher_recovery_test.go` (notify),
|
||||
`seed_test.go` (claim), `preferences_guard_test.go` (api); `dispatcher_test.go` seam signatures.
|
||||
- Docs: `hub/CHANGELOG.md`, `CONTEXT.md` (4 rulings), `REUSE.md` (§Alerting rewritten — serves as
|
||||
the notification-plane map; no standalone notification feature doc exists, noted below),
|
||||
`00-capability-map.md` (§F rows), `ROADMAP.md` (R-68 SHIPPED, R-69 minted for F14-full).
|
||||
## Product findings + docs
|
||||
|
||||
## Tests — 449 → 466 (17 new), full suite green (`go build && go vet && go test ./...` exit 0)
|
||||
- **R-70 (P2-HIGH)** minted: the offsite last mile is invisible on both surfaces (hub can't tell
|
||||
staged/consumed/applied; box shows the generic empty state). Coupled to R-31's status-card idiom
|
||||
and the R-39 consumed_at honesty-gauge precedent.
|
||||
- **R-71 (P1)** minted: the race itself — recurs structurally on every fresh onboarding whose ISO
|
||||
floor lags the managed floor. Spec-first directions listed in the row (ordering / two-phase
|
||||
consume / hub-side auto-restage with the R-39(a) mint-race guard).
|
||||
- Audit F10 row annotated: **offsite leg resolved**; PBS-DR half explicitly stays open (F13 +
|
||||
DR ceremony R-moment). CONTEXT.md updated.
|
||||
|
||||
| Group | Test | Result |
|
||||
|---|---|---|
|
||||
| A | `TestRecovery_PairedCustomerMail` (down→recovery: both channels, Hungarian subject, ✅ subject, both log rows) | PASS |
|
||||
| B | `TestRecovery_UnpairedStaysCustomerSilent` (operator only; enabled_events ignored) | PASS |
|
||||
| C | `TestRecovery_FlapDamping` (suppressed down ⇒ no 2nd recovery mail; op 1 h cooldown) | PASS |
|
||||
| D | `TestMarkClaimed_SeedsNotificationPrefs` / `_NeverOverwritesExistingPrefs` / `_EmptyEmail_NoRow_ClaimSucceeds` + store `TestSeedNotificationPrefs_*` (2) + `TestLastCustomerSentAt_PairingQuery` | PASS |
|
||||
| E | `TestSavePreferences_EmptyEmailCannotClobber` / `_NonEmptyEmailStillUpdates` / `_EmptyEmailNoStoredRow` | PASS |
|
||||
| F | `TestTestEvent_BothChannels` / `TestTestEvent_NoPrefsRow_NoPanic` | PASS |
|
||||
| G | `TestPriorityHeaders` (table) / `TestPriorityHeaders_EndToEnd` (headers on error both channels, absent on warning) | PASS |
|
||||
| guard | `TestRecovery_SeverityStaysInfo` (severityNotifies unwidened; non-recovery info silent) | PASS |
|
||||
## Observed, not acted on
|
||||
|
||||
**Red-proofs (each: mutation applied → FAIL confirmed → reverted → suite green):**
|
||||
1. Pairing check forced open (`if false && …`) → `TestRecovery_UnpairedStaysCustomerSilent` FAILED
|
||||
("unpaired recovery must NOT mail the customer, got 1"). ✔
|
||||
2. Seed via `SaveNotificationPrefs` upsert → `TestMarkClaimed_NeverOverwritesExistingPrefs` FAILED
|
||||
("claim seed MODIFIED an existing row"). ✔
|
||||
3. Clobber guard removed → `TestSavePreferences_EmptyEmailCannotClobber` FAILED ("CLOBBERED the
|
||||
stored address: email=\"\""). ✔
|
||||
4. Headers unconditional → `TestPriorityHeaders_EndToEnd` FAILED ("warning mail … must carry NO
|
||||
headers") (+ table test). ✔
|
||||
|
||||
## Deploy verification
|
||||
|
||||
Image built + pushed (`build.sh 0.71.0 --push`), manifest bumped (`f1da171`), ArgoCD hard-refresh +
|
||||
deliberate sync → **Synced / Healthy**, pod `hub-6ffbf574d8-p2w5p` Running, deployed image
|
||||
`gitea.dooplex.hu/admin/felhom-hub:0.71.0`, startup log clean (checkers seeded 2 ok, no dispatch
|
||||
errors).
|
||||
|
||||
## Live test-mail leg (the exact endpoint the UI invokes)
|
||||
|
||||
`POST /settings/notifications/test` on the demo-felhom guest controller (in-guest, session +
|
||||
CSRF): controller returned „Teszt email elküldve." Hub log:
|
||||
`Event from demo-felhom: test (info)` → `Test email sent to doodoo21@freemail.hu` →
|
||||
`Operator test email sent for demo-felhom`. `notification_log` rows (UTC):
|
||||
|
||||
```
|
||||
demo-felhom test customer sent Teszt értesítés 2026-07-22 18:59:34
|
||||
demo-felhom test operator sent operator test copy 2026-07-22 18:59:34
|
||||
```
|
||||
|
||||
Resend accepted both (no `failed` rows). **HANDED TO VIKTOR:** confirm the operator mail landed in
|
||||
Primary, the phone buzzed (the new Gmail filter's proof), and the client shows it high-priority.
|
||||
|
||||
## NOT live-validated in this task (verbatim, per spec)
|
||||
|
||||
- The natural `*_recovered` mail on a real staleness cycle — rides the next real event or the
|
||||
deliberate reboot drill; NOT fabricated by blocking reports (F9-bypass-shaped).
|
||||
- Seed-at-claim on a real claim — next onboarding; Peti's Friday reinstall is the natural candidate.
|
||||
|
||||
## Observations (not acted on)
|
||||
|
||||
- `sendTestEmail` nil-prefs panic was latent-reachable live: a „Teszt értesítés" click on a
|
||||
customer with no `customer_notifications` row (demo-hp today) would have panicked the dispatcher
|
||||
goroutine. Fixed in-scope (the function was already an edit surface); noted here because it
|
||||
predates this train.
|
||||
- No standalone notification-plane feature doc exists under `documentation/` — N.4 was satisfied
|
||||
in `REUSE.md` §Alerting (the de-facto plane map) + the capability map row; if Viktor wants a
|
||||
dedicated `architecture/notifications.md`, that is a docs-only follow-up.
|
||||
- `handleNotify` (legacy), mailrelay, `SendClaimEmail`/`SendSelfBindEmail` byte-untouched (claim/
|
||||
selfbind sends now pass explicit `nil` headers — no behavior change).
|
||||
- Website shows no hub version → N.6 n/a.
|
||||
- The `notification_log` is now pairing evidence (recovery gate reads it) — future retention/prune
|
||||
work must keep customer-channel stale/down/recovered rows long enough to pair (noted in REUSE).
|
||||
- demo-felhom's 07-21 staged offsite secret is still unconsumed (residue of the mistaken R-39-day
|
||||
offsite Re-issue; box recovered via key-auth-first, which never consumes). Harmless; supports R-70.
|
||||
- The 3 dead unclaimed-appliance records from the ISO train remain for operator discard.
|
||||
- Hub pod log only reaches back to 07-22 20:58Z (restart); the 07-21 correlation came from the DB.
|
||||
|
||||
Reference in New Issue
Block a user