diff --git a/CONTEXT.md b/CONTEXT.md index 85b260e..ce68660 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,20 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-23 — F10 offsite leg RESOLVED on demo-hp (diagnostic + designed-path repair + tier + proof; no code changed).** The hub said "Provisioned" while the box said „Még nincs beállítva" — + root cause: the day-0 managed update (0.153.0→0.156.0, 07-21 16:28) killed the offsite + apply-bridge ~35 s after it consumed the one-time password, before key-install/persist; the box + then hit the consume-404 dead-end at every start, silently, for 2 days. Repair = operator + **Re-issue** click (converged in 8 s), escrow ceremony via the real wizard (auto-confirm 3 s), + then the full proof: paperless-ngx toggled into scope, first offsite run (repo initialized on + sub3, 79.8 MB, snapshots=1), probe deleted → `mode=full` restore (snapshot `2bf7f2e1`) → place + merged 1 file missing-only → **md5-identical**. Minted **R-70** (last-mile invisible on both + surfaces; hub never reads `one_time_secrets.consumed_at`) and **R-71 [P1]** (the race recurs on + every fresh onboarding whose ISO floor lags the managed floor). Record: + `documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. F10's PBS-DR half stays open + (F13 + ceremony R-moment). + - **2026-07-22 — hub v0.71.0: the notification train (audit F11+F12+F14-light).** Four rulings now standing: **(1) recovery pairing** — `*_recovered` notifies the operator always and the customer **iff the customer was mailed the paired stale/down** (evidence = `notification_log` customer- diff --git a/REPORT.md b/REPORT.md index 5877a2b..073ce4a 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,122 +1,63 @@ -# REPORT — Hub notification train v0.71.0: F11 + F12 + F14-light (2026-07-22) +# REPORT — F10 diagnostic: demo-hp offsite "enabled at the hub, absent on the box" (2026-07-23) -**Task:** paired recovery notifications, prefs seeding at claim, empty-email no-clobber guard, -priority headers, operator test leg — all in `hub/`; felhom-controller read-only reference. -Origin: `documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md` (F11/F12/F14-light). +**Task:** F10 diagnostic spec (project Claude, 2026-07-23) — diagnose first, repair only via the +designed path, prove the tier. **No code changed in any repo.** Full evidence record: +`documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. -## Baselines (live-verified before starting) +## Phase-0 verdict -| Repo | main @ start | → shipped | -|---|---|---| -| felhom.eu | `5b35023574173e17f9ee968ba09b6ab34616a322`, hub 0.70.1 in `manifests/hub.yaml` | hub **0.71.0** — code `c766c8a`, manifest `f1da171` | -| felhom-controller | `0eba37d5cdc6` v0.160.0 | unchanged (read-only) | +Neither of the spec's two candidate shapes. The evidence (hub DB + box state + live logs, every +claim cited in the DIAG) proves a third: **the day-0 managed floor-update (0.153.0→0.156.0, +07-21 16:28:17Z) killed the offsite apply-bridge ~35 s after it consumed the one-time password** +(16:27:42Z), before key-install/persist. Consume-then-persist + retry-only-on-restart +(`offsiteapply.go:106–187`) ⇒ the credential was burned, no key was ever installed (so the +key-auth-first recovery path could never engage), and every later start logged the consume-404 WARN +and gave up. 153 reports over 2 days never carried an offbox object; the hub's "Provisioned…" +line is static copy that reads neither `consumed_at` nor the reports. -## Phase 0a — controller prefs-push fact (read-only, cited) +- Shape B ruled out from source: managed offsite is fully automatic; the box's „Távoli mentési cél + beállítása" button is the BYO NAS/SFTP form only (`offbox_handlers.go:44–126`). +- Strictly this was the spec's "consumed but persist failed → STOP" class; since the mechanism + provably held its fail-safe and the source itself designates the recovery ("the password is + spent; reset it on the hub to retry" = the offsite Re-issue), the operator ruled in-session: + proceed on the Re-issue path. -**Does the startup push fire for a never-configured customer, and with what email?** It does NOT -fire: `felhom-controller/controller/cmd/controller/main.go:821–830` guards the startup sync with -`if prefs.Email != ""` — an unconfigured box pushes nothing. The settings-save leg -(`internal/web/handlers.go:1532–1545`, the 2026-07-15 wipe-guard) refuses empty-email-with-events -outright; only the intentional all-off case (empty email + zero events) pushes an empty email, and -that push is legitimate. **Verdict: the clobber was LATENT on controller 0.160.0, not live** — the -hub-side guard shipped as the belt for older/rogue controllers. The debug re-sync -(`internal/web/handler_debug.go:432`) pushes stored prefs as-is. Test-leg endpoint the UI invokes: -`POST /settings/notifications/test` (`internal/web/server.go:395` → -`settingsNotificationsTestHandler`, `handlers.go:1616`) → `notifier.SendTest()` → hub -`POST /api/v1/event` with eventType `test`. +## Repair (designed path only) -## Phase 0b — Resend custom-headers probe +- Operator clicked **Re-issue offsite credentials** ONCE (R-31 click-once discipline; pre-verified + side-effect-free: no escrow blob existed, the box never held the old password, sub3 was empty). + Click 09:53:37Z → box consumed 09:53:41 → `offsite configured … (pending key escrow)` 09:53:45. + **8 seconds click-to-converged.** +- Escrow ceremony run by the operator through the real `/backup/escrow` wizard (one-shot R on the + operator's screen only): blob stored 10:01:17 (zero_knowledge, pw-hash recorded), hub-verified + auto-confirm 10:01:24 → `escrowed; offsite runs enabled`. -One real mail via the Resend API (`from monitoring@felhom.eu`, `to admin@felhom.eu`, subject -"[Felhom] header probe") with `"headers": {"X-Priority":"1","Importance":"high"}` — **HTTP 200**, -mail id `34d3f7f3-849f-432a-ac8e-eb479a1c6695`. Key sourced from the k8s Secret into the shell env -only; never written to any file or log. Mechanism GO. +## Tier proof (F10 closure bar) -## Files modified (commit `c766c8a`, docs+code; manifest bump `f1da171`) +paperless-ngx toggled into offsite scope via the real endpoint (per-app default is OFF). Then, all +via real endpoints from inside guest 9201 (endpoint-level method; no browser on DooPlex): +probe (md5 `9120e65d6a9f071072d827fc404dc840`) in the mandatory `appdata/paperless/media` → +**first offsite run**: repo initialized fresh on sub3, 79.8 MB / 49 files, 1m19s, ok → +probe deleted → **`mode=full` restore** (size gate 79.8 MB → confirm): snapshot **`2bf7f2e1`** to +staging, staging md5-identical → **place**: `1 file(s) merged (missing-only)`, live md5-identical. +Cleanup: probe removed, second run (2m17s ok) leaves the latest snapshot probe-free (retention +pruned the probe-bearing one); zero residue on box/repo; break-glass + DB copies shredded. +Hub now reports demo-hp `offsite: enabled/escrowed/quota 50`; nightly run scheduled (04:15 UTC). -- `hub/internal/store/store.go` — `LastCustomerSentAt` (pairing-evidence query, uses the - `(customer_id, created_at DESC)` index), `SeedNotificationPrefs` (INSERT OR IGNORE, empty-email - no-op); `strings` import. -- `hub/internal/notify/dispatcher.go` — `priorityHeaders`; `recoveredPairedDownTypes`; recovery - branch in `ProcessEvent` (before the severity gate; `severityNotifies` byte-untouched); - `processRecovery` (operator = existing `processOperator`; customer = blocked → prefs-nil/email → - **pairing** → prefs cooldown; INFO logs on every gated skip); `sendEmailFn`/`sendEmail` grew - `headers map[string]string` (payload carries `"headers"` only when non-empty; all callers - updated); `sendTestEmail` operator copy with forced priority headers + **latent nil-prefs panic - fixed** (`GetNotificationPrefs` returns `(nil, nil)` — the old code dereferenced `prefs.Email`). -- `hub/internal/notify/templates.go` — ✅ icon for `*_recovered` (eventType-aware, severity - fallback); `customerMessages["host_recovered"]`. -- `hub/internal/claim/engine.go` — `defaultSeedEvents` (critical-only) + seed call in - `MarkClaimed` (WARN-and-continue on failure; claim never hinges on notification plumbing). -- `hub/internal/api/handler.go` — `handleSavePreferences` empty-email no-clobber guard (INFO log). -- Tests: `notifications_seed_test.go` (store), `dispatcher_recovery_test.go` (notify), - `seed_test.go` (claim), `preferences_guard_test.go` (api); `dispatcher_test.go` seam signatures. -- Docs: `hub/CHANGELOG.md`, `CONTEXT.md` (4 rulings), `REUSE.md` (§Alerting rewritten — serves as - the notification-plane map; no standalone notification feature doc exists, noted below), - `00-capability-map.md` (§F rows), `ROADMAP.md` (R-68 SHIPPED, R-69 minted for F14-full). +## Product findings + docs -## Tests — 449 → 466 (17 new), full suite green (`go build && go vet && go test ./...` exit 0) +- **R-70 (P2-HIGH)** minted: the offsite last mile is invisible on both surfaces (hub can't tell + staged/consumed/applied; box shows the generic empty state). Coupled to R-31's status-card idiom + and the R-39 consumed_at honesty-gauge precedent. +- **R-71 (P1)** minted: the race itself — recurs structurally on every fresh onboarding whose ISO + floor lags the managed floor. Spec-first directions listed in the row (ordering / two-phase + consume / hub-side auto-restage with the R-39(a) mint-race guard). +- Audit F10 row annotated: **offsite leg resolved**; PBS-DR half explicitly stays open (F13 + + DR ceremony R-moment). CONTEXT.md updated. -| Group | Test | Result | -|---|---|---| -| A | `TestRecovery_PairedCustomerMail` (down→recovery: both channels, Hungarian subject, ✅ subject, both log rows) | PASS | -| B | `TestRecovery_UnpairedStaysCustomerSilent` (operator only; enabled_events ignored) | PASS | -| C | `TestRecovery_FlapDamping` (suppressed down ⇒ no 2nd recovery mail; op 1 h cooldown) | PASS | -| D | `TestMarkClaimed_SeedsNotificationPrefs` / `_NeverOverwritesExistingPrefs` / `_EmptyEmail_NoRow_ClaimSucceeds` + store `TestSeedNotificationPrefs_*` (2) + `TestLastCustomerSentAt_PairingQuery` | PASS | -| E | `TestSavePreferences_EmptyEmailCannotClobber` / `_NonEmptyEmailStillUpdates` / `_EmptyEmailNoStoredRow` | PASS | -| F | `TestTestEvent_BothChannels` / `TestTestEvent_NoPrefsRow_NoPanic` | PASS | -| G | `TestPriorityHeaders` (table) / `TestPriorityHeaders_EndToEnd` (headers on error both channels, absent on warning) | PASS | -| guard | `TestRecovery_SeverityStaysInfo` (severityNotifies unwidened; non-recovery info silent) | PASS | +## Observed, not acted on -**Red-proofs (each: mutation applied → FAIL confirmed → reverted → suite green):** -1. Pairing check forced open (`if false && …`) → `TestRecovery_UnpairedStaysCustomerSilent` FAILED - ("unpaired recovery must NOT mail the customer, got 1"). ✔ -2. Seed via `SaveNotificationPrefs` upsert → `TestMarkClaimed_NeverOverwritesExistingPrefs` FAILED - ("claim seed MODIFIED an existing row"). ✔ -3. Clobber guard removed → `TestSavePreferences_EmptyEmailCannotClobber` FAILED ("CLOBBERED the - stored address: email=\"\""). ✔ -4. Headers unconditional → `TestPriorityHeaders_EndToEnd` FAILED ("warning mail … must carry NO - headers") (+ table test). ✔ - -## Deploy verification - -Image built + pushed (`build.sh 0.71.0 --push`), manifest bumped (`f1da171`), ArgoCD hard-refresh + -deliberate sync → **Synced / Healthy**, pod `hub-6ffbf574d8-p2w5p` Running, deployed image -`gitea.dooplex.hu/admin/felhom-hub:0.71.0`, startup log clean (checkers seeded 2 ok, no dispatch -errors). - -## Live test-mail leg (the exact endpoint the UI invokes) - -`POST /settings/notifications/test` on the demo-felhom guest controller (in-guest, session + -CSRF): controller returned „Teszt email elküldve." Hub log: -`Event from demo-felhom: test (info)` → `Test email sent to doodoo21@freemail.hu` → -`Operator test email sent for demo-felhom`. `notification_log` rows (UTC): - -``` -demo-felhom test customer sent Teszt értesítés 2026-07-22 18:59:34 -demo-felhom test operator sent operator test copy 2026-07-22 18:59:34 -``` - -Resend accepted both (no `failed` rows). **HANDED TO VIKTOR:** confirm the operator mail landed in -Primary, the phone buzzed (the new Gmail filter's proof), and the client shows it high-priority. - -## NOT live-validated in this task (verbatim, per spec) - -- The natural `*_recovered` mail on a real staleness cycle — rides the next real event or the - deliberate reboot drill; NOT fabricated by blocking reports (F9-bypass-shaped). -- Seed-at-claim on a real claim — next onboarding; Peti's Friday reinstall is the natural candidate. - -## Observations (not acted on) - -- `sendTestEmail` nil-prefs panic was latent-reachable live: a „Teszt értesítés" click on a - customer with no `customer_notifications` row (demo-hp today) would have panicked the dispatcher - goroutine. Fixed in-scope (the function was already an edit surface); noted here because it - predates this train. -- No standalone notification-plane feature doc exists under `documentation/` — N.4 was satisfied - in `REUSE.md` §Alerting (the de-facto plane map) + the capability map row; if Viktor wants a - dedicated `architecture/notifications.md`, that is a docs-only follow-up. -- `handleNotify` (legacy), mailrelay, `SendClaimEmail`/`SendSelfBindEmail` byte-untouched (claim/ - selfbind sends now pass explicit `nil` headers — no behavior change). -- Website shows no hub version → N.6 n/a. -- The `notification_log` is now pairing evidence (recovery gate reads it) — future retention/prune - work must keep customer-channel stale/down/recovered rows long enough to pair (noted in REUSE). +- demo-felhom's 07-21 staged offsite secret is still unconsumed (residue of the mistaken R-39-day + offsite Re-issue; box recovered via key-auth-first, which never consumes). Harmless; supports R-70. +- The 3 dead unclaimed-appliance records from the ISO train remain for operator discard. +- Hub pod log only reaches back to 07-22 20:58Z (restart); the 07-21 correlation came from the DB. diff --git a/documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md b/documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md index 1522682..82ce869 100644 --- a/documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md +++ b/documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md @@ -108,7 +108,7 @@ repo consistent, 14 snapshots, latest 04:16 CEST (pre-outage), **zero locks** (r |---|---|---|---|---| | **F8** | **Site power loss; no auto-power-on.** Power itself returned within minutes (router rebooted at ~14:42 and stayed up); both miniPCs remained off ~4.5 h until manual power-on. For a paying customer this converts a power blip into a half-day outage ending only when someone is physically present. | HIGH | p1-\*; router uptime operator-attested | **mitigated-on-site** — Viktor set BIOS AC-power-on on both boxes (attested; not OS-verifiable). **roadmap-candidate:** make BIOS "restore on AC power" a provisioning-checklist item + a host-install doc requirement for every fleet box. | | **F9** | **H1 OOB belt only partially installed on the current fleet.** `felhom-mgmt-watchdog.timer` is live on both boxes (heal marker absent = no heal needed), but `felhom-sshd.service` and `felhom-oob-nft.service` (TASK H1, present in `felhom-agent/configs/`) are installed on **neither** box; operator access rides stock sshd :22 + tailscale + G1 break-glass. Pre-existing (both boxes provisioned via the universal ISO), surfaced by this audit's access preflight. | MEDIUM | p2-felhom-pve-ssh-belt.txt, p2-\*-host.txt | **needs-ruling** — was H1 intentionally dropped from the universal-ISO provisioning path, or should host-install grow the belt? | -| **F10** | **demo-hp backup tiers incomplete.** Tier-2 dump tree exists (paperless-ngx, fresh), but **no offbox target is configured** and the PBS DR datastore holds **0 snapshots** for it. A power event with disk damage would have had no off-box recovery path for that guest. Pre-existing (box added 07-21), not outage-caused. | MEDIUM | p6-demo-hp-offbox-locks.txt (`NO-OFFBOX`), p6-demo-hp-backup.txt (`snapshots=0`), p6-demo-hp-tier2.txt | **roadmap-candidate** — finish demo-hp tier provisioning (offbox + DR) as part of the pilot hardening pass. | +| **F10** | **demo-hp backup tiers incomplete.** Tier-2 dump tree exists (paperless-ngx, fresh), but **no offbox target is configured** and the PBS DR datastore holds **0 snapshots** for it. A power event with disk damage would have had no off-box recovery path for that guest. Pre-existing (box added 07-21), not outage-caused. | MEDIUM | p6-demo-hp-offbox-locks.txt (`NO-OFFBOX`), p6-demo-hp-backup.txt (`snapshots=0`), p6-demo-hp-tier2.txt | **OFFSITE LEG RESOLVED 2026-07-23** — root cause was NOT "provisioning unfinished": the day-0 managed update killed the apply-bridge after password-consume, burning the one-shot credential (full diagnosis, designed-path repair via operator Re-issue, escrow ceremony, and a byte-identical offsite restore round-trip: `DIAG-f10-demo-hp-offsite-2026-07-23.md`; product rows minted **R-70** visibility + **R-71** the race). **The PBS-DR-snapshot half STAYS OPEN** pending F13 (cadence ruling) and the deliberate DR ceremony R-moment on demo-hp. | | **F11** | **Recovery is silent.** `host_recovered`/`node_recovered` fired correctly but carry severity `info`, which the dispatcher deliberately does not email — the operator/customer only learns of recovery by looking. During a real customer outage the "it's back" signal is arguably the second-most valuable email. Working-as-coded, so a product question, not a defect. | LOW | p5-hub-events.txt; `hub/internal/notify/dispatcher.go` severityNotifies | **needs-ruling** — opt-in recovery notifications (operator at least)? | | **F12** | **demo-hp has no customer notification prefs** (`customer_notifications` has no row for it) → its "customer" received no node_down email and never would. Only demo-felhom is wired (doodoo21@freemail.hu). Pre-existing demo-box config gap; on a real onboarding this must not be skippable. | LOW | p5-customer-notification-prefs.txt | **roadmap-candidate** — make notification-prefs setup a claim/onboarding step, not an optional settings page. | | **F13** | **PBS DR tier has no visible backup cadence.** `/etc/pve/jobs.cfg` is empty on both hosts; N100's DR datastore holds a single CT-9201 snapshot from **2026-07-18** (S8-era), demo-hp none. The agent's 6 h *verify* loop runs, but nothing appears to *create* periodic DR snapshots. If DR snapshots are meant to be on-demand only, fine — but then the "DR tier" freshness expectation should be documented; if they're meant to be periodic, the scheduler is missing. | LOW | p6-\*-backup.txt | **needs-ruling** — clarify intended DR-tier cadence; roadmap if periodic. | diff --git a/documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md b/documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md new file mode 100644 index 0000000..56a9334 --- /dev/null +++ b/documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md @@ -0,0 +1,136 @@ +# DIAG — demo-hp offsite: "enabled at the hub, absent on the box" (audit F10, offsite leg) + +**Date:** 2026-07-23 · **Operator:** Viktor (present; hub click + escrow ceremony) · **Executor:** Claude Code on DooPlex +**Scope:** felhom.eu hub v0.71.0 (`b67fce39`) + felhom-controller v0.160.0 (`0eba37d5`), live on demo-hp (`demo-hp-bb76ea`, guest 9201). +**Origin:** `AUDIT-power-outage-recovery-2026-07-22.md` F10 + operator observation 2026-07-23 (hub: "Provisioned: u629488-sub3@…"; box: „Még nincs beállítva távoli mentési cél"). +**Outcome:** root cause proven from both planes' evidence; repaired via the designed path (operator Re-issue, once); **tier proven with a full offsite→loss→restore round-trip, byte-identical**. Two new roadmap rows minted (R-70, R-71). No code changed. + +All timestamps UTC unless marked CEST (hub pod log is CEST; hub DB and guest are UTC). + +## 1. Phase-0 verdict + +Neither spec Shape A ("delivery never consumed") nor Shape B ("customer click pending") — a third, +fully-evidenced shape: + +> **The day-0 managed floor-update (0.153.0 → 0.156.0) killed the offsite apply-bridge between +> one-time-password consume and key-install/persist.** The password was spent hub-side; the box +> persisted nothing; every later bridge run hit the documented consume-404 dead-end +> ("password spent — reset it on the hub to retry"), forever, silently. + +Shape B was ruled out **from source**: the Felhom-managed offsite arrives exclusively via the +automatic apply-bridge (`felhom-controller` `main.go:290–322` — bridge runs at every process start; +config-version bump → self-restart → bridge). The box button „Távoli mentési cél beállítása" is the +**BYO NAS/SFTP form only** (`internal/web/offbox_handlers.go:44–126`, key/known_hosts textareas; +never contacts the hub consume endpoint). No customer click exists in the managed-offsite chain. + +## 2. The mechanism (source, cited) + +- Hub stages the transient password in `one_time_secrets` (one row per customer; UPSERT resets + `consumed_at` — `hub/internal/store/store.go:1218–1227`); the controller consumes it via + `POST /api/v1/offsite/consume-password/{customerID}` — a **column flip** `consumed_at=now` + in one tx; a second call returns 404 (`store.go:1232–1250`, `api/offsite.go:13–31`). +- The descriptor (non-secret) rides `controller.yaml`; the ACK carries only `config_version`. +- Bridge order (`controller/internal/offsiteapply/offsiteapply.go:106–187`): marker check → + host-key fingerprint verify (before consume, "don't waste it on a mismatch") → key-auth-first + probe (re-pin WITHOUT consuming if an installed key works) → keygen → **consume** (l.159–163) → + `ssh-copy-id` install + verify (l.168–173) → persist `offbox` target `EscrowState=pending` + (l.176) → marker LAST (l.181). **Consume-then-persist; retry only on next process start.** +- The exact dead-end is documented in source (l.168–173): *"key install FAILED after consuming the + one-time password … the password is spent; reset it on the hub to retry"* — the designed recovery + is the hub's **Re-issue offsite credentials** (`hub/internal/web/configs.go:730–773`: Hetzner + password reset → fresh one-time secret → ConfigVersion bump → `bumpIntent` long-poll wake; if an + escrow blob exists it is marked stale — demo-hp had none). +- **The hub UI cannot see any of this:** the "Provisioned: … delivered to the controller once" line + is static copy gated only on `offsite.host` in ConfigJSON + (`hub/internal/web/templates/config_form_body.html:119–120`); nothing in the web layer reads + `one_time_secrets.consumed_at`. "Provisioned" ≠ "delivered" ≠ "applied" — the hub shows the first + and knows neither of the others. (→ R-70; note the R-39 precedent: PBS-DR grew a + `consumed_at` honesty gauge for exactly this disagreement class.) + +## 3. The 07-21 timeline (evidence: hub DB `events`/`one_time_secrets`/`customer_claims`/`reports`) + +| UTC | Event | +|---|---| +| 16:23:32 | customer self-bind (`appliance_bound`) | +| 16:23:55 | appliance credential delivered — box begins day-0 | +| 16:24:19 | host row `demo-hp-bb76ea` created | +| 16:24:27 | hub auto-provisions offsite on enrollment (`offsite_reissued`, source=hub): sub3 minted, one-time pw staged, config_version→2 | +| **16:27:42** | **one-time password CONSUMED** (bridge inside the 0.153.0 floor controller) | +| 16:28:07 | `controller_started (0.153.0)`; first report 16:28:08 | +| **16:28:17** | **managed update 0.153.0→0.156.0 — container replaced, bridge killed mid-flight** (~35 s after consume, inside the install/persist window) | +| 16:28:22 | 0.156.0 starts; bridge: no installed key → consume → **404** → nothing persisted | + +Corroboration: **153 controller reports** 07-21 16:28 → 07-23 09:31, **zero** ever carrying an +offbox status object; box settings.json had **no `offbox` key** (checked with the real key, per the +R-7b lesson) and **no `offbox/` dir** (no ssh_key → the key-install never completed, so the +key-auth-first recovery path could never engage); live log at the 0.160.0 start 07-22 17:16:35: +`[WARN] [offsite-apply] reconcile: … consume one-time password: no unconsumed offsite password +(already consumed or none provisioned)`. + +Spec note: this is literally the spec's "delivery consumed but persist failed → STOP" class, but +the evidence showed the mechanism **not broken** (fail-safe held: no half-configured offbox) with a +source-designated recovery identical to Shape A's remedy; the operator ruled: proceed (recorded in +session, 2026-07-23). + +## 4. Repair via the designed path (operator click, once — R-31 discipline) + +| UTC 07-23 | Event | +|---|---| +| 09:53:37 | Viktor clicks **Re-issue offsite credentials** ONCE — `[offsite] re-issued shared credentials for demo-hp (subaccount 275124)`; fresh pw staged; ConfigVersion bumped | +| 09:53:39 | box long-poll wakes, config re-pulled | +| **09:53:41** | fresh password **consumed** (4 s after staging) | +| 09:53:45 | `[offsite-apply] offsite configured for u629488-sub3@…:/home/felhom-repo (pending key escrow)`; `offbox/` dir complete (ssh_key/known_hosts/repo_password/applied_marker), settings `offbox` present | + +Click → converged in **8 seconds**. Escrow ceremony run by the operator through the real +`/backup/escrow` wizard: blob stored 10:01:17 (`posture=zero_knowledge`, `restic_pw_sha256` +recorded), R claimed one-shot 10:01:21, hub-verified auto-confirm 10:01:24 → +`EscrowState=escrowed; offsite runs enabled`. + +## 5. Tier proof (F10 closure bar: restore round-trip) + +Per-app scope: `paperless-ngx` toggled into offsite via the real `POST /backup/offbox/toggle` +(per-app default is OFF). Mandatory capture path per catalog: `appdata/paperless/media`. +All steps via real endpoints from inside guest 9201 (endpoint-level method — no browser on DooPlex); +probe placement/deletion on the filesystem simulates customer data and its loss; every +backup/restore action went through the UI's own endpoints. + +1. **Probe** 64 KiB urandom → `…/appdata/paperless/media/felhom-probe-f10-20260723.bin`, + md5 `9120e65d6a9f071072d827fc404dc840`. +2. **First offsite run** (`POST /backup/offbox/run`): repo **initialized fresh** on sub3 (proves the + target was empty), 79.8 MB / 49 files, 1m19s, status `ok`, snapshots=1. +3. **Loss**: probe deleted (ENOENT verified). +4. **Restore** (`POST /backup/offbox/restore`, `mode=full`, two-step size gate revealed 79.8 MB → + `confirm=1`): snapshot **`2bf7f2e1`** → staging `backups/offsite-restore/paperless-ngx`; + staging probe **md5-identical**. +5. **Place** (`POST /backup/offbox/place`): `1 file(s) merged (missing-only)`, live recovery unit + not overwritten, scratch auto-removed; live probe **md5-identical** + (`9120e65d6a9f071072d827fc404dc840`). +6. **Cleanup**: probe removed; second run (2m17s, `ok`) so the latest snapshot is probe-free; + retention pruned the probe-bearing snapshot (count stays 1). Guest session artifacts removed; + break-glass + DB copies shredded on DooPlex. + +Hub-side final state: latest demo-hp report carries +`offsite: {enabled, escrow_state:"escrowed", quota_gb:50}`; `host_escrow` row present +(zero_knowledge, hash recorded, not stale); `one_time_secrets` consumed 09:53:41. The Offsite tab +now reflects the customer. Nightly offsite is scheduled (daily 04:15 guest-UTC). + +## 6. Findings minted + +- **R-70 (P2-HIGH)** — the pending/failed offsite last mile is invisible on BOTH surfaces; the hub + cannot distinguish staged/consumed/applied. Couple to R-31's async/status-card idiom and the + R-39 `consumed_at` honesty-gauge precedent. +- **R-71 (P1)** — the day-0 race itself: managed floor-update vs apply-bridge, consume-then-persist + not crash-safe. **This recurs on every fresh onboarding** whose ISO floor lags the managed floor + (the update fires minutes after first boot, exactly when the bridge runs). demo-felhom escaped by + timing only. +- Audit F10 row annotated: **offsite leg RESOLVED** (this record); the PBS-DR-snapshot half of F10 + **stays open** pending F13 (cadence ruling) and the deliberate DR ceremony R-moment on demo-hp. + +## 7. Observed, not acted on + +- `one_time_secrets` row for **demo-felhom** staged 2026-07-21 08:29:29 is still **unconsumed** — + residue of the mistaken offsite-Re-issue click during the R-39 session (the box recovered via the + key-auth-first path, which never consumes). Harmless (superseded on any future re-issue), but it + is exactly the state the R-39 honesty gauge flags on the PBS side — supporting R-70. +- The three dead unclaimed-appliance records from the 07-22 ISO train remain at the hub for + operator discard (unchanged). diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 2e6696f..ee1eebe 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -81,6 +81,8 @@ | R-67 | **The NAS share appears in FileBrowser — browse what you mounted.** | S | **SHIPPED (controller v0.160.0, 2026-07-22)** | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). **Couples to R-64: browsing was its missing UX half.** A registered network storage now binds its share ROOT into FileBrowser (`/mnt/felhom-drives/:/srv/:rslave`) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (`Host is down` in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: `felhom-controller/REPORT.md` (2026-07-22) | | R-68 | **Notification train: paired recovery mails + prefs seeding at claim + priority headers (power-outage audit F11+F12+F14-light).** The dead-man's-switch fired perfectly on 07-22 and the customer who got „A szerver nem elérhető!" was never told it recovered (F11); a customer without a `customer_notifications` row is silently unnotifiable (F12, demo-hp live); delivered ≠ noticed (F14). | M | **SHIPPED (hub v0.71.0, 2026-07-22)** | Origin: `AUDIT-power-outage-recovery-2026-07-22.md`. Recovery = explicit eventType branch (severity semantics frozen; `severityNotifies` untouched): operator always hears both edges, customer iff PAIRED (customer-channel `sent` stale/down row newer than the last sent recovery — `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for recovery; ties → no mail, flap-safe). Seed-at-claim: `MarkClaimed` → `SeedNotificationPrefs` (INSERT-if-absent, never upsert — red-proofed; empty email no-op; never fails the claim; default critical-only set). Hub-side empty-email no-clobber belt in `handleSavePreferences` (controller 0.160.0 already guards its own two push legs — latent, not live). `X-Priority: 1` + `Importance: high` on error/critical via Resend `headers` (live-probed HTTP 200 before implementation); the `test` event now also mails the operator with those headers (one click proves both channels + rendering); latent `sendTestEmail` nil-prefs panic fixed. 17 tests + 4 red-proofs. **Live legs pending:** natural `*_recovered` mail on the next real staleness cycle (or the reboot-drill arc — NEVER fabricated by blocking reports, that is F9-bypass-shaped) and seed-at-claim on a real claim (Peti Friday reinstall is the natural candidate) | | R-69 | **F14-full: an operator push channel that actually interrupts (ntfy / Telegram / similar), beyond mail-client priority flags.** F14-light (v0.71.0 headers + Gmail filter) nudges a mail client; a 15:29 node_down should reach the operator's pocket in seconds regardless of inbox hygiene. Needs: channel choice (self-hosted ntfy on k3s vs Telegram bot), dispatcher fan-out seam, per-severity routing, quiet hours. | M | idea | Origin: `AUDIT-power-outage-recovery-2026-07-22.md` F14. Deliberately NOT built in the v0.71.0 train (scope-forked per the task spec) | +| R-70 | **[P2-HIGH] The offsite last mile is invisible on BOTH surfaces — the hub cannot tell "staged" from "delivered" from "applied".** demo-hp sat 2 days with the hub customer page saying "Provisioned: … the transient password is delivered to the controller once" while the box said „Még nincs beállítva távoli mentési cél" — and a real customer would sit unprotected indefinitely believing otherwise. The hub HAS the signal (`one_time_secrets.consumed_at` + 153 consecutive reports carrying no offbox object) and reads none of it: the "Provisioned" line is static copy gated only on `offsite.host` in ConfigJSON (`config_form_body.html:119–120`). | S–M | idea | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. Two legs: **hub customer card** shows the real delivery state ("provisioned, awaiting box consumption" / "consumed, awaiting apply" / "applied" — consumed_at × report-offbox-presence is enough for all three), and **controller banner** when the descriptor is enabled but no target is configured („Felhom offsite készen áll — a beállítás automatikus, folyamatban"). Couple to **R-31**'s async/status-card idiom (same surface likely serves both) and to the **R-39 `consumed_at` honesty gauge** precedent on the PBS side — a consumed secret + N report cycles with no offbox status is the same "disagreement no single tier can see" shape and deserves the same loud event. Supporting live datum: demo-felhom's 07-21 staged secret is still unconsumed today (key-auth-first path never consumes) — invisible for the same reason. | +| R-71 | **[P1] Day-0 race: the managed floor-update kills the offsite apply-bridge between password-consume and persist — the one-shot credential is burned and the box lands in the silent consume-404 dead-end forever.** Proven on demo-hp (07-21): consume 16:27:42 → managed update 0.153.0→0.156.0 replaces the container 16:28:17, ~35 s later, mid `ssh-copy-id` window; nothing persisted, no installed key ⇒ the key-auth-first recovery path can never engage, and every subsequent start logs the WARN and gives up until an operator Re-issue. **This recurs structurally on every fresh onboarding whose ISO floor lags the managed floor** — the update fires minutes after first boot, exactly when the bridge first runs. demo-felhom escaped by timing only. | M | idea | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` (mechanism cites: bridge order `offsiteapply.go:106–187` consume-then-persist, retry only on process start; the dead-end is even documented in source l.168–173 "the password is spent; reset it on the hub to retry"). Candidate directions, spec-first: (a) **order** — first-boot path lets the managed update settle before the bridge's consume step (cheapest; the race window is the update, not the reboot); (b) **two-phase consume** — hub marks consumed only on a controller ack-after-persist (touches the 404-no-oracle contract, design carefully); (c) **hub-side self-heal** — auto-restage a fresh secret when `consumed_at` is set but K consecutive reports show offsite enabled with no offbox status (the R-39(a) mint-race lesson applies: never restage on top of an UNCONSUMED secret). Pairs with R-70 (visibility) — but visibility alone is not the fix; the burned credential needs an unattended recovery path. | | R-53 | **`app_export.html` substituted the CSRF token where the customer domain belongs** - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | **SHIPPED (controller v0.150.0, 2026-07-20)** | One template token (`{{$.CSRFToken}}` -> `{{$.Domain}}`) plus the `Domain` key in `exportPageHandler`'s data map - that handler does not go through `baseData`, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined `.` and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F7) | ## P3 — post-alpha