F10 offsite leg resolved on demo-hp: day-0 update-vs-bridge race diagnosed, designed-path repair + tier round-trip proven; R-70/R-71 minted

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR

CONTEXT.md

@@ -3,6 +3,20 @@
 > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
 > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
 
+- **2026-07-23 — F10 offsite leg RESOLVED on demo-hp (diagnostic + designed-path repair + tier
+  proof; no code changed).** The hub said "Provisioned" while the box said „Még nincs beállítva" —
+  root cause: the day-0 managed update (0.153.0→0.156.0, 07-21 16:28) killed the offsite
+  apply-bridge ~35 s after it consumed the one-time password, before key-install/persist; the box
+  then hit the consume-404 dead-end at every start, silently, for 2 days. Repair = operator
+  **Re-issue** click (converged in 8 s), escrow ceremony via the real wizard (auto-confirm 3 s),
+  then the full proof: paperless-ngx toggled into scope, first offsite run (repo initialized on
+  sub3, 79.8 MB, snapshots=1), probe deleted → `mode=full` restore (snapshot `2bf7f2e1`) → place
+  merged 1 file missing-only → **md5-identical**. Minted **R-70** (last-mile invisible on both
+  surfaces; hub never reads `one_time_secrets.consumed_at`) and **R-71 [P1]** (the race recurs on
+  every fresh onboarding whose ISO floor lags the managed floor). Record:
+  `documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. F10's PBS-DR half stays open
+  (F13 + ceremony R-moment).
+
 - **2026-07-22 — hub v0.71.0: the notification train (audit F11+F12+F14-light).** Four rulings now
   standing: **(1) recovery pairing** — `*_recovered` notifies the operator always and the customer
   **iff the customer was mailed the paired stale/down** (evidence = `notification_log` customer-

REPORT.md

@@ -1,122 +1,63 @@
-# REPORT — Hub notification train v0.71.0: F11 + F12 + F14-light (2026-07-22)
+# REPORT — F10 diagnostic: demo-hp offsite "enabled at the hub, absent on the box" (2026-07-23)
 
-**Task:** paired recovery notifications, prefs seeding at claim, empty-email no-clobber guard,
+**Task:** F10 diagnostic spec (project Claude, 2026-07-23) — diagnose first, repair only via the
-priority headers, operator test leg — all in `hub/`; felhom-controller read-only reference.
+designed path, prove the tier. **No code changed in any repo.** Full evidence record:
-Origin: `documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md` (F11/F12/F14-light).
+`documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`.
 
-## Baselines (live-verified before starting)
+## Phase-0 verdict
 
-| Repo | main @ start | → shipped |
+Neither of the spec's two candidate shapes. The evidence (hub DB + box state + live logs, every
-|---|---|---|
+claim cited in the DIAG) proves a third: **the day-0 managed floor-update (0.153.0→0.156.0,
-| felhom.eu | `5b35023574173e17f9ee968ba09b6ab34616a322`, hub 0.70.1 in `manifests/hub.yaml` | hub **0.71.0** — code `c766c8a`, manifest `f1da171` |
+07-21 16:28:17Z) killed the offsite apply-bridge ~35 s after it consumed the one-time password**
-| felhom-controller | `0eba37d5cdc6` v0.160.0 | unchanged (read-only) |
+(16:27:42Z), before key-install/persist. Consume-then-persist + retry-only-on-restart
+(`offsiteapply.go:106–187`) ⇒ the credential was burned, no key was ever installed (so the
+key-auth-first recovery path could never engage), and every later start logged the consume-404 WARN
+and gave up. 153 reports over 2 days never carried an offbox object; the hub's "Provisioned…"
+line is static copy that reads neither `consumed_at` nor the reports.
 
-## Phase 0a — controller prefs-push fact (read-only, cited)
+- Shape B ruled out from source: managed offsite is fully automatic; the box's „Távoli mentési cél
+  beállítása" button is the BYO NAS/SFTP form only (`offbox_handlers.go:44–126`).
+- Strictly this was the spec's "consumed but persist failed → STOP" class; since the mechanism
+  provably held its fail-safe and the source itself designates the recovery ("the password is
+  spent; reset it on the hub to retry" = the offsite Re-issue), the operator ruled in-session:
+  proceed on the Re-issue path.
 
-**Does the startup push fire for a never-configured customer, and with what email?** It does NOT
+## Repair (designed path only)
-fire: `felhom-controller/controller/cmd/controller/main.go:821–830` guards the startup sync with
-`if prefs.Email != ""` — an unconfigured box pushes nothing. The settings-save leg
-(`internal/web/handlers.go:1532–1545`, the 2026-07-15 wipe-guard) refuses empty-email-with-events
-outright; only the intentional all-off case (empty email + zero events) pushes an empty email, and
-that push is legitimate. **Verdict: the clobber was LATENT on controller 0.160.0, not live** — the
-hub-side guard shipped as the belt for older/rogue controllers. The debug re-sync
-(`internal/web/handler_debug.go:432`) pushes stored prefs as-is. Test-leg endpoint the UI invokes:
-`POST /settings/notifications/test` (`internal/web/server.go:395` →
-`settingsNotificationsTestHandler`, `handlers.go:1616`) → `notifier.SendTest()` → hub
-`POST /api/v1/event` with eventType `test`.
 
-## Phase 0b — Resend custom-headers probe
+- Operator clicked **Re-issue offsite credentials** ONCE (R-31 click-once discipline; pre-verified
+  side-effect-free: no escrow blob existed, the box never held the old password, sub3 was empty).
+  Click 09:53:37Z → box consumed 09:53:41 → `offsite configured … (pending key escrow)` 09:53:45.
+  **8 seconds click-to-converged.**
+- Escrow ceremony run by the operator through the real `/backup/escrow` wizard (one-shot R on the
+  operator's screen only): blob stored 10:01:17 (zero_knowledge, pw-hash recorded), hub-verified
+  auto-confirm 10:01:24 → `escrowed; offsite runs enabled`.
 
-One real mail via the Resend API (`from monitoring@felhom.eu`, `to admin@felhom.eu`, subject
+## Tier proof (F10 closure bar)
-"[Felhom] header probe") with `"headers": {"X-Priority":"1","Importance":"high"}` — **HTTP 200**,
-mail id `34d3f7f3-849f-432a-ac8e-eb479a1c6695`. Key sourced from the k8s Secret into the shell env
-only; never written to any file or log. Mechanism GO.
 
-## Files modified (commit `c766c8a`, docs+code; manifest bump `f1da171`)
+paperless-ngx toggled into offsite scope via the real endpoint (per-app default is OFF). Then, all
+via real endpoints from inside guest 9201 (endpoint-level method; no browser on DooPlex):
+probe (md5 `9120e65d6a9f071072d827fc404dc840`) in the mandatory `appdata/paperless/media` →
+**first offsite run**: repo initialized fresh on sub3, 79.8 MB / 49 files, 1m19s, ok →
+probe deleted → **`mode=full` restore** (size gate 79.8 MB → confirm): snapshot **`2bf7f2e1`** to
+staging, staging md5-identical → **place**: `1 file(s) merged (missing-only)`, live md5-identical.
+Cleanup: probe removed, second run (2m17s ok) leaves the latest snapshot probe-free (retention
+pruned the probe-bearing one); zero residue on box/repo; break-glass + DB copies shredded.
+Hub now reports demo-hp `offsite: enabled/escrowed/quota 50`; nightly run scheduled (04:15 UTC).
 
-- `hub/internal/store/store.go` — `LastCustomerSentAt` (pairing-evidence query, uses the
+## Product findings + docs
-  `(customer_id, created_at DESC)` index), `SeedNotificationPrefs` (INSERT OR IGNORE, empty-email
-  no-op); `strings` import.
-- `hub/internal/notify/dispatcher.go` — `priorityHeaders`; `recoveredPairedDownTypes`; recovery
-  branch in `ProcessEvent` (before the severity gate; `severityNotifies` byte-untouched);
-  `processRecovery` (operator = existing `processOperator`; customer = blocked → prefs-nil/email →
-  **pairing** → prefs cooldown; INFO logs on every gated skip); `sendEmailFn`/`sendEmail` grew
-  `headers map[string]string` (payload carries `"headers"` only when non-empty; all callers
-  updated); `sendTestEmail` operator copy with forced priority headers + **latent nil-prefs panic
-  fixed** (`GetNotificationPrefs` returns `(nil, nil)` — the old code dereferenced `prefs.Email`).
-- `hub/internal/notify/templates.go` — ✅ icon for `*_recovered` (eventType-aware, severity
-  fallback); `customerMessages["host_recovered"]`.
-- `hub/internal/claim/engine.go` — `defaultSeedEvents` (critical-only) + seed call in
-  `MarkClaimed` (WARN-and-continue on failure; claim never hinges on notification plumbing).
-- `hub/internal/api/handler.go` — `handleSavePreferences` empty-email no-clobber guard (INFO log).
-- Tests: `notifications_seed_test.go` (store), `dispatcher_recovery_test.go` (notify),
-  `seed_test.go` (claim), `preferences_guard_test.go` (api); `dispatcher_test.go` seam signatures.
-- Docs: `hub/CHANGELOG.md`, `CONTEXT.md` (4 rulings), `REUSE.md` (§Alerting rewritten — serves as
-  the notification-plane map; no standalone notification feature doc exists, noted below),
-  `00-capability-map.md` (§F rows), `ROADMAP.md` (R-68 SHIPPED, R-69 minted for F14-full).
 
-## Tests — 449 → 466 (17 new), full suite green (`go build && go vet && go test ./...` exit 0)
+- **R-70 (P2-HIGH)** minted: the offsite last mile is invisible on both surfaces (hub can't tell
+  staged/consumed/applied; box shows the generic empty state). Coupled to R-31's status-card idiom
+  and the R-39 consumed_at honesty-gauge precedent.
+- **R-71 (P1)** minted: the race itself — recurs structurally on every fresh onboarding whose ISO
+  floor lags the managed floor. Spec-first directions listed in the row (ordering / two-phase
+  consume / hub-side auto-restage with the R-39(a) mint-race guard).
+- Audit F10 row annotated: **offsite leg resolved**; PBS-DR half explicitly stays open (F13 +
+  DR ceremony R-moment). CONTEXT.md updated.
 
-| Group | Test | Result |
+## Observed, not acted on
-|---|---|---|
-| A | `TestRecovery_PairedCustomerMail` (down→recovery: both channels, Hungarian subject, ✅ subject, both log rows) | PASS |
-| B | `TestRecovery_UnpairedStaysCustomerSilent` (operator only; enabled_events ignored) | PASS |
-| C | `TestRecovery_FlapDamping` (suppressed down ⇒ no 2nd recovery mail; op 1 h cooldown) | PASS |
-| D | `TestMarkClaimed_SeedsNotificationPrefs` / `_NeverOverwritesExistingPrefs` / `_EmptyEmail_NoRow_ClaimSucceeds` + store `TestSeedNotificationPrefs_*` (2) + `TestLastCustomerSentAt_PairingQuery` | PASS |
-| E | `TestSavePreferences_EmptyEmailCannotClobber` / `_NonEmptyEmailStillUpdates` / `_EmptyEmailNoStoredRow` | PASS |
-| F | `TestTestEvent_BothChannels` / `TestTestEvent_NoPrefsRow_NoPanic` | PASS |
-| G | `TestPriorityHeaders` (table) / `TestPriorityHeaders_EndToEnd` (headers on error both channels, absent on warning) | PASS |
-| guard | `TestRecovery_SeverityStaysInfo` (severityNotifies unwidened; non-recovery info silent) | PASS |
 
-**Red-proofs (each: mutation applied → FAIL confirmed → reverted → suite green):**
+- demo-felhom's 07-21 staged offsite secret is still unconsumed (residue of the mistaken R-39-day
-1. Pairing check forced open (`if false && …`) → `TestRecovery_UnpairedStaysCustomerSilent` FAILED
+  offsite Re-issue; box recovered via key-auth-first, which never consumes). Harmless; supports R-70.
-   ("unpaired recovery must NOT mail the customer, got 1"). ✔
+- The 3 dead unclaimed-appliance records from the ISO train remain for operator discard.
-2. Seed via `SaveNotificationPrefs` upsert → `TestMarkClaimed_NeverOverwritesExistingPrefs` FAILED
+- Hub pod log only reaches back to 07-22 20:58Z (restart); the 07-21 correlation came from the DB.
-   ("claim seed MODIFIED an existing row"). ✔
-3. Clobber guard removed → `TestSavePreferences_EmptyEmailCannotClobber` FAILED ("CLOBBERED the
-   stored address: email=\"\""). ✔
-4. Headers unconditional → `TestPriorityHeaders_EndToEnd` FAILED ("warning mail … must carry NO
-   headers") (+ table test). ✔
-
-## Deploy verification
-
-Image built + pushed (`build.sh 0.71.0 --push`), manifest bumped (`f1da171`), ArgoCD hard-refresh +
-deliberate sync → **Synced / Healthy**, pod `hub-6ffbf574d8-p2w5p` Running, deployed image
-`gitea.dooplex.hu/admin/felhom-hub:0.71.0`, startup log clean (checkers seeded 2 ok, no dispatch
-errors).
-
-## Live test-mail leg (the exact endpoint the UI invokes)
-
-`POST /settings/notifications/test` on the demo-felhom guest controller (in-guest, session +
-CSRF): controller returned „Teszt email elküldve." Hub log:
-`Event from demo-felhom: test (info)` → `Test email sent to doodoo21@freemail.hu` →
-`Operator test email sent for demo-felhom`. `notification_log` rows (UTC):
-
-```
-demo-felhom  test  customer  sent  Teszt értesítés      2026-07-22 18:59:34
-demo-felhom  test  operator  sent  operator test copy   2026-07-22 18:59:34
-```
-
-Resend accepted both (no `failed` rows). **HANDED TO VIKTOR:** confirm the operator mail landed in
-Primary, the phone buzzed (the new Gmail filter's proof), and the client shows it high-priority.
-
-## NOT live-validated in this task (verbatim, per spec)
-
-- The natural `*_recovered` mail on a real staleness cycle — rides the next real event or the
-  deliberate reboot drill; NOT fabricated by blocking reports (F9-bypass-shaped).
-- Seed-at-claim on a real claim — next onboarding; Peti's Friday reinstall is the natural candidate.
-
-## Observations (not acted on)
-
-- `sendTestEmail` nil-prefs panic was latent-reachable live: a „Teszt értesítés" click on a
-  customer with no `customer_notifications` row (demo-hp today) would have panicked the dispatcher
-  goroutine. Fixed in-scope (the function was already an edit surface); noted here because it
-  predates this train.
-- No standalone notification-plane feature doc exists under `documentation/` — N.4 was satisfied
-  in `REUSE.md` §Alerting (the de-facto plane map) + the capability map row; if Viktor wants a
-  dedicated `architecture/notifications.md`, that is a docs-only follow-up.
-- `handleNotify` (legacy), mailrelay, `SendClaimEmail`/`SendSelfBindEmail` byte-untouched (claim/
-  selfbind sends now pass explicit `nil` headers — no behavior change).
-- Website shows no hub version → N.6 n/a.
-- The `notification_log` is now pairing evidence (recovery gate reads it) — future retention/prune
-  work must keep customer-channel stale/down/recovered rows long enough to pair (noted in REUSE).

documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md

@@ -108,7 +108,7 @@ repo consistent, 14 snapshots, latest 04:16 CEST (pre-outage), **zero locks** (r
 |---|---|---|---|---|
 | **F8** | **Site power loss; no auto-power-on.** Power itself returned within minutes (router rebooted at ~14:42 and stayed up); both miniPCs remained off ~4.5 h until manual power-on. For a paying customer this converts a power blip into a half-day outage ending only when someone is physically present. | HIGH | p1-\*; router uptime operator-attested | **mitigated-on-site** — Viktor set BIOS AC-power-on on both boxes (attested; not OS-verifiable). **roadmap-candidate:** make BIOS "restore on AC power" a provisioning-checklist item + a host-install doc requirement for every fleet box. |
 | **F9** | **H1 OOB belt only partially installed on the current fleet.** `felhom-mgmt-watchdog.timer` is live on both boxes (heal marker absent = no heal needed), but `felhom-sshd.service` and `felhom-oob-nft.service` (TASK H1, present in `felhom-agent/configs/`) are installed on **neither** box; operator access rides stock sshd :22 + tailscale + G1 break-glass. Pre-existing (both boxes provisioned via the universal ISO), surfaced by this audit's access preflight. | MEDIUM | p2-felhom-pve-ssh-belt.txt, p2-\*-host.txt | **needs-ruling** — was H1 intentionally dropped from the universal-ISO provisioning path, or should host-install grow the belt? |
-| **F10** | **demo-hp backup tiers incomplete.** Tier-2 dump tree exists (paperless-ngx, fresh), but **no offbox target is configured** and the PBS DR datastore holds **0 snapshots** for it. A power event with disk damage would have had no off-box recovery path for that guest. Pre-existing (box added 07-21), not outage-caused. | MEDIUM | p6-demo-hp-offbox-locks.txt (`NO-OFFBOX`), p6-demo-hp-backup.txt (`snapshots=0`), p6-demo-hp-tier2.txt | **roadmap-candidate** — finish demo-hp tier provisioning (offbox + DR) as part of the pilot hardening pass. |
+| **F10** | **demo-hp backup tiers incomplete.** Tier-2 dump tree exists (paperless-ngx, fresh), but **no offbox target is configured** and the PBS DR datastore holds **0 snapshots** for it. A power event with disk damage would have had no off-box recovery path for that guest. Pre-existing (box added 07-21), not outage-caused. | MEDIUM | p6-demo-hp-offbox-locks.txt (`NO-OFFBOX`), p6-demo-hp-backup.txt (`snapshots=0`), p6-demo-hp-tier2.txt | **OFFSITE LEG RESOLVED 2026-07-23** — root cause was NOT "provisioning unfinished": the day-0 managed update killed the apply-bridge after password-consume, burning the one-shot credential (full diagnosis, designed-path repair via operator Re-issue, escrow ceremony, and a byte-identical offsite restore round-trip: `DIAG-f10-demo-hp-offsite-2026-07-23.md`; product rows minted **R-70** visibility + **R-71** the race). **The PBS-DR-snapshot half STAYS OPEN** pending F13 (cadence ruling) and the deliberate DR ceremony R-moment on demo-hp. |
 | **F11** | **Recovery is silent.** `host_recovered`/`node_recovered` fired correctly but carry severity `info`, which the dispatcher deliberately does not email — the operator/customer only learns of recovery by looking. During a real customer outage the "it's back" signal is arguably the second-most valuable email. Working-as-coded, so a product question, not a defect. | LOW | p5-hub-events.txt; `hub/internal/notify/dispatcher.go` severityNotifies | **needs-ruling** — opt-in recovery notifications (operator at least)? |
 | **F12** | **demo-hp has no customer notification prefs** (`customer_notifications` has no row for it) → its "customer" received no node_down email and never would. Only demo-felhom is wired (doodoo21@freemail.hu). Pre-existing demo-box config gap; on a real onboarding this must not be skippable. | LOW | p5-customer-notification-prefs.txt | **roadmap-candidate** — make notification-prefs setup a claim/onboarding step, not an optional settings page. |
 | **F13** | **PBS DR tier has no visible backup cadence.** `/etc/pve/jobs.cfg` is empty on both hosts; N100's DR datastore holds a single CT-9201 snapshot from **2026-07-18** (S8-era), demo-hp none. The agent's 6 h *verify* loop runs, but nothing appears to *create* periodic DR snapshots. If DR snapshots are meant to be on-demand only, fine — but then the "DR tier" freshness expectation should be documented; if they're meant to be periodic, the scheduler is missing. | LOW | p6-\*-backup.txt | **needs-ruling** — clarify intended DR-tier cadence; roadmap if periodic. |

documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md

@@ -0,0 +1,136 @@
+# DIAG — demo-hp offsite: "enabled at the hub, absent on the box" (audit F10, offsite leg)
+
+**Date:** 2026-07-23 · **Operator:** Viktor (present; hub click + escrow ceremony) · **Executor:** Claude Code on DooPlex
+**Scope:** felhom.eu hub v0.71.0 (`b67fce39`) + felhom-controller v0.160.0 (`0eba37d5`), live on demo-hp (`demo-hp-bb76ea`, guest 9201).
+**Origin:** `AUDIT-power-outage-recovery-2026-07-22.md` F10 + operator observation 2026-07-23 (hub: "Provisioned: u629488-sub3@…"; box: „Még nincs beállítva távoli mentési cél").
+**Outcome:** root cause proven from both planes' evidence; repaired via the designed path (operator Re-issue, once); **tier proven with a full offsite→loss→restore round-trip, byte-identical**. Two new roadmap rows minted (R-70, R-71). No code changed.
+
+All timestamps UTC unless marked CEST (hub pod log is CEST; hub DB and guest are UTC).
+
+## 1. Phase-0 verdict
+
+Neither spec Shape A ("delivery never consumed") nor Shape B ("customer click pending") — a third,
+fully-evidenced shape:
+
+> **The day-0 managed floor-update (0.153.0 → 0.156.0) killed the offsite apply-bridge between
+> one-time-password consume and key-install/persist.** The password was spent hub-side; the box
+> persisted nothing; every later bridge run hit the documented consume-404 dead-end
+> ("password spent — reset it on the hub to retry"), forever, silently.
+
+Shape B was ruled out **from source**: the Felhom-managed offsite arrives exclusively via the
+automatic apply-bridge (`felhom-controller` `main.go:290–322` — bridge runs at every process start;
+config-version bump → self-restart → bridge). The box button „Távoli mentési cél beállítása" is the
+**BYO NAS/SFTP form only** (`internal/web/offbox_handlers.go:44–126`, key/known_hosts textareas;
+never contacts the hub consume endpoint). No customer click exists in the managed-offsite chain.
+
+## 2. The mechanism (source, cited)
+
+- Hub stages the transient password in `one_time_secrets` (one row per customer; UPSERT resets
+  `consumed_at` — `hub/internal/store/store.go:1218–1227`); the controller consumes it via
+  `POST /api/v1/offsite/consume-password/{customerID}` — a **column flip** `consumed_at=now`
+  in one tx; a second call returns 404 (`store.go:1232–1250`, `api/offsite.go:13–31`).
+- The descriptor (non-secret) rides `controller.yaml`; the ACK carries only `config_version`.
+- Bridge order (`controller/internal/offsiteapply/offsiteapply.go:106–187`): marker check →
+  host-key fingerprint verify (before consume, "don't waste it on a mismatch") → key-auth-first
+  probe (re-pin WITHOUT consuming if an installed key works) → keygen → **consume** (l.159–163) →
+  `ssh-copy-id` install + verify (l.168–173) → persist `offbox` target `EscrowState=pending`
+  (l.176) → marker LAST (l.181). **Consume-then-persist; retry only on next process start.**
+- The exact dead-end is documented in source (l.168–173): *"key install FAILED after consuming the
+  one-time password … the password is spent; reset it on the hub to retry"* — the designed recovery
+  is the hub's **Re-issue offsite credentials** (`hub/internal/web/configs.go:730–773`: Hetzner
+  password reset → fresh one-time secret → ConfigVersion bump → `bumpIntent` long-poll wake; if an
+  escrow blob exists it is marked stale — demo-hp had none).
+- **The hub UI cannot see any of this:** the "Provisioned: … delivered to the controller once" line
+  is static copy gated only on `offsite.host` in ConfigJSON
+  (`hub/internal/web/templates/config_form_body.html:119–120`); nothing in the web layer reads
+  `one_time_secrets.consumed_at`. "Provisioned" ≠ "delivered" ≠ "applied" — the hub shows the first
+  and knows neither of the others. (→ R-70; note the R-39 precedent: PBS-DR grew a
+  `consumed_at` honesty gauge for exactly this disagreement class.)
+
+## 3. The 07-21 timeline (evidence: hub DB `events`/`one_time_secrets`/`customer_claims`/`reports`)
+
+| UTC | Event |
+|---|---|
+| 16:23:32 | customer self-bind (`appliance_bound`) |
+| 16:23:55 | appliance credential delivered — box begins day-0 |
+| 16:24:19 | host row `demo-hp-bb76ea` created |
+| 16:24:27 | hub auto-provisions offsite on enrollment (`offsite_reissued`, source=hub): sub3 minted, one-time pw staged, config_version→2 |
+| **16:27:42** | **one-time password CONSUMED** (bridge inside the 0.153.0 floor controller) |
+| 16:28:07 | `controller_started (0.153.0)`; first report 16:28:08 |
+| **16:28:17** | **managed update 0.153.0→0.156.0 — container replaced, bridge killed mid-flight** (~35 s after consume, inside the install/persist window) |
+| 16:28:22 | 0.156.0 starts; bridge: no installed key → consume → **404** → nothing persisted |
+
+Corroboration: **153 controller reports** 07-21 16:28 → 07-23 09:31, **zero** ever carrying an
+offbox status object; box settings.json had **no `offbox` key** (checked with the real key, per the
+R-7b lesson) and **no `offbox/` dir** (no ssh_key → the key-install never completed, so the
+key-auth-first recovery path could never engage); live log at the 0.160.0 start 07-22 17:16:35:
+`[WARN] [offsite-apply] reconcile: … consume one-time password: no unconsumed offsite password
+(already consumed or none provisioned)`.
+
+Spec note: this is literally the spec's "delivery consumed but persist failed → STOP" class, but
+the evidence showed the mechanism **not broken** (fail-safe held: no half-configured offbox) with a
+source-designated recovery identical to Shape A's remedy; the operator ruled: proceed (recorded in
+session, 2026-07-23).
+
+## 4. Repair via the designed path (operator click, once — R-31 discipline)
+
+| UTC 07-23 | Event |
+|---|---|
+| 09:53:37 | Viktor clicks **Re-issue offsite credentials** ONCE — `[offsite] re-issued shared credentials for demo-hp (subaccount 275124)`; fresh pw staged; ConfigVersion bumped |
+| 09:53:39 | box long-poll wakes, config re-pulled |
+| **09:53:41** | fresh password **consumed** (4 s after staging) |
+| 09:53:45 | `[offsite-apply] offsite configured for u629488-sub3@…:/home/felhom-repo (pending key escrow)`; `offbox/` dir complete (ssh_key/known_hosts/repo_password/applied_marker), settings `offbox` present |
+
+Click → converged in **8 seconds**. Escrow ceremony run by the operator through the real
+`/backup/escrow` wizard: blob stored 10:01:17 (`posture=zero_knowledge`, `restic_pw_sha256`
+recorded), R claimed one-shot 10:01:21, hub-verified auto-confirm 10:01:24 →
+`EscrowState=escrowed; offsite runs enabled`.
+
+## 5. Tier proof (F10 closure bar: restore round-trip)
+
+Per-app scope: `paperless-ngx` toggled into offsite via the real `POST /backup/offbox/toggle`
+(per-app default is OFF). Mandatory capture path per catalog: `appdata/paperless/media`.
+All steps via real endpoints from inside guest 9201 (endpoint-level method — no browser on DooPlex);
+probe placement/deletion on the filesystem simulates customer data and its loss; every
+backup/restore action went through the UI's own endpoints.
+
+1. **Probe** 64 KiB urandom → `…/appdata/paperless/media/felhom-probe-f10-20260723.bin`,
+   md5 `9120e65d6a9f071072d827fc404dc840`.
+2. **First offsite run** (`POST /backup/offbox/run`): repo **initialized fresh** on sub3 (proves the
+   target was empty), 79.8 MB / 49 files, 1m19s, status `ok`, snapshots=1.
+3. **Loss**: probe deleted (ENOENT verified).
+4. **Restore** (`POST /backup/offbox/restore`, `mode=full`, two-step size gate revealed 79.8 MB →
+   `confirm=1`): snapshot **`2bf7f2e1`** → staging `backups/offsite-restore/paperless-ngx`;
+   staging probe **md5-identical**.
+5. **Place** (`POST /backup/offbox/place`): `1 file(s) merged (missing-only)`, live recovery unit
+   not overwritten, scratch auto-removed; live probe **md5-identical**
+   (`9120e65d6a9f071072d827fc404dc840`).
+6. **Cleanup**: probe removed; second run (2m17s, `ok`) so the latest snapshot is probe-free;
+   retention pruned the probe-bearing snapshot (count stays 1). Guest session artifacts removed;
+   break-glass + DB copies shredded on DooPlex.
+
+Hub-side final state: latest demo-hp report carries
+`offsite: {enabled, escrow_state:"escrowed", quota_gb:50}`; `host_escrow` row present
+(zero_knowledge, hash recorded, not stale); `one_time_secrets` consumed 09:53:41. The Offsite tab
+now reflects the customer. Nightly offsite is scheduled (daily 04:15 guest-UTC).
+
+## 6. Findings minted
+
+- **R-70 (P2-HIGH)** — the pending/failed offsite last mile is invisible on BOTH surfaces; the hub
+  cannot distinguish staged/consumed/applied. Couple to R-31's async/status-card idiom and the
+  R-39 `consumed_at` honesty-gauge precedent.
+- **R-71 (P1)** — the day-0 race itself: managed floor-update vs apply-bridge, consume-then-persist
+  not crash-safe. **This recurs on every fresh onboarding** whose ISO floor lags the managed floor
+  (the update fires minutes after first boot, exactly when the bridge runs). demo-felhom escaped by
+  timing only.
+- Audit F10 row annotated: **offsite leg RESOLVED** (this record); the PBS-DR-snapshot half of F10
+  **stays open** pending F13 (cadence ruling) and the deliberate DR ceremony R-moment on demo-hp.
+
+## 7. Observed, not acted on
+
+- `one_time_secrets` row for **demo-felhom** staged 2026-07-21 08:29:29 is still **unconsumed** —
+  residue of the mistaken offsite-Re-issue click during the R-39 session (the box recovered via the
+  key-auth-first path, which never consumes). Harmless (superseded on any future re-issue), but it
+  is exactly the state the R-39 honesty gauge flags on the PBS side — supporting R-70.
+- The three dead unclaimed-appliance records from the 07-22 ISO train remain at the hub for
+  operator discard (unchanged).

documentation/backlog/ROADMAP.md

@@ -81,6 +81,8 @@
 | R-67 | **The NAS share appears in FileBrowser — browse what you mounted.** | S | **SHIPPED (controller v0.160.0, 2026-07-22)** | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). **Couples to R-64: browsing was its missing UX half.** A registered network storage now binds its share ROOT into FileBrowser (`/mnt/felhom-drives/<name>:/srv/<name>:rslave`) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (`Host is down` in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: `felhom-controller/REPORT.md` (2026-07-22) |
 | R-68 | **Notification train: paired recovery mails + prefs seeding at claim + priority headers (power-outage audit F11+F12+F14-light).** The dead-man's-switch fired perfectly on 07-22 and the customer who got „A szerver nem elérhető!" was never told it recovered (F11); a customer without a `customer_notifications` row is silently unnotifiable (F12, demo-hp live); delivered ≠ noticed (F14). | M | **SHIPPED (hub v0.71.0, 2026-07-22)** | Origin: `AUDIT-power-outage-recovery-2026-07-22.md`. Recovery = explicit eventType branch (severity semantics frozen; `severityNotifies` untouched): operator always hears both edges, customer iff PAIRED (customer-channel `sent` stale/down row newer than the last sent recovery — `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for recovery; ties → no mail, flap-safe). Seed-at-claim: `MarkClaimed` → `SeedNotificationPrefs` (INSERT-if-absent, never upsert — red-proofed; empty email no-op; never fails the claim; default critical-only set). Hub-side empty-email no-clobber belt in `handleSavePreferences` (controller 0.160.0 already guards its own two push legs — latent, not live). `X-Priority: 1` + `Importance: high` on error/critical via Resend `headers` (live-probed HTTP 200 before implementation); the `test` event now also mails the operator with those headers (one click proves both channels + rendering); latent `sendTestEmail` nil-prefs panic fixed. 17 tests + 4 red-proofs. **Live legs pending:** natural `*_recovered` mail on the next real staleness cycle (or the reboot-drill arc — NEVER fabricated by blocking reports, that is F9-bypass-shaped) and seed-at-claim on a real claim (Peti Friday reinstall is the natural candidate) |
 | R-69 | **F14-full: an operator push channel that actually interrupts (ntfy / Telegram / similar), beyond mail-client priority flags.** F14-light (v0.71.0 headers + Gmail filter) nudges a mail client; a 15:29 node_down should reach the operator's pocket in seconds regardless of inbox hygiene. Needs: channel choice (self-hosted ntfy on k3s vs Telegram bot), dispatcher fan-out seam, per-severity routing, quiet hours. | M | idea | Origin: `AUDIT-power-outage-recovery-2026-07-22.md` F14. Deliberately NOT built in the v0.71.0 train (scope-forked per the task spec) |
+| R-70 | **[P2-HIGH] The offsite last mile is invisible on BOTH surfaces — the hub cannot tell "staged" from "delivered" from "applied".** demo-hp sat 2 days with the hub customer page saying "Provisioned: … the transient password is delivered to the controller once" while the box said „Még nincs beállítva távoli mentési cél" — and a real customer would sit unprotected indefinitely believing otherwise. The hub HAS the signal (`one_time_secrets.consumed_at` + 153 consecutive reports carrying no offbox object) and reads none of it: the "Provisioned" line is static copy gated only on `offsite.host` in ConfigJSON (`config_form_body.html:119–120`). | S–M | idea | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. Two legs: **hub customer card** shows the real delivery state ("provisioned, awaiting box consumption" / "consumed, awaiting apply" / "applied" — consumed_at × report-offbox-presence is enough for all three), and **controller banner** when the descriptor is enabled but no target is configured („Felhom offsite készen áll — a beállítás automatikus, folyamatban"). Couple to **R-31**'s async/status-card idiom (same surface likely serves both) and to the **R-39 `consumed_at` honesty gauge** precedent on the PBS side — a consumed secret + N report cycles with no offbox status is the same "disagreement no single tier can see" shape and deserves the same loud event. Supporting live datum: demo-felhom's 07-21 staged secret is still unconsumed today (key-auth-first path never consumes) — invisible for the same reason. |
+| R-71 | **[P1] Day-0 race: the managed floor-update kills the offsite apply-bridge between password-consume and persist — the one-shot credential is burned and the box lands in the silent consume-404 dead-end forever.** Proven on demo-hp (07-21): consume 16:27:42 → managed update 0.153.0→0.156.0 replaces the container 16:28:17, ~35 s later, mid `ssh-copy-id` window; nothing persisted, no installed key ⇒ the key-auth-first recovery path can never engage, and every subsequent start logs the WARN and gives up until an operator Re-issue. **This recurs structurally on every fresh onboarding whose ISO floor lags the managed floor** — the update fires minutes after first boot, exactly when the bridge first runs. demo-felhom escaped by timing only. | M | idea | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` (mechanism cites: bridge order `offsiteapply.go:106–187` consume-then-persist, retry only on process start; the dead-end is even documented in source l.168–173 "the password is spent; reset it on the hub to retry"). Candidate directions, spec-first: (a) **order** — first-boot path lets the managed update settle before the bridge's consume step (cheapest; the race window is the update, not the reboot); (b) **two-phase consume** — hub marks consumed only on a controller ack-after-persist (touches the 404-no-oracle contract, design carefully); (c) **hub-side self-heal** — auto-restage a fresh secret when `consumed_at` is set but K consecutive reports show offsite enabled with no offbox status (the R-39(a) mint-race lesson applies: never restage on top of an UNCONSUMED secret). Pairs with R-70 (visibility) — but visibility alone is not the fix; the burned credential needs an unattended recovery path. |
 | R-53 | **`app_export.html` substituted the CSRF token where the customer domain belongs** - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | **SHIPPED (controller v0.150.0, 2026-07-20)** | One template token (`{{$.CSRFToken}}` -> `{{$.Domain}}`) plus the `Domain` key in `exportPageHandler`'s data map - that handler does not go through `baseData`, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined `<sub>.<domain>` and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F7) |
 
 ## P3 — post-alpha