F10 offsite leg resolved on demo-hp: day-0 update-vs-bridge race diagnosed, designed-path repair + tier round-trip proven; R-70/R-71 minted

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR

documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md

@@ -108,7 +108,7 @@ repo consistent, 14 snapshots, latest 04:16 CEST (pre-outage), **zero locks** (r
 |---|---|---|---|---|
 | **F8** | **Site power loss; no auto-power-on.** Power itself returned within minutes (router rebooted at ~14:42 and stayed up); both miniPCs remained off ~4.5 h until manual power-on. For a paying customer this converts a power blip into a half-day outage ending only when someone is physically present. | HIGH | p1-\*; router uptime operator-attested | **mitigated-on-site** — Viktor set BIOS AC-power-on on both boxes (attested; not OS-verifiable). **roadmap-candidate:** make BIOS "restore on AC power" a provisioning-checklist item + a host-install doc requirement for every fleet box. |
 | **F9** | **H1 OOB belt only partially installed on the current fleet.** `felhom-mgmt-watchdog.timer` is live on both boxes (heal marker absent = no heal needed), but `felhom-sshd.service` and `felhom-oob-nft.service` (TASK H1, present in `felhom-agent/configs/`) are installed on **neither** box; operator access rides stock sshd :22 + tailscale + G1 break-glass. Pre-existing (both boxes provisioned via the universal ISO), surfaced by this audit's access preflight. | MEDIUM | p2-felhom-pve-ssh-belt.txt, p2-\*-host.txt | **needs-ruling** — was H1 intentionally dropped from the universal-ISO provisioning path, or should host-install grow the belt? |
-| **F10** | **demo-hp backup tiers incomplete.** Tier-2 dump tree exists (paperless-ngx, fresh), but **no offbox target is configured** and the PBS DR datastore holds **0 snapshots** for it. A power event with disk damage would have had no off-box recovery path for that guest. Pre-existing (box added 07-21), not outage-caused. | MEDIUM | p6-demo-hp-offbox-locks.txt (`NO-OFFBOX`), p6-demo-hp-backup.txt (`snapshots=0`), p6-demo-hp-tier2.txt | **roadmap-candidate** — finish demo-hp tier provisioning (offbox + DR) as part of the pilot hardening pass. |
+| **F10** | **demo-hp backup tiers incomplete.** Tier-2 dump tree exists (paperless-ngx, fresh), but **no offbox target is configured** and the PBS DR datastore holds **0 snapshots** for it. A power event with disk damage would have had no off-box recovery path for that guest. Pre-existing (box added 07-21), not outage-caused. | MEDIUM | p6-demo-hp-offbox-locks.txt (`NO-OFFBOX`), p6-demo-hp-backup.txt (`snapshots=0`), p6-demo-hp-tier2.txt | **OFFSITE LEG RESOLVED 2026-07-23** — root cause was NOT "provisioning unfinished": the day-0 managed update killed the apply-bridge after password-consume, burning the one-shot credential (full diagnosis, designed-path repair via operator Re-issue, escrow ceremony, and a byte-identical offsite restore round-trip: `DIAG-f10-demo-hp-offsite-2026-07-23.md`; product rows minted **R-70** visibility + **R-71** the race). **The PBS-DR-snapshot half STAYS OPEN** pending F13 (cadence ruling) and the deliberate DR ceremony R-moment on demo-hp. |
 | **F11** | **Recovery is silent.** `host_recovered`/`node_recovered` fired correctly but carry severity `info`, which the dispatcher deliberately does not email — the operator/customer only learns of recovery by looking. During a real customer outage the "it's back" signal is arguably the second-most valuable email. Working-as-coded, so a product question, not a defect. | LOW | p5-hub-events.txt; `hub/internal/notify/dispatcher.go` severityNotifies | **needs-ruling** — opt-in recovery notifications (operator at least)? |
 | **F12** | **demo-hp has no customer notification prefs** (`customer_notifications` has no row for it) → its "customer" received no node_down email and never would. Only demo-felhom is wired (doodoo21@freemail.hu). Pre-existing demo-box config gap; on a real onboarding this must not be skippable. | LOW | p5-customer-notification-prefs.txt | **roadmap-candidate** — make notification-prefs setup a claim/onboarding step, not an optional settings page. |
 | **F13** | **PBS DR tier has no visible backup cadence.** `/etc/pve/jobs.cfg` is empty on both hosts; N100's DR datastore holds a single CT-9201 snapshot from **2026-07-18** (S8-era), demo-hp none. The agent's 6 h *verify* loop runs, but nothing appears to *create* periodic DR snapshots. If DR snapshots are meant to be on-demand only, fine — but then the "DR tier" freshness expectation should be documented; if they're meant to be periodic, the scheduler is missing. | LOW | p6-\*-backup.txt | **needs-ruling** — clarify intended DR-tier cadence; roadmap if periodic. |

documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md

@@ -0,0 +1,136 @@
+# DIAG — demo-hp offsite: "enabled at the hub, absent on the box" (audit F10, offsite leg)
+
+**Date:** 2026-07-23 · **Operator:** Viktor (present; hub click + escrow ceremony) · **Executor:** Claude Code on DooPlex
+**Scope:** felhom.eu hub v0.71.0 (`b67fce39`) + felhom-controller v0.160.0 (`0eba37d5`), live on demo-hp (`demo-hp-bb76ea`, guest 9201).
+**Origin:** `AUDIT-power-outage-recovery-2026-07-22.md` F10 + operator observation 2026-07-23 (hub: "Provisioned: u629488-sub3@…"; box: „Még nincs beállítva távoli mentési cél").
+**Outcome:** root cause proven from both planes' evidence; repaired via the designed path (operator Re-issue, once); **tier proven with a full offsite→loss→restore round-trip, byte-identical**. Two new roadmap rows minted (R-70, R-71). No code changed.
+
+All timestamps UTC unless marked CEST (hub pod log is CEST; hub DB and guest are UTC).
+
+## 1. Phase-0 verdict
+
+Neither spec Shape A ("delivery never consumed") nor Shape B ("customer click pending") — a third,
+fully-evidenced shape:
+
+> **The day-0 managed floor-update (0.153.0 → 0.156.0) killed the offsite apply-bridge between
+> one-time-password consume and key-install/persist.** The password was spent hub-side; the box
+> persisted nothing; every later bridge run hit the documented consume-404 dead-end
+> ("password spent — reset it on the hub to retry"), forever, silently.
+
+Shape B was ruled out **from source**: the Felhom-managed offsite arrives exclusively via the
+automatic apply-bridge (`felhom-controller` `main.go:290–322` — bridge runs at every process start;
+config-version bump → self-restart → bridge). The box button „Távoli mentési cél beállítása" is the
+**BYO NAS/SFTP form only** (`internal/web/offbox_handlers.go:44–126`, key/known_hosts textareas;
+never contacts the hub consume endpoint). No customer click exists in the managed-offsite chain.
+
+## 2. The mechanism (source, cited)
+
+- Hub stages the transient password in `one_time_secrets` (one row per customer; UPSERT resets
+  `consumed_at` — `hub/internal/store/store.go:1218–1227`); the controller consumes it via
+  `POST /api/v1/offsite/consume-password/{customerID}` — a **column flip** `consumed_at=now`
+  in one tx; a second call returns 404 (`store.go:1232–1250`, `api/offsite.go:13–31`).
+- The descriptor (non-secret) rides `controller.yaml`; the ACK carries only `config_version`.
+- Bridge order (`controller/internal/offsiteapply/offsiteapply.go:106–187`): marker check →
+  host-key fingerprint verify (before consume, "don't waste it on a mismatch") → key-auth-first
+  probe (re-pin WITHOUT consuming if an installed key works) → keygen → **consume** (l.159–163) →
+  `ssh-copy-id` install + verify (l.168–173) → persist `offbox` target `EscrowState=pending`
+  (l.176) → marker LAST (l.181). **Consume-then-persist; retry only on next process start.**
+- The exact dead-end is documented in source (l.168–173): *"key install FAILED after consuming the
+  one-time password … the password is spent; reset it on the hub to retry"* — the designed recovery
+  is the hub's **Re-issue offsite credentials** (`hub/internal/web/configs.go:730–773`: Hetzner
+  password reset → fresh one-time secret → ConfigVersion bump → `bumpIntent` long-poll wake; if an
+  escrow blob exists it is marked stale — demo-hp had none).
+- **The hub UI cannot see any of this:** the "Provisioned: … delivered to the controller once" line
+  is static copy gated only on `offsite.host` in ConfigJSON
+  (`hub/internal/web/templates/config_form_body.html:119–120`); nothing in the web layer reads
+  `one_time_secrets.consumed_at`. "Provisioned" ≠ "delivered" ≠ "applied" — the hub shows the first
+  and knows neither of the others. (→ R-70; note the R-39 precedent: PBS-DR grew a
+  `consumed_at` honesty gauge for exactly this disagreement class.)
+
+## 3. The 07-21 timeline (evidence: hub DB `events`/`one_time_secrets`/`customer_claims`/`reports`)
+
+| UTC | Event |
+|---|---|
+| 16:23:32 | customer self-bind (`appliance_bound`) |
+| 16:23:55 | appliance credential delivered — box begins day-0 |
+| 16:24:19 | host row `demo-hp-bb76ea` created |
+| 16:24:27 | hub auto-provisions offsite on enrollment (`offsite_reissued`, source=hub): sub3 minted, one-time pw staged, config_version→2 |
+| **16:27:42** | **one-time password CONSUMED** (bridge inside the 0.153.0 floor controller) |
+| 16:28:07 | `controller_started (0.153.0)`; first report 16:28:08 |
+| **16:28:17** | **managed update 0.153.0→0.156.0 — container replaced, bridge killed mid-flight** (~35 s after consume, inside the install/persist window) |
+| 16:28:22 | 0.156.0 starts; bridge: no installed key → consume → **404** → nothing persisted |
+
+Corroboration: **153 controller reports** 07-21 16:28 → 07-23 09:31, **zero** ever carrying an
+offbox status object; box settings.json had **no `offbox` key** (checked with the real key, per the
+R-7b lesson) and **no `offbox/` dir** (no ssh_key → the key-install never completed, so the
+key-auth-first recovery path could never engage); live log at the 0.160.0 start 07-22 17:16:35:
+`[WARN] [offsite-apply] reconcile: … consume one-time password: no unconsumed offsite password
+(already consumed or none provisioned)`.
+
+Spec note: this is literally the spec's "delivery consumed but persist failed → STOP" class, but
+the evidence showed the mechanism **not broken** (fail-safe held: no half-configured offbox) with a
+source-designated recovery identical to Shape A's remedy; the operator ruled: proceed (recorded in
+session, 2026-07-23).
+
+## 4. Repair via the designed path (operator click, once — R-31 discipline)
+
+| UTC 07-23 | Event |
+|---|---|
+| 09:53:37 | Viktor clicks **Re-issue offsite credentials** ONCE — `[offsite] re-issued shared credentials for demo-hp (subaccount 275124)`; fresh pw staged; ConfigVersion bumped |
+| 09:53:39 | box long-poll wakes, config re-pulled |
+| **09:53:41** | fresh password **consumed** (4 s after staging) |
+| 09:53:45 | `[offsite-apply] offsite configured for u629488-sub3@…:/home/felhom-repo (pending key escrow)`; `offbox/` dir complete (ssh_key/known_hosts/repo_password/applied_marker), settings `offbox` present |
+
+Click → converged in **8 seconds**. Escrow ceremony run by the operator through the real
+`/backup/escrow` wizard: blob stored 10:01:17 (`posture=zero_knowledge`, `restic_pw_sha256`
+recorded), R claimed one-shot 10:01:21, hub-verified auto-confirm 10:01:24 →
+`EscrowState=escrowed; offsite runs enabled`.
+
+## 5. Tier proof (F10 closure bar: restore round-trip)
+
+Per-app scope: `paperless-ngx` toggled into offsite via the real `POST /backup/offbox/toggle`
+(per-app default is OFF). Mandatory capture path per catalog: `appdata/paperless/media`.
+All steps via real endpoints from inside guest 9201 (endpoint-level method — no browser on DooPlex);
+probe placement/deletion on the filesystem simulates customer data and its loss; every
+backup/restore action went through the UI's own endpoints.
+
+1. **Probe** 64 KiB urandom → `…/appdata/paperless/media/felhom-probe-f10-20260723.bin`,
+   md5 `9120e65d6a9f071072d827fc404dc840`.
+2. **First offsite run** (`POST /backup/offbox/run`): repo **initialized fresh** on sub3 (proves the
+   target was empty), 79.8 MB / 49 files, 1m19s, status `ok`, snapshots=1.
+3. **Loss**: probe deleted (ENOENT verified).
+4. **Restore** (`POST /backup/offbox/restore`, `mode=full`, two-step size gate revealed 79.8 MB →
+   `confirm=1`): snapshot **`2bf7f2e1`** → staging `backups/offsite-restore/paperless-ngx`;
+   staging probe **md5-identical**.
+5. **Place** (`POST /backup/offbox/place`): `1 file(s) merged (missing-only)`, live recovery unit
+   not overwritten, scratch auto-removed; live probe **md5-identical**
+   (`9120e65d6a9f071072d827fc404dc840`).
+6. **Cleanup**: probe removed; second run (2m17s, `ok`) so the latest snapshot is probe-free;
+   retention pruned the probe-bearing snapshot (count stays 1). Guest session artifacts removed;
+   break-glass + DB copies shredded on DooPlex.
+
+Hub-side final state: latest demo-hp report carries
+`offsite: {enabled, escrow_state:"escrowed", quota_gb:50}`; `host_escrow` row present
+(zero_knowledge, hash recorded, not stale); `one_time_secrets` consumed 09:53:41. The Offsite tab
+now reflects the customer. Nightly offsite is scheduled (daily 04:15 guest-UTC).
+
+## 6. Findings minted
+
+- **R-70 (P2-HIGH)** — the pending/failed offsite last mile is invisible on BOTH surfaces; the hub
+  cannot distinguish staged/consumed/applied. Couple to R-31's async/status-card idiom and the
+  R-39 `consumed_at` honesty-gauge precedent.
+- **R-71 (P1)** — the day-0 race itself: managed floor-update vs apply-bridge, consume-then-persist
+  not crash-safe. **This recurs on every fresh onboarding** whose ISO floor lags the managed floor
+  (the update fires minutes after first boot, exactly when the bridge runs). demo-felhom escaped by
+  timing only.
+- Audit F10 row annotated: **offsite leg RESOLVED** (this record); the PBS-DR-snapshot half of F10
+  **stays open** pending F13 (cadence ruling) and the deliberate DR ceremony R-moment on demo-hp.
+
+## 7. Observed, not acted on
+
+- `one_time_secrets` row for **demo-felhom** staged 2026-07-21 08:29:29 is still **unconsumed** —
+  residue of the mistaken offsite-Re-issue click during the R-39 session (the box recovered via the
+  key-auth-first path, which never consumes). Harmless (superseded on any future re-issue), but it
+  is exactly the state the R-39 honesty gauge flags on the PBS side — supporting R-70.
+- The three dead unclaimed-appliance records from the 07-22 ISO train remain at the hub for
+  operator discard (unchanged).

documentation/backlog/ROADMAP.md

@@ -81,6 +81,8 @@
 | R-67 | **The NAS share appears in FileBrowser — browse what you mounted.** | S | **SHIPPED (controller v0.160.0, 2026-07-22)** | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). **Couples to R-64: browsing was its missing UX half.** A registered network storage now binds its share ROOT into FileBrowser (`/mnt/felhom-drives/<name>:/srv/<name>:rslave`) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (`Host is down` in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: `felhom-controller/REPORT.md` (2026-07-22) |
 | R-68 | **Notification train: paired recovery mails + prefs seeding at claim + priority headers (power-outage audit F11+F12+F14-light).** The dead-man's-switch fired perfectly on 07-22 and the customer who got „A szerver nem elérhető!" was never told it recovered (F11); a customer without a `customer_notifications` row is silently unnotifiable (F12, demo-hp live); delivered ≠ noticed (F14). | M | **SHIPPED (hub v0.71.0, 2026-07-22)** | Origin: `AUDIT-power-outage-recovery-2026-07-22.md`. Recovery = explicit eventType branch (severity semantics frozen; `severityNotifies` untouched): operator always hears both edges, customer iff PAIRED (customer-channel `sent` stale/down row newer than the last sent recovery — `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for recovery; ties → no mail, flap-safe). Seed-at-claim: `MarkClaimed` → `SeedNotificationPrefs` (INSERT-if-absent, never upsert — red-proofed; empty email no-op; never fails the claim; default critical-only set). Hub-side empty-email no-clobber belt in `handleSavePreferences` (controller 0.160.0 already guards its own two push legs — latent, not live). `X-Priority: 1` + `Importance: high` on error/critical via Resend `headers` (live-probed HTTP 200 before implementation); the `test` event now also mails the operator with those headers (one click proves both channels + rendering); latent `sendTestEmail` nil-prefs panic fixed. 17 tests + 4 red-proofs. **Live legs pending:** natural `*_recovered` mail on the next real staleness cycle (or the reboot-drill arc — NEVER fabricated by blocking reports, that is F9-bypass-shaped) and seed-at-claim on a real claim (Peti Friday reinstall is the natural candidate) |
 | R-69 | **F14-full: an operator push channel that actually interrupts (ntfy / Telegram / similar), beyond mail-client priority flags.** F14-light (v0.71.0 headers + Gmail filter) nudges a mail client; a 15:29 node_down should reach the operator's pocket in seconds regardless of inbox hygiene. Needs: channel choice (self-hosted ntfy on k3s vs Telegram bot), dispatcher fan-out seam, per-severity routing, quiet hours. | M | idea | Origin: `AUDIT-power-outage-recovery-2026-07-22.md` F14. Deliberately NOT built in the v0.71.0 train (scope-forked per the task spec) |
+| R-70 | **[P2-HIGH] The offsite last mile is invisible on BOTH surfaces — the hub cannot tell "staged" from "delivered" from "applied".** demo-hp sat 2 days with the hub customer page saying "Provisioned: … the transient password is delivered to the controller once" while the box said „Még nincs beállítva távoli mentési cél" — and a real customer would sit unprotected indefinitely believing otherwise. The hub HAS the signal (`one_time_secrets.consumed_at` + 153 consecutive reports carrying no offbox object) and reads none of it: the "Provisioned" line is static copy gated only on `offsite.host` in ConfigJSON (`config_form_body.html:119–120`). | S–M | idea | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. Two legs: **hub customer card** shows the real delivery state ("provisioned, awaiting box consumption" / "consumed, awaiting apply" / "applied" — consumed_at × report-offbox-presence is enough for all three), and **controller banner** when the descriptor is enabled but no target is configured („Felhom offsite készen áll — a beállítás automatikus, folyamatban"). Couple to **R-31**'s async/status-card idiom (same surface likely serves both) and to the **R-39 `consumed_at` honesty gauge** precedent on the PBS side — a consumed secret + N report cycles with no offbox status is the same "disagreement no single tier can see" shape and deserves the same loud event. Supporting live datum: demo-felhom's 07-21 staged secret is still unconsumed today (key-auth-first path never consumes) — invisible for the same reason. |
+| R-71 | **[P1] Day-0 race: the managed floor-update kills the offsite apply-bridge between password-consume and persist — the one-shot credential is burned and the box lands in the silent consume-404 dead-end forever.** Proven on demo-hp (07-21): consume 16:27:42 → managed update 0.153.0→0.156.0 replaces the container 16:28:17, ~35 s later, mid `ssh-copy-id` window; nothing persisted, no installed key ⇒ the key-auth-first recovery path can never engage, and every subsequent start logs the WARN and gives up until an operator Re-issue. **This recurs structurally on every fresh onboarding whose ISO floor lags the managed floor** — the update fires minutes after first boot, exactly when the bridge first runs. demo-felhom escaped by timing only. | M | idea | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` (mechanism cites: bridge order `offsiteapply.go:106–187` consume-then-persist, retry only on process start; the dead-end is even documented in source l.168–173 "the password is spent; reset it on the hub to retry"). Candidate directions, spec-first: (a) **order** — first-boot path lets the managed update settle before the bridge's consume step (cheapest; the race window is the update, not the reboot); (b) **two-phase consume** — hub marks consumed only on a controller ack-after-persist (touches the 404-no-oracle contract, design carefully); (c) **hub-side self-heal** — auto-restage a fresh secret when `consumed_at` is set but K consecutive reports show offsite enabled with no offbox status (the R-39(a) mint-race lesson applies: never restage on top of an UNCONSUMED secret). Pairs with R-70 (visibility) — but visibility alone is not the fix; the burned credential needs an unattended recovery path. |
 | R-53 | **`app_export.html` substituted the CSRF token where the customer domain belongs** - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | **SHIPPED (controller v0.150.0, 2026-07-20)** | One template token (`{{$.CSRFToken}}` -> `{{$.Domain}}`) plus the `Domain` key in `exportPageHandler`'s data map - that handler does not go through `baseData`, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined `<sub>.<domain>` and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F7) |
 
 ## P3 — post-alpha