felhom.eu/documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md

# DIAG — demo-hp offsite: "enabled at the hub, absent on the box" (audit F10, offsite leg)

**Date:** 2026-07-23 · **Operator:** Viktor (present; hub click + escrow ceremony) · **Executor:** Claude Code on DooPlex
**Scope:** felhom.eu hub v0.71.0 (`b67fce39`) + felhom-controller v0.160.0 (`0eba37d5`), live on demo-hp (`demo-hp-bb76ea`, guest 9201).
**Origin:** `AUDIT-power-outage-recovery-2026-07-22.md` F10 + operator observation 2026-07-23 (hub: "Provisioned: u629488-sub3@…"; box: „Még nincs beállítva távoli mentési cél").
**Outcome:** root cause proven from both planes' evidence; repaired via the designed path (operator Re-issue, once); **tier proven with a full offsite→loss→restore round-trip, byte-identical**. Two new roadmap rows minted (R-70, R-71). No code changed.

All timestamps UTC unless marked CEST (hub pod log is CEST; hub DB and guest are UTC).

## 1. Phase-0 verdict

Neither spec Shape A ("delivery never consumed") nor Shape B ("customer click pending") — a third,
fully-evidenced shape:

> **The day-0 managed floor-update (0.153.0 → 0.156.0) killed the offsite apply-bridge between
> one-time-password consume and key-install/persist.** The password was spent hub-side; the box
> persisted nothing; every later bridge run hit the documented consume-404 dead-end
> ("password spent — reset it on the hub to retry"), forever, silently.

Shape B was ruled out **from source**: the Felhom-managed offsite arrives exclusively via the
automatic apply-bridge (`felhom-controller` `main.go:290–322` — bridge runs at every process start;
config-version bump → self-restart → bridge). The box button „Távoli mentési cél beállítása" is the
**BYO NAS/SFTP form only** (`internal/web/offbox_handlers.go:44–126`, key/known_hosts textareas;
never contacts the hub consume endpoint). No customer click exists in the managed-offsite chain.

## 2. The mechanism (source, cited)

- Hub stages the transient password in `one_time_secrets` (one row per customer; UPSERT resets
  `consumed_at` — `hub/internal/store/store.go:1218–1227`); the controller consumes it via
  `POST /api/v1/offsite/consume-password/{customerID}` — a **column flip** `consumed_at=now`
  in one tx; a second call returns 404 (`store.go:1232–1250`, `api/offsite.go:13–31`).
- The descriptor (non-secret) rides `controller.yaml`; the ACK carries only `config_version`.
- Bridge order (`controller/internal/offsiteapply/offsiteapply.go:106–187`): marker check →
  host-key fingerprint verify (before consume, "don't waste it on a mismatch") → key-auth-first
  probe (re-pin WITHOUT consuming if an installed key works) → keygen → **consume** (l.159–163) →
  `ssh-copy-id` install + verify (l.168–173) → persist `offbox` target `EscrowState=pending`
  (l.176) → marker LAST (l.181). **Consume-then-persist; retry only on next process start.**
- The exact dead-end is documented in source (l.168–173): *"key install FAILED after consuming the
  one-time password … the password is spent; reset it on the hub to retry"* — the designed recovery
  is the hub's **Re-issue offsite credentials** (`hub/internal/web/configs.go:730–773`: Hetzner
  password reset → fresh one-time secret → ConfigVersion bump → `bumpIntent` long-poll wake; if an
  escrow blob exists it is marked stale — demo-hp had none).
- **The hub UI cannot see any of this:** the "Provisioned: … delivered to the controller once" line
  is static copy gated only on `offsite.host` in ConfigJSON
  (`hub/internal/web/templates/config_form_body.html:119–120`); nothing in the web layer reads
  `one_time_secrets.consumed_at`. "Provisioned" ≠ "delivered" ≠ "applied" — the hub shows the first
  and knows neither of the others. (→ R-70; note the R-39 precedent: PBS-DR grew a
  `consumed_at` honesty gauge for exactly this disagreement class.)

## 3. The 07-21 timeline (evidence: hub DB `events`/`one_time_secrets`/`customer_claims`/`reports`)

| UTC | Event |
|---|---|
| 16:23:32 | customer self-bind (`appliance_bound`) |
| 16:23:55 | appliance credential delivered — box begins day-0 |
| 16:24:19 | host row `demo-hp-bb76ea` created |
| 16:24:27 | hub auto-provisions offsite on enrollment (`offsite_reissued`, source=hub): sub3 minted, one-time pw staged, config_version→2 |
| **16:27:42** | **one-time password CONSUMED** (bridge inside the 0.153.0 floor controller) |
| 16:28:07 | `controller_started (0.153.0)`; first report 16:28:08 |
| **16:28:17** | **managed update 0.153.0→0.156.0 — container replaced, bridge killed mid-flight** (~35 s after consume, inside the install/persist window) |
| 16:28:22 | 0.156.0 starts; bridge: no installed key → consume → **404** → nothing persisted |

Corroboration: **153 controller reports** 07-21 16:28 → 07-23 09:31, **zero** ever carrying an
offbox status object; box settings.json had **no `offbox` key** (checked with the real key, per the
R-7b lesson) and **no `offbox/` dir** (no ssh_key → the key-install never completed, so the
key-auth-first recovery path could never engage); live log at the 0.160.0 start 07-22 17:16:35:
`[WARN] [offsite-apply] reconcile: … consume one-time password: no unconsumed offsite password
(already consumed or none provisioned)`.

Spec note: this is literally the spec's "delivery consumed but persist failed → STOP" class, but
the evidence showed the mechanism **not broken** (fail-safe held: no half-configured offbox) with a
source-designated recovery identical to Shape A's remedy; the operator ruled: proceed (recorded in
session, 2026-07-23).

## 4. Repair via the designed path (operator click, once — R-31 discipline)

| UTC 07-23 | Event |
|---|---|
| 09:53:37 | Viktor clicks **Re-issue offsite credentials** ONCE — `[offsite] re-issued shared credentials for demo-hp (subaccount 275124)`; fresh pw staged; ConfigVersion bumped |
| 09:53:39 | box long-poll wakes, config re-pulled |
| **09:53:41** | fresh password **consumed** (4 s after staging) |
| 09:53:45 | `[offsite-apply] offsite configured for u629488-sub3@…:/home/felhom-repo (pending key escrow)`; `offbox/` dir complete (ssh_key/known_hosts/repo_password/applied_marker), settings `offbox` present |

Click → converged in **8 seconds**. Escrow ceremony run by the operator through the real
`/backup/escrow` wizard: blob stored 10:01:17 (`posture=zero_knowledge`, `restic_pw_sha256`
recorded), R claimed one-shot 10:01:21, hub-verified auto-confirm 10:01:24 →
`EscrowState=escrowed; offsite runs enabled`.

## 5. Tier proof (F10 closure bar: restore round-trip)

Per-app scope: `paperless-ngx` toggled into offsite via the real `POST /backup/offbox/toggle`
(per-app default is OFF). Mandatory capture path per catalog: `appdata/paperless/media`.
All steps via real endpoints from inside guest 9201 (endpoint-level method — no browser on DooPlex);
probe placement/deletion on the filesystem simulates customer data and its loss; every
backup/restore action went through the UI's own endpoints.

1. **Probe** 64 KiB urandom → `…/appdata/paperless/media/felhom-probe-f10-20260723.bin`,
   md5 `9120e65d6a9f071072d827fc404dc840`.
2. **First offsite run** (`POST /backup/offbox/run`): repo **initialized fresh** on sub3 (proves the
   target was empty), 79.8 MB / 49 files, 1m19s, status `ok`, snapshots=1.
3. **Loss**: probe deleted (ENOENT verified).
4. **Restore** (`POST /backup/offbox/restore`, `mode=full`, two-step size gate revealed 79.8 MB →
   `confirm=1`): snapshot **`2bf7f2e1`** → staging `backups/offsite-restore/paperless-ngx`;
   staging probe **md5-identical**.
5. **Place** (`POST /backup/offbox/place`): `1 file(s) merged (missing-only)`, live recovery unit
   not overwritten, scratch auto-removed; live probe **md5-identical**
   (`9120e65d6a9f071072d827fc404dc840`).
6. **Cleanup**: probe removed; second run (2m17s, `ok`) so the latest snapshot is probe-free;
   retention pruned the probe-bearing snapshot (count stays 1). Guest session artifacts removed;
   break-glass + DB copies shredded on DooPlex.

Hub-side final state: latest demo-hp report carries
`offsite: {enabled, escrow_state:"escrowed", quota_gb:50}`; `host_escrow` row present
(zero_knowledge, hash recorded, not stale); `one_time_secrets` consumed 09:53:41. The Offsite tab
now reflects the customer. Nightly offsite is scheduled (daily 04:15 guest-UTC).

## 6. Findings minted

- **R-70 (P2-HIGH)** — the pending/failed offsite last mile is invisible on BOTH surfaces; the hub
  cannot distinguish staged/consumed/applied. Couple to R-31's async/status-card idiom and the
  R-39 `consumed_at` honesty-gauge precedent.
  **→ SHIPPED 2026-07-23 (hub v0.72.0 + controller v0.161.0):** detector `offsite.DeliveryStateFor`,
  operator-card state line (static copy deleted), `offsite_delivery_stuck` warning, controller
  truthful empty-state banner. See `felhom.eu/REPORT.md` (2026-07-23) + hub CHANGELOG v0.72.0.
- **R-71 (P1)** — the day-0 race itself: managed floor-update vs apply-bridge, consume-then-persist
  not crash-safe. **This recurs on every fresh onboarding** whose ISO floor lags the managed floor
  (the update fires minutes after first boot, exactly when the bridge runs). demo-felhom escaped by
  timing only.
  **→ PARTIAL 2026-07-23: (c) self-heal restage SHIPPED in hub v0.72.0** (R-39(a)-guarded, one
  restage/customer/24 h, every firing a warning event; unit-proven + red-proofed, NOT live-fired —
  arms on the next natural occurrence). (a) day-0 ordering stays open — its own spec.
  **→ SHIPPED 2026-07-24: (a) the apply-bridge settle-gate (controller v0.162.0).** `AwaitSettle`
  defers the one-time-password consume while a managed update is running OR the box is below the
  known operator floor (the auto-floor update is imminent), reading the updater's OWN
  `GetFloor()`/`IsUpdateRunning()` via a new `SettleProvider` seam (no second floor path). At/above
  floor it GOes on the first poll with zero added latency (B′). Bounds 10 s / 90 s floor sub-bound /
  5 min overall — both GO+WARN on the argument that **a hub which cannot serve a floor cannot serve a
  consume, so proceeding on floor-unknown carries no burn risk** (R-71c is the belt if a genuinely
  stuck update later burns one). Ordering-only: the consume/persist order and the 404-no-oracle
  contract are untouched — **(b) two-phase consume is REJECTED-BY-DESIGN.** Finding: the floor is
  in-memory (report-ACK ~5–10 s), not persisted → unknown on any restart until the first ACK; the
  90 s sub-bound is sized to that. 5 scenarios + 4 red-proofs (all observed FAIL then restored);
  deferral paths NOT live-fired — the v1.25.0 golden≥floor build gate now structurally PREVENTS the
  precondition (the point). **R-71 → SHIPPED (a)+(c). Layering: build gate prevents, (a) defers,
  (c) heals.** See `felhom-controller/REPORT.md` (2026-07-24) + controller CHANGELOG v0.162.0.
- Audit F10 row annotated: **offsite leg RESOLVED** (this record); the PBS-DR-snapshot half of F10
  **stays open** pending F13 (cadence ruling) and the deliberate DR ceremony R-moment on demo-hp.

## 7. Observed, not acted on

- `one_time_secrets` row for **demo-felhom** staged 2026-07-21 08:29:29 is still **unconsumed** —
  residue of the mistaken offsite-Re-issue click during the R-39 session (the box recovered via the
  key-auth-first path, which never consumes). Harmless (superseded on any future re-issue), but it
  is exactly the state the R-39 honesty gauge flags on the PBS side — supporting R-70.
- The three dead unclaimed-appliance records from the 07-22 ISO train remain at the hub for
  operator discard (unchanged).