felhom.eu/documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md

DIAG — demo-hp offsite: "enabled at the hub, absent on the box" (audit F10, offsite leg)

Date: 2026-07-23 · Operator: Viktor (present; hub click + escrow ceremony) · Executor: Claude Code on DooPlex Scope: felhom.eu hub v0.71.0 (b67fce39) + felhom-controller v0.160.0 (0eba37d5), live on demo-hp (demo-hp-bb76ea, guest 9201). Origin: AUDIT-power-outage-recovery-2026-07-22.md F10 + operator observation 2026-07-23 (hub: "Provisioned: u629488-sub3@…"; box: „Még nincs beállítva távoli mentési cél"). Outcome: root cause proven from both planes' evidence; repaired via the designed path (operator Re-issue, once); tier proven with a full offsite→loss→restore round-trip, byte-identical. Two new roadmap rows minted (R-70, R-71). No code changed.

All timestamps UTC unless marked CEST (hub pod log is CEST; hub DB and guest are UTC).

1. Phase-0 verdict

Neither spec Shape A ("delivery never consumed") nor Shape B ("customer click pending") — a third, fully-evidenced shape:

The day-0 managed floor-update (0.153.0 → 0.156.0) killed the offsite apply-bridge between one-time-password consume and key-install/persist. The password was spent hub-side; the box persisted nothing; every later bridge run hit the documented consume-404 dead-end ("password spent — reset it on the hub to retry"), forever, silently.

Shape B was ruled out from source: the Felhom-managed offsite arrives exclusively via the automatic apply-bridge (felhom-controller main.go:290–322 — bridge runs at every process start; config-version bump → self-restart → bridge). The box button „Távoli mentési cél beállítása" is the BYO NAS/SFTP form only (internal/web/offbox_handlers.go:44–126, key/known_hosts textareas; never contacts the hub consume endpoint). No customer click exists in the managed-offsite chain.

2. The mechanism (source, cited)

• Hub stages the transient password in one_time_secrets (one row per customer; UPSERT resets consumed_at — hub/internal/store/store.go:1218–1227); the controller consumes it via POST /api/v1/offsite/consume-password/{customerID} — a column flip consumed_at=now in one tx; a second call returns 404 (store.go:1232–1250, api/offsite.go:13–31).
• The descriptor (non-secret) rides controller.yaml; the ACK carries only config_version.
• Bridge order (controller/internal/offsiteapply/offsiteapply.go:106–187): marker check → host-key fingerprint verify (before consume, "don't waste it on a mismatch") → key-auth-first probe (re-pin WITHOUT consuming if an installed key works) → keygen → consume (l.159–163) → ssh-copy-id install + verify (l.168–173) → persist offbox target EscrowState=pending (l.176) → marker LAST (l.181). Consume-then-persist; retry only on next process start.
• The exact dead-end is documented in source (l.168–173): "key install FAILED after consuming the one-time password … the password is spent; reset it on the hub to retry" — the designed recovery is the hub's Re-issue offsite credentials (hub/internal/web/configs.go:730–773: Hetzner password reset → fresh one-time secret → ConfigVersion bump → bumpIntent long-poll wake; if an escrow blob exists it is marked stale — demo-hp had none).
• The hub UI cannot see any of this: the "Provisioned: … delivered to the controller once" line is static copy gated only on offsite.host in ConfigJSON (hub/internal/web/templates/config_form_body.html:119–120); nothing in the web layer reads one_time_secrets.consumed_at. "Provisioned" ≠ "delivered" ≠ "applied" — the hub shows the first and knows neither of the others. (→ R-70; note the R-39 precedent: PBS-DR grew a consumed_at honesty gauge for exactly this disagreement class.)

3. The 07-21 timeline (evidence: hub DB events/one_time_secrets/customer_claims/reports)

UTC	Event
16:23:32	customer self-bind (appliance_bound)
16:23:55	appliance credential delivered — box begins day-0
16:24:19	host row demo-hp-bb76ea created
16:24:27	hub auto-provisions offsite on enrollment (offsite_reissued, source=hub): sub3 minted, one-time pw staged, config_version→2
16:27:42	one-time password CONSUMED (bridge inside the 0.153.0 floor controller)
16:28:07	controller_started (0.153.0); first report 16:28:08
16:28:17	managed update 0.153.0→0.156.0 — container replaced, bridge killed mid-flight (~35 s after consume, inside the install/persist window)
16:28:22	0.156.0 starts; bridge: no installed key → consume → 404 → nothing persisted

Corroboration: 153 controller reports 07-21 16:28 → 07-23 09:31, zero ever carrying an offbox status object; box settings.json had no offbox key (checked with the real key, per the R-7b lesson) and no offbox/ dir (no ssh_key → the key-install never completed, so the key-auth-first recovery path could never engage); live log at the 0.160.0 start 07-22 17:16:35: [WARN] [offsite-apply] reconcile: … consume one-time password: no unconsumed offsite password (already consumed or none provisioned).

Spec note: this is literally the spec's "delivery consumed but persist failed → STOP" class, but the evidence showed the mechanism not broken (fail-safe held: no half-configured offbox) with a source-designated recovery identical to Shape A's remedy; the operator ruled: proceed (recorded in session, 2026-07-23).

4. Repair via the designed path (operator click, once — R-31 discipline)

UTC 07-23	Event
09:53:37	Viktor clicks Re-issue offsite credentials ONCE — [offsite] re-issued shared credentials for demo-hp (subaccount 275124); fresh pw staged; ConfigVersion bumped
09:53:39	box long-poll wakes, config re-pulled
09:53:41	fresh password consumed (4 s after staging)
09:53:45	[offsite-apply] offsite configured for u629488-sub3@…:/home/felhom-repo (pending key escrow); offbox/ dir complete (ssh_key/known_hosts/repo_password/applied_marker), settings offbox present

Click → converged in 8 seconds. Escrow ceremony run by the operator through the real /backup/escrow wizard: blob stored 10:01:17 (posture=zero_knowledge, restic_pw_sha256 recorded), R claimed one-shot 10:01:21, hub-verified auto-confirm 10:01:24 → EscrowState=escrowed; offsite runs enabled.

5. Tier proof (F10 closure bar: restore round-trip)

Per-app scope: paperless-ngx toggled into offsite via the real POST /backup/offbox/toggle (per-app default is OFF). Mandatory capture path per catalog: appdata/paperless/media. All steps via real endpoints from inside guest 9201 (endpoint-level method — no browser on DooPlex); probe placement/deletion on the filesystem simulates customer data and its loss; every backup/restore action went through the UI's own endpoints.

1. Probe 64 KiB urandom → …/appdata/paperless/media/felhom-probe-f10-20260723.bin, md5 9120e65d6a9f071072d827fc404dc840.
2. First offsite run (POST /backup/offbox/run): repo initialized fresh on sub3 (proves the target was empty), 79.8 MB / 49 files, 1m19s, status ok, snapshots=1.
3. Loss: probe deleted (ENOENT verified).
4. Restore (POST /backup/offbox/restore, mode=full, two-step size gate revealed 79.8 MB → confirm=1): snapshot 2bf7f2e1 → staging backups/offsite-restore/paperless-ngx; staging probe md5-identical.
5. Place (POST /backup/offbox/place): 1 file(s) merged (missing-only), live recovery unit not overwritten, scratch auto-removed; live probe md5-identical (9120e65d6a9f071072d827fc404dc840).
6. Cleanup: probe removed; second run (2m17s, ok) so the latest snapshot is probe-free; retention pruned the probe-bearing snapshot (count stays 1). Guest session artifacts removed; break-glass + DB copies shredded on DooPlex.

Hub-side final state: latest demo-hp report carries offsite: {enabled, escrow_state:"escrowed", quota_gb:50}; host_escrow row present (zero_knowledge, hash recorded, not stale); one_time_secrets consumed 09:53:41. The Offsite tab now reflects the customer. Nightly offsite is scheduled (daily 04:15 guest-UTC).

6. Findings minted

• R-70 (P2-HIGH) — the pending/failed offsite last mile is invisible on BOTH surfaces; the hub cannot distinguish staged/consumed/applied. Couple to R-31's async/status-card idiom and the R-39 consumed_at honesty-gauge precedent.
• R-71 (P1) — the day-0 race itself: managed floor-update vs apply-bridge, consume-then-persist not crash-safe. This recurs on every fresh onboarding whose ISO floor lags the managed floor (the update fires minutes after first boot, exactly when the bridge runs). demo-felhom escaped by timing only.
• Audit F10 row annotated: offsite leg RESOLVED (this record); the PBS-DR-snapshot half of F10 stays open pending F13 (cadence ruling) and the deliberate DR ceremony R-moment on demo-hp.

7. Observed, not acted on

• one_time_secrets row for demo-felhom staged 2026-07-21 08:29:29 is still unconsumed — residue of the mistaken offsite-Re-issue click during the R-39 session (the box recovered via the key-auth-first path, which never consumes). Harmless (superseded on any future re-issue), but it is exactly the state the R-39 honesty gauge flags on the PBS side — supporting R-70.
• The three dead unclaimed-appliance records from the 07-22 ISO train remain at the hub for operator discard (unchanged).