F10 offsite leg resolved on demo-hp: day-0 update-vs-bridge race diagnosed, designed-path repair + tier round-trip proven; R-70/R-71 minted
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
This commit is contained in:
@@ -0,0 +1,136 @@
|
||||
# DIAG — demo-hp offsite: "enabled at the hub, absent on the box" (audit F10, offsite leg)
|
||||
|
||||
**Date:** 2026-07-23 · **Operator:** Viktor (present; hub click + escrow ceremony) · **Executor:** Claude Code on DooPlex
|
||||
**Scope:** felhom.eu hub v0.71.0 (`b67fce39`) + felhom-controller v0.160.0 (`0eba37d5`), live on demo-hp (`demo-hp-bb76ea`, guest 9201).
|
||||
**Origin:** `AUDIT-power-outage-recovery-2026-07-22.md` F10 + operator observation 2026-07-23 (hub: "Provisioned: u629488-sub3@…"; box: „Még nincs beállítva távoli mentési cél").
|
||||
**Outcome:** root cause proven from both planes' evidence; repaired via the designed path (operator Re-issue, once); **tier proven with a full offsite→loss→restore round-trip, byte-identical**. Two new roadmap rows minted (R-70, R-71). No code changed.
|
||||
|
||||
All timestamps UTC unless marked CEST (hub pod log is CEST; hub DB and guest are UTC).
|
||||
|
||||
## 1. Phase-0 verdict
|
||||
|
||||
Neither spec Shape A ("delivery never consumed") nor Shape B ("customer click pending") — a third,
|
||||
fully-evidenced shape:
|
||||
|
||||
> **The day-0 managed floor-update (0.153.0 → 0.156.0) killed the offsite apply-bridge between
|
||||
> one-time-password consume and key-install/persist.** The password was spent hub-side; the box
|
||||
> persisted nothing; every later bridge run hit the documented consume-404 dead-end
|
||||
> ("password spent — reset it on the hub to retry"), forever, silently.
|
||||
|
||||
Shape B was ruled out **from source**: the Felhom-managed offsite arrives exclusively via the
|
||||
automatic apply-bridge (`felhom-controller` `main.go:290–322` — bridge runs at every process start;
|
||||
config-version bump → self-restart → bridge). The box button „Távoli mentési cél beállítása" is the
|
||||
**BYO NAS/SFTP form only** (`internal/web/offbox_handlers.go:44–126`, key/known_hosts textareas;
|
||||
never contacts the hub consume endpoint). No customer click exists in the managed-offsite chain.
|
||||
|
||||
## 2. The mechanism (source, cited)
|
||||
|
||||
- Hub stages the transient password in `one_time_secrets` (one row per customer; UPSERT resets
|
||||
`consumed_at` — `hub/internal/store/store.go:1218–1227`); the controller consumes it via
|
||||
`POST /api/v1/offsite/consume-password/{customerID}` — a **column flip** `consumed_at=now`
|
||||
in one tx; a second call returns 404 (`store.go:1232–1250`, `api/offsite.go:13–31`).
|
||||
- The descriptor (non-secret) rides `controller.yaml`; the ACK carries only `config_version`.
|
||||
- Bridge order (`controller/internal/offsiteapply/offsiteapply.go:106–187`): marker check →
|
||||
host-key fingerprint verify (before consume, "don't waste it on a mismatch") → key-auth-first
|
||||
probe (re-pin WITHOUT consuming if an installed key works) → keygen → **consume** (l.159–163) →
|
||||
`ssh-copy-id` install + verify (l.168–173) → persist `offbox` target `EscrowState=pending`
|
||||
(l.176) → marker LAST (l.181). **Consume-then-persist; retry only on next process start.**
|
||||
- The exact dead-end is documented in source (l.168–173): *"key install FAILED after consuming the
|
||||
one-time password … the password is spent; reset it on the hub to retry"* — the designed recovery
|
||||
is the hub's **Re-issue offsite credentials** (`hub/internal/web/configs.go:730–773`: Hetzner
|
||||
password reset → fresh one-time secret → ConfigVersion bump → `bumpIntent` long-poll wake; if an
|
||||
escrow blob exists it is marked stale — demo-hp had none).
|
||||
- **The hub UI cannot see any of this:** the "Provisioned: … delivered to the controller once" line
|
||||
is static copy gated only on `offsite.host` in ConfigJSON
|
||||
(`hub/internal/web/templates/config_form_body.html:119–120`); nothing in the web layer reads
|
||||
`one_time_secrets.consumed_at`. "Provisioned" ≠ "delivered" ≠ "applied" — the hub shows the first
|
||||
and knows neither of the others. (→ R-70; note the R-39 precedent: PBS-DR grew a
|
||||
`consumed_at` honesty gauge for exactly this disagreement class.)
|
||||
|
||||
## 3. The 07-21 timeline (evidence: hub DB `events`/`one_time_secrets`/`customer_claims`/`reports`)
|
||||
|
||||
| UTC | Event |
|
||||
|---|---|
|
||||
| 16:23:32 | customer self-bind (`appliance_bound`) |
|
||||
| 16:23:55 | appliance credential delivered — box begins day-0 |
|
||||
| 16:24:19 | host row `demo-hp-bb76ea` created |
|
||||
| 16:24:27 | hub auto-provisions offsite on enrollment (`offsite_reissued`, source=hub): sub3 minted, one-time pw staged, config_version→2 |
|
||||
| **16:27:42** | **one-time password CONSUMED** (bridge inside the 0.153.0 floor controller) |
|
||||
| 16:28:07 | `controller_started (0.153.0)`; first report 16:28:08 |
|
||||
| **16:28:17** | **managed update 0.153.0→0.156.0 — container replaced, bridge killed mid-flight** (~35 s after consume, inside the install/persist window) |
|
||||
| 16:28:22 | 0.156.0 starts; bridge: no installed key → consume → **404** → nothing persisted |
|
||||
|
||||
Corroboration: **153 controller reports** 07-21 16:28 → 07-23 09:31, **zero** ever carrying an
|
||||
offbox status object; box settings.json had **no `offbox` key** (checked with the real key, per the
|
||||
R-7b lesson) and **no `offbox/` dir** (no ssh_key → the key-install never completed, so the
|
||||
key-auth-first recovery path could never engage); live log at the 0.160.0 start 07-22 17:16:35:
|
||||
`[WARN] [offsite-apply] reconcile: … consume one-time password: no unconsumed offsite password
|
||||
(already consumed or none provisioned)`.
|
||||
|
||||
Spec note: this is literally the spec's "delivery consumed but persist failed → STOP" class, but
|
||||
the evidence showed the mechanism **not broken** (fail-safe held: no half-configured offbox) with a
|
||||
source-designated recovery identical to Shape A's remedy; the operator ruled: proceed (recorded in
|
||||
session, 2026-07-23).
|
||||
|
||||
## 4. Repair via the designed path (operator click, once — R-31 discipline)
|
||||
|
||||
| UTC 07-23 | Event |
|
||||
|---|---|
|
||||
| 09:53:37 | Viktor clicks **Re-issue offsite credentials** ONCE — `[offsite] re-issued shared credentials for demo-hp (subaccount 275124)`; fresh pw staged; ConfigVersion bumped |
|
||||
| 09:53:39 | box long-poll wakes, config re-pulled |
|
||||
| **09:53:41** | fresh password **consumed** (4 s after staging) |
|
||||
| 09:53:45 | `[offsite-apply] offsite configured for u629488-sub3@…:/home/felhom-repo (pending key escrow)`; `offbox/` dir complete (ssh_key/known_hosts/repo_password/applied_marker), settings `offbox` present |
|
||||
|
||||
Click → converged in **8 seconds**. Escrow ceremony run by the operator through the real
|
||||
`/backup/escrow` wizard: blob stored 10:01:17 (`posture=zero_knowledge`, `restic_pw_sha256`
|
||||
recorded), R claimed one-shot 10:01:21, hub-verified auto-confirm 10:01:24 →
|
||||
`EscrowState=escrowed; offsite runs enabled`.
|
||||
|
||||
## 5. Tier proof (F10 closure bar: restore round-trip)
|
||||
|
||||
Per-app scope: `paperless-ngx` toggled into offsite via the real `POST /backup/offbox/toggle`
|
||||
(per-app default is OFF). Mandatory capture path per catalog: `appdata/paperless/media`.
|
||||
All steps via real endpoints from inside guest 9201 (endpoint-level method — no browser on DooPlex);
|
||||
probe placement/deletion on the filesystem simulates customer data and its loss; every
|
||||
backup/restore action went through the UI's own endpoints.
|
||||
|
||||
1. **Probe** 64 KiB urandom → `…/appdata/paperless/media/felhom-probe-f10-20260723.bin`,
|
||||
md5 `9120e65d6a9f071072d827fc404dc840`.
|
||||
2. **First offsite run** (`POST /backup/offbox/run`): repo **initialized fresh** on sub3 (proves the
|
||||
target was empty), 79.8 MB / 49 files, 1m19s, status `ok`, snapshots=1.
|
||||
3. **Loss**: probe deleted (ENOENT verified).
|
||||
4. **Restore** (`POST /backup/offbox/restore`, `mode=full`, two-step size gate revealed 79.8 MB →
|
||||
`confirm=1`): snapshot **`2bf7f2e1`** → staging `backups/offsite-restore/paperless-ngx`;
|
||||
staging probe **md5-identical**.
|
||||
5. **Place** (`POST /backup/offbox/place`): `1 file(s) merged (missing-only)`, live recovery unit
|
||||
not overwritten, scratch auto-removed; live probe **md5-identical**
|
||||
(`9120e65d6a9f071072d827fc404dc840`).
|
||||
6. **Cleanup**: probe removed; second run (2m17s, `ok`) so the latest snapshot is probe-free;
|
||||
retention pruned the probe-bearing snapshot (count stays 1). Guest session artifacts removed;
|
||||
break-glass + DB copies shredded on DooPlex.
|
||||
|
||||
Hub-side final state: latest demo-hp report carries
|
||||
`offsite: {enabled, escrow_state:"escrowed", quota_gb:50}`; `host_escrow` row present
|
||||
(zero_knowledge, hash recorded, not stale); `one_time_secrets` consumed 09:53:41. The Offsite tab
|
||||
now reflects the customer. Nightly offsite is scheduled (daily 04:15 guest-UTC).
|
||||
|
||||
## 6. Findings minted
|
||||
|
||||
- **R-70 (P2-HIGH)** — the pending/failed offsite last mile is invisible on BOTH surfaces; the hub
|
||||
cannot distinguish staged/consumed/applied. Couple to R-31's async/status-card idiom and the
|
||||
R-39 `consumed_at` honesty-gauge precedent.
|
||||
- **R-71 (P1)** — the day-0 race itself: managed floor-update vs apply-bridge, consume-then-persist
|
||||
not crash-safe. **This recurs on every fresh onboarding** whose ISO floor lags the managed floor
|
||||
(the update fires minutes after first boot, exactly when the bridge runs). demo-felhom escaped by
|
||||
timing only.
|
||||
- Audit F10 row annotated: **offsite leg RESOLVED** (this record); the PBS-DR-snapshot half of F10
|
||||
**stays open** pending F13 (cadence ruling) and the deliberate DR ceremony R-moment on demo-hp.
|
||||
|
||||
## 7. Observed, not acted on
|
||||
|
||||
- `one_time_secrets` row for **demo-felhom** staged 2026-07-21 08:29:29 is still **unconsumed** —
|
||||
residue of the mistaken offsite-Re-issue click during the R-39 session (the box recovered via the
|
||||
key-auth-first path, which never consumes). Harmless (superseded on any future re-issue), but it
|
||||
is exactly the state the R-39 honesty gauge flags on the PBS side — supporting R-70.
|
||||
- The three dead unclaimed-appliance records from the 07-22 ISO train remain at the hub for
|
||||
operator discard (unchanged).
|
||||
Reference in New Issue
Block a user