R-195: a customer with no machine ever bound does not alarm (hub v0.92.0) + R-193/R-192 spike
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Part 4 (ships): `david` — a prospective customer with hosts=0, host_deletions=0, reports=0 — e-mailed an expected_dbdump_missed ERROR at 03:00 UTC three mornings running. The existing down-skip could never cover it: it reads the staleness checker's state, which is seeded from a query over the `reports` table, so a customer that never reported has no state at all and GetState() returns "" rather than "down". store.HasEverBoundHost (hosts row OR host_deletions tombstone) is consulted once per customer at the top of the deadline loop. The discriminator is "was a host EVER bound", never "has a report arrived" — a box installed and never heard from is a real fault and keeps alarming. Fail-OPEN on a read error. Red-proof observed: removing the guard fails with `got [expected_dbdump_missed]`, verbatim the event david sent. Parts 0-3 (spike, NO production code for R-193/R-192): audits/SPIKE-offsite-credential-recovery-2026-08-04.md establishes that the one-shot provider password is the RECOVERABLE secret and the restic repository password is the irreplaceable one — and that a guest rebuild mints a fresh one, orphaning the previous off-site history. Measured without touching a box, by comparing host_escrow.restic_pw_sha256 against host_escrow_superseded: BOTH demo boxes changed (demo-hp 15 snapshots / 40.9 MB, demo-felhom 36 snapshots / 1.14 GB). demo-felhom's "lucky" 76-second recovery restored delivery and not the repository, silently, for 13h. ReissueCredentials does NOT rotate the restic password (R-39's record and two hub comments are wrong -> R-196); candidate (b) is not implementable against a zero-knowledge escrow; candidate (a) already exists as F3 and is wired to the wrong event. Ends in ranked options and an unanswered question for the operator. R-195 SHIPPED; R-196 + R-197 filed; R-192 + R-193 updated, neither closed.
This commit is contained in:
@@ -0,0 +1,501 @@
|
||||
# SPIKE — what is actually one-shot, and what a guest rebuild really costs
|
||||
|
||||
**Date:** 2026-08-04 · **Items:** R-193 (rebuild drops offsite), R-192 (the alert says the opposite of
|
||||
what it measured) · **Class:** spike — **no production code shipped for R-193 or R-192**
|
||||
**Baselines read:** `felhom.eu` @ `f456835bbcc4` (hub v0.91.1) · `felhom-controller` @ `0887fd67` ·
|
||||
`felhom-agent` @ `856a127`
|
||||
|
||||
> **This document ends in ranked options and a STOP.** It establishes facts; it does not choose.
|
||||
> The operator's question is stated, unanswered, at the end.
|
||||
|
||||
---
|
||||
|
||||
## 0. Why this exists
|
||||
|
||||
Two candidate fixes were named when R-193 was filed, and neither could be weighed, because two prior
|
||||
session reports described the same mechanism in contradictory terms:
|
||||
|
||||
- R-39's record: Re-issue *"rotates the restic password and makes the escrow stale"*.
|
||||
- R-193's own resolution note: the one-time password is *"only the transport credential used once to
|
||||
install the box's own SSH key"*.
|
||||
|
||||
Those two worlds imply very different fates for the fifteen existing snapshots. This spike settles it
|
||||
**from source first, live state second**, so a measurement can contradict the code rather than be
|
||||
interpreted by it.
|
||||
|
||||
**The headline, before the detail:** *both prior claims are wrong in the way that matters.* Re-issue
|
||||
does **not** touch the restic repository password — but a **guest rebuild does**, silently, on every
|
||||
box, with nothing marking the escrow stale and nothing telling anyone. Both demo boxes lost repository
|
||||
continuity on 2026-08-03/04. The one that "recovered by luck" lost it too.
|
||||
|
||||
---
|
||||
|
||||
## 1. The instruments, and what each one can and cannot answer
|
||||
|
||||
Stated up front so a wrong instrument is visible here rather than inside a conclusion (§8 rule 2).
|
||||
|
||||
| # | Instrument | Exact query / read | Known limit |
|
||||
|---|---|---|---|
|
||||
| I1 | hub SQLite snapshot | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale at copy time (mtime 08:34 vs wal 10:45 CEST). Copying only the main file would have produced confident answers from a stale file — the exact R-3 shape. |
|
||||
| I1-check | freshness proof | `PRAGMA integrity_check` → `ok`; `MAX(host_reports.received_at)` = `2026-08-04 08:45:38`, `datetime('now')` = `08:49:46` | **Positive observable**: newest row 4 min old. Not "the query returned no error". |
|
||||
| I2 | `host_escrow.restic_pw_sha256` + `host_escrow_superseded.restic_pw_sha256` | see §2 | sha256 of a 256-bit random secret — non-reversible, safe to compare. Answers *"did the repo password change"* **without touching a box or a credential**. |
|
||||
| I3 | `reports.report_json → $.offsite` | grouped distinct-object query with first/last seen | The report echoes the box's own target state; absence of the key = no target configured. |
|
||||
| I4 | `one_time_secrets` | `SELECT customer_id, created_at, consumed_at` | **`customer_id` is the PRIMARY KEY** — one row per customer, last-write-wins. It **cannot** answer "how many credentials were delivered"; a Re-issue destroys the prior row. demo-hp's 2026-07-23 consume is no longer in this table. |
|
||||
| I5 | source | `hub/internal/offsite/*`, `hub/internal/api/*`, `controller/internal/{backup,offsiteapply}`, `agent/internal/escrow` | — |
|
||||
|
||||
**Not used, deliberately:** no `ReissueCredentials`, no ceremony, no config change, no deletion, no
|
||||
write to any box. Nothing on either box or on the storage endpoint was altered by this session.
|
||||
|
||||
---
|
||||
|
||||
## Q2 — Does a rebuilt controller reuse the escrowed repository password, or mint a new one?
|
||||
|
||||
### **RULING: it MINTS A NEW ONE. The escrowed password is never consulted on any automatic path.**
|
||||
|
||||
**Method — source.** The apply-bridge's terminal step is
|
||||
`Enabler.ConfigureOffbox(...)` (`controller/internal/offsiteapply/offsiteapply.go:221,256`), which
|
||||
reaches `Manager.ApplyOffsiteTarget` → `WriteOffboxSecrets`
|
||||
(`controller/internal/backup/offbox.go:370`). Quoted verbatim, `offbox.go:392-401`:
|
||||
|
||||
```go
|
||||
// Auto-generate the repo password once (0600), never log it.
|
||||
if _, err := os.Stat(m.offboxPwPath()); os.IsNotExist(err) {
|
||||
pw, gerr := generateOffboxPassword()
|
||||
...
|
||||
if werr := os.WriteFile(m.offboxPwPath(), []byte(pw), 0o600); werr != nil {
|
||||
```
|
||||
|
||||
`offboxPwPath()` is `<DataDir>/offbox/repo_password` (`offbox.go:363`). A rebuilt guest has a fresh
|
||||
data dir, so the file is absent, so **a fresh 256-bit password is minted**.
|
||||
|
||||
**The only path that recovers the old one is `InjectOffboxPassword`** (`offbox.go:541`), whose sole
|
||||
caller in the entire repo is `offboxInjectPasswordHandler`
|
||||
(`controller/internal/web/offbox_handlers.go:189`) — a **web form where a human pastes the recovered
|
||||
password**. Verified by census: `grep -rn "InjectOffboxPassword" --include=*.go .` → three hits, one
|
||||
definition, one caller, one comment. **The apply-bridge never calls it.**
|
||||
|
||||
**Method — live, and it is decisive without touching a box (I2).** The hub already stores the sha256
|
||||
of the escrowed repo password, so the question is answerable as a hash comparison:
|
||||
|
||||
| host | superseded (pre-rebuild) | current (post-rebuild) | verdict |
|
||||
|---|---|---|---|
|
||||
| `demo-hp-bb76ea` | `8e03eddf9ff7…` created `2026-07-23T10:01:17Z` | `8a9e33aa4da6…` created `2026-08-04T07:15:36Z` | **CHANGED** |
|
||||
| `demo-felhom-8363b5` | `48741892f0ef…` created `2026-07-21T08:38:33Z` | `c60c8bc737a6…` created `2026-08-04T07:20:07Z` | **CHANGED** |
|
||||
|
||||
**Both boxes minted a new repository password.** demo-hp's pre-rebuild escrow (`8e03…`, sealed eight
|
||||
minutes after its 2026-07-23 09:53:41 apply) is the key to its 15 snapshots. That key now exists
|
||||
**only inside a superseded, R-wrapped escrow blob** that the hub cannot open.
|
||||
|
||||
### **This is bigger than R-193 as filed, and it hits the box R-193 called lucky.**
|
||||
|
||||
R-193 records demo-felhom as having *"survived the SAME rebuild by luck"* — 76 seconds of downtime.
|
||||
Measured (I3), demo-felhom's pre-rebuild offsite object was:
|
||||
|
||||
```
|
||||
last_run 2026-08-03T02:17:44Z · last_status ok · snapshot_count 36 · repo_size_bytes 1 136 685 919
|
||||
```
|
||||
|
||||
**36 snapshots, 1.14 GB.** Since 2026-08-03 07:19:10 UTC it has reported `snapshot_count: 0,
|
||||
repo_size_bytes: 0` in every one of 109 reports, and its repo password hash changed.
|
||||
|
||||
> **The luck recovered DELIVERY. It did not recover the REPOSITORY.** The staged secret restored the
|
||||
> transport in 76 seconds and the box then minted a brand-new repo password anyway. The contrast
|
||||
> R-193 draws between the two boxes is real for the *credential*, and **false for the data**: both
|
||||
> boxes lost repository continuity, one loudly and one silently. The silent one is worse.
|
||||
|
||||
---
|
||||
|
||||
## Q3 — What did the 04:15 run report?
|
||||
|
||||
### **RULING: UNMEASURED — the decisive run has not happened yet. And the binary the question offers is the wrong one; source says the answer will be NEITHER 15 nor 1.**
|
||||
|
||||
**Why it has not happened.** The scheduled off-box run fires at ~02:15 UTC (04:15 CEST). Measured
|
||||
from I3 — `last_run` values `2026-08-02T02:15:52Z`, `2026-08-03T02:16:39Z` on demo-hp;
|
||||
`2026-08-01T02:17:54Z`, `2026-08-02T02:17:01Z`, `2026-08-03T02:17:44Z` on demo-felhom. **Daily, both
|
||||
boxes.** Then:
|
||||
|
||||
| box | why 2026-08-04 02:15 UTC produced nothing |
|
||||
|---|---|
|
||||
| demo-hp | no offsite target existed at all — the `offsite` key was **ABSENT** from every report between `2026-08-03 06:12:19` and `2026-08-04 07:12:01` UTC (111 reports). The re-issue landed at 07:15:47, five hours after the window. |
|
||||
| demo-felhom | the target existed but `escrow_state: pending` from `2026-08-03 07:19:10` to `2026-08-04 07:20:13`. `OffboxRunnable` requires configured **AND** escrowed (`offbox.go:564-569`), so the fork-4 gate blocked the run. The ceremony landed at 07:20:28. |
|
||||
|
||||
Both boxes became runnable only this morning. **The decisive run is 2026-08-05 ~02:15 UTC on both.**
|
||||
Reported as unmeasured rather than inferred.
|
||||
|
||||
### What source says will happen — and it is a third outcome
|
||||
|
||||
Same sub-account, same repo path, new password. Measured coordinates (I3, `$.dr_recipe`):
|
||||
`u629488-sub3.your-storagebox.de:/home/felhom-repo` — **unchanged** across the whole incident, and
|
||||
`repoPath` is a compile-time constant (`hub/internal/offsite/offsite.go:85`). So the next run opens an
|
||||
**existing repo with the wrong key**, which the codebase already has a name for
|
||||
(`controller/internal/backup/offbox.go:67-93`):
|
||||
|
||||
```go
|
||||
// ErrOffboxOrphaned is the sentinel returned when the offsite repo exists but is keyed under a
|
||||
// passphrase this controller no longer has (the reinstall shape) ...
|
||||
case strings.Contains(s, "wrong password or no key found"):
|
||||
return "orphaned"
|
||||
```
|
||||
|
||||
and `ensureOffboxRepo` (`offbox.go:663-679`) branches on claim state:
|
||||
|
||||
```go
|
||||
if !m.settings.GetClaimed() { // UNCLAIMED → auto move-aside + re-init
|
||||
...
|
||||
m.markOrphaned()
|
||||
return ErrOffboxOrphaned // CLAIMED → skip the run, show the orphan card
|
||||
```
|
||||
|
||||
Both boxes report `claimed: 1` (I3, `$.claimed`, latest report each). **So the predicted outcome is:
|
||||
the run REFUSES with `ErrOffboxOrphaned`, produces no snapshot at all, and waits for the customer to
|
||||
confirm a reset.** Not 15 (reattached) and not 1 (silently fresh) — a **third** outcome the question's
|
||||
framing did not contain. That is the good news half: the system is not going to quietly start a new
|
||||
history over the old one. It will stop and say so.
|
||||
|
||||
**Consequence if the customer confirms the reset:** `resetOrphanedRepo` (`offbox.go:270-314`) moves the
|
||||
old repo aside and re-inits — *"move-aside, not deleted"*. The 40.9 MB + 1.14 GB of old ciphertext
|
||||
survives, unreadable without the superseded escrow, **and keeps consuming the 50 GB soft quota
|
||||
indefinitely.** Nothing prunes a moved-aside restic repo (S-24 covers ep0's PBS namespaces, a
|
||||
different tier).
|
||||
|
||||
**To measure it tomorrow:** re-read `$.offsite.last_status` / `last_error` on both boxes after
|
||||
02:20 UTC, and `$.offsite.repo_state` for the orphan flag. **Predicted `ErrOffboxOrphaned`, not a
|
||||
snapshot count.** Whoever picks this up should record which of the three actually occurred — a
|
||||
prediction from source is not a measurement.
|
||||
|
||||
---
|
||||
|
||||
## Q1 — What is one-shot, and what is not?
|
||||
|
||||
Three secrets, and they have almost nothing in common.
|
||||
|
||||
| | **① storage-provider password** | **② the box's SFTP key** | **③ restic repository password** |
|
||||
|---|---|---|---|
|
||||
| **What it is** | the Hetzner sub-account / box password | ed25519 keypair for `sftp` transport | the **data** key — restic's repo encryption |
|
||||
| **Generated by** | **hub** — `genPassword()` (`offsite.go:481`) | **controller** — `KeyGen.Generate()` (`offsiteapply.go:234`) | **controller** — `generateOffboxPassword()`, 32 random bytes hex (`offbox.go:406`) |
|
||||
| **Stored where** | `one_time_secrets` (hub, plaintext, **one row per customer**) | `<DataDir>/offbox/ssh_key` 0600 (box only) | `<DataDir>/offbox/repo_password` 0600 (box) + inside the R-wrapped escrow blob (hub, opaque) |
|
||||
| **Delivered how** | `GET …/offsite/consume-password`, **served exactly once** then marked consumed (`api/offsite.go:9-31`) | never delivered — installed onto the provider by ssh-copy-id using ① | never delivered anywhere; it never leaves the box except into the escrow ceremony |
|
||||
| **Re-issuable?** | **YES** — `ReissueCredentials` resets it at the provider, any time, operator-initiated | **YES** — regenerated on every full apply; `dr_recipe.go:45` states it plainly: *"the SFTP access key is regenerated at DR"* | **NO automatic path.** Recoverable only by unsealing the escrow with the customer's recovery code and pasting it into `offboxInjectPasswordHandler` |
|
||||
| **What a guest rebuild does to it** | nothing (it lives on the hub / at the provider) | destroys it; a fresh one is minted and installed — **harmless** | **destroys it; a fresh one is minted — CATASTROPHIC for the existing repo** |
|
||||
| **Escrowed?** | no | no | **yes** — `IdentityBundle.ResticRepoPassword` |
|
||||
|
||||
The agent's own source names the asymmetry, and it is the sentence the whole spike turns on
|
||||
(`felhom-agent/internal/escrow/identity.go:35-39`):
|
||||
|
||||
> `ResticRepoPassword` … **It is the DATA key for the offsite tier — irreplaceable (unlike the SFTP
|
||||
> access key, which is regenerable at DR).**
|
||||
|
||||
**The one-shot thing (①) is the recoverable one. The irreplaceable thing (③) is the one nothing
|
||||
re-stages.** R-193 was filed against ①. The damage is in ③.
|
||||
|
||||
**Instrument caveat, worth carrying (I4):** `one_time_secrets` has `customer_id` as PRIMARY KEY, and
|
||||
`SaveOneTimeSecret` is last-write-wins **by design** (the R-39(a) guard comment in
|
||||
`monitor/offsite_delivery.go:26-30` depends on it). So the delivery ledger holds only the newest
|
||||
credential per customer: demo-hp's 2026-07-23 consume, on which R-192's whole diagnosis rests, was
|
||||
**overwritten** by the 2026-08-04 Re-issue and is no longer readable from that table.
|
||||
|
||||
---
|
||||
|
||||
## Q4 — Which claim about Re-issue is true?
|
||||
|
||||
### **RULING: `ReissueCredentials` touches ONLY the provider credential (①). It does NOT touch the restic repository password. R-39's record is wrong — and the hub's own comments repeat the same wrong claim in three places.**
|
||||
|
||||
**Method — source, quoted rather than summarised.** `ReissueCredentials`
|
||||
(`hub/internal/offsite/offsite.go:150-228`) does exactly four things:
|
||||
|
||||
1. `genPassword()`;
|
||||
2. `ResetSubaccountPassword` / `ResetBoxPassword` + `WaitAction` — **the provider account password**;
|
||||
3. `Store.SaveOneTimeSecret(customerID, pw)` — stage ① for delivery;
|
||||
4. bookkeeping: `MarkEscrowStale` + `offsite_reissued` + `escrow_stale` events.
|
||||
|
||||
There is **no reference to a restic password anywhere in the function**, and none is possible — the
|
||||
repo password is generated on the box and never leaves it except into the escrow blob. The hub does
|
||||
not hold it in any openable form (`api/handler.go`: *"The hub stores the bytes and NEVER decrypts them
|
||||
(it has no recovery code)"*).
|
||||
|
||||
**But step 4 is justified by a claim that is false**, `offsite.go:198-201`:
|
||||
|
||||
> `// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing`
|
||||
> `// key-escrow blob — which sealed the OLD password — is now STALE.`
|
||||
|
||||
The escrow *is* marked stale; the stated reason for marking it is not something this function does.
|
||||
The same false premise appears twice more: `api/handler.go:1067-1069` (*"The re-issuer resets the
|
||||
restic repo password, which makes the OLD escrow blob stale"*) and `OPEN-ITEMS.md`'s R-193 row, which
|
||||
inherited it from R-39.
|
||||
|
||||
> **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does
|
||||
> not provide** — and the first where the comment is not merely unenforced but factually describes a
|
||||
> different function. It survived because it reads as settled and because its *effect* (a stale escrow)
|
||||
> is real, so nobody checked the *cause*.
|
||||
|
||||
**Consequence, and it is a live defect, not a documentation nit.** On the ordinary Re-issue shape —
|
||||
a consumed-but-failed install on a box that still has its `repo_password` file — the box re-applies,
|
||||
`WriteOffboxSecrets` finds the file present and **keeps it**, and the repo password is unchanged. The
|
||||
hub has nonetheless told the customer, in Hungarian, that their recovery escrow is stale and asked
|
||||
them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** → **R-196**.
|
||||
|
||||
**Corroboration against what actually happened on demo-hp yesterday.** The escrow went `pending` after
|
||||
the Re-issue and the task asks: if the repo password was untouched, what explains that? **It is fully
|
||||
explained, and not by the Re-issue.** Two independent mechanisms, both hub-side-invisible:
|
||||
|
||||
1. `ApplyOffsiteTarget` (`offbox.go:482-494`) carries `EscrowState` over from the *existing* target —
|
||||
but the rebuilt guest had **no existing target**, so `cur` was nil, so the fallthrough
|
||||
`if tgt.EscrowState != "escrowed" { tgt.EscrowState = "pending" }` fired.
|
||||
2. The box had minted a **new** repo password moments earlier (Q2), so `pending` was *correct* — the
|
||||
new key genuinely was not escrowed.
|
||||
|
||||
**The Re-issue's `escrow_stale` event fired at 07:11:51 for the wrong reason and happened to be true
|
||||
for a different one.** Coincidence, not correctness. The proof that these are independent is
|
||||
**demo-felhom**: it received **no Re-issue at all** (measured — its only `escrow_stale` /
|
||||
`offsite_reissued` events in the entire history are dated `2026-07-21 08:29:29`), its repo password
|
||||
changed anyway, and **nothing marked its escrow stale for thirteen hours.**
|
||||
|
||||
> **The mechanism is exactly inverted.** `escrow_stale` is wired to the one path that does **not**
|
||||
> change the repo password, and absent from the path that **does**.
|
||||
|
||||
---
|
||||
|
||||
## Q5 — Why did one box recover itself and the other not?
|
||||
|
||||
### **RULING: confirmed from the ledger — an unconsumed staged secret, thirteen days old. And the recovery was partial in a way the narrative missed.**
|
||||
|
||||
**Method — I4, the `one_time_secrets` ledger, no values read:**
|
||||
|
||||
| customer | `created_at` | `consumed_at` |
|
||||
|---|---|---|
|
||||
| `demo-felhom` | `2026-07-21 08:29:29` | `2026-08-03 07:17:58` |
|
||||
| `demo-hp` | `2026-08-04 07:11:51` | `2026-08-04 07:12:06` |
|
||||
|
||||
demo-felhom's row was **created 2026-07-21 and sat UNCONSUMED for 13 days** — traceable to a Re-issue
|
||||
at that exact timestamp (its `offsite_reissued` + `escrow_stale` event pair, `2026-07-21 08:29:29`).
|
||||
When the rebuild dropped its target on 2026-08-03, the fresh controller consumed the spare at 07:17:58
|
||||
and was reporting `offsite` again by 07:19:10. **76 seconds, on a credential nobody had planned to
|
||||
leave there.** Confirmed from the ledger, not the narrative.
|
||||
|
||||
**What would have happened without one:** exactly demo-hp — `consume` → **404** (`api/offsite.go:20-22`,
|
||||
`no unconsumed offsite password`), the apply-bridge returns
|
||||
`offsite-apply: consume one-time password: …` and retries forever, and the tier stays absent until an
|
||||
operator acts. demo-hp sat there **25 hours** (`2026-08-03 06:12:19` → `2026-08-04 07:12:01`).
|
||||
|
||||
**The correction to the contrast (see Q2):** demo-felhom's spare restored *delivery* only. Its 36
|
||||
snapshots / 1.14 GB were orphaned by the same rebuild. **Neither box's data survived; one box's
|
||||
plumbing did.**
|
||||
|
||||
*Not established:* demo-felhom's escrow generation before 2026-07-21. `host_escrow_superseded` holds
|
||||
only two rows in the whole database (ids 3 and 4, one per box, both superseded 2026-08-04), so whether
|
||||
the 2026-07-21 Re-issue's ceremony re-sealed an *unchanged* password — the predicted false-staleness
|
||||
shape — **cannot be shown from this data**. Stated as unknown rather than asserted.
|
||||
|
||||
---
|
||||
|
||||
## Q6 — Why is delivery one-shot at all?
|
||||
|
||||
### **RULING: the design's reason is that ① is a LIVE PROVIDER PASSWORD, not a bootstrap token; and the honest answer to the security question is that an automatic restage weakens EXPOSURE WINDOW, not authentication.**
|
||||
|
||||
**The design's own reasons, from source:**
|
||||
|
||||
- `api/offsite.go:9-12` — *"serves the one-time transient offsite password to the controller **EXACTLY
|
||||
ONCE** … The value is returned once then marked consumed — a second call 404s. NEVER logged."*
|
||||
- `offsite.go:145-149` — *"the **EXPLICIT** operator recovery for a consumed-password dead-end … It is
|
||||
**NOT** implicit rotation: ProvisionOffsite never calls this."*
|
||||
- `offsiteapply.go:1-6` — the apply-bridge is *"idempotent (a descriptor hash marker prevents
|
||||
**re-consuming a spent password**)"*, i.e. the design treats a wasted consume as a real hazard, which
|
||||
R-71a's whole settle-gate exists to prevent.
|
||||
|
||||
**The security half, stated rather than skipped.** The question is: *could a machine that is not the
|
||||
real one obtain a credential via an automatic restage?*
|
||||
|
||||
**What stands there today is authentication, and a restage does not remove it.** The consume endpoint
|
||||
is gated by `checkAuthCustomer` (`api/offsite.go:14-18`) — **the customer's API key, the same
|
||||
credential as the config pull**. Anything that could consume a restaged password can already pull the
|
||||
customer's whole config. So an automatic restage **grants no new access to an unauthenticated party**.
|
||||
|
||||
**What it genuinely weakens is the exposure window.** Today a live Hetzner sub-account password is
|
||||
fetchable only in the seconds after an operator deliberately staged one. Under an automatic restage it
|
||||
becomes fetchable **whenever the box reports no offsite target** — a condition an attacker holding the
|
||||
API key can arguably induce. The credential's blast radius at the provider is the customer's own
|
||||
sub-account (label-scoped; `ReissueCredentials` refuses unless the label lookup finds **exactly one**,
|
||||
`offsite.go:164,180`), so this is a widening of *when*, not of *what*.
|
||||
|
||||
**And one further guard must survive any change** — R-39(a), `monitor/offsite_delivery.go:26-30`:
|
||||
`SaveOneTimeSecret` is last-write-wins, so restaging on top of an **unconsumed** secret clobbers a
|
||||
password a box may be about to consume. Any automatic restage must re-read the row immediately before
|
||||
acting and refuse unless it is still a **consumed** row. That guard is correct and is not what R-192 is
|
||||
about.
|
||||
|
||||
---
|
||||
|
||||
## Q7 — R-192's guard: exactly what it reads, and why it read wrong
|
||||
|
||||
### **RULING: both halves confirmed, with numbers. The guard reads the OLDEST 500 reports since the consume, and for demo-hp all 500 predated the rebuild by six days.**
|
||||
|
||||
**The query, quoted** (`hub/internal/store/store.go:987`):
|
||||
|
||||
```sql
|
||||
SELECT report_json FROM reports WHERE customer_id = ? AND received_at > ? ORDER BY id LIMIT 500
|
||||
```
|
||||
|
||||
`ORDER BY id` ascending with `LIMIT 500` = **the oldest 500**, not the newest.
|
||||
|
||||
**Reproduced against the live data (I1)**, with demo-hp's real consume anchor `2026-07-23 09:53:41`
|
||||
(recovered from R-192's record — it is no longer in `one_time_secrets`, see I4):
|
||||
|
||||
| measurement | value |
|
||||
|---|---|
|
||||
| `total` (what the guard sees) | **500** |
|
||||
| `withOffsite` (what the guard sees) | **500** |
|
||||
| oldest report in that window | `2026-07-23 09:53:47` |
|
||||
| newest report in that window | **`2026-07-28 11:17:40`** |
|
||||
| true `total` over the same window, no LIMIT | 1174 |
|
||||
| true `withOffsite` over the same window, no LIMIT | 1063 (⇒ 111 without — matching the 111 ABSENT reports in I3) |
|
||||
|
||||
**The whole 500-report evidence set ends 2026-07-28 — six days before the 2026-08-03 rebuild.**
|
||||
|
||||
**Defect (a) — the message is exactly inverted.** `maybeEmitStuck`
|
||||
(`monitor/offsite_delivery.go:~110`) interpolates `status.ReportsSinceConsume` into a string that
|
||||
hardcodes the phrase *"carry no offbox target"*, and never reads `OffsiteReportsSinceConsume`. The
|
||||
live e-mail, quoted in full from the events table:
|
||||
|
||||
> *"Offsite delivery stuck: one-time password consumed 284h19m0s ago and **500 report(s) since carry
|
||||
> no offbox target** — the credential is likely burned (apply died between consume and persist).
|
||||
> Re-issue delivers a fresh one."*
|
||||
|
||||
`OffsiteReportsSinceConsume` for that same call was **500**. Every one of them carried a target. The
|
||||
message states the precise negation of its own measurement, and prescribes a remedy for a failure mode
|
||||
that did not occur.
|
||||
|
||||
**Defect (b) — the heal refuses silently, on stale evidence.** `maybeHeal` returns bare on
|
||||
`status.OffsiteReportsSinceConsume != 0` (it was 500), with **no log line**. `offsite_credential_restaged`
|
||||
has never fired for any customer — confirmed: zero rows of that event type in the database.
|
||||
|
||||
**What the guard SHOULD discriminate on — the shape, not the code.** The guard's intent is right:
|
||||
*"there is offbox evidence, so this is a regressed apply, not a burned credential — the operator's
|
||||
call."* Its error is that **it asks the question of the wrong time period**. Offbox evidence from
|
||||
before a rebuild is not evidence that the credential still works. The discriminator must be
|
||||
**recency-bounded and rebuild-aware**: judge on evidence *after the newest `controller_started` /
|
||||
config-hash change*, or on the latest N reports — never on "everything since the consume". And a
|
||||
refusal must **name its reason in the log**, because "we chose not to heal" and "the heal never ran"
|
||||
must not look identical (the S-16 rule, one layer down).
|
||||
|
||||
**Do not fix it in isolation.** Under Q2's finding, a successful auto-restage on demo-hp would have
|
||||
restored the transport and the box would *still* have minted a new repo password. **The heal cannot
|
||||
protect the data; it can only protect the plumbing.** Whatever shape it takes must say so.
|
||||
|
||||
---
|
||||
|
||||
## Q8 — What each candidate would cost
|
||||
|
||||
**A fourth option exists and is the important one**, because (a) and (b) both address ① and Q2 shows
|
||||
the damage is in ③.
|
||||
|
||||
### (a) The hub restages automatically when a re-enrolled box reports no offsite
|
||||
|
||||
**A version of this ALREADY EXISTS and is wired to the wrong event.** `reissueOnReenroll`
|
||||
(`hub/internal/api/handler.go:1051-1084`), leg **F3**:
|
||||
|
||||
> *"F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
|
||||
> one-time password only ever reached the OLD controller)."*
|
||||
|
||||
It is called from `handleHostEnroll` — but **after** a mint-once-reuse short-circuit
|
||||
(`handler.go:1004-1016`): `if existing != nil { …return… }`. **A guest rebuild leaves the `hosts` row
|
||||
intact** (measured: `demo-hp-bb76ea` is the same host_id throughout the incident; only the agent leaf
|
||||
re-keyed, `host_leaf_changed` 2026-08-03 06:09:40), so re-enroll returns the existing credential and
|
||||
**F3 is never reached.**
|
||||
|
||||
- **Changes:** hub only. Either lower the trigger from host-enrollment to a controller-level rebuild
|
||||
signal (the `config_hash` change + `controller_started` pair the hub already receives and already
|
||||
logs), or fix R-192's guard so the existing R-71c self-heal can act.
|
||||
- **Weakens:** the exposure window in Q6. Needs the R-39(a) unconsumed-secret guard kept intact.
|
||||
- **Does NOT solve:** ③. The box still mints a new repo password and still orphans the repo. **This
|
||||
option restores the plumbing to a box whose data key is already gone.**
|
||||
- **Cost:** small. **Value against the actual harm: near zero.**
|
||||
|
||||
### (b) The credential becomes recoverable from escrow at re-bootstrap
|
||||
|
||||
- **BLOCKED AS STATED, and this is a hard constraint, not an estimate.** The escrow blob is
|
||||
**R-wrapped and zero-knowledge** — the hub holds opaque bytes and has no recovery code
|
||||
(`api/handler.go`, and D6 in `CONTEXT.md` S-3). **A rebuilt box cannot unseal it without the
|
||||
customer's recovery code.** Any design that says "recover it at re-bootstrap" is describing a
|
||||
customer-present ceremony, i.e. what the existing manual `offboxInjectPasswordHandler` already is.
|
||||
- **Does NOT solve:** an unattended rebuild, which is the whole scenario.
|
||||
- **Cost: not implementable as specified.**
|
||||
|
||||
### (c) — NOT PREVIOUSLY NAMED — the agent retains the repo password across a guest rebuild
|
||||
|
||||
The agent lives **on the Proxmox host** and survives a guest rebuild. It already receives the repo
|
||||
password today: the controller pushes it over the pinned local API
|
||||
(`POST /escrow/stage-secret` → `agent/internal/localapi/escrow_stage.go`), and the agent writes it to
|
||||
the fixed 0600 path `/var/lib/felhom-agent/escrow-stage/restic_repo_password`
|
||||
(`agent/internal/escrow/identity.go:44-46`) **transiently — wiped by the ceremony**
|
||||
(`WipeStagedResticPassword`). The seam, the transport, the pinning and the file are all already built.
|
||||
|
||||
- **Changes:** agent (retain rather than wipe; serve back over the same pinned local API) + controller
|
||||
(on a fresh data dir, ask the agent before minting — the `InjectOffboxPassword` seam already exists
|
||||
and is exactly the right shape).
|
||||
- **Weakens:** the escrow's *zero-knowledge* posture in one specific way — the irreplaceable data key
|
||||
would now sit at rest on the Proxmox host, which the operator has root on. **Measured against D6,
|
||||
that is not a new exposure in principle** (D6 already states plainly that *"the operator cannot read
|
||||
customer data"* was never the security property, and the operator holds root on every box) — but it
|
||||
IS a new copy, and copies are the thing R-133 is already open about for the hub DB. **This is a real
|
||||
trade and it is the operator's to make.**
|
||||
- **Solves:** exactly the failure that occurred, unattended, on every rebuild, without any hub
|
||||
involvement and without any credential leaving the premises.
|
||||
- **Cost:** medium. Two repos, one existing seam each.
|
||||
|
||||
### (d) Do neither — keep the manual Re-issue, and fix what LIES
|
||||
|
||||
- **Changes:** R-192's message + refusal log; correct the three false comments about what Re-issue
|
||||
rotates; and — the part that actually matters — **make a repo-password change VISIBLE**. The hub
|
||||
already receives `restic_pw_sha256` on every escrow upload and already stores it. Comparing the new
|
||||
hash against the superseded one is a two-line verdict, and it is the signal that was missing on
|
||||
demo-felhom for thirteen hours → **R-197**. (Note the shape: **this spike answered its own hardest
|
||||
question with a comparison the hub could be making automatically and is not.** Both values were
|
||||
already in the database; nothing read them.)
|
||||
- **Weakens:** nothing.
|
||||
- **Does NOT solve:** the data loss. Every rebuild still orphans the repo; the operator simply finds
|
||||
out the same day instead of never.
|
||||
- **Cost:** small.
|
||||
|
||||
### Ranked, with a recommendation
|
||||
|
||||
| rank | option | why |
|
||||
|---|---|---|
|
||||
| **1** | **(d) now — the honesty pass** | It is cheap, it weakens nothing, and **it is the only option that helps the incident already in flight.** The alarm that lies is worse than no alarm; the repo-password-changed detector is the one signal whose absence let demo-felhom lose 1.14 GB of history silently. Ship this regardless of what is decided about the rest. |
|
||||
| **2** | **(c) — agent-retained repo password** | The only candidate that addresses ③, which is where the harm is. Every seam it needs exists. It costs one deliberate trade-off the operator must actually make. |
|
||||
| **3** | **(a) — auto-restage** | Worth doing **after** (c), never instead of it. On its own it restores the transport to a box whose data key is already gone, and would have made both boxes look healthy on 2026-08-04 while the snapshots were orphaned — a *strictly worse* outcome than the current loud failure. |
|
||||
| **4** | **(b) — recover from escrow at re-bootstrap** | Not implementable as stated; the escrow is zero-knowledge by design. |
|
||||
|
||||
**Recommendation: ship (d), then decide (c).** And do not ship (a) first — it would have hidden this.
|
||||
|
||||
---
|
||||
|
||||
## What could not be established
|
||||
|
||||
Named rather than guessed (§8 rule 4).
|
||||
|
||||
1. **The 2026-08-05 02:15 UTC run's actual outcome** (Q3). Predicted `ErrOffboxOrphaned` from source +
|
||||
claim state; **not measured**.
|
||||
2. **Whether the orphaned ciphertext still exists at the provider.** 40.9 MB (demo-hp) + 1.14 GB
|
||||
(demo-felhom) are *presumed* present at `/home/felhom-repo` — reading the endpoint needs the SFTP
|
||||
credential, which this session did not touch. The Hetzner API has no directory-listing surface.
|
||||
3. **Whether demo-felhom's 2026-07-21 Re-issue re-sealed an unchanged password** (the predicted
|
||||
false-staleness shape, Q4/Q5). Only two `host_escrow_superseded` rows exist in the whole database;
|
||||
the generation before 2026-07-21 is not retained.
|
||||
4. **What removed demo-hp's offsite target at the rebuild** is fully explained (fresh data dir), but
|
||||
**why the guest was rebuilt on both boxes on consecutive mornings** is outside this spike.
|
||||
|
||||
---
|
||||
|
||||
## The operator's question — stated, and NOT answered here
|
||||
|
||||
> **The irreplaceable secret in the offsite tier is the restic repository password. It is generated on
|
||||
> the box, it is destroyed by a guest rebuild, and the only copy that survives is sealed under the
|
||||
> customer's recovery code — which nothing but a human can open. Every machine is going to be
|
||||
> reinstalled.**
|
||||
>
|
||||
> **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host, where
|
||||
> you already have root — or do you want it to stay sealed under the customer's recovery code and
|
||||
> accept that every rebuild starts a new repository and orphans the old one, provided the system says
|
||||
> so loudly on the day it happens?**
|
||||
|
||||
Option (c) is the first. Option (d) alone is the second. **This spike does not choose.**
|
||||
Reference in New Issue
Block a user