admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

RECON: trace the offsite DR chain link by link — it does not join up (R-198..R-201)
gates / gates (push) Successful in 7s
Details

Browse Source 
Read-only recon of the escrow -> recovery chain, from a dead node to an open
repository. No production code, no build, no version bump.

Headline: the hub's superseded-escrow retention does NOT retain the offsite
repository password. host_escrow_superseded has no identity_blob column and
demoteCurrentEscrowTx copies only the K-escrow blob, so what survives a
supersession is the PBS datastore key, not the restic repo password. The next
escrow ceremony -- which the system tells a rebuilt box's customer to run --
destroys the last copy. Both demo boxes crossed that line on 2026-08-04.

Also established:
- the hub's blob-serving endpoints (re-enroll / restore-directive) have zero
  callers anywhere: agent, hub UI, scripts, runbooks (R-199)
- POST /backup/offbox/inject-password is routed and handled but no template
  contains the form (R-200)
- nothing in the recovery path has ever been exercised; the one live
  round-trip proof (2026-06-10) predates the ResticRepoPassword field (R-201)
- a fail-closed mint refusal IS implementable: the report ACK already carries
  escrow{identity_blob_present, restic_pw_sha256} and the controller discards
  it whenever no offbox target exists

Corrections: yesterday's spike annotated (candidate (b) overturned in part --
unattended recovery is impossible, customer-present is not); capability-map
retention claim struck through and replaced with what the code does.

Deliverable: documentation/audits/RECON-offsite-dr-chain-2026-08-04.md
Register: new R-198..R-201; R-193 and R-192 updated; STATUS.md refreshed.
This commit is contained in:
authentik Default Admin
2026-08-04 12:16:04 +02:00
parent d26f49ad68
commit 3f2b7bc023
6 changed files with 726 additions and 310 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+99 -307
View File
@@ -1,329 +1,121 @@
# REPORT — R-193/R-192 spike (no code) + R-195 phantom-customer alarm (hub v0.92.0)
# REPORT — RECON: can a customer actually get their backups back? (2026-08-04)
**Date:** 2026-08-04 · **Class:** spike (Parts 0–3) + one fenced implementation (Part 4)
**Boundary held:** the spike shipped **no production code for R-193 or R-192**, and Part 4 never
touched the spike's subject matter.
**Class:** recon / spike. **No production code. No build, no deploy, no version bump.**
**Baselines re-read on arrival:** `felhom-agent` `856a127cd60b` (v0.124.1) · `felhom-controller`
`0887fd676d60` (v0.194.0) · `felhom.eu` `d26f49ad680e` (hub v0.92.0). All three trees clean.
> Supersedes the earlier 2026-08-04 report (R-191 + R-190), per the overwrite convention. Nothing
> durable from it lived only here — it is recorded in `hub/CHANGELOG.md`, `CONTEXT.md` S-24/S-25 and
> `OPEN-ITEMS.md` R-190/R-191.
> Supersedes the earlier 2026-08-04 report (R-193/R-192 spike + R-195), per the overwrite convention.
> Nothing durable from it lived only here — it is recorded in `hub/CHANGELOG.md`, `OPEN-ITEMS.md`
> R-192/R-193/R-195/R-196/R-197 and `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`.
**Deliverable:** `documentation/audits/RECON-offsite-dr-chain-2026-08-04.md` — Q1–Q8 with method,
measurement and ruling per question, the eleven-link chain table, the drill design, every query used,
and the operator's open decisions.
---
## 1. Confirmed baselines, re-read at the cited commit
## 1. What was established
| Repo | `main` @ commit on arrival | Version | End state |
|---|---|---|---|
| `felhom.eu` | `f456835bbcc4` ✔ matches the spec | hub `v0.91.1` (deployed `felhom-hub:0.91.1`) | **`v0.92.0`** built, pushed, manifest bumped, synced, **live** |
| `felhom-controller` | `0887fd676d60` | — | **untouched** (read-only) |
| `felhom-agent` | `856a127` | — | **untouched** (read-only) |
**The chain from a dead node to an open repository is not assembled, and the failure is deeper than
"some links are manual".**
Clean-tree gate passed before the build: `git status --porcelain` empty, `HEAD == origin/main`.
Highest register ID on arrival was **R-194**; grepped before minting.
1. **Links 1–4 are PROVEN-LIVE** — mint → stage to the agent → seal under R → store on the hub.
2. **Link 5 is lossy, and this is the finding of the session.** `host_escrow_superseded` has **no
`identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`)
copies only the K-escrow `blob`. The **restic repository password lives in `identity_blob`** — so
the retention feature built for the reinstall-orphan incident does not retain the secret that
incident is about. The **next escrow ceremony destroys the last copy**, and that ceremony is exactly
what the system tells a rebuilt box's customer to run. Both demo boxes crossed that line on
2026-08-04 at 07:15:36 and 07:20:08. → **R-198**
3. **Link 6 has no client anywhere.** `handleReEnroll` / `handleGetRestoreDirective` serve the blobs;
zero callers in the agent, the hub UI, any script or any runbook. Retrieval is `sqlite3` by hand on
the hub pod. → **R-199**
4. **Links 7–8 are manual-only.** `UnwrapIdentityBundle`'s only production caller is
`--selftest=identity-consume`, reading R from `FELHOM_RECOVERY_CODE`; nothing extracts
`restic_repo_password` from the recovered bundle.
5. **Link 9's form does not exist.** `POST /backup/offbox/inject-password` is routed, handled and
unit-tested; **no template contains it.** → **R-200**
6. **Link 11 has never happened.** No blob served, no fork-4 bundle unsealed with a real R outside a
unit test, no recovered password injected, no repo reopened, no restore. The one live round-trip
proof (2026-06-10) **predates the `ResticRepoPassword` field by a month**. → **R-201**
---
**And the most useful thing settled: a fail-closed refusal IS implementable.** The hub already sends
every box `escrow{identity_blob_present, restic_pw_sha256, created_at}` on every report ACK
(`api/handler.go:504-510`), and the controller **discards it** whenever no offbox target exists
(`report/escrow_confirm.go:75-84`). Persisting it needs no new hub API and no new secret; the
`ClaimSync` set-only pattern is the precedent.
## 2. Q1–Q8 — each answered or explicitly marked unmeasured
**The operator's ruled recovery-screen shape is buildable** and is priced row by row in §9 of the
findings doc: fresh-install signal exists (the mint branch's own `os.Stat`); hub-has-a-blob exists on
the wire; the preview is cheap and genuinely read-only (`restic snapshots --json` +
`stats --mode raw-data --json` are already the box's own counters); the one real cost is crossing
agent→controller for the unseal, because the controller image ships no `age` — one new agent local-API
endpoint mirroring `/escrow/ceremony/claim`, plus a narrow hub `GET /hosts/<id>/escrow`.
Full method, quotes and query output: **`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md`**.
## 2. Corrections owed and made
### Q2 and Q3 first — they decide whether the rest matters
- **Yesterday's spike annotated, not rewritten** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`):
a dated block at the top plus an inline note on Q8(b). Its candidate (b) verdict *"not implementable
against a zero-knowledge escrow"* is **too strong** — unattended recovery is impossible; a
customer-present recovery is not. Everything else in that spike stands, and its Q3 remains unmeasured.
- **Capability map corrected** (`architecture/00-capability-map.md`, escrow-ceremony row): the hub
v0.60.0 retention claim *"a previous passphrase stays recoverable with its recovery code"* is struck
through and replaced with what the code does — K-escrow only, no read path, `ListSupersededEscrow`
has zero production callers.
- **The same false promise is made to the customer in Hungarian** on the orphan card
(`controller/internal/web/templates/backups_remote.html:66,69`). Recorded in R-198; **not edited** —
this session ships no controller change.
- **The prompt's Q4 premise is a conflation, and it is corrected in the doc:** `escrow_confirm.go`
**does** fire on a mint (pending branch, hash mismatch, one WARN). The "wired to the wrong path"
finding was about the hub's `escrow_stale` event (R-196), a different mechanism. What is missing is
not the detection but the consequence — the warning reaches a log file and prescribes the act that
closes the recovery window.
**Q2 — does a rebuilt controller reuse the escrowed repository password, or mint a new one?**
**RULING: it MINTS A NEW ONE.**
*Method — source:* `ConfigureOffbox` → `ApplyOffsiteTarget` → `WriteOffboxSecrets`, which generates a
fresh 256-bit password whenever `<DataDir>/offbox/repo_password` is absent
(`felhom-controller/controller/internal/backup/offbox.go:392-401`). A rebuilt guest has a fresh data
dir. The only recovery path, `InjectOffboxPassword`, has **exactly one caller in the entire repo**
(census: `grep -rn "InjectOffboxPassword" --include=*.go .` → 3 hits, 1 definition, 1 caller, 1
comment) — `web/offbox_handlers.go:189`, a web form a human pastes into. **The apply-bridge never
calls it.**
*Method — live, without touching a box:* `host_escrow.restic_pw_sha256` vs
`host_escrow_superseded.restic_pw_sha256`. **demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom
`48741892…` → `c60c8bc7…`. Both CHANGED.**
**This is bigger than R-193 as filed:** demo-felhom, which R-193 records as having survived by luck,
had **36 snapshots / 1,136,685,919 B** before its rebuild and has reported `snapshot_count: 0` in all
109 reports since. Its luck restored **delivery**, not the **repository**.
## 3. Register
**Q3 — what did the 04:15 run report?** **UNMEASURED, and the binary in the question is wrong.**
The run is daily at ~02:15 UTC (measured `last_run` values on both boxes). Neither box could run on
2026-08-04: demo-hp had no target at all until 07:15:47 UTC; demo-felhom's target was
`escrow_state: pending` from 08-03 07:19:10 to 08-04 07:20:13, which `OffboxRunnable` blocks
(configured **AND** escrowed). **The decisive run is 2026-08-05 ~02:15 UTC.** Source predicts a
**third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`;
`repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies
**orphaned** → `ErrOffboxOrphaned` → **the run refuses and shows the orphan card.** Recorded as a
prediction, not a measurement.
- **New: R-198, R-199, R-200, R-201** (highest previous ID was R-197; grepped across `OPEN-ITEMS.md`,
`ROADMAP.md`, `STATUS.md`, `CONTEXT.md` before minting).
- **Updated: R-193** (seven-part addendum: (b) overturned in part, the chain census, the retention
finding, fail-closed implementable, the operator ruling priced, the drill, Q3 still unmeasured) and
**R-192** (two inputs, no verdict change).
- **`STATUS.md`** updated in operator language: two new "what's broken" entries, three new
"waiting on you" items, a "changed since" line.
### The rest
## 4. Live measurements taken (read-only)
- **Q1 — what is one-shot.** Full table in the spike. Headline: the hub's one-time provider password
is one-shot **but freely re-issuable**; the box's SFTP key is **regenerated on every apply by
design**; the **restic repository password** is the one nothing can restage — the agent's own source
calls it *"irreplaceable (unlike the SFTP access key…)"*
(`felhom-agent/internal/escrow/identity.go:35-39`). **R-193 was filed against the recoverable
secret; the damage is in the irreplaceable one.**
- **Q4 — see §3 below.**
- **Q5 — why one box recovered and the other not. CONFIRMED FROM THE LEDGER.** demo-felhom's
`one_time_secrets` row was created `2026-07-21 08:29:29` (traceable to a Re-issue at that exact
timestamp) and sat **UNCONSUMED for 13 days** until the rebuild consumed it at
`2026-08-03 07:17:58`. Without one: exactly demo-hp — `consume` → **404** and a retry loop, 25 hours
unprotected. **Correction to the contrast:** the recovery was of the plumbing only (Q2).
- **Q6 — why delivery is one-shot.** The design's own reasons quoted from three sites. **The security
half, stated:** the consume endpoint is gated by `checkAuthCustomer` — **the customer's API key, the
same credential as the config pull** — so an automatic restage grants **no new access to an
unauthenticated party**. What it genuinely widens is the **exposure window**: a live provider
password becomes fetchable whenever the box reports no offsite target, rather than only in the
seconds after an operator acts. Blast radius stays the customer's own label-scoped sub-account. The
R-39(a) unconsumed-secret guard must survive any change.
- **Q7 — R-192's guard. BOTH HALVES CONFIRMED WITH NUMBERS.** See §4.
- **Q8 — what each candidate costs.** See §5.
Hub SQLite snapshot with the `-wal` (`kubectl exec … cat`), `PRAGMA integrity_check` → `ok`, freshness
proven positively (`MAX(host_reports.received_at)` = `2026-08-04 10:00:40` vs `datetime('now')` =
`10:03:48`). Escrow tables, `PRAGMA table_info` on both, latest-report `$.offsite.*` and `$.claimed`
per customer, the `claimed` 1→0→1 transition across both rebuilds, and the events table since
2026-08-03. Plus one `ls -la --time-style=full-iso` inside guest 9201 proving the data dir, the
`encryption.key` and `offbox/repo_password` are all newborn as of `2026-08-03 07:17:25`–`07:18:02`.
Every query with its field names and values is in §11 of the findings doc.
---
**Nothing was written, run, rotated, re-issued, deleted or reconfigured anywhere.** No ceremony, no
`selftest=identity-consume`, no touch on the storage endpoint.
## 3. Which of the two contradictory claims about Re-issue is true
## 5. What could not be established
**The claim that Re-issue "only resets the transport credential" is TRUE about what
`ReissueCredentials` does. R-39's "rotates the restic password" is FALSE.**
The 2026-08-05 ~02:15 UTC run's outcome (still the decisive measurement; neither box has run —
`last_run` absent on both); whether demo-hp's data dir was fresh (measured only on demo-felhom;
demo-hp needs the break-glass credential this read-only session did not use); whether the orphaned
ciphertext still exists at the provider; whether `age` is present on either demo host. Named in §12
of the findings doc.
From source (`hub/internal/offsite/offsite.go:150-228`), the function does four things: `genPassword`,
`ResetSubaccountPassword`/`ResetBoxPassword`, `SaveOneTimeSecret`, and bookkeeping. **There is no
reference to a restic password anywhere in it, and none is possible** — that password is generated on
the box and never leaves it except into the R-wrapped escrow, which the hub cannot open.
## 6. Observations recorded, not acted on
**But the bookkeeping is justified by the false premise**, `offsite.go:198-201`: *"the restic repo
password just changed, so any existing key-escrow blob … is now STALE"*. The same wrong claim appears
at `api/handler.go:1067-1069` and in R-193's own row. **Eighth entry in `CLAUDE.md`'s table of
comments asserting a guarantee the code does not provide** — the first where the comment factually
describes a *different function*. → **R-196**.
A fresh `encryption.key` on every rebuild makes every pre-rebuild `app.yaml` `ENC:` secret
undecryptable — circular dependency **C-1** of `_recovery-inventory-2026-07-28.md`, which fired live on
both demo boxes this week and is a strictly larger blast radius than the offsite repo. `Claimed`
returning to 1 with no event means "the customer claimed this box" has no audit trail.
`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting. `handleReEnroll` returns a
freshly minted API key in its response body on an endpoint nothing currently calls.
**What corroborated it, and the corroboration is the strongest part.** The task asked: if the repo
password was untouched, what explains demo-hp's escrow going `pending` after the Re-issue? **Fully
explained, and not by the Re-issue:** `ApplyOffsiteTarget` carries `EscrowState` over from the
*existing* target, the rebuilt guest had none, so the `pending` fallthrough fired — and the box had
just minted a new repo password, so `pending` was *correct for a different reason*.
**The proof they are independent is demo-felhom:** it received **no Re-issue at all** (its only
`escrow_stale`/`offsite_reissued` pair is dated `2026-07-21 08:29:29`), its repo password changed
anyway, and **nothing marked its escrow stale for 13 hours.** The mechanism is exactly inverted:
`escrow_stale` is wired to the one path that does **not** change the repo password and absent from the
path that does.
## 7. Conventions
---
## 4. Every query used, with its fields and values
The DB snapshot was taken from the live pod and its **freshness proven with a positive observable**,
not with "the query returned no error":
| # | Query / read | Result |
|---|---|---|
| 1 | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale (mtime 08:34 vs wal 10:45 CEST). Copying only the main file is the R-3 shape. |
| 2 | `PRAGMA integrity_check` | `ok` |
| 3 | `MAX(host_reports.received_at)` vs `datetime('now')` | `2026-08-04 08:45:38` vs `08:49:46` — **newest row 4 min old** |
| 4 | `SELECT customer_id, status, created_at FROM customer_configs` | `demo-felhom` active · `peti-felhom` active · `demo-hp` active · `drill-r50` **blocked** · `david` active (created `2026-08-01 16:51:49`). **`sess-f` is gone — confirmed, not assumed.** `drill-r50` untouched. |
| 5 | `hosts` / `host_deletions` / `host_reports` / `reports` counts for `david` | **0 / 0 / 0 / 0** — never bound |
| 6 | `events WHERE event_type='expected_dbdump_missed' AND created_at > '2026-07-25'` | **only `david`**, 03:00 UTC on 08-02, 08-03, 08-04 |
| 7 | `SELECT customer_id, MAX(received_at), COUNT(*) FROM reports GROUP BY customer_id` | `demo-hp` 1329 · `demo-felhom` 6690 · `drill-r50` 183 (last 07-29) · `peti-felhom` 482 (last **2026-07-15**) — the reason peti-felhom is `down` and does not alarm |
| 8 | distinct `$.offsite` objects per box with first/last seen | demo-hp: `15 snapshots / 40 927 044 B` until `08-03 05:59:21`, **ABSENT** `06:12:19`→`08-04 07:12:01` (111 reports), then `pending` 07:15:47, `escrowed` 07:16:02, `snapshot_count 0`. demo-felhom: `36 snapshots / 1 136 685 919 B` until `08-03 07:01:28`, ABSENT ×1, then `pending` `07:19:10`→`08-04 07:20:13`, `escrowed` 07:20:28, `snapshot_count 0` |
| 9 | `host_escrow` + `host_escrow_superseded` `restic_pw_sha256` | **the decisive measurement** — both changed (§2) |
| 10 | `one_time_secrets` (no values read) | demo-felhom created `2026-07-21 08:29:29`, consumed `2026-08-03 07:17:58`; demo-hp created+consumed `2026-08-04 07:11:51`/`07:12:06` |
| 11 | `$.claimed` on the latest report of each box | **`1` on both** — which selects `ensureOffboxRepo`'s refuse branch |
| 12 | `CountReportsOffsiteSince` reproduced verbatim (`ORDER BY id LIMIT 500`) for demo-hp @ `2026-07-23 09:53:41` | **`total=500, withOffsite=500`**, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** |
| 13 | same window, **no LIMIT** | `total=1174, withOffsite=1063` (⇒ 111 without — matches #8) |
| 14 | full text of the live `offsite_delivery_stuck` message | *"…and **500 report(s) since carry no offbox target**…"* while `OffsiteReportsSinceConsume` was **500** |
| 15 | `COUNT(*)` of `offsite_credential_restaged` | **0, for every customer, ever** |
| 16 | the shipped predicate run verbatim against live data (post-deploy) | `david` → **0**; `demo-felhom`/`demo-hp`/`drill-r50`/`peti-felhom` → **1** |
**Instrument limits recorded rather than discovered later:** `one_time_secrets` has `customer_id` as
PRIMARY KEY and is last-write-wins **by design**, so it holds only the newest credential per customer —
demo-hp's `2026-07-23` consume, on which R-192's whole diagnosis rests, was **overwritten** by
yesterday's Re-issue and is no longer readable from that table. The `2026-07-23 09:53:41` anchor in
query #12 comes from R-192's record, and that is stated in the spike rather than presented as a live
read.
---
## 5. Ranked options, with a recommendation
| rank | option | verdict |
|---|---|---|
| **1** | **(d) the honesty pass — now** | Cheap, weakens nothing, and **the only option that helps the incident already in flight**. Two parts: correct R-192's message + log its refusal (**R-196**), and **detect a changed `restic_pw_sha256`** — both hashes are already in the hub's DB and **nothing reads them** (**R-197**). |
| **2** | **(c) agent-retained repo password — NOT PREVIOUSLY NAMED** | The only candidate aimed at the irreplaceable secret. The agent lives on the Proxmox host, **survives a guest rebuild**, already receives the password over the pinned local API and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Every seam exists. Price: a copy of the data key at rest on the host. **The operator's trade to make.** |
| **3** | **(a) hub restages automatically** | **Already exists** as `reissueOnReenroll`'s F3 leg, wired to the wrong event (behind `handleHostEnroll`'s mint-once-reuse short-circuit; a guest rebuild leaves the `hosts` row intact). Worth doing **after** (c), never instead: on its own it restores the transport to a box whose data key is already gone, and **would have made both boxes look healthy on 2026-08-04 while their snapshots were orphaned** — strictly worse than the current loud failure. |
| **4** | **(b) recover from escrow at re-bootstrap** | **Not implementable as stated.** The escrow is R-wrapped and zero-knowledge; the hub has no recovery code. "Recoverable at re-bootstrap" describes a customer-present ceremony — i.e. the manual form that already exists. |
**Recommendation: ship (d) now; then decide (c). Do not ship (a) first.**
### The operator's question — stated, and NOT answered
> The irreplaceable secret is the restic repository password. It is generated on the box, destroyed by
> a guest rebuild, and the only surviving copy is sealed under the customer's recovery code. Every
> machine will be reinstalled.
> **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host,
> where you already have root — or do you want it to stay sealed and accept that every rebuild starts
> a new repository, provided the system says so loudly on the day it happens?**
**STOPPED here. Neither candidate was begun.**
---
## 6. Part 4 — the phantom-customer alarm (SHIPPED)
**Files changed**
- `hub/internal/store/store.go` — **new** `(*Store).HasEverBoundHost(customerID) (bool, error)`:
a live `hosts` row **OR** a `host_deletions` tombstone.
- `hub/internal/monitor/deadline.go` — one guard at the top of `CheckBackupDeadlines`' customer loop,
plus an `unbound` counter in the summary line.
- `hub/internal/monitor/deadline_unbound_test.go` — **new**, 5 tests.
**Commit:** `7fff45d688fa` (code + tests + spike + docs) · `f581ac134964` (manifest bump).
**The mechanism, established at source and corroborated live.** The existing down-skip could never
have covered this: it reads `StalenessChecker.GetState()`, whose map is seeded from
`store.GetCustomers()` — **a query over the `reports` table** (`store.go:1055-1067`). A customer with
zero reports is in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. **The
skip misses exactly the customer it would most obviously cover.** Corroborated by query #7 and by the
live post-deploy startup log — `Staleness checker initialized: 2 ok, 0 stale, 1 down` — where the
`1 down` is `peti-felhom` (host deleted 2026-07-15, 482 old reports) and `david` is **absent from the
census entirely**.
**The discriminator is "was a host EVER bound", not "has a report arrived"**, which is the point:
a box that was installed, bound and went silent has a `hosts` row and keeps alarming. **Fail-open on a
read error** — an unreadable binding must never suppress a real alarm. The deferral is **logged with
its own counter** (the v0.73.0 Part-7 precedent). The anchored-verdict structure is untouched, and
`david`'s record was not modified.
**Sibling verdicts checked rather than assumed.** The census of every customer-iterating checker:
`deadline.go` backup half (already guarded by `reportJSON == ""`), `restoretest.go` (guarded by
`latest == ""`), `staleness.go` + `offsite.go` (iterate `GetCustomers()`, report-derived — a hostless
customer never appears), `offsite_delivery.go` (a hostless customer derives `DeliveryNoSecret` and is
skipped). **The DB-dump half was the only verdict that could fire**; the guard covers both verdicts in
that function uniformly.
**Test results — the suite run and the exit code read BEFORE any commit, never in one command.**
```
go build ./... → BUILD OK
go vet ./... → VET OK
go test ./... → rc=0, 17 packages ok, no FAIL lines
```
The 13 `TestCheckBackupDeadlines*`/`TestHasEverBoundHost` tests were run with `-v` and **the `=== RUN`
lines were checked**, per `CLAUDE.md`'s rule that a `-run` pattern matching nothing prints `ok` and
exits 0.
**THE RED-PROOF THAT MATTERS.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms` is the
case the change could break — a real box that was installed and never phoned home, indistinguishable
from `david` on every signal *except* the one the guard reads. With the guard removed it **still
passes**, which is the correct result and proves the test is aimed at the guard's *risk*, not its
presence. The companion red-proof of the fix itself, **observed**:
```
=== RUN TestCheckBackupDeadlines_NeverBoundHost_Silent
deadline_unbound_test.go:58: a customer with NO host ever bound must raise NOTHING;
got [expected_dbdump_missed]
--- FAIL: TestCheckBackupDeadlines_NeverBoundHost_Silent
```
— verbatim the event `david` sent three mornings running. `deadline.go` restored from a backup taken
before the mutation; full suite re-run green afterwards.
**Deployed version.** `felhom-hub:0.92.0` built + pushed (`sha256:ca71140cf54a…`),
`manifests/hub.yaml` bumped `0.91.1 → 0.92.0` in git, ArgoCD hard-refresh + **deliberate sync** (never
`kubectl set image`). Verified: app **Synced / Healthy**, `rollout status` → `deployment "hub"
successfully rolled out` (rc=0), running image `gitea.dooplex.hu/admin/felhom-hub:0.92.0`, pod
`hub-6775889d95-gc8sk` 1/1 Running, clean startup log.
**Is the 05:00 alarm confirmed silent? NO — CONFIRMATION IS PENDING, and it is stated rather than
assumed.** The hub's own log says `deadline-check: next run at 2026-08-05 05:00 CEST (in 17h53m57s)`;
there is no manual trigger (`CheckBackupDeadlines` has exactly one caller, the scheduler at
`cmd/hub/main.go:598`). An absent alarm tomorrow would in any case be weak evidence, so a **positive**
observable was taken instead: the shipped predicate was run verbatim against a fresh copy of the live
DB (query #16) — `david` → **0** (will be skipped), every other customer → **1** (behaviour
unchanged). **The 05:00 firing itself is still owed.**
---
## 7. `OPEN-ITEMS.md` rows, and the capability map
- **R-192** — updated with Q7's confirmed numbers; **NOT closed**, and now carries a *new* reason not
to fix it in isolation (a successful auto-restage would have protected the plumbing while the data
was already gone).
- **R-193** — updated with the whole spike; **NOT closed**, state moved to **WAITING-ON-OPERATOR**,
owner `CC + operator`. The old "15+ vs 1 decides it" line is struck through and replaced with the
measured answer plus the third predicted outcome.
- **R-195** *(new)* — the phantom-customer alarm. **SHIPPED**, hub v0.92.0.
- **R-196** *(new)* — `escrow_stale` wired to the path that does not change the repo password, absent
from the one that does. **OPEN.**
- **R-197** *(new)* — the hub holds both `restic_pw_sha256` values and reads neither. **OPEN.**
**The capability map DID need touching, and the §N.5 escalation did NOT fire — the row was not
overclaiming.** `00-capability-map.md`'s offsite row already recorded reinstall-continuity: *"a
recreated data volume that orphaned the repo … is now CLASSIFIED … + a move-aside (never-delete)
reset"*, shipped controller v0.142.0 and live-fired 2026-07-18. **That independently corroborates the
Q3 prediction from a different direction.** What it did not record — and now does — is that this fires
on an **ordinary, planned, unattended guest rebuild** on every box, with the measured 2026-08-03/04
instance, the hash evidence, and the inverted `escrow_stale` wiring. **Status unchanged
(PROVEN-LIVE):** the guard is correct and is not the gap; the cadence is.
`ROADMAP.md`'s newest row is R-171 and none of R-190…R-197 has one, so **there was nothing to
collapse.** One correction *was* warranted under the coupling rule: R-1's row claimed the v0.57.0
reinstall arc makes offsite re-issue *"auto-fire on re-enrollment"* — scope-corrected in place, since
that means **host** re-enrollment after a host **delete**, and a guest rebuild never reaches it.
`CONTEXT.md` gained **S-26** (the spike's Q1/Q2 headlines + the three corrections that must not be
re-inherited) and **S-27** (the R-195 ruling and its mechanism). `REUSE.md` gained the
`HasEverBoundHost` row with its "never a substitute for a liveness check" trap. `STATUS.md` rewritten
for the operator, one screen.
---
## 8. CI, task ids, and `--no-verify`
| commit | what | CI task id | run # | conclusion |
|---|---|---|---|---|
| `7fff45d688fa` | R-195 code + tests + the spike + all docs | **126** | 72 | **success** |
| `f581ac134964` | `manifests/hub.yaml` 0.91.1 → 0.92.0 | **127** | 73 | **success** |
| `b7c5cfc06188` | this report + `STATUS.md` + the ROADMAP scope correction | **128** | 74 | **success** |
*(A report cannot contain the run id of the commit that adds it; the docs-only follow-up that appended
run 74 to this table is reported in the session summary.)*
Matched by `head_sha` via
`curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/felhom.eu/actions/tasks?limit=5"` — a PULL
check, not a reliance on the failure mail.
**`--no-verify` was NOT used.** The pre-push hook ran and reported `gates OK - push proceeding`;
`python3 scripts/repo_gates.py` was additionally run in full (all five gates OK) before committing.
**Teardown:** the spike provisioned nothing — no guest, no VM, no storage, no credential. The only
artefact is a hub-DB snapshot in the session scratchpad, outside the repo.
**Read-only discipline held:** no `ReissueCredentials`, no ceremony, no credential rotation, no
change to any box's offsite configuration, no snapshot deleted, no customer record modified.
`drill-r50` untouched; `sess-f`'s removal **confirmed** rather than assumed.
---
## 9. Observations — noticed and NOT acted on
1. **The orphaned ciphertext is retained and keeps consuming quota, with nothing to prune it.**
`resetOrphanedRepo` is move-aside-never-delete by design (correct), but 40.9 MB + 1.14 GB of
unopenable data will sit against a 50 GB soft quota indefinitely. S-24's ep0 prune jobs cover the
**PBS** namespaces, a different tier. Not filed — it belongs to whichever option is chosen for
R-193.
2. **`host_escrow_superseded` holds only two rows in the entire database**, both from 2026-08-04. That
is why the spike could not establish whether demo-felhom's 2026-07-21 Re-issue re-sealed an
*unchanged* password — the predicted false-staleness shape of R-196. Retention policy for that
table was not investigated.
3. **Both demo guests were rebuilt on consecutive mornings** (demo-hp 08-03 06:12, demo-felhom 08-03
07:17, both to controller 0.192.0 with new config hashes). *Why* is outside this spike's scope and
was not chased.
4. **`api/handler.go:1067-1069`'s comment is the third copy of R-196's false premise**, and it sits in
a different package from the other two. Whoever fixes R-196 should grep rather than fix the one
they were shown.
5. **`peti-felhom` is `active` with its host deleted since 2026-07-15**, judged only by the `down`
skip. It is correct today and would start alarming the moment it reported once. Not a defect, but
the shape is worth knowing.
6. **The `-shm`/`-wal` trap generalises.** Any future session reading the hub DB from the pod must
copy all three files. Reading `hub.db` alone today would have produced confident answers from a
file 2 h 15 m stale — the R-3 shape with a different filename.
- **No `CHANGELOG.md` entry.** `felhom.eu` has no root changelog — they are per-area (`hub/`,
`scripts/`, `website/`) and log code changes. This session changed none of those areas. Stated
rather than silently skipped.
- **`REUSE.md`** untouched — no shared helper or pattern moved.
- **Gates + CI run id** are recorded in the session summary.
</content>
STATUS.md
+50
View File
@@ -35,6 +35,25 @@ Proven end to end on real hardware.
**The next scheduled off-site run is tomorrow at 04:15, and it will refuse rather than quietly start
a new history** — the machine already knows how to recognise this and asks before resetting.
**There is a decision here for you — see "Waiting on you".** *(R-193, R-192, and two new items)*
- **And the safety net we believed was under all of this is not there.** We traced the whole path today —
from a dead machine to reopened backups — and it does not join up. The important part is not that
several steps are manual; it is this: **the central hub keeps the old sealed key when a machine
re-seals, and we found it keeps the wrong one.** It keeps the key for the whole-machine backups and
**not** the key for the off-site file backups — the one this entire problem is about. So the moment
a rebuilt machine is asked to make a new recovery code, which is exactly what it asks the customer to
do, **the last copy of the old key is gone.** Both demo machines crossed that line yesterday morning.
**This means the 51 orphaned backups were beyond reach even if you had kept the recovery codes** —
which is a relief in one narrow sense and much worse in every other. It also means the customer is
currently told, in Hungarian on their own screen, that their old backups "may later be restorable
with the matching recovery code". That sentence is not true today. It is a small fix — one missing
column — and it must land before anything else here. *(R-198)*
- **Nothing in the recovery path has ever been performed.** Making a recovery code is proven, live,
with a real customer. **Using one is not.** No sealed package has ever been handed back to a machine
(the hub has the buttons for it; nothing anywhere presses them), no recovered key has ever been put
back, no old backup store has ever been reopened, and no file has ever been restored from a recovered
key. There is a form missing, a step missing, and a connection missing between two parts of the
system. We designed the exercise that would prove it end to end — see "Waiting on you".
*(R-199, R-200, R-201)*
- *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
@@ -106,6 +125,32 @@ Proven end to end on real hardware.
nothing else, and it is the piece whose absence let a gigabyte of history go quiet for thirteen
hours. What we are asking you to choose is (a). *(R-193; the full reasoning is in the findings
document)*
**Correction to (b), from today:** "old backups are kept" is true, but "openable later with the
recovery code" is **not** — see R-198 above. Option (b) is only honest once that is fixed.
- **Your ruling about the recovery screen has been priced, and it can be built.** You said: a freshly
installed machine that finds a sealed package waiting should say so loudly, offer the customer a box
to type their recovery code into, and show what would come back before doing anything. **All of it is
buildable, and one part is already free** — the hub is *already* telling every machine, on every
check-in, that a sealed package exists and which key it covers, and the machine currently throws that
message away. Showing a preview is also cheap: listing what is in an off-site store reads it without
writing to it, so the customer can see how many backups, from when, and for which apps before
committing. The real work is one new connection: the unsealing has to happen in the part that runs on
the Proxmox host, because the customer-facing part cannot do it — the same crossing your recovery-code
ceremony already makes in the other direction. **One thing for you to weigh, which we deliberately did
not decide:** a screen that takes a recovery code and then shows what is in a backup store is reachable
by anyone with the household's dashboard password, and the preview reveals backup dates and app names.
*(R-193)*
- **Should we run the proof?** Nothing about recovery has ever actually been done, so we designed the
exercise: on the spare demo machine, make a recovery code and **keep it**, put a marked file into an
app, back it up off-site, wipe the machine, reinstall, recover with the saved code, and check the
marked file comes back byte for byte. About half a day, of which under an hour needs you. The failure
we are watching for is precise: **if the recovered machine reports one backup instead of the ones we
put there, it started a fresh history and the proof failed** — "the store opened" is not good enough.
It costs that machine's contents if it goes wrong, which is acceptable there. **Do R-198 first**, or
the exercise walks a path that is missing a piece. *(R-201)*
- **The orphaned backups still on the storage box.** About 1.2 GB across the two demo machines, sitting
in set-aside stores nobody can open and nothing prunes, using the 50 GB allowance. We did not touch
them. **Delete, or leave?** *(R-193)*
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
session log; nothing suggests anyone else saw it. *(R-132)*
- **One small question, not urgent.** The automatic check cannot see which version you have told
@@ -115,6 +160,11 @@ Proven end to end on real hardware.
## Changed since last update
- **2026-08-04 (latest)** — Traced the whole recovery path end to end and it does not join up. The hub
keeps the wrong sealed key when a machine re-seals, so a rebuilt machine's old off-site key is lost the
moment the customer is asked for a new recovery code. Nothing in the recovery path has ever been
performed. Your ruling about the recovery screen is priced and buildable. The proof exercise is
designed and awaiting your go-ahead. *(R-198, R-199, R-200, R-201, R-193)*
- **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage
password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost
access to their off-site history, one of them silently. A decision is now waiting on you. The daily
documentation/architecture/00-capability-map.md
+1 -1
View File
File diff suppressed because one or more lines are too long
documentation/audits/RECON-offsite-dr-chain-2026-08-04.md
+531
View File
@@ -0,0 +1,531 @@
# RECON — can a customer actually get their backups back?
**Date:** 2026-08-04 · **Items:** R-193, R-192 (+ new: R-198…R-201) · **Class:** recon — **no production code**
**Session scope:** read-only everywhere. Nothing built, deployed, wiped, rotated, re-issued or deleted.
> **Headline, before the detail.** The chain is **not** assembled, and the reason is worse than "two
> links are manual". **The hub's superseded-escrow retention — the feature built for exactly this
> incident — does not retain the offsite repository password.** `host_escrow_superseded` has no
> `identity_blob` column, and `demoteCurrentEscrowTx` copies only the K-escrow blob. So the last copy
> of a repo password is destroyed by the **next escrow ceremony** — the very act the system tells the
> customer to perform when their escrow goes `pending` after a rebuild. The recovery window closes
> because the remedy closes it.
---
## 1. Confirmed baselines (re-read on arrival)
| Repo | `main` @ commit | tree | Version |
|------|-----------------|------|---------|
| `felhom-agent` | `856a127cd60b0be797867c162f0ae49c7b64993e` | clean | v0.124.1 |
| `felhom-controller` | `0887fd676d608e7c103fa13b3e87536ad2481811` | clean | v0.194.0 |
| `felhom.eu` | `d26f49ad680e2be2ce2888fd58dff7dee886e4ad` | clean | hub v0.92.0 |
All three match the prompt's cited commits exactly. `git status --porcelain` empty in each.
**Highest register ID: R-197** (`OPEN-ITEMS.md` R-196/R-197 were minted by yesterday's spike;
R-194/R-195 by the R-190 arc). Grepped across `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`,
`CONTEXT.md`. New items in this session therefore start at **R-198**.
---
## 2. Instruments — what each can and cannot answer
| # | Instrument | Exact read | Known limit |
|---|---|---|---|
| I1 | source | the three repos at the commits above | the authority; every claim below cites a file:line |
| I2 | hub SQLite snapshot | `kubectl -n felhom-system exec pod/hub-6775889d95-gc8sk -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | the `-wal` is load-bearing (copied; `PRAGMA integrity_check` → `ok`) |
| I2-fresh | freshness proof | `MAX(host_reports.received_at)` = `2026-08-04 10:00:40`, `datetime('now')` = `2026-08-04 10:03:48` | **positive observable**: newest row 3 m 08 s old, not "the query did not error" |
| I3 | live filesystem, demo-felhom guest 9201 | `ssh felhom-pve "pct exec 9201 -- ls -la --time-style=full-iso …"` | read-only `ls`; mtimes only, no file contents read |
| I4 | caller census | `grep -rn <symbol> --include=*.go` across all four repos | proves absence of a caller in Go; templates checked separately |
**Not used, deliberately:** no ceremony, no re-issue, no rotation, no `selftest=identity-consume`, no
write of any kind, nothing touched on the storage endpoint.
---
## 3. Q1 — the chain, link by link
From a dead node to an open repository. **Status vocabulary as specified.**
| # | Link | Component / function | Who invokes it | Status |
|---|---|---|---|---|
| 1 | repo password is minted | controller `Manager.WriteOffboxSecrets` → mint at `internal/backup/offbox.go:392-401`; generator `:405` | `ApplyOffsiteTarget` (`offbox.go:474`) ← `ConfigureOffbox` ← the apply-bridge (`offsiteapply.go:221,256`) | **PROVEN-LIVE** — fires on every fresh data dir; measured twice this week (I2 hash change, I3 mtime) |
| 2 | password staged to the agent | controller `PushOffboxPasswordForEscrow` (`offbox.go:530`) → agent `POST /escrow/stage-secret` (`localapi/server.go:518`) → `escrow.StagedResticPasswordPath()` = `/var/lib/felhom-agent/escrow-stage/restic_repo_password`, 0600 (`identity.go:44-47`) | the controller enable/ceremony path | **PROVEN-LIVE** |
| 3 | sealed under R at the ceremony | `AttachResticPassword` (`identity.go:73`) → `WrapIdentityBundle` → `WrapIdentity` (`identity.go:114`, `age -p -a`, scrypt + ChaCha20-Poly1305) | agent `--selftest=escrow-create`, driven by the controller wizard (`/escrow/ceremony`, `localapi/server.go:525-527`) | **PROVEN-LIVE** — customer-facing wizard first live firing 2026-07-18 (`00-capability-map.md`, escrow row) |
| 4 | stored by the hub | `PUT /api/v1/hosts/<id>/escrow` (`api/handler.go:239`, `handleHostEscrowPut :1104`) → `host_escrow.identity_blob` | agent `main.go:2937` | **PROVEN-LIVE** — measured: `identity_blob` present, 572 B, for **both** hosts (I2) |
| 5 | **old blob retained on supersession** | `demoteCurrentEscrowTx` (`store/store.go:2547-2556`) → `host_escrow_superseded` | `SaveHostEscrow`, `DeleteHost` | **BROKEN FOR THIS PURPOSE.** The INSERT copies `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256` — **not `identity_blob`**, and the table has **no such column** (`PRAGMA table_info`, I2). The K-escrow survives; **the restic repo password does not** |
| 6 | served back to a re-enrolling box | hub `handleReEnroll` (`api/dr.go:101`) and `handleGetRestoreDirective` (`:155`), returning `identity_escrow_b64`; recovery-mode gated, armed with the **global operator key** (`:34`) | **NOBODY.** I4: zero callers in `felhom-agent` (no `ReEnroll` symbol, no `/re-enroll` or `/restore-directive` string anywhere in the agent's hub client, `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`); zero in the hub UI (no template mentions them); zero in `scripts/` or any runbook | **implemented-never-exercised, and CLIENTLESS.** The only documented retrieval is by hand: `sqlite3 "SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …"` on a `kubectl cp`-ed `hub.db` (project memory, 2026-07-04 S5 prep) |
| 7 | unsealed with R | `UnwrapIdentityBundle` (`identity.go:172`) → `UnwrapIdentity` (`:139`) | **`runSelftestIdentityConsume` only** (`cmd/felhom-agent/main.go:2845-2900`), reading R from `FELHOM_RECOVERY_CODE`. I4: no other production caller | **manual-only — the only caller is a `--selftest` mode** |
| 8 | the repo password extracted | none. The selftest writes the **whole bundle JSON** to `-keydest` 0600 (`main.go:2872`) and prints *"identity recovered (tunnel_token + pbs_token)"* — it does not mention, extract or route `restic_repo_password` at all | a human reads the JSON | **missing** |
| 9 | placed on the rebuilt controller | `InjectOffboxPassword` (`offbox.go:541`) | one caller: `offboxInjectPasswordHandler` (`web/offbox_handlers.go:189`) at `POST /backup/offbox/inject-password` (`web/server.go:510`). **I4: no template in the repo contains that path or any form posting to it** | **manual-only, and UI-unreachable** — the handler exists, the form does not. A person must POST it by hand with a valid session cookie + CSRF token |
| 10 | the existing repo opens | `ensureOffboxRepo` (`offbox.go:653`): `restic cat config` → success clears `RepoState` | the scheduled/manual offbox run | **implemented-never-exercised** on a *recovered* password (exercised constantly on a live one) |
| 11 | a backup or restore proves it | — | — | **never** |
### The chain in one sentence
Links 1–4 are real and proven. **Link 5 silently drops the payload.** Links 6–9 are each individually
manual, and link 6 has no client at all. Link 11 has never happened.
---
## 4. Q2 — where the chain stops today
**The first link that is not automatic is #5, and it is not merely manual — it is lossy.**
Two distinct stopping points, and the order matters:
**(a) If the box has NOT re-escrowed since the rebuild** — the current `host_escrow.identity_blob`
still seals the *old* password. Recovery is possible in principle, and a person must:
1. Arm recovery mode on the hub (`PUT /api/v1/admin/hosts/<id>/recovery-mode`, global operator key) —
or skip it entirely, because nothing consumes the endpoint it gates;
2. `kubectl cp` / `exec cat` the hub DB and `sqlite3 … writefile('/root/idblob', identity_blob)`;
3. copy the blob to a machine with the agent binary and `age`;
4. `FELHOM_RECOVERY_CODE=… felhom-agent --selftest=identity-consume -blob /root/idblob -keydest /root/bundle.json`;
5. open `bundle.json` by hand and read `restic_repo_password` out of it;
6. log in to the customer's dashboard and `POST /backup/offbox/inject-password` **by hand** (curl with
session + CSRF) — there is no form;
7. re-run the apply so `WriteOffboxSecrets` finds the file present and keeps it;
8. trigger a run and hope `restic cat config` succeeds.
Eight manual steps, five of them requiring operator-tier access to the hub, one requiring the customer
to be present with R, and one (step 6) with no user interface at all.
**(b) If the box HAS re-escrowed** — which is what the system pushes the customer to do, because a
rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`,
`offbox.go:570`), and the card says *„Helyreállítási kód szükséges"* — then the old `identity_blob`
has been **overwritten** by `SaveHostEscrow` and **not carried** into the superseded row. At that point
**no procedure exists**, with or without R, at any level of effort. The password is gone.
Both demo boxes are in state (b) as of 2026-08-04 07:15:36 (demo-hp) and 07:20:08 (demo-felhom) — I2,
`host_escrow.updated_at`. This is an **independent, and much stronger, reason** the 51 orphaned
snapshots are unrecoverable than "nobody kept the recovery codes". Keeping R would not have helped.
---
## 5. Q3 — can a rebuilt controller know it should recover?
### RULING: **YES — the hub already tells it, on every single report, and the controller throws the answer away.**
**What the box knows at the mint decision.** `WriteOffboxSecrets` is called from
`ApplyOffsiteTarget`, driven by the apply-bridge with the hub's offsite **descriptor**, whose fields
are `host, user, port, repo_path, host_fingerprint, quota_gb` (`offsiteapply.go:198-204`). **Nothing
about escrow is in scope at that point.** Locally, the only signal is the one the mint branch already
reads: `os.Stat(m.offboxPwPath())` returning `IsNotExist` — which *is* the fresh-data-dir signal, at
exactly the right line, but cannot distinguish "never configured" from "rebuilt".
**What the box is nonetheless already told.** Every report ACK carries an `escrow` object
(`api/handler.go:504-510`), consumed as `report.EscrowStatus`
(`internal/report/escrow_confirm.go:19-23`):
```go
type EscrowStatus struct {
IdentityBlobPresent bool `json:"identity_blob_present"`
ResticPwSHA256 string `json:"restic_pw_sha256"`
CreatedAt string `json:"created_at"`
}
```
That is precisely *"the hub has a blob, and here is the hash of the password it covers"* — arriving
every report cycle, on a channel that already exists, with no new API and no new secret.
**And it is discarded.** `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:75-84`) returns
immediately unless `Pending()` or `Escrowed()` — both of which require `OffboxConfigured()` **and** a
non-nil target (`cmd/controller/main.go:580-590`). On a freshly rebuilt box, before the offsite apply,
there is no target, so the status is dropped on the floor. It is never persisted anywhere: I4 finds
exactly **one** reference to the ACK field in the whole controller
(`cmd/controller/main.go:641: escrowConfirmer.Reconcile(resp.Escrow)`).
**Therefore a fail-closed refusal IS implementable**, with no new hub endpoint:
- persist the last-seen ACK escrow status alongside settings (the precedent exists — `ClaimSync`
caches the hub's claim-code state into `settings.json` idempotently by generation,
`internal/report/claim_sync.go:39-53`, and it does so *set-only, never cleared on hub silence*,
which is the correct shape here too);
- at the mint branch, refuse when the password file is absent **and** the cached status says
`identity_blob_present && restic_pw_sha256 != ""`.
**One ordering caveat, stated rather than assumed.** On a truly fresh box the apply can race the first
ACK. Measured on demo-hp: first post-rebuild report `2026-08-03 06:12:19`, offsite apply
`2026-08-04 07:12` — the ACK preceded the apply by 25 hours, but that gap was an incident, not a
design. A fail-closed rule handles the race correctly by construction: refuse to mint until the escrow
status is known, and retry next cycle (the apply-bridge is already built to retry and to persist
nothing on a partial failure, `offsiteapply.go:1-6`).
**Corroborating measurement that the data dir really is fresh on a rebuild** (I3, demo-felhom guest
9201, `/var/lib/docker/volumes/felhom-controller-data/_data/data/`):
| file | mtime |
|---|---|
| the data directory's parent | `2026-08-03 07:17:25` |
| `encryption.key` (32 B) | `2026-08-03 07:17:26` |
| `offbox/` | `2026-08-03 07:18:02` |
| `offbox/repo_password` (64 B) | `2026-08-03 07:18:02` |
The volume, the app-secret encryption key and the repo password are all newborn, 36 seconds apart.
Corroborated hub-side (I2): `json_extract(report_json,'$.claimed')` reads **1 → 0 → 1** across the
rebuild on both boxes — demo-hp `0` in 5 reports `2026-08-03 06:12:19`–`06:21:03`; demo-felhom `0` in 4
reports `07:17:54`–`07:19:46`. `Claimed` is set only by `SetClaimed()`, whose sole caller is the claim
submit handler (`web/claim.go:349`), so the return to `1` was a **human re-claim**, not automation.
> **Note, not acted on:** a fresh `encryption.key` means every pre-rebuild `app.yaml` `ENC:` secret is
> undecryptable — the C-1 circular dependency in `_recovery-inventory-2026-07-28.md` fired on both
> demo boxes this week. That is outside R-193's scope and is recorded as an observation (§13).
---
## 6. Q4 — what the stale-escrow detector actually covers
### RULING: **the prompt's premise conflates two different mechanisms. `escrow_confirm.go` DOES fire on a mint. But it fires into a log file, and the remedy it prescribes destroys the last copy of the old key.**
**The conflation, stated because it matters.** Yesterday's spike's sentence *"wired to the one path
that doesn't change the key and absent from the one that does"* is about the **hub's `escrow_stale`
event**, emitted by `ReissueCredentials` (`hub/internal/offsite/offsite.go:198-201`) — that is R-196,
and it stands. It is **not** about `controller/internal/report/escrow_confirm.go`. Two different
detectors; only one of them was ever claimed to be misplaced.
**What `escrow_confirm.go` does on a mint, from source.** After a rebuild the apply creates a target
with `EscrowState: "pending"` (the `cur == nil` fallthrough, `offbox.go:482-494`). So `Pending()` is
true, the pending branch runs (`escrow_confirm.go:86-107`), the hub's hash (sealing the OLD password)
is compared against `LocalHash()` (the NEW minted one), they differ, and it logs:
```
[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password
(hub hash %.12s… != local %.12s…) — run the escrow ceremony … staying pending
```
**So: confirmed, it fires.** The detector is not absent from the mint path.
**Three limits, each of which matters more than the fact that it fires.**
1. **It is a controller-local log line.** No event, no e-mail, no hub-side signal. `StaleBlob()` — the
flag the web card renders — is set **only** by the escrowed branch (`reconcileEscrowed`,
`:130-165`); the pending branch sets no display flag. The operator learns nothing.
2. **Its prescription is the destructive act.** *"run the escrow ceremony"* → a new ceremony →
`SaveHostEscrow` overwrites `host_escrow.identity_blob` → `demoteCurrentEscrowTx` does not carry it
→ **the old repo password ceases to exist**. The detector correctly identifies the situation and
then tells the customer to close the door.
3. **It cannot distinguish the two situations that need different answers** — "you re-keyed
deliberately" and "your machine was rebuilt and your history is about to be orphaned" produce the
identical warning.
So: **less needs building than the prompt feared for the *detection*, and more needs building for the
*consequence*.** The comparison exists; what does not exist is a path from that comparison to either a
signal or a recovery.
---
## 7. Q5 — is the superseded-escrow retention real, and what does it retain?
### RULING: **the retention is real, it retains the wrong blob, and nothing can read it back.**
**What it retains — source and live, agreeing.** `demoteCurrentEscrowTx` (`store/store.go:2547-2556`):
```sql
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
FROM host_escrow WHERE host_id = ?
```
`identity_blob` is not in the column list. Live schema (I2, `PRAGMA table_info`):
| table | columns |
|---|---|
| `host_escrow` | `host_id, blob, key_fingerprint, posture, created_at, updated_at, **identity_blob**, directive_json, restic_pw_sha256, stale_at` |
| `host_escrow_superseded` | `id, host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at` |
**`blob` is the K-escrow — the PBS datastore encryption key**, wrapped by the PBS-native scrypt path.
**`identity_blob` is the age-wrapped `IdentityBundle`** — the one carrying `ResticRepoPassword`
(`felhom-agent/internal/escrow/identity.go:34-39`), plus `TunnelToken`, `PBSToken` and
`WGPrivateKey`. Live sizes confirm they are different objects: both hosts' current rows hold
`blob` = 383 B **and** `identity_blob` = 572 B; both superseded rows hold `blob` = 383 B and nothing
else.
**So, exactly what a customer with the matching OLD recovery code could recover from a retained blob:**
- ✅ the PBS datastore encryption key **K** — i.e. the Plane-2 whole-guest offsite ciphertext;
- ❌ **not** the restic repository password — the Tier-3 offsite app-data history;
- ❌ not the tunnel token, not the PBS access token, not the WireGuard private key.
**And there is no read path.** I4: `ListSupersededEscrow` (`store/store.go:2617`) has **zero
production callers** — only two tests. The only production surface is `CountSupersededEscrow`, used
for a number on the host page (`web/hosts.go:573`) and a log line (`api/handler.go:1151`). The schema
comment says so itself: *"The ACK and restore-serving read `host_escrow` (the CURRENT row) — never
this table."* Retrieval is `sqlite3` on the hub pod, by hand, by the operator.
**The claim this contradicts.** `00-capability-map.md` (escrow-ceremony row) records hub v0.60.0 as:
> *"a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS
> it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code
> (turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable')."*
The same claim appears in the schema comment (`store.go:370-375`) and, in Hungarian, on the customer's
own orphan card (`controller/internal/web/templates/backups_remote.html:66,69`):
> *„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók
> lehetnek."* · *„A régi előzmény félretéve marad (nem törlődik), és a hozzá tartozó helyreállítási
> kóddal később visszaállítható lehet."*
**For the offsite restic repository — the incident it names — that is false.** The retained blob does
not contain the key that opens it. This is the **ninth** entry in `CLAUDE.md`'s table of comments
asserting an invariant the code does not provide, and the first that is also **customer-facing copy**.
→ **R-198**, and the capability-map correction in §9.
---
## 8. Q6 — has any part of this ever been exercised?
**Method:** read every audit, validation doc, findings file and REPORT that mentions escrow consume,
identity-consume, re-enroll or inject-password; then check what the cited evidence actually covers.
### Proven
| what | evidence | scope of the proof |
|---|---|---|
| ceremony (create + seal + upload) | `00-capability-map.md` escrow row; `tests/VALIDATION-n100-rehearsal-2026-07-18.md` S6 — customer drove the wizard, R claimed one-shot 16:56:39, auto-confirmed 16:56:41 | the **create** side only |
| identity-bundle round-trip with a real R on a secret-less box | `tests/slice10d-identity-restore-spike-findings.md` §1, 2026-06-10 — wrap→unwrap byte-identical (`sha256` match); wrong-R **fails closed**, no plaintext emitted | **PRE-FORK-4.** The bundle then was `{tunnel_token, pbs_token}`. `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** (`felhom-agent/CHANGELOG.md:2344`) — a month later. **The proof does not cover the field this whole question is about** |
| `ResticRepoPassword` survives the wrap/unwrap | agent v0.77.0 CHANGELOG: *"`IdentityBundle` round-trip carries `ResticRepoPassword` byte-exact + not-in-blob + wrong-R fails closed"* | **unit test only** |
| `InjectOffboxPassword` places and refuses correctly | `controller/internal/backup/offbox_test.go:952-990`, incl. the companion showing a fresh mint without inject | **unit test only** |
### Never exercised — named in those words
- **A blob has never been served to a box.** Link 6 has no client (§3).
- **A fork-4 bundle has never been unsealed with a real R outside a unit test.** The only production
caller is a `--selftest` mode; the one live consume ever prepared (S5 Part 4-B, 2026-07-04) was
explicitly **deferred and never run** — its own resume note still reads *"DEFERRED 2026-07-04, not
run"* (project memory; `06-offsite-connectivity.md:327` marks the destructive drill operator-gated).
- **A recovered repo password has never been injected into a controller.**
- **An existing offsite repository has never been reopened with a recovered password.**
- **No restore of any kind has ever been performed from a recovered secret.**
`_recovery-inventory-2026-07-28.md` already said most of this and said it accurately — A.2.7 records
*"the **consume** side is proven at spike level … the destructive in-place drill is explicitly
operator-gated and unrun"*, and C.1 row 1 says *"escrow **creation** proven live; **consume never
exercised destructively**"*. **That document is right and this session confirms it.** What it did not
catch — because it was written from `host_escrow`, not from the supersession path — is Q5.
**Capability-map correction owed:** not for a "recovery proven" claim (no row makes one), but for the
**retention** claim quoted in §7. See §9.
---
## 9. Q7 — the operator's ruled shape, assessed row by row
> **Operator ruling, 2026-08-04, verbatim:** *If a node is a fresh install AND the hub has a recovery
> blob, then the controller should yell that recovery is available, and provide a form for the customer
> to enter the recovery key. After unlocking the blob, the controller should show what will be
> recovered before proceeding.*
| The ruling requires | Exists? | What it needs, and what it costs |
|---|---|---|
| the controller knows it is a **fresh install** | **YES, at the right line** | `os.Stat(m.offboxPwPath())` → `IsNotExist` is already the mint branch (`offbox.go:392`). It is trustworthy as *"this controller has no repo password"*, which is the only fact the decision needs; it does **not** by itself distinguish first-ever install from rebuild — that is what the next row supplies. Corroborators exist but are weaker: `Claimed` is set-only and was measured re-set by a human within minutes (§5); a fresh `encryption.key` is a stronger signal but is not currently read by anything. **Cost: zero — the branch already exists.** |
| the controller knows **the hub has a blob** | **YES on the wire, NO in the code** | `EscrowStatus{IdentityBlobPresent, ResticPwSHA256, CreatedAt}` arrives on every report ACK and is discarded when no offbox target exists (§5). **Needs:** persist it (the `ClaimSync` set-only-by-generation pattern is the precedent, `claim_sync.go:39-53`). **Cost: small — controller only, one settings field, one reconciler, no hub change.** |
| it can **yell** rather than silently mint | **NO** | The refusal belongs **inside `WriteOffboxSecrets`, at the mint branch** — the one place that decides. Returning a new sentinel (`ErrOffboxRecoveryAvailable`, sibling of `ErrOffboxOrphaned`) propagates through `ApplyOffsiteTarget` → `ConfigureOffbox` → the apply-bridge, which already treats a failed configure as "persist nothing, retry next cycle" (`offsiteapply.go:5-6, 256`) — so a refusal is **fail-safe by construction**, not a new failure mode. Meanwhile the customer sees a card on `/backups/remote` (the orphan card at `backups_remote.html:60-70` is the exact precedent) and offsite runs stay blocked, which they already are (`OffboxRunnable`). **Cost: small-medium — controller only.** **Design hazard to name:** a first-ever install of a *re-used customer record* would also trip this; the refusal must be dismissible ("start fresh anyway"), and dismissal must be the act that mints. |
| a **form takes R** | **NO — and R is not a password** | The controller UI **emits** R (`backups_escrow.html:235-236`, `/backup/escrow/…/claim` returns it once) and has **never accepted** one. The only thing resembling an entry point is `POST /backup/offbox/inject-password`, which takes a **64-hex repo password** (`offboxRepoPwPattern`, `offbox.go:543`) — a different secret at a different layer — and **has no form in any template** (§3 link 9). **Cost: small for the form; the hard part is the next row.** |
| the blob is **unsealed** with R | **NO — and there is a real boundary** | Unsealing is `age -d`. `ageBinary = "/usr/bin/age"` (`identity.go:24`) and the code live in the **agent**. The **controller image does not ship `age`** (`controller/Dockerfile:56-70` — restic, openssh-client, sshpass, sqlite3, docker-cli; no age) and must not: the agent is the privileged tier by design. **The crossing already has a transport and a precedent:** the pinned local API carries `POST /escrow/ceremony` + `GET /escrow/ceremony/status` + `POST /escrow/ceremony/claim` (`localapi/server.go:524-527`) — i.e. the agent **already** handles R server-side for the create direction and hands the result back exactly once. **Needs: one new agent local-API endpoint** (`POST /escrow/unseal`: blob + R in, the single `restic_repo_password` field out, R never logged, R never persisted) — the exact mirror of the claim endpoint. **Plus** a way for the blob to reach the agent: the agent's hub client has no escrow **GET** (only the `PUT` at `main.go:2937`), and `dr.go`'s re-enroll is the wrong tool (it rotates the host API key and needs operator-armed recovery mode). **Needs: a second small hub endpoint** — `GET /hosts/<id>/escrow`, per-host-key authed, serving opaque bytes to the box that owns them. Zero-knowledge is untouched: the hub still never decrypts, and the bytes are useless without R. **Cost: medium — three repos, two new endpoints, both narrow.** |
| **"show what will be recovered" before proceeding** | **the primitive EXISTS; the flow does not** | `restic snapshots --json` + `restic stats --mode raw-data --json` are already how the box counts snapshots and sizes the repo (`offboxRecordStats`, `offbox.go:1234-1265`). Both are **read-only against the repository** — nothing is written, no lock is taken, no `init`. So a preview can show: **snapshot count, per-snapshot time, host, tags (= app names), paths, and the deduplicated repo size** — all before committing. **Committing requires only that the recovered password be written to `offboxPwPath()`**, and even that is reversible in principle. **The clean shape:** unseal → hold the recovered password **in memory** → run the two read-only probes against the existing repo → render → and only on the customer's confirmation call `InjectOffboxPassword`. **Cost: small — one handler, one template, no new restic capability.** **Caveat, measured not assumed:** `offboxRecordStats` today also persists `RepoSizeBytes` into settings as a side effect; a preview path must not reuse it verbatim. |
| **proceeding** actually recovers | **NO end-to-end** | `InjectOffboxPassword` → next `WriteOffboxSecrets` keeps the file → `ensureOffboxRepo` probes and clears `RepoState`. Every piece is implemented; **none has ever run against a recovered password** (§8). **Cost: zero new code, but it is precisely what the drill in §10 has to prove.** |
### Things the ruling implies that do not exist, priced
| missing piece | cost | note |
|---|---|---|
| persisted ACK escrow status + fail-closed mint refusal | **S** | controller only; the `ClaimSync` pattern applies directly |
| the "recovery is available" card + dismissal | **S** | one template, one flag; the orphan card is the template precedent |
| a customer-facing **recovery-code** form (distinct from the password form) | **S** | the password form's handler exists but has no UI; both need building |
| agent local-API `POST /escrow/unseal` | **M** | new privileged surface; mirror of `/escrow/ceremony/claim` |
| hub `GET /hosts/<id>/escrow` (opaque, per-host key) | **S** | new read surface on the hub; zero-knowledge preserved |
| preview handler + template driven by read-only restic probes | **S** | primitive already exists |
| **carrying `identity_blob` into `host_escrow_superseded`** | **S** | one column, one `SELECT` list — **and without it the whole feature protects only boxes that have not yet re-escrowed** (→ R-198). This is a prerequisite, not a nice-to-have |
### The security question the ruling raises — stated, not answered
**What gates the surface today.** The controller dashboard is behind `RequireAuth`
(`internal/web/auth.go:52`): a bcrypt password (`effectivePasswordHash()` — customer-set in
`settings.json`, else operator-provisioned in `controller.yaml`), a 7-day session cookie, 5 failed
logins per minute then lockout, and `CsrfProtect` on every mutating route. The claim gate serves only
the claim page on an unclaimed box. So a recovery form would sit **behind the customer's dashboard
password**, on a controller reachable at whatever address that box's dashboard is reachable at.
**What a preview would expose to someone who guesses or steals R.** Not file contents — but the
snapshot list is not nothing: **backup dates and cadence, the app names (restic tags), the paths, and
the repository size.** That is a map of the household's digital life, and it is precisely the metadata
the zero-knowledge posture otherwise keeps out of everyone's hands including the operator's.
**Three specifics the operator should weigh, without this session choosing:**
1. **Guessing R is not the threat; holding the dashboard password is.** R is ten EFF words
(~129 bits, `00-capability-map.md` agent v0.93.0 note) and `age`'s scrypt is deliberately slow —
online guessing is not a route. But **anyone with the dashboard password can already restore live
offsite data** (`_recovery-inventory-2026-07-28.md` A.1.3–A.1.6). The marginal new exposure is
confined to the *orphaned/previous* repository, which today nobody can read at all.
2. **The form is an oracle.** A correct/incorrect R answer distinguishable in the UI turns the
dashboard into a verification service for a stolen recovery code. `UnwrapIdentity` already
fails closed with a single generic error (`identity.go:157`) — the UI must not do better than that,
and should rate-limit like the login path does.
3. **R will pass through the agent.** Whatever the form does, R crosses the local API to a
root-adjacent daemon. The ceremony already does this in the other direction, so the precedent is
set — but it means R is momentarily present in agent process memory on the Proxmox host, which is
the same host D6 says the operator has root on. **This is the same trade as spike option (c), in a
smaller and time-bounded form**, and it should be decided alongside it rather than separately.
---
## 10. Q8 — the drill design
**Why a deliberate cycle.** No recovery code exists for either demo box, and — per §7 — even one would
no longer open the old repositories. The proof therefore requires creating R, keeping it, and walking
the full circle. **This section is the design; the drill is not run in this session.**
### Target and preconditions
| | |
|---|---|
| **Box** | **`demo-hp`** (HP t740, node `felhom-host`, host `demo-hp-bb76ea`). Tier 0 — disposable, and the designated drill host by the 2026-07-25 operator ruling. **Not demo-felhom** — it is the N100 that also carries the ISO/rehearsal lineage; leave one demo box in a known-good state throughout |
| **Guest** | the customer guest on that host (rebuild target) |
| **Data** | one deliberately identifiable app + a sentinel file whose content is a known string and whose sha256 is recorded before the wipe |
| **Prereq** | the box is healthy, offsite target configured and `escrow_state: escrowed`, and its **current** `host_escrow.identity_blob` seals the password of the repo that will be tested |
| **Duration** | ~3–4 h wall clock, of which ~45 min is operator-attended (steps 2, 5, 8, 9) |
| **Blocking dependency** | **steps 8–9 cannot be executed as designed today** — links 6, 8 and 9 of the chain do not exist as automation. The drill as written uses the manual substitutes (§4a) and **that is the point**: it measures the *existing* chain honestly before anything is built |
### The cycle
| # | Step | Who | Observable that must be recorded |
|---|---|---|---|
| 1 | Record the starting state: `host_escrow.restic_pw_sha256`, `identity_blob` length, `$.offsite.snapshot_count`, `repo_size_bytes` | CC | a hash and two counters, from the hub DB |
| 2 | **Run the escrow ceremony through the customer wizard and KEEP R** | **HUMAN** | R written to the operator's password manager under a named entry; **its storage location recorded in the drill report — the code itself never** |
| 3 | Deploy an app, write the sentinel file, record its sha256 | CC | sha256 + path |
| 4 | Run an off-site backup through the real UI endpoint; wait for completion | CC | `$.offsite.last_status = ok`, `snapshot_count` increased by ≥1, `last_success` stamped |
| 5 | **Verify the snapshot from outside**: `restic snapshots --json` against the repo | CC | snapshot id + tag + timestamp — the pre-wipe truth to compare against |
| 6 | **Wipe**: destroy the guest and reinstall it (the same procedure that produced the 2026-08-03 rebuild) | **HUMAN** decision, CC executes | new `controller_started`; `claimed` drops to 0; a new `encryption.key` mtime |
| 7 | Let the box come back and reach the mint. **Do NOT run a new ceremony** | CC | the controller log carries the `escrow-confirm` mismatch WARN (§6); `escrow_state: pending`; `host_escrow.identity_blob` **unchanged** (this is the assertion that keeps recovery possible) |
| 8 | **Recover**: export `identity_blob` from the hub, unseal with the kept R, extract `restic_repo_password` | **HUMAN** (R is theirs) | `identity-consume` exits 0; the bundle JSON contains a non-empty `restic_repo_password`; **the value is never printed or logged** |
| 9 | **Inject** the recovered password (`POST /backup/offbox/inject-password`, by hand — there is no form), re-apply, run an off-site backup | **HUMAN**-assisted | `restic cat config` succeeds; **`snapshot_count` ≥ the step-5 count + 1**, not 1 |
| 10 | **Restore the sentinel** through the customer restore flow and compare sha256 | CC | **byte-identical sha256 to step 3** |
| 11 | Record everything, tear down nothing that would hide the evidence | CC | drill report |
### The single observation that makes this a FAILURE rather than a pass
> **Step 10's sha256 does not match step 3's — or step 9 reports a snapshot count of 1.**
A snapshot count of 1 means a **new, empty history** was started over the old one: the repository did
not reopen, and every earlier snapshot is orphaned. **Explicitly: "the repository opened" is not the
pass condition.** `restic cat config` succeeding proves the password is right; it does not prove the
customer's data came back. Only the sha256 comparison does.
### Secondary failures worth recording separately (each is a finding, not a pass/fail)
- Step 7 finds `identity_blob` **changed** — something re-escrowed automatically → the recovery window
closes without a human act (would be new and serious).
- Step 8's bundle has an **empty** `restic_repo_password` — a pre-fork-4 or hash-less blob shape.
- Step 9's `ensureOffboxRepo` classifies `orphaned` despite the correct password → the classifier or
the transport, not the key.
### What could make it fail for uninteresting reasons
The `age` binary absent on the machine running step 8; the hub DB export producing a truncated blob
(use `writefile`, verify the byte length against `length(identity_blob)` = 572); the CSRF token in
step 9 (`felhom_session` is dropped by curl's cookie jar — project memory
`controller-api-headless-access`); and the R-71a settle gate deferring the apply in step 9 (wait for
its GO line rather than retrying).
### Rollback, and whether the loss is acceptable
**If the recovery does not work, what has been lost is:** demo-hp's guest, its app data since the last
whole-guest archive, and its off-site history. **On that box that is acceptable** — it is Tier 0,
explicitly disposable, and it is the box the target-selection runbook names for drills. **The
rollback** is a whole-guest restore from the local/PBS archive taken immediately before step 6, which
is a proven path on this fleet (restore-test passed on demo-hp's host tier in 109.3 s on 2026-08-04).
**Take that archive deliberately as part of step 6 and verify it exists before destroying anything** —
do not rely on the nightly.
**Not acceptable, and therefore not proposed:** running this on demo-felhom (leaves no healthy demo
box), on DooPlex (Tier 2, it *is* the recovery chain), or on any real customer's box.
---
## 11. Every query used
| # | Source | Query / read | Value read |
|---|---|---|---|
| 1 | git | `git -C <repo> rev-parse HEAD`; `git status --porcelain` | the three commits in §1; all empty |
| 2 | hub DB | `PRAGMA integrity_check` | `ok` |
| 3 | hub DB | `SELECT MAX(received_at) FROM host_reports; SELECT datetime('now')` | `2026-08-04 10:00:40` / `2026-08-04 10:03:48` |
| 4 | hub DB | `SELECT host_id, length(blob), length(identity_blob), substr(restic_pw_sha256,1,12), created_at, updated_at FROM host_escrow` | `demo-felhom-8363b5, 383, 572, c60c8bc737a6, 2026-08-04T07:20:07Z, 2026-08-04 07:20:08` · `demo-hp-bb76ea, 383, 572, 8a9e33aa4da6, 2026-08-04T07:15:36Z, 2026-08-04 07:15:36` |
| 5 | hub DB | `SELECT id, host_id, length(blob), substr(restic_pw_sha256,1,12), created_at, superseded_at FROM host_escrow_superseded` | `3, demo-hp-bb76ea, 383, 8e03eddf9ff7, 2026-07-23T10:01:17Z, 2026-08-04 07:15:36` · `4, demo-felhom-8363b5, 383, 48741892f0ef, 2026-07-21T08:38:33Z, 2026-08-04 07:20:08` |
| 6 | hub DB | `PRAGMA table_info(host_escrow)` / `(host_escrow_superseded)` | the two column lists in §7 — `identity_blob` present in the first, **absent from the second** |
| 7 | hub DB | latest report per customer: `json_extract(report_json,'$.claimed' / '$.offsite.escrow_state' / '.snapshot_count' / '.repo_state' / '.last_status' / '.last_run')` | demo-felhom `2026-08-04 09:49:42`: `1, escrowed, 0, NULL, NULL, NULL` · demo-hp `2026-08-04 09:56:57`: `1, escrowed, 0, NULL, NULL, NULL` |
| 8 | hub DB | `claimed` grouped over reports since `2026-08-03 05:00` | demo-hp: `1` ×124, **`0` ×5 (06:12:19→06:21:03)** · demo-felhom: `1` ×120, **`0` ×4 (07:17:54→07:19:46)** |
| 9 | hub DB | `SELECT customer_id, event_type, severity, created_at FROM events WHERE created_at > '2026-08-03 00:00'` | no `escrow_stale`, no `offsite_reissued`, no `offbox_repo_orphaned`, no claim event in the window; `controller_started` ×4 demo-hp / ×4 demo-felhom around the rebuilds |
| 10 | guest 9201 (I3) | `ls -la --time-style=full-iso …/felhom-controller-data/_data/data{,/offbox}` | the mtime table in §5 |
| 11 | source (I4) | `grep -rn "InjectOffboxPassword" --include=*.go .` | 1 definition, 1 handler caller, 5 test uses, 1 comment — **no production caller besides the handler** |
| 12 | source (I4) | `grep -rn "inject" internal/web/templates/` | one unrelated hit (an XSS comment) — **no form** |
| 13 | source (I4) | `grep -rni "reenroll\|re_enroll" --include=*.go .` in `felhom-agent` | **zero hits** |
| 14 | source (I4) | `grep -rn "re-enroll\|recovery-mode\|restore-directive" --include=*.sh --include=*.md --include=*.py` workspace-wide | only design docs + project memory; no script, no runbook step |
| 15 | source (I4) | `grep -rn "ListSupersededEscrow"` in `felhom.eu` | 2 test callers, 0 production |
| 16 | source (I4) | `grep -rn "\.Escrow\b" --include=*.go .` in `felhom-controller` | one hit: `cmd/controller/main.go:641` |
| 17 | source | `felhom-agent/CHANGELOG.md` | `v0.77.0 — fork-4: escrow the offsite restic repo password under R (2026-07-09)` |
| 18 | source | `controller/Dockerfile:36-70` | runtime package list — **no `age`** |
---
## 12. What could not be established
Named, not guessed.
1. **The 2026-08-05 ~02:15 UTC scheduled run's outcome.** Still unmeasured — both boxes became
runnable only on 2026-08-04 morning and neither has run since (query 7: `last_run` absent on both).
Yesterday's spike predicts `ErrOffboxOrphaned`; that prediction is **unchanged and still unproven**.
2. **Whether demo-hp's data dir was fresh.** Established for demo-felhom by direct mtime measurement
(I3). demo-hp was not measured — it has no baked SSH key and reaching it needs the break-glass
credential, which this read-only session did not use. The hub-side `claimed 1→0→1` signature is
identical on both boxes, which is suggestive, not decisive.
3. **Whether the orphaned ciphertext still exists at the provider.** Unchanged from yesterday: reading
the endpoint needs the SFTP credential, which was not touched.
4. **Whether any pre-2026-07-21 escrow generation ever existed for demo-felhom.** The superseded table
holds two rows in the entire database; earlier generations are not retained (and, per §7, would not
have carried the identity blob anyway).
5. **Whether `age` is present on any machine that would run step 8 of the drill.** Not checked — the
agent declares `/usr/bin/age` as a runtime dependency but this session did not verify it on either
demo host.
---
## 13. Observations — noticed, not acted on
1. **A fresh `encryption.key` on every rebuild** (I3: 32 B, mtime `2026-08-03 07:17:26`) means every
pre-rebuild `app.yaml` `ENC:` secret on that box is undecryptable. This is exactly circular
dependency **C-1** in `_recovery-inventory-2026-07-28.md`, and it fired live on both demo boxes this
week without anyone naming it. It is a strictly larger blast radius than the offsite repo and it is
**not** covered by R-193. Worth its own item if the operator agrees it is not already implied by C-1.
2. **`Claimed` returning to `1` within minutes** of a rebuild, with no event of any kind recording it,
means "the customer claimed this box" has no audit trail. There is no `customer_claimed` event type.
3. **`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting** — harmless today,
a trap for anyone reusing it as a read-only preview primitive (§9).
4. **`handleReEnroll` returns the new API key in the response body** (`dr.go:148`) while also logging
the rotation. Not a defect (the box needs it), but it is a secret in an HTTP response on an endpoint
nothing currently calls — worth a second look before anything starts calling it.
---
## 14. The operator's decisions — stated, and NOT answered here
1. **Run the drill (§10) or not?** It is the only thing that converts *"the pieces look right"* into
*"a customer got their data back"*. It costs demo-hp for an afternoon and, if it fails, that box's
history. **Recommendation: yes — but fix `identity_blob` retention (R-198) first**, because a drill
that walks the current chain will pass or fail on a chain that is missing a link everyone believed
was there.
2. **The orphaned snapshots at the endpoint** (40.9 MB + 1.14 GB, presumed present, §12.3). They are
**unrecoverable by two independent mechanisms** now: no recovery code was kept, *and* the sealed
passwords were destroyed by the 2026-08-04 re-ceremonies. They consume the 50 GB soft quota
indefinitely; nothing prunes a moved-aside restic repo. **Delete, or leave?** This session did not
act on it and states it as the operator's.
3. **The unchanged R-193 question** — agent-retained repo password (spike option c) versus accepting
that every rebuild starts a new repository. **This session adds one input to that decision:** the
"accept it, provided the system says so loudly" branch currently rests on a retention feature that
does not retain the relevant secret. Under option (d)-alone as it stands today, *"the old history
stays recoverable with your recovery code"* is not true. Either R-198 is fixed, or that branch has
to be re-described honestly to the customer.
</content>
</invoke>
documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md
+39
View File
@@ -10,6 +10,39 @@ what it measured) · **Class:** spike — **no production code shipped for R-193
---
> ### ANNOTATION 2026-08-04 (later the same day) — read this before acting on §Q8
>
> A follow-up recon traced the escrow→recovery chain link by link
> (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **Two corrections to this document, and they pull
> in opposite directions.**
>
> **(1) Q8 candidate (b) — "not implementable as specified" — is TOO STRONG and is overturned in
> part.** Zero-knowledge means the *hub* cannot open the blob; the *customer* can, with R, which is
> what the ceremony exists for. The mechanism to hand a rebuilt box its old repo password back —
> customer-present, not unattended — is a real design, and the operator has since ruled on its shape.
> What (b) got right is narrower than what it said: **an UNATTENDED rebuild cannot recover from a
> zero-knowledge escrow.** A customer-present one can. The recon prices that shape row by row.
>
> **(2) A LOAD-BEARING FACT THIS SPIKE DID NOT CHECK, and it makes (d)-alone weaker than ranked
> here.** `host_escrow_superseded` **has no `identity_blob` column**, and `demoteCurrentEscrowTx`
> (`hub/internal/store/store.go:2547-2556`) copies only the K-escrow `blob`. So the retained
> "superseded" blob holds the **PBS datastore key**, not the **restic repository password**. The
> retention feature does not retain the secret it was built to retain, and the **next escrow ceremony
> — which the system tells the customer to run whenever a rebuild leaves the escrow `pending` — is
> what destroys the last copy.** Both demo boxes crossed that line on 2026-08-04 at 07:15:36 and
> 07:20:08. → **R-198**.
>
> **Consequence for this document's recommendation.** "Ship (d), then decide (c)" still holds, but
> (d)'s implicit promise — *the old history stays recoverable with the recovery code* — is **false as
> the code stands**, and is repeated verbatim to the customer in Hungarian on the orphan card. (d) is
> not complete without R-198.
>
> **Unchanged and still correct:** Q1, Q2, Q4 (R-196), Q5, Q6, Q7 (R-192) and the Q8 (a)/(c)/(d)
> analyses. **Q3 remains UNMEASURED** — neither box has run since; the 2026-08-05 ~02:15 UTC run is
> still the decisive one.
---
## 0. Why this exists
Two candidate fixes were named when R-193 was filed, and neither could be weighed, because two prior
@@ -412,6 +445,12 @@ re-keyed, `host_leaf_changed` 2026-08-03 06:09:40), so re-enroll returns the exi
### (b) The credential becomes recoverable from escrow at re-bootstrap
> **ANNOTATED 2026-08-04 (recon) — this ruling is overturned in part. See the annotation at the top.**
> The constraint is real but narrower than written: an **unattended** rebuild cannot recover from a
> zero-knowledge escrow; a **customer-present** one can, and the operator has ruled on that shape.
> The recon also found that the chain this option would use is clientless at three links and lossy at
> a fourth (`RECON-offsite-dr-chain-2026-08-04.md` §3–§4).
- **BLOCKED AS STATED, and this is a hard constraint, not an estimate.** The escrow blob is
**R-wrapped and zero-knowledge** — the hub holds opaque bytes and has no recovery code
(`api/handler.go`, and D6 in `CONTEXT.md` S-3). **A rebuilt box cannot unseal it without the
documentation/backlog/OPEN-ITEMS.md
+6 -2
View File
File diff suppressed because one or more lines are too long