# REPORT — R-193/R-192 spike (no code) + R-195 phantom-customer alarm (hub v0.92.0) **Date:** 2026-08-04 · **Class:** spike (Parts 0–3) + one fenced implementation (Part 4) **Boundary held:** the spike shipped **no production code for R-193 or R-192**, and Part 4 never touched the spike's subject matter. > Supersedes the earlier 2026-08-04 report (R-191 + R-190), per the overwrite convention. Nothing > durable from it lived only here — it is recorded in `hub/CHANGELOG.md`, `CONTEXT.md` S-24/S-25 and > `OPEN-ITEMS.md` R-190/R-191. --- ## 1. Confirmed baselines, re-read at the cited commit | Repo | `main` @ commit on arrival | Version | End state | |---|---|---|---| | `felhom.eu` | `f456835bbcc4` ✔ matches the spec | hub `v0.91.1` (deployed `felhom-hub:0.91.1`) | **`v0.92.0`** built, pushed, manifest bumped, synced, **live** | | `felhom-controller` | `0887fd676d60` | — | **untouched** (read-only) | | `felhom-agent` | `856a127` | — | **untouched** (read-only) | Clean-tree gate passed before the build: `git status --porcelain` empty, `HEAD == origin/main`. Highest register ID on arrival was **R-194**; grepped before minting. --- ## 2. Q1–Q8 — each answered or explicitly marked unmeasured Full method, quotes and query output: **`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md`**. ### Q2 and Q3 first — they decide whether the rest matters **Q2 — does a rebuilt controller reuse the escrowed repository password, or mint a new one?** **RULING: it MINTS A NEW ONE.** *Method — source:* `ConfigureOffbox` → `ApplyOffsiteTarget` → `WriteOffboxSecrets`, which generates a fresh 256-bit password whenever `/offbox/repo_password` is absent (`felhom-controller/controller/internal/backup/offbox.go:392-401`). A rebuilt guest has a fresh data dir. The only recovery path, `InjectOffboxPassword`, has **exactly one caller in the entire repo** (census: `grep -rn "InjectOffboxPassword" --include=*.go .` → 3 hits, 1 definition, 1 caller, 1 comment) — `web/offbox_handlers.go:189`, a web form a human pastes into. **The apply-bridge never calls it.** *Method — live, without touching a box:* `host_escrow.restic_pw_sha256` vs `host_escrow_superseded.restic_pw_sha256`. **demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom `48741892…` → `c60c8bc7…`. Both CHANGED.** **This is bigger than R-193 as filed:** demo-felhom, which R-193 records as having survived by luck, had **36 snapshots / 1,136,685,919 B** before its rebuild and has reported `snapshot_count: 0` in all 109 reports since. Its luck restored **delivery**, not the **repository**. **Q3 — what did the 04:15 run report?** **UNMEASURED, and the binary in the question is wrong.** The run is daily at ~02:15 UTC (measured `last_run` values on both boxes). Neither box could run on 2026-08-04: demo-hp had no target at all until 07:15:47 UTC; demo-felhom's target was `escrow_state: pending` from 08-03 07:19:10 to 08-04 07:20:13, which `OffboxRunnable` blocks (configured **AND** escrowed). **The decisive run is 2026-08-05 ~02:15 UTC.** Source predicts a **third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`; `repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies **orphaned** → `ErrOffboxOrphaned` → **the run refuses and shows the orphan card.** Recorded as a prediction, not a measurement. ### The rest - **Q1 — what is one-shot.** Full table in the spike. Headline: the hub's one-time provider password is one-shot **but freely re-issuable**; the box's SFTP key is **regenerated on every apply by design**; the **restic repository password** is the one nothing can restage — the agent's own source calls it *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/escrow/identity.go:35-39`). **R-193 was filed against the recoverable secret; the damage is in the irreplaceable one.** - **Q4 — see §3 below.** - **Q5 — why one box recovered and the other not. CONFIRMED FROM THE LEDGER.** demo-felhom's `one_time_secrets` row was created `2026-07-21 08:29:29` (traceable to a Re-issue at that exact timestamp) and sat **UNCONSUMED for 13 days** until the rebuild consumed it at `2026-08-03 07:17:58`. Without one: exactly demo-hp — `consume` → **404** and a retry loop, 25 hours unprotected. **Correction to the contrast:** the recovery was of the plumbing only (Q2). - **Q6 — why delivery is one-shot.** The design's own reasons quoted from three sites. **The security half, stated:** the consume endpoint is gated by `checkAuthCustomer` — **the customer's API key, the same credential as the config pull** — so an automatic restage grants **no new access to an unauthenticated party**. What it genuinely widens is the **exposure window**: a live provider password becomes fetchable whenever the box reports no offsite target, rather than only in the seconds after an operator acts. Blast radius stays the customer's own label-scoped sub-account. The R-39(a) unconsumed-secret guard must survive any change. - **Q7 — R-192's guard. BOTH HALVES CONFIRMED WITH NUMBERS.** See §4. - **Q8 — what each candidate costs.** See §5. --- ## 3. Which of the two contradictory claims about Re-issue is true **The claim that Re-issue "only resets the transport credential" is TRUE about what `ReissueCredentials` does. R-39's "rotates the restic password" is FALSE.** From source (`hub/internal/offsite/offsite.go:150-228`), the function does four things: `genPassword`, `ResetSubaccountPassword`/`ResetBoxPassword`, `SaveOneTimeSecret`, and bookkeeping. **There is no reference to a restic password anywhere in it, and none is possible** — that password is generated on the box and never leaves it except into the R-wrapped escrow, which the hub cannot open. **But the bookkeeping is justified by the false premise**, `offsite.go:198-201`: *"the restic repo password just changed, so any existing key-escrow blob … is now STALE"*. The same wrong claim appears at `api/handler.go:1067-1069` and in R-193's own row. **Eighth entry in `CLAUDE.md`'s table of comments asserting a guarantee the code does not provide** — the first where the comment factually describes a *different function*. → **R-196**. **What corroborated it, and the corroboration is the strongest part.** The task asked: if the repo password was untouched, what explains demo-hp's escrow going `pending` after the Re-issue? **Fully explained, and not by the Re-issue:** `ApplyOffsiteTarget` carries `EscrowState` over from the *existing* target, the rebuilt guest had none, so the `pending` fallthrough fired — and the box had just minted a new repo password, so `pending` was *correct for a different reason*. **The proof they are independent is demo-felhom:** it received **no Re-issue at all** (its only `escrow_stale`/`offsite_reissued` pair is dated `2026-07-21 08:29:29`), its repo password changed anyway, and **nothing marked its escrow stale for 13 hours.** The mechanism is exactly inverted: `escrow_stale` is wired to the one path that does **not** change the repo password and absent from the path that does. --- ## 4. Every query used, with its fields and values The DB snapshot was taken from the live pod and its **freshness proven with a positive observable**, not with "the query returned no error": | # | Query / read | Result | |---|---|---| | 1 | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale (mtime 08:34 vs wal 10:45 CEST). Copying only the main file is the R-3 shape. | | 2 | `PRAGMA integrity_check` | `ok` | | 3 | `MAX(host_reports.received_at)` vs `datetime('now')` | `2026-08-04 08:45:38` vs `08:49:46` — **newest row 4 min old** | | 4 | `SELECT customer_id, status, created_at FROM customer_configs` | `demo-felhom` active · `peti-felhom` active · `demo-hp` active · `drill-r50` **blocked** · `david` active (created `2026-08-01 16:51:49`). **`sess-f` is gone — confirmed, not assumed.** `drill-r50` untouched. | | 5 | `hosts` / `host_deletions` / `host_reports` / `reports` counts for `david` | **0 / 0 / 0 / 0** — never bound | | 6 | `events WHERE event_type='expected_dbdump_missed' AND created_at > '2026-07-25'` | **only `david`**, 03:00 UTC on 08-02, 08-03, 08-04 | | 7 | `SELECT customer_id, MAX(received_at), COUNT(*) FROM reports GROUP BY customer_id` | `demo-hp` 1329 · `demo-felhom` 6690 · `drill-r50` 183 (last 07-29) · `peti-felhom` 482 (last **2026-07-15**) — the reason peti-felhom is `down` and does not alarm | | 8 | distinct `$.offsite` objects per box with first/last seen | demo-hp: `15 snapshots / 40 927 044 B` until `08-03 05:59:21`, **ABSENT** `06:12:19`→`08-04 07:12:01` (111 reports), then `pending` 07:15:47, `escrowed` 07:16:02, `snapshot_count 0`. demo-felhom: `36 snapshots / 1 136 685 919 B` until `08-03 07:01:28`, ABSENT ×1, then `pending` `07:19:10`→`08-04 07:20:13`, `escrowed` 07:20:28, `snapshot_count 0` | | 9 | `host_escrow` + `host_escrow_superseded` `restic_pw_sha256` | **the decisive measurement** — both changed (§2) | | 10 | `one_time_secrets` (no values read) | demo-felhom created `2026-07-21 08:29:29`, consumed `2026-08-03 07:17:58`; demo-hp created+consumed `2026-08-04 07:11:51`/`07:12:06` | | 11 | `$.claimed` on the latest report of each box | **`1` on both** — which selects `ensureOffboxRepo`'s refuse branch | | 12 | `CountReportsOffsiteSince` reproduced verbatim (`ORDER BY id LIMIT 500`) for demo-hp @ `2026-07-23 09:53:41` | **`total=500, withOffsite=500`**, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** | | 13 | same window, **no LIMIT** | `total=1174, withOffsite=1063` (⇒ 111 without — matches #8) | | 14 | full text of the live `offsite_delivery_stuck` message | *"…and **500 report(s) since carry no offbox target**…"* while `OffsiteReportsSinceConsume` was **500** | | 15 | `COUNT(*)` of `offsite_credential_restaged` | **0, for every customer, ever** | | 16 | the shipped predicate run verbatim against live data (post-deploy) | `david` → **0**; `demo-felhom`/`demo-hp`/`drill-r50`/`peti-felhom` → **1** | **Instrument limits recorded rather than discovered later:** `one_time_secrets` has `customer_id` as PRIMARY KEY and is last-write-wins **by design**, so it holds only the newest credential per customer — demo-hp's `2026-07-23` consume, on which R-192's whole diagnosis rests, was **overwritten** by yesterday's Re-issue and is no longer readable from that table. The `2026-07-23 09:53:41` anchor in query #12 comes from R-192's record, and that is stated in the spike rather than presented as a live read. --- ## 5. Ranked options, with a recommendation | rank | option | verdict | |---|---|---| | **1** | **(d) the honesty pass — now** | Cheap, weakens nothing, and **the only option that helps the incident already in flight**. Two parts: correct R-192's message + log its refusal (**R-196**), and **detect a changed `restic_pw_sha256`** — both hashes are already in the hub's DB and **nothing reads them** (**R-197**). | | **2** | **(c) agent-retained repo password — NOT PREVIOUSLY NAMED** | The only candidate aimed at the irreplaceable secret. The agent lives on the Proxmox host, **survives a guest rebuild**, already receives the password over the pinned local API and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Every seam exists. Price: a copy of the data key at rest on the host. **The operator's trade to make.** | | **3** | **(a) hub restages automatically** | **Already exists** as `reissueOnReenroll`'s F3 leg, wired to the wrong event (behind `handleHostEnroll`'s mint-once-reuse short-circuit; a guest rebuild leaves the `hosts` row intact). Worth doing **after** (c), never instead: on its own it restores the transport to a box whose data key is already gone, and **would have made both boxes look healthy on 2026-08-04 while their snapshots were orphaned** — strictly worse than the current loud failure. | | **4** | **(b) recover from escrow at re-bootstrap** | **Not implementable as stated.** The escrow is R-wrapped and zero-knowledge; the hub has no recovery code. "Recoverable at re-bootstrap" describes a customer-present ceremony — i.e. the manual form that already exists. | **Recommendation: ship (d) now; then decide (c). Do not ship (a) first.** ### The operator's question — stated, and NOT answered > The irreplaceable secret is the restic repository password. It is generated on the box, destroyed by > a guest rebuild, and the only surviving copy is sealed under the customer's recovery code. Every > machine will be reinstalled. > **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host, > where you already have root — or do you want it to stay sealed and accept that every rebuild starts > a new repository, provided the system says so loudly on the day it happens?** **STOPPED here. Neither candidate was begun.** --- ## 6. Part 4 — the phantom-customer alarm (SHIPPED) **Files changed** - `hub/internal/store/store.go` — **new** `(*Store).HasEverBoundHost(customerID) (bool, error)`: a live `hosts` row **OR** a `host_deletions` tombstone. - `hub/internal/monitor/deadline.go` — one guard at the top of `CheckBackupDeadlines`' customer loop, plus an `unbound` counter in the summary line. - `hub/internal/monitor/deadline_unbound_test.go` — **new**, 5 tests. **Commit:** `7fff45d688fa` (code + tests + spike + docs) · `f581ac134964` (manifest bump). **The mechanism, established at source and corroborated live.** The existing down-skip could never have covered this: it reads `StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — **a query over the `reports` table** (`store.go:1055-1067`). A customer with zero reports is in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. **The skip misses exactly the customer it would most obviously cover.** Corroborated by query #7 and by the live post-deploy startup log — `Staleness checker initialized: 2 ok, 0 stale, 1 down` — where the `1 down` is `peti-felhom` (host deleted 2026-07-15, 482 old reports) and `david` is **absent from the census entirely**. **The discriminator is "was a host EVER bound", not "has a report arrived"**, which is the point: a box that was installed, bound and went silent has a `hosts` row and keeps alarming. **Fail-open on a read error** — an unreadable binding must never suppress a real alarm. The deferral is **logged with its own counter** (the v0.73.0 Part-7 precedent). The anchored-verdict structure is untouched, and `david`'s record was not modified. **Sibling verdicts checked rather than assumed.** The census of every customer-iterating checker: `deadline.go` backup half (already guarded by `reportJSON == ""`), `restoretest.go` (guarded by `latest == ""`), `staleness.go` + `offsite.go` (iterate `GetCustomers()`, report-derived — a hostless customer never appears), `offsite_delivery.go` (a hostless customer derives `DeliveryNoSecret` and is skipped). **The DB-dump half was the only verdict that could fire**; the guard covers both verdicts in that function uniformly. **Test results — the suite run and the exit code read BEFORE any commit, never in one command.** ``` go build ./... → BUILD OK go vet ./... → VET OK go test ./... → rc=0, 17 packages ok, no FAIL lines ``` The 13 `TestCheckBackupDeadlines*`/`TestHasEverBoundHost` tests were run with `-v` and **the `=== RUN` lines were checked**, per `CLAUDE.md`'s rule that a `-run` pattern matching nothing prints `ok` and exits 0. **THE RED-PROOF THAT MATTERS.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms` is the case the change could break — a real box that was installed and never phoned home, indistinguishable from `david` on every signal *except* the one the guard reads. With the guard removed it **still passes**, which is the correct result and proves the test is aimed at the guard's *risk*, not its presence. The companion red-proof of the fix itself, **observed**: ``` === RUN TestCheckBackupDeadlines_NeverBoundHost_Silent deadline_unbound_test.go:58: a customer with NO host ever bound must raise NOTHING; got [expected_dbdump_missed] --- FAIL: TestCheckBackupDeadlines_NeverBoundHost_Silent ``` — verbatim the event `david` sent three mornings running. `deadline.go` restored from a backup taken before the mutation; full suite re-run green afterwards. **Deployed version.** `felhom-hub:0.92.0` built + pushed (`sha256:ca71140cf54a…`), `manifests/hub.yaml` bumped `0.91.1 → 0.92.0` in git, ArgoCD hard-refresh + **deliberate sync** (never `kubectl set image`). Verified: app **Synced / Healthy**, `rollout status` → `deployment "hub" successfully rolled out` (rc=0), running image `gitea.dooplex.hu/admin/felhom-hub:0.92.0`, pod `hub-6775889d95-gc8sk` 1/1 Running, clean startup log. **Is the 05:00 alarm confirmed silent? NO — CONFIRMATION IS PENDING, and it is stated rather than assumed.** The hub's own log says `deadline-check: next run at 2026-08-05 05:00 CEST (in 17h53m57s)`; there is no manual trigger (`CheckBackupDeadlines` has exactly one caller, the scheduler at `cmd/hub/main.go:598`). An absent alarm tomorrow would in any case be weak evidence, so a **positive** observable was taken instead: the shipped predicate was run verbatim against a fresh copy of the live DB (query #16) — `david` → **0** (will be skipped), every other customer → **1** (behaviour unchanged). **The 05:00 firing itself is still owed.** --- ## 7. `OPEN-ITEMS.md` rows, and the capability map - **R-192** — updated with Q7's confirmed numbers; **NOT closed**, and now carries a *new* reason not to fix it in isolation (a successful auto-restage would have protected the plumbing while the data was already gone). - **R-193** — updated with the whole spike; **NOT closed**, state moved to **WAITING-ON-OPERATOR**, owner `CC + operator`. The old "15+ vs 1 decides it" line is struck through and replaced with the measured answer plus the third predicted outcome. - **R-195** *(new)* — the phantom-customer alarm. **SHIPPED**, hub v0.92.0. - **R-196** *(new)* — `escrow_stale` wired to the path that does not change the repo password, absent from the one that does. **OPEN.** - **R-197** *(new)* — the hub holds both `restic_pw_sha256` values and reads neither. **OPEN.** **The capability map DID need touching, and the §N.5 escalation did NOT fire — the row was not overclaiming.** `00-capability-map.md`'s offsite row already recorded reinstall-continuity: *"a recreated data volume that orphaned the repo … is now CLASSIFIED … + a move-aside (never-delete) reset"*, shipped controller v0.142.0 and live-fired 2026-07-18. **That independently corroborates the Q3 prediction from a different direction.** What it did not record — and now does — is that this fires on an **ordinary, planned, unattended guest rebuild** on every box, with the measured 2026-08-03/04 instance, the hash evidence, and the inverted `escrow_stale` wiring. **Status unchanged (PROVEN-LIVE):** the guard is correct and is not the gap; the cadence is. `ROADMAP.md`'s newest row is R-171 and none of R-190…R-197 has one, so **there was nothing to collapse.** One correction *was* warranted under the coupling rule: R-1's row claimed the v0.57.0 reinstall arc makes offsite re-issue *"auto-fire on re-enrollment"* — scope-corrected in place, since that means **host** re-enrollment after a host **delete**, and a guest rebuild never reaches it. `CONTEXT.md` gained **S-26** (the spike's Q1/Q2 headlines + the three corrections that must not be re-inherited) and **S-27** (the R-195 ruling and its mechanism). `REUSE.md` gained the `HasEverBoundHost` row with its "never a substitute for a liveness check" trap. `STATUS.md` rewritten for the operator, one screen. --- ## 8. CI, task ids, and `--no-verify` | commit | what | CI task id | run # | conclusion | |---|---|---|---|---| | `7fff45d688fa` | R-195 code + tests + the spike + all docs | **126** | 72 | **success** | | `f581ac134964` | `manifests/hub.yaml` 0.91.1 → 0.92.0 | **127** | 73 | **success** | Matched by `head_sha` via `curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/felhom.eu/actions/tasks?limit=5"` — a PULL check, not a reliance on the failure mail. **`--no-verify` was NOT used.** The pre-push hook ran and reported `gates OK - push proceeding`; `python3 scripts/repo_gates.py` was additionally run in full (all five gates OK) before committing. **Teardown:** the spike provisioned nothing — no guest, no VM, no storage, no credential. The only artefact is a hub-DB snapshot in the session scratchpad, outside the repo. **Read-only discipline held:** no `ReissueCredentials`, no ceremony, no credential rotation, no change to any box's offsite configuration, no snapshot deleted, no customer record modified. `drill-r50` untouched; `sess-f`'s removal **confirmed** rather than assumed. --- ## 9. Observations — noticed and NOT acted on 1. **The orphaned ciphertext is retained and keeps consuming quota, with nothing to prune it.** `resetOrphanedRepo` is move-aside-never-delete by design (correct), but 40.9 MB + 1.14 GB of unopenable data will sit against a 50 GB soft quota indefinitely. S-24's ep0 prune jobs cover the **PBS** namespaces, a different tier. Not filed — it belongs to whichever option is chosen for R-193. 2. **`host_escrow_superseded` holds only two rows in the entire database**, both from 2026-08-04. That is why the spike could not establish whether demo-felhom's 2026-07-21 Re-issue re-sealed an *unchanged* password — the predicted false-staleness shape of R-196. Retention policy for that table was not investigated. 3. **Both demo guests were rebuilt on consecutive mornings** (demo-hp 08-03 06:12, demo-felhom 08-03 07:17, both to controller 0.192.0 with new config hashes). *Why* is outside this spike's scope and was not chased. 4. **`api/handler.go:1067-1069`'s comment is the third copy of R-196's false premise**, and it sits in a different package from the other two. Whoever fixes R-196 should grep rather than fix the one they were shown. 5. **`peti-felhom` is `active` with its host deleted since 2026-07-15**, judged only by the `down` skip. It is correct today and would start alarming the moment it reported once. Not a defect, but the shape is worth knowing. 6. **The `-shm`/`-wal` trap generalises.** Any future session reading the hub DB from the pod must copy all three files. Reading `hub.db` alone today would have produced confident answers from a file 2 h 15 m stale — the R-3 shape with a different filename.