felhom.eu/REPORT.md

# REPORT — R-193/R-192 spike (no code) + R-195 phantom-customer alarm (hub v0.92.0)

**Date:** 2026-08-04 · **Class:** spike (Parts 0–3) + one fenced implementation (Part 4)
**Boundary held:** the spike shipped **no production code for R-193 or R-192**, and Part 4 never
touched the spike's subject matter.

> Supersedes the earlier 2026-08-04 report (R-191 + R-190), per the overwrite convention. Nothing
> durable from it lived only here — it is recorded in `hub/CHANGELOG.md`, `CONTEXT.md` S-24/S-25 and
> `OPEN-ITEMS.md` R-190/R-191.

---

## 1. Confirmed baselines, re-read at the cited commit

| Repo | `main` @ commit on arrival | Version | End state |
|---|---|---|---|
| `felhom.eu` | `f456835bbcc4` ✔ matches the spec | hub `v0.91.1` (deployed `felhom-hub:0.91.1`) | **`v0.92.0`** built, pushed, manifest bumped, synced, **live** |
| `felhom-controller` | `0887fd676d60` | — | **untouched** (read-only) |
| `felhom-agent` | `856a127` | — | **untouched** (read-only) |

Clean-tree gate passed before the build: `git status --porcelain` empty, `HEAD == origin/main`.
Highest register ID on arrival was **R-194**; grepped before minting.

---

## 2. Q1–Q8 — each answered or explicitly marked unmeasured

Full method, quotes and query output: **`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md`**.

### Q2 and Q3 first — they decide whether the rest matters

**Q2 — does a rebuilt controller reuse the escrowed repository password, or mint a new one?**
**RULING: it MINTS A NEW ONE.**
*Method — source:* `ConfigureOffbox` → `ApplyOffsiteTarget` → `WriteOffboxSecrets`, which generates a
fresh 256-bit password whenever `<DataDir>/offbox/repo_password` is absent
(`felhom-controller/controller/internal/backup/offbox.go:392-401`). A rebuilt guest has a fresh data
dir. The only recovery path, `InjectOffboxPassword`, has **exactly one caller in the entire repo**
(census: `grep -rn "InjectOffboxPassword" --include=*.go .` → 3 hits, 1 definition, 1 caller, 1
comment) — `web/offbox_handlers.go:189`, a web form a human pastes into. **The apply-bridge never
calls it.**
*Method — live, without touching a box:* `host_escrow.restic_pw_sha256` vs
`host_escrow_superseded.restic_pw_sha256`. **demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom
`48741892…` → `c60c8bc7…`. Both CHANGED.**
**This is bigger than R-193 as filed:** demo-felhom, which R-193 records as having survived by luck,
had **36 snapshots / 1,136,685,919 B** before its rebuild and has reported `snapshot_count: 0` in all
109 reports since. Its luck restored **delivery**, not the **repository**.

**Q3 — what did the 04:15 run report?** **UNMEASURED, and the binary in the question is wrong.**
The run is daily at ~02:15 UTC (measured `last_run` values on both boxes). Neither box could run on
2026-08-04: demo-hp had no target at all until 07:15:47 UTC; demo-felhom's target was
`escrow_state: pending` from 08-03 07:19:10 to 08-04 07:20:13, which `OffboxRunnable` blocks
(configured **AND** escrowed). **The decisive run is 2026-08-05 ~02:15 UTC.** Source predicts a
**third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`;
`repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies
**orphaned** → `ErrOffboxOrphaned` → **the run refuses and shows the orphan card.** Recorded as a
prediction, not a measurement.

### The rest

- **Q1 — what is one-shot.** Full table in the spike. Headline: the hub's one-time provider password
  is one-shot **but freely re-issuable**; the box's SFTP key is **regenerated on every apply by
  design**; the **restic repository password** is the one nothing can restage — the agent's own source
  calls it *"irreplaceable (unlike the SFTP access key…)"*
  (`felhom-agent/internal/escrow/identity.go:35-39`). **R-193 was filed against the recoverable
  secret; the damage is in the irreplaceable one.**
- **Q4 — see §3 below.**
- **Q5 — why one box recovered and the other not. CONFIRMED FROM THE LEDGER.** demo-felhom's
  `one_time_secrets` row was created `2026-07-21 08:29:29` (traceable to a Re-issue at that exact
  timestamp) and sat **UNCONSUMED for 13 days** until the rebuild consumed it at
  `2026-08-03 07:17:58`. Without one: exactly demo-hp — `consume` → **404** and a retry loop, 25 hours
  unprotected. **Correction to the contrast:** the recovery was of the plumbing only (Q2).
- **Q6 — why delivery is one-shot.** The design's own reasons quoted from three sites. **The security
  half, stated:** the consume endpoint is gated by `checkAuthCustomer` — **the customer's API key, the
  same credential as the config pull** — so an automatic restage grants **no new access to an
  unauthenticated party**. What it genuinely widens is the **exposure window**: a live provider
  password becomes fetchable whenever the box reports no offsite target, rather than only in the
  seconds after an operator acts. Blast radius stays the customer's own label-scoped sub-account. The
  R-39(a) unconsumed-secret guard must survive any change.
- **Q7 — R-192's guard. BOTH HALVES CONFIRMED WITH NUMBERS.** See §4.
- **Q8 — what each candidate costs.** See §5.

---

## 3. Which of the two contradictory claims about Re-issue is true

**The claim that Re-issue "only resets the transport credential" is TRUE about what
`ReissueCredentials` does. R-39's "rotates the restic password" is FALSE.**

From source (`hub/internal/offsite/offsite.go:150-228`), the function does four things: `genPassword`,
`ResetSubaccountPassword`/`ResetBoxPassword`, `SaveOneTimeSecret`, and bookkeeping. **There is no
reference to a restic password anywhere in it, and none is possible** — that password is generated on
the box and never leaves it except into the R-wrapped escrow, which the hub cannot open.

**But the bookkeeping is justified by the false premise**, `offsite.go:198-201`: *"the restic repo
password just changed, so any existing key-escrow blob … is now STALE"*. The same wrong claim appears
at `api/handler.go:1067-1069` and in R-193's own row. **Eighth entry in `CLAUDE.md`'s table of
comments asserting a guarantee the code does not provide** — the first where the comment factually
describes a *different function*. → **R-196**.

**What corroborated it, and the corroboration is the strongest part.** The task asked: if the repo
password was untouched, what explains demo-hp's escrow going `pending` after the Re-issue? **Fully
explained, and not by the Re-issue:** `ApplyOffsiteTarget` carries `EscrowState` over from the
*existing* target, the rebuilt guest had none, so the `pending` fallthrough fired — and the box had
just minted a new repo password, so `pending` was *correct for a different reason*.
**The proof they are independent is demo-felhom:** it received **no Re-issue at all** (its only
`escrow_stale`/`offsite_reissued` pair is dated `2026-07-21 08:29:29`), its repo password changed
anyway, and **nothing marked its escrow stale for 13 hours.** The mechanism is exactly inverted:
`escrow_stale` is wired to the one path that does **not** change the repo password and absent from the
path that does.

---

## 4. Every query used, with its fields and values

The DB snapshot was taken from the live pod and its **freshness proven with a positive observable**,
not with "the query returned no error":

| # | Query / read | Result |
|---|---|---|
| 1 | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale (mtime 08:34 vs wal 10:45 CEST). Copying only the main file is the R-3 shape. |
| 2 | `PRAGMA integrity_check` | `ok` |
| 3 | `MAX(host_reports.received_at)` vs `datetime('now')` | `2026-08-04 08:45:38` vs `08:49:46` — **newest row 4 min old** |
| 4 | `SELECT customer_id, status, created_at FROM customer_configs` | `demo-felhom` active · `peti-felhom` active · `demo-hp` active · `drill-r50` **blocked** · `david` active (created `2026-08-01 16:51:49`). **`sess-f` is gone — confirmed, not assumed.** `drill-r50` untouched. |
| 5 | `hosts` / `host_deletions` / `host_reports` / `reports` counts for `david` | **0 / 0 / 0 / 0** — never bound |
| 6 | `events WHERE event_type='expected_dbdump_missed' AND created_at > '2026-07-25'` | **only `david`**, 03:00 UTC on 08-02, 08-03, 08-04 |
| 7 | `SELECT customer_id, MAX(received_at), COUNT(*) FROM reports GROUP BY customer_id` | `demo-hp` 1329 · `demo-felhom` 6690 · `drill-r50` 183 (last 07-29) · `peti-felhom` 482 (last **2026-07-15**) — the reason peti-felhom is `down` and does not alarm |
| 8 | distinct `$.offsite` objects per box with first/last seen | demo-hp: `15 snapshots / 40 927 044 B` until `08-03 05:59:21`, **ABSENT** `06:12:19`→`08-04 07:12:01` (111 reports), then `pending` 07:15:47, `escrowed` 07:16:02, `snapshot_count 0`. demo-felhom: `36 snapshots / 1 136 685 919 B` until `08-03 07:01:28`, ABSENT ×1, then `pending` `07:19:10`→`08-04 07:20:13`, `escrowed` 07:20:28, `snapshot_count 0` |
| 9 | `host_escrow` + `host_escrow_superseded` `restic_pw_sha256` | **the decisive measurement** — both changed (§2) |
| 10 | `one_time_secrets` (no values read) | demo-felhom created `2026-07-21 08:29:29`, consumed `2026-08-03 07:17:58`; demo-hp created+consumed `2026-08-04 07:11:51`/`07:12:06` |
| 11 | `$.claimed` on the latest report of each box | **`1` on both** — which selects `ensureOffboxRepo`'s refuse branch |
| 12 | `CountReportsOffsiteSince` reproduced verbatim (`ORDER BY id LIMIT 500`) for demo-hp @ `2026-07-23 09:53:41` | **`total=500, withOffsite=500`**, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** |
| 13 | same window, **no LIMIT** | `total=1174, withOffsite=1063` (⇒ 111 without — matches #8) |
| 14 | full text of the live `offsite_delivery_stuck` message | *"…and **500 report(s) since carry no offbox target**…"* while `OffsiteReportsSinceConsume` was **500** |
| 15 | `COUNT(*)` of `offsite_credential_restaged` | **0, for every customer, ever** |
| 16 | the shipped predicate run verbatim against live data (post-deploy) | `david` → **0**; `demo-felhom`/`demo-hp`/`drill-r50`/`peti-felhom` → **1** |

**Instrument limits recorded rather than discovered later:** `one_time_secrets` has `customer_id` as
PRIMARY KEY and is last-write-wins **by design**, so it holds only the newest credential per customer —
demo-hp's `2026-07-23` consume, on which R-192's whole diagnosis rests, was **overwritten** by
yesterday's Re-issue and is no longer readable from that table. The `2026-07-23 09:53:41` anchor in
query #12 comes from R-192's record, and that is stated in the spike rather than presented as a live
read.

---

## 5. Ranked options, with a recommendation

| rank | option | verdict |
|---|---|---|
| **1** | **(d) the honesty pass — now** | Cheap, weakens nothing, and **the only option that helps the incident already in flight**. Two parts: correct R-192's message + log its refusal (**R-196**), and **detect a changed `restic_pw_sha256`** — both hashes are already in the hub's DB and **nothing reads them** (**R-197**). |
| **2** | **(c) agent-retained repo password — NOT PREVIOUSLY NAMED** | The only candidate aimed at the irreplaceable secret. The agent lives on the Proxmox host, **survives a guest rebuild**, already receives the password over the pinned local API and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Every seam exists. Price: a copy of the data key at rest on the host. **The operator's trade to make.** |
| **3** | **(a) hub restages automatically** | **Already exists** as `reissueOnReenroll`'s F3 leg, wired to the wrong event (behind `handleHostEnroll`'s mint-once-reuse short-circuit; a guest rebuild leaves the `hosts` row intact). Worth doing **after** (c), never instead: on its own it restores the transport to a box whose data key is already gone, and **would have made both boxes look healthy on 2026-08-04 while their snapshots were orphaned** — strictly worse than the current loud failure. |
| **4** | **(b) recover from escrow at re-bootstrap** | **Not implementable as stated.** The escrow is R-wrapped and zero-knowledge; the hub has no recovery code. "Recoverable at re-bootstrap" describes a customer-present ceremony — i.e. the manual form that already exists. |

**Recommendation: ship (d) now; then decide (c). Do not ship (a) first.**

### The operator's question — stated, and NOT answered

> The irreplaceable secret is the restic repository password. It is generated on the box, destroyed by
> a guest rebuild, and the only surviving copy is sealed under the customer's recovery code. Every
> machine will be reinstalled.
> **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host,
> where you already have root — or do you want it to stay sealed and accept that every rebuild starts
> a new repository, provided the system says so loudly on the day it happens?**

**STOPPED here. Neither candidate was begun.**

---

## 6. Part 4 — the phantom-customer alarm (SHIPPED)

**Files changed**
- `hub/internal/store/store.go` — **new** `(*Store).HasEverBoundHost(customerID) (bool, error)`:
  a live `hosts` row **OR** a `host_deletions` tombstone.
- `hub/internal/monitor/deadline.go` — one guard at the top of `CheckBackupDeadlines`' customer loop,
  plus an `unbound` counter in the summary line.
- `hub/internal/monitor/deadline_unbound_test.go` — **new**, 5 tests.

**Commit:** `7fff45d688fa` (code + tests + spike + docs) · `f581ac134964` (manifest bump).

**The mechanism, established at source and corroborated live.** The existing down-skip could never
have covered this: it reads `StalenessChecker.GetState()`, whose map is seeded from
`store.GetCustomers()` — **a query over the `reports` table** (`store.go:1055-1067`). A customer with
zero reports is in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. **The
skip misses exactly the customer it would most obviously cover.** Corroborated by query #7 and by the
live post-deploy startup log — `Staleness checker initialized: 2 ok, 0 stale, 1 down` — where the
`1 down` is `peti-felhom` (host deleted 2026-07-15, 482 old reports) and `david` is **absent from the
census entirely**.

**The discriminator is "was a host EVER bound", not "has a report arrived"**, which is the point:
a box that was installed, bound and went silent has a `hosts` row and keeps alarming. **Fail-open on a
read error** — an unreadable binding must never suppress a real alarm. The deferral is **logged with
its own counter** (the v0.73.0 Part-7 precedent). The anchored-verdict structure is untouched, and
`david`'s record was not modified.

**Sibling verdicts checked rather than assumed.** The census of every customer-iterating checker:
`deadline.go` backup half (already guarded by `reportJSON == ""`), `restoretest.go` (guarded by
`latest == ""`), `staleness.go` + `offsite.go` (iterate `GetCustomers()`, report-derived — a hostless
customer never appears), `offsite_delivery.go` (a hostless customer derives `DeliveryNoSecret` and is
skipped). **The DB-dump half was the only verdict that could fire**; the guard covers both verdicts in
that function uniformly.

**Test results — the suite run and the exit code read BEFORE any commit, never in one command.**

```
go build ./...  → BUILD OK
go vet ./...    → VET OK
go test ./...   → rc=0, 17 packages ok, no FAIL lines
```

The 13 `TestCheckBackupDeadlines*`/`TestHasEverBoundHost` tests were run with `-v` and **the `=== RUN`
lines were checked**, per `CLAUDE.md`'s rule that a `-run` pattern matching nothing prints `ok` and
exits 0.

**THE RED-PROOF THAT MATTERS.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms` is the
case the change could break — a real box that was installed and never phoned home, indistinguishable
from `david` on every signal *except* the one the guard reads. With the guard removed it **still
passes**, which is the correct result and proves the test is aimed at the guard's *risk*, not its
presence. The companion red-proof of the fix itself, **observed**:

```
=== RUN   TestCheckBackupDeadlines_NeverBoundHost_Silent
    deadline_unbound_test.go:58: a customer with NO host ever bound must raise NOTHING;
                                 got [expected_dbdump_missed]
--- FAIL: TestCheckBackupDeadlines_NeverBoundHost_Silent
```

— verbatim the event `david` sent three mornings running. `deadline.go` restored from a backup taken
before the mutation; full suite re-run green afterwards.

**Deployed version.** `felhom-hub:0.92.0` built + pushed (`sha256:ca71140cf54a…`),
`manifests/hub.yaml` bumped `0.91.1 → 0.92.0` in git, ArgoCD hard-refresh + **deliberate sync** (never
`kubectl set image`). Verified: app **Synced / Healthy**, `rollout status` → `deployment "hub"
successfully rolled out` (rc=0), running image `gitea.dooplex.hu/admin/felhom-hub:0.92.0`, pod
`hub-6775889d95-gc8sk` 1/1 Running, clean startup log.

**Is the 05:00 alarm confirmed silent? NO — CONFIRMATION IS PENDING, and it is stated rather than
assumed.** The hub's own log says `deadline-check: next run at 2026-08-05 05:00 CEST (in 17h53m57s)`;
there is no manual trigger (`CheckBackupDeadlines` has exactly one caller, the scheduler at
`cmd/hub/main.go:598`). An absent alarm tomorrow would in any case be weak evidence, so a **positive**
observable was taken instead: the shipped predicate was run verbatim against a fresh copy of the live
DB (query #16) — `david` → **0** (will be skipped), every other customer → **1** (behaviour
unchanged). **The 05:00 firing itself is still owed.**

---

## 7. `OPEN-ITEMS.md` rows, and the capability map

- **R-192** — updated with Q7's confirmed numbers; **NOT closed**, and now carries a *new* reason not
  to fix it in isolation (a successful auto-restage would have protected the plumbing while the data
  was already gone).
- **R-193** — updated with the whole spike; **NOT closed**, state moved to **WAITING-ON-OPERATOR**,
  owner `CC + operator`. The old "15+ vs 1 decides it" line is struck through and replaced with the
  measured answer plus the third predicted outcome.
- **R-195** *(new)* — the phantom-customer alarm. **SHIPPED**, hub v0.92.0.
- **R-196** *(new)* — `escrow_stale` wired to the path that does not change the repo password, absent
  from the one that does. **OPEN.**
- **R-197** *(new)* — the hub holds both `restic_pw_sha256` values and reads neither. **OPEN.**

**The capability map DID need touching, and the §N.5 escalation did NOT fire — the row was not
overclaiming.** `00-capability-map.md`'s offsite row already recorded reinstall-continuity: *"a
recreated data volume that orphaned the repo … is now CLASSIFIED … + a move-aside (never-delete)
reset"*, shipped controller v0.142.0 and live-fired 2026-07-18. **That independently corroborates the
Q3 prediction from a different direction.** What it did not record — and now does — is that this fires
on an **ordinary, planned, unattended guest rebuild** on every box, with the measured 2026-08-03/04
instance, the hash evidence, and the inverted `escrow_stale` wiring. **Status unchanged
(PROVEN-LIVE):** the guard is correct and is not the gap; the cadence is.

`ROADMAP.md`'s newest row is R-171 and none of R-190…R-197 has one, so **there was nothing to
collapse.** One correction *was* warranted under the coupling rule: R-1's row claimed the v0.57.0
reinstall arc makes offsite re-issue *"auto-fire on re-enrollment"* — scope-corrected in place, since
that means **host** re-enrollment after a host **delete**, and a guest rebuild never reaches it.

`CONTEXT.md` gained **S-26** (the spike's Q1/Q2 headlines + the three corrections that must not be
re-inherited) and **S-27** (the R-195 ruling and its mechanism). `REUSE.md` gained the
`HasEverBoundHost` row with its "never a substitute for a liveness check" trap. `STATUS.md` rewritten
for the operator, one screen.

---

## 8. CI, task ids, and `--no-verify`

| commit | what | CI task id | run # | conclusion |
|---|---|---|---|---|
| `7fff45d688fa` | R-195 code + tests + the spike + all docs | **126** | 72 | **success** |
| `f581ac134964` | `manifests/hub.yaml` 0.91.1 → 0.92.0 | **127** | 73 | **success** |

Matched by `head_sha` via
`curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/felhom.eu/actions/tasks?limit=5"` — a PULL
check, not a reliance on the failure mail.
**`--no-verify` was NOT used.** The pre-push hook ran and reported `gates OK - push proceeding`;
`python3 scripts/repo_gates.py` was additionally run in full (all five gates OK) before committing.

**Teardown:** the spike provisioned nothing — no guest, no VM, no storage, no credential. The only
artefact is a hub-DB snapshot in the session scratchpad, outside the repo.

**Read-only discipline held:** no `ReissueCredentials`, no ceremony, no credential rotation, no
change to any box's offsite configuration, no snapshot deleted, no customer record modified.
`drill-r50` untouched; `sess-f`'s removal **confirmed** rather than assumed.

---

## 9. Observations — noticed and NOT acted on

1. **The orphaned ciphertext is retained and keeps consuming quota, with nothing to prune it.**
   `resetOrphanedRepo` is move-aside-never-delete by design (correct), but 40.9 MB + 1.14 GB of
   unopenable data will sit against a 50 GB soft quota indefinitely. S-24's ep0 prune jobs cover the
   **PBS** namespaces, a different tier. Not filed — it belongs to whichever option is chosen for
   R-193.
2. **`host_escrow_superseded` holds only two rows in the entire database**, both from 2026-08-04. That
   is why the spike could not establish whether demo-felhom's 2026-07-21 Re-issue re-sealed an
   *unchanged* password — the predicted false-staleness shape of R-196. Retention policy for that
   table was not investigated.
3. **Both demo guests were rebuilt on consecutive mornings** (demo-hp 08-03 06:12, demo-felhom 08-03
   07:17, both to controller 0.192.0 with new config hashes). *Why* is outside this spike's scope and
   was not chased.
4. **`api/handler.go:1067-1069`'s comment is the third copy of R-196's false premise**, and it sits in
   a different package from the other two. Whoever fixes R-196 should grep rather than fix the one
   they were shown.
5. **`peti-felhom` is `active` with its host deleted since 2026-07-15**, judged only by the `down`
   skip. It is correct today and would start alarming the moment it reported once. Not a defect, but
   the shape is worth knowing.
6. **The `-shm`/`-wal` trap generalises.** Any future session reading the hub DB from the pod must
   copy all three files. Reading `hub.db` alone today would have produced confident answers from a
   file 2 h 15 m stale — the R-3 shape with a different filename.