R-195: a customer with no machine ever bound does not alarm (hub v0.92.0) + R-193/R-192 spike
gates / gates (push) Successful in 7s

Part 4 (ships): `david` — a prospective customer with hosts=0, host_deletions=0,
reports=0 — e-mailed an expected_dbdump_missed ERROR at 03:00 UTC three mornings
running. The existing down-skip could never cover it: it reads the staleness
checker's state, which is seeded from a query over the `reports` table, so a
customer that never reported has no state at all and GetState() returns "" rather
than "down". store.HasEverBoundHost (hosts row OR host_deletions tombstone) is
consulted once per customer at the top of the deadline loop. The discriminator is
"was a host EVER bound", never "has a report arrived" — a box installed and never
heard from is a real fault and keeps alarming. Fail-OPEN on a read error. Red-proof
observed: removing the guard fails with `got [expected_dbdump_missed]`, verbatim the
event david sent.

Parts 0-3 (spike, NO production code for R-193/R-192):
audits/SPIKE-offsite-credential-recovery-2026-08-04.md establishes that the one-shot
provider password is the RECOVERABLE secret and the restic repository password is the
irreplaceable one — and that a guest rebuild mints a fresh one, orphaning the previous
off-site history. Measured without touching a box, by comparing
host_escrow.restic_pw_sha256 against host_escrow_superseded: BOTH demo boxes changed
(demo-hp 15 snapshots / 40.9 MB, demo-felhom 36 snapshots / 1.14 GB). demo-felhom's
"lucky" 76-second recovery restored delivery and not the repository, silently, for 13h.
ReissueCredentials does NOT rotate the restic password (R-39's record and two hub
comments are wrong -> R-196); candidate (b) is not implementable against a
zero-knowledge escrow; candidate (a) already exists as F3 and is wired to the wrong
event. Ends in ranked options and an unanswered question for the operator.

R-195 SHIPPED; R-196 + R-197 filed; R-192 + R-193 updated, neither closed.
This commit is contained in:
admin committed 2026-08-04 11:04:39 +02:00
1 parent f456835bbc
commit 7fff45d688
9 files changed
+815 -6

No files matched your search

@@ -76,7 +76,7 @@
| Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded **DEGRADED** rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | `E2D-fresh-vm-2026-07-29` C1 (real 1.22.0 install, rc=0, `Day-0 provision SUCCESS`) + C2 (both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort) | Case A (a second drive already present at install) has never fired naturally — only Case B has |
| Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 **DB replay route → `07-backup-architecture.md` §8 row 3** |
| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | **Route + RTO → `07-backup-architecture.md` §8 rows 1, 2, 4, 5.** The matrix records that the copy's `recovery-unit/` mirror is read by no path (→ R-102) |
| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` **Route + RTO → `07-backup-architecture.md` §8 rows 4, 10, 12, 15** (incl. the R-95 delete exposure and the R-104 stale-lock defect) |
| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` **Route + RTO → `07-backup-architecture.md` §8 rows 4, 10, 12, 15** (incl. the R-95 delete exposure and the R-104 stale-lock defect). **2026-08-04 (R-193/R-197, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`) — the row is NOT overclaiming and the guard is not the gap; the CADENCE is.** This row already recorded that a recreated data volume orphans the repo, and the spike confirms the mechanism at source: `WriteOffboxSecrets` (`offbox.go:392`) mints a fresh 256-bit repo password whenever `<DataDir>/offbox/repo_password` is absent, and **no automatic path ever consults the escrowed one** — `InjectOffboxPassword` has exactly one caller, a web form a human pastes into. What was NOT recorded is that this now fires on an **ordinary, planned, unattended guest rebuild**, on every box: measured on BOTH demo boxes 2026-08-03/04 by comparing `host_escrow.restic_pw_sha256` against `host_escrow_superseded.restic_pw_sha256` (demo-hp `8e03eddf…`→`8a9e33aa…`, demo-felhom `48741892…`→`c60c8bc7…`), orphaning **15 snapshots / 40.9 MB** and **36 snapshots / 1.14 GB** respectively. **demo-felhom is the important half:** it kept its DELIVERY (a stale staged secret restored the target in 76 s) and lost its REPOSITORY anyway, with **nothing marking the escrow stale for 13 h** — `escrow_stale` is wired to `ReissueCredentials`, the one path that does NOT change the repo password (**R-196**), and absent from the rebuild path that does. Status unchanged: the classify-and-move-aside guard remains PROVEN-LIVE and correct, and is predicted (not yet measured) to refuse the 2026-08-05 run on both boxes rather than start a silent fresh history |
| Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** (2026-07-20) | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | **2026-07-19:** `audits/DIAG-immich-restore-2026-07-19.md` finds **no offsite path loads a DB dump** — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "**immich end-to-end from offsite alone**" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. **RULED 2026-07-19 (Viktor): 6D's destruction hit the FILE TREE ONLY — the database survived in its named volume** (`immich_postgres_data` is a named volume in both the v2 and v3 template eras), so "immich end-to-end from offsite alone" **overclaimed scope**: the file half was proven, the DB half was never destroyed and therefore never restored. Row downgraded PROVEN-LIVE → **PARTIAL**, scope-corrected. Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (no offsite path could replay a DB at all) + the P-FAB destructive re-import (the proven-replay evidence, on the LOCAL path). **2026-07-19, controller v0.148.0:** the DB half now exists in code (R-43 + R-44) and its replay reached a live box — but **round 2 found it aborts against a running app** (`audits/DIAG-immich-restore-round2-2026-07-19.md`, H4: the replay races immich's own schema repair; `clip_index` recreated by the app 2 s before the dump's CREATE INDEX). **2026-07-20, controller v0.153.0: H4 IS CLOSED (R-47)** — both restore paths now replay into a DB-ONLY window (`StartStackServices` brings up the database service alone; the app starts only after the replay exits 0), with a fail-closed refusal when a dump has no identifiable DB service. *(The earlier note here said "closes in v0.149" — that was wrong: v0.149.0 was the F3 dashboard `BackupStatus` fix. R-47 shipped in v0.153.0.)* **2026-07-20: the clean run HAPPENED** — endpoint-level supervised reconstitute of immich from snapshot `49e7cb46` (the very snapshot that aborted in round 2): stop → DB-service-only start → replay rc-0 → full start, no `already exists`, operation reported SUCCESS, immich's own DatabaseService logged `No schema drift detected` twice, 11 assets `active`, 4/4 containers healthy, 231 `public` indexes. **Operator confirmed the immich timeline renders correctly after the reconstitute** (screenshot held, 2026-07-20). Evidence: `felhom-controller/REPORT.md` §4b. **2026-07-20, LATER THE SAME DAY — the destructive drill RAN and the row now earns PROVEN-LIVE.** The operator deleted the photos in immich own UI **and emptied the trash** (the step whose absence makes a drill prove nothing — the round-1 lesson), then restored through the customer-facing UI. **`40 file(s) placed`** against the 6 of the earlier non-destructive run — the files were really gone and really came back — plus 1 DB dump replayed rc-0, 11 assets `active`, `No schema drift detected`, timeline confirmed by the operator. This is the destroy-then-recover proof the 6D downgrade asked for, and it was taken through the customer own buttons, not endpoint shortcuts. Evidence: `felhom-controller/REPORT.md` 4e **Route + RTO → `07-backup-architecture.md` §8 rows 3, 4** — the matrix also records that no offsite action unpacks the named-volume tars it captures (→ R-107) |
| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill **Route + RTO → `07-backup-architecture.md` §8 rows 6, 8, 9** — measured 84–112 s local / 1101 s PBS into a scratch guest; a restore to a DIFFERENT host is unmeasured |
@@ -0,0 +1,501 @@
# SPIKE — what is actually one-shot, and what a guest rebuild really costs
**Date:** 2026-08-04 · **Items:** R-193 (rebuild drops offsite), R-192 (the alert says the opposite of
what it measured) · **Class:** spike — **no production code shipped for R-193 or R-192**
**Baselines read:** `felhom.eu` @ `f456835bbcc4` (hub v0.91.1) · `felhom-controller` @ `0887fd67` ·
`felhom-agent` @ `856a127`
> **This document ends in ranked options and a STOP.** It establishes facts; it does not choose.
> The operator's question is stated, unanswered, at the end.
---
## 0. Why this exists
Two candidate fixes were named when R-193 was filed, and neither could be weighed, because two prior
session reports described the same mechanism in contradictory terms:
- R-39's record: Re-issue *"rotates the restic password and makes the escrow stale"*.
- R-193's own resolution note: the one-time password is *"only the transport credential used once to
install the box's own SSH key"*.
Those two worlds imply very different fates for the fifteen existing snapshots. This spike settles it
**from source first, live state second**, so a measurement can contradict the code rather than be
interpreted by it.
**The headline, before the detail:** *both prior claims are wrong in the way that matters.* Re-issue
does **not** touch the restic repository password — but a **guest rebuild does**, silently, on every
box, with nothing marking the escrow stale and nothing telling anyone. Both demo boxes lost repository
continuity on 2026-08-03/04. The one that "recovered by luck" lost it too.
---
## 1. The instruments, and what each one can and cannot answer
Stated up front so a wrong instrument is visible here rather than inside a conclusion (§8 rule 2).
| # | Instrument | Exact query / read | Known limit |
|---|---|---|---|
| I1 | hub SQLite snapshot | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale at copy time (mtime 08:34 vs wal 10:45 CEST). Copying only the main file would have produced confident answers from a stale file — the exact R-3 shape. |
| I1-check | freshness proof | `PRAGMA integrity_check` → `ok`; `MAX(host_reports.received_at)` = `2026-08-04 08:45:38`, `datetime('now')` = `08:49:46` | **Positive observable**: newest row 4 min old. Not "the query returned no error". |
| I2 | `host_escrow.restic_pw_sha256` + `host_escrow_superseded.restic_pw_sha256` | see §2 | sha256 of a 256-bit random secret — non-reversible, safe to compare. Answers *"did the repo password change"* **without touching a box or a credential**. |
| I3 | `reports.report_json → $.offsite` | grouped distinct-object query with first/last seen | The report echoes the box's own target state; absence of the key = no target configured. |
| I4 | `one_time_secrets` | `SELECT customer_id, created_at, consumed_at` | **`customer_id` is the PRIMARY KEY** — one row per customer, last-write-wins. It **cannot** answer "how many credentials were delivered"; a Re-issue destroys the prior row. demo-hp's 2026-07-23 consume is no longer in this table. |
| I5 | source | `hub/internal/offsite/*`, `hub/internal/api/*`, `controller/internal/{backup,offsiteapply}`, `agent/internal/escrow` | — |
**Not used, deliberately:** no `ReissueCredentials`, no ceremony, no config change, no deletion, no
write to any box. Nothing on either box or on the storage endpoint was altered by this session.
---
## Q2 — Does a rebuilt controller reuse the escrowed repository password, or mint a new one?
### **RULING: it MINTS A NEW ONE. The escrowed password is never consulted on any automatic path.**
**Method — source.** The apply-bridge's terminal step is
`Enabler.ConfigureOffbox(...)` (`controller/internal/offsiteapply/offsiteapply.go:221,256`), which
reaches `Manager.ApplyOffsiteTarget` → `WriteOffboxSecrets`
(`controller/internal/backup/offbox.go:370`). Quoted verbatim, `offbox.go:392-401`:
```go
// Auto-generate the repo password once (0600), never log it.
if _, err := os.Stat(m.offboxPwPath()); os.IsNotExist(err) {
pw, gerr := generateOffboxPassword()
...
if werr := os.WriteFile(m.offboxPwPath(), []byte(pw), 0o600); werr != nil {
```
`offboxPwPath()` is `<DataDir>/offbox/repo_password` (`offbox.go:363`). A rebuilt guest has a fresh
data dir, so the file is absent, so **a fresh 256-bit password is minted**.
**The only path that recovers the old one is `InjectOffboxPassword`** (`offbox.go:541`), whose sole
caller in the entire repo is `offboxInjectPasswordHandler`
(`controller/internal/web/offbox_handlers.go:189`) — a **web form where a human pastes the recovered
password**. Verified by census: `grep -rn "InjectOffboxPassword" --include=*.go .` → three hits, one
definition, one caller, one comment. **The apply-bridge never calls it.**
**Method — live, and it is decisive without touching a box (I2).** The hub already stores the sha256
of the escrowed repo password, so the question is answerable as a hash comparison:
| host | superseded (pre-rebuild) | current (post-rebuild) | verdict |
|---|---|---|---|
| `demo-hp-bb76ea` | `8e03eddf9ff7…` created `2026-07-23T10:01:17Z` | `8a9e33aa4da6…` created `2026-08-04T07:15:36Z` | **CHANGED** |
| `demo-felhom-8363b5` | `48741892f0ef…` created `2026-07-21T08:38:33Z` | `c60c8bc737a6…` created `2026-08-04T07:20:07Z` | **CHANGED** |
**Both boxes minted a new repository password.** demo-hp's pre-rebuild escrow (`8e03…`, sealed eight
minutes after its 2026-07-23 09:53:41 apply) is the key to its 15 snapshots. That key now exists
**only inside a superseded, R-wrapped escrow blob** that the hub cannot open.
### **This is bigger than R-193 as filed, and it hits the box R-193 called lucky.**
R-193 records demo-felhom as having *"survived the SAME rebuild by luck"* — 76 seconds of downtime.
Measured (I3), demo-felhom's pre-rebuild offsite object was:
```
last_run 2026-08-03T02:17:44Z · last_status ok · snapshot_count 36 · repo_size_bytes 1 136 685 919
```
**36 snapshots, 1.14 GB.** Since 2026-08-03 07:19:10 UTC it has reported `snapshot_count: 0,
repo_size_bytes: 0` in every one of 109 reports, and its repo password hash changed.
> **The luck recovered DELIVERY. It did not recover the REPOSITORY.** The staged secret restored the
> transport in 76 seconds and the box then minted a brand-new repo password anyway. The contrast
> R-193 draws between the two boxes is real for the *credential*, and **false for the data**: both
> boxes lost repository continuity, one loudly and one silently. The silent one is worse.
---
## Q3 — What did the 04:15 run report?
### **RULING: UNMEASURED — the decisive run has not happened yet. And the binary the question offers is the wrong one; source says the answer will be NEITHER 15 nor 1.**
**Why it has not happened.** The scheduled off-box run fires at ~02:15 UTC (04:15 CEST). Measured
from I3 — `last_run` values `2026-08-02T02:15:52Z`, `2026-08-03T02:16:39Z` on demo-hp;
`2026-08-01T02:17:54Z`, `2026-08-02T02:17:01Z`, `2026-08-03T02:17:44Z` on demo-felhom. **Daily, both
boxes.** Then:
| box | why 2026-08-04 02:15 UTC produced nothing |
|---|---|
| demo-hp | no offsite target existed at all — the `offsite` key was **ABSENT** from every report between `2026-08-03 06:12:19` and `2026-08-04 07:12:01` UTC (111 reports). The re-issue landed at 07:15:47, five hours after the window. |
| demo-felhom | the target existed but `escrow_state: pending` from `2026-08-03 07:19:10` to `2026-08-04 07:20:13`. `OffboxRunnable` requires configured **AND** escrowed (`offbox.go:564-569`), so the fork-4 gate blocked the run. The ceremony landed at 07:20:28. |
Both boxes became runnable only this morning. **The decisive run is 2026-08-05 ~02:15 UTC on both.**
Reported as unmeasured rather than inferred.
### What source says will happen — and it is a third outcome
Same sub-account, same repo path, new password. Measured coordinates (I3, `$.dr_recipe`):
`u629488-sub3.your-storagebox.de:/home/felhom-repo` — **unchanged** across the whole incident, and
`repoPath` is a compile-time constant (`hub/internal/offsite/offsite.go:85`). So the next run opens an
**existing repo with the wrong key**, which the codebase already has a name for
(`controller/internal/backup/offbox.go:67-93`):
```go
// ErrOffboxOrphaned is the sentinel returned when the offsite repo exists but is keyed under a
// passphrase this controller no longer has (the reinstall shape) ...
case strings.Contains(s, "wrong password or no key found"):
return "orphaned"
```
and `ensureOffboxRepo` (`offbox.go:663-679`) branches on claim state:
```go
if !m.settings.GetClaimed() { // UNCLAIMED → auto move-aside + re-init
...
m.markOrphaned()
return ErrOffboxOrphaned // CLAIMED → skip the run, show the orphan card
```
Both boxes report `claimed: 1` (I3, `$.claimed`, latest report each). **So the predicted outcome is:
the run REFUSES with `ErrOffboxOrphaned`, produces no snapshot at all, and waits for the customer to
confirm a reset.** Not 15 (reattached) and not 1 (silently fresh) — a **third** outcome the question's
framing did not contain. That is the good news half: the system is not going to quietly start a new
history over the old one. It will stop and say so.
**Consequence if the customer confirms the reset:** `resetOrphanedRepo` (`offbox.go:270-314`) moves the
old repo aside and re-inits — *"move-aside, not deleted"*. The 40.9 MB + 1.14 GB of old ciphertext
survives, unreadable without the superseded escrow, **and keeps consuming the 50 GB soft quota
indefinitely.** Nothing prunes a moved-aside restic repo (S-24 covers ep0's PBS namespaces, a
different tier).
**To measure it tomorrow:** re-read `$.offsite.last_status` / `last_error` on both boxes after
02:20 UTC, and `$.offsite.repo_state` for the orphan flag. **Predicted `ErrOffboxOrphaned`, not a
snapshot count.** Whoever picks this up should record which of the three actually occurred — a
prediction from source is not a measurement.
---
## Q1 — What is one-shot, and what is not?
Three secrets, and they have almost nothing in common.
| | **① storage-provider password** | **② the box's SFTP key** | **③ restic repository password** |
|---|---|---|---|
| **What it is** | the Hetzner sub-account / box password | ed25519 keypair for `sftp` transport | the **data** key — restic's repo encryption |
| **Generated by** | **hub** — `genPassword()` (`offsite.go:481`) | **controller** — `KeyGen.Generate()` (`offsiteapply.go:234`) | **controller** — `generateOffboxPassword()`, 32 random bytes hex (`offbox.go:406`) |
| **Stored where** | `one_time_secrets` (hub, plaintext, **one row per customer**) | `<DataDir>/offbox/ssh_key` 0600 (box only) | `<DataDir>/offbox/repo_password` 0600 (box) + inside the R-wrapped escrow blob (hub, opaque) |
| **Delivered how** | `GET …/offsite/consume-password`, **served exactly once** then marked consumed (`api/offsite.go:9-31`) | never delivered — installed onto the provider by ssh-copy-id using ① | never delivered anywhere; it never leaves the box except into the escrow ceremony |
| **Re-issuable?** | **YES** — `ReissueCredentials` resets it at the provider, any time, operator-initiated | **YES** — regenerated on every full apply; `dr_recipe.go:45` states it plainly: *"the SFTP access key is regenerated at DR"* | **NO automatic path.** Recoverable only by unsealing the escrow with the customer's recovery code and pasting it into `offboxInjectPasswordHandler` |
| **What a guest rebuild does to it** | nothing (it lives on the hub / at the provider) | destroys it; a fresh one is minted and installed — **harmless** | **destroys it; a fresh one is minted — CATASTROPHIC for the existing repo** |
| **Escrowed?** | no | no | **yes** — `IdentityBundle.ResticRepoPassword` |
The agent's own source names the asymmetry, and it is the sentence the whole spike turns on
(`felhom-agent/internal/escrow/identity.go:35-39`):
> `ResticRepoPassword` … **It is the DATA key for the offsite tier — irreplaceable (unlike the SFTP
> access key, which is regenerable at DR).**
**The one-shot thing (①) is the recoverable one. The irreplaceable thing (③) is the one nothing
re-stages.** R-193 was filed against ①. The damage is in ③.
**Instrument caveat, worth carrying (I4):** `one_time_secrets` has `customer_id` as PRIMARY KEY, and
`SaveOneTimeSecret` is last-write-wins **by design** (the R-39(a) guard comment in
`monitor/offsite_delivery.go:26-30` depends on it). So the delivery ledger holds only the newest
credential per customer: demo-hp's 2026-07-23 consume, on which R-192's whole diagnosis rests, was
**overwritten** by the 2026-08-04 Re-issue and is no longer readable from that table.
---
## Q4 — Which claim about Re-issue is true?
### **RULING: `ReissueCredentials` touches ONLY the provider credential (①). It does NOT touch the restic repository password. R-39's record is wrong — and the hub's own comments repeat the same wrong claim in three places.**
**Method — source, quoted rather than summarised.** `ReissueCredentials`
(`hub/internal/offsite/offsite.go:150-228`) does exactly four things:
1. `genPassword()`;
2. `ResetSubaccountPassword` / `ResetBoxPassword` + `WaitAction` — **the provider account password**;
3. `Store.SaveOneTimeSecret(customerID, pw)` — stage ① for delivery;
4. bookkeeping: `MarkEscrowStale` + `offsite_reissued` + `escrow_stale` events.
There is **no reference to a restic password anywhere in the function**, and none is possible — the
repo password is generated on the box and never leaves it except into the escrow blob. The hub does
not hold it in any openable form (`api/handler.go`: *"The hub stores the bytes and NEVER decrypts them
(it has no recovery code)"*).
**But step 4 is justified by a claim that is false**, `offsite.go:198-201`:
> `// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing`
> `// key-escrow blob — which sealed the OLD password — is now STALE.`
The escrow *is* marked stale; the stated reason for marking it is not something this function does.
The same false premise appears twice more: `api/handler.go:1067-1069` (*"The re-issuer resets the
restic repo password, which makes the OLD escrow blob stale"*) and `OPEN-ITEMS.md`'s R-193 row, which
inherited it from R-39.
> **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does
> not provide** — and the first where the comment is not merely unenforced but factually describes a
> different function. It survived because it reads as settled and because its *effect* (a stale escrow)
> is real, so nobody checked the *cause*.
**Consequence, and it is a live defect, not a documentation nit.** On the ordinary Re-issue shape —
a consumed-but-failed install on a box that still has its `repo_password` file — the box re-applies,
`WriteOffboxSecrets` finds the file present and **keeps it**, and the repo password is unchanged. The
hub has nonetheless told the customer, in Hungarian, that their recovery escrow is stale and asked
them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** → **R-196**.
**Corroboration against what actually happened on demo-hp yesterday.** The escrow went `pending` after
the Re-issue and the task asks: if the repo password was untouched, what explains that? **It is fully
explained, and not by the Re-issue.** Two independent mechanisms, both hub-side-invisible:
1. `ApplyOffsiteTarget` (`offbox.go:482-494`) carries `EscrowState` over from the *existing* target —
but the rebuilt guest had **no existing target**, so `cur` was nil, so the fallthrough
`if tgt.EscrowState != "escrowed" { tgt.EscrowState = "pending" }` fired.
2. The box had minted a **new** repo password moments earlier (Q2), so `pending` was *correct* — the
new key genuinely was not escrowed.
**The Re-issue's `escrow_stale` event fired at 07:11:51 for the wrong reason and happened to be true
for a different one.** Coincidence, not correctness. The proof that these are independent is
**demo-felhom**: it received **no Re-issue at all** (measured — its only `escrow_stale` /
`offsite_reissued` events in the entire history are dated `2026-07-21 08:29:29`), its repo password
changed anyway, and **nothing marked its escrow stale for thirteen hours.**
> **The mechanism is exactly inverted.** `escrow_stale` is wired to the one path that does **not**
> change the repo password, and absent from the path that **does**.
---
## Q5 — Why did one box recover itself and the other not?
### **RULING: confirmed from the ledger — an unconsumed staged secret, thirteen days old. And the recovery was partial in a way the narrative missed.**
**Method — I4, the `one_time_secrets` ledger, no values read:**
| customer | `created_at` | `consumed_at` |
|---|---|---|
| `demo-felhom` | `2026-07-21 08:29:29` | `2026-08-03 07:17:58` |
| `demo-hp` | `2026-08-04 07:11:51` | `2026-08-04 07:12:06` |
demo-felhom's row was **created 2026-07-21 and sat UNCONSUMED for 13 days** — traceable to a Re-issue
at that exact timestamp (its `offsite_reissued` + `escrow_stale` event pair, `2026-07-21 08:29:29`).
When the rebuild dropped its target on 2026-08-03, the fresh controller consumed the spare at 07:17:58
and was reporting `offsite` again by 07:19:10. **76 seconds, on a credential nobody had planned to
leave there.** Confirmed from the ledger, not the narrative.
**What would have happened without one:** exactly demo-hp — `consume` → **404** (`api/offsite.go:20-22`,
`no unconsumed offsite password`), the apply-bridge returns
`offsite-apply: consume one-time password: …` and retries forever, and the tier stays absent until an
operator acts. demo-hp sat there **25 hours** (`2026-08-03 06:12:19` → `2026-08-04 07:12:01`).
**The correction to the contrast (see Q2):** demo-felhom's spare restored *delivery* only. Its 36
snapshots / 1.14 GB were orphaned by the same rebuild. **Neither box's data survived; one box's
plumbing did.**
*Not established:* demo-felhom's escrow generation before 2026-07-21. `host_escrow_superseded` holds
only two rows in the whole database (ids 3 and 4, one per box, both superseded 2026-08-04), so whether
the 2026-07-21 Re-issue's ceremony re-sealed an *unchanged* password — the predicted false-staleness
shape — **cannot be shown from this data**. Stated as unknown rather than asserted.
---
## Q6 — Why is delivery one-shot at all?
### **RULING: the design's reason is that ① is a LIVE PROVIDER PASSWORD, not a bootstrap token; and the honest answer to the security question is that an automatic restage weakens EXPOSURE WINDOW, not authentication.**
**The design's own reasons, from source:**
- `api/offsite.go:9-12` — *"serves the one-time transient offsite password to the controller **EXACTLY
ONCE** … The value is returned once then marked consumed — a second call 404s. NEVER logged."*
- `offsite.go:145-149` — *"the **EXPLICIT** operator recovery for a consumed-password dead-end … It is
**NOT** implicit rotation: ProvisionOffsite never calls this."*
- `offsiteapply.go:1-6` — the apply-bridge is *"idempotent (a descriptor hash marker prevents
**re-consuming a spent password**)"*, i.e. the design treats a wasted consume as a real hazard, which
R-71a's whole settle-gate exists to prevent.
**The security half, stated rather than skipped.** The question is: *could a machine that is not the
real one obtain a credential via an automatic restage?*
**What stands there today is authentication, and a restage does not remove it.** The consume endpoint
is gated by `checkAuthCustomer` (`api/offsite.go:14-18`) — **the customer's API key, the same
credential as the config pull**. Anything that could consume a restaged password can already pull the
customer's whole config. So an automatic restage **grants no new access to an unauthenticated party**.
**What it genuinely weakens is the exposure window.** Today a live Hetzner sub-account password is
fetchable only in the seconds after an operator deliberately staged one. Under an automatic restage it
becomes fetchable **whenever the box reports no offsite target** — a condition an attacker holding the
API key can arguably induce. The credential's blast radius at the provider is the customer's own
sub-account (label-scoped; `ReissueCredentials` refuses unless the label lookup finds **exactly one**,
`offsite.go:164,180`), so this is a widening of *when*, not of *what*.
**And one further guard must survive any change** — R-39(a), `monitor/offsite_delivery.go:26-30`:
`SaveOneTimeSecret` is last-write-wins, so restaging on top of an **unconsumed** secret clobbers a
password a box may be about to consume. Any automatic restage must re-read the row immediately before
acting and refuse unless it is still a **consumed** row. That guard is correct and is not what R-192 is
about.
---
## Q7 — R-192's guard: exactly what it reads, and why it read wrong
### **RULING: both halves confirmed, with numbers. The guard reads the OLDEST 500 reports since the consume, and for demo-hp all 500 predated the rebuild by six days.**
**The query, quoted** (`hub/internal/store/store.go:987`):
```sql
SELECT report_json FROM reports WHERE customer_id = ? AND received_at > ? ORDER BY id LIMIT 500
```
`ORDER BY id` ascending with `LIMIT 500` = **the oldest 500**, not the newest.
**Reproduced against the live data (I1)**, with demo-hp's real consume anchor `2026-07-23 09:53:41`
(recovered from R-192's record — it is no longer in `one_time_secrets`, see I4):
| measurement | value |
|---|---|
| `total` (what the guard sees) | **500** |
| `withOffsite` (what the guard sees) | **500** |
| oldest report in that window | `2026-07-23 09:53:47` |
| newest report in that window | **`2026-07-28 11:17:40`** |
| true `total` over the same window, no LIMIT | 1174 |
| true `withOffsite` over the same window, no LIMIT | 1063 (⇒ 111 without — matching the 111 ABSENT reports in I3) |
**The whole 500-report evidence set ends 2026-07-28 — six days before the 2026-08-03 rebuild.**
**Defect (a) — the message is exactly inverted.** `maybeEmitStuck`
(`monitor/offsite_delivery.go:~110`) interpolates `status.ReportsSinceConsume` into a string that
hardcodes the phrase *"carry no offbox target"*, and never reads `OffsiteReportsSinceConsume`. The
live e-mail, quoted in full from the events table:
> *"Offsite delivery stuck: one-time password consumed 284h19m0s ago and **500 report(s) since carry
> no offbox target** — the credential is likely burned (apply died between consume and persist).
> Re-issue delivers a fresh one."*
`OffsiteReportsSinceConsume` for that same call was **500**. Every one of them carried a target. The
message states the precise negation of its own measurement, and prescribes a remedy for a failure mode
that did not occur.
**Defect (b) — the heal refuses silently, on stale evidence.** `maybeHeal` returns bare on
`status.OffsiteReportsSinceConsume != 0` (it was 500), with **no log line**. `offsite_credential_restaged`
has never fired for any customer — confirmed: zero rows of that event type in the database.
**What the guard SHOULD discriminate on — the shape, not the code.** The guard's intent is right:
*"there is offbox evidence, so this is a regressed apply, not a burned credential — the operator's
call."* Its error is that **it asks the question of the wrong time period**. Offbox evidence from
before a rebuild is not evidence that the credential still works. The discriminator must be
**recency-bounded and rebuild-aware**: judge on evidence *after the newest `controller_started` /
config-hash change*, or on the latest N reports — never on "everything since the consume". And a
refusal must **name its reason in the log**, because "we chose not to heal" and "the heal never ran"
must not look identical (the S-16 rule, one layer down).
**Do not fix it in isolation.** Under Q2's finding, a successful auto-restage on demo-hp would have
restored the transport and the box would *still* have minted a new repo password. **The heal cannot
protect the data; it can only protect the plumbing.** Whatever shape it takes must say so.
---
## Q8 — What each candidate would cost
**A fourth option exists and is the important one**, because (a) and (b) both address ① and Q2 shows
the damage is in ③.
### (a) The hub restages automatically when a re-enrolled box reports no offsite
**A version of this ALREADY EXISTS and is wired to the wrong event.** `reissueOnReenroll`
(`hub/internal/api/handler.go:1051-1084`), leg **F3**:
> *"F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
> one-time password only ever reached the OLD controller)."*
It is called from `handleHostEnroll` — but **after** a mint-once-reuse short-circuit
(`handler.go:1004-1016`): `if existing != nil { …return… }`. **A guest rebuild leaves the `hosts` row
intact** (measured: `demo-hp-bb76ea` is the same host_id throughout the incident; only the agent leaf
re-keyed, `host_leaf_changed` 2026-08-03 06:09:40), so re-enroll returns the existing credential and
**F3 is never reached.**
- **Changes:** hub only. Either lower the trigger from host-enrollment to a controller-level rebuild
signal (the `config_hash` change + `controller_started` pair the hub already receives and already
logs), or fix R-192's guard so the existing R-71c self-heal can act.
- **Weakens:** the exposure window in Q6. Needs the R-39(a) unconsumed-secret guard kept intact.
- **Does NOT solve:** ③. The box still mints a new repo password and still orphans the repo. **This
option restores the plumbing to a box whose data key is already gone.**
- **Cost:** small. **Value against the actual harm: near zero.**
### (b) The credential becomes recoverable from escrow at re-bootstrap
- **BLOCKED AS STATED, and this is a hard constraint, not an estimate.** The escrow blob is
**R-wrapped and zero-knowledge** — the hub holds opaque bytes and has no recovery code
(`api/handler.go`, and D6 in `CONTEXT.md` S-3). **A rebuilt box cannot unseal it without the
customer's recovery code.** Any design that says "recover it at re-bootstrap" is describing a
customer-present ceremony, i.e. what the existing manual `offboxInjectPasswordHandler` already is.
- **Does NOT solve:** an unattended rebuild, which is the whole scenario.
- **Cost: not implementable as specified.**
### (c) — NOT PREVIOUSLY NAMED — the agent retains the repo password across a guest rebuild
The agent lives **on the Proxmox host** and survives a guest rebuild. It already receives the repo
password today: the controller pushes it over the pinned local API
(`POST /escrow/stage-secret` → `agent/internal/localapi/escrow_stage.go`), and the agent writes it to
the fixed 0600 path `/var/lib/felhom-agent/escrow-stage/restic_repo_password`
(`agent/internal/escrow/identity.go:44-46`) **transiently — wiped by the ceremony**
(`WipeStagedResticPassword`). The seam, the transport, the pinning and the file are all already built.
- **Changes:** agent (retain rather than wipe; serve back over the same pinned local API) + controller
(on a fresh data dir, ask the agent before minting — the `InjectOffboxPassword` seam already exists
and is exactly the right shape).
- **Weakens:** the escrow's *zero-knowledge* posture in one specific way — the irreplaceable data key
would now sit at rest on the Proxmox host, which the operator has root on. **Measured against D6,
that is not a new exposure in principle** (D6 already states plainly that *"the operator cannot read
customer data"* was never the security property, and the operator holds root on every box) — but it
IS a new copy, and copies are the thing R-133 is already open about for the hub DB. **This is a real
trade and it is the operator's to make.**
- **Solves:** exactly the failure that occurred, unattended, on every rebuild, without any hub
involvement and without any credential leaving the premises.
- **Cost:** medium. Two repos, one existing seam each.
### (d) Do neither — keep the manual Re-issue, and fix what LIES
- **Changes:** R-192's message + refusal log; correct the three false comments about what Re-issue
rotates; and — the part that actually matters — **make a repo-password change VISIBLE**. The hub
already receives `restic_pw_sha256` on every escrow upload and already stores it. Comparing the new
hash against the superseded one is a two-line verdict, and it is the signal that was missing on
demo-felhom for thirteen hours → **R-197**. (Note the shape: **this spike answered its own hardest
question with a comparison the hub could be making automatically and is not.** Both values were
already in the database; nothing read them.)
- **Weakens:** nothing.
- **Does NOT solve:** the data loss. Every rebuild still orphans the repo; the operator simply finds
out the same day instead of never.
- **Cost:** small.
### Ranked, with a recommendation
| rank | option | why |
|---|---|---|
| **1** | **(d) now — the honesty pass** | It is cheap, it weakens nothing, and **it is the only option that helps the incident already in flight.** The alarm that lies is worse than no alarm; the repo-password-changed detector is the one signal whose absence let demo-felhom lose 1.14 GB of history silently. Ship this regardless of what is decided about the rest. |
| **2** | **(c) — agent-retained repo password** | The only candidate that addresses ③, which is where the harm is. Every seam it needs exists. It costs one deliberate trade-off the operator must actually make. |
| **3** | **(a) — auto-restage** | Worth doing **after** (c), never instead of it. On its own it restores the transport to a box whose data key is already gone, and would have made both boxes look healthy on 2026-08-04 while the snapshots were orphaned — a *strictly worse* outcome than the current loud failure. |
| **4** | **(b) — recover from escrow at re-bootstrap** | Not implementable as stated; the escrow is zero-knowledge by design. |
**Recommendation: ship (d), then decide (c).** And do not ship (a) first — it would have hidden this.
---
## What could not be established
Named rather than guessed (§8 rule 4).
1. **The 2026-08-05 02:15 UTC run's actual outcome** (Q3). Predicted `ErrOffboxOrphaned` from source +
claim state; **not measured**.
2. **Whether the orphaned ciphertext still exists at the provider.** 40.9 MB (demo-hp) + 1.14 GB
(demo-felhom) are *presumed* present at `/home/felhom-repo` — reading the endpoint needs the SFTP
credential, which this session did not touch. The Hetzner API has no directory-listing surface.
3. **Whether demo-felhom's 2026-07-21 Re-issue re-sealed an unchanged password** (the predicted
false-staleness shape, Q4/Q5). Only two `host_escrow_superseded` rows exist in the whole database;
the generation before 2026-07-21 is not retained.
4. **What removed demo-hp's offsite target at the rebuild** is fully explained (fresh data dir), but
**why the guest was rebuilt on both boxes on consecutive mornings** is outside this spike.
---
## The operator's question — stated, and NOT answered here
> **The irreplaceable secret in the offsite tier is the restic repository password. It is generated on
> the box, it is destroyed by a guest rebuild, and the only copy that survives is sealed under the
> customer's recovery code — which nothing but a human can open. Every machine is going to be
> reinstalled.**
>
> **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host, where
> you already have root — or do you want it to stay sealed under the customer's recovery code and
> accept that every rebuild starts a new repository and orphans the old one, provided the system says
> so loudly on the day it happens?**
Option (c) is the first. Option (d) alone is the second. **This spike does not choose.**
+5 -2
View File
@@ -36,8 +36,11 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
| **R-190** | **A storage ACL that demonstrably WORKED in the morning was gone by mid-morning, and nothing recorded its removal.** On demo-felhom, a `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50 CEST 2026-08-03** (task log read in full). From **09:24:56** the same path returned `HTTP 403 … missing privilege Datastore.Allocate at /storage/felhom-backup`, six times through the day, until the grant was re-applied by hand at 18:54. By ~14:50 `pveum acl list` showed **no row at all** for that path | **MITIGATION SHIPPED 2026-08-04** (agent **v0.124.0 → v0.124.1**) — **MECHANISM STILL OPEN** | — | **Why this is not just R-185 restated:** R-185's mechanism (the installer's Scenario-F arm resolves a pre-existing target without granting) explains a box that NEVER had the grant. This box HAD it and lost it, inside five hours, with the machine up throughout. **Ruled out, each by measurement:** a host reinstall (`uptime` = 12 days); any `pveum`/ACL/`user.cfg` activity in syslog between 04:00 and 10:00 (none); any ACL entry in `/cluster/log` (none). **Correlated, not established:** `host_leaf_changed` at 09:15 and `controller_started` at 09:19 — guest 9201 was reprovisioned nine minutes before the first 403. PVE removes ACLs at `/vms/<vmid>` when a guest is destroyed (`AccessControl::remove_vm_access`, the F-LEAK mechanism); whether any path can take a `/storage/<id>` row with it has NOT been established and is the first thing to check. **Why it matters more than the grant did:** a permission that can vanish silently makes every ACL-based guarantee on these hosts provisional, and the agent's new store-grant probe (v0.123.0) now detects the STATE but says nothing about the TRANSITION. **Worth pairing with:** whether the probe should report a grant it once had and no longer has as a distinct, louder signal than one it never had **THE ROW NOW REFLECTS THE MITIGATION, NOT THE CAUSE — stated plainly because the two are different things.** The box is resilient; the loss is still unexplained. **Mitigation:** when the store-grant probe finds the grant absent, the agent runs the EXISTING root wrapper (`felhom-backup-target-apply grant <id>`) and **re-reads once** to confirm — the pbsdr R-22 self-grant shape, including its restraint. **No new privileged surface:** the sudoers vector `grant *` already covers any storage id (confirmed in `configs/felhom-agent.sudoers`, not assumed), and the verb already grants BOTH user and token. The verb existed, was permitted, and had only ever been called at storage CREATION — the *built but never wired* shape in a verb rather than a seam, this project's seventh instance. Bounded at one attempt per tier per hour (a storage can be unreadable for reasons an ACL cannot fix; re-granting every cycle is a repair loop wearing a fix's clothes). **THE RECORD IS THE HALF THIS ROW IS ABOUT, and v0.124.0 got it wrong in production while every unit test passed.** It reported degraded for *one cycle* — meaning the probe call that repaired. But `probeAll` is invoked INDEPENDENTLY by the self-check log and by the collector building a host-report: on the box the repairing call was the log's (`09:39:34`, journal shows the repair and `degraded=1`) and the report three seconds later found the grant present and sent **`ok`**. The agent's journal had the record, the hub had nothing, and the operator would have learned nothing — the exact silence this row exists for, re-created inside its own mitigation. **v0.124.1** replaces it with a latch on TIME (20 min > the 900 s report interval), so at least one report must carry it. **PROVEN LIVE, twice, on demo-felhom** (grant deleted by hand, both rows): agent logs `store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190) target=felhom-backup privilege=Datastore.AllocateSpace action="felhom-backup-target-apply grant felhom-backup" confirmed_by=re-read`; the ACL rows return; and on v0.124.1 the **host-report at 08:00:30Z carried `status=degraded`** with the explanation and the hub raised `agent_capability_degraded` and **e-mailed the operator** at 08:00:40. Nothing new was built to carry it — the hub's existing ok→degraded→ok edge is the channel, and the text rides `Feature` because that is the field the hub interpolates into the e-mail (`Reason` does not travel). **PART 3 — the single bounded pass at the mechanism, with the negatives named.** The lead §8.6 nominated is real as a CLASS and is documented in our own installer: *"`pveum user token remove` purges the token's ACL, so re-applying post-rotate is mandatory"*. **It does NOT fit this box.** A rotation purges ALL of the token's ACLs and mints a NEW secret; demo-felhom's token still authenticates with the same secret (`--selftest` OK), it retained its other three storage grants throughout, and only `felhom-backup` was refused. No installer run is evidenced (no 2026-08-03 install log; host uptime 12 days at thLine truncated
| **R-191** | **Every weekly offsite backup UPLOADS successfully and then FAILS the job on a prune the box is deliberately not allowed to do — on both demo boxes.** Measured on demo-felhom 2026-08-04 06:49–06:53: the upload completed (223 s, 629 MiB of 1.874 GiB, **67.2 % reused incrementally**), then `ERROR: prune 'ct/9201': proxmox-backup-client failed: Error: permission check failed - missing Datastore.Modify\|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `ERROR: Backup of VM 9201 failed - error pruning backups` → `TASK ERROR: job errors`. The hub raised `whole_guest_backup_failed` | **CLOSED — SHIPPED 2026-08-04** (installer **1.25.0**; both live boxes corrected) | — | **This is R-89's rule not reaching the config.** R-89 moved PBS pruning SERVER-SIDE — *"boxes set `keep_last: 0`, ep0 runs prune jobs; box tokens stay write-only, never widen the grant"*. The token behaves exactly as designed: it refuses. But **both** demo boxes still arm the offsite tier with `keep_last=2 prune_pbs_allowed=true` (`backup_targets: [{target_id: felhom-pbs, cadence_seconds: 604800, keep_last: 2}]`), so every run asks for a prune that must fail. **The data is SAFE and that is why this is not a P1:** the snapshot lands before the prune is attempted; what is wrong is the job's VERDICT and the weekly operator e-mail it produces. **But it is corrosive in the specific way this project keeps finding:** a backup that reports FAILED while succeeding trains the operator to discount `whole_guest_backup_failed`, which is the same alert that would carry a real one — and it is exactly the failure the R-100 corollary warns about, an alarm whose text is true and whose trigger is not the thing you would act on. **Fix is one config line per box** (`keep_last: 0` on the PBS tier) plus whatever writes it on a fresh install; **deliberately NOT applied in this session** — the session was a runbook with an explicit "change nothing, and if a change appears necessary, stop and report" rule, and a retention field on a live backup tier is not a change to slip into an observation run. **Check before fixing:** whether ep0's prune jobs actually cover these two namespaces, or the snapshots simply accumulate once the box stops asking **THE GATE WAS RUN FIRST, AND IT MATTERED.** Before disabling anything, ep0 was read (read-only, Tier 2): prune jobs `prune-demo-felhom` and `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, `schedule 03:30`, `keep-last 2`, comment *"R-82 retention keep-last=2, server-side (box tokens are write-only)"* — and they have run **every day since 2026-07-27: 18 tasks, all `status=OK`**. The newest task log reads `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` / `keep ct/9201/2026-07-27…` / `keep ct/9201/2026-07-28…` / `TASK OK`. Retention happens, and it happens there. **A METHODOLOGICAL WARNING WORTH MORE THAN THE FIX.** Three separate queries said the OPPOSITE — *no prune jobs have ever run* — and **all three were broken instruments**: `worker-type` where the field is `worker_type`; the value `prune` where the worker type is `prunejob`; and `journalctl -u proxmox-backup` where the unit is `proxmox-backup-proxy`. A fourth reading (3 snapshots under keep-last 2) was mis-framed by CC and self-corrected — the third snapshot had landed AFTER that day's 03:30 window. Acting on any of them would have disabled the only pruning ATTEMPT while reporting that nothing prunes: a weekly false alarm traded for unbounded growth on the protected endpoint, invisible for months. **The gate is what caught it, and only because it demanded evidence rather than a verdict.** **Shipped:** installer **1.25.0** writes `keep_last: 0` on the offsite tier (the agent's existing guard `allowPBSPrune = !primary && keep_last > 0` already reads that as *never prune from the box* — no agent change), the justifying paragraph is rewritten to say where retention lives and cite R-89, and `hostinstall_gates.py` asserts it (red-proved: pinning `keep_last: 2` back fails the gate). **Both live boxes corrected in their own config** — `backup tier armed target=felhom-pbs … keep_last=0 … prune_pbs_allowed=false` on demo-felhom and demo-hp, with the local tier untouched at `keep_last=3`. Served over HTTPS at `1.25.0` with `"keep_last":0` in the served bytes. **STILL TO OBSERVE:** the next weekly offsite run completing OK end-to-end. The change removes the failing step; the *schedule* proving it is next week's event, and this row should carry that line when it happens. | CC |
| **R-194** | **PVE's permission cache delays every grant-state verdict by an unknown amount, so "the agent can read it" and "the ACL exists" are not the same measurement.** Observed twice while validating R-190's self-repair on demo-felhom 2026-08-04: both ACL rows for `/storage/felhom-backup` were deleted, and `GET /access/permissions` continued to report `Datastore.AllocateSpace` present — for **~40 s** in one run and **~16 minutes** in another. During that window the capability probe reads healthy and the self-repair does not fire | **OPEN** | — | **Why it matters beyond the delay:** it puts a floor under how fast a lost grant can be noticed, it makes any single permission read a lagging indicator, and — the interesting part — **it is a candidate contributor to R-190's own timeline**: a grant removed at an unknown moment could keep working until a cache expiry, which is exactly the shape of *worked at 04:44, refused at 09:24*. That does not explain what removed it, but it may explain when the refusal SURFACED, and the two have been treated as the same instant. **Not a defect in our code** — it is PVE behaviour, and the mitigation already tolerates it (the repair fires on the next probe after the cache clears). **What is worth deciding:** whether the store-grant probe should ALSO consult the storage content listing as a second signal, since that appeared to reflect the loss immediately (`{"data":[]}` while the permission read still said present) — two signals disagreeing is itself information, and today only one of them is read | CC |
| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **OPEN** | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. | CC |
| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **OPEN** | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. **The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained. Record the answer here. | CC |
| **R-195** | ~~**A customer with no machine ever bound e-mailed an `expected_dbdump_missed` ERROR every morning.** `david` — a real prospective customer whose record was created 2026-08-01 16:51:49 with **hosts=0, host_deletions=0, host_reports=0, reports=0** — raised the alarm at 03:00 UTC on 08-02, 08-03 and 08-04~~ | **SHIPPED** (hub **v0.92.0**, 2026-08-04) | — | **The mechanism, and it is the interesting half: the skip that protects every other silent customer is keyed off having reported at least once.** `CheckBackupDeadlines`' down-skip reads `StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — a query **over the `reports` table**. A customer with zero reports appears in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. The skip therefore misses exactly the customer it would most obviously cover. (Corroborated live: `peti-felhom` is active with a host deleted 2026-07-15 and **does not** alarm — it has 482 old reports, so it is `down` and skipped.) The backup half was already safe (`reportJSON == ""` → skip); the DB-dump half had no guard at all. **Fix:** `store.HasEverBoundHost(customerID)` = a live `hosts` row **OR** a `host_deletions` tombstone, consulted once per customer at the top of the deadline loop. **The discriminator is deliberately "was a host EVER bound", NOT "has a report arrived"** — a box that was installed and went silent is a real fault and must keep alarming; that is the case the change could break and it has its own test. Fail-**open** on a read error (an unreadable binding must never SUPPRESS a real alarm), and the deferral is LOGGED with its own counter (the v0.73.0 Part-7 precedent: a quiet check must not look like a check that did not run). The anchored-verdict structure is untouched. **Red-proof observed:** deleting the guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]` — verbatim the event `david` sent three mornings running. `david`'s record was NOT modified; the record was correct and the alarm was what was wrong | — |
| **R-196** | **`escrow_stale` is wired to the ONE path that does not change the repo password, and absent from the path that does.** `ReissueCredentials` (`hub/internal/offsite/offsite.go:150-228`) resets **only** the Hetzner sub-account/box password and stages a fresh one-time secret — it contains no reference to a restic password and *cannot*, since that password is generated on the box and never leaves it except into the R-wrapped escrow. Yet it calls `MarkEscrowStale` on the stated grounds that *"the restic repo password just changed"* (`offsite.go:198-201`), and the same false premise is repeated at `api/handler.go:1067-1069` and in R-39's record | **OPEN** | — | **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting a guarantee the code does not provide** — and the first where the comment factually describes a *different function*. It survived because its EFFECT (a stale escrow) is real, so nobody checked its CAUSE. **The live defect, not a documentation nit:** on the ordinary Re-issue shape (a consumed-but-failed install on a box that still holds its `repo_password` file) the box re-applies, `WriteOffboxSecrets` finds the file present and **keeps it**, the repo password is unchanged — and the hub has told the customer in Hungarian that their recovery escrow is stale and asked them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** **The inverse is the worse half and is R-193's:** demo-felhom's repo password *did* change on 2026-08-03 with **no Re-issue anywhere in its history** (its only `escrow_stale`/`offsite_reissued` pair is dated 2026-07-21 08:29:29) and therefore **nothing marked its escrow stale for 13 h**. **Fix shape:** mark the escrow stale on the evidence that it IS stale — a changed `restic_pw_sha256` (→ R-197) — not on a Re-issue; and correct all three comments in the same commit. **Not established, so not asserted:** whether the 2026-07-21 Re-issue on demo-felhom re-sealed an unchanged password (the predicted false-staleness shape) — `host_escrow_superseded` holds only two rows in the whole DB, both from 2026-08-04, so the prior generation is not retained. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q4 | CC |
| **R-197** | **The hub holds both halves of the evidence that a box's offsite DATA key changed, and reads neither.** `restic_pw_sha256` is stored on `host_escrow` and carried to `host_escrow_superseded` on every re-escrow. Comparing the two is what let this spike answer its hardest question in one query — and **nothing in the hub does it** | **OPEN** | — | **Why this is the cheapest real fix on the table.** A changed repo password means the previous offsite repository is now unopenable by the box, i.e. the customer's off-site history is orphaned. That is the single most consequential state change in the backup system, it is **already fully observable from data the hub owns**, and today it produces **no event, no e-mail, no card and no log line** — demo-felhom's went unremarked for 13 hours and would have gone unremarked indefinitely had this spike not run. **Two-line verdict:** on `SaveHostEscrow`, if the incoming `restic_pw_sha256` differs from the row being superseded, emit a distinct operator event naming the orphaned generation. **Deliberately an EVENT, not a heal** — nothing should act on this automatically until R-193's (c)-vs-accept decision is taken; the point is that the operator learns on the day. **Pair with R-196**, which is the same signal aimed at the right trigger. **Generalises past this row:** *a comparison the system could be making from data it already stores, and is not, is a silence with no cost of entry* — cf. R-190's store-grant probe, where the state was read and the TRANSITION was not. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q8 option (d) | CC |
| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **OPEN** | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the same spike:** under R-193's Q2 finding a successful auto-restage would have restoLine truncated
| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **OPEN** | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the one a rebuild destroys, and **nothing automaLine truncated
| — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |
| — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator |
| **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — |