docs: R-193 spike report, operator STATUS, and the ROADMAP R-1 scope correction (F3 covers HOST re-enroll, not a guest rebuild)

2026-08-04 11:12:08 +02:00
parent f581ac1349
commit b7c5cfc061
3 changed files with 342 additions and 236 deletions
@@ -1,263 +1,325 @@
-# REPORT — R-191 (a backup that worked must not report failure) + R-190 (a permission must repair itself)
+# REPORT — R-193/R-192 spike (no code) + R-195 phantom-customer alarm (hub v0.92.0)
-**Date:** 2026-08-04 · **Repos:** `felhom-agent` **v0.123.0 → v0.124.0 → v0.124.1** (`257c4d8`, and
+**Date:** 2026-08-04 · **Class:** spike (Parts 0–3) + one fenced implementation (Part 4)
-the v0.124.1 fix) · `felhom.eu` installer **1.24.0 → 1.25.0** (tag `installer-v1.25.0`) ·
+**Boundary held:** the spike shipped **no production code for R-193 or R-192**, and Part 4 never
-**no hub change and no hub bump**.
+touched the spike's subject matter.
 > Supersedes the earlier 2026-08-04 report (R-191 + R-190), per the overwrite convention. Nothing
 > durable from it lived only here — it is recorded in `hub/CHANGELOG.md`, `CONTEXT.md` S-24/S-25 and
 > `OPEN-ITEMS.md` R-190/R-191.
 ---
-## 1. Baselines
+## 1. Confirmed baselines, re-read at the cited commit
-| Repo | cited in §1 | found on arrival | matched? |
+| Repo | `main` @ commit on arrival | Version | End state |
 |---|---|---|---|
-| `felhom-agent` | `72161f6cf010` / `v0.123.0` | same | **yes** |
+| `felhom.eu` | `f456835bbcc4` ✔ matches the spec | hub `v0.91.1` (deployed `felhom-hub:0.91.1`) | **`v0.92.0`** built, pushed, manifest bumped, synced, **live** |
-| `felhom.eu` | `d692c46db08f` / installer `1.24.0` | **`33f0ab2f33f7`** / installer `1.24.0` | **version yes, commit NO** |
+| `felhom-controller` | `0887fd676d60` | — | **untouched** (read-only) |
 | `felhom-agent` | `856a127` | — | **untouched** (read-only) |
-`felhom.eu` had moved four commits forward since the brief was written — this morning's alert
+Clean-tree gate passed before the build: `git status --porcelain` empty, `HEAD == origin/main`.
-diagnosis (R-190/191/192/193 rows, the demo-hp offsite re-issue), all pushed. Recorded rather than
+Highest register ID on arrival was **R-194**; grepped before minting.
 glossed. Both `--ref=installer-v1.24.0` lines confirmed at 327 and 372.
-## 2. Part 0 — the gate, and the three broken instruments that nearly failed it
+---
-**VERDICT: PASS.** ep0 prunes **both** namespaces on schedule.
+## 2. Q1–Q8 — each answered or explicitly marked unmeasured
 Full method, quotes and query output: **`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md`**.
 ### Q2 and Q3 first — they decide whether the rest matters
 **Q2 — does a rebuilt controller reuse the escrowed repository password, or mint a new one?**
 **RULING: it MINTS A NEW ONE.**
 *Method — source:* `ConfigureOffbox` → `ApplyOffsiteTarget` → `WriteOffboxSecrets`, which generates a
 fresh 256-bit password whenever `<DataDir>/offbox/repo_password` is absent
 (`felhom-controller/controller/internal/backup/offbox.go:392-401`). A rebuilt guest has a fresh data
 dir. The only recovery path, `InjectOffboxPassword`, has **exactly one caller in the entire repo**
 (census: `grep -rn "InjectOffboxPassword" --include=*.go .` → 3 hits, 1 definition, 1 caller, 1
 comment) — `web/offbox_handlers.go:189`, a web form a human pastes into. **The apply-bridge never
 calls it.**
 *Method — live, without touching a box:* `host_escrow.restic_pw_sha256` vs
 `host_escrow_superseded.restic_pw_sha256`. **demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom
 `48741892…` → `c60c8bc7…`. Both CHANGED.**
 **This is bigger than R-193 as filed:** demo-felhom, which R-193 records as having survived by luck,
 had **36 snapshots / 1,136,685,919 B** before its rebuild and has reported `snapshot_count: 0` in all
 109 reports since. Its luck restored **delivery**, not the **repository**.
 **Q3 — what did the 04:15 run report?** **UNMEASURED, and the binary in the question is wrong.**
 The run is daily at ~02:15 UTC (measured `last_run` values on both boxes). Neither box could run on
 2026-08-04: demo-hp had no target at all until 07:15:47 UTC; demo-felhom's target was
 `escrow_state: pending` from 08-03 07:19:10 to 08-04 07:20:13, which `OffboxRunnable` blocks
 (configured **AND** escrowed). **The decisive run is 2026-08-05 ~02:15 UTC.** Source predicts a
 **third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`;
 `repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies
 **orphaned** → `ErrOffboxOrphaned` → **the run refuses and shows the orphan card.** Recorded as a
 prediction, not a measurement.
 ### The rest
 - **Q1 — what is one-shot.** Full table in the spike. Headline: the hub's one-time provider password
  is one-shot **but freely re-issuable**; the box's SFTP key is **regenerated on every apply by
  design**; the **restic repository password** is the one nothing can restage — the agent's own source
  calls it *"irreplaceable (unlike the SFTP access key…)"*
  (`felhom-agent/internal/escrow/identity.go:35-39`). **R-193 was filed against the recoverable
  secret; the damage is in the irreplaceable one.**
 - **Q4 — see §3 below.**
 - **Q5 — why one box recovered and the other not. CONFIRMED FROM THE LEDGER.** demo-felhom's
  `one_time_secrets` row was created `2026-07-21 08:29:29` (traceable to a Re-issue at that exact
  timestamp) and sat **UNCONSUMED for 13 days** until the rebuild consumed it at
  `2026-08-03 07:17:58`. Without one: exactly demo-hp — `consume` → **404** and a retry loop, 25 hours
  unprotected. **Correction to the contrast:** the recovery was of the plumbing only (Q2).
 - **Q6 — why delivery is one-shot.** The design's own reasons quoted from three sites. **The security
  half, stated:** the consume endpoint is gated by `checkAuthCustomer` — **the customer's API key, the
  same credential as the config pull** — so an automatic restage grants **no new access to an
  unauthenticated party**. What it genuinely widens is the **exposure window**: a live provider
  password becomes fetchable whenever the box reports no offsite target, rather than only in the
  seconds after an operator acts. Blast radius stays the customer's own label-scoped sub-account. The
  R-39(a) unconsumed-secret guard must survive any change.
 - **Q7 — R-192's guard. BOTH HALVES CONFIRMED WITH NUMBERS.** See §4.
 - **Q8 — what each candidate costs.** See §5.
 ---
 ## 3. Which of the two contradictory claims about Re-issue is true
 **The claim that Re-issue "only resets the transport credential" is TRUE about what
 `ReissueCredentials` does. R-39's "rotates the restic password" is FALSE.**
 From source (`hub/internal/offsite/offsite.go:150-228`), the function does four things: `genPassword`,
 `ResetSubaccountPassword`/`ResetBoxPassword`, `SaveOneTimeSecret`, and bookkeeping. **There is no
 reference to a restic password anywhere in it, and none is possible** — that password is generated on
 the box and never leaves it except into the R-wrapped escrow, which the hub cannot open.
 **But the bookkeeping is justified by the false premise**, `offsite.go:198-201`: *"the restic repo
 password just changed, so any existing key-escrow blob … is now STALE"*. The same wrong claim appears
 at `api/handler.go:1067-1069` and in R-193's own row. **Eighth entry in `CLAUDE.md`'s table of
 comments asserting a guarantee the code does not provide** — the first where the comment factually
 describes a *different function*. → **R-196**.
 **What corroborated it, and the corroboration is the strongest part.** The task asked: if the repo
 password was untouched, what explains demo-hp's escrow going `pending` after the Re-issue? **Fully
 explained, and not by the Re-issue:** `ApplyOffsiteTarget` carries `EscrowState` over from the
 *existing* target, the rebuilt guest had none, so the `pending` fallthrough fired — and the box had
 just minted a new repo password, so `pending` was *correct for a different reason*.
 **The proof they are independent is demo-felhom:** it received **no Re-issue at all** (its only
 `escrow_stale`/`offsite_reissued` pair is dated `2026-07-21 08:29:29`), its repo password changed
 anyway, and **nothing marked its escrow stale for 13 hours.** The mechanism is exactly inverted:
 `escrow_stale` is wired to the one path that does **not** change the repo password and absent from the
 path that does.
 ---
 ## 4. Every query used, with its fields and values
 The DB snapshot was taken from the live pod and its **freshness proven with a positive observable**,
 not with "the query returned no error":
 | # | Query / read | Result |
 |---|---|---|
 | 1 | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale (mtime 08:34 vs wal 10:45 CEST). Copying only the main file is the R-3 shape. |
 | 2 | `PRAGMA integrity_check` | `ok` |
 | 3 | `MAX(host_reports.received_at)` vs `datetime('now')` | `2026-08-04 08:45:38` vs `08:49:46` — **newest row 4 min old** |
 | 4 | `SELECT customer_id, status, created_at FROM customer_configs` | `demo-felhom` active · `peti-felhom` active · `demo-hp` active · `drill-r50` **blocked** · `david` active (created `2026-08-01 16:51:49`). **`sess-f` is gone — confirmed, not assumed.** `drill-r50` untouched. |
 | 5 | `hosts` / `host_deletions` / `host_reports` / `reports` counts for `david` | **0 / 0 / 0 / 0** — never bound |
 | 6 | `events WHERE event_type='expected_dbdump_missed' AND created_at > '2026-07-25'` | **only `david`**, 03:00 UTC on 08-02, 08-03, 08-04 |
 | 7 | `SELECT customer_id, MAX(received_at), COUNT(*) FROM reports GROUP BY customer_id` | `demo-hp` 1329 · `demo-felhom` 6690 · `drill-r50` 183 (last 07-29) · `peti-felhom` 482 (last **2026-07-15**) — the reason peti-felhom is `down` and does not alarm |
 | 8 | distinct `$.offsite` objects per box with first/last seen | demo-hp: `15 snapshots / 40 927 044 B` until `08-03 05:59:21`, **ABSENT** `06:12:19`→`08-04 07:12:01` (111 reports), then `pending` 07:15:47, `escrowed` 07:16:02, `snapshot_count 0`. demo-felhom: `36 snapshots / 1 136 685 919 B` until `08-03 07:01:28`, ABSENT ×1, then `pending` `07:19:10`→`08-04 07:20:13`, `escrowed` 07:20:28, `snapshot_count 0` |
 | 9 | `host_escrow` + `host_escrow_superseded` `restic_pw_sha256` | **the decisive measurement** — both changed (§2) |
 | 10 | `one_time_secrets` (no values read) | demo-felhom created `2026-07-21 08:29:29`, consumed `2026-08-03 07:17:58`; demo-hp created+consumed `2026-08-04 07:11:51`/`07:12:06` |
 | 11 | `$.claimed` on the latest report of each box | **`1` on both** — which selects `ensureOffboxRepo`'s refuse branch |
 | 12 | `CountReportsOffsiteSince` reproduced verbatim (`ORDER BY id LIMIT 500`) for demo-hp @ `2026-07-23 09:53:41` | **`total=500, withOffsite=500`**, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** |
 | 13 | same window, **no LIMIT** | `total=1174, withOffsite=1063` (⇒ 111 without — matches #8) |
 | 14 | full text of the live `offsite_delivery_stuck` message | *"…and **500 report(s) since carry no offbox target**…"* while `OffsiteReportsSinceConsume` was **500** |
 | 15 | `COUNT(*)` of `offsite_credential_restaged` | **0, for every customer, ever** |
 | 16 | the shipped predicate run verbatim against live data (post-deploy) | `david` → **0**; `demo-felhom`/`demo-hp`/`drill-r50`/`peti-felhom` → **1** |
 **Instrument limits recorded rather than discovered later:** `one_time_secrets` has `customer_id` as
 PRIMARY KEY and is last-write-wins **by design**, so it holds only the newest credential per customer —
 demo-hp's `2026-07-23` consume, on which R-192's whole diagnosis rests, was **overwritten** by
 yesterday's Re-issue and is no longer readable from that table. The `2026-07-23 09:53:41` anchor in
 query #12 comes from R-192's record, and that is stated in the spike rather than presented as a live
 read.
 ---
 ## 5. Ranked options, with a recommendation
 | rank | option | verdict |
 |---|---|---|
 | **1** | **(d) the honesty pass — now** | Cheap, weakens nothing, and **the only option that helps the incident already in flight**. Two parts: correct R-192's message + log its refusal (**R-196**), and **detect a changed `restic_pw_sha256`** — both hashes are already in the hub's DB and **nothing reads them** (**R-197**). |
 | **2** | **(c) agent-retained repo password — NOT PREVIOUSLY NAMED** | The only candidate aimed at the irreplaceable secret. The agent lives on the Proxmox host, **survives a guest rebuild**, already receives the password over the pinned local API and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Every seam exists. Price: a copy of the data key at rest on the host. **The operator's trade to make.** |
 | **3** | **(a) hub restages automatically** | **Already exists** as `reissueOnReenroll`'s F3 leg, wired to the wrong event (behind `handleHostEnroll`'s mint-once-reuse short-circuit; a guest rebuild leaves the `hosts` row intact). Worth doing **after** (c), never instead: on its own it restores the transport to a box whose data key is already gone, and **would have made both boxes look healthy on 2026-08-04 while their snapshots were orphaned** — strictly worse than the current loud failure. |
 | **4** | **(b) recover from escrow at re-bootstrap** | **Not implementable as stated.** The escrow is R-wrapped and zero-knowledge; the hub has no recovery code. "Recoverable at re-bootstrap" describes a customer-present ceremony — i.e. the manual form that already exists. |
 **Recommendation: ship (d) now; then decide (c). Do not ship (a) first.**
 ### The operator's question — stated, and NOT answered
 > The irreplaceable secret is the restic repository password. It is generated on the box, destroyed by
 > a guest rebuild, and the only surviving copy is sealed under the customer's recovery code. Every
 > machine will be reinstalled.
 > **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host,
 > where you already have root — or do you want it to stay sealed and accept that every rebuild starts
 > a new repository, provided the system says so loudly on the day it happens?**
 **STOPPED here. Neither candidate was begun.**
 ---
 ## 6. Part 4 — the phantom-customer alarm (SHIPPED)
 **Files changed**
 - `hub/internal/store/store.go` — **new** `(*Store).HasEverBoundHost(customerID) (bool, error)`:
  a live `hosts` row **OR** a `host_deletions` tombstone.
 - `hub/internal/monitor/deadline.go` — one guard at the top of `CheckBackupDeadlines`' customer loop,
  plus an `unbound` counter in the summary line.
 - `hub/internal/monitor/deadline_unbound_test.go` — **new**, 5 tests.
 **Commit:** `7fff45d688fa` (code + tests + spike + docs) · `f581ac134964` (manifest bump).
 **The mechanism, established at source and corroborated live.** The existing down-skip could never
 have covered this: it reads `StalenessChecker.GetState()`, whose map is seeded from
 `store.GetCustomers()` — **a query over the `reports` table** (`store.go:1055-1067`). A customer with
 zero reports is in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. **The
 skip misses exactly the customer it would most obviously cover.** Corroborated by query #7 and by the
 live post-deploy startup log — `Staleness checker initialized: 2 ok, 0 stale, 1 down` — where the
 `1 down` is `peti-felhom` (host deleted 2026-07-15, 482 old reports) and `david` is **absent from the
 census entirely**.
 **The discriminator is "was a host EVER bound", not "has a report arrived"**, which is the point:
 a box that was installed, bound and went silent has a `hosts` row and keeps alarming. **Fail-open on a
 read error** — an unreadable binding must never suppress a real alarm. The deferral is **logged with
 its own counter** (the v0.73.0 Part-7 precedent). The anchored-verdict structure is untouched, and
 `david`'s record was not modified.
 **Sibling verdicts checked rather than assumed.** The census of every customer-iterating checker:
 `deadline.go` backup half (already guarded by `reportJSON == ""`), `restoretest.go` (guarded by
 `latest == ""`), `staleness.go` + `offsite.go` (iterate `GetCustomers()`, report-derived — a hostless
 customer never appears), `offsite_delivery.go` (a hostless customer derives `DeliveryNoSecret` and is
 skipped). **The DB-dump half was the only verdict that could fire**; the guard covers both verdicts in
 that function uniformly.
 **Test results — the suite run and the exit code read BEFORE any commit, never in one command.**
 ```
-prune-demo-felhom   store=felhom-offsite  ns=demo-felhom  schedule=03:30  keep-last 2
+go build ./...  → BUILD OK
-prune-demo-hp       store=felhom-offsite  ns=demo-hp      schedule=03:30  keep-last 2
+go vet ./...    → VET OK
 go test ./...   → rc=0, 17 packages ok, no FAIL lines
 ```
-**18 prunejob tasks, all `status=OK`, every day from 2026-07-27 to 2026-08-04**, two per day, one per
+The 13 `TestCheckBackupDeadlines*`/`TestHasEverBoundHost` tests were run with `-v` and **the `=== RUN`
-namespace. The newest task log, read in full:
+lines were checked**, per `CLAUDE.md`'s rule that a `-run` pattern matching nothing prints `ok` and
 exits 0.
 **THE RED-PROOF THAT MATTERS.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms` is the
 case the change could break — a real box that was installed and never phoned home, indistinguishable
 from `david` on every signal *except* the one the guard reads. With the guard removed it **still
 passes**, which is the correct result and proves the test is aimed at the guard's *risk*, not its
 presence. The companion red-proof of the fix itself, **observed**:
 ```
-prune job 'prune-demo-felhom' / task triggered by schedule '03:30'
+=== RUN   TestCheckBackupDeadlines_NeverBoundHost_Silent
-retention options: --ns demo-felhom --max-depth 0 --keep-last 2
+    deadline_unbound_test.go:58: a customer with NO host ever bound must raise NOTHING;
-Pruning group demo-felhom:"ct/9201"
+                                 got [expected_dbdump_missed]
-keep ct/9201/2026-07-27T19:55:41Z
+--- FAIL: TestCheckBackupDeadlines_NeverBoundHost_Silent
 keep ct/9201/2026-07-28T04:49:43Z
 TASK OK
 ```
-Read-only throughout: no job created, edited or run; nothing deleted. ep0 is Tier 2.
+— verbatim the event `david` sent three mornings running. `deadline.go` restored from a backup taken
 before the mutation; full suite re-run green afterwards.
-### The part that matters more than the verdict
+**Deployed version.** `felhom-hub:0.92.0` built + pushed (`sha256:ca71140cf54a…`),
 `manifests/hub.yaml` bumped `0.91.1 → 0.92.0` in git, ArgoCD hard-refresh + **deliberate sync** (never
 `kubectl set image`). Verified: app **Synced / Healthy**, `rollout status` → `deployment "hub"
 successfully rolled out` (rc=0), running image `gitea.dooplex.hu/admin/felhom-hub:0.92.0`, pod
 `hub-6775889d95-gc8sk` 1/1 Running, clean startup log.
-**Three of my own queries said the endpoint prunes NOTHING, and all three were broken instruments:**
+**Is the 05:00 alarm confirmed silent? NO — CONFIRMATION IS PENDING, and it is stated rather than
 assumed.** The hub's own log says `deadline-check: next run at 2026-08-05 05:00 CEST (in 17h53m57s)`;
 there is no manual trigger (`CheckBackupDeadlines` has exactly one caller, the scheduler at
 `cmd/hub/main.go:598`). An absent alarm tomorrow would in any case be weak evidence, so a **positive**
 observable was taken instead: the shipped predicate was run verbatim against a fresh copy of the live
 DB (query #16) — `david` → **0** (will be skipped), every other customer → **1** (behaviour
 unchanged). **The 05:00 firing itself is still owed.**
-| query | why it was wrong |
+---
 |---|---|
 | `'prune' in t['worker-type']` → *0 prune tasks* | the field is **`worker_type`**; `worker-type` does not exist, so every task read `None` |
 | `worker_type == 'prune'` → *0* | the worker type is **`prunejob`**, not `prune` |
 | `journalctl -u proxmox-backup \| grep -c prune` → *0* | the unit is **`proxmox-backup-proxy`** |
-A fourth reading was mine to mis-frame: *"3 snapshots under keep-last 2 proves nothing is pruning"* —
+## 7. `OPEN-ITEMS.md` rows, and the capability map
 the third snapshot landed at 04:49, **after** that day's 03:30 window, so it proved nothing. I caught
 that one myself; the other three were caught only by checking the field names when the answer looked
 too categorical.
-**Had I acted on them, I would have disabled the only pruning ATTEMPT while filing a finding that
+- **R-192** — updated with Q7's confirmed numbers; **NOT closed**, and now carries a *new* reason not
-nothing prunes anywhere** — a weekly false alarm traded for unbounded growth on the protected
+  to fix it in isolation (a successful auto-restage would have protected the plumbing while the data
-endpoint, invisible for months. The gate did its job precisely because it demanded evidence rather
+  was already gone).
-than a verdict.
+- **R-193** — updated with the whole spike; **NOT closed**, state moved to **WAITING-ON-OPERATOR**,
  owner `CC + operator`. The old "15+ vs 1 decides it" line is struck through and replaced with the
  measured answer plus the third predicted outcome.
 - **R-195** *(new)* — the phantom-customer alarm. **SHIPPED**, hub v0.92.0.
 - **R-196** *(new)* — `escrow_stale` wired to the path that does not change the repo password, absent
  from the one that does. **OPEN.**
 - **R-197** *(new)* — the hub holds both `restic_pw_sha256` values and reads neither. **OPEN.**
-## 3. R-191 — the off-site tier stops asking to prune
+**The capability map DID need touching, and the §N.5 escalation did NOT fire — the row was not
 overclaiming.** `00-capability-map.md`'s offsite row already recorded reinstall-continuity: *"a
 recreated data volume that orphaned the repo … is now CLASSIFIED … + a move-aside (never-delete)
 reset"*, shipped controller v0.142.0 and live-fired 2026-07-18. **That independently corroborates the
 Q3 prediction from a different direction.** What it did not record — and now does — is that this fires
 on an **ordinary, planned, unattended guest rebuild** on every box, with the measured 2026-08-03/04
 instance, the hash evidence, and the inverted `escrow_stale` wiring. **Status unchanged
 (PROVEN-LIVE):** the guard is correct and is not the gap; the cadence is.
-**The defect:** `keep_last: 2` on the offsite tier, so vzdump uploaded and then failed the whole job
+`ROADMAP.md`'s newest row is R-171 and none of R-190…R-197 has one, so **there was nothing to
-on `missing Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` →
+collapse.** One correction *was* warranted under the coupling rule: R-1's row claimed the v0.57.0
-`whole_guest_backup_failed`. R-89 moved offsite pruning server-side; the 2026-07-26 "two weeks" ruling
+reinstall arc makes offsite re-issue *"auto-fire on re-enrollment"* — scope-corrected in place, since
-was not reversed — **where it is enforced moved and this value did not follow.**
+that means **host** re-enrollment after a host **delete**, and a guest rebuild never reaches it.
- **Installer 1.25.0**: `keep_last: 0`, and the justifying paragraph rewritten to say where retention
+`CONTEXT.md` gained **S-26** (the spike's Q1/Q2 headlines + the three corrections that must not be
-  now lives, citing R-89 and the Part 0 evidence, plus the warning that if ep0 ever stops pruning this
+re-inherited) and **S-27** (the R-195 ruling and its mechanism). `REUSE.md` gained the
-  zero is unbounded growth. The agent's guard `allowPBSPrune = !primary && keep_last > 0` already
+`HasEverBoundHost` row with its "never a substitute for a liveness check" trap. `STATUS.md` rewritten
-  reads 0 as *never prune* — **no agent change**.
+for the operator, one screen.
 - **A gate asserts it** (`hostinstall_gates.py`), red-proved: pinning `keep_last: 2` back →
  `FAIL: the offsite backup_target arms a CLIENT-SIDE prune (keep_last=2)`.
 - **Both live boxes**, quoted:
-```
+---
 demo-felhom  backup tier armed target=felhom-backup … keep_last=3 … prune_pbs_allowed=false primary=true
 demo-felhom  backup tier armed target=felhom-pbs    … keep_last=0 … prune_pbs_allowed=false primary=false
 felhom-host  backup tier armed target=felhom-backup … keep_last=3 … prune_pbs_allowed=false primary=true
 felhom-host  backup tier armed target=felhom-pbs    … keep_last=0 … prune_pbs_allowed=false primary=false
 ```
-The local tier is untouched (Scenario C).
+## 8. CI, task ids, and `--no-verify`
-**Not yet observed:** a weekly offsite run completing OK end to end. The failing step is removed and
+| commit | what | CI task id | run # | conclusion |
-the boxes are armed correctly; the *schedule* proving it is next week's event. The row says so rather
+|---|---|---|---|---|
-than implying the run has happened.
+| `7fff45d688fa` | R-195 code + tests + the spike + all docs | **126** | 72 | **success** |
 | `f581ac134964` | `manifests/hub.yaml` 0.91.1 → 0.92.0 | **127** | 73 | **success** |
-## 4. R-190 — the grant repairs itself, and the repair is visible
+Matched by `head_sha` via
 `curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/felhom.eu/actions/tasks?limit=5"` — a PULL
 check, not a reliance on the failure mail.
 **`--no-verify` was NOT used.** The pre-push hook ran and reported `gates OK - push proceeding`;
 `python3 scripts/repo_gates.py` was additionally run in full (all five gates OK) before committing.
-**Mitigation.** On a missing grant the agent runs the existing root wrapper
+**Teardown:** the spike provisioned nothing — no guest, no VM, no storage, no credential. The only
-`felhom-backup-target-apply grant <id>` and **re-reads once** — the pbsdr R-22 shape and its
+artefact is a hub-DB snapshot in the session scratchpad, outside the repo.
 restraint. **No new privileged surface:** `grant *` was already in the sudoers allowlist for any
 storage id (confirmed in `configs/felhom-agent.sudoers`, not assumed), and the verb already grants
 both user and token. It existed, was permitted, and had only ever been called at storage **creation**
 — the *built but never wired* shape in a verb rather than a seam. Bounded: one attempt per tier per
 hour.
-### The record — and how v0.124.0 got it wrong in production with a green test suite
+**Read-only discipline held:** no `ReissueCredentials`, no ceremony, no credential rotation, no
 change to any box's offsite configuration, no snapshot deleted, no customer record modified.
 `drill-r50` untouched; `sess-f`'s removal **confirmed** rather than assumed.
-§8.5 asked whether the hub's existing degraded↔ok edge sufficed **before building anything new**. It
+---
 does, as a *channel* — but only if the agent deliberately reports one degraded cycle, because a silent
 repair produces no edge at all. So **nothing new was built**: no wire change, no hub change, no new
 event type. The explanation rides `Feature`, because that is the field the hub interpolates into the
 operator's e-mail (`monitor/host_capability.go` builds its message from names and features; `Reason`
 does not travel).
-**v0.124.0 implemented "one cycle" as one probe CALL, and that failed live:**
+## 9. Observations — noticed and NOT acted on
-```
+1. **The orphaned ciphertext is retained and keeps consuming quota, with nothing to prune it.**
-09:39:34  agent: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED … confirmed_by=re-read
+   `resetOrphanedRepo` is move-aside-never-delete by design (correct), but 40.9 MB + 1.14 GB of
-09:39:34  agent: capabilities self-check ok=69 total=70 degraded=1
+   unopenable data will sit against a 50 GB soft quota indefinitely. S-24's ep0 prune jobs cover the
-09:39:37  hub:   host-report stored → pve:store-grant:felhom-backup = ok      ← the record never arrived
+   **PBS** namespaces, a different tier. Not filed — it belongs to whichever option is chosen for
-```
+   R-193.
-
+2. **`host_escrow_superseded` holds only two rows in the entire database**, both from 2026-08-04. That
-`probeAll` is invoked **independently** by the self-check log and by the collector building a report.
+   is why the spike could not establish whether demo-felhom's 2026-07-21 Re-issue re-sealed an
-The repairing call was the log's; the report three seconds later found the grant present and sent
+   *unchanged* password — the predicted false-staleness shape of R-196. Retention policy for that
-`ok`. The agent's journal had the record, the hub had nothing, and the operator would have learned
+   table was not investigated.
-nothing — the exact silence R-190 exists for, re-created inside its own mitigation, **while every unit
+3. **Both demo guests were rebuilt on consecutive mornings** (demo-hp 08-03 06:12, demo-felhom 08-03
-test passed**. **v0.124.1** replaces it with a latch on **time** (20 min > the 900 s report interval).
+   07:17, both to controller 0.192.0 with new config hashes). *Why* is outside this spike's scope and
-
+   was not chased.
-### Live proof, twice, on demo-felhom
+4. **`api/handler.go:1067-1069`'s comment is the third copy of R-196's false premise**, and it sits in
-
+   a different package from the other two. Whoever fixes R-196 should grep rather than fix the one
-Grant deleted by hand (both rows — user and token), which reproduces R-190 and the R-185 symptom
+   they were shown.
-(`{"data":[]}` through the token):
+5. **`peti-felhom` is `active` with its host deleted since 2026-07-15**, judged only by the `down`
-
+   skip. It is correct today and would start alarming the moment it reported once. Not a defect, but
-```
+   the shape is worth knowing.
-10:00:29  agent: store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)
+6. **The `-shm`/`-wal` trap generalises.** Any future session reading the hub DB from the pod must
-          target=felhom-backup privilege=Datastore.AllocateSpace
+   copy all three files. Reading `hub.db` alone today would have produced confident answers from a
-          action="felhom-backup-target-apply grant felhom-backup" confirmed_by=re-read
+   file 2 h 15 m stale — the R-3 shape with a different filename.
 10:00:30  hub:   host-report → pve:store-grant:felhom-backup = DEGRADED
          feature: "…the agent's storage grant was MISSING and has been AUTOMATICALLY RESTORED…"
 10:00:40  hub:   Host capability: demo-felhom-8363b5 ok → degraded (agent_capability_degraded)
 10:00:40  hub:   Operator email sent for demo-felhom/agent_capability_degraded
          pveum acl list | grep -c felhom-backup → 2   (both rows restored)
 ```
 **And the pair CLOSED**, which is the other half of "exactly one alert per loss":
 ```
 10:20:29  the 20-minute latch expires
 10:30:40  hub: Host capability: demo-felhom-8363b5 degraded → ok (agent_capability_recovered)
 ```
 So one lost grant produced **one degraded event + one operator e-mail + one recovery**, and nothing
 further. Final state on both boxes: agent `0.124.1`, two ACL rows on `/storage/felhom-backup`.
 Scenario E is unchanged from v0.123.0 — a failed repair, or one that does not survive the re-read,
 leaves the capability degraded with the failure in its reason.
 ## 5. Part 3 — one bounded pass at the mechanism, negatives named
 **The lead is real as a CLASS and is documented in our own installer:** *"`pveum user token remove`
 purges the token's ACL, so re-applying post-rotate is mandatory"* — a rotation would take every grant
 and re-apply only the hardcoded `PVE_STORAGES` set, which is exactly the observed end state.
 **It does not fit this box.** A rotation purges **all** the token's ACLs and mints a **new secret**;
 demo-felhom's token still authenticates with the same secret (`--selftest` OK), it kept its other
 three storage grants throughout, and only `felhom-backup` was refused. No installer run is evidenced
 (no 2026-08-03 install log; host uptime 12 days at the time).
 **Ruled out, by name:** a host reinstall; any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00;
 any cluster-log ACL entry; a token rotation **on this box**. **Not ruled out fleet-wide** — any
 installer run still purges and re-grants only `PVE_STORAGES`, though installer 1.24.0's reuse-arm fix
 now re-grants the backup target on that path.
 **The mechanism remains open**, and the row says which half it reflects: mitigation shipped, cause
 unexplained. No reproduction was attempted and the session was not extended for it.
 ## 6. Tests and red-proofs
 Green gate both repos: `go build ./... && go vet ./... && go test ./...` rc=0; `agent_gates.py` and
 `repo_gates.py` all OK. Test runs and commits always separate commands.
 | # | Test | Mutation | Observed |
 |---|---|---|---|
 | G | `hostinstall_gates.py` offsite assertion | pin `keep_last: 2` back | **FAIL** — `arms a CLIENT-SIDE prune (keep_last=2)` |
 | F | `TestGrantRepair_IsBounded` | drop the interval check | **FAIL** — `5 cycles produced 5 attempt(s)` |
 | D(record) | `TestGrantRepair_ARepairedGrantIsReportedAsATransition` | report `ok` after a repair | **FAIL** — `got "ok" — the loss would be invisible` |
 | D(latch) | `TestGrantRepair_TransitionSurvivesALaterProbe` | ignore the latch in the healthy path | **FAIL** — `got "ok" — the host-report would carry ok` |
 | H | `TestMainWiresTheGrantRepair` | pass `nil` instead of the repairer | **FAIL** — `not passed the repairer` |
 | — | `TestGrantRepair_CallsTheExistingWrapperVerb` | — | pass (argv is exactly `grant <id>`; anything else is a policy change) |
 | — | `TestGrantRepair_FailureIsReturned`, `_NilIsSafe` | — | pass |
 **Two red-proofs did not fail on the first attempt, and both are reported rather than quietly fixed:**
 1. **The seam mutation (`grantRepairer` → `nil`) failed to COMPILE** (`declared and not used`), so the
   test never ran. The guard holds — you cannot ship it — but it was not demonstrated. The mutation
   that does fail it keeps the variable used (`_ = grantRepairer`) and passes `nil`.
 2. **The latch red-proof passed**, because the test asserted the latch *helper* rather than the path
   consuming it. Fixed by extracting `storeGrantHealthyVerdict` and pointing the test at it — after
   which the mutation fails as it should. **This is the second hollow test caught in this file**; the
   first asserted a `Status` it had constructed itself. Both are now production functions under test.
 Timestamps in the repair tests carry jitter (`09:39:34`, `+13m07s`, `+27m51s`, …) rather than round
 intervals, per §10.
 ## 7. Files, commits, tag
 `felhom-agent`: `cmd/felhom-agent/main.go` (repairer, latch, `storeGrantHealthyVerdict`,
 `storeGrantRepairedVerdict`, wiring), `cmd/felhom-agent/storegrant_test.go`, `CHANGELOG.md`.
 Commits `257c4d8` (v0.124.0) and the v0.124.1 fix. Released shas: **`5e417938…`** (0.124.0),
 **`5c279bda…`** (0.124.1), both round-trip verified.
 `felhom.eu`: `scripts/felhom-host-install.sh`, `scripts/hostinstall_gates.py`, `scripts/CHANGELOG.md`,
 `manifests/webpage.yaml`, `documentation/backlog/OPEN-ITEMS.md`,
 `documentation/architecture/07-backup-architecture.md`, `CONTEXT.md`, `STATUS.md`, `REPORT.md`.
 **Installer tag: `installer-v1.25.0`.**
 ## 8. The served installer
 ```
 $ curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 SCRIPT_VERSION
 SCRIPT_VERSION="1.25.0"
 $ … | grep -o '"keep_last":[0-9]*'
 "keep_last":0
 ```
 Both `--ref=` lines bumped to `installer-v1.25.0` (327, 372); ArgoCD Synced/Healthy; webpage rollout
 complete. The served bytes carry the change, not just the version.
 ## 9. Deployed versions
 Both demo hosts run **`felhom-agent 0.124.1`**, deployed from the **published** artifact (downloaded
 from Gitea; deployed sha matches the published sha on both). `systemctl is-active` → active on both.
 **NOT VOUCHED** — that stays the operator's act.
 ## 10. Registers
 - **R-191 → CLOSED** (installer 1.25.0 + both boxes), carrying the Part 0 evidence *and* the
  broken-instrument warning, because the latter is the more reusable lesson.
 - **R-190 → MITIGATION SHIPPED, MECHANISM STILL OPEN** — stated in the status field itself so the row
  cannot be misread as solved.
 - **R-194 → filed** (grep established it free): PVE caches permissions — ~40 s in one run and
  ~16 minutes in another between deleting both ACL rows and the permission read reflecting it. It puts
  a floor under detection speed, makes a single permission read a lagging indicator, and is a
  candidate explanation for *when* R-190's refusal surfaced (not for what removed the grant).
 - `07-backup-architecture.md`: the offsite retention row was **already correct** — the configuration
  disagreed with it. Annotated, because a doc that states a contract does not enforce it; the gate does.
 - `ROADMAP.md` holds none of these rows.
 ## 11. Teardown
 The hand-removed grant was restored **by the agent**, which is the test, and confirmed present (2 ACL
 rows). Nothing else was provisioned: no scratch storage, no fixture, no prune job, no probe tag.
 Nothing was deleted from ep0 or from any tier.
 ## 12. Observations — noticed, NOT acted on
 - **PVE's permission cache is the reason detection is not instant** (R-194). Notably, the storage
  *content* listing reflected the loss immediately (`{"data":[]}`) while the *permission* read still
  said present — two signals disagreeing, and today only one is read. Worth deciding whether the probe
  should consult both.
 - **`logCapabilities` runs `probeAll` independently of the report path.** The latch makes that safe
  for this capability, but any future probe with one-shot semantics will hit the same trap. The
  general fix would be one probe per report cycle, shared — deliberately not attempted here.
 - **demo-hp's offsite tier is still `escrow_state: escrowed, snapshot_count: 0`** after this morning's
  re-issue; whether the pre-rebuild 15 snapshots reattach is answered by its next offbox run (04:15),
  and belongs to R-193 rather than to this session.
 - **The `felhom-pbs` tier is armed on both boxes with `cadence=168h`**, so the R-191 fix cannot be
  observed end-to-end until that cadence fires. Nothing was triggered manually to shortcut it.
@@ -1,6 +1,6 @@
 # STATUS — what works, what's broken, what's next
-**Updated 2026-08-03.**
+**Updated 2026-08-04.**
 > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
 > page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
@@ -20,10 +20,21 @@ Proven end to end on real hardware.
 ## What's broken
- **Rebuilding a machine silently takes away its off-site app-data backup.** Fixed on demo-hp on
+- **Rebuilding a machine throws away its off-site backup HISTORY — and that is worse than we thought
-  4 August (re-issued, picked up in 15 seconds, same repository, you escrowed the key), but the
+  yesterday.** We spent today establishing what a rebuild actually destroys, and the answer changes the
-  underlying fault stands: the password is delivered once and a rebuilt machine cannot ask again.
+  picture. There are three secrets involved. The one we have been talking about — the password to the
-  The daily email about it also tells the wrong story. *(R-193, R-192)*
+  storage provider — is the one you can re-issue any time, and we did. **The one that matters is the
  key that encrypts the backups themselves. The machine invents that key, and a rebuilt machine
  invents a brand-new one**, so the old backups can no longer be opened by the machine that wrote
  them.
  **Both demo machines did exactly this, and one of them did it silently.** demo-hp lost access to
  15 backups (41 MB). demo-felhom — the one we called lucky, because its off-site copy came back on
  its own in 76 seconds — **lost access to 36 backups (1.14 GB), and nothing said a word for
  thirteen hours.** The luck restored the delivery, not the history.
  Nothing has been deleted, and this was measured without touching either machine or any password.
  **The next scheduled off-site run is tomorrow at 04:15, and it will refuse rather than quietly start
  a new history** — the machine already knows how to recognise this and asks before resetting.
  **There is a decision here for you — see "Waiting on you".** *(R-193, R-192, and two new items)*
 - *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
  and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
  asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
@@ -34,6 +45,14 @@ Proven end to end on real hardware.
 ## What shipped recently
 - **The daily error email about David is gone.** David's record was created when he said he would try
  Felhom; no machine was ever attached to it, so nothing has ever been expected of it — yet every
  morning at five the system reported that a database backup had not happened. **A customer with no
  machine ever attached now produces no alarm.** A machine that *was* attached and went quiet still
  does, because that is a real fault, and there is a test whose whole job is to make sure a future
  change cannot take that away. David's record was not touched — the record was right; the alarm was
  wrong. **The first quiet morning will be 5 August, so this is not confirmed yet.** *(R-195)*
 - **A permission the machine needs can now repair itself — and it tells you it had to.** Yesterday we
  found a permission that vanished with no reinstall and no recorded action; the machine now notices,
  restores it through the same controlled helper that grants it in the first place, and **raises an
@@ -58,7 +77,12 @@ Proven end to end on real hardware.
 ## What we're working on
- **Now:** nothing outstanding.
+- **Now:** waiting on your answer about the backup key (above). Two small honesty fixes are ready to
  go the moment you say so — making a changed backup key raise an alarm, and correcting an email that
  currently states the opposite of what it measured. Neither needs a decision.
 - **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
  refuse and ask, rather than start fresh — but that is a prediction from the code, not something
  anyone has watched happen.
 - **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
  tests unattended. Most of the machinery it needed arrived with the restore-test change below.
  *(R-87)*
@@ -66,6 +90,22 @@ Proven end to end on real hardware.
 ## Waiting on you
 - **A real decision, and it is the important one on this page.** Every machine is going to be
  reinstalled at some point, and each reinstall throws away the key that opens its off-site backups.
  There are only two honest answers, and picking either is fine — picking neither is what leaves it
  where it is.
  **(a) Keep a copy of that key on the Proxmox host**, which survives a rebuild. The machinery for
  this already exists and is barely any work. The price: the key would then sit on a machine you have
  root on, so "only the customer's recovery code can open these backups" stops being true in the
  strict sense. It was never quite true anyway — you hold root everywhere — but this makes a real
  second copy, and copies are exactly what we are already uneasy about elsewhere.
  **(b) Leave the key sealed under the customer's recovery code**, accept that every rebuild starts a
  fresh off-site history, and make sure the system says so loudly on the day it happens instead of
  never. Old backups are kept, not deleted, but they keep consuming the customer's off-site space.
  **We recommend doing the "says so loudly" part immediately regardless** — it is cheap, it costs
  nothing else, and it is the piece whose absence let a gigabyte of history go quiet for thirteen
  hours. What we are asking you to choose is (a). *(R-193; the full reasoning is in the findings
  document)*
 - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
  session log; nothing suggests anyone else saw it. *(R-132)*
 - **One small question, not urgent.** The automatic check cannot see which version you have told
@@ -75,6 +115,10 @@ Proven end to end on real hardware.
 ## Changed since last update
 - **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage
  password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost
  access to their off-site history, one of them silently. A decision is now waiting on you. The daily
  false alarm about David is fixed. *(R-193, R-195)*
 - **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
  overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
  weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
@@ -26,7 +26,7 @@
 | R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 |
 | R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 |
 | R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/<customer>`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 |
-| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path |
+| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path. **SCOPE-CORRECTED 2026-08-04 (R-193 spike, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`): "auto-fire on re-enrollment" means HOST re-enrollment, and only when the host record was DELETED first.** `reissueOnReenroll` sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`, `api/handler.go:1004-1016`), so a **guest** rebuild — which leaves the `hosts` row intact — never reaches F2 or F3. Measured live: demo-hp's guest was rebuilt 2026-08-03, `demo-hp-bb76ea` was unchanged throughout, F3 did not fire, and the box had **no off-site copy for 25 hours** until an operator ran Re-issue by hand. Peti's clean-slate (a real host delete + re-enroll) still walks the supported path; **the arc does not cover a guest rebuild, and that is the gap R-193 owns.** Note also that F3 alone would NOT have saved the data — the rebuilt controller mints a fresh restic repo password regardless (spike Q2) |
 | R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
 | R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
 | R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none` → `p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** |