# REPORT — R-241 spike · golden 0.205.0 · finalwalk teardown (2026-08-07) *A sibling report: `REPORT.md` is overwritten per-session and a parallel session shares this clone.* **Three parts, in the order the venue's perishability required.** The spike read VM 324 before anything else touched the fleet; the bake and the vouch followed; the teardown went last. --- ## 1. THE SESSION'S ANSWER — R-241 is a MINTING defect, not a screen-predicate defect **This reverses the fix.** The recovery screen was telling the truth: there genuinely was nothing recoverable under the key the box held, because **the box minted that key itself, over the top of a sealed package it already knew the hub was holding for it.** Mending the predicate would have papered over a box quietly making its own backup history unopenable. **Three measurements, taken from the venue this session, not copied from last night's journal:** 1. **`WriteOffboxSecrets` (`offbox.go:411`) mints on ONE input — does the file exist.** No settings read at all, while its two neighbours in the same file, `OffsiteRecoveryOffer()` (`:1412`) and `needsOffsiteCredential()` (`:1377`), both consult `GetHubEscrowIdentityPresent()`. **The same fact is available on three paths and used on two.** 2. **That flag was not merely available — it was the precondition of the chain that reached the minting.** The 5-minute retry job logs only when `RetryIfDeclared` fires, which requires the declaration, which requires the flag. The venue logged `credential retry: … (the box still declares a need; retrying)` at **02:48:03Z** and five times after — **thirty minutes and six ticks before the mint at 03:18:06Z.** 3. **The box computed the right answer and threw it away.** At **03:28:03Z** — thirty-five minutes before the customer looked — `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:154`) logged `the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local 9b4a9a9dcec7…) … staying pending`. It is recomputed on every report cycle, **never persisted, never surfaced.** **And the hub explicitly disclaims doing this** — `offsiteheal`'s package doc: *"it never runs, or asks for, an escrow ceremony… **credential automatic, key customer-present** — the ruling this session implements and must not quietly widen."* The repository key is created **in the seam between two sides that each honoured their contract**, by a helper doing exactly what its doc comment says. That is why it survived review: every individual comment is accurate. **Corroboration from both sides, measured independently here:** the two key hashes on the venue (`30ef574f…` current, `9b4a9a9d…` in `repo_password.selfheal-aside`, mtime 03:18:06) reproduce the journal's figures exactly; and the hub's `host_escrow` row for `finalwalk-ed05d6` seals `restic_pw_sha256 = 30ef574f…` with a 572-byte `identity_blob` and **no superseded row** — so the hash the box logged as *"hub hash"* is confirmed from the database, not inferred. **Q2 — shape (b) is structurally unreachable**, not merely unfired: `markOrphaned()` has one producer, `ensureOffboxRepo()`, reached only from inside a run, and every run passes `if !m.offboxEscrowed()` (`offbox.go:743`) **first**. The box was `pending` and could not stop being pending, because the auto-confirm flips only on a hash match. *Positive control, because an absent log line is not evidence:* the scheduler was alive throughout (241 `agent-channel-health`, 120 `stack-scan`, 48 `offsite-credential-retry`, 16 `hub-report`), and `offbox-backup` is a `sched.Daily` leg whose slot fell **before** the destruction — the absence is explained, not just observed. **Q7 — the „Helyreállítási kód létrehozása" button.** It does **not** destroy the data: `SaveHostEscrow` demotes the current row into `host_escrow_superseded` **copy-before-delete, `identity_blob` included** (R-198). But the read path for a retained package is **unbuilt** (R-199), so it converts a one-screen-away self-service recovery into one needing an operator and tooling that does not exist. **And it re-enables the screen while invalidating the code that screen accepts** — a worse trap than a plain dead end, because it looks like progress. *(Reasoned from code and the hub schema; I did not press it — that is a state change and would have destroyed the evidence.)* **Two findings nobody asked for**, both filed: **R-243** (a box in this state silently stops backing up and **no alarm fires** — three individually-correct exclusions leave one state unobserved) and the trap in the obvious fix (`ResetOrphanedRepo` clears the orphan flag without a ceremony, so a hash-mismatch discriminator alone would re-offer the screen forever to a customer who declined the old data). **Output:** `audits/SPIKE-r241-recovery-offer-2026-08-07.md` — question/method/measurement/ruling, the Q4 seven-state table, ranked options, and **four operator decisions stated and left unanswered.** **No product code was written, in any part of this session.** --- ## 2. GOLDEN 0.205.0 — baked, verified, VOUCHED (R-239 CLOSED) | | | |---|---| | version | **0.205.0** · sha256 `8f49b2e8ccbc86a49df821fee9fb00c07293758811d3d0f0512dd0cf5fd54ee8` | | size | 656,937,561 bytes (uncompressed 2,003,343,360) | | MinAgent | 0.127.0 | **Round-trip verified rather than trusted:** the published bytes were fetched back (HTTP 200, byte count matches), re-hashed **independently of the baker** (matches), `zstd -t`'d clean, and `./etc/felhom-controller-image` was read **out of the downloaded archive** → `felhom-controller:0.205.0`. That last step is the one that matters, because `GOLDEN_VERSION` is derived from the tag argument and could have been right over stale content. A **third witness**: the hub's own dropdown lists `0.205.0` with `data-sha="8f49b2e8…"`. All acceptance markers pass (`docker OK (overlay2` ×1, `including mount point` ×2 = rootfs + mp0, `upload OK (HTTP 201)` ×1, and 0 each for `excluding`/`FATAL`/`ERROR:`/`WARN:`); unit `Result=success`, `ExecMainStatus=0`. Bake VM torn down: CT 9100 purged, secrets shredded, qemu observed gone via `ps -eo comm`, `drill.qcow2` reverted to `virgin`. **Secret hygiene:** token copied file→file; the invocation lives in an in-VM runner that reads the token itself, so it never reached a command line (`systemctl show … | grep -c -F ` → **0**); literal-value leak grep on the **committed** log → **0**, and **the instrument was proven first** — token appended to a throwaway copy → 1 hit → copy shredded → the 0 is a measurement. **Vouched, with the operator's approval.** In the event it was a **ONE-field change, not three** — read from live `hub_settings` before and after, not assumed: | field | before | after | |---|---|---| | `artifact_golden_version` | 0.203.0 | **0.205.0** | | `artifact_golden_sha256` | `3039c6ff…` | **`8f49b2e8…`** | | `artifact_agent_version` | 0.127.0 | 0.127.0 — **unchanged** | | `artifact_min_agent` | 0.127.0 | 0.127.0 — **unchanged** | **A trap worth naming:** `wrapper_sha256` is read from the form and **cleared when omitted**. A headless POST that forgets it silently drops the PBS-DR wrapper hash. It was carried through explicitly and verified present afterwards. **The R-120 gate passed exactly** — the newest controller the fleet reports is 0.205.0 (demo-hp), so a **0.204.0 golden would have been REFUSED**. Vouching is reversible; a bake never deletes an older golden's package. **§4.1's systemic half is recorded, NOT built** → **R-242**, with three proposed shapes and a stated earliest-catch (a `repo_gates.py` comparison of the manifest's `golden_version` against the newest released controller — it fires on the push that creates the gap, before any box is installed). --- ## 3. TEARDOWN — `finalwalk`, all five layers Enumerated first and cross-checked against the hub's **own** delete-preview, which agreed in every field. Matched on **identity**, never on size. The cascade refuses a live host, so VM 324 was stopped at 07:23:56Z (guarded on `qm config 324` reading `name: finalwalk-appliance` — `demo-hp` also carries a guest 9201) and aged past the 30 m `stale_threshold`, read from the deployed config. `POST /configs/finalwalk/delete` with all six gates → the hub's leg-by-leg log shows host deleted, off-site sub-account 285071 deprovisioned, PBS namespace deprovisioned (`existed=true` — the positive observable), claim reset, residue purged (**124 rows, matching the preview exactly**), `COMPLETE … full teardown`. Then `qm destroy 324 --purge`. **Every layer verified absent with a positive control that must survive, and does:** VM 324 gone (VM 300 `drill-r50` remains) · all 13 hub tables at 0 including **both** escrow tables (demo-felhom 50,823 / demo-hp 7,640 / peti 1,827 rows remain) · Storage Box `u629488-sub4` gone (sub1/2/3 remain) · `ep0` namespace `finalwalk` gone (demo-felhom, demo-hp remain) · WireGuard `10.77.0.5` gone **from the live `wg show` on ep0, not merely from the hub DB** (`.2/.3/.4/.250` remain). **14.06 GiB reclaimed**, against 15 G measured before deletion. **R shredded with a planted-copy control** — plant → search finds both → shred → same search finds 0. The zero was not believed until the instrument was proven. ### ⚠ And a correction I am reporting rather than quietly fixing A **full census** (every table, every column) after the "COMPLETE" cascade found **61 rows still matching `finalwalk`**. Four sources are deliberate (`events`, `notification_log`, `host_deletions`, `customer_resets` — the cascade's header says provenance outlives every tier). **The fifth is a gap:** `app_log_issues`, 29 rows, not covered by the residue purge — and **systematic**, with `c11` 40, `rewalk` 20 and `part4` 24 still present from the 2026-08-06 teardown, **whose ledger recorded "0 occurrences"**. That claim used a narrower query than a census and does not hold. Both the prior ledger and the register now carry the correction. **No secret material is involved.** The table is a fleet-wide aggregate; 12 of the 29 rows are `finalwalk`-only orphans and **17 are shared with live customers and must be de-referenced, not deleted** — very likely why the leg was never written. Filed as **R-244**, not fixed: a cascade change needs its own red-proof. **The reusable lesson: a per-table absence query is not a census.** --- ## Register | ID | Movement | |---|---| | **R-241** | **RULED** — minting defect. Diagnosed, **not fixed**; four operator decisions owed | | **R-239** | **CLOSED** — golden 0.205.0 baked, verified and vouched | | **R-242** | **NEW** — a release is not delivered until a golden carries it; recorded, **not built** | | **R-243** | **NEW** — the R-241 state silently stops off-site backups with no alarm | | **R-244** | **NEW** — `app_log_issues` survives the delete cascade, across all four torn-down venues | **Highest register ID moved R-241 → R-244.** ## Verification | | | |---|---| | commits | `71c43f87c240` (spike) · `08b75e602e01` (bake evidence) · `db578cd44d3e` (R-239 closed, map + STATUS) · this one | | CI | run **233** `71c43f87c240` success · run **234** `08b75e602e01` success — matched by `head_sha`, pulled not assumed | | `--no-verify` | **not used.** The pre-push hook ran `repo_gates.py --fast` on every push and reported `gates OK` | | gates | `python3 scripts/repo_gates.py --fast` → all six OK before each commit | ## What did not run, and why - **No product code, deliberately** — Part 1 was a question, and the answer changes what the fix should be. Beginning a candidate before the operator rules on §"THE OPERATOR'S DECISION" would prejudge it. - **Q4 row 7 and Q7's post-button behaviour are reasoned from code, not measured.** Both need a state change on the venue; either would have destroyed the evidence for everything else. They need a fresh fixture and their own session — and **row 7 shapes the fix**, so it matters. - **R-242, R-243 and R-244 are filed and not built** — R-242 because the task scoped it record-only; the other two because they surfaced inside a spike and an operation, and each needs its own red-proof. - **The exact wall-clock at which `hub_escrow_identity_present` first became true is not measured** — `SetHubEscrowIdentityPresent` writes only on change and logs nothing. The bound that matters is established by control flow: true at or before **02:48:03Z**.