felhom.eu/REPORT-r241-spike-golden-teardown-2026-08-07.md

REPORT — R-241 spike · golden 0.205.0 · finalwalk teardown (2026-08-07)

A sibling report: REPORT.md is overwritten per-session and a parallel session shares this clone.

Three parts, in the order the venue's perishability required. The spike read VM 324 before anything else touched the fleet; the bake and the vouch followed; the teardown went last.

---

1. THE SESSION'S ANSWER — R-241 is a MINTING defect, not a screen-predicate defect

This reverses the fix. The recovery screen was telling the truth: there genuinely was nothing recoverable under the key the box held, because the box minted that key itself, over the top of a sealed package it already knew the hub was holding for it. Mending the predicate would have papered over a box quietly making its own backup history unopenable.

Three measurements, taken from the venue this session, not copied from last night's journal:

1. WriteOffboxSecrets (offbox.go:411) mints on ONE input — does the file exist. No settings read at all, while its two neighbours in the same file, OffsiteRecoveryOffer() (:1412) and needsOffsiteCredential() (:1377), both consult GetHubEscrowIdentityPresent(). The same fact is available on three paths and used on two.
2. That flag was not merely available — it was the precondition of the chain that reached the minting. The 5-minute retry job logs only when RetryIfDeclared fires, which requires the declaration, which requires the flag. The venue logged credential retry: … (the box still declares a need; retrying) at 02:48:03Z and five times after — thirty minutes and six ticks before the mint at 03:18:06Z.
3. The box computed the right answer and threw it away. At 03:28:03Z — thirty-five minutes before the customer looked — EscrowAutoConfirmer.Reconcile (escrow_confirm.go:154) logged the hub's escrow blob does not cover the CURRENT repo password (hub hash 30ef574fe492… != local 9b4a9a9dcec7…) … staying pending. It is recomputed on every report cycle, never persisted, never surfaced.

And the hub explicitly disclaims doing this — offsiteheal's package doc: "it never runs, or asks for, an escrow ceremony… credential automatic, key customer-present — the ruling this session implements and must not quietly widen." The repository key is created in the seam between two sides that each honoured their contract, by a helper doing exactly what its doc comment says. That is why it survived review: every individual comment is accurate.

Corroboration from both sides, measured independently here: the two key hashes on the venue (30ef574f… current, 9b4a9a9d… in repo_password.selfheal-aside, mtime 03:18:06) reproduce the journal's figures exactly; and the hub's host_escrow row for finalwalk-ed05d6 seals restic_pw_sha256 = 30ef574f… with a 572-byte identity_blob and no superseded row — so the hash the box logged as "hub hash" is confirmed from the database, not inferred.

Q2 — shape (b) is structurally unreachable, not merely unfired: markOrphaned() has one producer, ensureOffboxRepo(), reached only from inside a run, and every run passes if !m.offboxEscrowed() (offbox.go:743) first. The box was pending and could not stop being pending, because the auto-confirm flips only on a hash match. Positive control, because an absent log line is not evidence: the scheduler was alive throughout (241 agent-channel-health, 120 stack-scan, 48 offsite-credential-retry, 16 hub-report), and offbox-backup is a sched.Daily leg whose slot fell before the destruction — the absence is explained, not just observed.

Q7 — the „Helyreállítási kód létrehozása" button. It does not destroy the data: SaveHostEscrow demotes the current row into host_escrow_superseded copy-before-delete, identity_blob included (R-198). But the read path for a retained package is unbuilt (R-199), so it converts a one-screen-away self-service recovery into one needing an operator and tooling that does not exist. And it re-enables the screen while invalidating the code that screen accepts — a worse trap than a plain dead end, because it looks like progress. (Reasoned from code and the hub schema; I did not press it — that is a state change and would have destroyed the evidence.)

Two findings nobody asked for, both filed: R-243 (a box in this state silently stops backing up and no alarm fires — three individually-correct exclusions leave one state unobserved) and the trap in the obvious fix (ResetOrphanedRepo clears the orphan flag without a ceremony, so a hash-mismatch discriminator alone would re-offer the screen forever to a customer who declined the old data).

Output: audits/SPIKE-r241-recovery-offer-2026-08-07.md — question/method/measurement/ruling, the Q4 seven-state table, ranked options, and four operator decisions stated and left unanswered. No product code was written, in any part of this session.

---

2. GOLDEN 0.205.0 — baked, verified, VOUCHED (R-239 CLOSED)

version	0.205.0 · sha256 8f49b2e8ccbc86a49df821fee9fb00c07293758811d3d0f0512dd0cf5fd54ee8
size	656,937,561 bytes (uncompressed 2,003,343,360)
MinAgent	0.127.0

Round-trip verified rather than trusted: the published bytes were fetched back (HTTP 200, byte count matches), re-hashed independently of the baker (matches), zstd -t'd clean, and ./etc/felhom-controller-image was read out of the downloaded archive → felhom-controller:0.205.0. That last step is the one that matters, because GOLDEN_VERSION is derived from the tag argument and could have been right over stale content. A third witness: the hub's own dropdown lists 0.205.0 with data-sha="8f49b2e8…".

All acceptance markers pass (docker OK (overlay2 ×1, including mount point ×2 = rootfs + mp0, upload OK (HTTP 201) ×1, and 0 each for excluding/FATAL/ERROR:/WARN:); unit Result=success, ExecMainStatus=0. Bake VM torn down: CT 9100 purged, secrets shredded, qemu observed gone via ps -eo comm, drill.qcow2 reverted to virgin.

Secret hygiene: token copied file→file; the invocation lives in an in-VM runner that reads the token itself, so it never reached a command line (systemctl show … | grep -c -F <token> → 0); literal-value leak grep on the committed log → 0, and the instrument was proven first — token appended to a throwaway copy → 1 hit → copy shredded → the 0 is a measurement.

Vouched, with the operator's approval. In the event it was a ONE-field change, not three — read from live hub_settings before and after, not assumed:

field	before	after
artifact_golden_version	0.203.0	0.205.0
artifact_golden_sha256	3039c6ff…	8f49b2e8…
artifact_agent_version	0.127.0	0.127.0 — unchanged
artifact_min_agent	0.127.0	0.127.0 — unchanged

A trap worth naming: wrapper_sha256 is read from the form and cleared when omitted. A headless POST that forgets it silently drops the PBS-DR wrapper hash. It was carried through explicitly and verified present afterwards. The R-120 gate passed exactly — the newest controller the fleet reports is 0.205.0 (demo-hp), so a 0.204.0 golden would have been REFUSED. Vouching is reversible; a bake never deletes an older golden's package.

§4.1's systemic half is recorded, NOT built → R-242, with three proposed shapes and a stated earliest-catch (a repo_gates.py comparison of the manifest's golden_version against the newest released controller — it fires on the push that creates the gap, before any box is installed).

---

3. TEARDOWN — finalwalk, all five layers

Enumerated first and cross-checked against the hub's own delete-preview, which agreed in every field. Matched on identity, never on size.

The cascade refuses a live host, so VM 324 was stopped at 07:23:56Z (guarded on qm config 324 reading name: finalwalk-appliance — demo-hp also carries a guest 9201) and aged past the 30 m stale_threshold, read from the deployed config. POST /configs/finalwalk/delete with all six gates → the hub's leg-by-leg log shows host deleted, off-site sub-account 285071 deprovisioned, PBS namespace deprovisioned (existed=true — the positive observable), claim reset, residue purged (124 rows, matching the preview exactly), COMPLETE … full teardown. Then qm destroy 324 --purge.

Every layer verified absent with a positive control that must survive, and does: VM 324 gone (VM 300 drill-r50 remains) · all 13 hub tables at 0 including both escrow tables (demo-felhom 50,823 / demo-hp 7,640 / peti 1,827 rows remain) · Storage Box u629488-sub4 gone (sub1/2/3 remain) · ep0 namespace finalwalk gone (demo-felhom, demo-hp remain) · WireGuard 10.77.0.5 gone from the live wg show on ep0, not merely from the hub DB (.2/.3/.4/.250 remain). 14.06 GiB reclaimed, against 15 G measured before deletion.

R shredded with a planted-copy control — plant → search finds both → shred → same search finds 0. The zero was not believed until the instrument was proven.

⚠ And a correction I am reporting rather than quietly fixing

A full census (every table, every column) after the "COMPLETE" cascade found 61 rows still matching finalwalk. Four sources are deliberate (events, notification_log, host_deletions, customer_resets — the cascade's header says provenance outlives every tier). The fifth is a gap: app_log_issues, 29 rows, not covered by the residue purge — and systematic, with c11 40, rewalk 20 and part4 24 still present from the 2026-08-06 teardown, whose ledger recorded "0 occurrences". That claim used a narrower query than a census and does not hold. Both the prior ledger and the register now carry the correction.

No secret material is involved. The table is a fleet-wide aggregate; 12 of the 29 rows are finalwalk-only orphans and 17 are shared with live customers and must be de-referenced, not deleted — very likely why the leg was never written. Filed as R-244, not fixed: a cascade change needs its own red-proof. The reusable lesson: a per-table absence query is not a census.

---

Register

ID	Movement
R-241	RULED — minting defect. Diagnosed, not fixed; four operator decisions owed
R-239	CLOSED — golden 0.205.0 baked, verified and vouched
R-242	NEW — a release is not delivered until a golden carries it; recorded, not built
R-243	NEW — the R-241 state silently stops off-site backups with no alarm
R-244	NEW — app_log_issues survives the delete cascade, across all four torn-down venues

Highest register ID moved R-241 → R-244.

Verification

commits	71c43f87c240 (spike) · 08b75e602e01 (bake evidence) · db578cd44d3e (R-239 closed, map + STATUS) · this one
CI	run 233 71c43f87c240 success · run 234 08b75e602e01 success — matched by head_sha, pulled not assumed
--no-verify	not used. The pre-push hook ran repo_gates.py --fast on every push and reported gates OK
gates	python3 scripts/repo_gates.py --fast → all six OK before each commit

What did not run, and why

• No product code, deliberately — Part 1 was a question, and the answer changes what the fix should be. Beginning a candidate before the operator rules on §"THE OPERATOR'S DECISION" would prejudge it.
• Q4 row 7 and Q7's post-button behaviour are reasoned from code, not measured. Both need a state change on the venue; either would have destroyed the evidence for everything else. They need a fresh fixture and their own session — and row 7 shapes the fix, so it matters.
• R-242, R-243 and R-244 are filed and not built — R-242 because the task scoped it record-only; the other two because they surfaced inside a spike and an operation, and each needs its own red-proof.
• The exact wall-clock at which hub_escrow_identity_present first became true is not measured — SetHubEscrowIdentityPresent writes only on change and logs nothing. The bound that matters is established by control flow: true at or before 02:48:03Z.