Files
felhom.eu/documentation/audits/DRILL-retained-key-2026-08-12.md
T
admin 1d5f2b8bb6
gates / gates (push) Successful in 23s
DRILL: the retained key works, and the customer cannot reach it
Three verdicts, kept separate because collapsing them is how this assumption
survived a week.

(a) The material IS retained. host_escrow_superseded id 11 is the first retained
row in fleet history to carry identity_blob (572 B), byte-identical to the
pre-supersession row (sha256 a10032341c8584ed...).

(b) The retained material DOES open the old store. Unsealed with the old recovery
code it yielded a password byte-identical to the pre-change one, and restored
three planted files byte-identical from a store the box itself could no longer
open - including a Hungarian accented filename verified as raw bytes. Negative
control ran first and failed closed.

(c) The customer has NO route, and is misinformed. ListSupersededEscrow has zero
production callers; the recovery path selects FROM host_escrow. Asked with the
code that had just worked by hand, the product answered "the recovery code did
not open the sealed bundle". A valid code for retained history is reported as a
bad code - the R-224 class again. R-304, rank 1.

Both installer faults were watched happening first, so installer-v1.27.0 is now
published (tag + both webpage.yaml refs). Pre-fix: the box came up on controller
0.98.3 against a vouched 0.213.0, below the floor and below the version carrying
the recovery screen; and our own uninstall left dnsmasq on 0.0.0.0:53 so our own
next install refused. R-297 and R-300 CLOSED.

Also filed R-305 (the dnsmasq fix fires once per machine - the leftover returns
on the second reinstall, proven), R-306 (--preflight-only writes state it says it
does not), R-307 (a live abandon countdown on demo-felhom, firing 2026-08-24 -
operator decision), R-308 (stored controller password stale), R-309 (the day-0
runbook's publication claim has been false since R-110), R-310 (two edges).

Ceiling R-303 -> R-310. Capability map moved: the retention claim is now marked
operator-only. Phase A logs did not survive the intermediate revert; recorded.
2026-08-12 17:41:56 +02:00

28 KiB
Raw Blame History

DRILL — the retained key, and the two fixes nobody had watched work

Date: 2026-08-12 · Class: drill (unattended, destructive on Tier 0; spike for Phase C's first step) Venues: drill-r50 (nested PVE on DooPlex) · demo-felhom (guest 9201 on felhom-pve) — both Tier 0 Baselines re-confirmed live on arrival, not assumed: controller 0.213.0 on both demo boxes (pct exec 9201 -- docker ps), agent 0.128.0 (hub hosts), hub 0.102.0 (deployment image), register ceiling R-303 (grep), felhom.eu clean at fbe1155f and level with origin/main.


1. The three verdicts, first, in plain language

(a) Is the material retained? — YES. Proven for the first time in the fleet's history.

(b) Does the retained material, once unsealed, yield a password that actually opens the old store? — YES. Three planted files came back byte-identical from a store the box itself could no longer open, including the Hungarian accented filename.

(c) Does the customer have a route to (b) through the product? — NO. And the failure is worse than absence: the customer's old recovery code — the one that demonstrably opens their old backups — is rejected by the product with "the recovery code did not open the sealed bundle". They are told their correct code is wrong, and given no next step.

The headline the operator asked for. The brief said to be ready for the answer to be no, and our own records predicted the retention would be "a box we fill and cannot open". That prediction was half right, and the wrong half was the one nobody checked. The box can be opened — the crypto, the retention and the copy are all sound. What does not exist is the door: no code path reads a retained row, so the only way in is an operator with SQLite, age and a shell. Everything I did in §4 a customer cannot do.

That distinction matters for the three pieces of work resting on this:

Rests on Verdict
The census answer ("nobody else is exposed") Stands. It was a statement about retention, and retention works.
The countdown banner's promise (old backups stay recoverable) True in substance, false in practice. The bytes are recoverable; the customer cannot recover them. The promise is made to someone who has no way to act on it.
The capability map's recovery claims Must move. Any claim of the form "the customer can recover the old history with their recovery code" is false today.

2. §7.0 — the probe, run first and read-only

Run before anything was installed or planted, exactly as the order required.

Q1 — does any host today hold a superseded record with key material present? NO. Zero, fleet-wide.

host_id             kfp16             blob_len  identity_blob  superseded_at
demo-hp-bb76ea      3f:4f:65:c0:d8:f  383       NULL           2026-08-04 07:15:36
demo-felhom-8363b5  7e:a6:af:f7:ea:6  383       NULL           2026-08-04 07:20:08

Both are the pre-R-198 rows written hours before the fix. SELECT COUNT(*) … WHERE identity_blob IS NOT NULL returned 0. The retention as fixed had never once been exercised.

Instrument: hub SQLite copied WITH hub.db-wal + -shm (the standing gotcha). Freshness proved by a positive observable — newest host_reports row 61 s old — not by "the query returned no error". PRAGMA integrity_check = ok.

Q2 — what act produces a retaining supersession? Two, both read from source:

  • hub/internal/store/store.go:2807SaveHostEscrow demotes when the newly sealed restic_pw_sha256 differs from the stored one.
  • hub/internal/store/store.go:2636DeleteHost demotes the current row unconditionally.

Q3 — is there any code path that USES a retained record to recover a password? NO.

The product's recovery route is real and wired (R-199) but reaches the current row only:

controller  →  agent POST /escrow/recover-offsite-password
            →  escrow.OffsiteKeyRecoverer.Fetch      felhom-agent/cmd/felhom-agent/main.go:1757
            →  hubClient.FetchIdentityEscrow
            →  hub GET identity-escrow handler       hub/internal/api/handler.go:1325
            →  store.GetHostDRBundle                 hub/internal/store/store.go:3152
               SELECT blob, identity_blob, directive_json FROM host_escrow WHERE host_id = ?

ListSupersededEscrow (store.go:2841) is the sole reader of a retained identity_blob. It has zero production callers — five call sites, all in _test.go. CountSupersededEscrow is used twice and only to render a count (api/handler.go:1209, web/hosts.go:572).

Feasibility verdict: producible. Phase C proceeded.


3. Phase C — producing the state

3.1 The planted dataset

Planted in /mnt/sys_drive/felhom-data/userdata/drill-20260812 on demo-felhom.

file sha256 bytes
payload.bin 6863824a67bf6cb636defca3cbddc1d58092bcdda3af939224dfef4663c15359 65536
plain-ascii.txt e294e0ee5b8fd131dc0fa71f5cf1253dc39473241aa090be36404a568285c83a 56
árvíztűrő-tükörfúrógép.txt a1957a87104c63be22b34e4f937e700edcdc925da5984bfcca407df4989f7190 37

The accented filename never crossed a shell as text (rule 6). It was built from explicit bytes and verified as hex, both as intended and as listed on disk — identical:

c3 a1 72 76 c3 ad 7a 74 c5 b1 72 c5 91 2d 74 c3 bc 6b c3 b6 72 66 c3 ba 72 c3 b3 67 c3 a9 70 2e 74 78 74

3.2 Positive control on the comparator — done BEFORE it was relied on

One byte of a copy of payload.bin flipped at offset 40000:

unaltered  f5de179b5ca941645819a879187db953db5c973d4c1a07b2a3ba96d3d79bfb6c
altered    515a65d96326bd847a87ea35c9fb5e6afebc9499ec0c1cd80ca3531b22cb2324
payload.bin: FAILED · sha256sum: WARNING: 1 computed checksum did NOT match

The comparator convicts. The copy was discarded and the original re-hashed unchanged.

3.3 The store, written under P1

restic backup into demo-felhom's own repository path (sftp:u629488-sub1@…your-storagebox.de:/home/felhom-repo) → snapshot 6ea85413, confirmed by identity and by listing its contents, not by the success line:

6ea85413  2026-08-12 15:15:33  demo-felhom  felhom-offbox,drill-retained-key-20260812
  …/drill-20260812/payload.bin
  …/drill-20260812/plain-ascii.txt
  …/drill-20260812/M-CM-!rvM-CM--ztM-EM-1rM-EM-^Q-tM-CM-<kM-CM-6rfM-CM-:rM-CM-3gM-CM-)p.txt

Key fingerprint of the store's password (P1): sha256 c60c8bc737a6b7c6… — value never recorded.

3.4 The supersession, by the product's own ceremony

The repository password was replaced (P1 → P2, d4aac4f1c0ef9ac0…), making /home/felhom-repo the old store, then the ceremony was driven through the same local-API calls the dashboard wizard makes: POST /escrow/stage-secretPOST /escrow/ceremony…/status…/claim.

phase=done  restic_pw_sealed=true  uploaded=true

3.5 (a) — the material IS retained

The hub grew its first ever retained row carrying key material:

id  host_id             identity_blob  restic_pw_sha256    superseded_at
11  demo-felhom-8363b5  572            c60c8bc737a6b7c6…   2026-08-12 15:18:55

Confirmed by identity and byte length, not by presence:

a10032341c8584edfd87ad941a50d9fd6f34c23a4802f55885cb1d4cab086bd3  current-identity.age   (pre-supersession)
a10032341c8584edfd87ad941a50d9fd6f34c23a4802f55885cb1d4cab086bd3  retained-identity.age  (retained row 11)
572 bytes both — IDENTICAL

3.6 (b) — the retained key DOES open the old store

Negative control first — the box's current password against the old store:

restic exit code: 1
Fatal: wrong password or no key found

Then the retained key. Retained blob → age -d with the old recovery code → bundle → restic_repo_password, sha256 c60c8bc737a6b7c6…, byte-identical to the P1 recorded before the supersession:

restic snapshots  → exit 0, snapshot 6ea85413 listed
restic restore    → exit 0
restored accented filename bytes: c3 a1 72 76 c3 ad 7a 74 c5 b1 72 c5 91 2d … (identical)
payload.bin: OK · plain-ascii.txt: OK · árvíztűrő-tükörfúrógép.txt: OK   (compare exit code 0)

All three files byte-identical, from a store the box itself could not open.

3.7 (c) — the customer has no route, and is actively misinformed

The product's own recovery endpoint was asked for the old password, using the same old recovery code that had just worked by hand:

OLD code (opens the retained row by hand → P1)   REFUSED: "the recovery code did not open the
                                                  sealed bundle — nothing was written"
NEW code (today's ceremony)                      RETURNED a password, sha d4aac4f1c0ef9ac0 = P2

P2 does not open the old store. The one code that works is reported as the code that does not. This is the R-224 defect class in a new guise: there, an unreachable hub was reported as a bad recovery code; here, a valid code for retained history is reported as a bad code. The message is not merely unhelpful — it is wrong, and it terminates the customer's attempt.


4. Every step taken off the customer's path

Stated plainly, because (b)'s "yes" is worth exactly as much as this list is short.

Step Off-path how Cost to the walk's fidelity
Read identity_blob out of the hub's SQLite No API serves a retained row — this is (c) None to the finding; it is the finding
age -d via a hand-written pty wrapper on DooPlex The product's unwrap is reachable only for the CURRENT blob None — same age -p primitive the agent uses
restic invoked directly with the recovered password The controller's recovery path compares, it never installs None — same restic, same repo, same credentials
Recovery code taken from the operator's credentials file A customer would read it from their card None — a customer plausibly has it
Planted dataset pushed with restic backup directly The dashboard button was unreachable (see F6) The unit-snapshot chain was not exercised; the store is identical
--skip-provision + --force on Phase A installs Saved a guest restore per cycle None — dnsmasq handling is independent of provisioning

5. Phase A — our removal undoes what our installation did

Venue drill-r50, reverted to virgin before each clean run.

5.1 The chain re-established, not assumed

Virgin baseline: dnsmasq not-installed, unit not-found, :53 free. After an appliance install the agent installed dnsmasq (internal/lanresolver, not the script) and the Felhom snippet constrained it to 10.0.2.15:53 + 127.0.0.1:53.

5.2 The wrong outcome, demonstrated first

Pre-fix uninstall (v1.26.0) — snippet removed, README only left, and:

enabled: enabled · active: active
udp UNCONN 0.0.0.0:53 · udp UNCONN [::]:53 · tcp LISTEN 0.0.0.0:53 · tcp LISTEN [::]:53

Pre-fix byo install, exit 1 — the refusal the publication had been waiting for since Sunday:

[ERROR]   a resolver is already bound to :53 on this host:
      udp UNCONN 0 0 0.0.0.0:53 … users:(("dnsmasq",pid=4934,fd=4)) …
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
  Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
  (Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
[ERROR] PRE-FLIGHT FAIL (exit 1) — fix the finding above and re-run

Nothing in it suggests the resolver is Felhom's own leftover.

5.3 The pass

Fixed path (v1.27.0) from virgin: preflight recorded dnsmasq: not present before Felhom; uninstall logged dnsmasq was installed by Felhom (recorded at install) — stopping + disabling it; result enabled: disabled · active: inactive · :53 FREE; and the second install:

[INFO]   host DNS (:53): free
[OK] pre-flight passed
[OK] PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed

5.4 Both sides of the ownership record

  • no (Felhom's): stop + disable. Quoted above.
  • yes (the owner's): dnsmasq pre-dates Felhom (recorded at install) — leaving it running, restarting only → left enabled/active. Felhom does not disable a resolver it did not install. Correct.
  • No record (the field case): every box installed before this change. Exercised by running the fixed byo preflight on a box whose state file the uninstall had already deleted — the improved refusal fired (§5.5).

5.5 The improved refusal, judged

  THIS LOOKS LIKE OURS. A previous Felhom install leaves the dnsmasq PACKAGE installed and its unit
  enabled (only our config snippet is removed), and unconstrained it binds 0.0.0.0:53 — which is what
  this gate is seeing. If this host had no dnsmasq before Felhom, clear it with:
      systemctl disable --now dnsmasq
  Then re-run this installer. If dnsmasq is YOURS, leave it and use one of the two routes above.

It names the leftover, gives the exact command, and does not overclaim — it says looks like, and leaves the owner's case intact. Good. One weakness: it asks the reader to answer "did this host have dnsmasq before Felhom?" — a question the installer itself now records the answer to, but in a state file the uninstall has already deleted. English throughout, consistent with the rest of the installer (this surface is operator/tester-facing, not the customer dashboard).

5.6 The finding this phase was not looking for — the fix works exactly once per machine

Proven, not deduced. Cycle 2 on the same box:

install : [INFO]   dnsmasq: already installed BEFORE Felhom — recorded; uninstall will not touch it
uninstall: [INFO]  dnsmasq pre-dates Felhom (recorded at install) — leaving it running, restarting only
after   : enabled: enabled · active: active · 0.0.0.0:53
cycle-3 byo preflight: PRE-FLIGHT FAIL (exit 1) — "a resolver is already bound to :53"

Why: the uninstall disables the unit but never purges the package (deliberately — purging on a host we may not own is the wrong blast radius). So on every later install dpkg-query says installed, preflight records yes, the agent re-enables it, and the uninstall then only restarts it. The wall returns on the second reinstall. → F2


6. Phase B — an install takes the image that was approved

Venue drill-r50, reverted to virgin. A genuinely stale golden (golden-0.98.3.tar.zst, controller 0.98.3, marker verified before use) was placed as local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst — newest by filename, which is exactly what discovery sorts on.

6.1 The fault, observed

Pre-fix (v1.25.0, the published tag) reported both facts and used the wrong one:

[INFO]   manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0
[STEP] 7/8 golden archive
[SKIP]   using local golden: local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[OK] Day-0 provision SUCCESS — vmid=120 … golden=local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst

No digest, no version compare, no warning. The box came up on controller 0.98.3 — confirmed twice, from the running container and from the in-guest marker:

gitea.dooplex.hu/admin/felhom-controller:0.98.3     (docker ps)
gitea.dooplex.hu/admin/felhom-controller:0.98.3     (/etc/felhom-controller-image)

The sharp end (§6 item 4). Vouched golden 0.213.0; floor 0.213.0; the box landed on 0.98.3 — below the floor, and below the version carrying the off-site recovery screen. A box born this way cannot run the ceremony its own data depends on, and is born below the update floor. The row was right.

6.2 Both fixed shapes

Explicitly named archive → refuses (exit 1):

[STEP] 7/8 golden archive
[ERROR] refusing the golden you named (local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst):
        it is controller 0.98.3, but the vouched golden is 0.213.0.
  The vouched golden is 0.213.0. Either pass the archive that matches it,
  or re-run with --force-gitea-golden to fetch the vouched one from Gitea.

Auto-discovered archive → warns and re-fetches: see §6.3.

No local goldens were deleted and teardown's keep-behaviour was not changed.

6.3 Auto-discovery result

[STEP] 7/8 golden archive
[WARN]   ignoring the local golden local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
         — it is controller 0.98.3, but the vouched golden is 0.213.0
[WARN]   fetching the vouched golden instead (this is what the manifest is for)
[INFO]   fetching golden v0.213.0 from Gitea → …/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst
[OK]   verified sha256 77429bfc7e39ffc8… matches the hub manifest

Landed on 0.213.0, confirmed from both the running container and the in-guest marker.

Observation, not a fault: the re-fetched archive is named by its real timestamp (17_31_32), which sorts below the planted stale one (23_59_59). So the stale archive stays the newest-by-filename and is re-discovered — and correctly re-rejected — on every subsequent install. The fix is not defeated; it just never displaces the bad archive, so the warning recurs forever.


7. Findings, ranked by what they cost the person in front of you

# Rank Finding
F1 1 — stops the visit The retained key has no product route, and the product tells a customer holding the correct old recovery code that it "did not open the sealed bundle". Recoverable data, valid code, and a dead end with no next step. (§3.7)
F2 2 — costs time The R-300 cleanup fires only on a machine that never had Felhom. The package is never purged, so every later install records dnsmasq_preexisting=yes and the leftover returns on the second reinstall. (§5.6)
F3 3 — misleads --preflight-only claims "no state written" (banner and the flag's own comment) but writes /var/lib/felhom-install/state.json_state_put short-circuits on DRY_RUN only. Worse, on a box carrying a Felhom leftover it records dnsmasq_preexisting=yes before the real install, baking in the wrong ownership answer.
F4 3 — misleads The pre-fix uninstall's closing note called dnsmasq a "system package". On these boxes Felhom installed it. (Already corrected in v1.27.0's wording; confirmed live.)
F5 4 — cosmetic The shape-2 refusal states the vouched version twice in consecutive sentences.
F6 2 — costs time The stored PASSWORD credential no longer opens demo-felhom's dashboard (Hibás jelszó). Not the known quoting trap — quotes were stripped and the value is unquoted. The dashboard was unreachable headlessly for this drill.
F7 — operator decision demo-felhom has a live abandon countdown: abandon_started_at 2026-08-10T08:06:31Z, abandon_at 2026-08-24T08:06:31Z, abandon_repo_path /home/felhom-repo.orphaned-20260810. Not started by this drill. See §9.
F8 4 — cosmetic --uninstall requires /dev/tty for the typed vmid confirmation when a guest exists, and --force does not bypass it (deliberate, and correct for an irreversible destroy) — but it means teardown cannot be scripted without a pty. Worth one line in the runbook.

A drill that finds nothing is a suspicious result. This one found eight, and the instrument was demonstrably working: the comparator was shown convicting a one-byte change before it was trusted (§3.2), the negative control on the old store failed closed before the positive one succeeded (§3.6), and both installer faults were reproduced from a virgin machine before either fix was watched.


8. Publication

Both Phase A's and Phase B's faults were observed live. The reward is earned.

  • Phase A fault: observed (§5.2) and fixed path quoted (§5.3).
  • Phase B fault: observed (§6.1) — the box came up on 0.98.3 against a vouched 0.213.0.

installer-v1.27.0 cut, covering both fixes (R-297 stale-golden comparison, R-300 removal leftover — the only two commits touching the installer since installer-v1.25.0).

And publishing is not what the day-0 runbook says it is. §C.1 states the URL "is always the current main script … pushing scripts/felhom-host-install.sh publishes it." That has been false since R-110: manifests/webpage.yaml runs two git-syncs, and /scripts/ follows the installer tag, not main. Measured before publishing — https://felhom.eu/scripts/felhom-host-install.sh served SCRIPT_VERSION="1.25.0" while main carried 1.27.0, three and a half hours after the push. So the fixes were genuinely unpublished, as the brief assumed — but for a reason the runbook denies. Publication here therefore meant: cut the tag and bump both --refs (sidecar at line 327 and init container at line 372). → R-309


9. The countdown — an operator decision, deliberately not taken

demo-felhom carries a running abandon countdown, started 2026-08-10, firing 2026-08-24, for /home/felhom-repo.orphaned-20260810. This drill did not start, shorten or trigger it, per the fence.

But §9 of the brief requires an end state of "no abandon countdown anywhere", and one exists. The two ways to satisfy that are not equivalent and the choice is not mine:

  1. Cancel it — the orphaned repository is kept indefinitely (storage cost, no data risk).
  2. Let it run — on 2026-08-24 the orphaned repository is deleted, irreversibly.

Doing nothing selects option 2 by default. Flagged, not resolved.


10. Teardown — four layers

Layer State
The machine demo-felhom: planted dataset and working directory removed; eight secret-bearing files shredded, not unlinked; repo_password restored to P1 (c60c8bc737a6b7c6…); escrow re-sealed and uploaded; offsite repo reachable (restic snapshots exit 0, 4 snapshots). The agent's escrow-stage/ directory is empty — the ceremony wiped the staged secret (positive observable). demo-hp: not touched at any point. drill-r50: see §11.
The host felhom-pve: no drill artefacts left outside guest 9201. drill-r50 host: see §11.
The hub Two new retained rows (id 11 = the P1 blob, id 12 = the P2 blob), both identity_blob 572 bytes — deliberately kept as the fixture that proves the retention works; they are append-only, tiny, and exactly what the design intends. drill-r50-0a4f9a host record: pre-existing since 2026-07-25, re-used rather than duplicated — no new hub-side sprawl. customer_configs gained nothing.
The off-site side Only demo-felhom's own repository path was touched. No prune, no forget, no delete, no rename — anywhere, at any point. One snapshot was added: 6ea85413, 66 KiB, tagged drill-retained-key-20260812. Deliberate residue — removing it would have meant a forget on the endpoint, which the fence forbids in spirit; the operator can remove it by ID if desired. Nothing outside /home/felhom-repo was read or written; peti-felhom was never contacted.

How I know the off-site side is untouched outside this machine's path: every restic invocation in this drill went through one wrapper pinned to sftp:u629488-sub1@u629488-sub1.your-storagebox.de:/home/felhom-repo — a per-customer sub-account whose credentials reach nothing else — and the only mutating verb used was backup.


11. Close of run

11.1 Wall clocks (CEST)

Phase Span Elapsed
§7.0 probe (read-only, source + hub) 16:42 → 16:52 ~9 min
Phase A — removal leftover 16:50 → 17:12 ~22 min (4 installs, 4 uninstalls, 3 preflights)
Phase B — stale golden 17:13 → 17:34 ~21 min (2 full provisions)
Phase C — retained key 17:14 → 17:23 ~25 min (overlapped Phase B; different machine)
Record, register, publication, teardown 17:35 → 17:55 ~20 min
Total 16:42 → 17:55 ≈ 1 h 13 min

Phase C ran concurrently with Phase B deliberately — different machines, no shared state. The 45 hour envelope was not needed; nothing was hurried and nothing was dropped for time.

11.2 Register

Ceiling moved R-303 → R-310. Opened: R-304 (retained key has no product route — rank 1), R-305 (R-300 fix is single-shot per machine — rank 2), R-306 (--preflight-only writes state it says it does not — rank 3), R-307 (live abandon countdown, operator decision), R-308 (stored controller password stale for demo-felhom — rank 2), R-309 (day-0 runbook's publication claim false since R-110 — rank 3), R-310 (two installer edges — rank 4). Closed: R-297 and R-300, both observed live and published.

11.3 Teardown

drill-r50: reverted to snapshot virgin and left powered offqemu-img snapshot -a virgin, qemu exited first, qemu.pid removed. Its three in-VM goldens went with the revert (the VM's normal lifecycle); the source archive on DooPlex, /mnt/5_hdd/felhom.eu/drill/golden-0.98.3.tar.zst, is untouched (mtime still 2026-07-03). Full four-layer detail in §10.

11.4 What was dropped, named plainly

Nothing was dropped. Phase A, Phase B and Phase C all ran in full, in the required order, and the publication was earned rather than assumed.

One thing was not done and it was not in scope: the planted dataset reached the off-site store via restic directly rather than via the dashboard's own button, because the stored controller password no longer opens demo-felhom (R-308). The store, the credentials and the restic binary were the product's own; what went unexercised is the app-backup → recovery-unit → offsite chain, which is not what this drill was measuring.

11.5 Evidence, including a gap

Logs at audits/evidence-drill-retained-key-2026-08-12/: B1-install-preB.log, B2-uninstall.log, B3-explicit-stale.log, B4-autodiscover-stale.log, Z-final-state.txt.

The Phase A logs did not survive. They lived on the drill VM's disk and were destroyed by the revert to virgin between Phase A and Phase B — I copied evidence out before the final teardown but not before the intermediate one. Every Phase A quotation in §5 is verbatim from the live run as read at the time, but the raw files are gone and cannot be re-read. This is the GL-1 discipline the golden-bake runbook already states ("scp the log OUT first"), applied to the last revert and not the middle one. Recorded rather than glossed; the fix is procedural, not a code change.

11.6 Observations — noticed, not acted on

  • The stale archive is never displaced. A re-fetched golden carries its real timestamp, which sorts below a planted 23_59_59, so the bad archive stays newest-by-filename and is re-rejected on every future install. Correct, but the warning recurs forever and nothing removes the cause.
  • The ceremony wipes its staged secret. After every run /var/lib/felhom-agent/escrow-stage/ was empty — checked as a positive observable, not inferred from an absent log line.
  • restic_pw_sealed: true in the ceremony status is a genuinely useful positive observable — it distinguishes "a ceremony ran" from "a ceremony sealed the thing that matters".
  • demo-felhom already had an orphaned store (/home/felhom-repo.orphaned-20260810) from the 10 August rebuild, and its password never changed — the current escrow's restic_pw_sha256 matched the box's live repo_password exactly on arrival. So that orphaning was a rename, not a key change.
  • R-241's mint guard makes password-changing supersessions rare by design. A rebuilt guest no longer mints a fresh repository password while the hub holds a sealed package, which is why the state Phase C needed had to be produced deliberately rather than found.
  • The Hungarian locale warning (perl: LC_CTYPE = "UTF-8") prefixes almost every pct exec on felhom-pve. Harmless, but it is noise on every operator transcript.