admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

DRILL: the retained key works, and the customer cannot reach it
gates / gates (push) Successful in 23s
Details

Browse Source 
Three verdicts, kept separate because collapsing them is how this assumption
survived a week.

(a) The material IS retained. host_escrow_superseded id 11 is the first retained
row in fleet history to carry identity_blob (572 B), byte-identical to the
pre-supersession row (sha256 a10032341c8584ed...).

(b) The retained material DOES open the old store. Unsealed with the old recovery
code it yielded a password byte-identical to the pre-change one, and restored
three planted files byte-identical from a store the box itself could no longer
open - including a Hungarian accented filename verified as raw bytes. Negative
control ran first and failed closed.

(c) The customer has NO route, and is misinformed. ListSupersededEscrow has zero
production callers; the recovery path selects FROM host_escrow. Asked with the
code that had just worked by hand, the product answered "the recovery code did
not open the sealed bundle". A valid code for retained history is reported as a
bad code - the R-224 class again. R-304, rank 1.

Both installer faults were watched happening first, so installer-v1.27.0 is now
published (tag + both webpage.yaml refs). Pre-fix: the box came up on controller
0.98.3 against a vouched 0.213.0, below the floor and below the version carrying
the recovery screen; and our own uninstall left dnsmasq on 0.0.0.0:53 so our own
next install refused. R-297 and R-300 CLOSED.

Also filed R-305 (the dnsmasq fix fires once per machine - the leftover returns
on the second reinstall, proven), R-306 (--preflight-only writes state it says it
does not), R-307 (a live abandon countdown on demo-felhom, firing 2026-08-24 -
operator decision), R-308 (stored controller password stale), R-309 (the day-0
runbook's publication claim has been false since R-110), R-310 (two edges).

Ceiling R-303 -> R-310. Capability map moved: the retention claim is now marked
operator-only. Phase A logs did not survive the intermediate revert; recorded.
This commit is contained in:
authentik Default Admin
2026-08-12 17:41:56 +02:00
parent fbe1155fbb
commit 1d5f2b8bb6
12 changed files with 2445 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/DRILL-retained-key-2026-08-12.md
+514
View File
@@ -0,0 +1,514 @@
# DRILL — the retained key, and the two fixes nobody had watched work
**Date:** 2026-08-12 · **Class:** drill (unattended, destructive on Tier 0; spike for Phase C's first step)
**Venues:** `drill-r50` (nested PVE on DooPlex) · `demo-felhom` (guest 9201 on felhom-pve) — both Tier 0
**Baselines re-confirmed live on arrival, not assumed:** controller **0.213.0** on both demo boxes
(`pct exec 9201 -- docker ps`), agent **0.128.0** (hub `hosts`), hub **0.102.0** (deployment image),
register ceiling **R-303** (grep), `felhom.eu` clean at `fbe1155f` and level with `origin/main`.
---
## 1. The three verdicts, first, in plain language
> **(a) Is the material retained? — YES.** Proven for the first time in the fleet's history.
>
> **(b) Does the retained material, once unsealed, yield a password that actually opens the old
> store? — YES.** Three planted files came back byte-identical from a store the box itself could no
> longer open, including the Hungarian accented filename.
>
> **(c) Does the customer have a route to (b) through the product? — NO. And the failure is worse
> than absence:** the customer's old recovery code — the one that demonstrably opens their old
> backups — is rejected by the product with *"the recovery code did not open the sealed bundle"*.
> They are told their correct code is wrong, and given no next step.
**The headline the operator asked for.** The brief said to be ready for the answer to be no, and our
own records predicted the retention would be *"a box we fill and cannot open"*. That prediction was
**half right, and the wrong half was the one nobody checked.** The box can be opened — the crypto,
the retention and the copy are all sound. What does not exist is the door: no code path reads a
retained row, so the only way in is an operator with SQLite, `age` and a shell. Everything I did in
§4 a customer cannot do.
That distinction matters for the three pieces of work resting on this:
| Rests on | Verdict |
|---|---|
| The census answer ("nobody else is exposed") | **Stands.** It was a statement about retention, and retention works. |
| The countdown banner's promise (old backups stay recoverable) | **True in substance, false in practice.** The bytes are recoverable; the customer cannot recover them. The promise is made to someone who has no way to act on it. |
| The capability map's recovery claims | **Must move.** Any claim of the form "the customer can recover the old history with their recovery code" is false today. |
---
## 2. §7.0 — the probe, run first and read-only
Run before anything was installed or planted, exactly as the order required.
**Q1 — does any host today hold a superseded record with key material present? NO. Zero, fleet-wide.**
```
host_id kfp16 blob_len identity_blob superseded_at
demo-hp-bb76ea 3f:4f:65:c0:d8:f 383 NULL 2026-08-04 07:15:36
demo-felhom-8363b5 7e:a6:af:f7:ea:6 383 NULL 2026-08-04 07:20:08
```
Both are the pre-R-198 rows written hours before the fix. `SELECT COUNT(*) … WHERE identity_blob IS
NOT NULL` returned **0**. The retention as fixed had never once been exercised.
*Instrument:* hub SQLite copied WITH `hub.db-wal` + `-shm` (the standing gotcha). Freshness proved by
a **positive observable** — newest `host_reports` row 61 s old — not by "the query returned no error".
`PRAGMA integrity_check` = ok.
**Q2 — what act produces a retaining supersession? Two, both read from source:**
- `hub/internal/store/store.go:2807` — `SaveHostEscrow` demotes when the newly sealed
`restic_pw_sha256` **differs** from the stored one.
- `hub/internal/store/store.go:2636` — `DeleteHost` demotes the current row unconditionally.
**Q3 — is there any code path that USES a retained record to recover a password? NO.**
The product's recovery route is real and wired (R-199) but reaches the **current** row only:
```
controller → agent POST /escrow/recover-offsite-password
→ escrow.OffsiteKeyRecoverer.Fetch felhom-agent/cmd/felhom-agent/main.go:1757
→ hubClient.FetchIdentityEscrow
→ hub GET identity-escrow handler hub/internal/api/handler.go:1325
→ store.GetHostDRBundle hub/internal/store/store.go:3152
SELECT blob, identity_blob, directive_json FROM host_escrow WHERE host_id = ?
```
`ListSupersededEscrow` (`store.go:2841`) is the **sole** reader of a retained `identity_blob`. It has
**zero production callers** — five call sites, all in `_test.go`. `CountSupersededEscrow` is used
twice and only to render a *count* (`api/handler.go:1209`, `web/hosts.go:572`).
**Feasibility verdict: producible.** Phase C proceeded.
---
## 3. Phase C — producing the state
### 3.1 The planted dataset
Planted in `/mnt/sys_drive/felhom-data/userdata/drill-20260812` on `demo-felhom`.
| file | sha256 | bytes |
|---|---|---|
| `payload.bin` | `6863824a67bf6cb636defca3cbddc1d58092bcdda3af939224dfef4663c15359` | 65536 |
| `plain-ascii.txt` | `e294e0ee5b8fd131dc0fa71f5cf1253dc39473241aa090be36404a568285c83a` | 56 |
| `árvíztűrő-tükörfúrógép.txt` | `a1957a87104c63be22b34e4f937e700edcdc925da5984bfcca407df4989f7190` | 37 |
**The accented filename never crossed a shell as text** (rule 6). It was built from explicit bytes and
verified as hex, both as intended and as listed on disk — identical:
```
c3 a1 72 76 c3 ad 7a 74 c5 b1 72 c5 91 2d 74 c3 bc 6b c3 b6 72 66 c3 ba 72 c3 b3 67 c3 a9 70 2e 74 78 74
```
### 3.2 Positive control on the comparator — done BEFORE it was relied on
One byte of a copy of `payload.bin` flipped at offset 40000:
```
unaltered f5de179b5ca941645819a879187db953db5c973d4c1a07b2a3ba96d3d79bfb6c
altered 515a65d96326bd847a87ea35c9fb5e6afebc9499ec0c1cd80ca3531b22cb2324
payload.bin: FAILED · sha256sum: WARNING: 1 computed checksum did NOT match
```
The comparator convicts. The copy was discarded and the original re-hashed unchanged.
### 3.3 The store, written under P1
`restic backup` into demo-felhom's **own** repository path
(`sftp:u629488-sub1@…your-storagebox.de:/home/felhom-repo`) → snapshot **`6ea85413`**, confirmed by
identity and by **listing its contents**, not by the success line:
```
6ea85413 2026-08-12 15:15:33 demo-felhom felhom-offbox,drill-retained-key-20260812
…/drill-20260812/payload.bin
…/drill-20260812/plain-ascii.txt
…/drill-20260812/M-CM-!rvM-CM--ztM-EM-1rM-EM-^Q-tM-CM-<kM-CM-6rfM-CM-:rM-CM-3gM-CM-)p.txt
```
**Key fingerprint of the store's password (P1): sha256 `c60c8bc737a6b7c6…`** — value never recorded.
### 3.4 The supersession, by the product's own ceremony
The repository password was replaced (P1 → P2, `d4aac4f1c0ef9ac0…`), making `/home/felhom-repo` the
**old** store, then the ceremony was driven through the same local-API calls the dashboard wizard
makes: `POST /escrow/stage-secret` → `POST /escrow/ceremony` → `…/status` → `…/claim`.
```
phase=done restic_pw_sealed=true uploaded=true
```
### 3.5 (a) — the material IS retained
The hub grew its first ever retained row carrying key material:
```
id host_id identity_blob restic_pw_sha256 superseded_at
11 demo-felhom-8363b5 572 c60c8bc737a6b7c6… 2026-08-12 15:18:55
```
Confirmed **by identity and byte length**, not by presence:
```
a10032341c8584edfd87ad941a50d9fd6f34c23a4802f55885cb1d4cab086bd3 current-identity.age (pre-supersession)
a10032341c8584edfd87ad941a50d9fd6f34c23a4802f55885cb1d4cab086bd3 retained-identity.age (retained row 11)
572 bytes both — IDENTICAL
```
### 3.6 (b) — the retained key DOES open the old store
**Negative control first** — the box's current password against the old store:
```
restic exit code: 1
Fatal: wrong password or no key found
```
**Then the retained key.** Retained blob → `age -d` with the old recovery code → bundle →
`restic_repo_password`, sha256 `c60c8bc737a6b7c6…`, **byte-identical to the P1 recorded before the
supersession**:
```
restic snapshots → exit 0, snapshot 6ea85413 listed
restic restore → exit 0
restored accented filename bytes: c3 a1 72 76 c3 ad 7a 74 c5 b1 72 c5 91 2d … (identical)
payload.bin: OK · plain-ascii.txt: OK · árvíztűrő-tükörfúrógép.txt: OK (compare exit code 0)
```
**All three files byte-identical, from a store the box itself could not open.**
### 3.7 (c) — the customer has no route, and is actively misinformed
The product's own recovery endpoint was asked for the old password, using the same old recovery code
that had just worked by hand:
```
OLD code (opens the retained row by hand → P1) REFUSED: "the recovery code did not open the
sealed bundle — nothing was written"
NEW code (today's ceremony) RETURNED a password, sha d4aac4f1c0ef9ac0 = P2
```
P2 does not open the old store. **The one code that works is reported as the code that does not.**
This is the R-224 defect class in a new guise: there, an unreachable hub was reported as a bad
recovery code; here, a *valid code for retained history* is reported as a bad code. The message is
not merely unhelpful — it is wrong, and it terminates the customer's attempt.
---
## 4. Every step taken off the customer's path
Stated plainly, because (b)'s "yes" is worth exactly as much as this list is short.
| Step | Off-path how | Cost to the walk's fidelity |
|---|---|---|
| Read `identity_blob` out of the hub's SQLite | No API serves a retained row — **this is (c)** | None to the finding; it *is* the finding |
| `age -d` via a hand-written pty wrapper on DooPlex | The product's unwrap is reachable only for the CURRENT blob | None — same `age -p` primitive the agent uses |
| `restic` invoked directly with the recovered password | The controller's recovery path **compares**, it never installs | None — same restic, same repo, same credentials |
| Recovery code taken from the operator's credentials file | A customer would read it from their card | None — a customer plausibly has it |
| Planted dataset pushed with `restic backup` directly | The dashboard button was unreachable (see F6) | The unit-snapshot chain was not exercised; the *store* is identical |
| `--skip-provision` + `--force` on Phase A installs | Saved a guest restore per cycle | None — dnsmasq handling is independent of provisioning |
---
## 5. Phase A — our removal undoes what our installation did
**Venue `drill-r50`, reverted to `virgin` before each clean run.**
### 5.1 The chain re-established, not assumed
Virgin baseline: `dnsmasq` **not-installed**, unit **not-found**, **`:53` free**. After an appliance
install the **agent** installed dnsmasq (`internal/lanresolver`, not the script) and the Felhom
snippet **constrained** it to `10.0.2.15:53` + `127.0.0.1:53`.
### 5.2 The wrong outcome, demonstrated first
Pre-fix uninstall (v1.26.0) — snippet removed, `README` only left, and:
```
enabled: enabled · active: active
udp UNCONN 0.0.0.0:53 · udp UNCONN [::]:53 · tcp LISTEN 0.0.0.0:53 · tcp LISTEN [::]:53
```
Pre-fix byo install, **exit 1** — the refusal the publication had been waiting for since Sunday:
```
[ERROR] a resolver is already bound to :53 on this host:
udp UNCONN 0 0 0.0.0.0:53 … users:(("dnsmasq",pid=4934,fd=4)) …
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
(Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
[ERROR] PRE-FLIGHT FAIL (exit 1) — fix the finding above and re-run
```
Nothing in it suggests the resolver is Felhom's own leftover.
### 5.3 The pass
Fixed path (v1.27.0) from virgin: preflight recorded `dnsmasq: not present before Felhom`; uninstall
logged `dnsmasq was installed by Felhom (recorded at install) — stopping + disabling it`; result
`enabled: disabled · active: inactive · :53 FREE`; and the second install:
```
[INFO] host DNS (:53): free
[OK] pre-flight passed
[OK] PRE-FLIGHT PASS (mode=byo) — no state written, no install step executed
```
### 5.4 Both sides of the ownership record
- **`no` (Felhom's):** stop + disable. Quoted above.
- **`yes` (the owner's):** `dnsmasq pre-dates Felhom (recorded at install) — leaving it running,
restarting only` → left `enabled`/`active`. **Felhom does not disable a resolver it did not
install.** Correct.
- **No record (the field case):** every box installed before this change. Exercised by running the
fixed byo preflight on a box whose state file the uninstall had already deleted — the improved
refusal fired (§5.5).
### 5.5 The improved refusal, judged
```
THIS LOOKS LIKE OURS. A previous Felhom install leaves the dnsmasq PACKAGE installed and its unit
enabled (only our config snippet is removed), and unconstrained it binds 0.0.0.0:53 — which is what
this gate is seeing. If this host had no dnsmasq before Felhom, clear it with:
systemctl disable --now dnsmasq
Then re-run this installer. If dnsmasq is YOURS, leave it and use one of the two routes above.
```
It names the leftover, gives the exact command, and does not overclaim — it says *looks like*, and
leaves the owner's case intact. **Good.** One weakness: it asks the reader to answer *"did this host
have dnsmasq before Felhom?"* — a question the installer itself now records the answer to, but in a
state file the uninstall has already deleted. English throughout, consistent with the rest of the
installer (this surface is operator/tester-facing, not the customer dashboard).
### 5.6 The finding this phase was not looking for — the fix works exactly once per machine
Proven, not deduced. Cycle 2 on the same box:
```
install : [INFO] dnsmasq: already installed BEFORE Felhom — recorded; uninstall will not touch it
uninstall: [INFO] dnsmasq pre-dates Felhom (recorded at install) — leaving it running, restarting only
after : enabled: enabled · active: active · 0.0.0.0:53
cycle-3 byo preflight: PRE-FLIGHT FAIL (exit 1) — "a resolver is already bound to :53"
```
**Why:** the uninstall disables the unit but never **purges** the package (deliberately — purging on
a host we may not own is the wrong blast radius). So on every later install `dpkg-query` says
installed, preflight records `yes`, the agent re-enables it, and the uninstall then only restarts it.
The wall returns on the second reinstall. → **F2**
---
## 6. Phase B — an install takes the image that was approved
**Venue `drill-r50`, reverted to `virgin`.** A genuinely stale golden (`golden-0.98.3.tar.zst`,
controller **0.98.3**, marker verified before use) was placed as
`local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst` — newest by filename, which is exactly what
discovery sorts on.
### 6.1 The fault, observed
Pre-fix (v1.25.0, the published tag) reported both facts and used the wrong one:
```
[INFO] manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.213.0
[STEP] 7/8 golden archive
[SKIP] using local golden: local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
[OK] Day-0 provision SUCCESS — vmid=120 … golden=local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
```
No digest, no version compare, no warning. **The box came up on controller 0.98.3** — confirmed twice,
from the running container and from the in-guest marker:
```
gitea.dooplex.hu/admin/felhom-controller:0.98.3 (docker ps)
gitea.dooplex.hu/admin/felhom-controller:0.98.3 (/etc/felhom-controller-image)
```
**The sharp end (§6 item 4).** Vouched golden **0.213.0**; floor **0.213.0**; the box landed on
**0.98.3** — below the floor, and below the version carrying the off-site recovery screen. A box born
this way cannot run the ceremony its own data depends on, and is born below the update floor. **The
row was right.**
### 6.2 Both fixed shapes
**Explicitly named archive → refuses** (exit 1):
```
[STEP] 7/8 golden archive
[ERROR] refusing the golden you named (local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst):
it is controller 0.98.3, but the vouched golden is 0.213.0.
The vouched golden is 0.213.0. Either pass the archive that matches it,
or re-run with --force-gitea-golden to fetch the vouched one from Gitea.
```
**Auto-discovered archive → warns and re-fetches:** see §6.3.
No local goldens were deleted and teardown's keep-behaviour was not changed.
### 6.3 Auto-discovery result
```
[STEP] 7/8 golden archive
[WARN] ignoring the local golden local:backup/vzdump-lxc-9100-2026_08_12-23_59_59.tar.zst
— it is controller 0.98.3, but the vouched golden is 0.213.0
[WARN] fetching the vouched golden instead (this is what the manifest is for)
[INFO] fetching golden v0.213.0 from Gitea → …/vzdump-lxc-9100-2026_08_12-17_31_32.tar.zst
[OK] verified sha256 77429bfc7e39ffc8… matches the hub manifest
```
**Landed on 0.213.0**, confirmed from both the running container and the in-guest marker.
**Observation, not a fault:** the re-fetched archive is named by its real timestamp
(`17_31_32`), which sorts *below* the planted stale one (`23_59_59`). So the stale archive stays the
newest-by-filename and is re-discovered — and correctly re-rejected — on every subsequent install.
The fix is not defeated; it just never displaces the bad archive, so the warning recurs forever.
---
## 7. Findings, ranked by what they cost the person in front of you
| # | Rank | Finding |
|---|---|---|
| **F1** | **1 — stops the visit** | The retained key has **no product route**, and the product tells a customer holding the **correct** old recovery code that it *"did not open the sealed bundle"*. Recoverable data, valid code, and a dead end with no next step. (§3.7) |
| **F2** | 2 — costs time | The R-300 cleanup fires **only on a machine that never had Felhom**. The package is never purged, so every later install records `dnsmasq_preexisting=yes` and the leftover returns on the second reinstall. (§5.6) |
| **F3** | 3 — misleads | `--preflight-only` claims *"no state written"* (banner and the flag's own comment) but **writes `/var/lib/felhom-install/state.json`** — `_state_put` short-circuits on `DRY_RUN` only. Worse, on a box carrying a Felhom leftover it records `dnsmasq_preexisting=yes` **before** the real install, baking in the wrong ownership answer. |
| **F4** | 3 — misleads | The pre-fix uninstall's closing note called dnsmasq a *"system package"*. On these boxes Felhom installed it. (Already corrected in v1.27.0's wording; confirmed live.) |
| **F5** | 4 — cosmetic | The shape-2 refusal states the vouched version twice in consecutive sentences. |
| **F6** | 2 — costs time | The stored `PASSWORD` credential no longer opens **demo-felhom**'s dashboard (`Hibás jelszó`). Not the known quoting trap — quotes were stripped and the value is unquoted. The dashboard was unreachable headlessly for this drill. |
| **F7** | — operator decision | **`demo-felhom` has a live abandon countdown**: `abandon_started_at 2026-08-10T08:06:31Z`, `abandon_at 2026-08-24T08:06:31Z`, `abandon_repo_path /home/felhom-repo.orphaned-20260810`. Not started by this drill. See §9. |
| **F8** | 4 — cosmetic | `--uninstall` requires `/dev/tty` for the typed vmid confirmation when a guest exists, and `--force` does **not** bypass it (deliberate, and correct for an irreversible destroy) — but it means teardown cannot be scripted without a pty. Worth one line in the runbook. |
**A drill that finds nothing is a suspicious result.** This one found eight, and the instrument was
demonstrably working: the comparator was shown convicting a one-byte change before it was trusted
(§3.2), the negative control on the old store failed closed before the positive one succeeded (§3.6),
and both installer faults were reproduced from a virgin machine before either fix was watched.
---
## 8. Publication
Both Phase A's and Phase B's faults were observed live. **The reward is earned.**
- Phase A fault: observed (§5.2) and fixed path quoted (§5.3).
- Phase B fault: observed (§6.1) — the box came up on 0.98.3 against a vouched 0.213.0.
**`installer-v1.27.0` cut, covering both fixes** (R-297 stale-golden comparison, R-300 removal
leftover — the only two commits touching the installer since `installer-v1.25.0`).
**And publishing is not what the day-0 runbook says it is.** §C.1 states the URL *"is always the
current `main` script … pushing `scripts/felhom-host-install.sh` publishes it."* That has been false
since R-110: `manifests/webpage.yaml` runs two git-syncs, and `/scripts/` follows the **installer
tag**, not `main`. Measured before publishing — `https://felhom.eu/scripts/felhom-host-install.sh`
served `SCRIPT_VERSION="1.25.0"` while `main` carried `1.27.0`, three and a half hours after the push.
So the fixes were genuinely unpublished, as the brief assumed — but for a reason the runbook denies.
Publication here therefore meant: cut the tag **and** bump **both** `--ref`s (sidecar at line 327 and
init container at line 372). → **R-309**
---
## 9. The countdown — an operator decision, deliberately not taken
`demo-felhom` carries a **running abandon countdown**, started **2026-08-10**, firing **2026-08-24**,
for `/home/felhom-repo.orphaned-20260810`. **This drill did not start, shorten or trigger it**, per
the fence.
But §9 of the brief requires an end state of *"no abandon countdown anywhere"*, and one exists. The
two ways to satisfy that are not equivalent and the choice is not mine:
1. **Cancel it** — the orphaned repository is kept indefinitely (storage cost, no data risk).
2. **Let it run** — on 2026-08-24 the orphaned repository is **deleted, irreversibly**.
Doing nothing selects option 2 by default. **Flagged, not resolved.**
---
## 10. Teardown — four layers
| Layer | State |
|---|---|
| **The machine** | `demo-felhom`: planted dataset and working directory removed; eight secret-bearing files **shredded**, not unlinked; `repo_password` restored to P1 (`c60c8bc737a6b7c6…`); escrow re-sealed and uploaded; offsite repo reachable (`restic snapshots` exit 0, 4 snapshots). The agent's `escrow-stage/` directory is **empty** — the ceremony wiped the staged secret (positive observable). `demo-hp`: **not touched at any point**. `drill-r50`: see §11. |
| **The host** | `felhom-pve`: no drill artefacts left outside guest 9201. `drill-r50` host: see §11. |
| **The hub** | Two new retained rows (id 11 = the P1 blob, id 12 = the P2 blob), both `identity_blob` 572 bytes — **deliberately kept as the fixture that proves the retention works**; they are append-only, tiny, and exactly what the design intends. `drill-r50-0a4f9a` host record: pre-existing since 2026-07-25, **re-used rather than duplicated** — no new hub-side sprawl. `customer_configs` gained nothing. |
| **The off-site side** | Only `demo-felhom`'s own repository path was touched. **No `prune`, no `forget`, no delete, no rename** — anywhere, at any point. One snapshot was **added**: `6ea85413`, 66 KiB, tagged `drill-retained-key-20260812`. **Deliberate residue** — removing it would have meant a `forget` on the endpoint, which the fence forbids in spirit; the operator can remove it by ID if desired. Nothing outside `/home/felhom-repo` was read or written; `peti-felhom` was never contacted. |
**How I know the off-site side is untouched outside this machine's path:** every restic invocation in
this drill went through one wrapper pinned to
`sftp:u629488-sub1@u629488-sub1.your-storagebox.de:/home/felhom-repo` — a per-customer sub-account
whose credentials reach nothing else — and the only mutating verb used was `backup`.
---
## 11. Close of run
### 11.1 Wall clocks (CEST)
| Phase | Span | Elapsed |
|---|---|---|
| §7.0 probe (read-only, source + hub) | 16:42 → 16:52 | **~9 min** |
| Phase A — removal leftover | 16:50 → 17:12 | **~22 min** (4 installs, 4 uninstalls, 3 preflights) |
| Phase B — stale golden | 17:13 → 17:34 | **~21 min** (2 full provisions) |
| Phase C — retained key | 17:14 → 17:23 | **~25 min** (overlapped Phase B; different machine) |
| Record, register, publication, teardown | 17:35 → 17:55 | **~20 min** |
| **Total** | **16:42 → 17:55** | **≈ 1 h 13 min** |
Phase C ran concurrently with Phase B deliberately — different machines, no shared state. The
4–5 hour envelope was not needed; nothing was hurried and nothing was dropped for time.
### 11.2 Register
**Ceiling moved R-303 → R-310.** Opened: **R-304** (retained key has no product route — rank 1),
**R-305** (R-300 fix is single-shot per machine — rank 2), **R-306** (`--preflight-only` writes state
it says it does not — rank 3), **R-307** (live abandon countdown, operator decision), **R-308**
(stored controller password stale for demo-felhom — rank 2), **R-309** (day-0 runbook's publication
claim false since R-110 — rank 3), **R-310** (two installer edges — rank 4).
**Closed: R-297 and R-300**, both observed live and published.
### 11.3 Teardown
`drill-r50`: **reverted to snapshot `virgin` and left powered off** — `qemu-img snapshot -a virgin`,
qemu exited first, `qemu.pid` removed. Its three in-VM goldens went with the revert (the VM's normal
lifecycle); the **source** archive on DooPlex, `/mnt/5_hdd/felhom.eu/drill/golden-0.98.3.tar.zst`, is
untouched (mtime still 2026-07-03). Full four-layer detail in §10.
### 11.4 What was dropped, named plainly
**Nothing was dropped.** Phase A, Phase B and Phase C all ran in full, in the required order, and the
publication was earned rather than assumed.
**One thing was not done and it was not in scope:** the planted dataset reached the off-site store via
`restic` directly rather than via the dashboard's own button, because the stored controller password
no longer opens demo-felhom (**R-308**). The store, the credentials and the restic binary were the
product's own; what went unexercised is the *app-backup → recovery-unit → offsite* chain, which is not
what this drill was measuring.
### 11.5 Evidence, including a gap
Logs at `audits/evidence-drill-retained-key-2026-08-12/`: `B1-install-preB.log`, `B2-uninstall.log`,
`B3-explicit-stale.log`, `B4-autodiscover-stale.log`, `Z-final-state.txt`.
**The Phase A logs did not survive.** They lived on the drill VM's disk and were destroyed by the
revert to `virgin` between Phase A and Phase B — I copied evidence out before the *final* teardown but
not before the *intermediate* one. Every Phase A quotation in §5 is verbatim from the live run as read
at the time, but the raw files are gone and cannot be re-read. **This is the GL-1 discipline the golden-bake
runbook already states ("scp the log OUT first"), applied to the last revert and not the middle one.**
Recorded rather than glossed; the fix is procedural, not a code change.
### 11.6 Observations — noticed, not acted on
- **The stale archive is never displaced.** A re-fetched golden carries its real timestamp, which sorts
below a planted `23_59_59`, so the bad archive stays newest-by-filename and is re-rejected on every
future install. Correct, but the warning recurs forever and nothing removes the cause.
- **The ceremony wipes its staged secret.** After every run `/var/lib/felhom-agent/escrow-stage/` was
**empty** — checked as a positive observable, not inferred from an absent log line.
- **`restic_pw_sealed: true` in the ceremony status is a genuinely useful positive observable** — it
distinguishes "a ceremony ran" from "a ceremony sealed the thing that matters".
- **`demo-felhom` already had an orphaned store** (`/home/felhom-repo.orphaned-20260810`) from the
10 August rebuild, and its password never changed — the current escrow's `restic_pw_sha256` matched
the box's live `repo_password` exactly on arrival. So that orphaning was a rename, not a key change.
- **R-241's mint guard makes password-changing supersessions rare by design.** A rebuilt guest no longer
mints a fresh repository password while the hub holds a sealed package, which is why the state Phase C
needed had to be produced deliberately rather than found.
- The Hungarian locale warning (`perl: LC_CTYPE = "UTF-8"`) prefixes almost every `pct exec` on
felhom-pve. Harmless, but it is noise on every operator transcript.