felhom.eu/REPORT-rehearsal-2026-08-09.md

# REPORT — BYO reinstall rehearsal, 2026-08-09 (session report)

*Written as `REPORT-<topic>.md` rather than `REPORT.md` per the repo's parallel-session rule.*

## The answer to the runbook's question, first

**Yes, the data comes back byte for byte. No, not in one sitting, and not without a shell.**

All four planted files returned **BYTE-IDENTICAL** — including two Hungarian accented filenames
verified as *raw name bytes*, not as rendered text. The unlock took **21 s**, the restore **13.2 s**.

But the walk completed only because two hard stops were cleared by someone who could open a terminal
and read source. **R-273**: the install died at step 5/8 on an agent version that was published as a
package but never git-tagged — cleared by completing the release. **R-280**: the reinstalled machine
could not re-attach its own data drive through any dashboard route, while the restore page said
„**Ez két kattintás**" and pointed at an empty list — cleared by POSTing an internal path
(`/mnt/sys_drive`) that no household could produce.

Neither is a data-integrity problem. Both stop a household dead. **This is the same shape the R-201
walks kept finding: the data half passes, the journey half fails.**

## Venue

**demo-hp (t740)**, operator-approved at STOP 1. It won every fidelity criterion that distinguishes
the two demo boxes: three customer apps against one, a registered storage path against none, and a
Secure-Boot `shim` install — the customer shape — against demo-felhom's SB-off `mkimage` firmware
workaround. `drill-r50` (VM 300) was verified not at risk before proceeding: it is outside the
`felhom` pool, on `local-lvm`, and `--uninstall` removes no storage and no non-pool guest.

## Findings, ranked by what they cost the person in front of you

**1 — stops the visit**
- **R-273** · the vouched agent (0.128.0) had no git tag; every install died at 5/8. **CLOSED** — tag
  pushed on your instruction after an independent sha check; install then succeeded in 3 m 49 s. The
  two guards that would prevent a recurrence are still owed.
- **R-280** · a reinstalled machine cannot re-attach its data drive through any route, and the restore
  page promises „két kattintás" at an empty list. **The one to fix before the tester's visit.**
- **R-272** · Felhom's uninstall restarts its own dnsmasq unconstrained; it grabs `:53`; the next
  install refuses and appears to blame the owner's network.

**2 — costs the visit**
- **R-274** · a local golden is adopted with no version and no checksum check. The copy on demo-hp is
  controller **0.192.0** against a vouched **0.210.0** — and below 0.200.0, where the recovery screen
  the customer needs actually shipped.
- **R-276** · an uninstalled box keeps a live WireGuard tunnel into the off-site endpoint; declared
  in neither the KEPT nor the WIPED list.

**3 — misleads**
- **R-281** · the hub said nothing at all through the whole reinstall, and the tripwire for a
  sealed-backup unseal did not fire on a real one.
- **R-282 / R-283** · one code, three names; the mail points at a page the box is not showing; the hub
  reads "Claimed 18d ago" while the box serves its setup page.
- **R-269** · a rotated-out local-API token still authorises until an unrelated lookup forces a
  reload. The shipped test passes only because of its lookup order.
- **R-270** · R-268's own rotation recipe is a step short; the controller never re-reads the mount.
- **R-271** · the `agent_channel_unauthorized` alarm can never close — its own advice silences the
  all-clear.
- **R-277** · three hub surfaces present a healthy off-site tier as absent. **This one caught me.**
- **R-278** · demo-felhom has had no off-site backup for six days, waiting on a ceremony nobody ran.

**4 — cosmetic / hygiene**
- **R-275** · five orphaned 0600 credential backups survive, and uid reuse hands them to the new
  service account. Superseded keys here; live if the backups were recent.
- **R-279** · no operator-triggerable off-site backup exists.

## What I got wrong, and corrected

I reported to the operator that the off-site tier had not run on **either** box since 2026-08-03.
That was true of demo-felhom and **false of demo-hp**, which had 18 unbroken daily snapshots. I had
read three hub surfaces that agreed with each other and none of which said what I took them to say
(now **R-277**). I corrected it before it changed any decision, and the operator's "repair off-site
first" ruling turned out to be unnecessary for the chosen venue.

I also raised **R-275**'s sudoers half as a likely privilege-escalation on reinstall, then **tested it
and refuted my own hypothesis**: sudo skips filenames containing dots, so the leftover file is inert.
`visudo -c -f` parsing a file OK is not evidence that sudo loads it.

## Integrity verdict — BYTE-IDENTICAL

```
expected 4 file(s); found 4
VERDICT: BYTE-IDENTICAL
```

Four expected, four restored, zero differences, compared against
`evidence-rehearsal-2026-08-09/GATE0-before-manifest.json` — a manifest keyed on **raw name bytes**.
`árvíztűrő-tükörfúrógép.txt` and `nested/őszibarack.md` came back with their name bytes intact (NFC
preserved, `c3a1…`), which is the discriminator the Gate 0 positive control was built to enforce: the
comparator had been watched **failing** on an NFC→NFD rename that renders identically to the eye.
Restored out of snapshot `41c830db` into the verification folder the product names, with live data
untouched.

## Wall clocks

| phase | duration |
|---|---|
| Pre-phase (R-268 rotation, proved both ways) | ~25 min |
| Gate 0 (venue, dataset, positive control, off-site run, capture) | ~55 min |
| P1 uninstall | **60 s** (08:37:23 → 08:38:23 UTC) |
| P1 leave-behind measurement | ~12 min |
| P2 preflight (3 runs: 2 refusals, 1 pass) | ~6 min |
| P3 install — first attempt, FAILED | 44 s (08:51:33 → 08:52:17 UTC) |
| P3 install — resumed, SUCCESS | **3 m 49 s** |
| P4 first contact (box live on its own URL) | within ~4 min of install |
| STOP 3 unlock | **21 s** |
| app redeploy (calibre-web) | **1 m 36 s** |
| restore prepare + execute | **8 s + 13.2 s** |
| **bare machine → verified files** | **1 h 49 m 22 s** (08:38:23 → 10:27:45 UTC) |
| — of which the product's own work | **≈ 7 m 47 s** |

**Neither figure is the customer number.** The 1 h 49 m is dominated by the R-273 diagnosis and release
fix (~38 min) and two waits on a human. The 7 m 47 s is what the product costs when the operator
already knows every answer. **The honest unaided figure is undefined, because an unaided household does
not finish.**

## Steps taken off-path, and what they cost

1. **R-268 rotation on demo-felhom** — required by the runbook's pre-phase; not the venue.
2. **demo-hp's dashboard password re-set to the credentials-file value**, on operator instruction.
   The customer-owned password was unknown to this session and no operator route to the off-site
   button exists (R-279). Prior hash preserved in-guest; destroyed with the guest at P1. Cost: none —
   P1 wiped it and P4 re-claims.
3. **The off-site run was started by a script pressing the dashboard's own endpoint** with a real
   session and CSRF token, not by a person clicking. Identical server path; only the click synthetic.
4. **`--passphrase-file` instead of the no-echo prompt** — a first-class documented option with a
   permission check, so the secret still never touched argv. A person would type it.
5. **`systemctl stop dnsmasq && systemctl disable dnsmasq`** — the action the refusal message tells
   the owner to take, used as the counterfactual that confirmed R-272.
6. **Pushed the `v0.128.0` git tag** — outward-facing, done on your "proceed", and only after an
   independent download proved the published package's sha256 equalled the hub's vouched value. It
   completes a half-finished release rather than changing code; the release script's own recovery text
   is the same line. **Cost to the walk: the install that followed was a `--resume`, not a fresh run,
   which is why R-274 is only half-observed.**
7. **`POST /settings/storage/add` with `/mnt/sys_drive`** — the manual escape hatch, typed. This is the
   R-280 wall; a customer could not produce that path. **The biggest fidelity cost of the run.**
8. **SSH into the guest to fingerprint the restored tree.** This is my *instrument*, not a customer
   step — the customer's step (the restore) finished at the dashboard. Byte-comparison inherently needs
   file access; nothing about the product was driven this way.

Everything else after the install returned was read-only, and no repair was attempted on the box.

## Teardown — all four layers

1. **The machine** — nothing created beyond the half-install itself, which is **left in place
   deliberately** for inspection and resumption (`state.json` completed: preflight, token, grows,
   enroll). demo-hp is **not serving** right now: no guest, agent installed but no unit.
2. **The host** — `local-lvm` 20 904 790 → 12 355 143 KiB (**≈8.5 GiB returned**); `local` ≈64 MiB;
   NVMe unchanged, backups deliberately kept. Pre-existing leftovers found and **not** removed
   (not this run's, recorded instead): storage `c11-scratch`, the orphaned `vzdump-lxc-9100` archive,
   and `/root/.dpw`, `.h`, `.sec.html` in the old guest (now destroyed with it).
3. **The hub** — **no customer or appliance record was created**; the `demo-hp` customer is retained
   deliberately, as the runbook requires. Nothing to delete.
4. **The off-site side** — **one write, and it was the intended one**: the Gate 0 backup that created
   snapshots `41c830db`, `9e38b84c`, `78b93f04`. **No prune, no forget, no delete.** How I know: every
   restic call was `snapshots`, `ls`, or the product's own `POST /backup/offbox/run`; retention runs
   inside that product path and is ep0's server-side job (R-89, boxes keep `keep_last: 0`).

## Secrets handling

No secret reached stdout. Token values, the retrieval passphrase, the controller password and the hub
DB copy were handled file→file at 0600 and shredded; the hub DB copy (which carries every host's
break-glass credential) was shredded immediately after the one hash comparison it was taken for.
Credential comparisons were done by sha256 prefix, never by value.

## State demo-hp was left in

**Back in service and healthy** — agent 0.128.0, controller 0.210.0, guest 9201 running and onboot,
claimed, storage path registered, `calibre-web` deployed, off-site repository unlocked and intact at
18 snapshots. `drill-r50` (VM 300) untouched throughout.

**Deliberately left alone, and named rather than tidied:** the pre-existing `c11-scratch` storage and
the three `vzdump-lxc-9100` golden archives on `local` (the teardown keeps goldens by design, and they
now number three). The restored files sit in the product's verification folder, not back in place —
that is R-213 and the product says so.

## Still owed

- **R-280** — the drive wall. The one finding that would stop the tester's visit outright.
- **R-273's two guards** — refuse a vouch whose tag does not resolve; check that a package and its tag
  ship together. The tag push fixed one box, not the class.
- **R-274's missing observation** — a *fresh* (non-resume) install taking a stale local golden.

Full account: `documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md`.