felhom.eu/documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md

# REHEARSAL — the BYO reinstall walk (2026-08-09)

> **Status: COMPLETE.** All phases walked; the integrity verdict is **BYTE-IDENTICAL**. §1–§8 were
> written before the destructive phase deliberately — a finding that exists only in a session that
> later crashes is a finding nobody has — and are left as written, including one claim later refuted
> by test (§8.3 → §9a) and one by measurement (§7.2 F-9).

**Venue: `demo-hp` (HP t740, `felhom-host`, guest 9201, customer `demo-hp`).** Operator-approved at
STOP 1. **Driven from DooPlex.** All times UTC unless marked; the host runs CEST (UTC+2).

---

## 1. Baselines — re-confirmed live on arrival, not taken from the spec

| | spec said | live reading | source |
|---|---|---|---|
| `felhom-agent` | v0.128.0 @ `28ba8593b8` | **0.128.0** on demo-felhom, **0.127.0** on demo-hp; HEAD == `origin/main` == `28ba8593b8` | `felhom-agent --version` on both nodes; `git rev-parse` |
| `felhom-controller` | v0.210.0 @ `c732fe1283` | **0.210.0** demo-felhom, **0.208.0** demo-hp; HEAD == `origin/main` == `c732fe1283` | hub `/configs`; `git rev-parse` |
| hub | v0.101.0 @ `56f8aa611c` | **0.101.0** (deployed image tag matches) | `kubectl get deploy hub`; page footer |
| Day-0 manifest | golden 0.210.0 / agent 0.128.0 / min 0.127.0 | **all three already saved** — golden `0.210.0` (`b9f701fa…`), agent `0.128.0` (`c6eba73b…`), min agent `0.127.0`, wrapper `104db0a4…` | hub `/configuration`, selected `<option>` values |

All three repos were clean and at `origin/main` before anything ran.

**The fleet is not uniform, and the venue is the box that is behind** — demo-hp runs agent 0.127.0 and
controller 0.208.0 against a manifest vouching 0.128.0/0.210.0. That is an argument *for* the venue: the
reinstall genuinely exercises the manifest rather than re-installing what is already there.

---

## 2. Venue comparison — the fidelity table that decided it

The runbook's criterion is *"the box whose current state most resembles what the tester will have"*.

| criterion | `demo-felhom` (N100) | `demo-hp` (t740) |
|---|---|---|
| customer apps deployed | 1 — opengist | **3 — privatebin, opengist, calibre-web** |
| enrolled user-data drive | **none registered.** `/mnt/felhom-drives` empty; controller logs `[WARN] Storage paths: no storage paths registered` | `/mnt/felhom-drives/Felhom-Share` + NVMe enrolled 2026-07-22; one registered storage path `/mnt/sys_drive` |
| off-site tier has actually run | **NO** — `offsite.state=needs_credential`, no run has ever succeeded | **YES** — 18 restic snapshots, daily, unbroken |
| escrow / recovery ceremony | present, generation 2 | present, generation 2 |
| install shape | mkimage loader, **Secure Boot OFF** (firmware workaround) | shim loader, **Secure Boot ENABLED** — the customer shape |
| collateral on the box | none | `drill-r50` (VM 300); leftover `c11-scratch` storage |

**Chosen: demo-hp.** It wins every fidelity criterion that distinguishes them.

**`drill-r50` is not at risk, and this was checked rather than assumed:** VM 300 is **not** in the
`felhom` pool (`pvesh get /pools/felhom` → members `[9201]`), its disks are on `local-lvm`, and
`--uninstall` removes no storage definitions and no non-pool guest — its own end-of-teardown statement
says the enrolled drives are *"unmounted only, NEVER wiped"* and that it never deletes PBS backups, hub
records or escrow blobs.

**Correction recorded, because it was part of the comparison put in front of the operator:** the
enrolled-drive row was first reported as *"real files on an enrolled drive"*. `Felhom-Share` is
**empty** — the share exists and is enrolled, but holds no files. demo-hp still wins the row (demo-hp
has a registered storage path; demo-felhom has none), on a narrower margin than first stated.

---

## 3. Pre-phase — R-268, the leaked per-guest local-API token

**Done first and independently, on `demo-felhom` guest 9201, as the runbook required.** No token value
reached stdout at any point: old value captured file→file from `bootstrap.json`, new value minted into
a file, both shredded at the end.

### 3.1 The rotation, and its proof

| step | old token | new token |
|---|---|---|
| before rotation | **HTTP 200** | — |
| bogus token (negative control) | **HTTP 401** | — |
| after the on-disk rotation, before any reload | **HTTP 200** ← *see 3.2* | 200 |
| after that lookup forced a reload | **HTTP 401** | — |
| after `systemctl restart felhom-agent` | **HTTP 401** | **HTTP 200** |

**Positive observable for the controller half:** `local-api: channel up (agent 169.254.253.1:8443) —
guest 9201, 3 mount(s) visible`, enumerating mp8/mp9/mp0 — data only the agent can supply. An absent
error would not have been evidence.

**R-268 is CLOSED.** The old token is refused; the new one works; the controller's channel is up.

### 3.2 FINDING — an out-of-process token rotation does not revoke the old token

`localapi.TokenStore.Mint`'s doc comment states *"last-write wins — any previous token for this guest is
revoked"*. **Across processes that is false until something else forces a reload.** The long-lived agent
serves `Lookup` from an in-memory index and re-reads the store **only on a miss** (the B3
reload-on-miss optimisation). A superseded token is still a direct map **hit**, so it authorises.

**Red-proved twice.** A temporary unit probe (Scenario C with the lookup order reversed — present the
rotated-out token *first*) fails on the claim; the shipped
`TestTokenStore_ReloadOnMiss_RemintCoherence` passes **only because it looks up the NEW token first**,
which is what triggers the reload that then evicts the old hash. Then proved on real hardware: the row
marked ← above is a live 200 from a token that the store on disk had already superseded.

This is the `CLAUDE.md` case exactly — a comment asserting an invariant with a test that appears to pin
it and does not. The honest fix is either a test that pins the reversed order, or an eviction that does
not depend on an unrelated lookup missing first.

### 3.3 FINDING — R-268's own stated rotation recipe is incomplete

The register row says the new plaintext *"must also be written into the guest's
`/etc/felhom-bootstrap/bootstrap.json` or the in-guest controller loses its agent access"*. **Writing
bootstrap.json is not sufficient.** `bootstrap.ensureLocalAPI` returns early when
`cfg.LocalAPI.Endpoint != ""` — it fills an *absent* block and never refreshes a present one — so after
a rotation the controller keeps serving the old token from its own `controller.yaml` **across a
restart**. Proved live: two controller restarts, still HTTP 401. The rotation only completed once
`local_api.token` was written into `controller.yaml` itself.

The neighbouring `DetectEndpointDrift` compares the **endpoint** and deliberately does not compare the
token (*"a token mismatch is a different failure"*). So this shape is known to be unmodelled, and R-78
("which file is authoritative") is its parent.

### 3.4 FINDING — the agent-channel alarm never closes, and its own prescribed remedy is why

`channelhealth` alerted correctly and promptly: `agent_channel_unauthorized`, no debounce, pushed to
the hub — *"agent rejected the controller token (HTTP 401) — token stale/rotated (**re-bootstrap**)"*.

The channel then recovered, and **no recovery event was ever sent**. In `Checker.Check`'s UP branch the
notify is guarded by `prev != "" && prev != "up"`, and a controller restart resets `state` to `""`. The
alert's own instruction is to re-bootstrap — i.e. restart the controller — so **following the alarm's
advice structurally guarantees the recovery is silent.** The down side is deliberately asymmetric
(F2: a born-down channel alerts on cycle 1); the up side is not.

Operator-visible result on the hub: two `agent_channel_unauthorized` errors, one `sent` and one
`suppressed` by cooldown, and nothing afterwards. The customer dashboard is fine — `SetDashboard`
reflects current state every cycle. It is the operator's trail that ends on "down".

---

## 4. Gate 0

### 4.1 The recovery code — CONFIRMED IN THE OPERATOR'S HANDS (STOP 1)
Not printed, not logged, not requested. It will be supplied at the screen at STOP 3.

### 4.2 The dataset — planted and fingerprinted

Planted at `/var/lib/felhom/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/` — inside
the **Calibre library**, which is real app-owned customer data and is **demonstrably** carried off-site
(the path appears in every `felhom-offbox,calibre-web` snapshot). Chosen after checking coverage rather
than assuming it: the per-app tier-1 unit carries only the named docker volume, so a file dropped in an
arbitrary place would not have travelled.

| sha256 (16) | bytes | name | name bytes (hex) |
|---|---|---|---|
| `54b773c46bbfd994…` | 3 145 728 | `binary-3mb.bin` | `62696e6172792d336d622e62696e` |
| `52a5c5ebfcac247f…` | 59 | **`árvíztűrő-tükörfúrógép.txt`** | `c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874` |
| `15d2024dfc224162…` | 25 | **`nested/őszibarack.md`** | `6e65737465642fc591737a6962617261636b2e6d64` |
| `924497918e55fe6d…` | 21 | `plain.txt` | `706c61696e2e747874` |

Manifest: `documentation/audits/evidence/` → recorded in the session scratchpad as
`GATE0-before-manifest.json`. The manifest keys on **raw name bytes**, not rendered text.

### 4.3 The positive control — the comparator was watched failing, three ways

| control | expected | result |
|---|---|---|
| identical copy | PASS | **PASS**, exit 0 |
| one bit flipped at byte 1 500 000 of the 3 MB binary | FAIL | **FAIL** — named the file and both sha256s |
| **NFC→NFD rename of the accented name** (renders *identically*) | FAIL | **FAIL** — caught as MISSING+EXTRA on the name bytes |
| one file deleted | FAIL | **FAIL** — named the missing file |

The NFD case is the one that matters: a comparison of *rendered* filenames would have passed it. Every
altered copy was discarded after the control ran.

### 4.4 The off-site tier — has run since the dataset was planted

Triggered through the product's own dashboard button (`POST /backup/offbox/run` → flash *„A távoli
mentés elindult"*), not by invoking restic by hand.

```
[offbox] backup run started (3 app(s) toggled)
[offbox] pre-push dump leg completed in 7.329s — snapshot pair is coherent
[offbox] backed up calibre-web (…/backups/primary/calibre-web, 1 mandatory path(s))
[offbox] backup OK: 3 app(s) backed up, 18 snapshot(s), 1m22s
```

**The restore point, by identity — never by size:**

| snapshot | time (UTC) | tag | paths |
|---|---|---|---|
| **`41c830db`** | 2026-08-09 08:30:38 | `felhom-offbox,calibre-web` | `…/backups/primary/calibre-web` + `…/userdata/media/books` |
| `9e38b84c` | 2026-08-09 08:30:49 | `felhom-offbox,opengist` | `…/backups/primary/opengist` |
| `78b93f04` | 2026-08-09 08:30:53 | `felhom-offbox,privatebin` | `…/backups/primary/privatebin` |

`restic ls 41c830db` confirms all four planted files are present, **both accented filenames included**.

### 4.5 The box, captured
`GATE0-demo-hp-before.txt` (282 lines): agent version, `pveversion`, `pct`/`qm` lists, pool membership,
`pvesm status`, `storage.cfg`, `df`, `lsblk` with serials, the agent unit + service user, sudoers, the
pveum users/tokens/roles/ACL, the guest hook, the WireGuard tunnel, the OOB belt, the agent state dir,
the install state file, PBS snapshots, local vzdump archives, and the enrolled-drive tree.

---

## 5. Findings so far, ranked by what they cost the person in front of you

Nothing yet is rank 1 or 2 — the walk has not started.

### Rank 3 — misleads

- **F-1 · A rotated-out local-API token still authorises** (§3.2). *A customer would do nothing — they
  cannot see it. An operator rotating a leaked token would believe it revoked when it is not.* The
  shipped test passes for the wrong reason.
- **F-2 · R-268's rotation recipe is incomplete** (§3.3). *Whoever follows it gets a box whose agent
  channel is dead across restarts, with a correct-sounding alarm and no hint that `controller.yaml` is
  the file that matters.*
- **F-3 · The agent-channel alarm never closes** (§3.4). *The operator fixes it as instructed and is
  never told it worked; the hub's last word stays "down".*
- **F-4 · Three hub surfaces jointly misrepresent a healthy off-site tier as absent.** The customer page
  reads `Snapshots 0 / Repo Size 0 MB / Integrity Unknown` (that panel renders the **local** tier, while
  the healthy `offsite` object — `snapshot_count: 18`, `last_status: ok` — sits unrendered in the same
  report); the Offsite page shows `0.0 GB` (true, but a 162 KB repo rounds to nothing); and a stale
  `offsite_delivery_stuck` event from 2026-08-07 reads as current. *Three independent surfaces agreeing
  on a wrong picture is how a working backup gets "fixed".* **This one caught me** — it produced a wrong
  statement to the operator at STOP 1, corrected before it changed anything.
- **F-5 · There is no operator-triggerable off-site backup.** The only route is the customer's dashboard
  button; `signed_jobs` needs a signing key the hub does not hold. Sibling of R-177.

### Rank 4 — cosmetic / hygiene

- **F-6 · Leave-behinds from earlier sessions**, found while capturing state:
  - demo-hp host: storage `c11-scratch` (Campaign 11), and `felhom-backup:…vzdump-lxc-9100-2026_08_03…`
    for a guest that no longer exists;
  - demo-hp guest 9201 `/root`: `.dpw` (13 bytes), `.h`, `.sec.html`, all 2026-08-07 16:08;
  - demo-felhom guest 9201: `/root/r204-backup/offbox.orig`.
- **F-7 · The managed-update floor is `0.200.0`, not `0.156.0`** — the runbook's §8.3 premise is stale.
  Confirmed twice: hub `/configuration` reads *"Effective floor v0.200.0 — source: DB (hub_settings);
  env fallback would be v0.120.0"*, and both boxes log `settle-gate: GO — at/above floor 0.200.0`. The
  ruling's **substance** still needs recording; its **number** must not be written down as 0.156.0.

### Not a finding, checked and cleared
- The two `userdata` trees under `sys_drive` are the **same directory** surfaced twice, not a
  backed-up copy beside an unprotected one. Checked against calibre-web's actual bind mounts.

---

## 6. Steps taken off-path, and what they cost

Rule 2 (no shell after the install command returns) does not bind yet — the walk has not begun. These
are Gate 0 preparation acts by the operator, recorded because they are mutations:

1. **R-268 rotation on demo-felhom** — token store appended, `bootstrap.json` and `controller.yaml`
   rewritten, agent and controller restarted. Required by the runbook's pre-phase. *Not the venue.*
2. **demo-hp's dashboard password was re-set to the value in the operator's credentials file.** The
   customer-owned password was not known to this session, and no operator-side route to the off-site
   button exists (F-5). The prior `password_hash` is preserved at `/root/.orighash` and
   `/root/settings.json.rehearsal-orig` in guest 9201; `claimed` was left `true`. **Done on operator
   instruction.** Cost to fidelity: none for the walk — P1 destroys this guest, and P4 re-claims with a
   password the operator chooses.
3. **The off-site run was started by a script pressing the dashboard's own endpoint**, with a real
   session and a real CSRF token, rather than by a person clicking. The server path is identical; only
   the click is synthetic.

---

## 7. P1 — removing Felhom from the host

`--uninstall`, the product's own path, dry-run first. **60 seconds** (08:37:23 → 08:38:23 UTC), exit 0.
The typed-vmid confirmation was satisfied, not bypassed.

### 7.1 The reversal, proved rather than trusted

| component | before | after |
|---|---|---|
| agent binary | `/usr/local/bin/felhom-agent` 0.127.0 | **ABSENT** |
| `felhom-agent` service user | present | **ABSENT** |
| `felhom-op` (OOB) user | present | **ABSENT** |
| `/etc/felhom-sshd` (OOB belt) | present | **ABSENT** |
| systemd units | agent, mgmt-watchdog, felhom-sshd, oob-nft, shared-parent | **all removed** |
| pveum roles | `FelhomAgentBase/Guest/Store` | **none** |
| pveum user + token | `felhom-agent@pve!agent` | **gone**; user list is `root@pam` only |
| pveum ACL | pool + 9 scratch-vmid + 3 storage grants | **empty** |
| `felhom` pool | members `[9201]` | **`[]` — pool removed** |
| guest 9201 | running | **destroyed** |
| guest-hook snippet | present | **removed** |
| dnsmasq snippets | present | **removed** |
| `/var/lib/felhom-agent`, `/var/lib/felhom-install` | present | **removed** |
| **KEPT as declared** | | PBS snapshots (2), local vzdump (4), `/mnt/felhom-drives/Felhom-Share`, storage defs |

Space returned: `local-lvm` 20 904 790 → 12 355 143 KiB (**≈8.5 GiB**); `local` ≈64 MiB; the NVMe
unchanged (backups deliberately kept). `drill-r50` (VM 300) untouched, as predicted.

### 7.2 What it left behind that it did not declare

**F-8 · Five orphaned `agent.json.*` backups survive, each carrying a hub API key and a Proxmox
token.** `/etc/felhom-agent/` remains with `agent.json.campaign8-before`, `.campaign9-before`,
`.campaign9-prev`, `.pre-e-target-move`, `.pre-prunegate.bak` — all 0600, three owned by the **now-deleted
service user's uid 999**, each holding a 64-char `hub.api_key` and a 59-char `proxmox.token`.

The uninstall's own WIPED statement claims it removes *"config (+ its .bak backups)"*, and
`scripts/CHANGELOG` records F1 as *"uninstall now purges the agent config's `.bak*` siblings (one held a
live hub api_key)"*. **That fix does not match the filenames in use** — and note it misses
`agent.json.pre-prunegate.bak`, a file that literally ends in `.bak`.

**Assessed rather than assumed: these particular secrets are SUPERSEDED, not live.** The orphaned
`hub.api_key` hashes to `a5d2222a…`; the hub's current key for demo-hp is `8c59d1b6…`. The Proxmox token
was deleted by this same uninstall. **The exposure is timing-dependent, not absent** — a backup taken
shortly before an uninstall would hold a current key, which is exactly what R-179's precedent records.
The broken thing is the stated guarantee.

**And the reinstall hands them to the new service account.** The rebuilt box recreates `felhom-agent` at
**uid 999 — the same uid the deleted one had** — so the three backups owned by uid 999 become the new
account's files. Verified: `su -s /bin/sh felhom-agent -c 'test -r …'` succeeds on
`agent.json.campaign8-before` and `agent.json.pre-prunegate.bak`. A fresh install's service account
inherits read access to the previous install's credentials. Superseded here; live if the backups are recent.

**F-9 · A stale sudoers file survives — and it is INERT. Hypothesis raised, then refuted by test.**
`/etc/sudoers.d/felhom-agent.bak-pre-e2a` (21 833 bytes, 2026-07-28, 0440 root:root) survives the
uninstall and grants `felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, …`,
and `visudo -c -f` on it says **`parsed OK`**.

The obvious worry — that the reinstall recreates the `felhom-agent` user and the box silently regains a
pre-E-2a privilege set — **was tested against the rebuilt box and is FALSE.** With the user recreated
(uid 999) and this as the *only* felhom file in `/etc/sudoers.d`, `sudo -l -U felhom-agent` returns
**"User felhom-agent is not allowed to run sudo on felhom-host."** sudo ignores files in `sudoers.d`
whose names contain a dot, so `…​.bak-pre-e2a` is never loaded. That is a robust reason, not luck —
`visudo -c -f <file>` checks a file explicitly and therefore does **not** tell you whether sudo loads it.

**What is left is rank 4:** 21 KB of the privilege grammar lingering on a host Felhom was told to leave,
inside a teardown that claims to have removed the sudoers file.

**F-10 · The WireGuard tunnel to Felhom's off-site endpoint stays up.** After an uninstall,
`wg-quick@wg-felhom` is **enabled and active**, `/etc/wireguard/wg-felhom.conf` present, with a live
handshake to `167.233.158.164:443` 52 seconds old and 5.86 GiB received / 2.48 GiB sent on the counter.
It appears in **neither** the WIPED nor the KEPT list. A box told to leave Felhom keeps a live inbound
route into Felhom's infrastructure, and the operator is not told.

**F-11 · Smaller undeclared residue:** `/etc/felhom/` (`.bootstrap-done`, `appliance-pairing-code` —
the console pairing code); `/etc/systemd/system/felhom-bootstrap.service` + `/usr/local/sbin/felhom-bootstrap.sh`;
the `vmbr9` island-bridge stanza (169.254.253.1/30) still in `/etc/network/interfaces`.

**F-12 · A teardown-ordering warning:** `user config - ignore invalid acl role 'FelhomAgentStore'` —
the role is deleted before an ACL that still references it. Harmless here; noise that would mask a real one.

---

## 8. P2 — the preflight, as the owner runs it

### 8.1 The refusals, judged on their messages

**First run, no caps:** `byo mode requires explicit --cores and --memory (noisy-neighbor caps on a host
you do not own)`. **Good** — names the missing thing and the reason.

**Second run:** it refused again —

```
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
  Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
  (Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
```

As a *message* this is excellent: the finding, the evidence, two routes out, and an explicit promise
about what it will not do.

### 8.2 F-13 (RANK 1 — STOPS THE VISIT) · Felhom's uninstall leaves the exact condition that makes Felhom's reinstall refuse

The resolver it is refusing over **is Felhom's own**, and the chain is fully evidenced:

| fact | evidence |
|---|---|
| Felhom installed `dnsmasq` | `/var/lib/dpkg/info/dnsmasq.list` dated **2026-07-21 18:24 CEST** — demo-hp's Felhom day-0 |
| it was constrained by a Felhom snippet | `/etc/dnsmasq.d/` held Felhom snippets before; only `README` after |
| **the uninstall restarted it unconstrained** | the running process started **2026-08-09 10:37:39 CEST — inside the uninstall window (10:37:23–10:38:23)** |
| unconstrained, it grabs the wildcard | `0.0.0.0:53` and `[::]:53`, `--local-service`, empty `dnsmasq.conf` |
| it is not PVE's | `/etc/pve/sdn/` empty; unit is stock `/usr/lib/systemd/system/dnsmasq.service`, **enabled** |

The uninstall mentions it only as *"the 'sudo' and 'dnsmasq' packages were left installed (**system
packages**)"* — but dnsmasq is not a system package here; Felhom installed it, and it left it **enabled
and running as a wildcard resolver**.

**What a customer would do next:** read a message that says a resolver is in the way, conclude their own
network setup is at fault, and start debugging DNS they never configured. On a BYO box with no shell
guidance, this is where the visit stops.

**Confirmed by counterfactual:** `systemctl stop dnsmasq && systemctl disable dnsmasq` → `host DNS (:53):
free` → **PRE-FLIGHT PASS**. Nothing else changed. *(Recorded as an intervention — it is the action the
message tells the owner to take.)*

### 8.3 F-14 (RANK 2) · A local golden is adopted with NO version and NO checksum check

Preflight reported `golden (local): local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst`. Step 7 is:

```sh
if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then
    log_skip "  using local golden: $GOLDEN_VOLID"
    _state_mark golden; return 0
fi
```

**The hub manifest's `golden.sha256` — the whole point of the hub vouching a different trust root than
Gitea — is consulted only on the fetch path.** A locally-present archive bypasses the vouch entirely: no
version comparison, no digest, no warning.

**What that archive actually contains, read out of the tar:**
`gitea.dooplex.hu/admin/felhom-controller:`**`0.192.0`**, against a vouched golden of **0.210.0**.

Two consequences, and the second is the serious one:

1. The rebuilt box comes up **18 controller releases stale**, reporting success.
2. **0.192.0 is below 0.200.0 — the release where R-193's off-site recovery SCREEN shipped.** So a
   customer reinstalled today returns on a controller that cannot run the recovery ceremony their data
   depends on. It is also born below the managed-update floor (0.200.0), and the updater's auto-target
   is the floor, never the newest — so it converges to 0.200.0, still not the vouched build.

**This is the R-111/R-115/R-120 drift family, one layer down:** the R-120 gate guards what the operator
may *vouch*; nothing guards what an install actually *takes*. It also compounds with P1 — the uninstall
deliberately keeps the old golden (*"golden vzdump left in place"*), and the reinstall then adopts it.

**The walk continues unflagged** (`--force-gitea-golden` NOT passed), because measuring the path a real
reinstall takes is the point. §9 records which controller version actually came up.

### 8.4 The BYO mutation disclosure — read as a customer

Comprehensive and honest: every pveum artifact, the sudoers file, the state and config paths, the
packages it will install, the guest's capped size, the self-update authority, and — creditably — the
**outbound WireGuard tunnel**, in plain words, with the note that the operator can disable it per
customer. It closes with `NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd.`

**One material omission, given §8.3:** it does not say that a pre-existing local golden may be used
instead of the vouched one. The disclosure lists everything the install will *create* and nothing about
what it may *reuse*.

---

## 9. P3 — the install. IT FAILED, AND THE WALK STOPS HERE

Started 08:51:33 UTC, **failed 08:52:17 UTC (44 s), exit 1, at step 5/8.** Run unflagged, as a real
owner would. The typed hostname acknowledgement was given, not bypassed.

```
[INFO]   manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.210.0
[OK]     verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK]     installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[OK]     created service user felhom-agent
curl: (22) The requested URL returned error: 404
[ERROR] raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)
```

### 9.1 F-15 (RANK 1 — STOPS EVERY INSTALL, FLEET-WIDE) · The vouched agent version was never tagged

**`v0.128.0` does not exist as a git tag** — not locally, not on Gitea. The newest tag is `v0.127.0`.
The *binary* was published to the Gitea package registry (which is why it is selectable in the hub's
artifact dropdown and why its sha256 verified), but the release's tag step never happened.

| ref | HTTP |
|---|---|
| `raw/branch/main/configs/felhom-mkfs-guarded.sh` | **200** |
| `raw/tag/v0.127.0/configs/felhom-mkfs-guarded.sh` | **200** |
| **`raw/tag/v0.128.0/configs/felhom-mkfs-guarded.sh`** | **404** |

The commit that *is* 0.128.0 exists and is on `main`: `28ba8593b8 — v0.128.0 — R-221: the escrow seed is
asserted every tick, not remembered once`. Only the tag is missing.

**This is R-184 happening.** R-184 reads: *"Nothing prevents the hub from vouching an agent version that
was never released. The R-115 gate proves every RELEASED version is installable, but it works from git
tags."* It was filed as a gap. It is now a live outage.

**It is the collision of a good fix with an open gap.** R-183 correctly hardened the installer to pin
config fetches to the vouched tag (`hostinstall_gates.py` even asserts the pin exists —
`ok: fetch_raw pins the agent configs to the vouched agent version`). Nothing asserts that the pinned ref
**resolves**. So the first vouch naming an untagged version fails every install closed.

**The release script already knows.** `felhom-agent/scripts/release-agent.sh:23` says in as many words:
*"`raw/tag/v<version>`, a released version without a git tag 404s a box mid-install, as root."* The
hazard was understood, documented, and the tag step was still missed.

**Blast radius — stated precisely, not inflated:**
- **Every fresh install and every reinstall, on every host, fails at step 5/8** for as long as the Day-0
  manifest vouches agent 0.128.0. That manifest was approved on 2026-08-09; the approval is what armed it.
- **Existing boxes are unaffected.** demo-felhom runs 0.128.0 happily — the self-update path takes the
  binary from the package registry, not from a tag.
- So the damage is confined to the install path. That is also the only path a new customer, or a
  reinstalling tester, ever walks.

**What a customer would do next:** nothing. `curl: (22) … 404` and a question about a git tag is not a
message a household can act on. The visit ends here and a developer is required.

### 9.2 The box's state, left as found for inspection

`--uninstall` is not re-run and no manual repair is attempted (rule 1). demo-hp is mid-install:

```
state.json  completed: [preflight, token, grows, enroll]   mode: byo
agent       0.128.0 installed, unit NOT created (not-found), inactive
user        felhom-agent uid=999 created
pveum       2 users, 3 roles, 63 ACL lines, pool 'felhom' recreated (empty)
guest       none
```

The documented `--resume` path exists and **was deliberately not used: it cannot help.** Resume replays
from `enroll`, hits the same 404, and fails identically — the missing tag is not a transient.

### 9.3 The one-line unblock, for the operator to decide

```sh
cd /mnt/5_hdd/felhom.eu/git/felhom-agent
git tag v0.128.0 28ba8593b8 && git push origin v0.128.0
```

This publishes the tag for a commit that is already on `main` and already released as a package — it
completes a half-finished release rather than changing any code. The alternative, rolling the manifest
back to agent 0.127.0, costs R-221 — the fix `STATUS.md` calls *"the last item blocking a customer from
something we promise them"* — so it is the worse option.

**Not done in this session.** Publishing a release tag is an outward-facing act and the runbook is
explicit: *"If a defect is so severe that the run cannot continue, stop and report; do not fix it here."*

### 9.4 What F-14 still owes

Step 7 never ran, so **whether the install would actually have adopted the stale 0.192.0 golden is not
yet observed** — F-14 rests on the code path plus the preflight's `golden (local): …9100…2026_08_03…`
line and the archive's own `felhom-controller:0.192.0` marker. That is strong, but it is not the
observation. **When the tag is published and the walk resumes, the first thing to record at step 7/8 is
which golden it takes.**

---

## 9a. P3 resumed — and the install succeeded

**Unblocked on operator instruction ("proceed").** `v0.128.0` was annotated at `28ba8593b8` and pushed
after an **independent download** confirmed the published package's sha256 equals the hub's vouched
`c6eba73b…`. Both config URLs then served 200. `--resume` completed in **3 m 49 s**
(09:30:30 → 09:34:19 UTC): `Day-0 provision SUCCESS`, controller **0.210.0**, agent **0.128.0**, guest
9201 onboot, pool member, all 16 ACL assertions green, controller healthy in ~18 s.

**R-274 partly refuted, and recorded as such.** Step 7 fetched the **vouched 0.210.0** — because
`--resume` skips preflight, and preflight is where local auto-discovery sets `GOLDEN_VOLID`. So the
fresh and resume paths disagree, and the resume path is the safe one. What survives: discovery is
`sort | tail -1` (newest local) with **no manifest comparison**, so a box whose newest local golden
predates the vouched one still installs stale. Full text in the register.

## 10. P4 — first contact, as the customer

**A clean pass, and worth saying so plainly.** Within four minutes of the install finishing, the
customer's own URL — fetched from outside the box, over the public internet — served:

> **A szerver beállítása** · Demo HP
> *Add meg az e-mailben kapott beállító kódot, majd válassz saját jelszót a vezérlőpult védelméhez.*
> Beállító kód · Új jelszó (min. 12 karakter) · Új jelszó megerősítése
> *Nem kaptad meg a kódot? Új kód kérése*

Unprompted, in Hungarian, naming the customer, with a self-service route if the code never arrived,
and nothing anywhere asking for a command line. The hub showed the host **ONLINE** at agent 0.128.0.

**Two findings here, neither fatal:** the hub still read *"Claimed 18d ago"* while the box was serving
its first-run page (**R-283**), and the code that arrives is named three different things across the
three surfaces, with the mail pointing at an „Elfelejtett jelszó" page the box does not show
(**R-282**). The reset code was nonetheless **accepted on the setup page** — 302 and a session — so
this is naming, not function.

## 11. P5 — getting the machine back to work

### 11.1 The recovery screen, unsought — the headline pass

The **first thing** on the dashboard after claiming, with nothing sought:

> **Adatok visszaszerzése** — *Ezt a gépet újratelepítették. A korábbi, házon kívüli mentéseid
> megvannak* — a sealed package held centrally, **sealed at 2026-08-04T11:11:37Z**, openable only with
> the customer's code; *nobody can replace it — not Felhom, not support, not the operator*; and
> **„Ebben a lépésben semmit nem állítunk vissza és semmi nem változik."**
> Field: **Helyreállítási kód (tíz szó)**.

R-193's screen meeting reality on a genuinely rebuilt box. It answered all three of its questions and
its seal date matches `host_escrow.created_at` exactly.

*(Nit: the seal date is rendered raw as `2026-08-04T11:11:37Z` to a Hungarian household — ISO-8601 in
UTC where a localised date belongs.)*

### 11.2 STOP 3 — the unlock

Entered at the box's own screen, from a file, never on a command line. **21 seconds**, and it listed
what it found without restoring anything:

| app | legutóbbi mentés | méret |
|---|---|---|
| **calibre-web** | 2026-08-09 10:30 (CEST) | **3.8 MB** |
| felhom-offbox | ″ | 6.8 KB |
| opengist | ″ | 182.3 KB |
| privatebin | ″ | 6.8 KB |

That is exactly the Gate 0 snapshot set, seen from the customer's side.

### 11.3 The wall — R-280, and it is rank 1

The restore page diagnoses the situation perfectly and then sends the customer to an empty page:

> *„Előbb csatold vissza az adatmeghajtót. … **Ez két kattintás:** Tárhely → Meghajtók, »Meglévő
> meghajtó csatolása«. Utána gyere vissza ide."*

**It is not two clicks; it is zero possible clicks.** `GET /api/disks/candidates` →
`{"initialize":[],"attach":[]}`. The agent is fine — `GET /api/disks` returns the NVMe in full — but
`handleDiskCandidates` builds *both* lists from the **unclaimed-disk** scan, and demo-hp's NVMe is
deliberately both the user-data drive and the `felhom-backup` target, so it is claimed and never
offered. Correct for `initialize`; over-broad for `attach`, which is non-destructive by definition.

It cascades: no store → Calibre-Web's install page degrades to *„Nincs regisztrált adattároló — adja
meg kézzel az útvonalat"*; no app → every restore row reads „Nincs telepítve".

**Escape hatch used, and recorded as off-path:** `POST /settings/storage/add` with
`storage_path=/mnt/sys_drive` succeeded first try — an internal path, the very one registered before
the wipe, that no household customer could produce. Everything unblocked immediately afterwards and
the deploy form became a proper picker („Tárhely (sys_drive) — 64.2 GB szabad").

### 11.4 App redeploy

`calibre-web` deployed from the catalogue through the dashboard's own API in **1 m 36 s**, running and
healthy. The catalogue itself was complete and in Hungarian, every app „Nincs telepítve".

*(Also observed: the deploy page warned „A kiválasztott tárhely majdnem megtelt." over an option
carrying `data-free-percent="93"` and „64.2 GB szabad" — an apparent inverted threshold, **R-284**.)*

## 12. P6 — getting the data back

Three options, each explained, none starting without a question. **R-204 item 3 is visibly fixed:**
option 1 states in as many words that it does *not* bring the customer's own files back.

| step | wall clock |
|---|---|
| full-restore **prepare** (3.8 MB) | **8 s** |
| full-restore **execute** | **13.2 s** (10:27:32.08 → 10:27:45.29 UTC) |

> *„A(z) calibre-web teljes mentése visszaállítva ellenőrző mappába:
> `/mnt/sys_drive/felhom-data/backups/offsite-restore/calibre-web` — a saját fájljaiddal együtt. A
> meglévő adatok változatlanok."*

Restored to a **verification folder**, not into place — honest about it, and R-213's territory.

### THE INTEGRITY VERDICT — **BYTE-IDENTICAL**

```
expected 4 file(s); found 4
VERDICT: BYTE-IDENTICAL
```

Compared against `GATE0-before-manifest.json`, keyed on **raw name bytes**:

| sha256 | bytes | name |
|---|---|---|
| `54b773c46bbfd994…` | 3 145 728 | `binary-3mb.bin` |
| `52a5c5ebfcac247f…` | 59 | **`árvíztűrő-tükörfúrógép.txt`** — name bytes `c3a1…` identical, NFC preserved |
| `15d2024dfc224162…` | 25 | **`nested/őszibarack.md`** — name bytes identical |
| `924497918e55fe6d…` | 21 | `plain.txt` |

Four expected, four restored, zero differences. The accented filenames survived as **bytes**, not
merely as rendered text — the discriminator the Gate 0 positive control was built to enforce.

## 13. P7 — what the hub thought was happening

**It thought nothing.** Across the whole reinstall the hub recorded **zero events and zero
notifications** for demo-hp. Positive control (standing rule 3): the hub logged **2 events all day
across all customers**, newest `db_dump_completed` at 00:30:07 — the store is reachable and the
silence is real.

- **The good half:** no false alarm fired during a legitimate reinstall, which is what P7 watches for.
- **The owed half (R-281):** `escrow_blob_served` exists as the tripwire for exactly this moment —
  *"If no recovery is in progress on that box, investigate"* — and **has fired for demo-hp before**
  (last 2026-08-04 20:12:54). Today's real unseal fired it not at all. A reinstall and a stolen
  machine look identical to the operator.

## 14. Teardown — all four layers

1. **The machine** — my instruments (`fp.py`, `restored.json`, the installer, the passphrase file)
   removed. Guest 9201 running and healthy. **`drill-r50` (VM 300) untouched throughout**, as predicted.
2. **The host** — `local-lvm` 36.97 % → 21.85 % (bare) → **32.35 %** (rebuilt). Pre-existing leftovers
   found and deliberately **not** removed, recorded instead: `c11-scratch`, the orphaned
   `vzdump-lxc-9100` archives (now three).
3. **The hub** — **no customer or appliance record created**; `demo-hp` retained deliberately per the
   runbook. Nothing to delete.
4. **The off-site side** — **18 snapshots, newest still `9e38b84c` / `78b93f04` at 08:30 UTC**, i.e.
   unchanged since Gate 0. The restore was a pure read. **No prune, no forget, no delete by me**; the
   only retention that ran was inside the product's own backup call at Gate 0, which reported
   `18 snapshot(s)` itself.

## 15. The answer to §2's question

**Yes — the data comes back, byte for byte. No — not in one sitting, and not without a shell.**

The walk completed: **P1 ✓ P2 ✓ P3 ✓ P4 ✓ P5 ✓ P6 ✓ P7 ✓**, and the verdict is BYTE-IDENTICAL.
But it completed only because two hard stops were cleared by someone who could open a terminal and
read source code:

1. **R-273** — the install died at 5/8 on a git tag that was never pushed. Cleared by a release action.
2. **R-280** — the data drive could not be re-attached through any dashboard route, while the page
   promised „két kattintás". Cleared by POSTing an internal path a customer could not know.

Neither is a data-integrity problem. Both are **journey** problems, and both stop a household dead.
This is the same shape the R-201 walks kept finding: **the data half passes, the journey half fails.**

**What the product did beautifully, and should not be lost in the finding count:** the box came up on
its own at the vouched version; the setup page appeared unprompted, in Hungarian, naming the customer;
the recovery screen appeared **without being sought** and stated plainly that nothing would be changed
by unlocking; the unlock took **21 s**; the restore took **13.2 s** and said honestly that it had put
the files in a verification folder rather than back in place. Sixteen ACL assertions verified
themselves. No false alarm fired.

### Wall clocks

| segment | |
|---|---|
| P1 uninstall | **60 s** |
| P2 preflight (2 refusals, then pass) | ~6 min |
| P3 install — first attempt, FAILED | 44 s |
| P3 install — resumed, SUCCESS | **3 m 49 s** |
| P4 first contact (box live on its own URL) | within ~4 min of install |
| STOP 3 unlock | **21 s** |
| app redeploy (calibre-web) | **1 m 36 s** |
| restore prepare + execute | **8 s + 13.2 s** |
| **bare machine → verified files** | **1 h 49 m 22 s** (08:38:23 → 10:27:45 UTC) |
| — of which the product's own work | **≈ 7 m 47 s** |

**The 1 h 49 m must not be quoted as the customer number** — it is dominated by the R-273 diagnosis and
release fix (~38 min) and by two waits on a human. **The ≈ 7 m 47 s must not be quoted either**: it is
what the product costs when someone already knows every answer. The honest figure for an unaided
household is **undefined, because an unaided household does not finish.**