felhom.eu/documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md

# REHEARSAL — the BYO reinstall walk (2026-08-09)

> **Status: IN PROGRESS.** Pre-phase and Gate 0 are complete; STOP 1 passed. The walk (P1–P7) has not
> started. This file is written before the destructive phase deliberately — a finding that exists only
> in a session that later crashes is a finding nobody has.

**Venue: `demo-hp` (HP t740, `felhom-host`, guest 9201, customer `demo-hp`).** Operator-approved at
STOP 1. **Driven from DooPlex.** All times UTC unless marked; the host runs CEST (UTC+2).

---

## 1. Baselines — re-confirmed live on arrival, not taken from the spec

| | spec said | live reading | source |
|---|---|---|---|
| `felhom-agent` | v0.128.0 @ `28ba8593b8` | **0.128.0** on demo-felhom, **0.127.0** on demo-hp; HEAD == `origin/main` == `28ba8593b8` | `felhom-agent --version` on both nodes; `git rev-parse` |
| `felhom-controller` | v0.210.0 @ `c732fe1283` | **0.210.0** demo-felhom, **0.208.0** demo-hp; HEAD == `origin/main` == `c732fe1283` | hub `/configs`; `git rev-parse` |
| hub | v0.101.0 @ `56f8aa611c` | **0.101.0** (deployed image tag matches) | `kubectl get deploy hub`; page footer |
| Day-0 manifest | golden 0.210.0 / agent 0.128.0 / min 0.127.0 | **all three already saved** — golden `0.210.0` (`b9f701fa…`), agent `0.128.0` (`c6eba73b…`), min agent `0.127.0`, wrapper `104db0a4…` | hub `/configuration`, selected `<option>` values |

All three repos were clean and at `origin/main` before anything ran.

**The fleet is not uniform, and the venue is the box that is behind** — demo-hp runs agent 0.127.0 and
controller 0.208.0 against a manifest vouching 0.128.0/0.210.0. That is an argument *for* the venue: the
reinstall genuinely exercises the manifest rather than re-installing what is already there.

---

## 2. Venue comparison — the fidelity table that decided it

The runbook's criterion is *"the box whose current state most resembles what the tester will have"*.

| criterion | `demo-felhom` (N100) | `demo-hp` (t740) |
|---|---|---|
| customer apps deployed | 1 — opengist | **3 — privatebin, opengist, calibre-web** |
| enrolled user-data drive | **none registered.** `/mnt/felhom-drives` empty; controller logs `[WARN] Storage paths: no storage paths registered` | `/mnt/felhom-drives/Felhom-Share` + NVMe enrolled 2026-07-22; one registered storage path `/mnt/sys_drive` |
| off-site tier has actually run | **NO** — `offsite.state=needs_credential`, no run has ever succeeded | **YES** — 18 restic snapshots, daily, unbroken |
| escrow / recovery ceremony | present, generation 2 | present, generation 2 |
| install shape | mkimage loader, **Secure Boot OFF** (firmware workaround) | shim loader, **Secure Boot ENABLED** — the customer shape |
| collateral on the box | none | `drill-r50` (VM 300); leftover `c11-scratch` storage |

**Chosen: demo-hp.** It wins every fidelity criterion that distinguishes them.

**`drill-r50` is not at risk, and this was checked rather than assumed:** VM 300 is **not** in the
`felhom` pool (`pvesh get /pools/felhom` → members `[9201]`), its disks are on `local-lvm`, and
`--uninstall` removes no storage definitions and no non-pool guest — its own end-of-teardown statement
says the enrolled drives are *"unmounted only, NEVER wiped"* and that it never deletes PBS backups, hub
records or escrow blobs.

**Correction recorded, because it was part of the comparison put in front of the operator:** the
enrolled-drive row was first reported as *"real files on an enrolled drive"*. `Felhom-Share` is
**empty** — the share exists and is enrolled, but holds no files. demo-hp still wins the row (demo-hp
has a registered storage path; demo-felhom has none), on a narrower margin than first stated.

---

## 3. Pre-phase — R-268, the leaked per-guest local-API token

**Done first and independently, on `demo-felhom` guest 9201, as the runbook required.** No token value
reached stdout at any point: old value captured file→file from `bootstrap.json`, new value minted into
a file, both shredded at the end.

### 3.1 The rotation, and its proof

| step | old token | new token |
|---|---|---|
| before rotation | **HTTP 200** | — |
| bogus token (negative control) | **HTTP 401** | — |
| after the on-disk rotation, before any reload | **HTTP 200** ← *see 3.2* | 200 |
| after that lookup forced a reload | **HTTP 401** | — |
| after `systemctl restart felhom-agent` | **HTTP 401** | **HTTP 200** |

**Positive observable for the controller half:** `local-api: channel up (agent 169.254.253.1:8443) —
guest 9201, 3 mount(s) visible`, enumerating mp8/mp9/mp0 — data only the agent can supply. An absent
error would not have been evidence.

**R-268 is CLOSED.** The old token is refused; the new one works; the controller's channel is up.

### 3.2 FINDING — an out-of-process token rotation does not revoke the old token

`localapi.TokenStore.Mint`'s doc comment states *"last-write wins — any previous token for this guest is
revoked"*. **Across processes that is false until something else forces a reload.** The long-lived agent
serves `Lookup` from an in-memory index and re-reads the store **only on a miss** (the B3
reload-on-miss optimisation). A superseded token is still a direct map **hit**, so it authorises.

**Red-proved twice.** A temporary unit probe (Scenario C with the lookup order reversed — present the
rotated-out token *first*) fails on the claim; the shipped
`TestTokenStore_ReloadOnMiss_RemintCoherence` passes **only because it looks up the NEW token first**,
which is what triggers the reload that then evicts the old hash. Then proved on real hardware: the row
marked ← above is a live 200 from a token that the store on disk had already superseded.

This is the `CLAUDE.md` case exactly — a comment asserting an invariant with a test that appears to pin
it and does not. The honest fix is either a test that pins the reversed order, or an eviction that does
not depend on an unrelated lookup missing first.

### 3.3 FINDING — R-268's own stated rotation recipe is incomplete

The register row says the new plaintext *"must also be written into the guest's
`/etc/felhom-bootstrap/bootstrap.json` or the in-guest controller loses its agent access"*. **Writing
bootstrap.json is not sufficient.** `bootstrap.ensureLocalAPI` returns early when
`cfg.LocalAPI.Endpoint != ""` — it fills an *absent* block and never refreshes a present one — so after
a rotation the controller keeps serving the old token from its own `controller.yaml` **across a
restart**. Proved live: two controller restarts, still HTTP 401. The rotation only completed once
`local_api.token` was written into `controller.yaml` itself.

The neighbouring `DetectEndpointDrift` compares the **endpoint** and deliberately does not compare the
token (*"a token mismatch is a different failure"*). So this shape is known to be unmodelled, and R-78
("which file is authoritative") is its parent.

### 3.4 FINDING — the agent-channel alarm never closes, and its own prescribed remedy is why

`channelhealth` alerted correctly and promptly: `agent_channel_unauthorized`, no debounce, pushed to
the hub — *"agent rejected the controller token (HTTP 401) — token stale/rotated (**re-bootstrap**)"*.

The channel then recovered, and **no recovery event was ever sent**. In `Checker.Check`'s UP branch the
notify is guarded by `prev != "" && prev != "up"`, and a controller restart resets `state` to `""`. The
alert's own instruction is to re-bootstrap — i.e. restart the controller — so **following the alarm's
advice structurally guarantees the recovery is silent.** The down side is deliberately asymmetric
(F2: a born-down channel alerts on cycle 1); the up side is not.

Operator-visible result on the hub: two `agent_channel_unauthorized` errors, one `sent` and one
`suppressed` by cooldown, and nothing afterwards. The customer dashboard is fine — `SetDashboard`
reflects current state every cycle. It is the operator's trail that ends on "down".

---

## 4. Gate 0

### 4.1 The recovery code — CONFIRMED IN THE OPERATOR'S HANDS (STOP 1)
Not printed, not logged, not requested. It will be supplied at the screen at STOP 3.

### 4.2 The dataset — planted and fingerprinted

Planted at `/var/lib/felhom/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/` — inside
the **Calibre library**, which is real app-owned customer data and is **demonstrably** carried off-site
(the path appears in every `felhom-offbox,calibre-web` snapshot). Chosen after checking coverage rather
than assuming it: the per-app tier-1 unit carries only the named docker volume, so a file dropped in an
arbitrary place would not have travelled.

| sha256 (16) | bytes | name | name bytes (hex) |
|---|---|---|---|
| `54b773c46bbfd994…` | 3 145 728 | `binary-3mb.bin` | `62696e6172792d336d622e62696e` |
| `52a5c5ebfcac247f…` | 59 | **`árvíztűrő-tükörfúrógép.txt`** | `c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874` |
| `15d2024dfc224162…` | 25 | **`nested/őszibarack.md`** | `6e65737465642fc591737a6962617261636b2e6d64` |
| `924497918e55fe6d…` | 21 | `plain.txt` | `706c61696e2e747874` |

Manifest: `documentation/audits/evidence/` → recorded in the session scratchpad as
`GATE0-before-manifest.json`. The manifest keys on **raw name bytes**, not rendered text.

### 4.3 The positive control — the comparator was watched failing, three ways

| control | expected | result |
|---|---|---|
| identical copy | PASS | **PASS**, exit 0 |
| one bit flipped at byte 1 500 000 of the 3 MB binary | FAIL | **FAIL** — named the file and both sha256s |
| **NFC→NFD rename of the accented name** (renders *identically*) | FAIL | **FAIL** — caught as MISSING+EXTRA on the name bytes |
| one file deleted | FAIL | **FAIL** — named the missing file |

The NFD case is the one that matters: a comparison of *rendered* filenames would have passed it. Every
altered copy was discarded after the control ran.

### 4.4 The off-site tier — has run since the dataset was planted

Triggered through the product's own dashboard button (`POST /backup/offbox/run` → flash *„A távoli
mentés elindult"*), not by invoking restic by hand.

```
[offbox] backup run started (3 app(s) toggled)
[offbox] pre-push dump leg completed in 7.329s — snapshot pair is coherent
[offbox] backed up calibre-web (…/backups/primary/calibre-web, 1 mandatory path(s))
[offbox] backup OK: 3 app(s) backed up, 18 snapshot(s), 1m22s
```

**The restore point, by identity — never by size:**

| snapshot | time (UTC) | tag | paths |
|---|---|---|---|
| **`41c830db`** | 2026-08-09 08:30:38 | `felhom-offbox,calibre-web` | `…/backups/primary/calibre-web` + `…/userdata/media/books` |
| `9e38b84c` | 2026-08-09 08:30:49 | `felhom-offbox,opengist` | `…/backups/primary/opengist` |
| `78b93f04` | 2026-08-09 08:30:53 | `felhom-offbox,privatebin` | `…/backups/primary/privatebin` |

`restic ls 41c830db` confirms all four planted files are present, **both accented filenames included**.

### 4.5 The box, captured
`GATE0-demo-hp-before.txt` (282 lines): agent version, `pveversion`, `pct`/`qm` lists, pool membership,
`pvesm status`, `storage.cfg`, `df`, `lsblk` with serials, the agent unit + service user, sudoers, the
pveum users/tokens/roles/ACL, the guest hook, the WireGuard tunnel, the OOB belt, the agent state dir,
the install state file, PBS snapshots, local vzdump archives, and the enrolled-drive tree.

---

## 5. Findings so far, ranked by what they cost the person in front of you

Nothing yet is rank 1 or 2 — the walk has not started.

### Rank 3 — misleads

- **F-1 · A rotated-out local-API token still authorises** (§3.2). *A customer would do nothing — they
  cannot see it. An operator rotating a leaked token would believe it revoked when it is not.* The
  shipped test passes for the wrong reason.
- **F-2 · R-268's rotation recipe is incomplete** (§3.3). *Whoever follows it gets a box whose agent
  channel is dead across restarts, with a correct-sounding alarm and no hint that `controller.yaml` is
  the file that matters.*
- **F-3 · The agent-channel alarm never closes** (§3.4). *The operator fixes it as instructed and is
  never told it worked; the hub's last word stays "down".*
- **F-4 · Three hub surfaces jointly misrepresent a healthy off-site tier as absent.** The customer page
  reads `Snapshots 0 / Repo Size 0 MB / Integrity Unknown` (that panel renders the **local** tier, while
  the healthy `offsite` object — `snapshot_count: 18`, `last_status: ok` — sits unrendered in the same
  report); the Offsite page shows `0.0 GB` (true, but a 162 KB repo rounds to nothing); and a stale
  `offsite_delivery_stuck` event from 2026-08-07 reads as current. *Three independent surfaces agreeing
  on a wrong picture is how a working backup gets "fixed".* **This one caught me** — it produced a wrong
  statement to the operator at STOP 1, corrected before it changed anything.
- **F-5 · There is no operator-triggerable off-site backup.** The only route is the customer's dashboard
  button; `signed_jobs` needs a signing key the hub does not hold. Sibling of R-177.

### Rank 4 — cosmetic / hygiene

- **F-6 · Leave-behinds from earlier sessions**, found while capturing state:
  - demo-hp host: storage `c11-scratch` (Campaign 11), and `felhom-backup:…vzdump-lxc-9100-2026_08_03…`
    for a guest that no longer exists;
  - demo-hp guest 9201 `/root`: `.dpw` (13 bytes), `.h`, `.sec.html`, all 2026-08-07 16:08;
  - demo-felhom guest 9201: `/root/r204-backup/offbox.orig`.
- **F-7 · The managed-update floor is `0.200.0`, not `0.156.0`** — the runbook's §8.3 premise is stale.
  Confirmed twice: hub `/configuration` reads *"Effective floor v0.200.0 — source: DB (hub_settings);
  env fallback would be v0.120.0"*, and both boxes log `settle-gate: GO — at/above floor 0.200.0`. The
  ruling's **substance** still needs recording; its **number** must not be written down as 0.156.0.

### Not a finding, checked and cleared
- The two `userdata` trees under `sys_drive` are the **same directory** surfaced twice, not a
  backed-up copy beside an unprotected one. Checked against calibre-web's actual bind mounts.

---

## 6. Steps taken off-path, and what they cost

Rule 2 (no shell after the install command returns) does not bind yet — the walk has not begun. These
are Gate 0 preparation acts by the operator, recorded because they are mutations:

1. **R-268 rotation on demo-felhom** — token store appended, `bootstrap.json` and `controller.yaml`
   rewritten, agent and controller restarted. Required by the runbook's pre-phase. *Not the venue.*
2. **demo-hp's dashboard password was re-set to the value in the operator's credentials file.** The
   customer-owned password was not known to this session, and no operator-side route to the off-site
   button exists (F-5). The prior `password_hash` is preserved at `/root/.orighash` and
   `/root/settings.json.rehearsal-orig` in guest 9201; `claimed` was left `true`. **Done on operator
   instruction.** Cost to fidelity: none for the walk — P1 destroys this guest, and P4 re-claims with a
   password the operator chooses.
3. **The off-site run was started by a script pressing the dashboard's own endpoint**, with a real
   session and a real CSRF token, rather than by a person clicking. The server path is identical; only
   the click is synthetic.

---

## 7. P1 — removing Felhom from the host

`--uninstall`, the product's own path, dry-run first. **60 seconds** (08:37:23 → 08:38:23 UTC), exit 0.
The typed-vmid confirmation was satisfied, not bypassed.

### 7.1 The reversal, proved rather than trusted

| component | before | after |
|---|---|---|
| agent binary | `/usr/local/bin/felhom-agent` 0.127.0 | **ABSENT** |
| `felhom-agent` service user | present | **ABSENT** |
| `felhom-op` (OOB) user | present | **ABSENT** |
| `/etc/felhom-sshd` (OOB belt) | present | **ABSENT** |
| systemd units | agent, mgmt-watchdog, felhom-sshd, oob-nft, shared-parent | **all removed** |
| pveum roles | `FelhomAgentBase/Guest/Store` | **none** |
| pveum user + token | `felhom-agent@pve!agent` | **gone**; user list is `root@pam` only |
| pveum ACL | pool + 9 scratch-vmid + 3 storage grants | **empty** |
| `felhom` pool | members `[9201]` | **`[]` — pool removed** |
| guest 9201 | running | **destroyed** |
| guest-hook snippet | present | **removed** |
| dnsmasq snippets | present | **removed** |
| `/var/lib/felhom-agent`, `/var/lib/felhom-install` | present | **removed** |
| **KEPT as declared** | | PBS snapshots (2), local vzdump (4), `/mnt/felhom-drives/Felhom-Share`, storage defs |

Space returned: `local-lvm` 20 904 790 → 12 355 143 KiB (**≈8.5 GiB**); `local` ≈64 MiB; the NVMe
unchanged (backups deliberately kept). `drill-r50` (VM 300) untouched, as predicted.

### 7.2 What it left behind that it did not declare

**F-8 · Five orphaned `agent.json.*` backups survive, each carrying a hub API key and a Proxmox
token.** `/etc/felhom-agent/` remains with `agent.json.campaign8-before`, `.campaign9-before`,
`.campaign9-prev`, `.pre-e-target-move`, `.pre-prunegate.bak` — all 0600, three owned by the **now-deleted
service user's uid 999**, each holding a 64-char `hub.api_key` and a 59-char `proxmox.token`.

The uninstall's own WIPED statement claims it removes *"config (+ its .bak backups)"*, and
`scripts/CHANGELOG` records F1 as *"uninstall now purges the agent config's `.bak*` siblings (one held a
live hub api_key)"*. **That fix does not match the filenames in use** — and note it misses
`agent.json.pre-prunegate.bak`, a file that literally ends in `.bak`.

**Assessed rather than assumed: these particular secrets are SUPERSEDED, not live.** The orphaned
`hub.api_key` hashes to `a5d2222a…`; the hub's current key for demo-hp is `8c59d1b6…`. The Proxmox token
was deleted by this same uninstall. **The exposure is timing-dependent, not absent** — a backup taken
shortly before an uninstall would hold a current key, which is exactly what R-179's precedent records.
The broken thing is the stated guarantee.

**And the reinstall hands them to the new service account.** The rebuilt box recreates `felhom-agent` at
**uid 999 — the same uid the deleted one had** — so the three backups owned by uid 999 become the new
account's files. Verified: `su -s /bin/sh felhom-agent -c 'test -r …'` succeeds on
`agent.json.campaign8-before` and `agent.json.pre-prunegate.bak`. A fresh install's service account
inherits read access to the previous install's credentials. Superseded here; live if the backups are recent.

**F-9 · A stale sudoers file survives — and it is INERT. Hypothesis raised, then refuted by test.**
`/etc/sudoers.d/felhom-agent.bak-pre-e2a` (21 833 bytes, 2026-07-28, 0440 root:root) survives the
uninstall and grants `felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, …`,
and `visudo -c -f` on it says **`parsed OK`**.

The obvious worry — that the reinstall recreates the `felhom-agent` user and the box silently regains a
pre-E-2a privilege set — **was tested against the rebuilt box and is FALSE.** With the user recreated
(uid 999) and this as the *only* felhom file in `/etc/sudoers.d`, `sudo -l -U felhom-agent` returns
**"User felhom-agent is not allowed to run sudo on felhom-host."** sudo ignores files in `sudoers.d`
whose names contain a dot, so `…​.bak-pre-e2a` is never loaded. That is a robust reason, not luck —
`visudo -c -f <file>` checks a file explicitly and therefore does **not** tell you whether sudo loads it.

**What is left is rank 4:** 21 KB of the privilege grammar lingering on a host Felhom was told to leave,
inside a teardown that claims to have removed the sudoers file.

**F-10 · The WireGuard tunnel to Felhom's off-site endpoint stays up.** After an uninstall,
`wg-quick@wg-felhom` is **enabled and active**, `/etc/wireguard/wg-felhom.conf` present, with a live
handshake to `167.233.158.164:443` 52 seconds old and 5.86 GiB received / 2.48 GiB sent on the counter.
It appears in **neither** the WIPED nor the KEPT list. A box told to leave Felhom keeps a live inbound
route into Felhom's infrastructure, and the operator is not told.

**F-11 · Smaller undeclared residue:** `/etc/felhom/` (`.bootstrap-done`, `appliance-pairing-code` —
the console pairing code); `/etc/systemd/system/felhom-bootstrap.service` + `/usr/local/sbin/felhom-bootstrap.sh`;
the `vmbr9` island-bridge stanza (169.254.253.1/30) still in `/etc/network/interfaces`.

**F-12 · A teardown-ordering warning:** `user config - ignore invalid acl role 'FelhomAgentStore'` —
the role is deleted before an ACL that still references it. Harmless here; noise that would mask a real one.

---

## 8. P2 — the preflight, as the owner runs it

### 8.1 The refusals, judged on their messages

**First run, no caps:** `byo mode requires explicit --cores and --memory (noisy-neighbor caps on a host
you do not own)`. **Good** — names the missing thing and the reason.

**Second run:** it refused again —

```
[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
  Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
  (Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)
```

As a *message* this is excellent: the finding, the evidence, two routes out, and an explicit promise
about what it will not do.

### 8.2 F-13 (RANK 1 — STOPS THE VISIT) · Felhom's uninstall leaves the exact condition that makes Felhom's reinstall refuse

The resolver it is refusing over **is Felhom's own**, and the chain is fully evidenced:

| fact | evidence |
|---|---|
| Felhom installed `dnsmasq` | `/var/lib/dpkg/info/dnsmasq.list` dated **2026-07-21 18:24 CEST** — demo-hp's Felhom day-0 |
| it was constrained by a Felhom snippet | `/etc/dnsmasq.d/` held Felhom snippets before; only `README` after |
| **the uninstall restarted it unconstrained** | the running process started **2026-08-09 10:37:39 CEST — inside the uninstall window (10:37:23–10:38:23)** |
| unconstrained, it grabs the wildcard | `0.0.0.0:53` and `[::]:53`, `--local-service`, empty `dnsmasq.conf` |
| it is not PVE's | `/etc/pve/sdn/` empty; unit is stock `/usr/lib/systemd/system/dnsmasq.service`, **enabled** |

The uninstall mentions it only as *"the 'sudo' and 'dnsmasq' packages were left installed (**system
packages**)"* — but dnsmasq is not a system package here; Felhom installed it, and it left it **enabled
and running as a wildcard resolver**.

**What a customer would do next:** read a message that says a resolver is in the way, conclude their own
network setup is at fault, and start debugging DNS they never configured. On a BYO box with no shell
guidance, this is where the visit stops.

**Confirmed by counterfactual:** `systemctl stop dnsmasq && systemctl disable dnsmasq` → `host DNS (:53):
free` → **PRE-FLIGHT PASS**. Nothing else changed. *(Recorded as an intervention — it is the action the
message tells the owner to take.)*

### 8.3 F-14 (RANK 2) · A local golden is adopted with NO version and NO checksum check

Preflight reported `golden (local): local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst`. Step 7 is:

```sh
if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then
    log_skip "  using local golden: $GOLDEN_VOLID"
    _state_mark golden; return 0
fi
```

**The hub manifest's `golden.sha256` — the whole point of the hub vouching a different trust root than
Gitea — is consulted only on the fetch path.** A locally-present archive bypasses the vouch entirely: no
version comparison, no digest, no warning.

**What that archive actually contains, read out of the tar:**
`gitea.dooplex.hu/admin/felhom-controller:`**`0.192.0`**, against a vouched golden of **0.210.0**.

Two consequences, and the second is the serious one:

1. The rebuilt box comes up **18 controller releases stale**, reporting success.
2. **0.192.0 is below 0.200.0 — the release where R-193's off-site recovery SCREEN shipped.** So a
   customer reinstalled today returns on a controller that cannot run the recovery ceremony their data
   depends on. It is also born below the managed-update floor (0.200.0), and the updater's auto-target
   is the floor, never the newest — so it converges to 0.200.0, still not the vouched build.

**This is the R-111/R-115/R-120 drift family, one layer down:** the R-120 gate guards what the operator
may *vouch*; nothing guards what an install actually *takes*. It also compounds with P1 — the uninstall
deliberately keeps the old golden (*"golden vzdump left in place"*), and the reinstall then adopts it.

**The walk continues unflagged** (`--force-gitea-golden` NOT passed), because measuring the path a real
reinstall takes is the point. §9 records which controller version actually came up.

### 8.4 The BYO mutation disclosure — read as a customer

Comprehensive and honest: every pveum artifact, the sudoers file, the state and config paths, the
packages it will install, the guest's capped size, the self-update authority, and — creditably — the
**outbound WireGuard tunnel**, in plain words, with the note that the operator can disable it per
customer. It closes with `NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd.`

**One material omission, given §8.3:** it does not say that a pre-existing local golden may be used
instead of the vouched one. The disclosure lists everything the install will *create* and nothing about
what it may *reuse*.

---

## 9. P3 — the install. IT FAILED, AND THE WALK STOPS HERE

Started 08:51:33 UTC, **failed 08:52:17 UTC (44 s), exit 1, at step 5/8.** Run unflagged, as a real
owner would. The typed hostname acknowledgement was given, not bypassed.

```
[INFO]   manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.210.0
[OK]     verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK]     installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[OK]     created service user felhom-agent
curl: (22) The requested URL returned error: 404
[ERROR] raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)
```

### 9.1 F-15 (RANK 1 — STOPS EVERY INSTALL, FLEET-WIDE) · The vouched agent version was never tagged

**`v0.128.0` does not exist as a git tag** — not locally, not on Gitea. The newest tag is `v0.127.0`.
The *binary* was published to the Gitea package registry (which is why it is selectable in the hub's
artifact dropdown and why its sha256 verified), but the release's tag step never happened.

| ref | HTTP |
|---|---|
| `raw/branch/main/configs/felhom-mkfs-guarded.sh` | **200** |
| `raw/tag/v0.127.0/configs/felhom-mkfs-guarded.sh` | **200** |
| **`raw/tag/v0.128.0/configs/felhom-mkfs-guarded.sh`** | **404** |

The commit that *is* 0.128.0 exists and is on `main`: `28ba8593b8 — v0.128.0 — R-221: the escrow seed is
asserted every tick, not remembered once`. Only the tag is missing.

**This is R-184 happening.** R-184 reads: *"Nothing prevents the hub from vouching an agent version that
was never released. The R-115 gate proves every RELEASED version is installable, but it works from git
tags."* It was filed as a gap. It is now a live outage.

**It is the collision of a good fix with an open gap.** R-183 correctly hardened the installer to pin
config fetches to the vouched tag (`hostinstall_gates.py` even asserts the pin exists —
`ok: fetch_raw pins the agent configs to the vouched agent version`). Nothing asserts that the pinned ref
**resolves**. So the first vouch naming an untagged version fails every install closed.

**The release script already knows.** `felhom-agent/scripts/release-agent.sh:23` says in as many words:
*"`raw/tag/v<version>`, a released version without a git tag 404s a box mid-install, as root."* The
hazard was understood, documented, and the tag step was still missed.

**Blast radius — stated precisely, not inflated:**
- **Every fresh install and every reinstall, on every host, fails at step 5/8** for as long as the Day-0
  manifest vouches agent 0.128.0. That manifest was approved on 2026-08-09; the approval is what armed it.
- **Existing boxes are unaffected.** demo-felhom runs 0.128.0 happily — the self-update path takes the
  binary from the package registry, not from a tag.
- So the damage is confined to the install path. That is also the only path a new customer, or a
  reinstalling tester, ever walks.

**What a customer would do next:** nothing. `curl: (22) … 404` and a question about a git tag is not a
message a household can act on. The visit ends here and a developer is required.

### 9.2 The box's state, left as found for inspection

`--uninstall` is not re-run and no manual repair is attempted (rule 1). demo-hp is mid-install:

```
state.json  completed: [preflight, token, grows, enroll]   mode: byo
agent       0.128.0 installed, unit NOT created (not-found), inactive
user        felhom-agent uid=999 created
pveum       2 users, 3 roles, 63 ACL lines, pool 'felhom' recreated (empty)
guest       none
```

The documented `--resume` path exists and **was deliberately not used: it cannot help.** Resume replays
from `enroll`, hits the same 404, and fails identically — the missing tag is not a transient.

### 9.3 The one-line unblock, for the operator to decide

```sh
cd /mnt/5_hdd/felhom.eu/git/felhom-agent
git tag v0.128.0 28ba8593b8 && git push origin v0.128.0
```

This publishes the tag for a commit that is already on `main` and already released as a package — it
completes a half-finished release rather than changing any code. The alternative, rolling the manifest
back to agent 0.127.0, costs R-221 — the fix `STATUS.md` calls *"the last item blocking a customer from
something we promise them"* — so it is the worse option.

**Not done in this session.** Publishing a release tag is an outward-facing act and the runbook is
explicit: *"If a defect is so severe that the run cannot continue, stop and report; do not fix it here."*

### 9.4 What F-14 still owes

Step 7 never ran, so **whether the install would actually have adopted the stale 0.192.0 golden is not
yet observed** — F-14 rests on the code path plus the preflight's `golden (local): …9100…2026_08_03…`
line and the archive's own `felhom-controller:0.192.0` marker. That is strong, but it is not the
observation. **When the tag is published and the walk resumes, the first thing to record at step 7/8 is
which golden it takes.**

---

## 10. Where the run stands

**Reached: P1 ✓, P2 ✓, P3 ✗ (blocked).** P4–P7 not attempted.

**The honest answer to §2's question is: NOT YET ANSWERABLE, and the reason is itself the answer for
today.** A machine wiped today cannot be set up again at all — not because the data is gone (it is
safe, in snapshot `41c830db` and in two local tiers), but because the install refuses at step 5 of 8 on
a missing git tag. The walk found a hard stop before it ever reached the question about data.

**The dataset is intact and waiting.** Nothing about Gate 0 needs redoing when the walk resumes:
snapshot `41c830db` holds all four files, the before-manifest is committed, and the comparator is proven.