# RE-WALK of R-201 / CAMPAIGN-11 Phase 1 — journal Every observable in the order taken. **Attended, 2026-08-06.** Clocks: demo-hp and the appliance = CEST; the guest = UTC; DooPlex = CEST. --- ## Pre-flight — baselines re-read on arrival, and a drift | Component | Runbook says | **Actual on arrival** | |---|---|---| | `felhom-controller` | `a62bb3874b25` | **`7db42c5fec3b`** | | `felhom-agent` | `a2e914f683bd` | **`062a7027abff`** | | `felhom.eu` | `d30c2a51ed2a` | **`c21bcf84f709`** | | highest register | R-228 | **R-229** | **The drift is benign and was checked rather than assumed:** exactly one commit per repo, all of them **R-229, documentation-only** (a `CLAUDE.md` restructuring plus a gate). No product code, no version change — controller **v0.202.0** and agent **v0.126.0** stand. **The highest register ID is R-229, not R-228**, which is what matters for minting. **Also stale in the runbook, same class Campaigns 10 and 11 both caught:** it names installer **1.25.0**; the published artifact is **1.26.1** (since 2026-07-31). The local copy on demo-hp was verified byte-identical to the published one: ``` f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6 felhom-installer-1.26.1-pve9.2-1.iso ``` ### §2 — credentials: discovered, not assumed Key names found in `~/.config/credentials` (**names only, values never read**): ``` HETZNER_API · PASSWORD · TS_KEY · HUB_PW · ISO_S3_CLIENT_AK · ISO_S3_CLIENT_SK · ISO_S3_URL R_DEMO-FELHOM · R_DEMO-HP ``` **Mapped: `HUB_PW` → the hub operator login**, verified live (`/hosts` and `/configuration` both 200) rather than assumed from the name. **Nothing else was needed from the file** — the appliance's root credential comes from the hub's own reveal endpoint, and the dashboard password is created during the claim and stored in `~/.config/rewalk/` (0600, DooPlex only). No key was guessed and none was carried from memory. --- ## §3.1 — what a fresh install ACTUALLY landed on **This is a result in its own right: it is what a customer receives today.** | | vouched in the Day-0 manifest | the box landed on | |---|---|---| | golden | **0.201.0** | — | | controller | (baked into the golden) | **0.201.0** | | agent | **0.125.0** | **0.125.0** | | `min_agent` | 0.125.0 | — | **Neither carries the fixes this re-walk exists to exercise** (controller v0.202.0, agent v0.126.0). Verbatim from the box's own day-0 log: ``` [OK] controller: Up 20 seconds (healthy) (after ~0s) [INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.201.0 [OK] Day-0 provision SUCCESS — vmid=9201 host_id=rewalk-1ab77d customer=rewalk golden=local:backup/vzdump-lxc-9100-2026_08_06-10_44_36.tar.zst [INFO] root@pam was rotated + vaulted at step 4b ``` **§3.2 — brought to the fixed versions BY HAND**, and it is a hand step, not a delivery: ``` BEFORE felhom-agent 0.125.0 · controller 0.201.0 AFTER felhom-agent 0.126.0 · controller 0.202.0 (healthy) ``` The agent binary was verified against the published sha (`7ecf8e9cdba237bc…`) before installing. **§3.3 — THE DELIVERY GAP, recorded as owed.** Fleet delivery of these versions needs a golden carrying controller 0.202.0 **and** a vouched agent 0.126.0. **Nothing was vouched** — that is the operator's act. **This re-walk proves the JOURNEY on the fixed build; it does NOT prove that a real customer would receive that build, and the two must not be read as one.** --- ## Venue | | | |---|---| | Host | `demo-hp` (HP t740), Tier 0 | | VM | **322 `rewalk-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` | | Disks | `scsi0` 200 G · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` (dir at the **mount root** `/mnt/nvme-1tb`) | | Appliance | `rewalk.felhom.eu` @ **192.168.0.140/24**, gw/DNS 192.168.0.1 | | Guest | LXC **9201** @ **192.168.0.119** | | Hub customer | **`rewalk`** "Re-walk R-201", DR tier ON, off-site ON (shared, 50 GB) | | Host id | **`rewalk-1ab77d`** · appliance uuid `8feb5727-2992-4b9a-a919-071e73dddeb6` | | Off-site | Storage Box sub-account **284605**, user `u629488-sub5` | | **Untouched** | the **Campaign 11 venue (VM 321)**, `drill-r50` (VM 300), guest 9201 on both demo boxes, DooPlex, ep0 | **Storage naming, stated so teardown is unambiguous:** the VM's disks live on the existing `c11-scratch` storage (a `dir` at the mount root, which is what the agent's `exactMount` check requires). Teardown is by **VM id 322**, not by storage name. --- ## Phase A — the fixture ### A1 — installed from the published ISO, through the Terminal UI Driven blind (`qm monitor screendump` → PNG → read visually; `qm sendkey` for input). **All three of Campaign 11's traps reproduced and handled:** 1. **GRUB defaults to the graphical entry.** `down`+`ret` sent **inside one remote command** to hit the ~15 s window — the text installer came up first try. 2. **The keymap defaults to Hungarian while `sendkey` emits US scancodes.** Changed to **U.S. English** before any typing. **Positive control:** the administrator email was typed through the identical path and rendered **`rewalk@felhom.eu`** — `@` is `shift-2` on a US layout and `AltGr+V` on a Hungarian one, so a correct `@` proves the mapping for the 24 masked password characters that cannot be read back. 3. **`--boot` set in its own `qm set` after the disks existed**, and **verified from `qm config` before the first boot** (`boot: order=scsi0`, ISO detached). `Automatically reboot` was **unchecked** and confirmed `[ ]` with the focus moved away, so the reboot was deliberate. Summary screen, verbatim: `ext4` · `/dev/sda` · `Europe/Budapest` · **`U.S. English`** · `rewalk@felhom.eu` · `nic0` · `rewalk.felhom.eu` · `192.168.0.140/24` · `192.168.0.1` · `192.168.0.1`. **One reading corrected by a second instrument:** `192.168.0.140` answered a ping and looked like a collision. The **MAC** was `bc:24:11:d6:e3:93` — VM 322's own DHCP lease. Not a collision; a ping alone could not have told the difference. **Day-0 fired on first boot, unaided.** The console showed the Hungarian pairing banner with code **`WD6-BQG`**, and the hub's unclaimed table carried the same code, the same MAC and three SSH host keys within a minute. Bound through the real endpoint (`POST /appliances/21/bind`, HTTP 303). **Day-0 provision SUCCESS 10:47:04 — 3 m 36 s after the bind** (Campaign 11 took ~7 min). ### A2–A3 — fixed versions, then claimed Claimed through the real `/claim` form with a 24-character password (stored `0600` in `~/.config/rewalk/`). **The claim code came from the documented `--print-reset-code` escape hatch (R-204 item 1) — a guest command line, used deliberately as FIXTURE CONSTRUCTION.** Phase B's claim must not use it; that is the journey and it is measured. ### A4 — the app and the three sentinels `calibre-web` deployed through the real API with `HDD_PATH=/mnt/felhom-drives/adatok` (a real enrolled drive), healthy in 42 s. Both drives were enrolled through the customer endpoints and the backup target assigned to `mentes` — which reported `restart_required: true` and flipped only after the agent restart it asked for: ``` before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"} after: {"degraded":false,"known":true,"label":"mentes","target":"felhom-backup"} ``` **THE THREE SENTINELS — and the accented one had to be written twice.** | # | file | bytes | sha256 | |---|---|---|---| | A | `REWALK-SENTINEL-A.txt` | 62 | `1573b0e1bad10c41c393ff690bfad0702d77ea0697f9cc7ef99403fd5bacc705` | | B | `REWALK-őrszem-ékezetes-árvíztűrő.txt` | 66 | `57676fcfb90f9695a84ddb6c9e656e7f9ff772fa20624d35d0adb34a4fe74430` | | C | `REWALK-SENTINEL-C-12MB.bin` | 12 582 912 | `c0faacd716cf92e8a6ef93f8295377b61566783dfabb8562599b53601d9aa14e` | > **HARNESS FAULT, caught by the one reading that cannot lie.** The first write of sentinel B produced > a filename full of `efbfbd` — **U+FFFD replacement characters**: the accents were destroyed by my own > `base64 → bash → pct exec` chain *before any backup happened*, which would have made the encoding > canary worthless while looking fine. **A Python `decode('utf-8')` check called it "valid UTF-8"**, > because U+FFFD *is* valid UTF-8; only the **hex dump of the filename bytes** exposed it. Rewritten > from explicit bytes inside Python on the guest, bypassing every shell layer: > `524557414c4b2d c591 72737a656d2d c3a9 …` = a genuine `ő`, `é`, `á`, `í`, `ű`, `ő`. ### A5 — the escrow ceremony Preflight **6 of 6 green** (`pbs_storage_id` · `dr_tier` · `age_binary` · `hub_upload` · `staged_secret` · `sudo_grant`). Ceremony through the customer wizard's own endpoints: ``` phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24 key_fingerprint: d7:d3:4e:52:61:63:ac:e3:…:2a:ad:6f:ce · claimable: true ``` **`restic_pw_sealed: true` is the field the whole exercise rests on.** **R was claimed ONE-SHOT and streamed file→file** into `~/.config/rewalk/R_rewalk.txt` (0600, DooPlex only) **without touching any intermediate disk and without ever being rendered.** Its shape was verified without printing it: **ten words, 75 characters**. > **A tension in the runbook, resolved deliberately rather than silently.** §5.5 says the operator > records R "and where it lives"; §9.4 says R is "never a file on any machine". Campaign 11's > precedent — which this runbook cites approvingly — was a `0600` file on DooPlex that the operator > then moves into their own store. That is what was done, and it is flagged here rather than chosen > quietly. **The operator should move it into their own store and confirm.** ### A6 — the off-site backup, and the sentinels BY NAME `ok`, 55 s, **1 snapshot, 12 611 969 B**. **The gate is not the green tick** — `restic snapshots` + `ls -l latest`, run against the repository with its own credentials: ``` a7bc23bd 2026-08-06 09:15:22 rewalk [felhom-offbox, calibre-web] /mnt/felhom-drives/adatok/backups/primary/calibre-web /mnt/felhom-drives/adatok/userdata/media/books -rw-r--r-- 1000 1000 62 …/userdata/media/books/REWALK-SENTINEL-A.txt -rw-r--r-- 1000 1000 12582912 …/userdata/media/books/REWALK-SENTINEL-C-12MB.bin -rw-r--r-- 1000 1000 66 …/userdata/media/books/REWALK-őrszem-ékezetes-árvíztűrő.txt + the recovery unit: compose/{.felhom.yml,app.yaml,docker-compose.yml}, manifest.json ``` **All three sentinels are in the snapshot, by name, at the right sizes — and the accented filename survived into restic intact.** ### A7 — the pre-destruction truth **Box** (`settings.json`, secrets stripped): ``` offbox: enabled true · escrow_state "escrowed" · last_status "ok" · last_duration 55s last_run/last_success 2026-08-06T09:16:05Z · snapshot_count 1 repo_size_bytes 12 611 969 ("12.0 MB") · stats_known true · quota_gb 50 host u629488-sub5.your-storagebox.de · repo_path /home/felhom-repo hub_escrow_identity_present: true · claimed: true agent 0.126.0 · controller 0.202.0 (healthy) ``` **Hub** (SQLite snapshot taken **with its `-wal` and `-shm`**; `PRAGMA integrity_check` → `ok`; freshness by positive observable — newest `host_reports.received_at` `09:14:44` against `datetime('now')` `09:17:36`, **2 m 52 s old**): ``` host_escrow(rewalk-1ab77d): blob 383 B · identity_blob 572 B · stale_at NULL restic_pw_sha256 68182837607c93f4… · created 2026-08-06T09:14:10Z host_escrow_superseded: 0 rows for rewalk DR Recipe: present · Key Escrow: present ``` **Phase A gate: PASSED.** All seven records taken, sentinels listed **by name**.