From 6a835ad9ecbfd513c9b6fede3a5d90f50a26d3e1 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 6 Aug 2026 11:19:19 +0200 Subject: [PATCH] Re-walk R-201: Phase A record, written BEFORE anything is destroyed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fixture is built and the gate passed: all three sentinels are in snapshot a7bc23bd BY NAME and at the right sizes, and the accented filename survived into restic intact. Recorded because they are results, not incidentals: - what a FRESH install actually landed on — controller 0.201.0, agent 0.125.0, golden 0.201.0. Neither carries the fixes. Brought to 0.202.0/0.126.0 BY HAND, and the delivery gap is recorded as owed: this proves the journey, NOT that a customer would receive that build. - baseline drift caught on arrival: one documentation-only R-229 commit per repo, and the highest register ID is R-229, not the runbook's R-228. - the runbook's installer baseline (1.25.0) is stale; the published artifact is 1.26.1, verified byte-identical to demo-hp's local copy. - credentials DISCOVERED not assumed: HUB_PW mapped and verified live; nothing else was needed from the file. A harness fault worth keeping: the accented sentinel's filename was destroyed at creation by my own base64->bash->pct exec chain, producing U+FFFD replacement characters. A Python decode('utf-8') check called it VALID UTF-8, because U+FFFD is valid UTF-8 — only a hex dump of the filename bytes exposed it. Rewritten from explicit bytes inside Python on the guest. R claimed one-shot, streamed file->file to a 0600 store on DooPlex, never rendered, shape verified without printing (10 words, 75 chars). The runbook's §5.5-vs-§9.4 tension about where R may live is flagged in the journal rather than resolved silently. --- .../tests/rewalk-r201-2026-08-06/journal.md | 236 ++++++++++++++++++ 1 file changed, 236 insertions(+) create mode 100644 documentation/tests/rewalk-r201-2026-08-06/journal.md diff --git a/documentation/tests/rewalk-r201-2026-08-06/journal.md b/documentation/tests/rewalk-r201-2026-08-06/journal.md new file mode 100644 index 0000000..ec8b24d --- /dev/null +++ b/documentation/tests/rewalk-r201-2026-08-06/journal.md @@ -0,0 +1,236 @@ +# RE-WALK of R-201 / CAMPAIGN-11 Phase 1 — journal + +Every observable in the order taken. **Attended, 2026-08-06.** Clocks: demo-hp and the appliance = +CEST; the guest = UTC; DooPlex = CEST. + +--- + +## Pre-flight — baselines re-read on arrival, and a drift + +| Component | Runbook says | **Actual on arrival** | +|---|---|---| +| `felhom-controller` | `a62bb3874b25` | **`7db42c5fec3b`** | +| `felhom-agent` | `a2e914f683bd` | **`062a7027abff`** | +| `felhom.eu` | `d30c2a51ed2a` | **`c21bcf84f709`** | +| highest register | R-228 | **R-229** | + +**The drift is benign and was checked rather than assumed:** exactly one commit per repo, all of them +**R-229, documentation-only** (a `CLAUDE.md` restructuring plus a gate). No product code, no version +change — controller **v0.202.0** and agent **v0.126.0** stand. **The highest register ID is R-229, not +R-228**, which is what matters for minting. + +**Also stale in the runbook, same class Campaigns 10 and 11 both caught:** it names installer +**1.25.0**; the published artifact is **1.26.1** (since 2026-07-31). The local copy on demo-hp was +verified byte-identical to the published one: + +``` +f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6 felhom-installer-1.26.1-pve9.2-1.iso +``` + +### §2 — credentials: discovered, not assumed + +Key names found in `~/.config/credentials` (**names only, values never read**): + +``` +HETZNER_API · PASSWORD · TS_KEY · HUB_PW · ISO_S3_CLIENT_AK · ISO_S3_CLIENT_SK · ISO_S3_URL +R_DEMO-FELHOM · R_DEMO-HP +``` + +**Mapped: `HUB_PW` → the hub operator login**, verified live (`/hosts` and `/configuration` both 200) +rather than assumed from the name. **Nothing else was needed from the file** — the appliance's root +credential comes from the hub's own reveal endpoint, and the dashboard password is created during the +claim and stored in `~/.config/rewalk/` (0600, DooPlex only). No key was guessed and none was carried +from memory. + +--- + +## §3.1 — what a fresh install ACTUALLY landed on + +**This is a result in its own right: it is what a customer receives today.** + +| | vouched in the Day-0 manifest | the box landed on | +|---|---|---| +| golden | **0.201.0** | — | +| controller | (baked into the golden) | **0.201.0** | +| agent | **0.125.0** | **0.125.0** | +| `min_agent` | 0.125.0 | — | + +**Neither carries the fixes this re-walk exists to exercise** (controller v0.202.0, agent v0.126.0). +Verbatim from the box's own day-0 log: + +``` +[OK] controller: Up 20 seconds (healthy) (after ~0s) +[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.201.0 +[OK] Day-0 provision SUCCESS — vmid=9201 host_id=rewalk-1ab77d customer=rewalk + golden=local:backup/vzdump-lxc-9100-2026_08_06-10_44_36.tar.zst +[INFO] root@pam was rotated + vaulted at step 4b +``` + +**§3.2 — brought to the fixed versions BY HAND**, and it is a hand step, not a delivery: + +``` +BEFORE felhom-agent 0.125.0 · controller 0.201.0 +AFTER felhom-agent 0.126.0 · controller 0.202.0 (healthy) +``` + +The agent binary was verified against the published sha (`7ecf8e9cdba237bc…`) before installing. + +**§3.3 — THE DELIVERY GAP, recorded as owed.** Fleet delivery of these versions needs a golden +carrying controller 0.202.0 **and** a vouched agent 0.126.0. **Nothing was vouched** — that is the +operator's act. **This re-walk proves the JOURNEY on the fixed build; it does NOT prove that a real +customer would receive that build, and the two must not be read as one.** + +--- + +## Venue + +| | | +|---|---| +| Host | `demo-hp` (HP t740), Tier 0 | +| VM | **322 `rewalk-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` | +| Disks | `scsi0` 200 G · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` (dir at the **mount root** `/mnt/nvme-1tb`) | +| Appliance | `rewalk.felhom.eu` @ **192.168.0.140/24**, gw/DNS 192.168.0.1 | +| Guest | LXC **9201** @ **192.168.0.119** | +| Hub customer | **`rewalk`** "Re-walk R-201", DR tier ON, off-site ON (shared, 50 GB) | +| Host id | **`rewalk-1ab77d`** · appliance uuid `8feb5727-2992-4b9a-a919-071e73dddeb6` | +| Off-site | Storage Box sub-account **284605**, user `u629488-sub5` | +| **Untouched** | the **Campaign 11 venue (VM 321)**, `drill-r50` (VM 300), guest 9201 on both demo boxes, DooPlex, ep0 | + +**Storage naming, stated so teardown is unambiguous:** the VM's disks live on the existing +`c11-scratch` storage (a `dir` at the mount root, which is what the agent's `exactMount` check +requires). Teardown is by **VM id 322**, not by storage name. + +--- + +## Phase A — the fixture + +### A1 — installed from the published ISO, through the Terminal UI + +Driven blind (`qm monitor screendump` → PNG → read visually; `qm sendkey` for input). **All three of +Campaign 11's traps reproduced and handled:** + +1. **GRUB defaults to the graphical entry.** `down`+`ret` sent **inside one remote command** to hit + the ~15 s window — the text installer came up first try. +2. **The keymap defaults to Hungarian while `sendkey` emits US scancodes.** Changed to **U.S. + English** before any typing. **Positive control:** the administrator email was typed through the + identical path and rendered **`rewalk@felhom.eu`** — `@` is `shift-2` on a US layout and `AltGr+V` + on a Hungarian one, so a correct `@` proves the mapping for the 24 masked password characters that + cannot be read back. +3. **`--boot` set in its own `qm set` after the disks existed**, and **verified from `qm config` + before the first boot** (`boot: order=scsi0`, ISO detached). `Automatically reboot` was + **unchecked** and confirmed `[ ]` with the focus moved away, so the reboot was deliberate. + +Summary screen, verbatim: `ext4` · `/dev/sda` · `Europe/Budapest` · **`U.S. English`** · +`rewalk@felhom.eu` · `nic0` · `rewalk.felhom.eu` · `192.168.0.140/24` · `192.168.0.1` · `192.168.0.1`. + +**One reading corrected by a second instrument:** `192.168.0.140` answered a ping and looked like a +collision. The **MAC** was `bc:24:11:d6:e3:93` — VM 322's own DHCP lease. Not a collision; a ping +alone could not have told the difference. + +**Day-0 fired on first boot, unaided.** The console showed the Hungarian pairing banner with code +**`WD6-BQG`**, and the hub's unclaimed table carried the same code, the same MAC and three SSH host +keys within a minute. Bound through the real endpoint (`POST /appliances/21/bind`, HTTP 303). + +**Day-0 provision SUCCESS 10:47:04 — 3 m 36 s after the bind** (Campaign 11 took ~7 min). + +### A2–A3 — fixed versions, then claimed + +Claimed through the real `/claim` form with a 24-character password (stored `0600` in +`~/.config/rewalk/`). **The claim code came from the documented `--print-reset-code` escape hatch +(R-204 item 1) — a guest command line, used deliberately as FIXTURE CONSTRUCTION.** Phase B's claim +must not use it; that is the journey and it is measured. + +### A4 — the app and the three sentinels + +`calibre-web` deployed through the real API with `HDD_PATH=/mnt/felhom-drives/adatok` (a real enrolled +drive), healthy in 42 s. Both drives were enrolled through the customer endpoints and the backup +target assigned to `mentes` — which reported `restart_required: true` and flipped only after the +agent restart it asked for: + +``` +before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"} +after: {"degraded":false,"known":true,"label":"mentes","target":"felhom-backup"} +``` + +**THE THREE SENTINELS — and the accented one had to be written twice.** + +| # | file | bytes | sha256 | +|---|---|---|---| +| A | `REWALK-SENTINEL-A.txt` | 62 | `1573b0e1bad10c41c393ff690bfad0702d77ea0697f9cc7ef99403fd5bacc705` | +| B | `REWALK-őrszem-ékezetes-árvíztűrő.txt` | 66 | `57676fcfb90f9695a84ddb6c9e656e7f9ff772fa20624d35d0adb34a4fe74430` | +| C | `REWALK-SENTINEL-C-12MB.bin` | 12 582 912 | `c0faacd716cf92e8a6ef93f8295377b61566783dfabb8562599b53601d9aa14e` | + +> **HARNESS FAULT, caught by the one reading that cannot lie.** The first write of sentinel B produced +> a filename full of `efbfbd` — **U+FFFD replacement characters**: the accents were destroyed by my own +> `base64 → bash → pct exec` chain *before any backup happened*, which would have made the encoding +> canary worthless while looking fine. **A Python `decode('utf-8')` check called it "valid UTF-8"**, +> because U+FFFD *is* valid UTF-8; only the **hex dump of the filename bytes** exposed it. Rewritten +> from explicit bytes inside Python on the guest, bypassing every shell layer: +> `524557414c4b2d c591 72737a656d2d c3a9 …` = a genuine `ő`, `é`, `á`, `í`, `ű`, `ő`. + +### A5 — the escrow ceremony + +Preflight **6 of 6 green** (`pbs_storage_id` · `dr_tier` · `age_binary` · `hub_upload` · +`staged_secret` · `sudo_grant`). Ceremony through the customer wizard's own endpoints: + +``` +phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24 +key_fingerprint: d7:d3:4e:52:61:63:ac:e3:…:2a:ad:6f:ce · claimable: true +``` + +**`restic_pw_sealed: true` is the field the whole exercise rests on.** + +**R was claimed ONE-SHOT and streamed file→file** into `~/.config/rewalk/R_rewalk.txt` (0600, DooPlex +only) **without touching any intermediate disk and without ever being rendered.** Its shape was +verified without printing it: **ten words, 75 characters**. + +> **A tension in the runbook, resolved deliberately rather than silently.** §5.5 says the operator +> records R "and where it lives"; §9.4 says R is "never a file on any machine". Campaign 11's +> precedent — which this runbook cites approvingly — was a `0600` file on DooPlex that the operator +> then moves into their own store. That is what was done, and it is flagged here rather than chosen +> quietly. **The operator should move it into their own store and confirm.** + +### A6 — the off-site backup, and the sentinels BY NAME + +`ok`, 55 s, **1 snapshot, 12 611 969 B**. **The gate is not the green tick** — +`restic snapshots` + `ls -l latest`, run against the repository with its own credentials: + +``` +a7bc23bd 2026-08-06 09:15:22 rewalk [felhom-offbox, calibre-web] + /mnt/felhom-drives/adatok/backups/primary/calibre-web + /mnt/felhom-drives/adatok/userdata/media/books + +-rw-r--r-- 1000 1000 62 …/userdata/media/books/REWALK-SENTINEL-A.txt +-rw-r--r-- 1000 1000 12582912 …/userdata/media/books/REWALK-SENTINEL-C-12MB.bin +-rw-r--r-- 1000 1000 66 …/userdata/media/books/REWALK-őrszem-ékezetes-árvíztűrő.txt ++ the recovery unit: compose/{.felhom.yml,app.yaml,docker-compose.yml}, manifest.json +``` + +**All three sentinels are in the snapshot, by name, at the right sizes — and the accented filename +survived into restic intact.** + +### A7 — the pre-destruction truth + +**Box** (`settings.json`, secrets stripped): + +``` +offbox: enabled true · escrow_state "escrowed" · last_status "ok" · last_duration 55s + last_run/last_success 2026-08-06T09:16:05Z · snapshot_count 1 + repo_size_bytes 12 611 969 ("12.0 MB") · stats_known true · quota_gb 50 + host u629488-sub5.your-storagebox.de · repo_path /home/felhom-repo +hub_escrow_identity_present: true · claimed: true +agent 0.126.0 · controller 0.202.0 (healthy) +``` + +**Hub** (SQLite snapshot taken **with its `-wal` and `-shm`**; `PRAGMA integrity_check` → `ok`; +freshness by positive observable — newest `host_reports.received_at` `09:14:44` against +`datetime('now')` `09:17:36`, **2 m 52 s old**): + +``` +host_escrow(rewalk-1ab77d): blob 383 B · identity_blob 572 B · stale_at NULL + restic_pw_sha256 68182837607c93f4… · created 2026-08-06T09:14:10Z +host_escrow_superseded: 0 rows for rewalk +DR Recipe: present · Key Escrow: present +``` + +**Phase A gate: PASSED.** All seven records taken, sentinels listed **by name**.