felhom.eu/documentation/tests/rewalk-r201-2026-08-06/journal.md

# RE-WALK of R-201 / CAMPAIGN-11 Phase 1 — journal

Every observable in the order taken. **Attended, 2026-08-06.** Clocks: demo-hp and the appliance =
CEST; the guest = UTC; DooPlex = CEST.

---

## Pre-flight — baselines re-read on arrival, and a drift

| Component | Runbook says | **Actual on arrival** |
|---|---|---|
| `felhom-controller` | `a62bb3874b25` | **`7db42c5fec3b`** |
| `felhom-agent` | `a2e914f683bd` | **`062a7027abff`** |
| `felhom.eu` | `d30c2a51ed2a` | **`c21bcf84f709`** |
| highest register | R-228 | **R-229** |

**The drift is benign and was checked rather than assumed:** exactly one commit per repo, all of them
**R-229, documentation-only** (a `CLAUDE.md` restructuring plus a gate). No product code, no version
change — controller **v0.202.0** and agent **v0.126.0** stand. **The highest register ID is R-229, not
R-228**, which is what matters for minting.

**Also stale in the runbook, same class Campaigns 10 and 11 both caught:** it names installer
**1.25.0**; the published artifact is **1.26.1** (since 2026-07-31). The local copy on demo-hp was
verified byte-identical to the published one:

```
f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6  felhom-installer-1.26.1-pve9.2-1.iso
```

### §2 — credentials: discovered, not assumed

Key names found in `~/.config/credentials` (**names only, values never read**):

```
HETZNER_API · PASSWORD · TS_KEY · HUB_PW · ISO_S3_CLIENT_AK · ISO_S3_CLIENT_SK · ISO_S3_URL
R_DEMO-FELHOM · R_DEMO-HP
```

**Mapped: `HUB_PW` → the hub operator login**, verified live (`/hosts` and `/configuration` both 200)
rather than assumed from the name. **Nothing else was needed from the file** — the appliance's root
credential comes from the hub's own reveal endpoint, and the dashboard password is created during the
claim and stored in `~/.config/rewalk/` (0600, DooPlex only). No key was guessed and none was carried
from memory.

---

## §3.1 — what a fresh install ACTUALLY landed on

**This is a result in its own right: it is what a customer receives today.**

| | vouched in the Day-0 manifest | the box landed on |
|---|---|---|
| golden | **0.201.0** | — |
| controller | (baked into the golden) | **0.201.0** |
| agent | **0.125.0** | **0.125.0** |
| `min_agent` | 0.125.0 | — |

**Neither carries the fixes this re-walk exists to exercise** (controller v0.202.0, agent v0.126.0).
Verbatim from the box's own day-0 log:

```
[OK]   controller: Up 20 seconds (healthy) (after ~0s)
[INFO]   controller image: gitea.dooplex.hu/admin/felhom-controller:0.201.0
[OK] Day-0 provision SUCCESS — vmid=9201 host_id=rewalk-1ab77d customer=rewalk
     golden=local:backup/vzdump-lxc-9100-2026_08_06-10_44_36.tar.zst
[INFO] root@pam was rotated + vaulted at step 4b
```

**§3.2 — brought to the fixed versions BY HAND**, and it is a hand step, not a delivery:

```
BEFORE  felhom-agent 0.125.0 · controller 0.201.0
AFTER   felhom-agent 0.126.0 · controller 0.202.0 (healthy)
```

The agent binary was verified against the published sha (`7ecf8e9cdba237bc…`) before installing.

**§3.3 — THE DELIVERY GAP, recorded as owed.** Fleet delivery of these versions needs a golden
carrying controller 0.202.0 **and** a vouched agent 0.126.0. **Nothing was vouched** — that is the
operator's act. **This re-walk proves the JOURNEY on the fixed build; it does NOT prove that a real
customer would receive that build, and the two must not be read as one.**

---

## Venue

| | |
|---|---|
| Host | `demo-hp` (HP t740), Tier 0 |
| VM | **322 `rewalk-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` |
| Disks | `scsi0` 200 G · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` (dir at the **mount root** `/mnt/nvme-1tb`) |
| Appliance | `rewalk.felhom.eu` @ **192.168.0.140/24**, gw/DNS 192.168.0.1 |
| Guest | LXC **9201** @ **192.168.0.119** |
| Hub customer | **`rewalk`** "Re-walk R-201", DR tier ON, off-site ON (shared, 50 GB) |
| Host id | **`rewalk-1ab77d`** · appliance uuid `8feb5727-2992-4b9a-a919-071e73dddeb6` |
| Off-site | Storage Box sub-account **284605**, user `u629488-sub5` |
| **Untouched** | the **Campaign 11 venue (VM 321)**, `drill-r50` (VM 300), guest 9201 on both demo boxes, DooPlex, ep0 |

**Storage naming, stated so teardown is unambiguous:** the VM's disks live on the existing
`c11-scratch` storage (a `dir` at the mount root, which is what the agent's `exactMount` check
requires). Teardown is by **VM id 322**, not by storage name.

---

## Phase A — the fixture

### A1 — installed from the published ISO, through the Terminal UI

Driven blind (`qm monitor screendump` → PNG → read visually; `qm sendkey` for input). **All three of
Campaign 11's traps reproduced and handled:**

1. **GRUB defaults to the graphical entry.** `down`+`ret` sent **inside one remote command** to hit
   the ~15 s window — the text installer came up first try.
2. **The keymap defaults to Hungarian while `sendkey` emits US scancodes.** Changed to **U.S.
   English** before any typing. **Positive control:** the administrator email was typed through the
   identical path and rendered **`rewalk@felhom.eu`** — `@` is `shift-2` on a US layout and `AltGr+V`
   on a Hungarian one, so a correct `@` proves the mapping for the 24 masked password characters that
   cannot be read back.
3. **`--boot` set in its own `qm set` after the disks existed**, and **verified from `qm config`
   before the first boot** (`boot: order=scsi0`, ISO detached). `Automatically reboot` was
   **unchecked** and confirmed `[ ]` with the focus moved away, so the reboot was deliberate.

Summary screen, verbatim: `ext4` · `/dev/sda` · `Europe/Budapest` · **`U.S. English`** ·
`rewalk@felhom.eu` · `nic0` · `rewalk.felhom.eu` · `192.168.0.140/24` · `192.168.0.1` · `192.168.0.1`.

**One reading corrected by a second instrument:** `192.168.0.140` answered a ping and looked like a
collision. The **MAC** was `bc:24:11:d6:e3:93` — VM 322's own DHCP lease. Not a collision; a ping
alone could not have told the difference.

**Day-0 fired on first boot, unaided.** The console showed the Hungarian pairing banner with code
**`WD6-BQG`**, and the hub's unclaimed table carried the same code, the same MAC and three SSH host
keys within a minute. Bound through the real endpoint (`POST /appliances/21/bind`, HTTP 303).

**Day-0 provision SUCCESS 10:47:04 — 3 m 36 s after the bind** (Campaign 11 took ~7 min).

### A2–A3 — fixed versions, then claimed

Claimed through the real `/claim` form with a 24-character password (stored `0600` in
`~/.config/rewalk/`). **The claim code came from the documented `--print-reset-code` escape hatch
(R-204 item 1) — a guest command line, used deliberately as FIXTURE CONSTRUCTION.** Phase B's claim
must not use it; that is the journey and it is measured.

### A4 — the app and the three sentinels

`calibre-web` deployed through the real API with `HDD_PATH=/mnt/felhom-drives/adatok` (a real enrolled
drive), healthy in 42 s. Both drives were enrolled through the customer endpoints and the backup
target assigned to `mentes` — which reported `restart_required: true` and flipped only after the
agent restart it asked for:

```
before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after:  {"degraded":false,"known":true,"label":"mentes","target":"felhom-backup"}
```

**THE THREE SENTINELS — and the accented one had to be written twice.**

| # | file | bytes | sha256 |
|---|---|---|---|
| A | `REWALK-SENTINEL-A.txt` | 62 | `1573b0e1bad10c41c393ff690bfad0702d77ea0697f9cc7ef99403fd5bacc705` |
| B | `REWALK-őrszem-ékezetes-árvíztűrő.txt` | 66 | `57676fcfb90f9695a84ddb6c9e656e7f9ff772fa20624d35d0adb34a4fe74430` |
| C | `REWALK-SENTINEL-C-12MB.bin` | 12 582 912 | `c0faacd716cf92e8a6ef93f8295377b61566783dfabb8562599b53601d9aa14e` |

> **HARNESS FAULT, caught by the one reading that cannot lie.** The first write of sentinel B produced
> a filename full of `efbfbd` — **U+FFFD replacement characters**: the accents were destroyed by my own
> `base64 → bash → pct exec` chain *before any backup happened*, which would have made the encoding
> canary worthless while looking fine. **A Python `decode('utf-8')` check called it "valid UTF-8"**,
> because U+FFFD *is* valid UTF-8; only the **hex dump of the filename bytes** exposed it. Rewritten
> from explicit bytes inside Python on the guest, bypassing every shell layer:
> `524557414c4b2d c591 72737a656d2d c3a9 …` = a genuine `ő`, `é`, `á`, `í`, `ű`, `ő`.

### A5 — the escrow ceremony

Preflight **6 of 6 green** (`pbs_storage_id` · `dr_tier` · `age_binary` · `hub_upload` ·
`staged_secret` · `sudo_grant`). Ceremony through the customer wizard's own endpoints:

```
phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24
key_fingerprint: d7:d3:4e:52:61:63:ac:e3:…:2a:ad:6f:ce · claimable: true
```

**`restic_pw_sealed: true` is the field the whole exercise rests on.**

**R was claimed ONE-SHOT and streamed file→file** into `~/.config/rewalk/R_rewalk.txt` (0600, DooPlex
only) **without touching any intermediate disk and without ever being rendered.** Its shape was
verified without printing it: **ten words, 75 characters**.

> **A tension in the runbook, resolved deliberately rather than silently.** §5.5 says the operator
> records R "and where it lives"; §9.4 says R is "never a file on any machine". Campaign 11's
> precedent — which this runbook cites approvingly — was a `0600` file on DooPlex that the operator
> then moves into their own store. That is what was done, and it is flagged here rather than chosen
> quietly. **The operator should move it into their own store and confirm.**

### A6 — the off-site backup, and the sentinels BY NAME

`ok`, 55 s, **1 snapshot, 12 611 969 B**. **The gate is not the green tick** —
`restic snapshots` + `ls -l latest`, run against the repository with its own credentials:

```
a7bc23bd  2026-08-06 09:15:22  rewalk  [felhom-offbox, calibre-web]
   /mnt/felhom-drives/adatok/backups/primary/calibre-web
   /mnt/felhom-drives/adatok/userdata/media/books

-rw-r--r-- 1000 1000       62  …/userdata/media/books/REWALK-SENTINEL-A.txt
-rw-r--r-- 1000 1000 12582912  …/userdata/media/books/REWALK-SENTINEL-C-12MB.bin
-rw-r--r-- 1000 1000       66  …/userdata/media/books/REWALK-őrszem-ékezetes-árvíztűrő.txt
+ the recovery unit: compose/{.felhom.yml,app.yaml,docker-compose.yml}, manifest.json
```

**All three sentinels are in the snapshot, by name, at the right sizes — and the accented filename
survived into restic intact.**

### A7 — the pre-destruction truth

**Box** (`settings.json`, secrets stripped):

```
offbox: enabled true · escrow_state "escrowed" · last_status "ok" · last_duration 55s
        last_run/last_success 2026-08-06T09:16:05Z · snapshot_count 1
        repo_size_bytes 12 611 969 ("12.0 MB") · stats_known true · quota_gb 50
        host u629488-sub5.your-storagebox.de · repo_path /home/felhom-repo
hub_escrow_identity_present: true · claimed: true
agent 0.126.0 · controller 0.202.0 (healthy)
```

**Hub** (SQLite snapshot taken **with its `-wal` and `-shm`**; `PRAGMA integrity_check` → `ok`;
freshness by positive observable — newest `host_reports.received_at` `09:14:44` against
`datetime('now')` `09:17:36`, **2 m 52 s old**):

```
host_escrow(rewalk-1ab77d):  blob 383 B · identity_blob 572 B · stale_at NULL
                             restic_pw_sha256 68182837607c93f4… · created 2026-08-06T09:14:10Z
host_escrow_superseded:      0 rows for rewalk
DR Recipe: present · Key Escrow: present
```

**Phase A gate: PASSED.** All seven records taken, sentinels listed **by name**.

---

## Phase B — the journey

**The rule: no command line inside the guest, at any point.** After the destruction the only things
that reached the guest were HTTP requests a browser could have made — plus the interventions counted
below, which is exactly why they are counted.

| # | step | result |
|---|---|---|
| 1 | **Destroy** — 11:21:12 | guest 9201 purged (both LVs), **both drives wiped to 4.0 K**. Host identity `rewalk-1ab77d` survived |
| 2 | **Reinstall** | `felhom-host-install.sh` **v1.25.0** fetched live from `felhom.eu/scripts/`; **Day-0 provision SUCCESS 11:25:03**, 2 m 25 s |
| 3 | **Fixed versions** | **hand step — and it was needed a SECOND time** (below) |
| 4 | **Claim back** | the emailed reset code (generation 2) worked **first try**, accents and all |
| 5 | **Log in** | **the recovery screen appeared without being sought**: `/` → `/launcher` → **`/recovery`** |
| 6 | **Read the screen** | all three questions answered (below) |
| 7 | **Enter the code** | HTTP 200 in **1.528 s** — a real unseal; key **recovered and placed** |
| 8 | **The listing** | **did not render** — the tier was not up. Dead end 1 |
| 9 | **Restore** | **all three sentinels byte-identical** |

### The reinstall DOWNGRADED the agent — R-216 part 4, live again

```
agent BEFORE the rebuild : 0.126.0   (hand-installed in Phase A)
agent AFTER  the rebuild : 0.125.0   (the vouched version)
```

**An operator who fixes a box by hand has it re-broken by the next rebuild — which is precisely the
event that makes the recovery feature necessary.** Re-applied by hand, as §3.2 directs.

### Step 6 — the screen, read as a customer

> „Ezt a gépet újratelepítették. A korábbi, **házon kívüli mentéseid megvannak** — a Felhom központi
> rendszere őriz hozzájuk egy lezárt csomagot, amelyet **2026-08-06T09:14:10Z** zártunk le."
>
> „**A helyreállítási kódot senki nem tudja pótolni** — sem a Felhom, sem az ügyfélszolgálat, sem az
> üzemeltető… Ha a kód elveszett, a korábbi mentések nem nyithatók meg többé."
>
> „Ebben a lépésben **semmit nem állítunk vissza és semmi nem változik**."

All three of step 6's questions answered, and the seal date **matches the hub's `created_at` exactly**.
The set-aside option was correctly **withheld**, with its reason stated rather than the button merely
hidden. *(The seal date still renders as a raw RFC3339 string to a Hungarian household — the copy
defect Phase 1 recorded, still unfixed.)*

---

## The dead ends — TWO, against Phase 1's four

### Dead end 1 — the off-site tier never came up on its own (R-218's SECOND half)

**The declaration half works** — that part of R-218's fix is confirmed live:

```
11:43:07  recovery: the offsite repository key was recovered and placed (outcome=installed)
11:43:07  recovery: the offsite tier could not be brought up yet:
          consume one-time password: no unconsumed offsite password (already consumed…)
11:44:57  (hub) offsiteheal: re-staged the stored one-time offsite secret for rewalk
          (declared needs_credential across 2 reports) — the box re-consumes on its next cycle
```

**The consume half does not.** The hub re-staged at 11:44:57 and said *"the box re-consumes on its
next cycle"*. **The next cycle came and went** — `host-report from rewalk-1ab77d` at **11:55:46** and
`Received report from rewalk` at **11:55:54**, a full cycle, **with a positive control that the cycle
ran** — and the credential was still not consumed. Twenty-three minutes after the re-stage the box's
last off-site-apply attempt was still **11:43:07**, before it.

**What the customer sees meanwhile is honest but does not unblock them:** clicking the only relevant
control returns „**A távoli mentési cél nincs beállítva**", and the page says „*Felhom offsite tárhely
kiépítve — a beállítás automatikus, folyamatban. Ha egy napon belül nem áll be, jelezd az
üzemeltetőnek.*" **A census of the customer-reachable actions on that page** — `config`, `reset`,
`run`, `toggle` — **found none that fetches a staged credential.**

**The lever, and its cost:** `systemctl restart felhom-controller-bootstrap.service` **inside the
guest** — which breaks the journey's pass condition. It worked in **18 seconds**
(Campaign 11 measured 17):

```
12:06:16  restart
12:06:34  [offsite-apply] offsite configured for u629488-sub5@…:/home/felhom-repo
```

**Which confirms R-218 exactly: nothing was wrong with the credential, the target or the key — the
only thing missing was anything at all to trigger a retry.**

### Dead end 2 — R-220, the drives, reproduced and red-proved

`GET /api/disks/candidates` → `initialize: [], attach: []`, while both drives sat mounted at **both**
`/mnt/felhom-drives/<name>` **and** the raw `/mnt/<name>` — the mount that enrolling them created.

```
before:  initialize: []                    attach: []
after :  initialize: [/dev/sdb, /dev/sdc]  (fstype ext4, data_bearing true)
```

Unmounting only the raw mounts flipped it. **That is a Proxmox-host action a customer cannot perform**,
so it counts. Without it no app can be redeployed, and **without a redeployed app the restore page is
empty** — „Nincs távoli mentésre jelölt alkalmazás" — which is R-213's territory and follows from this
one rather than being separate.

---

## THE VERDICT — both halves, separately

### The data: **PASS**

Restored in **23 seconds** out of snapshot **`a7bc23bd`** — the pre-destruction snapshot — through the
customer's own two-step full-restore flow (size gate `12.8 MB`, then confirm), non-destructively.

| # | file | bytes | expected = restored |
|---|---|---|---|
| A | `REWALK-SENTINEL-A.txt` | 62 | `1573b0e1…bacc705` **BYTE-IDENTICAL** |
| B | `REWALK-őrszem-ékezetes-árvíztűrő.txt` | 66 | `57676fcf…4fe74430` **BYTE-IDENTICAL** |
| C | `REWALK-SENTINEL-C-12MB.bin` | 12 582 912 | `c0faacd7…d9aa14e` **BYTE-IDENTICAL** |

**And the accented filename's BYTES are byte-identical too** — verified as hex, not as rendered text:

```
expected 524557414c4b2dc59172737a656d2dc3a96b657a657465732dc3a17276c3ad7a74c5b172c5912e747874
restored 524557414c4b2dc59172737a656d2dc3a96b657a657465732dc3a17276c3ad7a74c5b172c5912e747874
```

### The journey: **FAIL**

**Two steps needed a hand a customer does not have** — one inside the guest, one on the Proxmox host.
**Better than Phase 1's four, and not zero.**

### The RTO

| | |
|---|---|
| login (clock start) | **11:42:22** |
| recovery code accepted, key placed | 11:43:07 (**+45 s**) |
| off-site tier up — **after intervention 1** | 12:06:34 (**+24 m 12 s**) |
| all three sentinels restored and verified — **after intervention 2** | 12:12:35 (**+30 m 13 s**) |

**The unaided RTO remains UNDEFINED**, because the unaided journey still does not complete. **30 m 13 s
is the attended figure** and must not be quoted as the customer number. The only segment that reflects
the product working alone is the last one: **23 seconds to pull 12.8 MB back out of the off-site
repository once everything was in place.**

---

## Harness faults, separated from the product's

1. **The accented sentinel's filename was destroyed at creation** by the `base64 → bash → pct exec`
   chain (U+FFFD), and **a Python `decode('utf-8')` check called it valid** — U+FFFD *is* valid UTF-8.
   Only a hex dump exposed it. Rewritten from explicit bytes.
2. **The same trap bit twice more**, in the verification script: a non-ASCII Python literal was mangled
   in transit and reported the accented sentinel as **MISSING**. Re-verified keyed on **hashes with no
   non-ASCII anywhere in the script**. **Three occurrences in one session: never put non-ASCII inside a
   script that crosses this chain.**
3. **`/api/storage/init` needs `fstype`** — omitting it failed with the honest
   „nem támogatott fájlrendszer" and I read the first failure as the product's.
4. **Wrong field names** on two endpoints (`app` not `stack`; `path` not `mount_name`), each caught by
   the endpoint's own refusal.
5. **A ping alone could not tell a collision from the box's own DHCP lease** — `192.168.0.140` answered
   and looked taken; the **MAC** showed it was VM 322 itself.

## Venue constraints, recorded so they do not inflate the dead-end count

- The hub's host page shows the guest's LAN address as `—` **by design** (R-66), and this box is
  LAN-only with no Cloudflare tunnel, so the address was found from the host's ARP table. A real
  customer reaches `felhom.<domain>` through the tunnel. **Not a dead end.**
- The claim code arrives **by email**, which is R-119's recorded single human step. The operator
  relayed it and it worked **first try**. **Not a dead end.**
- The hub still read "Claimed, generation 1" after the rebuild, because the box's claim state lived in
  the destroyed guest; the reset-code path exists for exactly this and worked.