admin
6a835ad9ec
gates / gates (push) Successful in 8s
The fixture is built and the gate passed: all three sentinels are in snapshot
a7bc23bd BY NAME and at the right sizes, and the accented filename survived
into restic intact.
Recorded because they are results, not incidentals:
- what a FRESH install actually landed on — controller 0.201.0, agent 0.125.0,
golden 0.201.0. Neither carries the fixes. Brought to 0.202.0/0.126.0 BY HAND,
and the delivery gap is recorded as owed: this proves the journey, NOT that a
customer would receive that build.
- baseline drift caught on arrival: one documentation-only R-229 commit per repo,
and the highest register ID is R-229, not the runbook's R-228.
- the runbook's installer baseline (1.25.0) is stale; the published artifact is
1.26.1, verified byte-identical to demo-hp's local copy.
- credentials DISCOVERED not assumed: HUB_PW mapped and verified live; nothing
else was needed from the file.
A harness fault worth keeping: the accented sentinel's filename was destroyed at
creation by my own base64->bash->pct exec chain, producing U+FFFD replacement
characters. A Python decode('utf-8') check called it VALID UTF-8, because U+FFFD
is valid UTF-8 — only a hex dump of the filename bytes exposed it. Rewritten
from explicit bytes inside Python on the guest.
R claimed one-shot, streamed file->file to a 0600 store on DooPlex, never
rendered, shape verified without printing (10 words, 75 chars). The runbook's
§5.5-vs-§9.4 tension about where R may live is flagged in the journal rather
than resolved silently.
237 lines
12 KiB
Markdown
237 lines
12 KiB
Markdown
# RE-WALK of R-201 / CAMPAIGN-11 Phase 1 — journal
|
||
|
||
Every observable in the order taken. **Attended, 2026-08-06.** Clocks: demo-hp and the appliance =
|
||
CEST; the guest = UTC; DooPlex = CEST.
|
||
|
||
---
|
||
|
||
## Pre-flight — baselines re-read on arrival, and a drift
|
||
|
||
| Component | Runbook says | **Actual on arrival** |
|
||
|---|---|---|
|
||
| `felhom-controller` | `a62bb3874b25` | **`7db42c5fec3b`** |
|
||
| `felhom-agent` | `a2e914f683bd` | **`062a7027abff`** |
|
||
| `felhom.eu` | `d30c2a51ed2a` | **`c21bcf84f709`** |
|
||
| highest register | R-228 | **R-229** |
|
||
|
||
**The drift is benign and was checked rather than assumed:** exactly one commit per repo, all of them
|
||
**R-229, documentation-only** (a `CLAUDE.md` restructuring plus a gate). No product code, no version
|
||
change — controller **v0.202.0** and agent **v0.126.0** stand. **The highest register ID is R-229, not
|
||
R-228**, which is what matters for minting.
|
||
|
||
**Also stale in the runbook, same class Campaigns 10 and 11 both caught:** it names installer
|
||
**1.25.0**; the published artifact is **1.26.1** (since 2026-07-31). The local copy on demo-hp was
|
||
verified byte-identical to the published one:
|
||
|
||
```
|
||
f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6 felhom-installer-1.26.1-pve9.2-1.iso
|
||
```
|
||
|
||
### §2 — credentials: discovered, not assumed
|
||
|
||
Key names found in `~/.config/credentials` (**names only, values never read**):
|
||
|
||
```
|
||
HETZNER_API · PASSWORD · TS_KEY · HUB_PW · ISO_S3_CLIENT_AK · ISO_S3_CLIENT_SK · ISO_S3_URL
|
||
R_DEMO-FELHOM · R_DEMO-HP
|
||
```
|
||
|
||
**Mapped: `HUB_PW` → the hub operator login**, verified live (`/hosts` and `/configuration` both 200)
|
||
rather than assumed from the name. **Nothing else was needed from the file** — the appliance's root
|
||
credential comes from the hub's own reveal endpoint, and the dashboard password is created during the
|
||
claim and stored in `~/.config/rewalk/` (0600, DooPlex only). No key was guessed and none was carried
|
||
from memory.
|
||
|
||
---
|
||
|
||
## §3.1 — what a fresh install ACTUALLY landed on
|
||
|
||
**This is a result in its own right: it is what a customer receives today.**
|
||
|
||
| | vouched in the Day-0 manifest | the box landed on |
|
||
|---|---|---|
|
||
| golden | **0.201.0** | — |
|
||
| controller | (baked into the golden) | **0.201.0** |
|
||
| agent | **0.125.0** | **0.125.0** |
|
||
| `min_agent` | 0.125.0 | — |
|
||
|
||
**Neither carries the fixes this re-walk exists to exercise** (controller v0.202.0, agent v0.126.0).
|
||
Verbatim from the box's own day-0 log:
|
||
|
||
```
|
||
[OK] controller: Up 20 seconds (healthy) (after ~0s)
|
||
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.201.0
|
||
[OK] Day-0 provision SUCCESS — vmid=9201 host_id=rewalk-1ab77d customer=rewalk
|
||
golden=local:backup/vzdump-lxc-9100-2026_08_06-10_44_36.tar.zst
|
||
[INFO] root@pam was rotated + vaulted at step 4b
|
||
```
|
||
|
||
**§3.2 — brought to the fixed versions BY HAND**, and it is a hand step, not a delivery:
|
||
|
||
```
|
||
BEFORE felhom-agent 0.125.0 · controller 0.201.0
|
||
AFTER felhom-agent 0.126.0 · controller 0.202.0 (healthy)
|
||
```
|
||
|
||
The agent binary was verified against the published sha (`7ecf8e9cdba237bc…`) before installing.
|
||
|
||
**§3.3 — THE DELIVERY GAP, recorded as owed.** Fleet delivery of these versions needs a golden
|
||
carrying controller 0.202.0 **and** a vouched agent 0.126.0. **Nothing was vouched** — that is the
|
||
operator's act. **This re-walk proves the JOURNEY on the fixed build; it does NOT prove that a real
|
||
customer would receive that build, and the two must not be read as one.**
|
||
|
||
---
|
||
|
||
## Venue
|
||
|
||
| | |
|
||
|---|---|
|
||
| Host | `demo-hp` (HP t740), Tier 0 |
|
||
| VM | **322 `rewalk-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` |
|
||
| Disks | `scsi0` 200 G · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` (dir at the **mount root** `/mnt/nvme-1tb`) |
|
||
| Appliance | `rewalk.felhom.eu` @ **192.168.0.140/24**, gw/DNS 192.168.0.1 |
|
||
| Guest | LXC **9201** @ **192.168.0.119** |
|
||
| Hub customer | **`rewalk`** "Re-walk R-201", DR tier ON, off-site ON (shared, 50 GB) |
|
||
| Host id | **`rewalk-1ab77d`** · appliance uuid `8feb5727-2992-4b9a-a919-071e73dddeb6` |
|
||
| Off-site | Storage Box sub-account **284605**, user `u629488-sub5` |
|
||
| **Untouched** | the **Campaign 11 venue (VM 321)**, `drill-r50` (VM 300), guest 9201 on both demo boxes, DooPlex, ep0 |
|
||
|
||
**Storage naming, stated so teardown is unambiguous:** the VM's disks live on the existing
|
||
`c11-scratch` storage (a `dir` at the mount root, which is what the agent's `exactMount` check
|
||
requires). Teardown is by **VM id 322**, not by storage name.
|
||
|
||
---
|
||
|
||
## Phase A — the fixture
|
||
|
||
### A1 — installed from the published ISO, through the Terminal UI
|
||
|
||
Driven blind (`qm monitor screendump` → PNG → read visually; `qm sendkey` for input). **All three of
|
||
Campaign 11's traps reproduced and handled:**
|
||
|
||
1. **GRUB defaults to the graphical entry.** `down`+`ret` sent **inside one remote command** to hit
|
||
the ~15 s window — the text installer came up first try.
|
||
2. **The keymap defaults to Hungarian while `sendkey` emits US scancodes.** Changed to **U.S.
|
||
English** before any typing. **Positive control:** the administrator email was typed through the
|
||
identical path and rendered **`rewalk@felhom.eu`** — `@` is `shift-2` on a US layout and `AltGr+V`
|
||
on a Hungarian one, so a correct `@` proves the mapping for the 24 masked password characters that
|
||
cannot be read back.
|
||
3. **`--boot` set in its own `qm set` after the disks existed**, and **verified from `qm config`
|
||
before the first boot** (`boot: order=scsi0`, ISO detached). `Automatically reboot` was
|
||
**unchecked** and confirmed `[ ]` with the focus moved away, so the reboot was deliberate.
|
||
|
||
Summary screen, verbatim: `ext4` · `/dev/sda` · `Europe/Budapest` · **`U.S. English`** ·
|
||
`rewalk@felhom.eu` · `nic0` · `rewalk.felhom.eu` · `192.168.0.140/24` · `192.168.0.1` · `192.168.0.1`.
|
||
|
||
**One reading corrected by a second instrument:** `192.168.0.140` answered a ping and looked like a
|
||
collision. The **MAC** was `bc:24:11:d6:e3:93` — VM 322's own DHCP lease. Not a collision; a ping
|
||
alone could not have told the difference.
|
||
|
||
**Day-0 fired on first boot, unaided.** The console showed the Hungarian pairing banner with code
|
||
**`WD6-BQG`**, and the hub's unclaimed table carried the same code, the same MAC and three SSH host
|
||
keys within a minute. Bound through the real endpoint (`POST /appliances/21/bind`, HTTP 303).
|
||
|
||
**Day-0 provision SUCCESS 10:47:04 — 3 m 36 s after the bind** (Campaign 11 took ~7 min).
|
||
|
||
### A2–A3 — fixed versions, then claimed
|
||
|
||
Claimed through the real `/claim` form with a 24-character password (stored `0600` in
|
||
`~/.config/rewalk/`). **The claim code came from the documented `--print-reset-code` escape hatch
|
||
(R-204 item 1) — a guest command line, used deliberately as FIXTURE CONSTRUCTION.** Phase B's claim
|
||
must not use it; that is the journey and it is measured.
|
||
|
||
### A4 — the app and the three sentinels
|
||
|
||
`calibre-web` deployed through the real API with `HDD_PATH=/mnt/felhom-drives/adatok` (a real enrolled
|
||
drive), healthy in 42 s. Both drives were enrolled through the customer endpoints and the backup
|
||
target assigned to `mentes` — which reported `restart_required: true` and flipped only after the
|
||
agent restart it asked for:
|
||
|
||
```
|
||
before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"}
|
||
after: {"degraded":false,"known":true,"label":"mentes","target":"felhom-backup"}
|
||
```
|
||
|
||
**THE THREE SENTINELS — and the accented one had to be written twice.**
|
||
|
||
| # | file | bytes | sha256 |
|
||
|---|---|---|---|
|
||
| A | `REWALK-SENTINEL-A.txt` | 62 | `1573b0e1bad10c41c393ff690bfad0702d77ea0697f9cc7ef99403fd5bacc705` |
|
||
| B | `REWALK-őrszem-ékezetes-árvíztűrő.txt` | 66 | `57676fcfb90f9695a84ddb6c9e656e7f9ff772fa20624d35d0adb34a4fe74430` |
|
||
| C | `REWALK-SENTINEL-C-12MB.bin` | 12 582 912 | `c0faacd716cf92e8a6ef93f8295377b61566783dfabb8562599b53601d9aa14e` |
|
||
|
||
> **HARNESS FAULT, caught by the one reading that cannot lie.** The first write of sentinel B produced
|
||
> a filename full of `efbfbd` — **U+FFFD replacement characters**: the accents were destroyed by my own
|
||
> `base64 → bash → pct exec` chain *before any backup happened*, which would have made the encoding
|
||
> canary worthless while looking fine. **A Python `decode('utf-8')` check called it "valid UTF-8"**,
|
||
> because U+FFFD *is* valid UTF-8; only the **hex dump of the filename bytes** exposed it. Rewritten
|
||
> from explicit bytes inside Python on the guest, bypassing every shell layer:
|
||
> `524557414c4b2d c591 72737a656d2d c3a9 …` = a genuine `ő`, `é`, `á`, `í`, `ű`, `ő`.
|
||
|
||
### A5 — the escrow ceremony
|
||
|
||
Preflight **6 of 6 green** (`pbs_storage_id` · `dr_tier` · `age_binary` · `hub_upload` ·
|
||
`staged_secret` · `sudo_grant`). Ceremony through the customer wizard's own endpoints:
|
||
|
||
```
|
||
phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24
|
||
key_fingerprint: d7:d3:4e:52:61:63:ac:e3:…:2a:ad:6f:ce · claimable: true
|
||
```
|
||
|
||
**`restic_pw_sealed: true` is the field the whole exercise rests on.**
|
||
|
||
**R was claimed ONE-SHOT and streamed file→file** into `~/.config/rewalk/R_rewalk.txt` (0600, DooPlex
|
||
only) **without touching any intermediate disk and without ever being rendered.** Its shape was
|
||
verified without printing it: **ten words, 75 characters**.
|
||
|
||
> **A tension in the runbook, resolved deliberately rather than silently.** §5.5 says the operator
|
||
> records R "and where it lives"; §9.4 says R is "never a file on any machine". Campaign 11's
|
||
> precedent — which this runbook cites approvingly — was a `0600` file on DooPlex that the operator
|
||
> then moves into their own store. That is what was done, and it is flagged here rather than chosen
|
||
> quietly. **The operator should move it into their own store and confirm.**
|
||
|
||
### A6 — the off-site backup, and the sentinels BY NAME
|
||
|
||
`ok`, 55 s, **1 snapshot, 12 611 969 B**. **The gate is not the green tick** —
|
||
`restic snapshots` + `ls -l latest`, run against the repository with its own credentials:
|
||
|
||
```
|
||
a7bc23bd 2026-08-06 09:15:22 rewalk [felhom-offbox, calibre-web]
|
||
/mnt/felhom-drives/adatok/backups/primary/calibre-web
|
||
/mnt/felhom-drives/adatok/userdata/media/books
|
||
|
||
-rw-r--r-- 1000 1000 62 …/userdata/media/books/REWALK-SENTINEL-A.txt
|
||
-rw-r--r-- 1000 1000 12582912 …/userdata/media/books/REWALK-SENTINEL-C-12MB.bin
|
||
-rw-r--r-- 1000 1000 66 …/userdata/media/books/REWALK-őrszem-ékezetes-árvíztűrő.txt
|
||
+ the recovery unit: compose/{.felhom.yml,app.yaml,docker-compose.yml}, manifest.json
|
||
```
|
||
|
||
**All three sentinels are in the snapshot, by name, at the right sizes — and the accented filename
|
||
survived into restic intact.**
|
||
|
||
### A7 — the pre-destruction truth
|
||
|
||
**Box** (`settings.json`, secrets stripped):
|
||
|
||
```
|
||
offbox: enabled true · escrow_state "escrowed" · last_status "ok" · last_duration 55s
|
||
last_run/last_success 2026-08-06T09:16:05Z · snapshot_count 1
|
||
repo_size_bytes 12 611 969 ("12.0 MB") · stats_known true · quota_gb 50
|
||
host u629488-sub5.your-storagebox.de · repo_path /home/felhom-repo
|
||
hub_escrow_identity_present: true · claimed: true
|
||
agent 0.126.0 · controller 0.202.0 (healthy)
|
||
```
|
||
|
||
**Hub** (SQLite snapshot taken **with its `-wal` and `-shm`**; `PRAGMA integrity_check` → `ok`;
|
||
freshness by positive observable — newest `host_reports.received_at` `09:14:44` against
|
||
`datetime('now')` `09:17:36`, **2 m 52 s old**):
|
||
|
||
```
|
||
host_escrow(rewalk-1ab77d): blob 383 B · identity_blob 572 B · stale_at NULL
|
||
restic_pw_sha256 68182837607c93f4… · created 2026-08-06T09:14:10Z
|
||
host_escrow_superseded: 0 rows for rewalk
|
||
DR Recipe: present · Key Escrow: present
|
||
```
|
||
|
||
**Phase A gate: PASSED.** All seven records taken, sentinels listed **by name**.
|