Files

T

gates / gates (push) Successful in 8s

Details

Re-walk R-201: Phase A record, written BEFORE anything is destroyed

The fixture is built and the gate passed: all three sentinels are in snapshot
a7bc23bd BY NAME and at the right sizes, and the accented filename survived
into restic intact.

Recorded because they are results, not incidentals:
- what a FRESH install actually landed on — controller 0.201.0, agent 0.125.0,
  golden 0.201.0. Neither carries the fixes. Brought to 0.202.0/0.126.0 BY HAND,
  and the delivery gap is recorded as owed: this proves the journey, NOT that a
  customer would receive that build.
- baseline drift caught on arrival: one documentation-only R-229 commit per repo,
  and the highest register ID is R-229, not the runbook's R-228.
- the runbook's installer baseline (1.25.0) is stale; the published artifact is
  1.26.1, verified byte-identical to demo-hp's local copy.
- credentials DISCOVERED not assumed: HUB_PW mapped and verified live; nothing
  else was needed from the file.

A harness fault worth keeping: the accented sentinel's filename was destroyed at
creation by my own base64->bash->pct exec chain, producing U+FFFD replacement
characters. A Python decode('utf-8') check called it VALID UTF-8, because U+FFFD
is valid UTF-8 — only a hex dump of the filename bytes exposed it. Rewritten
from explicit bytes inside Python on the guest.

R claimed one-shot, streamed file->file to a 0600 store on DooPlex, never
rendered, shape verified without printing (10 words, 75 chars). The runbook's
§5.5-vs-§9.4 tension about where R may live is flagged in the journal rather
than resolved silently.

2026-08-06 11:19:19 +02:00

12 KiB

Raw Blame History

RE-WALK of R-201 / CAMPAIGN-11 Phase 1 — journal

Every observable in the order taken. Attended, 2026-08-06. Clocks: demo-hp and the appliance = CEST; the guest = UTC; DooPlex = CEST.

Pre-flight — baselines re-read on arrival, and a drift

Component	Runbook says	Actual on arrival
`felhom-controller`	`a62bb3874b25`	`7db42c5fec3b`
`felhom-agent`	`a2e914f683bd`	`062a7027abff`
`felhom.eu`	`d30c2a51ed2a`	`c21bcf84f709`
highest register	R-228	R-229

The drift is benign and was checked rather than assumed: exactly one commit per repo, all of them R-229, documentation-only (a CLAUDE.md restructuring plus a gate). No product code, no version change — controller v0.202.0 and agent v0.126.0 stand. The highest register ID is R-229, not R-228, which is what matters for minting.

Also stale in the runbook, same class Campaigns 10 and 11 both caught: it names installer 1.25.0; the published artifact is 1.26.1 (since 2026-07-31). The local copy on demo-hp was verified byte-identical to the published one:

f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6  felhom-installer-1.26.1-pve9.2-1.iso

§2 — credentials: discovered, not assumed

Key names found in ~/.config/credentials (names only, values never read):

HETZNER_API · PASSWORD · TS_KEY · HUB_PW · ISO_S3_CLIENT_AK · ISO_S3_CLIENT_SK · ISO_S3_URL
R_DEMO-FELHOM · R_DEMO-HP

Mapped: HUB_PW → the hub operator login, verified live (/hosts and /configuration both 200) rather than assumed from the name. Nothing else was needed from the file — the appliance's root credential comes from the hub's own reveal endpoint, and the dashboard password is created during the claim and stored in ~/.config/rewalk/ (0600, DooPlex only). No key was guessed and none was carried from memory.

§3.1 — what a fresh install ACTUALLY landed on

This is a result in its own right: it is what a customer receives today.

	vouched in the Day-0 manifest	the box landed on
golden	0.201.0	—
controller	(baked into the golden)	0.201.0
agent	0.125.0	0.125.0
`min_agent`	0.125.0	—

Neither carries the fixes this re-walk exists to exercise (controller v0.202.0, agent v0.126.0). Verbatim from the box's own day-0 log:

[OK]   controller: Up 20 seconds (healthy) (after ~0s)
[INFO]   controller image: gitea.dooplex.hu/admin/felhom-controller:0.201.0
[OK] Day-0 provision SUCCESS — vmid=9201 host_id=rewalk-1ab77d customer=rewalk
     golden=local:backup/vzdump-lxc-9100-2026_08_06-10_44_36.tar.zst
[INFO] root@pam was rotated + vaulted at step 4b

§3.2 — brought to the fixed versions BY HAND, and it is a hand step, not a delivery:

BEFORE  felhom-agent 0.125.0 · controller 0.201.0
AFTER   felhom-agent 0.126.0 · controller 0.202.0 (healthy)

The agent binary was verified against the published sha (7ecf8e9cdba237bc…) before installing.

§3.3 — THE DELIVERY GAP, recorded as owed. Fleet delivery of these versions needs a golden carrying controller 0.202.0 and a vouched agent 0.126.0. Nothing was vouched — that is the operator's act. This re-walk proves the JOURNEY on the fixed build; it does NOT prove that a real customer would receive that build, and the two must not be read as one.

Venue


Host	`demo-hp` (HP t740), Tier 0
VM	322 `rewalk-appliance` — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host`
Disks	`scsi0` 200 G · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` (dir at the mount root `/mnt/nvme-1tb`)
Appliance	`rewalk.felhom.eu` @ 192.168.0.140/24, gw/DNS 192.168.0.1
Guest	LXC 9201 @ 192.168.0.119
Hub customer	`rewalk` "Re-walk R-201", DR tier ON, off-site ON (shared, 50 GB)
Host id	`rewalk-1ab77d` · appliance uuid `8feb5727-2992-4b9a-a919-071e73dddeb6`
Off-site	Storage Box sub-account 284605, user `u629488-sub5`
Untouched	the Campaign 11 venue (VM 321), `drill-r50` (VM 300), guest 9201 on both demo boxes, DooPlex, ep0

Storage naming, stated so teardown is unambiguous: the VM's disks live on the existing c11-scratch storage (a dir at the mount root, which is what the agent's exactMount check requires). Teardown is by VM id 322, not by storage name.

Phase A — the fixture

A1 — installed from the published ISO, through the Terminal UI

Driven blind (qm monitor screendump → PNG → read visually; qm sendkey for input). All three of Campaign 11's traps reproduced and handled:

GRUB defaults to the graphical entry. down+ret sent inside one remote command to hit the ~15 s window — the text installer came up first try.
The keymap defaults to Hungarian while sendkey emits US scancodes. Changed to U.S. English before any typing. Positive control: the administrator email was typed through the identical path and rendered rewalk@felhom.eu — @ is shift-2 on a US layout and AltGr+V on a Hungarian one, so a correct @ proves the mapping for the 24 masked password characters that cannot be read back.
--boot set in its own qm set after the disks existed, and verified from qm config before the first boot (boot: order=scsi0, ISO detached). Automatically reboot was unchecked and confirmed [ ] with the focus moved away, so the reboot was deliberate.

Summary screen, verbatim: ext4 · /dev/sda · Europe/Budapest · U.S. English · rewalk@felhom.eu · nic0 · rewalk.felhom.eu · 192.168.0.140/24 · 192.168.0.1 · 192.168.0.1.

One reading corrected by a second instrument: 192.168.0.140 answered a ping and looked like a collision. The MAC was bc:24:11:d6:e3:93 — VM 322's own DHCP lease. Not a collision; a ping alone could not have told the difference.

Day-0 fired on first boot, unaided. The console showed the Hungarian pairing banner with code WD6-BQG, and the hub's unclaimed table carried the same code, the same MAC and three SSH host keys within a minute. Bound through the real endpoint (POST /appliances/21/bind, HTTP 303).

Day-0 provision SUCCESS 10:47:04 — 3 m 36 s after the bind (Campaign 11 took ~7 min).

A2–A3 — fixed versions, then claimed

Claimed through the real /claim form with a 24-character password (stored 0600 in ~/.config/rewalk/). The claim code came from the documented --print-reset-code escape hatch (R-204 item 1) — a guest command line, used deliberately as FIXTURE CONSTRUCTION. Phase B's claim must not use it; that is the journey and it is measured.

A4 — the app and the three sentinels

calibre-web deployed through the real API with HDD_PATH=/mnt/felhom-drives/adatok (a real enrolled drive), healthy in 42 s. Both drives were enrolled through the customer endpoints and the backup target assigned to mentes — which reported restart_required: true and flipped only after the agent restart it asked for:

before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after:  {"degraded":false,"known":true,"label":"mentes","target":"felhom-backup"}

THE THREE SENTINELS — and the accented one had to be written twice.

#	file	bytes	sha256
A	`REWALK-SENTINEL-A.txt`	62	`1573b0e1bad10c41c393ff690bfad0702d77ea0697f9cc7ef99403fd5bacc705`
B	`REWALK-őrszem-ékezetes-árvíztűrő.txt`	66	`57676fcfb90f9695a84ddb6c9e656e7f9ff772fa20624d35d0adb34a4fe74430`
C	`REWALK-SENTINEL-C-12MB.bin`	12 582 912	`c0faacd716cf92e8a6ef93f8295377b61566783dfabb8562599b53601d9aa14e`

HARNESS FAULT, caught by the one reading that cannot lie. The first write of sentinel B produced a filename full of efbfbd — U+FFFD replacement characters: the accents were destroyed by my own base64 → bash → pct exec chain before any backup happened, which would have made the encoding canary worthless while looking fine. A Python decode('utf-8') check called it "valid UTF-8", because U+FFFD is valid UTF-8; only the hex dump of the filename bytes exposed it. Rewritten from explicit bytes inside Python on the guest, bypassing every shell layer: 524557414c4b2d c591 72737a656d2d c3a9 … = a genuine ő, é, á, í, ű, ő.

A5 — the escrow ceremony

Preflight 6 of 6 green (pbs_storage_id · dr_tier · age_binary · hub_upload · staged_secret · sudo_grant). Ceremony through the customer wizard's own endpoints:

phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24
key_fingerprint: d7:d3:4e:52:61:63:ac:e3:…:2a:ad:6f:ce · claimable: true

restic_pw_sealed: true is the field the whole exercise rests on.

R was claimed ONE-SHOT and streamed file→file into ~/.config/rewalk/R_rewalk.txt (0600, DooPlex only) without touching any intermediate disk and without ever being rendered. Its shape was verified without printing it: ten words, 75 characters.

A tension in the runbook, resolved deliberately rather than silently. §5.5 says the operator records R "and where it lives"; §9.4 says R is "never a file on any machine". Campaign 11's precedent — which this runbook cites approvingly — was a 0600 file on DooPlex that the operator then moves into their own store. That is what was done, and it is flagged here rather than chosen quietly. The operator should move it into their own store and confirm.

A6 — the off-site backup, and the sentinels BY NAME

ok, 55 s, 1 snapshot, 12 611 969 B. The gate is not the green tick — restic snapshots + ls -l latest, run against the repository with its own credentials:

a7bc23bd  2026-08-06 09:15:22  rewalk  [felhom-offbox, calibre-web]
   /mnt/felhom-drives/adatok/backups/primary/calibre-web
   /mnt/felhom-drives/adatok/userdata/media/books

-rw-r--r-- 1000 1000       62  …/userdata/media/books/REWALK-SENTINEL-A.txt
-rw-r--r-- 1000 1000 12582912  …/userdata/media/books/REWALK-SENTINEL-C-12MB.bin
-rw-r--r-- 1000 1000       66  …/userdata/media/books/REWALK-őrszem-ékezetes-árvíztűrő.txt
+ the recovery unit: compose/{.felhom.yml,app.yaml,docker-compose.yml}, manifest.json

All three sentinels are in the snapshot, by name, at the right sizes — and the accented filename survived into restic intact.

A7 — the pre-destruction truth

Box (settings.json, secrets stripped):

offbox: enabled true · escrow_state "escrowed" · last_status "ok" · last_duration 55s
        last_run/last_success 2026-08-06T09:16:05Z · snapshot_count 1
        repo_size_bytes 12 611 969 ("12.0 MB") · stats_known true · quota_gb 50
        host u629488-sub5.your-storagebox.de · repo_path /home/felhom-repo
hub_escrow_identity_present: true · claimed: true
agent 0.126.0 · controller 0.202.0 (healthy)

Hub (SQLite snapshot taken with its -wal and -shm; PRAGMA integrity_check → ok; freshness by positive observable — newest host_reports.received_at 09:14:44 against datetime('now') 09:17:36, 2 m 52 s old):

host_escrow(rewalk-1ab77d):  blob 383 B · identity_blob 572 B · stale_at NULL
                             restic_pw_sha256 68182837607c93f4… · created 2026-08-06T09:14:10Z
host_escrow_superseded:      0 rows for rewalk
DR Recipe: present · Key Escrow: present

Phase A gate: PASSED. All seven records taken, sentinels listed by name.

12 KiB Raw Blame History Unescape Escape