felhom.eu/documentation/tests/part4-rewalk-2026-08-06/journal.md

PART 4 — the re-walk on a real installation, with no hand upgrade (2026-08-06)

Task 4 Part 4. Half 1 (the fixture) is complete and the venue is STANDING. Nothing has been destroyed; half 2 (destroy → reinstall → recover → compare) is owed and deliberately not started.

The question this venue exists to answer: does a box built from the published artifacts get R-218's and R-220's fixes without anyone touching it? The previous re-walk could not ask that — its reinstall downgraded the agent, because the vouched agent was older than the box needed.

The venue

Host	demo-hp (HP t740), Tier 0 — VM 323 part4-appliance (321 and 322 untouched)
Appliance	part4.felhom.eu @ 192.168.0.141, q35/OVMF, 4 cores, 8 GB
Disks	scsi0 200 G · scsi1 50 G · scsi2 50 G, qcow2 on c11-scratch
Guest	LXC 9201 part4, island 169.254.253.2, LAN 192.168.0.228
Hub customer	part4 "Part 4 - no hand upgrade" · host id part4-905bdb · appliance uuid 79fc7b81-…
Off-site	Storage Box sub-account, user u629488-sub6, /home/felhom-repo, 50 GB
Teardown	owed — VM 323, the hub customer, and the off-site repo. Teardown is by VM id 323.

Routing fact worth recording: DooPlex cannot reach either the appliance or the guest — no ping, no port. demo-hp can. Every customer-side call in this journal went through demo-hp as a jump host. The guest is born-on-island (R-50) and has no route from the workspace.

The vouch that made this test possible

Before the walk, three fields moved together (never one — golden_version alone ships a controller onto an older agent than it declares it needs):

field	before	after
golden_version	0.201.0	0.203.0
agent_version	0.125.0	0.127.0
min_agent	0.125.0	0.127.0

Rollback values are recorded in the scratch ROLLBACK-artifacts.txt; vouching is reversible by re-selecting them and saving — no rebuild, because a bake never deletes the previous package.

A consequence that must not be glossed: raising min_agent to 0.127.0 newly holds the controller floor on three boxes that were not held before — c11-36d660 and rewalk-1ab77d (agent 0.126.0) and demo-felhom-8363b5 (0.125.0). That is the R-216 machinery working as designed — the hub refuses to serve a controller past what the box's agent supports — and it is the safe state, but it means those boxes stay where they are until their agents move. The two campaign venues are meant to be frozen; the N100 demo box is not, and unfreezing it is an owed follow-up.

The headline: no hand upgrade

13:53:40  bind (POST /appliances/22/bind, HTTP 303)
13:54:53  host appears — "— floor held", NO REPORT
13:55:34  agent 0.127.0, ONLINE          <-- 1 m 54 s after the bind
          guest 9201 running; controller 0.203.0 reporting

No "floor held" on the new box, and no hand upgrade anywhere. The agent it installed is the agent the hub vouches; the controller it runs is the one baked into golden 0.203.0. The PBS wrapper reads "matches vouched" and root@pam was rotated and vaulted by day-0.

Install — three known traps, all reproduced and handled

Driven blind: qm monitor screendump → PPM → read visually, qm sendkey for input.

1. GRUB defaults to the graphical entry. down+ret sent inside one remote command to hit the ~15 s window; the terminal installer came up first try.
2. The keymap defaults to Hungarian while sendkey emits US scancodes. Switched to U.S. English before any typing — the list had to be opened with ret (arrow keys do not cycle it) and U.S. English is 15 rows below Hungarian. Positive control: the administrator email was typed through the identical path and rendered part4@felhom.eu. @ is shift-2 on a US layout and AltGr+V on a Hungarian one, so a correct @ is the only available evidence that the 24 masked password characters went in as intended.
3. --boot set in its own qm set after the disks existed, ISO detached, and both verified from qm config before the first boot (boot: order=scsi0, no ide2). Automatically reboot was unchecked and confirmed [ ] with the focus moved away, so the reboot was deliberate.

Summary screen, verbatim: ext4 · /dev/sda · Europe/Budapest · U.S. English · part4@felhom.eu · nic0 · part4.felhom.eu · 192.168.0.141/24 · 192.168.0.1 · 192.168.0.1.

The install-time root password was generated into a 0600 file and typed by a host-side script that read that file — it never entered a command line, a unit property or a transcript. Day-0 replaced it anyway and escrowed the replacement to the hub.

No guest command line — this walk, unlike the last one, actually holds the line

The previous re-walk's fixture used the --print-reset-code escape hatch, which is a guest command line, and said so. This one did not need it:

• The claim used the code the hub emailed the operator, entered through the real POST /claim form. The hub stores only a hash — there is no plaintext to reveal — so this genuinely required the customer's own channel. Verified by a positive observable, not the 302: the session then reached „Indítópult" (36 962 B) with zero claim-gate markers.
• The sentinels were written over SMB — a real customer path (R-7) — into a share created through the UI, with the share mounted so filenames never crossed a shell.

Appliance-level SSH was used (host, not guest) to restart the agent when the backup-target assign asked for it, and to read the guest's addresses. The break-glass credential came from the hub's own reveal-recovery-credential endpoint, file→file, and was never rendered.

The fixture

Drives. Baseline first, because it is what makes half 2 legible — on a fresh install GET /api/disks/candidates returned initialize: [/dev/sdb, /dev/sdc], attach: []. That is exactly the call R-220 breaks after a rebuild, so this is the measurement half 2 is compared against.

Both enrolled through the wizard's own pipeline (/api/storage/init → format → mount → register, polled via /api/storage/init/status), not by hand-assembling format+assign — the F9 episode is on record as proof that a hand-assembled enrolment proves nothing. The format-safety gate fired correctly, returning needs_confirmation — "device has a ext4 filesystem" — and only proceeding on an explicit confirmation carrying the durable id the agent itself had returned.

drive	device	managed path	raw path
adatok (default)	/dev/sdb	/mnt/felhom-drives/adatok	/mnt/adatok
mentes	/dev/sdc	/mnt/felhom-drives/mentes	/mnt/mentes

Both mounts exist for each drive — that double mount is precisely what R-220 is about.

Backup target assigned to mentes, which reported restart_required: true and flipped only after the agent restart it asked for:

before: {"degraded":true,  …"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after:  {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}

App. calibre-web deployed through POST /api/stacks/calibre-web/deploy with HDD_PATH=/mnt/felhom-drives/adatok; state: running and health_probe.healthy: true — the .State field alone is known to say "running" for unhealthy containers, so the probe is the claim.

The three sentinels, written over SMB and read back from the share:

#	file	bytes	sha256
A	PART4-SENTINEL-A.txt	72	2de4e291989b5aa8771b361d25198ea389edcee50f4014b77a10345f91cf39d0
B	PART4-őrszem-ékezetes-árvíztűrő.txt	76	86ac38586de64e3ba80103b39b63ffd47f1e699dad386f5d0b7feddf83f9c5f6
C	PART4-SENTINEL-C-12MB.bin	12 582 912	461f32bda8323d6955fb09dd0dd4457b5acb0081155cefa4d692c93e8b1148ae

Sentinel B is the encoding canary. Its filename was built from explicit code points and verified as hex, never as rendered text — the previous walk lost this file's accents to a base64 → bash → pct exec chain, and a Python decode('utf-8') check called the result valid because U+FFFD is valid UTF-8. Here the name reads 50415254342d c591 72737a656d2d c3a9 … on both sides — ő é á í ű ő, no efbfbd anywhere.

Escrow ceremony. Preflight 6 of 6 green (pbs_storage_id · dr_tier · age_binary · hub_upload · staged_secret · sudo_grant), agent_supported: true. Result:

phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24
key_fingerprint: 4c:e2:45:5f:48:9c:04:f0:…:71:ff:99:2a · claimable: true

restic_pw_sealed: true is the field the whole exercise rests on.

R was claimed ONE-SHOT and streamed file→file into ~/.config/part4/R_part4.txt (0600, DooPlex only); the intermediate on the jump host was shred -u'd and the raw response deleted. It was never rendered. Shape only: 10 words, 82 characters. The same runbook tension as last time applies — §5.5 says record that a code exists and where, §9.4 says R is never a file on any machine — and it is flagged here rather than resolved quietly: the operator should move R into their own store and confirm. Half 2 cannot run without it.

R-218, measured on a real installation

11:57:30Z  [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s)
11:57:40Z  [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.203.0)
11:57:44Z  [offsite-apply] offsite configured for u629488-sub6@…:/home/felhom-repo
12:02:31Z  [scheduler] Running job: offsite-credential-retry     (and every 5 min after)

credential retry log lines: 0.

The previous walk's dead end 1 — "the off-site tier never came up on its own" — is gone: the tier was configured 14 s after the settle gate, unaided. And the retry job behaves exactly as designed on a healthy box: registered, ticking, completing in 0 s, and saying nothing, because the declaration it keys on is false.

Stated precisely, because the distinction matters: this venue proves the silent half. It did NOT reproduce the failure condition itself — the credential was already staged before the box's first reconcile, so there was never anything for the retry to collect. The collecting half was proven earlier in the session on demo-hp, with a positive control. A venue that proves a retry never fires is not a venue that proves a retry works.

R-225, visible on a live box in both states

The remote-backup card read „Tároló méret · a pillanatképek száma még ismeretlen" and „Tárhelykeret: még nem tudjuk, mennyi van a tárolóban — legfeljebb 50 GB" before any snapshot existed, and „12.0 MB / 50 GB (0%)" after. Unknown rendered as unknown, not as zero — the R-225 fix, on a real installation, in both of its states.

The off-site copy was proven readable BEFORE anything was destroyed

restic-level listing needs the repository password, which is sealed — so the proof used the customer's own verification restore (mode=unit), which restores config+DB into a separate folder and leaves live data alone:

A(z) calibre-web beállításai és adatbázisa visszaállítva ellenőrző mappába:
/mnt/felhom-drives/adatok/backups/

The repository opens, decrypts and restores end-to-end. The gate is not the green tick — and that is not a slogan here, because running this check is what caught R-234 (below). Had the venue been destroyed on the strength of the tick, half 2 would have failed with the evidence already gone.

Two findings, both from the same family

• R-234 — an off-site run reports success while silently omitting an app the customer just enabled. Run with no app selected → 1 snapshot. Toggle calibre-web on (302; the "no app selected" warning disappears, so the selection landed). Next run: „✓ Rendben · 12.0 MB · 1 pillanatkép" — still one. The app restore then refused: „nincs pillanatkép a(z) calibre-web alkalmazáshoz". A third run produced the second snapshot and the same restore succeeded. Nothing on the card distinguishes the green run that carried the app from the green run that did not.
• R-235 — the console keeps telling an already-paired box to pair itself, 25 minutes after the bind, while promising „Ez a képernyő magától frissül".

Both are "presence is not success": a surface reporting that something happened rather than that it worked.

Pre-destruction truth

Read back from the share immediately before stopping — all three match their originals byte for byte:

PART4-SENTINEL-A.txt                          72  2de4e291989b5aa8771b361d25198ea389edcee50f4014b77a10345f91cf39d0
PART4-SENTINEL-C-12MB.bin               12582912  461f32bda8323d6955fb09dd0dd4457b5acb0081155cefa4d692c93e8b1148ae
PART4-őrszem-ékezetes-árvíztűrő.txt           76  86ac38586de64e3ba80103b39b63ffd47f1e699dad386f5d0b7feddf83f9c5f6

Box state: felhom-agent 0.127.0 · controller 0.203.0 · 2 snapshots off-site · 12.0 MB / 50 GB.

Half 2 — owed, not started

Destroy the guest, reinstall, recover with R, and compare. PASS = the three sentinels' sha256 are byte-identical AND no guest command line was used. The R-220 measurement is the same GET /api/disks/candidates call recorded above: after the rebuild it must still offer both drives, where before the fix it returned initialize: [], attach: [] with the deploy refusing by naming a list that was empty.

Harness faults, separated from the product's

• The first /api/disks/assign attempt failed (HTTP 400) because where is a path, not a drive name, and the uuid must be bare. Mine, not the product's — and it is why the enrolment was then done through /api/storage/init instead of hand-assembled.
• The first deploy POST was rejected for a missing HDD_PATH that I had sent: the body wraps the fields in values. The endpoint's own refusal caught it.
• I pre-formatted both disks via /api/disks/format before switching to the wizard endpoint, which is why the wizard then reported needs_confirmation ("device has a ext4 filesystem"). The gate behaved correctly; the extra state was mine.
• Grepping the rendered page for Hiba matched JavaScript template strings, not errors. Counted and discarded rather than reported as failures.