felhom.eu/documentation/tests/part4-rewalk-2026-08-06/journal.md

PART 4 — the re-walk on a real installation, with no hand upgrade (2026-08-06)

Task 4 Part 4. Both halves were walked. Half 1 (the fixture) is below; half 2 (destroy → reinstall → recover) is at the end of this file, with its verdict stated separately as required: the upgrade path PASSED, the data half FAILED — not reached, because the recovery journey dead-ends before any file comes back (R-238). The venue is still standing and its teardown is owed.

The question this venue exists to answer: does a box built from the published artifacts get R-218's and R-220's fixes without anyone touching it? The previous re-walk could not ask that — its reinstall downgraded the agent, because the vouched agent was older than the box needed.

The venue

Host	demo-hp (HP t740), Tier 0 — VM 323 part4-appliance (321 and 322 untouched)
Appliance	part4.felhom.eu @ 192.168.0.141, q35/OVMF, 4 cores, 8 GB
Disks	scsi0 200 G · scsi1 50 G · scsi2 50 G, qcow2 on c11-scratch
Guest	LXC 9201 part4, island 169.254.253.2, LAN 192.168.0.228
Hub customer	part4 "Part 4 - no hand upgrade" · host id part4-905bdb · appliance uuid 79fc7b81-…
Off-site	Storage Box sub-account, user u629488-sub6, /home/felhom-repo, 50 GB
Teardown	owed — VM 323, the hub customer, and the off-site repo. Teardown is by VM id 323.

Routing fact worth recording: DooPlex cannot reach either the appliance or the guest — no ping, no port. demo-hp can. Every customer-side call in this journal went through demo-hp as a jump host. The guest is born-on-island (R-50) and has no route from the workspace.

The vouch that made this test possible

Before the walk, three fields moved together (never one — golden_version alone ships a controller onto an older agent than it declares it needs):

field	before	after
golden_version	0.201.0	0.203.0
agent_version	0.125.0	0.127.0
min_agent	0.125.0	0.127.0

Rollback values are recorded in the scratch ROLLBACK-artifacts.txt; vouching is reversible by re-selecting them and saving — no rebuild, because a bake never deletes the previous package.

A consequence that must not be glossed: raising min_agent to 0.127.0 newly holds the controller floor on three boxes that were not held before — c11-36d660 and rewalk-1ab77d (agent 0.126.0) and demo-felhom-8363b5 (0.125.0). That is the R-216 machinery working as designed — the hub refuses to serve a controller past what the box's agent supports — and it is the safe state, but it means those boxes stay where they are until their agents move. The two campaign venues are meant to be frozen; the N100 demo box is not, and unfreezing it is an owed follow-up.

The headline: no hand upgrade

13:53:40  bind (POST /appliances/22/bind, HTTP 303)
13:54:53  host appears — "— floor held", NO REPORT
13:55:34  agent 0.127.0, ONLINE          <-- 1 m 54 s after the bind
          guest 9201 running; controller 0.203.0 reporting

No "floor held" on the new box, and no hand upgrade anywhere. The agent it installed is the agent the hub vouches; the controller it runs is the one baked into golden 0.203.0. The PBS wrapper reads "matches vouched" and root@pam was rotated and vaulted by day-0.

Install — three known traps, all reproduced and handled

Driven blind: qm monitor screendump → PPM → read visually, qm sendkey for input.

1. GRUB defaults to the graphical entry. down+ret sent inside one remote command to hit the ~15 s window; the terminal installer came up first try.
2. The keymap defaults to Hungarian while sendkey emits US scancodes. Switched to U.S. English before any typing — the list had to be opened with ret (arrow keys do not cycle it) and U.S. English is 15 rows below Hungarian. Positive control: the administrator email was typed through the identical path and rendered part4@felhom.eu. @ is shift-2 on a US layout and AltGr+V on a Hungarian one, so a correct @ is the only available evidence that the 24 masked password characters went in as intended.
3. --boot set in its own qm set after the disks existed, ISO detached, and both verified from qm config before the first boot (boot: order=scsi0, no ide2). Automatically reboot was unchecked and confirmed [ ] with the focus moved away, so the reboot was deliberate.

Summary screen, verbatim: ext4 · /dev/sda · Europe/Budapest · U.S. English · part4@felhom.eu · nic0 · part4.felhom.eu · 192.168.0.141/24 · 192.168.0.1 · 192.168.0.1.

The install-time root password was generated into a 0600 file and typed by a host-side script that read that file — it never entered a command line, a unit property or a transcript. Day-0 replaced it anyway and escrowed the replacement to the hub.

No guest command line — this walk, unlike the last one, actually holds the line

The previous re-walk's fixture used the --print-reset-code escape hatch, which is a guest command line, and said so. This one did not need it:

• The claim used the code the hub emailed the operator, entered through the real POST /claim form. The hub stores only a hash — there is no plaintext to reveal — so this genuinely required the customer's own channel. Verified by a positive observable, not the 302: the session then reached „Indítópult" (36 962 B) with zero claim-gate markers.
• The sentinels were written over SMB — a real customer path (R-7) — into a share created through the UI, with the share mounted so filenames never crossed a shell.

Appliance-level SSH was used (host, not guest) to restart the agent when the backup-target assign asked for it, and to read the guest's addresses. The break-glass credential came from the hub's own reveal-recovery-credential endpoint, file→file, and was never rendered.

The fixture

Drives. Baseline first, because it is what makes half 2 legible — on a fresh install GET /api/disks/candidates returned initialize: [/dev/sdb, /dev/sdc], attach: []. That is exactly the call R-220 breaks after a rebuild, so this is the measurement half 2 is compared against.

Both enrolled through the wizard's own pipeline (/api/storage/init → format → mount → register, polled via /api/storage/init/status), not by hand-assembling format+assign — the F9 episode is on record as proof that a hand-assembled enrolment proves nothing. The format-safety gate fired correctly, returning needs_confirmation — "device has a ext4 filesystem" — and only proceeding on an explicit confirmation carrying the durable id the agent itself had returned.

drive	device	managed path	raw path
adatok (default)	/dev/sdb	/mnt/felhom-drives/adatok	/mnt/adatok
mentes	/dev/sdc	/mnt/felhom-drives/mentes	/mnt/mentes

Both mounts exist for each drive — that double mount is precisely what R-220 is about.

Backup target assigned to mentes, which reported restart_required: true and flipped only after the agent restart it asked for:

before: {"degraded":true,  …"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after:  {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}

App. calibre-web deployed through POST /api/stacks/calibre-web/deploy with HDD_PATH=/mnt/felhom-drives/adatok; state: running and health_probe.healthy: true — the .State field alone is known to say "running" for unhealthy containers, so the probe is the claim.

The three sentinels, written over SMB and read back from the share:

#	file	bytes	sha256
A	PART4-SENTINEL-A.txt	72	2de4e291989b5aa8771b361d25198ea389edcee50f4014b77a10345f91cf39d0
B	PART4-őrszem-ékezetes-árvíztűrő.txt	76	86ac38586de64e3ba80103b39b63ffd47f1e699dad386f5d0b7feddf83f9c5f6
C	PART4-SENTINEL-C-12MB.bin	12 582 912	461f32bda8323d6955fb09dd0dd4457b5acb0081155cefa4d692c93e8b1148ae

Sentinel B is the encoding canary. Its filename was built from explicit code points and verified as hex, never as rendered text — the previous walk lost this file's accents to a base64 → bash → pct exec chain, and a Python decode('utf-8') check called the result valid because U+FFFD is valid UTF-8. Here the name reads 50415254342d c591 72737a656d2d c3a9 … on both sides — ő é á í ű ő, no efbfbd anywhere.

Escrow ceremony. Preflight 6 of 6 green (pbs_storage_id · dr_tier · age_binary · hub_upload · staged_secret · sudo_grant), agent_supported: true. Result:

phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24
key_fingerprint: 4c:e2:45:5f:48:9c:04:f0:…:71:ff:99:2a · claimable: true

restic_pw_sealed: true is the field the whole exercise rests on.

R was claimed ONE-SHOT and streamed file→file into ~/.config/part4/R_part4.txt (0600, DooPlex only); the intermediate on the jump host was shred -u'd and the raw response deleted. It was never rendered. Shape only: 10 words, 82 characters. The same runbook tension as last time applies — §5.5 says record that a code exists and where, §9.4 says R is never a file on any machine — and it is flagged here rather than resolved quietly: the operator should move R into their own store and confirm. Half 2 cannot run without it.

R-218, measured on a real installation

11:57:30Z  [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s)
11:57:40Z  [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.203.0)
11:57:44Z  [offsite-apply] offsite configured for u629488-sub6@…:/home/felhom-repo
12:02:31Z  [scheduler] Running job: offsite-credential-retry     (and every 5 min after)

credential retry log lines: 0.

The previous walk's dead end 1 — "the off-site tier never came up on its own" — is gone: the tier was configured 14 s after the settle gate, unaided. And the retry job behaves exactly as designed on a healthy box: registered, ticking, completing in 0 s, and saying nothing, because the declaration it keys on is false.

Stated precisely, because the distinction matters: this venue proves the silent half. It did NOT reproduce the failure condition itself — the credential was already staged before the box's first reconcile, so there was never anything for the retry to collect. The collecting half was proven earlier in the session on demo-hp, with a positive control. A venue that proves a retry never fires is not a venue that proves a retry works.

R-225, visible on a live box in both states

The remote-backup card read „Tároló méret · a pillanatképek száma még ismeretlen" and „Tárhelykeret: még nem tudjuk, mennyi van a tárolóban — legfeljebb 50 GB" before any snapshot existed, and „12.0 MB / 50 GB (0%)" after. Unknown rendered as unknown, not as zero — the R-225 fix, on a real installation, in both of its states.

The off-site copy was proven readable BEFORE anything was destroyed

restic-level listing needs the repository password, which is sealed — so the proof used the customer's own verification restore (mode=unit), which restores config+DB into a separate folder and leaves live data alone:

A(z) calibre-web beállításai és adatbázisa visszaállítva ellenőrző mappába:
/mnt/felhom-drives/adatok/backups/

The repository opens, decrypts and restores end-to-end. The gate is not the green tick — and that is not a slogan here, because running this check is what caught R-234 (below). Had the venue been destroyed on the strength of the tick, half 2 would have failed with the evidence already gone.

Two findings, both from the same family

• R-234 — an off-site run reports success while silently omitting an app the customer just enabled. Run with no app selected → 1 snapshot. Toggle calibre-web on (302; the "no app selected" warning disappears, so the selection landed). Next run: „✓ Rendben · 12.0 MB · 1 pillanatkép" — still one. The app restore then refused: „nincs pillanatkép a(z) calibre-web alkalmazáshoz". A third run produced the second snapshot and the same restore succeeded. Nothing on the card distinguishes the green run that carried the app from the green run that did not.
• R-235 — the console keeps telling an already-paired box to pair itself, 25 minutes after the bind, while promising „Ez a képernyő magától frissül".

Both are "presence is not success": a surface reporting that something happened rather than that it worked.

Pre-destruction truth

Read back from the share immediately before stopping — all three match their originals byte for byte:

PART4-SENTINEL-A.txt                          72  2de4e291989b5aa8771b361d25198ea389edcee50f4014b77a10345f91cf39d0
PART4-SENTINEL-C-12MB.bin               12582912  461f32bda8323d6955fb09dd0dd4457b5acb0081155cefa4d692c93e8b1148ae
PART4-őrszem-ékezetes-árvíztűrő.txt           76  86ac38586de64e3ba80103b39b63ffd47f1e699dad386f5d0b7feddf83f9c5f6

Box state: felhom-agent 0.127.0 · controller 0.203.0 · 2 snapshots off-site · 12.0 MB / 50 GB.

Half 2 — owed, not started

Destroy the guest, reinstall, recover with R, and compare. PASS = the three sentinels' sha256 are byte-identical AND no guest command line was used. The R-220 measurement is the same GET /api/disks/candidates call recorded above: after the rebuild it must still offer both drives, where before the fix it returned initialize: [], attach: [] with the deploy refusing by naming a list that was empty.

Harness faults, separated from the product's

• The first /api/disks/assign attempt failed (HTTP 400) because where is a path, not a drive name, and the uuid must be bare. Mine, not the product's — and it is why the enrolment was then done through /api/storage/init instead of hand-assembled.
• The first deploy POST was rejected for a missing HDD_PATH that I had sent: the body wraps the fields in values. The endpoint's own refusal caught it.
• I pre-formatted both disks via /api/disks/format before switching to the wizard endpoint, which is why the wizard then reported needs_confirmation ("device has a ext4 filesystem"). The gate behaved correctly; the extra state was mine.
• Grepping the rendered page for Hiba matched JavaScript template strings, not errors. Counted and discarded rather than reported as failures.

---

HALF 2 — destroy, reinstall, recover (2026-08-06, same day)

VERDICT, both halves separately:

half	verdict
The upgrade path — does a real installation get the fixes with no hand upgrade?	PASS
The data — are the sentinels byte-identical after recovery?	FAIL — not reached. The journey dead-ends before any file comes back (R-238).

The destruction

pct destroy 9201 --purge at 15:04:15 on the appliance (guarded on hostname = part4, because demo-hp also has a guest 9201 and destroying the wrong one would have been serious). Both LVs removed; the host identity part4-905bdb survived. Both drives then wiped to 20 K with their mounts left in place — that is deliberate, because the surviving raw /mnt/<name> mount is the R-220 condition.

The originals of all three sentinels are on DooPlex with known hashes, so the wipe risked the test outcome, never the data.

A stale-report window worth recording: immediately after the purge the hub still showed 1/1 running for a guest that no longer existed, until the next agent report. pct list on the box was already empty. Presence is not success — and a report is not a measurement.

The box does not self-heal a purged guest. Nothing re-provisioned it; the reinstall was needed.

The reinstall — and the R-216 recurrence is GONE

felhom-host-install.sh v1.25.0 fetched live from felhom.eu/scripts/, run as a transient unit with the retrieval passphrase read from a 0600 file (--passphrase-file), never a command line.

Day-0 provision SUCCESS — vmid=9201 host_id=part4-905bdb customer=part4    (~2 m 50 s)
controller image: gitea.dooplex.hu/admin/felhom-controller:0.203.0

	previous re-walk	this one
agent across the rebuild	0.126.0 → 0.125.0 (DOWNGRADE)	0.127.0 → 0.127.0

That downgrade was R-216 part 4 recurring: an operator who fixes a box by hand has it re-broken by the very event that makes recovery necessary. It does not happen when the vouched agent is the newest — which is what Part 3's three-field vouch achieved.

R-220 — FIXED, proven on a genuinely rebuilt box

Precondition verified before measuring — both devices mounted twice, raw and managed:

/dev/sdb /mnt/adatok              ext4      /dev/sdb /mnt/felhom-drives/adatok  ext4
/dev/sdc /mnt/mentes              ext4      /dev/sdc /mnt/felhom-drives/mentes  ext4

The agent's own /disks/candidates (measured at the agent, because that is where the fix lives):

attach     : /dev/sdb, /dev/sdc
initialize : /dev/sdb, /dev/sdc

Before the fix this returned initialize: [], attach: [], and the deploy refused by telling the customer to choose from a list that was empty. Both drives then re-attached through the customer endpoint (registered: true) — the end-to-end confirmation.

R-218 — the consume half, proven live, end to end

The unlock screen put the box in exactly R-218's situation: „A gép még várja a házon kívüli tárhely kapcsolódási adatait — amint megvannak… Nincs teendőd."

The retry job then did precisely what it was built to do, and said why:

[offsite-apply] credential retry: consume one-time password: no unconsumed offsite password
                (already consumed or none provisioned) (the box still declares a need; retrying)

The blocker was upstream, not in the retry — after a rebuild nothing re-stages the one-time password (R-236). Operator re-issue at 13:26:32Z, and the box collected it on a tick:

[offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.203.0)
[offsite-apply] offsite configured for u629488-sub6@…:/home/felhom-repo (pending key escrow)

A credential staged after start-up, collected without intervention and without a guest command line. In the previous walk this same situation was dead end 1 and needed a guest shell.

The customer journey, as read from the screen

• The recovery screen appeared without being sought — / landed on „Adatok visszaszerzése", claim-gate markers 0.
• It states the seal date, that nobody can replace the code, and that this step changes nothing. (The seal date still renders as a raw RFC3339 string — 2026-08-06T12:41:55Z — to a Hungarian household. Third walk running; still unfixed.)
• The unlock took 1.804 s — a real unseal. For contrast, a submission with an empty field returned in 25 ms with „Add meg a helyreállítási kódot" — a neutral prompt, not a wrong-code accusation. That is R-224/R-226 behaving as shipped: unknown defaults to neutral, and the typing message is reachable only after a real refusal.
• Escrow re-sealed ~15 min later („A helyreállítási kód letétbe helyezve").

Where the journey ends — and it ends short

Everything cryptographic succeeded. The customer still cannot reach a single file:

1. R-237 — the restore surface is keyed on apps that are currently installed and currently marked for future remote backup. Post-rebuild there are none, so „Nincs telepített alkalmazás", and the restore page points at a page that says the same thing. Reaching a restore at all required three undocumented steps in order: re-attach the drives, redeploy the app, toggle it on for future backups. A household that just lost its box does not know what it used to run.
2. R-238 — and then the button does nothing. „Teljes visszaállítás előkészítése" (mode=full) returns 302 and starts no job, leaves the wizard on step 1, shows no error, and writes no line to the controller's own debug ring. mode=unit on the identical form works.

That is why the data half is FAIL rather than PASS. It is not that a sentinel came back wrong — nothing came back at all. The sentinels' pre-destruction hashes are recorded above and their originals are intact on DooPlex, so the comparison can be completed the moment R-238 is fixed.

Honest accounting of what half 2 did NOT establish

• No sentinel was restored, so the byte-identity claim is unproven, in either direction.
• The off-site repository was proven readable before the destruction (verification restore), and the repo held 12.0 MB of unique data — consistent with sentinel C and inconsistent with config+DB alone. That is corroboration, not proof, that the sentinels are recoverable.
• I used appliance-level SSH (host, not guest) throughout. No guest command line was used at any point in either half — the property the previous walk could not claim.

Harness faults in half 2, separated from the product's

• Posted the unlock with field code instead of recovery_code. The product's response to that was correct and informative (the neutral empty-field prompt), which is how I noticed.
• Nested quoting through the double SSH hop (DooPlex → demo-hp → appliance) corrupted several commands; switched to scp'd script files. Two "0 results" readings were my broken parser, not absence — re-measured by grepping the raw file, exactly the trap the project's own rule names.
• A Hiba grep matched JavaScript template strings, not errors.