felhom.eu/documentation/tests/part4-rewalk-2026-08-06/journal.md

# PART 4 — the re-walk on a real installation, with no hand upgrade (2026-08-06)

Task 4 Part 4. **Both halves were walked.** Half 1 (the fixture) is below; **half 2 (destroy →
reinstall → recover) is at the end of this file**, with its verdict stated separately as required:
the upgrade path **PASSED**, the data half **FAILED — not reached**, because the recovery journey
dead-ends before any file comes back (R-238). The venue is still standing and its teardown is owed.

The question this venue exists to answer: **does a box built from the published artifacts get R-218's
and R-220's fixes without anyone touching it?** The previous re-walk could not ask that — its reinstall
*downgraded* the agent, because the vouched agent was older than the box needed.

## The venue

| | |
|---|---|
| Host | `demo-hp` (HP t740), Tier 0 — **VM 323 `part4-appliance`** (321 and 322 untouched) |
| Appliance | `part4.felhom.eu` @ **192.168.0.141**, q35/OVMF, 4 cores, 8 GB |
| Disks | `scsi0` 200 G · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` |
| Guest | LXC **9201 `part4`**, island `169.254.253.2`, LAN `192.168.0.228` |
| Hub customer | **`part4`** "Part 4 - no hand upgrade" · host id **`part4-905bdb`** · appliance uuid `79fc7b81-…` |
| Off-site | Storage Box sub-account, user `u629488-sub6`, `/home/felhom-repo`, 50 GB |
| Teardown | **owed** — VM 323, the hub customer, and the off-site repo. Teardown is by **VM id 323**. |

**Routing fact worth recording:** DooPlex cannot reach either the appliance or the guest — no ping, no
port. `demo-hp` can. Every customer-side call in this journal went through `demo-hp` as a jump host.
The guest is **born-on-island** (R-50) and has no route from the workspace.

## The vouch that made this test possible

Before the walk, three fields moved together (never one — `golden_version` alone ships a controller onto
an older agent than it declares it needs):

| field | before | after |
|---|---|---|
| `golden_version` | 0.201.0 | **0.203.0** |
| `agent_version` | 0.125.0 | **0.127.0** |
| `min_agent` | 0.125.0 | **0.127.0** |

Rollback values are recorded in the scratch `ROLLBACK-artifacts.txt`; vouching is reversible by
re-selecting them and saving — no rebuild, because a bake never deletes the previous package.

**A consequence that must not be glossed:** raising `min_agent` to 0.127.0 newly **holds the controller
floor on three boxes** that were not held before — `c11-36d660` and `rewalk-1ab77d` (agent 0.126.0) and
`demo-felhom-8363b5` (0.125.0). That is the R-216 machinery working as designed — the hub refuses to
serve a controller past what the box's agent supports — and it is the *safe* state, but it means those
boxes stay where they are until their agents move. The two campaign venues are meant to be frozen; the
**N100 demo box is not**, and unfreezing it is an owed follow-up.

## The headline: no hand upgrade

```
13:53:40  bind (POST /appliances/22/bind, HTTP 303)
13:54:53  host appears — "— floor held", NO REPORT
13:55:34  agent 0.127.0, ONLINE          <-- 1 m 54 s after the bind
          guest 9201 running; controller 0.203.0 reporting
```

**No "floor held" on the new box, and no hand upgrade anywhere.** The agent it installed is the agent
the hub vouches; the controller it runs is the one baked into golden 0.203.0. The PBS wrapper reads
"matches vouched" and root@pam was rotated and vaulted by day-0.

## Install — three known traps, all reproduced and handled

Driven blind: `qm monitor screendump` → PPM → read visually, `qm sendkey` for input.

1. **GRUB defaults to the graphical entry.** `down`+`ret` sent inside one remote command to hit the
   ~15 s window; the terminal installer came up first try.
2. **The keymap defaults to Hungarian while `sendkey` emits US scancodes.** Switched to **U.S. English**
   before any typing — the list had to be opened with `ret` (arrow keys do not cycle it) and U.S. English
   is 15 rows below Hungarian. **Positive control:** the administrator email was typed through the
   identical path and rendered **`part4@felhom.eu`**. `@` is `shift-2` on a US layout and `AltGr+V` on a
   Hungarian one, so a correct `@` is the only available evidence that the 24 masked password characters
   went in as intended.
3. **`--boot` set in its own `qm set` after the disks existed**, ISO detached, and both verified from
   `qm config` **before the first boot** (`boot: order=scsi0`, no `ide2`). `Automatically reboot` was
   unchecked and confirmed `[ ]` **with the focus moved away**, so the reboot was deliberate.

Summary screen, verbatim: `ext4` · `/dev/sda` · `Europe/Budapest` · **`U.S. English`** ·
`part4@felhom.eu` · `nic0` · `part4.felhom.eu` · `192.168.0.141/24` · `192.168.0.1` · `192.168.0.1`.

The install-time root password was generated into a `0600` file and typed by a host-side script that
read that file — it never entered a command line, a unit property or a transcript. Day-0 replaced it
anyway and escrowed the replacement to the hub.

## No guest command line — this walk, unlike the last one, actually holds the line

The previous re-walk's fixture used the `--print-reset-code` escape hatch, which is a guest command
line, and said so. This one did not need it:

- **The claim** used the code the hub emailed the operator, entered through the real `POST /claim`
  form. The hub stores only a hash — there is no plaintext to reveal — so this genuinely required the
  customer's own channel. Verified by a positive observable, not the 302: the session then reached
  „Indítópult" (36 962 B) with **zero** claim-gate markers.
- **The sentinels** were written over **SMB** — a real customer path (R-7) — into a share created
  through the UI, with the share **mounted** so filenames never crossed a shell.

Appliance-level SSH *was* used (host, not guest) to restart the agent when the backup-target assign
asked for it, and to read the guest's addresses. The break-glass credential came from the hub's own
`reveal-recovery-credential` endpoint, file→file, and was never rendered.

## The fixture

**Drives.** Baseline first, because it is what makes half 2 legible — on a fresh install
`GET /api/disks/candidates` returned **`initialize: [/dev/sdb, /dev/sdc]`, `attach: []`**. That is
exactly the call R-220 breaks after a rebuild, so this is the measurement half 2 is compared against.

Both enrolled through the wizard's own pipeline (`/api/storage/init` → format → mount → register,
polled via `/api/storage/init/status`), not by hand-assembling format+assign — the F9 episode is on
record as proof that a hand-assembled enrolment proves nothing. The **format-safety gate fired
correctly**, returning `needs_confirmation` — "device has a ext4 filesystem" — and only proceeding on
an explicit confirmation carrying the durable id the agent itself had returned.

| drive | device | managed path | raw path |
|---|---|---|---|
| `adatok` (default) | /dev/sdb | `/mnt/felhom-drives/adatok` | `/mnt/adatok` |
| `mentes` | /dev/sdc | `/mnt/felhom-drives/mentes` | `/mnt/mentes` |

Both mounts exist for each drive — **that double mount is precisely what R-220 is about**.

**Backup target** assigned to `mentes`, which reported `restart_required: true` and flipped only after
the agent restart it asked for:

```
before: {"degraded":true,  …"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after:  {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}
```

**App.** `calibre-web` deployed through `POST /api/stacks/calibre-web/deploy` with
`HDD_PATH=/mnt/felhom-drives/adatok`; `state: running` **and `health_probe.healthy: true`** — the
`.State` field alone is known to say "running" for unhealthy containers, so the probe is the claim.

**The three sentinels**, written over SMB and read back from the share:

| # | file | bytes | sha256 |
|---|---|---|---|
| A | `PART4-SENTINEL-A.txt` | 72 | `2de4e291989b5aa8771b361d25198ea389edcee50f4014b77a10345f91cf39d0` |
| B | `PART4-őrszem-ékezetes-árvíztűrő.txt` | 76 | `86ac38586de64e3ba80103b39b63ffd47f1e699dad386f5d0b7feddf83f9c5f6` |
| C | `PART4-SENTINEL-C-12MB.bin` | 12 582 912 | `461f32bda8323d6955fb09dd0dd4457b5acb0081155cefa4d692c93e8b1148ae` |

Sentinel B is the encoding canary. Its filename was built from explicit code points and verified **as
hex**, never as rendered text — the previous walk lost this file's accents to a
`base64 → bash → pct exec` chain, and a Python `decode('utf-8')` check called the result valid because
U+FFFD *is* valid UTF-8. Here the name reads
`50415254342d c591 72737a656d2d c3a9 …` on both sides — `ő é á í ű ő`, no `efbfbd` anywhere.

**Escrow ceremony.** Preflight **6 of 6 green** (`pbs_storage_id` · `dr_tier` · `age_binary` ·
`hub_upload` · `staged_secret` · `sudo_grant`), `agent_supported: true`. Result:

```
phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24
key_fingerprint: 4c:e2:45:5f:48:9c:04:f0:…:71:ff:99:2a · claimable: true
```

`restic_pw_sealed: true` is the field the whole exercise rests on.

**R** was claimed ONE-SHOT and streamed file→file into `~/.config/part4/R_part4.txt` (`0600`, DooPlex
only); the intermediate on the jump host was `shred -u`'d and the raw response deleted. It was never
rendered. Shape only: **10 words, 82 characters**. The same runbook tension as last time applies —
§5.5 says record *that* a code exists and where, §9.4 says R is never a file on any machine — and it is
flagged here rather than resolved quietly: **the operator should move R into their own store and
confirm.** Half 2 cannot run without it.

## R-218, measured on a real installation

```
11:57:30Z  [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s)
11:57:40Z  [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.203.0)
11:57:44Z  [offsite-apply] offsite configured for u629488-sub6@…:/home/felhom-repo
12:02:31Z  [scheduler] Running job: offsite-credential-retry     (and every 5 min after)
```

`credential retry` log lines: **0**.

**The previous walk's dead end 1 — "the off-site tier never came up on its own" — is gone**: the tier
was configured 14 s after the settle gate, unaided. And the retry job behaves exactly as designed on a
healthy box: registered, ticking, completing in 0 s, and **saying nothing**, because the declaration it
keys on is false.

**Stated precisely, because the distinction matters:** this venue proves the *silent* half. It did NOT
reproduce the failure condition itself — the credential was already staged before the box's first
reconcile, so there was never anything for the retry to collect. The *collecting* half was proven
earlier in the session on demo-hp, with a positive control. A venue that proves a retry never fires is
not a venue that proves a retry works.

## R-225, visible on a live box in both states

The remote-backup card read „Tároló méret · a pillanatképek száma **még ismeretlen**" and
„Tárhelykeret: **még nem tudjuk**, mennyi van a tárolóban — legfeljebb 50 GB" **before** any snapshot
existed, and „12.0 MB / 50 GB (0%)" **after**. Unknown rendered as unknown, not as zero — the R-225 fix,
on a real installation, in both of its states.

## The off-site copy was proven readable BEFORE anything was destroyed

`restic`-level listing needs the repository password, which is sealed — so the proof used the
customer's own verification restore (`mode=unit`), which restores config+DB into a **separate** folder
and leaves live data alone:

```
A(z) calibre-web beállításai és adatbázisa visszaállítva ellenőrző mappába:
/mnt/felhom-drives/adatok/backups/
```

The repository opens, decrypts and restores end-to-end. **The gate is not the green tick** — and that
is not a slogan here, because running this check is what caught R-234 (below). Had the venue been
destroyed on the strength of the tick, half 2 would have failed with the evidence already gone.

## Two findings, both from the same family

- **R-234 — an off-site run reports success while silently omitting an app the customer just enabled.**
  Run with no app selected → 1 snapshot. Toggle `calibre-web` on (302; the "no app selected" warning
  disappears, so the selection landed). Next run: **„✓ Rendben · 12.0 MB · 1 pillanatkép"** — still one.
  The app restore then refused: **„nincs pillanatkép a(z) calibre-web alkalmazáshoz"**. A third run
  produced the second snapshot and the same restore succeeded. Nothing on the card distinguishes the
  green run that carried the app from the green run that did not.
- **R-235 — the console keeps telling an already-paired box to pair itself**, 25 minutes after the bind,
  while promising „Ez a képernyő magától frissül".

Both are "presence is not success": a surface reporting that something *happened* rather than that it
*worked*.

## Pre-destruction truth

Read back from the share immediately before stopping — all three match their originals byte for byte:

```
PART4-SENTINEL-A.txt                          72  2de4e291989b5aa8771b361d25198ea389edcee50f4014b77a10345f91cf39d0
PART4-SENTINEL-C-12MB.bin               12582912  461f32bda8323d6955fb09dd0dd4457b5acb0081155cefa4d692c93e8b1148ae
PART4-őrszem-ékezetes-árvíztűrő.txt           76  86ac38586de64e3ba80103b39b63ffd47f1e699dad386f5d0b7feddf83f9c5f6
```

Box state: `felhom-agent 0.127.0` · controller `0.203.0` · **2 snapshots** off-site · 12.0 MB / 50 GB.

## Half 2 — owed, not started

Destroy the guest, reinstall, recover with R, and compare. **PASS = the three sentinels' sha256 are
byte-identical AND no guest command line was used.** The R-220 measurement is the same
`GET /api/disks/candidates` call recorded above: after the rebuild it must still offer both drives,
where before the fix it returned `initialize: [], attach: []` with the deploy refusing by naming a list
that was empty.

## Harness faults, separated from the product's

- The first `/api/disks/assign` attempt failed (HTTP 400) because `where` is a **path**, not a drive
  name, and the uuid must be bare. Mine, not the product's — and it is why the enrolment was then done
  through `/api/storage/init` instead of hand-assembled.
- The first `deploy` POST was rejected for a missing `HDD_PATH` that I *had* sent: the body wraps the
  fields in `values`. The endpoint's own refusal caught it.
- I pre-formatted both disks via `/api/disks/format` before switching to the wizard endpoint, which is
  why the wizard then reported `needs_confirmation` ("device has a ext4 filesystem"). The gate behaved
  correctly; the extra state was mine.
- Grepping the rendered page for `Hiba` matched **JavaScript template strings**, not errors. Counted
  and discarded rather than reported as failures.

---

# HALF 2 — destroy, reinstall, recover (2026-08-06, same day)

**VERDICT, both halves separately:**

| half | verdict |
|---|---|
| **The upgrade path** — does a real installation get the fixes with no hand upgrade? | **PASS** |
| **The data** — are the sentinels byte-identical after recovery? | **FAIL — not reached.** The journey dead-ends before any file comes back (R-238). |

## The destruction

`pct destroy 9201 --purge` at **15:04:15** on the appliance (guarded on `hostname = part4`, because
`demo-hp` also has a guest 9201 and destroying the wrong one would have been serious). Both LVs
removed; the host identity `part4-905bdb` survived. Both drives then wiped to 20 K **with their mounts
left in place** — that is deliberate, because the surviving raw `/mnt/<name>` mount **is** the R-220
condition.

The originals of all three sentinels are on DooPlex with known hashes, so the wipe risked the *test
outcome*, never the data.

**A stale-report window worth recording:** immediately after the purge the hub still showed
`1/1 running` for a guest that no longer existed, until the next agent report. `pct list` on the box
was already empty. Presence is not success — and a report is not a measurement.

**The box does not self-heal a purged guest.** Nothing re-provisioned it; the reinstall was needed.

## The reinstall — and the R-216 recurrence is GONE

`felhom-host-install.sh` **v1.25.0** fetched live from `felhom.eu/scripts/`, run as a transient unit
with the retrieval passphrase read from a `0600` file (`--passphrase-file`), never a command line.

```
Day-0 provision SUCCESS — vmid=9201 host_id=part4-905bdb customer=part4    (~2 m 50 s)
controller image: gitea.dooplex.hu/admin/felhom-controller:0.203.0
```

| | previous re-walk | this one |
|---|---|---|
| agent across the rebuild | **0.126.0 → 0.125.0 (DOWNGRADE)** | **0.127.0 → 0.127.0** |

That downgrade was R-216 part 4 recurring: an operator who fixes a box by hand has it re-broken by the
very event that makes recovery necessary. It does not happen when the vouched agent *is* the newest —
which is what Part 3's three-field vouch achieved.

## R-220 — FIXED, proven on a genuinely rebuilt box

Precondition verified **before** measuring — both devices mounted twice, raw and managed:

```
/dev/sdb /mnt/adatok              ext4      /dev/sdb /mnt/felhom-drives/adatok  ext4
/dev/sdc /mnt/mentes              ext4      /dev/sdc /mnt/felhom-drives/mentes  ext4
```

The agent's own `/disks/candidates` (measured at the agent, because that is where the fix lives):

```
attach     : /dev/sdb, /dev/sdc
initialize : /dev/sdb, /dev/sdc
```

Before the fix this returned `initialize: [], attach: []`, and the deploy refused by telling the
customer to choose from a list that was empty. Both drives then **re-attached through the customer
endpoint** (`registered: true`) — the end-to-end confirmation.

## R-218 — the consume half, proven live, end to end

The unlock screen put the box in exactly R-218's situation: „A gép még várja a házon kívüli tárhely
kapcsolódási adatait — amint megvannak… **Nincs teendőd**."

The retry job then did precisely what it was built to do, and said why:

```
[offsite-apply] credential retry: consume one-time password: no unconsumed offsite password
                (already consumed or none provisioned) (the box still declares a need; retrying)
```

**The blocker was upstream, not in the retry** — after a rebuild nothing re-stages the one-time
password (**R-236**). Operator re-issue at **13:26:32Z**, and the box collected it on a tick:

```
[offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.203.0)
[offsite-apply] offsite configured for u629488-sub6@…:/home/felhom-repo (pending key escrow)
```

**A credential staged after start-up, collected without intervention and without a guest command
line.** In the previous walk this same situation was dead end 1 and needed a guest shell.

## The customer journey, as read from the screen

- **The recovery screen appeared without being sought** — `/` landed on „Adatok visszaszerzése",
  claim-gate markers 0.
- It states the seal date, that nobody can replace the code, and that **this step changes nothing**.
  *(The seal date still renders as a raw RFC3339 string — `2026-08-06T12:41:55Z` — to a Hungarian
  household. Third walk running; still unfixed.)*
- **The unlock took 1.804 s** — a real unseal. For contrast, a submission with an empty field returned
  in **25 ms** with „Add meg a helyreállítási kódot" — a **neutral prompt, not a wrong-code
  accusation**. That is R-224/R-226 behaving as shipped: unknown defaults to neutral, and the typing
  message is reachable only after a real refusal.
- Escrow re-sealed ~15 min later („A helyreállítási kód letétbe helyezve").

## Where the journey ends — and it ends short

Everything cryptographic succeeded. The customer still cannot reach a single file:

1. **R-237** — the restore surface is keyed on apps that are *currently installed* and *currently
   marked for future remote backup*. Post-rebuild there are none, so „Nincs telepített alkalmazás",
   and the restore page points at a page that says the same thing. Reaching a restore at all required
   three undocumented steps in order: re-attach the drives, redeploy the app, toggle it on for
   **future** backups. A household that just lost its box does not know what it used to run.
2. **R-238** — and then the button does nothing. „Teljes visszaállítás előkészítése"
   (`mode=full`) returns 302 and starts **no job**, leaves the wizard on step 1, shows no error, and
   writes **no line to the controller's own debug ring**. `mode=unit` on the identical form works.

**That is why the data half is FAIL rather than PASS.** It is not that a sentinel came back wrong —
nothing came back at all. The sentinels' pre-destruction hashes are recorded above and their originals
are intact on DooPlex, so the comparison can be completed the moment R-238 is fixed.

## Honest accounting of what half 2 did NOT establish

- No sentinel was restored, so **the byte-identity claim is unproven**, in either direction.
- The off-site repository was proven readable *before* the destruction (verification restore), and the
  repo held 12.0 MB of unique data — consistent with sentinel C and inconsistent with config+DB alone.
  That is corroboration, not proof, that the sentinels are recoverable.
- I used appliance-level SSH (host, not guest) throughout. **No guest command line was used at any
  point in either half** — the property the previous walk could not claim.

## Harness faults in half 2, separated from the product's

- Posted the unlock with field `code` instead of `recovery_code`. The product's response to that was
  *correct and informative* (the neutral empty-field prompt), which is how I noticed.
- Nested quoting through the double SSH hop (DooPlex → demo-hp → appliance) corrupted several
  commands; switched to scp'd script files. Two "0 results" readings were **my broken parser**, not
  absence — re-measured by grepping the raw file, exactly the trap the project's own rule names.
- A `Hiba` grep matched JavaScript template strings, not errors.