Part 4 half 2: upgrade path PASSES, recovery journey dead-ends (R-236/237/238)
gates / gates (push) Successful in 8s

Destroyed guest 9201 (guarded on hostname — demo-hp also has a 9201), wiped both
drives leaving the raw /mnt/<name> mounts in place, reinstalled from the
published installer v1.25.0.

PASS — the upgrade path:
  agent 0.127.0 -> 0.127.0 across the rebuild. The previous re-walk DOWNGRADED
  0.126.0 -> 0.125.0 (R-216 part 4). Controller 0.203.0 from the baked golden,
  Day-0 provision SUCCESS in ~2m50s, nothing applied by hand.

R-220 FIXED, proven on a genuinely rebuilt box: with both devices mounted raw
AND managed, /disks/candidates offered both drives (before the fix: two empty
lists), and both re-attached through the customer endpoint.

R-218's consume half proven live end to end: credential re-staged at 13:26:32Z,
collected on a tick, tier configured — no guest command line. Previously this
exact situation was dead end 1 and needed a guest shell.

FAIL — the data half, NOT reached. Three new findings, in the order they bite:
  R-236 nothing re-stages the off-site one-time password after a rebuild, so the
        "Nincs teendod" promise stalls until an operator re-issues. The retry job
        is NOT at fault — it retries correctly and logs why.
  R-237 the restore surface is keyed on installed + remote-marked apps, so after
        a rebuild the customer is shown no backups at all — a circular dead end.
  R-238 "Teljes visszaallitas elokeszitese" returns 302 and does NOTHING: no job,
        no state change, no error, no line in the controller's own debug ring.
        mode=unit on the same form works. This is why the sentinel byte-identity
        criterion was not met — nothing came back, rather than coming back wrong.

No guest command line was used at any point in either half — the property the
previous walk could not claim.
This commit is contained in:
2026-08-06 16:03:04 +02:00
parent ac451f9667
commit 5810e1f552
2 changed files with 145 additions and 2 deletions
@@ -1,7 +1,9 @@
# PART 4 — the re-walk on a real installation, with no hand upgrade (2026-08-06)
Task 4 Part 4. **Half 1 (the fixture) is complete and the venue is STANDING.** Nothing has been
destroyed; half 2 (destroy → reinstall → recover → compare) is owed and deliberately not started.
Task 4 Part 4. **Both halves were walked.** Half 1 (the fixture) is below; **half 2 (destroy →
reinstall → recover) is at the end of this file**, with its verdict stated separately as required:
the upgrade path **PASSED**, the data half **FAILED — not reached**, because the recovery journey
dead-ends before any file comes back (R-238). The venue is still standing and its teardown is owed.
The question this venue exists to answer: **does a box built from the published artifacts get R-218's
and R-220's fixes without anyone touching it?** The previous re-walk could not ask that — its reinstall
@@ -248,3 +250,140 @@ that was empty.
correctly; the extra state was mine.
- Grepping the rendered page for `Hiba` matched **JavaScript template strings**, not errors. Counted
and discarded rather than reported as failures.
---
# HALF 2 — destroy, reinstall, recover (2026-08-06, same day)
**VERDICT, both halves separately:**
| half | verdict |
|---|---|
| **The upgrade path** — does a real installation get the fixes with no hand upgrade? | **PASS** |
| **The data** — are the sentinels byte-identical after recovery? | **FAIL — not reached.** The journey dead-ends before any file comes back (R-238). |
## The destruction
`pct destroy 9201 --purge` at **15:04:15** on the appliance (guarded on `hostname = part4`, because
`demo-hp` also has a guest 9201 and destroying the wrong one would have been serious). Both LVs
removed; the host identity `part4-905bdb` survived. Both drives then wiped to 20 K **with their mounts
left in place** — that is deliberate, because the surviving raw `/mnt/<name>` mount **is** the R-220
condition.
The originals of all three sentinels are on DooPlex with known hashes, so the wipe risked the *test
outcome*, never the data.
**A stale-report window worth recording:** immediately after the purge the hub still showed
`1/1 running` for a guest that no longer existed, until the next agent report. `pct list` on the box
was already empty. Presence is not success — and a report is not a measurement.
**The box does not self-heal a purged guest.** Nothing re-provisioned it; the reinstall was needed.
## The reinstall — and the R-216 recurrence is GONE
`felhom-host-install.sh` **v1.25.0** fetched live from `felhom.eu/scripts/`, run as a transient unit
with the retrieval passphrase read from a `0600` file (`--passphrase-file`), never a command line.
```
Day-0 provision SUCCESS — vmid=9201 host_id=part4-905bdb customer=part4 (~2 m 50 s)
controller image: gitea.dooplex.hu/admin/felhom-controller:0.203.0
```
| | previous re-walk | this one |
|---|---|---|
| agent across the rebuild | **0.126.0 → 0.125.0 (DOWNGRADE)** | **0.127.0 → 0.127.0** |
That downgrade was R-216 part 4 recurring: an operator who fixes a box by hand has it re-broken by the
very event that makes recovery necessary. It does not happen when the vouched agent *is* the newest —
which is what Part 3's three-field vouch achieved.
## R-220 — FIXED, proven on a genuinely rebuilt box
Precondition verified **before** measuring — both devices mounted twice, raw and managed:
```
/dev/sdb /mnt/adatok ext4 /dev/sdb /mnt/felhom-drives/adatok ext4
/dev/sdc /mnt/mentes ext4 /dev/sdc /mnt/felhom-drives/mentes ext4
```
The agent's own `/disks/candidates` (measured at the agent, because that is where the fix lives):
```
attach : /dev/sdb, /dev/sdc
initialize : /dev/sdb, /dev/sdc
```
Before the fix this returned `initialize: [], attach: []`, and the deploy refused by telling the
customer to choose from a list that was empty. Both drives then **re-attached through the customer
endpoint** (`registered: true`) — the end-to-end confirmation.
## R-218 — the consume half, proven live, end to end
The unlock screen put the box in exactly R-218's situation: „A gép még várja a házon kívüli tárhely
kapcsolódási adatait — amint megvannak… **Nincs teendőd**."
The retry job then did precisely what it was built to do, and said why:
```
[offsite-apply] credential retry: consume one-time password: no unconsumed offsite password
(already consumed or none provisioned) (the box still declares a need; retrying)
```
**The blocker was upstream, not in the retry** — after a rebuild nothing re-stages the one-time
password (**R-236**). Operator re-issue at **13:26:32Z**, and the box collected it on a tick:
```
[offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.203.0)
[offsite-apply] offsite configured for u629488-sub6@…:/home/felhom-repo (pending key escrow)
```
**A credential staged after start-up, collected without intervention and without a guest command
line.** In the previous walk this same situation was dead end 1 and needed a guest shell.
## The customer journey, as read from the screen
- **The recovery screen appeared without being sought** — `/` landed on „Adatok visszaszerzése",
claim-gate markers 0.
- It states the seal date, that nobody can replace the code, and that **this step changes nothing**.
*(The seal date still renders as a raw RFC3339 string — `2026-08-06T12:41:55Z` — to a Hungarian
household. Third walk running; still unfixed.)*
- **The unlock took 1.804 s** — a real unseal. For contrast, a submission with an empty field returned
in **25 ms** with „Add meg a helyreállítási kódot" — a **neutral prompt, not a wrong-code
accusation**. That is R-224/R-226 behaving as shipped: unknown defaults to neutral, and the typing
message is reachable only after a real refusal.
- Escrow re-sealed ~15 min later („A helyreállítási kód letétbe helyezve").
## Where the journey ends — and it ends short
Everything cryptographic succeeded. The customer still cannot reach a single file:
1. **R-237** — the restore surface is keyed on apps that are *currently installed* and *currently
marked for future remote backup*. Post-rebuild there are none, so „Nincs telepített alkalmazás",
and the restore page points at a page that says the same thing. Reaching a restore at all required
three undocumented steps in order: re-attach the drives, redeploy the app, toggle it on for
**future** backups. A household that just lost its box does not know what it used to run.
2. **R-238** — and then the button does nothing. „Teljes visszaállítás előkészítése"
(`mode=full`) returns 302 and starts **no job**, leaves the wizard on step 1, shows no error, and
writes **no line to the controller's own debug ring**. `mode=unit` on the identical form works.
**That is why the data half is FAIL rather than PASS.** It is not that a sentinel came back wrong —
nothing came back at all. The sentinels' pre-destruction hashes are recorded above and their originals
are intact on DooPlex, so the comparison can be completed the moment R-238 is fixed.
## Honest accounting of what half 2 did NOT establish
- No sentinel was restored, so **the byte-identity claim is unproven**, in either direction.
- The off-site repository was proven readable *before* the destruction (verification restore), and the
repo held 12.0 MB of unique data — consistent with sentinel C and inconsistent with config+DB alone.
That is corroboration, not proof, that the sentinels are recoverable.
- I used appliance-level SSH (host, not guest) throughout. **No guest command line was used at any
point in either half** — the property the previous walk could not claim.
## Harness faults in half 2, separated from the product's
- Posted the unlock with field `code` instead of `recovery_code`. The product's response to that was
*correct and informative* (the neutral empty-field prompt), which is how I noticed.
- Nested quoting through the double SSH hop (DooPlex → demo-hp → appliance) corrupted several
commands; switched to scp'd script files. Two "0 results" readings were **my broken parser**, not
absence — re-measured by grepping the raw file, exactly the trap the project's own rule names.
- A `Hiba` grep matched JavaScript template strings, not errors.