walk5 (R-201): Phase B + the verdict — BOTH HALVES PASS
THE DATA: PASS. All three sentinels byte-identical out of snapshot 5b0f20f7, including the accented filename's bytes, read back as bytes from the live path. THE JOURNEY: PASS — the first time in five walks. Zero guest command lines were needed to progress; the previous walk needed three. The reset-code hatch was used once, in Phase A only. §5's observation, which stands on its own whatever the verdict: at 14:58:52Z the rebuilt box collected its re-staged credential, configured the transport, and REFUSED TO MINT a repository password over the sealed package the hub holds. At the equivalent moment the previous walk minted a fresh key and lost the journey silently at 03:18. Sampled every 20s from T0: no key at any moment. Honest about which shape fired: with the mint guard holding there is no local key, so the offer comes from shape (a), not shape (c). Shape (c) was measured in Phase A in its NEGATIVE half (equal hashes, correctly silent). This proves the mint guard positively and the discriminator negatively. RTO 71.7s login to open store, of which 12.44s was the unseal and ~22s my own CSRF harness retry. Two new customer-facing obstacles, neither needing a shell but neither signposted: the restore refuses on unattached drives, and refuses because the app is not installed on a page that says the restore reinstalls it.
This commit is contained in:
@@ -411,3 +411,297 @@ DR Recipe: present · Key Escrow: present
|
||||
**Phase A gate: PASSED.** All seven records taken; the three sentinels listed **by name**.
|
||||
|
||||
---
|
||||
|
||||
## Phase B — the journey (§7)
|
||||
|
||||
**The rule: no command line inside the guest, at any point.** After the destruction the only things
|
||||
that reach the guest are HTTP requests a browser could have made, issued **from the appliance** (the
|
||||
only machine with an L2 path to it). **Instrumentation reads are counted separately and named**: the
|
||||
`w5watch.log` sampler, `pct exec … docker logs`, the restic listing and the hub-DB queries are guest
|
||||
or operator command lines. **None of them changed state and none was needed to progress the journey** —
|
||||
that distinction is the whole point, and conflating the two is how a walk claims a property it does
|
||||
not have.
|
||||
|
||||
### B1 — the destruction (14:19:26Z)
|
||||
|
||||
Guarded **four ways** before anything ran, because `demo-hp` also carries a guest 9201:
|
||||
|
||||
```
|
||||
hostname == walk5 ✓
|
||||
pct config 9201 → hostname: walk5 ✓
|
||||
mountpoint -q /mnt/adatok ✓
|
||||
mountpoint -q /mnt/mentes ✓
|
||||
```
|
||||
|
||||
```
|
||||
14:19:18Z pct stop 9201
|
||||
14:19:26Z pct destroy 9201 --purge
|
||||
→ "Logical volume vm-9201-disk-0 successfully removed"
|
||||
→ "Logical volume vm-9201-disk-1 successfully removed"
|
||||
→ pct list EMPTY · lvs | grep 9201 → NONE
|
||||
14:19:27Z /mnt/adatok and /mnt/mentes wiped 13M/1.2G → 4.0K each,
|
||||
MOUNTS LEFT IN PLACE — deliberately: the surviving raw mount IS the R-220 condition
|
||||
```
|
||||
|
||||
### B2 — the rebuild (14:24:09Z, 3 m 31 s)
|
||||
|
||||
`felhom-host-install.sh` fetched **live** from `felhom.eu/scripts/` — `v1.25.0`,
|
||||
sha256 `cb4d02059f126f8bfdddfe98607e87a3cb45dd2592c6b40b43c9659e2b687534`. The retrieval passphrase
|
||||
was passed as `--passphrase-file` (a `0600` file), never on the command line.
|
||||
|
||||
```
|
||||
[OK] Day-0 provision SUCCESS — vmid=9201 host_id=walk5-4bada5 customer=walk5
|
||||
golden=local:backup/vzdump-lxc-9100-2026_08_07-16_21_39.tar.zst
|
||||
[OK] controller: Up 19 seconds (healthy)
|
||||
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.206.0
|
||||
[OK] authz signers: 2 (operator-signed self-update armed)
|
||||
[INFO] root@pam was rotated + vaulted at step 4b
|
||||
```
|
||||
|
||||
**What the rebuild landed on:**
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| agent | 0.127.0 | **0.127.0** — no downgrade, no hand upgrade |
|
||||
| controller | 0.206.0 | **0.206.0** — same, and it is the vouched build |
|
||||
|
||||
**No hand upgrade, either time.** R-216 part 4's reinstall-downgrade did not recur.
|
||||
|
||||
Guest re-provisioned at **192.168.0.83** (it was .149 before — a fresh DHCP lease, not a fault).
|
||||
|
||||
### B3 — §5's observation window: THE MOMENT THAT MATTERS
|
||||
|
||||
**T0 — 14:24:31Z, 22 seconds after provision, nobody has logged in:**
|
||||
|
||||
```
|
||||
offbox directory ABSENT ("No such file or directory")
|
||||
repository key NONE
|
||||
settings.offbox null
|
||||
settings.hub_escrow_identity_present TRUE ← the box already knows a package is held for it
|
||||
settings.hub_escrow_key_sha256 eabf427c7274…144f ← and already knows WHICH key it seals
|
||||
settings.claimed null (pristine, unclaimed)
|
||||
```
|
||||
|
||||
**The scheduler is alive — the positive control, because an absent line is not evidence.** In the
|
||||
first five minutes the controller logged `agent-channel-health` ×5 (1 m cadence), `stack-scan` ×2,
|
||||
`system-health`, `backup-cache` and `offsite-credential-retry`. So "no key was written" is a
|
||||
measurement, not a silence.
|
||||
|
||||
**The window, minute by minute — nobody logged in at any point during it:**
|
||||
|
||||
| time (UTC) | event | source |
|
||||
|---|---|---|
|
||||
| 14:24:09 | Day-0 provision SUCCESS; rebuilt guest, controller **0.206.0** | installer |
|
||||
| 14:24:31 | **T0** — no offbox dir, no key, `offbox: null`, but `hub_escrow_identity_present` already **true** and the sealed key's hash already learned | box, **measured** |
|
||||
| 14:24:53 | hub: `offsite_delivery_stuck` (warning) + operator `notification_log` row — **REFUSED**, "regressed-apply shape … a guest rebuild does, R-193" | hub, **measured** |
|
||||
| 14:23:54 | first post-rebuild report carries **no** `offsite` object — it predates the ACK that set the flag | hub `reports`, **measured** |
|
||||
| **14:38:49** | **first declaration**: `offsite={"enabled":false,"state":"needs_credential"}` | hub `reports`, **measured** |
|
||||
| 14:38:53 | the checker **hands off**: *"the box DECLARES offsite.state=needs_credential; internal/offsiteheal owns this remediation"* | hub, **measured** |
|
||||
| 14:53:49 | second declaration, same shape | hub `reports`, **measured** |
|
||||
| **14:56:34** | `offsiteheal: re-staged the stored one-time offsite secret … (declared needs_credential across 2 reports) … no provider credential was minted` | hub, **measured** |
|
||||
| **14:58:52** | **the box collects it and applies the tier — AND REFUSES TO MINT** | box log, **measured** |
|
||||
|
||||
### THE LINE THIS WALK EXISTED TO SEE
|
||||
|
||||
```
|
||||
14:58:52 [WARN] [offbox] NOT minting a repository password: the hub holds a sealed recovery package
|
||||
for this box, and a fresh key would orphan the history that package protects (R-241).
|
||||
The transport is configured; the tier stays down until the customer's recovery code places
|
||||
the escrowed key.
|
||||
14:58:52 [INFO] [offbox] apply-offsite: transport configured for u629488-sub4@…:/home/felhom-repo,
|
||||
tier HELD awaiting the escrowed key (the hub holds a sealed package; no key was minted — R-241)
|
||||
14:58:52 [INFO] [offsite-apply] credential retry: the staged credential was collected and the tier applied
|
||||
```
|
||||
|
||||
**At the equivalent moment the previous walk minted `9b4a9a9d…` over a package sealing `30ef574f…`,
|
||||
and lost the journey silently at 03:18 in the morning. This build did not.**
|
||||
|
||||
### §5's four items, answered
|
||||
|
||||
**1. Does the box declare a need, and when is it staged and collected?** **Yes** — declared
|
||||
`needs_credential` at **14:38:49Z** and **14:53:49Z**, staged by the hub unaided at **14:56:34Z**,
|
||||
collected and applied at **14:58:52Z**. **Zero human actions**; the whole chain ran on a box that had
|
||||
not even been claimed.
|
||||
|
||||
**2. Is any repository key written?** **NO.** Sampled every ~20 s from 14:24:52Z onward:
|
||||
`repo_password_sha=NONE` at every single sample. At 15:00:24Z the directory holds exactly:
|
||||
|
||||
```
|
||||
-rw------- applied_marker 64
|
||||
-rw-r--r-- known_hosts 290
|
||||
-rw------- ssh_key 400
|
||||
repo_password: ABSENT
|
||||
```
|
||||
|
||||
**The transport is present and the key is not** — which is the guard's designed shape, not a partial
|
||||
failure: `ApplyOffsiteTarget` deliberately keeps the transport so the recovery screen can bring the
|
||||
tier up the instant the key arrives (R-219).
|
||||
|
||||
**3. What state does it report instead?** `offbox.enabled = true`, `escrow_state = pending`,
|
||||
`repo_state` absent — and `OffboxAwaitingRecoveryKey()` is therefore **true** (target exists, target
|
||||
enabled, no password), so the box declares the new **`awaiting_recovery_key`** holding state rather
|
||||
than coming up under a fresh key.
|
||||
|
||||
**4. The two fingerprints, before anyone logged in (15:00:24Z):**
|
||||
|
||||
```
|
||||
the escrowed key (hub's package, ACK-cached) eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
|
||||
whatever the box holds NONE — there is no repository key on this box
|
||||
```
|
||||
|
||||
> **HONEST NOTE ON WHICH SHAPE FIRES, because it changes what has been proven.** With the mint guard
|
||||
> holding, the box has **no local key at all**, so `OffsiteRecoveryOffer` returns true on **shape (a)**
|
||||
> — the pristine-rebuild shape — not on shape (c). **Shape (c) was measured in Phase A, in its
|
||||
> negative half**: equal hashes, correctly silent. So this walk proves the mint guard **positively**
|
||||
> and the discriminator **negatively**; a positive shape-(c) firing needs a box holding a *different*
|
||||
> key, which v0.206.0 now prevents from arising by itself. That is the intended end state (the fix
|
||||
> removes the state its own second half was written to describe), but it should not be reported as
|
||||
> "shape (c) proven".
|
||||
|
||||
### B4 — the journey, step by step (§7)
|
||||
|
||||
| # | step | result |
|
||||
|---|---|---|
|
||||
| 1 | **Destroy** — 14:19:26Z | guest 9201 purged (both LVs), both drives wiped 13M/1.2G → 4.0K. Host identity `walk5-4bada5` survived |
|
||||
| 2 | **Reinstall** | `felhom-host-install.sh` **v1.25.0** fetched live; **Day-0 provision SUCCESS 14:24:09Z**, 3 m 31 s. **controller 0.206.0, agent 0.127.0 — no hand upgrade, no downgrade** |
|
||||
| 3 | **§5's window** | **no key minted**; transport configured; `awaiting_recovery_key` — see B3 |
|
||||
| 4 | **Claim back** | new code requested through the customer-facing „Új kód kérése" path (15:01:27Z), hub rotated to **generation 2** and emailed it; the operator relayed it; **worked FIRST TRY** — 302 in **0.151 s**, accents intact (`á`=`c3a1`, `í`=`c3ad`, sha256 identical source→appliance) |
|
||||
| 5 | **Log in** | `/` → 302 `/launcher` → 302 **`/recovery`** — **the recovery screen appeared WITHOUT BEING SOUGHT** |
|
||||
| 6 | **Read the screen** | all three questions answered — quoted below |
|
||||
| 7 | **Enter R** | **HTTP 200 in 12.44 s** — a real unseal. `[web] recovery: the offsite repository key was recovered and placed (outcome=installed)` |
|
||||
| 8 | **The listing** | **rendered** — `calibre-web · 2026-08-07 14:57 · 12.8 MB` |
|
||||
| 9 | **Restore** | **all three sentinels byte-identical** — after two unsignposted obstacles, both cleared from the dashboard |
|
||||
|
||||
### The recovery screen, as a customer reads it (§7.6)
|
||||
|
||||
> **„Adatok visszaszerzése"**
|
||||
>
|
||||
> „Ezt a gépet újratelepítették. **A korábbi, házon kívüli mentéseid megvannak** — a Felhom központi
|
||||
> rendszere őriz hozzájuk egy lezárt csomagot, amelyet **2026-08-07T12:51:02Z** zártunk le. A csomagot
|
||||
> csak a **te helyreállítási kódoddal** lehet kinyitni.
|
||||
>
|
||||
> **A helyreállítási kódot senki nem tudja pótolni** — sem a Felhom, sem az ügyfélszolgálat, sem az
|
||||
> üzemeltető. Ez szándékos: így a mentéseidet rajtad kívül senki nem tudja megnyitni. Ha a kód
|
||||
> elveszett, a korábbi mentések nem nyithatók meg többé.
|
||||
>
|
||||
> Ha megadod a kódot, **feloldjuk a mentéseid zárolását és megmutatjuk, mi van bennük** — melyik
|
||||
> alkalmazás, mikorról, mekkora. **Ebben a lépésben semmit nem állítunk vissza és semmi nem változik.**"
|
||||
|
||||
**All three questions answered.** The sealed-at timestamp **`2026-08-07T12:51:02Z` matches the hub's
|
||||
`host_escrow.created_at` exactly**. The field label reads „Helyreállítási kód (tíz szó)" and R is
|
||||
indeed ten tokens.
|
||||
|
||||
*(One copy wart, recorded not filed: the sealed-at is rendered as a raw ISO-8601 string on a
|
||||
customer-facing Hungarian screen, where every other date on the box is formatted `2026-08-07 14:57`.)*
|
||||
|
||||
### The listing, against Phase A's record (§7.8)
|
||||
|
||||
| | Phase A recorded | the screen showed |
|
||||
|---|---|---|
|
||||
| app | `calibre-web` | **`calibre-web`** ✅ |
|
||||
| when | snapshot `5b0f20f7` at 12:57:41Z | **`2026-08-07 14:57`** ✅ (12:57Z rendered in CEST) |
|
||||
| size | `restic stats` 12.784 MiB | **`12.8 MB`** ✅ |
|
||||
|
||||
*(A second row `felhom-offbox · 12.8 MB` also appears — the snapshot carries both tags and the screen
|
||||
renders one row per tag. A customer sees an "app" they have never installed, and the same 12.8 MB
|
||||
counted twice. Cosmetic; filed as R-251.)*
|
||||
|
||||
### THE RTO (§7.7)
|
||||
|
||||
```
|
||||
login 15:04:25.417Z
|
||||
unlock returns 200 15:05:37.156Z
|
||||
------------------------------------------
|
||||
wall clock, login → open store 71.7 s
|
||||
```
|
||||
|
||||
**Of which 12.44 s was the unseal itself.** ~22 s of the remainder was **my own harness retry** — I
|
||||
scraped the CSRF token from a `<meta>` tag the recovery page does not carry, got a 403, and re-read it
|
||||
from the form (the controller logged `CSRF rejected … (token mismatch)` at 15:05:02). **A customer
|
||||
clicking the button would have seen ≈50 s.** Both numbers are stated because the 71.7 s is what was
|
||||
actually measured.
|
||||
|
||||
### The restore, and the two obstacles (§7.9)
|
||||
|
||||
`POST /backup/offbox/restore` (mode=full) → **refused**: „nincs elérhető adatmeghajtó a
|
||||
visszaállításhoz". Then, after that was cleared, `POST /backup/offbox/reconstitute` → **refused**:
|
||||
„a(z) calibre-web nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat".
|
||||
|
||||
Both were cleared **from the dashboard, with no guest shell**:
|
||||
|
||||
1. **Re-attach the drives** — `GET /api/disks/candidates` offered both surviving disks
|
||||
(`mountable: true`, `data_bearing: true`), and Tárhely → Meghajtók → „Meglévő meghajtó csatolása"
|
||||
(`POST /api/storage/attach`) registered them: `/mnt/felhom-drives/adatok` and `…/mentes`.
|
||||
2. **Redeploy the app** — `POST /api/stacks/calibre-web/deploy`, running in 1 m 30 s.
|
||||
|
||||
Then prepare (12.8 MB) → confirm → reconstitute:
|
||||
|
||||
```
|
||||
15:10:18 [offbox] restored calibre-web (5b0f20f7, full=true) → …/backups/offsite-restore/calibre-web
|
||||
15:14:30 [offbox] reconstituted calibre-web from snapshot 5b0f20f7: 6 file(s) placed,
|
||||
0 DB dump(s) replayed, safety dump=., skewed=false
|
||||
```
|
||||
|
||||
### THE THREE SENTINELS, read back as BYTES from the live path
|
||||
|
||||
```
|
||||
MATCH [A] 11eb7fb2d1891f62a685e3d9a9fda44e0a342d437d4aebaa15fcb4c0fd79d539 61
|
||||
name_hex 57414c4b352d53454e54494e454c2d412e747874
|
||||
MATCH [B] 6b504d1e83bf4c673334d0b9e701f3e5f07311d954fdc4463a6b0a55b261ae32 66
|
||||
name_hex 57414c4b352d c591 72737a656d2d c3a9 6b657a657465732d c3a1 7276 c3ad 7a74 c5b1 72 c591 2e747874
|
||||
MATCH [C] 0baaf402733e6a85f3d9ce8c4c3a57b05692f5f6733da6a23da76d2cd7635210 12582912
|
||||
name_hex 57414c4b352d53454e54494e454c2d432d31324d422e62696e
|
||||
all three present: True
|
||||
```
|
||||
|
||||
**Identical to Phase A in every byte, including the accented filename.** Read with `os.listdir` on a
|
||||
**bytes** path, so no decode/encode round trip could launder a `U+FFFD` into looking correct.
|
||||
|
||||
### The end state — the loop closed
|
||||
|
||||
```
|
||||
escrow_state escrowed (auto-confirmed: hub hash == local hash)
|
||||
repo_password sha256 eabf427c7274…144f == the escrowed key, == Phase A's key
|
||||
GET /launcher 200 (the recovery screen no longer interrupts)
|
||||
GET /recovery 302 (correctly retired — shape (c) compares equal again)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# THE VERDICT — both halves, separately
|
||||
|
||||
## THE DATA: **PASS**
|
||||
|
||||
All three sentinels came back **byte-identical**, out of snapshot `5b0f20f7`, under the key recovered
|
||||
from the sealed package with R — including the accented filename's **bytes**. Disk → restic → SFTP →
|
||||
Storage Box → a destroyed machine → a rebuilt machine → disk, intact.
|
||||
|
||||
## THE JOURNEY: **PASS — the first time in five walks**
|
||||
|
||||
**No step needed a command line inside the guest.** Every step that *progressed* the journey — request
|
||||
a code, claim, log in, read the screen, enter R, read the listing, re-attach the drives, redeploy the
|
||||
app, prepare, confirm, restore — was an HTTP request a browser makes, issued from the appliance
|
||||
because DooPlex has no L2 path to that LAN.
|
||||
|
||||
**Guest command lines used, named per §3 so the claim is not read wider than it is:** the `w5watch.log`
|
||||
sampler, `docker logs`, the settings reads, the restic listing and the final sentinel verification.
|
||||
**Every one is instrumentation** — none changed state, none was needed to progress, and removing them
|
||||
all would have changed nothing except my ability to describe what happened. The previous walk needed
|
||||
**three** guest command lines to progress; this one needed **zero**.
|
||||
|
||||
**The reset-code hatch was used ONCE, in Phase A**, where §3 permits it. It was **not** used in Phase B.
|
||||
|
||||
### But the journey is not smooth, and two obstacles are new
|
||||
|
||||
Neither is a dead end by §3's definition (neither needed a guest shell), and both were cleared from
|
||||
the dashboard — **but a customer would not know to do either**, because the product does not say so
|
||||
and, in the second case, says the opposite:
|
||||
|
||||
1. **The restore refuses with „nincs elérhető adatmeghajtó" and nothing on the recovery path mentions
|
||||
that the drives must be re-attached first.** → **R-252**
|
||||
2. **„a(z) calibre-web nincs telepítve — előbb állítsd helyre az alkalmazást" — on a page that says,
|
||||
three lines above, „Nincs telepítve — a visszaállítás előbb újratelepíti."** The product
|
||||
contradicts itself in the customer's own language, at the last step of a recovery. → **R-253**
|
||||
|
||||
**So: the machinery works end to end and the data is provably safe. The unaided journey now succeeds,
|
||||
and it succeeds through two obstacles the customer must guess their way past.**
|
||||
|
||||
Reference in New Issue
Block a user