Part 4 half 1: fresh install lands on vouched 0.127.0/0.203.0 with no hand upgrade
gates / gates (push) Successful in 9s
gates / gates (push) Successful in 9s
Venue: demo-hp VM 323 part4-appliance / guest 9201 / hub customer part4.
Bind 13:53:40 -> agent 0.127.0 ONLINE 13:55:34, no "floor held", controller
0.203.0 from the baked golden. The previous re-walk's reinstall DOWNGRADED the
agent (R-216 part 4); this one did not, because the vouch moved all three fields
(golden_version + agent_version + min_agent) together.
R-218 on a real installation: offsite-credential-retry registered (every 5m0s)
and ticking, 0 "credential retry" lines, and the tier configured unaided 14s
after the settle gate — the previous walk's dead end 1 is gone. Stated precisely
in the journal: this venue proves the SILENT half only; the collecting half was
proven separately with a positive control.
R-225 visible in both states on a live box: "még ismeretlen" before any
snapshot, "12.0 MB / 50 GB" after. Unknown is not zero.
Fixture complete with NO guest command line — the claim used the emailed code
through the real /claim form (the hub keeps only a hash), and the sentinels were
written over SMB with the share mounted so the accented filename never crossed a
shell (verified as hex on both sides, no U+FFFD).
Two new findings, both "presence is not success":
R-234 an offsite run reports success while omitting an app just toggled on —
found ONLY because the pre-destruction verification restore was run
instead of trusting the green tick.
R-235 the console still shows the pairing banner 25 min after the bind, while
promising it refreshes itself.
Half 2 (destroy -> reinstall -> recover -> compare) is owed and NOT started; the
venue is standing and its teardown is owed.
This commit is contained in:
@@ -157,6 +157,9 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour
|
||||
| **R-231** | **`/opt/backup/scripts/` on DooPlex is unversioned host state** — found 2026-08-06 while adding the auto-memory store to the backup set. No repository tracks the scripts that protect the recovery chain, so the edit made that day (`CLAUDE_MEMORY_DIR` in `backup-config.sh`, multi-path restic call in `backup-data.sh`) exists only on the box. This is the same class the part-2 session was closing, found inside the fix for it; the change is transcribed in `felhom.eu/workspace/README.md` so it is at least *recorded*. **Two related facts, both understating current safety:** the backup destination (`/mnt/5_hdd/backup`) is on the **same physical disk** as the workspace it protects, and the DooPlex backup set has **no off-site leg** (`sync-hetzner-backups.sh` is jarrs.eu and pulls *from* Hetzner *to* DooPlex). Bringing a root-owned production backup script under version control, and deciding what installs it, is its own scoped change. | **READY** — owner Viktor |
|
||||
| **R-233** | **The golden bake's acceptance checks were a list of strings the script does not print** — found 2026-08-06 while baking golden 0.203.0 by following `runbooks/RUNBOOK-manual-build.md` §4.1 verbatim. Two of the three named pass markers **cannot ever match**: `overlay2 OK` is not in `build-golden.sh` at all (the line it means is ` docker OK (overlay2; data-root /var/lib/docker)`), and `including mount point … mp1` refers to a volume that stopped existing in `build-golden.sh` **v3.0.0**, when R-165 collapsed the two data volumes into one. The 404 pre-gate's URL was also wrong — the published filename is `golden.tar.zst`, not `felhom-golden-<VER>.tar.zst`, so the pre-gate would 404 for the wrong reason and pass **even when the version already existed**. This is the *"an instrument that can silently drop results is not a measurement"* class landing on the bake's own acceptance check: a grep for an impossible string reads `0` forever, and `0` is indistinguishable from failure. **The bake was never actually unguarded** — the script's own `[ "$drv" = "overlay2" ] || { echo FATAL; exit 1; }` is fail-closed and the run exited 0 with no `FATAL`. The **document** was the broken part, which is why nothing had ever gone wrong and nobody had noticed. **FIXED in the same session:** all markers re-captured from the real log rather than paraphrased, the corrected pre-gate URL, the token handling moved off the command line into an in-VM runner script (the old `--setenv=GITEA_TOKEN=$GT` form put the value where `systemctl show` prints it), a required **positive control** on the token-leak grep, and the vouch step rewritten as the three-field change it actually is. **The general lesson:** a runbook's pass markers must be **copied from a captured log, never written from memory** — §4.0 of that same file already learned this for the qemu launch line and says so; §4.1 had not. | **CLOSED 2026-08-06** — fixed in `RUNBOOK-manual-build.md` |
|
||||
|
||||
| **R-234** | **An off-site run reports success while silently omitting an app the customer just switched on.** Found 2026-08-06 on the Part 4 venue (`part4`, VM 323), and found ONLY because the pre-destruction verification restore was run instead of trusting the green tick. Sequence, measured: a run with no app selected produced **1 snapshot**; `POST /backup/offbox/toggle` enabled `calibre-web` (HTTP 302, and the „Nincs távoli mentésre jelölt alkalmazás" warning disappeared, so the selection HAD landed); the next run finished in 30 s and reported **„✓ Rendben · 12.0 MB · 1 pillanatkép"** — still one snapshot. The app restore then refused: **„offbox: nincs pillanatkép a(z) calibre-web alkalmazáshoz"**. A THIRD run took the snapshot count to 2 and the same restore then succeeded. So a run that the customer sees as a green success did not carry the app they had just enabled, and **nothing in the card distinguishes that from a run that did**. The customer's belief ("my app is off-site") and the truth diverge silently, and they would discover it only at restore — the worst possible moment. The snapshot count is on the same card, which is what makes the omission detectable in hindsight and invisible in the moment. **Not yet root-caused**: the likely shape is that the run captured the app-selection set before the toggle committed, but that is a hypothesis, not a measurement. **This is the exact class the project already has a rule for** — "presence is not success": the run's timestamp and tick record that a run HAPPENED, not that it carried what the customer asked for. | **READY** — owner Viktor |
|
||||
| **R-235** | **The appliance console keeps telling an already-paired box to go and pair itself.** Measured 2026-08-06 on VM 323: **25 minutes after** the operator bind, with the guest provisioned, the controller reporting 0.203.0 and the agent ONLINE, the physical console still displayed „Felhom — a doboz készen áll, és a **párosításra vár**" together with the now-spent pairing code `US3-6GP` — and, in the same panel, the promise **„Ez a képernyő magától frissül — nincs teendő a doboznál"**. It does not refresh. A customer looking at their screen is told the setup has not happened, and is told the screen would have updated if it had. Cosmetic in mechanism, not in effect: it invites the customer to re-pair a working box, or to call for help about a box that is already fine. Same family as R-234 — a surface asserting a state that stopped being true. | **READY** — owner Viktor |
|
||||
|
||||
**Recorded against existing rows by Phase 2:**
|
||||
|
||||
- **R-216 — §4.1 is now MEASURED, not deduced.** The previous session could only offer two absences.
|
||||
|
||||
@@ -0,0 +1,250 @@
|
||||
# PART 4 — the re-walk on a real installation, with no hand upgrade (2026-08-06)
|
||||
|
||||
Task 4 Part 4. **Half 1 (the fixture) is complete and the venue is STANDING.** Nothing has been
|
||||
destroyed; half 2 (destroy → reinstall → recover → compare) is owed and deliberately not started.
|
||||
|
||||
The question this venue exists to answer: **does a box built from the published artifacts get R-218's
|
||||
and R-220's fixes without anyone touching it?** The previous re-walk could not ask that — its reinstall
|
||||
*downgraded* the agent, because the vouched agent was older than the box needed.
|
||||
|
||||
## The venue
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host | `demo-hp` (HP t740), Tier 0 — **VM 323 `part4-appliance`** (321 and 322 untouched) |
|
||||
| Appliance | `part4.felhom.eu` @ **192.168.0.141**, q35/OVMF, 4 cores, 8 GB |
|
||||
| Disks | `scsi0` 200 G · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` |
|
||||
| Guest | LXC **9201 `part4`**, island `169.254.253.2`, LAN `192.168.0.228` |
|
||||
| Hub customer | **`part4`** "Part 4 - no hand upgrade" · host id **`part4-905bdb`** · appliance uuid `79fc7b81-…` |
|
||||
| Off-site | Storage Box sub-account, user `u629488-sub6`, `/home/felhom-repo`, 50 GB |
|
||||
| Teardown | **owed** — VM 323, the hub customer, and the off-site repo. Teardown is by **VM id 323**. |
|
||||
|
||||
**Routing fact worth recording:** DooPlex cannot reach either the appliance or the guest — no ping, no
|
||||
port. `demo-hp` can. Every customer-side call in this journal went through `demo-hp` as a jump host.
|
||||
The guest is **born-on-island** (R-50) and has no route from the workspace.
|
||||
|
||||
## The vouch that made this test possible
|
||||
|
||||
Before the walk, three fields moved together (never one — `golden_version` alone ships a controller onto
|
||||
an older agent than it declares it needs):
|
||||
|
||||
| field | before | after |
|
||||
|---|---|---|
|
||||
| `golden_version` | 0.201.0 | **0.203.0** |
|
||||
| `agent_version` | 0.125.0 | **0.127.0** |
|
||||
| `min_agent` | 0.125.0 | **0.127.0** |
|
||||
|
||||
Rollback values are recorded in the scratch `ROLLBACK-artifacts.txt`; vouching is reversible by
|
||||
re-selecting them and saving — no rebuild, because a bake never deletes the previous package.
|
||||
|
||||
**A consequence that must not be glossed:** raising `min_agent` to 0.127.0 newly **holds the controller
|
||||
floor on three boxes** that were not held before — `c11-36d660` and `rewalk-1ab77d` (agent 0.126.0) and
|
||||
`demo-felhom-8363b5` (0.125.0). That is the R-216 machinery working as designed — the hub refuses to
|
||||
serve a controller past what the box's agent supports — and it is the *safe* state, but it means those
|
||||
boxes stay where they are until their agents move. The two campaign venues are meant to be frozen; the
|
||||
**N100 demo box is not**, and unfreezing it is an owed follow-up.
|
||||
|
||||
## The headline: no hand upgrade
|
||||
|
||||
```
|
||||
13:53:40 bind (POST /appliances/22/bind, HTTP 303)
|
||||
13:54:53 host appears — "— floor held", NO REPORT
|
||||
13:55:34 agent 0.127.0, ONLINE <-- 1 m 54 s after the bind
|
||||
guest 9201 running; controller 0.203.0 reporting
|
||||
```
|
||||
|
||||
**No "floor held" on the new box, and no hand upgrade anywhere.** The agent it installed is the agent
|
||||
the hub vouches; the controller it runs is the one baked into golden 0.203.0. The PBS wrapper reads
|
||||
"matches vouched" and root@pam was rotated and vaulted by day-0.
|
||||
|
||||
## Install — three known traps, all reproduced and handled
|
||||
|
||||
Driven blind: `qm monitor screendump` → PPM → read visually, `qm sendkey` for input.
|
||||
|
||||
1. **GRUB defaults to the graphical entry.** `down`+`ret` sent inside one remote command to hit the
|
||||
~15 s window; the terminal installer came up first try.
|
||||
2. **The keymap defaults to Hungarian while `sendkey` emits US scancodes.** Switched to **U.S. English**
|
||||
before any typing — the list had to be opened with `ret` (arrow keys do not cycle it) and U.S. English
|
||||
is 15 rows below Hungarian. **Positive control:** the administrator email was typed through the
|
||||
identical path and rendered **`part4@felhom.eu`**. `@` is `shift-2` on a US layout and `AltGr+V` on a
|
||||
Hungarian one, so a correct `@` is the only available evidence that the 24 masked password characters
|
||||
went in as intended.
|
||||
3. **`--boot` set in its own `qm set` after the disks existed**, ISO detached, and both verified from
|
||||
`qm config` **before the first boot** (`boot: order=scsi0`, no `ide2`). `Automatically reboot` was
|
||||
unchecked and confirmed `[ ]` **with the focus moved away**, so the reboot was deliberate.
|
||||
|
||||
Summary screen, verbatim: `ext4` · `/dev/sda` · `Europe/Budapest` · **`U.S. English`** ·
|
||||
`part4@felhom.eu` · `nic0` · `part4.felhom.eu` · `192.168.0.141/24` · `192.168.0.1` · `192.168.0.1`.
|
||||
|
||||
The install-time root password was generated into a `0600` file and typed by a host-side script that
|
||||
read that file — it never entered a command line, a unit property or a transcript. Day-0 replaced it
|
||||
anyway and escrowed the replacement to the hub.
|
||||
|
||||
## No guest command line — this walk, unlike the last one, actually holds the line
|
||||
|
||||
The previous re-walk's fixture used the `--print-reset-code` escape hatch, which is a guest command
|
||||
line, and said so. This one did not need it:
|
||||
|
||||
- **The claim** used the code the hub emailed the operator, entered through the real `POST /claim`
|
||||
form. The hub stores only a hash — there is no plaintext to reveal — so this genuinely required the
|
||||
customer's own channel. Verified by a positive observable, not the 302: the session then reached
|
||||
„Indítópult" (36 962 B) with **zero** claim-gate markers.
|
||||
- **The sentinels** were written over **SMB** — a real customer path (R-7) — into a share created
|
||||
through the UI, with the share **mounted** so filenames never crossed a shell.
|
||||
|
||||
Appliance-level SSH *was* used (host, not guest) to restart the agent when the backup-target assign
|
||||
asked for it, and to read the guest's addresses. The break-glass credential came from the hub's own
|
||||
`reveal-recovery-credential` endpoint, file→file, and was never rendered.
|
||||
|
||||
## The fixture
|
||||
|
||||
**Drives.** Baseline first, because it is what makes half 2 legible — on a fresh install
|
||||
`GET /api/disks/candidates` returned **`initialize: [/dev/sdb, /dev/sdc]`, `attach: []`**. That is
|
||||
exactly the call R-220 breaks after a rebuild, so this is the measurement half 2 is compared against.
|
||||
|
||||
Both enrolled through the wizard's own pipeline (`/api/storage/init` → format → mount → register,
|
||||
polled via `/api/storage/init/status`), not by hand-assembling format+assign — the F9 episode is on
|
||||
record as proof that a hand-assembled enrolment proves nothing. The **format-safety gate fired
|
||||
correctly**, returning `needs_confirmation` — "device has a ext4 filesystem" — and only proceeding on
|
||||
an explicit confirmation carrying the durable id the agent itself had returned.
|
||||
|
||||
| drive | device | managed path | raw path |
|
||||
|---|---|---|---|
|
||||
| `adatok` (default) | /dev/sdb | `/mnt/felhom-drives/adatok` | `/mnt/adatok` |
|
||||
| `mentes` | /dev/sdc | `/mnt/felhom-drives/mentes` | `/mnt/mentes` |
|
||||
|
||||
Both mounts exist for each drive — **that double mount is precisely what R-220 is about**.
|
||||
|
||||
**Backup target** assigned to `mentes`, which reported `restart_required: true` and flipped only after
|
||||
the agent restart it asked for:
|
||||
|
||||
```
|
||||
before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"}
|
||||
after: {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}
|
||||
```
|
||||
|
||||
**App.** `calibre-web` deployed through `POST /api/stacks/calibre-web/deploy` with
|
||||
`HDD_PATH=/mnt/felhom-drives/adatok`; `state: running` **and `health_probe.healthy: true`** — the
|
||||
`.State` field alone is known to say "running" for unhealthy containers, so the probe is the claim.
|
||||
|
||||
**The three sentinels**, written over SMB and read back from the share:
|
||||
|
||||
| # | file | bytes | sha256 |
|
||||
|---|---|---|---|
|
||||
| A | `PART4-SENTINEL-A.txt` | 72 | `2de4e291989b5aa8771b361d25198ea389edcee50f4014b77a10345f91cf39d0` |
|
||||
| B | `PART4-őrszem-ékezetes-árvíztűrő.txt` | 76 | `86ac38586de64e3ba80103b39b63ffd47f1e699dad386f5d0b7feddf83f9c5f6` |
|
||||
| C | `PART4-SENTINEL-C-12MB.bin` | 12 582 912 | `461f32bda8323d6955fb09dd0dd4457b5acb0081155cefa4d692c93e8b1148ae` |
|
||||
|
||||
Sentinel B is the encoding canary. Its filename was built from explicit code points and verified **as
|
||||
hex**, never as rendered text — the previous walk lost this file's accents to a
|
||||
`base64 → bash → pct exec` chain, and a Python `decode('utf-8')` check called the result valid because
|
||||
U+FFFD *is* valid UTF-8. Here the name reads
|
||||
`50415254342d c591 72737a656d2d c3a9 …` on both sides — `ő é á í ű ő`, no `efbfbd` anywhere.
|
||||
|
||||
**Escrow ceremony.** Preflight **6 of 6 green** (`pbs_storage_id` · `dr_tier` · `age_binary` ·
|
||||
`hub_upload` · `staged_secret` · `sudo_grant`), `agent_supported: true`. Result:
|
||||
|
||||
```
|
||||
phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24
|
||||
key_fingerprint: 4c:e2:45:5f:48:9c:04:f0:…:71:ff:99:2a · claimable: true
|
||||
```
|
||||
|
||||
`restic_pw_sealed: true` is the field the whole exercise rests on.
|
||||
|
||||
**R** was claimed ONE-SHOT and streamed file→file into `~/.config/part4/R_part4.txt` (`0600`, DooPlex
|
||||
only); the intermediate on the jump host was `shred -u`'d and the raw response deleted. It was never
|
||||
rendered. Shape only: **10 words, 82 characters**. The same runbook tension as last time applies —
|
||||
§5.5 says record *that* a code exists and where, §9.4 says R is never a file on any machine — and it is
|
||||
flagged here rather than resolved quietly: **the operator should move R into their own store and
|
||||
confirm.** Half 2 cannot run without it.
|
||||
|
||||
## R-218, measured on a real installation
|
||||
|
||||
```
|
||||
11:57:30Z [scheduler] Registered periodic job: offsite-credential-retry (every 5m0s)
|
||||
11:57:40Z [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.203.0)
|
||||
11:57:44Z [offsite-apply] offsite configured for u629488-sub6@…:/home/felhom-repo
|
||||
12:02:31Z [scheduler] Running job: offsite-credential-retry (and every 5 min after)
|
||||
```
|
||||
|
||||
`credential retry` log lines: **0**.
|
||||
|
||||
**The previous walk's dead end 1 — "the off-site tier never came up on its own" — is gone**: the tier
|
||||
was configured 14 s after the settle gate, unaided. And the retry job behaves exactly as designed on a
|
||||
healthy box: registered, ticking, completing in 0 s, and **saying nothing**, because the declaration it
|
||||
keys on is false.
|
||||
|
||||
**Stated precisely, because the distinction matters:** this venue proves the *silent* half. It did NOT
|
||||
reproduce the failure condition itself — the credential was already staged before the box's first
|
||||
reconcile, so there was never anything for the retry to collect. The *collecting* half was proven
|
||||
earlier in the session on demo-hp, with a positive control. A venue that proves a retry never fires is
|
||||
not a venue that proves a retry works.
|
||||
|
||||
## R-225, visible on a live box in both states
|
||||
|
||||
The remote-backup card read „Tároló méret · a pillanatképek száma **még ismeretlen**" and
|
||||
„Tárhelykeret: **még nem tudjuk**, mennyi van a tárolóban — legfeljebb 50 GB" **before** any snapshot
|
||||
existed, and „12.0 MB / 50 GB (0%)" **after**. Unknown rendered as unknown, not as zero — the R-225 fix,
|
||||
on a real installation, in both of its states.
|
||||
|
||||
## The off-site copy was proven readable BEFORE anything was destroyed
|
||||
|
||||
`restic`-level listing needs the repository password, which is sealed — so the proof used the
|
||||
customer's own verification restore (`mode=unit`), which restores config+DB into a **separate** folder
|
||||
and leaves live data alone:
|
||||
|
||||
```
|
||||
A(z) calibre-web beállításai és adatbázisa visszaállítva ellenőrző mappába:
|
||||
/mnt/felhom-drives/adatok/backups/
|
||||
```
|
||||
|
||||
The repository opens, decrypts and restores end-to-end. **The gate is not the green tick** — and that
|
||||
is not a slogan here, because running this check is what caught R-234 (below). Had the venue been
|
||||
destroyed on the strength of the tick, half 2 would have failed with the evidence already gone.
|
||||
|
||||
## Two findings, both from the same family
|
||||
|
||||
- **R-234 — an off-site run reports success while silently omitting an app the customer just enabled.**
|
||||
Run with no app selected → 1 snapshot. Toggle `calibre-web` on (302; the "no app selected" warning
|
||||
disappears, so the selection landed). Next run: **„✓ Rendben · 12.0 MB · 1 pillanatkép"** — still one.
|
||||
The app restore then refused: **„nincs pillanatkép a(z) calibre-web alkalmazáshoz"**. A third run
|
||||
produced the second snapshot and the same restore succeeded. Nothing on the card distinguishes the
|
||||
green run that carried the app from the green run that did not.
|
||||
- **R-235 — the console keeps telling an already-paired box to pair itself**, 25 minutes after the bind,
|
||||
while promising „Ez a képernyő magától frissül".
|
||||
|
||||
Both are "presence is not success": a surface reporting that something *happened* rather than that it
|
||||
*worked*.
|
||||
|
||||
## Pre-destruction truth
|
||||
|
||||
Read back from the share immediately before stopping — all three match their originals byte for byte:
|
||||
|
||||
```
|
||||
PART4-SENTINEL-A.txt 72 2de4e291989b5aa8771b361d25198ea389edcee50f4014b77a10345f91cf39d0
|
||||
PART4-SENTINEL-C-12MB.bin 12582912 461f32bda8323d6955fb09dd0dd4457b5acb0081155cefa4d692c93e8b1148ae
|
||||
PART4-őrszem-ékezetes-árvíztűrő.txt 76 86ac38586de64e3ba80103b39b63ffd47f1e699dad386f5d0b7feddf83f9c5f6
|
||||
```
|
||||
|
||||
Box state: `felhom-agent 0.127.0` · controller `0.203.0` · **2 snapshots** off-site · 12.0 MB / 50 GB.
|
||||
|
||||
## Half 2 — owed, not started
|
||||
|
||||
Destroy the guest, reinstall, recover with R, and compare. **PASS = the three sentinels' sha256 are
|
||||
byte-identical AND no guest command line was used.** The R-220 measurement is the same
|
||||
`GET /api/disks/candidates` call recorded above: after the rebuild it must still offer both drives,
|
||||
where before the fix it returned `initialize: [], attach: []` with the deploy refusing by naming a list
|
||||
that was empty.
|
||||
|
||||
## Harness faults, separated from the product's
|
||||
|
||||
- The first `/api/disks/assign` attempt failed (HTTP 400) because `where` is a **path**, not a drive
|
||||
name, and the uuid must be bare. Mine, not the product's — and it is why the enrolment was then done
|
||||
through `/api/storage/init` instead of hand-assembled.
|
||||
- The first `deploy` POST was rejected for a missing `HDD_PATH` that I *had* sent: the body wraps the
|
||||
fields in `values`. The endpoint's own refusal caught it.
|
||||
- I pre-formatted both disks via `/api/disks/format` before switching to the wizard endpoint, which is
|
||||
why the wizard then reported `needs_confirmation` ("device has a ext4 filesystem"). The gate behaved
|
||||
correctly; the extra state was mine.
|
||||
- Grepping the rendered page for `Hiba` matched **JavaScript template strings**, not errors. Counted
|
||||
and discarded rather than reported as failures.
|
||||
Reference in New Issue
Block a user