diff --git a/STATUS.md b/STATUS.md index 900260d..40364a5 100644 --- a/STATUS.md +++ b/STATUS.md @@ -46,6 +46,13 @@ code was wrong. *(CAMPAIGN 11)* ## What shipped recently +- **2026-08-05 (late)** — **New machines now get current software again — and the disc image had been + quietly un-updatable for days.** Approving the newer in-house service turned out to be impossible on + its own: the system correctly refuses to publish a set where the pre-built machine image is older + than the software the fleet already runs, and that image had been behind since late July. **So the + image was rebuilt and both were published together.** A machine installed from now on lands on + current software and can open a recovery package on day one. *(R-223)* + - **2026-08-05 (evening)** — **Four sentences where there was one, and none of them blames you.** "We did not accept your recovery code" used to appear when the code was wrong, when the machine could not ask, when the store could not be read, and when the customer held the code for an older @@ -118,13 +125,6 @@ code was wrong. *(CAMPAIGN 11)* ## Waiting on you -- **One click, and it is the most valuable one available: approve host-service version 0.125.0 for new - machines.** Hub → Configuration → Day-0 artifacts → agent. Today new machines get 0.120.0, which - **cannot** open a recovery package — and a reinstall actively puts the older one back over a machine - we fixed by hand, so every rebuild re-breaks the very thing a rebuild needs. 0.125.0 has run on both - demo machines since 4 August and through the entire campaign. **Until you do this, new machines are - correctly held back rather than lied to — which is better, but the feature does not work for them.** - *(R-223)* - **One thing to read after the machine next restarts — and nothing to do until then.** You told me not to restart DooPlex, so I did not, and the move to the second SSD has therefore never been diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index c40bf85..f07a096 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -73,7 +73,7 @@ open** and each blocks a real flow. Evidence: `tests/campaign11-evidence-2026-08 | **R-214** | **The physical console never stops asking to be paired.** Half an hour after `Day-0 provision SUCCESS`, with the host ONLINE, the console still showed the pairing banner and a stale code — on a screen whose own text promises *„Ez a képernyő magától frissül"*. Census: exactly two `/dev/console` writers in the whole day-0 path, both in the pairing loop; `felhom-host-install.sh` writes to the console not at all | **OPEN — NOT FIXED** | | **R-220** | **After a rebuild the customer's drives cannot be re-enrolled, and the refusal names an impossible action.** The deploy refuses (*„Válasszon a listából csatlakoztatott meghajtót"*) and the list is empty: `claim.go:84` treats a device mounted outside `/mnt/felhom-drives` as claimed, and the raw `/mnt/` mount that enrolment itself creates survives the guest rebuild while the controller's registry does not. **Red-proved**: unmounting only the raw mounts flipped `attach: []` → both drives. **This is the state Campaign 10 reached by hand and recorded as its own harness error; the product's rebuild path now arrives there.** Breaches I3 | **OPEN — NOT FIXED** | | **R-221** | **A rebuilt box cannot run the escrow ceremony at all.** The preflight refuses on `escrow.pbs_storage_id`, which the pbsdr bridge seeds into `agent.json` only via `finishConverged`. The convergence marker lives on the HOST and survives a guest rebuild; `agent.json` is rewritten by the installer. Unchanged descriptor → same hash → early return → the seed never runs into a config that no longer has it. **Red-proved**: moving only the marker aside seeded it instantly (`grep -c escrow`: 0 → 1). A real blocker for re-escrow, which is exactly what a rebuilt box must do | **OPEN — NOT FIXED** | -| **R-223** | **The Day-0 manifest vouches agent 0.120.0 while the recovery feature needs 0.125.0 — and a reinstall DOWNGRADES a box that was fixed by hand.** Verbatim from the second reinstall: `agent (existing): felhom-agent 0.125.0` → `manifest: agent v0.120.0` → `installed /usr/local/bin/felhom-agent (felhom-agent 0.120.0)`. So every rebuild re-breaks the recovery path — the one event that makes the feature necessary | **WAITING-ON-OPERATOR** — vouch agent 0.125.0 in Day-0 artifacts. One dropdown; it has run on both demo boxes since 2026-08-04 and through the whole campaign | +| **R-223** | **The Day-0 manifest vouched agent 0.120.0 while the recovery feature needs 0.125.0 — and a reinstall DOWNGRADES a box that was fixed by hand.** Verbatim from the second reinstall: `agent (existing): felhom-agent 0.125.0` → `manifest: agent v0.120.0` → `installed /usr/local/bin/felhom-agent (felhom-agent 0.120.0)`. So every rebuild re-broke the recovery path — the one event that makes the feature necessary. **⚠ AND IT WAS NOT A DROPDOWN.** The first vouch attempt was REFUSED by R-120's gate (`configs.go:1162`): *"golden 0.192.0 is older than the newest controller the fleet reports (0.201.0)"*. **The artifacts form saves as a unit, so the agent could not be vouched while the golden was stale — and the golden had been stale since controller 0.193.0, meaning the Day-0 manifest had been effectively UNVOUCHABLE for days and nobody had cause to notice.** The real remedy was a golden rebake | **CLOSED 2026-08-05.** Golden **0.201.0** baked in the drill VM (658 165 766 B, sha `e730d7cab343eb35…f007654`, **round-trip verified from Gitea**), then manifest set in one save: `agent=0.125.0 golden=0.201.0 min_agent=0.125.0`. A fresh install now lands on current agent AND current controller | **Untouched by this session, stated so nothing is presumed closed by association:** **R-213** (putting files back — the half the recovery screen deliberately does not do) and **R-202** (the orphan card's diff --git a/documentation/tests/campaign11-evidence-2026-08-05/journal.md b/documentation/tests/campaign11-evidence-2026-08-05/journal.md index b11222f..56af378 100644 --- a/documentation/tests/campaign11-evidence-2026-08-05/journal.md +++ b/documentation/tests/campaign11-evidence-2026-08-05/journal.md @@ -1127,3 +1127,78 @@ This is also the live proof of **R-202**'s open concern: the orphan card promise hozzá tartozó helyreállítási kóddal később visszaállítható lehet"*. Here is a customer **with** that code, and the answer is "wrong code". → **R-222.** Per the brief: the gap is recorded, **nothing was built.** + + +--- + +## The fix pass (2026-08-05, evening) — and the golden rebake it turned into + +Six findings fixed in controller **v0.201.0** + hub **v0.97.0/0.97.1**, each red-proofed. Full account: +`felhom-controller/REPORT.md`. + +### R-223 — vouching agent 0.125.0 was NOT a dropdown, and the manifest had been frozen + +The first vouch attempt was **REFUSED**, correctly, by R-120's gate (`hub/internal/web/configs.go:1162`): + +``` +[WARN] artifact vouch REFUSED: golden 0.192.0 is older than the newest controller + the fleet reports (0.201.0) — a fresh install would land on stale application code (R-120) +``` + +The manifest was **verified unchanged afterwards** rather than inferred from the HTTP 303 (which is a +redirect carrying the refusal, not a success). + +**The artifacts form saves as a unit**, so the agent could not be vouched while the golden was stale — +and the golden had been 0.192.0 since before controller 0.193.0 shipped. **The Day-0 manifest had +therefore been effectively unvouchable for days, and nobody had cause to notice because nobody tried.** +That is the finding; the "one dropdown" I first reported was wrong. + +### The rebake — golden 0.201.0 + +Baked in the drill VM on DooPlex (the sanctioned exception for bakes, `target-selection.md`). +Preconditions checked first: 6.6 T free, **no qemu held the qcow2 — proved by the revert succeeding**, +`virgin` intact, Debian template `13.6-1` listed rather than assumed (the point release rots) and +checksum-verified on download. + +| | | +|---|---| +| artifact | **658 165 766 B**, sha256 `e730d7cab343eb35bcef627829b40c160f83945e0a9bd57da546efc21f007654` | +| markers | `Result=success` · `ExecMainStatus=0` · **no** `FATAL`/`excluding`/`ERROR` · `docker OK (overlay2; data-root /var/lib/docker)` · `including mount point rootfs ('/')` **and** `mp0 ('/var/lib/felhom')` · `upload OK (HTTP 201)` | +| round trip | **the bytes downloaded from Gitea hash identically.** A local checksum proves only that the local file is what it is | +| token hygiene | token read from a 0600 file **inside** the VM, never on a command line; literal-value grep on the saved log → **0 hits** | +| teardown | guest 9100 `--purge`d · token/script/log shredded **after** the log was copied out for evidence · qemu exited (checked by `ps -eo comm`, not `pgrep -f`, which self-matches) · disk reverted to `virgin` | + +*Doc drift noticed, not acted on: the runbook's pass markers say `including mount point` for rootfs +**and mp0 and mp1**. `build-golden.sh` v3.0.0 bakes ONE data volume, so two mount points is correct +and the runbook describes an older layout.* + +### The manifest, read back after the save + +``` +agent_version = 0.125.0 agent_sha256 = f7d8339b53d92a6c…7016b3 +golden_version = 0.201.0 golden_sha256 = e730d7cab343eb35…f007654 +min_agent = 0.125.0 +``` + +**A machine installed from now on lands on current agent AND current controller**, and can open a +recovery package on day one. R-223 closed. + +### The held floor — what is and is not established + +**Established, positively, that the hold WAS in force**: the hub logged it with the corrected reason +(*"floor 0.200.0 is ABOVE the vouched golden 0.192.0…"*, 22:12:06), and the box observed it from its own +side — `[offsite-apply] settle-gate: GO — floor still unknown after 1m30s`, which is what a withheld +directive looks like to a controller. + +**NOT established by a positive observable: that the hold is now released.** After the vouch, c11's +next report (22:27:06) produced **no** HELD line and the Hosts dashboard renders **no** held reason — +but both of those are absences, and this project's own rule is that an absent log line is not evidence. +The hub logs nothing on the served path by design, and `SetFloor` is DEBUG-only and fires only on a +change, so the box-side positive would need a controller restart — **which is not taken, because Phase 2 +needs the venue as it is.** + +What can be said with certainty is the arithmetic the code performs: floor `0.200.0` ≤ golden `0.201.0` +(so the beyond-golden branch cannot fire) and agent `0.125.0` ≥ min_agent `0.125.0` (so the +below-MinAgent branch cannot fire) — the exact combination +`TestResolveManagedFloor_R216_FloorAboveGolden/floor_at_the_golden_→_unchanged_behaviour` pins as +served. **Deduction plus a pinned test, not a live positive. Recorded as such.**