docs: R-223 CLOSED by a golden rebake — and it was never a dropdown
gates / gates (push) Successful in 11s
gates / gates (push) Successful in 11s
The first vouch of agent 0.125.0 was REFUSED by R-120's gate: the artifacts form saves as a unit, and golden 0.192.0 was older than the newest controller the fleet reports. The golden had been stale since before controller 0.193.0, so the Day-0 manifest had been effectively unvouchable for days and nobody had cause to notice. Correcting STATUS.md and OPEN-ITEMS, which said "one dropdown". Golden 0.201.0 baked in the drill VM, round-trip verified from Gitea (e730d7cab343eb35...f007654, 658165766 B), teardown clean, 0 token hits in the saved log. Manifest now agent=0.125.0 golden=0.201.0 min_agent=0.125.0, read back after the save. The held floor: the HOLD is established positively (hub log + the box's own "floor still unknown"). Its RELEASE is not — no HELD line and no dashboard reason are both absences, the served path logs nothing by design, and the box-side positive needs a restart the venue cannot take before Phase 2. Recorded as deduction plus a pinned test, not a live positive.
This commit is contained in:
@@ -46,6 +46,13 @@ code was wrong. *(CAMPAIGN 11)*
|
|||||||
|
|
||||||
## What shipped recently
|
## What shipped recently
|
||||||
|
|
||||||
|
- **2026-08-05 (late)** — **New machines now get current software again — and the disc image had been
|
||||||
|
quietly un-updatable for days.** Approving the newer in-house service turned out to be impossible on
|
||||||
|
its own: the system correctly refuses to publish a set where the pre-built machine image is older
|
||||||
|
than the software the fleet already runs, and that image had been behind since late July. **So the
|
||||||
|
image was rebuilt and both were published together.** A machine installed from now on lands on
|
||||||
|
current software and can open a recovery package on day one. *(R-223)*
|
||||||
|
|
||||||
- **2026-08-05 (evening)** — **Four sentences where there was one, and none of them blames you.**
|
- **2026-08-05 (evening)** — **Four sentences where there was one, and none of them blames you.**
|
||||||
"We did not accept your recovery code" used to appear when the code was wrong, when the machine
|
"We did not accept your recovery code" used to appear when the code was wrong, when the machine
|
||||||
could not ask, when the store could not be read, and when the customer held the code for an older
|
could not ask, when the store could not be read, and when the customer held the code for an older
|
||||||
@@ -118,13 +125,6 @@ code was wrong. *(CAMPAIGN 11)*
|
|||||||
|
|
||||||
## Waiting on you
|
## Waiting on you
|
||||||
|
|
||||||
- **One click, and it is the most valuable one available: approve host-service version 0.125.0 for new
|
|
||||||
machines.** Hub → Configuration → Day-0 artifacts → agent. Today new machines get 0.120.0, which
|
|
||||||
**cannot** open a recovery package — and a reinstall actively puts the older one back over a machine
|
|
||||||
we fixed by hand, so every rebuild re-breaks the very thing a rebuild needs. 0.125.0 has run on both
|
|
||||||
demo machines since 4 August and through the entire campaign. **Until you do this, new machines are
|
|
||||||
correctly held back rather than lied to — which is better, but the feature does not work for them.**
|
|
||||||
*(R-223)*
|
|
||||||
|
|
||||||
- **One thing to read after the machine next restarts — and nothing to do until then.** You told me
|
- **One thing to read after the machine next restarts — and nothing to do until then.** You told me
|
||||||
not to restart DooPlex, so I did not, and the move to the second SSD has therefore never been
|
not to restart DooPlex, so I did not, and the move to the second SSD has therefore never been
|
||||||
|
|||||||
@@ -73,7 +73,7 @@ open** and each blocks a real flow. Evidence: `tests/campaign11-evidence-2026-08
|
|||||||
| **R-214** | **The physical console never stops asking to be paired.** Half an hour after `Day-0 provision SUCCESS`, with the host ONLINE, the console still showed the pairing banner and a stale code — on a screen whose own text promises *„Ez a képernyő magától frissül"*. Census: exactly two `/dev/console` writers in the whole day-0 path, both in the pairing loop; `felhom-host-install.sh` writes to the console not at all | **OPEN — NOT FIXED** |
|
| **R-214** | **The physical console never stops asking to be paired.** Half an hour after `Day-0 provision SUCCESS`, with the host ONLINE, the console still showed the pairing banner and a stale code — on a screen whose own text promises *„Ez a képernyő magától frissül"*. Census: exactly two `/dev/console` writers in the whole day-0 path, both in the pairing loop; `felhom-host-install.sh` writes to the console not at all | **OPEN — NOT FIXED** |
|
||||||
| **R-220** | **After a rebuild the customer's drives cannot be re-enrolled, and the refusal names an impossible action.** The deploy refuses (*„Válasszon a listából csatlakoztatott meghajtót"*) and the list is empty: `claim.go:84` treats a device mounted outside `/mnt/felhom-drives` as claimed, and the raw `/mnt/<name>` mount that enrolment itself creates survives the guest rebuild while the controller's registry does not. **Red-proved**: unmounting only the raw mounts flipped `attach: []` → both drives. **This is the state Campaign 10 reached by hand and recorded as its own harness error; the product's rebuild path now arrives there.** Breaches I3 | **OPEN — NOT FIXED** |
|
| **R-220** | **After a rebuild the customer's drives cannot be re-enrolled, and the refusal names an impossible action.** The deploy refuses (*„Válasszon a listából csatlakoztatott meghajtót"*) and the list is empty: `claim.go:84` treats a device mounted outside `/mnt/felhom-drives` as claimed, and the raw `/mnt/<name>` mount that enrolment itself creates survives the guest rebuild while the controller's registry does not. **Red-proved**: unmounting only the raw mounts flipped `attach: []` → both drives. **This is the state Campaign 10 reached by hand and recorded as its own harness error; the product's rebuild path now arrives there.** Breaches I3 | **OPEN — NOT FIXED** |
|
||||||
| **R-221** | **A rebuilt box cannot run the escrow ceremony at all.** The preflight refuses on `escrow.pbs_storage_id`, which the pbsdr bridge seeds into `agent.json` only via `finishConverged`. The convergence marker lives on the HOST and survives a guest rebuild; `agent.json` is rewritten by the installer. Unchanged descriptor → same hash → early return → the seed never runs into a config that no longer has it. **Red-proved**: moving only the marker aside seeded it instantly (`grep -c escrow`: 0 → 1). A real blocker for re-escrow, which is exactly what a rebuilt box must do | **OPEN — NOT FIXED** |
|
| **R-221** | **A rebuilt box cannot run the escrow ceremony at all.** The preflight refuses on `escrow.pbs_storage_id`, which the pbsdr bridge seeds into `agent.json` only via `finishConverged`. The convergence marker lives on the HOST and survives a guest rebuild; `agent.json` is rewritten by the installer. Unchanged descriptor → same hash → early return → the seed never runs into a config that no longer has it. **Red-proved**: moving only the marker aside seeded it instantly (`grep -c escrow`: 0 → 1). A real blocker for re-escrow, which is exactly what a rebuilt box must do | **OPEN — NOT FIXED** |
|
||||||
| **R-223** | **The Day-0 manifest vouches agent 0.120.0 while the recovery feature needs 0.125.0 — and a reinstall DOWNGRADES a box that was fixed by hand.** Verbatim from the second reinstall: `agent (existing): felhom-agent 0.125.0` → `manifest: agent v0.120.0` → `installed /usr/local/bin/felhom-agent (felhom-agent 0.120.0)`. So every rebuild re-breaks the recovery path — the one event that makes the feature necessary | **WAITING-ON-OPERATOR** — vouch agent 0.125.0 in Day-0 artifacts. One dropdown; it has run on both demo boxes since 2026-08-04 and through the whole campaign |
|
| **R-223** | **The Day-0 manifest vouched agent 0.120.0 while the recovery feature needs 0.125.0 — and a reinstall DOWNGRADES a box that was fixed by hand.** Verbatim from the second reinstall: `agent (existing): felhom-agent 0.125.0` → `manifest: agent v0.120.0` → `installed /usr/local/bin/felhom-agent (felhom-agent 0.120.0)`. So every rebuild re-broke the recovery path — the one event that makes the feature necessary. **⚠ AND IT WAS NOT A DROPDOWN.** The first vouch attempt was REFUSED by R-120's gate (`configs.go:1162`): *"golden 0.192.0 is older than the newest controller the fleet reports (0.201.0)"*. **The artifacts form saves as a unit, so the agent could not be vouched while the golden was stale — and the golden had been stale since controller 0.193.0, meaning the Day-0 manifest had been effectively UNVOUCHABLE for days and nobody had cause to notice.** The real remedy was a golden rebake | **CLOSED 2026-08-05.** Golden **0.201.0** baked in the drill VM (658 165 766 B, sha `e730d7cab343eb35…f007654`, **round-trip verified from Gitea**), then manifest set in one save: `agent=0.125.0 golden=0.201.0 min_agent=0.125.0`. A fresh install now lands on current agent AND current controller |
|
||||||
|
|
||||||
**Untouched by this session, stated so nothing is presumed closed by association:** **R-213** (putting
|
**Untouched by this session, stated so nothing is presumed closed by association:** **R-213** (putting
|
||||||
files back — the half the recovery screen deliberately does not do) and **R-202** (the orphan card's
|
files back — the half the recovery screen deliberately does not do) and **R-202** (the orphan card's
|
||||||
|
|||||||
@@ -1127,3 +1127,78 @@ This is also the live proof of **R-202**'s open concern: the orphan card promise
|
|||||||
hozzá tartozó helyreállítási kóddal később visszaállítható lehet"*. Here is a customer **with** that
|
hozzá tartozó helyreállítási kóddal később visszaállítható lehet"*. Here is a customer **with** that
|
||||||
code, and the answer is "wrong code". → **R-222.** Per the brief: the gap is recorded, **nothing was
|
code, and the answer is "wrong code". → **R-222.** Per the brief: the gap is recorded, **nothing was
|
||||||
built.**
|
built.**
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The fix pass (2026-08-05, evening) — and the golden rebake it turned into
|
||||||
|
|
||||||
|
Six findings fixed in controller **v0.201.0** + hub **v0.97.0/0.97.1**, each red-proofed. Full account:
|
||||||
|
`felhom-controller/REPORT.md`.
|
||||||
|
|
||||||
|
### R-223 — vouching agent 0.125.0 was NOT a dropdown, and the manifest had been frozen
|
||||||
|
|
||||||
|
The first vouch attempt was **REFUSED**, correctly, by R-120's gate (`hub/internal/web/configs.go:1162`):
|
||||||
|
|
||||||
|
```
|
||||||
|
[WARN] artifact vouch REFUSED: golden 0.192.0 is older than the newest controller
|
||||||
|
the fleet reports (0.201.0) — a fresh install would land on stale application code (R-120)
|
||||||
|
```
|
||||||
|
|
||||||
|
The manifest was **verified unchanged afterwards** rather than inferred from the HTTP 303 (which is a
|
||||||
|
redirect carrying the refusal, not a success).
|
||||||
|
|
||||||
|
**The artifacts form saves as a unit**, so the agent could not be vouched while the golden was stale —
|
||||||
|
and the golden had been 0.192.0 since before controller 0.193.0 shipped. **The Day-0 manifest had
|
||||||
|
therefore been effectively unvouchable for days, and nobody had cause to notice because nobody tried.**
|
||||||
|
That is the finding; the "one dropdown" I first reported was wrong.
|
||||||
|
|
||||||
|
### The rebake — golden 0.201.0
|
||||||
|
|
||||||
|
Baked in the drill VM on DooPlex (the sanctioned exception for bakes, `target-selection.md`).
|
||||||
|
Preconditions checked first: 6.6 T free, **no qemu held the qcow2 — proved by the revert succeeding**,
|
||||||
|
`virgin` intact, Debian template `13.6-1` listed rather than assumed (the point release rots) and
|
||||||
|
checksum-verified on download.
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| artifact | **658 165 766 B**, sha256 `e730d7cab343eb35bcef627829b40c160f83945e0a9bd57da546efc21f007654` |
|
||||||
|
| markers | `Result=success` · `ExecMainStatus=0` · **no** `FATAL`/`excluding`/`ERROR` · `docker OK (overlay2; data-root /var/lib/docker)` · `including mount point rootfs ('/')` **and** `mp0 ('/var/lib/felhom')` · `upload OK (HTTP 201)` |
|
||||||
|
| round trip | **the bytes downloaded from Gitea hash identically.** A local checksum proves only that the local file is what it is |
|
||||||
|
| token hygiene | token read from a 0600 file **inside** the VM, never on a command line; literal-value grep on the saved log → **0 hits** |
|
||||||
|
| teardown | guest 9100 `--purge`d · token/script/log shredded **after** the log was copied out for evidence · qemu exited (checked by `ps -eo comm`, not `pgrep -f`, which self-matches) · disk reverted to `virgin` |
|
||||||
|
|
||||||
|
*Doc drift noticed, not acted on: the runbook's pass markers say `including mount point` for rootfs
|
||||||
|
**and mp0 and mp1**. `build-golden.sh` v3.0.0 bakes ONE data volume, so two mount points is correct
|
||||||
|
and the runbook describes an older layout.*
|
||||||
|
|
||||||
|
### The manifest, read back after the save
|
||||||
|
|
||||||
|
```
|
||||||
|
agent_version = 0.125.0 agent_sha256 = f7d8339b53d92a6c…7016b3
|
||||||
|
golden_version = 0.201.0 golden_sha256 = e730d7cab343eb35…f007654
|
||||||
|
min_agent = 0.125.0
|
||||||
|
```
|
||||||
|
|
||||||
|
**A machine installed from now on lands on current agent AND current controller**, and can open a
|
||||||
|
recovery package on day one. R-223 closed.
|
||||||
|
|
||||||
|
### The held floor — what is and is not established
|
||||||
|
|
||||||
|
**Established, positively, that the hold WAS in force**: the hub logged it with the corrected reason
|
||||||
|
(*"floor 0.200.0 is ABOVE the vouched golden 0.192.0…"*, 22:12:06), and the box observed it from its own
|
||||||
|
side — `[offsite-apply] settle-gate: GO — floor still unknown after 1m30s`, which is what a withheld
|
||||||
|
directive looks like to a controller.
|
||||||
|
|
||||||
|
**NOT established by a positive observable: that the hold is now released.** After the vouch, c11's
|
||||||
|
next report (22:27:06) produced **no** HELD line and the Hosts dashboard renders **no** held reason —
|
||||||
|
but both of those are absences, and this project's own rule is that an absent log line is not evidence.
|
||||||
|
The hub logs nothing on the served path by design, and `SetFloor` is DEBUG-only and fires only on a
|
||||||
|
change, so the box-side positive would need a controller restart — **which is not taken, because Phase 2
|
||||||
|
needs the venue as it is.**
|
||||||
|
|
||||||
|
What can be said with certainty is the arithmetic the code performs: floor `0.200.0` ≤ golden `0.201.0`
|
||||||
|
(so the beyond-golden branch cannot fire) and agent `0.125.0` ≥ min_agent `0.125.0` (so the
|
||||||
|
below-MinAgent branch cannot fire) — the exact combination
|
||||||
|
`TestResolveManagedFloor_R216_FloorAboveGolden/floor_at_the_golden_→_unchanged_behaviour` pins as
|
||||||
|
served. **Deduction plus a pinned test, not a live positive. Recorded as such.**
|
||||||
|
|||||||
Reference in New Issue
Block a user