admin
a2c60edefd
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
100 lines
7.2 KiB
Markdown
100 lines
7.2 KiB
Markdown
# REPORT — ISO train v1.25.0 + hub v0.73.0 (2026-07-23)
|
||
|
||
**Task:** the four-ruling ISO train (F9 belt-everywhere, apt no-subscription, R-63 console ő/ű,
|
||
R-71 golden-lift mitigation) + Part 6 root-vault rider + Part 7 hub `offsite_stale` fix.
|
||
scripts: host-install **1.17.0→1.18.0**, build-felhom-iso **1.24.0→1.25.0**, felhom-bootstrap R-63;
|
||
hub **v0.72.0→v0.73.0**. Commits: `b620435` (scripts code), hub `0.73.0` train, docs follow.
|
||
|
||
## Status by part
|
||
|
||
| Part | State |
|
||
|---|---|
|
||
| **7 — hub `offsite_stale` newborn anchor** | ✅ SHIPPED + deployed (hub v0.73.0, Synced/Healthy); red-proofed; live: demo-hp/demo-felhom silent |
|
||
| **1 — OOB belt (F9)** | ✅ code (default appliance leg) + **LIVE on BOTH boxes** + **login PROVEN on felhom-pve** |
|
||
| **2 — apt no-subscription** | ✅ code + **LIVE aligned on both boxes** (`apt-get update` clean) |
|
||
| **3 — R-63 console font** | ✅ code (Lat2 font, verified font names) — nested-render proof pending the drill |
|
||
| **4 — R-71 build-gate + rule** | ✅ code + red-proofed + **LIVE-demonstrated blocking the real gap**; standing rule 5 added |
|
||
| **4 — golden republish** | ✅ **built + published + vouched** — golden **0.161.0** (sha `77624408…`), floor stays 0.156.0; the LIVE gap is CLOSED |
|
||
| **5 — ISO v1.25.0 build** | ✅ **built** (`…v1.25.0-nested-vm-generic-mkimage.iso`, sha `99b38420…`); gate passed live; rootpw+R-63 present |
|
||
| **5 — nested drill** | ⏳ the last validation leg (R-63 console capture + fresh-install code-path proof) |
|
||
| **6 — felhom-pve root vault** | ✅ rotated + vaulted + **verified authenticating over :22** |
|
||
|
||
## CRITICAL FINDING — the R-71 gap was LIVE, now CLOSED
|
||
|
||
**RESOLVED 2026-07-23:** golden republished at controller **0.161.0** (built on felhom-pve via
|
||
`build-golden.sh`, published to Gitea 618 MB, sha `77624408bd86ec21f3de25eb57f0d9c30d9c374f6e2c19bfcea0816ccb347188`,
|
||
Gitea-served copy sha-verified), **operator-vouched** in the manifest (golden 0.153.0 → 0.161.0;
|
||
floor unchanged at 0.156.0; agent 0.93.0). The gate now passes live and the v1.25.0 ISO built. The
|
||
gap below is the finding as discovered; it no longer exists.
|
||
|
||
The hub's current artifacts *were* **golden 0.153.0 < managed floor 0.156.0**. Every fresh install
|
||
today boots a controller BELOW the floor, so the day-0 managed update fires within minutes —
|
||
the exact update-vs-bridge race that burned demo-hp's offsite credential (DIAG-f10 / R-71). The
|
||
new build-gate **correctly refuses** to build a v1.25.0 ISO in this state (demonstrated live:
|
||
`R-71 BUILD GATE FAILED: golden 0.153.0 < managed floor 0.156.0`). **The fix is Part 4's golden
|
||
republish at controller 0.161.0 + vouch** — until then no valid pairing ISO can be built, by
|
||
design. The vouch (`handleSetArtifacts`) is operator-password-gated (no admin API), so it needs
|
||
the operator.
|
||
|
||
## Part detail
|
||
|
||
**Part 1 (belt).** host-install: appliance mode installs the H1 belt by default (`--no-oob` opts
|
||
out; byo still refuses `--enable-oob` — deliberate exclusion, flagged not changed), and the config
|
||
writer now sets `oob.enabled=true` so the agent activates the belt instead of leaving the units
|
||
inert (the pre-v1.25.0 state that left it unusable — F9). Lockout-safe with no operator peer
|
||
(empty sets + no key, `:22` untouched). **Live:** belt installed on felhom-pve + demo-hp (both
|
||
8822, sets filled `{10.77.0.250}`/`{8822}`, felhom-op key installed, `:22` pid unchanged). **Login
|
||
PROVEN on felhom-pve** end-to-end (`felhom-op@demo-felhom`) — see the operator-identity note below.
|
||
|
||
**Operator identity re-established (latent gap fixed).** The registered `oob_operator_ssh_pubkey`
|
||
was ORPHANED — it matched no key the operator held (an OOB credential the operator can't reproduce
|
||
is no better than the lockout it prevents). Re-anchored to the operator's real machine: registered
|
||
its WireGuard pubkey as the operator peer `10.77.0.250` + its `felhom-op-1` SSH key, built the
|
||
operator wg-felhom tunnel config (`ep0.felhom.eu:443`), added + persisted ep0's `forward` rules
|
||
(`10.77.0.250 → 10.77.0.2/.3` in `/etc/nftables.conf`). Along the way: a corp-network UDP/443 block
|
||
was diagnosed (ep0 tcpdump — packets arriving on 51820 but not 443) and the login completed from a
|
||
home network on 443. All secrets file→file, shredded.
|
||
|
||
**Part 2 (apt).** `align_apt_repos` (appliance-only, idempotent, pre-apt): disable enterprise
|
||
deb822 `.sources` (`Enabled: no`) / legacy `.list`, add `pve-no-subscription.sources` (suite+keyring
|
||
derived), `apt-get update` clean. **Live:** both boxes were on enterprise (401 on update); aligned →
|
||
enterprise+ceph disabled, no-subscription added, `apt-get update` exit 0 on both.
|
||
|
||
**Part 3 (R-63).** felhom-bootstrap loads a Latin-2 console font (`Lat2-Terminus16` → `Lat2-Fixed16`
|
||
→ `Lat2-Terminus14`) once before the first paint (idempotent, best-effort, never blocks boot). Font
|
||
names verified present in the trixie console-setup package. Renders ő/ű in the R-59 network screen
|
||
(élő/telepítő/ellenőrizze) + pairing banner (képernyő/teendő). Nested-console capture pending drill.
|
||
|
||
**Part 4 (build-gate + rule).** `assert_golden_ge_floor` + `ver_ge` in build-felhom-iso; runs before
|
||
workspace/mint; `die`s with both versions on `golden < floor`; `FELHOM_ASSERT_GOLDEN/FLOOR` inputs;
|
||
unset warns loudly. Red-proofed both directions (floor>golden → exit 1; golden≥floor → OK; equal →
|
||
OK; unset → WARN). Standing rule 5 in `publish-train-rules.md`. Live-demonstrated against the real
|
||
production gap (see above).
|
||
|
||
**Part 6 (root vault).** On felhom-pve, generated a fresh strong root password (never printed),
|
||
`chpasswd`, vaulted via `PUT /api/v1/hosts/demo-felhom-8363b5/recovery-credential` (HTTP 200, the
|
||
same day-0 `step_break_glass` mechanism, host api_key). Verified: retrieved via the operator
|
||
endpoint (global key) and it AUTHENTICATED root over `:22` (`VAULT-AUTH-OK: demo-felhom`); copy
|
||
shredded. felhom-pve now has G1 vault parity with demo-hp. (Note: the vault's `set_at` reflects the
|
||
original day-0 stamp — not bumped on rotation; cosmetic, the SECRET is the rotated one, proven by
|
||
the auth test.)
|
||
|
||
## Remaining — the nested drill (last validation leg)
|
||
|
||
Golden republish + vouch + the v1.25.0 ISO build are DONE. The one remaining leg is the
|
||
v1.24.0-style **nested drill** — the deep end-to-end proof of the fresh-install code path:
|
||
belt installed+active by the default appliance leg, apt=no-subscription, **R-63 console capture**
|
||
(the ő/ű closure evidence the spec asks for — ellenőrizze/telepítő rendered), golden boots 0.161.0
|
||
with **NO managed-update firing** (structurally guaranteed now that golden 0.161 ≥ floor 0.156 —
|
||
the drill confirms it live), `.rootpw.txt` opens the console (R-61 regression), and the build-gate
|
||
red-proof in the nest. Heavy (~15–20 min: 1.7 GB ISO transfer + nested install + boot + verify on
|
||
the remote N100); the belt + apt legs are already proven live on the two real boxes, so this drill
|
||
adds the fresh-install-path + R-63-visual proofs specifically.
|
||
|
||
## Observed / not acted on
|
||
|
||
- The live golden(0.153)<floor(0.156) gap predates this train (an old golden bake vs a later floor
|
||
lift) — the new gate surfaces it; the golden republish fixes it.
|
||
- ep0 root is reachable from DooPlex (used for the operator-tunnel + forward-rule work); the forward
|
||
rules are persisted to survive reboot.
|