From 6dab0f6a2cc665cbbce70c0f8fa2a65458fc7cfb Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 23 Jul 2026 15:48:01 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20ISO=20train=20v1.25.0=20=E2=80=94=20REP?= =?UTF-8?q?ORT=20(belt/apt/R-63/gate/vault=20live),=20F9=20resolved,=20R-6?= =?UTF-8?q?3=20shipped,=20nodes=20belt+vault,=20F8=20checklist;=20critical?= =?UTF-8?q?=20golden Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR --- CONTEXT.md | 18 ++ REPORT.md | 156 +++++++++--------- .../AUDIT-power-outage-recovery-2026-07-22.md | 2 +- documentation/backlog/ROADMAP.md | 1 + documentation/operations/nodes.md | 30 ++++ documentation/pilot/GO-LIVE-PACKAGE.md | 5 + 6 files changed, 132 insertions(+), 80 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index e930c11..0fba6d3 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -22,6 +22,24 @@ intact (its cleanup is the operator's, post-validation). R-71(a) day-0 ordering OPEN → next spec. Evidence: `REPORT.md` (2026-07-23). +- **2026-07-23 — ISO train v1.25.0 + hub v0.73.0 (four rulings + rider).** host-install + 1.17.0→1.18.0, build-felhom-iso 1.24.0→1.25.0, felhom-bootstrap R-63, hub v0.73.0. **Done + + live-verified:** (1) **F9 belt everywhere** — host-install installs the H1 OOB belt by default in + appliance mode (`--no-oob` opts out; byo still refuses), `oob.enabled` written so the agent + activates it; belt LIVE on both boxes, **login PROVEN on felhom-pve** (`felhom-op@demo-felhom`); + the orphaned operator identity was re-anchored to the operator's real machine (WG peer + SSH key + re-registered, ep0 forward rules persisted). (2) **apt** — `align_apt_repos` (appliance-only) + switches enterprise→no-subscription; LIVE-aligned both boxes (`apt-get update` clean). (3) **R-63** + — Lat2 console font before first paint. (4) **R-71 build-gate** — `assert_golden_ge_floor` in + build-felhom-iso + rule 5; red-proofed. (6) **root vault** — felhom-pve root rotated+vaulted+ + verified over :22 (G1 parity). (7) **hub v0.73.0** — `offsite_stale` never-ran branch ANCHORED + (applied-only + consumed_at/escrow anchor; one-state-one-owner with `offsite_delivery_stuck`), + shipped+deployed, red-proofed. **CRITICAL live finding:** golden **0.153.0 < floor 0.156.0** in + production now — the R-71 gap is real; the gate correctly blocks a v1.25.0 build. **Remaining + (enforced-sequenced):** golden republish at 0.161.0 + operator vouch (password-gated) → then the + v1.25.0 ISO rebuild + nested drill. Evidence: `REPORT.md`, `scripts/CHANGELOG.md`, + `operations/nodes.md`. + - **2026-07-23 — F10 offsite leg RESOLVED on demo-hp (diagnostic + designed-path repair + tier proof; no code changed).** The hub said "Provisioned" while the box said „Még nincs beállítva" — root cause: the day-0 managed update (0.153.0→0.156.0, 07-21 16:28) killed the offsite diff --git a/REPORT.md b/REPORT.md index 58fc5f3..a8ebc3e 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,93 +1,91 @@ -# REPORT — hub v0.72.0: R-70 + R-71(c) — the offsite last mile becomes visible, burned credentials heal themselves (2026-07-23) +# REPORT — ISO train v1.25.0 + hub v0.73.0 (2026-07-23) -**Spec:** R-70 + R-71(c) prompt (project Claude, 2026-07-23). Companion controller leg: v0.161.0 -(see `felhom-controller/REPORT.md`). Origin: `documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. -R-71(a) (day-0 ordering) untouched — separate upcoming spec. +**Task:** the four-ruling ISO train (F9 belt-everywhere, apt no-subscription, R-63 console ő/ű, +R-71 golden-lift mitigation) + Part 6 root-vault rider + Part 7 hub `offsite_stale` fix. +scripts: host-install **1.17.0→1.18.0**, build-felhom-iso **1.24.0→1.25.0**, felhom-bootstrap R-63; +hub **v0.72.0→v0.73.0**. Commits: `b620435` (scripts code), hub `0.73.0` train, docs follow. -## Baselines +## Status by part -| Repo | start | shipped | -|---|---|---| -| felhom.eu | `c801cee6`, hub 0.71.0 | hub **0.72.0** (code + separate manifest chore commit), ArgoCD Synced/Healthy | -| felhom-controller | `0eba37d5`, v0.160.0 | **v0.161.0** (`ce85314`), deployed BOTH boxes, healthy | +| Part | State | +|---|---| +| **7 — hub `offsite_stale` newborn anchor** | ✅ SHIPPED + deployed (hub v0.73.0, Synced/Healthy); red-proofed; live: demo-hp/demo-felhom silent | +| **1 — OOB belt (F9)** | ✅ code (default appliance leg) + **LIVE on BOTH boxes** + **login PROVEN on felhom-pve** | +| **2 — apt no-subscription** | ✅ code + **LIVE aligned on both boxes** (`apt-get update` clean) | +| **3 — R-63 console font** | ✅ code (Lat2 font, verified font names) — nested-render proof pending the drill | +| **4 — R-71 build-gate + rule** | ✅ code + red-proofed + **LIVE-demonstrated blocking the real gap**; standing rule 5 added | +| **4 — golden republish** | ⏳ **REQUIRED NEXT** — see the critical finding below (blocks a valid v1.25.0 ISO) | +| **5 — ISO rebuild + nested drill** | ⏳ gate-blocked until golden lifted; drill pending | +| **6 — felhom-pve root vault** | ✅ rotated + vaulted + **verified authenticating over :22** | -## What shipped (one detector, four consumers) +## CRITICAL FINDING — the R-71 gap is LIVE in production right now -1. **Detector** — `internal/offsite/delivery.go` `DeliveryStateFor`: `applied` / - `consumed_awaiting_apply` / `staged_awaiting_consume` / `no_secret` from the secret-row - timestamps × report offsite-presence. Applied wins (the box's own report is the strongest - evidence); applied+unconsumed-staged (demo-felhom) = `applied` + `StaleStagedSince` flag. - New store reads (`GetOneTimeSecretInfo` — timestamps only, value never selected; - `LatestReportOffsitePresence`; `CountReportsOffsiteSince`; `LastEventAt`) + the PBSDR-style - test back-dater. -2. **Customer card** — `deliveryViewFor` + `config_form_body.html`: the static "delivered to the - controller once" claim is deleted; the card renders state + age (badge `n-ok/n-warn/n-neutral`, - consumed goes amber past 30 min, stale-staged info line). Render test per branch. -3. **Loud event** — `offsite_delivery_stuck` (WARNING) at ≥ 1 h of consumed_awaiting_apply; - 24 h/customer cooldown, durable via the events table (restart-proof). -4. **Self-heal (R-71c)** — `monitor.OffsiteDeliveryChecker` on the shared 60 s ticker invokes - `web.Server.ReissueOffsiteForCustomer` behind the narrow `monitor.OffsiteReissuer` interface - (pbsdrheal precedent; armed only when the provisioner exists — else a restaged event would - lie about a silent no-op). Trigger: consumed ≥ 1 h + ≥ 4 consecutive offbox-less reports + - zero offbox evidence since consume. One restage/customer/24 h (durable); every firing emits - `offsite_credential_restaged` (WARNING). **R-39(a) guard**: the heal re-reads the secret row - immediately before acting and refuses over an unconsumed row — the store's - `SaveOneTimeSecret` clobber semantics are untouched (Re-issue depends on supersede; the guard - lives in the caller, exactly as specced). +The hub's current artifacts are **golden 0.153.0 < managed floor 0.156.0**. Every fresh install +today boots a controller BELOW the floor, so the day-0 managed update fires within minutes — +the exact update-vs-bridge race that burned demo-hp's offsite credential (DIAG-f10 / R-71). The +new build-gate **correctly refuses** to build a v1.25.0 ISO in this state (demonstrated live: +`R-71 BUILD GATE FAILED: golden 0.153.0 < managed floor 0.156.0`). **The fix is Part 4's golden +republish at controller 0.161.0 + vouch** — until then no valid pairing ISO can be built, by +design. The vouch (`handleSetArtifacts`) is operator-password-gated (no admin API), so it needs +the operator. -## Red-proofs (run, observed, restored — verbatim failures) +## Part detail -1. **THE CLOBBER RED-PROOF** — R-39(a) guard block removed from `maybeHeal`; the TOCTOU test - (operator Re-issue staged mid-tick via the onEvent hook) failed with: - `reissue calls = 1, want 0 — the R-39(a) guard must refuse over an unconsumed secret` - — i.e. the operator's fresh unconsumed secret would have been clobbered - (the fake reissuer mimics the production `SaveOneTimeSecret` side effect, so the clobber is - observed on the row, not inferred). Guard restored → green. -2. **Heal rate-limit** — `LastEventAt`/`healCooldown` check removed; failed with: - `reissue calls after recurrence = 2, want STILL 1 (one restage per customer per 24h)`. Restored. -3. **Stuck-event cooldown** — cooldown check removed; failed with: - `stuck events = 2, want exactly 1 (24h per-customer cooldown)`. Restored. +**Part 1 (belt).** host-install: appliance mode installs the H1 belt by default (`--no-oob` opts +out; byo still refuses `--enable-oob` — deliberate exclusion, flagged not changed), and the config +writer now sets `oob.enabled=true` so the agent activates the belt instead of leaving the units +inert (the pre-v1.25.0 state that left it unusable — F9). Lockout-safe with no operator peer +(empty sets + no key, `:22` untouched). **Live:** belt installed on felhom-pve + demo-hp (both +8822, sets filled `{10.77.0.250}`/`{8822}`, felhom-op key installed, `:22` pid unchanged). **Login +PROVEN on felhom-pve** end-to-end (`felhom-op@demo-felhom`) — see the operator-identity note below. -Plus: the full-Check demo-felhom-shape test (applied + stale staged → zero events, zero calls, -row byte-untouched), evidence gates (<4 reports → no heal but stuck event still fires; mixed -offbox history → no heal), young-consumed silence, disabled/blocked skip, dispatcher severity -tests (warning routes operator-only; an info variant would be silent — pinned beside the v0.71.0 -guard, `severityNotifies` untouched), 5 card render tests + `deliveryViewFor` amber derivation. -Green gates: hub 17 packages ok + `hub_confirm_gate.py`; controller 25 packages ok + all template -gates (the pre-existing R-29 `docker_run_volume_path_gate` red noted, untouched). +**Operator identity re-established (latent gap fixed).** The registered `oob_operator_ssh_pubkey` +was ORPHANED — it matched no key the operator held (an OOB credential the operator can't reproduce +is no better than the lockout it prevents). Re-anchored to the operator's real machine: registered +its WireGuard pubkey as the operator peer `10.77.0.250` + its `felhom-op-1` SSH key, built the +operator wg-felhom tunnel config (`ep0.felhom.eu:443`), added + persisted ep0's `forward` rules +(`10.77.0.250 → 10.77.0.2/.3` in `/etc/nftables.conf`). Along the way: a corp-network UDP/443 block +was diagnosed (ep0 tcpdump — packets arriving on 51820 but not 443) and the login completed from a +home network on 443. All secrets file→file, shredded. -## Live validation (read-only, both fixtures intact) +**Part 2 (apt).** `align_apt_repos` (appliance-only, idempotent, pre-apt): disable enterprise +deb822 `.sources` (`Enabled: no`) / legacy `.list`, add `pve-no-subscription.sources` (suite+keyring +derived), `apt-get update` clean. **Live:** both boxes were on enterprise (401 on update); aligned → +enterprise+ceph disabled, no-subscription added, `apt-get update` exit 0 on both. -- **Checker silence:** 10 min of 60 s ticks on the live fleet → **0** `offsite-delivery` log - lines, **0** detector events in the DB — both fixtures are healthy and the detector agrees. -- **Fixture states from live data** (fresh DB copy, shredded after): - demo-hp `latest_report_offsite=True, secret consumed 09:53:41` → **applied**; - demo-felhom `latest_report_offsite=True, secret_row=(2026-07-21 08:29:29, None)` → - **applied + stale-staged since 07-21** — the live specimen SURVIVED the train untouched - (`consumed_at` still NULL, created_at unchanged); peti-felhom → applied. -- **demo-hp controller page** (authed endpoint fetch inside the guest, ASCII-safe greps): - banner count 0, `felhom-offsite-card` id count 0 (configured + 1 toggled app → no card at all), - configured markers present. v0.161.0 healthy on both boxes. -- **Method:** endpoint-level + DB-input derivation (no browser on DooPlex). The rendered card is - pinned by render tests; the operator's 10-second residual: demo-hp Edit page shows - `applied`, demo-felhom shows `applied` + the stale-staged note (since 2026-07-21). -- **The self-heal is NOT live-fired** — no broken box exists and none was broken for it (F9 - rule). It ships unit-proven + red-proofed, PARTIAL/IMPLEMENTED on the ROADMAP with the - explicit "fires on next natural occurrence or a staged drill" note — never PROVEN-LIVE. - The controller banner leg is likewise unit-proven/live-pending (no box occupies the - enabled+no-offbox window; the next fresh onboarding is its natural live leg). +**Part 3 (R-63).** felhom-bootstrap loads a Latin-2 console font (`Lat2-Terminus16` → `Lat2-Fixed16` +→ `Lat2-Terminus14`) once before the first paint (idempotent, best-effort, never blocks boot). Font +names verified present in the trixie console-setup package. Renders ő/ű in the R-59 network screen +(élő/telepítő/ellenőrizze) + pairing banner (képernyő/teendő). Nested-console capture pending drill. -## Rulings recorded (CONTEXT.md) +**Part 4 (build-gate + rule).** `assert_golden_ge_floor` + `ver_ge` in build-felhom-iso; runs before +workspace/mint; `die`s with both versions on `golden < floor`; `FELHOM_ASSERT_GOLDEN/FLOOR` inputs; +unset warns loudly. Red-proofed both directions (floor>golden → exit 1; golden≥floor → OK; equal → +OK; unset → WARN). Standing rule 5 in `publish-train-rules.md`. Live-demonstrated against the real +production gap (see above). -State precedence (applied wins; stale-staged is a flag, never a downgrade); durable cooldowns via -the events table (restart-proof by design); both detector events operator-only (no -customerMessages entry, not in allowedEventTypes — the pbsdr_* precedent) until the mechanism has -history; heal disabled without a provisioner. +**Part 6 (root vault).** On felhom-pve, generated a fresh strong root password (never printed), +`chpasswd`, vaulted via `PUT /api/v1/hosts/demo-felhom-8363b5/recovery-credential` (HTTP 200, the +same day-0 `step_break_glass` mechanism, host api_key). Verified: retrieved via the operator +endpoint (global key) and it AUTHENTICATED root over `:22` (`VAULT-AUTH-OK: demo-felhom`); copy +shredded. felhom-pve now has G1 vault parity with demo-hp. (Note: the vault's `set_at` reflects the +original day-0 stamp — not bumped on rotation; cosmetic, the SECRET is the rotated one, proven by +the auth test.) -## Observed, not acted on +## Remaining (the two heaviest legs, now enforced-sequenced) -- R-29: `docker_run_volume_path_gate` red on `appexport/estimate.go:179` (pre-existing, has its - own row — the 3-line allowlist fix remains undone by design of this train's scope). -- The `szolg` single-hit on the demo-hp page grep is an accent-truncated unrelated word (the - ASCII-only-grep trap documented in felhom-controller/CLAUDE.md — verified benign via the - card-id count of 0). +1. **Golden republish** (Part 4): `build-golden.sh 9100 … gitea.dooplex.hu/admin/felhom-controller:0.161.0` + on a Proxmox host → publish to Gitea → **operator vouches** `artifact_golden_version=0.161.0` + + sha in the manifest UI (password-gated). This closes the LIVE golden.rootpw.txt` (single record of truth — never logged/manifested/committed); drill-verified against the installed box's shadow hash. Follow-up (appliance-grade record-keeping) stays open** | The ISO mints a **fresh throwaway crypt hash per build** and the plaintext is discarded, so nobody — including the person holding the machine — can log into the console of a box they just installed. Today that meant reaching demo-hp only through the G1 break-glass credential vaulted in the hub, which is the right mechanism for a *lost* password and the wrong one for a *never-known* password: it requires a working hub, a working network, and operator tooling, at exactly the moment the likely reason you need the console is that one of those is broken. **Slice 1 (do this):** the ISO build emits the baked root password into the build REPORT and the operator cheat-sheet alongside the sha256 — it is already a per-build value, so surfacing it costs nothing and closes the lockout. **Follow-up (appliance-grade):** keep it per-build random and treat the build output as the record of truth. **A fixed well-known password is explicitly REJECTED (operator ruling 2026-07-21)** — a pre-pairing box sits on a stranger's LAN with a predictable root credential, which is a far worse exposure than the lockout it would fix. Relates to G1 break-glass (the vault stays; this is about the window before/without it) | | R-62 | **[P3] Hub delete dialog: show the customer-id the operator must type, and reword the three acks for the ghost shape.** | XS | **idea (operator, 2026-07-22)** | Cosmetic, hub-only, docs-only in the v1.24.0 train. The delete confirmation asks the operator to type the customer-id, but the id appears NOWHERE on the Edit page the dialog opens from — the operator has to fish it out of the URL or another tab. Also: for a GHOST customer (host already gone) the three acknowledgement checkboxes describe teardown steps that cannot happen; **wording only** — the server MUST keep requiring all three (the render-gate lesson of v0.70.1 stands: reachability and requirements are separate concerns). | +| R-63 | **The install console learns ő/ű** — the kernel default console font lacks the Hungarian double-acute glyphs, so the R-59 network screen and pairing banner rendered ő as blanks. | XS | **SHIPPED (scripts v1.25.0, 2026-07-23)** | `felhom-bootstrap.sh` loads a Latin-2 console font (`Lat2-Terminus16` → `Lat2-Fixed16` → `Lat2-Terminus14`) ONCE before the first paint — idempotent, best-effort (a missing font/ioctl never blocks boot). Lat2 ships in the trixie/PVE base (console-setup), so no copy rewording needed. Font names verified against the package. Nested-console capture proof rides the v1.25.0 drill. Also in the same train: **F9 belt-everywhere RESOLVED** (host-install default appliance leg + live on both boxes + login proven) and the **R-71 build-gate** (`build-felhom-iso` asserts golden ≥ managed floor, `publish-train-rules.md` rule 5). **Live finding: golden 0.153.0 < floor 0.156.0 in production NOW** — the gate catches it; the fix is the golden republish at 0.161.0 (Part 4, pending; the managed floor stays 0.156.0). See `REPORT.md` (2026-07-23). | | R-64 | **„Felhom↔Felhom media pairing blessed" — the two-box SMB pairing (one box shares, the other mounts it as NAS storage) becomes a supported, documented flow.** | XS–S | idea (2026-07-22) | Origin: the operator ran the pairing drill on the live demo pair and it WORKS — the drill itself is the pending evidence leg (a written run-through with the R-66 surfaces in play). R-66 shipped the enabling visibility: the serving box's address is now on its own Beállítások → Rendszer „Hálózat" card, and the add form names the NetBIOS trap. Blessing = a short customer-facing recipe (`documentation/controller/network-storage-nas.md` naming-caveat paragraph is the seed) + one supported-path sentence in the capability map. Flips: would add a "Felhom↔Felhom media pairing" capability row (currently unlisted). Pairs with R-65 (same two-box topology, entirely different transport + guarantees) | | R-66 | **The box's own address becomes visible — „Hálózat" card, Debug network dump, NetBIOS hint.** | XS | **SHIPPED (controller v0.159.0, 2026-07-22)** | Origin: the pairing drill — the serving box's IP was findable only as a hint buried on the OTHER box's Megosztás page, and the add form's failure for „FELHOM" taught nothing. Three legs: (A) „Hálózat" card on Beállítások → Rendszer (Helyi cím / Hálózati név only-while-sharing / Átjáró; live per render, stored nowhere — S-5; „—" when unavailable); (B) `network` section in the Debug dump (interfaces/route/DNS/lan_address, best-effort per item); (C) the NetBIOS trap named (Szerver helper text + a purely lexical hint on `unreachable` for single-label non-IP names). **Design decision recorded:** the controller is bridge-netns'd, so ALL guest-net reads go through the one netns door (docker exec into host-networked felhom-samba, `stacks/guestnet.go`) — with Megosztás off the card honestly shows „—" rather than the plausible-wrong 172.x answer. Deployed demo-felhom + demo-hp 2026-07-22; demo-hp live-shows the closed-door path (sharing off → dashes + in-place dump errors), demo-felhom the open one (real .104/.1/\\FELHOM values). Flips no capability-map row (diagnosability/UX polish); enables R-64 | | R-67 | **The NAS share appears in FileBrowser — browse what you mounted.** | S | **SHIPPED (controller v0.160.0, 2026-07-22)** | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). **Couples to R-64: browsing was its missing UX half.** A registered network storage now binds its share ROOT into FileBrowser (`/mnt/felhom-drives/:/srv/:rslave`) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (`Host is down` in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: `felhom-controller/REPORT.md` (2026-07-22) | diff --git a/documentation/operations/nodes.md b/documentation/operations/nodes.md index 5d8876b..9c03f7e 100644 --- a/documentation/operations/nodes.md +++ b/documentation/operations/nodes.md @@ -83,6 +83,36 @@ may be trying to fix. Slice 1 is to emit the baked password into the build repor `demo-hp-lan` (`192.168.0.87` via `ProxyJump felhom-pve`) is the fallback while the box is away. +## OOB belt (H1) — both boxes, since 2026-07-23 (ISO train v1.25.0) + +The dedicated OOB sshd belt (TASK H1: `felhom-sshd` + the static `inet felhom_oob` table + `felhom-op`) +is installed and **active on BOTH fleet boxes** — the F9 gap (belt on neither) is closed. From +v1.25.0 host-install installs it by default on every appliance install (`--no-oob` opts out; byo still +refuses). + +- **Claimed port: `8822` on both** (first-free from `[8822,2222,8022,62222]`; persisted per box). +- **Reachability: the wg-felhom offsite tunnel ONLY** — the belt admits the operator `/32` + (`10.77.0.250`) over `wg-felhom` to 8822 and drops everything else; `:22` and every other interface + are untouched. **tailscale does NOT reach the belt** (wrong fabric, dropped by design). +- **Operator login** (from the machine holding the wg-felhom operator tunnel + the registered + `oob_operator_ssh_pubkey`): `ssh -p 8822 felhom-op@10.77.0.2` (felhom-pve) / `@10.77.0.3` (demo-hp). + **PROVEN live 2026-07-23** on felhom-pve (`felhom-op@demo-felhom`). +- **Operator tunnel**: the Mac/Windows operator peer dials `ep0.felhom.eu:443` (WireGuard), address + `10.77.0.250/32`, AllowedIPs `10.77.0.0/24`, server pubkey `f3d1ZI7…`. ep0's `forward` chain + (persisted in its `/etc/nftables.conf`) allows `10.77.0.250 → 10.77.0.2/.3`. If a work-network blocks + UDP/443, the RheinMetall-style firewalls pass UDP/51820 — a home/hotspot network works on 443. +- Register/rotate the operator identity hub-side: `PUT /api/v1/admin/wg/operator-peer` (global key) + with `{pubkey, assigned_ip:"10.77.0.250", ssh_pubkey}`; wgsync pushes it to ep0 and the SSH key flows + to both boxes' `felhom-op` authorized_keys within a tick. + +## felhom-pve (the N100) — vault parity + access + +felhom-pve has **operator SSH-key access** (over tailscale `100.70.170.35`) AND, since 2026-07-23, +**G1 break-glass vault parity with demo-hp**: its root@pam password is freshly rotated and vaulted in +the hub `host_recovery` row **`demo-felhom-8363b5`** (same PUT `…/recovery-credential` mechanism day-0 +uses; verified retrievable + authenticating over `:22`). Retrieval + shred-the-copy recipe is identical +to demo-hp's below (swap the host_id). So a lost N100 key is recoverable the same way as the key-less HP. + ## Tailscale on demo-hp is an OPERATOR-LAB EXCEPTION > **Read this before any product-shape audit.** `demo-hp` is **customer-shaped** — it is a normal diff --git a/documentation/pilot/GO-LIVE-PACKAGE.md b/documentation/pilot/GO-LIVE-PACKAGE.md index a6a7894..c536fac 100644 --- a/documentation/pilot/GO-LIVE-PACKAGE.md +++ b/documentation/pilot/GO-LIVE-PACKAGE.md @@ -201,6 +201,11 @@ Recommended order: **GL-1 and GL-3 immediately** (operator-heavy, unblock everyt ## 6. Open questions & operator actions +**Provisioning-checklist items (every fleet box, bench prep):** +- **BIOS "restore on AC power" = ON** (power-outage audit F8, 2026-07-22): a site power blip must not + become a half-day outage waiting for someone to press the button. Set + attest per box (not + OS-verifiable). Both current demo boxes done (Viktor, 07-22). + **Operator actions (Viktor):** - **Hub manifest bump — DONE** (confirmed at GL-6 Phase 0: the manifest vouches agent `0.76.0` + golden `0.103.0`). No action.