docs: ISO train v1.25.0 — REPORT (belt/apt/R-63/gate/vault live), F9 resolved, R-63 shipped, nodes belt+vault, F8 checklist; critical golden<floor finding
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
This commit is contained in:
+18
@@ -22,6 +22,24 @@
|
||||
intact (its cleanup is the operator's, post-validation). R-71(a) day-0 ordering OPEN → next
|
||||
spec. Evidence: `REPORT.md` (2026-07-23).
|
||||
|
||||
- **2026-07-23 — ISO train v1.25.0 + hub v0.73.0 (four rulings + rider).** host-install
|
||||
1.17.0→1.18.0, build-felhom-iso 1.24.0→1.25.0, felhom-bootstrap R-63, hub v0.73.0. **Done +
|
||||
live-verified:** (1) **F9 belt everywhere** — host-install installs the H1 OOB belt by default in
|
||||
appliance mode (`--no-oob` opts out; byo still refuses), `oob.enabled` written so the agent
|
||||
activates it; belt LIVE on both boxes, **login PROVEN on felhom-pve** (`felhom-op@demo-felhom`);
|
||||
the orphaned operator identity was re-anchored to the operator's real machine (WG peer + SSH key
|
||||
re-registered, ep0 forward rules persisted). (2) **apt** — `align_apt_repos` (appliance-only)
|
||||
switches enterprise→no-subscription; LIVE-aligned both boxes (`apt-get update` clean). (3) **R-63**
|
||||
— Lat2 console font before first paint. (4) **R-71 build-gate** — `assert_golden_ge_floor` in
|
||||
build-felhom-iso + rule 5; red-proofed. (6) **root vault** — felhom-pve root rotated+vaulted+
|
||||
verified over :22 (G1 parity). (7) **hub v0.73.0** — `offsite_stale` never-ran branch ANCHORED
|
||||
(applied-only + consumed_at/escrow anchor; one-state-one-owner with `offsite_delivery_stuck`),
|
||||
shipped+deployed, red-proofed. **CRITICAL live finding:** golden **0.153.0 < floor 0.156.0** in
|
||||
production now — the R-71 gap is real; the gate correctly blocks a v1.25.0 build. **Remaining
|
||||
(enforced-sequenced):** golden republish at 0.161.0 + operator vouch (password-gated) → then the
|
||||
v1.25.0 ISO rebuild + nested drill. Evidence: `REPORT.md`, `scripts/CHANGELOG.md`,
|
||||
`operations/nodes.md`.
|
||||
|
||||
- **2026-07-23 — F10 offsite leg RESOLVED on demo-hp (diagnostic + designed-path repair + tier
|
||||
proof; no code changed).** The hub said "Provisioned" while the box said „Még nincs beállítva" —
|
||||
root cause: the day-0 managed update (0.153.0→0.156.0, 07-21 16:28) killed the offsite
|
||||
|
||||
@@ -1,93 +1,91 @@
|
||||
# REPORT — hub v0.72.0: R-70 + R-71(c) — the offsite last mile becomes visible, burned credentials heal themselves (2026-07-23)
|
||||
# REPORT — ISO train v1.25.0 + hub v0.73.0 (2026-07-23)
|
||||
|
||||
**Spec:** R-70 + R-71(c) prompt (project Claude, 2026-07-23). Companion controller leg: v0.161.0
|
||||
(see `felhom-controller/REPORT.md`). Origin: `documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`.
|
||||
R-71(a) (day-0 ordering) untouched — separate upcoming spec.
|
||||
**Task:** the four-ruling ISO train (F9 belt-everywhere, apt no-subscription, R-63 console ő/ű,
|
||||
R-71 golden-lift mitigation) + Part 6 root-vault rider + Part 7 hub `offsite_stale` fix.
|
||||
scripts: host-install **1.17.0→1.18.0**, build-felhom-iso **1.24.0→1.25.0**, felhom-bootstrap R-63;
|
||||
hub **v0.72.0→v0.73.0**. Commits: `b620435` (scripts code), hub `0.73.0` train, docs follow.
|
||||
|
||||
## Baselines
|
||||
## Status by part
|
||||
|
||||
| Repo | start | shipped |
|
||||
|---|---|---|
|
||||
| felhom.eu | `c801cee6`, hub 0.71.0 | hub **0.72.0** (code + separate manifest chore commit), ArgoCD Synced/Healthy |
|
||||
| felhom-controller | `0eba37d5`, v0.160.0 | **v0.161.0** (`ce85314`), deployed BOTH boxes, healthy |
|
||||
| Part | State |
|
||||
|---|---|
|
||||
| **7 — hub `offsite_stale` newborn anchor** | ✅ SHIPPED + deployed (hub v0.73.0, Synced/Healthy); red-proofed; live: demo-hp/demo-felhom silent |
|
||||
| **1 — OOB belt (F9)** | ✅ code (default appliance leg) + **LIVE on BOTH boxes** + **login PROVEN on felhom-pve** |
|
||||
| **2 — apt no-subscription** | ✅ code + **LIVE aligned on both boxes** (`apt-get update` clean) |
|
||||
| **3 — R-63 console font** | ✅ code (Lat2 font, verified font names) — nested-render proof pending the drill |
|
||||
| **4 — R-71 build-gate + rule** | ✅ code + red-proofed + **LIVE-demonstrated blocking the real gap**; standing rule 5 added |
|
||||
| **4 — golden republish** | ⏳ **REQUIRED NEXT** — see the critical finding below (blocks a valid v1.25.0 ISO) |
|
||||
| **5 — ISO rebuild + nested drill** | ⏳ gate-blocked until golden lifted; drill pending |
|
||||
| **6 — felhom-pve root vault** | ✅ rotated + vaulted + **verified authenticating over :22** |
|
||||
|
||||
## What shipped (one detector, four consumers)
|
||||
## CRITICAL FINDING — the R-71 gap is LIVE in production right now
|
||||
|
||||
1. **Detector** — `internal/offsite/delivery.go` `DeliveryStateFor`: `applied` /
|
||||
`consumed_awaiting_apply` / `staged_awaiting_consume` / `no_secret` from the secret-row
|
||||
timestamps × report offsite-presence. Applied wins (the box's own report is the strongest
|
||||
evidence); applied+unconsumed-staged (demo-felhom) = `applied` + `StaleStagedSince` flag.
|
||||
New store reads (`GetOneTimeSecretInfo` — timestamps only, value never selected;
|
||||
`LatestReportOffsitePresence`; `CountReportsOffsiteSince`; `LastEventAt`) + the PBSDR-style
|
||||
test back-dater.
|
||||
2. **Customer card** — `deliveryViewFor` + `config_form_body.html`: the static "delivered to the
|
||||
controller once" claim is deleted; the card renders state + age (badge `n-ok/n-warn/n-neutral`,
|
||||
consumed goes amber past 30 min, stale-staged info line). Render test per branch.
|
||||
3. **Loud event** — `offsite_delivery_stuck` (WARNING) at ≥ 1 h of consumed_awaiting_apply;
|
||||
24 h/customer cooldown, durable via the events table (restart-proof).
|
||||
4. **Self-heal (R-71c)** — `monitor.OffsiteDeliveryChecker` on the shared 60 s ticker invokes
|
||||
`web.Server.ReissueOffsiteForCustomer` behind the narrow `monitor.OffsiteReissuer` interface
|
||||
(pbsdrheal precedent; armed only when the provisioner exists — else a restaged event would
|
||||
lie about a silent no-op). Trigger: consumed ≥ 1 h + ≥ 4 consecutive offbox-less reports +
|
||||
zero offbox evidence since consume. One restage/customer/24 h (durable); every firing emits
|
||||
`offsite_credential_restaged` (WARNING). **R-39(a) guard**: the heal re-reads the secret row
|
||||
immediately before acting and refuses over an unconsumed row — the store's
|
||||
`SaveOneTimeSecret` clobber semantics are untouched (Re-issue depends on supersede; the guard
|
||||
lives in the caller, exactly as specced).
|
||||
The hub's current artifacts are **golden 0.153.0 < managed floor 0.156.0**. Every fresh install
|
||||
today boots a controller BELOW the floor, so the day-0 managed update fires within minutes —
|
||||
the exact update-vs-bridge race that burned demo-hp's offsite credential (DIAG-f10 / R-71). The
|
||||
new build-gate **correctly refuses** to build a v1.25.0 ISO in this state (demonstrated live:
|
||||
`R-71 BUILD GATE FAILED: golden 0.153.0 < managed floor 0.156.0`). **The fix is Part 4's golden
|
||||
republish at controller 0.161.0 + vouch** — until then no valid pairing ISO can be built, by
|
||||
design. The vouch (`handleSetArtifacts`) is operator-password-gated (no admin API), so it needs
|
||||
the operator.
|
||||
|
||||
## Red-proofs (run, observed, restored — verbatim failures)
|
||||
## Part detail
|
||||
|
||||
1. **THE CLOBBER RED-PROOF** — R-39(a) guard block removed from `maybeHeal`; the TOCTOU test
|
||||
(operator Re-issue staged mid-tick via the onEvent hook) failed with:
|
||||
`reissue calls = 1, want 0 — the R-39(a) guard must refuse over an unconsumed secret`
|
||||
— i.e. the operator's fresh unconsumed secret would have been clobbered
|
||||
(the fake reissuer mimics the production `SaveOneTimeSecret` side effect, so the clobber is
|
||||
observed on the row, not inferred). Guard restored → green.
|
||||
2. **Heal rate-limit** — `LastEventAt`/`healCooldown` check removed; failed with:
|
||||
`reissue calls after recurrence = 2, want STILL 1 (one restage per customer per 24h)`. Restored.
|
||||
3. **Stuck-event cooldown** — cooldown check removed; failed with:
|
||||
`stuck events = 2, want exactly 1 (24h per-customer cooldown)`. Restored.
|
||||
**Part 1 (belt).** host-install: appliance mode installs the H1 belt by default (`--no-oob` opts
|
||||
out; byo still refuses `--enable-oob` — deliberate exclusion, flagged not changed), and the config
|
||||
writer now sets `oob.enabled=true` so the agent activates the belt instead of leaving the units
|
||||
inert (the pre-v1.25.0 state that left it unusable — F9). Lockout-safe with no operator peer
|
||||
(empty sets + no key, `:22` untouched). **Live:** belt installed on felhom-pve + demo-hp (both
|
||||
8822, sets filled `{10.77.0.250}`/`{8822}`, felhom-op key installed, `:22` pid unchanged). **Login
|
||||
PROVEN on felhom-pve** end-to-end (`felhom-op@demo-felhom`) — see the operator-identity note below.
|
||||
|
||||
Plus: the full-Check demo-felhom-shape test (applied + stale staged → zero events, zero calls,
|
||||
row byte-untouched), evidence gates (<4 reports → no heal but stuck event still fires; mixed
|
||||
offbox history → no heal), young-consumed silence, disabled/blocked skip, dispatcher severity
|
||||
tests (warning routes operator-only; an info variant would be silent — pinned beside the v0.71.0
|
||||
guard, `severityNotifies` untouched), 5 card render tests + `deliveryViewFor` amber derivation.
|
||||
Green gates: hub 17 packages ok + `hub_confirm_gate.py`; controller 25 packages ok + all template
|
||||
gates (the pre-existing R-29 `docker_run_volume_path_gate` red noted, untouched).
|
||||
**Operator identity re-established (latent gap fixed).** The registered `oob_operator_ssh_pubkey`
|
||||
was ORPHANED — it matched no key the operator held (an OOB credential the operator can't reproduce
|
||||
is no better than the lockout it prevents). Re-anchored to the operator's real machine: registered
|
||||
its WireGuard pubkey as the operator peer `10.77.0.250` + its `felhom-op-1` SSH key, built the
|
||||
operator wg-felhom tunnel config (`ep0.felhom.eu:443`), added + persisted ep0's `forward` rules
|
||||
(`10.77.0.250 → 10.77.0.2/.3` in `/etc/nftables.conf`). Along the way: a corp-network UDP/443 block
|
||||
was diagnosed (ep0 tcpdump — packets arriving on 51820 but not 443) and the login completed from a
|
||||
home network on 443. All secrets file→file, shredded.
|
||||
|
||||
## Live validation (read-only, both fixtures intact)
|
||||
**Part 2 (apt).** `align_apt_repos` (appliance-only, idempotent, pre-apt): disable enterprise
|
||||
deb822 `.sources` (`Enabled: no`) / legacy `.list`, add `pve-no-subscription.sources` (suite+keyring
|
||||
derived), `apt-get update` clean. **Live:** both boxes were on enterprise (401 on update); aligned →
|
||||
enterprise+ceph disabled, no-subscription added, `apt-get update` exit 0 on both.
|
||||
|
||||
- **Checker silence:** 10 min of 60 s ticks on the live fleet → **0** `offsite-delivery` log
|
||||
lines, **0** detector events in the DB — both fixtures are healthy and the detector agrees.
|
||||
- **Fixture states from live data** (fresh DB copy, shredded after):
|
||||
demo-hp `latest_report_offsite=True, secret consumed 09:53:41` → **applied**;
|
||||
demo-felhom `latest_report_offsite=True, secret_row=(2026-07-21 08:29:29, None)` →
|
||||
**applied + stale-staged since 07-21** — the live specimen SURVIVED the train untouched
|
||||
(`consumed_at` still NULL, created_at unchanged); peti-felhom → applied.
|
||||
- **demo-hp controller page** (authed endpoint fetch inside the guest, ASCII-safe greps):
|
||||
banner count 0, `felhom-offsite-card` id count 0 (configured + 1 toggled app → no card at all),
|
||||
configured markers present. v0.161.0 healthy on both boxes.
|
||||
- **Method:** endpoint-level + DB-input derivation (no browser on DooPlex). The rendered card is
|
||||
pinned by render tests; the operator's 10-second residual: demo-hp Edit page shows
|
||||
`applied`, demo-felhom shows `applied` + the stale-staged note (since 2026-07-21).
|
||||
- **The self-heal is NOT live-fired** — no broken box exists and none was broken for it (F9
|
||||
rule). It ships unit-proven + red-proofed, PARTIAL/IMPLEMENTED on the ROADMAP with the
|
||||
explicit "fires on next natural occurrence or a staged drill" note — never PROVEN-LIVE.
|
||||
The controller banner leg is likewise unit-proven/live-pending (no box occupies the
|
||||
enabled+no-offbox window; the next fresh onboarding is its natural live leg).
|
||||
**Part 3 (R-63).** felhom-bootstrap loads a Latin-2 console font (`Lat2-Terminus16` → `Lat2-Fixed16`
|
||||
→ `Lat2-Terminus14`) once before the first paint (idempotent, best-effort, never blocks boot). Font
|
||||
names verified present in the trixie console-setup package. Renders ő/ű in the R-59 network screen
|
||||
(élő/telepítő/ellenőrizze) + pairing banner (képernyő/teendő). Nested-console capture pending drill.
|
||||
|
||||
## Rulings recorded (CONTEXT.md)
|
||||
**Part 4 (build-gate + rule).** `assert_golden_ge_floor` + `ver_ge` in build-felhom-iso; runs before
|
||||
workspace/mint; `die`s with both versions on `golden < floor`; `FELHOM_ASSERT_GOLDEN/FLOOR` inputs;
|
||||
unset warns loudly. Red-proofed both directions (floor>golden → exit 1; golden≥floor → OK; equal →
|
||||
OK; unset → WARN). Standing rule 5 in `publish-train-rules.md`. Live-demonstrated against the real
|
||||
production gap (see above).
|
||||
|
||||
State precedence (applied wins; stale-staged is a flag, never a downgrade); durable cooldowns via
|
||||
the events table (restart-proof by design); both detector events operator-only (no
|
||||
customerMessages entry, not in allowedEventTypes — the pbsdr_* precedent) until the mechanism has
|
||||
history; heal disabled without a provisioner.
|
||||
**Part 6 (root vault).** On felhom-pve, generated a fresh strong root password (never printed),
|
||||
`chpasswd`, vaulted via `PUT /api/v1/hosts/demo-felhom-8363b5/recovery-credential` (HTTP 200, the
|
||||
same day-0 `step_break_glass` mechanism, host api_key). Verified: retrieved via the operator
|
||||
endpoint (global key) and it AUTHENTICATED root over `:22` (`VAULT-AUTH-OK: demo-felhom`); copy
|
||||
shredded. felhom-pve now has G1 vault parity with demo-hp. (Note: the vault's `set_at` reflects the
|
||||
original day-0 stamp — not bumped on rotation; cosmetic, the SECRET is the rotated one, proven by
|
||||
the auth test.)
|
||||
|
||||
## Observed, not acted on
|
||||
## Remaining (the two heaviest legs, now enforced-sequenced)
|
||||
|
||||
- R-29: `docker_run_volume_path_gate` red on `appexport/estimate.go:179` (pre-existing, has its
|
||||
own row — the 3-line allowlist fix remains undone by design of this train's scope).
|
||||
- The `szolg` single-hit on the demo-hp page grep is an accent-truncated unrelated word (the
|
||||
ASCII-only-grep trap documented in felhom-controller/CLAUDE.md — verified benign via the
|
||||
card-id count of 0).
|
||||
1. **Golden republish** (Part 4): `build-golden.sh 9100 … gitea.dooplex.hu/admin/felhom-controller:0.161.0`
|
||||
on a Proxmox host → publish to Gitea → **operator vouches** `artifact_golden_version=0.161.0` +
|
||||
sha in the manifest UI (password-gated). This closes the LIVE golden<floor gap. Managed floor
|
||||
stays 0.156.0.
|
||||
2. **ISO v1.25.0 rebuild + nested drill** (Part 5): once golden ≥ floor, build the pairing ISO
|
||||
(gate now passes) and run the v1.24.0-style nested drill (belt present+active, apt=no-sub,
|
||||
R-63 console capture, golden boots 0.161.0 with NO managed-update firing = the R-71 proof,
|
||||
`.rootpw.txt` opens the console, build-gate red-proof in the nest).
|
||||
|
||||
## Observed / not acted on
|
||||
|
||||
- The live golden(0.153)<floor(0.156) gap predates this train (an old golden bake vs a later floor
|
||||
lift) — the new gate surfaces it; the golden republish fixes it.
|
||||
- ep0 root is reachable from DooPlex (used for the operator-tunnel + forward-rule work); the forward
|
||||
rules are persisted to survive reboot.
|
||||
|
||||
@@ -107,7 +107,7 @@ repo consistent, 14 snapshots, latest 04:16 CEST (pre-outage), **zero locks** (r
|
||||
| # | Finding | Severity | Evidence | Disposition |
|
||||
|---|---|---|---|---|
|
||||
| **F8** | **Site power loss; no auto-power-on.** Power itself returned within minutes (router rebooted at ~14:42 and stayed up); both miniPCs remained off ~4.5 h until manual power-on. For a paying customer this converts a power blip into a half-day outage ending only when someone is physically present. | HIGH | p1-\*; router uptime operator-attested | **mitigated-on-site** — Viktor set BIOS AC-power-on on both boxes (attested; not OS-verifiable). **roadmap-candidate:** make BIOS "restore on AC power" a provisioning-checklist item + a host-install doc requirement for every fleet box. |
|
||||
| **F9** | **H1 OOB belt only partially installed on the current fleet.** `felhom-mgmt-watchdog.timer` is live on both boxes (heal marker absent = no heal needed), but `felhom-sshd.service` and `felhom-oob-nft.service` (TASK H1, present in `felhom-agent/configs/`) are installed on **neither** box; operator access rides stock sshd :22 + tailscale + G1 break-glass. Pre-existing (both boxes provisioned via the universal ISO), surfaced by this audit's access preflight. | MEDIUM | p2-felhom-pve-ssh-belt.txt, p2-\*-host.txt | **needs-ruling** — was H1 intentionally dropped from the universal-ISO provisioning path, or should host-install grow the belt? |
|
||||
| **F9** | **H1 OOB belt only partially installed on the current fleet.** `felhom-mgmt-watchdog.timer` is live on both boxes (heal marker absent = no heal needed), but `felhom-sshd.service` and `felhom-oob-nft.service` (TASK H1, present in `felhom-agent/configs/`) are installed on **neither** box; operator access rides stock sshd :22 + tailscale + G1 break-glass. Pre-existing (both boxes provisioned via the universal ISO), surfaced by this audit's access preflight. | MEDIUM | p2-felhom-pve-ssh-belt.txt, p2-\*-host.txt | **RESOLVED 2026-07-23 (ISO train v1.25.0).** Ruling: install everywhere. host-install grows the belt as a DEFAULT appliance leg (`--no-oob` opts out; byo still refuses — deliberate). Phase-0 confirmed the omission was NOT a coded exclusion, just `--enable-oob` never passed by the universal ISO. Belt installed + `oob.enabled` on BOTH live boxes; **login PROVEN end-to-end on felhom-pve** (`felhom-op@demo-felhom` over wg-felhom → belt); also re-anchored the orphaned operator identity to the operator's real machine. See `REPORT.md` (2026-07-23) + `operations/nodes.md`. |
|
||||
| **F10** | **demo-hp backup tiers incomplete.** Tier-2 dump tree exists (paperless-ngx, fresh), but **no offbox target is configured** and the PBS DR datastore holds **0 snapshots** for it. A power event with disk damage would have had no off-box recovery path for that guest. Pre-existing (box added 07-21), not outage-caused. | MEDIUM | p6-demo-hp-offbox-locks.txt (`NO-OFFBOX`), p6-demo-hp-backup.txt (`snapshots=0`), p6-demo-hp-tier2.txt | **OFFSITE LEG RESOLVED 2026-07-23** — root cause was NOT "provisioning unfinished": the day-0 managed update killed the apply-bridge after password-consume, burning the one-shot credential (full diagnosis, designed-path repair via operator Re-issue, escrow ceremony, and a byte-identical offsite restore round-trip: `DIAG-f10-demo-hp-offsite-2026-07-23.md`; product rows minted **R-70** visibility + **R-71** the race). **The PBS-DR-snapshot half STAYS OPEN** pending F13 (cadence ruling) and the deliberate DR ceremony R-moment on demo-hp. |
|
||||
| **F11** | **Recovery is silent.** `host_recovered`/`node_recovered` fired correctly but carry severity `info`, which the dispatcher deliberately does not email — the operator/customer only learns of recovery by looking. During a real customer outage the "it's back" signal is arguably the second-most valuable email. Working-as-coded, so a product question, not a defect. | LOW | p5-hub-events.txt; `hub/internal/notify/dispatcher.go` severityNotifies | **needs-ruling** — opt-in recovery notifications (operator at least)? |
|
||||
| **F12** | **demo-hp has no customer notification prefs** (`customer_notifications` has no row for it) → its "customer" received no node_down email and never would. Only demo-felhom is wired (doodoo21@freemail.hu). Pre-existing demo-box config gap; on a real onboarding this must not be skippable. | LOW | p5-customer-notification-prefs.txt | **roadmap-candidate** — make notification-prefs setup a claim/onboarding step, not an optional settings page. |
|
||||
|
||||
@@ -76,6 +76,7 @@
|
||||
| R-60 | **[P2] First-boot NIC sweep self-heal: if the hub is unreachable, try DHCP across every carrier-bearing NIC before settling.** | S | **SHIPPED v1.24.0 (2026-07-22) — spike + nested drill proven (SPIKE-firstboot-nic-sweep-2026-07-22.md): cable move → sweep → heal + hub registration unaided in <1 min; sweep is structurally first-boot-only (state.json gate + the unit's done-flag condition); drill also surfaced and fixed the baked-fallback-default-route trap (flush before the bounded dhclient)** | `felhom-bootstrap` currently accepts whatever addressing the installer left behind and, if the hub cannot be reached, simply stays broken. On demo-hp the fix was a human moving one cable from the 4-port card to the onboard port — **a sweep would have healed it unaided**: enumerate NICs with `carrier=1`, DHCP each in turn, and keep the first that reaches the hub. Cheap because the box has nothing to lose at first boot (no customer data, no running guests) and the failure it repairs is total. Deliberately scoped to FIRST BOOT and to the hub-unreachable condition only — a running box must never re-shuffle its own networking. Complements **R-59**: that one refuses to produce an unreachable box, this one repairs the case where the truth changed after the install (cable moved, switch port died, the installer guessed the wrong port) |
|
||||
| R-61 | **[P1] The baked root password must be knowable by the operator — the recurring console lockout.** | S | **slice 1 SHIPPED v1.24.0 (2026-07-22): the build emits the plaintext into a 0600 sibling `<iso>.rootpw.txt` (single record of truth — never logged/manifested/committed); drill-verified against the installed box's shadow hash. Follow-up (appliance-grade record-keeping) stays open** | The ISO mints a **fresh throwaway crypt hash per build** and the plaintext is discarded, so nobody — including the person holding the machine — can log into the console of a box they just installed. Today that meant reaching demo-hp only through the G1 break-glass credential vaulted in the hub, which is the right mechanism for a *lost* password and the wrong one for a *never-known* password: it requires a working hub, a working network, and operator tooling, at exactly the moment the likely reason you need the console is that one of those is broken. **Slice 1 (do this):** the ISO build emits the baked root password into the build REPORT and the operator cheat-sheet alongside the sha256 — it is already a per-build value, so surfacing it costs nothing and closes the lockout. **Follow-up (appliance-grade):** keep it per-build random and treat the build output as the record of truth. **A fixed well-known password is explicitly REJECTED (operator ruling 2026-07-21)** — a pre-pairing box sits on a stranger's LAN with a predictable root credential, which is a far worse exposure than the lockout it would fix. Relates to G1 break-glass (the vault stays; this is about the window before/without it) |
|
||||
| R-62 | **[P3] Hub delete dialog: show the customer-id the operator must type, and reword the three acks for the ghost shape.** | XS | **idea (operator, 2026-07-22)** | Cosmetic, hub-only, docs-only in the v1.24.0 train. The delete confirmation asks the operator to type the customer-id, but the id appears NOWHERE on the Edit page the dialog opens from — the operator has to fish it out of the URL or another tab. Also: for a GHOST customer (host already gone) the three acknowledgement checkboxes describe teardown steps that cannot happen; **wording only** — the server MUST keep requiring all three (the render-gate lesson of v0.70.1 stands: reachability and requirements are separate concerns). |
|
||||
| R-63 | **The install console learns ő/ű** — the kernel default console font lacks the Hungarian double-acute glyphs, so the R-59 network screen and pairing banner rendered ő as blanks. | XS | **SHIPPED (scripts v1.25.0, 2026-07-23)** | `felhom-bootstrap.sh` loads a Latin-2 console font (`Lat2-Terminus16` → `Lat2-Fixed16` → `Lat2-Terminus14`) ONCE before the first paint — idempotent, best-effort (a missing font/ioctl never blocks boot). Lat2 ships in the trixie/PVE base (console-setup), so no copy rewording needed. Font names verified against the package. Nested-console capture proof rides the v1.25.0 drill. Also in the same train: **F9 belt-everywhere RESOLVED** (host-install default appliance leg + live on both boxes + login proven) and the **R-71 build-gate** (`build-felhom-iso` asserts golden ≥ managed floor, `publish-train-rules.md` rule 5). **Live finding: golden 0.153.0 < floor 0.156.0 in production NOW** — the gate catches it; the fix is the golden republish at 0.161.0 (Part 4, pending; the managed floor stays 0.156.0). See `REPORT.md` (2026-07-23). |
|
||||
| R-64 | **„Felhom↔Felhom media pairing blessed" — the two-box SMB pairing (one box shares, the other mounts it as NAS storage) becomes a supported, documented flow.** | XS–S | idea (2026-07-22) | Origin: the operator ran the pairing drill on the live demo pair and it WORKS — the drill itself is the pending evidence leg (a written run-through with the R-66 surfaces in play). R-66 shipped the enabling visibility: the serving box's address is now on its own Beállítások → Rendszer „Hálózat" card, and the add form names the NetBIOS trap. Blessing = a short customer-facing recipe (`documentation/controller/network-storage-nas.md` naming-caveat paragraph is the seed) + one supported-path sentence in the capability map. Flips: would add a "Felhom↔Felhom media pairing" capability row (currently unlisted). Pairs with R-65 (same two-box topology, entirely different transport + guarantees) |
|
||||
| R-66 | **The box's own address becomes visible — „Hálózat" card, Debug network dump, NetBIOS hint.** | XS | **SHIPPED (controller v0.159.0, 2026-07-22)** | Origin: the pairing drill — the serving box's IP was findable only as a hint buried on the OTHER box's Megosztás page, and the add form's failure for „FELHOM" taught nothing. Three legs: (A) „Hálózat" card on Beállítások → Rendszer (Helyi cím / Hálózati név only-while-sharing / Átjáró; live per render, stored nowhere — S-5; „—" when unavailable); (B) `network` section in the Debug dump (interfaces/route/DNS/lan_address, best-effort per item); (C) the NetBIOS trap named (Szerver helper text + a purely lexical hint on `unreachable` for single-label non-IP names). **Design decision recorded:** the controller is bridge-netns'd, so ALL guest-net reads go through the one netns door (docker exec into host-networked felhom-samba, `stacks/guestnet.go`) — with Megosztás off the card honestly shows „—" rather than the plausible-wrong 172.x answer. Deployed demo-felhom + demo-hp 2026-07-22; demo-hp live-shows the closed-door path (sharing off → dashes + in-place dump errors), demo-felhom the open one (real .104/.1/\\FELHOM values). Flips no capability-map row (diagnosability/UX polish); enables R-64 |
|
||||
| R-67 | **The NAS share appears in FileBrowser — browse what you mounted.** | S | **SHIPPED (controller v0.160.0, 2026-07-22)** | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). **Couples to R-64: browsing was its missing UX half.** A registered network storage now binds its share ROOT into FileBrowser (`/mnt/felhom-drives/<name>:/srv/<name>:rslave`) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (`Host is down` in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: `felhom-controller/REPORT.md` (2026-07-22) |
|
||||
|
||||
@@ -83,6 +83,36 @@ may be trying to fix. Slice 1 is to emit the baked password into the build repor
|
||||
|
||||
`demo-hp-lan` (`192.168.0.87` via `ProxyJump felhom-pve`) is the fallback while the box is away.
|
||||
|
||||
## OOB belt (H1) — both boxes, since 2026-07-23 (ISO train v1.25.0)
|
||||
|
||||
The dedicated OOB sshd belt (TASK H1: `felhom-sshd` + the static `inet felhom_oob` table + `felhom-op`)
|
||||
is installed and **active on BOTH fleet boxes** — the F9 gap (belt on neither) is closed. From
|
||||
v1.25.0 host-install installs it by default on every appliance install (`--no-oob` opts out; byo still
|
||||
refuses).
|
||||
|
||||
- **Claimed port: `8822` on both** (first-free from `[8822,2222,8022,62222]`; persisted per box).
|
||||
- **Reachability: the wg-felhom offsite tunnel ONLY** — the belt admits the operator `/32`
|
||||
(`10.77.0.250`) over `wg-felhom` to 8822 and drops everything else; `:22` and every other interface
|
||||
are untouched. **tailscale does NOT reach the belt** (wrong fabric, dropped by design).
|
||||
- **Operator login** (from the machine holding the wg-felhom operator tunnel + the registered
|
||||
`oob_operator_ssh_pubkey`): `ssh -p 8822 felhom-op@10.77.0.2` (felhom-pve) / `@10.77.0.3` (demo-hp).
|
||||
**PROVEN live 2026-07-23** on felhom-pve (`felhom-op@demo-felhom`).
|
||||
- **Operator tunnel**: the Mac/Windows operator peer dials `ep0.felhom.eu:443` (WireGuard), address
|
||||
`10.77.0.250/32`, AllowedIPs `10.77.0.0/24`, server pubkey `f3d1ZI7…`. ep0's `forward` chain
|
||||
(persisted in its `/etc/nftables.conf`) allows `10.77.0.250 → 10.77.0.2/.3`. If a work-network blocks
|
||||
UDP/443, the RheinMetall-style firewalls pass UDP/51820 — a home/hotspot network works on 443.
|
||||
- Register/rotate the operator identity hub-side: `PUT /api/v1/admin/wg/operator-peer` (global key)
|
||||
with `{pubkey, assigned_ip:"10.77.0.250", ssh_pubkey}`; wgsync pushes it to ep0 and the SSH key flows
|
||||
to both boxes' `felhom-op` authorized_keys within a tick.
|
||||
|
||||
## felhom-pve (the N100) — vault parity + access
|
||||
|
||||
felhom-pve has **operator SSH-key access** (over tailscale `100.70.170.35`) AND, since 2026-07-23,
|
||||
**G1 break-glass vault parity with demo-hp**: its root@pam password is freshly rotated and vaulted in
|
||||
the hub `host_recovery` row **`demo-felhom-8363b5`** (same PUT `…/recovery-credential` mechanism day-0
|
||||
uses; verified retrievable + authenticating over `:22`). Retrieval + shred-the-copy recipe is identical
|
||||
to demo-hp's below (swap the host_id). So a lost N100 key is recoverable the same way as the key-less HP.
|
||||
|
||||
## Tailscale on demo-hp is an OPERATOR-LAB EXCEPTION
|
||||
|
||||
> **Read this before any product-shape audit.** `demo-hp` is **customer-shaped** — it is a normal
|
||||
|
||||
@@ -201,6 +201,11 @@ Recommended order: **GL-1 and GL-3 immediately** (operator-heavy, unblock everyt
|
||||
|
||||
## 6. Open questions & operator actions
|
||||
|
||||
**Provisioning-checklist items (every fleet box, bench prep):**
|
||||
- **BIOS "restore on AC power" = ON** (power-outage audit F8, 2026-07-22): a site power blip must not
|
||||
become a half-day outage waiting for someone to press the button. Set + attest per box (not
|
||||
OS-verifiable). Both current demo boxes done (Viktor, 07-22).
|
||||
|
||||
**Operator actions (Viktor):**
|
||||
- **Hub manifest bump — DONE** (confirmed at GL-6 Phase 0: the manifest vouches agent `0.76.0` +
|
||||
golden `0.103.0`). No action.
|
||||
|
||||
Reference in New Issue
Block a user