diff --git a/REPORT.md b/REPORT.md index 4305988..d33fd4c 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,149 +2,178 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## N100 rehearsal S8 wrap — evidence, VALIDATION, map flips, ROADMAP, R-3 — 2026-07-18 +## Pre-travel train — R-39 heal · scripts v1.21.0 + ISOs · nav polish · (golden deferred) — 2026-07-18 -**Class:** documentation/evidence run. **Repos touched:** `felhom.eu` only, docs only. -**No code changed anywhere** — every finding is recorded and ranked, none fixed (campaign discipline). +**Commits (this repo):** `bcdb042` scripts v1.21.0 · ROADMAP touch (R-39 diagnosis + R-33 collapse). +**Sibling commits:** `felhom-agent` `9596d5a` (v0.90.1) + `f22f70c` (report) · `felhom-controller` +`24d23b8` (v0.146.0) + `fd93020` (accordion tests). -**Commit:** `aa9c08f` — *docs(rehearsal): N100 S8 wrap — VALIDATION, map flips, ROADMAP R-30..R-39, -R-3 draft* (pushed to `main`; parent `bbaa844`). - -No CHANGELOG entry: `felhom.eu` keeps per-area changelogs (`hub/`, `scripts/`, `website/`) and this -run touched none of those areas. - -### What the run proved - -The complete final-product flow ran **end to end on real metal in one pass**: RESET → generic -pairing ISO v1.20.0 → **customer self-bind** → credential delivery → day-0 → managed-floor lift → -drive enrolment → escrow ceremony → offsite snapshots. - -| Leg | Wall-clock | -|---|---| -| Blank-metal boot → registered as an unclaimed appliance | **< 1 s** | -| Bind → credential delivered | **26 s** | -| Bind → controller running the CURRENT version | **2 min 44 s** | -| Managed floor 0.143.0 → 0.145.0 (`initiated_by: auto-floor`) | **5 s**, unattended | -| Escrow ceremony → offsite runs enabled | **12 s** | -| Drive enrol (confirm → usable, incl. mkfs) | **30.3 s** | - -**No post-bind leg stalled** — that is the immediacy row's real-onboarding proof. +**Phases 1–3 shipped. Phase 4 skipped cleanly (its own "time-permitting"). Phase 5 (golden 0.146.0 ++ publish) NOT started — see the closing section.** --- -## Phase A — evidence bundle (completed before the box was unplugged) +## Phase 1 — R-39: diagnosis, and the brief's hypothesis refuted -`180:~/n100-rehearsal/` — 984 KB, 10 files + `MANIFEST.md`: +**The conditional hub fix was NOT shipped, because its condition proved false.** The brief said to +ship a generation-bump fix "only if step 1–2 pin the mechanism to *re-mint fails to bump the +generation*". It does not: -| File | Evidences | +- `store.SetHostDesired` bumps `desired_generation` **unconditionally** — it went **2 → 3** on the + re-issue. +- `web/configs.go`'s `applyPBSDR` is **exonerated**: its "idempotent … no re-key, no second secret, + no spurious generation bump" comment at ~L618 is accurate and guarded by the + `cur != nil && cur.Namespace != ""` early return. The hub log shows mint #2 came from the + **re-issue** path, not from an Edit-tab Save. The comment-vs-behaviour contradiction the brief + expected does not exist. + +**The real mechanism is a signal mismatch between two tiers.** The hub's re-consume signal is *a +generation bump + a poke*. The agent's re-apply trigger is *a change in the descriptor content hash* +(`felhom-agent internal/pbsdr/manager.go` ~L235): + +```go +if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { + return // idempotent: this exact descriptor already converged +} +``` + +An ep0 credential re-issue re-keys the **secret of an existing token**, so `token_id` and +`fingerprint` never change and the descriptor stays **byte-identical** — only the side-table +`host_pbs_secrets` row rotates. Same hash → converged agent short-circuits → the fresh secret is +never consumed → the box keeps presenting a revoked credential → **401 forever**. Proof in one line: +`consumed-failed.json` carries hash `a4e5424…`, **identical** to the `marker.json` written two +minutes before the re-issue. The comment at `hub/internal/web/pbsdr.go:320` asserts the reissue +refreshes the descriptor "with the NEW token_id/fingerprint" — false for this op. + +**Timeline (hub log is CEST; the hub DB is UTC — a split *within one service*):** + +| CEST | Event | |---|---| -| `bootstrap-journal.txt` | first boot 16:17:14 UTC, PAIRING mode, unclaimed registration, the 30 s poll loop and its **52 systemd FAILED lines**, day-0 SUCCESS 16:32:32 | -| `host-state.txt` | `pct config 9201` (12288 MB), `pveversion -v`, `pvesm status` (**felhom-pbs 401/inactive**), `lsblk`, `blkid` | -| `guest-docker.txt` | 11 containers, controller 0.145.0 healthy, image pin | -| `controller-log-full.txt` / `controller-log.txt` | the whole controller life: auto-floor update, 16:55 config-refresh restart, escrow ceremony, orphan guard, both backup runs | -| `agent-journal.txt` | agent 0.90.0 start, WG registration, storage-wipe gate pending→confirmed, format + enrol, PBS apply, and the repeating PBS verify-loop permission error | -| `pbs-dr-state.txt` | the R-39 failure: storage.cfg stanza, `pvesm status`, credential metadata, direct token probe (401 everywhere), WG health, agent errors | -| `restic-and-pool.txt` | 2 snapshots / **48.717 MiB** attributed, pool `/home` listing + `du` showing **1.4 G + 3.0 M unattributed** | -| `hub-state.txt` | hosts row, both RESET ledger rows, the **unconsumed** PBS secret, self-bind token lifecycle, appliance registration, events timeline | -| `hub-events-timeline.txt` | the 15-row demo-felhom event stream from 15:50 | -| `timezones.txt` | the live-captured CEST/UTC split | +| 18:30:51 | `pbsdr provisioned … gen 2; secret stored consume-once` — mint #1, via the WG-registration hook, **with** a generation bump | +| 18:45:51 | agent consumes mint #1 → `converged state=applied` | +| 18:47:52 | `pbsdr credentials **re-issued** … fresh consume-once secret stored` — mint #2, `consumed_at` stayed NULL | -Secrets were read only to run probes and are recorded as lengths and metadata, never as values. +**A second, independent defect, found while healing.** `configs/felhom-pbs-apply`'s `reconcile` +passed `--server` to `pvesm set`; PVE treats `server` as **create-only** and rejects the whole call +even when the value is byte-identical. So *every* re-apply exited 255 — and because the agent +consumes the one-time secret **before** invoking the wrapper, each re-issue **burned a credential**. +Proven live before writing code: with `--server` → rejected; without → **rc 0**. Fixed in **agent +v0.90.1** (one argv line + red-proof `TestReconcileNeverPassesServerToPvesmSet`, verified red then +green; it handles two vacuous-pass traps — CRLF line endings, and the WHY comment quoting the very +flag under test). -**Not collected:** the operator gauge screenshots (Viktor's set) and the **PBS DR tab**, which -remains uncaptured — and now matters more, because R-39 means it would have shown a fault. The hub -UI is password-gated, so CC cannot capture it. +**Cost I incurred:** proving the mechanism consumed the pending secret against the still-unfixed +wrapper, so it burned. The box was already 401 before and after — no functional regression — but the +recoverable state was gone until an operator re-issue. The agent parked correctly in +`consumed-failed.json` with `NOT retrying silently`: **no burn loop**, the fail-safe worked. + +**HEALED — Viktor's re-issue click closed the chain in 9 s:** hub re-issued 20:28:44 → agent +consumed 20:28:51 → `converged state=applied` 20:28:53, with the patched wrapper. + +| Check | Before | After | +|---|---|---| +| `pvesm status` | `401 Unauthorized` / `inactive` | **`active`** | +| Direct token probe `/api2/json/version` | `401` | **`200`** | +| Real backup | none possible | **`felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z`, 9 744 319 312 B, 13m36s** | + +Encrypted under fingerprint `7e:a6:af:f7:ea:6d:3e:d9` — the **escrowed** key, the one customer zero +holds the recovery code for. The DR tier's **first real backup on the reborn box**. Nothing was +destroyed: `.pw`, `.enc` (K) and the `storage.cfg` entry verified intact (PVE rejects atomically, so +the set-only law held). + +**Left for the fleet spec, deliberately not improvised:** (a) make a fresh unconsumed secret actually +un-converge the agent; (b) fix the verify loop's read path — it reads `/etc/pve/priv/storage/.pw` +directly as non-root, a file it can only ever *write* through the root wrapper (`/etc/pve/priv` is +`0700 root:www-data`; sudoers exposes `create|reconcile|grant`, **no read verb**); (c) an auth probe +so `applied` can never mean `401`. --- -## Phases B–E — what landed +## Phase 2 — scripts v1.21.0 + fresh ISOs -- **Phase B** — `documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md`: run context, a - **UTC-normalised** timeline (the host journal is CEST, everything else UTC — a 2 h trap), derived - wall-clocks, per-ledger verdicts for S1–S8 plus ledgers 8 and 9, 12 findings, the not-exercised - list, and the discrepancy table. -- **Phase C** — 10 capability-map rows annotated, each citing the Phase-B doc. -- **Phase D** — 5 items collapsed, 11 new ranked items, 1 pre-invite checklist. -- **Phase E** — Viktor's draft (which landed as `runbooks/RUNBOOK-onboarding-draft-v3.md`, not the - filename the brief predicted) refined: all four `[REFINE]` slots filled, self-bind made the default - bind path, the measured wall-clock table inlined, interim workarounds for R-31/R-36/R-39 added, and - C6→C7 marked as the single unexecuted step. **Committed as DRAFT** — it graduates when C7 has - evidence. +`run_pairing()` now loops **inside** the script (30s sleep — hub-side rate unchanged) instead of +exiting non-zero per poll, so the unit sits in `activating` and systemd prints nothing on the +customer's console. Registration split into `register_appliance()` whose transient failures the loop +retries. Journal quiet but not dark: logged once on entry, then a 10-minute heartbeat; `410` still +exits non-zero on purpose. Console banner every 5 min, single accented spelling, plus the missing +reassurance („Ez a képernyő magától frissül"). -### Capability-map rows flipped +**The load-bearing half is `TimeoutStartSec=infinity`** — a `Type=oneshot` ExecStart is killed at 90s, +so without it systemd would kill the new wait and `Restart=on-failure` would silently reinstate the +exact spam this removes, *after appearing to work for the first three polls*. -| Row | Change | +**Verified behaviourally**, in a container against a stub hub answering `204` five times then +delivering: **one log line plus one heartbeat, zero exits between polls**, then a clean fall-through +to the direct install and `exit 0`. The old design produced 5 unit invocations and 5 `Failed to +start` console lines for that same sequence. + +**ISOs rebuilt (both `--pairing`, `--loader mkimage`, same PVE input `proxmox-ve_9.2-1.iso` +sha `4e88fe41…`), `secret-bearing: no`:** + +| ISO | sha256 | |---|---| -| Bare-metal Felhom ISO | PARTIAL → **PROVEN-LIVE** — F1 closed on physical hardware | -| Customer self-bind (slice 1) | IMPLEMENTED → **PROVEN-LIVE** — `customer_selfbind`, 0 failed attempts | -| Guest RAM resize (R-24) | IMPLEMENTED → **PROVEN-LIVE** — shrink **and** grow, live cgroup | -| Customer RESET | + two real firings, verified external teardown, orphan-guard interaction | -| Escrow ceremony | + first live customer-facing wizard firing | -| Immediacy | "real-onboarding proof pending" **cleared** | -| Publish train / floor | + box-side floor lift proven on a fresh install | -| Customer claim | + R-4 gmail half (Inbox under `p=quarantine`) | -| Offsite orphan guard | staged live leg **fired on its own** | -| **DR tier by default** | candidate PROVEN-LIVE upgrade **WITHDRAWN** (R-39) | +| `felhom-pve-9.2-1-v1.21.0-n100-generic-mkimage.iso` (safety) | `b1b25fd412b779bcacbfaa4c59002ee80a8f35e4c0bd1248cc35d997f967cb90` | +| `felhom-pve-9.2-1-v1.21.0-n100-demo-generic-mkimage.iso` (real) | `90a0fb7da3f3f11d315bf1cdf55a7da84ab9e6d75e06b2942041470f7864031d` | -**Not flipped, as instructed:** customer-performs-restore (still MISSING), BYO, DLNA, multi-user. - -### ROADMAP IDs assigned - -| Rank | IDs | -|---|---| -| P2-HIGH | **R-39** PBS DR applied-but-dead · **R-30** liveness from the wait channel · **R-31** async offsite + status card · **R-32** RESET base-dir purge + orphan-card honesty + unattributed-bytes gauge · **R-33** bootstrap console quiet-poll | -| P2 | **R-34** backup data lifecycle · **R-35** config-apply session survival · **R-36** post-RESET re-enroll offsite prompt · **R-27c** console-passphrase bind | -| P3 | **R-37** post-RESET health card · **R-38** installer GRUB slice | - -Collapsed: **R-1** (appliance half done; Peti half survives), **R-21** (physically closed), **R-24**, -**R-27** slice 1, **R-4** (gmail half). No existing item renumbered. +Both at `180:/mnt/5_hdd/felhom.eu/felhom-iso/out/`. **Verified the fix actually shipped inside the +artifact**, not just in git: extracted the embedded first-boot payload with xorriso and decoded it — +the shipped `felhom-bootstrap.sh` is **byte-identical to the committed source**, carries the new +cadence constants and the `while true` loop, the old `will poll again in 30s` exit line is **gone**, +and the embedded unit carries `TimeoutStartSec=infinity`. Viktor flashes the stick. --- -## Discrepancies found vs the brief (live evidence won in each case) +## Phase 3 — controller v0.146.0 nav polish -1. **"expect ~11675 MB" vs `pct config` 12288 MB.** Both correct at different times: F5 auto-sized to - 11675, then the customer shrank to 8192 (16:50:22) and grew to 12288 (17:02:17). Not a fault — - it is what completes ledger 9. -2. **"2.6 GB pool vs ~0 attributed"** → measured **1.4 G + 3.0 M** unattributed vs **49 M** - attributed, across **two** `.orphaned-*` directories, not one. The R-32 ruling is unaffected; the - numbers are. -3. **"felhom-pbs entry present"** → present but **401 / inactive**. The brief treated the descriptor - as done; it is provisioned but dead. → R-39. -4. **"RESET ×2 … second + third live firings"** → the ledger holds **exactly two rows, both - tonight**, on **two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51). - Recorded as tonight's two firings rather than as ordinals. -5. **"keep R-29 if the number is free"** → **it is not free.** R-29 was committed earlier the same day - (`2eccda4`) for the design-v2 gate rot — which *is* the brief's item 5. So the liveness item took - **R-30**, and **no new ID was minted for the red-gate item**. -6. **Tip and filename:** the brief's tip `a03e142` is two commits behind `bbaa844`, and Viktor's R-3 - draft landed as `RUNBOOK-onboarding-draft-v3.md`, not `DRAFT-R3-onboarding-runbook.md`. -7. **R-39 has no counterpart in the brief at all** — it was found by probing the box, not by working - the finding list. Its P2-HIGH rank is therefore **provisional pending Viktor**. +Built, pushed and **deployed to guest 9201** (`0.146.0 Up (healthy)`). -### The one finding worth reading twice +- **Scrollbars:** thin + hairline-coloured; `scrollbar-width`/`scrollbar-color` for Firefox **and** + `::-webkit-scrollbar` (8px, thumb `--line`, hover `--text-3`, `--radius`) for WebKit/Blink, since + neither alone covers the browsers customers use. `.sidebar` → `--bg-2` track, `html` → `--bg-0`. + Tokens only. +- **Collapsible groups:** Tárhely / Biztonsági mentés / Megosztás as accordions, chevron, exactly one + open. Header is a **real `