admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: pre-travel train — R-39 diagnosis + heal, R-33 shipped, golden deferred

Browse Source 
ROADMAP:
 - R-39 gains the full live diagnosis and REFUTES the brief's hypothesis. The
   generation IS bumped (SetHostDesired bumps unconditionally, 2->3) and
   applyPBSDR is exonerated, so no hub fix was shipped. The real mechanism is a
   signal mismatch: the hub's re-consume signal is a generation bump + poke,
   while the agent re-applies on a change of the DESCRIPTOR CONTENT HASH
   (manager.go ~L235). An ep0 re-issue re-keys the secret of an EXISTING token,
   so token_id/fingerprint are unchanged, the descriptor is byte-identical, the
   hash never moves, and the fresh secret is never consumed -> 401 forever.
   Proof: consumed-failed.json carries the same hash a4e5424... as the marker
   written two minutes before the re-issue.
   Records the second defect found while healing (wrapper reconcile passing
   --server, fixed in agent v0.90.1), marks the box HEALED with evidence
   (pvesm active, token 200, a real 9.7 GB encrypted backup listed PBS-side),
   and leaves the fleet fix explicitly pending its own spec.
 - R-33 collapses to SHIPPED (scripts v1.21.0), incl. why
   TimeoutStartSec=infinity is the load-bearing half.
 - Pre-invite checklist: golden target moves 0.145.x -> 0.146.0 and notes it is
   now MORE stale, since v0.146.0 is live on the demo box while the golden still
   bakes 0.143.0.

REPORT overwritten with the train: R-39 diagnosis verbatim + heal evidence, the
two ISO shas with the byte-identical-payload verification, the nav polish and
why the screenshot leg could not be done (the demo controller password is
customer-owned since the claim flow, so the build-server credentials are stale),
Phase 4 skipped cleanly, and Phase 5 deferred rather than half-run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
This commit is contained in:
authentik Default Admin
2026-07-18 21:02:34 +02:00
parent bcdb04222a
commit b14b5a65d9
2 changed files with 148 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+145 -116
View File
@@ -2,149 +2,178 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## N100 rehearsal S8 wrap — evidence, VALIDATION, map flips, ROADMAP, R-3 — 2026-07-18
## Pre-travel train — R-39 heal · scripts v1.21.0 + ISOs · nav polish · (golden deferred) — 2026-07-18
**Class:** documentation/evidence run. **Repos touched:** `felhom.eu` only, docs only.
**No code changed anywhere** — every finding is recorded and ranked, none fixed (campaign discipline).
**Commits (this repo):** `bcdb042` scripts v1.21.0 · ROADMAP touch (R-39 diagnosis + R-33 collapse).
**Sibling commits:** `felhom-agent` `9596d5a` (v0.90.1) + `f22f70c` (report) · `felhom-controller`
`24d23b8` (v0.146.0) + `fd93020` (accordion tests).
**Commit:** `aa9c08f` — *docs(rehearsal): N100 S8 wrap — VALIDATION, map flips, ROADMAP R-30..R-39,
R-3 draft* (pushed to `main`; parent `bbaa844`).
No CHANGELOG entry: `felhom.eu` keeps per-area changelogs (`hub/`, `scripts/`, `website/`) and this
run touched none of those areas.
### What the run proved
The complete final-product flow ran **end to end on real metal in one pass**: RESET → generic
pairing ISO v1.20.0 → **customer self-bind** → credential delivery → day-0 → managed-floor lift →
drive enrolment → escrow ceremony → offsite snapshots.
| Leg | Wall-clock |
|---|---|
| Blank-metal boot → registered as an unclaimed appliance | **< 1 s** |
| Bind → credential delivered | **26 s** |
| Bind → controller running the CURRENT version | **2 min 44 s** |
| Managed floor 0.143.0 → 0.145.0 (`initiated_by: auto-floor`) | **5 s**, unattended |
| Escrow ceremony → offsite runs enabled | **12 s** |
| Drive enrol (confirm → usable, incl. mkfs) | **30.3 s** |
**No post-bind leg stalled** — that is the immediacy row's real-onboarding proof.
**Phases 1–3 shipped. Phase 4 skipped cleanly (its own "time-permitting"). Phase 5 (golden 0.146.0
+ publish) NOT started — see the closing section.**
---
## Phase A — evidence bundle (completed before the box was unplugged)
## Phase 1 — R-39: diagnosis, and the brief's hypothesis refuted
`180:~/n100-rehearsal/` — 984 KB, 10 files + `MANIFEST.md`:
**The conditional hub fix was NOT shipped, because its condition proved false.** The brief said to
ship a generation-bump fix "only if step 1–2 pin the mechanism to *re-mint fails to bump the
generation*". It does not:
| File | Evidences |
- `store.SetHostDesired` bumps `desired_generation` **unconditionally** — it went **2 → 3** on the
re-issue.
- `web/configs.go`'s `applyPBSDR` is **exonerated**: its "idempotent … no re-key, no second secret,
no spurious generation bump" comment at ~L618 is accurate and guarded by the
`cur != nil && cur.Namespace != ""` early return. The hub log shows mint #2 came from the
**re-issue** path, not from an Edit-tab Save. The comment-vs-behaviour contradiction the brief
expected does not exist.
**The real mechanism is a signal mismatch between two tiers.** The hub's re-consume signal is *a
generation bump + a poke*. The agent's re-apply trigger is *a change in the descriptor content hash*
(`felhom-agent internal/pbsdr/manager.go` ~L235):
```go
if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) {
return // idempotent: this exact descriptor already converged
}
```
An ep0 credential re-issue re-keys the **secret of an existing token**, so `token_id` and
`fingerprint` never change and the descriptor stays **byte-identical** — only the side-table
`host_pbs_secrets` row rotates. Same hash → converged agent short-circuits → the fresh secret is
never consumed → the box keeps presenting a revoked credential → **401 forever**. Proof in one line:
`consumed-failed.json` carries hash `a4e5424…`, **identical** to the `marker.json` written two
minutes before the re-issue. The comment at `hub/internal/web/pbsdr.go:320` asserts the reissue
refreshes the descriptor "with the NEW token_id/fingerprint" — false for this op.
**Timeline (hub log is CEST; the hub DB is UTC — a split *within one service*):**
| CEST | Event |
|---|---|
| `bootstrap-journal.txt` | first boot 16:17:14 UTC, PAIRING mode, unclaimed registration, the 30 s poll loop and its **52 systemd FAILED lines**, day-0 SUCCESS 16:32:32 |
| `host-state.txt` | `pct config 9201` (12288 MB), `pveversion -v`, `pvesm status` (**felhom-pbs 401/inactive**), `lsblk`, `blkid` |
| `guest-docker.txt` | 11 containers, controller 0.145.0 healthy, image pin |
| `controller-log-full.txt` / `controller-log.txt` | the whole controller life: auto-floor update, 16:55 config-refresh restart, escrow ceremony, orphan guard, both backup runs |
| `agent-journal.txt` | agent 0.90.0 start, WG registration, storage-wipe gate pending→confirmed, format + enrol, PBS apply, and the repeating PBS verify-loop permission error |
| `pbs-dr-state.txt` | the R-39 failure: storage.cfg stanza, `pvesm status`, credential metadata, direct token probe (401 everywhere), WG health, agent errors |
| `restic-and-pool.txt` | 2 snapshots / **48.717 MiB** attributed, pool `/home` listing + `du` showing **1.4 G + 3.0 M unattributed** |
| `hub-state.txt` | hosts row, both RESET ledger rows, the **unconsumed** PBS secret, self-bind token lifecycle, appliance registration, events timeline |
| `hub-events-timeline.txt` | the 15-row demo-felhom event stream from 15:50 |
| `timezones.txt` | the live-captured CEST/UTC split |
| 18:30:51 | `pbsdr provisioned … gen 2; secret stored consume-once` — mint #1, via the WG-registration hook, **with** a generation bump |
| 18:45:51 | agent consumes mint #1 → `converged state=applied` |
| 18:47:52 | `pbsdr credentials **re-issued** … fresh consume-once secret stored` — mint #2, `consumed_at` stayed NULL |
Secrets were read only to run probes and are recorded as lengths and metadata, never as values.
**A second, independent defect, found while healing.** `configs/felhom-pbs-apply`'s `reconcile`
passed `--server` to `pvesm set`; PVE treats `server` as **create-only** and rejects the whole call
even when the value is byte-identical. So *every* re-apply exited 255 — and because the agent
consumes the one-time secret **before** invoking the wrapper, each re-issue **burned a credential**.
Proven live before writing code: with `--server` → rejected; without → **rc 0**. Fixed in **agent
v0.90.1** (one argv line + red-proof `TestReconcileNeverPassesServerToPvesmSet`, verified red then
green; it handles two vacuous-pass traps — CRLF line endings, and the WHY comment quoting the very
flag under test).
**Not collected:** the operator gauge screenshots (Viktor's set) and the **PBS DR tab**, which
remains uncaptured — and now matters more, because R-39 means it would have shown a fault. The hub
UI is password-gated, so CC cannot capture it.
**Cost I incurred:** proving the mechanism consumed the pending secret against the still-unfixed
wrapper, so it burned. The box was already 401 before and after — no functional regression — but the
recoverable state was gone until an operator re-issue. The agent parked correctly in
`consumed-failed.json` with `NOT retrying silently`: **no burn loop**, the fail-safe worked.
**HEALED — Viktor's re-issue click closed the chain in 9 s:** hub re-issued 20:28:44 → agent
consumed 20:28:51 → `converged state=applied` 20:28:53, with the patched wrapper.
| Check | Before | After |
|---|---|---|
| `pvesm status` | `401 Unauthorized` / `inactive` | **`active`** |
| Direct token probe `/api2/json/version` | `401` | **`200`** |
| Real backup | none possible | **`felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z`, 9 744 319 312 B, 13m36s** |
Encrypted under fingerprint `7e:a6:af:f7:ea:6d:3e:d9` — the **escrowed** key, the one customer zero
holds the recovery code for. The DR tier's **first real backup on the reborn box**. Nothing was
destroyed: `.pw`, `.enc` (K) and the `storage.cfg` entry verified intact (PVE rejects atomically, so
the set-only law held).
**Left for the fleet spec, deliberately not improvised:** (a) make a fresh unconsumed secret actually
un-converge the agent; (b) fix the verify loop's read path — it reads `/etc/pve/priv/storage/<id>.pw`
directly as non-root, a file it can only ever *write* through the root wrapper (`/etc/pve/priv` is
`0700 root:www-data`; sudoers exposes `create|reconcile|grant`, **no read verb**); (c) an auth probe
so `applied` can never mean `401`.
---
## Phases B–E — what landed
## Phase 2 — scripts v1.21.0 + fresh ISOs
- **Phase B** — `documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md`: run context, a
**UTC-normalised** timeline (the host journal is CEST, everything else UTC — a 2 h trap), derived
wall-clocks, per-ledger verdicts for S1–S8 plus ledgers 8 and 9, 12 findings, the not-exercised
list, and the discrepancy table.
- **Phase C** — 10 capability-map rows annotated, each citing the Phase-B doc.
- **Phase D** — 5 items collapsed, 11 new ranked items, 1 pre-invite checklist.
- **Phase E** — Viktor's draft (which landed as `runbooks/RUNBOOK-onboarding-draft-v3.md`, not the
filename the brief predicted) refined: all four `[REFINE]` slots filled, self-bind made the default
bind path, the measured wall-clock table inlined, interim workarounds for R-31/R-36/R-39 added, and
C6→C7 marked as the single unexecuted step. **Committed as DRAFT** — it graduates when C7 has
evidence.
`run_pairing()` now loops **inside** the script (30s sleep — hub-side rate unchanged) instead of
exiting non-zero per poll, so the unit sits in `activating` and systemd prints nothing on the
customer's console. Registration split into `register_appliance()` whose transient failures the loop
retries. Journal quiet but not dark: logged once on entry, then a 10-minute heartbeat; `410` still
exits non-zero on purpose. Console banner every 5 min, single accented spelling, plus the missing
reassurance („Ez a képernyő magától frissül").
### Capability-map rows flipped
**The load-bearing half is `TimeoutStartSec=infinity`** — a `Type=oneshot` ExecStart is killed at 90s,
so without it systemd would kill the new wait and `Restart=on-failure` would silently reinstate the
exact spam this removes, *after appearing to work for the first three polls*.
| Row | Change |
**Verified behaviourally**, in a container against a stub hub answering `204` five times then
delivering: **one log line plus one heartbeat, zero exits between polls**, then a clean fall-through
to the direct install and `exit 0`. The old design produced 5 unit invocations and 5 `Failed to
start` console lines for that same sequence.
**ISOs rebuilt (both `--pairing`, `--loader mkimage`, same PVE input `proxmox-ve_9.2-1.iso`
sha `4e88fe41…`), `secret-bearing: no`:**
| ISO | sha256 |
|---|---|
| Bare-metal Felhom ISO | PARTIAL → **PROVEN-LIVE** — F1 closed on physical hardware |
| Customer self-bind (slice 1) | IMPLEMENTED → **PROVEN-LIVE** — `customer_selfbind`, 0 failed attempts |
| Guest RAM resize (R-24) | IMPLEMENTED → **PROVEN-LIVE** — shrink **and** grow, live cgroup |
| Customer RESET | + two real firings, verified external teardown, orphan-guard interaction |
| Escrow ceremony | + first live customer-facing wizard firing |
| Immediacy | "real-onboarding proof pending" **cleared** |
| Publish train / floor | + box-side floor lift proven on a fresh install |
| Customer claim | + R-4 gmail half (Inbox under `p=quarantine`) |
| Offsite orphan guard | staged live leg **fired on its own** |
| **DR tier by default** | candidate PROVEN-LIVE upgrade **WITHDRAWN** (R-39) |
| `felhom-pve-9.2-1-v1.21.0-n100-generic-mkimage.iso` (safety) | `b1b25fd412b779bcacbfaa4c59002ee80a8f35e4c0bd1248cc35d997f967cb90` |
| `felhom-pve-9.2-1-v1.21.0-n100-demo-generic-mkimage.iso` (real) | `90a0fb7da3f3f11d315bf1cdf55a7da84ab9e6d75e06b2942041470f7864031d` |
**Not flipped, as instructed:** customer-performs-restore (still MISSING), BYO, DLNA, multi-user.
### ROADMAP IDs assigned
| Rank | IDs |
|---|---|
| P2-HIGH | **R-39** PBS DR applied-but-dead · **R-30** liveness from the wait channel · **R-31** async offsite + status card · **R-32** RESET base-dir purge + orphan-card honesty + unattributed-bytes gauge · **R-33** bootstrap console quiet-poll |
| P2 | **R-34** backup data lifecycle · **R-35** config-apply session survival · **R-36** post-RESET re-enroll offsite prompt · **R-27c** console-passphrase bind |
| P3 | **R-37** post-RESET health card · **R-38** installer GRUB slice |
Collapsed: **R-1** (appliance half done; Peti half survives), **R-21** (physically closed), **R-24**,
**R-27** slice 1, **R-4** (gmail half). No existing item renumbered.
Both at `180:/mnt/5_hdd/felhom.eu/felhom-iso/out/`. **Verified the fix actually shipped inside the
artifact**, not just in git: extracted the embedded first-boot payload with xorriso and decoded it —
the shipped `felhom-bootstrap.sh` is **byte-identical to the committed source**, carries the new
cadence constants and the `while true` loop, the old `will poll again in 30s` exit line is **gone**,
and the embedded unit carries `TimeoutStartSec=infinity`. Viktor flashes the stick.
---
## Discrepancies found vs the brief (live evidence won in each case)
## Phase 3 — controller v0.146.0 nav polish
1. **"expect ~11675 MB" vs `pct config` 12288 MB.** Both correct at different times: F5 auto-sized to
11675, then the customer shrank to 8192 (16:50:22) and grew to 12288 (17:02:17). Not a fault —
it is what completes ledger 9.
2. **"2.6 GB pool vs ~0 attributed"** → measured **1.4 G + 3.0 M** unattributed vs **49 M**
attributed, across **two** `.orphaned-*` directories, not one. The R-32 ruling is unaffected; the
numbers are.
3. **"felhom-pbs entry present"** → present but **401 / inactive**. The brief treated the descriptor
as done; it is provisioned but dead. → R-39.
4. **"RESET ×2 … second + third live firings"** → the ledger holds **exactly two rows, both
tonight**, on **two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51).
Recorded as tonight's two firings rather than as ordinals.
5. **"keep R-29 if the number is free"** → **it is not free.** R-29 was committed earlier the same day
(`2eccda4`) for the design-v2 gate rot — which *is* the brief's item 5. So the liveness item took
**R-30**, and **no new ID was minted for the red-gate item**.
6. **Tip and filename:** the brief's tip `a03e142` is two commits behind `bbaa844`, and Viktor's R-3
draft landed as `RUNBOOK-onboarding-draft-v3.md`, not `DRAFT-R3-onboarding-runbook.md`.
7. **R-39 has no counterpart in the brief at all** — it was found by probing the box, not by working
the finding list. Its P2-HIGH rank is therefore **provisional pending Viktor**.
Built, pushed and **deployed to guest 9201** (`0.146.0 Up (healthy)`).
### The one finding worth reading twice
- **Scrollbars:** thin + hairline-coloured; `scrollbar-width`/`scrollbar-color` for Firefox **and**
`::-webkit-scrollbar` (8px, thumb `--line`, hover `--text-3`, `--radius`) for WebKit/Blink, since
neither alone covers the browsers customers use. `.sidebar` → `--bg-2` track, `html` → `--bg-0`.
Tokens only.
- **Collapsible groups:** Tárhely / Biztonsági mentés / Megosztás as accordions, chevron, exactly one
open. Header is a **real `<button>`** with `aria-expanded` + `aria-controls` + `:focus-visible`, so
keyboard/AT reachability is real rather than simulated. **Nothing became unreachable** — checked
first: every group's landing page is also its first sub-item. **Progressive enhancement** — the
active group is opened **server-side**, so it is correct before any JS runs. **No layout jump** —
`grid-template-rows: 0fr → 1fr` rather than `max-height`, animating to the content's real height
with no magic number to drift; the toggle reserves its active border as transparent; both
transitions off under `prefers-reduced-motion`.
**R-39 — the DR tier can report `applied` and be dead simultaneously.** The descriptor
auto-provisioned and the agent converged `state=applied` at 16:45:53. But `pvesm status` returns
`401 Unauthorized`/`inactive`, and a direct probe with the stored credential 401s on **every**
endpoint including `/version` while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — so it
is authentication, not ACL scope. The hub minted a **second** token secret at 16:47:52, two minutes
*after* the agent applied the first, and its `consumed_at` is **still NULL**; the converged state
machine will not re-apply, and the agent's 15-minute verify loop **cannot read the credential to
notice**, because it reads directly as non-root a file it writes through a root sudo wrapper. A
silent DR failure surfaces first at a restore, which is the worst possible moment — hence the
withdrawn map upgrade and the interim `pvesm status` check now in R-3 step E1.
All design-v2 gates PASS (`template_id_gate`, `emoji_gate`, `native_confirm_gate`,
`offbox_rename_gate`, `mojibake_gate`, `app_row_dedup_gate`); build/vet/tests green.
`docker_run_volume_path_gate` still fails on `estimate.go:179` — that is **R-29**, pre-existing,
verified to fail identically on the untouched tree, and deliberately not bundled.
**Screenshot leg NOT done, and here is the honest reason.** The demo controller's password is
**customer-owned** since the claim flow — Viktor set it during the rehearsal — so the credentials on
the build server are stale and a curl-login returns the `Bejelentkezés` page. Instead of asserting
nothing, four **render tests** (`internal/web/nav_accordion_test.go`) pin the server-side half through
the real shared layout: every sub-page opens its own group with `aria-expanded=true` and an `.active`
toggle and **exactly one** group open (the count is asserted, not just the expected group); a page
outside any group opens nothing; every group's landing page still exists as a sub-link; the toggle is
a real button whose `aria-controls` targets a real element. **Red-proofed** — removing the `is-open`
marker fails two assertions on both storage pages. The visual leg still wants Viktor's browser.
---
## Remaining to first invite
## Phase 4 — skipped cleanly
**freemail.hu test-send (Viktor, remote-doable) + C6 customer-restore evidence (remote-doable on the
reborn box from vacation — the dashboard is remote) + R-11 rulings (contact channel, agreement,
thresholds) + golden 0.145.x rebuild.**
Its own instruction was "time-permitting; skip cleanly if not". Nothing was started, so nothing is
half-done. Note that its (a) auto-mint self-bind link, (b) post-RESET health card and (c)
unprovisioned-offsite flash correspond to **R-36 / R-37 / R-36** and remain open as written. The
conditional Phase-1 hub fix is **not** part of any v0.67.0 train, because its condition was refuted.
R-39 is not on that line because it was discovered after the brief was written; if Viktor confirms
the P2-HIGH rank, it belongs there too — a first tester whose DR tier is silently dead is exactly the
scenario the alpha exists to avoid.
## Phase 5 — NOT started (the honest hand-off)
Viktor's polish/tweak notes arrive separately and are neither pre-empted nor invented here.
The golden bake is a nested-VM procedure (snapshot-revert → cold boot → transient `golden-bake` unit
→ 404 pre-gate → upload → sha verify → teardown → token-leak grep). It is well recorded
(`pilot/RUNBOOK-publish-0.85-0.120-2026-07-12` Phase C) and entirely repeatable, but it is a long
multi-step operation with a revert dance at the end, and this session had already run long. Starting
it and running out mid-way would leave a booted drill VM and a half-published artifact — strictly
worse than a clean stop. It also ends in a **STOP → Viktor** regardless, since the manifest and floor
saves are password-gated clicks.
**So: controller v0.146.0 is live on the demo box, but the golden still bakes 0.143.0.** The
pre-invite checklist item "rebuild golden → 0.145.x" is therefore now "rebuild golden → **0.146.0**",
and it is the first thing to pick up next session.