Files
felhom.eu/REPORT.md
T
admin b14b5a65d9 docs: pre-travel train — R-39 diagnosis + heal, R-33 shipped, golden deferred
ROADMAP:
 - R-39 gains the full live diagnosis and REFUTES the brief's hypothesis. The
   generation IS bumped (SetHostDesired bumps unconditionally, 2->3) and
   applyPBSDR is exonerated, so no hub fix was shipped. The real mechanism is a
   signal mismatch: the hub's re-consume signal is a generation bump + poke,
   while the agent re-applies on a change of the DESCRIPTOR CONTENT HASH
   (manager.go ~L235). An ep0 re-issue re-keys the secret of an EXISTING token,
   so token_id/fingerprint are unchanged, the descriptor is byte-identical, the
   hash never moves, and the fresh secret is never consumed -> 401 forever.
   Proof: consumed-failed.json carries the same hash a4e5424... as the marker
   written two minutes before the re-issue.
   Records the second defect found while healing (wrapper reconcile passing
   --server, fixed in agent v0.90.1), marks the box HEALED with evidence
   (pvesm active, token 200, a real 9.7 GB encrypted backup listed PBS-side),
   and leaves the fleet fix explicitly pending its own spec.
 - R-33 collapses to SHIPPED (scripts v1.21.0), incl. why
   TimeoutStartSec=infinity is the load-bearing half.
 - Pre-invite checklist: golden target moves 0.145.x -> 0.146.0 and notes it is
   now MORE stale, since v0.146.0 is live on the demo box while the golden still
   bakes 0.143.0.

REPORT overwritten with the train: R-39 diagnosis verbatim + heal evidence, the
two ISO shas with the byte-identical-payload verification, the nav polish and
why the screenshot leg could not be done (the demo controller password is
customer-owned since the claim flow, so the build-server credentials are stale),
Phase 4 skipped cleanly, and Phase 5 deferred rather than half-run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
2026-07-18 21:02:34 +02:00

180 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# felhom.eu — task reports
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## Pre-travel train — R-39 heal · scripts v1.21.0 + ISOs · nav polish · (golden deferred) — 2026-07-18
**Commits (this repo):** `bcdb042` scripts v1.21.0 · ROADMAP touch (R-39 diagnosis + R-33 collapse).
**Sibling commits:** `felhom-agent` `9596d5a` (v0.90.1) + `f22f70c` (report) · `felhom-controller`
`24d23b8` (v0.146.0) + `fd93020` (accordion tests).
**Phases 13 shipped. Phase 4 skipped cleanly (its own "time-permitting"). Phase 5 (golden 0.146.0
+ publish) NOT started — see the closing section.**
---
## Phase 1 — R-39: diagnosis, and the brief's hypothesis refuted
**The conditional hub fix was NOT shipped, because its condition proved false.** The brief said to
ship a generation-bump fix "only if step 12 pin the mechanism to *re-mint fails to bump the
generation*". It does not:
- `store.SetHostDesired` bumps `desired_generation` **unconditionally** — it went **2 → 3** on the
re-issue.
- `web/configs.go`'s `applyPBSDR` is **exonerated**: its "idempotent … no re-key, no second secret,
no spurious generation bump" comment at ~L618 is accurate and guarded by the
`cur != nil && cur.Namespace != ""` early return. The hub log shows mint #2 came from the
**re-issue** path, not from an Edit-tab Save. The comment-vs-behaviour contradiction the brief
expected does not exist.
**The real mechanism is a signal mismatch between two tiers.** The hub's re-consume signal is *a
generation bump + a poke*. The agent's re-apply trigger is *a change in the descriptor content hash*
(`felhom-agent internal/pbsdr/manager.go` ~L235):
```go
if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) {
return // idempotent: this exact descriptor already converged
}
```
An ep0 credential re-issue re-keys the **secret of an existing token**, so `token_id` and
`fingerprint` never change and the descriptor stays **byte-identical** — only the side-table
`host_pbs_secrets` row rotates. Same hash → converged agent short-circuits → the fresh secret is
never consumed → the box keeps presenting a revoked credential → **401 forever**. Proof in one line:
`consumed-failed.json` carries hash `a4e5424…`, **identical** to the `marker.json` written two
minutes before the re-issue. The comment at `hub/internal/web/pbsdr.go:320` asserts the reissue
refreshes the descriptor "with the NEW token_id/fingerprint" — false for this op.
**Timeline (hub log is CEST; the hub DB is UTC — a split *within one service*):**
| CEST | Event |
|---|---|
| 18:30:51 | `pbsdr provisioned … gen 2; secret stored consume-once` — mint #1, via the WG-registration hook, **with** a generation bump |
| 18:45:51 | agent consumes mint #1`converged state=applied` |
| 18:47:52 | `pbsdr credentials **re-issued** … fresh consume-once secret stored` — mint #2, `consumed_at` stayed NULL |
**A second, independent defect, found while healing.** `configs/felhom-pbs-apply`'s `reconcile`
passed `--server` to `pvesm set`; PVE treats `server` as **create-only** and rejects the whole call
even when the value is byte-identical. So *every* re-apply exited 255 — and because the agent
consumes the one-time secret **before** invoking the wrapper, each re-issue **burned a credential**.
Proven live before writing code: with `--server` → rejected; without → **rc 0**. Fixed in **agent
v0.90.1** (one argv line + red-proof `TestReconcileNeverPassesServerToPvesmSet`, verified red then
green; it handles two vacuous-pass traps — CRLF line endings, and the WHY comment quoting the very
flag under test).
**Cost I incurred:** proving the mechanism consumed the pending secret against the still-unfixed
wrapper, so it burned. The box was already 401 before and after — no functional regression — but the
recoverable state was gone until an operator re-issue. The agent parked correctly in
`consumed-failed.json` with `NOT retrying silently`: **no burn loop**, the fail-safe worked.
**HEALED — Viktor's re-issue click closed the chain in 9 s:** hub re-issued 20:28:44 → agent
consumed 20:28:51 → `converged state=applied` 20:28:53, with the patched wrapper.
| Check | Before | After |
|---|---|---|
| `pvesm status` | `401 Unauthorized` / `inactive` | **`active`** |
| Direct token probe `/api2/json/version` | `401` | **`200`** |
| Real backup | none possible | **`felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z`, 9 744 319 312 B, 13m36s** |
Encrypted under fingerprint `7e:a6:af:f7:ea:6d:3e:d9` — the **escrowed** key, the one customer zero
holds the recovery code for. The DR tier's **first real backup on the reborn box**. Nothing was
destroyed: `.pw`, `.enc` (K) and the `storage.cfg` entry verified intact (PVE rejects atomically, so
the set-only law held).
**Left for the fleet spec, deliberately not improvised:** (a) make a fresh unconsumed secret actually
un-converge the agent; (b) fix the verify loop's read path — it reads `/etc/pve/priv/storage/<id>.pw`
directly as non-root, a file it can only ever *write* through the root wrapper (`/etc/pve/priv` is
`0700 root:www-data`; sudoers exposes `create|reconcile|grant`, **no read verb**); (c) an auth probe
so `applied` can never mean `401`.
---
## Phase 2 — scripts v1.21.0 + fresh ISOs
`run_pairing()` now loops **inside** the script (30s sleep — hub-side rate unchanged) instead of
exiting non-zero per poll, so the unit sits in `activating` and systemd prints nothing on the
customer's console. Registration split into `register_appliance()` whose transient failures the loop
retries. Journal quiet but not dark: logged once on entry, then a 10-minute heartbeat; `410` still
exits non-zero on purpose. Console banner every 5 min, single accented spelling, plus the missing
reassurance („Ez a képernyő magától frissül").
**The load-bearing half is `TimeoutStartSec=infinity`** — a `Type=oneshot` ExecStart is killed at 90s,
so without it systemd would kill the new wait and `Restart=on-failure` would silently reinstate the
exact spam this removes, *after appearing to work for the first three polls*.
**Verified behaviourally**, in a container against a stub hub answering `204` five times then
delivering: **one log line plus one heartbeat, zero exits between polls**, then a clean fall-through
to the direct install and `exit 0`. The old design produced 5 unit invocations and 5 `Failed to
start` console lines for that same sequence.
**ISOs rebuilt (both `--pairing`, `--loader mkimage`, same PVE input `proxmox-ve_9.2-1.iso`
sha `4e88fe41…`), `secret-bearing: no`:**
| ISO | sha256 |
|---|---|
| `felhom-pve-9.2-1-v1.21.0-n100-generic-mkimage.iso` (safety) | `b1b25fd412b779bcacbfaa4c59002ee80a8f35e4c0bd1248cc35d997f967cb90` |
| `felhom-pve-9.2-1-v1.21.0-n100-demo-generic-mkimage.iso` (real) | `90a0fb7da3f3f11d315bf1cdf55a7da84ab9e6d75e06b2942041470f7864031d` |
Both at `180:/mnt/5_hdd/felhom.eu/felhom-iso/out/`. **Verified the fix actually shipped inside the
artifact**, not just in git: extracted the embedded first-boot payload with xorriso and decoded it —
the shipped `felhom-bootstrap.sh` is **byte-identical to the committed source**, carries the new
cadence constants and the `while true` loop, the old `will poll again in 30s` exit line is **gone**,
and the embedded unit carries `TimeoutStartSec=infinity`. Viktor flashes the stick.
---
## Phase 3 — controller v0.146.0 nav polish
Built, pushed and **deployed to guest 9201** (`0.146.0 Up (healthy)`).
- **Scrollbars:** thin + hairline-coloured; `scrollbar-width`/`scrollbar-color` for Firefox **and**
`::-webkit-scrollbar` (8px, thumb `--line`, hover `--text-3`, `--radius`) for WebKit/Blink, since
neither alone covers the browsers customers use. `.sidebar``--bg-2` track, `html``--bg-0`.
Tokens only.
- **Collapsible groups:** Tárhely / Biztonsági mentés / Megosztás as accordions, chevron, exactly one
open. Header is a **real `<button>`** with `aria-expanded` + `aria-controls` + `:focus-visible`, so
keyboard/AT reachability is real rather than simulated. **Nothing became unreachable** — checked
first: every group's landing page is also its first sub-item. **Progressive enhancement** — the
active group is opened **server-side**, so it is correct before any JS runs. **No layout jump**
`grid-template-rows: 0fr → 1fr` rather than `max-height`, animating to the content's real height
with no magic number to drift; the toggle reserves its active border as transparent; both
transitions off under `prefers-reduced-motion`.
All design-v2 gates PASS (`template_id_gate`, `emoji_gate`, `native_confirm_gate`,
`offbox_rename_gate`, `mojibake_gate`, `app_row_dedup_gate`); build/vet/tests green.
`docker_run_volume_path_gate` still fails on `estimate.go:179` — that is **R-29**, pre-existing,
verified to fail identically on the untouched tree, and deliberately not bundled.
**Screenshot leg NOT done, and here is the honest reason.** The demo controller's password is
**customer-owned** since the claim flow — Viktor set it during the rehearsal — so the credentials on
the build server are stale and a curl-login returns the `Bejelentkezés` page. Instead of asserting
nothing, four **render tests** (`internal/web/nav_accordion_test.go`) pin the server-side half through
the real shared layout: every sub-page opens its own group with `aria-expanded=true` and an `.active`
toggle and **exactly one** group open (the count is asserted, not just the expected group); a page
outside any group opens nothing; every group's landing page still exists as a sub-link; the toggle is
a real button whose `aria-controls` targets a real element. **Red-proofed** — removing the `is-open`
marker fails two assertions on both storage pages. The visual leg still wants Viktor's browser.
---
## Phase 4 — skipped cleanly
Its own instruction was "time-permitting; skip cleanly if not". Nothing was started, so nothing is
half-done. Note that its (a) auto-mint self-bind link, (b) post-RESET health card and (c)
unprovisioned-offsite flash correspond to **R-36 / R-37 / R-36** and remain open as written. The
conditional Phase-1 hub fix is **not** part of any v0.67.0 train, because its condition was refuted.
## Phase 5 — NOT started (the honest hand-off)
The golden bake is a nested-VM procedure (snapshot-revert → cold boot → transient `golden-bake` unit
→ 404 pre-gate → upload → sha verify → teardown → token-leak grep). It is well recorded
(`pilot/RUNBOOK-publish-0.85-0.120-2026-07-12` Phase C) and entirely repeatable, but it is a long
multi-step operation with a revert dance at the end, and this session had already run long. Starting
it and running out mid-way would leave a booted drill VM and a half-published artifact — strictly
worse than a clean stop. It also ends in a **STOP → Viktor** regardless, since the manifest and floor
saves are password-gated clicks.
**So: controller v0.146.0 is live on the demo box, but the golden still bakes 0.143.0.** The
pre-invite checklist item "rebuild golden → 0.145.x" is therefore now "rebuild golden → **0.146.0**",
and it is the first thing to pick up next session.