admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: pre-travel train — R-39 diagnosis + heal, R-33 shipped, golden deferred

Browse Source 
ROADMAP:
 - R-39 gains the full live diagnosis and REFUTES the brief's hypothesis. The
   generation IS bumped (SetHostDesired bumps unconditionally, 2->3) and
   applyPBSDR is exonerated, so no hub fix was shipped. The real mechanism is a
   signal mismatch: the hub's re-consume signal is a generation bump + poke,
   while the agent re-applies on a change of the DESCRIPTOR CONTENT HASH
   (manager.go ~L235). An ep0 re-issue re-keys the secret of an EXISTING token,
   so token_id/fingerprint are unchanged, the descriptor is byte-identical, the
   hash never moves, and the fresh secret is never consumed -> 401 forever.
   Proof: consumed-failed.json carries the same hash a4e5424... as the marker
   written two minutes before the re-issue.
   Records the second defect found while healing (wrapper reconcile passing
   --server, fixed in agent v0.90.1), marks the box HEALED with evidence
   (pvesm active, token 200, a real 9.7 GB encrypted backup listed PBS-side),
   and leaves the fleet fix explicitly pending its own spec.
 - R-33 collapses to SHIPPED (scripts v1.21.0), incl. why
   TimeoutStartSec=infinity is the load-bearing half.
 - Pre-invite checklist: golden target moves 0.145.x -> 0.146.0 and notes it is
   now MORE stale, since v0.146.0 is live on the demo box while the golden still
   bakes 0.143.0.

REPORT overwritten with the train: R-39 diagnosis verbatim + heal evidence, the
two ISO shas with the byte-identical-payload verification, the nav polish and
why the screenshot leg could not be done (the demo controller password is
customer-owned since the claim flow, so the build-server credentials are stale),
Phase 4 skipped cleanly, and Phase 5 deferred rather than half-run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
This commit is contained in:
authentik Default Admin
2026-07-18 21:02:34 +02:00
parent bcdb04222a
commit b14b5a65d9
2 changed files with 148 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+145 -116
View File
@@ -2,149 +2,178 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## N100 rehearsal S8 wrap — evidence, VALIDATION, map flips, ROADMAP, R-3 — 2026-07-18
## Pre-travel train — R-39 heal · scripts v1.21.0 + ISOs · nav polish · (golden deferred) — 2026-07-18
**Class:** documentation/evidence run. **Repos touched:** `felhom.eu` only, docs only.
**No code changed anywhere** — every finding is recorded and ranked, none fixed (campaign discipline).
**Commits (this repo):** `bcdb042` scripts v1.21.0 · ROADMAP touch (R-39 diagnosis + R-33 collapse).
**Sibling commits:** `felhom-agent` `9596d5a` (v0.90.1) + `f22f70c` (report) · `felhom-controller`
`24d23b8` (v0.146.0) + `fd93020` (accordion tests).
**Commit:** `aa9c08f` — *docs(rehearsal): N100 S8 wrap — VALIDATION, map flips, ROADMAP R-30..R-39,
R-3 draft* (pushed to `main`; parent `bbaa844`).
No CHANGELOG entry: `felhom.eu` keeps per-area changelogs (`hub/`, `scripts/`, `website/`) and this
run touched none of those areas.
### What the run proved
The complete final-product flow ran **end to end on real metal in one pass**: RESET → generic
pairing ISO v1.20.0 → **customer self-bind** → credential delivery → day-0 → managed-floor lift →
drive enrolment → escrow ceremony → offsite snapshots.
| Leg | Wall-clock |
|---|---|
| Blank-metal boot → registered as an unclaimed appliance | **< 1 s** |
| Bind → credential delivered | **26 s** |
| Bind → controller running the CURRENT version | **2 min 44 s** |
| Managed floor 0.143.0 → 0.145.0 (`initiated_by: auto-floor`) | **5 s**, unattended |
| Escrow ceremony → offsite runs enabled | **12 s** |
| Drive enrol (confirm → usable, incl. mkfs) | **30.3 s** |
**No post-bind leg stalled** — that is the immediacy row's real-onboarding proof.
**Phases 1–3 shipped. Phase 4 skipped cleanly (its own "time-permitting"). Phase 5 (golden 0.146.0
+ publish) NOT started — see the closing section.**
---
## Phase A — evidence bundle (completed before the box was unplugged)
## Phase 1 — R-39: diagnosis, and the brief's hypothesis refuted
`180:~/n100-rehearsal/` — 984 KB, 10 files + `MANIFEST.md`:
**The conditional hub fix was NOT shipped, because its condition proved false.** The brief said to
ship a generation-bump fix "only if step 1–2 pin the mechanism to *re-mint fails to bump the
generation*". It does not:
| File | Evidences |
- `store.SetHostDesired` bumps `desired_generation` **unconditionally** — it went **2 → 3** on the
re-issue.
- `web/configs.go`'s `applyPBSDR` is **exonerated**: its "idempotent … no re-key, no second secret,
no spurious generation bump" comment at ~L618 is accurate and guarded by the
`cur != nil && cur.Namespace != ""` early return. The hub log shows mint #2 came from the
**re-issue** path, not from an Edit-tab Save. The comment-vs-behaviour contradiction the brief
expected does not exist.
**The real mechanism is a signal mismatch between two tiers.** The hub's re-consume signal is *a
generation bump + a poke*. The agent's re-apply trigger is *a change in the descriptor content hash*
(`felhom-agent internal/pbsdr/manager.go` ~L235):
```go
if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) {
return // idempotent: this exact descriptor already converged
}
```
An ep0 credential re-issue re-keys the **secret of an existing token**, so `token_id` and
`fingerprint` never change and the descriptor stays **byte-identical** — only the side-table
`host_pbs_secrets` row rotates. Same hash → converged agent short-circuits → the fresh secret is
never consumed → the box keeps presenting a revoked credential → **401 forever**. Proof in one line:
`consumed-failed.json` carries hash `a4e5424…`, **identical** to the `marker.json` written two
minutes before the re-issue. The comment at `hub/internal/web/pbsdr.go:320` asserts the reissue
refreshes the descriptor "with the NEW token_id/fingerprint" — false for this op.
**Timeline (hub log is CEST; the hub DB is UTC — a split *within one service*):**
| CEST | Event |
|---|---|
| `bootstrap-journal.txt` | first boot 16:17:14 UTC, PAIRING mode, unclaimed registration, the 30 s poll loop and its **52 systemd FAILED lines**, day-0 SUCCESS 16:32:32 |
| `host-state.txt` | `pct config 9201` (12288 MB), `pveversion -v`, `pvesm status` (**felhom-pbs 401/inactive**), `lsblk`, `blkid` |
| `guest-docker.txt` | 11 containers, controller 0.145.0 healthy, image pin |
| `controller-log-full.txt` / `controller-log.txt` | the whole controller life: auto-floor update, 16:55 config-refresh restart, escrow ceremony, orphan guard, both backup runs |
| `agent-journal.txt` | agent 0.90.0 start, WG registration, storage-wipe gate pending→confirmed, format + enrol, PBS apply, and the repeating PBS verify-loop permission error |
| `pbs-dr-state.txt` | the R-39 failure: storage.cfg stanza, `pvesm status`, credential metadata, direct token probe (401 everywhere), WG health, agent errors |
| `restic-and-pool.txt` | 2 snapshots / **48.717 MiB** attributed, pool `/home` listing + `du` showing **1.4 G + 3.0 M unattributed** |
| `hub-state.txt` | hosts row, both RESET ledger rows, the **unconsumed** PBS secret, self-bind token lifecycle, appliance registration, events timeline |
| `hub-events-timeline.txt` | the 15-row demo-felhom event stream from 15:50 |
| `timezones.txt` | the live-captured CEST/UTC split |
| 18:30:51 | `pbsdr provisioned … gen 2; secret stored consume-once` — mint #1, via the WG-registration hook, **with** a generation bump |
| 18:45:51 | agent consumes mint #1 → `converged state=applied` |
| 18:47:52 | `pbsdr credentials **re-issued** … fresh consume-once secret stored` — mint #2, `consumed_at` stayed NULL |
Secrets were read only to run probes and are recorded as lengths and metadata, never as values.
**A second, independent defect, found while healing.** `configs/felhom-pbs-apply`'s `reconcile`
passed `--server` to `pvesm set`; PVE treats `server` as **create-only** and rejects the whole call
even when the value is byte-identical. So *every* re-apply exited 255 — and because the agent
consumes the one-time secret **before** invoking the wrapper, each re-issue **burned a credential**.
Proven live before writing code: with `--server` → rejected; without → **rc 0**. Fixed in **agent
v0.90.1** (one argv line + red-proof `TestReconcileNeverPassesServerToPvesmSet`, verified red then
green; it handles two vacuous-pass traps — CRLF line endings, and the WHY comment quoting the very
flag under test).
**Not collected:** the operator gauge screenshots (Viktor's set) and the **PBS DR tab**, which
remains uncaptured — and now matters more, because R-39 means it would have shown a fault. The hub
UI is password-gated, so CC cannot capture it.
**Cost I incurred:** proving the mechanism consumed the pending secret against the still-unfixed
wrapper, so it burned. The box was already 401 before and after — no functional regression — but the
recoverable state was gone until an operator re-issue. The agent parked correctly in
`consumed-failed.json` with `NOT retrying silently`: **no burn loop**, the fail-safe worked.
**HEALED — Viktor's re-issue click closed the chain in 9 s:** hub re-issued 20:28:44 → agent
consumed 20:28:51 → `converged state=applied` 20:28:53, with the patched wrapper.
| Check | Before | After |
|---|---|---|
| `pvesm status` | `401 Unauthorized` / `inactive` | **`active`** |
| Direct token probe `/api2/json/version` | `401` | **`200`** |
| Real backup | none possible | **`felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z`, 9 744 319 312 B, 13m36s** |
Encrypted under fingerprint `7e:a6:af:f7:ea:6d:3e:d9` — the **escrowed** key, the one customer zero
holds the recovery code for. The DR tier's **first real backup on the reborn box**. Nothing was
destroyed: `.pw`, `.enc` (K) and the `storage.cfg` entry verified intact (PVE rejects atomically, so
the set-only law held).
**Left for the fleet spec, deliberately not improvised:** (a) make a fresh unconsumed secret actually
un-converge the agent; (b) fix the verify loop's read path — it reads `/etc/pve/priv/storage/<id>.pw`
directly as non-root, a file it can only ever *write* through the root wrapper (`/etc/pve/priv` is
`0700 root:www-data`; sudoers exposes `create|reconcile|grant`, **no read verb**); (c) an auth probe
so `applied` can never mean `401`.
---
## Phases B–E — what landed
## Phase 2 — scripts v1.21.0 + fresh ISOs
- **Phase B** — `documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md`: run context, a
**UTC-normalised** timeline (the host journal is CEST, everything else UTC — a 2 h trap), derived
wall-clocks, per-ledger verdicts for S1–S8 plus ledgers 8 and 9, 12 findings, the not-exercised
list, and the discrepancy table.
- **Phase C** — 10 capability-map rows annotated, each citing the Phase-B doc.
- **Phase D** — 5 items collapsed, 11 new ranked items, 1 pre-invite checklist.
- **Phase E** — Viktor's draft (which landed as `runbooks/RUNBOOK-onboarding-draft-v3.md`, not the
filename the brief predicted) refined: all four `[REFINE]` slots filled, self-bind made the default
bind path, the measured wall-clock table inlined, interim workarounds for R-31/R-36/R-39 added, and
C6→C7 marked as the single unexecuted step. **Committed as DRAFT** — it graduates when C7 has
evidence.
`run_pairing()` now loops **inside** the script (30s sleep — hub-side rate unchanged) instead of
exiting non-zero per poll, so the unit sits in `activating` and systemd prints nothing on the
customer's console. Registration split into `register_appliance()` whose transient failures the loop
retries. Journal quiet but not dark: logged once on entry, then a 10-minute heartbeat; `410` still
exits non-zero on purpose. Console banner every 5 min, single accented spelling, plus the missing
reassurance („Ez a képernyő magától frissül").
### Capability-map rows flipped
**The load-bearing half is `TimeoutStartSec=infinity`** — a `Type=oneshot` ExecStart is killed at 90s,
so without it systemd would kill the new wait and `Restart=on-failure` would silently reinstate the
exact spam this removes, *after appearing to work for the first three polls*.
| Row | Change |
**Verified behaviourally**, in a container against a stub hub answering `204` five times then
delivering: **one log line plus one heartbeat, zero exits between polls**, then a clean fall-through
to the direct install and `exit 0`. The old design produced 5 unit invocations and 5 `Failed to
start` console lines for that same sequence.
**ISOs rebuilt (both `--pairing`, `--loader mkimage`, same PVE input `proxmox-ve_9.2-1.iso`
sha `4e88fe41…`), `secret-bearing: no`:**
| ISO | sha256 |
|---|---|
| Bare-metal Felhom ISO | PARTIAL → **PROVEN-LIVE** — F1 closed on physical hardware |
| Customer self-bind (slice 1) | IMPLEMENTED → **PROVEN-LIVE** — `customer_selfbind`, 0 failed attempts |
| Guest RAM resize (R-24) | IMPLEMENTED → **PROVEN-LIVE** — shrink **and** grow, live cgroup |
| Customer RESET | + two real firings, verified external teardown, orphan-guard interaction |
| Escrow ceremony | + first live customer-facing wizard firing |
| Immediacy | "real-onboarding proof pending" **cleared** |
| Publish train / floor | + box-side floor lift proven on a fresh install |
| Customer claim | + R-4 gmail half (Inbox under `p=quarantine`) |
| Offsite orphan guard | staged live leg **fired on its own** |
| **DR tier by default** | candidate PROVEN-LIVE upgrade **WITHDRAWN** (R-39) |
| `felhom-pve-9.2-1-v1.21.0-n100-generic-mkimage.iso` (safety) | `b1b25fd412b779bcacbfaa4c59002ee80a8f35e4c0bd1248cc35d997f967cb90` |
| `felhom-pve-9.2-1-v1.21.0-n100-demo-generic-mkimage.iso` (real) | `90a0fb7da3f3f11d315bf1cdf55a7da84ab9e6d75e06b2942041470f7864031d` |
**Not flipped, as instructed:** customer-performs-restore (still MISSING), BYO, DLNA, multi-user.
### ROADMAP IDs assigned
| Rank | IDs |
|---|---|
| P2-HIGH | **R-39** PBS DR applied-but-dead · **R-30** liveness from the wait channel · **R-31** async offsite + status card · **R-32** RESET base-dir purge + orphan-card honesty + unattributed-bytes gauge · **R-33** bootstrap console quiet-poll |
| P2 | **R-34** backup data lifecycle · **R-35** config-apply session survival · **R-36** post-RESET re-enroll offsite prompt · **R-27c** console-passphrase bind |
| P3 | **R-37** post-RESET health card · **R-38** installer GRUB slice |
Collapsed: **R-1** (appliance half done; Peti half survives), **R-21** (physically closed), **R-24**,
**R-27** slice 1, **R-4** (gmail half). No existing item renumbered.
Both at `180:/mnt/5_hdd/felhom.eu/felhom-iso/out/`. **Verified the fix actually shipped inside the
artifact**, not just in git: extracted the embedded first-boot payload with xorriso and decoded it —
the shipped `felhom-bootstrap.sh` is **byte-identical to the committed source**, carries the new
cadence constants and the `while true` loop, the old `will poll again in 30s` exit line is **gone**,
and the embedded unit carries `TimeoutStartSec=infinity`. Viktor flashes the stick.
---
## Discrepancies found vs the brief (live evidence won in each case)
## Phase 3 — controller v0.146.0 nav polish
1. **"expect ~11675 MB" vs `pct config` 12288 MB.** Both correct at different times: F5 auto-sized to
11675, then the customer shrank to 8192 (16:50:22) and grew to 12288 (17:02:17). Not a fault —
it is what completes ledger 9.
2. **"2.6 GB pool vs ~0 attributed"** → measured **1.4 G + 3.0 M** unattributed vs **49 M**
attributed, across **two** `.orphaned-*` directories, not one. The R-32 ruling is unaffected; the
numbers are.
3. **"felhom-pbs entry present"** → present but **401 / inactive**. The brief treated the descriptor
as done; it is provisioned but dead. → R-39.
4. **"RESET ×2 … second + third live firings"** → the ledger holds **exactly two rows, both
tonight**, on **two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51).
Recorded as tonight's two firings rather than as ordinals.
5. **"keep R-29 if the number is free"** → **it is not free.** R-29 was committed earlier the same day
(`2eccda4`) for the design-v2 gate rot — which *is* the brief's item 5. So the liveness item took
**R-30**, and **no new ID was minted for the red-gate item**.
6. **Tip and filename:** the brief's tip `a03e142` is two commits behind `bbaa844`, and Viktor's R-3
draft landed as `RUNBOOK-onboarding-draft-v3.md`, not `DRAFT-R3-onboarding-runbook.md`.
7. **R-39 has no counterpart in the brief at all** — it was found by probing the box, not by working
the finding list. Its P2-HIGH rank is therefore **provisional pending Viktor**.
Built, pushed and **deployed to guest 9201** (`0.146.0 Up (healthy)`).
### The one finding worth reading twice
- **Scrollbars:** thin + hairline-coloured; `scrollbar-width`/`scrollbar-color` for Firefox **and**
`::-webkit-scrollbar` (8px, thumb `--line`, hover `--text-3`, `--radius`) for WebKit/Blink, since
neither alone covers the browsers customers use. `.sidebar` → `--bg-2` track, `html` → `--bg-0`.
Tokens only.
- **Collapsible groups:** Tárhely / Biztonsági mentés / Megosztás as accordions, chevron, exactly one
open. Header is a **real `<button>`** with `aria-expanded` + `aria-controls` + `:focus-visible`, so
keyboard/AT reachability is real rather than simulated. **Nothing became unreachable** — checked
first: every group's landing page is also its first sub-item. **Progressive enhancement** — the
active group is opened **server-side**, so it is correct before any JS runs. **No layout jump** —
`grid-template-rows: 0fr → 1fr` rather than `max-height`, animating to the content's real height
with no magic number to drift; the toggle reserves its active border as transparent; both
transitions off under `prefers-reduced-motion`.
**R-39 — the DR tier can report `applied` and be dead simultaneously.** The descriptor
auto-provisioned and the agent converged `state=applied` at 16:45:53. But `pvesm status` returns
`401 Unauthorized`/`inactive`, and a direct probe with the stored credential 401s on **every**
endpoint including `/version` while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — so it
is authentication, not ACL scope. The hub minted a **second** token secret at 16:47:52, two minutes
*after* the agent applied the first, and its `consumed_at` is **still NULL**; the converged state
machine will not re-apply, and the agent's 15-minute verify loop **cannot read the credential to
notice**, because it reads directly as non-root a file it writes through a root sudo wrapper. A
silent DR failure surfaces first at a restore, which is the worst possible moment — hence the
withdrawn map upgrade and the interim `pvesm status` check now in R-3 step E1.
All design-v2 gates PASS (`template_id_gate`, `emoji_gate`, `native_confirm_gate`,
`offbox_rename_gate`, `mojibake_gate`, `app_row_dedup_gate`); build/vet/tests green.
`docker_run_volume_path_gate` still fails on `estimate.go:179` — that is **R-29**, pre-existing,
verified to fail identically on the untouched tree, and deliberately not bundled.
**Screenshot leg NOT done, and here is the honest reason.** The demo controller's password is
**customer-owned** since the claim flow — Viktor set it during the rehearsal — so the credentials on
the build server are stale and a curl-login returns the `Bejelentkezés` page. Instead of asserting
nothing, four **render tests** (`internal/web/nav_accordion_test.go`) pin the server-side half through
the real shared layout: every sub-page opens its own group with `aria-expanded=true` and an `.active`
toggle and **exactly one** group open (the count is asserted, not just the expected group); a page
outside any group opens nothing; every group's landing page still exists as a sub-link; the toggle is
a real button whose `aria-controls` targets a real element. **Red-proofed** — removing the `is-open`
marker fails two assertions on both storage pages. The visual leg still wants Viktor's browser.
---
## Remaining to first invite
## Phase 4 — skipped cleanly
**freemail.hu test-send (Viktor, remote-doable) + C6 customer-restore evidence (remote-doable on the
reborn box from vacation — the dashboard is remote) + R-11 rulings (contact channel, agreement,
thresholds) + golden 0.145.x rebuild.**
Its own instruction was "time-permitting; skip cleanly if not". Nothing was started, so nothing is
half-done. Note that its (a) auto-mint self-bind link, (b) post-RESET health card and (c)
unprovisioned-offsite flash correspond to **R-36 / R-37 / R-36** and remain open as written. The
conditional Phase-1 hub fix is **not** part of any v0.67.0 train, because its condition was refuted.
R-39 is not on that line because it was discovered after the brief was written; if Viktor confirms
the P2-HIGH rank, it belongs there too — a first tester whose DR tier is silently dead is exactly the
scenario the alpha exists to avoid.
## Phase 5 — NOT started (the honest hand-off)
Viktor's polish/tweak notes arrive separately and are neither pre-empted nor invented here.
The golden bake is a nested-VM procedure (snapshot-revert → cold boot → transient `golden-bake` unit
→ 404 pre-gate → upload → sha verify → teardown → token-leak grep). It is well recorded
(`pilot/RUNBOOK-publish-0.85-0.120-2026-07-12` Phase C) and entirely repeatable, but it is a long
multi-step operation with a revert dance at the end, and this session had already run long. Starting
it and running out mid-way would leave a booted drill VM and a half-published artifact — strictly
worse than a clean stop. It also ends in a **STOP → Viktor** regardless, since the manifest and floor
saves are password-gated clicks.
**So: controller v0.146.0 is live on the demo box, but the golden still bakes 0.143.0.** The
pre-invite checklist item "rebuild golden → 0.145.x" is therefore now "rebuild golden → **0.146.0**",
and it is the first thing to pick up next session.
documentation/backlog/ROADMAP.md
+3 -3
View File
@@ -35,11 +35,11 @@
| ID | Item | Size | Status | Notes |
|----|------|------|--------|-------|
| R-39 | **[P2-HIGH] The PBS DR tier can be `applied` and dead at the same time — and nothing notices.** On the reborn N100 the descriptor auto-provisioned and the agent converged `state=applied`, yet `pvesm status` reports `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive` and a direct probe with the stored credential 401s on **every** endpoint including `/version` (WG transport healthy: handshake 9 s, 27.9 ms RTT — so authentication, not ACL scope). Three compounding defects: **(a)** a **mint/consume race** — the hub minted a SECOND token secret at 16:47:52, two minutes *after* the agent applied the first, and `consumed_at` is still NULL; **(b)** the converged state machine will not re-apply, so the box is pinned to a stale secret; **(c)** the agent's 15-minute PBS verify loop **cannot read the credential to detect any of it** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — the non-root agent writes that file through a root sudo wrapper, then reads it directly). | M | idea | **NEW — discovered by CC while collecting Phase-A evidence; not on the brief's finding list, so its P2-HIGH rank is provisional pending Viktor.** The severity case: this is the DR tier, the failure is silent, and it would surface first at a real restore. Suggested shape: make `consumed_at` authoritative (a fresh unconsumed secret must un-converge the reconciler), fix the verify loop's read path (read via the same root wrapper that writes it), and make a failing `pvesm status` a LOUD state rather than a skipped datastore. **Blocks the DR-tier map row's candidate upgrade to PROVEN-LIVE — that upgrade is now explicitly WITHDRAWN.** Evidence `pbs-dr-state.txt`, `hub-state.txt` |
| R-39 | **[P2-HIGH] The PBS DR tier can be `applied` and dead at the same time — and nothing notices.** On the reborn N100 the descriptor auto-provisioned and the agent converged `state=applied`, yet `pvesm status` reports `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive` and a direct probe with the stored credential 401s on **every** endpoint including `/version` (WG transport healthy: handshake 9 s, 27.9 ms RTT — so authentication, not ACL scope). Three compounding defects: **(a)** a **mint/consume race** — the hub minted a SECOND token secret at 16:47:52, two minutes *after* the agent applied the first, and `consumed_at` is still NULL; **(b)** the converged state machine will not re-apply, so the box is pinned to a stale secret; **(c)** the agent's 15-minute PBS verify loop **cannot read the credential to detect any of it** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — the non-root agent writes that file through a root sudo wrapper, then reads it directly). | M | **diagnosed + box healed 2026-07-18; FLEET FIX PENDING SPEC** | **DIAGNOSIS (2026-07-18, live on the N100 — supersedes the initial hypothesis).** The brief guessed "the re-mint fails to bump the generation". **That is FALSE and no hub fix was shipped:** `store.SetHostDesired` bumps `desired_generation` unconditionally (it went 2→3 on the re-issue), and `web/configs.go`'s `applyPBSDR` is likewise exonerated — its "no re-key, no second secret, no spurious generation bump" comment is accurate, guarded by the `cur != nil && cur.Namespace != ""` early return, and the hub log shows mint #2 came from the **re-issue** path, not from an Edit-tab Save. **The real mechanism is a signal mismatch between the two tiers.** The hub's re-consume signal is *a generation bump + a poke*; the agent's re-apply trigger is *a change in the DESCRIPTOR CONTENT HASH* (`felhom-agent internal/pbsdr/manager.go` ~L235: `if mk := m.loadMarker(); mk != nil && mk.Hash == h && (cf == nil || cf.Hash != h) { return }`). An ep0 credential re-issue re-keys the **secret of an existing token**, so `token_id` and `fingerprint` are unchanged and the descriptor is **byte-identical** — only the side-table `host_pbs_secrets` row rotates. Same hash → the converged agent short-circuits → the fresh secret is never consumed → the box keeps presenting a revoked credential → **401 forever**. Proof in one line: the agent's `consumed-failed.json` carries hash `a4e5424…`, **identical** to the `marker.json` written 2 min before the re-issue. The comment at `hub/internal/web/pbsdr.go:320` asserts the reissue refreshes the descriptor "with the NEW token_id/fingerprint" — that assumption is simply false for this op. **A SECOND, independent defect was found while healing** and is fixed: `configs/felhom-pbs-apply`'s `reconcile` passed `--server` to `pvesm set`, which PVE rejects wholesale as a create-only parameter, so *every* re-apply exited 255 — and because the agent consumes the one-time secret BEFORE calling the wrapper, each re-issue **burned a credential**. Fixed in **agent v0.90.1** (one argv line + red-proof `TestReconcileNeverPassesServerToPvesmSet`); proven live (`pvesm set --server <same>` rejected, without it rc 0). **BOX HEALED 2026-07-18:** after the wrapper hotfix, Viktor's Re-issue click converged in 9 s and the tier went `401/inactive` → **`active`** (token probe 401 → 200) with a real backup landing PBS-side — `felhom-pbs:backup/ct/9201/2026-07-18T18:31:06Z`, 9 744 319 312 B, encrypted under the escrowed key. **REMAINING (fleet, needs its own spec — deliberately NOT improvised):** (a) make a fresh unconsumed secret actually un-converge the agent — either `consumed_at` becomes authoritative or the descriptor carries a secret generation/nonce so the hash moves; (b) fix the verify loop's read path — it reads `/etc/pve/priv/storage/<id>.pw` **directly as non-root**, a file it can only ever *write* through the root wrapper (`/etc/pve/priv` is `0700 root:www-data`; sudoers exposes `create|reconcile|grant` and **no read verb**), so the one loop that could catch this is permanently blind; (c) an auth probe in the reconciler/gauge/ceremony-precheck so `applied` can never mean `401`. — **Original finding note:** discovered by CC while collecting Phase-A evidence; not on the brief's finding list, so its P2-HIGH rank is provisional pending Viktor. The severity case: this is the DR tier, the failure is silent, and it would surface first at a real restore. Suggested shape: make `consumed_at` authoritative (a fresh unconsumed secret must un-converge the reconciler), fix the verify loop's read path (read via the same root wrapper that writes it), and make a failing `pvesm status` a LOUD state rather than a skipped datastore. **Blocks the DR-tier map row's candidate upgrade to PROVEN-LIVE — that upgrade is now explicitly WITHDRAWN.** Evidence `pbs-dr-state.txt`, `hub-state.txt` |
| R-30 | **[P2-HIGH] Liveness presence should come from the wait channel, not the report clock.** The box was powered off at the start of the rehearsal, yet the hub carried it as healthy until the staleness threshold expired ~30 min later (`host_stale` 16:05:24 "no report for 30m"; cleared 16:33:24 "was stale for 27m"). The host-delete guard compounds it: RESET refuses while any host row exists, so a stale-but-"Online" host stalls a forced teardown. | M | idea | Direction: derive presence from **Dir-2 long-poll connectedness (~90 s grace)**, decoupled from notification hysteresis (the hysteresis is right for *alerting*, wrong for *presence*); an agent/ep0 analog can follow. Pairs with R-13/R-23 — the transport already exists, this is about believing it. *(Discussed in-session as "R-29"; that number was already taken by the gate-rot item earlier the same day, so it is R-30.)* |
| R-31 | **[P2-HIGH] Offsite provisioning is synchronous with no status affordance.** Save runs the Hetzner sync in-request, so the request can hit the nginx 504 **while succeeding server-side**: the operator cannot tell failed from slow, and a retry races the first attempt. | M | idea | Direction: make it async + a status card, reusing the proven **awaiting-card/poll idiom** (v0.138.0 escrow card). **Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click.** |
| R-32 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. | M | idea | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` |
| R-33 | **[P2-HIGH] Bootstrap pairing-poll spams the customer-visible console.** Each 30 s pairing poll is a separate `felhom-bootstrap.service` run that exits 1, so systemd prints `Failed to start …` on the screen the customer is watching — **52 FAILED lines in ~11 minutes** while nothing was wrong (the box was correctly waiting to be bound). | S | idea | Fix in **scripts v1.21**: poll *inside* the script rather than via unit restarts, and reword the first-boot noise. Cheap, and it is the customer's literal first impression of the product. Evidence `bootstrap-journal.txt` |
| R-33 | **[P2-HIGH] Bootstrap pairing-poll spams the customer-visible console.** Each 30 s pairing poll is a separate `felhom-bootstrap.service` run that exits 1, so systemd prints `Failed to start …` on the screen the customer is watching — **52 FAILED lines in ~11 minutes** while nothing was wrong (the box was correctly waiting to be bound). | S | **SHIPPED (scripts v1.21.0, 2026-07-18)** | `run_pairing()` is now a `while true` loop sleeping `POLL_INTERVAL` (30s — the hub-side rate is unchanged) instead of exiting non-zero per poll, so the unit sits in `activating` and systemd prints nothing. Registration split into `register_appliance()`, whose transient failures the loop retries rather than taking the unit down. The journal is quiet without going dark: the wait is logged once on entry (worded so nobody reads it as an error), then on a 10-min heartbeat with elapsed minutes; `410` still exits non-zero on purpose (delivery consumed but no local env is a real crash window). Console banner every 5 min instead of every cycle, single accented spelling, plus the reassurance the rehearsal showed was missing („Ez a képernyő magától frissül"). **The load-bearing half is `TimeoutStartSec=infinity` in the unit** — a `Type=oneshot` ExecStart is killed at `DefaultTimeoutStartSec` (90s), so without it systemd would kill the new wait and `Restart=on-failure` would silently reinstate the exact spam this removes, *after appearing to work for the first three polls*. Verified behaviourally in a container against a stub hub answering 204 five times then delivering: **one log line plus one heartbeat, zero exits between polls**, then a clean fall-through to the direct install and `exit 0` — where the old design produced 5 unit invocations and 5 `Failed to start` console lines. ISOs rebuilt at v1.21.0 and the shipped payload verified byte-identical to the committed source. Evidence `bootstrap-journal.txt`; ship commit `bcdb042` |
| R-5 | Hub: offsite storage visibility — RESTIC box aggregate (v0.64.0) + PBS DR datastore (v0.65.0), each with fill + operator alert; Offsite tabs + dual dashboard gauges | M | **SHIPPED (hub v0.64.0 + v0.65.0 + tenantsync v1.2.0, 2026-07-17)** | Restic: `monitor.OffsiteBoxChecker` (throttled Hetzner GET; fill 80/90% + oversub 2.0×; scope `"pool-box"`). PBS DR (v0.65.0, Option A): read-only `usage` op on the ep0 `felhom-tenantsync` forced command (`df`, no mutation) → `monitor.PBSDRBoxChecker` (fill 80/90%, states ok/unavailable/degraded, scope `"pbsdr-box"`); `/offsite` Restic/PBS-DR tabs + two gauges; hub deploy ⟂ ep0 update (graceful "n/a"). **Both data pipelines PROVEN-LIVE** (restic `0.2% full`; PBS `19.1% full, 7.1 GB of 37.2 GB` after ep0 → v1.2.0). Thresholds pending Viktor (named keys; separate restic/PBS pairs). OPEN: UI render unit-only (password-gated), alert emails unit + red-proof only (both nominal). See hub v0.64.0 + v0.65.0 REPORTs |
| R-6 | **Spike: LAN service discovery from the guest** — SSDP multicast (UDP 1900, DLNA), WSD (Windows discovery), mDNS; host-network vs macvlan; is the customer LXC LAN-bridged in appliance deployments? | M | **spiked (2026-07-18)** | **VERDICT: appliance guest IS LAN-bridged (own DHCP lease on the household /24); multicast discovery works ONLY in the guest netns — guest-direct or Docker `--network host` (SSDP/mDNS/WSD all PASS both ways); the default docker bridge is categorically DEAF to LAN multicast (WSD/mDNS RX FAIL, unicast-publish PASS). Real samba+wsdd on host-net → Windows 11 ProbeMatch + FELHOM-SPIKE renders in Explorer + 445 + authenticated SMB round-trip all PASS; real SSDP `MediaServer:1` advert reaches both LAN clients. → R-7 SMB stack MUST be host-network LAN-bound; R-8 Jellyfin-DLNA plausible if host-network. Caveat: `vmbr0 multicast_snooping=1` worked only because the household router is a live querier — customer LANs w/ snooping+no-querier, and Peti's BYO bridge, are UNTESTED gaps.** **S4b (human leg, the sharpest finding): wsdd makes the box VISIBLE but the Explorer double-click FAILS `0x80070035` — WSD gives no name resolution; the flat `\\FELHOM-SPIKE` resolved by no path. Adding `nmbd` (NetBIOS) fixed it live (flat name resolves + mounts). → R-7 needs smbd+wsdd+nmbd (+avahi/.local for modern clients), not wsdd alone.** Doc: `audits/SPIKE-lan-discovery-2026-07-18.md`. |
| R-7 | SMB **server** share — LAN file sharing as an embedded controller feature („Megosztás") | L | **SHIPPED slice 1 (controller v0.144.0 + `felhom-samba:1.0.0`, 2026-07-18)** | Flips map row *Files from Windows Explorer / Mac Finder (SMB server)* MISSING → **IMPLEMENTED**. Delivered as the FOURTH protected infra stack (host-net, smbd+**nmbd**+wsdd, `bind interfaces only = lo eth0`), NOT a catalog app. „Megosztás" → „Hálózati megosztás": enable + ONE household password (STDIN→smbpasswd, never persisted — only `user_set`), shares table, create flow (new folder under `<storage>/shares/` or an existing folder via a guarded picker). Picker guard = absolute → EvalSymlinks → containment in a registered LIVE root → deny-listed system subtree (derived from `ProtectedHDDPaths`, provably a subset) → is-a-dir, with UNIFORM refusals (no filesystem oracle). Per-share read-only enforced in smb.conf AND as a `:ro` bind. `force user = felhom` → SMB writes land as uid 1000. Live-validated on demo through the real endpoints + a Win11 workstation (445 True, NetBIOS `FELHOM` resolves, write/read byte-compare PASS, **write to a read-only share REFUSED**, WSD ProbeMatch). **Explorer leg PASSED 2026-07-18 (Viktor):** both shares open from the Network view; an Explorer save into the writable share landed as uid 1000 and a write into the read-only share was refused, folder untouched. Slice 1 is fully PROVEN-LIVE. Docs: `controller/sharing.md`; ship report `felhom-controller/REPORT.md`. **Slice 2 remainder:** avahi/`.local` for modern+Apple clients, curated app-folder presets (one-click paperless `consume/`), per-share users + guest access, recycle bin. **Backup gap split out → R-7b.** |
@@ -87,7 +87,7 @@ Everything here is **remote-doable** — the N100 is packed, and none of it need
| Action | Owner | Note |
|---|---|---|
| **Rebuild the golden → 0.145.x** | Viktor + CC train | The rehearsal banked the floor-lift evidence (0.143.0→0.145.0 unattended in 5 s), so the mechanism is proven — but a fresh box should not *start* two versions stale and then self-update in front of the customer. Do this before anyone is invited |
| **Rebuild the golden → 0.146.0** (was 0.145.x) | Viktor + CC train | The rehearsal banked the floor-lift evidence (0.143.0→0.145.0 unattended in 5 s), so the mechanism is proven — but a fresh box should not *start* stale and then self-update in front of the customer. **Now more stale, not less:** controller **v0.146.0** shipped and is live on the demo box (2026-07-18), while the golden still bakes **0.143.0**. Procedure is recorded in `pilot/RUNBOOK-publish-0.85-0.120-2026-07-12` Phase C (snapshot-revert → cold boot → transient `golden-bake` unit → 404 pre-gate → upload → sha verify → teardown → token-leak grep), then STOP for the operator's manifest + floor saves (floor LAST). Deferred from the 2026-07-18 pre-travel train rather than half-run |
| **freemail.hu test-send** | Viktor | The open half of R-4; the gmail half closed on 2026-07-18 under `p=quarantine` |
| **C6 — customer performs a restore, unassisted** | Viktor as customer zero | The one open script step in R-3 and still MISSING as capability evidence. Remote-doable on the reborn box — the dashboard is remote |
| **R-11 rulings** | Viktor | Contact channel, tester agreement, alert thresholds (the R-5 gauge thresholds are still pending a ruling) |