diff --git a/CONTEXT.md b/CONTEXT.md index da3a348..809bd7b 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -561,7 +561,9 @@ the guest it backs up. Gaps minted the same session: **R-102 … R-108**. tester-agreement doc; per-customer hub disable remains possible.* Mechanics: installer ships `felhom-pbs-apply` + `age` + `wg_tunnel.enabled: true` (byo wg-off assert RETIRED — consent = typed disclosure ack + tester-agreement line), single VERSION source - gated by `scripts/hostinstall_gates.py` (cross-checks the hub Setup-tab const), default ACL + with a cross-check against the hub Setup-tab const written as `scripts/hostinstall_gates.py` — + **but NOT enforced: the script exists, is RED today (1.19.0 != 1.22.0) and is invoked by nothing** + (tracked as R-94 leg (b), class R-29) — default ACL keeps felhom-pbs (doc advice REVERSED — narrowing caused the drill 403). Hub: per-customer `dr_tier` flag (new default ON; legacy backfilled from descriptor reality, one-time), cascade-not-error semantics + per-stage status on the edit form, WG-registration diff --git a/REPORT-record-correction-2026-07-29.md b/REPORT-record-correction-2026-07-29.md new file mode 100644 index 0000000..060b2d3 --- /dev/null +++ b/REPORT-record-correction-2026-07-29.md @@ -0,0 +1,158 @@ +# REPORT — installer-channel record correction + R-29 filing (2026-07-29) + +Two commits, documentation only. No code, no version bump, no CHANGELOG entry, no build, no deploy, +no box touched. Written as `REPORT-.md` per `CLAUDE.md:82-87` so root `REPORT.md` (the E-2 +increment-1 report) is preserved. + +| # | Commit | Baseline | Scope | +|---|--------|----------|-------| +| 1 | `d4c07873ca0c3d3e547373a9fafc0e472a6535e8` | `36d635a4cdc1`, unmoved | Retract a false R-94/E-2d finding; open R-110 | +| 2 | (this commit) | `d4c07873`, unmoved | File R-29 to the register; three record defects; this report | + +--- + +## Commit 1 — `d4c07873` + +### What was false + +`36d635a4` recorded that `felhom-bootstrap.sh` fetches the installer **from the hub**, that the hub +therefore serves 1.19.0, and that a fresh ISO install runs the pre-E-2 installer. All three wrong. +The claim had propagated into two `OPEN-ITEMS.md` rows, the ranking rationale, and `ROADMAP.md:149`. + +### Confirmation table — all PASS + +| # | Claim | Read at | Result | +|---|-------|---------|--------| +| F1 | bootstrap fetches from the **website** | `scripts/iso/felhom-bootstrap.sh:96` | PASS — `INSTALL_URL="${FELHOM_INSTALL_URL:-https://felhom.eu/scripts/felhom-host-install.sh}"` | +| F2 | hub-rendered command points at the same URL | `customer_unified.html:563`, `:1262` | PASS — and **three** emission sites, not two: `:563` static, `:1262` JS error branch, **`:1267` JS success branch** | +| F3 | website serves `/scripts/` from a git-sync tree tracking `main` | `manifests/webpage.yaml` — nginx `:74-77`, sparse-checkout CM `:211-218`, git-sync `:272-281`, init `:299-307` | PASS — `--branch=main --period=30s --link=current`; sparse-checkout `/website/` + `/scripts/`; `location /scripts/ { root …/current; }`. No image build, no ArgoCD step | +| F4 | `hostInstallVersion` selects nothing | `configs.go:28`, `:487`; `render_test.go:219`; `customer_unified.html:494` | PASS — repo-wide grep returns exactly those 4 code sites; all other hits prose. Rendered as a text label | +| F5 | every generated flag is parsed by 1.22.0 | generator `customer_unified.html:1206-1239` vs parser `felhom-host-install.sh:1175-1212` | PASS — `--mode --cores --memory --vmid --node --acl-storages --operator-pubkey-file --preserve-state-from --skip-provision --dry-run --preflight-only --allow-new-leaf` (+ `--customer-id`); every one a parser case. **No functional gap** | +| F6 | installer is 1.22.0 | `scripts/felhom-host-install.sh:187` | PASS | +| F7 | the drift test is hollow | `render_test.go:219-221` | PASS — `strings.Contains(html, hostInstallVersion)` compares the constant to itself; passes at any value | + +### Live command 1 — what the URL actually serves + +``` +$ curl -fsS https://felhom.eu/scripts/felhom-host-install.sh | grep -m1 '^SCRIPT_VERSION=' +SCRIPT_VERSION="1.22.0" # the SINGLE version source (F-1): -h, the run banners, and the hub +``` + +### Live command 2 — the drift gate's real state + +``` +$ python3 scripts/hostinstall_gates.py; echo "exit=$?" + ok: SCRIPT_VERSION=1.22.0 + ok: header has no version literal +FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0 (F-1: bump both together) + ok: age is in the installed package set + … (six further ok lines) … +hostinstall gates: 1 FAILURE(S) +exit=1 +``` + +### Phase 0 source read — PAIRING reaches the same installer invocation + +Mode selection `felhom-bootstrap.sh:537-541`: a fresh VM with no baked customer-id calls +`run_pairing`. On HTTP 200 from `/api/v1/appliance/poll` the loop writes the hub-delivered +`FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it, and calls +`run_direct` **in the same invocation** (`:495-499`). `run_direct` is the single site that fetches +`$INSTALL_URL` (`:322-330`), builds the args (`:334`) and invokes `bash "$SCRIPT_TMP" "${args[@]}"` +(`:343`). The customer it yields is the one the operator bound — claimable. **So the ISO leg is the +spine for E-2d**, not an obstacle to it. + +### Rows changed + +| ID | Before | After | +|---|---|---| +| R-94 (line 14) | `READY — deferred until E-2d`, blocked on E-2d, false ISO/hub framing | `READY (XS)`, blocked on nothing, retracted + re-scoped to three legs | +| R-94 (line 16) | duplicate row, `READY #2`, 1.19.0 vs 1.20.0 | **deleted** — merged | +| R-110 | did not exist | opened, `WAITING-ON-OPERATOR (S)` | +| E-2d | ISO implied as obstacle | Next-action appended; ISO is the spine | +| ranked list | 1 R-95 · 2 R-94 (high-consequence) · 3 R-86 · 4 R-87 | 1 R-95 · 2 R-94 **de-ranked** · 3 R-86 · 4 R-87 · 5 R-110 | + +Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`, +`documentation/runbooks/day0-install.md`. + +--- + +## Commit 2 — R-29 filing + record hygiene + +### The §1.1 ruling — is R-29 the right home for a non-design-v2 gate? **Yes. Proceeded.** + +R-29's title says *"the design-v2 green gates"*, and `scripts/hostinstall_gates.py` is not one — it +comes from drill F-1 (2026-07-12) and postdates the item. Four things decide it anyway: + +1. **R-29's own audit list already spans well beyond design-v2 subject matter.** It names + `docker_run_volume_path_gate` (docker mount safety), `offbox_rename_gate`, `app_row_dedup_gate` + and `manifest_bearer_gate` (secrets — `runbooks/secrets.md:76`). The title is a misnomer relative + to the item's own body. +2. **Part (b) — "the systemic half is the real item" in R-29's words — is stated with no + design-v2 restriction.** It is about the *enforcement mechanism*: "the gates run only when a human + remembers to run them… decide where they run (pre-push hook, `build.sh` step, or a CI job) and make + a red gate block the train." That is gate-agnostic and repo-wide. +3. **`hub_confirm_gate.py` is already on R-29's list and lives in the same `scripts/` directory** as + `hostinstall_gates.py`. Wiring one and not the other would be arbitrary. +4. **Identical failure shape, identical genre.** Both self-describe as mechanical grep-assertions + (`hostinstall_gates.py:2`, `hub_confirm_gate.py:1-8`); both exist, assert something true, and are + invoked by nothing. + +R-29 has already absorbed one independent re-raise without minting an ID (2026-07-18 rehearsal note) +and says so explicitly. This is the third. No new ID minted. + +### Orphan-search evidence — `hostinstall_gates.py` and `hub_confirm_gate.py` + +Re-established at `d4c07873`. Pattern `hostinstall_gates\|hub_confirm_gate`. + +| # | Scope | Command | Result | +|---|-------|---------|--------| +| S1 | `felhom.eu`, all file types | `grep -rn "$PAT" . --exclude-dir=.git` | **19 hits, zero invocations.** All are docstrings (`scripts/hostinstall_gates.py:5`, `scripts/hub_confirm_gate.py:7`), code comments (`hub/internal/web/configs.go:27`, `scripts/felhom-host-install.sh:189`) or prose (`REUSE.md:62`, `CONTEXT.md:540,564`, `hub/CHANGELOG.md:371,1292,1351,1385`, `scripts/CHANGELOG.md:483,524`, 3 files under `documentation/audits/`, `ROADMAP.md:149,158`, `OPEN-ITEMS.md:14`) | +| S2 | sibling repos | `grep -rln "$PAT" /mnt/5_hdd/felhom.eu/git --exclude-dir=.git` minus this repo | 3 files, all in `.claude-memory/` (`MEMORY.md`, `drtier-by-default-2026-07-12.md`, `polish-batch-2026-07-13.md`). Notes, not invokers | +| S3 | `~/.claude` | `grep -rln "$PAT" /home/kisfenyo/.claude`, and targeted on `settings*.json`, `skills/`, `hooks/` | **Zero hits in settings, skills or hooks** — where an invoker would live. Remaining hits are `file-history/` (Claude Code's own backups of files edited in past sessions) and `paste-cache/` (pasted task specs). Neither is an invocation site | +| S4 | git hooks | `ls -1 .git/hooks/ \| grep -v '\.sample$'` | **Empty — every hook is a `.sample` stub** | +| S5 | build files | `find . -type f \( -iname Makefile -o -iname '*.mk' -o -iname justfile -o -iname 'Taskfile*' \)` | Only `hub/Makefile`; `grep -n gate hub/Makefile` → **zero occurrences** | +| S6 | CI | `find . -type d \( -name .github -o -name .gitea -o -name .woodpecker* -o -name .drone* -o -name .circleci \)` | **Empty — `felhom.eu` has no CI configuration at all** | + +Of the four gates in `scripts/`, only `site_gates.py` is mandated (`CLAUDE.md:153`); +`manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. + +### Rows changed + +| ID | Before | After | +|---|---|---| +| **R-29** | **absent from `OPEN-ITEMS.md`** (`grep -c` → 0) while present at `ROADMAP.md:158` since before the 2026-07-27 rebuild | **opened, `READY (S for (a) / M for (b))`**, owner CC | +| R-94 | leg (b) stood alone | leg (b) carries `→ R-29` as its class. Row otherwise untouched | +| E-2d | cited `:322-341` for an invocation at `:343` | cited `:322-343`, with the fetch / args / call each pinned separately | +| R-95, R-86, R-87 | `READY #1`, `READY #3`, `READY #4` | `READY` — markers dropped (see below) | + +Not added to the ranked list under *"Why the READY rows rank this way"*: that list is a top-N +rationale, not a complete ordering (R-99, R-102, R-103, R-104, R-108, R-109, R-89, R-92, R-93 and +E-2d are all READY and absent from it). Ranking R-29 was not asked for and would be a priority claim +this task has no basis to make. + +### §2.2 — markers dropped, not renumbered + +The `#1/#3/#4` markers duplicated ranked-list positions 1/3/4 for exactly those three rows; two +orderings of one set is what produced the hole when the `#2` row was merged in `d4c07873`. Removing +the duplicate leaves the ranked list as the single maintained ordering. + +### The other two record defects + +- `ROADMAP.md:147` — cited a non-existent **R-164**; it means controller **v0.164.0**'s + deliberate-stop filter. Corrected. (It had already cost one max-ID scan a false positive.) +- `CONTEXT.md:564` — asserted in the present tense that the single VERSION source is *"gated by + `scripts/hostinstall_gates.py`"*. The gate exists, asserts exactly that, is red, and runs nowhere. + Corrected to say the cross-check exists but is not enforced, tracked as R-94 leg (b) / R-29. + `hub/internal/web/configs.go:27` carries the same false claim in a code comment and was left alone + — R-94 leg (b) territory, needs a hub build. + +Files: `documentation/backlog/OPEN-ITEMS.md`, `documentation/backlog/ROADMAP.md`, +`documentation/backlog/README.md`, `CONTEXT.md`, this file. + +--- + +## Standing note + +**No gate was wired, fixed, run as a hook, or deleted by either commit.** Filing R-29 is not doing +R-29; the enforcement decision (pre-push hook / `build.sh` step / CI) and the audit of the remaining +eight gates are R-29 part (b), M-sized, and its own task. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 04724a9..4617e99 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -10,12 +10,13 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha |---|---|---|---|---|---| | **R-88a** | ~~Failing backup re-quiesces every 5 min, no backoff~~ | **SHIPPED** (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — | | **R-88b** | ~~`/backup/due` cannot say *unknown*~~ | **SHIPPED + PROVEN-LIVE** (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | `age_state=unknown` captured on real hardware during a deliberate ep0 outage; controller deferred, **zero app stacks stopped** | — | -| **E-2d** | **Prove E-2 on a fresh VM on the t740** — the only remaining route to four unproven items: a real `felhom-host-install.sh` **1.22.0** run (never done), Case B naturally (single-drive install renders the degraded banner without degrading a live box), a **claimable** customer so the three claim-gated items stop being gated, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **READY (M)** | — | **Space checked 2026-07-29 — NOT a blocker, with one constraint: the VM disk must NOT go on `local-lvm`.** That thin pool is over-subscribed (144 GB allocated against 54 GB, 38.8% used) on a box running a live customer guest, and a full thin pool corrupts every guest on it. `local` has only 23.7 GB and sits on `pve-root`. **Use `/mnt/nvme-1tb` (888 GB free).** Caveat found: a dir storage at a SUBDIRECTORY there will fail the agent's `exactMount` check and report `disconnected` in the host report — cosmetic, but decide the placement deliberately. **Do NOT unblock drill-r50** (deliberately blocked; unblocking it means the fixture stops representing anything real). **CORRECTED 2026-07-29 — the ISO is not an obstacle to the 1.22.0 proof, it is the best route to it.** `felhom-bootstrap.sh:96` fetches from `https://felhom.eu/scripts/felhom-host-install.sh`, not the hub, and that URL serves **1.22.0** (git-sync from `main`, ≤30 s — live fetch confirmed). A fresh ISO install therefore exercises 1.22.0 **automatically** — which makes the ISO leg the *stronger* proof, because it is the real customer path rather than a proxy for it, and it retires the "run 1.22.0 manually" step. **The Phase 0 question that decides the shape is ANSWERED — it was a source read, and the answer is yes.** A fresh VM with no baked customer-id lands in **PAIRING** mode (`felhom-bootstrap.sh:537-541`), not **DIRECT** (`:312`), and only DIRECT passes `--customer-id / --mode / --passphrase-file`. But on a 200 from `/api/v1/appliance/poll` the pairing loop writes the hub-delivered `FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it and calls `run_direct` **in the same invocation** (`:495-499`) — so pairing reaches the identical installer invocation (`:322-341`, the single `$INSTALL_URL` fetch + `--customer-id/--mode/--hub-url/--passphrase-file`) and the customer it yields is the one the operator bound, i.e. claimable. **So the ISO leg is the spine**; a manual 1.22.0 run is not needed as a separate scenario | CC | -| **R-94** | **A hand-synced version constant drifts, and the gate that would catch it is never run** — `hub/internal/web/configs.go:28` pins `hostInstallVersion = "1.19.0"` while `scripts/felhom-host-install.sh:187` is `SCRIPT_VERSION="1.22.0"` | **READY (XS)** | — | **CORRECTED 2026-07-29 — the earlier framing of this row was false and is retracted.** The constant selects no script: its only consumers are `configs.go:487` (`ScriptVersion`) and `render_test.go:219`, and it renders as a label at `customer_unified.html:494`. The install command beneath that label fetches `https://felhom.eu/scripts/felhom-host-install.sh` (`customer_unified.html:563`, `:1262`), which the website git-syncs from `main` on a 30 s period (`manifests/webpage.yaml`) — so **1.22.0 is what every install already gets** (live fetch, 2026-07-29). Every flag the generator emits is parsed by 1.22.0 (`customer_unified.html`~`:1210`–`:1238` vs `felhom-host-install.sh:1177`–`:1210`): **no functional gap, only a wrong number on the operator's screen.** Three legs, all XS: **(a)** derive the label from `SCRIPT_VERSION` rather than hand-syncing it, or delete it; **(b)** `scripts/hostinstall_gates.py` **fails today** and is invoked by no Makefile, hook or `CLAUDE.md` — wire it next to `site_gates.py` or delete it, because a gate nobody runs reads as coverage it is not providing; **(c)** `render_test.go:219` compares the constant to itself and passes at any value — replace it with the cross-file assertion. **No longer blocked on E-2d** — it never gated anything | CC | +| **E-2d** | **Prove E-2 on a fresh VM on the t740** — the only remaining route to four unproven items: a real `felhom-host-install.sh` **1.22.0** run (never done), Case B naturally (single-drive install renders the degraded banner without degrading a live box), a **claimable** customer so the three claim-gated items stop being gated, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **READY (M)** | — | **Space checked 2026-07-29 — NOT a blocker, with one constraint: the VM disk must NOT go on `local-lvm`.** That thin pool is over-subscribed (144 GB allocated against 54 GB, 38.8% used) on a box running a live customer guest, and a full thin pool corrupts every guest on it. `local` has only 23.7 GB and sits on `pve-root`. **Use `/mnt/nvme-1tb` (888 GB free).** Caveat found: a dir storage at a SUBDIRECTORY there will fail the agent's `exactMount` check and report `disconnected` in the host report — cosmetic, but decide the placement deliberately. **Do NOT unblock drill-r50** (deliberately blocked; unblocking it means the fixture stops representing anything real). **CORRECTED 2026-07-29 — the ISO is not an obstacle to the 1.22.0 proof, it is the best route to it.** `felhom-bootstrap.sh:96` fetches from `https://felhom.eu/scripts/felhom-host-install.sh`, not the hub, and that URL serves **1.22.0** (git-sync from `main`, ≤30 s — live fetch confirmed). A fresh ISO install therefore exercises 1.22.0 **automatically** — which makes the ISO leg the *stronger* proof, because it is the real customer path rather than a proxy for it, and it retires the "run 1.22.0 manually" step. **The Phase 0 question that decides the shape is ANSWERED — it was a source read, and the answer is yes.** A fresh VM with no baked customer-id lands in **PAIRING** mode (`felhom-bootstrap.sh:537-541`), not **DIRECT** (`:312`), and only DIRECT passes `--customer-id / --mode / --passphrase-file`. But on a 200 from `/api/v1/appliance/poll` the pairing loop writes the hub-delivered `FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it and calls `run_direct` **in the same invocation** (`:495-499`) — so pairing reaches the identical installer invocation (`:322-343` — the single `$INSTALL_URL` fetch at `:322-330`, the `--customer-id/--mode/--hub-url/--passphrase-file` args array at `:334`, and the `bash "$SCRIPT_TMP" "${args[@]}"` call itself at `:343`) and the customer it yields is the one the operator bound, i.e. claimable. **So the ISO leg is the spine**; a manual 1.22.0 run is not needed as a separate scenario | CC | +| **R-94** | **A hand-synced version constant drifts, and the gate that would catch it is never run** — `hub/internal/web/configs.go:28` pins `hostInstallVersion = "1.19.0"` while `scripts/felhom-host-install.sh:187` is `SCRIPT_VERSION="1.22.0"` | **READY (XS)** | — | **CORRECTED 2026-07-29 — the earlier framing of this row was false and is retracted.** The constant selects no script: its only consumers are `configs.go:487` (`ScriptVersion`) and `render_test.go:219`, and it renders as a label at `customer_unified.html:494`. The install command beneath that label fetches `https://felhom.eu/scripts/felhom-host-install.sh` (`customer_unified.html:563`, `:1262`), which the website git-syncs from `main` on a 30 s period (`manifests/webpage.yaml`) — so **1.22.0 is what every install already gets** (live fetch, 2026-07-29). Every flag the generator emits is parsed by 1.22.0 (`customer_unified.html`~`:1210`–`:1238` vs `felhom-host-install.sh:1177`–`:1210`): **no functional gap, only a wrong number on the operator's screen.** Three legs, all XS: **(a)** derive the label from `SCRIPT_VERSION` rather than hand-syncing it, or delete it; **(b)** `scripts/hostinstall_gates.py` **fails today** and is invoked by no Makefile, hook or `CLAUDE.md` — wire it next to `site_gates.py` or delete it, because a gate nobody runs reads as coverage it is not providing (**this leg is one instance of → R-29**, which is the class: gates are enforced nowhere, and the enforcement decision belongs there, not here); **(c)** `render_test.go:219` compares the constant to itself and passes at any value — replace it with the cross-file assertion. **No longer blocked on E-2d** — it never gated anything | CC | | **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now | CC | -| **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY #1** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC | -| **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY #3** | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after **its own** newest archive | CC | -| **R-87** | The restic tier is never restore-tested | **READY #4** | — | Design a controller-side test (no scratch-guest analogue transfers) | CC | +| **R-29** | **The green gates are not enforced anywhere — one was RED for 16 releases before anyone ran it.** This is the **class**, not an instance: a gate that exists, asserts something true, is red, and is invoked by nothing reads as coverage it is not providing. `controller/scripts/docker_run_volume_path_gate.py` failed continuously from **2026-07-14 (v0.129.0)** until R-7b's close-out ran it by hand at v0.145.0 — sixteen releases in which every REPORT said "green" | **READY (S for (a) / M for (b))** | — | **This item has existed at `ROADMAP.md:158` since before the register was rebuilt (2026-07-27) and was never carried across — that omission is itself part of the finding**, because it is an open item *about work not getting done* that then went missing from the page that decides what gets done. Two separable parts, per R-29's own analysis: **(a)** the `docker_run_volume_path_gate` finding is benign and the fix is a 3-line ALLOWLIST addition with its why — **not** a rewrite of the flagged call — and it gets its own reviewed diff, never bundled into a feature commit; **(b)** the systemic half, the real item: decide where gates run (pre-push hook, `build.sh` step, or CI) and make a red gate block the train the way the Go green gate does. **Two further orphans confirmed 2026-07-29** by repo-wide grep across all file types + sibling repos + `~/.claude` settings/skills/hooks + `.git/hooks` (none non-sample) + Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) + CI-directory find (**this repo has no CI at all**) — every one of the 19 hits is a docstring, a code comment or prose, and **not one is an invocation**: `scripts/hostinstall_gates.py` — **RED today** (`hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1), the same finding as **R-94 leg (b)** — and `scripts/hub_confirm_gate.py`. Of the four gates in `scripts/`, only `site_gates.py` is mandated anywhere (`CLAUDE.md:153`) and `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **In R-29's own words, carried forward deliberately: do not mint a new ID for a new instance** — the 2026-07-18 rehearsal independently re-raised this item and no second ID was minted then either | CC | +| **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC | +| **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY** | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after **its own** newest archive | CC | +| **R-87** | The restic tier is never restore-tested | **READY** | — | Design a controller-side test (no scratch-guest analogue transfers) | CC | | — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC | | — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator | | **R-90** | ep0 RAM headroom — 4 GiB swap **survived its first reboot** 2026-07-27; 3.8 GB RAM unchanged | **BLOCKED** (interim proven) | Hetzner CX33 availability — **confirmed unavailable even powered OFF**, so it is the Cost-Optimized "Limited availability", not the power state | Re-check CX33; escape hatch if urgent = CPX/CCX lines (no availability warning, higher cost) | operator | diff --git a/documentation/backlog/README.md b/documentation/backlog/README.md index 1c30deb..d2b614f 100644 --- a/documentation/backlog/README.md +++ b/documentation/backlog/README.md @@ -1,6 +1,12 @@ # documentation/backlog/ -Verified-LIVE findings with implementable fix plans that are **not yet implemented**. Preserved here +**`OPEN-ITEMS.md`** is the register of open work and the file to read first — it holds only what is +open, one row per item, every row with a state and an owner. **`ROADMAP.md`** is the full history and +reasoning behind the `R-n` IDs, including shipped and killed items; an ID is minted there, and a new +instance of an existing item attaches to that ID rather than getting its own. + +The rest of this folder: verified-LIVE findings with implementable fix plans that are **not yet +implemented**. Preserved here (instead of on git branches) per the trunk-based, no-branches rule — the fix itself is implemented later **directly on `main`**, during a normal/supervised session. diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 8d64be2..c50ea7c 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -144,7 +144,7 @@ Each attempt runs the **full quiesce cycle**, so every customer app stack is STO Self-resolves the moment the target answers (the storage read succeeds, sees the archive, tier stops being due) — which is why it can hide indefinitely: it needs an offsite outage to appear at all. **PHASE-0 ROOT CAUSE, established at source 2026-07-27 — it is AGENT-side, case (a).** The storage read **errored** (`could not read the backup storage for the due-check … err=…` at 09:02:57/09:07:58/09:12:57 CEST), so this was never an empty-success. The failure is a **type boundary**: `newestArchiveOn` (`localapi/server.go:1095-1111`) documents *"Errors and unsupported services degrade to unknown, never to 'no backup'"* — but its `(time.Time, bool)` signature **cannot represent unknown**, so an error and a genuinely-empty storage both collapse to `(zero, false)`, and `handleBackupDue` (`server.go:934-941`) then emits a POSITIVE claim: `Due: true, Reason: "no successful backup recorded yet", AgeSecs: nil`. The fail-safe that *does* exist — `targetStoragePresent`'s "a storage-view error must never be read as 'not there'" (`server.go:1131-1151`) — answers a different question (does the storage exist) and behaved correctly. **Decisive for scoping: the errored path and the genuine-never path are BYTE-IDENTICAL on the wire** — same `Due`, same `Reason` string, same nil `AgeSecs` — so the controller has nothing to discriminate on and Part 2 CANNOT be done controller-side. **Two further P0 findings:** the agent restarted **4× on 2026-07-27** (07:36:39, 07:54:06, 08:50:16, 11:31:52 CEST) — all deliberate (`NRestarts=0`, `Restart=on-failure`, `Result=success`), zero self-update — so the trigger is armed by ordinary operator/config work far more often than "only when ep0 is down"; and **the loop alerted NOBODY** — zero `backup_failed` events despite the hub allowlist carrying that type, because **`internal/quiesce` does not import `internal/notify` at all**. Its only trace was `07:13:27 info app_start_failed "Telepített alkalmazás nem fut: BookStack"` — a customer-tier, Hungarian, info-severity SYMPTOM of the third cycle catching BookStack mid-restart. **The whole-guest backup tier R-82 built has no failure signal to the hub → its own item.** **Shape:** distinguish *storage unreachable* from *storage readable and empty*. Unreachable is UNKNOWN — defer the due-verdict rather than resolving it either way, exactly as R-81 made the hub do with a missing report. Only a target that is reachable AND has no archive is genuinely due. **Fix the window bypass in the same slice:** `AgeSecs == nil` must stop meaning "run now regardless of the hour". Either the agent distinguishes *never backed up* from *cannot tell* in what it reports, or `scheduledRunAllowed` gates on the former only — otherwise any future nil-age path re-opens the same hole. Note this does NOT weaken R-84's fail-safe intent: a tier whose storage is merely slow or briefly unreadable should still err toward backing up — it is specifically the **cold-store + unreachable** pair that must defer, because there the fallback has no information at all, only an empty default that looks like a fact. | | R-89 | **Retention is a COMMERCIAL attribute — it belongs to the hub, not to ep0 or a box** | M | idea — operator ruling 2026-07-27, first increment SHIPPED same day | **Ruling (2026-07-27):** retention is a per-customer *commercial* attribute (a paid tier may buy longer retention), so the **hub owns the policy** and ep0 merely executes it. Execution stays **server-side**: a reconciler writes a **PBS prune job** and PBS's own scheduler runs it, so hub downtime leaves the last-known policy running rather than silently stopping retention. **Increment 1 SHIPPED 2026-07-27** (`runbooks/RUNBOOK-pbs-prune-serverside-2026-07-27.md`): boxes no longer attempt prune (`keep_last: 0` → `allowPBSPrune=false`, config only — no code, no grant), and per-namespace prune jobs run on ep0 daily 03:30 UTC (`keep-last 2`), dry-run gated and verified `TASK OK`. This also closed a live defect — **every** demo-hp PBS backup since the tier was created on 07-26 had reported `job errors` while the data landed correctly, because `DatastoreBackup` grants `Datastore.Backup` but not `Datastore.Prune`; a tier that cries wolf on every success makes a genuine failure invisible, which is exactly what happened during the 07-27 migration window. **Remaining work:** retention becomes a per-customer attribute on the hub — a **policy** (keep-last/daily/weekly/monthly), NOT a bare number, since a paid tier will want "12 weekly + 6 monthly"; a reconciler converges it into the ep0 prune job in the same descriptor-and-converge shape as `pbs_dr`. **SECURITY PROPERTY TO PRESERVE — do not "fix" a future prune error by widening the grant:** box tokens stay **write-only** (`DatastoreBackup`), never `DatastorePowerUser`/`DatastoreAdmin`. A compromised box must not be able to delete its own offsite backups — that is the scenario offsite DR exists to survive. **PARALLEL QUESTION, unanswered and arguably the more urgent half: does the restic key on `storage-box-pool-1` (`u629488`) have DELETE rights?** If so the daily app-data offsite tier carries the identical exposure, and restic's **append-only** mode is the equivalent answer. **Rule once for both tiers.** Flips the capability-map row for offsite retention (currently: retention configured but unenforceable) | -| R-97 | **The whole-guest backup tier has NO failure signal to the hub — `internal/quiesce` never notifies** | S | **SHIPPED (controller v0.177.0 + hub v0.78.0, 2026-07-27)** — **R-97a:** `quiesce.TierNotifier`, a seam (not an import) wired by an init-only setter, edge-triggered on the R-88 breaker ARMING so a failing tier is reported once per run rather than once per retry; recovery rides `recordSuccess`'s existing bool. **NEW operator-only event types** `whole_guest_backup_failed`/`_recovered` — deliberately NOT `backup_failed`, which carries a customer Hungarian template AND sits in demo-felhom's live `enabled_events`, so reusing it would have emailed the CUSTOMER about a backup they cannot act on while it was still retrying. The recovery joins `recoveredPairedDownTypes` because its `info` severity would otherwise be dropped by `severityNotifies` — the operator would hear it break and never hear it heal. **The hub's operator cooldown was keyed `customerID:eventType` alone**, so one tier would have masked the other for an hour; now narrowly extended with a `tier` suffix taken from the event details, leaving every other event type unchanged. **R-97b:** a suppression window keyed to the quiesce CYCLE (not a state test — v0.164.0's `!= StateStopped` filter cannot see an app caught MID-RESTART, which is exactly how BookStack alarmed), consumed at the same single derivation point `classifyRunStates`. Grace = **180 s**, derived from the deploy flow's 120 s health timeout and Mealie's 60 s `start_period`; it **expires**, so an app that genuinely fails to come back still alarms. **PROVEN LIVE end-to-end with a control:** the new type POSTs 200 from inside guest 9201 while a bogus type 400s, and `notification_log` shows **1 operator row, 0 customer rows**. The quiesce→notify link itself is unit-proven only. | On 2026-07-27 three whole-guest backups failed and three quiesce cycles stopped and restarted every customer app stack, and **not one `backup_failed` event reached the hub.** It is not the allowlist — the hub already carries `backup_failed` and `backup_completed` (they are emitted by the controller's *app-data* backup path). The cause is that **`internal/quiesce` does not import `internal/notify` at all**: the tier R-82 built has no route to the hub, so a whole-guest backup can fail indefinitely in silence. The loop's only trace was `app_start_failed` — **info** severity, **Hungarian**, on the **customer** channel — telling the customer BookStack was down (it had been caught mid-restart by the third cycle) without saying why, during an outage the system itself caused. So the one signal that did fire was both the wrong tier and the wrong story. **Shape:** emit `backup_failed`/`backup_completed` from `quiesceAndPollTiers` naming the TIER, operator-tier; and decide whether a quiesce-induced restart should suppress `app_start_failed` the way R-164's deliberate-stop filter does — an app the backup stopped on purpose is not a fault. R-88's breaker bounds the repetition but changes nothing about the silence | +| R-97 | **The whole-guest backup tier has NO failure signal to the hub — `internal/quiesce` never notifies** | S | **SHIPPED (controller v0.177.0 + hub v0.78.0, 2026-07-27)** — **R-97a:** `quiesce.TierNotifier`, a seam (not an import) wired by an init-only setter, edge-triggered on the R-88 breaker ARMING so a failing tier is reported once per run rather than once per retry; recovery rides `recordSuccess`'s existing bool. **NEW operator-only event types** `whole_guest_backup_failed`/`_recovered` — deliberately NOT `backup_failed`, which carries a customer Hungarian template AND sits in demo-felhom's live `enabled_events`, so reusing it would have emailed the CUSTOMER about a backup they cannot act on while it was still retrying. The recovery joins `recoveredPairedDownTypes` because its `info` severity would otherwise be dropped by `severityNotifies` — the operator would hear it break and never hear it heal. **The hub's operator cooldown was keyed `customerID:eventType` alone**, so one tier would have masked the other for an hour; now narrowly extended with a `tier` suffix taken from the event details, leaving every other event type unchanged. **R-97b:** a suppression window keyed to the quiesce CYCLE (not a state test — v0.164.0's `!= StateStopped` filter cannot see an app caught MID-RESTART, which is exactly how BookStack alarmed), consumed at the same single derivation point `classifyRunStates`. Grace = **180 s**, derived from the deploy flow's 120 s health timeout and Mealie's 60 s `start_period`; it **expires**, so an app that genuinely fails to come back still alarms. **PROVEN LIVE end-to-end with a control:** the new type POSTs 200 from inside guest 9201 while a bogus type 400s, and `notification_log` shows **1 operator row, 0 customer rows**. The quiesce→notify link itself is unit-proven only. | On 2026-07-27 three whole-guest backups failed and three quiesce cycles stopped and restarted every customer app stack, and **not one `backup_failed` event reached the hub.** It is not the allowlist — the hub already carries `backup_failed` and `backup_completed` (they are emitted by the controller's *app-data* backup path). The cause is that **`internal/quiesce` does not import `internal/notify` at all**: the tier R-82 built has no route to the hub, so a whole-guest backup can fail indefinitely in silence. The loop's only trace was `app_start_failed` — **info** severity, **Hungarian**, on the **customer** channel — telling the customer BookStack was down (it had been caught mid-restart by the third cycle) without saying why, during an outage the system itself caused. So the one signal that did fire was both the wrong tier and the wrong story. **Shape:** emit `backup_failed`/`backup_completed` from `quiesceAndPollTiers` naming the TIER, operator-tier; and decide whether a quiesce-induced restart should suppress `app_start_failed` the way controller **v0.164.0**'s deliberate-stop filter does — an app the backup stopped on purpose is not a fault. R-88's breaker bounds the repetition but changes nothing about the silence | | R-95 | **The restic offsite tier's credential CAN DELETE — R-89's "parallel question", now ANSWERED** | M | idea — established read-only 2026-07-27 | **The exposure closed on the weekly PBS tier is fully open on the daily restic tier**, which holds the customer's actual documents and photos and is the only tier that survives losing the box. Established without mutating anything: **(1) Identity** — a per-customer *subaccount* on `storage-box-pool-1` (box 611714, bx11, `u629488`): `u629488-sub1` home `felhom-demo-felhom`, `sub2` peti-felhom, `sub3` demo-hp, each labelled `felhom-customer`. Auth is an **SSH key stored ON THE BOX** (`…/felhom-controller-data/_data/data/offbox/ssh_key`, 0600, beside `repo_password` + a pinned `known_hosts`) — customer-side, not hub-side, so a compromised guest holds it. **(2) Read-write: YES** — the API reports **`readonly=False` on all three subaccounts**, and it is not merely latent: the controller runs `restic forget --group-by host,tags --keep-daily 7 --keep-weekly … --prune` **from the box** (`backup/offbox.go:984`, also `:1070`). Delete rights are exercised on every run. **(3) Append-only: NO, and not expressible** — the repo is built as `sftp:` (`offbox.go:482`); restic's append-only mode requires the **REST server** backend, which plain SFTP cannot provide. **(4) A zero-code mitigation exists and is unused:** the box type carries `snapshot_limit=10` and the API reports `snapshot_plan=null` with **0 snapshots** and `size_snapshots=0`. Hetzner Storage Box snapshots are taken **server-side, outside the SFTP namespace** — an SFTP subaccount cannot delete them — so they are a genuine immutability layer at no extra cost and with no code change. **Rule once for both tiers, per R-89.** Options, cheapest first: enable a snapshot plan (operator click, immediate); split backup-write from prune so pruning runs somewhere the box cannot reach; or move the repo to restic's REST server with `--append-only`. Flips the capability-map row for offsite immutability | | R-94 | **A hand-synced version constant drifts, and the gate that would catch it is never run** | XS | idea — found 2026-07-27, **corrected 2026-07-29** | **RETRACTION.** The prior detail's conclusion — *"a new install driven from the hub still gets the pre-R-82 default"* — **was false and is retracted.** It assumed `hostInstallVersion` selects a script. It selects nothing: its only two consumers are `configs.go:487` (`ScriptVersion`) and `render_test.go:219`, and it is rendered as a **text label** at `customer_unified.html:494` ("Day-0 host bootstrap for host-install {{.ScriptVersion}}"). The install command printed **beneath that label** fetches `https://felhom.eu/scripts/felhom-host-install.sh` (static block `customer_unified.html:563`; JS generator `:1262`) — as does the ISO's `felhom-bootstrap.sh:96`, which fetches from the **website, not the hub**. That URL is the website's `git-sync` working tree tracking `--branch=main --period=30s` with `/scripts/` in the sparse-checkout, served by nginx's `location /scripts/` (`manifests/webpage.yaml`) — no image build, no ArgoCD step. **So 1.22.0 is what every install already gets**, confirmed by live fetch 2026-07-29 (`curl https://felhom.eu/scripts/felhom-host-install.sh` → `SCRIPT_VERSION="1.22.0"`). Every flag the generator emits (`customer_unified.html`~`:1210`–`:1238`) is parsed by 1.22.0 (`felhom-host-install.sh:1177`–`:1210`): **no functional gap, only a wrong number on the operator's screen.** **What survives from the original entry is its diagnosis, which was right all along: a hand-synced version constant in a second repo drifts every time the first one ships.** `configs.go:28` is still `"1.19.0"` against `felhom-host-install.sh:187`'s `SCRIPT_VERSION="1.22.0"`. Three legs, all XS: **(a)** derive the label from `SCRIPT_VERSION` rather than hand-syncing it, or delete the label; **(b)** `scripts/hostinstall_gates.py` exists precisely to catch this and **fails today** (`FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1) — but it is invoked by no Makefile, git hook, CI config or `CLAUDE.md` (only `site_gates.py` is mandated, `CLAUDE.md:153`), so wire it or delete it: a gate nobody runs reads as coverage it is not providing; **(c)** `render_test.go:219` asserts `strings.Contains(html, hostInstallVersion)` — it compares the constant to itself, passes at any value, and cannot detect drift; replace it with the cross-file assertion. **v0.73.2 is NOT an undeployed artifact** — its content is inside the live hub. See also **R-110**, found by the same investigation: the fetch URL has no staging | | R-110 | **`main` is the installer's publish channel — there is no staging** | S | idea — found 2026-07-29, **WAITING-ON-OPERATOR (a ruling, not a defect)** | `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a `--period=30s`, and nginx serves that working tree directly (`location /scripts/`, `root /usr/share/nginx/html/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`, `:1262`; `runbooks/day0-install.md` C.1) receives. **There is no tag, no pinned-version path, no staging copy and no rollback other than another push** — for the one artifact that runs as **root on a virgin box**, the most privileged thing Felhom ships. **Two consequences worth stating plainly:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29, so the proof run confirms what customers already receive rather than clearing it for release; and the precaution the old R-94 row recorded ("do not point every new box at an installer that has never run") **was never available to take**, because nothing points boxes at a version. **Open question for the operator, not a defect to fix blind:** should `/scripts/` serve a pinned release — a tag-tracked git-sync ref, or a versioned directory (`/scripts/1.22.0/…`) with the hub's generated command naming a version — or is `main`-tracking the accepted shape for a one-operator product where the alternative is a release ritual nobody performs? **Exposure today is zero** (no boxes are installing), which is exactly why it is cheap to decide now. Whichever way it goes, it decides whether R-94 leg (a) makes the label a *fact* (derived from the served script) or keeps it a *claim*. Flips no capability-map row — the map states what the platform does, and this changes nothing about that | @@ -155,7 +155,7 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the | R-96 | **Two standing rules were agreed in chat and never committed** | XS | idea — found 2026-07-27 | Verified by grep across every `CLAUDE.md`, `CONTEXT.md`, `REUSE.md` and `skills/`: **(a) "never combine a test run and a commit in one command"** — zero hits, despite three separate occurrences (the worst pushed a red suite because `packages ok: 28` was read while `rc=1` was not); **(b) "a 'no access' claim must list what was tried"** — zero hits, despite a wrong "no access exists" verdict on ep0 on 2026-07-27 where the working route was documented in memory and only one path had been attempted. **The third rule IS committed** — `CONTEXT.md:8` S-1, N.5's third leg (architecture docs same-session coupled). A rule that lives only in a chat log binds nobody; both belong in `CONTEXT.md` beside S-1/S-2 | | R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**) -| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** | +| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** **2026-07-29 — audit list extended, and a THIRD independent re-raise absorbed under the same rule (again no new ID):** add `scripts/hostinstall_gates.py`, which **postdates this item** (it comes from drill F-1, 2026-07-12) and is therefore not a design-v2 gate — but it is the identical failure shape and is tracked as **R-94 leg (b)**. It is **RED as of 2026-07-29**: `hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1, with its nine other assertions green. `scripts/hub_confirm_gate.py`, already on the list above, was **verified orphan on the same date**. Both confirmed by repo-wide grep across all file types plus sibling repos, `~/.claude` settings/skills/hooks, `.git/hooks` (no non-sample hooks exist), a Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) and a CI-directory find (**`felhom.eu` has no CI configuration at all**) — all 19 hits are docstrings, code comments or prose; **zero are invocations.** Only `site_gates.py` is mandated (`CLAUDE.md:153`); `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **Now also filed in `OPEN-ITEMS.md`** — this item predates the 2026-07-27 register rebuild and was never carried across, so an open item about work not getting done was itself missing from the page that decides what gets done | | R-37 | **Post-RESET health card shows stale pre-RESET warnings.** After a RESET the card should read **„RESET óta nincs adat"** instead of carrying warnings about a lifecycle that no longer exists. | XS | **SHIPPED (hub v0.67.0, 2026-07-18)** | The customer page raises a banner when a RESET **completed** after the newest report, quoting „RESET óta nincs adat" and the reset timestamp, because until the box reports again every health figure describes a lifecycle that no longer exists. Deliberately narrow: an **in-flight** reset does not trigger it (only a completed one), and it **clears itself** on the first post-RESET report. Ties resolve to STALE — SQLite timestamps are second-resolution and a same-second report almost certainly arrived just before the reset destroyed what it describes; erring the other way would hide the banner exactly when it matters most. Red-proofed (neutering the predicate fails the assertion). — Origin: 2026-07-18 rehearsal. Same family as R-36 — the hub knows the state changed and the UI has not caught up | | R-38 | **Installer GRUB slice.** A single default „Felhom telepítés" entry; the **interactive installers REMOVED** (safety: an interactive entry is how a wrong-disk manual install happens); felhom background. | S | **SHIPPED (scripts v1.22.0, 2026-07-19)** | Origin: 2026-07-18 rehearsal, alongside R-21's physical closure. Every ISO is repacked after `prepare-iso`: exactly **one** entry („Felhom telepítés", default, 5 s), with Graphical / Terminal UI / serial and the whole **Advanced Options** submenu (nomodeset ×2, three debug variants, Rescue Boot, memtest, UEFI settings) **not emitted** — not hidden, not password-gated. Boot behavior unchanged: the `linux`/`initrd` lines are lifted **verbatim at repack time** from the ISO's own automated entry, so a PVE bump tracks automatically; the build fails if they are missing, if the append line has lost `proxmox-start-auto-installer`, or if `auto-installer-mode.toml` is absent. Gated for 1 entry / 0 submenus / no live banned references, then **re-verified by reading the menu back out of the finished ISO**. Background generated at repack time from `website/assets/og-image_2.png` (one source, no second copy to drift), with the card's own grid phase-locked across the letterbox fill. Live-validated on the nested canary (UEFI/OVMF): renders, auto-fires, reaches the same match-nothing abort as v1.21.0. **Squashfs/theme rebranding still DEFERRED** — post-GRUB screens remain Proxmox-branded. GRUB's `e`/`c` keys also remain reachable (suppressing them needs a superuser password — a separate decision); the menu no longer advertises them |