From e9a74a0019fdde2d0a1696ab1147fbfc66e2920e Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 31 Jul 2026 21:31:54 +0200 Subject: [PATCH] docs: remove a gate criterion that could never pass, and close three register rows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PART 1 — the release gate. G7 required the packaged .deb to sha256-match the one built from committed source. That is unsatisfiable BY CONSTRUCTION: dpkg-deb stamps the build time into every archive, so two builds of byte-identical source differ. It was already failing when the 1.26.1 release ran it. A criterion nobody can satisfy gets waived once and read as advisory ever after — which is how R-29's shelf of never-run gates was built. Sub-clause dropped, reason recorded in G7's own note the way G6's amendment was, so a future reader can restore it if SOURCE_DATE_EPOCH ever makes it meaningful. RULING ASKED FOR — is payload integrity covered by G9 alone? NO, and G9 is widened rather than a new criterion invented. The package ships TWO payload files (build-deb.sh:54-55); G9 checked only the script. The systemd UNIT was covered by nothing: G7 covered the container, G8 covers the postinst behaviourally, G13 covers directory presence. The unit is not incidental — its After=, its ConditionPathExists= and its Restart= decide WHEN AND WHETHER day-0 runs at all, so a drifted unit would have shipped silently. Same shape as the /etc/felhom miss that G13 exists to prevent: a check that proved the thing present and said nothing about what it depended on. The check passes today. G13 moved to sit after G12 — it was minted late and left between G10 and G11. PART 2 — register dispositions. BASELINE DISCREPANCY, reported rather than worked around: only R-128 had a row. R-154 and R-155 had NO row in either file — minted in a spike document and never carried across, which is R-123's class, not the drift the task described. Rows created, closed, with the reasoning, because in all three cases the reasoning is the durable part: R-128 closed by CORRECTING a false claim, not by making the assertion real — the coupling does not exist and asserting it would invent a constraint. Flagged so nobody 'restores' it. R-154 closed with the measurement and where it now lives in pushed source. R-155 NARROWED, not deleted — unchanged for FELHOM_MENU=single, inapplicable to release. Flagged so the guard is not later removed wholesale on the strength of 'R-155 closed it'. Documentation only: no code, no build, no ISO, no upload, no box touched. --- documentation/backlog/OPEN-ITEMS.md | 4 +- documentation/backlog/ROADMAP.md | 3 + documentation/runbooks/iso-release-gate.md | 71 ++++++++++++++++------ 3 files changed, 59 insertions(+), 19 deletions(-) diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index b39273b..755f5b9 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -70,7 +70,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC | | **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC | | **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC | -| **R-128** | `build-felhom-iso.sh:44` comments that `ISO_VERSION` "aligns with felhom-host-install SCRIPT_VERSION" — ISO is **1.25.0**, host-install is **1.22.0** | READY (XS) | — | A comment asserting an invariant nothing maintains, in a file whose whole job is to say what an ISO implies. Fix = drop the claim, or make the build read `SCRIPT_VERSION` and refuse a mismatch. Found during the 2026-07-31 tester-gate run (`audits/tester-gate-golden-0.188.0-2026-07-31.md` §2) | CC | +| **R-128** | ~~`build-felhom-iso.sh:44` comments that `ISO_VERSION` "aligns with felhom-host-install SCRIPT_VERSION" — a claim nothing evaluated~~ | **CLOSED** (iso v1.26.0, 2026-07-31) | — | **Closed by correcting the claim, NOT by making the assertion real — and that distinction is the durable part.** The coupling it asserted does not exist: the ISO is a frozen artifact, while `felhom-host-install.sh` is fetched at RUN TIME from the website's git-sync of `main` (→ **R-94**, **R-110**), so whatever version an ISO carries, the script a box runs is always current. Making the build read `SCRIPT_VERSION` and refuse a mismatch — the fix the old row proposed — would have **invented a constraint** and coupled two things that are deliberately decoupled. `build-felhom-iso.sh:45-52` now states the independence in place of the false claim. **Do not 'restore' the assertion on the strength of seeing a disabled one** | — | +| **R-154** | ~~`[first-boot]` is automated-install-only and nothing in the Felhom tree said so~~ | **CLOSED** (iso v1.26.0, 2026-07-31) | — | A property of the PVE installer, not a Felhom defect, but one that silently delivered nothing on the path we now ship. Measured with a same-image control in `audits/SPIKE-universal-iso-3-2026-07-31.md` §2: on an interactive install the hook never runs and the `proxmox-first-boot` **package is not even installed** (`Config.pm:118` defaults `first_boot.enabled=0`, `set_first_boot_opt` is never called, `Install.pm:746` returns early, `:1360` skips the package, and `proxinstall` contains **zero** occurrences of `first-boot`). Now recorded in pushed source at `scripts/iso/pkg/build-deb.sh:6-11` — the header of the mechanism that replaced it — and cited in `runbooks/iso-release-gate.md` G7 | — | +| **R-155** | ~~`iso-repack.sh` refuses any ISO without `auto-installer-mode.toml`, blocking the no-`answer.toml` posture~~ | **CLOSED** (iso v1.26.0, 2026-07-31) | — | **NARROWED, not deleted — and the difference matters.** The guard protects a real promise: in single-entry appliance mode the menu shows one button labelled "Felhom telepítés" that boots the AUTOMATED installer, and without that file the same button would drop the user into the manual disk-picker. That promise is unchanged and the guard still enforces it for `FELHOM_MENU=single` (`iso-repack.sh:121-128`). It simply does not apply to `release`, where the file's absence **is** release-gate criterion **G1**. **Do not remove the guard wholesale on the strength of "R-155 closed it"** — deleting it would put an unattended installer behind a button promising one | — | | **R-129** | **Every doc says demo-hp has "no baked SSH key"** and needs the G1 break-glass password — but `ssh -o BatchMode=yes demo-hp` authenticated **by key**, first try, 2026-07-31 | READY (XS) | — | Stale in the expensive direction: a session that believes it sends itself to the hub vault for a credential it does not need. Verify who owns the key and when it landed, then correct `CLAUDE.md`, `runbooks/target-selection.md:41-42`, `runbooks/workspace-CLAUDE.md` and `felhom-agent/CLAUDE.md` together — or remove the key if it was not deliberate | CC | | **R-130** | **A "hard min" that only warns.** A fresh box's `local-lvm` was ~75 GiB against `HARD_MIN_LVM_GIB=120` (`scripts/felhom-host-install.sh`); the installer logged `[WARN] local-lvm free ~75 GiB < hard min 120 GiB` and went on to a **fully successful** install | READY (S) | — | Either the minimum is not hard (rename it and state the real floor) or it is wrong (and 120 GiB is not what a working appliance needs). Leaving it is the R-29 shape: a check that reads as coverage while providing none. Evidence: same audit §8 | CC | | **R-131** | **`sess-f` is a fourth orphaned scratch customer** on the hub ("R-120 golden 0.186.0 proof", DOWN), left by the 2026-07-30 session | READY (XS) | — | After `drill-r50`, `sess-c`, `sess-d` — the accumulation `runbooks/target-selection.md:86-87` and `PROMPT-TEMPLATE.md` §13 both warn about, now on its fourth instance. Delete it (see the recorded command in `audits/tester-gate-golden-0.188.0-2026-07-31.md` §7.1); the recurrence itself argues for a periodic scratch-customer sweep rather than another reminder | CC | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 18c8e60..cee5587 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -158,6 +158,9 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the | R-95 | **The restic offsite tier's credential CAN DELETE — R-89's "parallel question", now ANSWERED** | M | idea — established read-only 2026-07-27 | **The exposure closed on the weekly PBS tier is fully open on the daily restic tier**, which holds the customer's actual documents and photos and is the only tier that survives losing the box. Established without mutating anything: **(1) Identity** — a per-customer *subaccount* on `storage-box-pool-1` (box 611714, bx11, `u629488`): `u629488-sub1` home `felhom-demo-felhom`, `sub2` peti-felhom, `sub3` demo-hp, each labelled `felhom-customer`. Auth is an **SSH key stored ON THE BOX** (`…/felhom-controller-data/_data/data/offbox/ssh_key`, 0600, beside `repo_password` + a pinned `known_hosts`) — customer-side, not hub-side, so a compromised guest holds it. **(2) Read-write: YES** — the API reports **`readonly=False` on all three subaccounts**, and it is not merely latent: the controller runs `restic forget --group-by host,tags --keep-daily 7 --keep-weekly … --prune` **from the box** (`backup/offbox.go:984`, also `:1070`). Delete rights are exercised on every run. **(3) Append-only: NO, and not expressible** — the repo is built as `sftp:` (`offbox.go:482`); restic's append-only mode requires the **REST server** backend, which plain SFTP cannot provide. **(4) A zero-code mitigation exists and is unused:** the box type carries `snapshot_limit=10` and the API reports `snapshot_plan=null` with **0 snapshots** and `size_snapshots=0`. Hetzner Storage Box snapshots are taken **server-side, outside the SFTP namespace** — an SFTP subaccount cannot delete them — so they are a genuine immutability layer at no extra cost and with no code change. **Rule once for both tiers, per R-89.** Options, cheapest first: enable a snapshot plan (operator click, immediate); split backup-write from prune so pruning runs somewhere the box cannot reach; or move the repo to restic's REST server with `--append-only`. Flips the capability-map row for offsite immutability | | R-94 | **A hand-synced version constant drifts, and the gate that would catch it is never run** | XS | idea — found 2026-07-27, **corrected 2026-07-29** | **RETRACTION.** The prior detail's conclusion — *"a new install driven from the hub still gets the pre-R-82 default"* — **was false and is retracted.** It assumed `hostInstallVersion` selects a script. It selects nothing: its only two consumers are `configs.go:487` (`ScriptVersion`) and `render_test.go:219`, and it is rendered as a **text label** at `customer_unified.html:494` ("Day-0 host bootstrap for host-install {{.ScriptVersion}}"). The install command printed **beneath that label** fetches `https://felhom.eu/scripts/felhom-host-install.sh` (static block `customer_unified.html:563`; JS generator `:1262`) — as does the ISO's `felhom-bootstrap.sh:96`, which fetches from the **website, not the hub**. That URL is the website's `git-sync` working tree tracking `--branch=main --period=30s` with `/scripts/` in the sparse-checkout, served by nginx's `location /scripts/` (`manifests/webpage.yaml`) — no image build, no ArgoCD step. **So 1.22.0 is what every install already gets**, confirmed by live fetch 2026-07-29 (`curl https://felhom.eu/scripts/felhom-host-install.sh` → `SCRIPT_VERSION="1.22.0"`). Every flag the generator emits (`customer_unified.html`~`:1210`–`:1238`) is parsed by 1.22.0 (`felhom-host-install.sh:1177`–`:1210`): **no functional gap, only a wrong number on the operator's screen.** **What survives from the original entry is its diagnosis, which was right all along: a hand-synced version constant in a second repo drifts every time the first one ships.** `configs.go:28` is still `"1.19.0"` against `felhom-host-install.sh:187`'s `SCRIPT_VERSION="1.22.0"`. Three legs, all XS: **(a)** derive the label from `SCRIPT_VERSION` rather than hand-syncing it, or delete the label; **(b)** `scripts/hostinstall_gates.py` exists precisely to catch this and **fails today** (`FAIL: hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1) — but it is invoked by no Makefile, git hook, CI config or `CLAUDE.md` (only `site_gates.py` is mandated, `CLAUDE.md:153`), so wire it or delete it: a gate nobody runs reads as coverage it is not providing; **(c)** `render_test.go:219` asserts `strings.Contains(html, hostInstallVersion)` — it compares the constant to itself, passes at any value, and cannot detect drift; replace it with the cross-file assertion. **v0.73.2 is NOT an undeployed artifact** — its content is inside the live hub. See also **R-110**, found by the same investigation: the fetch URL has no staging | | R-110 | **`main` is the installer's publish channel — there is no staging** | S | idea — found 2026-07-29, **WAITING-ON-OPERATOR (a ruling, not a defect)** | `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a `--period=30s`, and nginx serves that working tree directly (`location /scripts/`, `root /usr/share/nginx/html/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`, `:1262`; `runbooks/day0-install.md` C.1) receives. **There is no tag, no pinned-version path, no staging copy and no rollback other than another push** — for the one artifact that runs as **root on a virgin box**, the most privileged thing Felhom ships. **Two consequences worth stating plainly:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29, so the proof run confirms what customers already receive rather than clearing it for release; and the precaution the old R-94 row recorded ("do not point every new box at an installer that has never run") **was never available to take**, because nothing points boxes at a version. **Open question for the operator, not a defect to fix blind:** should `/scripts/` serve a pinned release — a tag-tracked git-sync ref, or a versioned directory (`/scripts/1.22.0/…`) with the hub's generated command naming a version — or is `main`-tracking the accepted shape for a one-operator product where the alternative is a release ritual nobody performs? **Exposure today is zero** (no boxes are installing), which is exactly why it is cheap to decide now. Whichever way it goes, it decides whether R-94 leg (a) makes the label a *fact* (derived from the served script) or keeps it a *claim*. Flips no capability-map row — the map states what the platform does, and this changes nothing about that | +| R-128 | **`ISO_VERSION` "aligns with SCRIPT_VERSION" was a comment nothing evaluated** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | Closed by **correcting the claim, not asserting it**: the ISO is frozen while `felhom-host-install.sh` is fetched at run time from `main` (R-94/R-110), so an assertion would invent a constraint. `build-felhom-iso.sh:45-52`. Full reasoning in `OPEN-ITEMS.md` | +| R-154 | **`[first-boot]` is automated-install-only, and nothing in the tree said so** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | A PVE property, measured with a same-image control (`audits/SPIKE-universal-iso-3-2026-07-31.md` §2); recorded at `scripts/iso/pkg/build-deb.sh:6-11`. Superseded in practice by the `.deb` delivery route | +| R-155 | **`iso-repack.sh` refused any ISO without `auto-installer-mode.toml`** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | **Narrowed, not deleted** — unchanged for `FELHOM_MENU=single` (`iso-repack.sh:121-128`), does not apply to `release` where the file's absence *is* gate G1. Do not remove it wholesale | | R-90 | **ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86** | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: `Swap: 0B`, nothing in `/etc/fstab`, no swap unit, no `/swapfile` — **swap was never added.** On 2026-07-27 06:58:12 a global OOM fired while `proxmox-backup-proxy` held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration `rsync` held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed **R-88** downstream. Still exposed today: 3.8 GB, no swap, one kill on record. **This gates R-86** — backup-aligned restore-testing puts a multi-GB read on this box *on a schedule*, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console | | R-91 | **The old 13 GB datastore copy is still on ep0's root disk** | XS | WATCHING — gated on demo-felhom's first post-migration PBS backup | The datastore moved to a Hetzner Cloud Volume on 2026-07-27 (`/dev/sdb`, 100 GiB, attached 06:29:40 UTC, now `/mnt/pbs-datastore`, 13 G used of 98 G). The pre-migration copy survives at **`/srv/pbs-felhom`, 13 G**, on `/` (38 G total, 16 G used, 21 G free). **Do not delete yet:** demo-hp has landed two post-migration snapshots (07-27 08:25:47Z, 09:37:29Z) but **demo-felhom's newest is 2026-07-26T12:21:48Z — before the migration**, so the new volume has not yet proven a write for that namespace. Delete once it has. **Doc drift to fix in the same commit:** `CONTEXT.md:1018` still records the datastore at `/srv/pbs-felhom` | | R-92 | **The hub's PBS-DR gauge is 0.1 GB-granular, so small deltas are unverifiable** | XS | idea — 2026-07-27 | The PBS-DR box card rounds to 0.1 GB, which is coarser than the changes an operator wants to confirm after a prune or a GC — a successful prune of a small namespace moves the number by less than one displayed digit, so the UI cannot distinguish "it worked" from "nothing happened". Cosmetic today; it becomes load-bearing the moment retention (R-89) is customer-visible and someone needs to see that a policy change took effect | diff --git a/documentation/runbooks/iso-release-gate.md b/documentation/runbooks/iso-release-gate.md index 70f41d5..296223f 100644 --- a/documentation/runbooks/iso-release-gate.md +++ b/documentation/runbooks/iso-release-gate.md @@ -134,12 +134,29 @@ osirrox -indev "$ISO" -find /proxmox/packages 2>/dev/null | grep -c 'felhom-' osirrox -indev "$ISO" -extract /proxmox/packages/.deb /tmp/p.deb dpkg-deb -I /tmp/p.deb | grep -E '^ (Package|Version):' ``` -**PASS =** exactly one `felhom-*.deb`, its version recorded in the manifest, and its sha256 matching -the package built from committed source. +**PASS =** exactly one `felhom-*.deb`, its version recorded in the manifest, and **no `Depends`** +(the payload is a script and a unit file; the binaries it calls run at first boot, not at postinst +time, so there is nothing for `dpkg --configure -a` to order against). *Why:* Spike 4 measured that this is the only delivery mechanism that survives an interactive install (`[first-boot]` is never placed on the system — `Config.pm:118`, `Install.pm:746`, `:1360`). +> **Amendment, 2026-07-31 — a sub-clause that could never pass is removed, and the reason recorded.** +> This criterion originally also required *"its sha256 matching the package built from committed +> source"*. **That is unsatisfiable by construction, not by accident:** `dpkg-deb` stamps the build +> time into every archive, so two builds of byte-identical source produce different `.deb` hashes. +> The check was written expecting reproducibility the tool does not offer, and it was already failing +> when the 1.26.1 release ran it. +> +> A criterion nobody can satisfy is worse than no criterion: it gets waived the first time and read +> as advisory ever after. This project already carries a shelf of gates that exist and are never run +> (**R-29**), and that is exactly how they got there. +> +> **The payload-integrity claim it was reaching for is carried by G9**, which compares the *contents* +> rather than the container — see G9's own amendment, which widens it to cover both payload files. +> If `dpkg-deb` is ever made reproducible here (`SOURCE_DATE_EPOCH`), this sub-clause can be restored +> and would then mean something. + ### G8 — The postinst cannot fail, and cannot need what it hasn't got ```bash @@ -157,17 +174,35 @@ installer's DHCP happened to hold, so a box installed with the cable out has non exits non-zero surfaces as an error in the middle of a customer's install — far worse than the stub not running. `set -e` is banned because it converts any unexpected non-zero into exactly that failure. -### G9 — `felhom-bootstrap.sh` matches repo HEAD byte-for-byte +### G9 — **both** payload files match repo HEAD byte-for-byte -Extract the script from the package and compare: +Extract each payload file from the package and compare: ```bash -dpkg-deb --fsys-tarfile /tmp/p.deb | tar -xO ./usr/local/sbin/felhom-bootstrap.sh | sha256sum +dpkg-deb --fsys-tarfile /tmp/p.deb | tar -xO ./usr/local/sbin/felhom-bootstrap.sh | sha256sum sha256sum scripts/iso/felhom-bootstrap.sh +dpkg-deb --fsys-tarfile /tmp/p.deb | tar -xO ./lib/systemd/system/felhom-bootstrap.service | sha256sum +sha256sum scripts/iso/felhom-bootstrap.service ``` -**PASS =** identical. +**PASS =** both identical. -*Why:* it is the one genuinely frozen, drift-capable payload — everything else the box uses is fetched -at run time from `main`. An ISO in a drawer for six months still runs this exact script. +*Why:* these are the genuinely frozen, drift-capable payloads — everything else the box uses is +fetched at run time from `main`. An ISO in a drawer for six months still runs exactly these. + +> **Amendment, 2026-07-31 — widened from one file to two, because the unit was uncovered.** +> G9 originally checked only `felhom-bootstrap.sh`. The package ships **two** payload files +> (`scripts/iso/pkg/build-deb.sh:54-55`), and the second one — the systemd unit — was checked by no +> criterion at all: G7 covered the container (and its sha sub-clause was unsatisfiable anyway), G8 +> covers the postinst *behaviourally*, G13 covers directory *presence*. +> +> The unit is not incidental. Its `After=network-online.target pve-cluster.service pveproxy.service`, +> its `ConditionPathExists=!/etc/felhom/.bootstrap-done` and its `Restart=on-failure` are what decide +> **when and whether day-0 runs at all**. A drifted or hand-edited unit would have shipped silently — +> the same shape as the `/etc/felhom/` miss that G13 exists to prevent, where a check proved the thing +> present and said nothing about what it depended on. +> +> The check passes today: `build-deb.sh` installs both files verbatim from the same source directory. +> **Note it self-asserts only the script** (`build-deb.sh:86-89`) — extending that assertion to the +> unit is a code change, filed as an observation rather than made here. ### G10 — The profile and every build input are committed @@ -181,6 +216,16 @@ git rev-parse HEAD; git rev-parse origin/main # equal reference boxes cannot be rebuilt from `main`. A published image must never reach that state, and R-144 records a lab ISO that already has (`nested-probe`'s profile no longer exists). +### G11 — A published checksum, and a verified round trip + +**PASS =** the `.sha256` and the manifest are uploaded beside the ISO, **and** the file downloaded from +`https://iso.felhom.eu/` checksums to the same value. The round trip is the claim — a local +checksum proves only that the local file is what it is. + +### G12 — The bucket stays private + +**PASS =** the bucket's `Public Access` remains **Disabled**; the custom domain is the only public path. + ### G13 — every directory the payload writes into is IN the package ```bash @@ -199,16 +244,6 @@ it. The installed box registered at the hub, failed to persist the token, and th Any future criterion of the form "the correct file is present" should be paired with one of the form "and everything it needs at run time is too". `build-deb.sh` asserts this itself and is red-proofed. -### G11 — A published checksum, and a verified round trip - -**PASS =** the `.sha256` and the manifest are uploaded beside the ISO, **and** the file downloaded from -`https://iso.felhom.eu/` checksums to the same value. The round trip is the claim — a local -checksum proves only that the local file is what it is. - -### G12 — The bucket stays private - -**PASS =** the bucket's `Public Access` remains **Disabled**; the custom domain is the only public path. - --- ## Result recording