diff --git a/CLAUDE.md b/CLAUDE.md index eb2022a..dd4816f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -213,6 +213,22 @@ local and skippable, and only CI is neither. - **Website** auto-deploys via git-sync; just push to `main` (live in 1–2 min). Website changes go through `repo_gates.py` above (it runs `site_gates.py`); new pages go into that gate's `PAGES` list. Emergency edits: https://files.felhom.eu. All `website/` HTML is **UTF-8 with BOM** — preserve it. +- **THE INSTALLER DOES NOT (R-110, 2026-08-03).** `manifests/webpage.yaml` runs **two** git-syncs: + the website from `main` as above, and `/scripts/` from the tag **`installer-v`**. + Pushing `scripts/felhom-host-install.sh` therefore changes nothing that any machine downloads — + which it used to, within thirty seconds, for the one artifact that runs as **root on a virgin box**. + - **To publish:** cut `installer-v`, bump the `--ref` in `webpage.yaml` + (both the sidecar and the init container), commit, and sync. `hostinstall_gates.py` gate 6 + fails if the manifest stops naming an `installer-v…` tag or if the website stops tracking `main`. + - **To roll back:** move the tag back to the previous commit and wait ~30 s. **No ArgoCD sync and + no deploy** — git-sync picks up a moved tag on its next period, measured live on 2026-08-03 in + both directions. That is the emergency lever; fix forward with a new version afterwards. + - **Do NOT pin the website to the tag.** The sparse-checkout used to cover `/website/` and + `/scripts/` in one sync, and pinning that would turn every copy edit into a release. + - The **URL never carries a ref** (`https://felhom.eu/scripts/felhom-host-install.sh`), so + `felhom-bootstrap.sh` and the hub's day-0 command follow the tag with no edit — do not add one. + - The installer's own sixteen run-time fetches are pinned separately, to `raw/tag/v$ART_AGENT_VER` + in the **agent** repo (R-183) — they are the agent's configs, not this repo's. - **Manifests** are GitOps via the `felhom` app — commit to `main`, then deliberate sync. ## Key patterns diff --git a/CONTEXT.md b/CONTEXT.md index 6fd5675..79d432f 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -97,6 +97,32 @@ not provide, and the fourth of those found on live hardware rather than by revie no run scope, so a refused app re-alerts on every status refresh (measured: a second identical alert pair 13 s after the run's). Pre-existing in v0.192.0; R-181 changed neither caller. +**S-15 — publishing is an act, not a side-effect of pushing (2026-08-03, R-110 + R-115 + R-183).** +Two rulings, one shape: something became live because someone pushed, not because anyone decided. + +- **The installer.** `/scripts/` now git-syncs the tag `installer-v`; the **website + keeps tracking `main`** in a second sync, because pinning both would make every copy edit a + release. Publish = cut the next tag + bump the manifest `--ref` + sync. **Roll back = move the tag + back**, which takes ~30 s and needs no ArgoCD sync at all — git-sync v4.4.0 follows a moved tag, + and that half was measured before the manifest was touched because the whole model rests on it. +- **The sixteen run-time fetches were NOT what the spec described** — sixteen, not nine, and from + `felhom-agent`, not this repo — so no tag here could cover them. They are pinned to + `raw/tag/v$ART_AGENT_VER` instead, which is strictly better: the agent's configs now come from the + same ref as the agent binary being installed. That closed a real skew (**R-183**), not just a + channel. +- **The URL needed no change**, and that is worth knowing rather than re-deriving: it never carried + a ref, so both producers follow the tag automatically — and no hub change means no hub bump. +- **The agent.** `scripts/release-agent.sh` is THE release path: build → tag → publish → **verify by + an independent download**. It does not vouch. `check-published-versions.py` refuses a `v` + tag with no downloadable package, and **CI now runs the full gate set** rather than `--fast`, + without which that gate would have been registered and never run. +- **The gate's invariant is not the one specified, and P-C is why:** the hub manifest and Gitea's + package listing are both **401** anonymously; the package download and the tags api are not. So CI + can ask *is this installable* but not *what is vouched*. The residue is **R-184**. +- **Neither gate asserts "the newest version is published."** That would go red on the very push + that bumps a version, before publishing — and a gate that fails on the normal path is one people + learn to ignore. + **S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).** Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure diff --git a/REPORT.md b/REPORT.md index 196ed85..d699157 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,216 +1,200 @@ -# REPORT — R-181 (the reserve guards the write that fills the disk) + R-156 (papra) + two operator rulings +# REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed -**Date:** 2026-08-03 · **Repos:** `felhom-controller` (v0.192.0 → **v0.193.1**), `app-catalog-felhom.eu`, `felhom.eu` (docs only — **no hub change, no hub version bump**) +**Date:** 2026-08-03 · **Repos:** `felhom.eu` (installer **v1.22.0 → v1.23.0**), `felhom-agent` (**no bump**) +**Nothing was built** — no image, no binary, no golden. **Hub stays v0.89.0.** -## 1. Baselines — re-read on arrival, all matched §1 +## 1. Baselines — re-read on arrival, both matched §1 -| Repo | `main` @ arrival | Version | Shipped | +| Repo | @ arrival | Version | Result | |---|---|---|---| -| `felhom-controller` | `4be6467b501b` | v0.192.0 | **v0.193.0 `fef07c3`** → **v0.193.1 `6c43bf6`** | -| `app-catalog-felhom.eu` | `7cb58ecdf8e7` | n/a | `122bbee` | -| `felhom.eu` | `6b5d64c1fa73` | hub v0.89.0 | docs only, **no bump** | +| `felhom.eu` | `8360f940bfb2` | hub v0.89.0, `SCRIPT_VERSION="1.22.0"`, **0 tags** (confirmed) | installer **v1.23.0**, first tag `installer-v1.23.0` | +| `felhom-agent` | `9dfd89cb947e` | v0.120.0 | **unchanged** — scripts and gates only | -All three clean (`git status --porcelain` empty, `HEAD == origin/main`) before every build. +## 2. Part 0 — the R-182 measurement, and it REVERSED the row -## 2. The fix +Filed yesterday as *"the reserve re-alerts on every status refresh"* — **too many** alerts, observed +at the sending end. Measured at the **receiving end**, it is the opposite. -**One admission verdict per app per run** (`controller/internal/backup/admission.go`), taken before -that app's **first** write and consulted by all three legs — DB dump, volume dump, unit capture. The -three write under one per-app root (`appbackup.RecoveryUnitPath`), which is what makes one verdict -able to cover them honestly. +Method: the hub's SQLite copied **with its `-wal`** (4 MB and newer than the db — copying `hub.db` +alone would have read stale data, the exact trap this project recorded before), freshness confirmed by +the newest `notification_log` row post-dating the session. -- **Lazy, not run-wide.** App A's dump can put app B under the reserve; a run-start verdict reads a - disk that no longer exists. **Never re-decided between an app's own legs** — that is the split being - closed. **Reset per run.** -- **Ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act, so a refused app is never - bounced. **After** the volume-less check, which has no write to gate. -- **Exactly one operator alert per refused app per run.** Leg order unchanged. -- **Size term added:** *would this app's write cross the reserve?* — estimated from its previous - `.sql` + `.tar`. **No history → headroom-only**, or the first backup becomes the one that can never - happen; the alert says so when that applies. +**9 `recovery_unit_capture_failed` events received today → 2 operator emails sent.** -## 3. Files +| time | apps refused (events in) | operator emails out | +|---|---|---| +| 06:40:03 | privatebin, opengist | **opengist only** | +| 08:59:46/47 | opengist, privatebin | **privatebin only** | +| 08:59:59 | privatebin, opengist | **none** | +| 09:03:00 | opengist | **none** | +| 09:07:06 | privatebin, opengist | **none** | -| File | | +**Cause, confirmed at source:** the operator cooldown key is +`customerID + ":" + eventType + cooldownTierSuffix(details)` (`dispatcher.go:268`, 1 hour hardcoded). +`RecoveryUnitFailureDetails` carries **`app`** and **no `tier`**, so the suffix is empty and the key +holds **no app identifier**. The first refused app takes the slot; every other app's refusal for the +next hour is dropped — and dropped **before `LogNotification`**, so it leaves **no row on any +channel** and cannot be audited afterwards. + +This is **R-97a's failure mode in a second event type**; that row's own comment states it +(*"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20"*). `cooldownTierSuffix` was +written narrow on purpose; `recovery_unit_capture_failed` simply never opted in. + +**A correction I owe on yesterday's report.** It said *"one `recovery_unit_capture_failed` per app, +HTTP 200"*. That was true of what the **controller pushed**, and a reader would take it as *the +operator was told about each app* — which is false. The gap between an accepted event and a sent +email is the whole of this row. + +**Nothing was changed** (§8.5). R-182 is re-scoped with the evidence and the fix shape. + +## 3. Probes + +| | Question | Method | Verdict | +|---|---|---|---| +| **P-A** | does git-sync v4.4.0 follow a tag, and notice a **moved** one? | throwaway `docker run` git-sync against this repo, tag moved under it | **PASS both halves** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period (~20 s) | +| **P-B** | does Gitea serve `raw/tag//`? | one fetch on a throwaway tag | **PASS** — HTTP 200, byte-identical to `raw/branch/main` | +| **P-C** | can CI read the package registry? | anonymous fetches | **PARTIAL, and it changed the gate's design** — package **download** 200 (and **404** for a fake version, so it discriminates), **tags** api 200; package **listing** api **401**, hub artifact manifest **401** | + +**Publish model P-A implies:** publishing is **moving the tag**; rollback is **moving it back**, in +~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone +(`git ls-remote --tags` → 0 at the time). + +## 4. §8.2's three channels — enumerated + +| Channel | Before | After | | +|---|---|---|---| +| 1. the served script | `main`, 30 s | **`installer-v1.23.0`** | **MOVED** — `webpage.yaml` split into two syncs | +| 2. the run-time fetches | `raw/branch/main` | **`raw/tag/v$ART_AGENT_VER`** | **MOVED** — but see below | +| 3. the URL producers | `main` | unchanged | **NO CHANGE NEEDED** — and that is a finding, not an omission | + +**Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable.** +There are **sixteen** fetches, not nine, and they come from **`felhom-agent`**, not `felhom.eu` — so +no tag on this repo could ever have covered them, and §8.1's *"derive the tag from `SCRIPT_VERSION`"* +was impossible for them. Raised before building; operator ruled to pin them to **the agent version +being installed**, which the installer already resolves from the hub manifest and already sha-verifies. +That is strictly better than any installer-derived tag: binary and configs now come from one ref. + +**Channel 3 needed no change because the URL never carried a ref** — +`https://felhom.eu/scripts/felhom-host-install.sh` is path-based; the ref lives in the manifest. So +`felhom-bootstrap.sh` and the hub's day-0 command follow the tag automatically. **No hub template +change ⇒ no hub bump**, so §1's rule was never in tension and the STOP it anticipated never arose. + +## 5. The tag convention + +- **Shape:** `installer-v` in `felhom.eu` (prefixed so it cannot be read as a hub, + agent, controller or golden version); `v` in `felhom-agent` (that repo versions one thing). + **No new constant in the installer** — channel 2 derives its ref from `$ART_AGENT_VER` at run time, + and channel 1's ref lives only in the manifest. +- **Publish:** cut `installer-v`, bump the `--ref` in `webpage.yaml` (sidecar *and* + init container), commit, sync. +- **Roll back:** move the tag back to the previous commit — takes ~30 s, **no ArgoCD sync, no deploy**. + +## 6. Scenario A — proven by HTTP + +A real commit was pushed to `main` (a marker comment in the installer) **without moving the tag**, and +three sync periods were allowed to pass so "unchanged" means "had every chance to change": + +``` +website tree (main): .worktrees/6a82719… <- ADVANCED to the new commit +scripts tree (tag): .worktrees/bee6848… <- STAYED +sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555 +sha256 after push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555 +marker present at the served URL? 0 +https://felhom.eu/ -> HTTP 200 +``` + +Both halves of the split in one observation: the site still tracks `main`, the installer does not. + +## 7. Scenario B — publish and rollback, both directions + +| act | result | |---|---| -| `controller/internal/backup/admission.go` | **new** — the gate, the memo, the estimator | -| `controller/internal/backup/admission_test.go` | **new** — 11 tests | -| `controller/internal/backup/backup.go` | run scope + gates in the DB and volume legs | -| `controller/internal/backup/recovery_unit.go` | `floorVerdict` size-aware; capture leg via `admitApp` | -| `controller/internal/backup/capture_floor_test.go` | 3 call sites updated for the new signature | -| `controller/README.md`, `REUSE.md`, `CHANGELOG.md` | | -| `app-catalog-felhom.eu/templates/papra/docker-compose.yml` | mount moved to `/app/app-data` | +| tag moved `bee6848 → 6a82719` | scripts tree moved in **~40 s**; served `sha256 ea2b4aa9…`; **marker present** | +| tag moved back `→ bee6848` | scripts tree back in **~40 s**; served `sha256 2f859555…` — **exactly** the pre-publish sha; **marker gone** | -## 4. Tests — 28 packages `ok`, `rc=0` (read separately from any commit) +`https://felhom.eu/` returned 200 throughout. The marker commit was then reverted, and the tag moved +to `main`'s head — a **byte no-op**, verified by the served sha not changing. -All 11 new tests pass, plus the pre-existing floor suite. Refusal assertions are **sha256 tree -fingerprints before and after**, never log lines — the defect being fixed *is* a log line the tree -contradicted. +## 8. Files, commits, tags -The DB leg cannot run without Docker (`DiscoverDatabases` shells out), so its gate is pinned by an -**AST walk** of `backup.go` asserting `admitApp` precedes `DumpOne`. `strings.Contains` is -insufficient: a commented-out call still contains the string. +**`felhom.eu`** — `bee6848` (installer + gate + manifest), `6a82719` (Scenario A marker), `e79a20b` +(marker removed), plus the docs commit below. +`scripts/felhom-host-install.sh` · `scripts/hostinstall_gates.py` · `scripts/CHANGELOG.md` · +`manifests/webpage.yaml` · `CLAUDE.md` · `CONTEXT.md` · `STATUS.md` · `REPORT.md` · +`documentation/backlog/{OPEN-ITEMS,ROADMAP}.md` · `documentation/architecture/00-capability-map.md` -### Red-proofs — each demonstrated failing, then restored +**`felhom-agent`** — `dd2d1fe` (release path + gate + CI), `0db7766` (REPORT). +`scripts/release-agent.sh` **(new)** · `scripts/check-published-versions.py` **(new)** · +`scripts/agent_gates.py` · `.gitea/workflows/gates.yml` · `CLAUDE.md` · `CHANGELOG.md` · `REPORT.md` + +**Tags created:** `felhom.eu/installer-v1.23.0` (the first tag this repo has ever had) and +`felhom-agent/v0.120.0` (retroactive, at `cd6e267` — the commit the published binary was built from; +`configs/` is byte-identical there and at `main`, so nothing depended on the choice). + +## 9. Tests and red-proofs + +| Check | Result | +|---|---| +| `felhom.eu` `repo_gates.py --fast` | all 5 gates OK | +| `felhom-agent` `go build ./... && go vet ./...` | OK | +| `felhom-agent` `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) | +| `agent_gates.py --fast` | `published` correctly **SKIPPED** (hook must not fail on a network blip) | +| `agent_gates.py` (full) | both OK | + +**Red-proofs, each demonstrated failing then restored:** | # | Mutation | Result | |---|---|---| -| 1 | **Both** dump-leg `admitApp` gates removed (= exactly v0.192.0) | Scenario A **RED** — *"the VOLUME leg ran for a refused app"*; with the leg assertions temporarily made non-fatal, the **tree fingerprint changed** too. Also red: Scenario C, Scenario D, and the AST wiring test (which named the DB leg specifically) | -| 2 | The entire size term removed from `floorVerdict` (both its thresholds) | Scenario D **RED** — 0 alerts where 1 was required | -| 3a | The reserve removed entirely | Scenario F **PASSED — recorded honestly.** The specified mutation does not exercise the assertion: removing the reserve makes every app write, which overwrites and adds but **deletes nothing**, so a deletion-watching test correctly stays green | -| 3b | A prune injected into the refusal path | Scenario F **RED** — this is the mutation that proves the test watches deletion | -| 4 | Floor moved above the warning band (90% / 6 GiB) | `TestFloorSitsBelowTheCriticalWarningBand` **RED** | +| C | one of the sixteen fetches reverted to `raw/branch/main` | **RED** — gate 6a *and* 6b both fired | +| D | assertions 6a **and** 6b removed (every guard the test covers), same bad installer | **zero** mentions of the regression — the guards are what catch it | +| 6c | the manifest before the split | **RED** on its own, before I fixed it — the gate was demonstrated red by the real pre-change state | +| F | `v9.9.9` tagged and not published | **RED**, `binary NOT downloadable (HTTP 404 …)`, rc=1 | +| F′ | the gate **deregistered** from `agent_gates.py`, same bad state | **rc=0, "all agent gates OK"** — restored → `CONVICTED: published`, rc=1 | -Every mutation removed **every** guard its test covers (#1 removed both dump-leg gates, not one). +**Scenario F measured on real CI, not inferred.** Runs **69** and **70** are on the *same commit* +`0db7766`: **success** before `v9.9.9` existed, **failure** after pushing it. One variable. This also +retrospectively explains runs 67/68. **One deliberate CI failure email reached the operator — that was +this proof, not an incident.** I could not read CI's own step log: the jobs endpoint needs a Gitea API +token, and the only credential available (`~/.docker/config.json`) is a registry password that the API +rejects — so the controlled before/after replaced the log rather than an assumption standing in for it. -## 5. Live validation — demo-hp guest 9201 (Tier 0), the method that found the defect +## 10. No version bumps, nothing built -**Method:** endpoint-level — `POST /api/debug/backup/dbdump`, the exact endpoint the debug UI button -calls, which runs the production `RunDBDumps`. No browser on DooPlex. +`felhom-agent` **v0.120.0** unchanged (no Go code changed). Hub **v0.89.0** unchanged (no hub file +touched). The installer's `SCRIPT_VERSION` **did** go 1.22.0 → 1.23.0 — the installer is not in §12's +no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or +golden was built. -**The instrument was re-proven before use.** demo-hp's thin pool is 53.93 GiB, so a real fill of a -70 G volume would exhaust it and corrupt every guest. A 5 GiB `fallocate` step moved guest `df` -1.2G → 6.2G while thin-pool `data_percent` held **36.83 → 36.83** — zero blocks allocated. Re-checked -at every step of the fill. +## 11. Register -### Headroom term — 08:59:46, 906 MB free / 99% used - -| Observable | Result | +| ID | Outcome | |---|---| -| Tree fingerprint before | `TREE_SHA=111d1760c18d3440f700634ab325f8b8` (10 files; opengist's tar **182,272 B** — R-181's own "before" figure) | -| Tree fingerprint after | **`111d1760c18d3440f700634ab325f8b8` — identical** | -| Volume dumps written | **0** (baseline run at 08:58 wrote 2) | -| `Stopping for safe volume dump` | **absent** — and this is evidence, not an absence, because that line **is** present in the 08:58 baseline | -| Operator alerts | one `recovery_unit_capture_failed` per app, severity `error`, HTTP 200 | +| **R-110** | **CLOSED — SHIPPED** (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed | +| **R-115** | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, no bump) | +| **R-182** | **RE-SCOPED — the direction reversed** by Part 0's measurement; still open, now correctly described | +| **R-183** | **NEW, and CLOSED the same session** — binary and configs came from two different refs | +| **R-184** | **NEW, open** — nothing stops the hub vouching a version that was never released | -Free space restored → re-run at **09:01:33**: both apps captured normally. +**IDs established free:** `^| \*\*R-183\*\*` / `^| \*\*R-184\*\*` in `OPEN-ITEMS.md` → **0 rows** each; +all other hits are this session's own code and changelogs (forward references I wrote). `R-185` → 0 +hits anywhere and remains free. -### Size term — 09:03:00, proven separately +## 12. Observations — noticed, documented, NOT acted on -Reproducing the original sequence: a real 2 GiB file planted in opengist's volume, backed up so its -**previous** tar became **2,147,666,432 B** (the exact live figure), then the filesystem set to -**91% used / 2.9 GB free — both headroom terms deliberately clear**. +1. **The gate cannot see what is vouched** — filed as R-184 rather than papered over. Closing it needs + either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails + closed where the mistake is made, needs no new credential). +2. **A suppressed operator alert leaves no row at all.** The cooldown returns before `LogNotification`, + so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside + R-182 because it is what made that row take a day to get the right way round. +3. **`on: [push]` fires CI for tag pushes too.** Useful (it is how Scenario F was measured), but it + means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate. +4. **`felhom.eu` CI still runs `--fast`.** Correct today, since all its gates are network-free; if a + network gate is ever added there, that workflow needs the same change the agent's just got. -- **opengist refused `(size)`** — *"this app's last backup was 2.0 GB and writing it again would cross the reserve"* -- **privatebin ADMITTED and dumped normally** — the term is per-app, not a global halt -- Tree unchanged; 1 volume dump instead of 2 +## 13. Teardown -### One honest correction to the "app not stopped" claim - -`StartedAt` on both apps *did* move, 26 s **after** the refusal. It was the **quiesce loop** for the -whole-guest PBS backup, which my fill had broken — not the app-data path. Its own backoff logic then -behaved correctly (*"deferring its next quiesce by 15m so the apps are not stopped again for a backup -that cannot succeed"*). The app-data claim rests on the **absence of the `Stopping … for safe volume -dump` line**, which is the line that appears when that leg bounces an app. - -## 6. The `du` measurement (§Part 1.3) — measured, then rejected - -**66 timed runs** on demo-hp guest 9201, `docker run --rm -v :/v alpine du -sb /v`: -**median ~355 ms per volume, range 341–404 ms** — on volumes holding **tens of KB**. The cost is -container start-up, not the walk, so it does not shrink for small apps and only grows for real ones. - -**Rejected**, on two grounds beyond the number: `docker run` needs the writable layer, so the -measurement mechanism can fail under exactly the disk pressure the reserve exists to handle; and the -previous-dump estimate measures the **artifact that will be written** rather than the live volume, -which is the truer predictor. The previous-dump estimate stands. - -## 7. The refusal message as shipped, and what it guarantees - -``` -[WARN] [backup] App backup REFUSED for opengist (headroom) — refused: backing up this app would -leave the filesystem below the reserve (reserve: 97% used or 1.0 GiB free; the filesystem is already -below it, before this app's estimated 178.0 KB write) — /mnt/sys_drive: 64.3/68.7 GB used (94%), -0.9 GB free; NO database dump, NO volume dump and NO recovery-unit capture was written for it, the -previous unit is untouched and NOTHING was deleted -``` - -**It guarantees, for that app in that run:** no DB dump, no volume dump and no capture were written; -every file under `backups/primary/` is byte-identical; the app was not stopped; nothing anywhere -was deleted; exactly one operator alert was sent. All five verified by fingerprint above. - -**The wording was not weakened to fit the behaviour** — the behaviour moved so the wording became -true. What was *added* is the bound term (`headroom` / `size`) and the estimate. - -**v0.193.1 — found by this very proof run.** The estimate was rendered fixed to two-decimal GiB, so -opengist's real **178 KB** printed as `estimated 0.00 GiB write`, which reads as *no estimate was -available* — the opposite of what happened. Shipped the same session because it is the same defect -class the whole task is about. Re-verified live after redeploy: `estimated 178.0 KB write`. - -## 8. papra (R-156, last leg) - -**Precondition checked, not inherited** — both boxes were wiped and rebuilt today, so the 2 August -evidence was re-measured: `docker ps -a` (**including stopped**) on **both** demo guests → no papra; -hub `/hosts` → exactly two enrolled hosts (`demo-felhom-8363b5`, `demo-hp-bb76ea`), **zero** papra. - -**Decided from the image, not the README:** `WORKDIR=/app`, `DATABASE_URL=file:./app-data/db/db.sqlite`, -`DOCUMENT_STORAGE_FILESYSTEM_ROOT=./app-data/documents`, `PAPRA_CONFIG_DIR=./app-data` — and -**`/app/data` does not exist in the image at all**. - -**Departure from the task's stated preference order, stated because it was deliberate.** Option (1) -(reconfigure the app to write to `/app/data`) *was* available — all three paths are env-settable. Not -taken: it enumerates data paths, so a fourth added upstream would silently escape to the writable -layer again — this defect re-armed and invisible. Mounting the app's own data **root** captures every -current and future path by construction. - -**Gate output — the arbiter, run in both directions:** - -- fixed → `papra CLEAN`, with the self-test passing on that run: *"prober flags the R-156 signature and clears a correct template — trustworthy"* -- reverted to `/app/data` (red-proof on the **real template**, not just the canary) → `BROKEN`: *"mount /app/data is NOT writable by the app's own uid=999"*, *"DATA in the writable layer at /app/app-data/db (db_signature=True, e.g. ['db.sqlite'])"*, *"declared volume /app/data is EMPTY"* -- `catalog_gates.py papra` (full, not `--fast`) → **rc=0**, all three gates OK - -**Two operational findings about the gate:** it needs **root** (it reads `/var/lib/docker/volumes`, -mode `drwx--x---`; as a normal user its own canary fails UNDETERMINED and it correctly refuses a -verdict — fail-closed working as designed), and it hardcodes scratch path `/srv/felhom-gate`, created -on DooPlex. Unscoped it deploys all 53 templates; that run was aborted after 10 minutes and its -`volgate-*` scratch projects were cleaned up. - -## 9. §3's correction — confirmed in passing, not chased - -`restore_points.go:57-59` takes the manifest's mtime and then `newestArtifact` over the `.sql` and -`.tar` files, so **the newest of the three wins**. The restore point does **not** show a stale -timestamp. Confirmed and dropped, as instructed. - -## 10. Register - -| ID | Change | -|---|---| -| **R-181** | **CLOSED — SHIPPED** (v0.193.0 + v0.193.1), with the live evidence above | -| **R-156** | **CLOSED** — all three apps fixed | -| **R-110** | WAITING-ON-OPERATOR → **READY**, ruling attached: **option (b), tag-tracked**, and it must cover **both** channels (the `/scripts/` git-sync *and* the nine files fetched from `raw/branch/main`) or it only half-works | -| **R-115** | WAITING-ON-OPERATOR → **READY**, ruling attached: **mechanism (b)**, a build-side gate refusing to deploy or vouch an unpublished version; the third instance (agent v0.120.0) would have silently downgraded both demo boxes while reporting success | -| **R-182** | **NEW.** ID established free: `grep -ro "R-182\b"` over `documentation/` and `*.md` → 2 hits, both prose in `REPORT.md` recording it as *"checked and left unused"*; `R-183` → 0 hits and remains free | - -**R-165** is collapsed to CLOSED/PROVEN-LIVE in `ROADMAP.md`; the capability map's local-backup row -moves to **PROVEN-LIVE, both halves**, because the live fill proved the fixed behaviour for **both** -reserve terms. - -## 11. Observations — noticed, documented, NOT acted on - -1. **R-182 (filed).** The periodic status refresh (`GetFullStatus` → `captureAllRecoveryUnits`) runs - with no admission scope, so a refused app re-alerts on every poll — measured live: a second - identical alert pair 13 s after the run's. **Pre-existing in v0.192.0**; R-181 changed neither - caller. Its mitigation is a *comment* claiming the hub owns cooldown — which is exactly the - "invariant asserted in a comment with no test pinning it" shape, so verify at the hub before - scoping. -2. **A reserve refusal does not make the run fail.** The DB and volume legs record `SKIP`, not `FAIL`, - so `lastDBDump.Success` stays true and the customer-facing status does not turn red. Deliberate and - consistent with v0.192.0 (the capture refusal never set it either), and the operator alert is the - signal — but it means "backup succeeded" and "every app was backed up" are not the same statement. -3. **`UnitSpace.UsedPercent` and `df` disagree** — `df` reported 99% where the alert said 94%, because - `df`'s figure accounts for ext4 reserved blocks and the floor's does not. Harmless here (the - free-byte term bound), but a percent-term threshold is being compared against a number the operator - cannot reproduce with `df`. -4. **The whole-guest PBS backup fails when the volume is near-full**, pushing - `whole_guest_backup_failed` (severity `error`). Expected under a deliberate fill, and its backoff - behaved correctly; noted because it is collateral any future fill test will also produce. - -## 12. Teardown - -Fill file removed; the planted 2 GiB file removed; a final backup regenerated a correct 178 KB tar; -`pct fstrim 9201` returned 67.5 GiB and the thin pool settled at **29.43%**, *below* its 36.83% -baseline. The backups tree is byte-identical to the pre-test fingerprint. Guest helper scripts and the -credential file `shred`-ed. `volgate-*` scratch compose projects removed; the unrelated 9-day-old -`jarr-*` containers on DooPlex were left untouched. papra is **not** left deployed. - -No `--no-verify` was used on any push; the `felhom-controller` pre-push hook ran and reported -`gates OK` on both pushes. +Probe container, probe sync tree and probe tag (`probe-r110-delete-me`) removed; the red-proof tag +`v9.9.9` deleted (`git ls-remote --tags` → only `v0.120.0`); the Scenario A marker reverted from +`main` and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl +pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file. diff --git a/STATUS.md b/STATUS.md index 87883e1..7da1922 100644 --- a/STATUS.md +++ b/STATUS.md @@ -27,34 +27,51 @@ time, and an app switched off deliberately stayed off every time. also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy. *(R-95, R-87)* -**A full disk emails you repeatedly instead of once.** When the reserve refuses an app's backup you -are told once by the backup run — correctly — but the page showing backup status re-checks on a timer -and sends the same message again each time. Not new: as old as the reserve itself, and seen only -because we watched the alerts closely while proving the fix below. Harmless if the hub already -collapses repeats — which a comment claims and nobody has checked. *(R-182)* +**A full disk tells you about ONE app and silently swallows the rest.** Yesterday this was written +down the wrong way round — as *too many* emails. Measuring the receiving end reversed it: of nine +refusals the machine reported today, **two emails were sent**. When two apps are refused in the same +second you are told about one of them, and the other leaves no trace anywhere — not an email, not +even a line in the log saying it was held back. So a second app can be going unbacked-up while you +have already been told the problem is handled. It is the same fault we fixed once before for +whole-machine backups, in a second place that never opted into the fix. *(R-182)* ## What shipped recently -**The backup partition is gone, and both demo machines run on the new shape.** Wiped and rebuilt on -3 August and taken through the whole customer journey — set up, install an app, back it up, restore -it. One storage area instead of two; the space a backup can use went from 19 GB to 65 GB on the small -machine and 45 GB to 233 GB on the big one. Three reboots each, correct every time. The two were -rebuilt deliberately differently — one from a local copy of the image, one by the ordinary customer -route with the published fingerprint checked — so the disk shape and the delivery route are both -proven, rather than one proven twice. Their previous demo apps and data are gone; that was the point -of a wipe, and you approved it. *(R-165, R-178)* +**Pushing the installer no longer publishes it.** The script that runs as root on a brand-new +machine was copied from the main branch and served within thirty seconds, so pushing it *was* +publishing it, with no staging and no way back but another push. It now comes from a **labelled** +version: publishing is moving the label, and undoing it is moving the label back — about half a +minute, no deploy. The website is untouched by this and still updates in thirty seconds, because a +typo fix must never need a release. Proven by actually doing it: a real push changed nothing that +anyone downloads, moving the label published it, moving it back restored the previous bytes exactly. -**What replaced the wall — and it now watches the right moment.** The wall was quietly doing a second -job: keeping a runaway backup from eating the space the machine needs to keep running. That job is now -explicit, and as first built it was checked too late — the big write happened first, unchecked, and -only the small write after it was refused, while the message still promised your last good copy was -untouched. **Fixed and proven on 3 August.** The machine now decides once, per app, **before it writes -anything at all**, and that one answer covers all three steps: a refused app writes nothing, is not -restarted, and the promise is now literally true — checked by fingerprinting every file before and -after. It also stopped being blind to size, so an app is no longer waved through at 96% full and then -allowed to write two gigabytes. Proven by deliberately filling a demo machine, once for each way it -can refuse. Nothing is ever deleted to make room: every app has only one local copy, so "delete the -oldest" would always mean destroying some other app's only copy. *(R-181)* +**The catch that would have made it cosmetic was found and covered.** While it runs, the installer +fetches sixteen more files — not nine, and from the *agent's* repository, not the website's. They now +come from the same version of the agent the machine is installing. That closed a real fault nobody +had noticed: a new machine was getting the agent's tested program and its untested settings files, in +one install, from two different places. *(R-110, R-183)* + +**Releasing the agent now publishes it, in one command.** Putting a built agent where a new machine +can download it was a step someone had to remember, and it was forgotten three times in five days — +the last time leaving both demo machines running a version nobody could download, so a rebuild would +have quietly installed the *older* one and reported success. There is now one command that builds, +labels, publishes and then **downloads it back to check** — and a check that refuses to stay quiet if +a released version cannot actually be fetched. Proven by making CI fail on purpose and then go green +again on the same code. *(R-115)* + +**The backup partition is gone and both demo machines run on the new shape** — wiped, rebuilt and +taken through the whole customer journey on 3 August, by two deliberately different routes so the disk +shape and the delivery route are both proven. The space a backup can use went from 19 GB to 65 GB on +the small machine and 45 GB to 233 GB on the big one. Their previous demo apps and data are gone; that +was the point of a wipe, and you approved it. *(R-165, R-178)* + +**What replaced the wall now watches the right moment.** The wall was quietly keeping a runaway +backup from eating the space the machine needs to run. As first built, that replacement was checked +too late — the big write happened first, unchecked — while still promising your last good copy was +untouched. Fixed and proven on 3 August: the machine decides once, per app, **before it writes +anything**, and that one answer covers all three steps, so a refused app writes nothing, is not +restarted, and the promise is now literally true. It also stopped being blind to size. Nothing is ever +deleted to make room. *(R-181)* **The last of the three apps that never saved their data is fixed.** Installed nowhere, so nothing was stranded — checked on both demo machines and in the fleet list rather than assumed. Proven by the check @@ -75,24 +92,27 @@ step — but it notices quickly and tells you. *(R-29, R-161, R-168, R-169)* ## What we're working on -- **Now:** both of today's items are done — the reserve and the last unsaved app. Your two decisions - are written down and are ours to build. -- **Next:** building those two — moving the installer onto a labelled version so publishing is one - step you can undo, and a check that refuses to install a version nobody can download *(R-110, R-115)*. +- **Now:** nothing outstanding from today — the reserve, the last unsaved app, and both of your + decisions are all built and proven. +- **Next:** the alert that tells you about one app and swallows the second *(R-182)*. - **After:** the off-site copy that the machine making it can still erase *(R-95, R-87)*. ## Waiting on you - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a session log; nothing suggests anyone else saw it. *(R-132)* -- **Nothing else.** You settled both open questions on 3 August — the installer moves onto a labelled - version, and a check will refuse to install a version nobody can download. Both are written down and - are ours to build. *(R-110, R-115)* +- **Nothing else.** Both decisions you took on 3 August are now built and proven. One small question + will come back later: the automatic check cannot see which version you have told machines to + install, only which ones exist — closing that either needs a password given to the build server or + a check inside the hub itself. Filed, not urgent. *(R-184)* ## Changed since last update -- **2026-08-03** — The reserve now guards the step that fills the disk, and its promise is true; the - last app whose data was never saved is fixed. Both proven on a demo machine, not just in tests. +- **2026-08-03** — Publishing became something you do rather than something that happens: the + installer and the agent both moved onto labelled versions with a way back, and a check now refuses + a release nobody can download. Earlier the same day: the reserve now guards the step that fills the + disk and its promise is true, and the last app whose data was never saved is fixed. All proven on + real machines, not just in tests. Earlier the same day: both demo machines wiped and rebuilt from the new base image and taken through set-up → install an app → back it up → restore it, with the backup space ceiling gone and measured. @@ -102,10 +122,6 @@ step — but it notices quickly and tells you. *(R-29, R-161, R-168, R-169)* the hub's own database is in no automatic backup** — it holds every machine's emergency password. Filed, not yet fixed. -- **2026-08-02** — Boot recovery finished; six hard resets, everything back every time. Two instances - of the same hole — starting an app whose external drive was missing — were found by reading the code - and fixed the same day. - - **2026-08-02** — Thirteen mechanical checks had built up and nothing ran most of them; two were failing quietly. Fixed. Decided the same day: the 20 GB backup partition goes away; and only this machine and the tester's box are protected, every other box may be broken or reinstalled freely. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 89fc4a9..c0a9f7e 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -31,6 +31,7 @@ |---|---|---|---|---| | Appliance day-0 install: golden image → first boot → auto-confirm (zero clicks) → claimable box | installer, agent, hub, golden | **PROVEN-LIVE** (nested VM) | `DRILL-day0-vm-2026-07-12`, `DRILL-day0-take2-2026-07-12` | First firing on real customer hardware pending → R-1 | | BYO install: `--mode byo`, mandatory caps, host-mutation disclosure, coexistence guards | installer v1.15+, agent | **PARTIAL** | `DRILL-GL6-2026-07-08` (demo box); GL-8 coexistence fixes | Peti clean-slate reinstall on proxmox2 is the first real BYO run of the current path → R-1 | +| **The installer is PUBLISHED, not pushed — the artifact that runs as root on a virgin box is served from a version-controlled ref, and rolling back is one act** | scripts **v1.23.0** + `manifests/webpage.yaml` (R-110, operator ruling option (b)) | **PROVEN-LIVE (2026-08-03)** | `scripts/CHANGELOG.md` v1.23.0 + `REPORT.md`. **Proven by HTTP against the real URL, not from a pod's filesystem.** *Scenario A:* a real push to `main` without moving the tag left the served script **byte-identical** (`sha256 2f859555…`), and a marker comment planted in that very commit was **absent** from the served bytes, while the website tree advanced to the new commit in the same observation — both halves of the split in one measurement. *Scenario B:* moving the tag published in **~40 s** (`sha → ea2b4aa9…`, marker present) and moving it back restored **exactly** the pre-publish sha. `https://felhom.eu/` returned 200 throughout. *P-A, measured BEFORE the manifest was touched because the model rests on it:* git-sync v4.4.0 follows a tag **and notices a moved one** (`update required … local: remote:` → `updated successfully`) | **Two syncs, deliberately: the WEBSITE still tracks `main`.** Pinning both would turn every copy edit into a release, which makes the release meaningless and the site slow to fix. **Publish** = cut `installer-v` + bump the manifest `--ref` + sync; **roll back** = move the tag back, which needs **no ArgoCD sync and no deploy**. Continuity is structural rather than lucky: both trees are seeded by init containers so a fresh pod is not Ready until the tag is checked out, and `maxUnavailable` rounds to 0 on one replica, so a failed scripts-init leaves the OLD pod serving — the failure direction is *no update*, never *no `/scripts/`*. **The URL never carried a ref**, so the bootstrap script and the hub's day-0 command follow the tag with no edit and **no hub change**. The installer's own sixteen run-time fetches are a separate channel pinned to the AGENT's version (**R-183**), because they are the agent's configs and not this repo's — leaving them on `main` would have made the whole change cosmetic | | Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; **universal secret-free / operator-bind** mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE on TWO different boards** (N100 2026-07-18; HP t740 2026-07-21) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — the full chain on real metal in a single pass:** the generic reusable pairing ISO (v1.20.0, `--loader mkimage`, SB off) booted the cheap AMI board that F1 had blocked, installed unattended, and the box **self-registered as an unclaimed appliance at 16:17:14 — the same second it first booted** (`appliance_registrations` id=3), then bound → credential-delivered → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware.** Prior nested legs: slice A `SPIKE-baremetal-iso-2026-07-16` (build gate, disk-filter fail-safe, stub→host-install fetch); slice B RUNBOOK-B (shim boots+installs OVMF SB-enforcing + SeaBIOS; `--loader mkimage` boots+installs SB-off; mkimage SB-enforcing **FAILS** `Access Denied`; surgery byte-identical); **slice C (2026-07-17): the GENERIC secret-free ISO** — box self-registers as an unclaimed appliance (`POST /api/v1/appliance/register`, one-shot poll delivery, 404-no-oracle — all live-verified through the public ingress), operator binds on the Hosts page, hub delivers credentials once; bootstrap harness proves direct(zero-appliance-calls)/pairing/delivery; artifact proven secret-free (baked env = hub URL only) | **F1 loader caveat:** `--loader mkimage` fixes cheap AMI firmware that can't USB-boot the stock GRUB — UNSIGNED → **Secure Boot must be OFF**; default `shim` keeps SB. **Slice C bind is operator-password-gated** (CC stages, Viktor binds) → the live boot→register→bind→day-0 composition + physical N100 boot fold into the supervised rehearsal (R-1). Customer-facing **self-bind page = R-27 slice 1 SHIPPED (hub v0.66.0, 2026-07-17)** — see the dedicated self-bind row | **Second board, 2026-07-21 (demo-hp, HP t740 / Ryzen V1756B / AMI M42):** the whole chain ran on virgin hardware in one pass — armed install → self-registration as an unclaimed appliance → operator bind → day-0 → running guest 9201 + agent 0.92.1 as `demo-hp-bb76ea`. **The shim loader booted with Secure Boot ENABLED**, which retires the assumption that Felhom installs need SB off — that was an N100-firmware workaround. The exact-serial disk filter took the system SSD and left the box's 1TB NVMe untouched/unenrolled on hardware it had never seen. Two failures filed rather than smoothed over: **R-59** (no DHCP → the installer baked a static fallback instead of aborting) and **R-61** (baked root password unknowable → no console access). | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`.rootpw.txt`) | scripts v1.24.0 (`scripts/iso/felhom-bootstrap.sh` `network_gate`/`sweep_nics`, `build-felhom-iso.sh` rootpw emission) | **PROVEN-LIVE (nested drill — nested ≠ metal: metal proof rides the next real multi-NIC install)** | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` — dead-NIC install from the virgin v1.24.0 ISO baked the 192.168.100.2 fallback (WITH a dead default gateway), the R-59 screen painted on the console (screendump captured), and after the cable move the box swept to the working NIC, re-leased and **self-registered at the hub unaided in under a minute**; the drill also caught + fixed the stale-fallback-route trap (flush before the bounded dhclient) and verified the emitted rootpw against the installed box's shadow hash | R-59 ships as a first-boot gate, not an install-time abort (recorded deviation — the fallback is the auto-installer's own, initrd hook out of scope); sweep is structurally first-boot-only (`state.json` gate + unit done-flag condition); a box past install-start gets the screen but its interfaces are never touched | | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. **Deliverability (R-4), gmail half DONE 2026-07-18:** the rehearsal's claim email was the first sent under the tightened DMARC `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu remains Viktor's open half.** (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index e045ace..7338c50 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -13,9 +13,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-88b** | ~~`/backup/due` cannot say *unknown*~~ | **SHIPPED + PROVEN-LIVE** (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | `age_state=unknown` captured on real hardware during a deliberate ep0 outage; controller deferred, **zero app stacks stopped** | — | | **E-2d** | **Prove E-2 on a fresh VM** — a real `felhom-host-install.sh` 1.22.0 run, Case B naturally, a claimable customer, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **CLOSED — PARTIALLY PROVEN** (2026-07-29) | — | **C1, C2 proven** (`audits/E2D-fresh-vm-2026-07-29.md`); **C3, C4 proven live** (`audits/SESSION-C-2026-07-29.md`); **C5 FAILED → R-116** — the gate fires and an alarm reaches the hub, but it is the generic event, so the alarm and its recovery cannot be paired. **R-116 is the single named open leg**; per the Session-C runbook §9, decided in advance, a failed claim closes the item as partially proven rather than triggering a re-run. Both audits carry the full record — the `local-lvm` fence, the ISO/PAIRING derivation, the Phase 0 answers, the per-claim observables and the teardown evidence — and are the place to read it, not this cell. **The arc's actual definition of done is R-106 + R-109, R-108 and D5**, none of which this detour touched | CC | | **R-94** | ~~A hand-synced version constant drifts, and the gate that would catch it is never run~~ | **CLOSED — SHIPPED** (hub v0.87.0, 2026-08-02) | — | **All three legs closed.** **(a) closed by DELETION, not derivation** — deriving is not achievable honestly: the Setup command fetches `felhom-host-install.sh` at RUN TIME from a website that git-syncs `main` every 30 s (R-110), so no build-time value in the hub can be true, and a number that is wrong carries a version number's authority while being a guess. The const, the `pageData.ScriptVersion` field, its assignment and the rendered label are gone; a NOTE stands where the const was so it is not helpfully re-added. **(b)** `hostinstall_gates.py` gate 1 INVERTED — it now asserts the hub carries **no** host-install version literal, in six code shapes across every `.go`/`.html` under `hub/`; and the gate is now invoked, by `scripts/repo_gates.py` and the pre-push hook (→ R-29). **(c)** the tautological `render_test.go:219` assertion is deleted, not replaced — there is no version to assert. It was demonstrated PASSING with the const at `9.9.9` while the script was 1.22.0. The label had been wrong for 19 days (since 2026-07-14) | — | -| **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **READY (S)** — operator ruling taken 2026-08-03: **option (b), tag-tracked** | — | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly. **OPERATOR RULING 2026-08-03 — option (b) chosen: the publish channel moves from `main`-tracking to a TAG.** Publishing becomes *moving the tag*, and rollback becomes *moving it back* — the property `main`-tracking cannot have at any price. Recorded here, **not built this session, by instruction**. **The ruling carries a condition that decides whether the fix works at all: it must cover BOTH channels.** (i) the nginx-served `/scripts/` git-sync (`manifests/webpage.yaml`, `--branch=main`, 30 s period) that every `felhom-bootstrap.sh` fetch and every operator day-0 command reads, AND (ii) **the nine files `felhom-host-install.sh` fetches from `raw/branch/main`** (`:2072`–`:2206`), of which the hub vouches a sha for exactly ONE. Fixing only (i) leaves a tagged installer pulling nine untagged files from `main` at run time — a staging story that is false in the place it matters most, since one of those nine (`felhom-backup-target-apply`) is installed **0755 into `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n`. **Exposure is still zero** (no boxes installing), which is exactly why it stays cheap. Now CC's to build | CC | +| **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **CLOSED — SHIPPED** (installer v1.23.0, 2026-08-03) | — | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly. **OPERATOR RULING 2026-08-03 — option (b) chosen: the publish channel moves from `main`-tracking to a TAG.** Publishing becomes *moving the tag*, and rollback becomes *moving it back* — the property `main`-tracking cannot have at any price. Recorded here, **not built this session, by instruction**. **The ruling carries a condition that decides whether the fix works at all: it must cover BOTH channels.** (i) the nginx-served `/scripts/` git-sync (`manifests/webpage.yaml`, `--branch=main`, 30 s period) that every `felhom-bootstrap.sh` fetch and every operator day-0 command reads, AND (ii) **the nine files `felhom-host-install.sh` fetches from `raw/branch/main`** (`:2072`–`:2206`), of which the hub vouches a sha for exactly ONE. Fixing only (i) leaves a tagged installer pulling nine untagged files from `main` at run time — a staging story that is false in the place it matters most, since one of those nine (`felhom-backup-target-apply`) is installed **0755 into `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n`. **Exposure is still zero** (no boxes installing), which is exactly why it stays cheap. **CLOSED 2026-08-03 — installer v1.23.0, and the both-channels condition was HONOURED, but not in the shape the ruling assumed.** **The spec's mechanism for channel 2 rested on a factual error, found by reading the code:** the run-time fetches are **sixteen, not nine**, and they come from the **`felhom-agent`** repo, not from `felhom.eu` — so no tag on this repo could ever have covered them, and `§8.1`'s *"derive the tag from `SCRIPT_VERSION`"* was unimplementable for them. Operator ruled on the alternative: pin them to **the agent version being installed**, which the installer already resolves from the hub manifest and already sha-verifies. `fetch_raw` now fetches `raw/tag/v$ART_AGENT_VER/`, with **no fallback to a branch** — a vouched version whose tag is missing fails loudly, because a silent fallback is the appearance of control with none of it. That also fixed a latent skew → **R-183**. **Channel 1:** `manifests/webpage.yaml` split into TWO git-syncs — the website still tracks `main` at 30 s (a copy edit must never need a release), `/scripts/` tracks **`installer-v1.23.0`**. Both trees are seeded by init containers, so a fresh pod is not Ready until the tag is checked out and there is no 404 window; `maxUnavailable` rounds to 0 on one replica, so a failed scripts-init leaves the OLD pod serving — the failure direction is *no update*, never *no /scripts/*. **Channel 3 needed no change, and that is recorded rather than left as a silence:** `https://felhom.eu/scripts/felhom-host-install.sh` never carried a ref — the ref lives in the manifest — so `felhom-bootstrap.sh` and the hub's day-0 command follow the tag with **no edit and no hub version bump**, which is why §1's no-bump rule was never in tension. **PROVEN LIVE, both scenarios, by HTTP against the real URL.** *P-A (measured before the manifest was touched):* git-sync v4.4.0 follows a tag **and notices a MOVED one** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period. *Scenario A:* a real push to `main` without moving the tag — the website tree advanced to the new commit while the scripts tree stayed put, the served sha stayed **byte-identical (`2f859555…`)** and a marker comment deliberately planted in that commit was **absent** from the served URL. *Scenario B:* moving the tag published it in ~40 s (sha → `ea2b4aa9…`, marker present), and moving it back rolled it back to **exactly** the pre-publish sha with the marker gone; `https://felhom.eu/` returned 200 throughout. **Gate 6 in `hostinstall_gates.py`** pins all of it structurally with no network, so it stays in `--fast` and runs in CI: no `raw/branch/` ref in the installer, `fetch_raw` still pinned, the manifest still splitting tag-vs-main. **It deliberately does NOT assert that a tag exists for the current `SCRIPT_VERSION`** — that would go red on the very push that bumps the version, before publishing, and publishing being a separate act is the whole ruling; the same reasoning §8.4 applies to the agent gate | — | | **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` **DEFERRED LEG, AND IT RECURRED → R-115.** This row's shipped half stands and is not reopened: the bump happened, was verified, and was proven end-to-end by the E-2d install. But its own closing line — *decide whether publishing joins the release train rather than staying a remembered step* — was never acted on, and agent 0.114.0 reproduced the exact condition the same afternoon. The recurrence is filed as **R-115**, not as a reopen, because the stale-channel finding is closed while the process defect that caused it is a distinct problem with a distinct fix. | CC | -| **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **READY (M)** — operator ruling taken 2026-08-03: **mechanism (b), build-side gate** | — | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design. **THIRD INSTANCE, 2026-08-03 — and it was found by a runbook that had been told there was nothing left to do.** Agent **v0.120.0** — the agent half of the R-165 merge — was built, committed at `cd6e267`, and deployed to BOTH demo hosts, and was **never published**: `GET …/generic/felhom-agent/0.120.0/felhom-agent` → **HTTP 404** (0.119.0 → 200), and the hub manifest accordingly vouched **0.119.0**. The consequence is the sharpest yet, because installer step 5's idempotent skip requires `installed == vouched` EXACTLY: a documented-path reinstall would have **downgraded both boxes** from the merge-aware 0.120.0 to the pre-merge 0.119.0 — silently, since the current `step_grows` sets `SYSDATA_GROW=0` so 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on error) never fires and the install would have *succeeded* while proving a stack nobody ships. **R-178's own row asserted `agent v0.120.0 is live on BOTH hosts` and `no code left to write`; both were true and both were beside the point** — the gap was publication, which no one checks. Fixed in-session on the operator's ruling: `scripts/publish-agent.sh 0.120.0` (sha `a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`, round-trip GET verified) then vouched, and both reinstalls then fetched and sha-verified it from Gitea. **This is the third instance of a row that has been WAITING-ON-OPERATOR since 2026-07-29; option (b) — a gate that refuses to deploy or vouch an unpublished version — would have caught all three.** **OPERATOR RULING 2026-08-03 — mechanism (b), build-side: a gate that REFUSES to deploy or vouch a version that is not published.** It is the strongest of the four options and the only one that fails closed; (c) and (d) were reminders, and R-29's whole finding is that reminders do not hold. Recorded here, **not built this session, by instruction**; it is now CC's to build. **The third instance is the argument for the ruling and belongs inside it:** agent **v0.120.0** (the agent half of the R-165 merge) was built, committed and deployed to BOTH demo hosts while `GET …/generic/felhom-agent/0.120.0/felhom-agent` returned **HTTP 404**, so the hub vouched 0.119.0. Because installer step 5's idempotent skip requires `installed == vouched` EXACTLY, a documented-path reinstall would have **silently downgraded both boxes** to the pre-merge agent — and would have *succeeded* while doing it, since the current `step_grows` sets `SYSDATA_GROW=0` so 0.119.0's fatal `mp1` resize never fires. A gate at deploy/vouch time is the only one of the four standing between that and the operator | CC | +| **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, 2026-08-03) | — | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design. **THIRD INSTANCE, 2026-08-03 — and it was found by a runbook that had been told there was nothing left to do.** Agent **v0.120.0** — the agent half of the R-165 merge — was built, committed at `cd6e267`, and deployed to BOTH demo hosts, and was **never published**: `GET …/generic/felhom-agent/0.120.0/felhom-agent` → **HTTP 404** (0.119.0 → 200), and the hub manifest accordingly vouched **0.119.0**. The consequence is the sharpest yet, because installer step 5's idempotent skip requires `installed == vouched` EXACTLY: a documented-path reinstall would have **downgraded both boxes** from the merge-aware 0.120.0 to the pre-merge 0.119.0 — silently, since the current `step_grows` sets `SYSDATA_GROW=0` so 0.119.0's `mp1` resize (`bringup.go` 4c, fatal on error) never fires and the install would have *succeeded* while proving a stack nobody ships. **R-178's own row asserted `agent v0.120.0 is live on BOTH hosts` and `no code left to write`; both were true and both were beside the point** — the gap was publication, which no one checks. Fixed in-session on the operator's ruling: `scripts/publish-agent.sh 0.120.0` (sha `a7763d31b55b5ce75457b4dba7b06aa300325811834b0be78af4587b47110b9d`, round-trip GET verified) then vouched, and both reinstalls then fetched and sha-verified it from Gitea. **This is the third instance of a row that has been WAITING-ON-OPERATOR since 2026-07-29; option (b) — a gate that refuses to deploy or vouch an unpublished version — would have caught all three.** **OPERATOR RULING 2026-08-03 — mechanism (b), build-side: a gate that REFUSES to deploy or vouch a version that is not published.** It is the strongest of the four options and the only one that fails closed; (c) and (d) were reminders, and R-29's whole finding is that reminders do not hold. Recorded here, **not built this session, by instruction**; it is now CC's to build. **The third instance is the argument for the ruling and belongs inside it:** agent **v0.120.0** (the agent half of the R-165 merge) was built, committed and deployed to BOTH demo hosts while `GET …/generic/felhom-agent/0.120.0/felhom-agent` returned **HTTP 404**, so the hub vouched 0.119.0. Because installer step 5's idempotent skip requires `installed == vouched` EXACTLY, a documented-path reinstall would have **silently downgraded both boxes** to the pre-merge agent — and would have *succeeded* while doing it, since the current `step_grows` sets `SYSDATA_GROW=0` so 0.119.0's fatal `mp1` resize never fires. A gate at deploy/vouch time is the only one of the four standing between that and the operator. **CLOSED 2026-08-03 — both halves, no version bump (no Go code changed).** **(1) `scripts/release-agent.sh` is now THE release path**: build → **tag** → publish → **verify by an INDEPENDENT download**. It calls the existing `publish-agent.sh` rather than reimplementing it, refuses a dirty or unpushed tree, refuses to re-release an existing version (one version name must never mean two binaries), and **deliberately does not vouch** — vouching points machines at a version and stays the operator's act. `CLAUDE.md`'s raw `go build` line is replaced by it, so the documented way to release cannot complete without publishing. It also **tags**, because R-183 made the tag part of the released artifact. **(2) `scripts/check-published-versions.py`**, registered in `agent_gates.py` as **not `--fast`** — it needs network, and a push must not fail because Gitea blinked. **The CI workflow now runs the FULL set instead of `--fast`**, without which the gate would have been registered and never run: the built-but-never-wired failure this project has shipped four times. **THE INVARIANT IS NOT THE ONE THE TASK SPECIFIED, and the reason was measured (P-C), not argued.** §8.4 asked for *"the version the hub tells machines to install must be downloadable"* — the better invariant, and **CI cannot see it**: the hub's artifact manifest is **401** without a per-customer passphrase and Gitea's package **listing** api is **401** without a token, while the package **download** url and the **tags** api are anonymous. Adding an operator credential to CI is the operator's call, not a gate author's. The implemented invariant — **every `v` tag must have a downloadable package and a tag tree serving the agent's configs** — needs no credential and **catches all three recorded instances**, because the release script creates the tag and publishes in one act. **What it does NOT catch is stated rather than assumed away: the hub vouching a version that was never released at all → R-184.** **Red-proof F, MEASURED ON REAL CI and not inferred:** runs **69** and **70** are on the *same commit* `0db7766` — **success** before a tagged-but-unpublished `v9.9.9` existed, **failure** after pushing it. Same code, same workflow, one variable. Locally: the gate exits 1 naming the 404; **deregistered from the entry point** the same bad state reports `all agent gates OK` **rc=0**; restored → `CONVICTED: published` rc=1. `v9.9.9` deleted afterwards (`git ls-remote --tags` → only `v0.120.0`). **One deliberate CI failure e-mail reached the operator at ~12:5x CEST — that was this proof, not an incident** | — | | **R-116** | ~~**The drive-absent alarm and its recovery were a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`; `backup_target_absent` never fired at all**~~ | **SHIPPED + PROVEN-LIVE** (agent v0.116.0, 2026-07-30) | — | **CLOSED. The full four-event sequence, on the wire, on a fresh box** (`audits/R116-v0116-2026-07-30.md`): `backup_target_absent (error)` on detach → `backup_target_restored (info)` on return for the TARGET, and `storage_disconnected (error)` → `storage_reconnected (info)` for a NON-target drive on the same box four minutes apart. **Two matched pairs, correctly discriminated — and discrimination is proven NON-trivially for the first time**, since both prior runs had the target itself emit the generic event. Gate fired in **3 s**; all four events reached the hub, so the specific alarm, its severity, its Hungarian copy and the hub routing are now exercised end-to-end. **Over-correction PASSES with a positive observable** (0 ABSENT lines / 0 drive events over 2m14s with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking). **NARROWED by the R-117 spike (`audits/SPIKE-r117-bind-liveness-2026-07-30.md` §12), and it stands as written:** the 2 `RETURNED` lines are a genuine positive observable, so rule 3 is satisfied — but `degraded:false` over that window was read off a drive whose bind was **dead** (R-117), so the window evidences **"the gate did not over-fire"** and **NOT** **"the drive was healthy."** No other part of this row changes: every input to the pairing fix is configuration-derived (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), which R-117 does not touch. Ran on a nested PVE on **demo-hp** per `runbooks/target-selection.md` — through the **real day-0** from the v1.25.0 ISO, with the agent **installed unaided from the vouched Day-0 manifest** (published sha `b47c5c4dab641ee5…`, independent registry GET verified, manifest read back), drives enrolled through the real endpoints, device loss a real hot-detach. **THE FIX, and the ruling is the substantive part:** the mechanism was first isolated from the captured payload (`DIAG-r116-disks-payload-2026-07-30.md`) after two fixes aimed at shapes that do not occur. **Both smaller-looking options were REJECTED because they regress R-114** — `backup_target_offer.go:79` reads `BackupTarget && MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns before its `TargetAbsent` branch, so back-filling `MountPath` on the Observe row **or** flagging the registry row (whose `MountPath` is the stale unit-file value) would have told the customer the backup target is fine while its drive was gone. **R-114's correctness was resting on R-116's bug** — a coupling invisible until the payload existed. Taken instead: the Observe row gets the **guest path only** (`mount_path` stays `""`, which is true) from a new `ConfigPath` (`json:"-"`, so the cross-repo golden + key-set contract is untouched), and the union row is deduped **on guest path** — the join being CONFIGURATION (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849; 4 red-proofs each asserted to land, and red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. v0.115.0 left in place (inert, harmless). Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded. **Caveat: the drill's controller was 0.185.1 from the golden, which PREDATES R-114**, so its absent-state banner showed the old false copy — the golden being a release behind, not a regression → **R-120** | — | | **R-120** | ~~**The golden baked a controller that predated R-114 + R-112, so a FRESH box showed the customer the WRONG absent-target message**~~ | **CLOSED — golden rebaked + PROVEN-LIVE, and the class now has an ENFORCED gate** (golden 0.186.0 + hub v0.82.0, 2026-07-30) | — | **`audits/R120-golden-rebake-2026-07-30.md`.** **Half 1 — the artifact.** Golden **0.186.0** baked from `main`'s controller in the DooPlex bake fixture (overlay2 OK, **3 mounts**, FATAL 0, exclusions 0, 618 MB, upload **201**, `GOLDEN_SHA256=b760ac6a33e70700…`, token-leak grep 0, GL-1 teardown, `drill.qcow2` back to `virgin`). Three observables: **published** — anonymous GET (what the installer does) 200 / 648930639 bytes / sha identical to the bake; **vouched** — manifest read BACK; **resolved** — `Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)`. Floor **untouched** per publish-train rule 2 (`min_controller_version` still 0.156.0; it is a separate form); MinAgent left 0.113.0 as 0.186.0 declares. **Proven on a REAL day-0, not the fixture** (per the Part-1 rule now in `runbooks/target-selection.md`): VM 9402 on demo-hp from the v1.25.0 ISO → `Controller elindult (0.186.0)`. With the target detached the endpoint returned the **`TargetAbsent`** copy — *„A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod…"* — **and `offer_path` absent entirely**; the day-old read on the 0.185.1 golden had returned the false system-disk message **plus** an offer of the other drive. **Half 2 — the mechanism, operator ruling REFUSE.** hub **v0.82.0**: the gate sits in `hub/internal/web/configs.go` `handleSetArtifacts` immediately before the only write — the sole UI path to `SetArtifactManifest` — so it runs on every vouch without anyone choosing to, and it **refuses** rather than warning. Signal: `store.NewestReportedControllerVersion()` over `reports.controller_version`, **semver-compared in Go** (`MAX()` in SQL ranks 0.99.0 above 0.186.0 — a pair this fleet has shipped). Fail-open in exactly two deliberate cases: empty golden field, unknown fleet version. **NEAR-MISS RECORDED: the first draft read `guests.controller_version`, a column that exists and that NOTHING writes** — it would always have seen `""` and failed open, i.e. inert, this gate's own failure shape, one grep from shipping. 4 tests through the **production handler** over httptest (never a seam), the refusal asserting **both** the flash **and** that the manifest was not written; red-proof: deleting the block makes the stale golden vouchable again. **PROVEN LIVE on the deployed hub by re-attempting the original mistake:** vouching 0.185.1 → `HTTP 303 …flash=golden_behind_fleet` + `[WARN] artifact vouch REFUSED: golden 0.185.1 is older than the newest controller the fleet reports (0.186.0)`, and the manifest read back **unchanged at 0.186.0**. Recorded on **R-29's audit list** (`ROADMAP.md`) as the **first enforced gate** beside its three orphans, so the contrast is kept — the orphans are unchanged. Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded, exactly as `sess-e` was (and `sess-e` was deleted this run) | — | | **R-117** | ~~**A drive's guest bind becomes a DEAD MOUNT while every signal reads healthy — and it happens in TWO ways, only one of which the original framing covered.** (a) *after a detach/return*: the host raw mount heals onto the NEW device via its fs-UUID-keyed unit while the bind still names the OLD one, so the gate takes its `Return` branch and restarts the customer's apps onto a namespace that `EIO`s on every call; (b) *in STEADY STATE, no cycle at all* — a device that errors without disappearing leaves the raw mount `active`, `BoundUnderParent` `true` and the drive never `Disconnected`, so **the gate produces no action and NOTHING is emitted on any channel**~~ | **SHIPPED + PROVEN-LIVE** (agent **v0.117.0**, 2026-07-30) | — | **CLOSED. `audits/R117-v0117-2026-07-30.md`.** `BoundUnderParent` gains a THIRD term at both /disks sites: `bindLiveness` reads `/proc` only and requires (a) **the bind names the same device as the raw mount** and (b) **the filesystem has not aborted** (`shutdown` **or** `emergency_ro`, both measured). **BOTH CHECKS ARE LOAD-BEARING and this is the substantive part:** R-117 was filed as a detach/return defect, but a device that fails WITHOUT disappearing gives the identical all-signals-healthy state with the **devnos EQUAL** and the drive never `Disconnected`, so the gate emits nothing at all, indefinitely (R-117a) — the device comparison alone cannot see it, and a P1-only fix passes every payload test (red-proof RP3 exists for exactly that). **THREE states, never a bool:** `{Unknown, Live, StaleDevice, Aborted}`, `Unknown` is the zero value, and every caller reads `Usable()` where unknown counts **PRESENT** (absent stops a customer's apps — the `newestArchiveOn` trap). **NO NEW RECOVERY PATH:** `AttachDrive`'s normalize leg already did the repair and three call sites already invoked it (20 s ticker, agent startup, and **the controller's `Return` branch BEFORE `restartStacks`**); all three were defeated by `if n == 1 && GuestSeesMount(...)` logging *"fully live, no-op"* about an EIO namespace. **RULING (asked for, given, flagged for overrule):** `StaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, guest never restarts — init PID identical); `Aborted` ⇒ **quiet no-op and SURFACE**, because a re-bind lands on the SAME dead superblock and this runs every 20 s = an infinite silent retry that masks the state. No operator decision required: it routes an already-broken state into the **existing** gate, event types and Hungarian copy — no new customer-facing concept — and the alternative is apps writing documents into a filesystem that rejects every write. **ORDERING TRAP caught by a test:** abort-first classifies the real return state as aborted (its stale bind carries `shutdown` too) and refuses the repair **while still reporting correctly**, so the abort flag is read off the RAW mount in the stale case. **LIVE on demo-hp** (brought 0.113.0 → 0.117.0 first — see R-121): RETURN `raw 8:32 / bind 8:16 shutdown` ⇒ `stale-device`, usable **false**; IN-PLACE `both 252:11 emergency_ro`, raw unit still `active` ⇒ `filesystem-aborted`, usable **false**; healthy ⇒ `live`; **340–497 µs**. **No block I/O proven by strace** (only `/proc/self/mountinfo`, **0** statfs) — the Part 1 `CLAUDE.md` fence applied to its own first consumer. **No regression through the REAL pipeline:** `GET /disks` with the controller's own credential shows the live backup-target drive `bound_under_parent=True`, with 32 gate lines in 3 min as the positive observable and zero spurious transitions. Tests **849→863**, 29/29 green, **6 red-proofs each verified to land** — and **RP1 failing to fail exposed a HOLLOW test**: the aborted fixture used a `/dev/mapper` device, for which `RoleForStorage` derives `role=system`, and a system row never runs the conjunction, so it reported false by DEFAULT and no mutation could fail it. Fixtures now assert the production row shape first. Teardown all 3 layers; hub layer = the vouched manifest, **retained** (it is the product, not scratch). **NOT covered:** the stale-bind repair on hardware — `StablePathForRaw` hardcodes the live parent, so it would write into guest 9201's namespace (R-117h); and sustained-load behaviour, still unmeasured. Follow-ups **R-117g** (no guided recovery for an aborted fs), **R-117h** (parent dir not test-seamable), **R-121** | CC | @@ -92,7 +92,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-174** | ~~**The app-stop guard's crash recovery started apps onto MISSING drives — a regression in v0.189.0 code.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0, 2026-08-02) | — | **Found by REVIEW on 2026-08-02, in code shipped 2026-08-01, and closed the same session — R-171 one path over.** `appStopGuard.SetStarter(stackMgr)` handed `Recover` the RAW stack manager, whose `StartStack` has no drive gate, and `Recover` runs **at startup** — exactly when an external drive may not have come back. So: a backup stops an app, the box loses power, the drive does not remount, and the app is started on a missing drive. The rule was not new — the API's own `startGatedByMissingDrive` already refused this to the customer; the guard bypassed it. **`bootDriveGate` could NOT be reused whole**, and the reason is recorded in the code: its holder #2 reads `bootAppStopGuard.HeldStacks()`, which during `Recover` is **the guard's own marker** — it would refuse every recovery it was meant to perform — and holders #1/#2 read package-level vars assigned AFTER `Recover()` runs, so a whole-gate reuse would be correct only by accident of nil-safety. Holder #3 is extracted into a shared `driveStartGate` with **two callers, one implementation**, and `TestBootDriveGateAndAppStopShareTheDrivePredicate` pins the delegation. **A REFUSAL IS NOT A FAILURE:** new `ErrStartRefused` + a `Refused` bucket — both keep the marker, only `Failed` alarms, because routing a deliberate hold into `NotifyBackupFailed` (customer-enabled by default) is the very R-171 false alarm this fixes. `main.go` guards on `Alarming()`, not `!= nil`, and the pre-existing seam test was TIGHTENED to require it. **Live on 9201, both directions:** drive held unmounted → `refusing to restart "calibre-web" … drive /mnt/felhom-drives/hdd_1 is not a live mountpoint`, marker retained byte-identical, zero containers started, `not alarming`; drive returned → `restarted calibre-web`, marker CLEARED. **ID established free:** `grep -ro "R-174\b" documentation/ *.md` → 0 hits | — | | **R-175** | ~~**`07-backup-architecture.md` §7.5 states ONE box's size bound as if it were the fleet's.**~~ | **CLOSED — FIXED 2026-08-03** (same pass as R-165) | — | **Measured, not inferred** (`audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1: `pct config 9201` on both hosts). Independent of the merge — the sentence is wrong today and will be wrong differently after R-165. **The fix is to state the bound as a FUNCTION of `mp1`, not a constant**, and to say which box any quoted figure came from. Same class as the comment-asserting-an-invariant rule: a doc stating a fleet-wide number that only one machine satisfies reads as settled and is not. **ID established free:** `grep -ro "R-175\b" documentation/ *.md` → 0 hits **FIXED.** §7.5 gained a **7.5.1** which (a) states plainly that the bound is a FUNCTION of `mp1` and applies only to a box still on the split layout, naming all three real shapes, and (b) records that the ceiling itself has been removed by R-165 for boxes built from golden ≥ 0.192.0. Fixed in the same pass as the merge rather than filed and forgotten, because the section would otherwise have been wrong in two ways at once | CC | | **R-176** | **Two prerequisites for the R-165 merge are UNMEASURED, and both are cheap.** (a) Whether a **pre-merge archive** (carrying `mp1`) restore-tests cleanly into a **merged-layout** guest — reading `mountParity` (`felhom-agent/internal/reconcile/restoretest.go:347`) says it should, because the restore recreates `mp1` from the archive so archive and restored guest agree; **that was reasoned from source and never executed.** (b) The in-place per-box migration (move `/felhom-data` onto `mp0`, drop the slot, verify) has **never been rehearsed even once**, so "is the box restorable at every point of it?" is currently unknown | **(a) ANSWERED 2026-08-03 (P1: PASS). (b) NOT REQUIRED — operator ruling: every node is reinstalled, none migrated** | blocks R-165 landing safely | **Filed because this project's own record is that FOUR production designs specced against unvalidated mechanisms were all wrong** — which is exactly why R-165's own spike refused to design. Both are one command on a **Tier-0** box (D-d: both demo boxes are disposable). (b) is only required work if Peti's box turns out to need migrating rather than reinstalling — the hub cannot answer that (M5: `peti-felhom` exists as a customer with **no host in the register**), so it is the operator's input. **ID established free:** `grep -ro "R-176\b" documentation/ *.md` → 0 hits **UPDATE 2026-08-03.** **(a) is measured and passed** — `audits/SPIKE-r165-phase0-2026-08-03.md` P1: a real pre-merge archive (`mp0+mp1`, confirmed from its own vzdump log) restore-tested on demo-hp, `pass: true`, `mount_parity: ok`, 84 s, with `mountParity` untouched. One limit stated rather than glossed: it ran with the pre-merge agent because the merged one did not exist yet, and the comparison is archive-vs-its-own-restore which never consults the host layout — **re-run it once against agent v0.120.0**, which is one command. **(b) is withdrawn, not deferred:** the operator ruled that every node is REINSTALLED rather than migrated in place (both demo boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed in a few weeks), so the in-place migration rehearsal has no consumer. Recorded explicitly rather than silently skipped | CC | -| **R-182** | **The reserve re-alerts on every status refresh, so a full disk pages the operator on a timer rather than once.** `captureAllRecoveryUnits` has TWO callers: the backup run (`runDBDumpsInternal`), which since R-181 holds a per-run admission memo so a refused app alerts exactly once — and `GetFullStatus` (`backup.go:967`), the **periodic status refresh**, which calls it with no run scope. Outside a run the verdict is decided fresh, so every refused app pushes another `recovery_unit_capture_failed` (severity `error`) on every refresh | **READY (S) — NEW 2026-08-03** | — | **Measured live, not inferred.** demo-hp 2026-08-03: the backup run at **08:59:46** produced exactly one alert per app (2 apps → 2 events, correct); a **second identical pair fired at 08:59:59**, thirteen seconds later, from the status refresh. Under a sustained full disk that is a repeating operator page for a condition already reported. **PRE-EXISTING, NOT INTRODUCED BY R-181 — stated because it decides the priority.** v0.192.0's `captureAllRecoveryUnits` called `unitFloorBlocked` → `unitNotify` on the same path with the same frequency; R-181 changed neither caller. It is filed now because R-181's live proof is the first time anyone watched the event stream closely enough to see it. **Deliberately NOT fixed in the R-181 task**, whose §9.2 mandates minimal changes and whose §15 routes observations here; the fix changes alerting semantics on a path that task did not own. **The mitigating claim should be checked before the fix is scoped:** `backup.go`'s own comment says *"NO CONTROLLER-SIDE COOLDOWN — the hub owns cooldown"*. If the hub genuinely dedupes `recovery_unit_capture_failed`, this is log noise and an S; if it does not, it is a real repeated page. **That is exactly the shape of a comment asserting an invariant with no test pinning it** (`CLAUDE.md`'s own table), so verify it at the hub rather than trusting it. **Fix shape (not implemented):** give the status-refresh sweep its own admission scope, or make the non-run path decide without alerting — the run path is already correct and must not be disturbed | CC | +| **R-183** | **A fresh install fetched the vouched agent BINARY and its sixteen CONFIG files from two different refs, and nothing compared them.** `felhom-host-install.sh` resolved the agent version from the hub manifest and sha-verified the binary — then took `felhom-agent.service`, `felhom-agent.sudoers` and fourteen more from `raw/branch/main`, i.e. whatever the agent repo's tip happened to hold at that second. One install, two refs, no comparison | **CLOSED — SHIPPED** (installer v1.23.0, 2026-08-03) | — | **Found while implementing R-110, by reading `fetch_raw`'s call sites rather than the spec's description of them** — the task said nine files from `felhom.eu`; they are **sixteen** and they come from **`felhom-agent`**. **Why it is a defect and not only untidiness:** these files are the agent's own operating surface — its systemd unit, its sudoers, its guarded wrappers — and `configs/felhom-backup-target-apply` is installed **0755 into `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n`. A config newer than the binary is a root-executed artifact the vouched version was never tested against. **Not hypothetical in shape:** the fleet has shipped exactly this class before, where a config and the code that reads it moved independently. **Fixed by pinning to the agent version the install is already committed to**, on the operator's ruling: `raw/tag/v$ART_AGENT_VER/`, resolved from the hub manifest that the binary's sha is already checked against — so binary and configs now come from ONE ref by construction. **No fallback to a branch**, deliberately: a missing tag dies loudly rather than quietly serving `main`. Pinned by `hostinstall_gates.py` gate 6 (no `raw/branch/` anywhere; the `$ART_AGENT_VER` pin still present), red-proofed by reverting one of the sixteen and by removing both assertions. `felhom-agent` now carries `v` tags (`v0.120.0` created retroactively at `cd6e267`, the commit the published binary was built from; `configs/` is byte-identical there and at `main`, so nothing depended on the choice) and `release-agent.sh` creates them as part of releasing | — | +| **R-184** | **Nothing prevents the hub from vouching an agent version that was never released.** The R-115 gate proves every RELEASED version is installable, but it works from git tags — so a hub artifact-manifest entry naming a version with no tag and no package is invisible to it. The installer would then die at step 5 on a virgin machine, as root | **READY (S) — NEW 2026-08-03** | — | **Filed BECAUSE the R-115 gate deliberately does not cover it, rather than leaving the gap unstated.** CI cannot check it: the hub's `/api/v1/artifacts/` answers **401** without a per-customer retrieval passphrase and Gitea's package **listing** api answers **401** without a token (both measured 2026-08-03, P-C), so a credential-free gate can ask *"is this version installable"* but never *"which version is vouched"*. **Two shapes, and the second is better:** (a) give CI a hub credential — expands what CI can reach, and is the operator's call not a gate author's; (b) **validate at vouch time, in the hub**: the operator UI's Day-0 artifact form refuses a version whose package is not downloadable. (b) fails closed at the moment of the decision, needs no new credential anywhere, and puts the check where the mistake is actually made. **Exposure is low and should be said so:** vouching is a deliberate operator action against a version they have just released, and R-115's release path now makes released-but-unpublished nearly impossible. This is the residue, not the main risk | CC | +| **R-182** | **A full disk tells the operator about ONE app and silently swallows every other app's refusal for an hour.** The hub's operator cooldown key is `customerID + ":" + eventType + cooldownTierSuffix(details)` (`hub/internal/notify/dispatcher.go:268`). `recovery_unit_capture_failed` carries **`app`** in its details and **no `tier`**, so the suffix is empty and the key contains **no app identifier**: the first refused app's alert takes the 1-hour slot and the second app's is dropped — and dropped BEFORE `LogNotification`, so it leaves **no row on any channel**. It cannot even be audited after the fact | **READY (M) — RE-SCOPED 2026-08-03, and the direction REVERSED** | — | **FILED AS THE OPPOSITE DEFECT AND THE MEASUREMENT OVERTURNED IT.** It was filed 2026-08-03 as *"the reserve re-alerts on every status refresh"* — too MANY alerts — from the controller-side observation that a second push followed 13 s after the first. That was the sending end. **Measured at the receiving end** (hub `notification_log` + `events`, read from a copy taken WITH its `-wal`, freshness confirmed by the newest row post-dating the session): **9 events received today → 2 operator e-mails sent.** **06:40:03** privatebin AND opengist both refused → **opengist e-mailed, privatebin's alert has no row at all**. **08:59:46/47** opengist AND privatebin both refused → **privatebin e-mailed, opengist's absent**. **08:59:59, 09:03:00, 09:07:06** → **no operator row whatsoever**, all inside the 1-hour cooldown opened at 08:59:47. So the controller pushing repeatedly is not the defect; the hub emitting at most one operator e-mail per customer per hour is, and the loser is silent. **This is R-97a's failure mode exactly, in a second event type.** That row's own comment states it: *"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20 for the whole hour"*. `cooldownTierSuffix` was written NARROW on purpose — empty unless the producer sends a `tier` — so no existing type's behaviour changed; `recovery_unit_capture_failed` simply never opted in. **CORRECTION OWED, and it is the reason this was worth measuring:** the 2026-08-03 R-181 report said *"one `recovery_unit_capture_failed` per app, HTTP 200"*. That was **true of what the CONTROLLER pushed** and would be read as *the operator was told about each app* — which is **false**. The distinction between an accepted event and a sent e-mail is the whole of this row. **Fix shape (NOT implemented — Part 0 was investigation only, by instruction):** let the producer opt into a per-app cooldown key, the way R-97a let the whole-guest producer opt into a per-tier one — the narrow mechanism already exists and needs no widening. **And a suppressed operator alert should leave a `skipped` row rather than nothing**, or this class stays undiagnosable from the hub's own records | CC | | **R-181** | **The capture floor guards the cheap leg and not the leg that fills the volume — and its refusal message asserts an invariant the code does not provide.** B2 (controller v0.192.0) is recorded on R-165 as the deliberate replacement for the bulkhead the `mp1` partition used to give. It is consulted in exactly one place — `m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which writes a manifest and a compose copy: **a few KB.** The leg that writes the bulk, `runVolumeDumps` (`backup.go:535`), has **no floor check at all** — its gates are protected-stack, volume-less, disconnected, decommissioned — and it runs FIRST, by design (*"MUST run before captureAllRecoveryUnits so the manifests enumerate the fresh tars"*, `backup.go:483`). So the write that fills the filesystem is unguarded, and the floor then refuses the write that would have cost almost nothing. **Second limb: the refusal message is false.** `recovery_unit.go:331` prints *"the previous unit is untouched and NOTHING was deleted"*. Nothing was deleted — true. Untouched — **measured false**: privatebin's `volume-dumps/privatebin_privatebin_data.tar` went `26c546c2…` → `b538ab89…` and opengist's went **182,272 B → 2,147,666,432 B**, both rewritten by the earlier leg, while each unit's `manifest.json` kept `created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so a unit's payload can be swapped under a stale descriptor and **nothing in the unit can detect it** | **CLOSED — SHIPPED** (controller v0.193.0 + v0.193.1, 2026-08-03) | unblocks **R-165** | **FIRST LIVE FIRING OF B2, and it is why the runbook asked for one.** Proven on demo-hp 2026-08-03 06:40:03 on a box reinstalled from the merged golden (R-178). Method: a real 2 GiB file in opengist's data volume, then `fallocate` to bring the filesystem to 96 % used / 3.0 GiB free — both floor terms deliberately still clear, so the run started. **The `fallocate` instrument was proven before use** (5 GiB moved guest `df` 977M→6.0G while thin-pool `data_percent` stayed 29.03 → 29.03: zero blocks allocated), because demo-hp's thin pool is 53.93 GiB and a real fill to 97 % of a 70 G volume would have exhausted it and corrupted every guest on the box including the `drill-r50` fixture. Sequence observed: opengist's volume dump wrote **2.0 GB unguarded** → free fell to 1.0 GB → **both** apps' recovery-unit captures were then REFUSED on the `1.0 GiB free` term, each pushing `recovery_unit_capture_failed` (severity `error`) to the hub, accepted HTTP 200. **What DOES hold: it refuses per app rather than aborting the run, it never deletes, and the alert reaches the operator.** **Fix shape, not written this session by design (§7 of the runbook):** the floor belongs before the write in `runVolumeDumps` too, the message must stop claiming what the earlier leg has already falsified, and per `CLAUDE.md` *"a comment asserting an invariant needs a test pinning it"* the pinning test must assert the **consequence** (after a refusal, is the previous unit's payload byte-identical?) and not the mechanism. **Class:** the sixth entry in `CLAUDE.md`'s own table of shipped guarantees the code did not provide — found, as four of those were, only on live hardware. **CLOSED 2026-08-03 — controller v0.193.0 (`fef07c3`) + v0.193.1 (`6c43bf6`), proven live on demo-hp.** **The fix is ONE admission verdict per app per run** (`internal/backup/admission.go`), taken before that app's FIRST write and consulted by all three legs — the three write under one per-app root (`appbackup.RecoveryUnitPath`), which is exactly why one verdict can honestly cover them. **Decided LAZILY at the app's first write, never once at run start**: app A's dump can put app B under the reserve, so a run-start verdict would read a disk that no longer exists — the same class of mistake one level up. **Never re-decided between an app's own legs** (that IS the split this closes) and **reset per run**. Placed ahead of `DumpAppVolumesSafe`, which stops the stack as its first act, so a refused app is never bounced; placed AFTER the volume-less check, which has no write to gate. Exactly ONE operator alert per refused app per run. Leg order unchanged. **The floor is now SIZE-AWARE**, which is the second half of the defect: it asks whether THIS app's write would cross the reserve, not only whether the filesystem is already below it — the term whose absence admitted an app at 96% and then let it write 2 GB. Estimate = the app's previous `.sql`+`.tar` on disk; **no history → headroom-only** deliberately, or the first backup becomes the one that can never happen, and the alert says so. **A container-based `du` was MEASURED and rejected, not assumed**: 66 timed runs on demo-hp guest 9201, **median ~355 ms/volume (341–404)** on volumes holding tens of KB — the cost is container start-up, not the walk. Decisive on top: `docker run` needs the writable layer, so the instrument can fail under exactly the pressure the reserve exists to handle; and the previous-dump estimate measures the ARTIFACT that will be written rather than the live volume. **THE MESSAGE WAS NOT WEAKENED — the behaviour moved so the wording became true**, and it is checked by sha256 tree fingerprint, not by reading the log line (which is what lied). **LIVE PROOF, demo-hp guest 9201, the same method that found it.** The `fallocate` instrument was RE-PROVEN on the rebuilt box before use (guest `df` 1.2G→6.2G on a 5 GiB step while thin-pool `data_percent` stayed **36.83 → 36.83**: zero blocks allocated), because a real fill of a 70 G volume would exhaust the 53.93 GiB pool. **Headroom term @ 08:59:46** — 906 MB free / 99%: both apps refused, **`TREE_SHA` 111d1760c18d3440f700634ab325f8b8 IDENTICAL before and after** (10 files, incl. opengist's tar still at 182,272 B — R-181's own 'before' figure), **no `Stopping for safe volume dump` line at all** (it is present in the 08:58 baseline, which is what makes its absence evidence), 0 volume dumps, one `recovery_unit_capture_failed` per app HTTP 200. **Freed and re-run @ 09:01:33** — both captured normally. **SIZE term proven separately @ 09:03:00**, reproducing the original sequence with a real 2 GiB file in opengist's volume (its previous tar then **2,147,666,432 B**, the exact live figure) and the filesystem at **91% used / 2.9 GB free — both headroom terms deliberately clear**: opengist refused `(size)` — *"this app's last backup was 2.0 GB and writing it again would cross the reserve"* — while **privatebin was ADMITTED and dumped normally**, proving the term is per-app and not a global halt. **Teardown complete**: fill removed, planted file removed, `pct fstrim 9201` returned 67.5 GiB, pool **29.43%** (below the 36.83% baseline), tree byte-identical to the pre-test fingerprint. **v0.193.1 shipped in the same session**, found by this very proof run: the estimate was rendered fixed to 2-decimal GiB, so opengist's real **178 KB** printed as `estimated 0.00 GiB write` — which reads as *no estimate was available* and is the opposite of what happened. Rendering moved to `humanizeBytes`; arithmetic unchanged. Re-verified live: `estimated 178.0 KB write`. **11 new tests + 4 red-proofs**, each demonstrated failing then restored: both dump-leg gates removed (= v0.192.0) → Scenario A red with the tree shown changing; the size term removed → Scenario D red; a prune injected into the refusal path → Scenario F red; the floor moved above the warning band → Scenario G red. **Recorded honestly: the specified Scenario-F mutation (remove the reserve entirely) did NOT turn F red** — removing it makes every app write, which overwrites and adds but deletes nothing, so a deletion-watching test correctly stays green; the prune mutation is the one that proves the assertion. The DB leg cannot run without Docker, so its gate is pinned by an **AST walk** of `backup.go` asserting `admitApp` precedes `DumpOne` — `strings.Contains` is insufficient, a commented-out call still contains the string. **§3's correction CONFIRMED in passing and not chased**: `restore_points.go:57-59` takes the manifest mtime then `newestArtifact` over `.sql` and `.tar`, so the newest of the three wins — the restore point does NOT show a stale timestamp. **New finding from the live run → R-182.** | — | | **R-180** | **`--archive-storage` is accepted without checking the agent's token will ever be granted on it, and the failure lands at step 8/8 — after the root@pam password has already been rotated.** `felhom-host-install.sh` validates the archive storage EXISTS (`pvesm status --storage`, `:1583`) and that the golden volid RESOLVES on it (`:1661`), both in pre-flight. It never checks that storage against the ACL set it is about to grant, which is the fixed default `local local-lvm felhom-pbs` (`--acl-storages`, which `runbooks/day0-install.md` tells the operator **not** to pass). A storage outside that set therefore passes every pre-flight gate and dies at the last step | **READY (S) — NEW 2026-08-03** | — | **Hit live on demo-hp 2026-08-03** during R-178 Phase A, self-inflicted and therefore a clean demonstration: the golden was staged on `felhom-backup` (the enrolled NVMe, where the box's vzdumps live) and `--archive-storage felhom-backup` passed. Pre-flight passed; steps 1–7 ran; step 8 returned `reconcile: bring-up restore: proxmox: POST /nodes/felhom-host/lxc -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)`. **The cost is the ORDER, not the error** — by the time it fires, step 2 has minted the PVE token, step 4b has **rotated root@pam and vaulted it** (so the old console password is already dead), and step 5 has installed the agent. Recovery was `--resume` after moving the golden to `local`, which worked cleanly. **This is statically checkable in pre-flight**: `ARCHIVE_STORAGE ∈ PVE_STORAGES` is a one-line assertion over two variables both known at `:1583`. Same class as R-29 — the checkable thing that nothing checks | CC | | **R-179** | **`--uninstall` leaves the NAS network-storage systemd units behind, with the automount in `failed` state and the parent bind still mounted.** The teardown's residue-diff provenance (`day0-install.md` Part E: *"a full-filesystem diff against the pre-install baseline showed zero Felhom-named leftovers"*) is from **v1.9.1**, which predates the NAS network-storage feature. A box that has ever had a network share configured keeps `/etc/systemd/system/mnt-felhom\x2ddrives-.mount` and `.automount` after a full uninstall | **READY (S) — NEW 2026-08-03** | — | **Observed on demo-hp 2026-08-03** after `--uninstall --vmid 9201`: `mnt-felhom\x2ddrives-Felhom\x2dShare.automount` **loaded failed failed**, its `.mount` `loaded inactive dead`, and `mnt-felhom\x2ddrives.mount` still `active mounted` — the uninstall's own output had warned `/mnt/felhom-drives/Felhom-Share is busy — NOT forcing` and `/mnt/felhom-drives root bind left mounted`, which is correct behaviour (it never forces an unmount) but is not teardown. Cleared by hand before the reinstall: stop both units, remove both unit files, `daemon-reload`, unmount the autofs then the parent. **NEGATIVE CONTROL, same day:** demo-felhom's uninstall left **nothing** (`ls /etc/systemd/system | grep -i felhom` → only the unrelated `felhom-bootstrap.service`; no felhom mounts) — because that box had no network share configured. **So the residue is conditional on the feature having been used, which is exactly why a diff taken on a box that never used it reported clean.** `felhom-bootstrap.service` is NOT residue — it is the ISO first-boot unit, `disabled`+`inactive`, exactly-once and already fired | CC | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 9631888..799c979 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -20,7 +20,7 @@ | ID | Item | Size | Status | Notes / map rows flipped | |----|------|------|--------|--------------------------| -| R-115 | **Publishing is a remembered step — forgotten within eight hours of being documented as forgettable** | M | **READY** — operator ruling 2026-08-03: **mechanism (b), a build-side gate that REFUSES to deploy or vouch an unpublished version.** THIRD instance the same day (agent v0.120.0 deployed to both boxes while unpublished; a documented-path reinstall would have silently downgraded them and *succeeded*). CC's to build | A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published, so **"deployed" and "installable" are independent states that drift silently**. **Instance 1 — R-111** (morning): 17 agent releases v0.97.0–v0.113.0 stranded; a new customer would have installed without the whole R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT. Found only because the E-2d Phase 0 gate happened to look. **Instance 2 — agent 0.114.0** (same afternoon): the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, never published — which blocked Session C, since a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix. **The finding is the RECURRENCE, not either instance** — both are fixed. R-111 named this leg in its own text (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; it recurred the same day, which is the evidence that **a note is not a mechanism**. **Class: → R-29, one layer up** (a control that exists and is never walked) — deliberately NOT given a second ID. **Filed as its own item rather than reopening R-111** because R-111's finding (the channel *was* stale) is closed and verified end-to-end by the E-2d install, while the process defect that caused it is a distinct problem with a distinct fix and a distinct owner. **Operator's decision, mechanisms first:** (a) publish as a step in the build/release path so deployed and installable cannot diverge; (b) a gate that refuses to deploy an unpublished+unvouched version — strongest, fails closed; (c) a session-end checklist entry; (d) accept manual + a pre-Session-C verification. **(a)/(b) are mechanisms, (c)/(d) are reminders — and R-29's whole finding is that reminders do not hold.** No code written when filed, by design | +| R-115 | ~~**Publishing is a remembered step — forgotten within eight hours of being documented as forgettable**~~ | M | **CLOSED — SHIPPED 2026-08-03** (`release-agent.sh` + `check-published-versions.py`, no version bump). Releasing now builds, tags, publishes and **verifies by an independent download** in one act; a `v` tag with no package fails the gate, and **CI runs the full gate set** so it actually runs. Red-proof measured on real CI: same commit, green before a tagged-unpublished version existed, red after. Residue → **R-184** | A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published, so **"deployed" and "installable" are independent states that drift silently**. **Instance 1 — R-111** (morning): 17 agent releases v0.97.0–v0.113.0 stranded; a new customer would have installed without the whole R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT. Found only because the E-2d Phase 0 gate happened to look. **Instance 2 — agent 0.114.0** (same afternoon): the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, never published — which blocked Session C, since a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix. **The finding is the RECURRENCE, not either instance** — both are fixed. R-111 named this leg in its own text (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; it recurred the same day, which is the evidence that **a note is not a mechanism**. **Class: → R-29, one layer up** (a control that exists and is never walked) — deliberately NOT given a second ID. **Filed as its own item rather than reopening R-111** because R-111's finding (the channel *was* stale) is closed and verified end-to-end by the E-2d install, while the process defect that caused it is a distinct problem with a distinct fix and a distinct owner. **Operator's decision, mechanisms first:** (a) publish as a step in the build/release path so deployed and installable cannot diverge; (b) a gate that refuses to deploy an unpublished+unvouched version — strongest, fails closed; (c) a session-end checklist entry; (d) accept manual + a pre-Session-C verification. **(a)/(b) are mechanisms, (c)/(d) are reminders — and R-29's whole finding is that reminders do not hold.** No code written when filed, by design | | R-116 | **The drive-absent alarm and its recovery are a mismatched pair — generic on the way out, specific on the way back** | S | idea — **PROVEN LIVE 2026-07-29** | Absent fires `storage_disconnected`; return fires `backup_target_restored`. `backup_target_absent` never fires at all (count 0 across a full Session-C run), so an operator gets an alarm they cannot match to its recovery — exactly what `notifyDriveReturned`'s own comment forbids. Root cause: `notifyDriveAbsent` (`intermediary.go:635-646`) branches on `isTarget[a.Path]` with `a.Path` the GUEST path, and `driveTargetByPath` (`:602-616`) builds it as `out[GuestPath] = d.BackupTarget` — but **the drive is TWO `/disks` rows and the flag and the guest path sit on different ones**: the `felhom-backup` storage row has `BackupTarget: true` (`felhom-agent/internal/localapi/disks.go:211`) and gets a guest path only while classified user-data, while the registry union row has the guest path and **never assigns `BackupTarget`** (`disks.go:265-267`). Absent ⇒ the flagged row loses its guest path ⇒ the union row writes `false` ⇒ generic. On return the rows rejoin ⇒ specific. v0.184.1 fixed the KEYING, not this. **Only reachable because R-113 made the gate fire at all.** Fix likely agent-side; decide the repo first. Blocks E-2's C5. Evidence: `audits/SESSION-C-2026-07-29.md` §5 | | R-113 | **The drive-absent gate cannot fire on device loss — E-2b's alarm is wired to an unreachable condition** | M | idea — **PROVEN LIVE 2026-07-29** | `planDriveGates` (`felhom-controller/internal/web/intermediary.go:216-262`) treats a path as present by OR-ing in `d.BoundUnderParent`, which the agent derives from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`internal/localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is not device-bound and its mountinfo entry outlives the device**, so the gate reads it as present and `notifyDriveAbsent` is never called. Live on a fresh box: target drive hot-detached, agent said `enrolled drive absent by UUID` every 20 s for 4½ min, controller logged **0** `[gate]` lines, hub received **zero** events — neither `backup_target_absent` nor the generic `storage_disconnected`. Not a virtualisation artefact (device-bound-mount vs manual-bind is the same on metal); caveat: SCSI hot-detach, physical unplug not staged. **Sixth instance of seam-built-but-never-wired — E-2b wired the seam to a condition that cannot occur.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 | | R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | @@ -157,7 +157,7 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the | R-97 | **The whole-guest backup tier has NO failure signal to the hub — `internal/quiesce` never notifies** | S | **SHIPPED (controller v0.177.0 + hub v0.78.0, 2026-07-27)** — **R-97a:** `quiesce.TierNotifier`, a seam (not an import) wired by an init-only setter, edge-triggered on the R-88 breaker ARMING so a failing tier is reported once per run rather than once per retry; recovery rides `recordSuccess`'s existing bool. **NEW operator-only event types** `whole_guest_backup_failed`/`_recovered` — deliberately NOT `backup_failed`, which carries a customer Hungarian template AND sits in demo-felhom's live `enabled_events`, so reusing it would have emailed the CUSTOMER about a backup they cannot act on while it was still retrying. The recovery joins `recoveredPairedDownTypes` because its `info` severity would otherwise be dropped by `severityNotifies` — the operator would hear it break and never hear it heal. **The hub's operator cooldown was keyed `customerID:eventType` alone**, so one tier would have masked the other for an hour; now narrowly extended with a `tier` suffix taken from the event details, leaving every other event type unchanged. **R-97b:** a suppression window keyed to the quiesce CYCLE (not a state test — v0.164.0's `!= StateStopped` filter cannot see an app caught MID-RESTART, which is exactly how BookStack alarmed), consumed at the same single derivation point `classifyRunStates`. Grace = **180 s**, derived from the deploy flow's 120 s health timeout and Mealie's 60 s `start_period`; it **expires**, so an app that genuinely fails to come back still alarms. **PROVEN LIVE end-to-end with a control:** the new type POSTs 200 from inside guest 9201 while a bogus type 400s, and `notification_log` shows **1 operator row, 0 customer rows**. The quiesce→notify link itself is unit-proven only. | On 2026-07-27 three whole-guest backups failed and three quiesce cycles stopped and restarted every customer app stack, and **not one `backup_failed` event reached the hub.** It is not the allowlist — the hub already carries `backup_failed` and `backup_completed` (they are emitted by the controller's *app-data* backup path). The cause is that **`internal/quiesce` does not import `internal/notify` at all**: the tier R-82 built has no route to the hub, so a whole-guest backup can fail indefinitely in silence. The loop's only trace was `app_start_failed` — **info** severity, **Hungarian**, on the **customer** channel — telling the customer BookStack was down (it had been caught mid-restart by the third cycle) without saying why, during an outage the system itself caused. So the one signal that did fire was both the wrong tier and the wrong story. **Shape:** emit `backup_failed`/`backup_completed` from `quiesceAndPollTiers` naming the TIER, operator-tier; and decide whether a quiesce-induced restart should suppress `app_start_failed` the way controller **v0.164.0**'s deliberate-stop filter does — an app the backup stopped on purpose is not a fault. R-88's breaker bounds the repetition but changes nothing about the silence | | R-95 | **The restic offsite tier's credential CAN DELETE — R-89's "parallel question", now ANSWERED** | M | idea — established read-only 2026-07-27 | **The exposure closed on the weekly PBS tier is fully open on the daily restic tier**, which holds the customer's actual documents and photos and is the only tier that survives losing the box. Established without mutating anything: **(1) Identity** — a per-customer *subaccount* on `storage-box-pool-1` (box 611714, bx11, `u629488`): `u629488-sub1` home `felhom-demo-felhom`, `sub2` peti-felhom, `sub3` demo-hp, each labelled `felhom-customer`. Auth is an **SSH key stored ON THE BOX** (`…/felhom-controller-data/_data/data/offbox/ssh_key`, 0600, beside `repo_password` + a pinned `known_hosts`) — customer-side, not hub-side, so a compromised guest holds it. **(2) Read-write: YES** — the API reports **`readonly=False` on all three subaccounts**, and it is not merely latent: the controller runs `restic forget --group-by host,tags --keep-daily 7 --keep-weekly … --prune` **from the box** (`backup/offbox.go:984`, also `:1070`). Delete rights are exercised on every run. **(3) Append-only: NO, and not expressible** — the repo is built as `sftp:` (`offbox.go:482`); restic's append-only mode requires the **REST server** backend, which plain SFTP cannot provide. **(4) A zero-code mitigation exists and is unused:** the box type carries `snapshot_limit=10` and the API reports `snapshot_plan=null` with **0 snapshots** and `size_snapshots=0`. Hetzner Storage Box snapshots are taken **server-side, outside the SFTP namespace** — an SFTP subaccount cannot delete them — so they are a genuine immutability layer at no extra cost and with no code change. **Rule once for both tiers, per R-89.** Options, cheapest first: enable a snapshot plan (operator click, immediate); split backup-write from prune so pruning runs somewhere the box cannot reach; or move the repo to restic's REST server with `--append-only`. Flips the capability-map row for offsite immutability | | R-94 | ~~A hand-synced version constant drifts, and the gate that would catch it is never run~~ | XS | **CLOSED — SHIPPED hub v0.87.0, 2026-08-02** | Closed by **deleting** the label rather than deriving it: the Setup command fetches the installer at run time from a 30 s-git-synced website (R-110), so no build-time value in the hub can be true. `hostinstall_gates.py` gate 1 inverted to pin the ABSENCE of a version literal; the tautological `render_test.go` assertion deleted (demonstrated passing at `9.9.9`). Detail: `OPEN-ITEMS.md` R-94 | -| R-110 | **`main` is the installer's publish channel — there is no staging** | S | **READY** — operator ruling 2026-08-03: **option (b), the channel moves to a TAG**, so publishing is moving the tag and rollback is moving it back. **Must cover BOTH channels** — the nginx `/scripts/` git-sync AND the nine files the installer fetches from `raw/branch/main` — or it only half-works. CC's to build | `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a `--period=30s`, and nginx serves that working tree directly (`location /scripts/`, `root /usr/share/nginx/html/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`, `:1262`; `runbooks/day0-install.md` C.1) receives. **There is no tag, no pinned-version path, no staging copy and no rollback other than another push** — for the one artifact that runs as **root on a virgin box**, the most privileged thing Felhom ships. **Two consequences worth stating plainly:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29, so the proof run confirms what customers already receive rather than clearing it for release; and the precaution the old R-94 row recorded ("do not point every new box at an installer that has never run") **was never available to take**, because nothing points boxes at a version. **Open question for the operator, not a defect to fix blind:** should `/scripts/` serve a pinned release — a tag-tracked git-sync ref, or a versioned directory (`/scripts/1.22.0/…`) with the hub's generated command naming a version — or is `main`-tracking the accepted shape for a one-operator product where the alternative is a release ritual nobody performs? **Exposure today is zero** (no boxes are installing), which is exactly why it is cheap to decide now. Whichever way it goes, it decides whether R-94 leg (a) makes the label a *fact* (derived from the served script) or keeps it a *claim*. Flips no capability-map row — the map states what the platform does, and this changes nothing about that | +| R-110 | ~~**`main` is the installer's publish channel — there is no staging**~~ | S | **CLOSED — SHIPPED 2026-08-03** (installer v1.23.0). `/scripts/` syncs `installer-v1.23.0`; the website still tracks `main`. Proven by HTTP: a push to `main` left the served bytes byte-identical, moving the tag published in ~40 s, moving it back restored the exact prior sha. The run-time fetches turned out to be **sixteen from the agent repo**, not nine from here — pinned to the agent version instead → **R-183** | `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a `--period=30s`, and nginx serves that working tree directly (`location /scripts/`, `root /usr/share/nginx/html/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`, `:1262`; `runbooks/day0-install.md` C.1) receives. **There is no tag, no pinned-version path, no staging copy and no rollback other than another push** — for the one artifact that runs as **root on a virgin box**, the most privileged thing Felhom ships. **Two consequences worth stating plainly:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29, so the proof run confirms what customers already receive rather than clearing it for release; and the precaution the old R-94 row recorded ("do not point every new box at an installer that has never run") **was never available to take**, because nothing points boxes at a version. **Open question for the operator, not a defect to fix blind:** should `/scripts/` serve a pinned release — a tag-tracked git-sync ref, or a versioned directory (`/scripts/1.22.0/…`) with the hub's generated command naming a version — or is `main`-tracking the accepted shape for a one-operator product where the alternative is a release ritual nobody performs? **Exposure today is zero** (no boxes are installing), which is exactly why it is cheap to decide now. Whichever way it goes, it decides whether R-94 leg (a) makes the label a *fact* (derived from the served script) or keeps it a *claim*. Flips no capability-map row — the map states what the platform does, and this changes nothing about that | | R-128 | **`ISO_VERSION` "aligns with SCRIPT_VERSION" was a comment nothing evaluated** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | Closed by **correcting the claim, not asserting it**: the ISO is frozen while `felhom-host-install.sh` is fetched at run time from `main` (R-94/R-110), so an assertion would invent a constraint. `build-felhom-iso.sh:45-52`. Full reasoning in `OPEN-ITEMS.md` | | R-154 | **`[first-boot]` is automated-install-only, and nothing in the tree said so** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | A PVE property, measured with a same-image control (`audits/SPIKE-universal-iso-3-2026-07-31.md` §2); recorded at `scripts/iso/pkg/build-deb.sh:6-11`. Superseded in practice by the `.deb` delivery route | | R-155 | **`iso-repack.sh` refused any ISO without `auto-installer-mode.toml`** | XS | **CLOSED — iso v1.26.0, 2026-07-31** | **Narrowed, not deleted** — unchanged for `FELHOM_MENU=single` (`iso-repack.sh:121-128`), does not apply to `release` where the file's absence *is* gate G1. Do not remove it wholesale |