docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected

R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The hub's notification_log for the run reads: two per-app failures RECORDED, one digest SENT naming both, and the customer channel SKIPPED with operator_only. Against the measured previous behaviour — two failures, one email naming one app, one leaving no trace anywhere. Scenario D proved itself on an event I had not planned: disk_critical alarmed on two filesystems, the second was collapsed by the cooldown, and that collapse is now visible WITH ITS KEY. Yesterday it would have left nothing at all. A gap the spec did not anticipate is recorded with its fix: the per-app event also fires from the periodic sweep, outside any run, so making it record-only would have created a NEW silence. The sweep emits a digest too, with no run_id, so it stays under the ordinary hourly cooldown. ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile SURVIVED the resize and is active (checked, because a resize is a stop/start). The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere. Five documents corrected — three of which the task's list did not name, found by searching. Two audit/evidence documents ANNOTATED, body untouched: they record what was true when written and that is their value. R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable, not unbounded — the original OOM was a 14.46 GB restore — so the restore-test cadence should still be paced, just not by fear of the endpoint. target-selection.md's "D-d did not name ep0 either way" is deliberately left standing. It is the operator's question, not CC's. STATUS.md 127 -> 83 lines, items rather than sentences.
2026-08-03 14:00:54 +02:00
parent f21e7caed1
commit e34b614e5b
13 changed files with 279 additions and 279 deletions
@@ -123,6 +123,33 @@ Two rulings, one shape: something became live because someone pushed, not becaus
  that bumps a version, before publishing — and a gate that fails on the normal path is one people
  learn to ignore.
 **S-16 — a backup run NOTIFIES ONCE and RECORDS ALWAYS, and those are different things
 (2026-08-03, R-182; controller v0.194.0 + hub v0.90.0/.1).** Measured: nine per-app capture failures
 reached the hub, two were mailed, seven were dropped by a cooldown whose key carries no app
 identifier — *before* `LogNotification`, so they left no row anywhere.
 - **The record:** `recovery_unit_capture_failed`, per app, unconditionally, now routed
  **record-only** by the hub (`recordOnlyEvents`) — stored and logged every time, never competing
  for an e-mail slot.
 - **The notification:** `backup_run_failures`, ONE per run, listing every failed app with its leg
  and reason plus the counts and free space. **A clean run emits nothing.**
 - **A suppressed operator event now leaves a `suppressed` row** carrying the key that suppressed it —
  for every operator type. *"We chose not to e-mail you"* and *"nothing happened"* must never look
  identical; that is the whole finding, stated as a rule.
 - **The periodic sweep gets a digest too, with NO `run_id`**, so it stays under the ordinary hourly
  cooldown. Without it the sweep's failures would be recorded and never notified — a new silence
  created while closing one. A real run's digest carries a unique `run_id` precisely so the cooldown
  can never collapse a manual run into the nightly one.
 - **Why the silence is safe:** the hub's deadline check raises `expected_backup_missed` from report
  freshness, independently of any mail the box sends (`monitor/deadline.go:396,417`). **If that check
  is ever weakened, this design loses its footing.**
 - **Not taken, and why:** putting `app` in the cooldown key. It fixes the swallowing by producing one
  mail per failing app — a dozen on a full disk.
 **ep0 was rescaled by the operator to a CX33 (2026-08-03): 4 vCPU, 8 GB RAM, measured on the box, and
 the 4 GiB swapfile survived. The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only, so no disk
 figure in any runbook needed correcting. That closed **R-90** and unblocked **R-86**.
 **S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).**
 Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the
 CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure
@@ -1,200 +1,188 @@
-# REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed
+# REPORT — R-182: one email per backup run, nothing dropped without a trace + ep0 rescaled (R-90, R-86)
-**Date:** 2026-08-03 · **Repos:** `felhom.eu` (installer **v1.22.0 → v1.23.0**), `felhom-agent` (**no bump**)
+**Date:** 2026-08-03 · **Repos:** `felhom-controller` **v0.194.0** (`88897a2`), `felhom.eu` hub
-**Nothing was built** — no image, no binary, no golden. **Hub stays v0.89.0.**
+**v0.90.0 → v0.90.1** (`dd40f85`, `f21e7ca`)
-## 1. Baselines — re-read on arrival, both matched §1
+## 1. Baselines — re-read on arrival
-| Repo | @ arrival | Version | Result |
+| Repo | @ arrival | Version | Matched §1? |
 |---|---|---|---|
-| `felhom.eu` | `8360f940bfb2` | hub v0.89.0, `SCRIPT_VERSION="1.22.0"`, **0 tags** (confirmed) | installer **v1.23.0**, first tag `installer-v1.23.0` |
+| `felhom-controller` | `db0d4b129d3b` | v0.193.1 | **yes** |
-| `felhom-agent` | `9dfd89cb947e` | v0.120.0 | **unchanged** — scripts and gates only |
+| `felhom.eu` | `7dc1744eece3` | hub v0.89.0 | **yes** — my local clone was one commit behind origin (another session had deleted 24 stale `REPORT-*.md` files); pulled before starting |
-## 2. Part 0 — the R-182 measurement, and it REVERSED the row
+## 2. §3's safety property — confirmed at file:line
-Filed yesterday as *"the reserve re-alerts on every status refresh"* — **too many** alerts, observed
+`hub/internal/monitor/deadline.go:396` raises `expected_backup_missed` and `:417` raises
-at the sending end. Measured at the **receiving end**, it is the opposite.
+`expected_dbdump_missed`, both from `assessBackupFreshness(reportJSON,…)` and `GetEventsByType` — the
 box's **report freshness and stored events**, never from an email. So the hub notices a run that never
 finished **independently of any mail the controller chooses to send**, and a digest's silence still
 means "the run finished and found nothing wrong". **The whole design rests on this; weakening that
 check re-opens a silent-failure path.**
-Method: the hub's SQLite copied **with its `-wal`** (4 MB and newer than the db — copying `hub.db`
+## 3. Files
 alone would have read stale data, the exact trap this project recorded before), freshness confirmed by
 the newest `notification_log` row post-dating the session.
-**9 `recovery_unit_capture_failed` events received today → 2 operator emails sent.**
+**Hub:** `internal/notify/dispatcher.go` (suppression row, `cooldownRunSuffix`, `recordOnlyEvents`,
 `operatorOnlyEvents`) · `internal/notify/templates.go` (digest renderer, `trimRepeatedUsage`) ·
 `internal/api/handler.go` (allowlist) · `internal/store/store.go` (status doc) · two new test files ·
 `recovery_unit_dispatch_test.go` (updated, deliberately) · `hub/CHANGELOG.md` · `manifests/hub.yaml`
-| time | apps refused (events in) | operator emails out |
+**Docs:** `CONTEXT.md` (S-16) · `STATUS.md` · `documentation/architecture/07-backup-architecture.md`
-|---|---|---|
+(the alerting contract, S-1) · `00-capability-map.md` · `backlog/{OPEN-ITEMS,ROADMAP}.md` · five
-| 06:40:03 | privatebin, opengist | **opengist only** |
+runbooks corrected · two audit/evidence documents annotated
 | 08:59:46/47 | opengist, privatebin | **privatebin only** |
 | 08:59:59 | privatebin, opengist | **none** |
 | 09:03:00 | opengist | **none** |
 | 09:07:06 | privatebin, opengist | **none** |
-**Cause, confirmed at source:** the operator cooldown key is
+## 4. Tests and red-proofs
 `customerID + ":" + eventType + cooldownTierSuffix(details)` (`dispatcher.go:268`, 1 hour hardcoded).
 `RecoveryUnitFailureDetails` carries **`app`** and **no `tier`**, so the suffix is empty and the key
 holds **no app identifier**. The first refused app takes the slot; every other app's refusal for the
 next hour is dropped — and dropped **before `LogNotification`**, so it leaves **no row on any
 channel** and cannot be audited afterwards.
-This is **R-97a's failure mode in a second event type**; that row's own comment states it
+`go build && go vet && go test ./...` in `hub/` → **17 packages ok, rc=0**. `repo_gates.py --fast` →
-(*"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20"*). `cooldownTierSuffix` was
+all 5 gates OK.
 written narrow on purpose; `recovery_unit_capture_failed` simply never opted in.
 **A correction I owe on yesterday's report.** It said *"one `recovery_unit_capture_failed` per app,
 HTTP 200"*. That was true of what the **controller pushed**, and a reader would take it as *the
 operator was told about each app* — which is false. The gap between an accepted event and a sent
 email is the whole of this row.
 **Nothing was changed** (§8.5). R-182 is re-scoped with the evidence and the fix shape.
 ## 3. Probes
 | | Question | Method | Verdict |
 |---|---|---|---|
 | **P-A** | does git-sync v4.4.0 follow a tag, and notice a **moved** one? | throwaway `docker run` git-sync against this repo, tag moved under it | **PASS both halves** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period (~20 s) |
 | **P-B** | does Gitea serve `raw/tag/<tag>/<path>`? | one fetch on a throwaway tag | **PASS** — HTTP 200, byte-identical to `raw/branch/main` |
 | **P-C** | can CI read the package registry? | anonymous fetches | **PARTIAL, and it changed the gate's design** — package **download** 200 (and **404** for a fake version, so it discriminates), **tags** api 200; package **listing** api **401**, hub artifact manifest **401** |
 **Publish model P-A implies:** publishing is **moving the tag**; rollback is **moving it back**, in
 ~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone
 (`git ls-remote --tags` → 0 at the time).
 ## 4. §8.2's three channels — enumerated
 | Channel | Before | After | |
 |---|---|---|---|
 | 1. the served script | `main`, 30 s | **`installer-v1.23.0`** | **MOVED** — `webpage.yaml` split into two syncs |
 | 2. the run-time fetches | `raw/branch/main` | **`raw/tag/v$ART_AGENT_VER`** | **MOVED** — but see below |
 | 3. the URL producers | `main` | unchanged | **NO CHANGE NEEDED** — and that is a finding, not an omission |
 **Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable.**
 There are **sixteen** fetches, not nine, and they come from **`felhom-agent`**, not `felhom.eu` — so
 no tag on this repo could ever have covered them, and §8.1's *"derive the tag from `SCRIPT_VERSION`"*
 was impossible for them. Raised before building; operator ruled to pin them to **the agent version
 being installed**, which the installer already resolves from the hub manifest and already sha-verifies.
 That is strictly better than any installer-derived tag: binary and configs now come from one ref.
 **Channel 3 needed no change because the URL never carried a ref** —
 `https://felhom.eu/scripts/felhom-host-install.sh` is path-based; the ref lives in the manifest. So
 `felhom-bootstrap.sh` and the hub's day-0 command follow the tag automatically. **No hub template
 change ⇒ no hub bump**, so §1's rule was never in tension and the STOP it anticipated never arose.
 ## 5. The tag convention
 - **Shape:** `installer-v<SCRIPT_VERSION>` in `felhom.eu` (prefixed so it cannot be read as a hub,
  agent, controller or golden version); `v<semver>` in `felhom-agent` (that repo versions one thing).
  **No new constant in the installer** — channel 2 derives its ref from `$ART_AGENT_VER` at run time,
  and channel 1's ref lives only in the manifest.
 - **Publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml` (sidecar *and*
  init container), commit, sync.
 - **Roll back:** move the tag back to the previous commit — takes ~30 s, **no ArgoCD sync, no deploy**.
 ## 6. Scenario A — proven by HTTP
 A real commit was pushed to `main` (a marker comment in the installer) **without moving the tag**, and
 three sync periods were allowed to pass so "unchanged" means "had every chance to change":
 ```
 website tree (main): .worktrees/6a82719…      <- ADVANCED to the new commit
 scripts tree (tag):  .worktrees/bee6848…      <- STAYED
 sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
 sha256 after  push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
 marker present at the served URL? 0
 https://felhom.eu/ -> HTTP 200
 ```
 Both halves of the split in one observation: the site still tracks `main`, the installer does not.
 ## 7. Scenario B — publish and rollback, both directions
 | act | result |
 |---|---|
 | tag moved `bee6848 → 6a82719` | scripts tree moved in **~40 s**; served `sha256 ea2b4aa9…`; **marker present** |
 | tag moved back `→ bee6848` | scripts tree back in **~40 s**; served `sha256 2f859555…` — **exactly** the pre-publish sha; **marker gone** |
 `https://felhom.eu/` returned 200 throughout. The marker commit was then reverted, and the tag moved
 to `main`'s head — a **byte no-op**, verified by the served sha not changing.
 ## 8. Files, commits, tags
 **`felhom.eu`** — `bee6848` (installer + gate + manifest), `6a82719` (Scenario A marker), `e79a20b`
 (marker removed), plus the docs commit below.
 `scripts/felhom-host-install.sh` · `scripts/hostinstall_gates.py` · `scripts/CHANGELOG.md` ·
 `manifests/webpage.yaml` · `CLAUDE.md` · `CONTEXT.md` · `STATUS.md` · `REPORT.md` ·
 `documentation/backlog/{OPEN-ITEMS,ROADMAP}.md` · `documentation/architecture/00-capability-map.md`
 **`felhom-agent`** — `dd2d1fe` (release path + gate + CI), `0db7766` (REPORT).
 `scripts/release-agent.sh` **(new)** · `scripts/check-published-versions.py` **(new)** ·
 `scripts/agent_gates.py` · `.gitea/workflows/gates.yml` · `CLAUDE.md` · `CHANGELOG.md` · `REPORT.md`
 **Tags created:** `felhom.eu/installer-v1.23.0` (the first tag this repo has ever had) and
 `felhom-agent/v0.120.0` (retroactive, at `cd6e267` — the commit the published binary was built from;
 `configs/` is byte-identical there and at `main`, so nothing depended on the choice).
 ## 9. Tests and red-proofs
 | Check | Result |
 |---|---|
 | `felhom.eu` `repo_gates.py --fast` | all 5 gates OK |
 | `felhom-agent` `go build ./... && go vet ./...` | OK |
 | `felhom-agent` `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
 | `agent_gates.py --fast` | `published` correctly **SKIPPED** (hook must not fail on a network blip) |
 | `agent_gates.py` (full) | both OK |
 **Red-proofs, each demonstrated failing then restored:**
 | # | Mutation | Result |
 |---|---|---|
-| C | one of the sixteen fetches reverted to `raw/branch/main` | **RED** — gate 6a *and* 6b both fired |
+| D | the suppression row removed (back to the bare `return`) | **RED** — `want 1 'suppressed' operator row, got 0` |
-| D | assertions 6a **and** 6b removed (every guard the test covers), same bad installer | **zero** mentions of the regression — the guards are what catch it |
+| E | `cooldownRunSuffix` removed from the key | **RED** — `operator mails = 1, want 2`; the manual run's digest is swallowed |
-| 6c | the manifest before the split | **RED** on its own, before I fixed it — the gate was demonstrated red by the real pre-change state |
+| G | the digest removed from `operatorOnlyEvents` | **RED** — and it showed **a customer actually receiving the operator digest**, demonstrated rather than argued |
-| F | `v9.9.9` tagged and not published | **RED**, `binary NOT downloadable (HTTP 404 …)`, rc=1 |
+| A | (controller) all four `noteFailure` feeds removed | **RED** |
-| F′ | the gate **deregistered** from `agent_gates.py`, same bad state | **rc=0, "all agent gates OK"** — restored → `CONVICTED: published`, rc=1 |
+| I | (controller) the `main.go` seam commented out | **DID NOT FAIL FIRST TIME** — see below |
-**Scenario F measured on real CI, not inferred.** Runs **69** and **70** are on the *same commit*
+**Red-proof I is the one worth reading.** The AST test walked the backup package but not `main.go`, so
-`0db7766`: **success** before `v9.9.9` existed, **failure** after pushing it. One variable. This also
+the seam could be disconnected with the whole suite green. Per §9.11 that is not a pass: **the test was
-retrospectively explains runs 67/68. **One deliberate CI failure email reached the operator — that was
+fixed and the mutation re-run**, at which point it failed correctly. The string `SetRunSummaryNotify`
-this proof, not an incident.** I could not read CI's own step log: the jobs endpoint needs a Gitea API
+is still present in the commented-out file, which is exactly why this is an AST walk.
 token, and the only credential available (`~/.docker/config.json`) is a registry password that the API
 rejects — so the controlled before/after replaced the log rather than an assumption standing in for it.
-## 10. No version bumps, nothing built
+**One passing test was changed, deliberately.**
 `TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer` required the OPERATOR to be emailed a per-app
 capture failure — correct when that event was the only signal, wrong now that it is the record and the
 digest is the notification. Its **customer-safety claim is unchanged** and is why the test still
 exists; the operator assertion is inverted with the reasoning written in place, and it now asserts the
 `recorded` row instead. R-158's guarantee **moved rather than weakened**.
-`felhom-agent` **v0.120.0** unchanged (no Go code changed). Hub **v0.89.0** unchanged (no hub file
+## 5. The live run
 touched). The installer's `SCRIPT_VERSION` **did** go 1.22.0 → 1.23.0 — the installer is not in §12's
 no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or
 golden was built.
-## 11. Register
+Demo-hp guest 9201, filled for real to **241 MB free / 100% used** (thin pool held **30.78 → 30.78**).
 Triggered through `POST /api/debug/backup/dbdump` — the exact endpoint the debug UI button calls.
 **The hub's `notification_log`, quoted:**
 ```
 11:50:45 | recovery_unit_capture_failed | operator | recorded  | record-only: the per-run digest … | Recovery unit capture FAILED for "opengist" …
 11:50:45 | recovery_unit_capture_failed | operator | recorded  | record-only: the per-run digest … | Recovery unit capture FAILED for "privatebin" …
 11:50:45 | backup_run_failures          | operator | sent      |                                  | 2 of 2 apps failed to back up in this manual run: opengist, …
 11:50:45 | backup_run_failures          | customer | skipped   | operator_only                    | 2 of 2 apps failed …
 11:51:00 | disk_critical                | operator | suppressed| operator cooldown 1h, key=demo-hp:disk_critical | …
 11:51:00 | disk_critical                | operator | sent      |                                  | …
 ```
 Two failures → **both recorded**, **one** email naming **both**, customer refused with the reason
 logged. Against the measured previous behaviour: two failures → one email naming one app, one leaving
 no trace.
 **Scenario D proved itself on an event I had not planned.** `disk_critical` alarmed on two
 filesystems; the second was collapsed by the cooldown, and that collapse is now **visible with its
 key**. Yesterday it would have left nothing at all — which is the entire finding, demonstrated by
 accident on a different event type.
 **Scenario E:** a second run in the same hour (11:51:28) produced a second digest.
 **Scenario B:** after freeing the space, the run completed with 2 volume dumps and **no** digest line.
 ## 6. The email as shipped
 ```
 SUBJECT: [Felhom] 🔴 demo-hp: 2 of 2 apps failed to back up (manual run)
 Customer: demo-hp
 Event:    backup_run_failures
 Severity: error
 Time:     2026-08-03 13:52 CEST
 Message:  2 of 2 apps failed to back up in this manual run: opengist, privatebin
 FAILED: 2 of 2 apps attempted in this manual run.
  opengist    whole app (refused before any write)  refused: backing up this app would leave the
                                                    filesystem below the reserve (reserve: 97% used
                                                    or 1.0 GiB free; the filesystem is already below
                                                    it, before this app's estimated 178.0 KB write)
  privatebin  whole app (refused before any write)  refused: … estimated 2.5 KB write
 Filesystem: /mnt/sys_drive — 65.0/68.7 GB used (95%), 0.2 GB free
 Every failure above is also recorded individually in the notification log,
 whether or not this mail was sent.
 Dashboard: https://hub.felhom.eu/customers/demo-hp
 ```
 **v0.90.1 came from reading exactly this.** In the first rendering every app row also ended with
 `— /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free` — the same clause the `Filesystem:` line
 already carries. On two apps that is untidy; down a list of a dozen it is forty characters twelve
 times, pushing the part that *differs* off a phone screen at 07:00. The reserve's message is unchanged
 (it is right for a single-app alert); the digest trims the duplicate when rendering, and only an exact
 `— <target path>:` suffix.
 ## 7. §8.1's table, in practice
 The run refused **both** deployed apps on the reserve → both in the digest. **No** deliberate skips
 occurred in the live run (no drive was disconnected or decommissioned on that box), so that row of the
 table was exercised by test only — `TestRunSummary_DeliberateSkipsAreNotFailures`, which marks one app
 disconnected and one decommissioned through the real `settings` store and asserts **no digest at all**.
 ## 8. ep0 — measured, then corrected
 | | measured on the box |
 |---|---|
 | hostname | `felhom-hetzner` |
 | RAM | **7757 MB (8 GB)** — was 3.8 |
 | vCPU | **4** |
 | swapfile | **survived the resize** — `/swapfile`, 4 GiB, `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`) |
 | local disk | `/` 38 G, 58% used — **UNCHANGED**, as expected for a CPU/RAM resize |
 | datastore volume | `/mnt/pbs-datastore` 98 G, 8% — unaffected |
 The swapfile was **checked rather than assumed**: a resize is a stop/start, so "the interim lever is
 still in place" was an assumption until measured.
 **Corrected** (five places, three of which the spec's list did not name — found by searching):
 `RUNBOOK-ep0-datastore-volume-2026-07-27.md:6` · `RUNBOOK-pbs-prune-serverside-2026-07-27.md:232,:310`
 · `runbooks/offsite-endpoint.md:8,:34` · **`runbooks/target-selection.md:101`**.
 **Annotated, body untouched** (they record what was true when written, and that is their value):
 `audits/SPIKE-connectivity-wireguard-2026-07-03.md:13` ·
 `tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md:142`.
 **No disk figure was touched anywhere** — the 40 GB local disk did not change.
 ## 9. Register
 | ID | Outcome |
 |---|---|
-| **R-110** | **CLOSED — SHIPPED** (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed |
+| **R-182** | **CLOSED — SHIPPED**, with the live evidence above |
-| **R-115** | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, no bump) |
+| **R-90** | **CLOSED** — ep0 rescaled to a CX33; measured on the box, swapfile confirmed surviving |
-| **R-182** | **RE-SCOPED — the direction reversed** by Part 0's measurement; still open, now correctly described |
+| **R-86** | **READY, unblocked and re-ranked** — endpoint headroom no longer sets the restore-test cadence. Stated honestly: 8 GB is comfortable, not unbounded (the original OOM was a 14.46 GB restore), so the cadence should still be paced — just not by fear of the endpoint |
 | **R-183** | **NEW, and CLOSED the same session** — binary and configs came from two different refs |
 | **R-184** | **NEW, open** — nothing stops the hub vouching a version that was never released |
-**IDs established free:** `^| \*\*R-183\*\*` / `^| \*\*R-184\*\*` in `OPEN-ITEMS.md` → **0 rows** each;
+**No new IDs were minted**, so no grep was needed. R-184 remains the highest.
 all other hits are this session's own code and changelogs (forward references I wrote). `R-185` → 0
 hits anywhere and remains free.
-## 12. Observations — noticed, documented, NOT acted on
+**`target-selection.md`'s ep0 protection question is STILL OPEN and deliberately untouched** —
 *"D-d did not name ep0 either way. Confirm it explicitly."* That is the operator's to settle, not CC's.
-1. **The gate cannot see what is vouched** — filed as R-184 rather than papered over. Closing it needs
+## 10. Deployed
   either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails
   closed where the mistake is made, needs no new credential).
 2. **A suppressed operator alert leaves no row at all.** The cooldown returns before `LogNotification`,
   so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside
   R-182 because it is what made that row take a day to get the right way round.
 3. **`on: [push]` fires CI for tag pushes too.** Useful (it is how Scenario F was measured), but it
   means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate.
 4. **`felhom.eu` CI still runs `--fast`.** Correct today, since all its gates are network-free; if a
   network gate is ever added there, that workflow needs the same change the agent's just got.
-## 13. Teardown
+- Controller **0.194.0** on demo-hp guest 9201 — `Up (healthy)`.
 - Hub **0.90.1** — ArgoCD `Synced` / `Healthy`, image `gitea.dooplex.hu/admin/felhom-hub:0.90.1`,
  startup log clean (`Listening on :8080`, deadline-check scheduled for 05:00 CEST).
-Probe container, probe sync tree and probe tag (`probe-r110-delete-me`) removed; the red-proof tag
+## 11. Teardown
-`v9.9.9` deleted (`git ls-remote --tags` → only `v0.120.0`); the Scenario A marker reverted from
+
-`main` and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl
+Fill file removed; guest helper scripts and the credential file `shred`-ed; `pct fstrim 9201` returned
-pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file.
+63.3 GiB; thin pool **30.76**; the guest is back to 1.9 G used / 64 G free. The scratch email renderer
 under `hub/tmprender/` was deleted after use. The hub DB copy is scratch-only.
 ## 12. Observations — NOT acted on
 1. **Only two apps are deployed on the demo box**, so "several apps" was 2 of 2 — exactly the measured
   case, but a five-app demonstration would be stronger.
 2. **`FormatOperatorEmail` puts an emoji in every operator subject.** Part 3 asked for none; the icon
   comes from the shared formatter and changing it globally would alter every other operator mail and
   its tests. The digest's own copy has no emoji.
 3. **A `recorded` row is written for record-only events on the `operator` channel.** It is not an
   operator *notification*, so the channel label is slightly overloaded — readable today, worth a
   thought if a third channel ever appears.
 4. **The customer-channel `skipped/operator_only` row no longer appears for the per-app event**,
   because record-only returns before the customer leg. That is correct, but anyone grepping history
   for that pattern will see it stop on 2026-08-03.
@@ -14,114 +14,70 @@ A blank machine boots the Felhom disc, installs itself unattended, and is claime
 who sets their own password. They install apps from a catalogue of fifty-three, share files over the
 home network, and open apps from a launcher or a shared link. Backups run on their own to three
 places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
-restore files and app data from the drive alone. Proven end to end on real hardware.
+restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
-
+box six times, everything returned every time, and an app switched off deliberately stayed off.
-**Apps come back after a power cut.** The machine tells an app the customer switched off from one
+Proven end to end on real hardware.
 that simply did not come back, and waits for the system to finish starting before deciding instead of
 glancing once, five seconds in. Hard-reset the demo box six times in a row: everything came back every
 time, and an app switched off deliberately stayed off every time.
 ## What's broken
-**The off-site copy can be erased by the machine that made it** — the credential that writes it can
+- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
-also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
+  also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
-*(R-95, R-87)*
+  *(R-95, R-87)*
-
+- **Nothing else open that affects a customer.** The three faults that were on this list yesterday —
-**A full disk tells you about ONE app and silently swallows the rest.** Yesterday this was written
+  the reserve watching the wrong step, the last app whose data was never saved, and the alert that
-down the wrong way round — as *too many* emails. Measuring the receiving end reversed it: of nine
+  told you about one app and swallowed the rest — are all fixed and proven on a real machine.
 refusals the machine reported today, **two emails were sent**. When two apps are refused in the same
 second you are told about one of them, and the other leaves no trace anywhere — not an email, not
 even a line in the log saying it was held back. So a second app can be going unbacked-up while you
 have already been told the problem is handled. It is the same fault we fixed once before for
 whole-machine backups, in a second place that never opted into the fix. *(R-182)*
 ## What shipped recently
-**Pushing the installer no longer publishes it.** The script that runs as root on a brand-new
+- **A failed backup now tells you about every app, once.** When several apps fail in one night you get
-machine was copied from the main branch and served within thirty seconds, so pushing it *was*
+  **one** email listing all of them, what kind of backup failed for each, why, and how much space is
-publishing it, with no staging and no way back but another push. It now comes from a **labelled**
+  left — instead of an email about one app and silence about the rest. Measured before the fix: nine
-version: publishing is moving the label, and undoing it is moving the label back — about half a
+  failures, two emails. Separately, **every failure is now written down the moment it happens**,
-minute, no deploy. The website is untouched by this and still updates in thirty seconds, because a
+  whether or not it is emailed, and **a message the machine decides not to send now leaves a record
-typo fix must never need a release. Proven by actually doing it: a real push changed nothing that
+  saying so**. A machine choosing not to tell you and nothing happening no longer look identical.
-anyone downloads, moving the label published it, moving it back restored the previous bytes exactly.
+  A run that finds nothing wrong stays silent, and a backup you trigger yourself always reports.
-
+  *(R-182)*
-**The catch that would have made it cosmetic was found and covered.** While it runs, the installer
+- **Publishing became something you do, not something that happens.** The installer that runs as root
-fetches sixteen more files — not nine, and from the *agent's* repository, not the website's. They now
+  on a new machine, and the agent, both moved onto labelled versions with a way back; a check now
-come from the same version of the agent the machine is installing. That closed a real fault nobody
+  refuses a release nobody can download. *(R-110, R-115)*
-had noticed: a new machine was getting the agent's tested program and its untested settings files, in
+- **The reserve now guards the step that fills the disk, and its promise is true.** It decides once
-one install, from two different places. *(R-110, R-183)*
+  per app, before writing anything, and no longer waves an app through at 96% full. *(R-181)*
-
+- **The backup partition is gone**; both demo machines were rebuilt on the new shape by two different
-**Releasing the agent now publishes it, in one command.** Putting a built agent where a new machine
+  routes. Backup space went from 19 GB to 65 GB on the small machine, 45 GB to 233 GB on the big one.
-can download it was a step someone had to remember, and it was forgotten three times in five days —
+  *(R-165, R-178)*
-the last time leaving both demo machines running a version nobody could download, so a rebuild would
+- **The last of the three apps that never saved their data is fixed.** *(R-156)*
-have quietly installed the *older* one and reported success. There is now one command that builds,
+- **A filling disk warns the customer before anything breaks**, naming the drive and the space left.
-labels, publishes and then **downloads it back to check** — and a check that refuses to stay quiet if
+  *(R-167, R-158)*
-a released version cannot actually be fetched. Proven by making CI fail on purpose and then go green
+- **The checks have two nets** — one before every push, one on the build server that emails you.
-again on the same code. *(R-115)*
+  *(R-29, R-161, R-168)*
 **The backup partition is gone and both demo machines run on the new shape** — wiped, rebuilt and
 taken through the whole customer journey on 3 August, by two deliberately different routes so the disk
 shape and the delivery route are both proven. The space a backup can use went from 19 GB to 65 GB on
 the small machine and 45 GB to 233 GB on the big one. Their previous demo apps and data are gone; that
 was the point of a wipe, and you approved it. *(R-165, R-178)*
 **What replaced the wall now watches the right moment.** The wall was quietly keeping a runaway
 backup from eating the space the machine needs to run. As first built, that replacement was checked
 too late — the big write happened first, unchecked — while still promising your last good copy was
 untouched. Fixed and proven on 3 August: the machine decides once, per app, **before it writes
 anything**, and that one answer covers all three steps, so a refused app writes nothing, is not
 restarted, and the promise is now literally true. It also stopped being blind to size. Nothing is ever
 deleted to make room. *(R-181)*
 **The last of the three apps that never saved their data is fixed.** Installed nowhere, so nothing was
 stranded — checked on both demo machines and in the fleet list rather than assumed. Proven by the check
 that caught it, run in both directions: it clears the fixed version and still convicts the old one.
 *(R-156)*
 **A filling disk warns the customer before anything breaks, and a failed backup reaches you** — the
 customer while there is still room to act, naming the drive and the space left; you when one app's
 backup fails, with the disk figures. The customer is deliberately not told about the second: they can
 free space, but they can do nothing about a failed backup. Both proven by filling a real disk. There
 are two rules and not one because the serious warning fired on free space while the disk was only 91%
 full — a percentage alone would have missed it. *(R-167, R-158)*
 **The checks have two nets and the second emails you.** Every repository has one command that runs all
 its checks, before every push. That one can be skipped, so the build server runs them again and emails
 you on failure. It cannot *stop* a change — everything goes straight to the main copy with no review
 step — but it notices quickly and tells you. *(R-29, R-161, R-168, R-169)*
 ## What we're working on
- **Now:** nothing outstanding from today — the reserve, the last unsaved app, and both of your
+- **Now:** nothing outstanding. Everything raised in the last three days is closed.
-  decisions are all built and proven.
+- **Next:** restore-tests scheduled around each backup rather than on a free-running timer — now
- **Next:** the alert that tells you about one app and swallows the second *(R-182)*.
+  unblocked, because the off-site endpoint has the memory for it. *(R-86)*
- **After:** the off-site copy that the machine making it can still erase *(R-95, R-87)*.
+- **After:** the off-site copy that the machine making it can still erase. *(R-95, R-87)*
 ## Waiting on you
 - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
  session log; nothing suggests anyone else saw it. *(R-132)*
- **Nothing else.** Both decisions you took on 3 August are now built and proven. One small question
+- **One small question, not urgent.** The automatic check cannot see which version you have told
-  will come back later: the automatic check cannot see which version you have told machines to
+  machines to install, only which ones exist. Closing that needs either a password given to the build
-  install, only which ones exist — closing that either needs a password given to the build server or
+  server or a check inside the hub itself. *(R-184)*
-  a check inside the hub itself. Filed, not urgent. *(R-184)*
+- **One older question still standing:** whether the off-site endpoint counts as protected. It was
  never named either way, and it is yours to settle. *(target-selection.md)*
 ## Changed since last update
- **2026-08-03** — Publishing became something you do rather than something that happens: the
+- **2026-08-03** — A failed backup now reports every app in one email, and nothing is dropped without
-  installer and the agent both moved onto labelled versions with a way back, and a check now refuses
+  a record. You rescaled the off-site endpoint to twice the memory, which closed one item and
-  a release nobody can download. Earlier the same day: the reserve now guards the step that fills the
+  unblocked another; the documents describing the old machine were corrected, and the audit records
-  disk and its promise is true, and the last app whose data was never saved is fixed. All proven on
+  of what it used to be were annotated rather than rewritten.
-  real machines, not just in tests.
+- **2026-08-03** — Publishing became deliberate for both the installer and the agent, with a way back
-  Earlier the same day: both demo machines wiped and rebuilt from the new base image and taken through
+  and a check that a release can actually be downloaded.
-  set-up → install an app → back it up → restore it, with the backup space ceiling gone and measured.
+- **2026-08-03** — The reserve was fixed to guard the step that fills the disk; the last unsaved app
-
+  was fixed; both demo machines were rebuilt on the merged disk layout and proven end to end.
 - **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in
  a mode where reading a page cannot block a machine's status update; a one-word difference meant that
  setting had **never taken effect**, for the hub's whole life. Fixed and verified live. **Also found:
  the hub's own database is in no automatic backup** — it holds every machine's emergency password.
  Filed, not yet fixed.
 - **2026-08-02** — Thirteen mechanical checks had built up and nothing ran most of them; two were
  failing quietly. Fixed. Decided the same day: the 20 GB backup partition goes away; and only this
  machine and the tester's box are protected, every other box may be broken or reinstalled freely.
@@ -86,7 +86,7 @@
 | An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 |
 | Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped |
 | **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait |
-| **A failed per-app Tier-1 backup reaches the OPERATOR** (app, error, and the target filesystem's used/free bytes at the moment of failure) | controller **v0.191.0**, hub **v0.89.0** (R-158, closed by R-167) | **PROVEN-LIVE (2026-08-02)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it |
+| **A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed** | controller **v0.194.0**, hub **v0.90.1** (R-158 → R-167 → **R-182**) | **PROVEN-LIVE (2026-08-03)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it. **ROW REWRITTEN 2026-08-03 (R-182) — the 2026-08-02 claim was TRUE OF ONE APP AND FALSE OF THE REST, and it is worth saying which.** The signal existed and worked; what it did not do was scale past the first failing app. Measured: nine per-app events reached the hub in one day and **two** operator mails went out, because the cooldown key is `customerID:eventType(+tier)` and this type carries `app` but no `tier` — so the first refused app took the hour and the rest were dropped **before `LogNotification`**, leaving no row on any channel. The old row said "One event per app, loop continuing", which was true of what the CONTROLLER emitted and not of what the operator received. **Now:** the per-app event is the RECORD (hub `recordOnlyEvents`: stored + logged every time, never mailed) and `backup_run_failures` is the NOTIFICATION — one mail per run listing every failed app, its leg and its reason, with the counts and free space. A suppressed operator event of ANY type now leaves a `suppressed` row naming its key. **Proven live on demo-hp 2026-08-03** by a real 64.6 GiB fill (thin pool held 30.78 → 30.78): `notification_log` shows `recovery_unit_capture_failed | operator | recorded` ×2, `backup_run_failures | operator | sent` ×1 naming BOTH apps, and `backup_run_failures | customer | skipped | operator_only`. A second run in the same hour produced a second digest; after freeing space a run completed with 2 volume dumps and **no** digest. The suppression row proved itself on an unplanned event — `disk_critical | suppressed | key=demo-hp:disk_critical` — a collapse that yesterday would have left nothing at all. **The digest's silence is safe only because** the hub's deadline check raises `expected_backup_missed` from report freshness independently of any mail (`monitor/deadline.go:396,417`); that check is load-bearing for this row |
 | **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.193.1** (R-165 / D-a / B2, completed by R-181) | **PROVEN-LIVE (2026-08-03) — BOTH halves** | `REPORT.md` (R-178 reinstalls) + `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately.** **Proven (R-178, 2026-08-03):** *"a local backup is bounded by the box's FREE SPACE, not by a partition set at build time"* — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp `--golden <local volid>`; demo-felhom the normal manifest route with **`verified sha256 54e2a4c431daf580… matches the hub manifest`**), each showing `mp0` at `/var/lib/felhom` with **no `mp1`**, both consumer paths real mounts on ONE filesystem (`stat -c %d` = `64519` on all three), 3/3 reboots each, and claim → deploy → backup → **restore** with a planted marker returning byte-identical. Space available to a recovery unit measured at **65 GiB / 233 GiB**, against the **19 GiB / 45 GiB** those boxes' `mp1` slices offered. **NOT proven — and measured FALSE in part:** *"a capture that would exhaust it is refused per app rather than allowed to stop the container runtime"*. The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — **but it is checked only in `captureAllRecoveryUnits`, while `runVolumeDumps` writes the bulk with no floor check at all**, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → **R-181, CLOSED THE SAME DAY (controller v0.193.0 + v0.193.1) and the second half is now PROVEN-LIVE TOO.** The reserve became a **per-app, per-run ADMISSION decision** taken before the app's FIRST write and covering all three legs (DB dump, volume dump, capture) — they write under one per-app root, which is what lets one verdict cover them honestly — and it gained a **size term**, so an app is no longer admitted at 96% and then allowed to write 2 GB. **Re-proven by filling demo-hp deliberately, once for EACH term, using the method that found the defect.** *Headroom @ 08:59:46* (906 MB free / 99%): both apps refused, **the whole `backups/primary` tree byte-identical — `TREE_SHA` 111d1760c18d3440f700634ab325f8b8 before and after**, opengist's tar still at its original 182,272 B; **no `Stopping <app> for safe volume dump` line at all**, which is the positive-by-absence observable that matters because that line IS present in the 08:58 baseline run; 0 volume dumps; one alert per app, HTTP 200. Space freed, re-run @ 09:01:33 → both captured normally. *Size @ 09:03:00*, reproducing the original sequence with a real 2 GiB file in opengist's volume (previous tar **2,147,666,432 B**, the exact figure the defect was measured at) and the filesystem at **91% used / 2.9 GB free — both headroom terms deliberately clear**: opengist refused `(size)` while **privatebin was ADMITTED and dumped normally**, proving the term is per-app rather than a global halt. **The refusal's wording was NOT weakened to fit** — the behaviour moved so the wording became true, and it is verified by tree fingerprint rather than by reading the log line, which is what lied. The `fallocate` instrument was re-proven on the rebuilt box before use (5 GiB step moved guest `df` while thin-pool `data_percent` held **36.83 → 36.83**), and teardown returned the pool to **29.43%**, below its own baseline. The golden **is now VOUCHED** (2026-08-03, hub `Artifact manifest set: … golden=0.192.0`), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity |
 | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
 | **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise |
@@ -574,6 +574,30 @@ because that was a different filesystem. On a merged box it can. Decision **B2**
 It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
 only destroy a different app's only local copy.
 **WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).**
 The two are deliberately different mechanisms, because conflating them is how seven failures went
 missing on 2026-08-03 without leaving a trace.
 | | Record | Notification |
 |---|---|---|
 | what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN |
 | when | every time, unconditionally | at the end of a run, **only if something failed** |
 | gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown |
 | where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox |
 - **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that
  is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's
  REPORT freshness, independent of any mail the box sends. That check is load-bearing for this
  design; weakening it re-opens a silent-failure path.
 - **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it.
  Deciding not to tell someone is itself an event worth recording.
 - **Deliberate skips are not failures** and never appear in the digest — a disconnected or
  decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the
  operator stops reading.
 - **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within
  the hour, because someone pressing the button is actively trying to get a backup. The periodic
  capture sweep is capped by the ordinary hourly cooldown.
 **THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a
 **per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately
 before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the
@@ -10,7 +10,7 @@
 **Class:** SPIKE (empirical validation; no product code). **Repos:** felhom.eu (this doc only);
 felhom-agent read-only for grounding (`internal/pbs/{client,pin}.go`, `configs/build-golden.sh`,
 `internal/hub/cloudflared.go`, `internal/escrow/identity.go`).
-**Probe ends:** `felhom-hetzner` = Hetzner CX23, Debian 13.4, public IPv4 `167.233.158.164`,
+**Probe ends:** `felhom-hetzner` = Hetzner CX23 (**note added 2026-08-03: rescaled to a CX33, 8 GB RAM — this spike records the machine as probed and its body is deliberately unchanged**), Debian 13.4, public IPv4 `167.233.158.164`,
 global IPv6 `2a01:4f8:...::/64` (throwaway — NOT the live jarrs.eu box) ⟷ **demo-felhom** =
 the real PVE 9.2.2 host on the operator's home line (One Hungary fixed cable, Budapest), driven
 over the existing LAN SSH path; the tunnel itself always dialed **out**.
@@ -144,7 +144,7 @@
 | R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. |
 | R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. |
 | R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). |
-| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — operator ruling 2026-07-27** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
+| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — UNBLOCKED 2026-08-03 (R-90 closed: ep0 is now a CX33, 8 GB + 4 GiB swap, so endpoint headroom no longer sets the cadence — though 8 GB is comfortable, not unbounded: the original OOM was a 14.46 GB restore)** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
 | R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. |
 | R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | **Part 1 SHIPPED (controller v0.176.0, 2026-07-27); Part 2 OPEN (agent wire change)** — **Part 1** added the failure breaker: consecutive failures tracked per TARGET, backoff `15m→30m→1h→2h→4h` capped, reset on success, gating the QUIESCE rather than the backup so no stack is stopped for a doomed attempt. Never permanent (the cap bounds the retry interval — a latched breaker is a silent backup outage), never global (a broken offsite tier cannot suppress a healthy local one), never applied to `TriggerNow`, and `stillRunning` is not a failure. In-memory on purpose: forgetting a backoff costs one attempt, persisting one could outlive the fix. **Part 2 is agent-side and needs a wire change** — see the P0 finding below. Live on both boxes; the loop reproduction is UNIT-TEST ONLY (reproducing it live would need PBS taken down on ep0, which has no swap). | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **NOT bounded, and it also DISABLES the business-hours protection — an availability fault, not a logging nuisance.** The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves, and a second reading ("it stopped after three tries, so something limits it") was wrong too — the live run stopped after three only because the storage came back. **Verified in code: there is no limiter.** `internal/quiesce` holds no failure counter, no backoff, no breaker, no attempt budget; `runOnce` keeps no state between ticks, and the driver is a plain 5-minute `time.Ticker` (`quiesce.go:149`, default `quiesce.go:107-109`). Had the target stayed down, the loop would have continued indefinitely.
@@ -175,7 +175,7 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
 | R-162 | **`docker diff` is the gate's only witness and its failure mode is quiet** | XS | WATCHING — 2026-08-02 | A limitation, not a defect. The gate's power is `docker diff` excluding mounted paths; on a driver where it is unsupported or lies, the gate degrades to mount-occupancy + writability **and would not say so**. It fails closed (the canary self-test stops reporting BROKEN and the gate then refuses to report), but the message blames the prober rather than the driver. Revisit only if a non-overlay driver ships |
 | R-163 | **`mp1` is RETENTION, not staging — and it is sized as if it were neither** | M | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | Settled by reading: a unit is the KEPT copy on the app's **own** drive (`backup.go:245-255`), falling back to the system SSD for driveless apps — *"the SSD-only system-data fallback"* (`appbackup/paths.go:26-27`); **no post-copy deletion**, the only prune is F5 residue-on-old-drives (`backup.go:1053-1112`). So `mp1` **20 G** retains every driveless app's unit while `mp0` permits **50 G** of volumes, and a DB app's unit is up to **~2×** its data (measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to 0 (`felhom-agent/.../main.go:178`), not derived from the drive; demo-hp's real guest ships 50G/20G. **`mp1` gates the whole chain** — Tier-2 mirrors the unit *"(always)"* (`tier2.go:302,368`) and Tier-3 carries it. Bounded: a unit is volume tars + DB dumps only, **never `mp8` userdata** (`recovery_unit.go:20-25`). **No number proposed.** → `07-backup-architecture.md` §7.5 |
 | R-164 | **C2's chain — the DB volume tar cannot be dropped until a sound dump predicate exists** | S | BLOCKED — on the predicate (2026-08-02) | The unit holds a volume tar **and** a SQL dump and the restore uses both: the dump is authoritative and replayed after the tar so it WINS (F17), with only the DB service up (R-47) — `restore_unit.go:262-266`. Dropping the DB tar would halve DB-app units and close R-127(b)'s initdb-skip trap. **The obvious gate is dead, measured:** `ValidateDump`'s empty-`accounts` warning was **correct** (the DB truly had 0 rows; seeding one stopped the warning and put the row in the dump) — but **a fresh appliance legitimately has zero accounts**, so gating on it blocks every new customer's first backup. Order: sound predicate (dump vs **live** per-table counts) → warn→gate → tar-drop. Pairs with **R-127** |
-| R-90 | **ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86** | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: `Swap: 0B`, nothing in `/etc/fstab`, no swap unit, no `/swapfile` — **swap was never added.** On 2026-07-27 06:58:12 a global OOM fired while `proxmox-backup-proxy` held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration `rsync` held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed **R-88** downstream. Still exposed today: 3.8 GB, no swap, one kill on record. **This gates R-86** — backup-aligned restore-testing puts a multi-GB read on this box *on a schedule*, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console |
+| R-90 | ~~**ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86**~~ | S | **CLOSED 2026-08-03 — the operator rescaled ep0 to a CX33.** MEASURED on the box, not read from an invoice: `Mem: 7757` MB (**8 GB**, was 3.8), `nproc` **4**, and the 4 GiB swapfile added 2026-07-27 **survived the resize** and is active (checked because a resize is a stop/start, so "still there" was an assumption until measured). **The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only — so no disk figure in any runbook needed correcting. Unblocks R-86 | Runbooks corrected; SPIKE/evidence documents ANNOTATED not revised. `target-selection.md`'s "D-d did not name ep0 either way" stays open — it is the operator's |
 | R-91 | **The old 13 GB datastore copy is still on ep0's root disk** | XS | WATCHING — gated on demo-felhom's first post-migration PBS backup | The datastore moved to a Hetzner Cloud Volume on 2026-07-27 (`/dev/sdb`, 100 GiB, attached 06:29:40 UTC, now `/mnt/pbs-datastore`, 13 G used of 98 G). The pre-migration copy survives at **`/srv/pbs-felhom`, 13 G**, on `/` (38 G total, 16 G used, 21 G free). **Do not delete yet:** demo-hp has landed two post-migration snapshots (07-27 08:25:47Z, 09:37:29Z) but **demo-felhom's newest is 2026-07-26T12:21:48Z — before the migration**, so the new volume has not yet proven a write for that namespace. Delete once it has. **Doc drift to fix in the same commit:** `CONTEXT.md:1018` still records the datastore at `/srv/pbs-felhom` |
 | R-92 | **The hub's PBS-DR gauge is 0.1 GB-granular, so small deltas are unverifiable** | XS | idea — 2026-07-27 | The PBS-DR box card rounds to 0.1 GB, which is coarser than the changes an operator wants to confirm after a prune or a GC — a successful prune of a small namespace moves the number by less than one displayed digit, so the UI cannot distinguish "it worked" from "nothing happened". Cosmetic today; it becomes load-bearing the moment retention (R-89) is customer-visible and someone needs to see that a policy change took effect |
 | R-93 | **`drill-r50` is both a blocked customer and the only drift fixture** | XS | idea — 2026-07-27 | The drill customer is blocked in the hub (so it stops alarming) yet it is also the only record exercising the endpoint-drift path R-77 added. Blocking hides it from `GetActiveCustomerIDs`, so the fixture it provides is silently inert — a monitor with no live subject reads exactly like a monitor that passes. Decide: retire it and build a synthetic fixture, or unblock it and silence per-customer instead (the operator has a per-alert silencing feature planned). Related to the R-50 drill VM, now shut down |
@@ -3,7 +3,8 @@
 **Class:** supervised operational run. **No repo version bump** — the only commits are this record
 and the capacity note. **Nothing was deleted.**
-**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner CX23, Nuremberg.
+**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner **CX33 (4 vCPU / 8 GB RAM)**, Nuremberg.
 > **Rescaled 2026-08-03** from the CX23 (2 vCPU / 3.8 GB) this runbook was written against. **The 40 GB local disk did NOT change** — this was a CPU/RAM resize — so every disk figure below still stands. The 4 GiB swapfile added on 2026-07-27 survived the resize.
 **Datastore moved:** `felhom-offsite`, `/srv/pbs-felhom` → **`/mnt/pbs-datastore`** (name unchanged).
 **Window:** 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC.
@@ -229,7 +229,9 @@ as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as
 `keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
 minutes of ep0 CPU per weekly backup.
-> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
+> Watch item (**superseded 2026-08-03**: ep0 is now a **CX33, 8 GB RAM**, and it HAS a 4 GiB swapfile
 > which survived the resize — so the pressure below is much reduced, though the shape of the concern
 > stands). As written: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
 > window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
 > `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.
@@ -307,7 +309,7 @@ Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13
   watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
 4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
   it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
-5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.
+5. ~~**ep0 has no swap** (3.7 GB CX23)~~ — **corrected 2026-08-03: ep0 is a CX33 with 8 GB RAM and an active 4 GiB swapfile.** See the volume runbook's OOM for the original incident.
 ## 11. Observations
@@ -5,7 +5,8 @@
 > firewall, and the hub-driven `felhom-peersync` reconcile surface. Re-running it on a fresh VM
 > re-creates the endpoint from nothing (that is the DR story, step 8).
 >
-> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23, Debian 13,
+> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23 **at the time — rescaled
 > to a CX33, 4 vCPU / 8 GB RAM, on 2026-08-03; the 40 GB local disk is unchanged**, Debian 13,
 > `167.233.158.164` / `2a01:4f8:1c16:7aa1::1`) with hub v0.32.0. The production endpoint is a
 > later re-run of this runbook on a production VM.
 >
@@ -31,7 +32,7 @@ Parameters used throughout (adjust for a new endpoint):
      points at nothing (live-run finding). Home-resolver propagation can lag public DNS by
      minutes — a client-side `wg-quick up` that fails to resolve right after record creation
      just needs a retry.
- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway CX23), not
+- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway box, **CX33 since 2026-08-03**), not
      any production box.
 ## 1. Base (on the box, as root)
@@ -98,7 +98,8 @@ enrolled host. No access route from DooPlex, and nothing here needs one.
 ### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
 Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
-path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune
+path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
 4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching. Do not delete datastores, prune
 jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
 customer documents and photos, on a credential that can still delete (R-95).
 **Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
@@ -139,7 +139,7 @@ is not reachable, for two reasons that are each **already-recorded deliberate po
   `endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
 So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is
-Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90) and fenced by §3 — or *DR tier off* →
+Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90 — **note added 2026-08-03: ep0 has since been rescaled to a CX33 with 8 GB RAM; this journal records what was true when it was written and is deliberately not revised**) and fenced by §3 — or *DR tier off* →
 no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits.
 Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the