Test-box check 2026-11-07: demo-hp space freed (R-548 note), N100 pve-step R6 diary entry (agent 0.172.1), STATUS do-list, R-1228 due date moved (asked)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

documentation/backlog/OPEN-ITEMS.md

@@ -181,7 +181,7 @@ stopping line that lies.
 | **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC |
 | **R-433** | Backup & restore | P3 | **A sub-account cannot reach ANY Storage Box snapshot, by any name — so clause (b) of the 2026-09-01 R-95 re-scope ("the rest is recoverable file by file") is NOT SUPPORTED.** MEASURED 2026-09-01 on `demo-hp` over the credential the box already holds, read-only, no delete verb issued. **The sweep:** a batched `stat -c %n` over Hetzner's port-23 restricted shell (500–600 paths per round trip, stdout carrying only paths that exist) tried **777,600** names of the vendor form `YYYY-MM-DDTHH-MM-SS` across nine full days at second granularity, and 126 alternative shapes — **zero resolved.** **The control is what makes the zero mean anything:** the identical 600-name batch with one real path appended returned it in 6 of 6 batches. **The structural cause:** `/home` (the customer data, `u629488-sub3`) is **st_dev 0,82**; `/.zfs/snapshot` is **st_dev 0,276**; `/home/.zfs` does not exist. A snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding `felhom-repo`. **What still stands:** clause (a) — the box can delete its live repository but cannot WRITE into `/.zfs/snapshot` — is unchanged and re-confirmed. **What is now open again:** the only routes to the older copy are a panel rollback of the WHOLE Storage Box (deletes newer snapshots, hits every customer on it) and the provider API (fenced by §11-D, and `hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method at all**, so it needs new code regardless). **NOT ESTABLISHED, and it is the question that decides whether per-file recovery exists for anyone:** whether the MAIN account can see the snapshots. No main-account credential exists in this project. **The ranking is Viktor's; I am not re-ranking R-95 — but the argument that moved it down is the argument this drill removed.** `audits/evidence-drill-r95-recovery-2026-09-01/` **BLOCKED-ON-PROVIDER 2026-09-01.** The one question that can move this is drafted and ready to send: **Question 1** of `documentation/runbooks/provider-questions-2026-09-01.md` — *can the MAIN account retrieve individual files from a snapshot, without a whole-box restore?* **Neither answer leaves this row where it is:** "yes" makes per-file recovery real but permanently operator-only, and closes this; "no, full restore only" means the snapshots do not bound a single customer's exposure at all, because using them costs every other customer on the box their newer snapshots — **and R-95 becomes urgent.** Nothing here can progress without it, and it is not CC's to send (§11-D). Tracked by a dated check in the DUE-CHECKS block. **⚠ A TRAP ON THE WAY TO THIS ANSWER, found by the operator 2026-09-01 and armed against in the questions file: HETZNER HAS TWO SIMILARLY-NAMED PRODUCTS WHOSE DOCS SAY OPPOSITE THINGS.** **Storage SHARE** (a managed Nextcloud, NOT us) documents *"Currently, we only support restores for the full backup ZFS snapshot to a specific point in time"* (`docs.hetzner.com/storage/storage-share/faq/backup-snapshot/`). **Storage BOX** (ours) documents the opposite — *"You can download individual files or entire directories as usual"* (`docs.hetzner.com/storage/storage-box/snapshots/`). **A web search for the obvious phrasing surfaces the SHARE page, and it reads like a definitive NO.** Tell them apart by the giveaways: the Share page talks about *Nextcloud's data cache*, a *database dump* and the *konsoleH* interface, and never mentions Storage Box. **Why this is filed and not left as trivia: a support agent could answer Question 1 from the wrong page, and that answer would push this row to the top of the register for no reason.** Question 1 now names the product, quotes the Storage Box line, and says explicitly that we know what the Share FAQ says. **If an answer cites full-snapshot-only, check WHICH PRODUCT it is about before acting on it.** **Nothing in this repository ever leaned on the Share claim** — verified by grep at the time; the only vendor line we cite is the Storage Box one. **RE-DATED 2026-09-15 → 2026-09-22 (due-checks gate):** the operator mailbox read through the Gmail connector (`(from:hetzner OR subject:hetzner OR "Storage Box") after:2026/09/01`) holds no Hetzner reply — one match, our own `offsite_snapshots_dropped` alarm. Whether the two tickets were ever sent is not visible to CC; if they were not, sending them is the operator's act. **-- ANSWERED. Hetzner replied on ticket #2026090103040671; the operator supplied the thread on 2026-09-22 after this session searched for it and WRONGLY reported no answer (see the instrument note below and R-628).** **Q1 - can the MAIN account read individual files and directories out of a specific snapshot, without restoring the whole box?** Hetzner: *"With the main account it should be possible to download files and directories from the snapshot as it is stated in the documentation"*, citing `docs.hetzner.com/storage/storage-box/snapshots#access-to-snapshots`; and separately *"A restLine truncated
 | **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC |
-| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. **2026-11-03: demo-hp reached it too.** Its local tier has refused every try since 2026-10-30 („local has 9.4 GiB free; the last archive of guest 9201 was 9.1 GiB"; the disk is 39 GB, the newest archive 10-29), every 15 minutes: the mails `whole_guest_backup_failed` (08:11 CET, 2026-11-03) and `expected_backup_missed` (05:00 CET, „newest backup is 120h old"). PBS and off-site are not this tier. Not caused by the release. | — | — | CC |
+| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. **2026-11-03: demo-hp reached it too.** Its local tier has refused every try since 2026-10-30 („local has 9.4 GiB free; the last archive of guest 9201 was 9.1 GiB"; the disk is 39 GB, the newest archive 10-29), every 15 minutes: the mails `whole_guest_backup_failed` (08:11 CET, 2026-11-03) and `expected_backup_missed` (05:00 CET, „newest backup is 120h old"). PBS and off-site are not this tier. Not caused by the release. **2026-11-07 — the box half again (operator's yes):** still refused every ~4 h („local has 9.4 GiB free … needs about 12.4 GiB"), `expected_backup_missed` daily to 216 h, and the OS updates held behind it (`os_update_stale`). The journal had grown to 3.8 GB (journald's default cap on a 39 GB root); shrunk to 500 MB (`journalctl --vacuum-size=500M`, tLine truncated
 | **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) |
 | **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
 | **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
@@ -210,7 +210,7 @@ stopping line that lies.
 | **R-1206** | Backup & restore | P3 | **Backups written while the box's clock ran ahead stay „the latest" after the clock is fixed.** The rehearsal's scenario #10 (clock set to 2026-10-28) left off-site snapshots dated 2026-10-28; on 2026-10-21 the rebuilt box's recovery page listed every app's „Legutóbbi mentés: 2026. 10. 28. 07:39", and the full restore said „mentés: 2026. 10. 28. 07:37" — a date in the future (`audits/fixes-2026-10-21/P1-live/07-unlock-right.txt`, `11-nextcloud-full-restore.txt`). restic orders snapshots by their own time, so the future ones are „newest" to every retention rule for a week. NOT measured: whether the box's keep-daily prune then drops a real snapshot. Related R-1202, R-1204 (clock). | NARROWED — the NAS tier only. **2026-10-22 night (read, `backup/offbox_window.go:22-24, 130-137`):** on the Felhom off-site tier no real snapshot is lost — the guard refuses a prune while any snapshot is future-dated (R-822, by design). The NAS tier prunes without that guard: open. **2026-10-25:** the kept-data choice no longer takes such a future-dated copy as the newest (R-1271, controller `bd9f4f7`); retention is unchanged — this row. | — | A pure unit test with the policy simulator on the NAS tier and one future-dated snapshot; if a real one is dropped, give the NAS tier the same guard | CC |
 | **R-1212** | Backup & restore | P4 | **Two household texts on the backup path do not follow the box's language:** the share-restore summary (`SharesRestoreResult.FlashMessage`, `controller/internal/backup/shares_restore.go`) is built from Hungarian Go literals, and the off-site `LastError` for an interrupted run is the Hungarian literal „megszakadt futás (a vezérlő újraindult futás közben)" (`controller/internal/backup/backup.go`). An English household reads Hungarian there. Found by the night helper 2026-10-21. | OPEN | — | Bundle keys for both (rendered in the box language when written, R-557), with parity fixtures | CC |
 | **R-1221** | Backup & restore | P4 | **The nightly whole-box backup starts ~1.5–5 min later each night, until one night is skipped (after ~70–80 nights with no press between).** REAL, low harm (night 2026-10-22; split from R-1189 (b)): agent `localapi/server.go:1184` is `age >= Cadence` with the age counted from the archive START; the controller polls every 5 min (`quiesce.go:150`) and the archive starts after the apps stop. Measured on demo-hp: 04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40. At W+6h one night is skipped (44 h is below the 48 h valve, `quiesce.go:807-818`); the hub's 26 h line (`deadline.go:29`) mails the operator. Any manual or off-cycle run resets the drift. Evidence `audits/night-2026-10-22/P3-candidates/demohp-whole-box-completions.txt`. | OPEN | — | Fix: inside the night window a tier is due at `age >= cadence - 4h`; the agent answers due-ness, so it is agent code (a unit test with an injected clock) and rides an agent release | CC |
-| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). **2026-10-30: SWITCHED** (operator yes, 10:02:35 UTC): `felhom.eu` and `www` are Cloudflare Pages (`felhom-site`, uploaded by `scripts/publish-site-cloudflare.sh`); 30 of 30 pages and the installer byte-identical to `installer-v1.36.1` from outside; a fresh install from the public ISO 1.30.0 (throwaway VM 3461 on demo-hp) fetched the installer from the test address and from `felhom.eu` and passed the preflight both times; the contact form posts to `api.felhom.eu` (DooPlex mailer, a test mail arrived). `audits/release-2026-10-30/p3-cloudflare/`. **What is left:** a new install still needs DooPlex for the hub (claim, config, first report) and Gitea (the agent package, the golden) — by design (D35); DooPlex's webpage copy runs until 2026-11-06 as the way back (DUE-CHECKS), then the operator decides to retire it. NARROWED | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) |
+| **R-1228** | Backup & restore | P3 | **The website, the installer script and the hub are all served from the home line — a DooPlex loss stops new installs and the public site.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G8): `felhom.eu`, `hub.felhom.eu` and `gitea.dooplex.hu` resolve to the home address; the installer fetches `felhom.eu/scripts/felhom-host-install.sh`; only the ISO (Cloudflare R2) survives. | OPEN — **2026-10-23 — D35 ruled YES, after the release week (operator):** serve the website and the installer from Cloudflare like the ISO. Dated: start on or after 2026-10-30 (DUE-CHECKS). **2026-10-30: SWITCHED** (operator yes, 10:02:35 UTC): `felhom.eu` and `www` are Cloudflare Pages (`felhom-site`, uploaded by `scripts/publish-site-cloudflare.sh`); 30 of 30 pages and the installer byte-identical to `installer-v1.36.1` from outside; a fresh install from the public ISO 1.30.0 (throwaway VM 3461 on demo-hp) fetched the installer from the test address and from `felhom.eu` and passed the preflight both times; the contact form posts to `api.felhom.eu` (DooPlex mailer, a test mail arrived). `audits/release-2026-10-30/p3-cloudflare/`. **What is left:** a new install still needs DooPlex for the hub (claim, config, first report) and Gitea (the agent package, the golden) — by design (D35); DooPlex's webpage copy runs until 2026-11-06 as the way back (DUE-CHECKS), then the operator decides to retire it. NARROWED **2026-11-07: asked** (demo-check session, in chat); the due date moved to 2026-11-14 to wait for the answer. | the operator's yes | Operator decides (STATUS): A) accept — Gitea and the hub first, the site after; B) the static site + the installer tag on Cloudflare like the ISO — CC's pick, after the release week | CC (ruled yes 2026-10-23, D35) |
 | **R-1229** | Backup & restore | P4 | **ArgoCD's per-app auto-sync settings live only on the cluster, not in git.** DooPlex survey 2026-10-23 night (`audits/dooplex-survival-2026-10-22/PLAN.md`, G9): the Applications differ (`enabled` true/false, prune, selfHeal); a rebuilt cluster from `homelab-manifests` would not have them. | OPEN | — | Write each Application's sync policy into `homelab-manifests` (a homelab change, the operator's yes for the commit that changes live behaviour) | operator |
 | **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC |
 | **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
@@ -406,6 +406,6 @@ stopping line that lies.
 | R-1035 | 2026-11-11 | „felhő tárhely” + „saját felhő tárhely” impressions and average position, and the new page's own impressions/clicks/position, 4 weeks after publishing; operator supplies the read; detail in the R-1035 row |
 | R-528 | 2026-11-12 | no failed oom_check / unexplained app_oom since 2026-10-12; detail in the R-528 row |
 | R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row |
-| R-1228 | 2026-11-06 | D35 way back ends: ask the operator to retire DooPlex's felhom.eu/www webpage copy (ingress + git-sync), or keep it longer; detail in the R-1228 row |
+| R-1228 | 2026-11-14 | D35 way back ends: ask the operator to retire DooPlex's felhom.eu/www webpage copy (ingress + git-sync), or keep it longer; detail in the R-1228 row |
 <!-- DUE-CHECKS-END -->
 | **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC |

documentation/backlog/STATUS-HISTORY.md

@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
 
 **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
 
+## Moved from STATUS on 2026-11-07
+
+**Updated 2026-11-03.** Every box runs controller 0.350.0 with Traefik v3.7.17; the hole is closed, proven on the scratch box. Golden 0.350.0 and agent 0.172.0 are vouched (10:18 CET), hub 0.179.0 is live, and Paperless 3, n8n 3, Mealie and Termix are offered.
+
 ## Moved from STATUS on 2026-11-03
 
 **Updated 2026-11-03 (morning).** On your word, golden 0.349.0 is baked; the code-push gate is green again. It needs your vouch (first line of the release list). It carries Traefik v3.7.16 until controller 0.350.0 (R-1354).