diff --git a/REPORT-night-2026-10-19.md b/REPORT-night-2026-10-19.md index 323064ec..5e494ea0 100644 --- a/REPORT-night-2026-10-19.md +++ b/REPORT-night-2026-10-19.md @@ -1,4 +1,129 @@ -# REPORT — evening/night brief 2026-10-19 (one stop less per night, no false tunnel error, R-893 measured, hub and agent fixes ready, a scanner on every push, ep0's first weekly clean-up) +# REPORT — the night briefs of 2026-10-19 and 2026-10-20 (part 2 run by day): one stop less per night, the safe undo for a failed off-site restore, hub 0.162.0 / agent 0.161.0 / installer 1.35.0 delivered, downloads out of the backups, a scanner on every push + +# PART 2 — the night brief part 2 (run by day, 2026-10-20): your four answers, the safe undo, downloads out of the backups, Immich, the update test + +**Safest readings taken alone (written first, as the brief asks):** +1. **R-1083 is not in agent 0.161.0.** An agent key would make 9202's controller book the agent's whole-guest backups, + one of which writes to ep0 — a change on ep0 the brief forbids; 9202 is also outside the `felhom` pool. +2. **All five downloaders' folders are `excluded`, not only the two named** (qBittorrent, Transmission; also NZBGet, + SABnzbd, MeTube): the brief's gate rule — a download folder is never optional or mandatory — covers all five. +3. **No update-test edge was run:** the bench (guest 9401) and the drill catalog are outside the brief's machines and + repos. The list done and the list left are written; the ask is on the morning list. +4. The brief was written for a night run and ran by day; its „stop by 05:00" was read as a time box, not a clock. + +## Part 2 — in plain words + +- **Morning list:** vouch agent 0.161.0 · decide D21 (Immich) · MeTube's videos out of the second-drive copy — keep? · + the bench and the drill catalog for the update test — yes? +- **Hub 0.162.0 is live.** Its own deploy was the real restart test: Tester-2 (down since 4 October) got no repeat mail + — read in the hub and, separately, in your mailbox. +- **Agent 0.161.0 runs on demo-hp, Tester 1 and demo-felhom.** On demo-hp the agent said „busy" during a backup and + „free" before and after. **Not vouched — that is yours.** +- **Installer 1.35.0 is published.** The served file equals the tag's file. +- **A failed off-site restore now puts the app back exactly as it was** (your D20 = A, controller 0.336.0, the floor). + Proven on the scratch box: after a forced failure the database matched its fingerprints from before, the app ran, + nothing was held; a normal restore then worked. +- **Downloads are out of the backups** (torrent, usenet, MeTube): the controller's own check copies 0 of them now. +- **Immich's off-site copy:** 1.33 GB, of which 102 MB must survive. Two options; pick B (leave out the AI model files). +- **ep0's first weekly clean-up:** done this morning (Part 10 of part 1 below). + +## Part 2 — each part + +| Part | Done? | Why / how | +|---|---|---| +| 1a hub deploy | **done** | `manifests/hub.yaml` → 0.162.0 (`e1d349f1`), only Deployment/hub synced (the only OutOfSync); image `sha256:69285e64…` = the build; Synced/Healthy; reports from all three boxes; floor read 0.335.0; Day-0 agent list has no `-step1`. R-1001 + R-1011 closed. | +| 1b agent release | **done** | R-1083 left out (above). `release-agent.sh 0.161.0` from a clean clone; signed `agent_update` per box: demo-hp 06:38, Tester 1 06:54, demo-felhom 06:55 UTC, each COMPLETED, 76/76, committed. `host_busy` live; tunnel `absent` = the fixture row (9202's real Docker 29 answer); R-1010 by its tests. R-1070 + R-1010 closed. | +| 1b R-921 live | **not done** | The no-stop case happens only on the night schedule (a press backs up the local tier only). Row R-1173 with a dated check after demo-hp's next off-site night. | +| 1c installer | **done** | tag `installer-v1.35.0`, both webpage pins (`b0ae2c57`), webpage Deployment synced alone; served sha256 `f7a196fd…` = tag. R-1167 closed. | +| 2 safe undo | **done** | below. R-893 + R-1169 closed. | +| 3 downloads | **done, one read left** | catalog `c50adfc`, gate `download-binds`; classifier proof. demo-felhom (qBittorrent) runs its next second-drive copy 2026-10-21 03:30 — dated check on R-1019 (NARROWED). | +| 4 Immich | **done (plan only)** | below. R-49 → READY, D21. | +| 5 update test | **not done** | fixture side complete for every installed app; the edges need the bench / drill catalog (above). R-462 NARROWED. | +| 6 ep0 clean-up | **done** | part 1, Part 10. | +| 7 controller release | **done** | 0.336.0 below. | + +## Part 2 — the safe undo, the 9202 proof (`07` §6.3; `audits/night2-2026-10-20/P2/`) + +- Code `93fb251`: the fit check before the stop, the copy right after it (`backups/restore-undo//`), the put-back on + every failure branch, the hold only when the put-back fails, the copy removed on success. 11 tests; nine mutations red. +- **9202, rc.1** (a helper with the fences in full; Rallly + postgres 18; the same forced failure as on 19 October): + +| | Live before | After the forced failure | +|---|---|---| +| public tables | 36 | 36 | +| markers | A, B, C, D | A, B, C, D | +| `r893_only_old` (backup only) | absent | **absent** (on 19 October: present) | +| schema / data / row-count md5 | `05828c77` / `e5c1a72f` / `d8bcbc4b` | **equal** | +| app | running | **running**, no hold | +| the copy | — | 69 MB in 0.9 s, removed after | + + The sentence: „…visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás + előtti állapotba, és az alkalmazás újra fut". Then a normal restore of the same snapshot: the snapshot's content + (A–C, the backup-only table present), the app running, the copy removed. 9202 back on 0.331.0 as found. +- Not measured live (each pinned by a red-proved test): the version-change branch, the `_files` sentence, the refusals + before the stop, a failing put-back, the operator event (9202 has no hub). + +## Part 2 — download binds (`15` §7; `audits/night2-2026-10-20/P3/`) + +| App | Bind (userdata) | Class now | Copied by a tier | +|---|---|---|---| +| qBittorrent | media/torrents | excluded (was optional) | none (was the second drive) | +| Transmission | media/transmission, media/transmission/watch | excluded, excluded | none | +| NZBGet | media/nzbget | excluded (was optional) | none | +| SABnzbd | media/usenet | excluded (was optional) | none | +| MeTube | media/metube | excluded (was optional) | none | +| Sonarr, Radarr | media (rw) | excluded | none | +| Jellyfin, Plex, Emby | media (:ro) | excluded | none | +| Prowlarr | no userdata bind | — | — | + +## Part 2 — Immich (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`) + +Model cache 824 MB (62 %) · raw database folder 283 MB (21 %) · Immich's own backups 115 MB (9 %) · logical dump 54 MB +· originals 49 MB · thumbnails 2 MB. **A** (leave out all three re-fetchable parts, −92 %) needs a never-run restore +from the dump alone on the vectorchord image; **B** (only the model cache, −62 %) needs one restore with an empty cache. +**Pick B now**, A's other parts after their proof. + +## Part 2 — update-test edges + +None run (above). Installed apps: 16; with a fixture: 15; without: bentopdf (keeps no data). Upstream within a major, +waiting for a tested step: PrivateBin 2.0.6→2.1.0, Home Assistant 2026.9.4→2026.10.2, RomM 5.3.1→5.4.0, Plex +1.41.4→1.43.4, Kimai 2.67.0→2.70.0 (`audits/night2-2026-10-20/P5/r462-list.txt`). + +## Part 2 — versions, commits, rows + +- Hub 0.162.0 deployed (`e1d349f1`). Agent 0.161.0 released (`5152123`, tag `v0.161.0`; notes `4e1621c`). Installer + 1.35.0 published (`b0ae2c57`). Controller 0.336.0 the floor (`93fb251`; notes `aa9e09e`, `7ca59e2`), image + `sha256:93cd67d8…8d73`. Catalog `c50adfc`. Evidence `301f4e9e`, `bcaa9576`. +- i18n: no page template changed; seven Go-side keys (HU + EN), one rewritten (R-1169); no fixture changed. +- **System poster:** no listed fact changed — none of the three. +- **Register 160 before, 156 after. Closed 7:** R-1001, R-1011, R-1167, R-1070, R-1010, R-893, R-1169. **Opened 3:** + R-1171 (demo-hp's local copy never restore-tested), R-1172 (the put-back logs at ERROR; an empty folder stays), + R-1173 (R-921's no-stop not yet seen live). Re-stated: R-1019 (NARROWED, dated 2026-10-21), R-1083, R-462 (NARROWED), + R-49 (READY, D21). **Both briefs together: 161 before, 156 after; closed 11, opened 6.** + +## Observations — part 2 + +1. demo-hp's local whole-guest copy is never restore-tested: one copy kept, a 24 h settle, replaced at 24 h. FILED: R-1171 +2. A successful put-back is logged at ERROR, and an empty `backups/restore-undo/` folder stays after a restore. FILED: R-1172 +3. R-921's no-stop branch has no live observation yet (it needs a night schedule). FILED: R-1173 +4. R-462's „only three apps are tested" was out of date: 58 apps have fixtures on both venues. NOT-A-FINDING: the row is re-stated with today's list. +5. The hub deploy at 06:03 UTC removed the old pod's log, which held the 07:10 Budapest off-site audit lines of today (R-895). NOT-A-FINDING: R-895's dated check is 2026-10-21; that run happens in the new pod. +6. In step 4 of the 9202 drill the data-only md5 differed from a read taken before snapshot 2, while the schema and every row count were equal. NOT-A-FINDING: the two reads were at different moments (the read, then the snapshot's own dump); the claim under test — before and after the forced failure — matched on all three fingerprints. +7. The helper's brief expected 37 tables after the normal restore; snapshot 2 has 36. NOT-A-FINDING: last night's 37 was the mixed state, not the snapshot (my arithmetic, corrected). +8. The first background wait loop for demo-hp's agent update printed a time and stopped before the box had updated. NOT-A-FINDING: the version was read directly and a foreground loop used instead. +9. A Python edit of the controller CHANGELOG failed on a quote character, and the commit carried only REPORT.md. NOT-A-FINDING: fixed by the next commit (`7ca59e2`). + +## Part 2 — teardown + +- Machine: 9202 back on 0.331.0 as found (Rallly removed through the product, the SFTP target, images and files removed + by name, settings and the off-box folder put back); demo-hp 9201's probe script removed. +- Host: demo-hp `/tmp` copies removed; nothing on `pve/data`; no whole-guest backup of 9202. +- Hub: per-box floors set and cleared; the global floor raised; nothing else. +- ep0: read only. + +--- + +# PART 1 — the evening/night brief of 2026-10-19 ## In plain words diff --git a/STATUS.md b/STATUS.md index e88d6e83..d6ac5e9b 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for the first real tester (Tester-2): yes. Tester 2 (a laptop) is off; nothing was sent to it.** -**Updated 2026-10-19 (night).** An off-site backup no longer stops the household's apps for nothing when the agent is busy (controller 0.335.0 is the floor since 20:39 local; it works once the next agent is released — morning list). Three agent fixes and three hub fixes are written and tested, NOT released or deployed; the installer's „delete the old host first" message is written, NOT published (morning list). A security scanner now runs on every push in all four repos (warn-first: it lists, it never refuses); the first full scan found no real secret and no called weakness. The ArgoCD danger to three live hub secrets is gone (ArgoCD no longer tracks them). A failed off-site restore was measured on the scratch box — **decide D20**. ep0's first weekly clean-up ran at 05:30 local: it removed only tester-1's 4 October copy, as predicted, and the hub sees a rule for every household. — `REPORT-night-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (late afternoon).** The household passphrase no longer gives a live box's keys (your D19): each box now fetches its config with its own key (controller 0.334.0 is the floor since 16:31 local; hub 0.161.0 is live, your yes). A reinstall over a live box needs you to delete the old host first; the installer does not word that refusal yet (row filed). Every new household on ep0 gets its weekly clean-up rule automatically, and the hub alarms on one without; tester-1 and Tester-2 got theirs (your yes; tester-1's 4 October copy goes tonight). demo-hp kept its own address for a 30-minute watch after your router change. — `REPORT-day3-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (afternoon).** ep0 now keeps one whole-box copy per week for 2 weeks (your yes; the change removed nothing today, the first run is tomorrow 03:30 UTC). Controller 0.333.0 is the floor for every box (12:16 local): signing out on the dashboard now also ends the file manager's own 2-hour pass, and the recovery-code page says the code becomes live after the two words. Hub 0.160.0 is live (your yes): a reinstall within 45 minutes also mails you; a refused backup that repeats is counted, and the alarm repeats at most once a day. The household passphrase was measured: alone it reaches the box's keys and through them every off-site key action — **decide D19**. demo-hp loses its address to another device every few minutes — **fix it in the router** (do-list). — `REPORT-day2-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (day).** One sign-in for the file manager is live on every box (your D16 = yes; controller 0.332.0 is the floor since 05:24 local): „Fájlok ↗" opens the app's folder with no second login; signing out on the dashboard asks again; no app on a box can pretend to be the household (tested from inside an app). Agent 0.160.0 runs on demo-hp, Tester 1 and demo-felhom (your yes): a new recovery code replaces the old one only after the household types two words back; a broken-into box cannot flood the whole-box backups, and the local copies are one per day for 3 days (your D17); an empty partition no longer asks „data will be erased". **Vouch agent 0.160.0** (morning list). Hub 0.159.0 is live (your yes): a refused backup has its own alarm; a reinstall over an old host now mails you instead of staying silent — a first draft that re-issued the codes by itself was stopped by a security review. The old backup program still reads demo-hp's off-site copies after the first real clean-up with the new one. The document-preview error was a broken test file, not the box — `REPORT-day-2026-10-19.md`. **Earlier:** **Updated 2026-10-18 (evening).** A brand-new box was installed from nothing on demo-hp (your test customer Tester4): the emergency password was saved in the hub first, then set, and the web login took it; the box came up on the approved golden and moved to the floor by itself in under 3 minutes; its agent was the approved one. OnlyOffice now opens AND saves documents from FileBrowser and from Nextcloud (before tonight: FileBrowser could not open one, and Nextcloud lost every save while saying „saved") — controller 0.331.0 is the floor for every box (22:48 local). Immich's and Nextcloud's Redis no longer have a disk volume. Three agent fixes are written and tested, NOT released (your yes). Golden 0.331.0 is baked, NOT vouched — `REPORT-evening-2026-10-18.md`. **Earlier:** **Updated 2026-10-18 (day).** Hub 0.158.0 is live (your yes): its own 90-day access log works and writes no secret (tested with a fake token); you can vouch an agent without a new golden; the off-site snapshot check gives its first verdict on 20 October. Controller 0.330.0 is the floor (13:10 local): if the host refuses „retire anyway", the drive's apps start again. Agent 0.159.2 runs on the three test hosts (your yes; no pool check on a good backup) — **vouch it**. Installer 1.34.0 is published (your yes): the emergency password is saved in the hub first, then set. The calculator shows Immich's bigger figure (your yes). A fresh reviewer read last night's code: one medium finding, already closed by today's installer change — `REPORT-day-2026-10-18.md`. **Earlier:** **Updated 2026-10-18 (night).** Controller 0.329.0 is the floor for every box (21:36 UTC, MinAgent 0.158.1), after the three test boxes: each app reaches only its own data folder on the drive; homepage no longer controls Docker, so no app has a special right; when the host refuses to retire a drive, nothing changes and the household reads why; a virtual disk without a serial gets a plain sentence. Immich gets 3 GB (it was killed twice at 2 GB on a 4,000-photo import). Golden 0.329.0 is baked, NOT vouched. Hub 0.158.0 is built, NOT deployed — `REPORT-night-2026-10-18.md`. **Earlier:** **Updated 2026-10-17 (evening).** Controller 0.328.0 is the floor for every box (17:22 local): the box refuses a catalog change that asks for dangerous container rights (your D2 = A). Agent 0.159.1 (a gone, decommissioned drive is forgotten; an orphan thin snapshot is named in the backup failure; new Go) — runs on demo-hp, Tester 1 and demo-felhom; **your vouch in the hub is still open**. New emergency root passwords on demo-hp, Tester 1, demo-felhom (your D1 = A; read them in the hub). Hub 0.157.0 deployed on your yes — `REPORT-agent-allowlist-rotation-2026-10-17.md`. **Earlier:** **Updated 2026-10-17 (afternoon, later).** Controller 0.327.1 is the floor for every box since 12:06 UTC: FileBrowser is published only after its default password is checked (your D11 = A), and the install check uses real memory amounts with a clear message (your D10 = A) — `REPORT-filebrowser-memory-ci-2026-10-17.md`. **Earlier:** Controller 0.326.0 is the floor for every box since 09:55 UTC: FileBrowser's default password is checked at every FileBrowser start. Tester 1 is back on the live app list (test apps and drives removed); six memory limits raised — `REPORT-tester1-limits-filebrowser-2026-10-17.md`. **Earlier today:** controller 0.325.0 was the floor since 08:21 UTC (after the three test boxes): two more password doors count a try before checking it, family sign-ins are stored as fingerprints, a file-share rename always applies; five apps' Redis saves snapshots now, which a full disk cannot damage — `REPORT-burndown-3-2026-10-17.md`. **Earlier (2026-10-17 night):** controller 0.324.0 was the floor for every box since 02:46 UTC (all three test boxes made tonight's off-site backup on it first): a restore now brings the database password back with the data (your 1A), restic 0.19.1 (your 2A), dashboard hardening — `REPORT-night-2026-10-17.md`. **Earlier (2026-10-16 evening):** 0.323.0, Debian 13 — `REPORT-debian13-restic-2026-10-16.md`. **Earlier (midday):** Controller 0.319.0 (the app page in panels, the login in one panel) is the floor, MinAgent 0.158.1 — `REPORT-app-page-tuning-2026-10-16.md`. **Earlier (morning):** Storage identity build B shipped: agent 0.158.1 and golden 0.318.0 vouched by you, controller 0.318.0 is the floor (MinAgent 0.158.1), installer 1.33.0 published — new one-disk boxes get their own data area. Open items: see `documentation/backlog/OPEN-ITEMS.md`. Older notes, day by day: +**Updated 2026-10-20 (day).** Your four answers are done: hub 0.162.0 is live (a restart no longer re-mails a box you know is down — proven on Tester-2), agent 0.161.0 runs on the three test boxes (**vouch it**), installer 1.35.0 is published. A failed off-site restore now puts the app back exactly as it was and starts it (your D20 = A; controller 0.336.0 is the floor since 10:54 local, proven on the scratch box). Downloads (torrent, usenet, MeTube) are out of the backups. Immich's off-site copy: **decide D21**. The bench for the update test needs your word (morning list). — `REPORT-night-2026-10-19.md` (part 2). **Earlier:** **Updated 2026-10-19 (night).** An off-site backup no longer stops the household's apps for nothing when the agent is busy (controller 0.335.0 is the floor since 20:39 local; it works once the next agent is released — morning list). Three agent fixes and three hub fixes are written and tested, NOT released or deployed; the installer's „delete the old host first" message is written, NOT published (morning list). A security scanner now runs on every push in all four repos (warn-first: it lists, it never refuses); the first full scan found no real secret and no called weakness. The ArgoCD danger to three live hub secrets is gone (ArgoCD no longer tracks them). A failed off-site restore was measured on the scratch box — **decide D20**. ep0's first weekly clean-up ran at 05:30 local: it removed only tester-1's 4 October copy, as predicted, and the hub sees a rule for every household. — `REPORT-night-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (late afternoon).** The household passphrase no longer gives a live box's keys (your D19): each box now fetches its config with its own key (controller 0.334.0 is the floor since 16:31 local; hub 0.161.0 is live, your yes). A reinstall over a live box needs you to delete the old host first; the installer does not word that refusal yet (row filed). Every new household on ep0 gets its weekly clean-up rule automatically, and the hub alarms on one without; tester-1 and Tester-2 got theirs (your yes; tester-1's 4 October copy goes tonight). demo-hp kept its own address for a 30-minute watch after your router change. — `REPORT-day3-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (afternoon).** ep0 now keeps one whole-box copy per week for 2 weeks (your yes; the change removed nothing today, the first run is tomorrow 03:30 UTC). Controller 0.333.0 is the floor for every box (12:16 local): signing out on the dashboard now also ends the file manager's own 2-hour pass, and the recovery-code page says the code becomes live after the two words. Hub 0.160.0 is live (your yes): a reinstall within 45 minutes also mails you; a refused backup that repeats is counted, and the alarm repeats at most once a day. The household passphrase was measured: alone it reaches the box's keys and through them every off-site key action — **decide D19**. demo-hp loses its address to another device every few minutes — **fix it in the router** (do-list). — `REPORT-day2-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (day).** One sign-in for the file manager is live on every box (your D16 = yes; controller 0.332.0 is the floor since 05:24 local): „Fájlok ↗" opens the app's folder with no second login; signing out on the dashboard asks again; no app on a box can pretend to be the household (tested from inside an app). Agent 0.160.0 runs on demo-hp, Tester 1 and demo-felhom (your yes): a new recovery code replaces the old one only after the household types two words back; a broken-into box cannot flood the whole-box backups, and the local copies are one per day for 3 days (your D17); an empty partition no longer asks „data will be erased". **Vouch agent 0.160.0** (morning list). Hub 0.159.0 is live (your yes): a refused backup has its own alarm; a reinstall over an old host now mails you instead of staying silent — a first draft that re-issued the codes by itself was stopped by a security review. The old backup program still reads demo-hp's off-site copies after the first real clean-up with the new one. The document-preview error was a broken test file, not the box — `REPORT-day-2026-10-19.md`. **Earlier:** **Updated 2026-10-18 (evening).** A brand-new box was installed from nothing on demo-hp (your test customer Tester4): the emergency password was saved in the hub first, then set, and the web login took it; the box came up on the approved golden and moved to the floor by itself in under 3 minutes; its agent was the approved one. OnlyOffice now opens AND saves documents from FileBrowser and from Nextcloud (before tonight: FileBrowser could not open one, and Nextcloud lost every save while saying „saved") — controller 0.331.0 is the floor for every box (22:48 local). Immich's and Nextcloud's Redis no longer have a disk volume. Three agent fixes are written and tested, NOT released (your yes). Golden 0.331.0 is baked, NOT vouched — `REPORT-evening-2026-10-18.md`. **Earlier:** **Updated 2026-10-18 (day).** Hub 0.158.0 is live (your yes): its own 90-day access log works and writes no secret (tested with a fake token); you can vouch an agent without a new golden; the off-site snapshot check gives its first verdict on 20 October. Controller 0.330.0 is the floor (13:10 local): if the host refuses „retire anyway", the drive's apps start again. Agent 0.159.2 runs on the three test hosts (your yes; no pool check on a good backup) — **vouch it**. Installer 1.34.0 is published (your yes): the emergency password is saved in the hub first, then set. The calculator shows Immich's bigger figure (your yes). A fresh reviewer read last night's code: one medium finding, already closed by today's installer change — `REPORT-day-2026-10-18.md`. **Earlier:** **Updated 2026-10-18 (night).** Controller 0.329.0 is the floor for every box (21:36 UTC, MinAgent 0.158.1), after the three test boxes: each app reaches only its own data folder on the drive; homepage no longer controls Docker, so no app has a special right; when the host refuses to retire a drive, nothing changes and the household reads why; a virtual disk without a serial gets a plain sentence. Immich gets 3 GB (it was killed twice at 2 GB on a 4,000-photo import). Golden 0.329.0 is baked, NOT vouched. Hub 0.158.0 is built, NOT deployed — `REPORT-night-2026-10-18.md`. **Earlier:** **Updated 2026-10-17 (evening).** Controller 0.328.0 is the floor for every box (17:22 local): the box refuses a catalog change that asks for dangerous container rights (your D2 = A). Agent 0.159.1 (a gone, decommissioned drive is forgotten; an orphan thin snapshot is named in the backup failure; new Go) — runs on demo-hp, Tester 1 and demo-felhom; **your vouch in the hub is still open**. New emergency root passwords on demo-hp, Tester 1, demo-felhom (your D1 = A; read them in the hub). Hub 0.157.0 deployed on your yes — `REPORT-agent-allowlist-rotation-2026-10-17.md`. **Earlier:** **Updated 2026-10-17 (afternoon, later).** Controller 0.327.1 is the floor for every box since 12:06 UTC: FileBrowser is published only after its default password is checked (your D11 = A), and the install check uses real memory amounts with a clear message (your D10 = A) — `REPORT-filebrowser-memory-ci-2026-10-17.md`. **Earlier:** Controller 0.326.0 is the floor for every box since 09:55 UTC: FileBrowser's default password is checked at every FileBrowser start. Tester 1 is back on the live app list (test apps and drives removed); six memory limits raised — `REPORT-tester1-limits-filebrowser-2026-10-17.md`. **Earlier today:** controller 0.325.0 was the floor since 08:21 UTC (after the three test boxes): two more password doors count a try before checking it, family sign-ins are stored as fingerprints, a file-share rename always applies; five apps' Redis saves snapshots now, which a full disk cannot damage — `REPORT-burndown-3-2026-10-17.md`. **Earlier (2026-10-17 night):** controller 0.324.0 was the floor for every box since 02:46 UTC (all three test boxes made tonight's off-site backup on it first): a restore now brings the database password back with the data (your 1A), restic 0.19.1 (your 2A), dashboard hardening — `REPORT-night-2026-10-17.md`. **Earlier (2026-10-16 evening):** 0.323.0, Debian 13 — `REPORT-debian13-restic-2026-10-16.md`. **Earlier (midday):** Controller 0.319.0 (the app page in panels, the login in one panel) is the floor, MinAgent 0.158.1 — `REPORT-app-page-tuning-2026-10-16.md`. **Earlier (morning):** Storage identity build B shipped: agent 0.158.1 and golden 0.318.0 vouched by you, controller 0.318.0 is the floor (MinAgent 0.158.1), installer 1.33.0 published — new one-disk boxes get their own data area. Open items: see `documentation/backlog/OPEN-ITEMS.md`. Older notes, day by day: `documentation/backlog/STATUS-HISTORY.md`. Reports: `REPORT-night-2026-10-14.md`, `REPORT-immich-rollout-2026-10-14.md`, `REPORT-seo-felho-tarhely-2026-10-14.md`, `REPORT-immich-repair-2026-10-14.md`, `REPORT-power-cut-fixes-2026-10-14.md`, `REPORT-night-power-2026-10-13.md`, `REPORT-poweroff.md`. ## What works @@ -67,14 +67,22 @@ | D16 | R-1100 | **DECIDED 2026-10-19: yes** — built with both safeguards and live on every box (controller 0.332.0): a one-time pass from the dashboard, and a header name only the box knows. | done | — | — | — | | D17 | R-1149 | **DECIDED 2026-10-19: one per day, 3 days** — agent 0.160.0 on the three test hosts. | done | — | — | — | | D18 | R-1162 | **DECIDED 2026-10-19: A** — ep0 keeps one whole-box copy per ISO week, 2 weeks (`keep-weekly 2` on demo-hp and demo-felhom; nothing removed today; the first run, 2026-10-20 03:30 UTC, removed only tester-1's 2026-10-04 copy, as predicted). tester-1's missing job is R-1165 (your pick: not by hand today). | done | — | — | — | -| D20 | R-893 | After a failed off-site restore of one app, put it back exactly as it was and start it (A), or keep holding it stopped and only make the database part exact (B)? Measured tonight on the scratch box: the box puts the newer rows over the backup's older volume and a table only the backup had stays; the app is held. Note: `audits/night-2026-10-19/P3/design-R-893.md` | **A — copy the app's volumes first** (your D8 already chose this direction): one copy's disk for the restore's minutes; a fit check refuses before any stop | B: no disk, but a bad undo copy leaves an empty database, and the household still waits for support | a failed off-site restore keeps ending in a stopped app only support can bring back | — | +| D21 | R-49 | Immich's off-site copy is 92 % things that come back by themselves (1.33 GB, of which 102 MB is your photos and the database). Leave out only the AI model files now (B: −62 %, no new risk), or also the raw database folder and Immich's own backups (A: −92 %, needs a restore path we have never run)? **CC's pick: B now, A's other parts after their proof.** If you do nothing: every Immich box keeps sending ~12× its real data off-site each night. Note: `audits/night2-2026-10-20/P4/immich-offsite-plan.md`. | open | — | — | — | +| D20 | R-893 | **DECIDED 2026-10-20: A** — a failed off-site restore of one app now puts the app back exactly as it was and starts it (controller 0.336.0, the floor since 10:54 local). Proven on the scratch box: a forced failure left the database exactly as before (same fingerprints), the app running, nothing held; a normal restore then worked. | done | — | — | — | | D19 | R-1159 | **DECIDED 2026-10-19: box key first, then lock** — a live box pulls its config with its own key (controller 0.334.0, the floor), and the passphrase gives no keys once the box has reported on it (hub 0.161.0). Proven on demo-hp: 409, no key, you got the alarm mail. A reinstall needs your delete of the old host first. | done | — | — | — | ## Waiting on you — do -- **Morning list, 2026-10-20 (from the night of 2026-10-19):** - 1. **Deploy hub 0.162.0?** (built `sha256:69285e64…`, from `d92a712e`): a hub restart no longer re-mails a box you already know is down (R-1001); the Day-0 agent list offers only real agent packages (R-1011); a box with no tunnel reads „none set up". **What the deploy does with the three Secrets** (`gitea-creds`, `umami-config`, `healthchecks-config`): **nothing** — ArgoCD no longer tracks them (read 18:21 UTC: none in the app's 25 resources, no tracking mark), so even a whole-app sync with prune leaves them; still sync only the hub, as always (R-953 closed). If not: Tester 2's restart mails continue on every hub restart. - 2. **Release agent 0.161.0?** (code `56aabc7`, CHANGELOG „Unreleased"): `host_busy` for the off-site stop (R-921 — without it the controller's new check has nothing to read), a box with no tunnel no longer fails its OS update (R-1070), the janitor clears a half-removed backup snapshot (R-1010). Then your vouch. If not: the extra app stop and the Tester4-type false error stay. - 3. **Publish installer 1.35.0?** (tag `installer-v1.35.0` + the `webpage.yaml` ref): a re-run on a live box says „the operator deletes the old host first" in EN and HU instead of „unexpected 409" (R-1167). If not: the next person to hit it reads only „409". +- **Morning list, 2026-10-20 (day) — after the night brief part 2, run by day:** + 1. **Vouch agent 0.161.0** (hub → Configuration → Day-0 artifacts): version `0.161.0`, sha256 + `ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5`, config bundle + `485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47` (read from the registry by download). It runs on + demo-hp, Tester 1 and demo-felhom. If not: a new box still installs 0.160.0. + 2. **Decide D21** (Immich's off-site size): CC's pick B — leave out only the AI model files. + 3. **MeTube's downloaded videos** are now out of the second-drive copy, like the torrent and usenet downloads. If you + want MeTube's kept: say so (one line back). If not: nothing to do. + 4. **The update test for more apps needs two things CC may not take alone:** start the test bench (guest 9401 on + demo-hp, now stopped) and commit to the drill catalog. Five installed apps wait for a tested step (PrivateBin, + Home Assistant, RomM, Plex, Kimai). If not: they stay on their tested version, which is safe. - **Nézd meg a demo-hp-n a Calibre-Web „Adatok és mentés" fülét és a Fájlkezelőt** (`felhom.enkisfelhom.hu`, controller 0.321.0, minden dobozon kiadva). A mentés egy részben, egyszerű szavakkal; a Fájlkezelő nem kér frissítést és nem ajánl „Install app"-ot. - **Look at `roms/roms/` on the N100's drive in the file manager (R-1099).** RomM now scans `roms//`. A game uploaded through RomM before today sits one folder deeper: if there are platform folders in `roms/roms/`, move them up into `roms/` and press „Scan” in RomM. If not: those games show under a platform called „roms”. - **Nézd meg a Calibre-Web oldalát a demo-hp-n** (`felhom.enkisfelhom.hu`, controller 0.319.0, minden dobozon kiadva). A fejléc, a fülsor és minden rész most külön panel (az „A” stílus). A belépési adatok egy panelben vannak, egymás mellett. A Calibre-Web jelszaváról a doboz nem tud, mert az alkalmazás előbb települt, mint hogy a sablonban jelszó lett volna — ezt most egy mondat mondja ki. Ha nem tetszik, írd meg, mit változtassak. Ha nem nézed meg: semmi nem romlik el. diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index d49dab91..a9430a1c 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -736,6 +736,19 @@ stayed (37 tables, the live app had 36); the definition did not move (same versi answered 409. The restore page's „the database is back as it was" is false for it. Design note with the two options and the pick (A: a copy of the app's live volumes, put back on failure — one copy's disk for the restore's length, a fit check before the stop): `audits/night-2026-10-19/P3/design-R-893.md`; the decision is on `STATUS.md`. +**`[DESIGN]` 2026-10-20 — „put back exactly as it was" (R-893 option A, operator decision D20 = A, `09` §3 decision 192 +second half; controller 0.336.0 is the floor).** Before the stop, every named volume the snapshot will replace is measured +and its copy is added to the restore's own free-space need (R-357's exact rule); too little room or an unmeasurable volume +refuses with nothing stopped. Right after the stop the volumes are tarred into `/backups/restore-undo//` +(outside every capture set). On ANY failure after a volume or the definition moved, the copies are poured back, a volume +the snapshot added is removed, the live definition is written back and the app is STARTED; the household reads that the +restore failed and nothing changed (with placed files: its database and volumes are back, the copied files stay); the +operator gets `backup_run_failures` leg `restore-put-back`. The hold (`restore_mixed`) remains only for a put-back that +fails — then the copy is kept for support — and the double failure (replay AND rollback) holds as before. On success the +copy is removed. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same +forced failure as on 2026-10-19 → 36 tables, the backup-only table absent, schema/data/row-count md5 equal to before, +the app running, no hold, the copy (69 MB, 0.9 s) gone; a normal restore of the same snapshot then worked. The R-893 +known limit above is closed for named volumes; placed FILES are still not taken back (the plain-files mix, as ruled). **[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.** Recorded here rather than only in a closed register row, because a decision that survives only inside diff --git a/documentation/audits/night2-2026-10-20/P4/immich-offsite-plan.md b/documentation/audits/night2-2026-10-20/P4/immich-offsite-plan.md new file mode 100644 index 00000000..e0156713 --- /dev/null +++ b/documentation/audits/night2-2026-10-20/P4/immich-offsite-plan.md @@ -0,0 +1,45 @@ +# R-49 — Immich's off-site copy: what it holds, and two ways to shrink it (2026-10-20) + +Status: **measure and plan only — nothing changed.** Night brief part 2 (run by day), Part 4. Architecture: `07-backup-architecture.md` +(the off-site tier = the recovery unit + the app's mandatory data). Measured read-only on scratch guest 9202 (demo-hp), Immich +v3.2.4 + `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0`, data on the drive `/mnt/sizing-bench`, 09:0x UTC. + +## What today's off-site copy holds + +The off-site tier copies the recovery unit (`backups/primary/immich/`, 1 160 553 825 B) and the mandatory drive folder +`appdata/immich/` (166 077 159 B) — about **1.33 GB** in all. The external library `media/photos` is `optional` (not off-site; +0 B here anyway). + +| Part | Where | Bytes | Share | Can it come back without the copy? | +|---|---|---:|---:|---| +| ML model cache | unit `volume-dumps/immich_immich_ml_cache.tar` (volume 823 835 298 B) | 823 764 480 | 62 % | **Yes** — Immich downloads the models again when a smart-search or face job first needs them | +| Raw database folder | unit `volume-dumps/immich_immich_postgres_data.tar` (volume 282 169 461 B) | 283 049 472 | 21 % | **Yes, if** a restore from the logical dump alone is proven (below) | +| Immich's own database backups | `appdata/immich/backups/` | 115 367 547 | 9 % | Yes — a second copy of what the logical dump already holds; it grows every night | +| Logical database dump | unit `db-dumps/immich-postgres.sql` | 53 711 763 | 4 % | **No** — this IS the household's albums, people, metadata | +| Originals | `appdata/immich/upload/` | 48 673 663 | 4 % | **No** — the photos themselves | +| Thumbnails | `appdata/immich/thumbs/` | 2 035 910 | <1 % | Yes — Immich regenerates them | +| Redis | volume `immich_immich_redis_data` (31 594 479 B) | — | — | not in the unit (a queue cache) | + +**What must survive: 102 MB of 1.33 GB (8 %).** The 2026-07-19 measurement (R-49) said the same shape: ~60 % model cache. + +## Two options + +| | A — leave out all three re-fetchable parts | B — leave out only the model cache | +|---|---|---| +| Off-site size here | ~105 MB (dump + originals + thumbnails + unit metadata) — **−92 %** | ~510 MB — **−62 %** | +| What a restore then needs | the models download again (~824 MB, from Immich's model host) before smart search and face recognition work again — the photos, albums and timeline work at once; the database is rebuilt from the logical dump alone into an EMPTY volume (no raw folder to lay under it) | the models download again, as in A; the database comes back as today (volume, then dump) | +| The proof it needs | (1) one off-site restore of Immich onto a box with an empty model cache: photos and albums read back, then a smart search answers after the download (time measured); (2) **one off-site restore from the dump alone** into an empty database volume, on this vectorchord image: the extensions and vector indexes come back, the row counts match, Immich starts and searches — today's restore always lays the raw folder first, so this has never been run | only proof (1) | +| Risk | the dump-only database restore is the unmeasured part; if the vector extension needs more than the dump gives, a restore would fail where it works today (option A's put-back keeps the app as it was, but the backup would be weaker) | none new: the cache is a cache | +| Also | Immich's own `backups/` folder needs a catalog class (`excluded`) — a template change in the `hdd` block | a catalog change: the model-cache volume marked as not captured | + +Not measured today: the model download's time and size on a real line (the 824 MB is the cache as it stands, which holds every +model this box used); the off-site bytes after restic's compression and de-duplication (the table is the plain size). + +## Pick: B now, then A's other two parts after proof (2) + +B removes 62 % of the copy with no new risk — the strongest case R-49 named first — and needs only proof (1). The raw database +folder and Immich's own backups are another 30 %, but leaving out the raw folder changes HOW the database comes back, and that +path has never run for this extension image; it should be a separate step with its own proof, not ride along. + +**If the operator does nothing:** every Immich household keeps sending ~12× its real data off-site each night, against its +quota and transfer. diff --git a/documentation/audits/night2-2026-10-20/P7/01-floors-and-move.txt b/documentation/audits/night2-2026-10-20/P7/01-floors-and-move.txt new file mode 100644 index 00000000..fc5c721a --- /dev/null +++ b/documentation/audits/night2-2026-10-20/P7/01-floors-and-move.txt @@ -0,0 +1,4 @@ +# per-box floors 0.336.0 / MinAgent 0.158.1 set 08:51:06Z; image sha256:93cd67d871125877e935a024765075b5a0b7c0e7683e1165976ff7a654fe8d73 (clean clone 93fb251; rc.1 on 9202 first) +demo-felhom|0.336.0|2026-10-20 08:54:55 +demo-hp|0.336.0|2026-10-20 08:54:56 +tester-1|0.336.0|2026-10-20 08:54:54 diff --git a/documentation/audits/night2-2026-10-20/P7/03-box-walk.txt b/documentation/audits/night2-2026-10-20/P7/03-box-walk.txt new file mode 100644 index 00000000..529a3c5a --- /dev/null +++ b/documentation/audits/night2-2026-10-20/P7/03-box-walk.txt @@ -0,0 +1,27 @@ +=== demo-hp (enkisfelhom.hu) 08:53:38 +containers: 24 running; not healthy/running: traefik Up 4 days; +unhealthy: 0 +2026/10/20 08:51:19 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.335.0 → 0.336.0) +allow-list refusals since start: 0 +2026/10/20 08:51:19 sync.go:142: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +installed apps (stacks with app.yaml): 12 +public https://felhom.enkisfelhom.hu/: 302 +public https://files.enkisfelhom.hu/: 200 +=== demo-felhom (demo-felhom.eu) 08:53:41 +containers: 9 running; not healthy/running: traefik Up 4 days; +unhealthy: 0 +2026/10/20 08:51:08 [INFO] [selfupdate] Post-update startup: update successful (0.335.0 → 0.336.0) +allow-list refusals since start: 0 +2026/10/20 08:51:11 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +installed apps (stacks with app.yaml): 5 +public https://felhom.demo-felhom.eu/: 302 +public https://files.demo-felhom.eu/: 200 +=== tester-1 (enkicsifelhom.hu) 08:53:42 +containers: 10 running; not healthy/running: traefik Up 4 days; +unhealthy: 0 +2026/10/20 08:51:18 [INFO] [selfupdate] Post-update startup: update successful (0.335.0 → 0.336.0) +allow-list refusals since start: 0 +2026/10/20 08:51:19 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +installed apps (stacks with app.yaml): 3 +public https://felhom.enkicsifelhom.hu/: 302 +public https://files.enkicsifelhom.hu/: 200 diff --git a/documentation/audits/night2-2026-10-20/P7/04-https-only.txt b/documentation/audits/night2-2026-10-20/P7/04-https-only.txt new file mode 100644 index 00000000..8682d689 --- /dev/null +++ b/documentation/audits/night2-2026-10-20/P7/04-https-only.txt @@ -0,0 +1,33 @@ +control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -) +== tester-1 (enkicsifelhom.hu): 6 public name(s) + OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1 + OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 + OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1 + OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1 + OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1 +== demo-hp (enkisfelhom.hu): 14 public name(s) + OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1 + OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1 + OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1 + OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1 + OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1 + OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1 + OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1 + OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 + OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1 + OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1 + OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1 + OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1 + OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1 + OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1 +== demo-felhom (demo-felhom.eu): 8 public name(s) + OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1 + OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1 + OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1 + OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 + OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1 + OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1 + OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1 + OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1 +== every public name redirects http -> https diff --git a/documentation/audits/night2-2026-10-20/P7/05-files-one-sign-in.txt b/documentation/audits/night2-2026-10-20/P7/05-files-one-sign-in.txt new file mode 100644 index 00000000..6c97134d --- /dev/null +++ b/documentation/audits/night2-2026-10-20/P7/05-files-one-sign-in.txt @@ -0,0 +1,16 @@ +== demo-hp +dashboard sign-in: 302 +the Fajlok button goes to: https://files.enkisfelhom.hu/_felhom/sso +the pass answer goes to: https://files.enkisfelhom.hu/files/; files cookie set: 1 +FileBrowser with the files cookie: felhom-household admin: False modify: True +FileBrowser without it, a fresh browser: HTTP 401 +dashboard sign-out: 302 +after the sign-out, the files cookie sends the browser to: https://felhom.enkisfelhom.hu/__files/open?to=%2Ffiles%2F +== tester-1 +dashboard sign-in: 302 +the Fajlok button goes to: https://files.enkicsifelhom.hu/_felhom/sso +the pass answer goes to: https://files.enkicsifelhom.hu/files/; files cookie set: 1 +FileBrowser with the files cookie: felhom-household admin: False modify: True +FileBrowser without it, a fresh browser: HTTP 401 +dashboard sign-out: 302 +after the sign-out, the files cookie sends the browser to: https://felhom.enkicsifelhom.hu/__files/open?to=%2Ffiles%2F diff --git a/documentation/audits/night2-2026-10-20/P7/06-global-floor.txt b/documentation/audits/night2-2026-10-20/P7/06-global-floor.txt new file mode 100644 index 00000000..e3b6a747 --- /dev/null +++ b/documentation/audits/night2-2026-10-20/P7/06-global-floor.txt @@ -0,0 +1,22 @@ +2026-10-20T08:54:35Z +global floor: +HTTP/1.1 303 See Other +Location: /configuration?flash=floor_set +read back (hub DB with WAL): +min_controller_version|0.336.0 +min_controller_version_declared_min_agent|0.336.0=0.158.1 +clear the per-box floors: +== tester-1 +HTTP/1.1 303 See Other +Location: /customers/tester-1?flash=floor_set +== demo-hp +HTTP/1.1 303 See Other +Location: /customers/demo-hp?flash=floor_set +== demo-felhom +HTTP/1.1 303 See Other +Location: /customers/demo-felhom?flash=floor_set +Error: in prepare, no such table: customers +per-box floors after the clear (customer_configs): +demo-felhom|''|'' +demo-hp|''|'' +tester-1|''|'' diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index a641952a..718dbd55 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,18 @@ --- +## 2026-10-20 (night brief part 2, run by day) — hub 0.162.0, agent 0.161.0, installer 1.35.0, the safe undo for a failed off-site restore (full text of each row: `git show bcaa9576:documentation/backlog/OPEN-ITEMS.md`) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-1001** | **Every hub restart re-mails the operator about a box that is already known to be down.** (P3) | CLOSED 2026-10-20 — **hub 0.162.0 deployed (operator yes) and the deploy itself was the real restart:** Tester-2 (down since 2026-10-04) had been re-mailed on every earlier restart (host_down + two restore_test_stale at 02:15, 09:30, 14:12 on 2026-10-19). After the 06:03 UTC restart: `Host staleness checker initialized: … 1 already alarmed (seeded silently, R-1001)`, `restore-test staleness: Tester-2 … already reported before the restart — not re-sent` for both tiers, 0 new events (id > 6129), and the mailbox (a different channel) holds no Tester-2 host_down / restore_test_stale mail after 14:12 on 2026-10-19 while the daily 03:00 mails are there. | `audits/night2-2026-10-20/P1/1a-hub-deploy.txt` | +| **R-1011** | **The Day-0 „Agent" list offers step-bundle packages (`0.156.1-step1`, `0.152.0-step1` …) as agent versions; one has no agent binary, so the hub shows the bundle JSON's sha (`FileSHA256` falls back to the first file) and a vouch would point fresh installs at a package with no binary.** (P3) | CLOSED 2026-10-20 — **hub 0.162.0 deployed:** the Day-0 agent list (GET /configuration with CC's token) offers 0.160.0 … 0.150.0 and no `-step1` package. | `audits/night2-2026-10-20/P1/1a-hub-deploy.txt` | +| **R-1167** | **The installer does not word the hub's new 409 („this customer already has a live box").** (P4) | CLOSED 2026-10-20 — **installer 1.35.0 published (operator yes):** tag `installer-v1.35.0` (→ `e1d349f1`), both webpage git-sync pins, the webpage Deployment synced alone; `https://felhom.eu/scripts/felhom-host-install.sh` serves `SCRIPT_VERSION="1.35.0"` with sha256 `f7a196fd…` = the tag's file (before: 1.34.0, `df722ee4…`). | `audits/night2-2026-10-20/P1/1c-installer-publish.txt` | +| **R-1070** | **A box with no Cloudflare tunnel gets an ERROR after its host OS update: „NOT healthy … (the tunnel is unknown)”.** (P3) | CLOSED 2026-10-20 — **agent 0.161.0 released and on demo-hp, Tester 1 and demo-felhom** (signed jobs, all committed, 76/76; hub 0.162.0 shows `absent` as „none set up"). The agent's own probe in 9202 (no tunnel) answers `error: no such object: cloudflared` rc 1 — exactly the `TestClassifyTunnel` „absent, docker 29" fixture row; 9201 answers `running`. | `audits/night2-2026-10-20/P1/1b-agent-release.txt` | +| **R-1010** | **The agent's stale-lock janitor cannot remove a half-removed `vzdump` snapshot: Proxmox's delete fails on the first missing volume („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump") and the janitor retries the same call every 10 min forever, leaving the other volume's snapshot and the snapshot entry.** (P3) | CLOSED 2026-10-20 — **agent 0.161.0 released and on the three test hosts.** Proven by its red-proved tests (`TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone`, `TestDeleteSnapshotForce_R1010`), rerun green on the released commit; a half-removed snapshot on `pve/data` was not made (rule 6). | `audits/night2-2026-10-20/P1/1b-agent-release.txt` | +| **R-893** | **After a failed OFF-SITE replay, the rollback pours the NEWER pre-restore copy over the OLDER volume just put back.** (P3) | CLOSED 2026-10-20 — **option A built and released (operator D20 = A): controller 0.336.0 is the floor (08:54 UTC).** Before the stop the live volumes the snapshot will replace are measured and the copy must fit (refused with nothing stopped otherwise); right after the stop they are tarred into `backups/restore-undo//`; on any failure after a volume or the definition moved they are poured back, a volume the snapshot added is removed, the live definition written back and the app started; a put-back that fails holds and keeps the copy. 11 unit tests, nine mutations red. **Live on 9202** (rc.1): a forced replay failure after the volume was replaced → 36 tables, markers A–D, the backup-only table ABSENT, schema/data/row-count md5 EQUAL to before, Rallly running, no hold, the copy gone; then a normal restore of the same snapshot worked. `07` §6.3 now states it as the design. Follow-up: R-1172. | `audits/night2-2026-10-20/P2/` (unit red-proofs, `live/` the 9202 drill), `P7/` (release) | +| **R-1169** | **After a failed off-site restore that is held (`restore_mixed`), the restore page says the database is back as it was and the app's files and other data come from the backup — measured false on both halves.** (P3) | CLOSED 2026-10-20 — **controller 0.336.0:** `err.backup.db_restore_failed_held_mixed` says only what is certain (stopped half-way, the data could not be returned, the app stays stopped, the copy is kept, support helps), HU + EN; the plain rolled-back case with placed files no longer claims „your data is back as it was" (it says the files stay). Pinned by `assertR893Held` / `TestR893A_AFailedPutBackHoldsTheAppAndKeepsTheCopy`. | `audits/night2-2026-10-20/P2/redproof-unit.txt` | + ## 2026-10-19 (evening/night brief) — no stop for a busy host, the scanner on every push, the ArgoCD prune danger gone (full text of each row: `git show f92fa5a8:documentation/backlog/OPEN-ITEMS.md`) | Row | What | Closed | Evidence | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 508eddb1..41ef26cc 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -112,7 +112,7 @@ match what the reader sees is how an instrument stops being believed (R-421). It **nothing was proven on the day this line was drawn** and a stopping line that moves a status is a stopping line that lies. -## Install & onboarding — 8 rows (P3 5, P4 3) +## Install & onboarding — 7 rows (P3 5, P4 2) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -123,7 +123,6 @@ stopping line that lies. | **R-993** | Install & onboarding | P3 | **Operator: approve the minimum machine in `documentation/audits/sizing-2026-10-13/SIZING.md` §5** (the input R-991's hardware page needs). MEASURED 2026-10-13 on the N100 demo box (61 apps deployed and removed through the product, 15 min at rest each; heavy apps with one real action). Proposal: **minimum** x86-64, 4 cores (N100 class), **8 GB**, 256 GB SSD; **recommended 16 GB**, 512 GB SSD; out: ARM / Raspberry Pi, Synology-QNAP, routers, cloud VPS. Worked from the floor (2.5 GB for the whole box with no apps) + Immich, Jellyfin, Nextcloud, Home Assistant, Vaultwarden, Paperless-ngx at their peaks (5.0 GB) + 50 % headroom ≈ 12.3 GB. 8 GB runs that set without Immich; 16 GB runs all of it twice over. Data for the estimator: `sizing.json`. **2026-10-13 (estimator session) — the installer accepts a host BELOW this minimum:** `autosize_guest_caps` has no floor on the host, only on the guest; on a 4 GB box (MemTotal ≈ 3.5 GiB) the guest gets host − 1 GB ≈ 2.4 GB and Proxmox keeps ≈ 1 GB, where it used ≈ 2.0 GB on the N100 (SIZING.md §1). Whatever minimum is approved, nothing enforces it at install today. The three-verdict wording of §5 now matches the estimator page (`/gepigeny`, R-997). | **WAITING-ON-OPERATOR** — filed 2026-10-13 | the operator's decision | Operator: approve or change §5. If nothing is done: no hardware page; a buyer with a Raspberry Pi can still download the installer (R-991 stays open) | operator | | **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC | | **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator | -| **R-1167** | Install & onboarding | P4 | **The installer does not word the hub's new 409 („this customer already has a live box").** Since hub 0.161.0 (R-1159) a re-run of `felhom-host-install.sh` against a customer whose box is live (a repair, a late `--resume`, a reinstall without the operator's delete) gets 409 from `GET /config` at its preflight; the script prints only „unexpected hub status 409 on config preflight" (`step_preflight`), not the hub's sentence (delete the old host first). The installer is published (felhom.eu/scripts), so the fix is an installer release. Found 2026-10-19. | **READY — written in installer 1.35.0 (`d92a712e`, `5d0dfeac`), NOT published** (the morning question): `die_live_box` says it in EN and HU at the preflight, host-enroll and the controller.yaml fetch. Harness cases (static + the real functions against a local fake hub), red-proved; without curl (the CI runner) the two behaviour cases skip out loud. Close after the publish. OPEN — filed 2026-10-19 | `scripts/felhom-host-install.sh` `step_preflight` | Next installer release: a 409 case that prints the hub's body | CC | ## Apps & catalog — 13 rows (P3 5, P4 8) @@ -149,19 +148,18 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-440** | App updates | P3 | **[P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible.** `compose pull` on a moving tag fetches whatever upstream published that day. **MEASURED 2026-09-01 over `app-catalog-felhom.eu` @ `29edad9c5bf4`: 79 `image:` lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version.** `postgres:16-alpine` (8 apps), `redis:7-alpine` (6), `mariadb:11.6` (2), plus one each of `postgres:15-alpine`, `postgis/postgis:16-3.5-alpine`, `mariadb:11.4`, `mariadb:12.3`, `ghcr.io/claperco/claper:2.5`, `ghcr.io/thomiceli/opengist:1.13`, `wger/server:2.6`. **A 24th is arguable and is recorded rather than rounded away:** `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. **Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists**, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. **MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control.** Running digests on demo-hp compared against what the registry serves for the same tag today: **`mariadb:11.4` MOVED** (`sha256:4f1d8d20...` -> `sha256:611a2fcc...`) and **`mariadb:12.3` MOVED** (`sha256:a02fe89c...` -> `sha256:dd9b303a...`), while `postgres:16-alpine`, `redis:7-alpine`, `mariadb:11.6` and `opengist:1.13` were SAME — **and both fully-pinned CONTROLS (`rommapp/romm:5.0.0`, `privatebin/pdo:2.0.5`) were SAME.** So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under `romm` and `bookstack`, with no catalog change and no record. **Compounding fact found while reading:** the recovery unit records `ImagePins` but the manifest comment says *"image NOT stored - re-pulled on restore"*, so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. **HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): `app.yaml.installed_images` now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade.** What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). **The row therefore stays OPEN and its rank is unchanged** — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. `audits/SPIKE-app-update-2026-09-01.md` **— NIGHT 2026-09-23:** every image moved tonight (and the 21 backfilled moves) carries its resolved digest in the catalog's test record; the floating pins themselves still float until the box pulls by digest (`09` §6.4 part 6). **-- MEASURED 2026-09-30 on demo-hp 9201: a fresh install or a guarded Update renders `name:tag@sha256` from the ladder (`digest.go` `RenderWithLadderDigests`) — 9 of 20 services there run such a definition (adventurelog, docmost, paperless-ngx). The other 11 run TAG-ONLY definitions: bookstack, kimai, opengist, privatebin, romm were installed before v0.269.0 and not updated since, and `CarryDigests` (`digest.go:141`) keeps only a digest the running definition already has; bentopdf and calibre-web have no ladder at all. So a re-pull of those (a restore) takes whatever the tag serves that day. `audits/pg-last-six-2026-09-30/E/E2-digests-demo-hp.txt`.** **-- 2026-09-30 (evening):** calibre-web now has a ladder entry (`53a4a1d`), so a new install or a guarded Update renders its tested digest; bentopdf still has none. `audits/more-night-apps-2026-09-30/` **Merged 2026-10-05 from R-446 (duplicate):** the household-visible consequence — for these apps the „Naprakész" badge can never turn „behind" (`felhom-controller controller/internal/stacks/updateorder.go:96` returns false with no catalog digests / no test date); R-740 (same-tag re-test) is closed, so floating-tag drift for laddered apps is handled. | **NARROWED 2026-09-30 — floating only for (1) apps with no ladder entry and (2) apps installed before v0.269.0 until their next guarded Update; owner: CC.** **Re-ranked 2026-10-03: P2→P3: narrowed to apps with no ladder entry and pre-v0.269 installs; each guarded update fixes the latter.** | — | — | CC | -| **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s, measured). The catalog currency and the next-apps ranking: `audits/catalog-currency-2026-09-30.md` — 25 of 53 apps behind upstream inside a major, 19 across one (morning); after the session **31 apps can update themselves at night (+ nextcloud with a whole copy)**, 32 carry a proven step, 21 have no ladder.** **-- 2026-09-30 (evening, more-night-apps brief): ten more steps published on both venues** — first ladders for calibre-web, gitea, wger, crafty-controller, uptime-kuma, zipline (new front-door fixtures; gitea's installer form), within-major steps for emby, ghost, home-assistant, outline, rallly. **Night-updatable by the audit's method (a): 30 + 2 conditional at the session's start → 35 + 3 conditional** (calibre-web conditional: `files_may_change`). zipline got a two-step ladder 4.6.1 → 4.7.0 → 4.8.0 (the direct jump fails, R-742). Stopped: wanderer (the bench cannot run it, R-739). Fixture/harness defects fixed on the way: R-735, gitea's installer, the HTTPS backend on the bench, per-file names for `files_may_change`. `audits/more-night-apps-2026-09-30/` | **READY — rank P2-MEDIUM; owner: CC (scope already ruled, `09` §3 decision 6)** **Re-ranked 2026-10-03: P2→P3: 35+ apps already update by proven steps; an app without a ladder simply does not move, which is safe.** | — | — | CC | +| **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s, measured). The catalog currency and the next-apps ranking: `audits/catalog-currency-2026-09-30.md` — 25 of 53 apps behind upstream inside a major, 19 across one (morning); after the session **31 apps can update themselves at night (+ nextcloud with a whole copy)**, 32 carry a proven step, 21 have no ladder.** **-- 2026-09-30 (evening, more-night-apps brief): ten more steps published on both venues** — first ladders for calibre-web, gitea, wger, crafty-controller, uptime-kuma, zipline (new front-door fixtures; gitea's installer form), within-major steps for emby, ghost, home-assistant, outline, rallly. **Night-updatable by the audit's method (a): 30 + 2 conditional at the session's start → 35 + 3 conditional** (calibre-web conditional: `files_may_change`). zipline got a two-step ladder 4.6.1 → 4.7.0 → 4.8.0 (the direct jump fails, R-742). Stopped: wanderer (the bench cannot run it, R-739). Fixture/harness defects fixed on the way: R-735, gitea's installer, the HTTPS backend on the bench, per-file names for `files_may_change`. `audits/more-night-apps-2026-09-30/` | **READY — rank P2-MEDIUM; owner: CC (scope already ruled, `09` §3 decision 6)** **Re-ranked 2026-10-03: P2→P3: 35+ apps already update by proven steps; an app without a ladder simply does not move, which is safe.** **NARROWED 2026-10-20 (night brief part 2, Part 5):** the fixture side is DONE for every app installed on a test box — 58 box fixtures run on both venues; of the 16 installed apps only bentopdf has none (it keeps no data). Upstream within a major (check-currency, registries read): privatebin 2.0.6→2.1.0, home-assistant 2026.9.4→2026.10.2, romm 5.3.1→5.4.0, plex 1.41.4→1.43.4, kimai 2.67.0→2.70.0 — the list left. **No edge was run:** the bench (guest 9401, stopped) and the drill catalog the box venue needs were outside the brief's machines and repos; the operator's word for both is on STATUS's morning list (`audits/night2-2026-10-20/P5/r462-list.txt`). | — | — | CC | | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC | | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC | -## Backup & restore — 26 rows (P2 3, P3 12, P4 11) +## Backup & restore — 27 rows (P2 3, P3 12, P4 12) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest | — | — | operator | -| **R-893** | Backup & restore | P3 | **After a failed OFF-SITE replay, the rollback pours the NEWER pre-restore copy over the OLDER volume just put back.** Read in source 2026-10-06 (R-638 option A, not measured): `internal/backup/offbox_reconstitute.go` writes the undo copy from the live (newer) database, replaces the volumes with the snapshot's older tars, then — when the replay fails — `rollbackSafetyDump` loads that newer dump over the older database volume. The loader only drops what the dump knows, so tables the newer migration removed stay; and when the snapshot's older definition was written, the rollback branch does not put the newer definition back, so the older app starts on rolled-back data; non-database volumes stay at the snapshot's state. An order change cannot fix it (the only undo is a logical dump, and its volume was replaced). Known limit in `07` §6.3. **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-893.md` — re-verified; and the screen's „your data is back as it was" (`err.backup.db_restore_failed_rolled_back`) is false for files, other volumes and the app version. Question D8 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D8 (`09` §3 decision 192):** yes, both, in that order — first the app stays stopped for support (option C), next „put back exactly as it was" (option A). | **READY — measured 2026-10-19 on 9202 and a design note written; the decision is the operator's** (STATUS D20). A forced replay failure after the volume was replaced (Rallly + postgres 18, scratch SFTP off-site on 9202's own disk): the undo dump's newest rows on top of the backup's volume, a table only the backup had left behind (37 tables against 36), definition unchanged (same version), app held (`restore_mixed`, start → 409). Note: `audits/night-2026-10-19/P3/design-R-893.md` — A (copy the app's live volumes before the replay, put them back on failure; one copy's disk for minutes, a fit check before the stop) or B (empty the DB volume before the undo load; no disk, a new way to lose the database). Pick A. The version-change branch was not measured (it needs a drill-catalog commit). The page's false sentence: R-1169. **NARROWED — 2026-10-09: the first half DELIVERED in controller 0.304.0. Slice 0 (the live measurement) NOT run: 9202 has no off-site, and a dump cannot be cut by hand inside an encrypted off-site copy on Tester 1; it needs a scratch off-site repo — a plan, not a finding.** **NARROWED 2026-10-08 — the first half (D8 option C, the app held stopped) is built on controller main and ships tomorrow; what stays open is the second half, „put back exactly as it was” (option A: a pre-restore copy of the app's volumes and placed files, a fit check, disk for one copy; the slice-0 measurement on 9202 first).** **OPEN — filed 2026-10-06** **2026-10-06 night: verified in source, no code** — `offbox_reconstitute.go:758` (undo dump from the live DB), `:778-860` (files and volumes from the snapshot), `:804` (the snapshot's definition is written when its version differs), `:898` (the rollback loads the newer dump over the older volume; nothing writes the live definition back). Not a reorder fix: it needs R-638 option B (a rebuilding loader) or a pre-restore volume copy (disk cost; R-685's class). Which state a household gets after a failed off-site replay is the operator's call. Next: the 9202 measurement, then the design. | a design: R-638 option B (a loader that rebuilds instead of overlays) or a pre-restore volume copy | Measure it once on 9202 (a forced replay failure after an off-site restore over a migrated app); then a design for the operator | CC | | **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):** `_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC | | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). | operator: print the sheet | Operator: print + store the sheet; then close | operator | -| **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC | +| **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** **READY 2026-10-20 — measured and planned, nothing changed; the decision is the operator's (STATUS D21).** Re-measured on 9202 (Immich v3.2.4): off-site ~1.33 GB = model cache 824 MB (62 %), raw database folder 283 MB (21 %), Immich's own backups 115 MB (9 %), logical dump 54 MB, originals 49 MB, thumbnails 2 MB — 102 MB must survive. A: leave out all three re-fetchable parts (−92 %; needs a never-run restore from the dump alone on the vectorchord image). B: only the model cache (−62 %; needs one restore with an empty cache). Pick: B now, A's other parts after their proof (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`). | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC | | **R-231** | Backup & restore | P3 | **`/opt/backup/scripts/` on DooPlex is unversioned host state** — found 2026-08-06 while adding the auto-memory store to the backup set. No repository tracks the scripts that protect the recovery chain, so the edit made that day (`CLAUDE_MEMORY_DIR` in `backup-config.sh`, multi-path restic call in `backup-data.sh`) exists only on the box. This is the same class the part-2 session was closing, found inside the fix for it; the change is transcribed in `felhom.eu/workspace/README.md` so it is at least *recorded*. **Two related facts, both understating current safety:** the backup destination (`/mnt/5_hdd/backup`) is on the **same physical disk** as the workspace it protects, and the DooPlex backup set has **no off-site leg** (`sync-hetzner-backups.sh` is jarrs.eu and pulls *from* Hetzner *to* DooPlex). Bringing a root-owned production backup script under version control, and deciding what installs it, is its own scoped change. | **VERIFY** — **MEASURED 2026-10-13 — the due check is GREEN** (read by the sizing session, two channels): `systemctl show dooplex-backup.service` → `Result=success`, `ExecMainStatus=0`, ran 03:27:39–03:33:29 CEST; its journal ends „All backup components completed successfully!”, and the manifests leg (the header-token change) logged „Successfully backed up homelab-manifests” with the plain URL `https://gitea.dooplex.hu/admin/homelab-manifests` (no token in it). Left open, not closed: the row also carries the same-disk and no-off-site facts, which closing would drop — the owner decides. **VERIFY** — **2026-10-12 evening: the token left the clone URL** (operator yes in chat; homelab-manifests `1943a70`, installed by the role): `backup-manifests.sh` sends it as an `http.extraHeader` through `GIT_CONFIG_*` env; read-only proof: `git ls-remote` of the private repo with the header → HEAD, without → refused. The 2026-10-13 03:10 run is the test of both changes. **VERIFY** — **2026-10-12: in git and installed from it** (operator yes in chat, operator in the session): homelab-manifests `fdd324b`, `homelab-ansible/roles/dooplex_backup/` (11 scripts + the timer units, byte-identical to the live ones; no secret in any file; `install.sh` dry run / `--apply`). The install changed only the units' mode (744 → 644). Baseline: last night's run `Result=success`, exit 0 (`audits/answers-2026-10-12/s20-backup-scripts.txt`). **Seen by the commit's security review, NOT changed (a DooPlex behaviour change needs the operator's word):** `backup-manifests.sh` puts the Gitea token into the clone URL for the mirror run (visible in the process list for those seconds) and strips it from the stored remote afterwards; no token VALUE is in any file, and the repo is private (anonymous API read = 404). A header-based fetch (`git -c http.extraHeader=…` from the token file) would remove it — one line in the decision list. **READY** — operator ruling 2026-10-12 (S20, `09` §3 decision 215): yes — the scripts into a repo + an install script; the operator present for the install. **READY** — owner Viktor | — | Owner: close the row (the install from git is proven) and file the same-disk / no-off-site facts as their own row if they still hold | CC | | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC | | **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC | @@ -170,7 +168,7 @@ stopping line that lies. | **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC | | **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** | — | — | CC | | **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) | -| **R-1019** | Backup & restore | P3 | **The live download-client templates put downloads into two backup tiers nobody chose.** MEASURED 2026-10-14 with the controller's own classifier over the live templates (`audits/media-all-drives-2026-10-14/``A-drives/A6-3-capture-sets.txt`): Transmission's `${USERDATA_PATH}/media/transmission/watch` bind is not listed in its `backup:` block, so it is classed MANDATORY and goes into the OFF-SITE copy (a writable unlisted bind defaults to mandatory); and qBittorrent's `media/torrents` and Transmission's `media/transmission` are `optional`, so on a box with a second drive every download is mirrored there nightly (Tier 2 = mandatory + optional) — it can fill the backup drive with films. | **NARROWED** — filed 2026-10-14; the `watch` half FIXED 2026-10-14 (declared optional, app-catalog `bc54c01`); open: the `optional` download folders still get a nightly Tier-2 mirror on a box with a second drive | — | Declare the media binds `excluded` (the watch folder too) in the live-fix task; `15` §7 states the rule for the per-drive bind | CC | +| **R-1019** | Backup & restore | P3 | **The live download-client templates put downloads into two backup tiers nobody chose.** MEASURED 2026-10-14 with the controller's own classifier over the live templates (`audits/media-all-drives-2026-10-14/``A-drives/A6-3-capture-sets.txt`): Transmission's `${USERDATA_PATH}/media/transmission/watch` bind is not listed in its `backup:` block, so it is classed MANDATORY and goes into the OFF-SITE copy (a writable unlisted bind defaults to mandatory); and qBittorrent's `media/torrents` and Transmission's `media/transmission` are `optional`, so on a box with a second drive every download is mirrored there nightly (Tier 2 = mandatory + optional) — it can fill the backup drive with films. | **NARROWED** — filed 2026-10-14; the `watch` half FIXED 2026-10-14 (declared optional, app-catalog `bc54c01`); open: the `optional` download folders still get a nightly Tier-2 mirror on a box with a second drive **NARROWED 2026-10-20 — catalog `c50adfc`:** the download folders of qBittorrent, Transmission (+ `watch`), NZBGet, SABnzbd and MeTube are `excluded` (all five downloaders; the brief named two, its gate rule covers all); gate `download-binds` (container target `/downloads` or `/watch` must be `excluded`; 11 decoys; it refused the live catalog 6 times before the fix). The controller's own classifier: before, each copied its folder to the second-drive tier; after, 0 paths on every tier (`audits/night2-2026-10-20/P3/`). Left: read demo-felhom's (qBittorrent) next second-drive run (2026-10-21 03:30 CEST) — dated check. | — | Declare the media binds `excluded` (the watch folder too) in the live-fix task; `15` §7 states the rule for the per-drive bind | CC | | **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC | | **R-164** | Backup & restore | P4 | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC | | **R-213** | Backup & restore | P4 | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **DEFERRED** — operator ruling 2026-10-12 (S18, `09` §3 decision 213): designed after the first customers. Stays P4; re-dated: look again when the first paying household has run 30 days. **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | After the first customers | Operator + CC | @@ -182,7 +180,9 @@ stopping line that lies. | **R-878** | Backup & restore | P4 | **A catch-up (R-871) runs the database-dump leg in the DAY, and that leg stops an app with a volume for its copy — the household may notice the stop, and a large volume makes it longer.** MEASURED 2026-10-05 on demo-felhom: the catch-up at 08:25:02 stopped opengist, copied 182.5 KB, started it again — about 1 s, then a few seconds of `health: starting`; the night does exactly the same, unseen. Nothing measured for a large volume. Fix direction (if it matters): skip the volume copy of a running app in a DAYTIME catch-up and leave it to the next night, or warn. `audits/catchup-2026-10-05/partA/live-demo-felhom.txt` | **READY — owner: CC** **2026-10-05 (burn-down night): NEEDS A LIVE MEASUREMENT** — a timed catch-up on 9202 for an app with a multi-GB volume, then the skip-or-warn choice. | — | measure a large volume first | CC | | **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.** **2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC | | **R-1087** | Backup & restore | P4 | **Whole-box copies left on the OLD local target are never pruned after the target moves.** MEASURED 2026-10-15 (storage-identity build B space check): the N100's host root (`local`) holds three copies from 2026-07-28 (5.5–5.7 GB each, ≈ 16.8 GB); its agent's `local_backup_target` is now `felhom-backup` (the USB disk, retention 3), and retention prunes only the current target. On a small-root box such leftovers would starve the local copies (R-548). Evidence: `audits/storage-identity-buildB-2026-10-15/space/felhom-pve-9201.txt`. | OPEN | — | Decide whether the agent prunes its own `vzdump-lxc--*` files on a former target (after a newer copy exists on the new one), or the target move asks; until then, the N100's three July files can be removed by hand with the operator's word. | CC | -| **R-1169** | Backup & restore | P3 | **After a failed off-site restore that is held (`restore_mixed`), the restore page says the database is back as it was and the app's files and other data come from the backup — measured false on both halves.** 2026-10-19 on 9202 (R-893's measurement): a table only the backup had stayed in the database (37 tables against 36), and Rallly had no files or other volumes from the backup. The sentence (`err.backup.db_restore_failed_held_mixed`) should say only what is certain: the restore stopped half-way, the app is held stopped, nothing is lost yet, support will help. A text change in the controller (HU + EN). | OPEN — filed 2026-10-19 | `audits/night-2026-10-19/P3/README.md` | Next controller release: the sentence rewritten in both bundles, a fixture for the held-mixed page | CC | +| **R-1171** | Backup & restore | P3 | **demo-hp's LOCAL whole-guest copy is never restore-tested: the box keeps one copy (keep-daily 1) and the restore test wants a copy 24 h old, so the only copy turns 24 h old exactly when the next night replaces it.** Read 2026-10-20 on demo-hp (agent journal): every 6-hourly evaluation since 2026-10-18 says `local: no settled archive yet — nothing to prove`; the local storage holds one 9201 archive; the last local proof was 2026-10-14 03:42 (`settle=24h`, `keep_last=1`, `prune=keep-daily=1`). The hub's `restore_test_stale` (168 h) will alarm from ~2026-10-21 03:42. A real household with retention 1 would be the same. | OPEN — filed 2026-10-20 | demo-hp `journalctl -u felhom-agent` („restore-test evaluated"), the hub's host report `restore_tests` | Agent: a tier that keeps ONE copy proves its newest archive once it is older than the settle time measured from the BACKUP's end, not 24 h — or a settle below the cadence (e.g. 6 h) for a 1-copy tier; a test; then read the next evaluation | CC | +| **R-1172** | Backup & restore | P4 | **After option A (R-893): a failed off-site restore that was PUT BACK is logged at ERROR, and an empty `backups/restore-undo/` folder stays on disk.** Measured on 9202 2026-10-20 (rc.1): the web handler logs `[ERROR] [web] off-box reconstitute rallly (async): … Nem változott semmi…` — good news at ERROR; the put-back removes the per-app folder but not the empty `restore-undo/` parent. Small; 0.336.0 was already pushed (a version is pushed once). | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P2/live/11-controller-log-window.txt`, `12-after-failed-restore.txt` | Next controller release: the put-back outcome logged at WARN (its own sentence is the household's), the empty parent removed; a test | CC | +| **R-1173** | Backup & restore | P3 | **R-921's consequence (no app stop for an off-site tier the agent will refuse) has not been seen live.** Agent 0.161.0 serves `host_busy` (proven live on demo-hp 2026-10-20 during a press: busy `backup:local`), and controller ≥ 0.335.0 reads it — but the case only arises on the night schedule (a manual press backs up the local tier only), so the no-stop branch is proven by its red-proved test alone. | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P1/1b-host-busy-poll.txt` | Read demo-hp's controller log after the night its off-site (pbs) tier is due (~2026-10-22): a `host is busy … no app is stopped (R-921)` line, or the off-site tier starting with nothing in flight | CC | ## Storage & devices — 9 rows (P3 3, P4 6) @@ -223,7 +223,7 @@ stopping line that lies. | **R-1118** | Security & access | P3 | **CC's secrets moved off the plain `~/.config/credentials` into an encrypted store with a broker and guards; what is left is retiring the old file.** BUILT 2026-10-17 (store B, operator ruling 11:28): private repo `admin/cc-secrets` (`ARCHITECTURE.md`, the design verbatim in `docs/`), `sec` (no get), PreToolUse refusals + PostToolUse masking for Bash and Read in `~/.claude/settings.json` (every CC session on DooPlex), nightly transcript sweep with mail. 55 tests, 18 red-proofs, 36-row child-session proof in both launch modes, fresh-session proof (`audits/cc-creds-build-2026-10-17/`). 25 names in the store (23 imported, `RESEND_API` from CC's environment, a self-test value). `12-agent-tooling.md` §5 points at it. | NARROWED — built; the old file's retirement is R-1134 | the week with no reader (R-1134) | `sec doctor` stays all OK; R-1134 | CC | | **R-1121** | Security & access | P3 | **`~/rotated-secrets-2026-10-09.txt` still exists on DooPlex** — 632 bytes, modified 2026-10-09 13:08 +0200, mode 600 (SPIKE Q6: `test -e` + `stat` only, not opened). Left by the R-925 rotation work. If any value in it is still valid, it is one more plaintext secrets file in the home folder, readable by every CC session. | WAITING-ON-OPERATOR | what the file holds — only the operator knows | Operator: confirm every value in it is revoked, then `shred -u` it; or move the live ones into the R-1118 store | operator | -## Box system & updates — 12 rows (P2 1, P3 7, P4 4) +## Box system & updates — 11 rows (P2 1, P3 6, P4 4) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -237,17 +237,14 @@ stopping line that lies. | **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement | — | — | CC | | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC | | **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC | -| **R-1010** | Box system & updates | P3 | **The agent's stale-lock janitor cannot remove a half-removed `vzdump` snapshot: Proxmox's delete fails on the first missing volume („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump") and the janitor retries the same call every 10 min forever, leaving the other volume's snapshot and the snapshot entry.** Seen 2026-10-14 after a backup ran out of space on a full host root (LVM could not write its metadata; `audits/power-cut-fixes-2026-10-14/README.md` C4). Fixed by hand with `pct delsnapshot 9201 vzdump --force`. | **READY — fixed in agent `56aabc7`, committed, NOT released** (the morning question): on „Failed to find logical volume …_vzdump" the janitor retries once with the API's `force=1` (`pvesh usage … -command delete` lists `--force`; same token privilege, no sudoers line); any other error unchanged; never a second force. Tests red-proved. Not exercised live (it needs a half-removed snapshot on `pve/data`). Close after the release. **READY** — filed 2026-10-14 | — | CC: on that error, retry once with `--force` (a sudoers line or the API's `force=1`), with a test that replays the error text; next agent release | CC | | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC | -## Monitoring & notifications — 12 rows (P2 1, P3 8, P4 3) +## Monitoring & notifications — 10 rows (P2 1, P3 6, P4 3) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1070** | Monitoring & notifications | P3 | **A box with no Cloudflare tunnel gets an ERROR after its host OS update: „NOT healthy … (the tunnel is unknown)”.** MEASURED 2026-10-15 16:30 on Tester4-68d1d8 (throwaway customer, placeholder domain `tester4.test`, so no tunnel token): hub event `os_update_health_failed` (error) „OS update (host) on Tester4-68d1d8: NOT healthy after 106 package(s) were installed (the tunnel is unknown). Nothing was undone automatically; put a host package back by hand…”. The box was healthy (dashboard 200, apps installing). A tunnel that never existed is read as a failed health check. Would hit every test customer and any box whose tunnel is not set yet. Evidence: `audits/omv-vm-2026-10-15/C-hub-events-Tester4.txt`. | **READY — fixed in agent `56aabc7`, committed, NOT released:** the cause measured — Docker 29 prints „error: no such object" in lower case, so the probe's case-sensitive match read a box with no cloudflared container as `unknown`, and the host rule failed it. New state `absent`; the host rule passes it only when the reading from before the step also had no cloudflared container; `not_running`/`unknown` still fail. The hub never alarms on `absent` and shows „none set up" (hub 0.162.0). Tests red-proved. Close after the agent release, on Tester4's next host OS update. Hub side of a VANISHED tunnel: R-1170. OPEN | — | Find the health verdict's tunnel input (hub os-update health); treat „no tunnel configured” as not-applicable, not as unhealthy; red-proof with a no-tunnel fixture. | CC | | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC | -| **R-1001** | Monitoring & notifications | P3 | **Every hub restart re-mails the operator about a box that is already known to be down.** MEASURED 2026-10-13 18:33 UTC (hub v0.149.0 rollout, R-998 session): Tester 2, silent 216 h, got a fresh `host_down` and `restore_test_stale` mail. `monitor/host_staleness.go` leaves a down host unseeded on start so the first Check emits (F2), and its comment says „the dispatcher's 1h cooldown dedups the re-emit" — but host liveness events bypass that cooldown (`dispatcher.go` `nodeLivenessEvents`, 5-min dedupe only); `restore_test_stale` keeps its edge state in memory. The comment states an invariant the code does not provide. | **READY — built in hub 0.162.0 (image `sha256:69285e64…`, from `d92a712e`), NOT deployed** (the night brief: no hub deploy at night; STATUS's morning question). Tests red-proved: `TestHostStaleness_R1001_RestartDoesNotReMail`, `…NewSilenceStillMails`, `TestRestoreTestStale_R1001_RestartDoesNotReMail` (each restart a new checker over the same DB). Close after the deploy, on the first restart read-back. **READY** — filed 2026-10-13 | — | Seed a host silently when a `host_down`/`host_stale` event for it is newer than its last report; the same for the restore-test edge; a test that restarts the checker | CC | | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | | **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** **REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc//fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC | | **R-388** | Monitoring & notifications | P3 | **PRODUCT DECISION (not a defect): the customer notification model is the wrong shape, and the settings page grows by one toggle per detector.** The operator's framing, recorded verbatim 2026-08-23: *"A customer should be notified only about things they can act on or are responsible for — the drive they unplugged, the storage they filled. **A failed backup is our incident, not theirs.** The intended shape is that we detect it, we tell them we noticed and are dealing with it, and they are not handed an error they cannot solve. The subscription should feel like being looked after, not like being on call."* Today's page is the opposite shape — one switch per detector, and it **grew from 12 to 15 in a single session** (one new alarm plus two compound toggles split into four). That growth is the argument, not an aside: a page that grows per detector keeps asking a household to make engineering decisions. | **READY** — **2026-10-12 evening: slice 1 delivered** (operator Q1/Q2 „as picked", `09` §3 decisions 234–235): the page shows three group switches; the per-detector switches and the cooldown sit under „Részletek"; an „egyedi" group is saved back unchanged; the default list fills groups 1 and 3 (households with no saved list), group 2 stays „egyedi" (`app_start_failed` off, R-329). Tests red-proved. controller 0.309.0, delivered 2026-10-12 18:08 to demo-hp, demo-felhom and Tester 1 (`controller-0.309.0-delivery.txt`). **READY** — **2026-10-12: the design is written** (`08` §8.1): three group switches („A gépem és a meghajtóim", „Az alkalmazásaim", „Amit mi intézünk") mapped onto today's 18 detectors; option A picked; slice 1 changes no wire. Two questions for the operator (group 3 default; the cooldown field). **READY** — operator ruling 2026-10-12 (S5, `09` §3 decision 200): design now (one page: options, a pick, the first slice), build in slices later. **OPEN — DIRECTION, operator's call** | a decision on scope; nothing here is a bug | Slice 2 (hub): group 3's mails say „we noticed X and are handling it" instead of the raw error | CC (+ operator: two answers) | @@ -259,7 +256,7 @@ stopping line that lies. | **R-1170** | Monitoring & notifications | P3 | **A box whose tunnel container disappears raises no alarm, even when its tunnel is configured.** The hub alarms on `not_running` only (`checkTunnel`). A guest with no cloudflared container read `unknown` since Docker 29 (lower-case „no such object", R-1070) and reads `absent` with the next agent release — neither alarms. On a box whose customer has a tunnel token, a missing container means the box is unreachable from outside. Found 2026-10-19 while fixing R-1070. | OPEN — filed 2026-10-19 | `hub/internal/api/handler.go` `checkTunnel` | Hub: treat `absent` as down when the customer's config carries a tunnel token (two reports in a row, as today); a test | CC | -## Hub & operator — 12 rows (P3 5, P4 7) +## Hub & operator — 11 rows (P3 4, P4 7) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -272,7 +269,6 @@ stopping line that lies. | **R-814** | Hub & operator | P4 | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | **VERIFY** (2026-10-03 triage: a July watch row with no id; given R-814. WAITING-ON-OPERATOR — no record found that the box was deleted.) — WAITING-ON-OPERATOR | operator console | Delete the box | operator | | **R-844** | Hub & operator | P4 | **The household's OS-update line exists only on the hub's customer timeline.** 2026-10-04: the box itself has no event surface for agent results (the controller UI shows no timeline), so `os_update_applied` is a hub customer event (info: recorded, never mailed). Its stored text is the hub's English sentence; the hu/en bundle text (`mail.event.os_update_applied`) is used only if it is ever mailed. Fix direction: a controller-side line (the controller already polls the agent's local API) when the box gets a household timeline. `audits/os-guest-lane-2026-10-04/partG/hub-customer-timeline-demo-hp.txt` | **READY — owner: CC** **2026-10-05 (burn-down night): NEEDS A DESIGN** — a household timeline on the box does not exist yet. | — | — | CC | | **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC | -| **R-1011** | Hub & operator | P3 | **The Day-0 „Agent" list offers step-bundle packages (`0.156.1-step1`, `0.152.0-step1` …) as agent versions; one has no agent binary, so the hub shows the bundle JSON's sha (`FileSHA256` falls back to the first file) and a vouch would point fresh installs at a package with no binary.** Seen 2026-10-14 (`/configuration`: two „0.156.1" entries, `ac83ea41…` is the step bundle). | **READY — built in hub 0.162.0, NOT deployed:** the sha lookup asks for exactly `felhom-agent` / `golden.tar.zst` (no fallback); step bundles leave the list; a vouch of one is refused (`artifact_file_missing`). Registry read 2026-10-19: 6 step-bundle versions, every release holds `felhom-agent`. Tests red-proved. Close after the deploy, on the Configuration page read. **READY** — filed 2026-10-14 | — | CC: list only versions whose package holds the `felhom-agent` file (refuse a missing preferred file instead of falling back), with a test; next hub release | CC | | **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator | | **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator | @@ -300,7 +296,7 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1083** | Process & tooling | P3 | **The scratch box 9202 cannot use any feature that needs the agent: its `bootstrap.json` has no `local_api` block, so its controller never gets an agent client.** MEASURED 2026-10-15 (storage-identity build A): 9202 put on controller 0.317.0 by hand to show that a NAS share no longer hides the one-copy notice (R-1075); its `bootstrap.json` (read on the box) holds no `local_api`, so the controller has no agent to ask and keeps the old `st_dev` rule — the check could not tell old from new — 9202 was put back on 0.314.0. Every agent-coupled feature (drives, same-disk, format, network storage) is untestable on the scratch box today, and the NAS leg of build A is proven by a red-proofed unit test only. Evidence: `audits/storage-identity-buildA-2026-10-15/9202-upgrade.txt`. | **OPEN — the exact gap, read 2026-10-19 (no agent change tonight):** the agent mints a per-guest local-API token in ONE place, `--selftest=provision`'s back half (`cmd/felhom-agent/main.go` `bh.Provision` after `engine.RunBringUp`), which only runs for a guest it brings up from a golden archive in the same call. There is no product way to mint for an EXISTING guest (9202) — no subcommand, no local-API route; the token store `/var/lib/felhom-agent/local-tokens.log` is written only by that path. Options: an agent subcommand `--mint-guest-token ` (owner-only, writes bootstrap's `local_api`), or re-provision 9202 through `selftest=provision` (destroys and rebuilds the scratch guest — not done; rule 6). Until then 9202 is controller-only. OPEN | — | Give 9202 a local-API token the normal way (the agent mints it for its guest), or record in `runbooks/target-selection.md` that 9202 is controller-only; then show the NAS notice live. | CC | +| **R-1083** | Process & tooling | P3 | **The scratch box 9202 cannot use any feature that needs the agent: its `bootstrap.json` has no `local_api` block, so its controller never gets an agent client.** MEASURED 2026-10-15 (storage-identity build A): 9202 put on controller 0.317.0 by hand to show that a NAS share no longer hides the one-copy notice (R-1075); its `bootstrap.json` (read on the box) holds no `local_api`, so the controller has no agent to ask and keeps the old `st_dev` rule — the check could not tell old from new — 9202 was put back on 0.314.0. Every agent-coupled feature (drives, same-disk, format, network storage) is untestable on the scratch box today, and the NAS leg of build A is proven by a red-proofed unit test only. Evidence: `audits/storage-identity-buildA-2026-10-15/9202-upgrade.txt`. | **OPEN — the exact gap, read 2026-10-19 (no agent change tonight):** the agent mints a per-guest local-API token in ONE place, `--selftest=provision`'s back half (`cmd/felhom-agent/main.go` `bh.Provision` after `engine.RunBringUp`), which only runs for a guest it brings up from a golden archive in the same call. There is no product way to mint for an EXISTING guest (9202) — no subcommand, no local-API route; the token store `/var/lib/felhom-agent/local-tokens.log` is written only by that path. Options: an agent subcommand `--mint-guest-token ` (owner-only, writes bootstrap's `local_api`), or re-provision 9202 through `selftest=provision` (destroys and rebuilds the scratch guest — not done; rule 6). Until then 9202 is controller-only. OPEN **2026-10-20 — left out of agent 0.161.0 on purpose:** the code would be small (a root-only subcommand that mints the token and adds `local_api` to an existing guest's `bootstrap.json`), but an agent key makes 9202's controller book the agent's whole-guest backups, one tier of which writes to ep0 (the demo-hp namespace) — and 9202 is outside the `felhom` pool. Needs, before it ships: a way to give a scratch guest a key with NO whole-guest tiers (or the operator's word that 9202 may write to ep0), and 9202 in the pool. | — | Give 9202 a local-API token the normal way (the agent mints it for its guest), or record in `runbooks/target-selection.md` that 9202 is controller-only; then show the NAS notice live. | CC | | **R-1081** | Process & tooling | P4 | **Two published controller tags hold content that is not what their number says: `0.315.0` and `0.316.3`.** Both were caused by CC on 2026-10-15, and both by the same shape — a guard that was not on the path the work takes. **`0.316.3`:** the build directory's `build.sh` is a symlink into the SHARED clone's working copy, which predated the `FELHOM_REPO_DIR` override, so the override was silently ignored and the build copied that tree — which held 31 uncommitted files of another session's physical-disk storage work. The banner printed `Commit: 0225255` with no `Repo:` line. **Nobody deployed it; that was luck, not a control.** Fixed at the root in controller v0.316.4: `build.sh` now refuses `--push` from a tree that is dirty, not at `origin/main`, or not a git checkout, with four test cases red-proved. **Neither tag is re-pushed or deleted** — a version number is pushed once, and that applies to numbers CC spoiled. **No box runs either:** demo-hp 9201 is on 0.317.0, the floor is 0.314.0, and the button offers the highest tag. **`0.315.0`:** Caused by CC on 2026-10-15 while proving R-1080's own fix: the guard was added to `controller/build.sh` in the repo, but the documented release command runs `$FELHOM_ROOT/build/felhom-controller/build.sh`, which was a **separate copy taken 2026-07-18** and had no guard. Running the refusal against a real existing tag therefore rebuilt and re-pushed 0.315.0 — a fourth time — with v0.316.0's code under it (digest `8887343011d6` → `b809201841f7`). **Fixed at the root in the same session:** the build directory's script is now a symlink to the repo's, the old copy kept beside it as `build.sh.stale-copy-2026-07-18`, and the refusal was then proven to fire from the documented path. **No box is affected:** nothing runs 0.315.0 (demo-hp 9201 is on 0.316.2), the floor is 0.314.0, and the household button offers the highest tag. What remains is two tags that must never be deployed and are no longer reproducible from the registry. | **OPEN** — filed 2026-10-15; nothing to do unless someone needs one of those two images, and then the answer is the current release | — | Leave them. If an audit ever needs the real 0.315.0 or 0.316.3, build it from the matching commit under a NEW number and say so | CC | | **R-992** | Process & tooling | P3 | **The drill-catalog route that `09` §6.5 documents cannot be followed: `admin/app-catalog-drill` is PRIVATE and no box has catalog credentials.** Measured 2026-10-12 while publishing the Usenet pair: the box's clone fails with `fatal: could not read Username for 'https://gitea.dooplex.hu'` (exit 128), the Gitea API reports the drill repo `private=true` and the live catalog `private=false`, and NEITHER the current controller.yaml on 9202 NOR any of its 30 saved copies has ever carried `git.username` or `git.token` (the controller supports both — `internal/config/config.go` GitConfig). The route worked for the campaigns of 2026-10-10 and the torrent pair earlier the same day, so the repository's visibility changed between then and now. **Worked around, not fixed:** that campaign staged its two templates in the box's own catalog cache (`audits/usenet-2026-10-12/tools/stage_in_cache.py`), which is where the sync copies a catalog change anyway, with the sync interval raised to 12h for the duration and rescan used in place of sync. The real fix is an operator decision: either a read-only catalog credential on the scratch box (a secret on a Tier-0 guest) or making the drill repository readable. Until then every new-app campaign hits this. | WAITING-ON-OPERATOR | the operator's choice between a box credential and a readable drill repo | Decide; then either set `git.username`/`git.token` on 9202 from the credentials store, or flip the drill repo, and correct `09` §6.5 either way | operator | | **R-1034** | Process & tooling | P3 | **Gate 17 checks the FAQ JSON-LD on two pages only, by a hard-coded list — a FAQPage anywhere else is unchecked.** `scripts/site_gates.py` gate 17 loops `for p in ("gyik.html", os.path.join("en", "faq.html"))`, so its „JSON-LD equals the visible questions and answers” guarantee stops at those two files. **Since 2026-10-14 two more pages carry a FAQPage block the gate never reads:** `/sajat-felho-tarhely` and `/en/own-cloud-storage`. Their JSON-LD was *derived* from the visible headings and paragraphs by a script and then verified verbatim against the rendered text, so it is correct today — but nothing stops the next copy edit moving the visible text and leaving the structured data behind, which is exactly the failure gate 17 exists to catch, and the one Google penalises. **Why it is not a one-line fix:** gate 17's extractor is written for the FAQ pages' accordion markup (`