diff --git a/CONTEXT.md b/CONTEXT.md index 87ad4776..8b47d2a6 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,18 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-19 (night) — controller 0.335.0 is the floor (MinAgent 0.158.1, 18:39 UTC); hub 0.162.0 BUILT, not deployed; +> agent `56aabc7` and installer 1.35.0 committed, not released/published.** R-921 (operator answer „add one field"): the +> agent serves its host-wide heavy-operation gate as `host_busy` on `GET /backup/status`; the controller reads it before +> any stop (held → contention, no stop). R-1070's cause: Docker 29 prints „error: no such object" in lower case, so a +> guest without cloudflared read `unknown` and failed the host OS-update rule; new state `absent` (passes when absent +> before the step too); `11` §8.2 had claimed `unknown` does not fail — corrected. R-1010: the janitor's forced snapshot +> delete is the API's `force=1`. Hub 0.162.0: R-1001 (restart seeding from the DB), R-1011 (exact-file sha). R-953 is +> gone: ArgoCD tracks none of the three Secrets now. **R-986: `scripts/security_scan_gate.py`, shared, in every repo's +> fast set and CI, WARN-FIRST**; govulncheck pin v1.1.4 → v1.8.0 (v1.1.4 cannot read Go 1.27); CI has no Go, so +> govulncheck runs only in the hook; a depth-1 CI clone scans the whole tree (the root commit), all accepted. R-893 +> measured on 9202 (`audits/night-2026-10-19/P3/`) → D20. 9202 is on DHCP (.121 now). + > **2026-10-19 (late afternoon) — controller 0.334.0 is the floor (MinAgent 0.158.1, 16:31 local); hub 0.161.0 DEPLOYED; > tenantsync v1.3.0 on ep0 (operator yes).** R-1159 (D19 „box key first, then lock"): the passphrase is used AFTER enroll > (installer `/config` for its git login, the agent bakes it into `bootstrap.json`, the guest's first-boot pull, and — diff --git a/REPORT-night-2026-10-19.md b/REPORT-night-2026-10-19.md new file mode 100644 index 00000000..4ad1c05d --- /dev/null +++ b/REPORT-night-2026-10-19.md @@ -0,0 +1,146 @@ +# REPORT — evening/night brief 2026-10-19 (one stop less per night, no false tunnel error, R-893 measured, hub and agent fixes ready, a scanner on every push, ep0's first weekly clean-up) + +## In plain words + +- **Morning list:** deploy hub 0.162.0? · release agent 0.161.0? · publish installer 1.35.0? · decide D20 (R-893). +- **One app stop less per night** (R-921): the controller (0.335.0, the floor) now asks the agent „are you busy?" before + it stops the apps. The agent's answer is written but not released, so it starts working with the agent release. +- **No false error for a box without a tunnel** (R-1070): the real cause was Docker 29 writing its „not found" message + in small letters. Fixed in the agent (not released). +- **A failed off-site restore, measured** (R-893): the box puts the newer rows over the backup's older volume; a table + only the backup had stays. The app is held stopped. Two options written; pick A (save the app's volumes first). +- **Hub fixes built, not installed:** no repeat mail after a restart; the agent list offers only real agent packages. + The ArgoCD danger to three hub secrets is already gone (ArgoCD no longer tracks them). +- **Installer message written, not published:** a re-run on a live box says „delete the old host first", EN and HU. +- **A scanner on every push, in all four repos** (warn-first). First scan: no real secret, no called weakness. +- **The scratch box cannot get its agent key without an agent change** (R-1083): the exact gap is written down. +- **ep0's first weekly clean-up:** read after 05:30 — see Part 10. + +Decisions CC took: none beyond the brief. The operator answered one question (R-921, 19:2x): „add one field". + +## Part 1 — R-921 (`07` §6.4; `03` local API) — DONE (live once the agent ships) + +- No existing agent answer served the host-wide gate (read: `/backup/due`, `/backup/status`, the 409 shape). Operator: + add a read-only field to the existing route, no new route. +- Agent `56aabc7`: `host_busy` `{busy, what}` on `GET /backup/status`. Controller 0.335.0: `precheckSkips` → held → + no stop, contention for each due tier (15-minute skip, one „BLOCKED" alarm after 3 h); free / older agent / failed + read → the old order; the window's own tier in flight → attach as before. +- **The race** (free at the read, busy at the start): one short stop, resumed at once — today's behaviour, pinned. +- Tests, red-proved: agent `TestBackupStatus_R921_HostBusy` (assignment removed → `free gate: host_busy = `); + controller `TestR921_NoStopWhileTheHostIsBusy` (branch removed → 3 apps stopped, the start refused), + `…SkipsStillAlarmAfterTheCeiling`, `…OtherAnswersKeepTheOldOrder` (4 rows), `TestHostBusy_ReadsTheAgentsGate`. +- Live: demo-hp's agent answers no `host_busy` (read inside 9201) — the old order, as designed. A live busy state was + not made (it needs the new agent on the box). R-921 closed per the brief. + +## Part 2 — R-1070 (`11` §8.2) — DONE in the agent (not released) + +- Where: `felhom-agent/internal/osupdate/leg.go` `HostHealthVerdict` (the tunnel check) and + `internal/hub/cloudflared.go` `ClassifyTunnel`. Measured on demo-hp: `docker inspect` of a missing name prints + `error: no such object: …` (Docker 29); the match wanted `No such object` → `unknown` → the host rule failed. +- Fix: case-insensitive match → new state `absent`; the host rule passes `absent` only when the reading from before + the step (Docker answered) also had no cloudflared; `not_running` and `unknown` still fail. Hub 0.162.0 shows + `absent` as „none set up". `11` §8.2 had said `unknown` does not fail — corrected (the code always failed it). +- Tests, red-proved: `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows), three `TestClassifyTunnel` rows, + `TestSystemRows_R1070_AbsentTunnelIsNotAWarning`. + +## Part 3 — R-893: measured and designed, no restore code changed + +Helper on 9202 (fences carried in full). Evidence and the note: `documentation/audits/night-2026-10-19/P3/`. + +- Rallly 4.15.3 + postgres 18; a scratch SFTP off-site on 9202's own disk; one failing SQL line in the prepared copy + of the dump, so the replay failed AFTER the volume was replaced. +- **What it left:** the newest rows (the undo dump) over the backup's volume; a table only the backup had stayed + (37 tables against 36); definition unchanged (same version); app held (`restore_mixed`), a start → 409. +- The restore page's sentence is false on both halves for this app → R-1169. +- **Options:** A — copy the app's live volumes before the replay, put them back on failure, start the app (one copy's + disk for minutes; Rallly 69 MB in ~1 s; a fit check before the stop). B — empty the DB volume before the undo load + (no disk; a bad undo copy leaves an empty database; usually still a support call). **Pick: A** (the operator's D8 + direction). Not measured: the version-change branch (needs a drill-catalog commit). +- Teardown: 9202 back on 0.331.0 as found, every throwaway removed (checked again by CC); host: `/tmp` only; hub: nothing. + +## Part 4 — hub 0.162.0, built, NOT deployed (`05`) + +- R-1001: start-up seeding from the DB for `host_stale`/`host_down`; `restore_test_stale` edge from the DB. Tests + red-proved (each restart = a new checker over the same DB). +- R-1011: exact-file sha (`ErrNoFile`); step bundles leave the list; a vouch of one → `artifact_file_missing`. Registry + read: 6 step-bundle versions, every release holds `felhom-agent`. Tests red-proved. +- R-953: **no change needed** — read back 18:21 UTC: ArgoCD tracks none of the three Secrets (no tracking mark), so the + next deploy does nothing with them. `P4/r953-readback.txt`. +- Image `sha256:69285e640a9c5e4ce59b1fbb5cd94623fa47a71798e9d86fa0767ee6c69a6ac8` from a clean clone at `d92a712e`. + +## Part 5 — agent, committed, NOT released (`03`) + +R-1010 (`56aabc7`): on „Failed to find logical volume …_vzdump" one retry with the API's `force=1` (`pvesh usage` +lists `--force`; the same token privilege; no sudoers line, no root command); never a second force. Red-proved. Plus +R-921's field and R-1070. CHANGELOG „Unreleased". Not exercised live (it needs a half-removed snapshot on `pve/data`). + +## Part 6 — installer 1.35.0, written, NOT published + +R-1167: `die_live_box` in EN and HU at the preflight, host-enroll and the controller.yaml fetch. Harness: a static case +and two behaviour cases (the real functions against a local fake hub), red-proved on a mutated copy. **CI red twice** +(2106, 2108): the CI runner has no curl; the behaviour cases now SKIP out loud there (`5d0dfeac`). + +## Part 7 — the scanner (R-986), warn-first — DONE + +- Shared `scripts/security_scan_gate.py` in all four runners' fast sets (hook + CI); `.security-accepted` per repo; + 8 decoys in scratch repos, red-proved; CI fetches the pinned gitleaks (both sha256 checked — seen working in CI 2108). +- govulncheck pin v1.1.4 → **v1.8.0** (v1.1.4 cannot read the Go 1.27 code). CI has no Go: govulncheck NOT CHECKED + there, out loud; the hook runs it. +- **First scan of today's `main`** (`P7/triage.md`): govulncheck 0 called in all three modules. gitleaks history: + controller 12, agent 15, catalog 1 (all as on 2026-10-12), felhom.eu 50 (+7: the usenet bench 5, a boot journal 1, + a test 1). Today's tree: controller 4, felhom.eu 42, agent 15, catalog 0 — every one triaged: **0 real**; accepted = + throwaway demo/bench app values (the bench LXC 9401 stopped, the apps gone from 9202); false = test constants, public + keys, an SSH fingerprint, a tunnel ID, hashes. + +## Part 8 — R-1083: the exact gap (no agent change) + +The agent mints a per-guest token only inside `--selftest=provision`, for a guest it brings up from a golden in the same +call. No product way exists for an existing guest. Row stays OPEN with the gap and two options. + +## Part 9 — controller 0.335.0, the floor + +rc.1 on demo-hp 18:08 UTC (0 errors) → `0.335.0` `sha256:262831f9…` → per-box floors 18:22 → all three moved by +18:37 → box walk green (0 unhealthy, 0 refusals, catalog in sync; https-only 28 OK, 0 FAIL; „Fájlok ↗" one sign-in on +demo-hp and Tester 1) → **global floor 0.335.0 / MinAgent 0.158.1 at 18:39:26 UTC**, read back; per-box floors cleared. +i18n: no page changed, no fixture changed. Evidence `P9/`. + +## Part 10 — ep0's first weekly clean-up + +PENDING at the time of writing (runs 03:30 UTC). + +## Versions, commits + +- Controller 0.335.0 (`15ea002`, REPORT `c4b3b0a`, scanner `a03c25a`). Hub 0.162.0 built (`5f31da04`). Agent + `56aabc7`, `84ef581` (not released). Installer 1.35.0 (`d92a712e`, `5d0dfeac`, not published). felhom.eu scanner + `15f74288`, docs `f92fa5a8`. Catalog `87f5588`, `57ddb20`. +- **System poster:** no listed fact changed tonight — none of the three. +- **Instruction/doc corrections:** `11-os-updates.md` §8.2 (the `unknown` tunnel sentence was false); `nodes.md` and + `nightly-rotation.md` (9202 is DHCP, .121 now). + +## Rows + +Register **161 before, 161 after** (before Part 10). **Closed 3:** R-921, R-953, R-986. **Opened 3:** R-1168 (warn → +fail), R-1169 (the false restore sentence), R-1170 (a vanished tunnel raises no alarm). Re-stated: R-1001, R-1011, +R-1010, R-1070, R-1167 (READY, waiting on the release/deploy/publish), R-893 (READY, D20), R-1083 (the gap). + +## Observations + +1. The restore page's held-mixed sentence is false for the measured app. FILED: R-1169 +2. A tunnel container that disappears on a box with a tunnel configured raises no alarm. FILED: R-1170 +3. The scanner warns only. FILED: R-1168 +4. `11` §8.2 said an `unknown` tunnel does not fail the host rule; the code always failed it. NOT-A-FINDING: corrected in the session (the size rule); it is R-1070's cause. +5. govulncheck v1.1.4 (the 2026-10-12 pin) cannot read Go 1.27 code. NOT-A-FINDING: the pin moved to v1.8.0 tonight, recorded in `SCANNERS.md` §5. +6. The CI runner has no curl; two new harness cases failed CI 2106 and 2108. NOT-A-FINDING: fixed the same night (`5d0dfeac`), they skip out loud there. +7. R-953's tracking marks went after 2026-10-12 with no record of who removed them. NOT-A-FINDING: the danger the row named is gone, nothing to fix. +8. 9202's DHCP address moved (.114 → .121) and the walk named the old one. NOT-A-FINDING: fixed (`box_walk.py` reads it; two docs corrected). +9. The brief said the installer already prints HU and EN; it printed English only. NOT-A-FINDING: the HU lines are new, named in the scripts CHANGELOG. +10. The agent's backup cap (R-973) can also refuse a POST after the stop. NOT-A-FINDING: it answers 409, so the apps resume at once (the BUSY path), and it fires only after a flood of presses that each stopped the apps anyway. +11. The manual off-site run dumped and restarted every app, not only the one switched on. NOT-A-FINDING: by design, the coherent app-data pair (`07`). +12. On CI's depth-1 clone, gitleaks scans the whole tree (the root commit), ~100 s for felhom.eu. NOT-A-FINDING: a stronger check than the diff, all current hits accepted; recorded in `CONTEXT.md`. + +## Teardown + +- Machine: 9202 back as found (0.331.0, no throwaway left — checked); 9201 on the release 0.335.0; probe scripts removed. +- Host: `/tmp` copies only, removed. Nothing on `pve/data`. +- Hub: per-box floors cleared; nothing else created. +- ep0: read only. diff --git a/STATUS.md b/STATUS.md index f4873c21..40e01ef3 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for the first real tester (Tester-2): yes. Tester 2 (a laptop) is off; nothing was sent to it.** -**Updated 2026-10-19 (late afternoon).** The household passphrase no longer gives a live box's keys (your D19): each box now fetches its config with its own key (controller 0.334.0 is the floor since 16:31 local; hub 0.161.0 is live, your yes). A reinstall over a live box needs you to delete the old host first; the installer does not word that refusal yet (row filed). Every new household on ep0 gets its weekly clean-up rule automatically, and the hub alarms on one without; tester-1 and Tester-2 got theirs (your yes; tester-1's 4 October copy goes tonight). demo-hp kept its own address for a 30-minute watch after your router change. — `REPORT-day3-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (afternoon).** ep0 now keeps one whole-box copy per week for 2 weeks (your yes; the change removed nothing today, the first run is tomorrow 03:30 UTC). Controller 0.333.0 is the floor for every box (12:16 local): signing out on the dashboard now also ends the file manager's own 2-hour pass, and the recovery-code page says the code becomes live after the two words. Hub 0.160.0 is live (your yes): a reinstall within 45 minutes also mails you; a refused backup that repeats is counted, and the alarm repeats at most once a day. The household passphrase was measured: alone it reaches the box's keys and through them every off-site key action — **decide D19**. demo-hp loses its address to another device every few minutes — **fix it in the router** (do-list). — `REPORT-day2-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (day).** One sign-in for the file manager is live on every box (your D16 = yes; controller 0.332.0 is the floor since 05:24 local): „Fájlok ↗" opens the app's folder with no second login; signing out on the dashboard asks again; no app on a box can pretend to be the household (tested from inside an app). Agent 0.160.0 runs on demo-hp, Tester 1 and demo-felhom (your yes): a new recovery code replaces the old one only after the household types two words back; a broken-into box cannot flood the whole-box backups, and the local copies are one per day for 3 days (your D17); an empty partition no longer asks „data will be erased". **Vouch agent 0.160.0** (morning list). Hub 0.159.0 is live (your yes): a refused backup has its own alarm; a reinstall over an old host now mails you instead of staying silent — a first draft that re-issued the codes by itself was stopped by a security review. The old backup program still reads demo-hp's off-site copies after the first real clean-up with the new one. The document-preview error was a broken test file, not the box — `REPORT-day-2026-10-19.md`. **Earlier:** **Updated 2026-10-18 (evening).** A brand-new box was installed from nothing on demo-hp (your test customer Tester4): the emergency password was saved in the hub first, then set, and the web login took it; the box came up on the approved golden and moved to the floor by itself in under 3 minutes; its agent was the approved one. OnlyOffice now opens AND saves documents from FileBrowser and from Nextcloud (before tonight: FileBrowser could not open one, and Nextcloud lost every save while saying „saved") — controller 0.331.0 is the floor for every box (22:48 local). Immich's and Nextcloud's Redis no longer have a disk volume. Three agent fixes are written and tested, NOT released (your yes). Golden 0.331.0 is baked, NOT vouched — `REPORT-evening-2026-10-18.md`. **Earlier:** **Updated 2026-10-18 (day).** Hub 0.158.0 is live (your yes): its own 90-day access log works and writes no secret (tested with a fake token); you can vouch an agent without a new golden; the off-site snapshot check gives its first verdict on 20 October. Controller 0.330.0 is the floor (13:10 local): if the host refuses „retire anyway", the drive's apps start again. Agent 0.159.2 runs on the three test hosts (your yes; no pool check on a good backup) — **vouch it**. Installer 1.34.0 is published (your yes): the emergency password is saved in the hub first, then set. The calculator shows Immich's bigger figure (your yes). A fresh reviewer read last night's code: one medium finding, already closed by today's installer change — `REPORT-day-2026-10-18.md`. **Earlier:** **Updated 2026-10-18 (night).** Controller 0.329.0 is the floor for every box (21:36 UTC, MinAgent 0.158.1), after the three test boxes: each app reaches only its own data folder on the drive; homepage no longer controls Docker, so no app has a special right; when the host refuses to retire a drive, nothing changes and the household reads why; a virtual disk without a serial gets a plain sentence. Immich gets 3 GB (it was killed twice at 2 GB on a 4,000-photo import). Golden 0.329.0 is baked, NOT vouched. Hub 0.158.0 is built, NOT deployed — `REPORT-night-2026-10-18.md`. **Earlier:** **Updated 2026-10-17 (evening).** Controller 0.328.0 is the floor for every box (17:22 local): the box refuses a catalog change that asks for dangerous container rights (your D2 = A). Agent 0.159.1 (a gone, decommissioned drive is forgotten; an orphan thin snapshot is named in the backup failure; new Go) — runs on demo-hp, Tester 1 and demo-felhom; **your vouch in the hub is still open**. New emergency root passwords on demo-hp, Tester 1, demo-felhom (your D1 = A; read them in the hub). Hub 0.157.0 deployed on your yes — `REPORT-agent-allowlist-rotation-2026-10-17.md`. **Earlier:** **Updated 2026-10-17 (afternoon, later).** Controller 0.327.1 is the floor for every box since 12:06 UTC: FileBrowser is published only after its default password is checked (your D11 = A), and the install check uses real memory amounts with a clear message (your D10 = A) — `REPORT-filebrowser-memory-ci-2026-10-17.md`. **Earlier:** Controller 0.326.0 is the floor for every box since 09:55 UTC: FileBrowser's default password is checked at every FileBrowser start. Tester 1 is back on the live app list (test apps and drives removed); six memory limits raised — `REPORT-tester1-limits-filebrowser-2026-10-17.md`. **Earlier today:** controller 0.325.0 was the floor since 08:21 UTC (after the three test boxes): two more password doors count a try before checking it, family sign-ins are stored as fingerprints, a file-share rename always applies; five apps' Redis saves snapshots now, which a full disk cannot damage — `REPORT-burndown-3-2026-10-17.md`. **Earlier (2026-10-17 night):** controller 0.324.0 was the floor for every box since 02:46 UTC (all three test boxes made tonight's off-site backup on it first): a restore now brings the database password back with the data (your 1A), restic 0.19.1 (your 2A), dashboard hardening — `REPORT-night-2026-10-17.md`. **Earlier (2026-10-16 evening):** 0.323.0, Debian 13 — `REPORT-debian13-restic-2026-10-16.md`. **Earlier (midday):** Controller 0.319.0 (the app page in panels, the login in one panel) is the floor, MinAgent 0.158.1 — `REPORT-app-page-tuning-2026-10-16.md`. **Earlier (morning):** Storage identity build B shipped: agent 0.158.1 and golden 0.318.0 vouched by you, controller 0.318.0 is the floor (MinAgent 0.158.1), installer 1.33.0 published — new one-disk boxes get their own data area. Open items: see `documentation/backlog/OPEN-ITEMS.md`. Older notes, day by day: +**Updated 2026-10-19 (night).** An off-site backup no longer stops the household's apps for nothing when the agent is busy (controller 0.335.0 is the floor since 20:39 local; it works once the next agent is released — morning list). Three agent fixes and three hub fixes are written and tested, NOT released or deployed; the installer's „delete the old host first" message is written, NOT published (morning list). A security scanner now runs on every push in all four repos (warn-first: it lists, it never refuses); the first full scan found no real secret and no called weakness. The ArgoCD danger to three live hub secrets is gone (ArgoCD no longer tracks them). A failed off-site restore was measured on the scratch box — **decide D20**. — `REPORT-night-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (late afternoon).** The household passphrase no longer gives a live box's keys (your D19): each box now fetches its config with its own key (controller 0.334.0 is the floor since 16:31 local; hub 0.161.0 is live, your yes). A reinstall over a live box needs you to delete the old host first; the installer does not word that refusal yet (row filed). Every new household on ep0 gets its weekly clean-up rule automatically, and the hub alarms on one without; tester-1 and Tester-2 got theirs (your yes; tester-1's 4 October copy goes tonight). demo-hp kept its own address for a 30-minute watch after your router change. — `REPORT-day3-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (afternoon).** ep0 now keeps one whole-box copy per week for 2 weeks (your yes; the change removed nothing today, the first run is tomorrow 03:30 UTC). Controller 0.333.0 is the floor for every box (12:16 local): signing out on the dashboard now also ends the file manager's own 2-hour pass, and the recovery-code page says the code becomes live after the two words. Hub 0.160.0 is live (your yes): a reinstall within 45 minutes also mails you; a refused backup that repeats is counted, and the alarm repeats at most once a day. The household passphrase was measured: alone it reaches the box's keys and through them every off-site key action — **decide D19**. demo-hp loses its address to another device every few minutes — **fix it in the router** (do-list). — `REPORT-day2-2026-10-19.md`. **Earlier:** **Updated 2026-10-19 (day).** One sign-in for the file manager is live on every box (your D16 = yes; controller 0.332.0 is the floor since 05:24 local): „Fájlok ↗" opens the app's folder with no second login; signing out on the dashboard asks again; no app on a box can pretend to be the household (tested from inside an app). Agent 0.160.0 runs on demo-hp, Tester 1 and demo-felhom (your yes): a new recovery code replaces the old one only after the household types two words back; a broken-into box cannot flood the whole-box backups, and the local copies are one per day for 3 days (your D17); an empty partition no longer asks „data will be erased". **Vouch agent 0.160.0** (morning list). Hub 0.159.0 is live (your yes): a refused backup has its own alarm; a reinstall over an old host now mails you instead of staying silent — a first draft that re-issued the codes by itself was stopped by a security review. The old backup program still reads demo-hp's off-site copies after the first real clean-up with the new one. The document-preview error was a broken test file, not the box — `REPORT-day-2026-10-19.md`. **Earlier:** **Updated 2026-10-18 (evening).** A brand-new box was installed from nothing on demo-hp (your test customer Tester4): the emergency password was saved in the hub first, then set, and the web login took it; the box came up on the approved golden and moved to the floor by itself in under 3 minutes; its agent was the approved one. OnlyOffice now opens AND saves documents from FileBrowser and from Nextcloud (before tonight: FileBrowser could not open one, and Nextcloud lost every save while saying „saved") — controller 0.331.0 is the floor for every box (22:48 local). Immich's and Nextcloud's Redis no longer have a disk volume. Three agent fixes are written and tested, NOT released (your yes). Golden 0.331.0 is baked, NOT vouched — `REPORT-evening-2026-10-18.md`. **Earlier:** **Updated 2026-10-18 (day).** Hub 0.158.0 is live (your yes): its own 90-day access log works and writes no secret (tested with a fake token); you can vouch an agent without a new golden; the off-site snapshot check gives its first verdict on 20 October. Controller 0.330.0 is the floor (13:10 local): if the host refuses „retire anyway", the drive's apps start again. Agent 0.159.2 runs on the three test hosts (your yes; no pool check on a good backup) — **vouch it**. Installer 1.34.0 is published (your yes): the emergency password is saved in the hub first, then set. The calculator shows Immich's bigger figure (your yes). A fresh reviewer read last night's code: one medium finding, already closed by today's installer change — `REPORT-day-2026-10-18.md`. **Earlier:** **Updated 2026-10-18 (night).** Controller 0.329.0 is the floor for every box (21:36 UTC, MinAgent 0.158.1), after the three test boxes: each app reaches only its own data folder on the drive; homepage no longer controls Docker, so no app has a special right; when the host refuses to retire a drive, nothing changes and the household reads why; a virtual disk without a serial gets a plain sentence. Immich gets 3 GB (it was killed twice at 2 GB on a 4,000-photo import). Golden 0.329.0 is baked, NOT vouched. Hub 0.158.0 is built, NOT deployed — `REPORT-night-2026-10-18.md`. **Earlier:** **Updated 2026-10-17 (evening).** Controller 0.328.0 is the floor for every box (17:22 local): the box refuses a catalog change that asks for dangerous container rights (your D2 = A). Agent 0.159.1 (a gone, decommissioned drive is forgotten; an orphan thin snapshot is named in the backup failure; new Go) — runs on demo-hp, Tester 1 and demo-felhom; **your vouch in the hub is still open**. New emergency root passwords on demo-hp, Tester 1, demo-felhom (your D1 = A; read them in the hub). Hub 0.157.0 deployed on your yes — `REPORT-agent-allowlist-rotation-2026-10-17.md`. **Earlier:** **Updated 2026-10-17 (afternoon, later).** Controller 0.327.1 is the floor for every box since 12:06 UTC: FileBrowser is published only after its default password is checked (your D11 = A), and the install check uses real memory amounts with a clear message (your D10 = A) — `REPORT-filebrowser-memory-ci-2026-10-17.md`. **Earlier:** Controller 0.326.0 is the floor for every box since 09:55 UTC: FileBrowser's default password is checked at every FileBrowser start. Tester 1 is back on the live app list (test apps and drives removed); six memory limits raised — `REPORT-tester1-limits-filebrowser-2026-10-17.md`. **Earlier today:** controller 0.325.0 was the floor since 08:21 UTC (after the three test boxes): two more password doors count a try before checking it, family sign-ins are stored as fingerprints, a file-share rename always applies; five apps' Redis saves snapshots now, which a full disk cannot damage — `REPORT-burndown-3-2026-10-17.md`. **Earlier (2026-10-17 night):** controller 0.324.0 was the floor for every box since 02:46 UTC (all three test boxes made tonight's off-site backup on it first): a restore now brings the database password back with the data (your 1A), restic 0.19.1 (your 2A), dashboard hardening — `REPORT-night-2026-10-17.md`. **Earlier (2026-10-16 evening):** 0.323.0, Debian 13 — `REPORT-debian13-restic-2026-10-16.md`. **Earlier (midday):** Controller 0.319.0 (the app page in panels, the login in one panel) is the floor, MinAgent 0.158.1 — `REPORT-app-page-tuning-2026-10-16.md`. **Earlier (morning):** Storage identity build B shipped: agent 0.158.1 and golden 0.318.0 vouched by you, controller 0.318.0 is the floor (MinAgent 0.158.1), installer 1.33.0 published — new one-disk boxes get their own data area. Open items: see `documentation/backlog/OPEN-ITEMS.md`. Older notes, day by day: `documentation/backlog/STATUS-HISTORY.md`. Reports: `REPORT-night-2026-10-14.md`, `REPORT-immich-rollout-2026-10-14.md`, `REPORT-seo-felho-tarhely-2026-10-14.md`, `REPORT-immich-repair-2026-10-14.md`, `REPORT-power-cut-fixes-2026-10-14.md`, `REPORT-night-power-2026-10-13.md`, `REPORT-poweroff.md`. ## What works @@ -67,9 +67,14 @@ | D16 | R-1100 | **DECIDED 2026-10-19: yes** — built with both safeguards and live on every box (controller 0.332.0): a one-time pass from the dashboard, and a header name only the box knows. | done | — | — | — | | D17 | R-1149 | **DECIDED 2026-10-19: one per day, 3 days** — agent 0.160.0 on the three test hosts. | done | — | — | — | | D18 | R-1162 | **DECIDED 2026-10-19: A** — ep0 keeps one whole-box copy per ISO week, 2 weeks (`keep-weekly 2` on demo-hp and demo-felhom; nothing removed today; first run 2026-10-20 03:30 UTC). tester-1's missing job is R-1165 (your pick: not by hand today). | done | — | — | — | +| D20 | R-893 | After a failed off-site restore of one app, put it back exactly as it was and start it (A), or keep holding it stopped and only make the database part exact (B)? Measured tonight on the scratch box: the box puts the newer rows over the backup's older volume and a table only the backup had stays; the app is held. Note: `audits/night-2026-10-19/P3/design-R-893.md` | **A — copy the app's volumes first** (your D8 already chose this direction): one copy's disk for the restore's minutes; a fit check refuses before any stop | B: no disk, but a bad undo copy leaves an empty database, and the household still waits for support | a failed off-site restore keeps ending in a stopped app only support can bring back | — | | D19 | R-1159 | **DECIDED 2026-10-19: box key first, then lock** — a live box pulls its config with its own key (controller 0.334.0, the floor), and the passphrase gives no keys once the box has reported on it (hub 0.161.0). Proven on demo-hp: 409, no key, you got the alarm mail. A reinstall needs your delete of the old host first. | done | — | — | — | ## Waiting on you — do +- **Morning list, 2026-10-20 (from the night of 2026-10-19):** + 1. **Deploy hub 0.162.0?** (built `sha256:69285e64…`, from `d92a712e`): a hub restart no longer re-mails a box you already know is down (R-1001); the Day-0 agent list offers only real agent packages (R-1011); a box with no tunnel reads „none set up". **What the deploy does with the three Secrets** (`gitea-creds`, `umami-config`, `healthchecks-config`): **nothing** — ArgoCD no longer tracks them (read 18:21 UTC: none in the app's 25 resources, no tracking mark), so even a whole-app sync with prune leaves them; still sync only the hub, as always (R-953 closed). If not: Tester 2's restart mails continue on every hub restart. + 2. **Release agent 0.161.0?** (code `56aabc7`, CHANGELOG „Unreleased"): `host_busy` for the off-site stop (R-921 — without it the controller's new check has nothing to read), a box with no tunnel no longer fails its OS update (R-1070), the janitor clears a half-removed backup snapshot (R-1010). Then your vouch. If not: the extra app stop and the Tester4-type false error stay. + 3. **Publish installer 1.35.0?** (tag `installer-v1.35.0` + the `webpage.yaml` ref): a re-run on a live box says „the operator deletes the old host first" in EN and HU instead of „unexpected 409" (R-1167). If not: the next person to hit it reads only „409". - **Nézd meg a demo-hp-n a Calibre-Web „Adatok és mentés" fülét és a Fájlkezelőt** (`felhom.enkisfelhom.hu`, controller 0.321.0, minden dobozon kiadva). A mentés egy részben, egyszerű szavakkal; a Fájlkezelő nem kér frissítést és nem ajánl „Install app"-ot. - **Look at `roms/roms/` on the N100's drive in the file manager (R-1099).** RomM now scans `roms//`. A game uploaded through RomM before today sits one folder deeper: if there are platform folders in `roms/roms/`, move them up into `roms/` and press „Scan” in RomM. If not: those games show under a platform called „roms”. - **Nézd meg a Calibre-Web oldalát a demo-hp-n** (`felhom.enkisfelhom.hu`, controller 0.319.0, minden dobozon kiadva). A fejléc, a fülsor és minden rész most külön panel (az „A” stílus). A belépési adatok egy panelben vannak, egymás mellett. A Calibre-Web jelszaváról a doboz nem tud, mert az alkalmazás előbb települt, mint hogy a sablonban jelszó lett volna — ezt most egy mondat mondja ki. Ha nem tetszik, írd meg, mit változtassak. Ha nem nézed meg: semmi nem romlik el. diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index 517847f1..456e1e93 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -232,7 +232,11 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg - `POST /rollback` — roll *this* guest back to a named snapshot (post-deploy failure recovery). - `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive). - `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8). - - `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. + - `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy` + (R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is + held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the + state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it + before it stops any app. A read, never a reservation. - **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record (`present`, `last_boot_at`, `last_boot_unclean`, `tripped`) read from `/var/lib/felhom-crash-guard/state.json`; a missing or garbled file answers 200 `present:false`, an older agent 404 — both read as a normal boot. The controller waits @@ -399,7 +403,10 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba > space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped. > - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10 > minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed -> teardown tries the operator is told. +> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased +> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's +> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no +> root command); any other error is returned unchanged, and a failed forced delete is never forced again. > - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data > and metadata, critical at 90 %, one alarm per pool per 6 hours (`08` §6.2). > diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index 30843d2f..a75dd7df 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -90,7 +90,11 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh 2026-09-17, **45 m** since operator ruling A on R-549; OK under it, down at 2× = 90 m; today: controller-report recency → `node_stale`/`down`/`recovered`): - **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's - liveness signal; a silent agent = the box is gone (the critical alert). + liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known + silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down` + event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge + from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed + lookup lets the alarm through. - **Guest up/down comes from the host report's per-guest status** — authoritative, every poll, faster than waiting for a guest report to go stale. - **Guest-report recency = secondary** app-level signal. @@ -134,7 +138,11 @@ walk is green; the golden is baked weekly and vouched by the operator; a new box itself to the floor (R-1052: not yet proven end to end). Setting the global floor today needs the operator page with the operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the -vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. +vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list +offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the +`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle +(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused +(`artifact_file_missing`). **The controller floor and its agent requirement (hub v0.112.0, R-472).** The managed controller floor rides the report ACK. `store.ResolveManagedFloor` decides per box whether to serve it, from three diff --git a/documentation/architecture/07-backup-architecture.md b/documentation/architecture/07-backup-architecture.md index 33b30445..33c5f7f7 100644 --- a/documentation/architecture/07-backup-architecture.md +++ b/documentation/architecture/07-backup-architecture.md @@ -728,7 +728,13 @@ definition is written or a named volume replaced — a failed placement, volume the live definition back and HOLDS the app stopped (`restore_mixed`, operator-cleared); the household reads that it needs our help; the operator gets a `backup_run_failures` line. The hold is persisted before the stop. Placed files ALONE (no version change, no volume replaced) still restart after a successful rollback (`TestR379_ScenarioA`): that -mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893. +mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893. **`[FACT]` +2026-10-19 (measured on 9202, controller 0.335.0, `audits/night-2026-10-19/P3/`):** a forced replay failure after the +volume was replaced left the NEWEST rows (the undo dump) on top of the backup's volume — a table only the backup had +stayed (37 tables, the live app had 36); the definition did not move (same version); the app was held and a start +answered 409. The restore page's „the database is back as it was" is false for it. Design note with the two options and +the pick (A: a copy of the app's live volumes, put back on failure — one copy's disk for the restore's length, a fit +check before the stop): `audits/night-2026-10-19/P3/design-R-893.md`; the decision is on `STATUS.md`. **[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.** Recorded here rather than only in a closed register row, because a decision that survives only inside @@ -826,8 +832,12 @@ parts above, not a measured press). demo-hp: the off-site tier stopped every app for ~1 min and the agent then refused it (BUSY: the night OS step held the agent's host-wide lock). Before stopping anything the controller now asks the agent which tiers have a job in flight (`GET /backup/status`); another tier's job running or snapshotted → no stop, the tier stays due. A refusal after the -stop still resumes the apps at once (pinned by test). **Not covered:** the host-wide busy lock (OS step, restore test, -fstrim) — the agent serves it on no endpoint, so the measured case needs an agent change too (R-921, LEFT). +stop still resumes the apps at once (pinned by test). **The host-wide busy lock (2026-10-19, operator answer „add one +field"):** the agent serves it as `host_busy` on `GET /backup/status` (agent Unreleased `56aabc7`); controller 0.335.0 +(the floor) reads it before any stop — held → no app stopped, each due tier recorded as contention (retry in 15 min, +one „BLOCKED" alarm after 3 h), exactly the verdict a 409 gets. An agent without the field keeps the old order (stop, +ask, resume at once). The check is a read, not a reservation: a gate taken between the read and the start costs one +short stop, resumed at once. The measured case is closed on a box once it runs the next agent release. **Measured 2026-10-05 on demo-hp (9 apps, controller v0.295.0):** „Mentés most" stopped the apps at 09:19:08Z, the local tier ran 09:19:29–09:24:09, the PBS tier was busy (the controller logged a retry in 15 min; no second stop was seen in the next 55 min), the last app was back at 09:24:55Z — the longest stop **5 min 47 s**, for the local tier alone. The button text and its confirm (v0.296.0) give both diff --git a/documentation/architecture/11-os-updates.md b/documentation/architecture/11-os-updates.md index 46befbb3..6ab2d929 100644 --- a/documentation/architecture/11-os-updates.md +++ b/documentation/architecture/11-os-updates.md @@ -724,7 +724,11 @@ Evidence: `audits/os-host-lane-2026-10-04/` (parts A–G). skips the host step. - **The host health rule** (`HostHealthVerdict`, pinned in `internal/osupdate`): `felhom-agent`, `pveproxy`, `pvedaemon`, `pvestatd` and `pve-cluster` are `active`; the customer guest runs; the guest health rule (§8.1) - passes; the tunnel is `running` — an `unknown` tunnel does not fail it, `not_running` does. Same 5-minute wait. + passes; the tunnel is `running` — `not_running` and `unknown` fail it (CORRECTED 2026-10-19: this line said + „an `unknown` tunnel does not fail it"; the code always failed it, which is what Tester4 hit, R-1070). **Since agent + Unreleased `56aabc7` (R-1070):** a guest with NO cloudflared container reads `absent` (Docker 29's lower-case + „no such object" used to fall through to `unknown`), and `absent` passes when the reading from before the step also + listed no cloudflared container (Docker answered) — a box with no tunnel set up. Same 5-minute wait. - **Separate approved sets:** host and guest releases are separate (`os-host-…`, `os-guest-…`), each by the same rule (24 h, 1 night of THAT layer, every ring-0 box). Ring 1 receives `host_release` beside `release`. - **Reboot needed:** reported when PID 1 or `lxc-start` maps a replaced file, with the date of the first scanned diff --git a/documentation/audits/night-2026-10-19/P3/00-baseline.txt b/documentation/audits/night-2026-10-19/P3/00-baseline.txt new file mode 100644 index 00000000..4d7e0896 --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/00-baseline.txt @@ -0,0 +1,25 @@ +Mon Oct 19 18:45:25 UTC 2026 +gitea.dooplex.hu/admin/felhom-controller:0.331.0 +filebrowser gtstef/filebrowser:1.5.6-stable Up 16 hours (healthy) +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 18 hours (healthy) +paperless-postgres postgres:18-alpine Up 18 hours (healthy) +paperless-redis redis:7-alpine Up 18 hours (healthy) +immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 18 hours (healthy) +immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 18 hours (healthy) +immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 18 hours (healthy) +immich-redis redis:7-alpine Up 18 hours (healthy) +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.331.0 Up 22 hours (healthy) +traefik traefik:v3.7.13 Up 3 days +/opt/docker/stacks/rallly +docker-compose.yml +hold-logs +Filesystem Size Used Avail Use% Mounted on +/dev/loop1 69G 11G 54G 17% /var/lib/felhom +total 20 +drwx------ 2 root root 4096 Sep 30 08:40 . +drwxr-xr-x 4 root root 4096 Oct 19 18:46 .. +-rw-r--r-- 1 root root 91 Sep 30 08:40 known_hosts +-rw------- 1 root root 64 Sep 30 08:40 repo_password +-rw------- 1 root root 411 Sep 30 08:40 ssh_key +83723cedb2aa77ace51345e585937720fde114d1f0b24da21eb37fba4bd39374 settings.json +-rw-r--r-- 1 root root 27003 Oct 19 03:14 /root/r893-settings.json.before diff --git a/documentation/audits/night-2026-10-19/P3/01-marker-A.txt b/documentation/audits/night-2026-10-19/P3/01-marker-A.txt new file mode 100644 index 00000000..6c1708a9 --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/01-marker-A.txt @@ -0,0 +1,13 @@ +INSERT 0 1 +=== A-written 18:49:07 UTC +rallly lukevella/rallly:4.15.3 Up About a minute (healthy) +rallly-postgres postgres:18-alpine Up About a minute (healthy) +markers (phase=rows): A-before-snapshot=1 +r893_after table exists: 0 +r893_after rows: ERROR: relation "r893_after" does not exist +LINE 1: select count(*) from r893_after; + ^ +public tables: 34 +prisma migrations: 153 +db volume created: 2026-10-19T18:47:54Z +definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine diff --git a/documentation/audits/night-2026-10-19/P3/02-snapshot-and-marker-B.txt b/documentation/audits/night-2026-10-19/P3/02-snapshot-and-marker-B.txt new file mode 100644 index 00000000..bab6db09 --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/02-snapshot-and-marker-B.txt @@ -0,0 +1,13 @@ +INSERT 0 1 +CREATE TABLE +INSERT 0 1 +=== B-written 18:51:46 UTC +rallly lukevella/rallly:4.15.3 Up 35 seconds (healthy) +rallly-postgres postgres:18-alpine Up 41 seconds (healthy) +markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1 +r893_after table exists: 1 +r893_after rows: 1 +public tables: 35 +prisma migrations: 153 +db volume created: 2026-10-19T18:47:54Z +definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine diff --git a/documentation/audits/night-2026-10-19/P3/03-forced-failure.txt b/documentation/audits/night-2026-10-19/P3/03-forced-failure.txt new file mode 100644 index 00000000..fcbdbab0 --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/03-forced-failure.txt @@ -0,0 +1,11 @@ +126 +34 +2660:-- PostgreSQL database dump complete +2660-SELECT r893_forced_replay_failure(); -- R-893 measurement 2026-10-19: forced error AFTER the volumes are replaced +2661:-- PostgreSQL database dump complete +2720-SELECT r893_forced_replay_failure(); -- R-893 measurement 2026-10-19: forced error AFTER the volumes are replaced +2721:-- PostgreSQL database dump complete +total 67608 +drwxr-xr-x 2 root root 4096 Oct 19 18:54 . +drwxr-xr-x 5 root root 4096 Oct 19 18:54 .. +-rw-r--r-- 1 root root 69221888 Oct 19 18:54 rallly_rallly_postgres_data.tar diff --git a/documentation/audits/night-2026-10-19/P3/04-before-snapshot-2.txt b/documentation/audits/night-2026-10-19/P3/04-before-snapshot-2.txt new file mode 100644 index 00000000..bd5974bd --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/04-before-snapshot-2.txt @@ -0,0 +1,15 @@ +CREATE TABLE +INSERT 0 1 +INSERT 0 1 +=== before-snapshot-2 18:53:21 UTC +rallly lukevella/rallly:4.15.3 Up 2 minutes (healthy) +rallly-postgres postgres:18-alpine Up 2 minutes (healthy) +markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1, C-in-snapshot-2=1 +r893_after table exists: 1 +r893_after rows: 1 +r893_only_old exists: 1 +r893_new2 exists: 0 +public tables: 36 +prisma migrations: 153 +db volume created: 2026-10-19T18:47:54Z +definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine diff --git a/documentation/audits/night-2026-10-19/P3/05-live-before-restore.txt b/documentation/audits/night-2026-10-19/P3/05-live-before-restore.txt new file mode 100644 index 00000000..591d6771 --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/05-live-before-restore.txt @@ -0,0 +1,16 @@ +DROP TABLE +INSERT 0 1 +CREATE TABLE +INSERT 0 1 +=== LIVE-before-the-restore 18:54:53 UTC +rallly lukevella/rallly:4.15.3 Up 30 seconds (healthy) +rallly-postgres postgres:18-alpine Up 35 seconds (healthy) +markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1, C-in-snapshot-2=1, D-after-snapshot-2=1 +r893_after table exists: 1 +r893_after rows: 1 +r893_only_old exists: 0 +r893_new2 exists: 1 +public tables: 36 +prisma migrations: 153 +db volume created: 2026-10-19T18:47:54Z +definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine diff --git a/documentation/audits/night-2026-10-19/P3/06-reconstitute-log.txt b/documentation/audits/night-2026-10-19/P3/06-reconstitute-log.txt new file mode 100644 index 00000000..3479ee45 --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/06-reconstitute-log.txt @@ -0,0 +1,40 @@ +2026/10/19 18:55:43 dbdump.go:277: [DEBUG] DumpOne: starting dump for container=rallly-postgres, stack=rallly, dbType=postgres, dest=/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql +2026/10/19 18:55:43 dbdump.go:314: [DEBUG] DumpOne: pg_dump command: docker exec 8ad4e7b50692 pg_dump -U rallly -d rallly --clean --if-exists --no-owner --no-privileges +2026/10/19 18:55:43 [WARN] [backup] ValidateDump: /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data +2026/10/19 18:55:43 dbdump.go:424: [DEBUG] DumpOne: completed rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (size=110.0 KB, valid=true, tables=36, duration=325ms) +2026/10/19 18:55:43 dbdump.go:429: [INFO] [backup] DB dump: rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB, 325ms, 36 tables) +2026/10/19 18:55:43 offbox_reconstitute.go:262: [INFO] [offbox] rallly: pre-restore safety dump written → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB) +2026/10/19 18:55:43 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=running deployed=true containers=2 +2026/10/19 18:55:43 manager.go:1440: [INFO] [stacks] Stopping stack: rallly +2026/10/19 18:55:43 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly) +2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/10/19 18:55:54 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.7s) +2026/10/19 18:55:54 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly +2026/10/19 18:55:55 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully +2026/10/19 18:55:55 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly +2026/10/19 18:55:55 manager.go:1404: [INFO] [stacks] Starting stack rallly services only: [rallly-postgres] +2026/10/19 18:55:55 manager.go:1695: [DEBUG] Running: docker compose up -d rallly-postgres (in /opt/docker/stacks/rallly) +2026/10/19 18:55:56 manager.go:1415: [INFO] [stacks] Stack rallly services [rallly-postgres] started (took 0.4s) +2026/10/19 18:55:56 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine +2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813) +2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly +2026/10/19 18:55:56 restore_db.go:87: [INFO] [backup] Restore rallly: replaying DB dump into rallly-postgres (postgres) +2026/10/19 18:55:56 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql into rallly-postgres (postgres) +2026/10/19 18:55:57 dbdump.go:814: [ERROR] [backup] postgres import into rallly-postgres FAILED: exit status 3 — engine output follows: +2026/10/19 18:55:57 offbox_reconstitute.go:1062: [ERROR] [offbox] rallly: database replay failed, rolling back to the pre-restore state: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3 +2026/10/19 18:55:57 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine +2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813) +2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly +2026/10/19 18:55:57 offbox_reconstitute.go:603: [DEBUG] [offbox] rallly: rallly-postgres was re-created during the restore (8ad4e7b50692 → b55136567813) — rolling back into the live container +2026/10/19 18:55:57 offbox_reconstitute.go:606: [INFO] [offbox] rallly: rolling back to the pre-restore state from pre-restore-20261019T185543Z-rallly-postgres.sql +2026/10/19 18:55:57 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres) +2026/10/19 18:55:58 dbdump.go:820: [INFO] [backup] Imported DB dump pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres) +2026/10/19 18:55:58 offbox_reconstitute.go:611: [INFO] [offbox] rallly: rollback complete — 1 database(s) returned to the pre-restore state +2026/10/19 18:55:58 settings.go:1951: [WARN] [settings] restore hold SET for rallly — the app stays stopped until it is cleared +2026/10/19 18:55:58 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=starting deployed=true containers=1 +2026/10/19 18:55:58 manager.go:1440: [INFO] [stacks] Stopping stack: rallly +2026/10/19 18:55:58 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly) +2026/10/19 18:55:59 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 0.9s) +2026/10/19 18:55:59 offbox_reconstitute.go:531: [ERROR] [offbox] rallly: the restore stopped half-way after files/volumes/version had moved — HOLDING the app stopped for support (R-893); error was: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3 +2026/10/19 18:55:59 offbox_handlers.go:547: [ERROR] [web] off-box reconstitute rallly (async): a(z) rallly adatbázisának visszaállítása sikertelen. Az adatbázis a visszaállítás előtti állapotban van, de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot diff --git a/documentation/audits/night-2026-10-19/P3/07-after-failed-replay.txt b/documentation/audits/night-2026-10-19/P3/07-after-failed-replay.txt new file mode 100644 index 00000000..a1e82441 --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/07-after-failed-replay.txt @@ -0,0 +1,14 @@ +=== after the failed replay 18:56:42 UTC +(no rallly container listed = compose down; app held) +definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine +Unable to find image 'alpine:3' locally +3: Pulling from library/alpine +Digest: sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6 +Status: Downloaded newer image for alpine:3 +markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1, C-in-snapshot-2=1, D-after-snapshot-2=1 +r893_after exists: 1 rows: 1 +r893_only_old exists: 1 rows: 1 +r893_new2 exists: 1 rows: 1 +public tables: 37 +prisma migrations: 153 +read copy removed diff --git a/documentation/audits/night-2026-10-19/P3/08-app-state-and-start.txt b/documentation/audits/night-2026-10-19/P3/08-app-state-and-start.txt new file mode 100644 index 00000000..701fcec6 --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/08-app-state-and-start.txt @@ -0,0 +1,4 @@ +GET stack: 200 {'status': None, 'state': 'stopped', 'held': None, 'hold': None, 'restore_hold': None, 'deployed': True} +POST start: 409 {"ok": false, "data": {"reason": "held"}, "error": "a(z) rallly visszaállítása 2026-10-19 18:55-kor félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás biztonsági okból leállítva marad. Ehhez segítség kell: vedd fel velünk a kapcsolatot"} +restore-status: 200 {"ok": true, "data": {"running": false, "op": "offbox-reconstitute", "stack": "rallly", "started_at": "2026-10-19T18:55:42.483513793Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": false, "message": "A teljes visszaállítás sikertelen: a(z) rallly adatbázisának visszaállítása sikertelen. Az adatbázis a visszaállítás előtti állapotban van, de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot", "finished_at": "2026-10-19T18:55:59.246941172 +settings restore hold: {'stack': 'rallly', 'at': '2026-10-19T18:55:58Z', 'replay_error': 'importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3', 'reason': 'restore_mixed'} diff --git a/documentation/audits/night-2026-10-19/P3/09-controller-log-window.txt b/documentation/audits/night-2026-10-19/P3/09-controller-log-window.txt new file mode 100644 index 00000000..495c8d1d --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/09-controller-log-window.txt @@ -0,0 +1,176 @@ +2026/10/19 18:55:42 auth.go:160: [DEBUG] [web] login attempt: visitor 192.168.0.180 (peer 172.18.0.3:41876, X-Forwarded-For "192.168.0.180", CF-Connecting-IP "") +2026/10/19 18:55:42 auth.go:192: [DEBUG] [web] login successful from 192.168.0.180, creating session +2026/10/19 18:55:42 session_store.go:140: [DEBUG] [web] sessions: saved 142 session(s) to dashboard-sessions.json +2026/10/19 18:55:42 auth.go:316: [DEBUG] [web] session created, expires=2026-10-26T18:55:42Z, active_sessions=146 +2026/10/19 18:55:42 auth.go:212: [INFO] [web] Login from 192.168.0.180 +2026/10/19 18:55:42 auth.go:144: [DEBUG] [web] auth: valid session for GET / +2026/10/19 18:55:42 server.go:637: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.3:41876 +2026/10/19 18:55:42 auth.go:144: [DEBUG] [web] auth: valid session for GET /launcher +2026/10/19 18:55:42 server.go:637: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.3:41876 +2026/10/19 18:55:42 auth.go:144: [DEBUG] [web] auth: valid session for POST /backup/offbox/reconstitute +2026/10/19 18:55:42 server.go:637: [DEBUG] [web] ServeHTTP: POST /backup/offbox/reconstitute from 172.18.0.3:41876 +2026/10/19 18:55:43 [WARN] [backup] ValidateDump: /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data +2026/10/19 18:55:43 offbox_reconstitute.go:1206: [WARN] [offbox] rallly: the snapshot dump rallly-postgres.sql has no account rows — it may predate the customer's data +2026/10/19 18:55:43 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106) +2026/10/19 18:55:43 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/10/19 18:55:43 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 5e59e7bdadc1 rallly rallly lukevella/rallly:4.15.3 +8ad4e7b50692 rallly-postgres rallly postgres:18-alpine +197253b4df3a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +f7b4c0e21fd3 paperless-redis paperless-ngx redis:7-alpine +8c1fda4f57c7 paperless-postgres paperless-ngx postgres:18-alpine +011c9b739e1a immich-server immich ghcr.io/immich-app/immich-server:v3.2.4 +224ee4af256c immich-machine-learning immich ghcr.io/immich-app/immich-machine-learning:v3.2.4 +c030b51054cb... +2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container rallly (image=lukevella/rallly:4.15.3, not a database) +2026/10/19 18:55:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=8ad4e7b50692) +2026/10/19 18:55:43 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly +2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/10/19 18:55:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=8c1fda4f57c7) +2026/10/19 18:55:43 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/10/19 18:55:43 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database) +2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database) +2026/10/19 18:55:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=c030b51054cb) +2026/10/19 18:55:43 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich +2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database) +2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893-sftp (image=atmoz/sftp:alpine, not a database) +2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database) +2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.335.0, not a database) +2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database) +2026/10/19 18:55:43 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 10 non-DB container(s) +2026/10/19 18:55:43 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/10/19 18:55:43 dbdump.go:277: [DEBUG] DumpOne: starting dump for container=rallly-postgres, stack=rallly, dbType=postgres, dest=/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql +2026/10/19 18:55:43 dbdump.go:314: [DEBUG] DumpOne: pg_dump command: docker exec 8ad4e7b50692 pg_dump -U rallly -d rallly --clean --if-exists --no-owner --no-privileges +2026/10/19 18:55:43 [WARN] [backup] ValidateDump: /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data +2026/10/19 18:55:43 dbdump.go:424: [DEBUG] DumpOne: completed rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (size=110.0 KB, valid=true, tables=36, duration=325ms) +2026/10/19 18:55:43 dbdump.go:429: [INFO] [backup] DB dump: rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB, 325ms, 36 tables) +2026/10/19 18:55:43 offbox_reconstitute.go:262: [INFO] [offbox] rallly: pre-restore safety dump written → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB) +2026/10/19 18:55:43 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=running deployed=true containers=2 +2026/10/19 18:55:43 manager.go:1440: [INFO] [stacks] Stopping stack: rallly +2026/10/19 18:55:43 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, IMPORT_PATH] (10 app + 0 system) +2026/10/19 18:55:43 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly) +2026/10/19 18:55:44 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714348KB) → used≈12771MB avail≈13127MB +2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714348 availKB=15567204 → total=25898MB avail=13127MB used=12771MB (49.3%) +2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=12.0GB avail=52.8GB (18.5%) +2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=175.4GB avail=714.7GB (19.7%) +2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.25 2.62 2.11 8/1611 4393" → 1m=2.25 5m=2.62 15m=2.11 +2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 57.6°C (hwmon2) +2026/10/19 18:55:44 info.go:13: [DEBUG] [system] GetInfo done in 59ms — mem=12771MB/25898MB (49.3%), rootDisk=12.0GB/68.4GB (18.5%), load=2.25/2.62/2.11, temp=57.6°C (hwmon2), cpu=13.9% +2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting +2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/10/19 18:55:44 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (3 skipped not due, 0 skipped no container) +2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting +2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/19 18:55:44 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks +2026/10/19 18:55:54 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/19 18:55:54 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/19 18:55:54 manager.go:838: [INFO] [stacks] Status refresh: 10 containers across 70 stacks +2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/10/19 18:55:54 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (3 skipped not due, 0 skipped no container) +2026/10/19 18:55:54 manager.go:1717: [DEBUG] Command completed: docker compose down (took 10.7s) +2026/10/19 18:55:54 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.7s) +2026/10/19 18:55:54 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks +2026/10/19 18:55:54 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly +2026/10/19 18:55:55 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully +2026/10/19 18:55:55 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly +2026/10/19 18:55:55 manager.go:1404: [INFO] [stacks] Starting stack rallly services only: [rallly-postgres] +2026/10/19 18:55:55 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SECRET_PASSWORD, SUBDOMAIN, DB_PASSWORD, IMPORT_PATH] (10 app + 0 system) +2026/10/19 18:55:55 manager.go:1695: [DEBUG] Running: docker compose up -d rallly-postgres (in /opt/docker/stacks/rallly) +2026/10/19 18:55:56 manager.go:1717: [DEBUG] Command completed: docker compose up -d rallly-postgres (took 0.4s) +2026/10/19 18:55:56 manager.go:1415: [INFO] [stacks] Stack rallly services [rallly-postgres] started (took 0.4s) +2026/10/19 18:55:56 manager.go:838: [INFO] [stacks] Status refresh: 10 containers across 70 stacks +2026/10/19 18:55:56 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/10/19 18:55:56 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine +197253b4df3a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +f7b4c0e21fd3 paperless-redis paperless-ngx redis:7-alpine +8c1fda4f57c7 paperless-postgres paperless-ngx postgres:18-alpine +011c9b739e1a immich-server immich ghcr.io/immich-app/immich-server:v3.2.4 +224ee4af256c immich-machine-learning immich ghcr.io/immich-app/immich-machine-learning:v3.2.4 +c030b51054cb immich-postgres immich ghcr.io/immich-app/postgres... +2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813) +2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly +2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=8c1fda4f57c7) +2026/10/19 18:55:56 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database) +2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database) +2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=c030b51054cb) +2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich +2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database) +2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893-sftp (image=atmoz/sftp:alpine, not a database) +2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database) +2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.335.0, not a database) +2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database) +2026/10/19 18:55:56 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 9 non-DB container(s) +2026/10/19 18:55:56 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/10/19 18:55:56 restore_db.go:87: [INFO] [backup] Restore rallly: replaying DB dump into rallly-postgres (postgres) +2026/10/19 18:55:56 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql into rallly-postgres (postgres) +2026/10/19 18:55:57 dbdump.go:814: [ERROR] [backup] postgres import into rallly-postgres FAILED: exit status 3 — engine output follows: +ERROR: function r893_forced_replay_failure() does not exist +LINE 1: SELECT r893_forced_replay_failure(); + ^ +HINT: No function matches the given name and argument types. You might need to add explicit type casts. +2026/10/19 18:55:57 offbox_reconstitute.go:1062: [ERROR] [offbox] rallly: database replay failed, rolling back to the pre-restore state: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3 +2026/10/19 18:55:57 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/10/19 18:55:57 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine +197253b4df3a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +f7b4c0e21fd3 paperless-redis paperless-ngx redis:7-alpine +8c1fda4f57c7 paperless-postgres paperless-ngx postgres:18-alpine +011c9b739e1a immich-server immich ghcr.io/immich-app/immich-server:v3.2.4 +224ee4af256c immich-machine-learning immich ghcr.io/immich-app/immich-machine-learning:v3.2.4 +c030b51054cb immich-postgres immich ghcr.io/immich-app/postgres... +2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813) +2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly +2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=8c1fda4f57c7) +2026/10/19 18:55:57 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database) +2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database) +2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=c030b51054cb) +2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich +2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database) +2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893-sftp (image=atmoz/sftp:alpine, not a database) +2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database) +2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.335.0, not a database) +2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database) +2026/10/19 18:55:57 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 9 non-DB container(s) +2026/10/19 18:55:57 dbdump.go:208: [INFO] [backup] Discovered 3 databases +2026/10/19 18:55:57 offbox_reconstitute.go:603: [DEBUG] [offbox] rallly: rallly-postgres was re-created during the restore (8ad4e7b50692 → b55136567813) — rolling back into the live container +2026/10/19 18:55:57 offbox_reconstitute.go:606: [INFO] [offbox] rallly: rolling back to the pre-restore state from pre-restore-20261019T185543Z-rallly-postgres.sql +2026/10/19 18:55:57 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres) +2026/10/19 18:55:58 dbdump.go:820: [INFO] [backup] Imported DB dump pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres) +2026/10/19 18:55:58 offbox_reconstitute.go:611: [INFO] [offbox] rallly: rollback complete — 1 database(s) returned to the pre-restore state +2026/10/19 18:55:58 settings.go:1951: [WARN] [settings] restore hold SET for rallly — the app stays stopped until it is cleared +2026/10/19 18:55:58 settings.go:922: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (27892 bytes) +2026/10/19 18:55:58 settings.go:925: [INFO] [settings] Settings saved +2026/10/19 18:55:58 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=starting deployed=true containers=1 +2026/10/19 18:55:58 manager.go:1440: [INFO] [stacks] Stopping stack: rallly +2026/10/19 18:55:58 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, IMPORT_PATH] (10 app + 0 system) +2026/10/19 18:55:58 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly) +2026/10/19 18:55:59 manager.go:1717: [DEBUG] Command completed: docker compose down (took 0.9s) +2026/10/19 18:55:59 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 0.9s) +2026/10/19 18:55:59 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks +2026/10/19 18:55:59 offbox_reconstitute.go:531: [ERROR] [offbox] rallly: the restore stopped half-way after files/volumes/version had moved — HOLDING the app stopped for support (R-893); error was: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3 +2026/10/19 18:55:59 notifier.go:1373: [WARN] notifier disabled (no hub configured): DROPPED event backup_run_failures (severity error) — further backup_run_failures events are logged at DEBUG only +2026/10/19 18:55:59 offbox_handlers.go:547: [ERROR] [web] off-box reconstitute rallly (async): a(z) rallly adatbázisának visszaállítása sikertelen. Az adatbázis a visszaállítás előtti állapotban van, de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot +2026/10/19 18:55:59 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting +2026/10/19 18:56:04 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting +2026/10/19 18:56:04 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting +2026/10/19 18:56:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/10/19 18:56:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/10/19 18:56:04 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (2 skipped not due, 0 skipped no container) +2026/10/19 18:56:04 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks diff --git a/documentation/audits/night-2026-10-19/P3/10-teardown.txt b/documentation/audits/night-2026-10-19/P3/10-teardown.txt new file mode 100644 index 00000000..e7762441 --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/10-teardown.txt @@ -0,0 +1,37 @@ +20:57:56 [X] stop -> 200 {'ok': True, 'message': 'Stack rallly stop completed'} +20:58:28 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'rallly', 'volumes_removed': ['rallly_rallly_postgres_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': ' +20:58:36 [X] after remove: deployed=False leftovers='/opt/docker/stacks/rallly' +remove: 200 +r893-sftp +images-removed +ls: cannot access '/mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly': No such file or directory +ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/rallly': No such file or directory +vol-check-done +Stopping 'felhom-controller-bootstrap.service', but its triggering units are still active: +felhom-controller-bootstrap.path +felhom-controller Up 2 seconds (health: starting) +83723cedb2aa77ace51345e585937720fde114d1f0b24da21eb37fba4bd39374 /var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json +total 20 +drwx------ 2 root root 4096 Sep 30 08:40 . +drwxr-xr-x 4 root root 4096 Oct 19 18:59 .. +-rw-r--r-- 1 root root 91 Sep 30 08:40 known_hosts +-rw------- 1 root root 64 Sep 30 08:40 repo_password +-rw------- 1 root root 411 Sep 30 08:40 ssh_key +0.335.0-image-removed +0 +gitea.dooplex.hu/admin/felhom-controller:0.331.0 Up 50 seconds (healthy) +3bc7298260da6566d4c09245124acd92f20f62bbebe46b0181c11d99a8314f2b /var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json +0 +filebrowser Up 2 minutes (healthy) +paperless-webserver Up 7 minutes (healthy) +paperless-redis Up 7 minutes (healthy) +paperless-postgres Up 7 minutes (healthy) +immich-server Up 7 minutes (healthy) +immich-machine-learning Up 8 minutes (healthy) +immich-postgres Up 8 minutes (healthy) +immich-redis Up 8 minutes (healthy) +traefik Up 3 days +/var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json {'immich': None, 'paperless-ngx': None, 'privatebin': None} +offbox_target set: False +restore holds: ['chaoscrash', 'chaosoom', 'chaosoomb', 'gokapi'] +0 diff --git a/documentation/audits/night-2026-10-19/P3/README.md b/documentation/audits/night-2026-10-19/P3/README.md new file mode 100644 index 00000000..ade8e5e8 --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/README.md @@ -0,0 +1,95 @@ +# P3 — R-893: a failed off-site replay, measured on scratch guest 9202 (2026-10-19 evening) + +Night brief 2026-10-19, Part 3 — **measure and design only; no restore code changed.** Tier 0 only (demo-hp, scratch +guest 9202). Code read live: `felhom-controller/controller/internal/backup/offbox_reconstitute.go` at `15ea002` +(controller 0.335.0). Architecture: `07-backup-architecture.md` §6.3 (the R-893 known limit, the failure ladder +„replay → rollback → hold"); `09` §3 decision 192 (D8: first hold, then „put back exactly as it was"). + +## How it was reproduced (the product path, one forced error) + +- 9202 put on controller **0.335.0** for the run (was 0.331.0; put back after). Hub is OFF on 9202. +- A **scratch off-site target**: an SFTP server container (`atmoz/sftp:alpine`, `r893-sftp`, port 2222) on 9202 itself, + repo on 9202's own disk (`/var/lib/felhom/r893-sftp`, nvme-scratch — not `pve/data`). Configured through the + household's own endpoint `POST /backup/offbox/config` (a throwaway ed25519 key, the server's host key pinned), + escrow confirmed with the legacy manual endpoint (9202 has no agent), only `rallly` switched on. +- Throwaway database app: **Rallly 4.15.3 + postgres:18-alpine**, the DB in the named volume + `rallly_rallly_postgres_data` (deployed through the product, `box_walk.deploy`). +- Markers in the real database (positive observables): table `r893_marker` with one row per phase, plus tables that + exist only at certain times. +- The **forced error**: one line `SELECT r893_forced_replay_failure();` put into the PREPARED scratch copy of the + snapshot's dump, just before its completion marker (so the completeness gate still passes). The real + `ReconstituteFromOffsite` then replaced the volume, started the DB alone and failed the replay — i.e. the replay + failed **after the volume was replaced**, as the brief asked. + +## Timeline (UTC) + +| Time | Act | File | +|---|---|---| +| 18:45 | baseline: 9202 on 0.331.0, no off-site target, settings + off-box folder saved | `00-baseline.txt` | +| 18:48 | Rallly deployed; marker **A** | `01-marker-A.txt` | +| 18:51:19 | off-site run → snapshot 1; then marker **B** + table `r893_after` | `02-snapshot-and-marker-B.txt` | +| 18:53 | table `r893_only_old` + marker **C**; off-site run → **snapshot 2** (18:54:29, `93dbb37d`) | `04-before-snapshot-2.txt` | +| 18:54:53 | after snapshot 2: `r893_only_old` DROPPED (like a table a newer version removes), marker **D**, table `r893_new2` — the live state the household has | `05-live-before-restore.txt` | +| 18:54:59 | full restore prepared into scratch (66.1 MB); the failing line injected | `03-forced-failure.txt` | +| 18:55:42 | `POST /backup/offbox/reconstitute` (confirm) | `06-reconstitute-log.txt`, `09-controller-log-window.txt` | +| 18:55:43 | undo dump of the LIVE db written (`pre-restore-…-rallly-postgres.sql`, 36 tables) | | +| 18:55:55 | the snapshot's volume put in place (`Restored 1 Docker volume(s)`) | | +| 18:55:57 | replay FAILED (`exit status 3`, single transaction — nothing of it applied) | | +| 18:55:58 | rollback: the undo dump loaded into the database **that now sits on snapshot 2's volume** — „rollback complete" | | +| 18:55:58–59 | hold SET (`reason: restore_mixed`), app stopped | | +| 18:56:42 | the held volume read through a throwaway COPY (the held state untouched) | `07-after-failed-replay.txt` | +| 18:57 | household start → **409 held** | `08-app-state-and-start.txt` | +| 18:58–19:01 | teardown | `10-teardown.txt` | + +## What the failed replay left + +| | Live before the restore | After the failed replay + rollback | +|---|---|---| +| marker rows | A, B, C, D | A, B, C, D (the newest rows — the undo dump won) | +| `r893_new2` (written after snapshot 2) | present, 1 row | present, 1 row | +| `r893_only_old` (in snapshot 2 only, dropped after) | **absent** | **present, 1 row** — left from the snapshot's volume | +| public tables | 36 | **37** | +| definition on disk | rallly 4.15.3, postgres 18 | rallly 4.15.3, postgres 18 (unchanged — same version; the R-1106 password overlay was empty, so the definition did not move) | +| app | running | **stopped, held** (`restore_mixed`); a start answers 409 „held" | +| other volumes / files | — | none for Rallly (its only volume is the DB's) | + +So the rollback gives the **newest rows on top of the older volume**: everything the undo dump knows is exact, and +whatever only the older volume had stays behind (the dump's `--clean --if-exists` drops only the objects IT names). +That is the R-638 / R-893 mix, now measured, not read. The 0.304.0 hold works as built: the app does not start on it. + +**The sentences the household sees:** +- the restore page (`err.backup.db_restore_failed_held_mixed`): „Az adatbázis a visszaállítás előtti állapotban van, + de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze." — **measured false on + both halves for this app**: the database is NOT as it was (a table from the backup stayed), and Rallly had no files + or other data from the backup (only its database volume moved). +- the start refusal: „…félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás + biztonsági okból leállítva marad." — true. + +## Not measured tonight (said plainly) + +- The **version-change** branch (the snapshot's older definition written, then the live one written back by + `holdAppAfterMixedRestore`). It needs an app installed at an older catalog step and updated before the restore; the + drill catalog that could do it is a git repo this helper was told not to commit to. The branch is pinned by + `r893_mixed_restore_test.go`; it ends in the same hold. +- What support does next: no product action releases a `restore_mixed` hold with the data made right — that is the + gap option A closes. + +## Side observations + +- The manual off-site run's pre-push dump leg dumped and restarted **every** app on the box (paperless, immich), not + only the one switched on (log: `App-data backup completed: 3 databases … 4 volume dump(s)`). By design (the + coherent pair), outside R-893's scope. +- 9202's address moved (DHCP: .114 → .121); `box_walk.py`'s 9202 row still names .114 — run with `BASE=`. + +## Teardown (three layers) + +- **Machine (9202):** Rallly removed through the product (`volumes_removed: rallly_rallly_postgres_data`, its backups + and the off-site scratch gone); `r893-sftp` container, `atmoz/sftp:alpine` and `alpine:3` images (pulled tonight), + the repo folder, the key, all `/root/r893-*` and `/tmp/r893-*` removed; `settings.json` and the `offbox/` folder put + back from the copies taken at 18:45 (off-site target unset, no per-app off-site switch, restore holds as before: + chaoscrash, chaosoom, chaosoomb, gokapi); controller back on **0.331.0** (healthy, 0 `[ERROR]`), the 0.335.0 image + removed from 9202. The template folder `/opt/docker/stacks/rallly` (catalog files + an old `hold-logs`) was there + before and stays. +- **Host (demo-hp):** only `/tmp/r893-*` copies, removed. Nothing on `pve/data`. +- **Hub:** nothing. 9202 has no hub; the hold's events were dropped on the box („notifier disabled (no hub + configured)"). diff --git a/documentation/audits/night-2026-10-19/P3/design-R-893.md b/documentation/audits/night-2026-10-19/P3/design-R-893.md new file mode 100644 index 00000000..a0c2f46b --- /dev/null +++ b/documentation/audits/night-2026-10-19/P3/design-R-893.md @@ -0,0 +1,51 @@ +# R-893 — after a failed off-site replay: put the app back exactly as it was? (design note, 2026-10-19) + +Status: **design only, for the operator.** Measured first (`README.md` in this folder, 9202, controller 0.335.0). +Architecture: `07` §6.3 (the failure ladder „replay → rollback → hold"), `09` §3 decision 192 (D8: first hold — +built in 0.304.0 — then „put back exactly as it was"). No restore code was changed. + +## What the measurement showed (plain words) + +An off-site restore of one app replaces the app's database volume with the backup's, then loads the backup's +database copy. When that load fails, the box puts the household's own database copy (taken a minute before) back +**on top of the backup's volume**. Measured on Rallly: all the newest rows came back, but a table that only the backup +had **stayed behind** (37 tables instead of 36). The app is then held stopped, and a start is refused — that part +works. But nothing in the product can make the data right again; the household waits for support. And the restore +page tells them the database „is back as it was before the restore" — measured not true. + +## The question + +**When an off-site restore of one app fails half-way, should the box put the app back exactly as it was and start +it, or keep holding it stopped for support?** + +## The options + +| | A — keep a copy of the app's volumes, and put it back | B — keep holding; only make the database part exact | +|---|---|---| +| What happens | After the undo dump and the stop, the box saves the app's live volumes (the same volume-dump helper the nightly copy uses) and keeps the live definition (already done since 0.304.0). On a failed replay it puts those volumes and the definition back — the state from one minute before — and starts the app. The undo dump stays as a second way back. | On a failed replay the box empties the database volume before it loads the undo dump (a rebuilding load, on the rollback path only). Then the database is exactly as it was. The app stays held while files or other volumes moved; an app whose only change was its database volume can start. | +| Disk | One extra copy of the app's volumes for the length of the restore (Rallly: 69 MB, made in about 1 s). A free-space check refuses BEFORE the stop when the copy does not fit — then nothing is touched. | None. | +| Code | ~1 session: the save step, the put-back on every failed branch, the fit check, tests; then one 9202 drill (tonight's setup can be reused). | ~½–1 session: the emptying step and its tests; the R-638 option-B loader, but only on this one path. | +| Risk | Low: both helpers exist and already run every night. A put-back that fails leaves the hold, as today. | Higher: the volume is emptied before the undo is loaded, so a bad undo dump leaves an empty database (the backup's volume is gone too). Files and other volumes are still the backup's. | +| What the household sees | „The restore did not finish. Nothing changed: the app runs as before." No support call. | Usually still „the app is stopped; contact us" — fewer cases than today, but the same support call. | + +## Pick: A + +It is the direction the operator already ruled (D8, second half), it makes the restore page's promise true, and the +measured cost is small: the extra disk is one copy of the app's own volumes, for minutes, and a fit check refuses +cleanly when there is no room. B removes only one of the mixed parts and adds a new way to lose the database. + +**One small fix either way, now:** the sentence `err.backup.db_restore_failed_held_mixed` says the database is back as +it was and that files and other data came from the backup. Measured: neither is reliably true (Rallly had no files or +other volumes from the backup, and its database was not as before). It should say only what is certain: the restore +stopped half-way, the app is held stopped, and nothing is lost yet. + +**If the operator does nothing:** a failed off-site restore keeps ending in a held app that only support can bring +back, and the page keeps telling the household their database is back as it was. + +## First slice and its red test (for A) + +- Red test beside `r893_mixed_restore_test.go`: a failed replay after a volume replay must call the volume put-back + with the live copy, write the live definition back, and START the app (no hold); with too little free space for the + copy, the restore must refuse with `StopStack` called 0 times. +- Then the 9202 drill (this folder's `README.md`, same steps): after the forced failure, the table count and the + markers must equal the live state before the restore (36 tables, A–D, no `r893_only_old`), and the app must run. diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 11efc819..8d916f36 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,14 @@ --- +## 2026-10-19 (evening/night brief) — no stop for a busy host, the scanner on every push, the ArgoCD prune danger gone (full text of each row: `git show f92fa5a8:documentation/backlog/OPEN-ITEMS.md`) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-921** | **When the off-site tier answers BUSY, the household still gets a short stop of every app — with no copy.** (P3) | CLOSED 2026-10-19 — **the measured case is now seen before the stop** (operator answer „add one field", 19:2x): the agent serves its host-wide gate as `host_busy` on `GET /backup/status` (agent `56aabc7`, committed, NOT released — it reaches a box with the next agent release, STATUS's morning question), and controller 0.335.0 (the floor since 18:39 UTC) reads it before any stop: held → no app stopped, each due tier recorded as contention (15-minute skip, one „BLOCKED" alarm after 3 h). An agent without the field keeps the old order (proven live: demo-hp's agent answers no `host_busy`). The race (free at the read, busy at the start) keeps the old short stop, resumed at once. Tests red-proved: agent `TestBackupStatus_R921_HostBusy`; controller `TestR921_NoStopWhileTheHostIsBusy` (without the branch: 3 apps stopped), `…SkipsStillAlarmAfterTheCeiling`, `…OtherAnswersKeepTheOldOrder`, `TestHostBusy_ReadsTheAgentsGate`. A live busy state was not made on 9201 (it needs the new agent there). | `audits/night-2026-10-19/P9/` (release), controller `15ea002`, agent `56aabc7` | +| **R-953** | **A full ArgoCD sync of the `felhom` app with prune would DELETE three live Secrets and so break the hub and Umami.** (P3) | CLOSED 2026-10-19 — **not reproducing: ArgoCD no longer tracks the three Secrets.** Read back 18:21 UTC (read only, no sync): app `felhom` Synced/Healthy at `a28f162c`, 25 tracked resources, none of them a Secret, none `requiresPruning`, `Deployment/umami` Synced; the three Secrets carry no labels and no annotations (no tracking id). A sync with prune touches only tracked resources, so the next deploy does nothing with them. No manifest change was possible or needed (a `Prune=false` annotation would live on the untracked live object). How the tracking marks went after 2026-10-12 is not recorded. The build-deploy skill's „sync only the hub" line stays. | `audits/night-2026-10-19/P4/r953-readback.txt` | +| **R-986** | **No dependency or secret scanner runs on any push, so a known-vulnerable library or a committed key is found only by a review like this one.** (P3) | CLOSED 2026-10-19 — **the warn-first gate runs on every push in all four repos.** `felhom.eu/scripts/security_scan_gate.py` (shared, never copied) in each runner's FAST set — pre-push hook and CI: govulncheck v1.8.0 (called vulnerabilities only) on each Go module, gitleaks 8.24.3 on the pushed commits with the default rules pinned; hits listed, exit 0. Accepted hits per repo in `.security-accepted` (gitleaks by rule + a hash of the line + file), reason per line. CI fetches the pinned gitleaks with both sha256 checked (seen in CI 2108: 42 hits, all accepted); CI has no Go, so govulncheck is NOT CHECKED there, out loud — the hook runs it. 8 decoys in scratch repos, red-proved. First scan of today's `main`: 0 called vulnerabilities, 0 real secrets, every gitleaks hit with a verdict. The govulncheck pin moved v1.1.4 → v1.8.0 (v1.1.4 cannot read Go 1.27 code). Warn → fail: R-1168. | `audits/night-2026-10-19/P7/triage.md`, `SCANNERS.md` §5 | + ## 2026-10-19 (third day brief) — the passphrase locks once the box uses its own key, every ep0 household gets a prune job, demo-hp's address (full text of each row: `git show 97f27a75:documentation/backlog/OPEN-ITEMS.md`) | Row | What | Closed | Evidence | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 4ee03e34..b1b7c279 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -123,7 +123,7 @@ stopping line that lies. | **R-993** | Install & onboarding | P3 | **Operator: approve the minimum machine in `documentation/audits/sizing-2026-10-13/SIZING.md` §5** (the input R-991's hardware page needs). MEASURED 2026-10-13 on the N100 demo box (61 apps deployed and removed through the product, 15 min at rest each; heavy apps with one real action). Proposal: **minimum** x86-64, 4 cores (N100 class), **8 GB**, 256 GB SSD; **recommended 16 GB**, 512 GB SSD; out: ARM / Raspberry Pi, Synology-QNAP, routers, cloud VPS. Worked from the floor (2.5 GB for the whole box with no apps) + Immich, Jellyfin, Nextcloud, Home Assistant, Vaultwarden, Paperless-ngx at their peaks (5.0 GB) + 50 % headroom ≈ 12.3 GB. 8 GB runs that set without Immich; 16 GB runs all of it twice over. Data for the estimator: `sizing.json`. **2026-10-13 (estimator session) — the installer accepts a host BELOW this minimum:** `autosize_guest_caps` has no floor on the host, only on the guest; on a 4 GB box (MemTotal ≈ 3.5 GiB) the guest gets host − 1 GB ≈ 2.4 GB and Proxmox keeps ≈ 1 GB, where it used ≈ 2.0 GB on the N100 (SIZING.md §1). Whatever minimum is approved, nothing enforces it at install today. The three-verdict wording of §5 now matches the estimator page (`/gepigeny`, R-997). | **WAITING-ON-OPERATOR** — filed 2026-10-13 | the operator's decision | Operator: approve or change §5. If nothing is done: no hardware page; a buyer with a Raspberry Pi can still download the installer (R-991 stays open) | operator | | **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC | | **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator | -| **R-1167** | Install & onboarding | P4 | **The installer does not word the hub's new 409 („this customer already has a live box").** Since hub 0.161.0 (R-1159) a re-run of `felhom-host-install.sh` against a customer whose box is live (a repair, a late `--resume`, a reinstall without the operator's delete) gets 409 from `GET /config` at its preflight; the script prints only „unexpected hub status 409 on config preflight" (`step_preflight`), not the hub's sentence (delete the old host first). The installer is published (felhom.eu/scripts), so the fix is an installer release. Found 2026-10-19. | OPEN — filed 2026-10-19 | `scripts/felhom-host-install.sh` `step_preflight` | Next installer release: a 409 case that prints the hub's body | CC | +| **R-1167** | Install & onboarding | P4 | **The installer does not word the hub's new 409 („this customer already has a live box").** Since hub 0.161.0 (R-1159) a re-run of `felhom-host-install.sh` against a customer whose box is live (a repair, a late `--resume`, a reinstall without the operator's delete) gets 409 from `GET /config` at its preflight; the script prints only „unexpected hub status 409 on config preflight" (`step_preflight`), not the hub's sentence (delete the old host first). The installer is published (felhom.eu/scripts), so the fix is an installer release. Found 2026-10-19. | **READY — written in installer 1.35.0 (`d92a712e`, `5d0dfeac`), NOT published** (the morning question): `die_live_box` says it in EN and HU at the preflight, host-enroll and the controller.yaml fetch. Harness cases (static + the real functions against a local fake hub), red-proved; without curl (the CI runner) the two behaviour cases skip out loud. Close after the publish. OPEN — filed 2026-10-19 | `scripts/felhom-host-install.sh` `step_preflight` | Next installer release: a 409 case that prints the hub's body | CC | ## Apps & catalog — 13 rows (P3 5, P4 8) @@ -158,7 +158,7 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest | — | — | operator | -| **R-893** | Backup & restore | P3 | **After a failed OFF-SITE replay, the rollback pours the NEWER pre-restore copy over the OLDER volume just put back.** Read in source 2026-10-06 (R-638 option A, not measured): `internal/backup/offbox_reconstitute.go` writes the undo copy from the live (newer) database, replaces the volumes with the snapshot's older tars, then — when the replay fails — `rollbackSafetyDump` loads that newer dump over the older database volume. The loader only drops what the dump knows, so tables the newer migration removed stay; and when the snapshot's older definition was written, the rollback branch does not put the newer definition back, so the older app starts on rolled-back data; non-database volumes stay at the snapshot's state. An order change cannot fix it (the only undo is a logical dump, and its volume was replaced). Known limit in `07` §6.3. **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-893.md` — re-verified; and the screen's „your data is back as it was" (`err.backup.db_restore_failed_rolled_back`) is false for files, other volumes and the app version. Question D8 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D8 (`09` §3 decision 192):** yes, both, in that order — first the app stays stopped for support (option C), next „put back exactly as it was" (option A). | **NARROWED — 2026-10-09: the first half DELIVERED in controller 0.304.0. Slice 0 (the live measurement) NOT run: 9202 has no off-site, and a dump cannot be cut by hand inside an encrypted off-site copy on Tester 1; it needs a scratch off-site repo — a plan, not a finding.** **NARROWED 2026-10-08 — the first half (D8 option C, the app held stopped) is built on controller main and ships tomorrow; what stays open is the second half, „put back exactly as it was” (option A: a pre-restore copy of the app's volumes and placed files, a fit check, disk for one copy; the slice-0 measurement on 9202 first).** **OPEN — filed 2026-10-06** **2026-10-06 night: verified in source, no code** — `offbox_reconstitute.go:758` (undo dump from the live DB), `:778-860` (files and volumes from the snapshot), `:804` (the snapshot's definition is written when its version differs), `:898` (the rollback loads the newer dump over the older volume; nothing writes the live definition back). Not a reorder fix: it needs R-638 option B (a rebuilding loader) or a pre-restore volume copy (disk cost; R-685's class). Which state a household gets after a failed off-site replay is the operator's call. Next: the 9202 measurement, then the design. | a design: R-638 option B (a loader that rebuilds instead of overlays) or a pre-restore volume copy | Measure it once on 9202 (a forced replay failure after an off-site restore over a migrated app); then a design for the operator | CC | +| **R-893** | Backup & restore | P3 | **After a failed OFF-SITE replay, the rollback pours the NEWER pre-restore copy over the OLDER volume just put back.** Read in source 2026-10-06 (R-638 option A, not measured): `internal/backup/offbox_reconstitute.go` writes the undo copy from the live (newer) database, replaces the volumes with the snapshot's older tars, then — when the replay fails — `rollbackSafetyDump` loads that newer dump over the older database volume. The loader only drops what the dump knows, so tables the newer migration removed stay; and when the snapshot's older definition was written, the rollback branch does not put the newer definition back, so the older app starts on rolled-back data; non-database volumes stay at the snapshot's state. An order change cannot fix it (the only undo is a logical dump, and its volume was replaced). Known limit in `07` §6.3. **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-893.md` — re-verified; and the screen's „your data is back as it was" (`err.backup.db_restore_failed_rolled_back`) is false for files, other volumes and the app version. Question D8 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D8 (`09` §3 decision 192):** yes, both, in that order — first the app stays stopped for support (option C), next „put back exactly as it was" (option A). | **READY — measured 2026-10-19 on 9202 and a design note written; the decision is the operator's** (STATUS D20). A forced replay failure after the volume was replaced (Rallly + postgres 18, scratch SFTP off-site on 9202's own disk): the undo dump's newest rows on top of the backup's volume, a table only the backup had left behind (37 tables against 36), definition unchanged (same version), app held (`restore_mixed`, start → 409). Note: `audits/night-2026-10-19/P3/design-R-893.md` — A (copy the app's live volumes before the replay, put them back on failure; one copy's disk for minutes, a fit check before the stop) or B (empty the DB volume before the undo load; no disk, a new way to lose the database). Pick A. The version-change branch was not measured (it needs a drill-catalog commit). The page's false sentence: R-1169. **NARROWED — 2026-10-09: the first half DELIVERED in controller 0.304.0. Slice 0 (the live measurement) NOT run: 9202 has no off-site, and a dump cannot be cut by hand inside an encrypted off-site copy on Tester 1; it needs a scratch off-site repo — a plan, not a finding.** **NARROWED 2026-10-08 — the first half (D8 option C, the app held stopped) is built on controller main and ships tomorrow; what stays open is the second half, „put back exactly as it was” (option A: a pre-restore copy of the app's volumes and placed files, a fit check, disk for one copy; the slice-0 measurement on 9202 first).** **OPEN — filed 2026-10-06** **2026-10-06 night: verified in source, no code** — `offbox_reconstitute.go:758` (undo dump from the live DB), `:778-860` (files and volumes from the snapshot), `:804` (the snapshot's definition is written when its version differs), `:898` (the rollback loads the newer dump over the older volume; nothing writes the live definition back). Not a reorder fix: it needs R-638 option B (a rebuilding loader) or a pre-restore volume copy (disk cost; R-685's class). Which state a household gets after a failed off-site replay is the operator's call. Next: the 9202 measurement, then the design. | a design: R-638 option B (a loader that rebuilds instead of overlays) or a pre-restore volume copy | Measure it once on 9202 (a forced replay failure after an off-site restore over a migrated app); then a design for the operator | CC | | **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):** `_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC | | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). | operator: print the sheet | Operator: print + store the sheet; then close | operator | | **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC | @@ -169,7 +169,6 @@ stopping line that lies. | **R-433** | Backup & restore | P3 | **A sub-account cannot reach ANY Storage Box snapshot, by any name — so clause (b) of the 2026-09-01 R-95 re-scope ("the rest is recoverable file by file") is NOT SUPPORTED.** MEASURED 2026-09-01 on `demo-hp` over the credential the box already holds, read-only, no delete verb issued. **The sweep:** a batched `stat -c %n` over Hetzner's port-23 restricted shell (500–600 paths per round trip, stdout carrying only paths that exist) tried **777,600** names of the vendor form `YYYY-MM-DDTHH-MM-SS` across nine full days at second granularity, and 126 alternative shapes — **zero resolved.** **The control is what makes the zero mean anything:** the identical 600-name batch with one real path appended returned it in 6 of 6 batches. **The structural cause:** `/home` (the customer data, `u629488-sub3`) is **st_dev 0,82**; `/.zfs/snapshot` is **st_dev 0,276**; `/home/.zfs` does not exist. A snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding `felhom-repo`. **What still stands:** clause (a) — the box can delete its live repository but cannot WRITE into `/.zfs/snapshot` — is unchanged and re-confirmed. **What is now open again:** the only routes to the older copy are a panel rollback of the WHOLE Storage Box (deletes newer snapshots, hits every customer on it) and the provider API (fenced by §11-D, and `hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method at all**, so it needs new code regardless). **NOT ESTABLISHED, and it is the question that decides whether per-file recovery exists for anyone:** whether the MAIN account can see the snapshots. No main-account credential exists in this project. **The ranking is Viktor's; I am not re-ranking R-95 — but the argument that moved it down is the argument this drill removed.** `audits/evidence-drill-r95-recovery-2026-09-01/` **BLOCKED-ON-PROVIDER 2026-09-01.** The one question that can move this is drafted and ready to send: **Question 1** of `documentation/runbooks/provider-questions-2026-09-01.md` — *can the MAIN account retrieve individual files from a snapshot, without a whole-box restore?* **Neither answer leaves this row where it is:** "yes" makes per-file recovery real but permanently operator-only, and closes this; "no, full restore only" means the snapshots do not bound a single customer's exposure at all, because using them costs every other customer on the box their newer snapshots — **and R-95 becomes urgent.** Nothing here can progress without it, and it is not CC's to send (§11-D). Tracked by a dated check in the DUE-CHECKS block. **⚠ A TRAP ON THE WAY TO THIS ANSWER, found by the operator 2026-09-01 and armed against in the questions file: HETZNER HAS TWO SIMILARLY-NAMED PRODUCTS WHOSE DOCS SAY OPPOSITE THINGS.** **Storage SHARE** (a managed Nextcloud, NOT us) documents *"Currently, we only support restores for the full backup ZFS snapshot to a specific point in time"* (`docs.hetzner.com/storage/storage-share/faq/backup-snapshot/`). **Storage BOX** (ours) documents the opposite — *"You can download individual files or entire directories as usual"* (`docs.hetzner.com/storage/storage-box/snapshots/`). **A web search for the obvious phrasing surfaces the SHARE page, and it reads like a definitive NO.** Tell them apart by the giveaways: the Share page talks about *Nextcloud's data cache*, a *database dump* and the *konsoleH* interface, and never mentions Storage Box. **Why this is filed and not left as trivia: a support agent could answer Question 1 from the wrong page, and that answer would push this row to the top of the register for no reason.** Question 1 now names the product, quotes the Storage Box line, and says explicitly that we know what the Share FAQ says. **If an answer cites full-snapshot-only, check WHICH PRODUCT it is about before acting on it.** **Nothing in this repository ever leaned on the Share claim** — verified by grep at the time; the only vendor line we cite is the Storage Box one. **RE-DATED 2026-09-15 → 2026-09-22 (due-checks gate):** the operator mailbox read through the Gmail connector (`(from:hetzner OR subject:hetzner OR "Storage Box") after:2026/09/01`) holds no Hetzner reply — one match, our own `offsite_snapshots_dropped` alarm. Whether the two tickets were ever sent is not visible to CC; if they were not, sending them is the operator's act. **-- ANSWERED. Hetzner replied on ticket #2026090103040671; the operator supplied the thread on 2026-09-22 after this session searched for it and WRONGLY reported no answer (see the instrument note below and R-628).** **Q1 - can the MAIN account read individual files and directories out of a specific snapshot, without restoring the whole box?** Hetzner: *"With the main account it should be possible to download files and directories from the snapshot as it is stated in the documentation"*, citing `docs.hetzner.com/storage/storage-box/snapshots#access-to-snapshots`; and separately *"A restore of a snapshot will revert the Storage Box completely to the state it was in when the snapshot was taken."* **So the Storage BOX documentation governs, not the Storage Share FAQ** - which is exactly the trap `provider-questions-2026-09-01.md` warned the reader about, and the answer came back on the right side of it. **File-level snapshot access is a MAIN-ACCOUNT capability; the sub-account cannot do it**, which matches R-433's own measured finding (777,600 names swept from a sub-account, zero hits). **HEDGE, kept because it is in the reply: "should be possible" is not "is", and nobody has yet read a file out of a snapshot from the main account on `u629488`. That is now the measurement this row needs, and it is cheap.** **Q2 - is `--append-only` enforced by Hetzner, or taken from what the client sends?** Hetzner: *"You can use --append-only and it would look like this: `command="rclone serve restic --stdio --append-only path/to/repo" `"*, with `fluix.one/blog/hetzner-restic-append-only/`. **So it is enforced by US, by pinning a FORCED COMMAND to the SSH key in the Storage Box's `authorized_keys`** - not by Hetzner globally, and not by anything the client asks for. A key without that prefix still gets a deleting server. **This is the answer R-95 has been blocked on** and it says append-only IS expressible on a Storage Box, which R-95 records as impossible over SFTP - see R-95 and R-436. **NOT YET MEASURED, and that distinction is the whole of what this row is worth: this is a support answer, not a proof.** What is owed is a real forced-command key on `u629488`, a restic `forget --prune` through it that is REFUSED, and a backup through it that still succeeds. | **BLOCKED** — **BLOCKED-ON-PROVIDER — Question 1 of `runbooks/provider-questions-2026-09-01.md`** | — | — | CC + operator | | **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC | | **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** | — | — | CC | -| **R-921** | Backup & restore | P3 | **When the off-site tier answers BUSY, the household still gets a short stop of every app — with no copy.** MEASURED 2026-10-08 night on demo-hp (read-back 2026-10-09, `audits/dooplex-survival-2026-10-09/partE/R-518.txt`): local tier 20:20–20:25Z (own stop, < 2 min), then at 20:26:14Z the agent refused the off-site request (`backup refused — a heavy operation is already in flight`, `busy=backup:local` — the night OS step ran 20:27–20:29Z), yet the controller's metrics show all 15 app containers down at 20:26:19Z and back at 20:27:19Z; the retry ran the off-site tier at 20:34Z with its own short stop. So a two-tier night costs THREE stops, one of them for nothing. This was R-518's noted "unmeasured" risk. Fix shape: ask the agent whether it is free (or reserve the slot) BEFORE quiescing, and resume at once on BUSY. | **NARROWED 2026-10-09 — built; RELEASED 2026-10-10 (controller 0.305.0 on demo-hp, demo-felhom, Tester 1):** before stopping apps the controller asks the agent which tiers have a job in flight (`GET /backup/status`); another tier's job in flight → no stop, the tier stays due (red-proved: 3 apps stopped for a refused tier before); a refusal after the stop still resumes the apps at once (pinned). Also fixes a stop after a long upload or a restart mid-upload. **LEFT: the measured case itself** — the agent's host-wide busy lock (the night OS step, restore test, fstrim) is served on NO endpoint, so the controller cannot see it before stopping; needs one agent field (e.g. `busy` on `GET /backup/status`) + a controller pre-check behind a feature probe. `07` §6.4. | an agent change | Build the agent `busy` field + its controller check; read back a two-tier night | CC | | **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) | | **R-1019** | Backup & restore | P3 | **The live download-client templates put downloads into two backup tiers nobody chose.** MEASURED 2026-10-14 with the controller's own classifier over the live templates (`audits/media-all-drives-2026-10-14/``A-drives/A6-3-capture-sets.txt`): Transmission's `${USERDATA_PATH}/media/transmission/watch` bind is not listed in its `backup:` block, so it is classed MANDATORY and goes into the OFF-SITE copy (a writable unlisted bind defaults to mandatory); and qBittorrent's `media/torrents` and Transmission's `media/transmission` are `optional`, so on a box with a second drive every download is mirrored there nightly (Tier 2 = mandatory + optional) — it can fill the backup drive with films. | **NARROWED** — filed 2026-10-14; the `watch` half FIXED 2026-10-14 (declared optional, app-catalog `bc54c01`); open: the `optional` download folders still get a nightly Tier-2 mirror on a box with a second drive | — | Declare the media binds `excluded` (the watch folder too) in the live-fix task; `15` §7 states the rule for the per-drive bind | CC | | **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC | @@ -184,6 +183,7 @@ stopping line that lies. | **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.** **2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC | | **R-1087** | Backup & restore | P4 | **Whole-box copies left on the OLD local target are never pruned after the target moves.** MEASURED 2026-10-15 (storage-identity build B space check): the N100's host root (`local`) holds three copies from 2026-07-28 (5.5–5.7 GB each, ≈ 16.8 GB); its agent's `local_backup_target` is now `felhom-backup` (the USB disk, retention 3), and retention prunes only the current target. On a small-root box such leftovers would starve the local copies (R-548). Evidence: `audits/storage-identity-buildB-2026-10-15/space/felhom-pve-9201.txt`. | OPEN | — | Decide whether the agent prunes its own `vzdump-lxc--*` files on a former target (after a newer copy exists on the new one), or the target move asks; until then, the N100's three July files can be removed by hand with the operator's word. | CC | | **R-1162** | Backup & restore | P3 | **ep0's whole-guest off-site prune was a count (keep-last 2), so a compromised controller could push out the pre-compromise copy, slowly.** Operator decision (STATUS D18, the brief's D1 = A, 2026-10-19 08:24): one copy per ISO week, 2 weeks. Before: PBS's own dry run and an offline model of its selection both showed `keep-weekly 2` removing nothing on demo-hp (2026-10-08, 10-15 kept) or demo-felhom (10-06, 10-13 kept); nothing deleted by the dry runs (8 snapshots before and after). On the operator's „yes": `prune-demo-hp` and `prune-demo-felhom` changed to `keep-weekly 2` (schedule 03:30 unchanged), read back 07:49 UTC. DooPlex's copy of ep0 already keeps `keep-weekly 8` (decision 71) — unchanged. Households without a job: R-1165. | NARROWED 2026-10-19 — changed and read back; the first scheduled run is 2026-10-20 03:30 UTC | `audits/day-2026-10-19b/P1/` | Read the 2026-10-20 03:30 UTC prune task on ep0 (`proxmox-backup-manager task list`, `prunejob` for both namespaces: OK, `keep-weekly 2`, two snapshots each), then close | CC | +| **R-1169** | Backup & restore | P3 | **After a failed off-site restore that is held (`restore_mixed`), the restore page says the database is back as it was and the app's files and other data come from the backup — measured false on both halves.** 2026-10-19 on 9202 (R-893's measurement): a table only the backup had stayed in the database (37 tables against 36), and Rallly had no files or other volumes from the backup. The sentence (`err.backup.db_restore_failed_held_mixed`) should say only what is certain: the restore stopped half-way, the app is held stopped, nothing is lost yet, support will help. A text change in the controller (HU + EN). | OPEN — filed 2026-10-19 | `audits/night-2026-10-19/P3/README.md` | Next controller release: the sentence rewritten in both bundles, a fixture for the held-mixed page | CC | ## Storage & devices — 9 rows (P3 3, P4 6) @@ -238,17 +238,17 @@ stopping line that lies. | **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement | — | — | CC | | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC | | **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC | -| **R-1010** | Box system & updates | P3 | **The agent's stale-lock janitor cannot remove a half-removed `vzdump` snapshot: Proxmox's delete fails on the first missing volume („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump") and the janitor retries the same call every 10 min forever, leaving the other volume's snapshot and the snapshot entry.** Seen 2026-10-14 after a backup ran out of space on a full host root (LVM could not write its metadata; `audits/power-cut-fixes-2026-10-14/README.md` C4). Fixed by hand with `pct delsnapshot 9201 vzdump --force`. | **READY** — filed 2026-10-14 | — | CC: on that error, retry once with `--force` (a sudoers line or the API's `force=1`), with a test that replays the error text; next agent release | CC | +| **R-1010** | Box system & updates | P3 | **The agent's stale-lock janitor cannot remove a half-removed `vzdump` snapshot: Proxmox's delete fails on the first missing volume („Failed to find logical volume pve/snap_vm-9201-disk-0_vzdump") and the janitor retries the same call every 10 min forever, leaving the other volume's snapshot and the snapshot entry.** Seen 2026-10-14 after a backup ran out of space on a full host root (LVM could not write its metadata; `audits/power-cut-fixes-2026-10-14/README.md` C4). Fixed by hand with `pct delsnapshot 9201 vzdump --force`. | **READY — fixed in agent `56aabc7`, committed, NOT released** (the morning question): on „Failed to find logical volume …_vzdump" the janitor retries once with the API's `force=1` (`pvesh usage … -command delete` lists `--force`; same token privilege, no sudoers line); any other error unchanged; never a second force. Tests red-proved. Not exercised live (it needs a half-removed snapshot on `pve/data`). Close after the release. **READY** — filed 2026-10-14 | — | CC: on that error, retry once with `--force` (a sudoers line or the API's `force=1`), with a test that replays the error text; next agent release | CC | | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC | -## Monitoring & notifications — 11 rows (P2 1, P3 7, P4 3) +## Monitoring & notifications — 12 rows (P2 1, P3 8, P4 3) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1070** | Monitoring & notifications | P3 | **A box with no Cloudflare tunnel gets an ERROR after its host OS update: „NOT healthy … (the tunnel is unknown)”.** MEASURED 2026-10-15 16:30 on Tester4-68d1d8 (throwaway customer, placeholder domain `tester4.test`, so no tunnel token): hub event `os_update_health_failed` (error) „OS update (host) on Tester4-68d1d8: NOT healthy after 106 package(s) were installed (the tunnel is unknown). Nothing was undone automatically; put a host package back by hand…”. The box was healthy (dashboard 200, apps installing). A tunnel that never existed is read as a failed health check. Would hit every test customer and any box whose tunnel is not set yet. Evidence: `audits/omv-vm-2026-10-15/C-hub-events-Tester4.txt`. | OPEN | — | Find the health verdict's tunnel input (hub os-update health); treat „no tunnel configured” as not-applicable, not as unhealthy; red-proof with a no-tunnel fixture. | CC | +| **R-1070** | Monitoring & notifications | P3 | **A box with no Cloudflare tunnel gets an ERROR after its host OS update: „NOT healthy … (the tunnel is unknown)”.** MEASURED 2026-10-15 16:30 on Tester4-68d1d8 (throwaway customer, placeholder domain `tester4.test`, so no tunnel token): hub event `os_update_health_failed` (error) „OS update (host) on Tester4-68d1d8: NOT healthy after 106 package(s) were installed (the tunnel is unknown). Nothing was undone automatically; put a host package back by hand…”. The box was healthy (dashboard 200, apps installing). A tunnel that never existed is read as a failed health check. Would hit every test customer and any box whose tunnel is not set yet. Evidence: `audits/omv-vm-2026-10-15/C-hub-events-Tester4.txt`. | **READY — fixed in agent `56aabc7`, committed, NOT released:** the cause measured — Docker 29 prints „error: no such object" in lower case, so the probe's case-sensitive match read a box with no cloudflared container as `unknown`, and the host rule failed it. New state `absent`; the host rule passes it only when the reading from before the step also had no cloudflared container; `not_running`/`unknown` still fail. The hub never alarms on `absent` and shows „none set up" (hub 0.162.0). Tests red-proved. Close after the agent release, on Tester4's next host OS update. Hub side of a VANISHED tunnel: R-1170. OPEN | — | Find the health verdict's tunnel input (hub os-update health); treat „no tunnel configured” as not-applicable, not as unhealthy; red-proof with a no-tunnel fixture. | CC | | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC | -| **R-1001** | Monitoring & notifications | P3 | **Every hub restart re-mails the operator about a box that is already known to be down.** MEASURED 2026-10-13 18:33 UTC (hub v0.149.0 rollout, R-998 session): Tester 2, silent 216 h, got a fresh `host_down` and `restore_test_stale` mail. `monitor/host_staleness.go` leaves a down host unseeded on start so the first Check emits (F2), and its comment says „the dispatcher's 1h cooldown dedups the re-emit" — but host liveness events bypass that cooldown (`dispatcher.go` `nodeLivenessEvents`, 5-min dedupe only); `restore_test_stale` keeps its edge state in memory. The comment states an invariant the code does not provide. | **READY** — filed 2026-10-13 | — | Seed a host silently when a `host_down`/`host_stale` event for it is newer than its last report; the same for the restore-test edge; a test that restarts the checker | CC | +| **R-1001** | Monitoring & notifications | P3 | **Every hub restart re-mails the operator about a box that is already known to be down.** MEASURED 2026-10-13 18:33 UTC (hub v0.149.0 rollout, R-998 session): Tester 2, silent 216 h, got a fresh `host_down` and `restore_test_stale` mail. `monitor/host_staleness.go` leaves a down host unseeded on start so the first Check emits (F2), and its comment says „the dispatcher's 1h cooldown dedups the re-emit" — but host liveness events bypass that cooldown (`dispatcher.go` `nodeLivenessEvents`, 5-min dedupe only); `restore_test_stale` keeps its edge state in memory. The comment states an invariant the code does not provide. | **READY — built in hub 0.162.0 (image `sha256:69285e64…`, from `d92a712e`), NOT deployed** (the night brief: no hub deploy at night; STATUS's morning question). Tests red-proved: `TestHostStaleness_R1001_RestartDoesNotReMail`, `…NewSilenceStillMails`, `TestRestoreTestStale_R1001_RestartDoesNotReMail` (each restart a new checker over the same DB). Close after the deploy, on the first restart read-back. **READY** — filed 2026-10-13 | — | Seed a host silently when a `host_down`/`host_stale` event for it is newer than its last report; the same for the restore-test edge; a test that restarts the checker | CC | | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | | **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** **REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc//fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC | | **R-388** | Monitoring & notifications | P3 | **PRODUCT DECISION (not a defect): the customer notification model is the wrong shape, and the settings page grows by one toggle per detector.** The operator's framing, recorded verbatim 2026-08-23: *"A customer should be notified only about things they can act on or are responsible for — the drive they unplugged, the storage they filled. **A failed backup is our incident, not theirs.** The intended shape is that we detect it, we tell them we noticed and are dealing with it, and they are not handed an error they cannot solve. The subscription should feel like being looked after, not like being on call."* Today's page is the opposite shape — one switch per detector, and it **grew from 12 to 15 in a single session** (one new alarm plus two compound toggles split into four). That growth is the argument, not an aside: a page that grows per detector keeps asking a household to make engineering decisions. | **READY** — **2026-10-12 evening: slice 1 delivered** (operator Q1/Q2 „as picked", `09` §3 decisions 234–235): the page shows three group switches; the per-detector switches and the cooldown sit under „Részletek"; an „egyedi" group is saved back unchanged; the default list fills groups 1 and 3 (households with no saved list), group 2 stays „egyedi" (`app_start_failed` off, R-329). Tests red-proved. controller 0.309.0, delivered 2026-10-12 18:08 to demo-hp, demo-felhom and Tester 1 (`controller-0.309.0-delivery.txt`). **READY** — **2026-10-12: the design is written** (`08` §8.1): three group switches („A gépem és a meghajtóim", „Az alkalmazásaim", „Amit mi intézünk") mapped onto today's 18 detectors; option A picked; slice 1 changes no wire. Two questions for the operator (group 3 default; the cooldown field). **READY** — operator ruling 2026-10-12 (S5, `09` §3 decision 200): design now (one page: options, a pick, the first slice), build in slices later. **OPEN — DIRECTION, operator's call** | a decision on scope; nothing here is a bug | Slice 2 (hub): group 3's mails say „we noticed X and are handling it" instead of the raw error | CC (+ operator: two answers) | @@ -257,14 +257,14 @@ stopping line that lies. | **R-285** | Monitoring & notifications | P4 | **A planned, supervised reinstall pages the operator as if the machine had died — there is no notion of expected downtime anywhere.** During the 2026-08-09 rehearsal the hub sent, all `status: sent` to the operator channel: `host_stale` 08:58 UTC, `node_stale` 09:00, **`host_down` 09:28 (error)**, **`node_down` 09:30 (error)**, `host_leaf_changed` 09:31, `host_recovered` 09:31, `node_recovered` 09:34, `offsite_delivery_stuck` 09:34 — eight operator mails for work that was deliberate, attended and announced. **This is the OPPOSITE gap from the one R-281 filed:** the alarms are not missing, they are indiscriminate. `host_stale` at 30 min and `host_down` at 60 min (`monitor/host_staleness.go:22-23`, `downAfter = 2 * threshold`) cannot distinguish a wiped-on-purpose box from a dead one, and `host_leaf_changed` firing on a reinstall is correct-but-expected. **Note the interaction with the mute used on 2026-08-09 evening:** blocking a customer silences everything, so today the only two settings are *page me for planned work* and *tell me nothing at all*. **What is owed is a middle:** a maintenance window, or an operator-set expected-downtime flag, that suppresses staleness and leaf-change while leaving genuine faults audible | **READY (M) — NEW 2026-08-09** | — | The evidence is the operator's mailbox plus `events`/`notification_log` for 2026-08-09 | CC | | **R-1005** | Monitoring & notifications | P4 | **A recovery mail says how long the box was „stale", not how long it was gone: after a 51-minute power cut demo-hp's `host_recovered` read „was stale for 8m" and `node_recovered` „23m".** An operator reading only the recovery mail under-reads the outage (`audits/night-power-2026-10-13/` R6). Same kind, the other way: the off-site mail of 2026-10-14 02:18 was titled „3 of 3 apps failed to back up (offsite run)" while 11 other apps were backed up in that run — only the 3 flagged apps were counted. | **READY** — filed 2026-10-14 | — | CC: add the time since the last report before the gap to both recovery messages, and count every app of the run in the off-site subject, with a test that asserts each text; next hub release | CC | | **R-1049** | Monitoring & notifications | P3 | **A pulled drive mails the operator „unexpectedly disconnected” (error), and nothing ever mails that it came back.** MEASURED 2026-10-14 night (`audits/media-drive-events-2026-10-14/K8-hub-mails.txt`, the hub DB with its WAL from the running pod, and the mailbox as the second channel): two pulls → two `storage_disconnected` mails; the hub RECEIVED both `storage_reconnected` and both `health_recovered` (severity info) and logged no notification for any of them — they are not in `recoveredPairedDownTypes` (`hub/internal/notify/dispatcher.go`), so the severity gate drops them. The R-97a shape: told it broke, never told it healed — the operator cannot tell a returned drive from one still gone without opening the hub. **BUILT 2026-10-15 — hub v0.151.0 (`12552aca`): both pairs in `recoveredPairedDownTypes`, cooldowns ended by the next loss (`TestR1049_DriveBackMailsBothEdges`, red-proved). Not deployed: the operator's „yes” first (decision 162); the image build waits for it too.** | **WAITING-ON-OPERATOR** — hub v0.151.0 code pushed, not built, not deployed | the operator's „yes” to the hub deploy | On the „yes”: build + deploy hub v0.151.0; read a „came back” mail on the next pull | CC | +| **R-1170** | Monitoring & notifications | P3 | **A box whose tunnel container disappears raises no alarm, even when its tunnel is configured.** The hub alarms on `not_running` only (`checkTunnel`). A guest with no cloudflared container read `unknown` since Docker 29 (lower-case „no such object", R-1070) and reads `absent` with the next agent release — neither alarms. On a box whose customer has a tunnel token, a missing container means the box is unreachable from outside. Found 2026-10-19 while fixing R-1070. | OPEN — filed 2026-10-19 | `hub/internal/api/handler.go` `checkTunnel` | Hub: treat `absent` as down when the customer's config carries a tunnel token (two reports in a row, as today); a test | CC | -## Hub & operator — 13 rows (P3 6, P4 7) +## Hub & operator — 12 rows (P3 5, P4 7) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-975** | Hub & operator | P3 | **The hub database keeps every customer's tunnel token, Cloudflare API token and git token unsealed in one column, so anyone with a copy of the database (or its nightly snapshots) gets them.** R-879 sealed the other secret columns but not this one. Fix: seal this column the same way. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Seal with a migration and a test | CC | -| **R-953** | Hub & operator | P3 | **A full ArgoCD sync of the `felhom` app with prune would DELETE three live Secrets and so break the hub and Umami.** Measured 2026-10-12: `Secret/gitea-creds` (read by `Deployment/hub`), `Secret/umami-config` (read by `Deployment/umami` and `Deployment/umami-db`) and `Secret/healthchecks-config` (read by nothing) are live but no longer in git since R-925 de-gitted `felhom.secret.yaml`; ArgoCD marks all three `requiresPruning`. `Deployment/umami` is also OutOfSync (not pruning). Today only the build-deploy skill's „sync only the hub" line prevents it. Fix: mark the three Secrets `argocd.argoproj.io/compare-options: IgnoreExtraneous` (or move them out of the app's scope), decide whether `healthchecks-config` is still needed, and bring `Deployment/umami` back in sync. | OPEN — filed 2026-10-12 | — | Annotate or move the three Secrets; then a whole-app diff shows nothing to prune | CC | | **R-31** | Hub & operator | P3 | **[P2-HIGH] Offsite provisioning is synchronous with no status affordance.** Save runs the Hetzner sync in-request, so the request can hit the nginx 504 **while succeeding server-side**: the operator cannot tell failed from slow, and a retry races the first attempt. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: operator-only; a known workaround (click once, wait, verify) exists.** **2026-10-06 night: the race half fixed on felhom.eu main (hub, unreleased):** a second Save while the first still provisions is refused with 409 („already running — wait about a minute, then reload; do not save again"); nothing saved, nothing created; per customer, in memory. `TestProvision_R31_*`, red-proof `audits/night-burndown-2026-10-06/hub/R-31-red.txt`. LEFT: the async save with a status card (the escrow-card idiom). | — | Direction: make it async + a status card, reusing the proven **awaiting-card/poll idiom** (v0.138.0 escrow card). **Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click.** | CC | | **R-882** | Hub & operator | P3 | **Longhorn on DooPlex could not grow a volume online: its `instance-manager` (116 days up) called a host process that no longer existed** — `nsenter: cannot open /host/proc/196610/ns/mnt` on every expansion retry, and an offline growth was blocked by the expansion's own attachment ticket (found 2026-10-05 growing `hub-data` to 2 Gi). A restart of the instance-manager (operator-approved) fixed it: 77/77 volumes back `attached/healthy` in 110 s. **Why the cached PID went stale was not established** (likely a containerd/k3s or iscsid restart after the instance-manager started), so it will recur after the next such restart and stay invisible until a volume needs to grow. `audits/hub-db-offsite-2026-10-05/partA/step1-*.txt` | **OPEN** | — | Find which host process the PID was and whether Longhorn 1.10.x re-resolves it; until then, before growing any volume, check the instance-manager's age against the last k3s/containerd restart | operator | | **R-883** | Hub & operator | P3 | **8 DooPlex workloads run an image by a moving tag (`:latest` or none), so any pod restart is a silent upgrade.** Measured 2026-10-05: the Longhorn restart restarted zipline on `ghcr.io/diced/zipline:latest` (pull Always), which pulled 4.8.0; 4.8.0 refused its database (`cannot safely migrate from prisma to drizzle: expected migration 20260508022000 … was not applied`) and crash-looped. Fixed for zipline by pinning `4.7.0` (homelab-manifests `90f60e4`, `4c8ec7a`; 4.7.0 applied the four missing migrations; a dump from before is kept out-of-band). The other 7 were counted, not named or changed. **Checked from source 2026-10-05 (burn-down round 2):** homelab-manifests@87dfc29 still has moving tags (grep image lines without a numeric tag): admin-system/toolbox.yaml:12 nicolaka/netshoot:latest (a bare Pod); calibre-system/cwa.yaml:826 calibre-web-automated:dev; outline-system/outline.yaml:270 minio/minio:latest; tandoor-system/recipe-importer.yaml:26 gitea.dooplex.hu/admin/recipe-importer:latest (pull Always :27); adventurelog-system/adventurelog.yaml:100 and :256 adventurelog-backend/frontend:latest (pull Always :101,:257); jarrs-system/jarr- | **OPEN** | — | List the 7 (`kubectl get deploy,sts -A` images without a fixed tag), pin each to the running version, and let Renovate move them | operator | @@ -273,7 +273,7 @@ stopping line that lies. | **R-814** | Hub & operator | P4 | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | **VERIFY** (2026-10-03 triage: a July watch row with no id; given R-814. WAITING-ON-OPERATOR — no record found that the box was deleted.) — WAITING-ON-OPERATOR | operator console | Delete the box | operator | | **R-844** | Hub & operator | P4 | **The household's OS-update line exists only on the hub's customer timeline.** 2026-10-04: the box itself has no event surface for agent results (the controller UI shows no timeline), so `os_update_applied` is a hub customer event (info: recorded, never mailed). Its stored text is the hub's English sentence; the hu/en bundle text (`mail.event.os_update_applied`) is used only if it is ever mailed. Fix direction: a controller-side line (the controller already polls the agent's local API) when the box gets a household timeline. `audits/os-guest-lane-2026-10-04/partG/hub-customer-timeline-demo-hp.txt` | **READY — owner: CC** **2026-10-05 (burn-down night): NEEDS A DESIGN** — a household timeline on the box does not exist yet. | — | — | CC | | **R-1002** | Hub & operator | P4 | **A box the household switched off reads ONLINE on the hub's hosts page for its first 45 minutes** (seen live 2026-10-13, R-998 proof, `audits/poweroff-2026-10-13/B-hub-watch.log`: the „Power: Kikapcsolva…" line at 18:53, the KIKAPCSOLVA badge only at 19:38, when the host report turned stale). The customer page read „warn" in part of that gap (seen, not captured). No alarm and no mail fired; the operator page is only slow to say it. | **READY** — filed 2026-10-13 | — | CC: let `hostDisplayStatus` read `deliberate_off_at` before staleness, with a render test per state; next hub release | CC | -| **R-1011** | Hub & operator | P3 | **The Day-0 „Agent" list offers step-bundle packages (`0.156.1-step1`, `0.152.0-step1` …) as agent versions; one has no agent binary, so the hub shows the bundle JSON's sha (`FileSHA256` falls back to the first file) and a vouch would point fresh installs at a package with no binary.** Seen 2026-10-14 (`/configuration`: two „0.156.1" entries, `ac83ea41…` is the step bundle). | **READY** — filed 2026-10-14 | — | CC: list only versions whose package holds the `felhom-agent` file (refuse a missing preferred file instead of falling back), with a test; next hub release | CC | +| **R-1011** | Hub & operator | P3 | **The Day-0 „Agent" list offers step-bundle packages (`0.156.1-step1`, `0.152.0-step1` …) as agent versions; one has no agent binary, so the hub shows the bundle JSON's sha (`FileSHA256` falls back to the first file) and a vouch would point fresh installs at a package with no binary.** Seen 2026-10-14 (`/configuration`: two „0.156.1" entries, `ac83ea41…` is the step bundle). | **READY — built in hub 0.162.0, NOT deployed:** the sha lookup asks for exactly `felhom-agent` / `golden.tar.zst` (no fallback); step bundles leave the list; a vouch of one is refused (`artifact_file_missing`). Registry read 2026-10-19: 6 step-bundle versions, every release holds `felhom-agent`. Tests red-proved. Close after the deploy, on the Configuration page read. **READY** — filed 2026-10-14 | — | CC: list only versions whose package holds the `felhom-agent` file (refuse a missing preferred file instead of falling back), with a test; next hub release | CC | | **R-1056** | Hub & operator | P4 | **Five operator routes fit no token group, so they are session only until the operator places them:** apps reset-telemetry and dismiss-issues, the Configuration asset re-seed, the off-site WireGuard endpoint save, the off-site abandon-cancel (`05` §17.3). Nothing needs them by token today. | WAITING-ON-OPERATOR — filed 2026-10-15 by the token build | operator decision | Place each in a group, or leave session only | operator | | **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator | @@ -301,12 +301,11 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1083** | Process & tooling | P3 | **The scratch box 9202 cannot use any feature that needs the agent: its `bootstrap.json` has no `local_api` block, so its controller never gets an agent client.** MEASURED 2026-10-15 (storage-identity build A): 9202 put on controller 0.317.0 by hand to show that a NAS share no longer hides the one-copy notice (R-1075); its `bootstrap.json` (read on the box) holds no `local_api`, so the controller has no agent to ask and keeps the old `st_dev` rule — the check could not tell old from new — 9202 was put back on 0.314.0. Every agent-coupled feature (drives, same-disk, format, network storage) is untestable on the scratch box today, and the NAS leg of build A is proven by a red-proofed unit test only. Evidence: `audits/storage-identity-buildA-2026-10-15/9202-upgrade.txt`. | OPEN | — | Give 9202 a local-API token the normal way (the agent mints it for its guest), or record in `runbooks/target-selection.md` that 9202 is controller-only; then show the NAS notice live. | CC | +| **R-1083** | Process & tooling | P3 | **The scratch box 9202 cannot use any feature that needs the agent: its `bootstrap.json` has no `local_api` block, so its controller never gets an agent client.** MEASURED 2026-10-15 (storage-identity build A): 9202 put on controller 0.317.0 by hand to show that a NAS share no longer hides the one-copy notice (R-1075); its `bootstrap.json` (read on the box) holds no `local_api`, so the controller has no agent to ask and keeps the old `st_dev` rule — the check could not tell old from new — 9202 was put back on 0.314.0. Every agent-coupled feature (drives, same-disk, format, network storage) is untestable on the scratch box today, and the NAS leg of build A is proven by a red-proofed unit test only. Evidence: `audits/storage-identity-buildA-2026-10-15/9202-upgrade.txt`. | **OPEN — the exact gap, read 2026-10-19 (no agent change tonight):** the agent mints a per-guest local-API token in ONE place, `--selftest=provision`'s back half (`cmd/felhom-agent/main.go` `bh.Provision` after `engine.RunBringUp`), which only runs for a guest it brings up from a golden archive in the same call. There is no product way to mint for an EXISTING guest (9202) — no subcommand, no local-API route; the token store `/var/lib/felhom-agent/local-tokens.log` is written only by that path. Options: an agent subcommand `--mint-guest-token ` (owner-only, writes bootstrap's `local_api`), or re-provision 9202 through `selftest=provision` (destroys and rebuilds the scratch guest — not done; rule 6). Until then 9202 is controller-only. OPEN | — | Give 9202 a local-API token the normal way (the agent mints it for its guest), or record in `runbooks/target-selection.md` that 9202 is controller-only; then show the NAS notice live. | CC | | **R-1081** | Process & tooling | P4 | **Two published controller tags hold content that is not what their number says: `0.315.0` and `0.316.3`.** Both were caused by CC on 2026-10-15, and both by the same shape — a guard that was not on the path the work takes. **`0.316.3`:** the build directory's `build.sh` is a symlink into the SHARED clone's working copy, which predated the `FELHOM_REPO_DIR` override, so the override was silently ignored and the build copied that tree — which held 31 uncommitted files of another session's physical-disk storage work. The banner printed `Commit: 0225255` with no `Repo:` line. **Nobody deployed it; that was luck, not a control.** Fixed at the root in controller v0.316.4: `build.sh` now refuses `--push` from a tree that is dirty, not at `origin/main`, or not a git checkout, with four test cases red-proved. **Neither tag is re-pushed or deleted** — a version number is pushed once, and that applies to numbers CC spoiled. **No box runs either:** demo-hp 9201 is on 0.317.0, the floor is 0.314.0, and the button offers the highest tag. **`0.315.0`:** Caused by CC on 2026-10-15 while proving R-1080's own fix: the guard was added to `controller/build.sh` in the repo, but the documented release command runs `$FELHOM_ROOT/build/felhom-controller/build.sh`, which was a **separate copy taken 2026-07-18** and had no guard. Running the refusal against a real existing tag therefore rebuilt and re-pushed 0.315.0 — a fourth time — with v0.316.0's code under it (digest `8887343011d6` → `b809201841f7`). **Fixed at the root in the same session:** the build directory's script is now a symlink to the repo's, the old copy kept beside it as `build.sh.stale-copy-2026-07-18`, and the refusal was then proven to fire from the documented path. **No box is affected:** nothing runs 0.315.0 (demo-hp 9201 is on 0.316.2), the floor is 0.314.0, and the household button offers the highest tag. What remains is two tags that must never be deployed and are no longer reproducible from the registry. | **OPEN** — filed 2026-10-15; nothing to do unless someone needs one of those two images, and then the answer is the current release | — | Leave them. If an audit ever needs the real 0.315.0 or 0.316.3, build it from the matching commit under a NEW number and say so | CC | | **R-992** | Process & tooling | P3 | **The drill-catalog route that `09` §6.5 documents cannot be followed: `admin/app-catalog-drill` is PRIVATE and no box has catalog credentials.** Measured 2026-10-12 while publishing the Usenet pair: the box's clone fails with `fatal: could not read Username for 'https://gitea.dooplex.hu'` (exit 128), the Gitea API reports the drill repo `private=true` and the live catalog `private=false`, and NEITHER the current controller.yaml on 9202 NOR any of its 30 saved copies has ever carried `git.username` or `git.token` (the controller supports both — `internal/config/config.go` GitConfig). The route worked for the campaigns of 2026-10-10 and the torrent pair earlier the same day, so the repository's visibility changed between then and now. **Worked around, not fixed:** that campaign staged its two templates in the box's own catalog cache (`audits/usenet-2026-10-12/tools/stage_in_cache.py`), which is where the sync copies a catalog change anyway, with the sync interval raised to 12h for the duration and rescan used in place of sync. The real fix is an operator decision: either a read-only catalog credential on the scratch box (a secret on a Tier-0 guest) or making the drill repository readable. Until then every new-app campaign hits this. | WAITING-ON-OPERATOR | the operator's choice between a box credential and a readable drill repo | Decide; then either set `git.username`/`git.token` on 9202 from the credentials store, or flip the drill repo, and correct `09` §6.5 either way | operator | | **R-1034** | Process & tooling | P3 | **Gate 17 checks the FAQ JSON-LD on two pages only, by a hard-coded list — a FAQPage anywhere else is unchecked.** `scripts/site_gates.py` gate 17 loops `for p in ("gyik.html", os.path.join("en", "faq.html"))`, so its „JSON-LD equals the visible questions and answers” guarantee stops at those two files. **Since 2026-10-14 two more pages carry a FAQPage block the gate never reads:** `/sajat-felho-tarhely` and `/en/own-cloud-storage`. Their JSON-LD was *derived* from the visible headings and paragraphs by a script and then verified verbatim against the rendered text, so it is correct today — but nothing stops the next copy edit moving the visible text and leaving the structured data behind, which is exactly the failure gate 17 exists to catch, and the one Google penalises. **Why it is not a one-line fix:** gate 17's extractor is written for the FAQ pages' accordion markup (`