diff --git a/CONTEXT.md b/CONTEXT.md index 378f0838..e20df14d 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,8 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-11-02/03 (night, unattended) — the security watch and the new notices; no release.** (1) **R-1334's fix held:** 0 Docker Hub 429 on five nights. A streak is per SOURCE, so seven pins that can never be read (`gitea.dooplex.hu/admin/*` ×6, `minio/minio`) held `registry:docker.io` and `runtime` open: `security_watch.py` `STATED_UNREADABLE` + private registry = GAP with a reason; `Transient` retried on every source (budget 30); `FAILED-SOURCE` lines; R-1310's rules (`PACKAGE_NOT_OURS`, `WORD_ONLY`, one entry per GHSA, date tags). (2) **Traefik v3.7.16 has GHSA-xv5x (high):** pin `c4b6844`, unreleased (R-1354). (3) **Catalog live `c760fca`:** gitea 28.3.0, uptime-kuma 2.5.9 (crosses 2.5.8's TLS removal on TCP monitors — decided by CC unattended, operator may reverse), grafana 13.2.5, actualbudget 26.11.0, n8n 2.46.4, vaultwarden 1.37.5-alpine. (4) **Commit only:** controller `c4b6844` + `3764445` (R-1347, R-1348, R-1339, R-1351, R-1323), agent `50eeccc` (R-1302), hub `2f7e836f` (R-1352). (5) **Walks:** 8 claims moved to walked (31 → 23 not walked). R-1361: an event sent while the hub is away is lost after 9 s. R-1362, R-1364 filed. Report: `REPORT-security-2026-11-02.md`; evidence `documentation/audits/security-2026-11-02/`. + > **2026-10-30 (day, attended) — RELEASED: hub v0.178.0 (1 GiB), agent v0.171.0 (VOUCHED 10:47:36 CET); the site on > Cloudflare (D35, switched 10:02:35 UTC).** (1) **R-1344 decided from the measurement** (4 733 probes through the night > chain, slowest 303 ms, no failure): the agent restarts a running-but-unanswering controller at 10 unhealthy sweeps diff --git a/REPORT-security-2026-11-02.md b/REPORT-security-2026-11-02.md new file mode 100644 index 00000000..b14bdada --- /dev/null +++ b/REPORT-security-2026-11-02.md @@ -0,0 +1,55 @@ +# REPORT — security night 2026-11-02/03 (unattended): the watch, the new notices, six small fixes, unproven claims + +Read first: `08-alarm-ladder.md` §11, `09-update-architecture.md` decision 255, `07-backup-architecture.md`, `03-host-agent.md`, `05-hub-architecture.md`, `00-capability-map.md`. + +## Decisions taken by CC (operator may reverse) + +- **Uptime Kuma's step crosses 2.5.8**, which the 10-27 batch refused: 2.5.8 turns a TCP-port monitor with TLS into a plain TCP one (upstream: „never worked as expected"). The fix (admin-only code execution; unauthenticated RSS cache poisoning) outweighs it, and no box runs Kuma. +- **R-1339's written fix was not built:** a sub-folder „not skipped" would write a `userdata/` inside the app's folder. The behaviour stays; only the false line goes. +- **A brief's expectation that turned out false, said here:** the watch was not blind to Docker Hub, and R-1334's fix had held. Traefik's line was no false alarm. + +## Six lines + +1. **The watch was not blind; it could not be heard.** A streak is per source, so seven pins that can never be read (six private images, `minio/minio`) held `registry:docker.io` and `runtime` open for a week: no real outage could have mailed. 0 Docker Hub 429 on five nights. Fixed: those seven are a stated GAP, timeouts are retried. Proven: run 2550 (runner), dry 2555 (runner-2), night 2565 (runner-2): `FAILED-SOURCES 0`, memory `failures {}`. +2. **The notices:** Vaultwarden, Uptime Kuma, n8n, PrivateBin and Gitea are real; **Traefik is real too, high, on every box** (GHSA-xv5x, hidden behind a word-only line). Live: gitea 28.3.0, uptime-kuma 2.5.9, grafana 13.2.5, actualbudget 26.11.0, n8n 2.46.4, vaultwarden 1.37.5-alpine. PrivateBin has no 2.1.1 image. +3. **The six waiters:** ActualBudget fixed (live). Paperless and SparkyFitness have a fixed release on a new main version (proven on the drill). cal.com, Calibre-Web, RomM: none. +4. **The small fixes:** 7 done with red first. Live-proven 3 (R-1347, R-1348, R-1339); R-1351 test only (the race fired on neither image in six boots); R-1323, R-1352, R-1302 test only, by the brief. +5. **RomM's night leg:** done, 2026-10-28 03:22, 88 s (closed; demo-hp's library is small). +6. **Claims:** 8 walked (31 → 23 not walked); 2 walked and failed in part: the hub down loses events (R-1361), the LAN name has gaps (R-1364). + +## The morning release list, and the one decision + +STATUS has both. First the golden (R-468). Controller v0.350.0: Traefik v3.7.17 first, R-1347, R-1348, R-1339, R-1351, R-1323. Agent v0.172.0 + vouch: R-1302. Hub v0.179.0: R-1352. Catalog, drill only: Paperless 3.4.1 and n8n 3.0.1 (bench, box, undo), SparkyFitness v1.11.0 (the same, behind the licence R-784), Mealie v3.30.0 and Termix 26.11.0 (bench, box). Decision D66 (R-1365): should a source that stays down mail again every 7 days? Pick: yes. + +## Commits and CI (by head_sha, the jobs endpoint) + +| Repo | Commit | CI job | What | +|---|---|---|---| +| felhom.eu | `4afa8c3c` | 2548 green | the watch: stated GAP, retries, R-1310's rules | +| felhom.eu | `2f7e836f` | 2553 green | hub R-1352, commit only | +| felhom.eu | `d02c0c2c`, `17820abb` | 2557, 2561 green | evidence | +| controller | `c4b6844`, `3764445`, `13e34d4` | 2551, 2560, 2562 green | Traefik pin, five fixes, commit only | +| agent | `50eeccc`, `ecbb0cc` | 2552, 2563 green | R-1302, commit only | +| catalog | **`c760fca`, the one live push** | 2559 green | six `security_fix` steps; negative control first (`walks/_negative-control/`: `failed`) | + +## Rows + +Before 228, after 235. **Closed (5):** R-1303, R-1310, R-1324, R-1334, R-1336. **Opened (12):** R-1354 Traefik, R-1355 DooPlex's notices, R-1356 n8n 3, R-1357 PrivateBin, R-1358 cloudflared, R-1359 Mealie + Termix, R-1360 Meilisearch, R-1361 events lost, R-1362 Backups page, R-1363 raw JSON in a mail, R-1364 LAN DNS, R-1365 D66. + +## Observations + +- **FILED: R-1361, R-1362, R-1363, R-1364** (the walks); **R-1354–R-1360** (the triage). +- **Noted in R-468: the golden gate went red at midnight:** the Sunday bake of 11-01 was missed (in no brief), golden 0.344.0 is 8 days old. Baking is not this night's work; no waiver written. A code push to felhom.eu is refused until a bake and the vouch. First on the list. +- **NOT-A-FINDING: the watch's 03:20 run** is `gates`, not the watch. +- **NOT-A-FINDING: 9202's „Tárhely (…)" sources with no mount, and its refused removal of a per-app `HDD_PATH`:** legacy of its own old installs (R-839 refuses that shape); no other box holds one. +- **NOT-A-FINDING: `CT990000 stopped create` on Tester 1:** the restore test at that moment; it passed at 22:45. +- **NOT-A-FINDING: DooPlex restarted 2026-11-01 16:11 CET:** two boots, read only; its only effect here is that R-1353's log covers 31 h. +- **NOT-A-FINDING: the watch's „0 new, 63 reminders" mail of 11-03:** the 7-day reminder, as designed. +- **Fixed without a row (the size rule):** the Vaultwarden fixture (1.37's register route), the n8n fixture (`/healthz` before the routes), the box-admin guard (the drill mirror), the Actual fixture's log, the undo tool (`tools/undo_proof3.py`: a probe with a `container:` line, services read by label). +- **A mistake, said plainly:** one dead session token of a throwaway Actual reached the evidence in `d02c0c2c`; redacted in `03a71b68`; the app was already removed; it stays in git history. Dead, and the fixture no longer logs it. + +## Teardown + +Machine: 9202 back on its image, its yaml byte-identical, on the live catalog; bench 9401 stopped; Tester 1's routes and firewall as found, address empty. Host: the drill reset to live; local test images removed; nothing else provisioned. Hub: untouched. **Left, on purpose:** a `restore_mixed` hold for `radicale` in 9202's settings (the fence forbids CC to clear one; the command is in `part3/README.md`); `email_cleared: true` in Tester 1's settings; 9202's Paperless is a fresh install, the throwaway of 09-22 went with the walks. No VM was installed, so no hub host to delete. + +Poster: no fact on it changed. Instruction files: none edited. Evidence: `documentation/audits/security-2026-11-02/`. diff --git a/STATUS.md b/STATUS.md index 3efc1ab4..7f79a12b 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,29 +1,30 @@ # STATUS — what works, what's broken, what's next -**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** +**Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-30.** On your yes, felhom.eu and the installer are served by Cloudflare (DooPlex's copy stays until 11-06 as the way back), and the contact form posts to api.felhom.eu. Hub 0.178.0 (1 GiB) saves your vouch in 0.13 s instead of 38 s and opens the apps page in under a second. Agent 0.171.0 (your vouch) restarts a controller that runs but does not answer — proven on Tester 1. +**Updated 2026-11-03.** The security watch was not half-blind: seven pins it can never read held two alarms open, so a real outage could not have mailed; fixed, clean on both runners. Live, each proven on the scratch box: Gitea 28.3.0, Uptime Kuma 2.5.9, Grafana 13.2.5, ActualBudget 26.11.0, n8n 2.46.4, Vaultwarden 1.37.5 (`c760fca`). Traefik on every box has a real high notice (R-1354): first on the list. ## What works -- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241). +- **The Plex attack did not reach Felhom's secrets** (R-1241). - A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4). - 65 catalog apps install, update one tested step at a time, and a failed update undoes itself (map B1, B3, B4, B6). - Backups: database dumps, a second-drive copy, an off-site copy and whole-box copies; restore tests run by themselves (map C1–C4, A17, A18). -- A restore brings an app back with its data and its database password; a failed or cut restore puts the app back as it was; a restore that does not start says so and keeps both copies (map C5, D6, D7, D16). -- After a power cut the box starts by itself, cleans up a half-written backup, and Immich repairs broken photos (map C10, F12). -- A second drive is offered as the backup target; media apps see every drive; a one-disk box gets its own data area (map D1, D13, D15). -- The three test boxes answer only over https; the hub's key opens nothing on a box; password guesses are counted (map E12, E13, E17). -- A hacked app that uploads or computes more than it explains alarms only you; each box tests its upload weekly; qBittorrent gets half (map F13). +- A restore brings an app back with its data; a failed or cut restore puts it back as it was; one that does not start says so (map C5, D6, D7, D16). +- After a power cut the box starts by itself and cleans up a half-written backup (map C10, F12). +- A second drive is offered as the backup target; media apps see every drive (map D1, D13, D15). +- The three test boxes answer only over https; the hub's key opens nothing on a box (map E12, E13, E17). +- A hacked app that uploads or computes more than it explains alarms only you (map F13). - An urgent security fix reaches the boxes at night after one test on the test box; the box backs up first and undoes a failed update (map B23). -- The hub mails alarms and recoveries and keeps a weekly whole-box copy on ep0; CC uses its own limited token (map F1, F11, G15). +- The hub mails alarms and recoveries and keeps a weekly whole-box copy on ep0 (map F1, F11, G15). ## What is broken - **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours). - Tester 2 off since 10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). Scratch box: no agent features (R-1083). - DooPlex: something sets the shared checkout to „bare" (R-935). +- A box's event is lost if the hub is away for 9 seconds; a DooPlex restart can swallow a mail (R-1361). ## What we work on -- Apps still waiting for the HTTP2Shell fix: 6 — ActualBudget, cal.com, Calibre-Web, Paperless, RomM and SparkyFitness have no fixed release yet. +- HTTP2Shell waiters: cal.com, Calibre-Web, RomM have no fix; Paperless, SparkyFitness have one on a new main version (R-1331; R-785, behind R-784). - Dated checks, run by CC. 2026-11-06: ask you to retire DooPlex's old copy of the website (R-1228). ## Waiting on you — decide @@ -32,6 +33,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| +| D66 | R-1365 | Should a security-watch source that stays down mail again every 7 days, as an open notice does? Today it mails once, then only the Monday summary lists it. | **Yes** (a real outage then cannot be forgotten; the seven pins that can never be read no longer count, so it stays quiet when all is well) | no: one line in a long Monday mail stays the only reminder | a source that stays down mails once; the Monday list still shows it | — | | D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | | D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — | | D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — | @@ -39,7 +41,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te ## Waiting on you — do -**Still needed? (yes/no)** — CC could not check these. Answer yes or no per line. +**Still needed? (yes/no)** — answer per line. 1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16) 2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources (`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). (since 10-15) 3. Look at these demo-hp pages when you have time: Calibre-Web, the file manager, AdventureLog, Plex, the app page, the Plex and Sonarr guides. (since 10-16) @@ -47,12 +49,11 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). (since 10-14) 6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. (since 10-14) 7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). (since 10-12) - -**2026-10-30.** On the hub's Hosts page: Discard the unclaimed appliance registered 10:44:59 (uuid d3529d2d…, CC's deleted proof VM). If not: a dead entry alarms. **2026-10-29 (release run).** 1. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes. +**2026-11-03 release list** (your word). 0. The Sunday golden was missed; a code push is red until a bake and your vouch (R-468). 1. Controller v0.350.0 (`3764445`): Traefik v3.7.17 (R-1354), R-1347, R-1348, R-1339, R-1351, R-1323. If not: every box keeps the hole. 2. Agent v0.172.0 + vouch: R-1302. 3. Hub v0.179.0: R-1352. 4. Catalog, proven on the drill: Paperless 3.4.1 (R-1331), n8n 3.0.1 (R-1356), Mealie, Termix (R-1359). 5. To the homelab session: R-1355. +**2026-10-30.** On the hub's Hosts page: Discard the appliance of 10:44:59 (uuid d3529d2d…) and appliance 49 (CC's proof VMs). If not: dead entries alarm. **2026-10-29 (release run).** 1. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes. **2026-10-23 do list.** 1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens. -2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms. -3. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups. +2. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups. **The rest** - System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the website now on Cloudflare, only its mail form on DooPlex (D35). If not: the drawing is out of date. - Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). (since 10-09) diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index b7bccdf4..4c80787c 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -91,7 +91,7 @@ | [A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.336.0, agent v0.161.0, hub v0.162.0, golden 0.331.0, installer 1.35.0, ISO 1.29.0 | **PARTIAL** | 2026-10-20: download → install → bind → claim → 8 apps → use → backup → remove → every restore path → power cut → mistyped codes all worked; the 8 GB box FROZE twice in the first afternoon and only a plug pull brought it back (R-1192), and the owner passphrase needed the operator | `audits/rehearsal-2026-10-20/journal.md` (2026-10-20); `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) | | [A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) | | [A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp; **re-check (2026-10-24):** mechanism changed after its evidence — agent 1b6e316 (local keep-daily, R-1149) | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) | -| [A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record | +| [A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | `audits/security-2026-11-02/part4/restore-test-demo-hp.txt`, `restore-test-tester1.txt`, `restore-test-n100.txt` (2026-11-02: the journals of all three boxes, every tier passed unattended in 10 days); earlier: the row cited per-repo CHANGELOGs and a spec | | [A19](capability-map-detail/A.md#a19) | A restore-test can never fill the box's disk: it sizes the restore first (uncompressed), keeps off the tested guest's pool when it can, refuses what does not fit, and retries a failed clean-up on a timer | agent v0.133.0 (RELEASED, NOT DELIVERED), hub v0.124.0 | **IMPLEMENTED** | written BUILT: the refusal proven on demo-hp 2026-09-24; the full-test path not live | `audits/r672-2026-09-24/` (2026-09-24); R-672 | | [A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0; purge-first hub v0.142.0 | **PROVEN-LIVE** | two real firings (2026-07-18); third 2026-10-22 on a real customer (Tester-2) with the R-32 purge-first leg — all legs ok, others untouched | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/tester2-reset-2026-10-22/` | | [A21](capability-map-detail/A.md#a21) | Customer DELETE cascade (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **IMPLEMENTED** | written UNIT-PROVEN: live leg pending | hub unit tests `hub/internal/web/customer_delete_test.go` (outside `documentation/`) | @@ -121,7 +121,7 @@ | [B17](capability-map-detail/B.md#b17) | An app's VERSION is frozen to what the customer has; only a deliberate Update moves it, while template CORRECTIONS still arrive | controller v0.235.0 | **PROVEN-LIVE** | two real catalog pushes, real 15-minute cycle | `tests/VALIDATION-update-slice3-2026-09-06.md` (2026-09-06) | | [B18](capability-map-detail/B.md#b18) | Catalog sync (git, 15 min) + orphan lifecycle + validation choke point (bad `backup:` block degrades to legacy, loudly) | controller v0.132, catalog | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) | | [B19](capability-map-detail/B.md#b19) | Lemez-egészség felügyelet: per-disk SMART kártya („Lemezek állapota") + degradáció-riasztás (Rendben/Figyelmeztetés/Hiba/Nincs adat) | agent v0.94.0→v0.95.0, controller v0.169.0→v0.171.0→v0.215.0, hub v0.73.1 | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the Hiba-from-counters path is unit-only (R-332) | `audits/SPIKE-smart-coverage-2026-07-25.md` (2026-07-25); `audits/DIAG-smart-passed-trap-2026-08-14.md` (2026-08-14) | -| [B20](capability-map-detail/B.md#b20) | App crashes → customer notified (one event per transition, no flapping spam) | controller v0.120, hub v0.48 | **IMPLEMENTED** | crash→customer mail never live-confirmed | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) | +| [B20](capability-map-detail/B.md#b20) | App crashes → customer notified (one event per transition, no flapping spam) | controller v0.120, hub v0.48 | **PROVEN-LIVE** | — (the household's leg needs an address on its page; Tester 1 had none) | `audits/security-2026-11-02/part5/crash-walk.log`, `crash-walk-customer.log`, `mail-crash-operator.txt`, `mail-crash-household.txt` (2026-11-02, Tester 1: a crash loop stopped the app, the operator's mail in 1 s, the household's Hungarian mail in 1 s, read in the mailbox); `audits/CAMPAIGN-3-2026-07-11.md` | | [B21](capability-map-detail/B.md#b21) | Post-deploy optional config (API keys etc.) with restart | controller, catalog `.felhom.yml` | **IMPLEMENTED** | config-save + restart in no campaign | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) | | [B22](capability-map-detail/B.md#b22) | Backup classification: 13 bind-bearing apps carry mandatory/optional/excluded classes | catalog, controller v0.132–133 | **PROVEN-LIVE** | — | `audits/SPIKE-backup-classification-2026-07-14.md` (2026-07-14); `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15) | | [B23](capability-map-detail/B.md#b23) | An urgent security fix reaches the boxes at night — a `security_fix` ladder step (box walk required, bench optional, a CVE/GHSA notice), taken by the night update (`09` §3 decision 255, D39 = B) | catalog `3898593`, controller v0.341.0 | **PROVEN-LIVE** | scratch 9202 (drill catalog), Plex 1.43.4 → 1.43.5; an app with no box fixture is checked by health, version and a soak only (`seed: none`) | `audits/security-lane-2026-10-23/` (2026-10-23); batch use: `audits/security-app-updates-2026-10-27/` (2026-10-27, 14 steps live) | @@ -148,7 +148,7 @@ | [C1](capability-map-detail/C.md#c1) | Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded DEGRADED rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | — | `audits/E2D-fresh-vm-2026-07-29.md` (2026-07-29) | | [C2](capability-map-detail/C.md#c2) | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) | | [C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 03c0a69 + 66a69d4 + 5929b8e (capped container R-1214, kept copies D25) | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) | -| [C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 8f5bea0 + d5f88a2 (abandon R-1182, off-site proof R-1189) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) | +| [C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 8f5bea0 + d5f88a2 (abandon R-1182, off-site proof R-1189) | `audits/security-2026-11-02/part5/offsite-demo-hp.txt`, `offsite-tester1.txt` (2026-11-02: snapshots per day read from the repositories, unbroken 10-21 → 11-02 on demo-hp, 10-27 → 11-02 on Tester 1); `audits/CAMPAIGN-6D-2026-07-15.md`; `audits/VALIDATION-offbox-storagebox-2026-07-09.md` | | [C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 93fb251 + d582feb (D20, D22) **2026-10-25:** changed again by controller 0.343.0 (D48 pull-first, R-1270 fix A: the verdict needs a stable start, the undo copy goes after 3 more minutes); on 0.343.0 Tester 1's local unit restore and set-aside restore passed and released the copy (`audits/release-2026-10-26/t1-restores-0.343.0/`); this path itself not re-walked. | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) | | [C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d5f88a2 + 5929b8e (R-1187 c, R-1191) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) | | [C7](capability-map-detail/C.md#c7) | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08) | @@ -157,8 +157,8 @@ | [C10](capability-map-detail/C.md#c10) | Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) | | [C11](capability-map-detail/C.md#c11) | An app can be withdrawn from the catalog without orphaning the customers running it (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** | guest 9201, real endpoints | *(evidence not on file — not re-verifiable)*: the row cites TASK-F Part 1 and `felhom-controller/REPORT.md` (2026-07-21), both outside `documentation/` | | [C12](capability-map-detail/C.md#c12) | Box survives a site/network change (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE** | — | `audits/SPIKE-island-bridge-2026-07-25.md` (2026-07-25) | -| [C13](capability-map-detail/C.md#c13) | The customer is warned BEFORE a filesystem fills — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller v0.191.0/.1/.2, hub v0.89.0 (R-167, decision D-c) | **PROVEN-LIVE** | guest 9201, a real filesystem | *(evidence not on file — not re-verifiable)*: the row cites `felhom-controller/REPORT.md` (2026-08-02); `audits/SPIKE-r165-mp1-merge-2026-08-02.md` is cited as context only | -| [C14](capability-map-detail/C.md#c14) | A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed | controller v0.194.0, hub v0.90.1 (R-158 → R-167 → R-182) | **PROVEN-LIVE** | guest 9201 | *(evidence not on file — not re-verifiable)*: the row cites `felhom-controller/REPORT.md` (2026-08-03), outside `documentation/` | +| [C13](capability-map-detail/C.md#c13) | The customer is warned BEFORE a filesystem fills — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller v0.191.0/.1/.2, hub v0.89.0 (R-167, decision D-c) | **PROVEN-LIVE** | guest 9201, a real filesystem | `audits/security-2026-11-02/part5/fill-walk.log`, `mail-fill-household.txt` (2026-11-03, Tester 1: a drive taken to 89 %, the watcher's next 10-minute tick, the household's Hungarian mail naming the drive and 5,2 GB free, the operator's copy, once) | +| [C14](capability-map-detail/C.md#c14) | A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed | controller v0.194.0, hub v0.90.1 (R-158 → R-167 → R-182) | **PROVEN-LIVE** | guest 9201 | `audits/security-2026-11-02/part5/mail-operator-digest.txt` (read in the mailbox 2026-11-03: „Tester4: 3 of 8 apps failed to back up (manual run)", each app named, one mail) | | [C15](capability-map-detail/C.md#c15) | A local backup is bounded by the box's FREE SPACE, not by a partition set at build time — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` v3.0.0, agent v0.120.0, controller v0.193.1 | **PROVEN-LIVE** | both halves | `audits/SPIKE-r165-phase0-2026-08-03.md` (2026-08-03) | | [C16](capability-map-detail/C.md#c16) | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) | | [C17](capability-map-detail/C.md#c17) | A customer (not the operator) performs a restore via UI alone | all | **MISSING** | as evidence | `audits/DIAG-immich-restore-2026-07-19.md` (2026-07-19); R-3 | @@ -189,7 +189,7 @@ | ID | Capability | Component | Status | Scope | Evidence | |---|---|---|---|---|---| | [E1](capability-map-detail/E.md#e1) | Remote access via Cloudflare Tunnel + Traefik (per-app subdomains) | cloudflared, traefik | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) | -| [E2](capability-map-detail/E.md#e2) | LAN access when internet is down (lan_resolver) | agent | **IMPLEMENTED** | never drilled as a customer experience | R-19 | +| [E2](capability-map-detail/E.md#e2) | LAN access when internet is down (lan_resolver) | agent | **PROVEN-LIVE** | the NAME works only if the home network's DNS points at the box, and on a host whose own FQDN is the dashboard's name it answers the host (R-1364) | `audits/security-2026-11-02/part5/outage-wan.log`, `lan-resolver.txt` (2026-11-03, Tester 1, internet and hub dropped 13 min: the dashboard answered at the box's address, 10 apps ran, the public name 530, back in 60 s after) | | [E3](capability-map-detail/E.md#e3) | Phone photo backup | immich (classified) | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15) | | [E4](capability-map-detail/E.md#e4) | Documents/OCR | paperless-ngx (classified) | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) | | [E5](capability-map-detail/E.md#e5) | Files from Windows Explorer / Mac Finder (SMB server) | controller v0.145.0 + `felhom-samba:1.0.0` | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites `felhom-controller` REPORT.md (v0.144.0, outside `documentation/`), the design `controller/sharing.md` and the transport verdict `audits/SPIKE-lan-discovery-2026-07-18.md` — none is a run of the shipped server | diff --git a/documentation/architecture/where-felhom-stands.html b/documentation/architecture/where-felhom-stands.html index 117a335c..4489cb86 100644 --- a/documentation/architecture/where-felhom-stands.html +++ b/documentation/architecture/where-felhom-stands.html @@ -6,17 +6,17 @@

Where Felhom stands

What we built, what happens when things go wrong, and what is still missing. Generated from where-felhom-stands.yaml, which cites the capability map, register row or evidence document behind every claim — and which is checked by scripts/check_stands.py.
-
Verified 2026-08-22 against felhom-agent 40d857b527, felhom-controller 2da259af38, hub 877fcd2a38. 17 claim(s) carry a recorded status move (7 re-checked in this pass) — each is marked on the page.
-
Walked (24)done end to end on real hardware, evidence on file
Built (12)shipped and tested, the real path never walked
Partial (15)some walked, some not — the note says which
Missing (4)does not exist
needs-hardwareonly a running box could settle it
+
Verified 2026-08-22 against felhom-agent 40d857b527, felhom-controller 2da259af38, hub 877fcd2a38. 17 claim(s) carry a recorded status move (4 re-checked in this pass) — each is marked on the page.
+
Walked (32)done end to end on real hardware, evidence on file
Built (4)shipped and tested, the real path never walked
Partial (15)some walked, some not — the note says which
Missing (4)does not exist
needs-hardwareonly a running box could settle it
The journey
Seven stages, left to right, as they happen to a person.
1  Getting a box
A blank machine installs itself from our own boot image and registers itself as unclaimed — proven on two different boardsTwo boards: N100 2026-07-18, HP t740 2026-07-21.capability-map Bare-metal Felhom ISO (blank hardware → zero-touch auto-ins… · evidence tests/VALIDATION-n100-rehearsal-2026-07-18.md
The installer is published rather than pushed: rolling it back is one actchanged 2026-08-09, was walkedGate 6 of hostinstall_gates asserts the manifest names an installer-v tag; ran green tonight.why it moved: gate 6 asserts the manifest names an installer tag, but no walk of a rollback is on filecapability-map The installer is PUBLISHED, not pushed · register R-110
Installing onto hardware the customer already owns — the path exists, the first real one has not happenedUPGRADED: a real --mode byo install completed on demo-hp 2026-08-09 (Day-0 provision SUCCESS, 3m49s). Still not a customer's own hardware, so not 'walked' — but 'has not happened' is now false.evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
A machine that ends up on the wrong network port explains itself on screen and finds its way back — proven in a virtual drill, never on metalMap says the same: PROVEN-LIVE (nested drill — nested != metal).capability-map Box survives a wrong-NIC install · evidence audits/SPIKE-firstboot-nic-sweep-2026-07-22.md
Reinstalling a machine we previously installed refuses, because our own removal leaves a name service holding the port our own installer checksWarning card. Confirmed live 2026-08-09; dnsmasq restarted by --uninstall seizes :53.register R-272 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
2  Making it theirs
An emailed one-time code; the customer sets their own password and the operator never sees itExercised live 2026-08-09: reset code accepted at /claim, 302 + session.capability-map Customer claim: one-time emailed code → customer sets own p… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
The customer can bind their own machine from a link, with no operator present — done once, for realattempts=0, locked=0, source customer_selfbind, 2026-07-18.capability-map Customer binds their own appliance (self-service) · evidence tests/VALIDATION-n100-rehearsal-2026-07-18.md
It has never been done by a person who is not the operatorMap records MISSING (as evidence). Still true after 2026-08-09.capability-map A customer (not the operator) performs a restore via UI alone
Each of the two secrets has one name on every surface, and the re-enrol mail points at a page a rebuilt machine showsShipped (controller v0.211.0, hub v0.104.0/v0.105.0); no customer has typed the passphrase yet. Operator ruling 2026-10-12 (S13). The old defect cost a real code on 2026-08-09.register R-282 · register R-295 · register R-323 · register R-327
-
3  Using it
Apps installed from a catalog of about 52, with a memory guard and health-aware progress53 templates listed on the rebuilt box 2026-08-09; deploy driven live.capability-map Deploy an app from the catalog (env config, memory guard, h… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
Start, stop, restart, update, logs, remove — and the parts that must not be stopped cannot bechanged 2026-08-09, was builtWalked 2026-10-28 night on scratch 9202: install, stop, start, restart, logs; traefik and cloudflared refuse a stop (403); remove with data leaves no volume and no container. The update half: the same day's guarded updates on 9202 (bench + box + undo walks).why it moved: a walk document is now cited (2026-10-28 night)capability-map App lifecycle: start/stop/restart/update/logs/remove/redeploy · register R-108 · evidence audits/night-2026-10-28/part5-claims/lifecycle-walk.log · evidence audits/dockerhub-bigsteps-2026-10-28
Reachable from anywhere through a tunnel, per-app addressesVerified tonight: the rebuilt box answered on its public URL from outside.capability-map Remote access via Cloudflare Tunnel + Traefik (per-app subd… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
Reachable on the home network when the internet is downneeds-hardwareOnly an observation on a running box with WAN pulled could settle it. Boxes are off.capability-map LAN access when internet is down (lan_resolver)
Phone photos, documents with text recognition, files from Windows Explorer or a Maccapability-map Files from Windows Explorer / Mac Finder (SMB server) · evidence audits/SPIKE-lan-discovery-2026-07-18.md
A one-tap launcher, and a read-only guest link for visitorschanged 2026-08-09, was builtWalked 2026-10-28 night on scratch 9202 through the pages the browser receives: the launcher lists each openable app; the guest link opens with no login and no admin link, a wrong token is 404, rotate and switch-off kill the old link.why it moved: walked live 2026-10-28 night (map E8, E9 → PROVEN-LIVE)capability-map Indítópult (app launcher) — one-tap grid of the household's… · evidence audits/night-2026-10-28/part5-claims/launcher-walk.log
Media to a TVMap: MISSING.capability-map Media to TV via DLNA
Separate accounts per household memberMap: MISSING.capability-map Multiple household users / per-person accounts
+
3  Using it
Apps installed from a catalog of about 52, with a memory guard and health-aware progress53 templates listed on the rebuilt box 2026-08-09; deploy driven live.capability-map Deploy an app from the catalog (env config, memory guard, h… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
Start, stop, restart, update, logs, remove — and the parts that must not be stopped cannot bechanged 2026-08-09, was builtWalked 2026-10-28 night on scratch 9202: install, stop, start, restart, logs; traefik and cloudflared refuse a stop (403); remove with data leaves no volume and no container. The update half: the same day's guarded updates on 9202 (bench + box + undo walks).why it moved: a walk document is now cited (2026-10-28 night)capability-map App lifecycle: start/stop/restart/update/logs/remove/redeploy · register R-108 · evidence audits/night-2026-10-28/part5-claims/lifecycle-walk.log · evidence audits/dockerhub-bigsteps-2026-10-28
Reachable from anywhere through a tunnel, per-app addressesVerified tonight: the rebuilt box answered on its public URL from outside.capability-map Remote access via Cloudflare Tunnel + Traefik (per-app subd… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
Reachable on the home network when the internet is downWalked 2026-11-03: with the internet gone the dashboard answers at the box's own address, and the box's DNS answers every app's name. By NAME it needs the home network's DNS to point at the box, and nothing tells the household; on a host whose FQDN is the dashboard's name (Tester 1) that one name answers the host and fails (R-1364).evidence audits/security-2026-11-02/part5/lan-resolver.txt · evidence audits/security-2026-11-02/part5/outage-wan.log · register R-1364 · capability-map LAN access when internet is down (lan_resolver)
Phone photos, documents with text recognition, files from Windows Explorer or a Maccapability-map Files from Windows Explorer / Mac Finder (SMB server) · evidence audits/SPIKE-lan-discovery-2026-07-18.md
A one-tap launcher, and a read-only guest link for visitorschanged 2026-08-09, was builtWalked 2026-10-28 night on scratch 9202 through the pages the browser receives: the launcher lists each openable app; the guest link opens with no login and no admin link, a wrong token is 404, rotate and switch-off kill the old link.why it moved: walked live 2026-10-28 night (map E8, E9 → PROVEN-LIVE)capability-map Indítópult (app launcher) — one-tap grid of the household's… · evidence audits/night-2026-10-28/part5-claims/launcher-walk.log
Media to a TVMap: MISSING.capability-map Media to TV via DLNA
Separate accounts per household memberMap: MISSING.capability-map Multiple household users / per-person accounts
4  Drives
A new drive is found, offered, formatted, mounted and enrolled — including on awkward older boot layoutsmoved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walkedApplies to a NEW drive. Re-attaching an existing one after a reinstall is R-280 and fails.why it moved: receipt found 2026-08-10: a live throwaway drive on felhom-pve walked scan -> format -> mount -> PVE dir storage -> one-click Regisztralas, with the resulting storage.cfg entry and systemd mount unit recorded. The map already read PROVEN-LIVE; the dataset was behind it. Scope unchanged: this is a NEW drive. Re-attaching an existing one after a reinstall is R-280 and still fails.capability-map Drive wizard: scan/format/mount/enroll, incl. legacy-boot L… · register R-220 · evidence audits/SPIKE-raw-drive-enroll-2026-06-15.md
Moving data between drives, crash-safe; removing a drive safely; unplug detectedchanged 2026-08-09, was walkedwhy it moved: no walk document citedcapability-map Data migration between drives (all / per-app), crash-safe
A network drive can be browsed and hold bulk media, but may not hold an app's data — enforcedRefuseAsAppNamespace is the fail-closed predicate.capability-map Network storage (NAS) is browse + bulk-media only · register R-108 · evidence audits/R108-network-app-namespace-2026-07-30.md
After a reinstall the data drive cannot be re-attached through any dashboard routeWarning card. /api/disks/candidates returns empty; the restore page promises two clicks.register R-280 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
-
5  Backing up
App data on the machine, nightly database dumps, a copy on a second drivemoved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walkedTrue for every app but one, and that one was found on 2026-08-21: paperless-ngx's 72-table PostgreSQL was dumped nightly and written into a folder named after an app that does not exist, so it never entered the recovery unit, the second-drive copy or the off-site copy. Fixed and proven live 2026-08-22 (R-355) — the dump is now in the app's own unit and in the off-site snapshot for the first time. A catalogue-wide sweep, itself proven able to convict a planted case, says 1 of 53 was affected.why it moved: receipt found 2026-08-10: an adversarial, destructive, unattended overnight campaign across both boxes and ep0, with A2 (one quiesce, two tiers) proven end to end. Map already read PROVEN-LIVE.capability-map Tier-2 secondary-drive copy: class-driven legs · register R-355 · evidence audits/CAMPAIGN-8-backup-restore-2026-07-27.md · evidence audits/DRILL-backup-truth-2026-08-21/evidence/phase4-paperl…
A whole-machine archive that lands off the guest's own disk — a single-drive machine is recorded as degraded rather than pretendingchanged 2026-08-09, was walkedwhy it moved: no walk document citedcapability-map Whole-guest backup lands OFF the guest's own physical device · register R-165
An encrypted off-site copy, sealed with a key the operator cannot readchanged 2026-08-09, was walkedThe COPY is real and the key is still unreadable to us — what failed is DAILY and UNBROKEN. The 2026-08-09 note said '18 snapshots, daily, unbroken'; the next snapshot after 2026-08-09 08:30 was 2026-08-21 22:17, put there by hand during the drill. Twelve days, no alarm. Two causes in series: the 2026-08-21 rebuild lost the off-box target (R-193's shape), and after the self-heal restored it EVERY per-app switch was still off, so the first run logged 'backup OK: 0 app(s) backed up, 14s'.why it moved: the claim is about a CONTINUING daily copy; a 12-day silent gap was found on 2026-08-21 and nothing reported itcapability-map Offsite (restic → Hetzner Storage Box) · register R-199 · register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
The backups prove themselves: a restore is actually performed, unattended, on every tier, on both machineschanged 2026-08-09, was walkedSTILL GREY, and for a sharper reason than in August. The scheduler runs and the check works — it failed again on 2026-08-21, unprompted, and named the cause exactly: after the reinstall the box presents a different PBS key than its own older archives were sealed with, so those archives cannot be opened at all (R-366). A tier whose archives are orphaned is a different alarm from a tier whose test failed, and only the second is being said.proof decay: PROOF-DECAY RULE FIRED (first time it has). A receipt EXISTS - architecture/_recovery-inventory-2026-07-28.md carries live journal lines for scheduled restore-tests on both boxes and both tiers - but it is superseded by later observation: demo-hp logged restore_test_failed on 2026-08-05, and the box has since been wiped and reinstalled (2026-08-09). The claim is about a CONTINUING scheduled behaviour, so a 2026-07-28 observation cannot carry it. Stays grey until a scheduled restore-test is seen passing on the rebuilt box. THE CAPABILITY MAP STILL READS PROVEN-LIVE (2026-08-03) AND IS NOW THE THING OUT OF STEP.why it moved: no walk document cited, and the last recorded restore-test on demo-hp FAILED (2026-08-05)capability-map Restore-proof is UNATTENDED — the scheduler covers EVERY tier · register R-86 · register R-366 · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
The customer is warned before a drive fills, per drive, in their own languagechanged 2026-08-09, was walkedThe warning is real and was SEEN firing on 2026-08-21 with the right Hungarian copy, naming the drive and the free space. What 'BEFORE' cannot survive is the cadence: the watcher runs once a day at 03:30 plus once at startup (R-363), so a filesystem that fills at 03:31 goes unannounced for ~24 h. Watched live: the 69 GB volume carrying all 40-class app data was filled to 99% and the watcher said nothing, while the backup reserve was already refusing an app per run and telling the hub about it.why it moved: a daily check cannot carry the word BEFORE; observed silent for the whole window a filesystem sat at 99%capability-map The customer is warned BEFORE a filesystem fills · register R-167 · register R-363 · evidence audits/SPIKE-r165-mp1-merge-2026-08-02.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
A backup that covered nothing still calls itself successfulWarning card — and the drill found two more of it, both live. (1) An off-site run with no app selected logs 'backup OK: 0 app(s) backed up'; the card does say 'nincs kijelölt alkalmazás' beside the green tick, so this one is honest if you read past the tick. (2) A restore that placed nothing reported success: '0 fájl visszaállítva', ok=true. The second is the one that matters and it is R-353, still open. The volume half of it is fixed (R-354): the message now names what came back.register R-240 · register R-353 · register R-354 · evidence audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experi…
-
6  When something goes wrong
The machine watches itself and repairs some faults without telling anyone it had tochanged 2026-08-09, was builtWalked 2026-10-29 night on scratch 9202: a crashed app part (kill -9 of paperless-redis) ran again in 1 s and the health probe read the app healthy; a guest reboot brought every app back in under 2 minutes, the boot sweep found nothing to start; no event, no banner, no mail. Docker's restart policy did the repair in both legs; the controller's own boot repair had nothing to do (its last proof: 6 of 6 hard resets, 2026-08-02). R-264 records that the self-heal counters reach the hub and are decoded nowhere.why it moved: walked 2026-10-29 on scratch 9202 (a walk document is now cited)capability-map Box survives an unattended app or guest-network failure · register R-264 · evidence audits/night-2026-10-28/part5-claims/selfheal/WALK.md · evidence audits/DIAG-bootrecon-drive-absent-2026-08-02.md
Failures reach the operator by email, one mail per run, every failing app namedchanged 2026-08-09, was walkedCONFIRMED against source: backup_run_failures digest is allowlisted (api/handler.go:1837), operator-only (dispatcher.go:423), templated (templates.go:48); recovery_unit_capture_failed is record-only (dispatcher.go:376). NOTE: R-182's register row still describes the PRE-FIX behaviour — see R-289.why it moved: the digest is wired and source-verified, but no run of it has been observed deliveringcapability-map A failed per-app Tier-1 backup reaches the OPERATOR — EVERY… · register R-182
Failures reach the customerThe customer leg is built; today's log shows customer-channel rows skipped as operator_only.capability-map App crashes → customer notified (one event per transition, …
An already-paired box no longer tells anyone to pair it — its console says there is nothing to dochanged 2026-08-09, was partialWas a warning card (the console kept showing the pairing code). Fixed by ISO 1.28.0 (R-535; R-214, R-235 closed). Walked 2026-10-29 00:04 on Tester 1 (paired, claimed): the console reads, in Hungarian and English, that there is nothing to do on this machine and no need to log in — no pairing code.why it moved: the defect is closed and the console was read live on a paired box (2026-10-29)register R-214 · register R-235 · evidence audits/night-2026-10-28/part5-claims/tester1-console-2026-1…
+
5  Backing up
App data on the machine, nightly database dumps, a copy on a second drivemoved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walkedTrue for every app but one, and that one was found on 2026-08-21: paperless-ngx's 72-table PostgreSQL was dumped nightly and written into a folder named after an app that does not exist, so it never entered the recovery unit, the second-drive copy or the off-site copy. Fixed and proven live 2026-08-22 (R-355) — the dump is now in the app's own unit and in the off-site snapshot for the first time. A catalogue-wide sweep, itself proven able to convict a planted case, says 1 of 53 was affected.why it moved: receipt found 2026-08-10: an adversarial, destructive, unattended overnight campaign across both boxes and ep0, with A2 (one quiesce, two tiers) proven end to end. Map already read PROVEN-LIVE.capability-map Tier-2 secondary-drive copy: class-driven legs · register R-355 · evidence audits/CAMPAIGN-8-backup-restore-2026-07-27.md · evidence audits/DRILL-backup-truth-2026-08-21/evidence/phase4-paperl…
A whole-machine archive that lands off the guest's own disk — a single-drive machine is recorded as degraded rather than pretendingchanged 2026-08-09, was walkedwhy it moved: no walk document citedcapability-map Whole-guest backup lands OFF the guest's own physical device · register R-165
An encrypted off-site copy, sealed with a key the operator cannot readchanged 2026-08-09, was walkedDAILY and UNBROKEN, read from the repositories 2026-11-02: demo-hp 14–15 snapshots every day 10-21 → 11-02, Tester 1 4 every day 10-27 → 11-02. The 2026-08 gap's two causes were fixed (R-199, R-193). The key stays the household's.why it moved: the claim is about a CONTINUING daily copy; a 12-day silent gap was found on 2026-08-21 and nothing reported itevidence audits/security-2026-11-02/part5/offsite-demo-hp.txt · evidence audits/security-2026-11-02/part5/offsite-tester1.txt · capability-map Offsite (restic → Hetzner Storage Box) · register R-199 · register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
The backups prove themselves: a restore is actually performed, unattended, on every tier, on both machineschanged 2026-08-09, was walkedRead 2026-11-02 from the journals of all three boxes: a scheduled restore-test passed on every tier in 10 days — demo-hp local and PBS, Tester 1 local and PBS, the N100 its USB drive and PBS, the newest on 2026-11-02 22:45. R-366's orphaned key was a one-off of the 2026-08 reinstall.proof decay: PROOF-DECAY RULE FIRED (first time it has). A receipt EXISTS - architecture/_recovery-inventory-2026-07-28.md carries live journal lines for scheduled restore-tests on both boxes and both tiers - but it is superseded by later observation: demo-hp logged restore_test_failed on 2026-08-05, and the box has since been wiped and reinstalled (2026-08-09). The claim is about a CONTINUING scheduled behaviour, so a 2026-07-28 observation cannot carry it. Stays grey until a scheduled restore-test is seen passing on the rebuilt box. THE CAPABILITY MAP STILL READS PROVEN-LIVE (2026-08-03) AND IS NOW THE THING OUT OF STEP.why it moved: no walk document cited, and the last recorded restore-test on demo-hp FAILED (2026-08-05)evidence audits/security-2026-11-02/part4/restore-test-demo-hp.txt · evidence audits/security-2026-11-02/part4/restore-test-tester1.txt · evidence audits/security-2026-11-02/part4/restore-test-n100.txt · capability-map Restore-proof is UNATTENDED — the scheduler covers EVERY tier · register R-86 · register R-366 · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
The customer is warned before a drive fills, per drive, in their own languagechanged 2026-08-09, was walkedWalked 2026-11-03 on Tester 1: a data drive taken to 89 %; the watcher (every 10 minutes since R-363) warned at its next tick, 2 minutes later: the household's Hungarian mail names the drive and 5,2 GB free, once. The August gap (a daily check) is gone. The cadence leaves a drive up to 10 minutes.why it moved: a daily check cannot carry the word BEFORE; observed silent for the whole window a filesystem sat at 99%evidence audits/security-2026-11-02/part5/fill-walk.log · evidence audits/security-2026-11-02/part5/mail-fill-household.txt · capability-map The customer is warned BEFORE a filesystem fills · register R-167 · register R-363 · evidence audits/SPIKE-r165-mp1-merge-2026-08-02.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
A backup that covered nothing still calls itself successfulWarning card — and the drill found two more of it, both live. (1) An off-site run with no app selected logs 'backup OK: 0 app(s) backed up'; the card does say 'nincs kijelölt alkalmazás' beside the green tick, so this one is honest if you read past the tick. (2) A restore that placed nothing reported success: '0 fájl visszaállítva', ok=true. The second is the one that matters and it is R-353, still open. The volume half of it is fixed (R-354): the message now names what came back.register R-240 · register R-353 · register R-354 · evidence audits/DRILL-backup-truth-2026-08-21/evidence/phase3-experi…
+
6  When something goes wrong
The machine watches itself and repairs some faults without telling anyone it had tochanged 2026-08-09, was builtWalked 2026-10-29 night on scratch 9202: a crashed app part (kill -9 of paperless-redis) ran again in 1 s and the health probe read the app healthy; a guest reboot brought every app back in under 2 minutes, the boot sweep found nothing to start; no event, no banner, no mail. Docker's restart policy did the repair in both legs; the controller's own boot repair had nothing to do (its last proof: 6 of 6 hard resets, 2026-08-02). R-264 records that the self-heal counters reach the hub and are decoded nowhere.why it moved: walked 2026-10-29 on scratch 9202 (a walk document is now cited)capability-map Box survives an unattended app or guest-network failure · register R-264 · evidence audits/night-2026-10-28/part5-claims/selfheal/WALK.md · evidence audits/DIAG-bootrecon-drive-absent-2026-08-02.md
Failures reach the operator by email, one mail per run, every failing app namedchanged 2026-08-09, was walkedSeen in the mailbox: „Tester4: 3 of 8 apps failed to back up (manual run)\" names immich, nextcloud and paperless-ngx in one mail (2026-10-20); seven of this shape in 60 days.why it moved: the digest is wired and source-verified, but no run of it has been observed deliveringevidence audits/security-2026-11-02/part5/mail-operator-digest.txt · capability-map A failed per-app Tier-1 backup reaches the OPERATOR — EVERY… · register R-182
Failures reach the customerSeen in the mailbox: the crash mail and the fill warning to tester1@ (2026-11-02/03), „A szerver nem elérhető!\" to tester4@ (2026-10-24), each in Hungarian. It needs an address on the household's page (Tester 1 had none).evidence audits/security-2026-11-02/part5/mail-crash-household.txt · evidence audits/security-2026-11-02/part5/mail-fill-household.txt · evidence audits/security-2026-11-02/part5/mail-operator-digest.txt · capability-map App crashes → customer notified (one event per transition, …
An already-paired box no longer tells anyone to pair it — its console says there is nothing to dochanged 2026-08-09, was partialWas a warning card (the console kept showing the pairing code). Fixed by ISO 1.28.0 (R-535; R-214, R-235 closed). Walked 2026-10-29 00:04 on Tester 1 (paired, claimed): the console reads, in Hungarian and English, that there is nothing to do on this machine and no need to log in — no pairing code.why it moved: the defect is closed and the console was read live on a paired box (2026-10-29)register R-214 · register R-235 · evidence audits/night-2026-10-28/part5-claims/tester1-console-2026-1…
7  Getting everything back
A rebuilt machine shows a full-page recovery screen without anyone looking for it, and says plainly that nobody can replace a lost recovery codeSeen unsought on the rebuilt demo-hp 2026-08-09, seal date matching host_escrow.created_at.register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
The customer's code opens the sealed package and the data returns byte for byte — including accented Hungarian filenames, verified as raw bytesStill true, and re-proven 2026-08-22 (5/5 byte-identical, both accented names as raw bytes) — but the SCOPE is narrower than the sentence sounds and was silently narrower still until v0.218.0. The drill found the off-site restore had NO named-volume leg at all: the archive sat in the unit, the snapshot and the checking folder and was never replayed, under a success message (R-354, fixed and proven 2026-08-22). AND 40 of the 53 catalogue apps STILL cannot run this route at all — it refuses first, saying a running app is not installed (R-356, open). So: proven for an app that declares a data drive; unproven and currently unreachable for the class whose entire dataset is a named volume.register R-201 · register R-354 · register R-356 · evidence tests/walk5-r201-2026-08-07/journal.md · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
Walked end to end with no command line inside the machine (2026-08-07)contestedCONTESTED-RESOLVED: not a contradiction. The walk proves the ROUTE needs no guest shell; the map's MISSING row is about a NON-OPERATOR doing it, which has still never happened. Two questions, one word 'customer'.register R-201 · evidence tests/walk5-r201-2026-08-07/journal.md · capability-map A customer (not the operator) performs a restore via UI alone
Putting restored files back where they belong is still manualWarning card. Confirmed 2026-08-09: the restore lands in a verification folder and says so.register R-213
The tripwire that says someone is opening this customer's backups does fireCONFIRMED tonight from the hub store: escrow_blob_served 2026-08-09 10:19:41Z = 12:19 CEST in the operator's mailbox. R-281's original claim of silence is WITHDRAWN.register R-281 · register R-285 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
When things go wrong
@@ -24,8 +24,8 @@
A disk starts failing
A disk starts failing — healthy path only; a genuinely failing disk has never been seenneeds-hardwareOnly a failing disk on a running box could settle it.capability-map Lemez-egészség felügyelet: per-disk SMART kártya · evidence audits/SPIKE-smart-coverage-2026-07-25.md
The backup drive is unplugged
The backup drive is unpluggedcapability-map An ABSENT backup-target drive raises its OWN alarm · evidence audits/R116-v0116-2026-07-30.md
-
A drive is filling up
A drive is filling upchanged 2026-08-09, was walkedwhy it moved: no walk document citedregister R-167
-
An app crashes
An app crashes — the email leg has never been confirmed end to endConsistent with fault.operator-email: the digest is wired but its delivery is unobserved.capability-map App crashes → customer notified (one event per transition, …
+
A drive is filling up
A drive is filling upchanged 2026-08-09, was walkedSame walk as backup.fill-warning: warned before it filled; the file removed, nothing else was needed.why it moved: no walk document citedevidence audits/security-2026-11-02/part5/fill-walk.log · register R-167
+
An app crashes
An app crashes — the box stops it and both the operator and the household get a mailWalked 2026-11-02 on Tester 1: kill -9 of a throwaway app, 15 s apart; at 6 restarts in 10 minutes the box stopped and held it, the operator's mail arrived in 1 s; with an address on the household's page the Hungarian mail arrived too. The household's Start lifted it. One flaw: the mail ends with raw JSON (R-1363).evidence audits/security-2026-11-02/part5/crash-walk.log · evidence audits/security-2026-11-02/part5/crash-walk-customer.log · evidence audits/security-2026-11-02/part5/mail-crash-operator.txt · evidence audits/security-2026-11-02/part5/mail-crash-household.txt · register R-1363 · capability-map App crashes → customer notified (one event per transition, …
Power cut mid-backup
Power cut mid-backupevidence audits/AUDIT-power-outage-recovery-2026-07-22.md
The guest is destroyed
The guest is destroyedregister R-201 · evidence audits/DRILL-r201-night-run-2026-08-04.md · evidence audits/DRILL-r201-night-run-2026-08-04.md
The whole machine is wiped and reinstalled
The whole machine is wiped and reinstalled — the data comes backchanged 2026-08-09, was walkedThe 2026-08-09 rehearsal really did return 4/4 byte-identical, and that stands. What the next real reinstall showed (2026-08-21, demo-hp) is that the rebuild ORPHANS BOTH OFF-PREMISES TIERS at once, quietly: the restic target was lost and needed a self-heal plus a per-app re-enable before any copy resumed (R-193), and the PBS archives from before the reinstall cannot be opened by the rebuilt box at all, because it now presents a different key (R-366). The data came back in the rehearsal because the rehearsal restored it immediately; a machine left alone after a reinstall is not protected in the meantime and nothing says so.why it moved: a real reinstall on 2026-08-21 left both off-premises tiers broken - one silently for 12 days, the other unreadable - so 'the data comes back' holds only if someone restores it at onceregister R-193 · register R-366 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md
@@ -34,8 +34,8 @@
The customer forgets their dashboard password
The customer forgets their dashboard passwordExercised 2026-08-09 via the reset code.evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
The customer loses their recovery code
The customer loses their recovery code — by design, the data is unrecoverablemoved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walkedThe recovery screen states it in Hungarian.why it moved: receipt found 2026-08-10, and then proven the hard way: the recovery screen states in Hungarian that nobody can replace a lost code, and on 2026-08-10 demo-felhom's pre-fix key was confirmed unrecoverable by construction (identity_blob NULL; the restic password lives only in that bundle). The by-design refusal is real.capability-map Escrow ceremony: customer-facing wizard, one-shot R claim, … · register R-198 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md
The machine moves house / new network
The machine moves house / new networkcapability-map Box survives a site/network change (relocation, different s… · evidence audits/AUDIT-vacation-remote-ops-2026-07-20.md
-
The internet goes down
The internet goes down — built, never walkedneeds-hardwareNeeds a running box with WAN pulled.capability-map LAN access when internet is down (lan_resolver)
-
The hub is down
The hub is downchanged 2026-08-09, was walkedwhy it moved: no walk document citedcapability-map Config/state change round-trips in seconds (hub↔box immediacy
+
The internet goes down
The internet goes downWalked 2026-11-03 on Tester 1: packets to the outside dropped on host and guest for 13 minutes, the line's route left in place. The dashboard answered on the home network, 10 apps ran, the public name gave 530; 60 s after the line came back the public name worked by itself, nothing to do. Slow: the Backups page took over 2 minutes (R-1362). An earlier try removed the route: the agent's self-heal put it back in about 100 s.evidence audits/security-2026-11-02/part5/outage-wan.log · evidence audits/security-2026-11-02/part5/outage-wan-3min.txt · register R-1362 · capability-map LAN access when internet is down (lan_resolver)
+
The hub is down
The hub is downchanged 2026-08-09, was walkedWalked 2026-11-03 on Tester 1 (the hub's address dropped for 13 minutes; DooPlex untouched): the box works on — dashboard, apps, an install from the cached catalog, a crash-loop stop. What fails: every event sent in that time is lost after 9 s and nothing re-sends it, so the operator never heard of the held app (R-1361).why it moved: no walk document citedevidence audits/security-2026-11-02/part5/outage-hub.log · register R-1361 · capability-map Config/state change round-trips in seconds (hub↔box immediacy
We ship a broken release
We ship a broken release — the guard is missingProven the hard way on 2026-08-09: a vouched agent version had no git tag and every install died at 5/8. Both guards still owed.register R-273 · register R-287
A fresh install picks up an old image
A fresh install picks up an old imageNarrowed 2026-08-09: the resume path fetched the vouched golden; a fresh install still takes the newest LOCAL archive with no manifest comparison.register R-274
The off-site provider account is deleted
The off-site provider account is deletedThe credential that holds the customer's documents can still delete.register R-95
diff --git a/documentation/architecture/where-felhom-stands.yaml b/documentation/architecture/where-felhom-stands.yaml index e315035a..162ef9ff 100644 --- a/documentation/architecture/where-felhom-stands.yaml +++ b/documentation/architecture/where-felhom-stands.yaml @@ -186,14 +186,17 @@ claims: band: journey stage: 3 title: "Reachable on the home network when the internet is down" - status: built - note: "Only an observation on a running box with WAN pulled could settle it. Boxes are off." + status: partial + note: "Walked 2026-11-03: with the internet gone the dashboard answers at the box's own address, and the box's DNS answers every app's name. By NAME it needs the home network's DNS to point at the box, and nothing tells the household; on a host whose FQDN is the dashboard's name (Tester 1) that one name answers the host and fails (R-1364)." sources: + - evidence: "audits/security-2026-11-02/part5/lan-resolver.txt" + - evidence: "audits/security-2026-11-02/part5/outage-wan.log" + - register: "R-1364" - capability-map: "LAN access when internet is down (lan_resolver)" verified: - date: 2026-08-09 - verdict: needs-hardware - depth: needs-hardware + date: 2026-11-03 + verdict: upgraded + depth: source-read - id: use.files band: journey stage: 3 @@ -345,9 +348,11 @@ claims: band: journey stage: 5 title: "An encrypted off-site copy, sealed with a key the operator cannot read" - status: partial - note: "The COPY is real and the key is still unreadable to us — what failed is DAILY and UNBROKEN. The 2026-08-09 note said '18 snapshots, daily, unbroken'; the next snapshot after 2026-08-09 08:30 was 2026-08-21 22:17, put there by hand during the drill. Twelve days, no alarm. Two causes in series: the 2026-08-21 rebuild lost the off-box target (R-193's shape), and after the self-heal restored it EVERY per-app switch was still off, so the first run logged 'backup OK: 0 app(s) backed up, 14s'." + status: walked + note: "DAILY and UNBROKEN, read from the repositories 2026-11-02: demo-hp 14–15 snapshots every day 10-21 → 11-02, Tester 1 4 every day 10-27 → 11-02. The 2026-08 gap's two causes were fixed (R-199, R-193). The key stays the household's." sources: + - evidence: "audits/security-2026-11-02/part5/offsite-demo-hp.txt" + - evidence: "audits/security-2026-11-02/part5/offsite-tester1.txt" - capability-map: "Offsite (restic → Hetzner Storage Box)" - register: "R-199" - register: "R-193" @@ -357,16 +362,19 @@ claims: from: walked reason: "the claim is about a CONTINUING daily copy; a 12-day silent gap was found on 2026-08-21 and nothing reported it" verified: - date: 2026-08-22 - verdict: downgraded + date: 2026-11-03 + verdict: upgraded depth: source-read - id: backup.restore-proof band: journey stage: 5 title: "The backups prove themselves: a restore is actually performed, unattended, on every tier, on both machines" - status: built - note: "STILL GREY, and for a sharper reason than in August. The scheduler runs and the check works — it failed again on 2026-08-21, unprompted, and named the cause exactly: after the reinstall the box presents a different PBS key than its own older archives were sealed with, so those archives cannot be opened at all (R-366). A tier whose archives are orphaned is a different alarm from a tier whose test failed, and only the second is being said." + status: walked + note: "Read 2026-11-02 from the journals of all three boxes: a scheduled restore-test passed on every tier in 10 days — demo-hp local and PBS, Tester 1 local and PBS, the N100 its USB drive and PBS, the newest on 2026-11-02 22:45. R-366's orphaned key was a one-off of the 2026-08 reinstall." sources: + - evidence: "audits/security-2026-11-02/part4/restore-test-demo-hp.txt" + - evidence: "audits/security-2026-11-02/part4/restore-test-tester1.txt" + - evidence: "audits/security-2026-11-02/part4/restore-test-n100.txt" - capability-map: "Restore-proof is UNATTENDED — the scheduler covers EVERY tier" - register: "R-86" - register: "R-366" @@ -377,16 +385,18 @@ claims: decay: "PROOF-DECAY RULE FIRED (first time it has). A receipt EXISTS - architecture/_recovery-inventory-2026-07-28.md carries live journal lines for scheduled restore-tests on both boxes and both tiers - but it is superseded by later observation: demo-hp logged restore_test_failed on 2026-08-05, and the box has since been wiped and reinstalled (2026-08-09). The claim is about a CONTINUING scheduled behaviour, so a 2026-07-28 observation cannot carry it. Stays grey until a scheduled restore-test is seen passing on the rebuilt box. THE CAPABILITY MAP STILL READS PROVEN-LIVE (2026-08-03) AND IS NOW THE THING OUT OF STEP." worse_2026_08_22: "It failed AGAIN, unprompted, on 2026-08-21 21:59 - and the cause is worse than 'untested'. Hub event 3016: the PBS archive of 2026-08-18 could not be restored because the manifest's key does not match the key the rebuilt box now presents. The archives that predate the 2026-08-21 reinstall are UNREADABLE to the machine that made them (R-366). Credit where due: the mechanism caught it and named the key mismatch precisely. The gap is that it is reported as 'a restore test failed' rather than 'your older whole-machine backups cannot be opened on this box'." verified: - date: 2026-08-22 - verdict: downgraded - depth: needs-hardware + date: 2026-11-03 + verdict: upgraded + depth: source-read - id: backup.fill-warning band: journey stage: 5 title: "The customer is warned before a drive fills, per drive, in their own language" - status: partial - note: "The warning is real and was SEEN firing on 2026-08-21 with the right Hungarian copy, naming the drive and the free space. What 'BEFORE' cannot survive is the cadence: the watcher runs once a day at 03:30 plus once at startup (R-363), so a filesystem that fills at 03:31 goes unannounced for ~24 h. Watched live: the 69 GB volume carrying all 40-class app data was filled to 99% and the watcher said nothing, while the backup reserve was already refusing an app per run and telling the hub about it." + status: walked + note: "Walked 2026-11-03 on Tester 1: a data drive taken to 89 %; the watcher (every 10 minutes since R-363) warned at its next tick, 2 minutes later: the household's Hungarian mail names the drive and 5,2 GB free, once. The August gap (a daily check) is gone. The cadence leaves a drive up to 10 minutes." sources: + - evidence: "audits/security-2026-11-02/part5/fill-walk.log" + - evidence: "audits/security-2026-11-02/part5/mail-fill-household.txt" - capability-map: "The customer is warned BEFORE a filesystem fills" - register: "R-167" - register: "R-363" @@ -396,8 +406,8 @@ claims: from: walked reason: "a daily check cannot carry the word BEFORE; observed silent for the whole window a filesystem sat at 99%" verified: - date: 2026-08-22 - verdict: downgraded + date: 2026-11-03 + verdict: upgraded depth: source-read - id: backup.sikeres band: journey @@ -436,29 +446,33 @@ claims: band: journey stage: 6 title: "Failures reach the operator by email, one mail per run, every failing app named" - status: built - note: "CONFIRMED against source: backup_run_failures digest is allowlisted (api/handler.go:1837), operator-only (dispatcher.go:423), templated (templates.go:48); recovery_unit_capture_failed is record-only (dispatcher.go:376). NOTE: R-182's register row still describes the PRE-FIX behaviour — see R-289." + status: walked + note: "Seen in the mailbox: „Tester4: 3 of 8 apps failed to back up (manual run)\" names immich, nextcloud and paperless-ngx in one mail (2026-10-20); seven of this shape in 60 days." sources: + - evidence: "audits/security-2026-11-02/part5/mail-operator-digest.txt" - capability-map: "A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run" - register: "R-182" changed: from: walked reason: "the digest is wired and source-verified, but no run of it has been observed delivering" verified: - date: 2026-08-09 - verdict: downgraded + date: 2026-11-03 + verdict: upgraded depth: source-read - id: fault.customer-email band: journey stage: 6 title: "Failures reach the customer" - status: built - note: "The customer leg is built; today's log shows customer-channel rows skipped as operator_only." + status: walked + note: "Seen in the mailbox: the crash mail and the fill warning to tester1@ (2026-11-02/03), „A szerver nem elérhető!\" to tester4@ (2026-10-24), each in Hungarian. It needs an address on the household's page (Tester 1 had none)." sources: + - evidence: "audits/security-2026-11-02/part5/mail-crash-household.txt" + - evidence: "audits/security-2026-11-02/part5/mail-fill-household.txt" + - evidence: "audits/security-2026-11-02/part5/mail-operator-digest.txt" - capability-map: "App crashes → customer notified (one event per transition, no flapping spam)" verified: - date: 2026-08-09 - verdict: confirmed + date: 2026-11-03 + verdict: upgraded depth: source-read - id: fault.already-paired band: journey @@ -573,26 +587,33 @@ claims: - id: fail.drive-filling band: failures title: "A drive is filling up" - status: built + status: walked + note: "Same walk as backup.fill-warning: warned before it filled; the file removed, nothing else was needed." sources: + - evidence: "audits/security-2026-11-02/part5/fill-walk.log" - register: "R-167" changed: from: walked reason: "no walk document cited" verified: - date: 2026-08-09 - verdict: downgraded - depth: register+map + date: 2026-11-03 + verdict: upgraded + depth: source-read - id: fail.app-crash band: failures - title: "An app crashes — the email leg has never been confirmed end to end" - status: built - note: "Consistent with fault.operator-email: the digest is wired but its delivery is unobserved." + title: "An app crashes — the box stops it and both the operator and the household get a mail" + status: walked + note: "Walked 2026-11-02 on Tester 1: kill -9 of a throwaway app, 15 s apart; at 6 restarts in 10 minutes the box stopped and held it, the operator's mail arrived in 1 s; with an address on the household's page the Hungarian mail arrived too. The household's Start lifted it. One flaw: the mail ends with raw JSON (R-1363)." sources: + - evidence: "audits/security-2026-11-02/part5/crash-walk.log" + - evidence: "audits/security-2026-11-02/part5/crash-walk-customer.log" + - evidence: "audits/security-2026-11-02/part5/mail-crash-operator.txt" + - evidence: "audits/security-2026-11-02/part5/mail-crash-household.txt" + - register: "R-1363" - capability-map: "App crashes → customer notified (one event per transition, no flapping spam)" verified: - date: 2026-08-09 - verdict: confirmed + date: 2026-11-03 + verdict: upgraded depth: source-read - id: fail.power-cut band: failures @@ -700,28 +721,34 @@ claims: depth: register+map - id: fail.internet-down band: failures - title: "The internet goes down — built, never walked" - status: built - note: "Needs a running box with WAN pulled." + title: "The internet goes down" + status: walked + note: "Walked 2026-11-03 on Tester 1: packets to the outside dropped on host and guest for 13 minutes, the line's route left in place. The dashboard answered on the home network, 10 apps ran, the public name gave 530; 60 s after the line came back the public name worked by itself, nothing to do. Slow: the Backups page took over 2 minutes (R-1362). An earlier try removed the route: the agent's self-heal put it back in about 100 s." sources: + - evidence: "audits/security-2026-11-02/part5/outage-wan.log" + - evidence: "audits/security-2026-11-02/part5/outage-wan-3min.txt" + - register: "R-1362" - capability-map: "LAN access when internet is down (lan_resolver)" verified: - date: 2026-08-09 - verdict: needs-hardware - depth: needs-hardware + date: 2026-11-03 + verdict: upgraded + depth: source-read - id: fail.hub-down band: failures title: "The hub is down" - status: built + status: partial + note: "Walked 2026-11-03 on Tester 1 (the hub's address dropped for 13 minutes; DooPlex untouched): the box works on — dashboard, apps, an install from the cached catalog, a crash-loop stop. What fails: every event sent in that time is lost after 9 s and nothing re-sends it, so the operator never heard of the held app (R-1361)." sources: + - evidence: "audits/security-2026-11-02/part5/outage-hub.log" + - register: "R-1361" - capability-map: "Config/state change round-trips in seconds (hub↔box immediacy" changed: from: walked reason: "no walk document cited" verified: - date: 2026-08-09 - verdict: downgraded - depth: register+map + date: 2026-11-03 + verdict: upgraded + depth: source-read - id: fail.broken-release band: failures title: "We ship a broken release — the guard is missing" diff --git a/documentation/audits/security-2026-11-02/part1/night-2026-11-03.log b/documentation/audits/security-2026-11-02/part1/night-2026-11-03.log new file mode 100644 index 00000000..3c8b28f5 --- /dev/null +++ b/documentation/audits/security-2026-11-02/part1/night-2026-11-03.log @@ -0,0 +1,295 @@ +2026-11-03T03:40:21.3774082Z felhom-gates-runner-2(version:v0.6.1) received task 2593 of job watch, be triggered by event: schedule +2026-11-03T03:40:21.3794120Z workflow prepared +2026-11-03T03:40:21.3795891Z evaluating expression 'always()' +2026-11-03T03:40:21.3797225Z expression 'always()' evaluated to 'true' +2026-11-03T03:40:21.3807020Z evaluating expression '' +2026-11-03T03:40:21.3808038Z expression '' evaluated to 'true' +2026-11-03T03:40:21.3808331Z ⭐ Run Main Fetch main (latest, for the memory) and the two sibling repos +2026-11-03T03:40:21.3814363Z Wrote command \n\necho "WATCH_STARTED_AT=$(date +%s)" >> "$GITHUB_ENV"\nbase=http://gitea.gitea-system.svc.cluster.local:3000/admin\ngit init -q . && git remote add origin $base/felhom.eu.git\ngit fetch -q --depth 1 origin main && git checkout -q -B main FETCH_HEAD\necho "felhom.eu main at $(git rev-parse --short=12 HEAD)"\nfor r in felhom-controller app-catalog-felhom.eu; do\n git init -q ../$r && git -C ../$r remote add origin $base/$r.git\n git -C ../$r fetch -q --depth 1 origin main && git -C ../$r checkout -q FETCH_HEAD\n echo "$r at $(git -C ../$r rev-parse --short=12 HEAD)"\ndone\n\n\n to 'workflow/0.sh' +2026-11-03T03:40:21.3816737Z ::group::Run echo "WATCH_STARTED_AT=$(date +%s)" >> "$GITHUB_ENV" +2026-11-03T03:40:21.3817177Z echo "WATCH_STARTED_AT=$(date +%s)" >> "$GITHUB_ENV" +2026-11-03T03:40:21.3817592Z base=http://gitea.gitea-system.svc.cluster.local:3000/admin +2026-11-03T03:40:21.3817833Z git init -q . && git remote add origin $base/felhom.eu.git +2026-11-03T03:40:21.3818070Z git fetch -q --depth 1 origin main && git checkout -q -B main FETCH_HEAD +2026-11-03T03:40:21.3818305Z echo "felhom.eu main at $(git rev-parse --short=12 HEAD)" +2026-11-03T03:40:21.3818537Z for r in felhom-controller app-catalog-felhom.eu; do +2026-11-03T03:40:21.3818819Z git init -q ../$r && git -C ../$r remote add origin $base/$r.git +2026-11-03T03:40:21.3819259Z git -C ../$r fetch -q --depth 1 origin main && git -C ../$r checkout -q FETCH_HEAD +2026-11-03T03:40:21.3819497Z echo "$r at $(git -C ../$r rev-parse --short=12 HEAD)" +2026-11-03T03:40:21.3819716Z done +2026-11-03T03:40:21.3819996Z shell: bash --noprofile --norc -e -o pipefail {0} +2026-11-03T03:40:21.3820238Z ::endgroup:: +2026-11-03T03:40:34.2379112Z felhom.eu main at 17820abbe363 +2026-11-03T03:40:37.5306666Z felhom-controller at 13e34d4ed4c3 +2026-11-03T03:40:38.3913639Z app-catalog-felhom.eu at c760fca93898 +2026-11-03T03:40:38.3999626Z ::group::Run set -u +2026-11-03T03:40:38.4000281Z set -u +2026-11-03T03:40:38.4000518Z mkdir -p documentation/security-watch +2026-11-03T03:40:38.4000738Z mode="${MODE:-run}" +2026-11-03T03:40:38.4001041Z [ "${GITHUB_EVENT_NAME:-}" = "schedule" ] && mode=run +2026-11-03T03:40:38.4001299Z echo "mode: $mode" +2026-11-03T03:40:38.4001482Z case "$mode" in +2026-11-03T03:40:38.4001676Z test-mail) python3 scripts/security_watch.py --test-mail; exit $? ;; +2026-11-03T03:40:38.4002145Z dry) extra="--no-mail" ; state=/tmp/state-dry.json +2026-11-03T03:40:38.4002374Z cp documentation/security-watch/state.json "$state" 2>/dev/null || true ;; +2026-11-03T03:40:38.4002591Z seed) extra="--seed" ; state=documentation/security-watch/state.json ;; +2026-11-03T03:40:38.4002905Z run) extra="" ; state=documentation/security-watch/state.json ;; +2026-11-03T03:40:38.4003179Z *) echo "unknown mode $mode"; exit 2 ;; +2026-11-03T03:40:38.4003380Z esac +2026-11-03T03:40:38.4003553Z python3 scripts/security_watch.py --state="$state" --catalog=../app-catalog-felhom.eu \ +2026-11-03T03:40:38.4003924Z --controller=../felhom-controller --json=/tmp/watch-results.json $extra +2026-11-03T03:40:38.4004172Z shell: bash --noprofile --norc -e -o pipefail {0} +2026-11-03T03:40:38.4004423Z env: +2026-11-03T03:40:38.4015597Z RESEND_API_KEY: *** +2026-11-03T03:40:38.4015882Z GH_ADVISORY_TOKEN: *** +2026-11-03T03:40:38.4016085Z MODE: +2026-11-03T03:40:38.4016291Z FELHOM_WHEEL_FETCH: 1 +2026-11-03T03:40:38.4016553Z ::endgroup:: +2026-11-03T03:40:38.4083949Z mode: run +2026-11-03T03:40:38.6736065Z /root/.cache/act/76abc955bc1c1a2a/hostexecutor/scripts/security_watch.py:1024: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC). +2026-11-03T03:40:38.6737029Z today = datetime.date.fromisoformat(opts["today"]) if "today" in opts else datetime.datetime.utcnow().date() +2026-11-03T03:40:38.7287299Z security watch 2026-11-03: 197 pins (11 box/build, 97 app, 89 DooPlex); GitHub key PRESENT — every repo's advisories tonight +2026-11-03T03:40:50.3040389Z RUNTIME-READ traefik:v3.7.17 7 s ok +2026-11-03T03:41:41.6296265Z RUNTIME-READ actualbudget/actual-server:26.11.0 18 s ok +2026-11-03T03:41:45.9805770Z RUNTIME-READ lscr.io/linuxserver/bookstack:26.09.2 16 s ok +2026-11-03T03:42:06.6399759Z RUNTIME-READ gitea/gitea:28.3.0 7 s ok +2026-11-03T03:42:31.5646844Z RUNTIME-READ grafana/grafana:13.2.5 30 s ok +2026-11-03T03:43:20.0272268Z RUNTIME-READ n8nio/n8n:2.46.4 47 s ok +2026-11-03T03:43:25.9441227Z RUNTIME-READ vaultwarden/server:1.37.5-alpine 9 s ok +2026-11-03T03:44:30.3181250Z RUNTIME-READ louislam/uptime-kuma:2.5.9 77 s ok +2026-11-03T03:44:35.6053776Z RUNTIME-READ mysql:8.4 24 s ok +2026-11-03T03:44:52.4270736Z ADVISORY doop actualbudget/actual-server 26.10.0 GHSA-p7p2-pm6f-rg6x (fixed 26.11.0); CVE-2026-111497 (fixed 24.22.1) +2026-11-03T03:44:52.4271671Z ADVISORY doop advplyr/audiobookshelf 2.37.1 CVE-2026-111497 (fixed 24.22.1) +2026-11-03T03:44:52.4272667Z ADVISORY box cloudflare/cloudflared 2026.10.4 feed:cloudflare-blog:01M5DK5J81T0VC8YTBR2FKD8A3 (fixed read the notice); vulnnotes:cloudflared@2026.10.6:VULN-184686 (fixed 2026.10.6); vulnnotes:cloudflared@2026.10.5:VULN-183456 (fixed 2026.10.5) +2026-11-03T03:44:52.4273109Z ADVISORY doop codewithcj/sparkyfitness v1.9.0 release:CodeWithCJ/SparkyFitness@v1.10.0 (fixed v1.10.0) +2026-11-03T03:44:52.4273409Z ADVISORY doop codewithcj/sparkyfitness_server v1.9.0 release:CodeWithCJ/SparkyFitness@v1.10.0 (fixed v1.10.0); CVE-2026-111497 (fixed 24.22.1) +2026-11-03T03:44:52.4273860Z ADVISORY doop crocodilestick/calibre-web-automated dev CVE-2026-111497 (fixed 1.2.1) +2026-11-03T03:44:52.4274144Z ADVISORY doop docker.io/calcom/cal.com v6.2.0 CVE-2026-111497 (fixed 22.23.4, 24.22.1, 26.12.1) +2026-11-03T03:44:52.4274491Z ADVISORY doop flomp/wanderer-db v0.21.0 GHSA-vprp-gr6m-j375 (fixed v0.21.2); GHSA-3xp7-7492-j2q6 (fixed v0.21.2); GHSA-j9m4-8qwf-jvx6 (fixed v0.21.2); GHSA-fprh-3pp9-68m9 (fixed v0.21.2) +2026-11-03T03:44:52.4274905Z ADVISORY doop flomp/wanderer-web v0.21.0 GHSA-vprp-gr6m-j375 (fixed v0.21.2); GHSA-3xp7-7492-j2q6 (fixed v0.21.2); GHSA-j9m4-8qwf-jvx6 (fixed v0.21.2); GHSA-fprh-3pp9-68m9 (fixed v0.21.2); … 1 more +2026-11-03T03:44:52.4275291Z ADVISORY doop getmeili/meilisearch v1.11.3 release:meilisearch/meilisearch@v1.55.1 (fixed v1.55.1); release:meilisearch/meilisearch@v1.56.1 (fixed v1.56.1) +2026-11-03T03:44:52.4275741Z ADVISORY doop ghcr.io/gethomepage/homepage v1.13.2 GHSA-669x-4pg4-w24r (fixed 2.2.0); CVE-2026-111497 (fixed 22.23.4) +2026-11-03T03:44:52.4276028Z ADVISORY doop ghcr.io/immich-app/immich-machine-learning v2.7.5 GHSA-q89f-h332-8q2h (fixed v3.2.4); GHSA-h5w4-vjv4-9r5q (fixed a version after 3.1.0 (the notice names none)); GHSA-g4w3-c888-cr93 (fixed v3.2.0); GHSA-72rm-x7xr-v679 (fixed v3.3.0); … 7 more +2026-11-03T03:44:52.4276413Z ADVISORY doop ghcr.io/immich-app/immich-server v2.7.5 GHSA-q89f-h332-8q2h (fixed v3.2.4); GHSA-h5w4-vjv4-9r5q (fixed a version after 3.1.0 (the notice names none)); GHSA-g4w3-c888-cr93 (fixed v3.2.0); GHSA-72rm-x7xr-v679 (fixed v3.3.0); … 8 more +2026-11-03T03:44:52.4276863Z ADVISORY doop ghcr.io/lukegus/termix release-2.9.2 CVE-2026-111497 (fixed 26.12.1) +2026-11-03T03:44:52.4277115Z ADVISORY doop ghcr.io/paperless-ngx/paperless-ngx 3.3.4 release:paperless-ngx/paperless-ngx@v3.4.0 (fixed v3.4.0); CVE-2026-111497 (fixed 1.2.1) +2026-11-03T03:44:52.4277441Z ADVISORY doop gitea/gitea 1.26.2 GHSA-mf6p-vqpw-3r7x (fixed 28.3.0); GHSA-32h8-5xqg-h466 (fixed 28.3.0); GHSA-9v6p-xpcp-7488 (fixed 28.3.0); GHSA-6h2f-c8jm-84hc (fixed 28.3.0); … 145 more +2026-11-03T03:44:52.4277770Z ADVISORY doop grafana/grafana 13.2.4 GHSA-8xmm-x63g-f6xv (fixed no fixed version named yet); GHSA-4724-7jwc-3fpw (fixed the range is unreadable — check by hand); GHSA-2x6g-h2hg-rq84 (fixed the range is unreadable — check by hand); GHSA-8wjh-59cw-9xh4 (fixed no fixed version named yet); … 2 more +2026-11-03T03:44:52.4278189Z ADVISORY box gtstef/filebrowser 1.5.8-stable GHSA-892f-9h5p-mfc8 (fixed 2.1.0-stable, 2.1.4-beta); GHSA-gfgm-886h-6r47 (fixed 2.0.2-stable, 2.1.4-beta); GHSA-wxj5-g3qp-g732 (fixed 2.0.2-stable, 2.1.3-beta); GHSA-3j46-5w28-f8p2 (fixed 2.1.0-stable, 2.1.3-beta); … 32 more +2026-11-03T03:44:52.4278635Z ADVISORY doop louislam/uptime-kuma 2.5.6 GHSA-86ch-2247-gr56 (fixed 2.5.9); GHSA-v2p3-6f5h-6hhf (fixed 2.5.8); GHSA-qjxc-h5jf-c7rj (fixed the range is unreadable — check by hand); release:louislam/uptime-kuma@2.5.7 (fixed 2.5.7); … 1 more +2026-11-03T03:44:52.4279095Z ADVISORY doop outlinewiki/outline 1.11.0 GHSA-x4w3-9p9j-cprh (fixed 1.12.0); GHSA-rq53-m97j-f2pg (fixed 1.12.0); GHSA-h6ff-rw9x-5mq2 (fixed 1.12.0); GHSA-rvj7-2gww-6xfq (fixed 1.12.0); … 3 more +2026-11-03T03:44:52.4279403Z ADVISORY doop redis 8.10.2 GHSA-c8h9-259x-jff4 (fixed the range is unreadable — check by hand); GHSA-8ghh-qpmp-7826 (fixed no fixed version named yet); GHSA-93m2-935m-8rj3 (fixed no fixed version named yet); GHSA-m8fj-85cg-7vhp (fixed the range is unreadable — check by hand); … 8 more +2026-11-03T03:44:52.4279755Z ADVISORY doop rommapp/romm 5.4.0 release:rommapp/romm@5.4.1 (fixed 5.4.1); CVE-2026-111497 (fixed 24.22.1) +2026-11-03T03:44:52.4280104Z ADVISORY doop vabene1111/recipes 2.6.15 CVE-2026-111497 (fixed 24.22.1) +2026-11-03T03:44:52.4280355Z ADVISORY doop vaultwarden/server 1.37.4 GHSA-x9cq-f669-vrr4 (fixed 1.37.5); GHSA-9pjj-f9g5-3x4c (fixed 1.37.5) +2026-11-03T03:44:52.4280613Z ADVISORY app calcom/cal.com v6.2.0 GHSA-p3f6-52gv-cj7m (fixed the range is unreadable — check by hand); GHSA-cpf2-q635-xrwx (fixed the range is unreadable — check by hand); CVE-2026-111497 (fixed 22.23.4, 24.22.1, 26.12.1) +2026-11-03T03:44:52.4280964Z ADVISORY app codewithcj/sparkyfitness v0.17.3 GHSA-v883-jx9h-qf29 (fixed a version after 1.7.2 (the notice names none)); GHSA-cmmf-53wf-r5gv (fixed a version after 1.6.5 (the notice names none)); GHSA-p8cf-g8rv-gvrq (fixed a version after 1.6.5 (the notice names none)) +2026-11-03T03:44:52.4281497Z ADVISORY app codewithcj/sparkyfitness_server v0.17.3 GHSA-v883-jx9h-qf29 (fixed a version after 1.7.2 (the notice names none)); GHSA-cmmf-53wf-r5gv (fixed a version after 1.6.5 (the notice names none)); GHSA-p8cf-g8rv-gvrq (fixed a version after 1.6.5 (the notice names none)); CVE-2026-111497 (fixed 24.22.1) +2026-11-03T03:44:52.4281886Z ADVISORY app crocodilestick/calibre-web-automated v4.0.8 CVE-2026-111497 (fixed 1.2.1) +2026-11-03T03:44:52.4282171Z ADVISORY app f0rc3/gokapi v1.9.6 GHSA-45vh-rpc8-hxpp (fixed 2.2.4); GHSA-qwc6-vc2v-2ggj (fixed 2.2.4); GHSA-j6jp-78w8-34x6 (fixed 2.2.4); GHSA-m2hx-wjxc-9fp4 (fixed 2.2.3); … 6 more +2026-11-03T03:44:52.4282847Z ADVISORY app getmeili/meilisearch v1.41.0 release:meilisearch/meilisearch@v1.55.1 (fixed v1.55.1); release:meilisearch/meilisearch@v1.56.1 (fixed v1.56.1) +2026-11-03T03:44:52.4283201Z ADVISORY app getmeili/meilisearch v1.36.0 release:meilisearch/meilisearch@v1.55.1 (fixed v1.55.1); release:meilisearch/meilisearch@v1.56.1 (fixed v1.56.1) +2026-11-03T03:44:52.4283653Z ADVISORY app ghcr.io/immich-app/immich-machine-learning v3.3.5 GHSA-8244-8vpr-vp9c (fixed the range is unreadable — check by hand) +2026-11-03T03:44:52.4283988Z ADVISORY app ghcr.io/immich-app/immich-server v3.3.5 GHSA-8244-8vpr-vp9c (fixed the range is unreadable — check by hand) +2026-11-03T03:44:52.4284583Z ADVISORY app ghcr.io/karakeep-app/karakeep 0.34.1 GHSA-pp34-2c9r-mchw (fixed no fixed version named yet) +2026-11-03T03:44:52.4284909Z ADVISORY app ghcr.io/lukegus/termix 26.10.5 GHSA-r9gw-3w87-mhh7 (fixed no fixed version named yet); release:Termix-SSH/Termix@v26.10.6 (fixed v26.10.6) +2026-11-03T03:44:52.4285306Z ADVISORY app ghcr.io/mealie-recipes/mealie v3.29.0 release:mealie-recipes/mealie@v3.30.0 (fixed v3.30.0) +2026-11-03T03:44:52.4285694Z ADVISORY app ghcr.io/paperless-ngx/paperless-ngx 2.20.15 GHSA-c8f3-pqr2-rr24 (fixed 3.3.2); GHSA-2jhj-xqrq-rmrq (fixed 3.1.2); CVE-2026-111497 (fixed 1.1.1) +2026-11-03T03:44:52.4286127Z ADVISORY app ghcr.io/tandoorrecipes/recipes 2.7.2 GHSA-c7mc-c9jr-38hf (fixed the range is unreadable — check by hand); GHSA-cqj3-64qw-4w52 (fixed the range is unreadable — check by hand) +2026-11-03T03:44:52.4286588Z ADVISORY app gitea/gitea 28.3.0 GHSA-rqhx-647v-wx32 (fixed no fixed version named yet) +2026-11-03T03:44:52.4287029Z ADVISORY app glanceapp/glance v0.8.5 GHSA-r225-jhr6-68cc (fixed >= 0.8.7) +2026-11-03T03:44:52.4287375Z ADVISORY app grafana/grafana 13.2.5 GHSA-8xmm-x63g-f6xv (fixed no fixed version named yet); GHSA-4724-7jwc-3fpw (fixed the range is unreadable — check by hand); GHSA-2x6g-h2hg-rq84 (fixed the range is unreadable — check by hand); GHSA-8wjh-59cw-9xh4 (fixed no fixed version named yet); … 1 more +2026-11-03T03:44:52.4287943Z ADVISORY app louislam/uptime-kuma 2.5.9 GHSA-qjxc-h5jf-c7rj (fixed the range is unreadable — check by hand) +2026-11-03T03:44:52.4288340Z ADVISORY app n8nio/n8n 2.46.4 GHSA-2x68-96jw-5r6c (fixed >= 3.0.1); GHSA-9p7v-45m2-wxgj (fixed >= 3.0.1); GHSA-mx4r-96jx-v278 (fixed >= 3.0.1); GHSA-4v22-g448-h4rg (fixed >= 3.0.1); … 9 more +2026-11-03T03:44:52.4288882Z ADVISORY app privatebin/pdo 2.1.0 GHSA-c4gc-8mcm-hh65 (fixed 2.1.1); GHSA-hw9p-r7fq-8mvr (fixed 2.1.1) +2026-11-03T03:44:52.4289157Z ADVISORY app rommapp/romm 5.4.1 CVE-2026-111497 (fixed 24.22.1) +2026-11-03T03:44:52.4289540Z BEHIND doop ghcr.io/alam00000/bentopdf 2.8.10 newest 2.8.11 +2026-11-03T03:44:52.4289838Z BEHIND doop ghcr.io/diced/zipline 4.8.1 newest 4.8.2 +2026-11-03T03:44:52.4290214Z BEHIND app freikin/dawarich 1.19.0 newest 1.20.1 +2026-11-03T03:44:52.4290478Z BEHIND app ghcr.io/alam00000/bentopdf v2.8.10 newest v2.8.11 +2026-11-03T03:44:52.4290753Z BEHIND app ghcr.io/alexta69/metube 2026.10.27 newest 2026.11.01 +2026-11-03T03:44:52.4291099Z BEHIND app ghcr.io/hargata/lubelogger v1.7.3 newest v1.7.4 +2026-11-03T03:44:52.4291336Z BEHIND app ghcr.io/kozea/radicale 3.8.1 newest 3.9.0 +2026-11-03T03:44:52.4291560Z BEHIND app gotson/komga 1.28.0 newest 1.29.2 +2026-11-03T03:44:52.4291838Z BEHIND app kimai/kimai2 2.67.0 newest 2.71.0 +2026-11-03T03:44:52.4292136Z BEHIND app lscr.io/linuxserver/code-server 4.143.1 newest 4.144.0 +2026-11-03T03:44:52.4292412Z BEHIND app lukevella/rallly 4.16.3 newest 4.16.4 +2026-11-03T03:44:52.4293002Z BEHIND app nextcloud 34.0.4-apache newest 34.0.5-apache +2026-11-03T03:44:52.4293302Z BEHIND app nginx 1.30.5-alpine newest 1.31.8-alpine +2026-11-03T03:44:52.4293598Z BEHIND app qbittorrentofficial/qbittorrent-nox 5.2.4-1 newest 5.3.0-2 +2026-11-03T03:44:52.4293847Z GAP buil alpine 3.24.2 newest 3.24.3 +2026-11-03T03:44:52.4294069Z GAP buil alpine 3.24.2 newest 3.24.3 +2026-11-03T03:44:52.4294428Z GAP doop alpine/k8s 1.37.1 newest 1.37.2 +2026-11-03T03:44:52.4294670Z GAP doop arcadiatechnology/crafty-4 4.11.0 +2026-11-03T03:44:52.4294905Z GAP doop bhgedigital/envsubst latest +2026-11-03T03:44:52.4295298Z GAP doop busybox 1.38 +2026-11-03T03:44:52.4295544Z GAP doop codercom/code-server 4.143.1 newest 4.144.0 +2026-11-03T03:44:52.4295874Z GAP buil debian 13.7-slim +2026-11-03T03:44:52.4296313Z GAP buil debian 13.7-slim +2026-11-03T03:44:52.4296799Z GAP doop docker.io/bitnamilegacy/postgresql 17.6.0-debian-12-r3 +2026-11-03T03:44:52.4297382Z GAP doop docker.io/library/nextcloud 34.0.5-apache +2026-11-03T03:44:52.4297795Z GAP doop docker.io/valkey/valkey 9.2-alpine +2026-11-03T03:44:52.4298298Z GAP doop filebrowser/filebrowser v2.63.23 +2026-11-03T03:44:52.4298735Z GAP doop ghcr.io/cloudnative-pg/postgresql 17.2 newest 17.11 +2026-11-03T03:44:52.4299240Z GAP doop ghcr.io/headlamp-k8s/headlamp v0.46.0 +2026-11-03T03:44:52.4299607Z GAP doop ghcr.io/immich-app/postgres 16-vectorchord0.3.0 +2026-11-03T03:44:52.4300100Z GAP doop ghcr.io/jimmidyson/configmap-reload v0.15.0 +2026-11-03T03:44:52.4300469Z GAP doop ghcr.io/onedr0p/exportarr v2.3.0 +2026-11-03T03:44:52.4300876Z GAP doop ghcr.io/seriousm4x/upsnap 5 +2026-11-03T03:44:52.4301320Z GAP doop ghcr.io/stakater/reloader v1.4.22 newest v1.4.23 +2026-11-03T03:44:52.4301730Z GAP doop ghcr.io/tautulli/tautulli v2.18.2 +2026-11-03T03:44:52.4302084Z GAP doop gitea.dooplex.hu/admin/felhom-act-runner 0.1.0 +2026-11-03T03:44:52.4302449Z GAP doop gitea.dooplex.hu/admin/jarr latest +2026-11-03T03:44:52.4302898Z GAP doop gitea.dooplex.hu/admin/recipe-importer latest +2026-11-03T03:44:52.4303390Z GAP doop gitea.dooplex.hu/admin/revfulop-calendar 1.6.2 +2026-11-03T03:44:52.4303749Z GAP doop gitea.dooplex.hu/admin/sparkyfitness-export 0.3.0 +2026-11-03T03:44:52.4304148Z GAP doop gitea.dooplex.hu/admin/wan-probe 0.1.0 +2026-11-03T03:44:52.4304533Z GAP doop griefed/serverpackcreator 8.1.2 +2026-11-03T03:44:52.4304919Z GAP doop gtstef/filebrowser 1.5.8-stable +2026-11-03T03:44:52.4305301Z GAP doop guacamole/guacamole 1.6.0 +2026-11-03T03:44:52.4305861Z GAP doop guacamole/guacd 1.6.0 +2026-11-03T03:44:52.4306551Z GAP doop index.docker.io/plexinc/pms-docker 1.43.5.11029-6bf8af3f0 +2026-11-03T03:44:52.4306960Z GAP doop itzg/mc-backup 2026.10.1 newest 2026.10.3 +2026-11-03T03:44:52.4307271Z GAP doop itzg/minecraft-server 2026.10.0-java21 newest 2026.10.1-java17 +2026-11-03T03:44:52.4307511Z GAP doop linuxserver/bookstack 26.09.20260929 +2026-11-03T03:44:52.4307987Z GAP doop linuxserver/prowlarr version-2.6.5.5623 +2026-11-03T03:44:52.4308403Z GAP doop linuxserver/qbittorrent 5.3.0 +2026-11-03T03:44:52.4310321Z GAP doop linuxserver/radarr version-6.4.4.10685 +2026-11-03T03:44:52.4311184Z GAP doop linuxserver/sonarr version-4.0.20.3014 +2026-11-03T03:44:52.4311489Z GAP doop mariadb 11 +2026-11-03T03:44:52.4311836Z GAP doop mikefarah/yq 4.54.1 +2026-11-03T03:44:52.4312076Z GAP doop minio/minio latest +2026-11-03T03:44:52.4312694Z GAP doop mongodb/mongodb-community-server 8.0.5-ubuntu2204 newest 8.3.13-ubuntu2204 +2026-11-03T03:44:52.4312983Z GAP doop msdeluise/plant-it-server 0.10.0 +2026-11-03T03:44:52.4313519Z GAP doop mysql 8.4 +2026-11-03T03:44:52.4313733Z GAP doop nicolaka/netshoot latest +2026-11-03T03:44:52.4313948Z GAP doop postgis/postgis 16-3.5 +2026-11-03T03:44:52.4314211Z GAP doop postgres 15-alpine +2026-11-03T03:44:52.4314575Z GAP doop postgres 16-alpine +2026-11-03T03:44:52.4314795Z GAP doop privatebin/nginx-fpm-alpine 2.0.6 newest 2.1.0 +2026-11-03T03:44:52.4315028Z GAP doop prom/alertmanager v0.34.1 newest v0.34.2 +2026-11-03T03:44:52.4315317Z GAP doop prom/node-exporter v1.12.1 +2026-11-03T03:44:52.4315646Z GAP doop prom/prometheus v3.15.0 newest v3.15.1 +2026-11-03T03:44:52.4315897Z GAP doop prometheuscommunity/smartctl-exporter v0.15.0 +2026-11-03T03:44:52.4316264Z GAP doop python 3.15-bookworm +2026-11-03T03:44:52.4316489Z GAP doop quay.io/jetstack/version-checker v0.11.0 +2026-11-03T03:44:52.4316709Z GAP doop quay.io/prometheus/blackbox-exporter v0.29.1 +2026-11-03T03:44:52.4316928Z GAP doop renovate/renovate 44.181.1 newest 44.206.2 +2026-11-03T03:44:52.4317180Z GAP doop tailscale/tailscale v1.104.2 newest v1.104.4 +2026-11-03T03:44:52.4317426Z GAP app emby/embyserver 4.11.0.4 newest 4.11.0.13 +2026-11-03T03:44:52.4317644Z GAP app ghcr.io/immich-app/postgres 16-vectorchord0.4.3-pg +2026-11-03T03:44:52.4317870Z GAP app ghcr.io/karakeep-app/karakeep-chrome 151.0.7922.47-r1 +2026-11-03T03:44:52.4318112Z GAP app mariadb 12.3 +2026-11-03T03:44:52.4318325Z GAP app mariadb 11.4 newest 11.8 +2026-11-03T03:44:52.4318538Z GAP app mariadb 11.8 +2026-11-03T03:44:52.4318760Z GAP app mariadb 12.3 +2026-11-03T03:44:52.4318971Z GAP app mariadb 11.8 +2026-11-03T03:44:52.4319178Z GAP app msdeluise/plant-it 0.10.0 +2026-11-03T03:44:52.4319391Z GAP app plexinc/pms-docker 1.43.5.11029-6bf8af3f0 +2026-11-03T03:44:52.4319667Z GAP app postgis/postgis 16-3.5-alpine +2026-11-03T03:44:52.4319907Z GAP app postgis/postgis 17-3.5-alpine +2026-11-03T03:44:52.4320128Z GAP app postgres 18-alpine +2026-11-03T03:44:52.4320353Z GAP app postgres 17-alpine +2026-11-03T03:44:52.4320569Z GAP app postgres 18-alpine +2026-11-03T03:44:52.4320780Z GAP app postgres 18-alpine +2026-11-03T03:44:52.4320990Z GAP app postgres 18-alpine +2026-11-03T03:44:52.4321198Z GAP app postgres 18-alpine +2026-11-03T03:44:52.4321406Z GAP app postgres 18-alpine +2026-11-03T03:44:52.4321626Z GAP app postgres 17-alpine +2026-11-03T03:44:52.4321848Z GAP app postgres 16-alpine +2026-11-03T03:44:52.4322061Z GAP app registry.gitlab.com/crafty-controller/crafty-4 4.11.0 +2026-11-03T03:44:52.4322293Z FLOATING doop ghcr.io/seanmorley15/adventurelog-backend latest +2026-11-03T03:44:52.4322539Z FLOATING doop ghcr.io/seanmorley15/adventurelog-frontend latest +2026-11-03T03:44:52.4322791Z FLOATING doop nginx 1.31-alpine +2026-11-03T03:44:52.4323001Z FLOATING doop redis 8-alpine +2026-11-03T03:44:52.4323247Z FLOATING doop redis 8.10-alpine +2026-11-03T03:44:52.4323502Z FLOATING app ghcr.io/claperco/claper 2.5 +2026-11-03T03:44:52.4323973Z FLOATING app ghcr.io/thomiceli/opengist 1.15 +2026-11-03T03:44:52.4324204Z FLOATING app jellyfin/jellyfin 12.4 +2026-11-03T03:44:52.4324425Z FLOATING app lscr.io/linuxserver/prowlarr version-2.6.5.5623 +2026-11-03T03:44:52.4324649Z FLOATING app lscr.io/linuxserver/sabnzbd version-5.1.3 +2026-11-03T03:44:52.4324873Z FLOATING app lscr.io/linuxserver/transmission version-4.1.3-r0 +2026-11-03T03:44:52.4325097Z FLOATING app nzbgetcom/nzbget v26.3 +2026-11-03T03:44:52.4325326Z FLOATING app redis 7.4-alpine +2026-11-03T03:44:52.4325544Z FLOATING app redis 7-alpine +2026-11-03T03:44:52.4325772Z FLOATING app redis 7-alpine +2026-11-03T03:44:52.4325980Z FLOATING app redis 7-alpine +2026-11-03T03:44:52.4326189Z FLOATING app redis 7-alpine +2026-11-03T03:44:52.4326410Z FLOATING app redis 7-alpine +2026-11-03T03:44:52.4326625Z FLOATING app redis 7-alpine +2026-11-03T03:44:52.4326869Z FLOATING app wger/server 2.7 +2026-11-03T03:44:52.4327096Z SELF box gitea.dooplex.hu/admin/felhom-samba 1.2.0 +2026-11-03T03:44:52.4327313Z SELF app gitea.dooplex.hu/admin/recipe-importer v0.9.11 +2026-11-03T03:44:52.4327651Z CURRENT doop f0rc3/gokapi v2.2.4 +2026-11-03T03:44:52.4328014Z CURRENT doop ghcr.io/seerr-team/seerr v3.5.1 +2026-11-03T03:44:52.4328381Z CURRENT doop ghcr.io/thomiceli/opengist 1.15.3 +2026-11-03T03:44:52.4328671Z CURRENT doop glanceapp/glance v0.8.7 +2026-11-03T03:44:52.4328955Z CURRENT buil golang 1.27.3-trixie +2026-11-03T03:44:52.4329293Z CURRENT buil golang 1.27.3-alpine3.24 +2026-11-03T03:44:52.4329704Z CURRENT doop onlyoffice/documentserver 9.4.0 +2026-11-03T03:44:52.4330136Z CURRENT buil restic 0.19.1 +2026-11-03T03:44:52.4330554Z CURRENT box traefik v3.7.17 +2026-11-03T03:44:52.4330963Z CURRENT app actualbudget/actual-server 26.11.0 +2026-11-03T03:44:52.4331393Z CURRENT app deluan/navidrome 0.64.2 +2026-11-03T03:44:52.4331851Z CURRENT app docmost/docmost 0.97.0 +2026-11-03T03:44:52.4332258Z CURRENT app flomp/wanderer-db v0.21.2 +2026-11-03T03:44:52.4332691Z CURRENT app flomp/wanderer-web v0.21.2 +2026-11-03T03:44:52.4333115Z CURRENT app ghcr.io/advplyr/audiobookshelf 2.37.2 +2026-11-03T03:44:52.4333541Z CURRENT app ghcr.io/cmintey/wishlist v0.69.0 +2026-11-03T03:44:52.4333983Z CURRENT app ghcr.io/diced/zipline 4.8.2 +2026-11-03T03:44:52.4334421Z CURRENT app ghcr.io/gethomepage/homepage v2.7.0 +2026-11-03T03:44:52.4334876Z CURRENT app ghcr.io/gramps-project/grampsweb v25.6.0 +2026-11-03T03:44:52.4335310Z CURRENT app ghcr.io/grimmory-tools/grimmory v3.5.0 +2026-11-03T03:44:52.4335763Z CURRENT app ghcr.io/home-assistant/home-assistant 2026.10.4 +2026-11-03T03:44:52.4336230Z CURRENT app ghcr.io/papra-hq/papra 26.7.1-rootless +2026-11-03T03:44:52.4336695Z CURRENT app ghcr.io/seanmorley15/adventurelog-backend v0.13.1 +2026-11-03T03:44:52.4337133Z CURRENT app ghcr.io/seanmorley15/adventurelog-frontend v0.13.1 +2026-11-03T03:44:52.4337584Z CURRENT app ghcr.io/seerr-team/seerr v3.5.1 +2026-11-03T03:44:52.4338077Z CURRENT app ghcr.io/sysadminsmedia/homebox 0.27.1 +2026-11-03T03:44:52.4338499Z CURRENT app ghost 6.73.0-alpine +2026-11-03T03:44:52.4338918Z CURRENT app lscr.io/linuxserver/bookstack 26.09.2 +2026-11-03T03:44:52.4339387Z CURRENT app lscr.io/linuxserver/grocy 4.7.1 +2026-11-03T03:44:52.4339811Z CURRENT app lscr.io/linuxserver/radarr 6.4.4 +2026-11-03T03:44:52.4340242Z CURRENT app lscr.io/linuxserver/sonarr 4.0.20 +2026-11-03T03:44:52.4340948Z CURRENT app onlyoffice/documentserver 9.4.0 +2026-11-03T03:44:52.4341429Z CURRENT app outlinewiki/outline 1.12.0 +2026-11-03T03:44:52.4341855Z CURRENT app vaultwarden/server 1.37.5-alpine +2026-11-03T03:44:52.4342302Z CURRENT app vikunja/vikunja 2.7.1 +2026-11-03T03:44:52.4342730Z COUNTS ADVISORY=45 BEHIND=14 CURRENT=35 FLOATING=20 GAP=81 SELF=2 +2026-11-03T03:44:52.4343172Z FAILED-SOURCES 0 +2026-11-03T03:44:52.4343610Z RUNTIME read=9 (235 s) from-cache=176 not-checked=0 stated-unreadable=10 zstd-decoder=zstandard 0.25.0 (pinned wheel, sha256 checked) +2026-11-03T03:44:52.4344113Z TAGS fetched=116 from-cache=2 retried-429=0 retried-timeout=2 (cache /root/.cache/felhom-watch-tags.json) +2026-11-03T03:44:52.4344644Z REQUESTS api.github.com=68 api.osv.dev=1 auth.docker.io=1 blog.cloudflare.com=1 ghcr.io=174 github.com=64 hub.docker.com=203 raw.githubusercontent.com=1 registry-1.docker.io=1 +2026-11-03T03:44:52.8773108Z MAIL RESEND-ACCEPTED id=01a19f82-821b-7937-8b20-d8eb97cd797b kind=advisory subject=[Felhom security] cal.com, termix, paperless-ngx, recipes … — 0 new advisories, 63 reminders +2026-11-03T03:44:52.8958850Z STATE written: documentation/security-watch/state.json (335 known advisories) +2026-11-03T03:44:53.0308611Z ::group::Run set -u +2026-11-03T03:44:53.0309403Z set -u +2026-11-03T03:44:53.0309670Z f=documentation/security-watch/state.json +2026-11-03T03:44:53.0309808Z if [ ! -f "$f" ]; then +2026-11-03T03:44:53.0309931Z echo "no memory file was written - nothing to commit"; exit 0 +2026-11-03T03:44:53.0310101Z fi +2026-11-03T03:44:53.0310207Z if git ls-files --error-unmatch "$f" >/dev/null 2>&1 && git diff --quiet -- "$f"; then +2026-11-03T03:44:53.0310341Z echo "memory unchanged - nothing to commit"; exit 0 +2026-11-03T03:44:53.0310522Z fi +2026-11-03T03:44:53.0310650Z cp "$f" /tmp/state.json +2026-11-03T03:44:53.0310754Z auth=$(python3 -c 'import base64,os;print(base64.b64encode(("x:"+os.environ["PUSH_TOKEN"]).encode()).decode())') +2026-11-03T03:44:53.0310900Z for attempt in 1 2 3; do +2026-11-03T03:44:53.0311046Z git fetch -q --depth 1 origin main && git checkout -q -B main FETCH_HEAD +2026-11-03T03:44:53.0311174Z mkdir -p "$(dirname "$f")" && cp /tmp/state.json "$f" +2026-11-03T03:44:53.0311292Z git add "$f" +2026-11-03T03:44:53.0311469Z git -c user.name="felhom security watch" -c user.email=monitoring@felhom.eu \ +2026-11-03T03:44:53.0311631Z commit -q -m "security-watch: memory of $(date -u +%F) (R-1303). Docs only." || { echo "nothing to commit"; exit 0; } +2026-11-03T03:44:53.0311782Z if git -c http.extraHeader="Authorization: Basic $auth" push -q origin HEAD:main; then +2026-11-03T03:44:53.0311984Z echo "memory pushed: $(git rev-parse --short=12 HEAD)"; exit 0 +2026-11-03T03:44:53.0312340Z fi +2026-11-03T03:44:53.0312561Z echo "push refused (attempt $attempt) - re-fetching main" +2026-11-03T03:44:53.0312782Z sleep 5 +2026-11-03T03:44:53.0312969Z done +2026-11-03T03:44:53.0313152Z echo "MEMORY NOT SAVED after 3 attempts"; exit 1 +2026-11-03T03:44:53.0313268Z shell: bash --noprofile --norc -e -o pipefail {0} +2026-11-03T03:44:53.0313406Z env: +2026-11-03T03:44:53.0313969Z PUSH_TOKEN: *** +2026-11-03T03:44:53.0314088Z ::endgroup:: +2026-11-03T03:44:54.3755222Z memory pushed: 329d769d89b8 +2026-11-03T03:44:54.3765322Z Cleaning up container for job watch +2026-11-03T03:44:54.9457960Z 🏁 Job succeeded diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 83b9383d..c77281dd 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,16 @@ --- +## 2026-11-03 — the security night (full text of the rows: `git show HEAD~1:documentation/backlog/OPEN-ITEMS.md` at the close-out commit) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-1334** | **Docker Hub's tag API refuses the watch (HTTP 429).** (P3) | CLOSED 2026-11-03 — the 2026-10-28 fix held: 0 Docker Hub 429 on five real nights and both runners. What looked like its failure was another cause (R-1303's streak): seven pins that can never be read, now a stated GAP, plus read timeouts, now retried. `failures` empty after run 2550 and the 03:40 night run | `audits/security-2026-11-02/part1/DIAGNOSIS.md`; `scripts/security_watch.py` W12, W15; felhom.eu `4afa8c3c` | +| **R-1310** | **The security watch flags notices that do not apply and misses fixes that do.** (P3) | CLOSED 2026-11-03 — another program's package is not listed (`PACKAGE_NOT_OURS`), every GHSA a release names is its own entry, a word-only line is no ADVISORY (`WORD_ONLY`), `YY.MM.YYYYMMDD` tags are not newer; seerr was already repointed in the catalog. First catch: Traefik's GHSA-xv5x sat under a word-only line | W16, red first, 13 mutants; felhom.eu `4afa8c3c` | +| **R-1336** | **ActualBudget had no release with the HTTP2Shell fix.** (P3) | CLOSED 2026-11-03 — 26.11.0 carries Node 24.22.1, read from the image; a `security_fix` step 26.9.0 → 26.11.0 proven on 9202, live | catalog `c760fca`; `audits/security-2026-11-02/walks/actualbudget/` | +| **R-1324** | **RomM 5.4's first start could outlast the update's check on a big library.** (P3) | CLOSED 2026-11-03 — demo-hp's night leg took the step: `last_auto_update` 2026-10-28 03:22:09Z, outcome `done`, images recorded 03:23:37Z (88 s), running 5.4.1, not undone. demo-hp's library is small: a big library's migration stays unmeasured, and no box waits on this step any more | `audits/security-2026-11-02/part4/romm-demo-hp-app-yaml.txt` | +| **R-1303** | **Nobody watched the security notices of what we ship.** (P2) | CLOSED 2026-11-03 — a week of nightly runs read; the Monday summary 2026-11-02 (`weekly_last`, mail „[Felhom security] weekly: 59 behind, 354 advisories open", in the mailbox) and real ADVISORY mails (10-28 onwards); both source streaks closed by R-1334's fix. The one design question left is STATUS D66 | `audits/security-2026-11-02/part1/`; state.json | + ## 2026-10-30 — R-1344 and R-1346 closed (full text of the rows: `git show e0e55bf8:documentation/backlog/OPEN-ITEMS.md`) | Row | What | Closed | Evidence | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index f4c09e1b..744ce009 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -113,7 +113,7 @@ match what the reader sees is how an instrument stops being believed (R-421). It **nothing was proven on the day this line was drawn** and a stopping line that moves a status is a stopping line that lies. -## Install & onboarding — 6 rows (P3 4, P4 2) +## Install & onboarding — 7 rows (P3 5, P4 2) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -121,10 +121,11 @@ stopping line that lies. | **R-250** | Install & onboarding | P3 | **A customer create can fail fail-closed because the host-key scan ladder is shorter than the DNS/AAAA settle time.** Found 2026-08-07 creating the fifth walk's venue. `POST /configs/new` with off-site enabled provisions the Storage Box sub-account and then scans its SSH host key to pin it — **fail-closed by design** (`offsite.go:111-121`, *"don't serve a descriptor the controller can't verify"*), with `defaultScanBackoff` = 2+4+8+16+30 ≈ **60 s**, sized by its own comment to *"the observed DNS propagation lag"*. **Measured, both halves:** the first create exhausted the ladder — five `no such host`, then `dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachable` (the name had just begun resolving, **AAAA-first, into a pod with no IPv6 route**) — and the hub logged `[ERROR] offsite provision for walk5`. An **identical second POST succeeded** ~70 s later on its own final rung (`shared already provisioned … (subaccount 285351)`). **Total settle ≈ 100 s against a 60 s budget.** The endpoint was never the problem: verified afterwards from both the node and the hub pod, by name, `SSH-2.0-OpenSSH_9.6p1`. **Two distinct things are wrong and should not be merged:** (1) the budget is sized against DNS *existence*, but what actually bit is the **AAAA-before-A** window, a different and longer phenomenon — lengthen the ladder and/or prefer the A record for the scan dial; (2) **the operator is told nothing actionable** — the create fails with a generic error, the remedy is "press it again", and nothing says so. The retry is genuinely safe (idempotent on the label; confirmed afterwards — `one_time_secrets` 1, one sub-account, no double-provision) **but that safety is invisible to the person deciding whether pressing again will double-charge them.** **Severity LOW-MEDIUM:** self-clearing, no data at risk — but it is the first thing a new customer's provisioning does and it fails looking like an outage. | **VERIFY** — **2026-10-12: hub 0.147.0 live** (S29, decision 224): the host-key scan dials IPv4 first, waits ~2 min (was ~1), and when it still runs out says the store was created and it is safe to press Save again (`TestScannerDialsIPv4First`, `TestScanOutOfRetriesSaysSafeToPressAgain`, red-proved). Known limit: the hub Ingress reads for 60 s, so a very slow first try can show a gateway timeout while provisioning finishes in the background (pressing again is safe). **READY** — operator ruling 2026-10-12 (S29, `09` §3 decision 224): yes — the scan prefers IPv4, waits longer, and the error says „safe to press again". **READY** — owner Viktor | — | The next real off-site setup (a new customer): read the hub log for the scan; success or the new message → close | CC | | **R-516** | Install & onboarding | P3 | **[P3-LOW] English a customer meets on a fresh box and its apps' first screens — enumerated by the big night.** MEASURED 2026-09-14 (BIGNIGHT, VM 333, ISO 1.27.1, controller 0.242.0). Felhom-owned: (1) the dashboard menu item **„Debug"**; (2) the dashboard CPU tile **„Load: 0.29 / 0.39 / 0.37"**; (3) the launcher tile **„Filebrowser"** opens a login in English with no Felhom text (R-513); (4) the storage page mixes formal „Adjon hozzá / Csatlakoztasson" with the product's „te". App first screens a household meets before any Felhom text helps: (5) **Uptime Kuma 2.4 opens on „Which database would you like to use?"** (SQLite / Embedded MariaDB, „Next") — the app card's „Első lépések" does not mention it; (6) PrivateBin, Gokapi, AdventureLog and FileBrowser UIs are English (the apps' own). Already rows: the Proxmox installer screens (R-495, answered by the guide), `wiki.DOMAIN` (R-498). **Fix shape:** rename „Debug"/„Load" (controller); add the Uptime Kuma database step to its card, or pre-seed `db-config.json` for SQLite in the template (catalog). **Added by F4 (20:04:54Z):** (7) the storage page prints the disconnect time as a raw ISO UTC string „Leválasztva: 2026-09-14T19:58:02Z"; (8) the „Meghajtó leválasztva" banner appears twice on every page; (9) „4 telepített alkalmazás nem fut — nézze meg a rendszermonitort" uses the formal form. **Added by F7 (20:50–21:00Z, system disk at 95 %):** (10) a banner on every page in English, „**SSD disk usage high: 90%**"; (11) the dashboard tile reads „Rendszer (/) 61.8 GB / 68.7 GB (**90%**)" while `df` reports **95 %** (reserved blocks ignored), and „(/)" labels the data volume `/mnt/sys_drive`; the deploy page says nothing about free disk. **Added by the i18n spike (2026-09-17, controller v0.247.0):** (12) **six formal („ön") forms in the converted dashboard copy** — „Olvassa be telefonnal" (launcher QR hint), „Biztosan kikapcsolja a megosztást?" (launcher), „Ha újratelepíti … importálnia kell" (the layout's remove-app modal), „Kérjük, vegye fel a kapcsolatot" (backups empty state). They were NOT fixed — a localisation release may not change Hungarian bytes — and are now COUNTED by `controller/scripts/i18n_missing_gate.py` (`HU_FORMAL_CEILING` = 6, a ratchet: a new one convicts, fixing one lowers it). The inventory (`audits/I18N-INVENTORY-2026-09-17.md`) is the list this row closes against in localisation slice 6 (R-561). **Extended 2026-09-17 by slice 1 (controller v0.248.0–v0.250.0):** the converted copy now counts **16** formal forms (`HU_FORMAL_CEILING` 6 → 10 → 12 → 16, each raise stated, Hungarian unchanged by rule) — and the count is an UNDER-count: the gate's stem list sees 4 on the release C pages (storage, drive wizards, debug), while a wider list („írja be”, „adja meg”, „adjon hozzá”, „válassza ki”, „biztosan eltávolítja”, „engedélyezze”, „hozzon létre”, „kattintson”) finds at least **22** keys there alone (login „Adja meg a jelszavát”, the NAS guide, the storage confirms). Widen the stems when this row is worked; the ceiling rises with them. **NARROWED 2026-09-20 by localisation slice 6's walk, item by item** (`audits/i18n-slice6-2026-09-20/R-516-item-by-item.md`): item 2 FIXED (slice 1), items 3 and 6 APP-OWNED (and for an English household item 6 is an advantage), item 11 is a wrong NUMBER rather than a language and needs its own row, item 1 is correct English for an English reader and still open for a Hungarian one. **Items 4, 7, 8, 9 and 10 are NOT DECIDABLE FROM AN ENGLISH WALK** - they are about Hungarian copy quality, a disconnected drive and a full disk, none of which this walk had. **What this row is now waiting for is a HUNGARIAN walk on a box with a second drive**, not another English one; saying it closed would be the kind of closure that makes a register stop meaning anything. **-- 2026-10-08:** the recovery screen's own answers are Go literals in Hungarian only — 14 `renderRecovery` messages in `controller/internal/web/recovery_handlers.go` (seen: the safe default „A művelet nem fejeződött be…" rendered on an English page in a red-proof run). Today's new R-304 sentence is a bundle key in both languages; the 14 old ones are not. | **NARROWED 2026-09-20 - rank P3-LOW; owner: CC; needs a Hungarian walk** **2026-10-05 (burn-down night): items 1 and 4 FIXED on controller `main`** (`d7efa7b`): the menu says „Hibakeresés” (English „Debug”); the storage pages use the te-form (formal-form ceiling 17 → 14; 116 parity fixtures changed by exactly those bytes). Item 2 was already fixed. **Left:** items 5–6 (apps' own screens) and the rest. **2026-10-06: items 1 and 4 DELIVERED** (controller v0.298.0). **2026-10-06 (burn-down night, later): items 7–10 and 12 FIXED on controller `main`** (`e4774e6`, `2e5ef7e`): the disconnect time in local time; one banner per unplugged drive; „nézd meg”; the disk/memory/CPU/temperature banners in the household's language (`health.*` keys; the wire text to the hub unchanged and pinned); the last eleven counted formal forms are te-form (ceiling 14 → 0; 119 parity fixtures changed by exactly those bytes). **Left:** items 5–6 (apps' own screens); about 20 formal forms the gate does not count yet („írja be”, „adja meg”, „hozzon létre”, …) on the debug, storage, network-storage and security pages. Item 11 moved to its own row **R-889**. **2026-10-06 02:40: the bundle has NO formal form left** (controller `4c3c203`, unreleased): 49 sentences in the te-form; the gate's stem list widened (it counted 61 on the previous bundle, 0 now) with decoys and one listed third-person exception. **Left (narrowed):** Felhom-owned formal forms in Go and template literals the gate cannot read yet — the network-storage attach errors, escrow and share handlers, the single-copy notice, a settings refusal, the SMART mail, a fill-watch „Kérjük”, and the setup wizard pages; each moves into the bundle in the te-form. Items 5–6 are the apps' own screens (not ours). **2026-10-06: items 7–10, 12 and the bundle's 49 formal forms DELIVERED** in controller v0.299.0. **2026-10-06 night: the Go-literal leftover moved** (controller main `3d6ba28`, unreleased): 22 Go-literal messages — escrow handler (6), share password page (2), export upload (1), network-storage uid/remove/attach failures (13) — are bundle keys (`api.escrow.*`, `api.share.*`, `api.export.*`, `api.netstorage.*`) in the te-form with English (14 were formal); a detached attach failure renders in the reader's language; the NAS refusal says „Válassz". `TestR516_GoLiteralFormalFormsAreGone` (source scan + positive control), `TestR516_NetAddFailureFollowsTheReader`; red-proofs `audits/night-burndown-2026-10-06/ctrl/R-516-red*.txt`. **Left:** the setup wizard and recovery-info.txt (R-554); the SMART and fill-watch texts (the hub event's message — wire text, needs the household-copy producer shape); items 5–6 (apps' own screens). | — | — | CC | | **R-1273** | Install & onboarding | P3 | **An install that fails on the image registry's rate limit tells the household only „exit code 1”.** Catalog sweep 2026-10-24, code-server 16:01:37: `lscr.io … toomanyrequests` → the product cleaned up right (R-649) but the event and the card say „Alkalmazás telepítése nem sikerült: Code-Server — exit code 1”; nothing says the download server is busy and a later try works. The same install went through later (`audits/catalog-sweep-2026-10-24/code-server/`). | OPEN | — | Recognise `toomanyrequests` (and a pull timeout) in the deploy error and say „the app's download server is busy — try again in an hour”; a test with the stderr line | CC | +| **R-1364** | Install & onboarding | P3 | **The box's own DNS (the LAN way in when the internet is down) has two gaps. (1) It works by name only if the home network's DNS points at the box, and no page or guide tells the household so. (2) On a host whose FQDN is the dashboard's name (Tester 1: `/etc/hosts` holds `felhom.enkicsifelhom.hu`) dnsmasq answers that one name with the HOST's address, where nothing serves 443: the app names work, the dashboard's fails (000), with the internet up or down.** demo-hp's host has another name and answers right. Measured 2026-11-03, `audits/security-2026-11-02/part5/lan-resolver.txt`. The file `/etc/dnsmasq.d/felhom-resolver-base.conf` is the agent's (`internal/lanresolver`). | OPEN — filed 2026-11-03 | — | `no-hosts` in the base file (agent, test red first); one sentence on the page or in the guide for the home network's DNS. Read which FQDN an ISO install gives its host | CC | | **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC | | **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator | -## Apps & catalog — 24 rows (P2 1, P3 12, P4 11) +## Apps & catalog — 27 rows (P2 1, P3 13, P4 13) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -147,14 +148,17 @@ stopping line that lies. | **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | WATCHING — **2026-10-22 night:** not reproducible tonight; the Immich server log of 19:38 is gone (Immich restarted at 02:16 for the off-site dump). What the household sees: the upload fails after 10 minutes and the Immich phone app sends it again. | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC | | **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. **Seen again 2026-10-24 night on Tester 1:** privatebin's image declares `/run`, `/tmp` and `/var/lib/nginx/tmp` as volumes — each recreate (the night dump's stop/start) leaves three empty anonymous volumes; the same class as Immich's `/data` (`audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt`). | OPEN **2026-10-24 (catalog sweep):** the same class for OnlyOffice and RomM (RomM's image volume `/romm` holds only the empty mount points of its binds); after every catalog app had been removed WITH data, the sweep box still held 52 such volumes (PrivateBin 21, OnlyOffice 8, RomM 8, Immich 3, and 12 from 12:50–12:52 before the batch, holding the same nginx folders as PrivateBin's) — remove-with-data does not see them (no compose project label). All small (4–72 KB), no household data (`audits/catalog-sweep-2026-10-24/_box/anon-volumes.txt`). | — | declare the image's VOLUME paths as tmpfs or named volumes in the catalog (Immich, privatebin), or remove the dangling ones a recreate leaves, by name | CC | | **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo//`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC | -| **R-1329** | Apps & catalog | P3 | **cal.com runs on Node.js 20, which gets no security fixes any more** — v6.2.0 carries Node 20.20.0 (no fixed release on the 20 line for CVE-2026-111497) and Debian 12's unpatched `libbrotli1` 1.0.9-2+b6 (fixed: 1.0.9-2+deb12u1); the same image runs in the catalog and on DooPlex. Read from the image 2026-10-27; 2026-10-28: v6.2.0 is still the newest tag, no Node 22/24 image (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN | — | a cal.com release on Node 22/24 (a big step: its own walk); until then the watch lists it nightly | CC | -| **R-1330** | Apps & catalog | P2 | **RomM's newest release (5.4.1) still ships Node.js 24.18.1 (CVE-2026-111497, fixed 24.22.1)** — no newer RomM exists (tags read 2026-10-27). Exposure: Node is IN the image but no Node process runs (RomM serves through Python gunicorn/rq and nginx — `ps` on demo-hp 9201), and RomM sits behind the setup gate. | OPEN — **RULED D61 = keep RomM offered** (operator 2026-10-28, starting the day brief; `09` §3 decision 277): the first RomM release with Node ≥ 24.22.1 is a lane step. 2026-10-28: 5.4.1 is still the newest tag (read from the registry). Earlier: CC's pick 2026-10-27 night | — | the watch flags it nightly (R-1327); the step when RomM ships | CC | -| **R-1331** | Apps & catalog | P3 | **Paperless-ngx 2.20.15 carries Python Brotli 1.1.0 (CVE-2026-111497, fixed 1.1.1/1.2.1)** — its web side only COMPRESSES (whitenoise, compression_middleware); the DECODERS are urllib3's and httpx's, for replies to requests Paperless itself makes (its own outbound calls), never for what a visitor sends. Read in the running container on demo-hp 9201, 2026-10-27. 2026-10-28: the newest release 3.3.4 still carries Brotli 1.2.0 and is a new main version — no fixed release yet (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN — no step tonight (decided by CC unattended — operator may reverse): the bad path needs a hostile server Paperless calls | — | take the first Paperless release with Brotli ≥ 1.2.1 as a normal step | CC | -| **R-1336** | Apps & catalog | P3 | **ActualBudget has no release with the HTTP2Shell fix (CVE-2026-111497): 26.9.0 carries Node.js 24.20.0 and the newest 26.10.0 still Node.js 24.21.0 (fixed: 24.22.1), read from both images.** `node app.js` serves the app; the setup gate covers it until the first account exists, then its own password. No box runs it. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Node ≥ 24.22.1 | the watch lists it nightly (R-1327); take that release as a lane step | CC | -| **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC | +| **R-1329** | Apps & catalog | P3 | **cal.com runs on Node.js 20, which gets no security fixes any more** — v6.2.0 carries Node 20.20.0 (no fixed release on the 20 line for CVE-2026-111497) and Debian 12's unpatched `libbrotli1` 1.0.9-2+b6 (fixed: 1.0.9-2+deb12u1); the same image runs in the catalog and on DooPlex. Read from the image 2026-10-27; 2026-10-28: v6.2.0 is still the newest tag, no Node 22/24 image (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN **2026-11-02:** v6.2.0 is still the newest; its image still holds Node 20.20.0. | — | a cal.com release on Node 22/24 (a big step: its own walk); until then the watch lists it nightly | CC | +| **R-1330** | Apps & catalog | P2 | **RomM's newest release (5.4.1) still ships Node.js 24.18.1 (CVE-2026-111497, fixed 24.22.1)** — no newer RomM exists (tags read 2026-10-27). Exposure: Node is IN the image but no Node process runs (RomM serves through Python gunicorn/rq and nginx — `ps` on demo-hp 9201), and RomM sits behind the setup gate. | OPEN — **RULED D61 = keep RomM offered** (operator 2026-10-28, starting the day brief; `09` §3 decision 277): the first RomM release with Node ≥ 24.22.1 is a lane step. 2026-10-28: 5.4.1 is still the newest tag (read from the registry). Earlier: CC's pick 2026-10-27 night **2026-11-02:** 5.4.1 is still the newest tag; its image still holds Node 24.18.1. | — | the watch flags it nightly (R-1327); the step when RomM ships | CC | +| **R-1331** | Apps & catalog | P3 | **Paperless-ngx 2.20.15 carries Python Brotli 1.1.0 (CVE-2026-111497, fixed 1.1.1/1.2.1)** — its web side only COMPRESSES (whitenoise, compression_middleware); the DECODERS are urllib3's and httpx's, for replies to requests Paperless itself makes (its own outbound calls), never for what a visitor sends. Read in the running container on demo-hp 9201, 2026-10-27. 2026-10-28: the newest release 3.3.4 still carries Brotli 1.2.0 and is a new main version — no fixed release yet (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN — no step tonight (decided by CC unattended — operator may reverse): the bad path needs a hostile server Paperless calls **2026-11-02: a fixed release exists, on a new main version.** 3.4.0 and 3.4.1 read clean (3.3.4 is still bad). **The big step 2.20.15 → 3.4.1 is proven on the drill:** bench `proven`, 9202 `proven` (peak 40.7 %), the undo `proven` (`audits/security-2026-11-02/bench/paperless-ngx/`, `walks/paperless-drill/`). Not written to the live catalog. | — | the operator's word (STATUS, morning list), then the writer | CC | +| **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) **2026-11-02:** v4.0.8 is still the newest; its image still holds Brotli 1.2.0. | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC | +| **R-1356** | Apps & catalog | P3 | **n8n keeps three notices fixed only in 3.0.1 (`GHSA-9p7v` high: code nodes run in the main process; `GHSA-mx4r` unauthenticated webhook-secret reset; `GHSA-2x68`), after the lane step to 2.46.4 fixed nine.** | OPEN — the big step 2.46.4 → 3.0.1 is proven on the drill: bench, 9202 (peak 23.4 %), the undo (`audits/security-2026-11-02/bench/n8n/`, `walks/n8n3-drill/`). Not written to the live catalog. | the operator's word | morning list; then the writer | CC | +| **R-1357** | Apps & catalog | P3 | **PrivateBin 2.1.0 has two public notices fixed in 2.1.1 (`GHSA-hw9p` comment replay by anyone who knows a paste id; `GHSA-c4gc` 2.1.0's XSS fix is ineffective), but no 2.1.1 exists: no release on GitHub and no `privatebin/pdo` tag (read 2026-11-02).** Public by design on a box; Tester 1 runs it. | OPEN — filed 2026-11-03 | upstream: a 2.1.1 image | the watch keeps it open; take the release as a lane step | CC | +| **R-1359** | Apps & catalog | P4 | **Two dependency fixes with no public id, so no lane: Mealie v3.30.0 (`pillow` to v13 „[security]") and Termix 26.11.0 (`body-parser` „for a security fix", in v26.10.6).** Mealie's release also removes `OPENAI_CUSTOM_PROMPT_DIR` (our template does not set it). | OPEN — both proven on the drill as normal steps: bench and 9202 (`audits/security-2026-11-02/bench/{mealie,termix}/`, `walks/{mealie,termix}-drill/`). Not written. | the operator's word | morning list; then the writer | CC | +| **R-1360** | Apps & catalog | P4 | **Meilisearch v1.36.0 (Wanderer) and v1.41.0 (Karakeep) are 15–20 versions behind releases with „important security fixes" (v1.55.1, v1.56.1: long filters, rustls).** The search engine sits on the app's private network, so a stranger cannot reach it; its data needs a dump migration across such a jump: a big step. | OPEN — filed 2026-11-03 | — | a big step when one of the two apps is next moved | CC | -## App updates — 10 rows (P3 10) +## App updates — 9 rows (P3 9) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -162,14 +166,13 @@ stopping line that lies. | **R-1318** | App updates | P3 | **Karakeep 0.33.2 + chrome 151: two CISA-KEV V8 bugs in the crawler's Chromium (runs `--no-sandbox`), reached by content the household saves; the fix is 0.34.1 + chrome 154.0.8037.57-r1** — an auth library swap (everyone signed out, e-mails lowercased) and a chrome rebuilt on full Chrome (memory limit and healthcheck to measure again). No box runs it. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step: measure chrome 154's memory; bench + 9202 + undo; a STOP | CC | | **R-1319** | App updates | P3 | **Paperless-ngx 2.20.15: GHSA-2jhj (low; needs a setting we do not set — not reachable); the fix is 3.3.4, a big step** (encryption and API v1 removed, the search index rebuilt). It runs on all four test boxes. Not urgent. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step 3.3.4 when the catalog takes Paperless 3 for other reasons; read our API v1 use first | CC | | **R-1321** | App updates | P3 | **Tandoor 2.6.15: GHSA-5f74 (high, an XXE file read) and GHSA-69q3 (any user); the fix is 2.7.2** — 2.7.0 removes settings and an endpoint and rewrites the login pages (our probe path — not sure), and 2.7.0/2.7.1 break invite links with sign-up off, so straight to 2.7.2. No box runs it. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | a normal step to 2.7.2 with the probe path checked; bench + 9202 (+ undo if a migration); a STOP | CC | -| **R-1324** | App updates | P3 | **RomM 5.4's first start can run a 15–20 minute migration on a big library (upstream notes) — longer than the update's check window, so a household's night step could be undone for slowness, not for a fault.** Today's walk (9202, a small seed) took the step in seconds; demo-hp's RomM takes it tonight (security-fix step, catalog `06e603e`). `audits/security-app-updates-2026-10-27/triage-notes/romm.md` | OPEN — filed 2026-10-27; WATCH demo-hp's night leg line | — | read demo-hp's night leg line for romm (done / undone, the time); if undone for time: give the step a longer check or the `needs_person` mark | CC | | **R-440** | App updates | P3 | **[P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible.** `compose pull` on a moving tag fetches whatever upstream published that day. **MEASURED 2026-09-01 over `app-catalog-felhom.eu` @ `29edad9c5bf4`: 79 `image:` lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version.** `postgres:16-alpine` (8 apps), `redis:7-alpine` (6), `mariadb:11.6` (2), plus one each of `postgres:15-alpine`, `postgis/postgis:16-3.5-alpine`, `mariadb:11.4`, `mariadb:12.3`, `ghcr.io/claperco/claper:2.5`, `ghcr.io/thomiceli/opengist:1.13`, `wger/server:2.6`. **A 24th is arguable and is recorded rather than rounded away:** `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. **Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists**, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. **MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control.** Running digests on demo-hp compared against what the registry serves for the same tag today: **`mariadb:11.4` MOVED** (`sha256:4f1d8d20...` -> `sha256:611a2fcc...`) and **`mariadb:12.3` MOVED** (`sha256:a02fe89c...` -> `sha256:dd9b303a...`), while `postgres:16-alpine`, `redis:7-alpine`, `mariadb:11.6` and `opengist:1.13` were SAME — **and both fully-pinned CONTROLS (`rommapp/romm:5.0.0`, `privatebin/pdo:2.0.5`) were SAME.** So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under `romm` and `bookstack`, with no catalog change and no record. **Compounding fact found while reading:** the recovery unit records `ImagePins` but the manifest comment says *"image NOT stored - re-pulled on restore"*, so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. **HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): `app.yaml.installed_images` now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade.** What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). **The row therefore stays OPEN and its rank is unchanged** — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. `audits/SPIKE-app-update-2026-09-01.md` **— NIGHT 2026-09-23:** every image moved tonight (and the 21 backfilled moves) carries its resolved digest in the catalog's test record; the floating pins themselves still float until the box pulls by digest (`09` §6.4 part 6). **-- MEASURED 2026-09-30 on demo-hp 9201: a fresh install or a guarded Update renders `name:tag@sha256` from the ladder (`digest.go` `RenderWithLadderDigests`) — 9 of 20 services there run such a definition (adventurelog, docmost, paperless-ngx). The other 11 run TAG-ONLY definitions: bookstack, kimai, opengist, privatebin, romm were installed before v0.269.0 and not updated since, and `CarryDigests` (`digest.go:141`) keeps only a digest the running definition already has; bentopdf and calibre-web have no ladder at all. So a re-pull of those (a restore) takes whatever the tag serves that day. `audits/pg-last-six-2026-09-30/E/E2-digests-demo-hp.txt`.** **-- 2026-09-30 (evening):** calibre-web now has a ladder entry (`53a4a1d`), so a new install or a guarded Update renders its tested digest; bentopdf still has none. `audits/more-night-apps-2026-09-30/` **Merged 2026-10-05 from R-446 (duplicate):** the household-visible consequence — for these apps the „Naprakész" badge can never turn „behind" (`felhom-controller controller/internal/stacks/updateorder.go:96` returns false with no catalog digests / no test date); R-740 (same-tag re-test) is closed, so floating-tag drift for laddered apps is handled. | **NARROWED 2026-09-30 — floating only for (1) apps with no ladder entry and (2) apps installed before v0.269.0 until their next guarded Update; owner: CC.** **Re-ranked 2026-10-03: P2→P3: narrowed to apps with no ladder entry and pre-v0.269 installs; each guarded update fixes the latter.** | — | — | CC | | **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s, measured). The catalog currency and the next-apps ranking: `audits/catalog-currency-2026-09-30.md` — 25 of 53 apps behind upstream inside a major, 19 across one (morning); after the session **31 apps can update themselves at night (+ nextcloud with a whole copy)**, 32 carry a proven step, 21 have no ladder.** **-- 2026-09-30 (evening, more-night-apps brief): ten more steps published on both venues** — first ladders for calibre-web, gitea, wger, crafty-controller, uptime-kuma, zipline (new front-door fixtures; gitea's installer form), within-major steps for emby, ghost, home-assistant, outline, rallly. **Night-updatable by the audit's method (a): 30 + 2 conditional at the session's start → 35 + 3 conditional** (calibre-web conditional: `files_may_change`). zipline got a two-step ladder 4.6.1 → 4.7.0 → 4.8.0 (the direct jump fails, R-742). Stopped: wanderer (the bench cannot run it, R-739). Fixture/harness defects fixed on the way: R-735, gitea's installer, the HTTPS backend on the bench, per-file names for `files_may_change`. `audits/more-night-apps-2026-09-30/` | **READY — rank P2-MEDIUM; owner: CC (scope already ruled, `09` §3 decision 6)** **Re-ranked 2026-10-03: P2→P3: 35+ apps already update by proven steps; an app without a ladder simply does not move, which is safe.** **NARROWED 2026-10-20 (night brief part 2, Part 5):** the fixture side is DONE for every app installed on a test box — 58 box fixtures run on both venues; of the 16 installed apps only bentopdf has none (it keeps no data). Upstream within a major (check-currency, registries read): privatebin 2.0.6→2.1.0, home-assistant 2026.9.4→2026.10.2, romm 5.3.1→5.4.0, plex 1.41.4→1.43.4, kimai 2.67.0→2.70.0 — the list left. **No edge was run:** the bench (guest 9401, stopped) and the drill catalog the box venue needs were outside the brief's machines and repos; the operator's word for both is on STATUS's morning list (`audits/night2-2026-10-20/P5/r462-list.txt`). | — | — | CC | | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC | -| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:** **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC | -| **R-1347** | App updates | P3 | **After a successful restore lifts an update hold, the stack manager's „last update ended held" flag stays set until the next update or a controller restart — until then the crash-loop watcher does not watch the app and the nightly update run skips it.** Found by the D64 work 2026-10-29 (controller `5b3b541` clears the flag for `clear_hold` only, `ForgetUpdateHeld`). `clearUpdateHoldAfterRestore` lifts the hold in settings but not this cached flag. | OPEN | — | the restore's lift calls ForgetUpdateHeld too, red-first test (restore after an update hold → the watcher watches again) | CC | +| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:** **OPEN — rank P3-LOW; owner: CC (after R-784)** **2026-11-02:** v1.10.0 and v1.11.0 read clean of HTTP2Shell (Node 24.22.1); v1.9.0 is still bad. **The big step v0.17.3 → v1.11.0 is proven on the drill:** bench, 9202 (peak 29.4 %), the undo (`audits/security-2026-11-02/bench/sparkyfitness/`, `walks/sparkyfitness-drill/`). Not written: R-784 (the licence) stands before it. | — | — | CC | +| **R-1347** | App updates | P3 | **After a successful restore lifts an update hold, the stack manager's „last update ended held" flag stays set until the next update or a controller restart — until then the crash-loop watcher does not watch the app and the nightly update run skips it.** Found by the D64 work 2026-10-29 (controller `5b3b541` clears the flag for `clear_hold` only, `ForgetUpdateHeld`). `clearUpdateHoldAfterRestore` lifts the hold in settings but not this cached flag. | **VERIFY — BUILT 2026-11-02, controller `c4b6844`, commit only; PROVEN LIVE on 9202 (test image):** an update with its undo copy cut off → hold `update_failed`; the household's restore → „the update hold is CLEARED", „update-held flag is forgotten"; six `kill -9` → the crash-loop watcher stopped the app, so it watches again (`audits/security-2026-11-02/part3/`). Tests `TestR1347_*`, red first. OPEN | — | ships with the next controller release | CC | -## Backup & restore — 45 rows (P2 4, P3 23, P4 18) +## Backup & restore — 46 rows (P2 4, P3 24, P4 18) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -197,7 +200,7 @@ stopping line that lies. | **R-878** | Backup & restore | P4 | **A catch-up (R-871) runs the database-dump leg in the DAY, and that leg stops an app with a volume for its copy — the household may notice the stop, and a large volume makes it longer.** MEASURED 2026-10-05 on demo-felhom: the catch-up at 08:25:02 stopped opengist, copied 182.5 KB, started it again — about 1 s, then a few seconds of `health: starting`; the night does exactly the same, unseen. Nothing measured for a large volume. Fix direction (if it matters): skip the volume copy of a running app in a DAYTIME catch-up and leave it to the next night, or warn. `audits/catchup-2026-10-05/partA/live-demo-felhom.txt` | **READY — owner: CC** **2026-10-05 (burn-down night): NEEDS A LIVE MEASUREMENT** — a timed catch-up on 9202 for an app with a multi-GB volume, then the skip-or-warn choice. | — | measure a large volume first | CC | | **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.** **2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** **2026-10-22: Tester 2's planned RESET (operator ruling 2026-10-22 14:11, `pilot/TESTER2-handover.md`) will be the purge's first live run — CC reads its leg journal after.** **2026-10-22 12:56 UTC — the first live run (Tester 2's RESET, `audits/tester2-reset-2026-10-22/`): PASS for what it could show.** The hetzner leg logged in with the sub-account's own password, listed the folder, found NO repository (`purged … 0 path(s) []` — the box never ran an off-site backup; the hub never held a listing for it), re-listed, then deleted sub-account 322460; leg `ok`. The other customers' repositories are unchanged (each listed after the reset with its box's own key: demo-hp 303, tester-1 62, demo-felhom 43 = the hub's 05:10 listing; Tester4 has no box — the hub's own listing on 2026-10-23 05:10 compares it). **Not yet seen live: the removal of a folder that holds data** (the `rm -rf` branch) — the next RESET of a customer with off-site history shows it. Open as before: only the old leftovers. **Dated check 2026-10-23 (DUE-CHECKS):** read the hub's 05:10 UTC listing of Tester4 (72 entries on 2026-10-22) and its `offsite_snapshots_vanished` events — the one other customer whose folder CC could not re-list. **2026-10-23 dated check — DONE:** the hub's 05:10 UTC listing of Tester4 holds 80 snapshot files = the 72 of 2026-10-22 plus 8 new ones from the night's box (`8 new snapshot file(s), each inside a reported box run`), none vanished, no `offsite_snapshots_vanished` (hub log 07:10:03 CEST, `audits/night-2026-10-23/P4-testnight/`). Tester 2's RESET touched no other customer's folder. | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC | | **R-1087** | Backup & restore | P4 | **Whole-box copies left on the OLD local target are never pruned after the target moves.** MEASURED 2026-10-15 (storage-identity build B space check): the N100's host root (`local`) holds three copies from 2026-07-28 (5.5–5.7 GB each, ≈ 16.8 GB); its agent's `local_backup_target` is now `felhom-backup` (the USB disk, retention 3), and retention prunes only the current target. On a small-root box such leftovers would starve the local copies (R-548). Evidence: `audits/storage-identity-buildB-2026-10-15/space/felhom-pve-9201.txt`. | OPEN | — | Decide whether the agent prunes its own `vzdump-lxc--*` files on a former target (after a newer copy exists on the new one), or the target move asks; until then, the N100's three July files can be removed by hand with the operator's word. | CC | -| **R-1171** | Backup & restore | P3 | **demo-hp's LOCAL whole-guest copy is never restore-tested: the box keeps one copy (keep-daily 1) and the restore test wants a copy 24 h old, so the only copy turns 24 h old exactly when the next night replaces it.** Read 2026-10-20 on demo-hp (agent journal): every 6-hourly evaluation since 2026-10-18 says `local: no settled archive yet — nothing to prove`; the local storage holds one 9201 archive; the last local proof was 2026-10-14 03:42 (`settle=24h`, `keep_last=1`, `prune=keep-daily=1`). The hub's `restore_test_stale` (168 h) will alarm from ~2026-10-21 03:42. A real household with retention 1 would be the same. | OPEN — filed 2026-10-20 | demo-hp `journalctl -u felhom-agent` („restore-test evaluated"), the hub's host report `restore_tests` | Agent: a tier that keeps ONE copy proves its newest archive once it is older than the settle time measured from the BACKUP's end, not 24 h — or a settle below the cadence (e.g. 6 h) for a 1-copy tier; a test; then read the next evaluation | CC | +| **R-1171** | Backup & restore | P3 | **demo-hp's LOCAL whole-guest copy is never restore-tested: the box keeps one copy (keep-daily 1) and the restore test wants a copy 24 h old, so the only copy turns 24 h old exactly when the next night replaces it.** Read 2026-10-20 on demo-hp (agent journal): every 6-hourly evaluation since 2026-10-18 says `local: no settled archive yet — nothing to prove`; the local storage holds one 9201 archive; the last local proof was 2026-10-14 03:42 (`settle=24h`, `keep_last=1`, `prune=keep-daily=1`). The hub's `restore_test_stale` (168 h) will alarm from ~2026-10-21 03:42. A real household with retention 1 would be the same. | OPEN — filed 2026-10-20 **2026-11-02 read (journal):** demo-hp's local copy passed a scheduled restore-test on 2026-10-30 10:44 (`audits/security-2026-11-02/part4/restore-test-demo-hp.txt`), so „never” no longer holds; one pass, read once. Close after the next one. | demo-hp `journalctl -u felhom-agent` („restore-test evaluated"), the hub's host report `restore_tests` | Agent: a tier that keeps ONE copy proves its newest archive once it is older than the settle time measured from the BACKUP's end, not 24 h — or a settle below the cadence (e.g. 6 h) for a 1-copy tier; a test; then read the next evaluation | CC | | **R-1172** | Backup & restore | P4 | **After option A (R-893): a failed off-site restore that was PUT BACK is logged at ERROR, and an empty `backups/restore-undo/` folder stays on disk.** Measured on 9202 2026-10-20 (rc.1): the web handler logs `[ERROR] [web] off-box reconstitute rallly (async): … Nem változott semmi…` — good news at ERROR; the put-back removes the per-app folder but not the empty `restore-undo/` parent. Small; 0.336.0 was already pushed (a version is pushed once). | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P2/live/11-controller-log-window.txt`, `12-after-failed-restore.txt` | Next controller release: the put-back outcome logged at WARN (its own sentence is the household's), the empty parent removed; a test | CC | | **R-1173** | Backup & restore | P3 | **R-921's consequence (no app stop for an off-site tier the agent will refuse) has not been seen live.** Agent 0.161.0 serves `host_busy` (proven live on demo-hp 2026-10-20 during a press: busy `backup:local`), and controller ≥ 0.335.0 reads it — but the case only arises on the night schedule (a manual press backs up the local tier only), so the no-stop branch is proven by its red-proved test alone. | OPEN — filed 2026-10-20 **2026-10-23 dated check — DONE (the other outcome):** demo-hp's off-site (PBS) tier ran 2026-10-22 20:42:53–20:49:48 UTC with nothing in flight (quiesce → started → snapshotted → early resume → done); no `host is busy` case arose, so the R-921 no-stop branch is still proven by its test alone. Tonight only the local tier was due (refused by space, R-548). Kept open; no new date — it reads itself the night a busy host meets a due off-site tier. | `audits/night2-2026-10-20/P1/1b-host-busy-poll.txt` | Read demo-hp's controller log after the night its off-site (pbs) tier is due (~2026-10-22): a `host is busy … no app is stopped (R-921)` line, or the off-site tier starting with nothing in flight | CC | | **R-1184** | Backup & restore | P3 | **A cut-off shared-folder restore can be „placed" and reported restored while files are missing; the download is then deleted.** Readiness is only „the scratch folder is not empty" (`internal/backup/shares_restore.go:131-140`) — the R-358 class fixed for apps, not for shares; 30-min download window (`web/offbox_handlers.go:716`); fixed 2 GiB free-space gate that fails open (`:107`). CONFIRMED in code. Data-loss hunt 2026-10-20 (C1 reviewer 1, R3). | OPEN | — | Fix: a completion marker after restic succeeds, as for apps | CC | @@ -217,7 +220,8 @@ stopping line that lies. | **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC | | **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC | | **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC | -| **R-1348** | Backup & restore | P3 | **The restore_mixed hold written at START-UP after an interrupted put-back fails sends no operator mail when it is set** — only the hold reporter's info `app_held` reaches the hub, and the hub's reminder after its grace period. Found by the R-1340 work 2026-10-29 (controller `5b3b541`): the mail sentences for unhealthy stop, failed update and waiting put-back exist and are tested, but the start-up path has no sender. | OPEN | — | send the operator mail from the start-up put-back failure (the R-1340 sentence), red-first test | CC | +| **R-1348** | Backup & restore | P3 | **The restore_mixed hold written at START-UP after an interrupted put-back fails sends no operator mail when it is set** — only the hold reporter's info `app_held` reaches the hub, and the hub's reminder after its grace period. Found by the R-1340 work 2026-10-29 (controller `5b3b541`): the mail sentences for unhealthy stop, failed update and waiting put-back exist and are tested, but the start-up path has no sender. | **VERIFY — BUILT 2026-11-02, controller `c4b6844` + `3764445`, commit only; PROVEN LIVE on 9202 (test image):** a restore cut at „the restore copy is recorded", its volume held, the controller started → hold `restore_mixed` / `interrupted_put_back`, and the event `backup_run_failures` with the R-1340 sentence reached the event sink (`part3/r1348-after.txt`). The hub's mail itself is the existing path. Tests `TestR1348_*`, red first. OPEN | — | ships with the next controller release | CC | +| **R-1362** | Backup & restore | P3 | **With the internet down, the Backups page takes over 2 minutes (135 s and 145 s measured; 5 s with only the hub down).** Walked 2026-11-03 on Tester 1: every other page answers at once on the home network. The page waits on something off-site inside its handler; a household that lost its line and opens exactly this page sees a hang. `audits/security-2026-11-02/part5/outage-wan.log` | OPEN — filed 2026-11-03 | — | name the wait with a timing, then give it a short deadline and show „the off-site state could not be read" | CC | ## Storage & devices — 14 rows (P3 4, P4 10) @@ -236,16 +240,15 @@ stopping line that lies. | **R-1213** | Storage & devices | P4 | **Operator detail still on two household pages:** the registered drive card's `ext4 · /dev/sdb1 · Model` line (`templates/storage.html`; W14 hid only the agent drive cards), and the heading „Távoli mentés (3. mentés) — titkosított, offsite" with the English „offsite". Found by the night helper 2026-10-21 (W11/W14 follow-up). | OPEN | — | Move the line behind „Részletek"; „…titkosított, házon kívüli" | CC | | **R-1275** | Storage & devices | P4 | **An English household sees the drive names in Hungarian** on the install page (and wherever a storage label shows): „Belső tárhely — 52.0 GB free (default)”, „Külső tárhely (hdd_1)” (catalog sweep 2026-10-24, `audits/catalog-sweep-2026-10-24/navidrome/page-install-en.txt`). The label is stored once, in Hungarian, when the drive is found (`felhom-controller/controller/internal/settings/settings.go` `InferStorageLabel`); a household may rename it. | OPEN | — | Render a label that still equals the inferred default in the reader's language; a test per language | CC | | **R-1276** | Storage & devices | P4 | **„Remove with data” leaves the app's empty parent folder** (`/appdata/`) when the template binds a subfolder: Grimmory, RomM, Paperless-ngx (catalog sweep 2026-10-24; 4 KB each; the bound folders were removed). Harmless (empty, not counted as old data), untidy for a household that browses the drive. | OPEN | — | Remove the parent when it is empty after the bound folders go; a test | CC | -| **R-1339** | Storage & devices | P4 | **At every start the controller logs „FileBrowser: drive …/userdata/ not mounted — skipping userdata skeleton" for per-app folders on a drive that IS mounted** (scratch 9202, 2026-10-29 03:35 UTC: nine lines, `scratch_hdd` a live mountpoint, mp8). `skipFileBrowserPath` (`controller/internal/web/handlers.go:3766`) asks whether the folder ITSELF is a mountpoint, and a folder under a mounted drive never is. No data is lost: the file browser shows the folders through the drive's own mount (`/srv/scratch_hdd`). The cost is a false „not mounted" in the log of a box whose drive works, which misleads a diagnosis. Fix: ask about the drive the folder lives on, with a test that a sub-folder of a mounted drive is not skipped. Evidence: `audits/night-2026-10-28/part5-claims/selfheal/controller-after-reboot.log`. | OPEN | — | the fix above, with its red-first test | CC | +| **R-1339** | Storage & devices | P4 | **At every start the controller logs „FileBrowser: drive …/userdata/ not mounted — skipping userdata skeleton" for per-app folders on a drive that IS mounted** (scratch 9202, 2026-10-29 03:35 UTC: nine lines, `scratch_hdd` a live mountpoint, mp8). `skipFileBrowserPath` (`controller/internal/web/handlers.go:3766`) asks whether the folder ITSELF is a mountpoint, and a folder under a mounted drive never is. No data is lost: the file browser shows the folders through the drive's own mount (`/srv/scratch_hdd`). The cost is a false „not mounted" in the log of a box whose drive works, which misleads a diagnosis. Fix: ask about the drive the folder lives on, with a test that a sub-folder of a mounted drive is not skipped. Evidence: `audits/night-2026-10-28/part5-claims/selfheal/controller-after-reboot.log`. | **VERIFY — BUILT 2026-11-02, controller `c4b6844`, commit only; PROVEN LIVE on 9202:** nine false lines at a boot before, none after (`part3/stdout-boot-before.txt`, `stdout-boot-after.txt`). The row's „not skipped" would have written a `userdata/` inside the app's folder and recreated the file manager, so the behaviour stays and only the line changes. The per-app folders in 9202's list are a legacy of its old installs (R-839 refuses that shape now): no other box holds one. OPEN | — | ships with the next controller release | CC | -## Security & access — 29 rows (P1 2, P2 5, P3 15, P4 7) +## Security & access — 30 rows (P1 2, P2 6, P3 15, P4 7) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-1241** | Security & access | P1 | **2026-10-24: the Felhom half is DONE** — hub 0.168.0 deployed (the tokens sealed; every copy leaving the volume is clean, the live file's old bytes accepted by the operator, R-975 closed), the hub volume out of the Longhorn backup (D40, R-1243 VERIFY), the two old plain copies deleted by the operator. **What stays here is the homelab half:** the open NFS export and Longhorn's login (the homelab session). **A hacked app pod on DooPlex can read the hub's nightly database copies — with every customer's unsealed Cloudflare, tunnel and git token (R-975) — through the Longhorn NFS export, and by a published file-handle escape the operator signing keys on the same disk: whoever breaks one homelab app can take over the boxes' tunnels and sign agent updates.** MEASURED 2026-10-23 (Plex2Shell impact audit, `audits/plex2shell-felhom-impact-2026-10-23/`): export `/mnt/5_hdd/backup/longhorn-pvc` rw, `no_root_squash`, `no_subtree_check`, a sub-directory of sda1, which also holds `/mnt/5_hdd/felhom.eu/felhom-op-*` (no passphrase, R-924); `secure` is void in pods (containerd sets `ip_unprivileged_port_start=0`; a uid-1000 probe with no capabilities bound port 700); the four `felhom-system` volumes have `encrypted=false`; Longhorn's API on :9500 answers any pod with no login (backups, targets, volumes). In the Plex2Shell window neither door was used — verdict NOT REACHED (nfsd accepted 0 new connections 10-10 → 10-23, positive control; no off-schedule Longhorn backup or volume). 2026-10-23 ~15:18Z, operator yes: the export was narrowed from `*` to `10.42.0.0/16` — the home network and the tailnet are closed, pods are not (Longhorn connects from its pod address). Not exercised: the escape itself (Tier-2 machine). | OPEN | the homelab session (DooPlex host + cluster); the operator (R-924) | Homelab: a NetworkPolicy so only `longhorn-system` reaches :2049/:111 and :9500, or the backup store on its own filesystem. Felhom: R-975 sealed in hub 0.168.0 (deploy pending); the hub's plain Longhorn copies → R-1243; a passphrase on the signing keys (R-924) | operator | -| **R-1303** | Security & access | P2 | **Nobody watched the security notices of what we ship** — now a nightly CI watch (`scripts/security_watch.py`, 03:40 UTC; 108 pins: 11 box/build, 97 app): ADVISORY mails that night, BEHIND weekly, a source down two nights mails. BUILT and live-proven 2026-10-26: the test mail reached the mailbox (CI job 2407); the seed run (job 2408) recorded 142 open notices on 30 pins without mailing and committed its memory itself. Spike: `audits/security-watch-2026-10-26/SPIKE.md`; gaps: R-1307, R-1308, R-1309 | OPEN — built; closes after a week of nightly runs with the key (watch the Monday summary and one real ADVISORY mail) | none — the GitHub key is in place since 2026-10-27 (`GH_ADVISORY_TOKEN`, store and CI; its own token, not Renovate's): a dry CI run with it (job 2415) read every repo's advisories in one night (68 GitHub requests, 38 ADVISORY, 1 NOT CHECKED = one Docker Hub timeout). The first real night (job 2413, no key yet) mailed nothing: 159 silent first reads of notices the seed could not read, memory committed by the job. Logs: `audits/security-watch-2026-10-26/ci-night1-2026-10-27.log`, `ci-dry-with-key.log` | read the nightly logs to 2026-11-03; close on the Monday summary and one real ADVISORY mail | CC | | **R-1139** | Security & access | P2 | **Tester 2's break-glass root password was not rotated with the others (R-954, operator decision D1 = A, 2026-10-17): the box has been offline since 2026-10-04.** demo-hp, Tester 1 and demo-felhom were rotated and proven on 2026-10-17 (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | OPEN — filed 2026-10-17 **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a fresh install sets and vaults a new console password).** | Tester 2 reporting again | When it reports: the installer's step 4b alone (`P3/rotate-step4b.sh` shape), then the vaulted value logs in (PVE API 200) and the old one is refused | CC | -| **R-1305** | Security & access | P2 | **The file manager on every box (FileBrowser Quantum 1.5.6-stable) is inside 36 published notices; 2 are fixed in our own channel (1.5.7-stable)** — GHSA-qx86-4v5r-26g5 (high: with the victim's password, an anonymous request re-enrolls TOTP and skips the second factor; our households have no second factor there, so low for us) and GHSA-3846-gh75-gp3m (medium: a public upload share replaces files). Most others name fixes only on the 2.x line. `files.` is public through the tunnel (behind the controller's forwardAuth, password sign-in on). Found by the first security-watch run (R-1303). `audits/security-watch-2026-10-26/ADVISORIES-2026-10-26.md` | **NARROWED 2026-10-27 — the two notices fixed in our channel are RELEASED: controller v0.346.1 pins `gtstef/filebrowser:1.5.8-stable` (1.5.7-stable skipped: it broke existing TOTP sign-ins), global floor 15:50:15Z; live on Tester 1, demo-hp, demo-felhom and 9202 (sign-in walk: listing 200, 401 without a cookie; R-1281 probe: an app container cannot reach it) — `audits/security-app-updates-2026-10-27/part5/`. LEFT: triage the 34 other notices against the 1.5.x line.** **2026-10-28 (R-1332): HTTP2Shell does not reach it** — 1.5.8-stable carries Alpine `brotli-libs` 1.2.0-r1 (2.0.2-stable: 1.2.1-r0, a new main version), but the Go program does not link libbrotli (`ldd`: musl only; no process maps it, all three boxes) — the library is there for libcurl. NOT a step today. Earlier: OPEN — filed 2026-10-26 | a controller release | raise `FileBrowserImage` to 1.5.7-stable in the next controller release; triage the 34 others against 1.5.x (which apply to the stable line) | CC | +| **R-1305** | Security & access | P2 | **The file manager on every box (FileBrowser Quantum 1.5.6-stable) is inside 36 published notices; 2 are fixed in our own channel (1.5.7-stable)** — GHSA-qx86-4v5r-26g5 (high: with the victim's password, an anonymous request re-enrolls TOTP and skips the second factor; our households have no second factor there, so low for us) and GHSA-3846-gh75-gp3m (medium: a public upload share replaces files). Most others name fixes only on the 2.x line. `files.` is public through the tunnel (behind the controller's forwardAuth, password sign-in on). Found by the first security-watch run (R-1303). `audits/security-watch-2026-10-26/ADVISORIES-2026-10-26.md` | **NARROWED 2026-10-27 — the two notices fixed in our channel are RELEASED: controller v0.346.1 pins `gtstef/filebrowser:1.5.8-stable` (1.5.7-stable skipped: it broke existing TOTP sign-ins), global floor 15:50:15Z; live on Tester 1, demo-hp, demo-felhom and 9202 (sign-in walk: listing 200, 401 without a cookie; R-1281 probe: an app container cannot reach it) — `audits/security-app-updates-2026-10-27/part5/`. LEFT: triage the 34 other notices against the 1.5.x line.** **2026-10-28 (R-1332): HTTP2Shell does not reach it** — 1.5.8-stable carries Alpine `brotli-libs` 1.2.0-r1 (2.0.2-stable: 1.2.1-r0, a new main version), but the Go program does not link libbrotli (`ldd`: musl only; no process maps it, all three boxes) — the library is there for libcurl. NOT a step today. Earlier: OPEN — filed 2026-10-26 **2026-11-02:** a new notice, GHSA-892f-9h5p-mfc8 (low: `/api/settings` shows secrets to an admin's token), fixed on 2.1.0-stable only. | a controller release | raise `FileBrowserImage` to 1.5.7-stable in the next controller release; triage the 34 others against 1.5.x (which apply to the stable line) | CC | | **R-1042** | Security & access | P3 | **The boxes' Cloudflare keys cannot change zone settings, so Cloudflare's own „Always Use HTTPS” cannot be switched on.** Measured 2026-10-14 on all three zones (`audits/https-only-2026-10-14/``1A-1-zones-and-keys.txt`): each `cf_api_token` is valid and sees only its own zone, but GET/PATCH `settings/always_use_https`, `ssl`, `security_header` answer 403; the redirect and transform rule phases too. The https-only fix therefore lives on the box (decision 246). A second layer at Cloudflare needs „Zone Settings: Edit” (and, for headers, „Transform Rules”) added to each customer's key — the hub's zone-reach check stays. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | a key permission only the operator can grant | Optional: widen each customer key, then switch „Always Use HTTPS” on per zone | operator | | **R-1242** | Security & access | P3 | **The demo NAS export `/mnt/5_hdd/felhom-data` → 192.168.0.162 (demo-felhom) is a sub-directory of DooPlex's sda1 with `no_subtree_check`, so a compromised demo-felhom host could walk the whole disk as uid 101000 and read every world-readable file there (the GPG secrets exports are 0644, every repo, the public keys).** Measured 2026-10-23 (`audits/plex2shell-felhom-impact-2026-10-23/` Part 1; `exportfs -v`: `all_squash,anonuid=101000`). 0600 files (the signing keys) stay out of reach. Not exercised. | OPEN | a DooPlex host change (fence) | Give the demo share its own filesystem, or `subtree_check`; or remove other-read from `/mnt/5_hdd/backup/secrets/exports` | operator | | **R-1104** | Security & access | P3 | **Check https-only on Tester 2 when it reports again.** R-1033 (plain http on every public name) is closed on the three boxes that report (35 names → 301, 2026-10-16). Tester 2 (`sajatfelhom.hu`) has been off since 2026-10-04 on controller 0.292.0; its addresses answer Cloudflare 530 (tunnel down), so nothing is served in plain http while it is off. When it comes back it takes the floor (≥ 0.312.0 carries the redirect). | OPEN — filed 2026-10-16 | Tester 2 reporting again | Run `scripts/https_only_check.py` with Tester 2 in it; close | CC | @@ -261,6 +264,8 @@ stopping line that lies. | **R-525** | Security & access | P4 | **[P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured.** Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. **What it needs:** a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | **READY — rank P3-LOW; owner: CC (spike)** **Re-ranked 2026-10-03: P3->P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed.** | — | — | CC | | **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restart**, because startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137). Raised to 1Gi, **in the manifest, not just live** (the `.claude/rules/manifests.md` rule: a bare `kubectl set` is reverted by the next sync and the fix silently disappears). Service restored and verified. New values are in a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`), never echoed; the operator moves them to the password manager and deletes it. **WHAT REMAINS, and it is bigger than what was fixed:** that one password is still the admin password in about **12 other namespaces** — `nextcloud`, `paperless`, `bookstack` (a **database root** password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent` (x2), `servarr`, `homepage` (x2). Rotating Gitea does not touch them: each is its own login and each is still the string that was published. **Also owed:** a Gitea token for user `kisfenyo` sits in plaintext in the `origin` URL of the local `homelab-manifests` clone (`.git/config`); CC printed it to a session transcript while investigating, so it should be rotated regardless — this is R-580's shape (store the remote without credentials). **Not a finding:** `homelab-manifests` itself is private (404 anonymously) and does not contain the password; ArgoCD's repo credential is a separate token, so the rotation did not touch it. | **NARROWED 2026-10-09 — the felhom.eu half is DONE and verified (Gitea admin password rotated, old one now 401; umami + healthchecks rotated; file de-gitted; gate carve-out removed). What remains is the ~12 reused logins elsewhere in the homelab and the repo-visibility decision; owner: operator** **2026-10-10 (release session), two consequences measured:** (a) DooPlex's own Docker login for `gitea.dooplex.hu` (`~/.docker/config.json`) still holds the OLD admin password — the registry token endpoint answers 401 for it and 200 for the current one — so every image push from DooPlex fails `unauthorized` until it is refreshed; the 0.145.0/0.305.0 pushes used a one-off login in a scratch `DOCKER_CONFIG`, shredded after. (b) The `felhom` ArgoCD app reads OutOfSync for `Secret/gitea-creds`, `Secret/healthchecks-config`, `Secret/umami-config` and `Deployment/umami`: a whole-app sync would push git's (de-gitted) view over the rotated live Secrets — the hub deploy synced ONLY `Deployment/hub` (`audits/release-2026-10-10/hub-0.145.0-deploy.txt`). Refreshing the DooPlex login is a DooPlex change (operator's word). **2026-10-22 night (Part 6, read only, `audits/night-2026-10-22/P6-repos-private/01-readers.md`):** homelab-manifests is already private; felhom-controller can go private after one change — CI clones with a read token (check first that the hub's `gitea-creds` token user can read a private `admin` repo); felhom.eu must stay public now (the website git-sync has no credential), felhom-agent too (every install fetches 16 config files from `raw/tag` with no login), and the catalog too (every box clones it with no login, and a token on a customer box is readable by its owner). Making a repository private does NOT hide its images or packages: they follow the user `admin`, which is public — and boxes need that (NOT-A-FINDING). **Count corrected:** the served `` comments are 61 on four pages (adatkezeles 33, feltetelek 15, teszteles 7, en/testing 6), not „32 on /adatkezeles”. The visibility question is decision D29 in STATUS. **2026-10-22 evening — D29 = no (operator):** the controller repository stays public for now; this row's secret rotation is unchanged. | — | **OPERATOR RULED 2026-10-09: (a) he rotates the remaining services himself — CC's scope stopped at the Felhom boundary; (b) the repo STAYS anonymously readable for now**, because making it private breaks the website git-sync and the installer tag fetch, which both clone with no credentials (R-110). The standing consequence of (b): no secret may ever enter this repo again, which `manifest_bearer_gate.py` now enforces with no exemption; and if (b) is reversed, the 61 served `` comments (four pages, 33 on /adatkezeles) must be stripped in the same change. **The exact checklist of the 12 remaining secrets** (namespace / secret / key, re-measured after the rotation, with the two traps that bite — a DB password is not changed by editing the Secret, and a pod that has run for months may not restart) is in `documentation/runbooks/secrets.md`. Operator: **(1)** work that checklist, `bookstack-db/root-password` first because it is a database root password and every one of these hosts answers on the public internet; (`bookstack` first — it is a database root password); **(2)** rotate the `kisfenyo` Gitea token embedded in the local `homelab-manifests` remote URL, and store the remote without credentials; **(3)** decide whether `gitea.dooplex.hu` should answer anonymously at all — remembering the website git-sync clones it with no credentials and the installer is fetched from a public tag, so making it private breaks both unless they get credentials first, and the 61 served `` comments become a map of it; **(4)** move `~/rotated-secrets-2026-10-09.txt` into the password manager and delete it. If nothing is done: the published password keeps opening a dozen services, even though Gitea itself is now safe | operator | | **R-1243** | Security & access | P2 | **The hub's Longhorn backups sit unencrypted in the NFS folder any DooPlex pod can read (R-1241) — the two there now (2026-10-18, 2026-10-23) hold every household's Cloudflare and tunnel tokens in plaintext, because they predate hub 0.168.0.** Measured 2026-10-23 night (`audits/hub-token-seal-2026-10-23/` §3): the hub volume is `encrypted=false` and in Longhorn's `default` backup group (daily 04:00 + weekly Sunday 05:00, retain 1 each); the hub's own copy to ep0 is encrypted at rest (16/16 snapshots `crypt-mode: encrypt`, key on DooPlex's root disk and on paper) and in transit (SSH tunnel + pinned PBS TLS), every 6 h, restore-tested weekly. DECIDE: (a) an encrypted Longhorn volume for the hub (~5–10 min downtime, a homelab StorageClass + key in the cluster) or (b) take the hub volume out of the Longhorn backup — ep0's copy is the backup; the access log then has no backup until it rides the ep0 push. **CC's pick: (b).** And: delete the two plain copies (the operator's call; with (b) they never rotate out). | **VERIFY** — **2026-10-24: D40 put into effect** (operator ruling 07:22): the hub PVC `hub-data` carries `recurring-job-group.longhorn.io/default: disabled` (felhom.eu `3363ffa8`, ArgoCD sync of the PVC only) and the Longhorn volume `pvc-486c9809…` reads `disabled` too (labelled live to match, unchanged after 2 min); the NFS folder holds no copy of the volume (no `pvc-486c9809*` directory, no backup config naming it; Longhorn lists no backup volume for it; 53 other volumes backed up this morning). Closes after the dated check: tonight's 04:00 job makes no new hub backup while the others run. WAITING-ON-OPERATOR **2026-10-25 dated check DONE (read-only, the restore-truth session):** `kubectl -n longhorn-system get backups.longhorn.io` lists 106 backups made today (the newest 04:08:28Z — the job ran) and none of `pvc-486c9809` — the hub volume is out of the Longhorn backup as D40 meant. | the operator's pick; a homelab change for either option | Operator: pick (a) or (b), and say whether to delete the two plain copies after hub 0.168.0 is live | CC (the dated check) | +| **R-1354** | Security & access | P2 | **Traefik v3.7.16 on every box has GHSA-xv5x-242w-7hf7 (high, published 2026-11-02): an unauthenticated client that sends a `Negotiate` or `NTLM` header holds a backend transport per connection until Traefik runs out of memory, NTLM backend or not (v3.7.14–v3.7.16).** Traefik fronts every box. The brief expected a false alarm: the watch had flagged only a word-only line of the same notes. | OPEN — pin committed, controller `c4b6844`; v3.7.17 ran on 9202 through 3 boots, dashboard and file manager healthy (`audits/security-2026-11-02/part3/`) | a controller release (the operator's word) | first on the morning list | CC | +| **R-1355** | Security & access | P2 | **DooPlex's own Gitea 1.26.2 is inside 19 new notices (2026-11-02, fixed 28.3.0: `GHSA-8p3x` OAuth2 account takeover, `GHSA-37xv` and `GHSA-8g99` public-only tokens reach private data, `GHSA-32h8` Actions secrets) — on the host that holds our code and registry.** Also new on DooPlex: Vaultwarden 1.37.4 (fix 1.37.5), Uptime Kuma 2.5.6 (fix 2.5.9), Paperless 3.3.4 (fix 3.4.0). Felhom changes nothing on DooPlex. Read in `documentation/security-watch/state.json`, group DooPlex. | OPEN — filed 2026-11-03 | the homelab session | add them to `audits/http2shell-2026-10-27/HOMELAB-LIST.md`'s work; which notices reach a self-hosted Gitea with our settings is the homelab session's first read | operator (homelab session) | | **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator | | **R-904** | Security & access | P4 | **Cloudflare can read every household's app traffic; replacing it with our own relay is a later item.** Facts (reviewer discussion 2026-10-08; `01`): app traffic and the dashboard reach the box through the Cloudflare Tunnel (`01` §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in `01` — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (`01` §4, §7); certificates (`01` §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (`01` §5 last row, §7); flood protection (not in `01`). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (`09` §3 decision 184): a later item. | **DEFERRED** — **2026-10-10: the asked sentence is in `01` §7** („TLS ends at Cloudflare's edge …”); the spike stays deferred. **DEFERRED — after the first customers (operator ruling 2026-10-08 09:07)** | — | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator | | **R-1040** | Security & access | P4 | **Until the household makes Sonarr's or Radarr's first admin, any container on the box can read the app's API key.** Measured on the bench 2026-10-14 with the R-1030 seed in place: `AuthenticationRequired=Enabled`, `AuthenticationMethod=None` — `/initialize.json` from another container answers 200 with `apiKey`. Outsiders are blocked by the box's setup gate in that window; a compromised other app is not. | **OPEN** — filed 2026-10-14 | — | Seed the API key at install (`SONARR__AUTH__APIKEY`, measured to work) so the box knows it, and close the window by telling the household in the first steps; or seed a Forms user | CC | @@ -272,11 +277,11 @@ stopping line that lies. | **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC | | **R-1312** | Backup & restore | P3 | **The hub customer page's off-site switch did not reach the box: on Tester 1 (2026-10-27, operator-attended, ~14:00–14:12 UTC) the box's saved off-site target stayed `enabled` and its `controller.yaml` `offsite.enabled: true` (unchanged since 2026-10-04) while the hub showed off-site OFF.** So what the switch stops on the box side is not known: the controller reads `cfg.Offsite.Enabled` only at start (`cmd/controller/main.go:890`), and nothing rewrote the file. Either the switch is hub-side only by design (then the page must say so), or a box keeps pushing off-site while the operator believes it is off. Found by the R-1218 live attempt (R-1218 closed 2026-10-27 with its live proof waived by the operator). | OPEN | — | Read the hub handler behind the switch and what it changes (config, tenancy, storage access); then one attended run: switch off, watch the box's next off-site push (does it run, does it fail at the storage) | CC | -## Box system & updates — 15 rows (P3 9, P4 6) +## Box system & updates — 16 rows (P3 10, P4 6) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1323** | Box system & updates | P3 | **When the file manager's image moves at start, the R-1281 step logs a false alarm: „could not leave traefik-public — every app can still reach it (R-1281): … No such container” — the container it tried to move was the OLD one, already replaced.** Seen on 9202 (0.343.0 → 0.346.1-rc.1, 2026-10-27 14:58:17Z); the probe a few seconds later showed the new container on `felhom-filebrowser` + `felhom-files` only (an app container: „bad address”; the control reached traefik). The text claims an exposure that is not there, which teaches the reader to ignore the real one. `audits/security-app-updates-2026-10-27/part5/9202-rc1-switch.txt` | OPEN — filed 2026-10-27 | — | in `filebrowser_gate.go`: on „No such container” read the CURRENT container's networks and log the truth (or retry once); a test with a replaced container ID; next controller release | CC | +| **R-1323** | Box system & updates | P3 | **When the file manager's image moves at start, the R-1281 step logs a false alarm: „could not leave traefik-public — every app can still reach it (R-1281): … No such container” — the container it tried to move was the OLD one, already replaced.** Seen on 9202 (0.343.0 → 0.346.1-rc.1, 2026-10-27 14:58:17Z); the probe a few seconds later showed the new container on `felhom-filebrowser` + `felhom-files` only (an app container: „bad address”; the control reached traefik). The text claims an exposure that is not there, which teaches the reader to ignore the real one. `audits/security-app-updates-2026-10-27/part5/9202-rc1-switch.txt` | **VERIFY — BUILT 2026-11-02, controller `c4b6844`, commit only.** `TestR1323_*`, red first. Not seen live: the image did not move on the test boots. OPEN — filed 2026-10-27 | — | ships with the next controller release | CC | | **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC | | **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC | @@ -285,21 +290,20 @@ stopping line that lies. | **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **2026-10-24 02:4xZ dated check read (weekend-night session, read only):** Tester 1 (8 GB host, guest 4 GB, controller 0.341.0) ran its 03:30 copy at 01:30:00–01:30:13Z (3 apps, 157 MB + 86 MB + 16 KB; no „capped copy container did not start" WARN, no leftover `felhom-tier2-copy-*`); the guest cgroup's memory „full" stall counter on the host (`/sys/fs/cgroup/lxc/9201/memory.pressure`) read **total 444 µs** over the whole 11.1 h since the host started (15:32Z), so „full" stayed far under 20 % (20 % of 10 s = 2 s); 0 OOM kills. **Not shown:** that the copy ran in the capped container (the INFO log does not name it; Docker keeps only ~200 events, from 02:36Z). **Weak load:** 0.24 GB, not the soak's large copy — so the row stays WATCHING; close after a night whose copy moves ≥ 1 GB, read from the same counter. WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC | | **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator | | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC | -| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement | — | — | CC | +| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC | +| **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC | | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC | | **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC | | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC | | **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC | -| **R-1351** | Box system & updates | P4 | **At a guest boot two start paths run `docker compose up` on FileBrowser in the same second, and one fails with „Conflict. The container name \"/_filebrowser\" is already in use" — two `[ERROR]` lines and a `[WARN] first-boot base-stack bring-up` for an app that then runs healthy.** Seen 2026-10-29 19:07:02 UTC on Tester 1 (controller 0.349.0) after the operator reboot from the hub: `[infra] filebrowser compose exists — starting without regenerating` and `[web] FileBrowser mounts synced (recreated)` at the same second; the recreate won (Docker's temporary rename `_filebrowser`), FileBrowser „Up (healthy)", files. 200. The first reboot an hour earlier showed no such line — a race, not every boot. Any guest boot can hit it (a power cut, the household's „A Felhom újraindítása"). The effect is a false ERROR in the box log (and in a box walk); nothing stays broken. | OPEN | `audits/release-2026-10-29c/final/walk-after.txt` | serialise the two FileBrowser start paths at boot (one waits for the other, or the bring-up skips a stack another path is recreating); test red first | CC | +| **R-1351** | Box system & updates | P4 | **At a guest boot two start paths run `docker compose up` on FileBrowser in the same second, and one fails with „Conflict. The container name \"/_filebrowser\" is already in use" — two `[ERROR]` lines and a `[WARN] first-boot base-stack bring-up` for an app that then runs healthy.** Seen 2026-10-29 19:07:02 UTC on Tester 1 (controller 0.349.0) after the operator reboot from the hub: `[infra] filebrowser compose exists — starting without regenerating` and `[web] FileBrowser mounts synced (recreated)` at the same second; the recreate won (Docker's temporary rename `_filebrowser`), FileBrowser „Up (healthy)", files. 200. The first reboot an hour earlier showed no such line — a race, not every boot. Any guest boot can hit it (a power cut, the household's „A Felhom újraindítása"). The effect is a false ERROR in the box log (and in a box walk); nothing stays broken. | **VERIFY — BUILT 2026-11-02, controller `c4b6844`, commit only.** One lock over both compose paths; `TestR1351_*`, seen failing without the lock. **Live: no Conflict and no first-boot WARN on any of 6 boots of 9202 (3 on each image, also with the sync recreating and the file manager held down so both paths ran in one second), so these boots do not tell the images apart: the race did not fire on the old image either. The proof is the test, stated plainly.** OPEN | `audits/release-2026-10-29c/final/walk-after.txt` | ships with the next controller release; read the next reboots' lines | CC | -## Monitoring & notifications — 24 rows (P2 2, P3 14, P4 8) +## Monitoring & notifications — 25 rows (P2 3, P3 14, P4 8) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-1262** | Monitoring & notifications | P3 | **The upload watch treats ANY viewer as an explanation for any amount of upload, so a hacked Plex can send freely while one family member watches.** Found 2026-10-24 (SEC-1 X1): `felhom-controller/controller/internal/upwatch/watch.go:275,299`; `08` §9 states the rule, its limits list did not name this case. | OPEN | — | propose a cap (viewers × a bitrate ceiling) for the operator's R-1244 design; `08`'s limits list names the case meanwhile | CC | -| **R-1310** | Monitoring & notifications | P3 | **The security watch (R-1303) flags notices that do not apply and misses fixes that do — measured by reading all 38 flagged pins by hand on 2026-10-27.** False: advisories of ecosystem `actions` (Home Assistant's CI), of another package in the repo (Gitea's runner), with a hostname or ≤-old range (cal.com 2023/2024, Grafana 2022), for the desktop/Electron client (Termix), for the rootless image only (Uptime Kuma); a release hit on a word (Dawarich „no longer receives security fixes", Rallly v4.16.0). Wrong „no fix": Vikunja's six (range `<= 2.6.0`, fixed in 2.7.0), karakeep-chrome (154/156 exist). Missed: Tandoor 2.7.0 names 8 GHSA ids (1 matched), Dawarich 1.16.0 (2FA, export deletion), Wishlist v0.69.0 (critical, no GHSA). BEHIND reads linuxserver `26.09.20260929` as newer than 26.09.2; Seerr follows the renamed-away `fallenbagel/jellyseerr`. `audits/security-app-updates-2026-10-27/TRIAGE.md` NOT-A-FINDING list and `triage-notes/` | OPEN — filed 2026-10-27; **two shapes FIXED the same day** (ecosystem `actions` not listed; an empty fix with a closed range reads „after X"; test W9, red first) | — | teach `security_watch.py`: skip a package that is not the pinned one; match every GHSA id a release body names; ignore `YY.MM.YYYYMMDD` tags; repoint seerr to `seerr-team/seerr`; each with a test seen failing | CC | -| **R-1334** | Monitoring & notifications | P3 | **Docker Hub's tag API refuses the watch (HTTP 429) since the DooPlex group was added: 19 DooPlex images NOT CHECKED in the 2026-10-28 night run (job 2446, `hub.docker.com` = 211 requests), and the „could not check registry:docker.io" mail went out (streak 2).** The runtime part is not affected (HEAD + cache). ghcr.io also answered one blob with 429 in the dry run of 07:02 (Immich v3.3.5, while this session read images from the same address). | **VERIFY** — BUILT 2026-10-28 midday, measured first (`audits/dockerhub-bigsteps-2026-10-28/part1/`): the 429s came from the BURST, not the count — night 2446 sent 211 tag-list requests in ~2 min from six workers (15 refused), dry run 2450 sent 224 in 3.5 min (none). `fe677bae`: Docker Hub requests spaced (1/s across the workers), a 429 retried with its Retry-After (NOT CHECKED only after the 4th try), tag lists cached 24 h per registry path on the runner (two names of one path fetch once); `8644450d`: the image reader retries a registry 429 too (ghcr.io refused two blob reads in back-to-back dry runs). Tests W14 + retry checks, red first. Dry runs 2458 and 2459: **0 Docker Hub 429, 0 retries** (210 requests over ~7 min). **No Docker Hub login needed.** Each of the two CI runners keeps its own tag cache. `minio/minio:latest` answers 401 — Docker Hub no longer serves it (homelab, R-1328). | — | close after the first CI night with 0 Docker Hub 429 (the 03:40 run of 2026-10-29) | CC | | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC | | **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC | | **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says )" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC | @@ -320,7 +324,10 @@ stopping line that lies. | **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC | | **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC | | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | -| **R-1352** | Monitoring & notifications | P4 | **When a controller stops answering AND the agent restarts it between two host reports, the operator gets the restart mail first and the „does not answer — the agent restarts it in 2 minutes" alarm second, in the same second.** Seen live 2026-10-30 (the R-1344 proof on Tester 1: alarm 09:29:21, restart 09:31:22, both reached the hub with the 09:42:56 report; `hub/internal/monitor/controller_supervisor.go` `observe()` emits the restart before the alarm). The text of each mail is true; only the order reads backwards. | OPEN | — | emit the alarm before the restart in `observe()`, or skip the alarm when the restart for the same episode arrives with it (one line + a test) | CC | +| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03 | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC | +| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC | +| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) | +| **R-1352** | Monitoring & notifications | P4 | **When a controller stops answering AND the agent restarts it between two host reports, the operator gets the restart mail first and the „does not answer — the agent restarts it in 2 minutes" alarm second, in the same second.** Seen live 2026-10-30 (the R-1344 proof on Tester 1: alarm 09:29:21, restart 09:31:22, both reached the hub with the 09:42:56 report; `hub/internal/monitor/controller_supervisor.go` `observe()` emits the restart before the alarm). The text of each mail is true; only the order reads backwards. | **VERIFY — BUILT 2026-11-02, hub `2f7e836f`, commit only.** The alarm of an episode whose restart arrives in the same report is not emitted; a new episode still alarms. `TestR1352_*`, red first. OPEN | — | ships with the next hub release | CC | ## Hub & operator — 14 rows (P3 3, P4 11) @@ -339,7 +346,7 @@ stopping line that lies. | **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator | | **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC | | **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC | -| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC | +| **R-1353** | Hub & operator | P4 | **Statements still over 1 s, named by the new slow-statement line (hub 0.178.0) in its first hour:** a background deadline/restore-test check reading days of full host-report JSON (`GetHostReportsSince`, `store.go`; 1.5–9.2 s cold, from `monitor/deadline.go` and `monitor/restoretest.go` — no page waits on it); the start-up prune of `events` and `app_log_issues` (1.1 s and 1.8 s; no index on `created_at` / `last_seen`); the customer page's newest-report read (`GetCustomer`, 1.3 s cold in the pod probe). With the 1 GiB limit most of this should come from the cache once warm. Evidence: the hub log 2026-10-30 09:22–10:05 UTC; `audits/release-2026-10-30/p2-r1346/`. | OPEN **2026-11-02 read (`audits/security-2026-11-02/part4/hub-slow-lines-2026-11-01-02.txt`; the log covers only the 31 h since DooPlex restarted on 2026-11-01 16:11 CET, the lines from 10-30 are gone with the old pod):** 35 lines, 15 statements. What repeats: `VACUUM INTO` 5× (avg 110 s, the 6-hourly snapshot, expected); `GetHostReportsSince` 9× (avg 7.9 s, max 26.6 s); `GetCustomer`'s newest report 3× (1.1 s). Once, at the cold start: the prune of `events` (10.4 s) and `app_log_issues` (21 s, again 04:30), the OS-release and host lists (2–22 s). No fix before the full week (11-06). | — | read a week of slow-statement lines; index or narrow what still repeats (the history read needs only a few fields, not the whole JSON) | CC | ## Business & legal — 14 rows (P2 4, P3 2, P4 8) @@ -412,4 +419,4 @@ stopping line that lies. | R-1228 | 2026-11-06 | D35 way back ends: ask the operator to retire DooPlex's felhom.eu/www webpage copy (ingress + git-sync), or keep it longer; detail in the R-1228 row | | **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC | -| **R-1302** | Monitoring & notifications | P4 | **A fresh install's agent logs ERROR „store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" and „capability DEGRADED" for `felhom-pbs` at its FIRST start, although the grant was simply never made yet** — the whole-guest tier is armed (21:00:45) before the PBS storage exists (created 21:00:57 on the hub's descriptor). Seen 2026-10-26 night on the R-1232 BYO box (VM 9375, `audits/night-fixes-2026-10-26/r1232/3-first-run.txt`) and on 2026-10-25's BYO box (`degraded=pve:store-grant:felhom-pbs`). An ERROR that says „investigate the loss" on every new box teaches the reader to skip it. | OPEN | — | on the first grant of a tier whose storage did not exist before, log INFO „granted" instead of the loss ERROR; keep the ERROR for a grant that existed and vanished; a test for both | CC | +| **R-1302** | Monitoring & notifications | P4 | **A fresh install's agent logs ERROR „store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" and „capability DEGRADED" for `felhom-pbs` at its FIRST start, although the grant was simply never made yet** — the whole-guest tier is armed (21:00:45) before the PBS storage exists (created 21:00:57 on the hub's descriptor). Seen 2026-10-26 night on the R-1232 BYO box (VM 9375, `audits/night-fixes-2026-10-26/r1232/3-first-run.txt`) and on 2026-10-25's BYO box (`degraded=pve:store-grant:felhom-pbs`). An ERROR that says „investigate the loss" on every new box teaches the reader to skip it. | **VERIFY — BUILT 2026-11-02, agent `50eeccc`, commit only.** `/store-grant-seen.json`: a first grant logs INFO and reports healthy; a grant that was held and vanished keeps the ERROR; a doubt keeps it too. `TestR1302_*`, red first, four mutants. Not seen live: no agent release tonight. OPEN | — | ships with the next agent release (vouch) | CC | diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index 3478cd76..c34173e5 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.** +## Moved from STATUS on 2026-11-03 + +**Updated 2026-10-30.** On your yes, felhom.eu and the installer are served by Cloudflare (DooPlex's copy stays until 11-06 as the way back), and the contact form posts to api.felhom.eu. Hub 0.178.0 (1 GiB) saves your vouch in 0.13 s instead of 38 s and opens the apps page in under a second. Agent 0.171.0 (your vouch) restarts a controller that runs but does not answer — proven on Tester 1. + ## Moved from STATUS on 2026-10-30 **Updated 2026-10-29 (evening release).** On your yes: hub 0.177.0, agent 0.170.0 (your vouch) and controller 0.349.0 for every box; „Reboot guest" on the hub works and refuses during backups, updates and twice in an hour, and no alarm follows your hold clear any more — proven on Tester 1. The slow vouch is the hub's own database step (38 of 38.3 s); the stuck-controller decision waits for tomorrow's night log.