> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
> **2026-11-03 (day, attended) — RELEASED: controller v0.350.0 (the global floor since 08:18 CET, MinAgent 0.158.1, `sha256:9ee53908…` from `5c336d82`), agent v0.172.0, golden 0.350.0 (both VOUCHED 10:18:10 CET), hub v0.179.0 (09:05 CET), catalog `f4c5c79`.** Traefik v3.7.17 on every box (R-1354): on 9202 v3.7.16 grew 18.7 → 79.0 MiB after 920 NTLM connections, v3.7.17 stayed near 25 MiB for 25,116. Golden 0.349.0 was never vouched. The fresh install ran the public installer with one added line naming the unvouched pair, because an install refuses any golden but the vouched one (R-297). A reinstall of a customer whose controller has reported is refused until the operator deletes the host, however long one waits (R-1159). R-1302's ERROR is a BYO box's: the appliance path pre-positions the grant. Report: `REPORT-release-2026-11-03.md`; evidence `documentation/audits/release-2026-11-03/`.
>
> **2026-11-02/03 (night, unattended) — the security watch and the new notices; no release.** (1) **R-1334's fix held:** 0 Docker Hub 429 on five nights. A streak is per SOURCE, so seven pins that can never be read (`gitea.dooplex.hu/admin/*`×6, `minio/minio`) held `registry:docker.io` and `runtime` open: `security_watch.py``STATED_UNREADABLE` + private registry = GAP with a reason; `Transient` retried on every source (budget 30); `FAILED-SOURCE` lines; R-1310's rules (`PACKAGE_NOT_OURS`, `WORD_ONLY`, one entry per GHSA, date tags). (2) **Traefik v3.7.16 has GHSA-xv5x (high):** pin `c4b6844`, unreleased (R-1354). (3) **Catalog live `c760fca`:** gitea 28.3.0, uptime-kuma 2.5.9 (crosses 2.5.8's TLS removal on TCP monitors — decided by CC unattended, operator may reverse), grafana 13.2.5, actualbudget 26.11.0, n8n 2.46.4, vaultwarden 1.37.5-alpine. (4) **Commit only:** controller `c4b6844` + `3764445` (R-1347, R-1348, R-1339, R-1351, R-1323), agent `50eeccc` (R-1302), hub `2f7e836f` (R-1352). (5) **Walks:** 8 claims moved to walked (31 → 23 not walked). R-1361: an event sent while the hub is away is lost after 9 s. R-1362, R-1364 filed. Report: `REPORT-security-2026-11-02.md`; evidence `documentation/audits/security-2026-11-02/`.
> **2026-11-02/03 (night, unattended) — the security watch and the new notices; no release.** (1) **R-1334's fix held:** 0 Docker Hub 429 on five nights. A streak is per SOURCE, so seven pins that can never be read (`gitea.dooplex.hu/admin/*`×6, `minio/minio`) held `registry:docker.io` and `runtime` open: `security_watch.py``STATED_UNREADABLE` + private registry = GAP with a reason; `Transient` retried on every source (budget 30); `FAILED-SOURCE` lines; R-1310's rules (`PACKAGE_NOT_OURS`, `WORD_ONLY`, one entry per GHSA, date tags). (2) **Traefik v3.7.16 has GHSA-xv5x (high):** pin `c4b6844`, unreleased (R-1354). (3) **Catalog live `c760fca`:** gitea 28.3.0, uptime-kuma 2.5.9 (crosses 2.5.8's TLS removal on TCP monitors — decided by CC unattended, operator may reverse), grafana 13.2.5, actualbudget 26.11.0, n8n 2.46.4, vaultwarden 1.37.5-alpine. (4) **Commit only:** controller `c4b6844` + `3764445` (R-1347, R-1348, R-1339, R-1351, R-1323), agent `50eeccc` (R-1302), hub `2f7e836f` (R-1352). (5) **Walks:** 8 claims moved to walked (31 → 23 not walked). R-1361: an event sent while the hub is away is lost after 9 s. R-1362, R-1364 filed. Report: `REPORT-security-2026-11-02.md`; evidence `documentation/audits/security-2026-11-02/`.
> **2026-10-30 (day, attended) — RELEASED: hub v0.178.0 (1 GiB), agent v0.171.0 (VOUCHED 10:47:36 CET); the site on
> **2026-10-30 (day, attended) — RELEASED: hub v0.178.0 (1 GiB), agent v0.171.0 (VOUCHED 10:47:36 CET); the site on
# REPORT — release run 2026-11-03 (attended): the Traefik fix first, then the golden, agent, hub, catalog
## Five lines
1.**Every box runs controller 0.350.0 with Traefik v3.7.17:** yes. The global floor at 08:18:16 CET; the three boxes moved at 08:05 CET on per-box floors; 9202 by hand.
2.**The load test (9202 only, from the LAN):** v3.7.16 grew from 18.7 to 79.0 MiB after 920 connections and did not release it (stopped at 11 s by the bound). v3.7.17 stayed near 25 MiB for 25,116 connections and the whole 300 s. Proven by behaviour.
3.**Golden 0.350.0 installed fresh and vouched:** yes, 10:18:10 CET. The install was the closest path, not the hub's: the public installer with one added line (the hub refuses an unvouched golden, R-297).
Evidence: `documentation/audits/release-2026-11-03/` (each part has a README), `documentation/tests/golden-0.350.0-2026-11-03/`.
## Said plainly
- **My mistakes.** The first fresh install did not use the test installer: I set the variable in a place the bootstrap blanks, and I read the result only at the end. Then I told the operator a 45-minute wait would open the reinstall; it does not (R-1159), and 12 retries got the same refusal. It cost one operator act (the host delete) and about 90 minutes.
- **Scripted on purpose:** the uninstall's typed confirmation went through `ssh -tt`, on my own test VM only.
- **Proof limits.** R-1351, R-1323 and R-1352 have test proof only: their conditions did not fire. R-1347 and R-1348 were proven on the test image of the same code; the release build ran a plain restore. The file manager sign-in was not driven on the N100, which is lived-in; its anonymous page answered 200. R-1302 was staged: a plain first start on the appliance path reaches neither line.
- **ep0.** I changed nothing there by hand. The fresh installs made the hub re-issue Tester4's PBS token on ep0, twice: the product's own cascade on a customer's new host.
## Observations
- FILED (appended): R-548 — demo-hp's local backup tier is refused since 10-30, the disk has too little room.
- FILED (appended): R-1361 — the night walk's lost `db_dump_completed` became a false `expected_dbdump_missed` mail on Tester 1.
- NOT-A-FINDING: `Tester4: passphrase_keys_refused`, `host_stale`, `node_stale`, `offsite_delivery_stuck` mails — CC's own removal of a test install and its retries; the guard was right.
- NOT-A-FINDING: Traefik's repo digest on 9202 differs from the other boxes — the image id is the same on all; 9202 pulled the tag on the night of 11-02, the others this morning. The tag's index was not read.
- NOT-A-FINDING: `FELHOM_INSTALL_URL` in a unit drop-in is ignored — the bootstrap reads it only from its env file, by design.
- NOT-A-FINDING: ArgoCD's `Ingress/contact-mailer` OutOfSync — the live object matches git field by field (the same reading as 2026-10-30).
- NOT-A-FINDING: the slow-statement WARN lines on the hub — R-1353's class, already a row.
## Teardown
VM 9383 destroyed with its disks; drill VM off, back on `virgin`; bench 9401 stopped; 9202 on 0.350.0, its `radicale` hold left for the operator; per-box floors cleared; DooPlex's clones and scratch secrets removed. **Hub:** host `Tester4-e286d3` waits for the operator, from about 11:00 CET.
**Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).**
**Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-11-03 (morning).** On your word, golden 0.349.0 is baked; the code-push gate is green again. It needs your vouch (first line of the release list). It carries Traefik v3.7.16 until controller 0.350.0 (R-1354).
**Updated 2026-11-03.** Every box runs controller 0.350.0 with Traefik v3.7.17; the hole is closed, proven on the scratch box. Golden 0.350.0 and agent 0.172.0 are vouched (10:18 CET), hub 0.179.0 is live, and Paperless 3, n8n 3, Mealie and Termix are offered.
## What works
## What works
- **The Plex attack did not reach Felhom's secrets** (R-1241).
- **The Plex attack did not reach Felhom's secrets** (R-1241).
@@ -24,7 +24,7 @@
- A box's event is lost if the hub is away for 9 seconds; a DooPlex restart can swallow a mail (R-1361).
- A box's event is lost if the hub is away for 9 seconds; a DooPlex restart can swallow a mail (R-1361).
## What we work on
## What we work on
- HTTP2Shell waiters: cal.com, Calibre-Web, RomM have no fix; Paperless, SparkyFitness have one on a new main version (R-1331; R-785, behind R-784).
- HTTP2Shell waiters: cal.com, Calibre-Web, RomM have no fix; SparkyFitness has one on a new main version (R-785, behind R-784).
- Dated checks, run by CC. 2026-11-06: ask you to retire DooPlex's old copy of the website (R-1228).
- Dated checks, run by CC. 2026-11-06: ask you to retire DooPlex's old copy of the website (R-1228).
## Waiting on you — decide
## Waiting on you — decide
@@ -49,7 +49,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). (since 10-14)
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). (since 10-14)
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. (since 10-14)
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. (since 10-14)
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). (since 10-12)
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). (since 10-12)
**2026-11-03 release list** (your word). 0. **Vouch golden 0.349.0** (agent 0.171.0, min_agent 0.158.1), sha `9d97e7c5ec3e…`, in full in `documentation/tests/golden-0.349.0-2026-11-03/README.md`. If not: a new install keeps getting 0.344.0, and the floor stays above the vouched golden. 1. Controller v0.350.0 (`3764445`): Traefik v3.7.17 (R-1354), R-1347, R-1348, R-1339, R-1351, R-1323. If not: every box keeps the hole. 2. Agent v0.172.0 + vouch: R-1302. 3. Hub v0.179.0: R-1352. 4. Catalog, proven on the drill: Paperless 3.4.1 (R-1331), n8n 3.0.1 (R-1356), Mealie, Termix (R-1359). 5. To the homelab session: R-1355.
**2026-11-03.** 1. From about 11:00 CET: delete host `Tester4-e286d3` on the Hosts page (CC's proof VM, gone). If not: stale mails. 2. To the homelab session: R-1355.
**2026-10-30.** On the hub's Hosts page: Discard the appliance of 10:44:59 (uuid d3529d2d…) and appliance 49 (CC's proof VMs). If not: dead entries alarm. **2026-10-29 (release run).** 1. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes.
**2026-10-30.** On the hub's Hosts page: Discard the appliance of 10:44:59 (uuid d3529d2d…) and appliance 49 (CC's proof VMs). If not: dead entries alarm. **2026-10-29 (release run).** 1. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes.
**2026-10-23 do list.**
**2026-10-23 do list.**
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
paperless-ngx {'hu': [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}], 'en': [{'title': 'A newer version of this app is available. Select
# Part 2 — golden 0.350.0, one fresh install, 2026-11-03
The bake: `documentation/tests/golden-0.350.0-2026-11-03/`. VM 9383 `release1103-box` on demo-hp (6 cores, 10 GiB, raw on `nvme-scratch`, never the thin pool), from the public ISO 1.30.0 (sha256 = the published one), graphical installer driven by `qm sendkey` (`install/`).
**The path, exactly.** A hub-offered install refuses any golden but the vouched one (R-297), and the brief wanted the proof before the vouch. So: the public installer `felhom-host-install.sh` 1.36.1, fetched from felhom.eu (sha `e18cdd17…`), with **one added line** (`installer-the-one-line.diff.txt`) that replaces the manifest's agent + golden with 0.172.0 + 0.350.0, run by hand as root in the DIRECT shape of `felhom-bootstrap.sh` (`--customer-id Tester4 --mode appliance --hub-url … --passphrase-file …`), not through the bind page. Everything else is the real path: the hub's enroll, the fetch from Gitea, the sha check, the provision.
**Three tries, said plainly.**
1. A unit drop-in set `FELHOM_INSTALL_URL`, but the bootstrap blanks that variable and reads only `/etc/felhom/bootstrap.env`, so the box bound (bind page, 07:30:55 UTC) and installed the VOUCHED pair: golden 0.344.0, agent 0.171.0, moved by the floor to 0.350.0 about 2 minutes later (`first-run-vouched-manifest-journal.txt`). It proves the normal path and the floor, not the new golden. CC's mistake; read only at the end. Removed with `--uninstall` (`uninstall-first-run.txt`; the typed confirmation fed through `ssh -tt` on CC's own test VM).
2. The hub refused the reinstall: 12 tries (CC expected a 45-minute limit; there is none: once a controller has reported, the passphrase stays closed until the operator deletes the host, R-1159). The operator deleted the host at 09:08:12 UTC.
**The hub's own acts on this customer:** the fresh installs made the hub re-issue Tester4's PBS token on ep0 (`pbsdr F-14 auto re-issue`, gen 2) and its off-site key, twice. That is the product's normal hands-free cascade; CC touched nothing on ep0 by hand.
**Teardown** (`teardown.txt`): VM 9383 destroyed with its disks; the typed passwords shredded. **Left for the operator:** delete host `Tester4-e286d3` on the Hosts page, from about 11:00 CET (the 45-minute rule).
**Vouch:** the operator, 10:18:10 CET: golden 0.350.0, agent 0.172.0, bundle `cc34b0bc…`, min_agent 0.158.1 (`vouch-readback.txt`: the hub's log and its database with the WAL agree).
`release-agent.sh` from a clean clone of `ecbb0cc` (CI 2563 green): agent sha256 `2c0222cd01713f65c627d6d17c18b4a8b2b1e0c39881db0df7ccb17de48e24c7`, bundle `cc34b0bc3346ffd6a4ba80fe9269852dced8c56c9503c8b14a50d9ded2961cce`, tag `v0.172.0`. `configs/` unchanged since 0.171.0, so the signed delivery was `agent_update` alone.
**Delivery** (`agent-0172-sign.txt`, `agent-0172-delivery.txt`): demo-felhom, demo-hp, Tester 1: signed op COMPLETED, self-check 77 of 77, `selfupdate: update committed` by 07:58:34 UTC.
**The fix live** (`r1302-live.txt`, `r1302_live.sh`, on the throwaway box). **A plain first start does not reach this code on the appliance path:** the installer pre-positions the grant, and the control run on agent 0.171.0 (Part 2's first try) logged no R-190 line either. The ERROR was seen on BYO boxes. So it was staged:
- A, a true first grant: no seen file, the grant taken away, the agent started → INFO „store-grant: granted — the first grant of this tier, nothing was lost (R-1302)", self-check degraded=0, 0 R-190 lines.
- B, a loss: the seen file there, the grant taken away, the agent started → ERROR „GRANT WAS MISSING … (R-190)", degraded=1.
**Vouch:** the operator, 10:18:10 CET, with golden 0.350.0. `pinned.day0_agent` = 0.172.0 (D50).
Nov 03 08:57:30 felhom felhom-agent[2609044]: time=2026-11-03T08:57:30.595+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=e7c906876dcfdb8f op=agent_update
Nov 03 08:56:42 demo-hp felhom-agent[1819483]: time=2026-11-03T08:56:42.162+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=55bc053ef33f6cf2 op=agent_update
Nov 03 08:49:36 demo-felhom felhom-agent[2017068]: time=2026-11-03T08:49:36.224+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=37085279fd83f188 op=agent_update
time=2026-11-03T10:12:24.251+01:00 level=INFO msg="store-grant: granted — the first grant of this tier, nothing was lost (R-1302)" target=felhom-pbs privilege=Datastore.AllocateSpace action="felhom-backup-target-apply grant felh
time=2026-11-03T10:12:50.661+01:00 level=ERROR msg="store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" target=felhom-pbs privilege=Datastore.AllocateSpace action="felhom-backup-target-apply
time=2026-11-03T10:12:50.661+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:store-grant:felhom-pbs feature="backup tier felhom-pbs: the agent's storage grant
# R-1302 staged on the throwaway box (VM 9383, agent 0.172.0). The appliance path pre-positions the grant, so a plain first start never reaches the repair (read in the install log).
# A: a true first grant: no seen file, the grant taken away, the agent started. B: a loss: the seen file there, the grant taken away, the agent started.
del(){for k in "--users felhom-agent@pve""--tokens felhom-agent@pve!agent";do pveum acl delete /storage/felhom-pbs $k --roles FelhomAgentStore;done;}
pveum acl list --output-format json | python3 -c "import json,sys;print('grant back:',sum(1 for a in json.load(sys.stdin) if a['path']=='/storage/felhom-pbs'))"
Tests green (`go build`, `vet`, `test`); image `sha256:9c52c1c3…` from a clean clone of `57d46201` (CI 2570 green). The operator's yes; manifest `01a44d56`; only `Deployment/hub` synced, 08:05:10 UTC. Running 0.179.0, Healthy; ArgoCD's `OutOfSync` is the known `Ingress/contact-mailer` line (the live object matches git field by field, as on 2026-10-30). `watch-after-deploy.txt`: 50 minutes, 0 ERROR lines, 0 restarts, no new event from the three boxes.
R-1352 itself did not fire in that time: it needs a controller that stops answering. Proof by test only (`TestR1352_*`).
| **R-1354** | **Traefik v3.7.16 on every box had GHSA-xv5x-242w-7hf7: unauthenticated Negotiate/NTLM headers hold a backend transport per connection until memory runs out.** (P2) | CLOSED 2026-11-03 — controller 0.350.0 (Traefik v3.7.17), the global floor since 08:18 CET; every box reports it (Tester 1, demo-hp, demo-felhom, 9202), golden 0.350.0 carries it. **Proven by behaviour on 9202:** v3.7.16 18.7 → 79.0 MiB after 920 connections, not released; v3.7.17 flat near 25 MiB for 25,116 | `audits/release-2026-11-03/``part1/README.md`, `load-before.txt`, `load-after.txt`; controller `c4b6844`, `5c336d82` |
| **R-1347** | **A restore that lifts an update hold left the stack manager's „update held" flag set.** (P3) | CLOSED 2026-11-03 — controller 0.350.0. Proven live on the test image of the same code (2026-11-03 night); a plain restore on Tester 1 on the release build ran clean, not the staged fault | `audits/security-2026-11-02/part3/`; `audits/release-2026-11-03/``part1/restore-tester1.txt`; controller `c4b6844` |
| **R-1348** | **A restore_mixed hold written at start-up sent no operator mail.** (P3) | CLOSED 2026-11-03 — controller 0.350.0. Proven live on the test image (the event with the R-1340 sentence reached the sink); the release build ran a plain restore, not the staged fault | `audits/security-2026-11-02/part3/r1348-after.txt`; controller `c4b6844`, `3764445` |
| **R-1339** | **False „drive … not mounted" lines for per-app folders at every start.** (P4) | CLOSED 2026-11-03 — controller 0.350.0: nine false lines → none on the test image; 0 ERROR lines on the three boxes. The row's own fix was not built (it would write `userdata/` into app folders): only the false line goes | `audits/security-2026-11-02/part3/`; controller `c4b6844` |
| **R-1351** | **FileBrowser's two start paths at boot raced: a false „Conflict" ERROR.** (P4) | CLOSED 2026-11-03 — controller 0.350.0, one lock. **Test proof only:** the race did not fire on six boots of either image, nor on this release's starts (0 ERROR lines). If it is seen again on ≥ 0.350.0, this row reopens | `TestR1351_*`; `audits/security-2026-11-02/part3/`; controller `c4b6844` |
| **R-1323** | **„No such container" was logged as a false alarm that every app can still reach the file manager.** (P3) | CLOSED 2026-11-03 — controller 0.350.0. **Test proof only:** the file manager's image did not move on these starts | `TestR1323_*`; controller `c4b6844` |
| **R-1302** | **A fresh install's agent logged ERROR „GRANT WAS MISSING … investigate the loss" at its first start.** (P4) | CLOSED 2026-11-03 — agent 0.172.0, vouched 10:18:10 CET. Staged live on a fresh box: a true first grant logs INFO „granted" and no R-190 line; a loss keeps the ERROR. Seen meanwhile: the appliance path pre-positions the grant, so a plain first start there logs neither; the ERROR was a BYO box's | `audits/release-2026-11-03/``part3/r1302-live.txt`; agent `50eeccc` |
| **R-1352** | **The restart mail arrived before the „does not answer" alarm of the same episode.** (P4) | CLOSED 2026-11-03 — hub 0.179.0, live since 09:05 CET. **Test proof only:** no controller stopped answering in the watch | `TestR1352_*`; `audits/release-2026-11-03/``part4/`; hub `2f7e836f` |
| **R-1331** | **Paperless-ngx 2.20.15 carried Python Brotli 1.1.0 (CVE-2026-111497).** (P3) | CLOSED 2026-11-03 — the big step 2.20.15 → 3.4.1 is pushed on the operator's yes (bench, 9202, undo proven); offered to every box | `audits/security-2026-11-02/bench/paperless-ngx/`, `walks/paperless-drill/`; catalog `f4c5c79` |
| **R-1356** | **n8n kept three notices fixed only in 3.0.1 (`GHSA-9p7v` high, `GHSA-mx4r`, `GHSA-2x68`).** (P3) | CLOSED 2026-11-03 — the big step 2.46.4 → 3.0.1 is pushed on the operator's yes (bench, 9202, undo proven); offered to every box | `audits/security-2026-11-02/bench/n8n/`, `walks/n8n3-drill/`; catalog `f4c5c79` |
| **R-1359** | **Two dependency fixes with no public id: Mealie v3.30.0 (`pillow`), Termix 26.11.0 (`body-parser`).** (P4) | CLOSED 2026-11-03 — normal steps pushed on the operator's yes (bench and 9202 proven) | `audits/security-2026-11-02/bench/{mealie,termix}/`, `walks/{mealie,termix}-drill/`; catalog `f4c5c79` |
| **R-1334** | **Docker Hub's tag API refuses the watch (HTTP 429).** (P3) | CLOSED 2026-11-03 — the 2026-10-28 fix held: 0 Docker Hub 429 on five real nights and both runners. What looked like its failure was another cause (R-1303's streak): seven pins that can never be read, now a stated GAP, plus read timeouts, now retried. `failures` empty after run 2550 and the 03:40 night run | `audits/security-2026-11-02/part1/DIAGNOSIS.md`; `scripts/security_watch.py` W12, W15; felhom.eu `4afa8c3c` |
| **R-1334** | **Docker Hub's tag API refuses the watch (HTTP 429).** (P3) | CLOSED 2026-11-03 — the 2026-10-28 fix held: 0 Docker Hub 429 on five real nights and both runners. What looked like its failure was another cause (R-1303's streak): seven pins that can never be read, now a stated GAP, plus read timeouts, now retried. `failures` empty after run 2550 and the 03:40 night run | `audits/security-2026-11-02/part1/DIAGNOSIS.md`; `scripts/security_watch.py` W12, W15; felhom.eu `4afa8c3c` |
| **R-1310** | **The security watch flags notices that do not apply and misses fixes that do.** (P3) | CLOSED 2026-11-03 — another program's package is not listed (`PACKAGE_NOT_OURS`), every GHSA a release names is its own entry, a word-only line is no ADVISORY (`WORD_ONLY`), `YY.MM.YYYYMMDD` tags are not newer; seerr was already repointed in the catalog. First catch: Traefik's GHSA-xv5x sat under a word-only line | W16, red first, 13 mutants; felhom.eu `4afa8c3c` |
| **R-1310** | **The security watch flags notices that do not apply and misses fixes that do.** (P3) | CLOSED 2026-11-03 — another program's package is not listed (`PACKAGE_NOT_OURS`), every GHSA a release names is its own entry, a word-only line is no ADVISORY (`WORD_ONLY`), `YY.MM.YYYYMMDD` tags are not newer; seerr was already repointed in the catalog. First catch: Traefik's GHSA-xv5x sat under a word-only line | W16, red first, 13 mutants; felhom.eu `4afa8c3c` |
| **R-1336** | **ActualBudget had no release with the HTTP2Shell fix.** (P3) | CLOSED 2026-11-03 — 26.11.0 carries Node 24.22.1, read from the image; a `security_fix` step 26.9.0 → 26.11.0 proven on 9202, live | catalog `c760fca`; `audits/security-2026-11-02/walks/actualbudget/` |
| **R-1336** | **ActualBudget had no release with the HTTP2Shell fix.** (P3) | CLOSED 2026-11-03 — 26.11.0 carries Node 24.22.1, read from the image; a `security_fix` step 26.9.0 → 26.11.0 proven on 9202, live | catalog `c760fca`; `audits/security-2026-11-02/walks/actualbudget/` |
| **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo/<app>/`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC |
| **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo/<app>/`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC |
| **R-1329** | Apps & catalog | P3 | **cal.com runs on Node.js 20, which gets no security fixes any more** — v6.2.0 carries Node 20.20.0 (no fixed release on the 20 line for CVE-2026-111497) and Debian 12's unpatched `libbrotli1` 1.0.9-2+b6 (fixed: 1.0.9-2+deb12u1); the same image runs in the catalog and on DooPlex. Read from the image 2026-10-27; 2026-10-28: v6.2.0 is still the newest tag, no Node 22/24 image (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN **2026-11-02:** v6.2.0 is still the newest; its image still holds Node 20.20.0. | — | a cal.com release on Node 22/24 (a big step: its own walk); until then the watch lists it nightly | CC |
| **R-1329** | Apps & catalog | P3 | **cal.com runs on Node.js 20, which gets no security fixes any more** — v6.2.0 carries Node 20.20.0 (no fixed release on the 20 line for CVE-2026-111497) and Debian 12's unpatched `libbrotli1` 1.0.9-2+b6 (fixed: 1.0.9-2+deb12u1); the same image runs in the catalog and on DooPlex. Read from the image 2026-10-27; 2026-10-28: v6.2.0 is still the newest tag, no Node 22/24 image (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN **2026-11-02:** v6.2.0 is still the newest; its image still holds Node 20.20.0. | — | a cal.com release on Node 22/24 (a big step: its own walk); until then the watch lists it nightly | CC |
| **R-1330** | Apps & catalog | P2 | **RomM's newest release (5.4.1) still ships Node.js 24.18.1 (CVE-2026-111497, fixed 24.22.1)** — no newer RomM exists (tags read 2026-10-27). Exposure: Node is IN the image but no Node process runs (RomM serves through Python gunicorn/rq and nginx — `ps` on demo-hp 9201), and RomM sits behind the setup gate. | OPEN — **RULED D61 = keep RomM offered** (operator 2026-10-28, starting the day brief; `09` §3 decision 277): the first RomM release with Node ≥ 24.22.1 is a lane step. 2026-10-28: 5.4.1 is still the newest tag (read from the registry). Earlier: CC's pick 2026-10-27 night **2026-11-02:** 5.4.1 is still the newest tag; its image still holds Node 24.18.1. | — | the watch flags it nightly (R-1327); the step when RomM ships | CC |
| **R-1330** | Apps & catalog | P2 | **RomM's newest release (5.4.1) still ships Node.js 24.18.1 (CVE-2026-111497, fixed 24.22.1)** — no newer RomM exists (tags read 2026-10-27). Exposure: Node is IN the image but no Node process runs (RomM serves through Python gunicorn/rq and nginx — `ps` on demo-hp 9201), and RomM sits behind the setup gate. | OPEN — **RULED D61 = keep RomM offered** (operator 2026-10-28, starting the day brief; `09` §3 decision 277): the first RomM release with Node ≥ 24.22.1 is a lane step. 2026-10-28: 5.4.1 is still the newest tag (read from the registry). Earlier: CC's pick 2026-10-27 night **2026-11-02:** 5.4.1 is still the newest tag; its image still holds Node 24.18.1. | — | the watch flags it nightly (R-1327); the step when RomM ships | CC |
| **R-1331** | Apps & catalog | P3 | **Paperless-ngx 2.20.15 carries Python Brotli 1.1.0 (CVE-2026-111497, fixed 1.1.1/1.2.1)** — its web side only COMPRESSES (whitenoise, compression_middleware); the DECODERS are urllib3's and httpx's, for replies to requests Paperless itself makes (its own outbound calls), never for what a visitor sends. Read in the running container on demo-hp 9201, 2026-10-27. 2026-10-28: the newest release 3.3.4 still carries Brotli 1.2.0 and is a new main version — no fixed release yet (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN — no step tonight (decided by CC unattended — operator may reverse): the bad path needs a hostile server Paperless calls **2026-11-02: a fixed release exists, on a new main version.** 3.4.0 and 3.4.1 read clean (3.3.4 is still bad). **The big step 2.20.15 → 3.4.1 is proven on the drill:** bench `proven`, 9202 `proven` (peak 40.7 %), the undo `proven` (`audits/security-2026-11-02/bench/paperless-ngx/`, `walks/paperless-drill/`). Not written to the live catalog. | — | the operator's word (STATUS, morning list), then the writer | CC |
| **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) **2026-11-02:** v4.0.8 is still the newest; its image still holds Brotli 1.2.0. | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC |
| **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) **2026-11-02:** v4.0.8 is still the newest; its image still holds Brotli 1.2.0. | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC |
| **R-1356** | Apps & catalog | P3 | **n8n keeps three notices fixed only in 3.0.1 (`GHSA-9p7v` high: code nodes run in the main process; `GHSA-mx4r` unauthenticated webhook-secret reset; `GHSA-2x68`), after the lane step to 2.46.4 fixed nine.** | OPEN — the big step 2.46.4 → 3.0.1 is proven on the drill: bench, 9202 (peak 23.4 %), the undo (`audits/security-2026-11-02/bench/n8n/`, `walks/n8n3-drill/`). Not written to the live catalog. | the operator's word | morning list; then the writer | CC |
| **R-1357** | Apps & catalog | P3 | **PrivateBin 2.1.0 has two public notices fixed in 2.1.1 (`GHSA-hw9p` comment replay by anyone who knows a paste id; `GHSA-c4gc` 2.1.0's XSS fix is ineffective), but no 2.1.1 exists: no release on GitHub and no `privatebin/pdo` tag (read 2026-11-02).** Public by design on a box; Tester 1 runs it. | OPEN — filed 2026-11-03 | upstream: a 2.1.1 image | the watch keeps it open; take the release as a lane step | CC |
| **R-1357** | Apps & catalog | P3 | **PrivateBin 2.1.0 has two public notices fixed in 2.1.1 (`GHSA-hw9p` comment replay by anyone who knows a paste id; `GHSA-c4gc` 2.1.0's XSS fix is ineffective), but no 2.1.1 exists: no release on GitHub and no `privatebin/pdo` tag (read 2026-11-02).** Public by design on a box; Tester 1 runs it. | OPEN — filed 2026-11-03 | upstream: a 2.1.1 image | the watch keeps it open; take the release as a lane step | CC |
| **R-1359** | Apps & catalog | P4 | **Two dependency fixes with no public id, so no lane: Mealie v3.30.0 (`pillow` to v13 „[security]") and Termix 26.11.0 (`body-parser` „for a security fix", in v26.10.6).** Mealie's release also removes `OPENAI_CUSTOM_PROMPT_DIR` (our template does not set it). | OPEN — both proven on the drill as normal steps: bench and 9202 (`audits/security-2026-11-02/bench/{mealie,termix}/`, `walks/{mealie,termix}-drill/`). Not written. | the operator's word | morning list; then the writer | CC |
| **R-1360** | Apps & catalog | P4 | **Meilisearch v1.36.0 (Wanderer) and v1.41.0 (Karakeep) are 15–20 versions behind releases with „important security fixes" (v1.55.1, v1.56.1: long filters, rustls).** The search engine sits on the app's private network, so a stranger cannot reach it; its data needs a dump migration across such a jump: a big step. | OPEN — filed 2026-11-03 | — | a big step when one of the two apps is next moved | CC |
| **R-1360** | Apps & catalog | P4 | **Meilisearch v1.36.0 (Wanderer) and v1.41.0 (Karakeep) are 15–20 versions behind releases with „important security fixes" (v1.55.1, v1.56.1: long filters, rustls).** The search engine sits on the app's private network, so a stranger cannot reach it; its data needs a dump migration across such a jump: a big step. | OPEN — filed 2026-11-03 | — | a big step when one of the two apps is next moved | CC |
@@ -170,7 +167,6 @@ stopping line that lies.
| **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.****The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.**`audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s,Line truncated
| **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.****The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.**`audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s,Line truncated
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:****OPEN — rank P3-LOW; owner: CC (after R-784)****2026-11-02:** v1.10.0 and v1.11.0 read clean of HTTP2Shell (Node 24.22.1); v1.9.0 is still bad. **The big step v0.17.3 → v1.11.0 is proven on the drill:** bench, 9202 (peak 29.4 %), the undo (`audits/security-2026-11-02/bench/sparkyfitness/`, `walks/sparkyfitness-drill/`). Not written: R-784 (the licence) stands before it. | — | — | CC |
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:****OPEN — rank P3-LOW; owner: CC (after R-784)****2026-11-02:** v1.10.0 and v1.11.0 read clean of HTTP2Shell (Node 24.22.1); v1.9.0 is still bad. **The big step v0.17.3 → v1.11.0 is proven on the drill:** bench, 9202 (peak 29.4 %), the undo (`audits/security-2026-11-02/bench/sparkyfitness/`, `walks/sparkyfitness-drill/`). Not written: R-784 (the licence) stands before it. | — | — | CC |
| **R-1347** | App updates | P3 | **After a successful restore lifts an update hold, the stack manager's „last update ended held" flag stays set until the next update or a controller restart — until then the crash-loop watcher does not watch the app and the nightly update run skips it.** Found by the D64 work 2026-10-29 (controller `5b3b541` clears the flag for `clear_hold` only, `ForgetUpdateHeld`). `clearUpdateHoldAfterRestore` lifts the hold in settings but not this cached flag. | **VERIFY — BUILT 2026-11-02, controller `c4b6844`, commit only; PROVEN LIVE on 9202 (test image):** an update with its undo copy cut off → hold `update_failed`; the household's restore → „the update hold is CLEARED", „update-held flag is forgotten"; six `kill -9` → the crash-loop watcher stopped the app, so it watches again (`audits/security-2026-11-02/part3/`). Tests `TestR1347_*`, red first. OPEN | — | ships with the next controller release | CC |
| **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.****What is NOT claimed here:**`restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC |
| **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.****What is NOT claimed here:**`restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC |
| **R-433** | Backup & restore | P3 | **A sub-account cannot reach ANY Storage Box snapshot, by any name — so clause (b) of the 2026-09-01 R-95 re-scope ("the rest is recoverable file by file") is NOT SUPPORTED.** MEASURED 2026-09-01 on `demo-hp` over the credential the box already holds, read-only, no delete verb issued. **The sweep:** a batched `stat -c %n` over Hetzner's port-23 restricted shell (500–600 paths per round trip, stdout carrying only paths that exist) tried **777,600** names of the vendor form `YYYY-MM-DDTHH-MM-SS` across nine full days at second granularity, and 126 alternative shapes — **zero resolved.****The control is what makes the zero mean anything:** the identical 600-name batch with one real path appended returned it in 6 of 6 batches. **The structural cause:**`/home` (the customer data, `u629488-sub3`) is **st_dev 0,82**; `/.zfs/snapshot` is **st_dev 0,276**; `/home/.zfs` does not exist. A snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding `felhom-repo`. **What still stands:** clause (a) — the box can delete its live repository but cannot WRITE into `/.zfs/snapshot` — is unchanged and re-confirmed. **What is now open again:** the only routes to the older copy are a panel rollback of the WHOLE Storage Box (deletes newer snapshots, hits every customer on it) and the provider API (fenced by §11-D, and `hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method at all**, so it needs new code regardless). **NOT ESTABLISHED, and it is the question that decides whether per-file recovery exists for anyone:** whether the MAIN account can see the snapshots. No main-account credential exists in this project. **The ranking is Viktor's; I am not re-ranking R-95 — but the argument that moved it down is the argument this drill removed.**`audits/evidence-drill-r95-recovery-2026-09-01/`**BLOCKED-ON-PROVIDER 2026-09-01.** The one question that can move this is drafted and ready to send: **Question 1** of `documentation/runbooks/provider-questions-2026-09-01.md` — *can the MAIN account retrieve individual files from a snapshot, without a whole-box restore?***Neither answer leaves this row where it is:** "yes" makes per-file recovery real but permanently operator-only, and closes this; "no, full restore only" means the snapshots do not bound a single customer's exposure at all, because using them costs every other customer on the box their newer snapshots — **and R-95 becomes urgent.** Nothing here can progress without it, and it is not CC's to send (§11-D). Tracked by a dated check in the DUE-CHECKS block. **⚠ A TRAP ON THE WAY TO THIS ANSWER, found by the operator 2026-09-01 and armed against in the questions file: HETZNER HAS TWO SIMILARLY-NAMED PRODUCTS WHOSE DOCS SAY OPPOSITE THINGS.** **Storage SHARE** (a managed Nextcloud, NOT us) documents *"Currently, we only support restores for the full backup ZFS snapshot to a specific point in time"* (`docs.hetzner.com/storage/storage-share/faq/backup-snapshot/`). **Storage BOX** (ours) documents the opposite — *"You can download individual files or entire directories as usual"* (`docs.hetzner.com/storage/storage-box/snapshots/`). **A web search for the obvious phrasing surfaces the SHARE page, and it reads like a definitive NO.** Tell them apart by the giveaways: the Share page talks about *Nextcloud's data cache*, a *database dump* and the *konsoleH* interface, and never mentions Storage Box. **Why this is filed and not left as trivia: a support agent could answer Question 1 from the wrong page, and that answer would push this row to the top of the register for no reason.** Question 1 now names the product, quotes the Storage Box line, and says explicitly that we know what the Share FAQ says. **If an answer cites full-snapshot-only, check WHICH PRODUCT it is about before acting on it.****Nothing in this repository ever leaned on the Share claim** — verified by grep at the time; the only vendor line we cite is the Storage Box one. **RE-DATED 2026-09-15 → 2026-09-22 (due-checks gate):** the operator mailbox read through the Gmail connector (`(from:hetzner OR subject:hetzner OR "Storage Box") after:2026/09/01`) holds no Hetzner reply — one match, our own `offsite_snapshots_dropped` alarm. Whether the two tickets were ever sent is not visible to CC; if they were not, sending them is the operator's act. **-- ANSWERED. Hetzner replied on ticket #2026090103040671; the operator supplied the thread on 2026-09-22 after this session searched for it and WRONGLY reported no answer (see the instrument note below and R-628).** **Q1 - can the MAIN account read individual files and directories out of a specific snapshot, without restoring the whole box?** Hetzner: *"With the main account it should be possible to download files and directories from the snapshot as it is stated in the documentation"*, citing `docs.hetzner.com/storage/storage-box/snapshots#access-to-snapshots`; and separately *"A restLine truncated
| **R-433** | Backup & restore | P3 | **A sub-account cannot reach ANY Storage Box snapshot, by any name — so clause (b) of the 2026-09-01 R-95 re-scope ("the rest is recoverable file by file") is NOT SUPPORTED.** MEASURED 2026-09-01 on `demo-hp` over the credential the box already holds, read-only, no delete verb issued. **The sweep:** a batched `stat -c %n` over Hetzner's port-23 restricted shell (500–600 paths per round trip, stdout carrying only paths that exist) tried **777,600** names of the vendor form `YYYY-MM-DDTHH-MM-SS` across nine full days at second granularity, and 126 alternative shapes — **zero resolved.****The control is what makes the zero mean anything:** the identical 600-name batch with one real path appended returned it in 6 of 6 batches. **The structural cause:**`/home` (the customer data, `u629488-sub3`) is **st_dev 0,82**; `/.zfs/snapshot` is **st_dev 0,276**; `/home/.zfs` does not exist. A snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding `felhom-repo`. **What still stands:** clause (a) — the box can delete its live repository but cannot WRITE into `/.zfs/snapshot` — is unchanged and re-confirmed. **What is now open again:** the only routes to the older copy are a panel rollback of the WHOLE Storage Box (deletes newer snapshots, hits every customer on it) and the provider API (fenced by §11-D, and `hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method at all**, so it needs new code regardless). **NOT ESTABLISHED, and it is the question that decides whether per-file recovery exists for anyone:** whether the MAIN account can see the snapshots. No main-account credential exists in this project. **The ranking is Viktor's; I am not re-ranking R-95 — but the argument that moved it down is the argument this drill removed.**`audits/evidence-drill-r95-recovery-2026-09-01/`**BLOCKED-ON-PROVIDER 2026-09-01.** The one question that can move this is drafted and ready to send: **Question 1** of `documentation/runbooks/provider-questions-2026-09-01.md` — *can the MAIN account retrieve individual files from a snapshot, without a whole-box restore?***Neither answer leaves this row where it is:** "yes" makes per-file recovery real but permanently operator-only, and closes this; "no, full restore only" means the snapshots do not bound a single customer's exposure at all, because using them costs every other customer on the box their newer snapshots — **and R-95 becomes urgent.** Nothing here can progress without it, and it is not CC's to send (§11-D). Tracked by a dated check in the DUE-CHECKS block. **⚠ A TRAP ON THE WAY TO THIS ANSWER, found by the operator 2026-09-01 and armed against in the questions file: HETZNER HAS TWO SIMILARLY-NAMED PRODUCTS WHOSE DOCS SAY OPPOSITE THINGS.** **Storage SHARE** (a managed Nextcloud, NOT us) documents *"Currently, we only support restores for the full backup ZFS snapshot to a specific point in time"* (`docs.hetzner.com/storage/storage-share/faq/backup-snapshot/`). **Storage BOX** (ours) documents the opposite — *"You can download individual files or entire directories as usual"* (`docs.hetzner.com/storage/storage-box/snapshots/`). **A web search for the obvious phrasing surfaces the SHARE page, and it reads like a definitive NO.** Tell them apart by the giveaways: the Share page talks about *Nextcloud's data cache*, a *database dump* and the *konsoleH* interface, and never mentions Storage Box. **Why this is filed and not left as trivia: a support agent could answer Question 1 from the wrong page, and that answer would push this row to the top of the register for no reason.** Question 1 now names the product, quotes the Storage Box line, and says explicitly that we know what the Share FAQ says. **If an answer cites full-snapshot-only, check WHICH PRODUCT it is about before acting on it.****Nothing in this repository ever leaned on the Share claim** — verified by grep at the time; the only vendor line we cite is the Storage Box one. **RE-DATED 2026-09-15 → 2026-09-22 (due-checks gate):** the operator mailbox read through the Gmail connector (`(from:hetzner OR subject:hetzner OR "Storage Box") after:2026/09/01`) holds no Hetzner reply — one match, our own `offsite_snapshots_dropped` alarm. Whether the two tickets were ever sent is not visible to CC; if they were not, sending them is the operator's act. **-- ANSWERED. Hetzner replied on ticket #2026090103040671; the operator supplied the thread on 2026-09-22 after this session searched for it and WRONGLY reported no answer (see the instrument note below and R-628).** **Q1 - can the MAIN account read individual files and directories out of a specific snapshot, without restoring the whole box?** Hetzner: *"With the main account it should be possible to download files and directories from the snapshot as it is stated in the documentation"*, citing `docs.hetzner.com/storage/storage-box/snapshots#access-to-snapshots`; and separately *"A restLine truncated
| **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)****2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC |
| **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)****2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC |
| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC****2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. | — | — | CC |
| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC****2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair.**2026-11-03: demo-hp reached it too.** Its local tier has refused every try since 2026-10-30 („local has 9.4 GiB free; the last archive of guest 9201 was 9.1 GiB"; the disk is 39 GB, the newest archive 10-29), every 15 minutes: the mails `whole_guest_backup_failed` (08:11 CET, 2026-11-03) and `expected_backup_missed` (05:00 CET, „newest backup is 120h old"). PBS and off-site are not this tier. Not caused by the release. | — | — | CC |
| **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) |
| **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) |
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
@@ -220,7 +216,6 @@ stopping line that lies.
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
| **R-1348** | Backup & restore | P3 | **The restore_mixed hold written at START-UP after an interrupted put-back fails sends no operator mail when it is set** — only the hold reporter's info `app_held` reaches the hub, and the hub's reminder after its grace period. Found by the R-1340 work 2026-10-29 (controller `5b3b541`): the mail sentences for unhealthy stop, failed update and waiting put-back exist and are tested, but the start-up path has no sender. | **VERIFY — BUILT 2026-11-02, controller `c4b6844` + `3764445`, commit only; PROVEN LIVE on 9202 (test image):** a restore cut at „the restore copy is recorded", its volume held, the controller started → hold `restore_mixed` / `interrupted_put_back`, and the event `backup_run_failures` with the R-1340 sentence reached the event sink (`part3/r1348-after.txt`). The hub's mail itself is the existing path. Tests `TestR1348_*`, red first. OPEN | — | ships with the next controller release | CC |
| **R-1362** | Backup & restore | P3 | **With the internet down, the Backups page takes over 2 minutes (135 s and 145 s measured; 5 s with only the hub down).** Walked 2026-11-03 on Tester 1: every other page answers at once on the home network. The page waits on something off-site inside its handler; a household that lost its line and opens exactly this page sees a hang. `audits/security-2026-11-02/part5/outage-wan.log` | OPEN — filed 2026-11-03 | — | name the wait with a timing, then give it a short deadline and show „the off-site state could not be read" | CC |
| **R-1362** | Backup & restore | P3 | **With the internet down, the Backups page takes over 2 minutes (135 s and 145 s measured; 5 s with only the hub down).** Walked 2026-11-03 on Tester 1: every other page answers at once on the home network. The page waits on something off-site inside its handler; a household that lost its line and opens exactly this page sees a hang. `audits/security-2026-11-02/part5/outage-wan.log` | OPEN — filed 2026-11-03 | — | name the wait with a timing, then give it a short deadline and show „the off-site state could not be read" | CC |
## Storage & devices — 14 rows (P3 4, P4 10)
## Storage & devices — 14 rows (P3 4, P4 10)
@@ -240,7 +235,6 @@ stopping line that lies.
| **R-1213** | Storage & devices | P4 | **Operator detail still on two household pages:** the registered drive card's `ext4 · /dev/sdb1 · Model` line (`templates/storage.html`; W14 hid only the agent drive cards), and the heading „Távoli mentés (3. mentés) — titkosított, offsite" with the English „offsite". Found by the night helper 2026-10-21 (W11/W14 follow-up). | OPEN | — | Move the line behind „Részletek"; „…titkosított, házon kívüli" | CC |
| **R-1213** | Storage & devices | P4 | **Operator detail still on two household pages:** the registered drive card's `ext4 · /dev/sdb1 · Model` line (`templates/storage.html`; W14 hid only the agent drive cards), and the heading „Távoli mentés (3. mentés) — titkosított, offsite" with the English „offsite". Found by the night helper 2026-10-21 (W11/W14 follow-up). | OPEN | — | Move the line behind „Részletek"; „…titkosított, házon kívüli" | CC |
| **R-1275** | Storage & devices | P4 | **An English household sees the drive names in Hungarian** on the install page (and wherever a storage label shows): „Belső tárhely — 52.0 GB free (default)”, „Külső tárhely (hdd_1)” (catalog sweep 2026-10-24, `audits/catalog-sweep-2026-10-24/navidrome/page-install-en.txt`). The label is stored once, in Hungarian, when the drive is found (`felhom-controller/controller/internal/settings/settings.go``InferStorageLabel`); a household may rename it. | OPEN | — | Render a label that still equals the inferred default in the reader's language; a test per language | CC |
| **R-1275** | Storage & devices | P4 | **An English household sees the drive names in Hungarian** on the install page (and wherever a storage label shows): „Belső tárhely — 52.0 GB free (default)”, „Külső tárhely (hdd_1)” (catalog sweep 2026-10-24, `audits/catalog-sweep-2026-10-24/navidrome/page-install-en.txt`). The label is stored once, in Hungarian, when the drive is found (`felhom-controller/controller/internal/settings/settings.go``InferStorageLabel`); a household may rename it. | OPEN | — | Render a label that still equals the inferred default in the reader's language; a test per language | CC |
| **R-1276** | Storage & devices | P4 | **„Remove with data” leaves the app's empty parent folder** (`<drive>/appdata/<app>`) when the template binds a subfolder: Grimmory, RomM, Paperless-ngx (catalog sweep 2026-10-24; 4 KB each; the bound folders were removed). Harmless (empty, not counted as old data), untidy for a household that browses the drive. | OPEN | — | Remove the parent when it is empty after the bound folders go; a test | CC |
| **R-1276** | Storage & devices | P4 | **„Remove with data” leaves the app's empty parent folder** (`<drive>/appdata/<app>`) when the template binds a subfolder: Grimmory, RomM, Paperless-ngx (catalog sweep 2026-10-24; 4 KB each; the bound folders were removed). Harmless (empty, not counted as old data), untidy for a household that browses the drive. | OPEN | — | Remove the parent when it is empty after the bound folders go; a test | CC |
| **R-1339** | Storage & devices | P4 | **At every start the controller logs „FileBrowser: drive …/userdata/<app> not mounted — skipping userdata skeleton" for per-app folders on a drive that IS mounted** (scratch 9202, 2026-10-29 03:35 UTC: nine lines, `scratch_hdd` a live mountpoint, mp8). `skipFileBrowserPath` (`controller/internal/web/handlers.go:3766`) asks whether the folder ITSELF is a mountpoint, and a folder under a mounted drive never is. No data is lost: the file browser shows the folders through the drive's own mount (`/srv/scratch_hdd`). The cost is a false „not mounted" in the log of a box whose drive works, which misleads a diagnosis. Fix: ask about the drive the folder lives on, with a test that a sub-folder of a mounted drive is not skipped. Evidence: `audits/night-2026-10-28/part5-claims/selfheal/controller-after-reboot.log`. | **VERIFY — BUILT 2026-11-02, controller `c4b6844`, commit only; PROVEN LIVE on 9202:** nine false lines at a boot before, none after (`part3/stdout-boot-before.txt`, `stdout-boot-after.txt`). The row's „not skipped" would have written a `userdata/` inside the app's folder and recreated the file manager, so the behaviour stays and only the line changes. The per-app folders in 9202's list are a legacy of its old installs (R-839 refuses that shape now): no other box holds one. OPEN | — | ships with the next controller release | CC |
| **R-525** | Security & access | P4 | **[P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured.** Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. **What it needs:** a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | **READY — rank P3-LOW; owner: CC (spike)****Re-ranked 2026-10-03: P3->P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed.** | — | — | CC |
| **R-525** | Security & access | P4 | **[P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured.** Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. **What it needs:** a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | **READY — rank P3-LOW; owner: CC (spike)****Re-ranked 2026-10-03: P3->P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed.** | — | — | CC |
| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `<!-- source -->` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous**`curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”***Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config``APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config``SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restarLine truncated
| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `<!-- source -->` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous**`curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”***Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config``APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config``SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restarLine truncated
| **R-1243** | Security & access | P2 | **The hub's Longhorn backups sit unencrypted in the NFS folder any DooPlex pod can read (R-1241) — the two there now (2026-10-18, 2026-10-23) hold every household's Cloudflare and tunnel tokens in plaintext, because they predate hub 0.168.0.** Measured 2026-10-23 night (`audits/hub-token-seal-2026-10-23/` §3): the hub volume is `encrypted=false` and in Longhorn's `default` backup group (daily 04:00 + weekly Sunday 05:00, retain 1 each); the hub's own copy to ep0 is encrypted at rest (16/16 snapshots `crypt-mode: encrypt`, key on DooPlex's root disk and on paper) and in transit (SSH tunnel + pinned PBS TLS), every 6 h, restore-tested weekly. DECIDE: (a) an encrypted Longhorn volume for the hub (~5–10 min downtime, a homelab StorageClass + key in the cluster) or (b) take the hub volume out of the Longhorn backup — ep0's copy is the backup; the access log then has no backup until it rides the ep0 push. **CC's pick: (b).** And: delete the two plain copies (the operator's call; with (b) they never rotate out). | **VERIFY** — **2026-10-24: D40 put into effect** (operator ruling 07:22): the hub PVC `hub-data` carries `recurring-job-group.longhorn.io/default: disabled` (felhom.eu `3363ffa8`, ArgoCD sync of the PVC only) and the Longhorn volume `pvc-486c9809…` reads `disabled` too (labelled live to match, unchanged after 2 min); the NFS folder holds no copy of the volume (no `pvc-486c9809*` directory, no backup config naming it; Longhorn lists no backup volume for it; 53 other volumes backed up this morning). Closes after the dated check: tonight's 04:00 job makes no new hub backup while the others run. WAITING-ON-OPERATOR **2026-10-25 dated check DONE (read-only, the restore-truth session):**`kubectl -n longhorn-system get backups.longhorn.io` lists 106 backups made today (the newest 04:08:28Z — the job ran) and none of `pvc-486c9809` — the hub volume is out of the Longhorn backup as D40 meant. | the operator's pick; a homelab change for either option | Operator: pick (a) or (b), and say whether to delete the two plain copies after hub 0.168.0 is live | CC (the dated check) |
| **R-1243** | Security & access | P2 | **The hub's Longhorn backups sit unencrypted in the NFS folder any DooPlex pod can read (R-1241) — the two there now (2026-10-18, 2026-10-23) hold every household's Cloudflare and tunnel tokens in plaintext, because they predate hub 0.168.0.** Measured 2026-10-23 night (`audits/hub-token-seal-2026-10-23/` §3): the hub volume is `encrypted=false` and in Longhorn's `default` backup group (daily 04:00 + weekly Sunday 05:00, retain 1 each); the hub's own copy to ep0 is encrypted at rest (16/16 snapshots `crypt-mode: encrypt`, key on DooPlex's root disk and on paper) and in transit (SSH tunnel + pinned PBS TLS), every 6 h, restore-tested weekly. DECIDE: (a) an encrypted Longhorn volume for the hub (~5–10 min downtime, a homelab StorageClass + key in the cluster) or (b) take the hub volume out of the Longhorn backup — ep0's copy is the backup; the access log then has no backup until it rides the ep0 push. **CC's pick: (b).** And: delete the two plain copies (the operator's call; with (b) they never rotate out). | **VERIFY** — **2026-10-24: D40 put into effect** (operator ruling 07:22): the hub PVC `hub-data` carries `recurring-job-group.longhorn.io/default: disabled` (felhom.eu `3363ffa8`, ArgoCD sync of the PVC only) and the Longhorn volume `pvc-486c9809…` reads `disabled` too (labelled live to match, unchanged after 2 min); the NFS folder holds no copy of the volume (no `pvc-486c9809*` directory, no backup config naming it; Longhorn lists no backup volume for it; 53 other volumes backed up this morning). Closes after the dated check: tonight's 04:00 job makes no new hub backup while the others run. WAITING-ON-OPERATOR **2026-10-25 dated check DONE (read-only, the restore-truth session):**`kubectl -n longhorn-system get backups.longhorn.io` lists 106 backups made today (the newest 04:08:28Z — the job ran) and none of `pvc-486c9809` — the hub volume is out of the Longhorn backup as D40 meant. | the operator's pick; a homelab change for either option | Operator: pick (a) or (b), and say whether to delete the two plain copies after hub 0.168.0 is live | CC (the dated check) |
| **R-1354** | Security & access | P2 | **Traefik v3.7.16 on every box has GHSA-xv5x-242w-7hf7 (high, published 2026-11-02): an unauthenticated client that sends a `Negotiate` or `NTLM` header holds a backend transport per connection until Traefik runs out of memory, NTLM backend or not (v3.7.14–v3.7.16).** Traefik fronts every box. The brief expected a false alarm: the watch had flagged only a word-only line of the same notes. | OPEN — pin committed, controller `c4b6844`; v3.7.17 ran on 9202 through 3 boots, dashboard and file manager healthy (`audits/security-2026-11-02/part3/`) | a controller release (the operator's word) | first on the morning list | CC |
| **R-1355** | Security & access | P2 | **DooPlex's own Gitea 1.26.2 is inside 19 new notices (2026-11-02, fixed 28.3.0: `GHSA-8p3x` OAuth2 account takeover, `GHSA-37xv` and `GHSA-8g99` public-only tokens reach private data, `GHSA-32h8` Actions secrets) — on the host that holds our code and registry.** Also new on DooPlex: Vaultwarden 1.37.4 (fix 1.37.5), Uptime Kuma 2.5.6 (fix 2.5.9), Paperless 3.3.4 (fix 3.4.0). Felhom changes nothing on DooPlex. Read in `documentation/security-watch/state.json`, group DooPlex. | OPEN — filed 2026-11-03 | the homelab session | add them to `audits/http2shell-2026-10-27/HOMELAB-LIST.md`'s work; which notices reach a self-hosted Gitea with our settings is the homelab session's first read | operator (homelab session) |
| **R-1355** | Security & access | P2 | **DooPlex's own Gitea 1.26.2 is inside 19 new notices (2026-11-02, fixed 28.3.0: `GHSA-8p3x` OAuth2 account takeover, `GHSA-37xv` and `GHSA-8g99` public-only tokens reach private data, `GHSA-32h8` Actions secrets) — on the host that holds our code and registry.** Also new on DooPlex: Vaultwarden 1.37.4 (fix 1.37.5), Uptime Kuma 2.5.6 (fix 2.5.9), Paperless 3.3.4 (fix 3.4.0). Felhom changes nothing on DooPlex. Read in `documentation/security-watch/state.json`, group DooPlex. | OPEN — filed 2026-11-03 | the homelab session | add them to `audits/http2shell-2026-10-27/HOMELAB-LIST.md`'s work; which notices reach a self-hosted Gitea with our settings is the homelab session's first read | operator (homelab session) |
| **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs****Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator |
| **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs****Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator |
| **R-904** | Security & access | P4 | **Cloudflare can read every household's app traffic; replacing it with our own relay is a later item.** Facts (reviewer discussion 2026-10-08; `01`): app traffic and the dashboard reach the box through the Cloudflare Tunnel (`01` §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in `01` — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (`01` §4, §7); certificates (`01` §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (`01` §5 last row, §7); flood protection (not in `01`). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (`09` §3 decision 184): a later item. | **DEFERRED** — **2026-10-10: the asked sentence is in `01` §7** („TLS ends at Cloudflare's edge …”); the spike stays deferred. **DEFERRED — after the first customers (operator ruling 2026-10-08 09:07)** | — | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator |
| **R-904** | Security & access | P4 | **Cloudflare can read every household's app traffic; replacing it with our own relay is a later item.** Facts (reviewer discussion 2026-10-08; `01`): app traffic and the dashboard reach the box through the Cloudflare Tunnel (`01` §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in `01` — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (`01` §4, §7); certificates (`01` §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (`01` §5 last row, §7); flood protection (not in `01`). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (`09` §3 decision 184): a later item. | **DEFERRED** — **2026-10-10: the asked sentence is in `01` §7** („TLS ends at Cloudflare's edge …”); the spike stays deferred. **DEFERRED — after the first customers (operator ruling 2026-10-08 09:07)** | — | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator |
@@ -281,7 +274,6 @@ stopping line that lies.
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
|---|---|---|---|---|---|---|---|
| **R-1323** | Box system & updates | P3 | **When the file manager's image moves at start, the R-1281 step logs a false alarm: „could not leave traefik-public — every app can still reach it (R-1281): … No such container” — the container it tried to move was the OLD one, already replaced.** Seen on 9202 (0.343.0 → 0.346.1-rc.1, 2026-10-27 14:58:17Z); the probe a few seconds later showed the new container on `felhom-filebrowser` + `felhom-files` only (an app container: „bad address”; the control reached traefik). The text claims an exposure that is not there, which teaches the reader to ignore the real one. `audits/security-app-updates-2026-10-27/part5/9202-rc1-switch.txt` | **VERIFY — BUILT 2026-11-02, controller `c4b6844`, commit only.**`TestR1323_*`, red first. Not seen live: the image did not move on the test boots. OPEN — filed 2026-10-27 | — | ships with the next controller release | CC |
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
@@ -296,7 +288,6 @@ stopping line that lies.
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:****VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:****OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:****VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:****VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:****OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:****VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
| **R-1351** | Box system & updates | P4 | **At a guest boot two start paths run `docker compose up` on FileBrowser in the same second, and one fails with „Conflict. The container name \"/<id>_filebrowser\" is already in use" — two `[ERROR]` lines and a `[WARN] first-boot base-stack bring-up` for an app that then runs healthy.** Seen 2026-10-29 19:07:02 UTC on Tester 1 (controller 0.349.0) after the operator reboot from the hub: `[infra] filebrowser compose exists — starting without regenerating` and `[web] FileBrowser mounts synced (recreated)` at the same second; the recreate won (Docker's temporary rename `<id>_filebrowser`), FileBrowser „Up (healthy)", files.<domain> 200. The first reboot an hour earlier showed no such line — a race, not every boot. Any guest boot can hit it (a power cut, the household's „A Felhom újraindítása"). The effect is a false ERROR in the box log (and in a box walk); nothing stays broken. | **VERIFY — BUILT 2026-11-02, controller `c4b6844`, commit only.** One lock over both compose paths; `TestR1351_*`, seen failing without the lock. **Live: no Conflict and no first-boot WARN on any of 6 boots of 9202 (3 on each image, also with the sync recreating and the file manager held down so both paths ran in one second), so these boots do not tell the images apart: the race did not fire on the old image either. The proof is the test, stated plainly.** OPEN | `audits/release-2026-10-29c/final/walk-after.txt` | ships with the next controller release; read the next reboots' lines | CC |
| **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go``parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC |
| **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go``parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC |
| **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC |
| **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC |
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03 | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC |
| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03**2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC |
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1352** | Monitoring & notifications | P4 | **When a controller stops answering AND the agent restarts it between two host reports, the operator gets the restart mail first and the „does not answer — the agent restarts it in 2 minutes" alarm second, in the same second.** Seen live 2026-10-30 (the R-1344 proof on Tester 1: alarm 09:29:21, restart 09:31:22, both reached the hub with the 09:42:56 report; `hub/internal/monitor/controller_supervisor.go``observe()` emits the restart before the alarm). The text of each mail is true; only the order reads backwards. | **VERIFY — BUILT 2026-11-02, hub `2f7e836f`, commit only.** The alarm of an episode whose restart arrives in the same report is not emitted; a new episode still alarms. `TestR1352_*`, red first. OPEN | — | ships with the next hub release | CC |
## Hub & operator — 14 rows (P3 3, P4 11)
## Hub & operator — 14 rows (P3 3, P4 11)
@@ -419,4 +409,3 @@ stopping line that lies.
| R-1228 | 2026-11-06 | D35 way back ends: ask the operator to retire DooPlex's felhom.eu/www webpage copy (ingress + git-sync), or keep it longer; detail in the R-1228 row |
| R-1228 | 2026-11-06 | D35 way back ends: ask the operator to retire DooPlex's felhom.eu/www webpage copy (ingress + git-sync), or keep it longer; detail in the R-1228 row |
<!-- DUE-CHECKS-END -->
<!-- DUE-CHECKS-END -->
| **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC |
| **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC |
| **R-1302** | Monitoring & notifications | P4 | **A fresh install's agent logs ERROR „store-grant: GRANT WAS MISSING AND HAS BEEN SELF-REPAIRED — investigate the loss (R-190)" and „capability DEGRADED" for `felhom-pbs` at its FIRST start, although the grant was simply never made yet** — the whole-guest tier is armed (21:00:45) before the PBS storage exists (created 21:00:57 on the hub's descriptor). Seen 2026-10-26 night on the R-1232 BYO box (VM 9375, `audits/night-fixes-2026-10-26/r1232/3-first-run.txt`) and on 2026-10-25's BYO box (`degraded=pve:store-grant:felhom-pbs`). An ERROR that says „investigate the loss" on every new box teaches the reader to skip it. | **VERIFY — BUILT 2026-11-02, agent `50eeccc`, commit only.**`<state dir>/store-grant-seen.json`: a first grant logs INFO and reports healthy; a grant that was held and vanished keeps the ERROR; a doubt keeps it too. `TestR1302_*`, red first, four mutants. Not seen live: no agent release tonight. OPEN | — | ships with the next agent release (vouch) | CC |
@@ -7,6 +7,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
## Moved from STATUS on 2026-11-03
## Moved from STATUS on 2026-11-03
**Updated 2026-11-03 (morning).** On your word, golden 0.349.0 is baked; the code-push gate is green again. It needs your vouch (first line of the release list). It carries Traefik v3.7.16 until controller 0.350.0 (R-1354).
**2026-11-03 release list** (your word). 0. **Vouch golden 0.349.0** (agent 0.171.0, min_agent 0.158.1), sha `9d97e7c5ec3e…`, in full in `documentation/tests/golden-0.349.0-2026-11-03/README.md`. If not: a new install keeps getting 0.344.0, and the floor stays above the vouched golden. 1. Controller v0.350.0 (`3764445`): Traefik v3.7.17 (R-1354), R-1347, R-1348, R-1339, R-1351, R-1323. If not: every box keeps the hole. 2. Agent v0.172.0 + vouch: R-1302. 3. Hub v0.179.0: R-1352. 4. Catalog, proven on the drill: Paperless 3.4.1 (R-1331), n8n 3.0.1 (R-1356), Mealie, Termix (R-1359). 5. To the homelab session: R-1355.
**Updated 2026-11-03.** The security watch was not half-blind: seven pins it can never read held two alarms open, so a real outage could not have mailed; fixed, clean on both runners. Live, each proven on the scratch box: Gitea 28.3.0, Uptime Kuma 2.5.9, Grafana 13.2.5, ActualBudget 26.11.0, n8n 2.46.4, Vaultwarden 1.37.5 (`c760fca`). Traefik on every box has a real high notice (R-1354): first on the list.
**Updated 2026-11-03.** The security watch was not half-blind: seven pins it can never read held two alarms open, so a real outage could not have mailed; fixed, clean on both runners. Live, each proven on the scratch box: Gitea 28.3.0, Uptime Kuma 2.5.9, Grafana 13.2.5, ActualBudget 26.11.0, n8n 2.46.4, Vaultwarden 1.37.5 (`c760fca`). Traefik on every box has a real high notice (R-1354): first on the list.
**Updated 2026-10-30.** On your yes, felhom.eu and the installer are served by Cloudflare (DooPlex's copy stays until 11-06 as the way back), and the contact form posts to api.felhom.eu. Hub 0.178.0 (1 GiB) saves your vouch in 0.13 s instead of 38 s and opens the apps page in under a second. Agent 0.171.0 (your vouch) restarts a controller that runs but does not answer — proven on Tester 1.
**Updated 2026-10-30.** On your yes, felhom.eu and the installer are served by Cloudflare (DooPlex's copy stays until 11-06 as the way back), and the contact form posts to api.felhom.eu. Hub 0.178.0 (1 GiB) saves your vouch in 0.13 s instead of 38 s and opens the apps page in under a second. Agent 0.171.0 (your vouch) restarts a controller that runs but does not answer — proven on Tester 1.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
admin committed