diff --git a/STATUS.md b/STATUS.md index 8de7389d..45e1d090 100644 --- a/STATUS.md +++ b/STATUS.md @@ -15,7 +15,7 @@ - The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15). ## What is broken -- **P1 (the data-loss hunt, 2026-10-20 — still running):** shared folders on two drives lose their second copy every night, and a dead drive's copy is deleted the same night (R-1175); a box rebuilt after „delete the old host" cannot reach the household's off-site history (R-1176); importing an app bundle makes a family-only app public (R-1191); on the 8 GB minimum box the family's first afternoon froze the box for an hour while the hub said OK (R-1192, R-1193). +- **P1 (the data-loss hunt, 2026-10-20 — still running):** shared folders on two drives lose their second copy every night, and a dead drive's copy is deleted the same night (R-1175); a box rebuilt after „delete the old host" cannot reach the household's off-site history (R-1176); importing an app bundle makes a family-only app public (R-1191); on the 8 GB minimum box the family's first afternoon froze the box for an hour, twice; the household was told nothing (R-1192, R-1193). - **P1:** about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours). - Two root-level weaknesses wait on your decisions D3 and D4 (R-985, R-982). - Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/03-progress-slow.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/03-progress-slow.txt index f567e7d6..5da36b9d 100644 --- a/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/03-progress-slow.txt +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/03-progress-slow.txt @@ -1 +1,15 @@ -16:32:09Z running pha \ No newline at end of file +16:32:09Z running phase=dump app= 0% / files 0/0 snaps=0 last_err='' +16:34:13Z running phase= app=bookstack 100% 153.6 MB/153.6 MB files 8/8 snaps=0 last_err='' +16:36:13Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err='' +16:38:14Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err='' +16:40:14Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err='' +16:42:14Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err='' +16:44:14Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err='' +16:46:14Z running phase= app=immich 12.32862472832852% 150.6 MB/1.2 GB files 787/790 snaps=0 last_err='' +16:48:14Z running phase= app=mealie 0% / files 0/0 snaps=0 last_err='' +16:50:14Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err='' +16:52:14Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err='' +16:54:14Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err='' +16:56:15Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err='' +16:58:15Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err='' +17:00:15Z running phase= app=nextcloud 41.02783378851586% 423.9 MB/1.0 GB files 3107/3108 snaps=0 last_err='' diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/04-netem-off.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/04-netem-off.txt new file mode 100644 index 00000000..5e16c2b7 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/04-netem-off.txt @@ -0,0 +1,5 @@ +qdisc netem 8001: root refcnt 2 limit 1000 loss 5% rate 1Mbit seed 8660933106302401915 + Sent 87470204 bytes 76388 pkt (dropped 2885, overlimits 0 requeues 0) + backlog 10598b 4p requeues 0 +17:01:02 +qdisc fq_codel 0: root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/05-progress-after.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/05-progress-after.txt new file mode 100644 index 00000000..9556c742 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/05-progress-after.txt @@ -0,0 +1,2 @@ +17:01:12Z running app= nextcloud 41.02783378851586 423.9 MB / 1.0 GB snaps= 0 size= dur= err= +17:03:12Z error app= 0 / snaps= 0 size= dur= 30m10s err= offbox backup immich: exit status 1 diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/06-controller-log.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/06-controller-log.txt new file mode 100644 index 00000000..9b281f0a --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/06-controller-log.txt @@ -0,0 +1,5 @@ +17:01:32 INFO [offbox] backed up nextcloud (/mnt/felhom-drives/hdd_1/backups/primary/nextcloud, 1 mandatory path(s)) +17:01:36 INFO [offbox] backed up paperless-ngx (/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx, 1 mandatory path(s)) +17:01:38 INFO [offbox] backed up radicale (/mnt/sys_drive/felhom-data/backups/primary/radicale, 0 mandatory path(s)) +17:01:49 ERROR [offbox] backup failed after 30m10s: offbox backup immich: exit status 1 +17:01:49 WARN [web] manual off-box backup failed: offbox backup immich: exit status 1 diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/06b-controller-log-info.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/06b-controller-log-info.txt new file mode 100644 index 00000000..fab6321b --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/06b-controller-log-info.txt @@ -0,0 +1,7 @@ +200 2026-10-20T16:50:57Z +17:01:32 INFO [offbox] backed up nextcloud (/mnt/felhom-drives/hdd_1/backups/primary/nextcloud, 1 mandatory path(s)) +17:01:36 INFO [offbox] backed up paperless-ngx (/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx, 1 mandatory path(s)) +17:01:38 INFO [offbox] backed up radicale (/mnt/sys_drive/felhom-data/backups/primary/radicale, 0 mandatory path(s)) +17:01:49 ERROR [offbox] backup failed after 30m10s: offbox backup immich: exit status 1 +17:01:49 INFO [offbox] manual run progress reporting ended after 30m10s +17:01:49 WARN [web] manual off-box backup failed: offbox backup immich: exit status 1 diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/07-second-press.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/07-second-press.txt new file mode 100644 index 00000000..4e71bc18 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/07-second-press.txt @@ -0,0 +1,6 @@ +17:04:13Z press 302 +17:04:43Z running app= 0 / snaps= 0 size= dur= 30m10s err= +17:05:43Z running app= 0 / snaps= 0 size= dur= 30m10s err= +17:06:43Z running app= immich 25 309.8 MB / 1.2 GB snaps= 0 size= dur= 30m10s err= +17:07:43Z running app= nextcloud 88 911.3 MB / 1.0 GB snaps= 0 size= dur= 30m10s err= +17:08:43Z ok app= 0 / snaps= 17 size= 1.1 GB dur= 3m38s err= diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/README.md b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/README.md new file mode 100644 index 00000000..0c00ffd9 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s5-slow-net/README.md @@ -0,0 +1,20 @@ +# B6 #5 — slow, lossy internet during the first big off-site run (2026-10-20) + +Expected (written before): the run slows but stays honest on the page; it either finishes or fails clearly, and the +next run picks up (restic resumes from what is already uploaded); nothing is reported as safe that is not. + +| UTC | act | observed | +|---|---|---| +| 16:31:28 | `tc netem` on the VM's own tap (demo-hp): upload 1 Mbit/s + 5 % loss, both directions 5 % loss (`01-netem-on.txt`) | — | +| 16:31:37 | „Távoli mentés most" | running, phase dump (`02-run-pressed.txt`) | +| 16:34 | progress | BookStack 100 % 153.6 MB | +| 16:36–16:46 | progress | **Immich frozen at „12.3 % · 150.6 MB / 1.2 GB · files 787/790" for 10+ minutes** — the number does not move while the line crawls | +| ~16:47 | | the run moves on to Mealie, then Nextcloud (41 % frozen 16:50–17:01) — Immich had failed (its restic message fell out of the controller's log ring) | +| 17:01:02 | shaping removed; only 87 MB had left the box in 30 min (`04-netem-off.txt`) | — | +| 17:01:32–38 | | Nextcloud, Paperless, Radicale „backed up" | +| 17:01:49 | | **run = error after 30 min 10 s: „offbox backup immich: exit status 1"; the page: „✗ Hiba … Utolsó hiba: offbox backup immich: exit status 1 … a pillanatképek száma még ismeretlen"** — English, technical, and silent about the apps that DID copy (`06*`, screens `b6-s5/`) | +| 17:04–17:08 | second „Távoli mentés most" at full speed | **ok in 3 min 38 s; 17 snapshots; 1.1 GB** (`07-second-press.txt`) | + +Verdict: alarm TRUE (the run is marked failed), text poor (W25); time to heal = one more press at full speed. The +per-app 2-hour limit (`offbox.go:39`) was not the cause — the failure was Immich's own upload breaking on the lossy +line. Not measured: whether the NIGHT's run would have retried Immich by itself (the household pressed again first). diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 45e5c04f..4efa8417 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -211,7 +211,7 @@ stopping line that lies. | **R-1077** | Storage & devices | P4 | **No USB enclosure that hides or fakes a disk's serial has been measured.** The storage-identity spike (2026-10-15) had none at hand and found none in earlier evidence; the design's `usb_bridge` unknown rule (`architecture/16-storage-physical-identity.md` §3) is therefore an assumption. Cheap single- and dual-bay bridges are known to report one bridge serial for any disk. Evidence: `audits/storage-identity-2026-10-15/README.md` case 4b. | OPEN | — | Buy or borrow one cheap USB enclosure (single and dual bay), run `tools/physdisk_probe.py` against it, and set the rule from what it reports. | Operator (a device) then CC | | **R-1094** | Storage & devices | P4 | **The whole-disk format refusal tells the household to format the free part — which is refused while the other part is in use.** SEEN 2026-10-16 (controller 0.318.0): 409 „Ezen a lemezen egy használatban lévő meghajtó van, ezért a teljes lemez nem formázható. Csak a szabad részét formázhatod." — since agent 0.158.1 a part next to a mounted part can only be USED (the root wrapper trusts no „Felhom's own" claim). Evidence: `audits/storage-identity-buildB-2026-10-15/live/14-samedisk-refusal-offer.txt`. | READY | — | Reword `err.web.meghajto_hasznalatban_formazas` (hu + en): „… nem formázható. A szabad részét használatba veheted." Next controller release; the copy gates run. | CC | -## Security & access — 19 rows (P1 2, P2 2, P3 9, P4 6) +## Security & access — 20 rows (P1 2, P2 2, P3 10, P4 6) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -234,12 +234,13 @@ stopping line that lies. | **R-1040** | Security & access | P4 | **Until the household makes Sonarr's or Radarr's first admin, any container on the box can read the app's API key.** Measured on the bench 2026-10-14 with the R-1030 seed in place: `AuthenticationRequired=Enabled`, `AuthenticationMethod=None` — `/initialize.json` from another container answers 200 with `apiKey`. Outsiders are blocked by the box's setup gate in that window; a compromised other app is not. | **OPEN** — filed 2026-10-14 | — | Seed the API key at install (`SONARR__AUTH__APIKEY`, measured to work) so the box knows it, and close the window by telling the household in the first steps; or seed a Forms user | CC | | **R-1050** | Security & access | P4 | **Raise HSTS to one year on or after 2026-11-15, if no https trouble was reported.** `09` §3 decision 249 (D2 = A, operator 2026-10-15): HSTS stays `max-age=86400` now; after a month with no https trouble it becomes one year (`31536000`); never preload, never `includeSubDomains`. The value is `HSTSMaxAgeSeconds` in `felhom-controller/controller/internal/infra/infra.go`, pinned by `TestRenderForwardedHeaders_HTTPSOnly`. „Trouble” = any report of a name that will not open, or a box answering http after the floor. | **WATCHING** — filed 2026-10-15 | the date | On or after 2026-11-15: read the register and the operator's mail for https trouble since 2026-10-15; none → a controller release with one year (test updated), bake → vouch → floor | CC | | **R-1054** | Security & access | P3 | **The hub's operator/global API key is still one bearer that can do everything on `/api/v1`, including reading every box's break-glass root password (`GET /admin/hosts/{id}/recovery-credential`).** Hub v0.152.0 gave the web surface scoped tokens (`05` §17) but tokens are refused on `/api/v1` by design. Who uses the global key today is not listed in one place. Fix: list its consumers; move each operator use to a scoped token on the web side (or a token class on the API), and keep the global key only where a box path needs it — or retire it. | OPEN — filed 2026-10-15 by the token build | — | List the global key's consumers (pod env, scripts, runbooks) | CC | +| **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). | VERIFY | — | Read the agent's felhom-sshd render with and without an operator key; decide whether a missing key is a warning at all | CC | ## Box system & updates — 12 rows (P1 1, P2 1, P3 6, P4 4) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1192** | Box system & updates | P1 | **On the minimum machine (8 GB), the household's first afternoon froze the box for about an hour: the dashboard and every app stopped answering, and only pulling the plug brought it back.** Rehearsal 2026-10-20 on a fresh box (controller 0.336.0, agent 0.161.0): the install page let the household book 8 apps to 3638 of 3712 MB (98 %) — the guest has 4096 MB in all — and told it on every page „Az alkalmazások csúcsterhelése meghaladhatja a rendelkezésre álló memóriát. **Normál használat mellett ez nem okoz problémát.**" Then the first uploads (53 PDFs to Paperless, photos to Immich, a file tree to Nextcloud — sequential per app) → from ~13:05 UTC `/api/health` gave no answer in 15 s; the VM read ~4 GB/min from its disk (158 → 476 GB) with the network still; the box's own history shows load 139–152 and no samples for 36 min; it came back at 14:00 for minutes and froze again; the plug pulled at 14:11 → dashboard in 2 min 22 s, stable after. Lost on the way: 8 Paperless documents failed (5 ordinary invoices among them), the Immich upload cut at photo 150, Nextcloud at 251 of 3 393 files. Cause NOT diagnosed (no shell in the box; memory read ~2.3 GB while load was 150 — not plain „memory full"). Evidence `audits/rehearsal-2026-10-20/journal.md` (12:56–14:23), `B4-use/box-metrics-after-plug.txt`, `B4-use/vm-counters-before-plug.txt`, `B4-use/after-plug-watch.txt`. | OPEN | a diagnosis on a box with a shell (rerun on a scratch nested VM with the vaulted password) | Diagnose (per-container memory.events, PSI, swap, IO wait during the same uploads); then either lower the admission ceiling well below 100 % of the guest, raise the guest's share of the host, or cap the apps' background work; and drop the „nem okoz problémát" sentence until it is true | CC | +| **R-1192** | Box system & updates | P1 | **On the minimum machine (8 GB), the household's first afternoon froze the box for about an hour: the dashboard and every app stopped answering, and only pulling the plug brought it back.** Rehearsal 2026-10-20 on a fresh box (controller 0.336.0, agent 0.161.0): the install page let the household book 8 apps to 3638 of 3712 MB (98 %) — the guest has 4096 MB in all — and told it on every page „Az alkalmazások csúcsterhelése meghaladhatja a rendelkezésre álló memóriát. **Normál használat mellett ez nem okoz problémát.**" Then the first uploads (53 PDFs to Paperless, photos to Immich, a file tree to Nextcloud — sequential per app) → from ~13:05 UTC `/api/health` gave no answer in 15 s; the VM read ~4 GB/min from its disk (158 → 476 GB) with the network still; the box's own history shows load 139–152 and no samples for 36 min; it came back at 14:00 for minutes and froze again; the plug pulled at 14:11 → dashboard in 2 min 22 s, stable after. Lost on the way: 8 Paperless documents failed (5 ordinary invoices among them), the Immich upload cut at photo 150, Nextcloud at 251 of 3 393 files. Cause NOT diagnosed (no shell in the box; memory read ~2.3 GB while load was 150 — not plain „memory full"); the one hint: the hub's `app_oom` at 13:34 — immich-machine-learning killed by its memory limit. Reproduced a SECOND time at 15:27:55 with only one Immich upload + one chunked Nextcloud upload running (~35 min, then froze again; plug pulled 16:13); per-app memory now logged every 30 s (`B4-use/memwatch.txt`). Evidence `audits/rehearsal-2026-10-20/journal.md` (12:56–14:23), `B4-use/box-metrics-after-plug.txt`, `B4-use/vm-counters-before-plug.txt`, `B4-use/after-plug-watch.txt`. | OPEN | a diagnosis on a box with a shell (rerun on a scratch nested VM with the vaulted password) | Diagnose (per-container memory.events, PSI, swap, IO wait during the same uploads); then either lower the admission ceiling well below 100 % of the guest, raise the guest's share of the host, or cap the apps' background work; and drop the „nem okoz problémát" sentence until it is true | CC | | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC | | **R-982** | Box system & updates | P3 | **The Proxmox host accepts root password login over SSH from the LAN (measured on demo-hp), so anyone on the household network can try the root password.** Design-shaped: the break-glass password exists for the console. Fix: key-only SSH for root; the password stays for the console. Evidence: `audits/security-review-2026-10-12/evidence/demo-hp-host-reads.txt`. | OPEN — design-proposal | — | Decide; then set it in the installer and the config bundle | CC | | **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC | @@ -253,13 +254,13 @@ stopping line that lies. | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC | -## Monitoring & notifications — 11 rows (P2 2, P3 6, P4 3) +## Monitoring & notifications — 11 rows (P2 1, P3 7, P4 3) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC | | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | -| **R-1193** | Monitoring & notifications | P2 | **While the household's box was frozen for an hour, the hub showed it ONLINE and OK and nobody was told.** During R-1192 (13:05–14:11 UTC) the hub's Hosts page read Tester4-816e21 ONLINE, controller OK, host memory 39 %: the host agent kept reporting from the Proxmox host while the guest (dashboard, apps) did not answer. No operator mail, no household message. The same-channel trap of standing rule 3 (R-286): the liveness signal comes from the part that was still alive. Evidence `audits/rehearsal-2026-10-20/journal.md` 13:22. | OPEN | — | Add a guest-side liveness signal the hub can judge (the controller's own report age, or the agent probing the dashboard's `/api/health` with a deadline) and alarm on it | CC | +| **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC | | **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** **REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc//fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC | | **R-388** | Monitoring & notifications | P3 | **PRODUCT DECISION (not a defect): the customer notification model is the wrong shape, and the settings page grows by one toggle per detector.** The operator's framing, recorded verbatim 2026-08-23: *"A customer should be notified only about things they can act on or are responsible for — the drive they unplugged, the storage they filled. **A failed backup is our incident, not theirs.** The intended shape is that we detect it, we tell them we noticed and are dealing with it, and they are not handed an error they cannot solve. The subscription should feel like being looked after, not like being on call."* Today's page is the opposite shape — one switch per detector, and it **grew from 12 to 15 in a single session** (one new alarm plus two compound toggles split into four). That growth is the argument, not an aside: a page that grows per detector keeps asking a household to make engineering decisions. | **READY** — **2026-10-12 evening: slice 1 delivered** (operator Q1/Q2 „as picked", `09` §3 decisions 234–235): the page shows three group switches; the per-detector switches and the cooldown sit under „Részletek"; an „egyedi" group is saved back unchanged; the default list fills groups 1 and 3 (households with no saved list), group 2 stays „egyedi" (`app_start_failed` off, R-329). Tests red-proved. controller 0.309.0, delivered 2026-10-12 18:08 to demo-hp, demo-felhom and Tester 1 (`controller-0.309.0-delivery.txt`). **READY** — **2026-10-12: the design is written** (`08` §8.1): three group switches („A gépem és a meghajtóim", „Az alkalmazásaim", „Amit mi intézünk") mapped onto today's 18 detectors; option A picked; slice 1 changes no wire. Two questions for the operator (group 3 default; the cooldown field). **READY** — operator ruling 2026-10-12 (S5, `09` §3 decision 200): design now (one page: options, a pick, the first slice), build in slices later. **OPEN — DIRECTION, operator's call** | a decision on scope; nothing here is a bug | Slice 2 (hub): group 3's mails say „we noticed X and are handling it" instead of the raw error | CC (+ operator: two answers) | | **R-435** | Monitoring & notifications | P3 | **The snapshot-drop detector is blind to a single-app deletion — the exact shape the box can produce.** `snapshotDropFraction = 0.5` and `snapshotDropFloor = 5` (`hub/internal/monitor/offsite.go`) require a fall of MORE than half the previous count. demo-hp's baseline is **69** across 9 apps, so ~35 snapshots must go before it speaks; **one app's tag is ~9 and is invisible.** `offbox.go:1388` runs `forget --prune` **grouped by host,tags** — a per-tag wipe is precisely the shape a faulty retention or a targeted deletion produces. **This is deliberate, not accidental:** the constant's own comment argues the insensitivity, and *"a detector that cries wolf is switched off within a fortnight"* is a lesson this project paid for. **So this row is NOT a demand to lower the threshold.** It is a demand that the blind spot be written where the operator reads it, because "an unexplained fall is noticed within a day" (R-431, and `STATUS.md`) is true only of falls above half. **Discovered by arithmetic while planning the drill's Phase 2/3 pairing, which could not have worked:** Phase 2 deletes one app and Phase 3 expects the alarm to fire. **LIMITATION NOW WRITTEN INTO THE ALARM'S OWN DOCUMENTATION, hub v0.111.1** — the comment above `snapshotDropFraction` in `hub/internal/monitor/offsite.go` now states what the detector does NOT see, with the demo-hp arithmetic and the `forget --prune` grouping that makes the blind spot sit on the most likely single-app failure. **It also says explicitly that the numbers must NOT be lowered to "fix" this** and that per-app detection needs a SECOND signal keyed on the per-tag count. **The row stays OPEN because documenting a blind spot is not covering it** — and because `STATUS.md` and R-431 both still say "noticed within a day", which is true only of falls above half. **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-435.md` — the box's key can no longer delete (decisions 68–69); deletion through any other credential still needs >50 % to alarm. Question D7 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D7 (`09` §3 decision 191):** yes — option C, an error mail when even one snapshot disappears outside a hub-opened clean-up window (pinned tiers). | **VERIFY** — **2026-10-12 read: not provable today.** No mail for a snapshot drop reached the operator since 2026-10-11 (Gmail), but the window itself could not be read: no hub page shows `offsite_windows`, Tester 1's controller log restarted with 0.308.0 at 13:49Z, and the guest-agent path to its VM did not answer. Re-dated after the next weekly clean-up. **VERIFY — 2026-10-09: DELIVERED in hub 0.144.0. Live proof NOT run: a window is granted only when the box asks for one, and Tester 1's weekly clean-up is next due ~2026-10-12 (last window 2026-10-05); prove it then (no alarm inside the window; the hub's window row as control).** **VERIFY — built on hub main 2026-10-08 (D7, decision 191), ships tomorrow; closes after the 9202 proof (a hand-granted window removes N, no alarm; the hub's window row as the control).** **OPEN — documented in code v0.111.1; the coverage gap itself is unclosed** **2026-10-05 (burn-down night): NEEDS A DESIGN.** Covering one-app deletions needs per-app snapshot counts from the controller and a per-tag threshold measured on real counts — two repos, a mechanism nobody has measured. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the night session that reached it was not briefed for this read (a Tester 1 log and the mailbox); the window row and the mails stay readable a day later. | — | 2026-10-20: read Tester 1's controller log (window opened/closed, count before/after) and the operator mailbox (no drop alarm) the morning after its weekly clean-up | CC |