Felhom in a VM on OpenMediaVault 8 (KVM plugin): tried on an exact copy of the pilot's box; Hungarian guide; R-1068..R-1073; capability A23
gates / gates (push) Successful in 6m53s

On demo-hp: OMV 8.5.11-1 + openmediavault-kvm 8.3.4 in a VM, every OMV step done in its web pages
(headless browser). Public ISO 1.29.0 on the whole passed NVMe, BIOS firmware, bound and claimed as
throwaway customer Tester4, Vaultwarden + Jellyfin, local backups, two OMV restarts (clean guest
shutdown, AutoStart back in 1 min 52 s). Guest memory 7,863 MiB of 12 GiB.
Traps for the guide: the bridge changes OMV's IP; the plugin boots the disk first (wipe the NVMe);
the host-disk list offers OMV's own disk. Rows: one-disk Jellyfin refusal (R-1068), virtio disk not
formattable (R-1069), tunnel-less OS-update alarm (R-1070), NAS films for Jellyfin (R-1071, operator),
mixed formal/informal backup text (R-1072), no SMART in a VM (R-1073, operator). Register 157 -> 163.
Everything removed; Tester 1 back ONLINE. Customer Tester4 waits for the operator's delete.

Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
This commit is contained in:
admin committed 2026-10-15 19:24:59 +02:00
1 parent 5d4538cc59
commit 340804f9af
81 files changed
+1334 -4

No files matched your search

+10 -4
View File
@@ -124,7 +124,7 @@ stopping line that lies.
| **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC |
| **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
## Apps & catalog — 15 rows (P2 1, P3 8, P4 6)
## Apps & catalog — 16 rows (P2 1, P3 9, P4 6)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -137,6 +137,7 @@ stopping line that lies.
| **R-1029** | Apps & catalog | P3 | **Plex's folder picker shows the drives as identical „media” folders.** With one bind per drive at the same path (`/mnt/felhom-drives/<drive>/userdata/media`), the leaf of every library folder is `media`; the drive's name sits two levels up, so in Plex's picker the household sees several folders all called „media” (reported by the operator during the 2026-10-14 test on Tester 1). It conflicts with `15` §2 reason 2 (the same path everywhere). | **READY** — filed 2026-10-14 | the layout decision (STATUS D9) | Decide in `15` §2: the players (`:ro`, never linked) may get a readable path per drive (e.g. `/media/<drive label>`) without breaking the hardlink rule, which binds only the download/library apps | CC |
| **R-1036** | Apps & catalog | P3 | **The apps page lags the catalog by two apps.** `app-catalog-felhom.eu/templates/` holds **64** apps; `/alkalmazasok` and `/en/apps` render **63** cards, one of which (`filebrowser`) is built in and not a catalog template — so **62 catalog apps are listed**. Missing: **`plant-it`** and **`wger`**, both in the catalog since 2026-10-06/08. Their website assets are ALREADY present (`plant-it-logo.svg` + 3 screenshots, `wger-logo.png` + 3 screenshots, renamed to the catalog slug by `a3507761`, 2026-10-08) — only the cards were never written, so this is not a new-asset job. Found 2026-10-14 while reconciling the app count for the „felhő tárhely” page, where four places said **60** and the rest said **62**. **Done in that session:** every stated count on the site set to **62** (`index.html`, `szolgaltatasok.html`, `en/services.html` were 60), which is what the apps page actually shows — so no visitor can count the cards and find the number wrong. **62 is still not the catalog's number.** **To close:** write the two cards on both apps pages (gate 15 keeps the two pages' app sets equal, so they move together), then raise every stated count 62 → 64 in the same commit. **The operator chose this split at the 2026-10-14 STOP** over doing it inside the SEO change. **2026-10-15 (R-1048):** Prowlarr's card added on both pages and every stated count 62 → 63 (the media group 19 → 20), so the site still states what it shows; the two owed cards would make it 65. **Note before writing them:** `wger` is `lifecycle: hidden` and `plant-it` `lifecycle: abandoned` — neither is offered for a new install, so a card would advertise an app the wizard refuses. | **OPEN** | — | write the `plant-it` + `wger` cards on both apps pages, then 62 → 64 everywhere, one commit | CC |
| **R-1065** | Apps & catalog | P3 | **After the Plex claim the box does not set Plex's libraries until the next wiring run.** The claim handler (`felhom-controller` `internal/web/media_events.go` `appPlexClaimHandler`) opens the gate when Plex says claimed and returns; the wiring runs only when a `media_library` app comes up, on a drive event, or on „Bekötés most". Plex's own install ran it while Plex was unclaimed (`plex=household(0)`), so a freshly claimed Plex has no libraries until one of those happens — the shape of R-1058. Read from code 2026-10-15; to be measured on the N100 when the operator claims Plex (the sheet asks for „Bekötés most" only if no library appears). | **OPEN** — filed 2026-10-15 | — | Run the wiring (RunMediaWiringSoon) after a successful claim; a test; measure on the N100's claim | CC |
| **R-1071** | Apps & catalog | P3 | **Jellyfin cannot use films that sit on a NAS share (the pilot's OMV), and the pilot's wish is exactly that.** MEASURED 2026-10-15: an OMV SMB share added as „Hálózati tárhely” mounts and verifies (`/mnt/felhom-drives/omv-filmek`) — but only with a user name AND a WRITABLE share (read-only → refused „az alkalmazások nem tudnak rá írni”) — and the household's „Újraindítás most” binds only local drives into Jellyfin (`per-drive binds 1 [/mnt/felhom-drives/filmlemez/userdata/media]`). By design: R-108 (a NAS is browse-only, never an app's namespace); `media_library.go` skips `Kind == "network"`. A design decision, not a defect (R-370). Evidence: `audits/omv-vm-2026-10-15/B6-omv-share.txt`. | WAITING-ON-OPERATOR | Operator: should a media app read a NAS share read-only (a new bind kind), or is the answer „put the films on a drive the box owns”? | Operator rules; if yes, a design note in `15-media-library.md` first. | Operator |
| **R-76** | Apps & catalog | P4 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-26, size S, roadmap state `idea (surfaced by the R-75 spike, 2026-07-26)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: not a catalog fix** — the setgid chain is set by the controller's FileBrowser setup; a controller row. | — | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say t | CC |
| **R-707** | Apps & catalog | P4 | **[P2] 37 apps still start with a login a stranger can take (`09` §3 decision 45).** Audit of all 53 apps: `app-catalog-felhom.eu/FIRST-ADMIN.md` (class, fix route, status, measured or read). Open: **3 hard-coded defaults** — calibre-web (`admin / admin123`, measured working on demo-hp and 9202; its own `cps.py -s` route needs a generated password WITH a special character — our generator is letters+digits, a controller change), mealie (`changeme@example.com / MyPassword`), wger (`admin / adminadmin`); **34 open first-run screens** (the first visitor creates the admin: actualbudget, adventurelog, audiobookshelf, calcom, docmost, emby, ghost, gitea, gramps-web, home-assistant, homebox, immich, jellyfin, komga, n8n, navidrome, opengist, outline, papra, plant-it, radarr, rallly, recipe-importer, romm, seerr, sonarr, sparkyfitness, tandoor, termix, uptime-kuma, vikunja, wanderer, wishlist, zipline). **Stale notes:** romm's `default_creds` `admin / admin` answers 401 on demo-hp (like a wrong password) — the page now warns with a login that does not exist; zipline's looks stale too. **Measured on demo-hp 2026-09-28 (read-only):** bookstack's default still logs in on the INSTALLED app (the fix is for new installs; the page now warns). Each fix: route (a) env or (b) the app's own CLI/API via `after_install:`, proven on 9202 with the default failing and the generated password working; route (c) a page sentence. Several sessions (operator, 2026-09-28). **2026-09-29 (controller v0.280.0, catalog `d0e7e2e`):** every class-3 app fixed — mealie, wger, calibre-web by `after_install` (calibre-web with the new `password:24:special`), proven on 9202 fresh installs (`audits/login-gate-2026-09-29/D/`); the setup gate (decision 46, spike PASSED) built and live on immich, n8n, audiobookshelf (probes measured) and uptime-kuma (button) (`…/C/`); romm's and zipline's stale notes removed. **Left: 30 class-4 apps** — gate each (probe measured on 9202 where one exists — 11 upstream candidates listed in `…/B/B-VERDICT.md` §3; the button otherwise). **2026-09-29 afternoon (controller v0.281.0, catalog `6faf432`):** 28 more class-4 apps gated — 32 of 34 — each proven on 9202 (`audits/gate-rollout-2026-09-29/`B): stranger → gate page / 401, household reached the first-setup screen, the gate opened (9 by a measured probe, the rest by the press), the app answered after. seerr, outline, rallly: gated, their opening needs a media server / e-mail (not proven). **Left:** wanderer (R-714); plant-it is not installable. | **NARROWED** (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — seerr, outline and rallly are gated, but the gate OPENING is not proven (needs a media server / e-mail)) — **CLOSED — 2026-09-29 (the rest → R-714)** | — | — | CC |
| **R-905** | Apps & catalog | P4 | **wger's collected style files (283 MB) go into every wger backup, though the app rebuilds them at every start.** Since R-762's fix (`DJANGO_DEBUG=False` + `collectstatic`, catalog `cf1ed43`) the static files sit in a named volume that the backup legs copy like data (measured 283 MB, `audits/design-build-2026-10-06/`); `collectstatic` regenerates them, so they are rebuildable bytes in every Tier-1 unit, Tier-2 copy and off-site snapshot. wger is `lifecycle: hidden` and no box runs it (hub read 2026-10-08). Options to weigh: a non-backed-up volume class for regenerable data, or an anonymous volume. Found by the R-762 helpers 2026-10-08. | **OPEN — needs a design before wger is shown** | — | Design the „regenerable volume" exclusion (catalog + controller backup legs) | CC |
@@ -154,7 +155,7 @@ stopping line that lies.
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
## Backup & restore — 30 rows (P2 3, P3 17, P4 10)
## Backup & restore — 31 rows (P2 3, P3 17, P4 11)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -188,11 +189,15 @@ stopping line that lies.
| **R-832** | Backup & restore | P4 | **ep0's copy in a place outside both Hetzner and the operator's home (roadmap).** Today DooPlex (the operator's home) holds it (decision 71). A Hetzner Storage Box would share a provider with ep0 and with every household's file backups, and cannot run PBS, so the copy could not be verified or restored from directly. | **DEFERRED — later, if the product grows** | — | — | operator |
| **R-878** | Backup & restore | P4 | **A catch-up (R-871) runs the database-dump leg in the DAY, and that leg stops an app with a volume for its copy — the household may notice the stop, and a large volume makes it longer.** MEASURED 2026-10-05 on demo-felhom: the catch-up at 08:25:02 stopped opengist, copied 182.5 KB, started it again — about 1 s, then a few seconds of `health: starting`; the night does exactly the same, unseen. Nothing measured for a large volume. Fix direction (if it matters): skip the volume copy of a running app in a DAYTIME catch-up and leave it to the next night, or warn. `audits/catchup-2026-10-05/partA/live-demo-felhom.txt` | **READY — owner: CC** **2026-10-05 (burn-down night): NEEDS A LIVE MEASUREMENT** — a timed catch-up on 9202 for an app with a multi-GB volume, then the skip-or-warn choice. | — | measure a large volume first | CC |
| **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.** **2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC |
| **R-1072** | Backup & restore | P4 | **The backup page speaks to the household in two registers in one card: „csatlakoztasson” (formal) and „Csatlakoztass” (informal).** MEASURED 2026-10-15 on controller 0.314.0, one-disk box, `/backups`: „Csak egy másolat készül (nincs második meghajtó) — a 3-2-1 mentéshez csatlakoztasson egy második meghajtót vagy offsite tárolót.” next to „… Csatlakoztass egy második meghajtót a teljes védelemhez.” The product's voice is informal (tegezés). Small; not fixed here because this run allowed no product change. Evidence: `audits/omv-vm-2026-10-15/B4-backup-pages.txt`. | READY | — | Change the first string to „csatlakoztass” (and „offsite tárolót” → „távoli tárhelyet”) in the controller's i18n file; the copy gates run. | CC |
## Storage & devices — 6 rows (P3 3, P4 3)
## Storage & devices — 9 rows (P3 6, P4 3)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1068** | Storage & devices | P3 | **On a box with one disk, a media app's (or any data-folder app's) Install is refused with a raw field error, and the way through is nowhere near the button.** MEASURED 2026-10-15 on the OMV-VM copy of the pilot's box (one passed NVMe, controller 0.314.0): `POST /api/stacks/jellyfin/deploy` with DOMAIN+SUBDOMAIN → **400 „a(z) "Médiatár útvonal" (HDD_PATH) mező kitöltése kötelező”** — the env var name reaches the household. R-1031's default-drive rule finds no drive because none is registered. The box DOES offer one: `/api/disks/candidates` listed `/mnt/sys_drive` (the installer's data volume, `already_mounted`), and `handleStorageRegisterMounted` (R-280) registers it — **seen in the candidate list and the code, NOT pressed** (the box was removed). With a second disk attached through „Csatlakoztatás”, the same Install succeeded in 30 s. Evidence: `audits/omv-vm-2026-10-15/README.md` „What happened”, `B4-deploy.txt`. | OPEN | — | Prove the `sys_drive` route live on a one-disk box (Tester 1 or a scratch VM); then make the Install refusal name it in household words (a link to Meghajtók), or register `sys_drive` by default on a one-disk box. | CC |
| **R-1069** | Storage & devices | P3 | **A disk passed into a Felhom VM on a virtio bus cannot be formatted, and the household reads the agent's raw English refusal.** MEASURED 2026-10-15 (OMV KVM plugin, „Add host disk”, default Bus `virtio`): `POST /api/disks/format` → 500 „agentapi: format: HTTP 409: format refused: device has no durable id to bind the format to (path-only formats are not permitted)”. A virtio-blk disk without a serial has no `/dev/disk/by-id` link inside the guest, so the anti-retarget guard refuses — correct and safe. The same disk on Bus `sata` got `byid:ata-QEMU_HARDDISK_QM00003` and formatted. Affects every hypervisor that passes disks as virtio without a serial (OMV KVM, Proxmox without `serial=`). The pilot guide says `sata`. Evidence: `audits/omv-vm-2026-10-15/B4c-second-disk-format.txt`. | OPEN | — | Translate the refusal for the household („Ezt a lemezt a Felhom nem tudja biztonságosan azonosítani — a virtuális gépben állítsd a lemez buszát SATA-ra”); optionally fall back to a filesystem-UUID binding after a blank-format. | CC |
| **R-1073** | Storage & devices | P3 | **Felhom in a virtual machine sees no disk health: SMART is UNKNOWN for its own system disk and any passed disk.** MEASURED 2026-10-15 on the OMV-VM copy: every storage target reported `smart.health = UNKNOWN`, all counters null (the guest sees QEMU disks). On the pilot's box the NVMe and the „old, not stable” Toshiba would be watched by OMV only, and Felhom's drive-health alarms (R-330/R-332's whole point) cannot fire. Evidence: the host report in `audits/omv-vm-2026-10-15/README.md` (memory/disk section) and `C-hub-events-Tester4.txt`. | WAITING-ON-OPERATOR | Operator: is a VM install a supported shape, and if so, is „OMV watches SMART” acceptable or must Felhom say on the dashboard that it cannot watch the disks? | Operator rules; then CC adds a household line on the Meghajtók page when SMART is unavailable. | Operator |
| **R-1062** | Storage & devices | P4 | **The N100's USB drive cannot be split in two without losing the box's whole-guest backup target.** The operator asked (2026-10-15 16:31) for two halves for multi-drive tests. Read first (`audits/n100-lived-in-2026-10-15/B/`): no family files — 2.7 GB nightly vzdumps of guest 9201 (`dump/`, PVE storage `felhom-backup` = the whole-guest backup target), 3.1 GB app backups + an off-site restore copy, 422 MB leftover app data of removed apps, empty media folders. The agent refuses a decommission of a backup target (`refuseIfBackupTarget`) and the N100 has no other drive to move that role to. Nothing changed. | **WAITING-ON-OPERATOR** — filed 2026-10-15 | operator decision | Keep one drive (CC's pick: the lived-in box keeps its local whole-guest backup), or give a word to drop that backup leg / add a second drive first | operator |
| **R-1063** | Storage & devices | P3 | **„Végleges leszerelés" leaves a half-state when the agent refuses: the dashboard marks the drive decommissioned first and never undoes it.** `handleStorageDecommission` → `finalizeDecommissionWith` calls `settings.SetDecommissioned` (and may move the default) BEFORE `agent.Decommission`; on the agent's 403 (protected role) or 409 (whole-guest backup target) it returns the error and keeps the mark — the dashboard would show a retired drive the agent kept serving. Read from code 2026-10-15 (`felhom-controller` `internal/web/storage_handlers.go`); NOT pressed on the lived-in N100 on purpose. | **OPEN** — filed 2026-10-15 | — | Measure on demo-hp (a backup-target drive), then: ask the agent first (or undo the mark and the default on a refusal); a test with a refusing agent seam, red first | CC |
| **R-1064** | Storage & devices | P4 | **Removing a storage path does not mention the cross-drive backup copies it holds.** demo-hp 2026-10-15: `/mnt/sizing-bench` removed through `/settings/storage/remove` (302, „Adattároló eltávolítva") while calibre-web's and paperless-ngx's `cross_drive.destination_path` pointed at it; the copies stay on disk, the records still name the removed path. The in-use check counts HDD_PATH only (`appsUsingPath`). | **OPEN** — filed 2026-10-15 | — | Read what tonight's tier-2 run (01:30) does with the two records; then: name the copies in the removal's confirmation (or count them as in use) | CC |
@@ -244,10 +249,11 @@ stopping line that lies.
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
## Monitoring & notifications — 10 rows (P2 1, P3 6, P4 3)
## Monitoring & notifications — 11 rows (P2 1, P3 7, P4 3)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1070** | Monitoring & notifications | P3 | **A box with no Cloudflare tunnel gets an ERROR after its host OS update: „NOT healthy … (the tunnel is unknown)”.** MEASURED 2026-10-15 16:30 on Tester4-68d1d8 (throwaway customer, placeholder domain `tester4.test`, so no tunnel token): hub event `os_update_health_failed` (error) „OS update (host) on Tester4-68d1d8: NOT healthy after 106 package(s) were installed (the tunnel is unknown). Nothing was undone automatically; put a host package back by hand…”. The box was healthy (dashboard 200, apps installing). A tunnel that never existed is read as a failed health check. Would hit every test customer and any box whose tunnel is not set yet. Evidence: `audits/omv-vm-2026-10-15/C-hub-events-Tester4.txt`. | OPEN | — | Find the health verdict's tunnel input (hub os-update health); treat „no tunnel configured” as not-applicable, not as unhealthy; red-proof with a no-tunnel fixture. | CC |
| **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
| **R-1001** | Monitoring & notifications | P3 | **Every hub restart re-mails the operator about a box that is already known to be down.** MEASURED 2026-10-13 18:33 UTC (hub v0.149.0 rollout, R-998 session): Tester 2, silent 216 h, got a fresh `host_down` and `restore_test_stale` mail. `monitor/host_staleness.go` leaves a down host unseeded on start so the first Check emits (F2), and its comment says „the dispatcher's 1h cooldown dedups the re-emit" — but host liveness events bypass that cooldown (`dispatcher.go` `nodeLivenessEvents`, 5-min dedupe only); `restore_test_stale` keeps its edge state in memory. The comment states an invariant the code does not provide. | **READY** — filed 2026-10-13 | — | Seed a host silently when a `host_down`/`host_stale` event for it is newer than its last report; the same for the restore-test edge; a test that restarts the checker | CC |
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |