Media folder fix: live read-back on demo-hp 9201 (hardlink, live catalog bc54c01); R-1014/1015/1016/1030 closed, R-1019 narrowed, R-1039/1040 filed; report

2026-10-14 22:01:09 +02:00
parent adc9dee0e3
commit 5a03fd4a57
10 changed files with 241 additions and 11 deletions
@@ -1,3 +1,5 @@
+**S1–S3 pushed; the live-fix task may start.** (Restored at the top 2026-10-14 ~21:20: the final-report edit had pushed it down. The live fix is done — `REPORT-media-folder-fix-2026-10-14.md`.)
+
 **Parts A–E done. Waiting on the operator: the layout decision (STATUS D9) and the teardown word (R-1022).**
 Read before the live fix: `media/` also holds photos (`15` §2); removal deletes literal binds (R-1027, R-1032).

@@ -0,0 +1,61 @@
+# REPORT — the media folder fix in the live catalog (one drive), 2026-10-14
+
+> A topic report (`CLAUDE.md`: parallel sessions do not share the root `REPORT.md`). Evidence:
+> `documentation/audits/media-folder-fix-2026-10-14/`. Architecture read: `15-media-library.md`, `07` §6,
+> `09` §1.2/§3; the first task's report.
+
+## In plain words
+
+1. **Does the media chain work now for a customer with one drive?** Yes. Proven on demo-hp guest 9201 with the live
+   catalog: Radarr and qBittorrent installed through the dashboard, one public-domain film handed to Radarr; the film in
+   the library is the downloaded file under a second name (link count 2, one inode). No extra space.
+2. **Can the download apps reach the photos?** No. Inside qBittorrent and Transmission the `media/photos` path does not
+   exist; the media folder shows only their own download folder. A write to the photos path lands only in the
+   container's own layer; the photos on the drive stay unchanged (bench).
+3. **What happens to an app that was already installed?** It gets the new template from the catalog sync and uses it
+   at its next start. Its old torrents keep seeding: the old `/downloads` path stays, as a second name of the same folder.
+   New downloads go to the same-path folder. Measured on the bench for both download apps: an install from the old
+   template, then updated.
+4. **Backup, export, remove: unchanged in meaning?** Yes. Sonarr/Radarr's new `media` bind is declared `excluded`:
+   no backup tier, no export, no removal touches it. Without that line the whole `media` folder, photos included, would
+   have gone into every tier and off-site — measured with the controller's own code. Transmission's `watch` folder is
+   now declared too; before, it went off-site by mistake.
+5. **Sonarr/Radarr sign-in default:** seeded. A new install starts with „Authentication Required: Enabled". An existing
+   install keeps the household's choice. Until the household makes its first admin, other apps on the box can still read
+   the API key — filed as R-1040.
+6. **Claims in the brief that turned out wrong:**
+   - „a write outside the download folder fails" — the write succeeds, but only inside the container; nothing reaches
+     the drive (B-3).
+   - „`/media/shows` in Sonarr and Jellyfin" — also in Plex, and in Jellyfin's prerequisites too.
+   - Part A: the hub's `/apps` page gives a COUNT per app, not the boxes. The boxes were read one by one; the hub's
+     „2" = Tester 1 (the rig) + an older report from demo-felhom's sizing bench.
+   - The first report's top line had been pushed down by my own final edit; restored at the top.
+
+## What changed
+
+- **app-catalog `bc54c01`** — sonarr, radarr (one `media` bind, `excluded`; first-start seed; first steps hu+en);
+  qbittorrent, transmission (only their folder at the same path; `/downloads` kept; default folders moved at start;
+  watch declared); plex, jellyfin (`/media/tv`); onboarding 2.2 rows; copy freeze (only these lines). CHANGELOG +
+  REPORT there.
+- **felhom.eu** — `09` §3 decisions 242 (D9 = A with a limit) and 243 (the N100 waits); `15` §2 ruled table; evidence.
+
+## Gates and measurements
+
+- `catalog_gates.py --fast` for the six apps: all OK (the copy gate first caught a Hungarian suffix error, „filmra" →
+  „filmre", fixed).
+- `check-volume-persistence.py` on the bench: sonarr, radarr, qbittorrent, transmission, jellyfin CLEAN; plex
+  UNDETERMINED (`/transcode` empty after the exercise; plex's folders did not change in this task).
+- Downloads: public domain only, archive.org; removed after the read-back (Radarr deleted the film with its files,
+  qBittorrent deleted the torrent with its data); both apps removed WITHOUT „delete data". qBittorrent stalled on
+  archive.org's last piece again; a throwaway Transmission on the bench gave it as a local peer (R-1039).
+
+## Register
+
+Rows: before 156 → after 154. Opened 2 (R-1039, R-1040). Closed 4 (R-1014, R-1015, R-1016, R-1030). Narrowed 1
+(R-1019: the watch half fixed; the Tier-2 mirror of optional downloads stays open). STATUS: 52 lines; D9 removed (ruled).
+
+## Teardown
+
+Machine: demo-hp 9201 — Radarr and qBittorrent removed, their volumes gone, the household folders kept (empty
+`movies`, `torrents/.temp`). Bench 9401 — throwaway containers, folders and the catalog copy removed; the bench is left
+RUNNING (it was stopped before this task). Host: nothing changed. Hub: read only. Tester 1 and the N100: untouched.
@@ -6,6 +6,7 @@
 `documentation/backlog/STATUS-HISTORY.md`. Reports: `REPORT-immich-rollout-2026-10-14.md`, `REPORT-seo-felho-tarhely-2026-10-14.md`, `REPORT-immich-repair-2026-10-14.md`, `REPORT-power-cut-fixes-2026-10-14.md`, `REPORT-night-power-2026-10-13.md`, `REPORT-poweroff.md`.

 ## What works
+- Media apps work together on one drive: a download reaches Sonarr/Radarr as one file, no copy; the download apps see only their own folder (live catalog, proven on demo-hp).
 - A power cut in the middle of a backup no longer blocks later backups: the box removes the half-written backup at start-up (proven with a real cut on demo-hp, on all three boxes now).
 - After a power cut, Immich repairs the photos the cut left broken, by itself, and the app page lists any lost photo (proven with a real cut on demo-hp: 41 of 41 repaired; on all three boxes now). An Immich update cannot switch it off quietly: the catalog refuses an untested Immich version.
 - A household can switch the machine off; the hub holds alarms for 14 days and clears the state by itself (N100).
@@ -37,7 +38,6 @@
 | D6 | R-992 | The scratch box cannot read the private drill catalog: give it a read-only login, or make that copy readable? | **A read-only login** | Readable copy: the drill catalog becomes public | every new-app test needs hand-staging | new-app tests |
 | D7 | R-990, R-989 | Rule yes / no / later on the forum ideas (`audits/forum-research-2026-10-12/FELHOM.md` §5) and the selfh.st ideas (`audits/selfhst-survey-2026-10-12/FELHOM.md` §6) | — | — | nothing changes; the research ages | — |
 | D8 | R-206 | Install DooPlex's Docker settings from git (restarts Docker), and prune the build cache weekly: yes or no? | **Yes, in a quiet window** | — | a rebuild of DooPlex still has the settings in git | — |
-| D9 | R-1023 | Media apps: bind the whole `media` folder (download apps can then write into Photos and Music), or move films and shows to their own folder first? | **Own folder** (`15-media-library.md` §2) | Whole `media`: no move, but the internet-facing apps reach the family photos | the build waits for this answer | the build |

 ## Waiting on you — do
 - **Clear demo-hp's own floor** (hub → Demo HP → floor: empty → Save floor). It still reads 0.311.0. If not: the next global raise passes demo-hp by.
@@ -15,8 +15,8 @@ import json, os, secrets, sys, time
 sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
 import box_walk as w

-HDD = "/mnt/felhom-drives/media_a"
-SEC = os.path.expanduser("~/.felhom-tester1/secrets.json")
+HDD = os.environ.get("HDD", "/mnt/felhom-drives/media_a")
+SEC = os.path.join(os.environ.get("SC", os.path.expanduser("~/.felhom-tester1")), "secrets.json")


 def load():
@@ -0,0 +1,13 @@
+radarr: delete movie 1 with files -> 200
+qbittorrent: torrents deleted with their data
+left in movies: 
+left in torrents: .temp 
+
+radarr stop -> 200
+radarr remove WITHOUT delete data -> 200 {'ok': True, 'data': {'removed': 'radarr', 'volumes_removed': ['radarr_radarr_config'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'userdata_kept': ['/mnt/felhom-drives/hdd_1/userdata/media (
+qbittorrent stop -> 200
+qbittorrent remove WITHOUT delete data -> 200 {'ok': True, 'data': {'removed': 'qbittorrent', 'volumes_removed': ['qbittorrent_qbittorrent_config'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'userdata_kept': ['/mnt/felhom-drives/hdd_1/u
+0
+/opt/docker/stacks/qbittorrent
+/opt/docker/stacks/radarr
+audiobooks books comics grimmory metube movies music nzbget photos podcasts torrents transmission tv usenet 
@@ -0,0 +1,22 @@
+21:41:40 qbittorrent: POST deploy {"DOMAIN": "enkisfelhom.hu", "SUBDOMAIN": "torrent", "HDD_PATH": "/mnt/felhom-drives/hdd_1", "QBT_ADMIN_PASSWORD": "<generated, kept in secrets.json>"} -> 202 {"ok": true, "message": "The install has started – the card shows how it is going"}
+21:41:56 qbittorrent: deployed, state=running, pinned={'qbittorrent': 'qbittorrentofficial/qbittorrent-nox:5.2.4-1'}
+running /var/lib/docker/volumes/qbittorrent_qbittorrent_config/_data->/config:true /mnt/felhom-drives/hdd_1/userdata/media/torrents->/downloads:true /mnt/felhom-drives/hdd_1/userdata/media/torrents->/mnt/felhom-drives/hdd_1/userdata/media/torrents:true 
+2026/10/14 19:41:40 router.go:449: [INFO] [api] Deploy requested for stack: qbittorrent
+2026/10/14 19:41:40 deploy.go:526: [INFO] [stacks] Deploying stack qbittorrent with 4 env vars: [DOMAIN, SUBDOMAIN, HDD_PATH, QBT_ADMIN_PASSWORD]
+2026/10/14 19:41:40 manager.go:1773: [INFO] [stacks] Deploying stack qbittorrent — checking 1 images...
+2026/10/14 19:41:41 notifier.go:353: [INFO] Event pushed: app_deploy_started (info) [+household(en)] — Alkalmazás telepítése elindult: qBittorrent
+2026/10/14 19:41:46 deploy.go:616: [INFO] [stacks] Stack qbittorrent deployed successfully (took 5.3s)
+2026/10/14 19:41:46 notifier.go:353: [INFO] Event pushed: app_deployed (info) [+household(en)] — Alkalmazás telepítve: qBittorrent
+2026/10/14 19:41:51 after_install.go:184: [INFO] [stacks] after_install qbittorrent: qbittorrent [/bin/sh -c curl -s -c /tmp/fc -o /dev/null -X POST -d "username=admin&password=${QBT_ADMIN_PASSWORD}" -H 'Referer: http://127.0.0.1:8080' http://127.0.0.1:8080/api/v2/auth/login && curl -s -b /tmp/fc -o /dev/null -X POST --data-urlencode 'json={"upnp":false,"web_ui_domain_list":"${SUBDOMAIN}.${DOMAIN};qbittorrent;localhost;127.0.0.1"}' -H 'Referer: http://127.0.0.1:8080' http://127.0.0.1:8080/api/v2/app/setPreferences && curl -s -b /tmp/fc http://127.0.0.1:8080/api/v2/app/preferences | grep -q '"upnp":false' && curl -s -o /dev/null -w '%{http_code}' -H 'Host: qbittorrent:8080' http://127.0.0.1:8080/ | grep -qE '^(200|401|403)$' && echo FELHOM_AFTER_INSTALL_OK
+2026/10/14 19:41:51 install_hold.go:135: [INFO] [stacks] qbittorrent: install hold OPENED by after_install — the app is reached as without a hold
+
+21:42:00 radarr: POST deploy {"DOMAIN": "enkisfelhom.hu", "SUBDOMAIN": "radarr", "HDD_PATH": "/mnt/felhom-drives/hdd_1"} -> 202 {"ok": true, "message": "The install has started – the card shows how it is going"}
+21:42:30 radarr: deployed, state=running, pinned={'radarr': 'lscr.io/linuxserver/radarr:6.4.4'}
+running /var/lib/docker/volumes/radarr_radarr_config/_data->/config:true /mnt/felhom-drives/hdd_1/userdata/media->/mnt/felhom-drives/hdd_1/userdata/media:true 
+2026/10/14 19:41:59 router.go:449: [INFO] [api] Deploy requested for stack: radarr
+2026/10/14 19:42:00 deploy.go:526: [INFO] [stacks] Deploying stack radarr with 3 env vars: [DOMAIN, SUBDOMAIN, HDD_PATH]
+2026/10/14 19:42:00 manager.go:1773: [INFO] [stacks] Deploying stack radarr — checking 1 images...
+2026/10/14 19:42:00 notifier.go:353: [INFO] Event pushed: app_deploy_started (info) [+household(en)] — Alkalmazás telepítése elindult: Radarr
+2026/10/14 19:42:10 deploy.go:616: [INFO] [stacks] Stack radarr deployed successfully (took 10.0s)
+2026/10/14 19:42:10 notifier.go:353: [INFO] Event pushed: app_deployed (info) [+household(en)] — Alkalmazás telepítve: Radarr
+
@@ -0,0 +1,27 @@
+== Radarr: what the first steps say
+drwxrwsr-x 2 root users 4096 Jul 21 19:08 /mnt/felhom-drives/hdd_1/userdata/media/movies
+drwxrwsr-x 2 abc  users 4096 Oct 13 21:26 /mnt/felhom-drives/hdd_1/userdata/media/torrents
+audiobooks books comics grimmory metube movies music nzbget photos podcasts torrents transmission tv usenet
+  root folder /mnt/felhom-drives/hdd_1/userdata/media/movies -> 201
+  download client qBittorrent (its own connection test) -> 201
+  movie added: Night of the Living Dead (1968) path=/mnt/felhom-drives/hdd_1/userdata/media/movies/Night of the Living Dead (1968)
+  release push -> 200 approved=True rejections=[]
+  grabbed by qBittorrent
+  +65s queue: downloading downloading left=594823937 output=None
+  +125s queue: downloading downloading left=300174081 output=None
+  +185s queue: downloading downloading left=300174081 output=None
+  +245s queue: downloading downloading left=173673216 output=None
+  +305s queue: downloading downloading left=72043264 output=None
+  +365s queue: downloading downloading left=72043264 output=None
+  +425s queue: downloading downloading left=35585 output=None
+  +485s queue: warning downloading left=35585 output=None
+  +545s queue: warning downloading left=35585 output=None
+  +605s queue: warning downloading left=35585 output=None
+  +665s queue: warning downloading left=35585 output=None
+  +725s queue: warning downloading left=35585 output=None
+  +735s history downloadFolderImported: {"downloadClientName": "qBittorrent", "droppedPath": "/mnt/felhom-drives/hdd_1/userdata/media/torrents/night-of-the-living-dead-1968_202508/Night of the Living Dead 1968.mp4", "importedPath": "/mnt/felhom-drives/hdd_1/userdata/media/movies/Night of the Living Dead (1968)/night-of-the-living-dead-1968_202508.mp4"}
+  download: links=2 inode=9568271  library: links=2 inode=9568271
+  RESULT: HARDLINK — one file, two names
+# NOTE: qBittorrent stalled on the LAST piece (archive.org's web seed; the same fault as on Tester 1 that afternoon). At
+# 21:53 a throwaway Transmission on the bench (192.168.0.118, LXC 9401) fetched the same public-domain torrent and was
+# given to qBittorrent as a peer (addPeers); qBittorrent completed at 21:54:43 and Radarr imported at +735 s.
@@ -0,0 +1,98 @@
+#!/usr/bin/env python3
+"""live_readback.py — Part B6: the one-drive chain on a box with the LIVE templates, run INSIDE the guest.
+
+Does what the new first steps tell the household to do — Radarr: root folder `.../media/movies`; one qBittorrent
+entry (host qbittorrent, port 8080, admin, the install's password, category `movies`) — then hands ONE public-domain
+archive.org film to Radarr by its release push and waits for the import. Proof: download path, library path, link
+count, inode, Radarr's history entry. Keys and passwords are read at runtime and never printed.
+
+    run_in_guest-style:  python3 live_readback.py <drive mount, e.g. /mnt/felhom-drives/hdd_1>
+"""
+import datetime, json, os, re, subprocess, sys, time, urllib.error, urllib.request
+
+HDD = sys.argv[1]
+LIB = f"{HDD}/userdata/media/movies"
+ITEM = "night-of-the-living-dead-1968_202508"   # CC Public Domain Mark, 597 MB
+SECRETS = []
+
+
+def red(s):
+    s = str(s)
+    for x in SECRETS:
+        s = s.replace(x, "<redacted>") if x else s
+    return s
+
+
+def say(*a):
+    print(red(" ".join(str(x) for x in a)), flush=True)
+
+
+def sh(c):
+    return subprocess.run(c, shell=True, capture_output=True, text=True).stdout.strip()
+
+
+def ip(c):
+    return sh(f"docker inspect {c} -f '{{{{range .NetworkSettings.Networks}}}}{{{{.IPAddress}}}}{{{{end}}}}'")
+
+
+key = re.search(r"<ApiKey>([^<]+)", open("/var/lib/docker/volumes/radarr_radarr_config/_data/config.xml").read()).group(1)
+pw = sh("docker exec qbittorrent printenv QBT_ADMIN_PASSWORD")
+SECRETS += [key, pw]
+RA = f"http://{ip('radarr')}:7878/api/v3"
+H = {"X-Api-Key": key, "Host": "radarr", "Content-Type": "application/json"}
+
+
+def ra(m, p, body=None):
+    r = urllib.request.Request(RA + p, data=json.dumps(body).encode() if body is not None else None, headers=H, method=m)
+    try:
+        with urllib.request.urlopen(r, timeout=120) as x:
+            t = x.read(); return x.status, (json.loads(t) if t else None)
+    except urllib.error.HTTPError as e:
+        b = e.read().decode()[:400]; say(f"  !! {m} {p} -> {e.code} {b}"); return e.code, None
+
+
+say("== Radarr: what the first steps say")
+print(sh(f"docker exec radarr sh -c 'ls -ld {LIB} {HDD}/userdata/media/torrents; ls {HDD}/userdata/media | tr \"\\n\" \" \"'"))
+if not any(r["path"].rstrip("/") == LIB for r in ra("GET", "/rootfolder")[1]):
+    c, r = ra("POST", "/rootfolder", {"path": LIB}); say(f"  root folder {LIB} -> {c}")
+sch = {s["implementation"]: s for s in ra("GET", "/downloadclient/schema")[1]}
+if not ra("GET", "/downloadclient")[1]:
+    b = json.loads(json.dumps(sch["QBittorrent"]))
+    vals = {"host": "qbittorrent", "port": 8080, "username": "admin", "password": pw, "movieCategory": "movies"}
+    for f in b["fields"]:
+        if f["name"] in vals:
+            f["value"] = vals[f["name"]]
+    b.update({"name": "qBittorrent", "enable": True})
+    c, r = ra("POST", "/downloadclient", b); say(f"  download client qBittorrent (its own connection test) -> {c}")
+c, lk = ra("GET", "/movie/lookup/tmdb?tmdbId=10331")
+qp = ra("GET", "/qualityprofile")[1][0]
+c, m = ra("POST", "/movie", dict(lk, qualityProfileId=qp["id"], rootFolderPath=LIB, monitored=True, tags=[],
+                                 minimumAvailability="released", addOptions={"searchForMovie": False}))
+say(f"  movie added: {m['title']} ({m['year']}) path={m['path']}")
+rel = {"title": "Night.of.the.Living.Dead.1968.480p.WEB-DL.x264-ARCHIVEORG", "protocol": "torrent", "size": 596645320,
+       "downloadUrl": f"https://archive.org/download/{ITEM}/{ITEM}_archive.torrent", "indexer": "archive.org (release push)",
+       "publishDate": datetime.datetime.now(datetime.timezone.utc).isoformat()}
+t0 = time.time()
+c, d = ra("POST", "/release/push", rel)
+dd = d[0] if isinstance(d, list) else (d or {})
+say(f"  release push -> {c} approved={dd.get('approved')} rejections={dd.get('rejections')}")
+imp = None
+for i in range(150):
+    time.sleep(10)
+    for e in ra("GET", f"/history/movie?movieId={m['id']}")[1] or []:
+        if e["eventType"] == "grabbed" and i == 0:
+            say(f"  grabbed by {e['data'].get('downloadClientName')}")
+        if e["eventType"] == "downloadFolderImported":
+            imp = e
+    if imp:
+        break
+    if i % 6 == 5:
+        for r in (ra("GET", f"/queue?movieIds={m['id']}")[1] or {}).get("records", []):
+            say(f"  +{int(time.time()-t0)}s queue: {r.get('status')} {r.get('trackedDownloadState')} left={r.get('sizeleft')} output={r.get('outputPath')}")
+if not imp:
+    say(f"  RESULT: not imported within {int(time.time()-t0)}s"); sys.exit(1)
+e = imp["data"]
+say(f"  +{int(time.time()-t0)}s history downloadFolderImported: {json.dumps({k: e.get(k) for k in ('downloadClientName', 'droppedPath', 'importedPath')})}")
+a, b = os.stat(e["droppedPath"]), os.stat(e["importedPath"])
+say(f"  download: links={a.st_nlink} inode={a.st_ino}  library: links={b.st_nlink} inode={b.st_ino}")
+say("  RESULT:", "HARDLINK — one file, two names" if (a.st_ino, a.st_dev) == (b.st_ino, b.st_dev) else "COPY")
@@ -26,6 +26,15 @@

 ---

+## 2026-10-14 — the media folders meet in the live catalog
+
+| Row | What | Closed | Evidence |
+|---|---|---|---|
+| **R-1014** | **The media chain could not work: the download folders did not meet, and the hardlink comment was false** (P2) | CLOSED 2026-10-14 — FIXED in the live catalog (decision 242, D9 = A with a limit): Sonarr/Radarr bind `media` at the same path, the clients only their own folder at the same path; the false comment replaced by the measured reason. Live read-back on demo-hp 9201: one public-domain film qBittorrent → Radarr imported as a HARDLINK (links 2, one inode). Installed clients keep their old torrents (bench). | app-catalog `bc54c01`; `audits/media-folder-fix-2026-10-14/``C-live-readback-9201.txt`, `B2-B3-*.txt` |
+| **R-1015** | **Three apps told the household to use `/media/shows`** (P3) | CLOSED 2026-10-14 — FIXED: Sonarr, Plex and Jellyfin first steps (hu + en) name `media/tv`; the copy freeze re-captured for those lines only. | app-catalog `bc54c01` |
+| **R-1016** | **qBittorrent's onboarding record named the wrong download folder** (P4) | CLOSED 2026-10-14 — FIXED: `onboarding/qbittorrent.md` and `transmission.md` row 2.2 name the real folders, with the bench evidence. | app-catalog `bc54c01` |
+| **R-1030** | **Sonarr's and Radarr's first admin must default to „Enabled”** (P2) | CLOSED 2026-10-14 — FIXED for new installs: the first start seeds `AuthenticationRequired=Enabled` into `config.xml`; the first-admin form then reads „enabled”; an existing `config.xml` is never touched (bench). The window BEFORE the household makes its admin stays open to other containers — filed as R-1040. | app-catalog `bc54c01`; `audits/media-folder-fix-2026-10-14/``B2-sonarr-radarr.txt` |
+
 ## 2026-10-14 — the Immich repair on every box; the shared checkout cleaned

 | Row | What | Closed | Evidence |