From 790ec845f7988971a5da3c6f2a802aaaf322bc9d Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 14 Jul 2026 16:39:55 +0200 Subject: [PATCH] =?UTF-8?q?docs(audit):=20backup-classification=20spike=20?= =?UTF-8?q?=E2=80=94=20coupling=20model=20VALIDATED=20(Immich=20DB-only=20?= =?UTF-8?q?restore=20broken-not-empty);=20offsite=3Dunit-only=20live-prove?= =?UTF-8?q?n;=20NEW=20F-S1=20(tier-2=20copies=20no=20userdata)=20+=20F-S2?= =?UTF-8?q?=20(paperless-ngx=20appdata=20skipped);=20round-trip=20verdict:?= =?UTF-8?q?=20exclusion=20scoping?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- REPORT.md | 75 ++-- .../SPIKE-backup-classification-2026-07-14.md | 405 ++++++++++++++++++ 2 files changed, 447 insertions(+), 33 deletions(-) create mode 100644 documentation/audits/SPIKE-backup-classification-2026-07-14.md diff --git a/REPORT.md b/REPORT.md index b7159b8..381016b 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,42 +2,51 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## CAMPAIGN-6B — unattended close-out of the C6 remainder — 2026-07-14 +## SPIKE — referential-coupling backup classification — 2026-07-14 -**controller 0.129.0 / agent 0.88.0 / hub 0.54.0 · demo + drill · findings-only, unattended** +**controller 0.130.0 (demo 9201) / agent 0.88.0 · findings-only, no production code · full audit: +`documentation/audits/SPIKE-backup-classification-2026-07-14.md`** -Continuation of CAMPAIGN-6A. Drove the `.fab` full-circle (upload/download + real Cloudflare edge), -the server-observable browser/DOM planes, backup-tier + NAS integrity, and regression spot-checks. -Full audit: `documentation/audits/CAMPAIGN-6B-2026-07-14.md`. Continuation: `CAMPAIGN-6C-2026-07-14-PROMPT.md`. +Empirical ground-truth pass for the userdata-backup-classification redesign +(mandatory=COUPLED / optional=DECOUPLED-precious / excluded=DECOUPLED-bulk). -### Headline finding — C6B-F1 (CRITICAL) -**`.fab` export silently produces hollow, data-free bundles for the standard `${USERDATA_PATH}` -`needs_hdd` app convention** — 12 of 13 media catalog apps (jellyfin, plex, emby, immich, sonarr, -radarr, navidrome, paperless-ngx, audiobookshelf, calibre-web, komga, romm). Live: sonarr with -**4.17 GB / 7 files** exported to a **2308-byte, config-only** bundle, reported `done` with no error, -past the v0.125.0 anti-hollow guard. Two compounding causes in the `needs_hdd` branch: `executeExport` -is either/or (`needs_hdd` apps never run `exportVolumeData`, dropping named volumes incl. the app DB), -and `ParseComposeHDDMounts` substitutes only `${HDD_PATH}`, never `${USERDATA_PATH}` → 0 mounts → all -userdata skipped. The guard only checks manifest-*claimed* tars, so 0-claimed passes. `.fab` is the -customer portability/backup-download path → fresh/cross-box restore = silent total data loss. Fix -direction in the audit; **priority for project Claude to spec.** +### What was established -### What passed -- **`.fab` transport is byte-clean end-to-end** — LAN + real Cloudflare edge, upload (64 MiB chunked) - and download, every sha256 == reference. -- **413 edge-cap re-proven** (120 MiB → CF 413 before origin; 80 MiB clears to origin). -- **Upload state machine 8/8** — wrong-ext 400, collision `(N)`, single-flight 409, strict-offset 409, - abort cleanup, oversize (both Hungarian numbers), restart-GC, idle-15min expiry. -- **F7 atomicity** (`.tar.tmp`→rename) code-confirmed; **native `alert()`/`confirm()` absent** on all - surfaces; CSRF stale → 403, session-expiry → 401-JSON/redirect; **F4** uid-range friendly 400; - agent-restart per-share re-arm verdicts logged. +- **SQ1 — offsite carries no userdata today, no exception (source + live).** All 12 restic + snapshots are `backups/primary/` units only; a full `restic ls` of the newest immich + snapshot shows compose + `immich-postgres.sql` + manifest — **immich is offbox-toggled and its + photo library is in none of its snapshots** (the latent broken-restore gap, now live-proven). +- **NEW F-S1:** tier-2 does **not** copy bulk userdata — `RunTier2` rsyncs only the unit + + `appdata/`; the `tier2.go:17` "recovery unit + bulk userdata" header and the + `main.go:1360` "namespace wholesale" comment are stale. Browsable HDD userdata has **zero + automated off-drive protection today** (live: `secondary/{radarr,jellyfin}` hold only + `recovery-unit/`). +- **NEW F-S2:** tier-2's appdata leg keys on the stack name → **paperless-ngx** + (`appdata/paperless`) is silently skipped (live contrast: `secondary/nextcloud` has `appdata/`, + `secondary/paperless-ngx` doesn't) — its document originals get no tier-2 copy. +- **SQ3 — the coupling model is VALIDATED (scratch Immich, catalog images v2.5.5).** DB-present / + library-absent restore: the server **refuses to boot** (its own `.immich` folder-check, with the + mount state remembered *in the DB*); with the skeleton recreated it boots into a **lying + gallery** — timeline reports all 3 assets, every thumbnail/original fetch is HTTP 500. No repair/ + re-import path. Restoring the files → all 200 again. **Worse-than-empty: confirmed.** +- **SQ2** — full per-bind classification for all 55 catalog apps (13 hdd + 42 volume-only); + shared-media-tree consistency resolved by the "class belongs to the bind; the writing app is the + authority; `:ro` readers back up nothing" rule. +- **SQ4** — size split measured live (state 0.5 M–933 M vs bulk tree ~100 MB demo / + TB-scale prod); the two-number estimate reuses the existing `estimate.go` per-mount du + + container-view volume sizer — zero new measurement plumbing. +- **SQ5 — round-trip verdict: EXCLUSION scoping** (tar the userdata root minus excluded subtrees). + Per-bind nested tars cannot ride the v1 manifest (`HDDSubdirs` basenames; CTRL-001 forbids path + separators); exclusion scoping needs no manifest/import change and stays backward-compatible. + Proposed `backup:` block schema + **two-level default** (no block → legacy per-tier behavior; + block present → unlisted writable binds mandatory) — the task's single-level mandatory-default + premise would have shipped unmigrated media trees offsite (quota blow-up) and was corrected. +- **SQ6** — `.fab` over-capture quantified on the live tree (radarr bundle carries ~1.6× its own + data even on the small demo tree, unbounded in prod); the live marker-file export leg is blocked + on the customer-owned dashboard login (same blocker as the open Viktor full-circle test). -### Split to CAMPAIGN-6C -Supervised browser planes (escrow wizard click-through, hub 8-tab UX); deep backup tiers (live F7 -6 s-cut, restic self-heal, offsite restore-to-verify, tier-2 F6, per-app toggles, snapshot coherence); -dead-app alert + email cooldown; the **C6B-F1 fix-verify** full-circle byte-compare. +### Cleanup -### Notes -- No code changed. A temporary break-glass credential was used on the demo controller to drive the - authed tests, then **fully reverted at cleanup** (original customer-claimed hash restored) — the box - is back to its pre-6B credential; **Viktor rotates** that. No secret/R/blob committed. +Scratch Immich fully removed (containers/volumes/network/files/images; verified 0 spike containers, +controller healthy). Restic inspection read-only (`--no-lock`), 12 snapshots before and after. No +drive namespace, tier-2 target, or deployed app touched. diff --git a/documentation/audits/SPIKE-backup-classification-2026-07-14.md b/documentation/audits/SPIKE-backup-classification-2026-07-14.md new file mode 100644 index 0000000..2295a5f --- /dev/null +++ b/documentation/audits/SPIKE-backup-classification-2026-07-14.md @@ -0,0 +1,405 @@ +# SPIKE — referential-coupling backup classification (2026-07-14) + +**Class:** spike (findings only — no production code, no schema edits, no repo mutations beyond this doc). +**Versions at spike time (recorded live):** controller **0.130.0** on demo guest 9201 (healthy), +agent **0.88.0** on felhom-pve (active), catalog @ HEAD (55 templates). Offsite repo: Hetzner +Storage Box sub-account, 12 snapshots, last run 2026-07-14 02:15 UTC, status ok, all inspection +**read-only** (`--no-lock`; no prune/forget/write). +**Empirical core:** a scratch Immich (catalog-pinned images v2.5.5) deployed in an isolated +namespace on 9201, exercised through the DB-present/library-absent restore scenario, then fully +removed. + +Product decision under test: classify each userdata bind by **referential coupling** — +**mandatory** (COUPLED: DB/config references it; restoring state without it yields a *broken* app), +**optional** (DECOUPLED-precious: user-specific, not re-downloadable, app re-derives on restore → +empty-not-broken), **excluded** (DECOUPLED-bulk: large, replaceable, cross-app). + +## Verdict up front + +1. **Offsite carries NO userdata today — confirmed at source and live, no exception.** Every restic + snapshot path is `backups/primary/` (unit only). The latent gap is real: **immich is + offbox-toggled on demo, yet its photo library is in none of its 5 snapshots** (§SQ1). +2. **The coupling model is VALIDATED by the Immich experiment** — DB-only restore is not merely + empty, it is broken at two escalating levels: the server **refuses to boot** (its own + folder-check), and once coaxed past that, the timeline still advertises every asset while **every + thumbnail/original fetch is HTTP 500**. No self-heal, no re-import path for managed uploads + (§SQ3). +3. **Two NEW capture-gap findings** beyond the task baselines: (F-S1) tier-2 does **not** copy bulk + userdata — the tier2.go header comment is stale; browsable HDD userdata has **zero automated + off-drive protection today**; (F-S2) tier-2's appdata leg keys on the **stack name**, so + paperless-ngx (`appdata/paperless`) is **silently skipped** — live-proven (§SQ1). +4. **Round-trip verdict: exclusion scoping** (tar the userdata root, skip excluded subtrees) is the + only shape that round-trips through the v1 `.fab` manifest without format changes; inclusion + scoping (per-bind tars) cannot carry nested paths in `HDDSubdirs` (CTRL-001 forbids separators) + (§SQ5). +5. **The migration-default premise as stated does NOT hold** and needs a two-level default — + detailed in §SQ5. + +--- + +## SQ1 — Capture map per tier (source + live) + +### Source-confirmed mechanics + +| Tier | What it captures | Code evidence | +|---|---|---| +| Recovery unit (primary) | `compose/` (docker-compose.yml, .felhom.yml, secret-stripped app.yaml) + enumerated `db-dumps/*.sql` + `volume-dumps/*.tar` + manifest.json. **No userdata, no HDD appdata.** | `backup/recovery_unit.go:91-106` (capture list is exactly the two config files + built app.yaml; dumps only *enumerated* from dirs the dump flow wrote) | +| Offsite (restic) | The recovery unit dir **only** — `discoverOffboxUnit` returns `backups/primary/`; one `restic backup ` per toggled app. | `backup/offbox.go:372` (`RunOffboxBackup`), `offbox.go:559-586` (`runOffboxInternal`: `src` = the unit path, tagged `felhom-offbox,`) | +| Tier-2 (cross-drive rsync) | Unit → `backups/secondary//recovery-unit` **plus** `appdata/` → `.../appdata`. **NOT `userdata/`** — the only two `rsyncMirror` call sites in the tree are these (`tier2.go:169,177`). | `backup/tier2.go:140-184`; `AppDataDir` = `/appdata/` (`appbackup/paths.go:68`) | +| Manual `.fab` (v0.130.0) | Config + DB dump + volume tars + `ExportDataMounts` = `${HDD_PATH}` binds ∪ the **userdata ROOT** as one tar. | `stacks/delete.go:575` (`ExportDataMounts`), `appexport/export.go:594-638` | +| PBS whole-guest | Guest rootfs + docker volumes; bind-mounted drive content is out of reach (platform docs; not re-derived here). | — | + +### F-S1 (NEW, HIGH for the redesign): tier-2 does not copy bulk userdata — the comment lies + +`tier2.go:17` says *"recovery unit + bulk userdata … the ONLY off-drive protection that browsable +HDD userdata can get"*, and `cmd/controller/main.go:1360-1361` says the tier-2 path *"copies the +felhom-data namespace wholesale"*. **The code copies neither**: `RunTier2` rsyncs exactly +`RecoveryUnitPath` and `AppDataDir(nsRoot, stack)` — `UserdataDir` appears nowhere in the backup +package (grep: only the two comment lines). Consequence for the model: **the shared media tree has +no automated off-drive copy at any tier today**; its only protection is a manual `.fab` export. +The task baseline "Tier-2 DOES copy bulk userdata" traced to these stale comments, not to behavior. + +Live proof (demo 9201): `backups/secondary/radarr/` and `backups/secondary/jellyfin/` on felhom-usb +contain **only** `recovery-unit/` — no media, while `nas-media/userdata/media/` holds 96 MB the +apps actively use. + +### F-S2 (NEW): tier-2 appdata leg silently skips apps whose appdata dir ≠ stack name + +`RunTier2` copies `appdata/`; the catalog binds `${HDD_PATH}/appdata/paperless/...` for +stack **paperless-ngx** (slug `paperless-ngx`). `AppDataDir(nsRoot,"paperless-ngx")` doesn't exist → +the `os.Stat` guard (`tier2.go:176`) skips the copy **silently** (status still records ok). +Live contrast on demo: `secondary/nextcloud/` contains `appdata/` (name matches); +`secondary/paperless-ngx/` (felhom-flash) contains **only** `recovery-unit/` while +`appdata/paperless/` exists on the same source drive. Affected today: **paperless-ngx** (its +document originals+archive!). immich/nextcloud/romm happen to match. Any future app with a nested +or differently-named appdata bind inherits the gap. The per-bind classification metadata fixes this +class of bug structurally (capture what the metadata declares, not what a name-convention guesses). + +### Live capture map (per catalog app; source-derived, spot-verified live where deployed) + +Legend: unit = config+dumps; ✔=captured, ✘=not captured. "app-named HDD appdata" = `${HDD_PATH}/appdata/` bind. + +| app | needs_hdd | offsite (restic) | tier-2 (cross-drive) | manual .fab (v0.130.0) | +|---|---|---|---|---| +| audiobookshelf | true | unit ✔ / userdata ✘ | unit ✔ / userdata ✘ | unit ✔ + userdata ROOT ✔ | +| calibre-web | true | unit ✔ / userdata ✘ | unit ✔ / userdata ✘ | unit ✔ + userdata ROOT ✔ | +| emby, jellyfin, plex | true | unit ✔ / media tree ✘ | unit ✔ / media tree ✘ | unit ✔ + userdata ROOT ✔ (RO shared tree over-captured) | +| immich | true | unit ✔ / **photo library ✘** | unit ✔ + `appdata/immich` ✔ | unit ✔ + `appdata/immich` ✔ + userdata ROOT ✔ | +| komga, navidrome, radarr, sonarr, romm | true | unit ✔ / userdata ✘ | unit ✔ / userdata ✘ (romm: + `appdata/romm` ✔) | unit ✔ + userdata ROOT ✔ (romm: + appdata bind ✔) | +| nextcloud | true | unit ✔ / **data dir ✘** | unit ✔ + `appdata/nextcloud` ✔ | unit ✔ + `appdata/nextcloud` ✔ | +| paperless-ngx | true | unit ✔ / **documents ✘** | unit ✔ / **`appdata/paperless` ✘ (F-S2)** | unit ✔ + appdata binds ✔ + userdata ROOT ✔ | +| 42 volume-only apps | false | unit ✔ (volume dumps ride in the unit) | unit ✔ (F6: they flow through tier-2 too) | unit ✔ (volume tars) | + +Live legs run: restic `snapshots` + `ls` (below); `find` on `backups/primary/radarr` (unit = +compose/ + volume-dumps/ + manifest.json exactly) and `backups/secondary/{radarr,jellyfin,nextcloud,paperless-ngx}`. + +### The offsite "no userdata" verdict — live evidence + +Offbox-toggled apps on demo: audiobookshelf, calibre-web, immich (settings.json `app_backup`). +`restic snapshots --no-lock`: 12 snapshots, **every** path is +`/mnt/felhom-drives/{felhom-usb,nas-media}/backups/primary/`. Full `restic ls` of the newest +immich snapshot (`81927042`, 2026-07-14 02:15): + +``` +.../backups/primary/immich/compose/{.felhom.yml,app.yaml,docker-compose.yml} +.../backups/primary/immich/db-dumps/immich-postgres.sql +.../backups/primary/immich/manifest.json +``` + +That is the complete file list. **The photo library (`appdata/immich`, 129 MB on the same drive) is +absent** → an offsite-only Immich restore today reproduces exactly the broken state measured in SQ3. +This is the "looks backed up but isn't" gap the redesign exists to close: the customer toggled +immich offsite ON, the status page says ok, and the photos are not there. + +--- + +## SQ2 — Per-bind classification inventory (all 55 catalog apps) + +Catalog ground truth (full read of all 55 templates): **13 apps** have `${USERDATA_PATH}`/ +`${HDD_PATH}` binds; **42 are named-volume-only** (classification moot — their state already rides +in the unit's volume dumps; the only host bind among them is homepage's RO docker.sock). No +`.felhom.yml` anywhere has a backup-classification field (grep across all 55: zero hits) — +greenfield confirmed. `needs_hdd` lives under `resources:`. + +Proposed class per bind (reason = coupling argument): + +| app | bind | class | reason | +|---|---|---|---| +| immich | `${HDD_PATH}/appdata/immich` → `/usr/src/app/upload` | **mandatory** | Managed upload library. DB-authoritative: asset rows + `mountChecks` state reference these paths; **SQ3 proves restore-without = broken, not empty** | +| immich | `${USERDATA_PATH}/media/photos` → `/external/photos:ro` | **optional** | External library: RO, registered post-deploy, re-scanned on demand → empty-not-broken if absent; but content is user-precious (not re-downloadable) → customer-selectable | +| paperless-ngx | `${HDD_PATH}/appdata/paperless/media` | **mandatory** | Document originals + archive; DB rows carry content hashes/paths → restore-without = broken documents (coupled by design; empirical leg deferred, see SQ3 note) | +| paperless-ngx | `${HDD_PATH}/appdata/paperless/export` | **excluded** | Transient export target, re-creatable | +| paperless-ngx | `${USERDATA_PATH}/import/paperless` | **excluded** | Consume inbox — transient by contract (paperless deletes after ingest) | +| nextcloud | `${HDD_PATH}/appdata/nextcloud` → `/var/www/html/data` | **mandatory** | THE user files + `oc_filecache`/shares reference them; not re-derivable content | +| calibre-web | `${USERDATA_PATH}/media/books` → `/calibre-library` | **mandatory** | The Calibre library dir contains `metadata.db` **inside** the tree — DB and book files are one referential unit; restore-without = broken library | +| calibre-web | `${USERDATA_PATH}/import/calibre` | **excluded** | Ingest inbox, transient | +| audiobookshelf | `${USERDATA_PATH}/media/audiobooks`, `media/podcasts` | **excluded** | ABS re-scans folders; DB keeps progress/metadata keyed by rescan-match; podcasts are re-downloadable by definition. (Judgment call on audiobooks — see shared-tree note) | +| komga | `${USERDATA_PATH}/media/comics` → `/data` | **optional** | Komga re-scans (empty-not-broken), but comic collections are typically hand-curated/irreplaceable → precious, customer-selectable | +| navidrome | `${USERDATA_PATH}/media/music:ro` | **excluded** | Classic re-rippable bulk; navidrome re-scans; shared tree | +| radarr | `${USERDATA_PATH}/media/movies` | **excluded** | Re-downloadable; radarr re-scans and re-acquires by design | +| radarr / sonarr | `${USERDATA_PATH}/downloads` (SAME host path, both apps) | **excluded** | Transient download queue; also cross-app shared — backing it up per-app would double-capture | +| sonarr | `${USERDATA_PATH}/media/tv` | **excluded** | As movies | +| emby / jellyfin / plex | `${USERDATA_PATH}/media` (whole tree) `:ro` | **excluded** | Pure READERS of the shared tree (watch-state/config in volumes). See consistency rule below | +| romm | `${USERDATA_PATH}/roms` → `/romm/library` | **optional** | Romm re-scans (empty-not-broken); curated ROM sets are often not re-acquirable → precious | +| romm | `${HDD_PATH}/appdata/romm/resources` | **excluded** | Scraped artwork/metadata cache, re-derivable from scrapers | + +**Shared-media-tree consistency (the model survives, with one rule).** emby/jellyfin/plex mount the +**whole** `media/` tree RO while seven apps own subtrees of it, and radarr+sonarr bind the identical +`downloads/` path. A naive per-app class assignment contradicts itself (books are *mandatory* to +calibre-web but sit inside the streamers' *excluded* tree; a per-app tar of `downloads` would +double-capture). The rule that dissolves both: **a class belongs to the BIND (the host path), and +the app that owns/writes the path is the classification authority; read-only binds are never +backup-relevant for the mounting app** (the streamers back up nothing from `media/`; calibre-web's +`media/books` = mandatory regardless of who else reads it). Overlaps of the same host path across +apps must be deduplicated at capture time by path, not by app — this already matches +`ExportDataMounts`' containment-dedup philosophy. + +**Flagged judgment calls** (not blockers, spec should confirm with the operator): audiobooks +(excluded vs optional), comics/roms (optional chosen — precious-decoupled). The mandatory class is +NOT in doubt anywhere — every mandatory above is DB-referenced content. + +--- + +## SQ3 — Immich DB-only restore (the empirical core) + +**Setup** (scratch, isolated): compose project `spikeimmich` in `/root/spike-immich/` on 9201's +rootfs — catalog-pinned images (`immich-server:v2.5.5`, `postgres:16-vectorchord0.3.0`, +`redis:7-alpine`), ML disabled, loopback port 127.0.0.1:12283 only, no traefik, library bind +`/root/spike-immich/library` → `/usr/src/app/upload`, postgres in `/root/spike-immich/pgdata`. +**No felhom namespace touched** (registered drives, real apps, controller state all untouched; +controller verified healthy before/after). + +**Healthy baseline:** admin created + logged in via API; 3 generated PNGs (640×480/800×600, +0.8–1.2 MB) uploaded via `POST /api/assets` → 3 assets `status:created`; after job settle, all 3 +thumbnails HTTP 200 (8–13 KB webp/jpeg) and originals HTTP 200. Library tree: `upload/` (3 +originals), `thumbs/` (preview+thumbnail per asset), plus Immich's own `.immich` marker files +(timestamp content) in all six subdirs. Sizes: library 3.6 MB, pgdata 302 MB. + +**The coupled-broken scenario** — stop server → `mv library library-hidden` → empty `library/` +→ start (DB fully intact = "restored state without userdata"): + +- **Level 1 — refuses to boot.** Server exits 1 in a crash-loop. Its StorageService logs + `Verifying system mount folder checks, current state: {"mountChecks":{"thumbs":true,"upload":true, + "backups":true,"library":true,"profile":true,"encoded-video":true}}` — i.e. **the DB itself + remembers the folders existed** — then + `Failed to read (/usr/src/app/upload/encoded-video/.immich): ENOENT` and + `microservices worker exited with code 1` (pointer to + docs.immich.app/administration/system-integrity#folder-checks). The coupling is not incidental: + Immich *actively encodes* it (markers written on first run, presence recorded in the DB, + verified on every boot). +- **Level 2 — skeleton-recreated restore is a lying gallery.** Recreating the six subdirs + copying + the `.immich` markers back (what a well-meaning restore script would do) lets the server boot. + Then: `GET /api/timeline/buckets` still reports `[{"timeBucket":"2026-07-01","count":3}]` — the + UI advertises all 3 photos — while **all 3 thumbnail fetches and all 3 original fetches return + HTTP 500** (`ENOENT` → `GlobalExceptionFilter Unknown error` per request in the server log). +- **Worse-than-empty verdict: CONFIRMED.** A clean install boots quietly with an empty timeline. The + DB-only restore either doesn't boot at all, or boots into a gallery where every remembered photo + errors — "looks recoverable but isn't", plus error spam. +- **No self-heal / re-import:** no repair API at `/api/repair` (404); job queues idle-clean; the + admin "repair" surface in Immich identifies orphans but cannot resurrect managed uploads whose + bytes are gone (nothing exists to re-import *from* — the library WAS the source). Reverse + direction (files present, DB absent) is a different scenario and irrelevant to offsite restore. +- **Control leg:** restoring the hidden library and restarting → all 3 thumbnails + originals + HTTP 200 again. The files alone were the broken link → **mandatory=coupled validated**. + +**Paperless second candidate: deferred** (time-boxed; Immich was decisive). Paperless's coupling is +design-documented (content-hash-indexed originals in `media/documents/originals` + archive +versions); recommend a 15-minute repeat of this protocol during the implementation spec's +acceptance run rather than blocking the model on it. + +**Cleanup:** done and verified — see §Cleanup. + +--- + +## SQ4 — Size split on live data + +Measured on demo (du; drive namespaces on felhom-flash/felhom-usb/nas-media): + +| quantity | example values | +|---|---| +| (a) app state = recovery unit (config + DB dump + volume dumps) | navidrome 548 K · jellyfin 1.0 M · radarr 4.0 M · paperless-ngx 71 M (usb) · immich 44 M · calibre-web 248 M · nextcloud 933 M | +| (b) coupled/precious data (HDD appdata) | immich `appdata/immich` 129 M · nextcloud `appdata/nextcloud` 37 M · paperless `appdata/paperless` 56 K (demo-tiny) | +| (c) bulk shared tree | nas-media `userdata/media` 96 M (movies 57 M, music 37 M, books 12 M) + `downloads` 8.1 M; felhom-flash `userdata/navidrome` 2.9 G (campaign canary .bins — debris, see Observations) | + +Ratios even at demo scale: jellyfin state (1 M) vs the media tree it reads (96 M) ≈ **1:100**; +radarr state (4 M) vs movies (57 M) ≈ 1:14. Production libraries are 100 GB–10 TB against +state+coupled in the 0.1–5 GB band → the two-number estimate ("state+coupled" vs "+bulk") separates +by 2–4 orders of magnitude and is exactly the number the opt-in warning needs. Counter-example that +justifies *measuring* rather than assuming: scratch Immich's postgres (302 MB) dwarfed its 3-photo +library (3.6 MB) — state is not always the small number; and nextcloud's unit (933 MB, volume dumps +carrying `nextcloud_html`+DB) exceeds its appdata (37 MB) on demo. + +**Where the numbers come from (reuse seams):** tier-2 already sizes with `du -sb` +(`tier2.go dirSizeBytes`); the export estimate already does a **per-mount** du loop +(`appexport/estimate.go:49-57 duBytes`) + the v0.129.0 F-A container-view `volumeSizer` for named +volumes, with the `SizeUnknown`-never-renders-as-fits guard. A classification-driven two-number +estimate needs **zero new measurement plumbing** — split the existing per-mount loop by the bind's +class and sum into two buckets. + +--- + +## SQ5 — Metadata schema + round-trip feasibility + +### Round-trip mechanics as-built (code evidence) + +- The `.fab` manifest keys HDD tars by **basename**: `HDDSubdirs []string` + (`appexport/manifest.go:27`); export tars each mount as `.tar` with a fatal + basename-collision guard (`export.go:611-619`). +- **CTRL-001 hard-blocks nested paths in the v1 field:** `UnmarshalManifest` → + `validateManifestPaths` → `ValidateSegment` rejects any segment containing a path separator + (`validate.go:35-36`), and restore re-checks per subdir (`restore.go:604`). `hdd_subdirs: + ["userdata/media/tv"]` fails the parse *by security design*. +- Import maps a basename → a resolved `${HDD_PATH}` mount (note: `resolveHDDMounts` matches only + literal `${HDD_PATH}` references — `restore.go:670` — never `${USERDATA_PATH}`), else **fallback** + `/` (`restore.go:616-626`). The v0.130.0 userdata ROOT round-trips purely via + this fallback: basename `userdata` → `/userdata` = `USERDATA_PATH`. Pinned by + `stacks/export_mounts_test.go:55-59`. + +### Feasibility verdict: **exclusion scoping wins** + +- **Inclusion scoping** (tar each mandatory bind separately) requires nested-path knowledge on the + import side → a **manifest v2** field (e.g. `hdd_entries: [{rel_path, tar}]`), a new multi-segment + validator (per-segment CTRL-001 checks, no `..`/absolute), import-side placement logic, and a + compatibility story for old-controller-imports-new-bundle (v1 readers would silently ignore the + new field → hollow restores — the exact C6B-F1 failure class resurrected). Feasible but expensive + and risk-shaped. +- **Exclusion scoping** (tar the userdata ROOT, **skip excluded-class subtrees**; per-class + variants: offsite/state-only captures nothing of userdata unless a mandatory bind exists) keeps + the manifest byte-identical in shape: still one `userdata` entry in `HDDSubdirs`, still restored + by the existing basename fallback, **zero import changes, old controllers import new bundles + correctly** (they just extract a root tar that happens to contain less). The only new code is an + exclude list on the export tar call — and mandatory-bind capture for the *other* tiers (offsite + unit enrichment) reuses the same per-bind path resolution `ParseComposeUserdataMounts` already + provides (`stacks/delete.go:513`). The v0.130.0 root-capture comment's dedup philosophy carries + over unchanged. +- Same-drive HDD-appdata binds (`appdata/immich` etc.) already round-trip per-bind today (their + basenames are direct `${HDD_PATH}` children) — classification of those needs no tar-shape change + at all. + +### Proposed `.felhom.yml` addition + +```yaml +backup: + userdata: + - path: "media/photos" # relative to USERDATA_PATH + class: optional + hdd: # optional block for ${HDD_PATH} binds (appdata/*) + - path: "appdata/immich" # relative to HDD_PATH + class: mandatory +``` + +Semantics: `class ∈ {mandatory, optional, excluded}`; a bind **listed in the compose but absent +from the block defaults to `mandatory`** (C6B-F1 lesson: back up rather than silently drop); +read-only binds (`:ro`) are exempt from that default (reader rule, §SQ2) and default `excluded`. +The `${HDD_PATH}/appdata/*` binds default `mandatory` even without a block — they are app-managed +stores (immich/nextcloud/paperless all mandatory; only romm's scraper cache warrants an explicit +`excluded`). + +### The migration default — the task's premise needs one correction + +The premise *"missing field → mandatory default means today's behavior (offsite=config-only) does +NOT auto-start shipping media"* is **internally contradictory as stated**: if a missing field +defaults each bind to mandatory AND the new offsite policy ships mandatory binds, then an +unmigrated sonarr ships its whole `media/tv` offsite on day one — a surprise size/cost regression +(demo quota is 50 GB; a modest TV library is 10–100× that; the SLICE-4 soft-quota gate would refuse +runs and alert-spam). The reasoning **holds with a two-level default**: + +- **No `backup:` block at all → legacy behavior per tier** (offsite = unit only, .fab = full root + capture as v0.130.0). No unmigrated app changes its offsite footprint or cost. +- **Block present → unlisted writable binds default `mandatory`** (within an app that has opted + into classification, the safe direction is capture). + +Since all 13 hdd apps get explicit classes in the same catalog change that ships the feature, the +block-absent branch exists only as a safety net for stragglers/third-party templates — and it fails +toward *today's* behavior, not toward a quota blow-up. The 42 volume-only apps need no block. + +--- + +## SQ6 — `.fab` over-capture (derived + quantified; live export leg blocked) + +**Code-proven:** `ExportDataMounts` returns the userdata **ROOT** for any app with a +`${USERDATA_PATH}` bind (`delete.go:575-598`; pinned by `TestExportDataMounts_UserdataConvention`), +so a sonarr export tars every sibling's content in the shared namespace by construction — the +question is only *how much*. + +**Quantified on the live demo tree (nas-media, the drive radarr/jellyfin/navidrome/calibre-web +share):** a radarr `.fab` today captures the whole `userdata/` root ≈ **105 MB** (media 96 MB + +downloads 8.1 MB + import/documents/roms), of which radarr's own binds (`media/movies` 57 MB + +`downloads` 8.1 MB) are ~65 MB → **~1.6× over-capture even on a small demo tree**, carrying +navidrome's music, calibre-web's books (with its `metadata.db`!), and jellyfin's whole view. Under +the proposed classes, radarr's default `.fab` = state only (both its binds are excluded); the +over-capture factor vs today is unbounded (grows with everyone else's library). + +**Live byte-level leg (deploy sonarr+jellyfin one-drive, marker files, export, inspect tar): NOT +run.** Triggering a real export requires the dashboard session (CSRF-protected UI flow) and the +demo box's login is customer-owned (Viktor) — the same blocker as the open `.fab` full-circle +upload test. Protocol for when it runs: drop `SPIKE-marker-sonarr.txt` in `media/tv` and +`SPIKE-marker-jelly.txt` in `media/movies`, export sonarr, `tar tf` the bundle's +`data/hdd/userdata.tar` for both markers (expected today: both present). Suitable as the +before-picture acceptance test of the implementation spec. + +--- + +## Recommendations for the implementation spec + +1. **Schema:** the `backup:` block above (per-bind, path-relative, three classes; reader-rule for + `:ro`; two-level default). Ship explicit classes for all 13 hdd apps in the same catalog commit. +2. **Per-tier policy:** + - **Offsite (restic)** = unit + **mandatory** binds, not deselectable. Fixes the live-proven + immich gap. Size impact is bounded because mandatory ≈ DB-coupled stores, not media trees. + - **Tier-2** = unit + mandatory + **optional** (it's the browsable local copy; a 2nd HDD has the + room) — and fix F-S2 by driving the appdata leg from the declared binds instead of the + stack-name guess; fix the F-S1/stale comments either by implementing the claimed userdata copy + for mandatory+optional or by correcting the comments in the same change. + - **Manual `.fab`** = state + mandatory always; optional pre-selected; excluded opt-in behind the + two-number size warning ("state+coupled ≈ X MB / +bulk ≈ Y GB", reusing the estimate.go + per-mount du split). UI states plainly: bulk media's intended path is FileBrowser, coupled data + rides in the `.fab`. +3. **Round-trip scoping: exclusion** (root tar minus excluded subtrees) — no manifest change, no + import change, backward-compatible both directions (§SQ5). Do NOT introduce per-bind nested tars + under manifest v1. +4. **Capture the class at deploy/unit time** into the recovery-unit manifest (the unit already + carries `.felhom.yml` — the offsite runner can read the classes without new sync paths). +5. **Acceptance tests to carry over:** the SQ3 protocol (immich DB-only restore must become + restorable end-to-end from offsite alone), the SQ6 marker test (sonarr `.fab` must stop carrying + jellyfin's markers), a paperless repeat of SQ3, and an F-S2 red-proof (paperless-ngx tier-2 must + copy `appdata/paperless`). +6. **Migration:** catalog commit with explicit classes for the 13 → controller feature gates on + block presence → stragglers keep legacy behavior. No `MinAgent` coupling expected (all + controller-side), but the offsite unit-content change should bump the recovery-unit + `SchemaVersion` consideration in the spec. + +## Cleanup checklist (all verified live) + +- [x] Scratch Immich: `docker compose down -v` (containers + volumes + `spikeimmich_spikenet` + network removed; `docker ps -a --filter name=spike` → 0). +- [x] `/root/spike-immich/` (library, pgdata, photos, token), `/root/spike-immich-compose.yml`, + `/root/genpng.py` deleted. +- [x] Pulled images removed (`immich-server:v2.5.5`, `postgres:16-vectorchord0.3.0`); + `redis:7-alpine` left in place (cannot prove it wasn't pre-existing; no container uses it; + zero-risk residue, noted here for honesty). +- [x] No restic write/prune/forget — inspection used `--no-lock` reads only; repo status unchanged + (12 snapshots before and after). +- [x] No tier-2 target, drive namespace, controller setting, or deployed app touched; controller + 0.130.0 healthy at spike end (`Up`, healthy). +- [x] No secret values recorded anywhere in this doc (scratch instance used throwaway credentials, + destroyed with the instance). + +## Observations (out of scope, documented not acted on) + +- **Userdata debris on demo drives:** felhom-usb `userdata/` contains non-skeleton per-app dirs + (`actualbudget`, `bookstack`, `mealie`, `navidrome`, `paperless-ngx`, `uptime-kuma`, + `vaultwarden`) and felhom-flash `userdata/navidrome/` holds 2.9 GB of campaign canary `.bin` + files — leftovers from earlier campaigns; harmless but they'd inflate any root-capture `.fab` + and the future size estimates. Candidate for a demo-hygiene pass. +- **audiobookshelf binds `media/podcasts`, which is not in `UserdataSkeleton()`** — the deploy belt + pre-creates it from the compose scan, so no bug, but skeleton and catalog have drifted. +- **immich's current recovery unit carries no volume-dumps** (db-dump only) — consistent with the + dump flow not having run for its named volumes since the last teardown; worth a look during the + spec's offsite enrichment work. +- calibre-web's tier-2 status records an error against destination `/mnt/felhom-drives/teszt_enroll` + (a stale pinned target from an old enroll test) — the pin-invalidation fallback may deserve a + glance. +- The task baseline described the catalog as "~13 apps"; it is **55** (13 hdd + 42 volume-only) — + the classification workload is smaller than "all apps" implies but the migration commit touches + 13 files, not 55.