felhom.eu/documentation/audits/catalog-sweep-2026-10-24/README.md

# Catalog sweep 2026-10-24 — every catalog app on a fresh box, the way a family uses it

Unattended validation sweep (Saturday 12:30Z → Sunday ~00:00Z). Brief: the operator's „every catalog app, one by one, on
a fresh box" prompt. Venue: demo-hp (Tier 0). Method: endpoint-level — every step is the request the dashboard or the
app's own page sends (no browser click-through except the screenshot harness; no `docker exec` to make state).

## Safest readings taken (the brief allows no question)

1. **ISO 1.30.0, not 1.36.0.** The brief said „1.36.0 or the current one"; the current one on felhom.eu/letoltes was
   1.30.0 (sha256 checked against the published `.sha256`). See `_box/README.md`.
2. **12 GiB for the VM, not 16.** demo-hp had 14 GiB free beside 9201, 9202 and Tester 1; 12 is the brief's minimum.
   The guest itself is 7.7 GiB (the golden's size); the box's own memory admission decided what could run.
3. **„Remove keeping data → install again with use my kept data" is a drive-folder question.** By design (`07` §6.5)
   the question is asked only for an app that keeps a folder on a drive (`HDD_PATH`). For the 40+ apps that keep their
   data in Docker volumes, „keep my data" keeps the recovery unit; the reinstall starts empty, and the item comes back by
   the unit restore („Visszaállítás"). So step 5 + 6 ran as: keep-remove → reinstall → read (empty, expected) → unit
   restore → read. That is a unit restore on **every** app with a seed, not only every fifth.
4. **„Mentés most" is the local copy only** (`07` §8, by design); the second-drive and off-site copies run in the night.
   The sweep did not wait for a night, so „the right tiers ran" is checked as „the local unit holds the app".
5. **Categories:** the brief names „Apps & catalog" for bigger issues; a row whose harm is in backup/restore or storage is
   filed in that category instead (`register_shape_gate` has one category per row, and the owner reads it there).
6. **Sev:** the brief's scale (data at risk P1/P2, a broken app P2, a confusing step P3). R-1269, R-1271 are P2 by that rule (the data is not deleted in either; the household loses access to it).
7. **The sweep box reads the LIVE catalog** (main `5e196e1`, then `227a544` — rules only, no template moved). The drill
   catalog was used only for the three apps whose fix needed proving; the box was pointed back after (`_drill-repoint.txt`).
8. **Harness fixes live in the sweep's copy** (`tools/`), not in the catalog's scripts; porting them is R-1277.

## The box (Part 0)

`_box/README.md`: VM **9360** `sweep1024-box` (6 cores, 12 GiB, disks 256 + 200 + 150 GB on `nvme-scratch`), ISO 1.30.0,
customer Tester4 (hub host `Tester4-aa9466`), guest 9201, domain `tester4.test` (no DNS: the browser LXC maps the
names). Bind 12:42:18Z, day-0 SUCCESS 12:45:40Z, claim, recovery code escrowed (stored out-of-band), drives `hdd_1`
(210 GB) and `hdd_2` (157 GB) plus the installer's internal place `belso`. Versions at the start: controller 0.342.0,
agent 0.164.0, hub 0.169.0. Browser: LXC **9361** `sweep1024-browser` (Chromium + Playwright), 60 dashboard screens
(15 pages × phone/computer × HU/EN) in `_box/screens-dashboard/`.

## Method per app (Part 1)

`tools/sweep.py <app>` — one app at a time, small first, heavy last (`tools/order.txt`); the next starts after the last
is removed with its data. Each `<app>/` holds `record.json` (the measured facts), `walk.log`, the page texts
(`page-{install,guide,overview}-{hu,en}.txt`) and `screens/` (phone + computer, HU + EN, install page and the app after
first login). Steps: install through `POST /api/stacks/<app>/deploy` (timed to the front door answering); a stranger
request without the family cookie (must be refused where the template gates); one item through the app's own API (the
catalog's fixture where one exists, a new one in `tools/sweep_fixtures.py` for seven more); „Mentés most"; remove
keeping data; reinstall (the kept-data question answered „use" where it is asked); read; unit restore; read; remove with
data; leftovers checked (containers, volumes, `app.yaml`, `appdata`, `kept`, the subdomain → 404). Memory: the peak of
each container's **anon** memory from its cgroup (`memory.stat`; the file cache is not counted — project rule) against
its compose limit; a `*` in the table means only `docker stats` was read for that app (it includes the cache). The anon reading joined mid-sweep, so 22 apps carry it and 40 carry the `docker stats` upper bound; of those 40 none passed 79 % of its limit (Tandoor 1009/1280) and none restarted. Above 80 % on anon: only OpenGist (F14).

## Results

<!-- TABLE START -->
| App | Verdict | Peak memory / limit (MiB) | Install → answering (s) | What we saw | Row / fix |
|---|---|---|---|---|---|
| actualbudget | **works** | 49* / 256 | 30 | the seeded item back after the restore |  |
| adventurelog | **works with a problem** | 381 / 896 | 207 | works; but the keep-backups → reinstall → restore path breaks it (DB password from the new install over the old data) — reproduced in its re-run | R-1269, R-1270 |
| audiobookshelf | **works** | 56* / 512 | 41 | the seeded item back after the restore | drive names: R-1275 |
| bentopdf | **works** | 3* / 384 | 30 | stateless (nothing to keep); installs, answers, removes clean |  |
| bookstack | **works** | 167* / 768 | 71 | the seeded item back after the restore |  |
| calcom | **works** | 807 / 1792 | 278 | the seeded item back after the restore; 2 start-up messages of the image itself |  |
| calibre-web | **works** | 211 / 768 | 76 | the book (its own upload) is listed again after the keep-data reinstall; the fixture's download check gets 401 (its sign-in for /download, not the app); the admin name is the box's generated one (re-run 2) | R-1277 |
| claper | **works** | 267* / 640 | 86 | the seeded item back after the restore |  |
| code-server | **works** | 81 / 1024 | 46 | first install refused by the registry's rate limit, told as „exit code 1”; the re-run installed and worked (no seed: the editor has no API) | R-1273 |
| crafty-controller | **works** | 79* / 2048 | 76 | the seeded item back after the restore |  |
| dawarich | **works** | 835* / 2688 | 136 | the seeded item back after the restore (every-5th: restore also before the reinstall) |  |
| docmost | **works** | 528* / 1152 | 126 | the seeded item back after the restore |  |
| emby | **works** | 95 / 2048 | 41 | installs, answers, removes clean; no seed (no fixture for its wizard) |  |
| ghost | **works** | 167* / 512 | 121 | the seeded item back after the restore |  |
| gitea | **works** | 112 / 512 | 25 | the seeded item back after the restore |  |
| glance | **works** | 7* / 128 | 15 | family gate holds a stranger; config-only (nothing to keep); the app's own headings are English |  |
| gokapi | **works** | 6 / 128 | 15 | installs, own login, removes clean; no seed (API key only in its UI) |  |
| grafana | **works** | 383* / 768 | 61 | the seeded item back after the restore |  |
| gramps-web | **works** | 347* / 1024 | 206 | installs, answers, removes clean; no seed fixture |  |
| grimmory | **works with a problem** | 594 / 1408 | 96 | first admin, library and an EPUB upload work; setup gate then family gate work as designed; the read-back through the family gate stayed inconclusive (the harness's gate logic, R-1277); remove-with-data leaves the empty parent folder | R-1276, R-1277 |
| grocy | **works** | 34* / 384 | 36 | the seeded item back after the restore |  |
| home-assistant | **works** | 311 / 1024 | 161 | the seeded item back after the restore |  |
| homebox | **works** | 205 / 256 | 20 | the seeded item back after the restore |  |
| homepage | **works with a problem** | 91* / 256 | 25 | no gate at all: a stranger sees the box's CPU/RAM/disk (and later the family's links); still the upstream sample page | R-1272 |
| immich | **works with a problem** | 1845 / 6016 | 121 | album (new fixture) in; „use my kept data” loaded the rehearsal's future-dated database and Immich never answered again (404, 20 min) | R-1271 |
| jellyfin | **works** | 110 / 2048 | 41 | first user (new fixture, its wizard API) back after the restore |  |
| karakeep | **works** | 649* / 2816 | 107 | the seeded item back after the restore |  |
| kimai | **works** | 303* / 640 | 111 | own login; a Hungarian browser lands on /hu/ (curl without a language gets /en/) |  |
| komga | **works** | 473* / 768 | 46 | the seeded item back after the restore | drive names: R-1275 |
| lubelogger | **works** | 60* / 512 | 25 | the seeded item back after the restore |  |
| mealie | **works** | 322 / 1000 | 72 | the seeded item back after the restore (the fixture raced after_install on the first run) |  |
| metube | **works** | 90* / 768 | 61 | family gate holds a stranger; download queue entry back |  |
| n8n | **works with a problem** | 346 / 1536 | 151 | the owner back after the restore; on the live template it tries to send usage data out ([Rudder] lines, 6 in the live re-run) — the drill fix: the variable in the running container, 0 error lines | drill a601950 (morning list) |
| navidrome | **works** | 23* / 256 | 25 | the seeded item back after the restore | drive names: R-1275 |
| nextcloud | **works with a problem** | 298 / 1664 | 122 | user in; „use my kept data” loaded the rehearsal's future-dated database beside today's files — the user was not there | R-1271 |
| nzbget | **works** | 6* / 512 | 25 | the seeded item back after the restore; „no news server” until the family adds one (the guide says so) |  |
| onlyoffice | **works** | 464 / 2048 | 71 | document server (no household data to keep); installs, answers, removes clean (every-5th restore) |  |
| opengist | **works with a problem** | 125 / 128 | 15 | the gist owner back after the restore; but 128M is too small: one sign-up + sign-in hit the limit 443 times and swapped 98 MiB (memprobe-128M.txt); at 256M (drill) anon 216 MiB | drill 405077f (morning list) |
| outline | **works** | 327 / 1152 | 151 | the seeded item back after the restore |  |
| paperless-ngx | **works with a problem** | 669* / 1664 | 146 | account in; „use my kept data” loaded the rehearsal's future-dated database — the account was not there | R-1271, R-1276 |
| papra | **works** | 320 / 768 | 76 | the seeded item back after the restore (Papra 26.6 needs an Origin header — a browser sends it) |  |
| plant-it | **not installed** | – | – | lifecycle: abandoned — not offered for a new install (by design) |  |
| plex | **works** | 223 / 2048 | 30 | installs, answers, removes clean; no seed (a server is claimed with a plex.tv account) |  |
| privatebin | **works** | 31* / 128 | 15 | the seeded item back after the restore; D22 put-back PASS |  |
| prowlarr | **works** | 114* / 512 | 21 | the seeded item back after the restore |  |
| qbittorrent | **works** | 77* / 1024 | 15 | the seeded item back after the restore |  |
| radarr | **works** | 100* / 512 | 41 | the seeded item back after the restore |  |
| radicale | **works** | 33* / 128 | 25 | the seeded item back after the restore |  |
| rallly | **works** | 346 / 1024 | 71 | installs, answers, removes clean; no seed: Rallly 4.15 closed password sign-up (login by e-mail code); the guide's polls need no account | R-1277 |
| recipe-importer | **works** | 67* / 128 | 25 | stateless helper; installs, answers, removes clean (every-5th restore) |  |
| romm | **works with a problem** | 748* / 1536 | 116 | item back through „use my kept data” (its own copy); remove-with-data left the empty parent folder | R-1276 |
| sabnzbd | **works** | 69* / 1024 | 25 | the seeded item back after the restore |  |
| seerr | **works** | 166 / 384 | 92 | installs, answers (307 to its setup), removes clean; no seed (needs a media server) |  |
| sonarr | **works** | 80* / 512 | 41 | the seeded item back after the restore |  |
| sparkyfitness | **works** | 417* / 1792 | 91 | the seeded item back after the restore |  |
| tandoor | **works with a problem** | 1052* / 1536 | 177 | works; but the keep-backups → reinstall → restore path breaks it (DB password from the new install over the old data), and the restore still said success — reproduced on its own | R-1269, R-1270 |
| termix | **works** | 111* / 256 | 61 | installs, answers, removes clean (every-5th restore); no seed fixture |  |
| transmission | **works** | 6* / 512 | 20 | the seeded item back after the restore |  |
| uptime-kuma | **works** | 120* / 384 | 76 | the seeded item back after the restore |  |
| vaultwarden | **works** | 37 / 256 | 25 | installs in 25 s, its own login page, sign-up closed (404); the box admin's invite works (run-live-2), but 1.37.4 has no route for the fixture's invited sign-up, so no item was seeded | R-1277 |
| vikunja | **works** | 37* / 256 | 10 | the seeded item back after the restore; D22 put-back PASS |  |
| wanderer | **not testable on this box** | 140* / 1024 | 605 | not testable on this box: its web part calls its database at the PUBLIC name, which tester4.test cannot resolve (known, R-739) — unhealthy, 404 | R-739 |
| wger | **not installed** | – | – | lifecycle: hidden — not offered for a new install (by design) |  |
| wishlist | **works** | 200* / 512 | 71 | the seeded item back after the restore |  |
| zipline | **works** | 233* / 768 | 81 | the seeded item back after the restore |  |

works: 52, works with a problem: 10, not installed: 2, not testable on this box: 1 (of 65)
<!-- TABLE END -->

## Findings — each a row, a fix or NOT-A-FINDING

| # | What | Disposition |
|---|---|---|
| F10 | A reinstall's own capture rewrites the unit that „remove — keep backups" kept; restoring it pairs the NEW install's DB password with the OLD data — the app never starts, the item is gone (Tandoor ×2, AdventureLog) | R-1269 (P2) |
| F11 | The unit restore reports `ok: true` „…az alkalmazás újraindult" while the controller logged „health check failed … did not reach running state" | R-1270 (P2) |
| F12 | „Use my kept data" loaded the rehearsal box's future-dated off-site database (2026-10-28) for Paperless-ngx, Nextcloud and Immich — another drive's install, which `07` §6.5 says is refused; Immich then never answered | R-1271 (P2) |
| F1 | Homepage has no gate: a stranger sees the box's CPU/RAM/disk (Glance, same class, is gated) | R-1272 (operator decision) |
| F7 | An install refused by the registry's rate limit (`toomanyrequests`) tells the household only „exit code 1" | R-1273 |
| F8 | The restore refusal says „turn on remote backup, or connect a second drive" when both exist but have not copied yet | R-1274 |
| F2 | English pages show the drive names in Hungarian („Belső tárhely", „Külső tárhely (hdd_1)") | R-1275 |
| F9 | „Remove with data" leaves the empty parent folder `appdata/<app>` (Grimmory, RomM, Paperless-ngx) | R-1276 |
| — | The catalog's box fixtures drifted from the apps (OpenGist, Calibre-Web, Mealie, Papra, Rallly; seven apps had none) | R-1277 |
| F13 | n8n sends usage data (`[Rudder]` lines; no `N8N_DIAGNOSTICS_ENABLED=false`) | **fixed in the drill** `a601950`, proven below; morning list |
| F14 | OpenGist at 125.8 of 128 MiB anon after one sign-up + sign-in; at 128M it hits the limit 443 times and swaps 98 MiB | **fixed in the drill** `405077f` + `492a17f` (256M), proven below; morning list |
| — | 52 anonymous Docker volumes stayed after every app was removed with data (PrivateBin, OnlyOffice, RomM, Immich: the images' own `VOLUME` paths; 4–72 KB, no household data). The sweep's own leftover check filters by the app's label and missed them; found in the end-state read (`_box/anon-volumes.txt`) | known: note on R-1267 (widened to OnlyOffice and RomM) |
| — | Wanderer unhealthy, 404: its web part calls its database by the public name, which a box without DNS cannot resolve | known: note on R-739 |
| F3 | The install page's default place is the internal place `belso`, not the 210 GB drive | NOT-A-FINDING: by design — `16` §5 registers the internal place as the default drive |
| F4 | Guide labels in English (LubeLogger „Settings", NZBGet „Settings / News-Servers") | NOT-A-FINDING: they are the app's own on-screen labels, and the Hungarian catalog copy is frozen by the copy-i18n gate (measured on the drill) |
| F5 | Kimai opens in English | NOT-A-FINDING: a Hungarian browser lands on `/hu/`; only a request with no language gets `/en/` |
| F6 | AdventureLog backend 95 % of its limit | NOT-A-FINDING: that was `docker stats` with the cache; on anon 288.7/384 MiB (75 %), = the ladder's 73 % |
| — | Database error lines (SparkyFitness `database "sparky" does not exist`, Dawarich `database "dawarich" already exists`; SparkyFitness `Invalid password` = the harness's wrong-password control) | NOT-A-FINDING: the images' own first-start and restore-time checks; no restart, the item read back after the restore |
| — | Radicale redirects to an IP | NOT-A-FINDING: a curl artefact (no Host on the redirect probe); the browser lands right |
| — | NZBGet / Transmission screenshots show the browser's basic-auth prompt | NOT-A-FINDING: the app's own login, by design |
| — | code-server's first install failed (`toomanyrequests`) | the failed install was removed through the product (`code-server.failed-remove.txt`); the re-run worked; the message is R-1273 |

## F10 proof — the kept unit is rewritten by the reinstall (`_unitprobe/tandoor/`)

`tools/unitprobe.py tandoor`: install, one account, „Mentés most" (unit 19:45:14–20), remove keeping data and
backups, install again (a volume app: no kept-data question), wait for the controller's next capture, „Visszaállítás".

- 19:50:32 `Recovery unit captured for tandoor` — the unit's `app.yaml` checksum `0f366c97…` → `4438c1c5…`,
  `created_at` 19:45:26 → 19:50:32; the data files unchanged (`unit-1/2/3*.txt`).
- The restore: `ok: true`, „A(z) tandoor: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult."
- After: state `starting`, the account not there, `password authentication failed` ×42 (app) and ×22 (postgres).
- The comment at `felhom-controller/controller/internal/backup/recovery_unit.go:101-103` says the refresh never rewrites
  the definition away from its data; after a remove + reinstall it does.

Seen three times: the sweep's own Tandoor cycle (17:27:41 reinstall → 17:30:34 capture → 17:30:47 restore), this probe,
and AdventureLog's re-run (20:03:18 → 20:05:32 → restore). AdventureLog's first run worked only because no capture fell
between its reinstall and its restore.

## D22 put-back — PASS on two apps (`_putback/`)

`tools/putback.py <app>`: the largest volume tar of the app's unit truncated to half, then „Visszaállítás". PrivateBin
and Vikunja: the restore failed honestly („Nem változott semmi…"), the pre-restore data was put back, the app ran, and the
seeded item read back through the front door.

## Drill fixes, proven on the sweep box pointed at the drill (Part 2)

Drill catalog `app-catalog-drill` (never the live catalog): `742a354` (merge of live main), then:

- `a601950` **n8n** — `N8N_DIAGNOSTICS_ENABLED=false` in the compose (usage data stays home).
- `405077f` + `492a17f` **OpenGist** — memory 128M → 256M (compose + `.felhom.yml` `mem_limit`), the comment carries the
  measured need.

**How the box was pointed at the drill.** The drill repo on Gitea is private (R-992) and the box holds no catalog
credential, so the box reads a local mirror: a bare repository in the controller's data volume, loaded from a git bundle,
`repo_url: file:///opt/docker/felhom-controller/drill-catalog.git` (the method of
`audits/burndown-3-2026-10-17/tools/repoint.py`; loading the mirror is a `docker exec git fetch` into the controller —
catalog plumbing, not app state). Only `repo_url` changes; `tools/repoint.py live` puts the saved `controller.yaml` back
and checks it byte for byte (`RESTORED-IDENTICAL`, twice in `_drill-repoint.txt`). The box ended on the live catalog
`227a544`.

**What went wrong first, and is kept as evidence.** The first drill pass (chain 3, 21:18Z) set `repo_url` to the Gitea
URL in the belief that the repo was public. The box's clone failed (`fatal: could not read Username`), the controller kept
the templates it had, and n8n, OpenGist and Vaultwarden ran on the LIVE templates. The leftover `[Rudder]` lines in that
n8n run showed it. Those three runs are kept as `run-live-2/` (valid live re-runs). The second pass (chain 4) refused to
install anything until the box's catalog cache read the drill's commit and the changed lines were in the box's copy of
both templates: `drill main fd23571, box cache fd23571: DRILL-CONFIRMED`. (The drill also carries a six-line Plex
ladder entry from an earlier session's test; Plex was not installed in this pass.)

**n8n proof.** The running container carried `N8N_DIAGNOSTICS_ENABLED=false` (read at 21:37:58, that line only); error
lines 0 against 6 `[Rudder]` lines in the live re-run just before it (the control); the owner account read back after the
unit restore.

**OpenGist proof — and a wrong turn.** At 256M (drill) OpenGist took 215.8 MiB anon. I first read that as „it grows into
its limit" and reverted the fix (`c420053`); an automated security review flagged the revert as a lowered resource cap.
A measurement settled it (`tools/memprobe-guest.sh`, `opengist/memprobe-128M.txt`): on the live 128M template one
sign-up + sign-in hit `memory.max` 443 times and pushed 98 MiB into swap (anon 125 MiB, 0 `oom_kill`, 0 restarts). The
real need is ~220 MiB; the fix went back in (`405077f`). Where the box has no swap room left, the same sign-up would end in a kill (not measured here).

Drill gates: `catalog_gates.py --fast opengist n8n` from the drill clone — all OK except `security-scan`, which looks for
a `felhom.eu` clone beside the drill and found none; the shared scanner run by hand on the drill clone: 0 hits.

## Teardown

Three layers, 2026-10-25 ~00:05Z:

- **Machine (demo-hp):** VM **9360** `sweep1024-box` and LXC **9361** `sweep1024-browser` — both created by this session —
  stopped and destroyed with `--purge`; no disk of either is left on `nvme-scratch`. Untouched: Tester 1 (VM 341), 9201,
  9202, 9401, the thin pool, and the 1.30.0 ISO in `local:iso` (there since 2026-10-22). Before the destroy, the box's
  controller log, the agent journal, its end state and its anonymous-volume list were copied here (`_box/`).
- **Host (DooPlex):** the session's secret files (`/run/user/1000/sweep-sc`: the dashboard password copy, cookies, CSRF
  tokens) shredded. The drill catalog keeps its commits (`a601950`, `405077f`, `492a17f`, and the reverted turn
  `c420053`) for the morning list; nothing went to the live catalog.
- **Hub:** host **Tester4-aa9466** stays for the operator to delete (CC holds no delete). Until then it reads as a box
  that stopped reporting.
- **ep0:** listed, nothing removed. The host's PBS link to ep0 (`felhom-pbs`) was set up at day-0 (`pbsdr: converged`,
  14:44:48 CEST, `_box/agent-journal.log.gz`); no whole-box copy went to it (weekly cadence — only the local copy ran,
  14:51). The app copies the box sent off-site went to the Tester4 restic repository on the storage box (unlocked with
  the rehearsal's recovery code; a new code escrowed — stored out-of-band), not to ep0.

## Evidence map

- `_box/` — Part 0 (VM, ISO, bind, day-0, claim, recovery, drives, dashboard screens).
- `<app>/` — one per app; re-run apps keep the earlier run in `first-run/`, `run-2/` or `run-drill-pre/`.
- `<app>.console.txt`, `progress.log`, `batch.out`, `chain2.out`, `chain3.out` — the runner's own output.
- `_unitprobe/tandoor/` — F10. `_putback/{privatebin,vikunja}/` — D22.
- `tools/` — the harness (sweep copies of the catalog's `box_walk.py` and `upgrade_fixtures_box.py`, marked SWEEP COPY;
  `sweep.py`, `sweep_fixtures.py`, `unitprobe.py`, `putback.py`, `repoint.py`, `table.py`, the rehearsal's VM and browser
  helpers). Secrets were read by name through `sec` and never written here; the evidence was scanned for every known
  secret value and for token shapes (one app session token found and blanked: Actual Budget's), with a positive control.
- Screenshots were reduced with `pngquant` (lossy palette) to keep the repository small.