Files
felhom.eu/documentation/runbooks/nightly-rotation.md
T
admin 8d786f7940
gates / gates (push) Successful in 28s
Update night 2026-09-21: the full record, twelve rows, and the answers to five of the seven questions
The drill is complete. Teardown done in three layers plus Gitea; the live catalog's every `image:`
line is proven identical to before.

WHAT WAS MEASURED. 21 edges across 19 apps, on scratch guest 9202 through the product's own
guarded Update, against a PRIVATE DRILL CATALOG so the live catalog carried no test reference at
any point: 14 proven, 3 failed, 4 inconclusive. Each app seeded and read back through its OWN
front door, with a negative control on every readback. Ten of the fourteen printed a verbatim
migration line. Up from the three apps this project had ever measured.

THE RESULT THAT MATTERS. R-618, P1: three of the 53 templates name a health probe the app does not
answer, and because the guarded update WAITS on that same probe, a SUCCESSFUL update ends by
STOPPING a working app. tandoor was measured serving HTTP 200 on the new version at four samples
across five minutes, docker's own healthcheck green, and was then stopped and the household sent
to a restore they did not need. zipline and wger are the same defect, both confirmed live. The
gate that catches all three is static and cheap: both health checks already sit in the same file.

WHAT THE NIGHT ANSWERED that was open. The UNATTENDED HOLD (312.9 s, pressed once, never again) —
which needed a purpose-built image store, because the rule that makes automatic updates safe is
the same rule that refuses the obvious way to break one. MariaDB across a major through the real
button, all four observables, first time. PostgreSQL across a major, refusing exactly as predicted,
with the conversion costed at ~9 s of engine work. There is NO single-flight: five updates ran at
once and all ended honest. And the two EARLY power-cut phases nobody had cut in.

TWELVE NEW ROWS (R-615..R-626), register 303 -> 315, and eight existing rows updated with what was
measured — including two CORRECTIONS: R-606 records the pre-flight refusals as reaching an English
household in English and they do not, and R-446/R-458 are both narrower than their rows state.

Two instrument fixes were needed before anything could be trusted: the unattended caller turned
every success into a timeout (R-623), and one of my own reproductions was wrong and is kept
labelled with what it actually measured.

Interventions: zero. No controller, agent or hub code written. The hub was never touched beyond
the floor the operator asked for.

Gates: repo_gates.py --fast, all 15 OK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-21 22:34:24 +02:00

8.3 KiB

Nightly rotation — "be a customer for the night" on demo-hp

One app per night, as a throwaway deploy on demo-hp, through the product's own endpoints: install, real data through the app's front door (never a volume or database directly — R-156), ten minutes of household use, backup through the backups page's endpoint, restore and read the data back, the guarded Update if the catalog offers one, removal with "delete my data", and a clean drive. Every surprise is a register row before the next step. An app with no non-browser front door is recorded, not faked. Tick with the date; the next night takes the first unticked app.

The nine standing apps of demo-hp come first. They cannot be a throwaway on guest 9201 (same stack name), so they are walked on the scratch guest 9202 (R-481, built 2026-09-13: under the demo-hp customer, hub off, tunnel off, https://192.168.0.114 with the same Host names and password).

Standing on demo-hp — walked on the scratch guest 9202

  • bentopdf — 2026-09-13 (night 2, scratch guest 9202): install, front door 200, no data route (browser-side toolbox — recorded, not faked), backup now + Tier 2, remove-keep → Tier-2 restore back, same-version Update (backed up first: the restore's deploy time is newer than the copies, R-478 by design), remove-all clean. No new rows. audits/nightly-2026-09-13b-bentopdf/.
  • bookstack — 2026-09-21 (update night, scratch guest 9202): install, seeded through php artisan with its own negative control, backup, real upstream Update 26.05.2->26.05.5 (45.1 s), read back, removed. DATABASE HALF ONLY (R-460). audits/DRILL-update-night-2026-09-21.md.
  • calibre-web
  • docmost — 2026-09-21 (update night, scratch guest 9202): install, seeded a workspace+user through its own API, read back, backup, real upstream Update 0.95.0->0.96.0 (103.6 s) with its own migration line quoted, read back, removed with data. audits/DRILL-update-night-2026-09-21.md.
  • kimai
  • opengist — NOT A FULL WALK; attempted 2026-09-21 (update night, scratch guest 9202): install, seeded an account through its own sign-up form, read back, backup, real upstream Update 1.13->1.15 — the UPDATE SUCCEEDED (14.4 s, all four observables agree) but the readback ran before /login was serving, so the edge is inconclusive, not proven. audits/DRILL-update-night-2026-09-21.md.
  • paperless-ngx
  • privatebin — 2026-09-21 (update night, scratch guest 9202): install, seeded a paste through its own JSON API, read back, „Mentés most", real upstream Update 2.0.5->2.0.6 (15.4 s), read back again, removed with data. audits/DRILL-update-night-2026-09-21.md.
  • romm — 2026-09-21 (update night, scratch guest 9202): install, seeded through its own CSRF-protected user API, backup, real upstream Update 5.0.0->5.3.0 (60.6 s), logged in again, removed. DATABASE HALF ONLY. audits/DRILL-update-night-2026-09-21.md.

The rest of the catalog

  • actualbudget — 2026-09-13: no non-browser front door (the client talks a sync protocol, no REST); deploy / update / remove were exercised the same day for R-475 (audits/rulings-r472-r475-2026-09-13/). Recorded, not faked.
  • adventurelog — 2026-09-13: full walk (install, sign-up + trip data through the API, backup now + Tier 2, remove-with-data-kept → second-drive restore REFUSED (R-486) → own-copy restore byte-identical, same-version guarded Update, remove-with-backups left 484 MB behind (R-474)). Photo upload needs a browser (R-483). Rows: R-482..R-487. audits/nightly-2026-09-13-adventurelog/.
  • audiobookshelf — 2026-09-21 (update night, scratch guest 9202): install with its drive path, seeded a root account through /init, read back, backup, real upstream Update 2.35.1->2.36.1 (23.6 s), read back, removed. DATABASE HALF ONLY. audits/DRILL-update-night-2026-09-21.md.
  • calcom
  • claper
  • code-server
  • crafty-controller
  • emby
  • ghost
  • gitea — NOT A FULL WALK; attempted 2026-09-21 (update night, scratch guest 9202): attempted 2026-09-21 and inconclusive: a fresh Gitea sits in its web-installer state so gitea admin user create refuses. Needs an installer-form fixture (R-624). audits/DRILL-update-night-2026-09-21.md.
  • glance — NOT A FULL WALK; attempted 2026-09-21 (update night, scratch guest 9202): used 2026-09-21 as the UNATTENDED HOLD subject on drill images (R-618's method), not as a catalog walk — deliberately not ticked. audits/DRILL-update-night-2026-09-21.md.
  • gokapi
  • grafana — 2026-09-21 (update night, scratch guest 9202): install, seeded a folder through its own API as the deploy-generated admin, read back, backup, real upstream Update 13.1.0->13.2.2 (26.7 s), read back, removed with data. audits/DRILL-update-night-2026-09-21.md.
  • gramps-web
  • home-assistant — 2026-09-21 (update night, scratch guest 9202): install, seeded the owner through its own onboarding API, logged in through its own login flow, backup, real upstream Update 2026.7.2->2026.9.3 (103.6 s), read back, removed with data. audits/DRILL-update-night-2026-09-21.md.
  • homebox
  • homepage
  • immich
  • jellyfin
  • komga
  • mealie — 2026-09-21 (update night, scratch guest 9202): install, seeded a recipe through its own API as the first-run admin, read back, backup, real upstream Update v3.20.1->v3.27.0 (18.5 s), read back, removed with data. audits/DRILL-update-night-2026-09-21.md.
  • n8n — 2026-09-21 (update night, scratch guest 9202): install, seeded the owner through its own setup API, read back, backup, real upstream Update 2.31.3->2.40.5 (117.9 s), read back, removed with data. audits/DRILL-update-night-2026-09-21.md.
  • navidrome — 2026-09-21 (update night, scratch guest 9202): install with its drive path, seeded the first admin through its own API, read back, backup, real upstream Update 0.63.2->0.64.0 (11.3 s), read back, removed. The removal left a container behind that a later reboot resurrected — R-626. audits/DRILL-update-night-2026-09-21.md.
  • nextcloud — 2026-09-21 (update night, scratch guest 9202): install, seeded a user through occ, read back, backup, MariaDB engine major 11.6->12.3 through the real Update button (217.4 s) with all four SPIKE-r459 observables, read back, removed. audits/DRILL-update-night-2026-09-21.md.
  • onlyoffice
  • outline
  • papra — 2026-09-21 (update night, scratch guest 9202): install, seeded an account through its own sign-up API, signed in again, backup, real upstream Update 26.6.1->26.6.2 (60.5 s), removed with data. audits/DRILL-update-night-2026-09-21.md.
  • plant-it
  • plex
  • radarr
  • rallly
  • recipe-importer
  • seerr
  • sonarr
  • sparkyfitness
  • tandoor — NOT A FULL WALK; attempted 2026-09-21 (update night, scratch guest 9202): install, seeded a superuser through its own Django CLI, read back, backup, real upstream Update 2.6.13->2.6.15 — the update SUCCEEDED and the app served 200 on the new version, then the box stopped it because the template probes the wrong port (R-618, P1). Restored in 32.4 s. failed, and the failure is ours. audits/DRILL-update-night-2026-09-21.md.
  • termix
  • uptime-kuma
  • vaultwarden — NOT A FULL WALK; attempted 2026-09-21 (update night, scratch guest 9202): attempted 2026-09-21 and inconclusive BY DESIGN: the catalog closes self-registration on purpose (R-512), so no headless account route exists (R-624). audits/DRILL-update-night-2026-09-21.md.
  • vikunja — 2026-09-21 (update night, scratch guest 9202): install, registered a user and created a project through its own API, read back, backup, real upstream Update 2.3.0->2.6.0 (24.6 s), read back, removed with data. audits/DRILL-update-night-2026-09-21.md.
  • wanderer
  • wger — NOT A FULL WALK; attempted 2026-09-21 (update night, scratch guest 9202): deployed 2026-09-21 ONLY to measure its health probe: it listens on 8000 and the template probes 80, so the box reads it unhealthy while it serves (R-618). Removed. NOT a catalog walk. audits/DRILL-update-night-2026-09-21.md.
  • wishlist
  • zipline — NOT A FULL WALK; attempted 2026-09-21 (update night, scratch guest 9202): attempted 2026-09-21 and inconclusive: registration disabled by the template; and its probe path is wrong (R-618), so an update would have been held by the probe (R-624). audits/DRILL-update-night-2026-09-21.md.