admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e78726314ea43aecaea37a2fc081edef922ea607
felhom.eu/documentation/audits/update-night-2026-09-21/apps/romm/app-logs-after.txt
T
admin 9c69b3ff07
gates / gates (push) Successful in 27s
Details
Update night: Phases 2-4 evidence — both engines, the unattended HOLD, and five new findings 
Evidence off the machine at the end of the phases that produced it (R-320). Teardown follows.

PHASE 2 — the two database engines, through the REAL Update button:
- MariaDB 11.6 -> 12.3 on nextcloud: PROVEN, and pressed through the button for the first time.
  All four SPIKE-r459 observables: the datadir's own record moved 11.6.2 -> 12.3.3; the engine
  itself says "already upgraded ... no need to run mariadb-upgrade again"; the entrypoint says
  "Major version upgrade detected ... Check required!" and then STARTED and FINISHED it (not the
  `skipped due to $MARIADB_AUTO_UPGRADE` line R-459 feared); and the engine took its own
  pre-upgrade backup, 631 905 B. The seeded Nextcloud account read back.
- PostgreSQL 16 -> 17 on docmost: FAILED exactly as R-463 predicted and nobody had measured.
  5.1 s to held; the pin named 17 while nothing ran; the restore brought it back in 29.1 s.
  The engine's REFUSAL LINE was destroyed by failAndHold before any probe could read it, so it
  was REPRODUCED INDEPENDENTLY with a control on every step (R-320).

PHASE 3 — the bad days. B1 produced THE UNATTENDED HOLD, which this project has never had: the
caller pressed once with nobody watching, the app held after 312.9 s, and passes 2 and 3 pressed
nothing. B2 put the pin back on a pull failure in 1.0 s. B3 refused `busy` six times. B4 showed
there is NO single-flight — 5 of 5 updates ran at once and all ended honest. B5 cut the power in
`backing-up` and the box recovered itself and said so. B7 refused under the 2 GB floor. B9 found
R-458's risk narrower than the row states.

PHASE 4 — every badge on the box is TRUE, and the held app answers all four of Q4's questions.

FINDINGS, five new and three corrections to existing rows. The one that matters: R-618 is P1 —
two templates name a health probe the app does not answer, and because the guarded update waits
on that same probe, a SUCCESSFUL update ends by STOPPING a working app. Measured: tandoor served
HTTP 200 on the new version at four samples across five minutes and was then stopped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-21 22:13:57 +02:00

13 KiB
Raw Blame History

romm | INFO: [RomM][init][2026-09-21 21:29:29] _____ __ __
romm | INFO: [RomM][init][2026-09-21 21:29:29] | __ \ | \/ |
romm | INFO: [RomM][init][2026-09-21 21:29:29] | |__) |___ _ __ ___ | \ / |
romm | INFO: [RomM][init][2026-09-21 21:29:29] | _ // _ \| '_ ` _ \| |\/| |
romm | INFO: [RomM][init][2026-09-21 21:29:29] | | \ \ (_) | | | | | | | | |
romm | INFO: [RomM][init][2026-09-21 21:29:29] |_| \_\___/|_| |_| |_|_| |_|
romm | INFO: [RomM][init][2026-09-21 21:29:29]
romm | INFO: [RomM][init][2026-09-21 21:29:29] The beautiful, powerful, self-hosted Rom manager and player
romm | INFO: [RomM][init][2026-09-21 21:29:29]
romm | INFO: [RomM][init][2026-09-21 21:29:29] Version: 5.3.0
romm | INFO: [RomM][init][2026-09-21 21:29:29]
romm | INFO: [RomM][init][2026-09-21 21:29:29] No OpenTelemetry environment variables found, disabling OpenTelemetry SDK
romm | INFO: [RomM][init][2026-09-21 21:29:29] REDIS_HOST is set, not starting internal valkey-server
romm | INFO: [RomM][init][2026-09-21 21:29:29] Running database migrations
romm | INFO: [RomM][init][2026-09-21 21:29:45] Database migrations succeeded
romm | INFO: [RomM][startup][2026-09-21 21:29:56] Running startup tasks
romm | INFO: [RomM][startup][2026-09-21 21:29:56] Cleared 2 job(s) left behind by the old scheduler
romm | INFO: [RomM][startup][2026-09-21 21:29:56] Initializing cache with fixtures data
romm | INFO: [RomM][startup][2026-09-21 21:29:56] Startup tasks completed
romm | INFO: [RomM][init][2026-09-21 21:29:57] Starting backend
romm | INFO: [RomM][init][2026-09-21 21:29:57] Starting RQ cron scheduler
romm | INFO: [RomM][init][2026-09-21 21:29:57] Starting RQ worker
romm | INFO: [RomM][init][2026-09-21 21:29:57] Starting RQ scan worker
romm | INFO: [RomM][init][2026-09-21 21:30:00] Starting nginx
romm | 2026/09/21 21:30:00 [notice] 131#131: js vm init njs: 0000710FFA2DDB00
romm | INFO: [RomM][init][2026-09-21 21:30:00] 🚀 RomM is now available at http://0.0.0.0:8080
romm | 21:30:00 Loading cron configuration from tasks.cron_config
romm | 21:30:00 Worker ae1444165f644e9294bf27d3ebcefdb8: started with PID 104, version 2.12.0
romm | 21:30:00 Worker ae1444165f644e9294bf27d3ebcefdb8: subscribing to channel rq:pubsub:ae1444165f644e9294bf27d3ebcefdb8
romm | 21:30:00 *** Listening on high, default, low...
romm | 21:30:00 Acquired scheduler lock for default
romm | 21:30:00 Acquired scheduler lock for low
romm | 21:30:00 Acquired scheduler lock for high
romm | 21:30:00 Scheduler for default, low, high started with PID 146
romm | 21:30:00 Worker e3223e339f794c82bec23fdda811216c: started with PID 107, version 2.12.0
romm | 21:30:00 Worker e3223e339f794c82bec23fdda811216c: subscribing to channel rq:pubsub:e3223e339f794c82bec23fdda811216c
romm | 21:30:00 *** Listening on scans...
romm | 21:30:00 Acquired scheduler lock for scans
romm | 21:30:00 Scheduler for scans started with PID 148
romm | INFO: [RomM][nginx][2026-09-21 21:30:05] 127.0.0.1 | - | GET / 200 | 5878 | Unknown Unknown | 0.000
romm | INFO: [RomM][nginx][2026-09-21 21:30:10] 172.18.0.7 | - | GET / 200 | 2206 | Unknown Unknown | 0.000
romm | INFO: [RomM][cron_config][2026-09-21 21:30:17] Scheduled 'build_recommendations' at '30 5 * * *'
romm-db | 2026-09-21 21:28:14+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.4.13+maria~ubu2404 started.
romm-db | 2026-09-21 21:28:14+02:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
romm-db | 2026-09-21 21:28:14+02:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
romm-db | 2026-09-21 21:28:14+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.4.13+maria~ubu2404 started.
romm-db | 2026-09-21 21:28:14+02:00 [Note] [Entrypoint]: MariaDB upgrade not required
romm | INFO: [RomM][cron_config][2026-09-21 21:30:17] Scheduled 'cleanup_zip_cache' at '0 4 * * *'
romm | INFO: [RomM][cron_config][2026-09-21 21:30:17] Scheduled 'cleanup_netplay' at '*/30 * * * *'
romm | INFO: [RomM][cron_config][2026-09-21 21:30:17] Scheduled 'cleanup_upload_tmp' at '0 * * * *'
romm | 21:30:17 Registered 'tasks.tasks.run_task_by_name' to run on low with cron schedule '30 5 * * *'
romm | 21:30:17 Registered 'tasks.tasks.run_task_by_name' to run on low with cron schedule '0 4 * * *'
romm-db | 2026-09-21 21:28:14 0 [Note] Starting MariaDB 11.4.13-MariaDB-ubu2404 source revision 170b1d70737be6f134448f51713cdc1ae215b420 server_uid +Q9/tt9ag22l9EQsPkibJVRvRoU= as process 1
romm | 21:30:17 Registered 'tasks.tasks.run_task_by_name' to run on low with cron schedule '*/30 * * * *'
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: Compressed tables use zlib 1.3
romm | 21:30:17 Registered 'tasks.tasks.run_task_by_name' to run on low with cron schedule '0 * * * *'
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: Number of transaction pools: 1
romm | 21:30:17 Successfully registered 4 cron jobs from 'tasks.cron_config'
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
romm | 21:30:17 CronScheduler 090f2c2a50a2:101:0f34e0: starting...
romm-db | 2026-09-21 21:28:14 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup).
romm | 21:30:17 CronScheduler 090f2c2a50a2:101:0f34e0: registering birth...
romm-db | create_uring failed: falling back to libaio
romm | INFO: [RomM][nginx][2026-09-21 21:30:25] 172.18.0.8 | 192.168.0.180 | GET /api/heartbeat 499 | 0 | Unknown Unknown | 15.016
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: Using Linux native AIO
romm | INFO: [RomM][server][2026-09-21 21:30:36] Started server process [119]
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: innodb_buffer_pool_size_max=8388608m, innodb_buffer_pool_size=128m
romm | INFO: [RomM][on][2026-09-21 21:30:36] Waiting for application startup.
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: Completed initialization of buffer pool
romm | INFO: [RomM][on][2026-09-21 21:30:37] Application startup complete.
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
romm | INFO: [RomM][server][2026-09-21 21:30:37] Started server process [121]
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: End of log at LSN=3058473
romm | INFO: [RomM][on][2026-09-21 21:30:37] Waiting for application startup.
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: Opened 3 undo tablespaces
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: log sequence number 3058473; transaction id 2446
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
romm | INFO: [RomM][on][2026-09-21 21:30:38] Application startup complete.
romm-db | 2026-09-21 21:28:14 0 [Note] Plugin 'FEEDBACK' is disabled.
romm | INFO: [RomM][logs][2026-09-21 21:30:38] Log stream forwarder started
romm | INFO: [RomM][server][2026-09-21 21:30:39] Started server process [118]
romm | INFO: [RomM][on][2026-09-21 21:30:39] Waiting for application startup.
romm | INFO: [RomM][on][2026-09-21 21:30:39] Application startup complete.
romm | INFO: [RomM][nginx][2026-09-21 21:30:35] 127.0.0.1 | - | GET / 200 | 5878 | Unknown Unknown | 0.000
romm | INFO: [RomM][nginx][2026-09-21 21:30:38] 172.18.0.8 | 192.168.0.180 | GET /api/heartbeat 200 | 543 | Unknown Unknown | 7.435
romm | INFO: [RomM][nginx][2026-09-21 21:30:38] 172.18.0.8 | 192.168.0.180 | GET /api/heartbeat 200 | 543 | Unknown Unknown | 0.013
romm | INFO: [RomM][nginx][2026-09-21 21:30:39] 172.18.0.8 | 192.168.0.180 | POST /api/login 401 | 43 | Unknown Unknown | 0.587
romm-db | 2026-09-21 21:28:14 0 [Note] Plugin 'wsrep-provider' is disabled.
romm-db | 2026-09-21 21:28:14 0 [Note] InnoDB: Buffer pool(s) load completed at 260921 21:28:14
romm-db | 2026-09-21 21:28:16 0 [Note] Server socket created on IP: '0.0.0.0', port: '3306'.
romm-db | 2026-09-21 21:28:16 0 [Note] Server socket created on IP: '::', port: '3306'.
romm-db | 2026-09-21 21:28:16 0 [Note] mariadbd: Event Scheduler: Loaded 0 events
romm-db | 2026-09-21 21:28:16 0 [Note] mariadbd: ready for connections.
romm-db | Version: '11.4.13-MariaDB-ubu2404' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
romm-db | 2026-09-21 21:29:45 22 [Warning] Aborted connection 22 to db: 'romm' user: 'romm' host: '172.20.0.4' (Got an error reading communication packets)
romm-redis | 1:C 21 Sep 2026 21:28:14.156 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
romm-redis | 1:C 21 Sep 2026 21:28:14.156 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
romm-redis | 1:C 21 Sep 2026 21:28:14.156 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started
romm-redis | 1:C 21 Sep 2026 21:28:14.156 * Configuration loaded
romm-redis | 1:M 21 Sep 2026 21:28:14.157 * Increased maximum number of open files to 10032 (it was originally set to 1024).
romm-redis | 1:M 21 Sep 2026 21:28:14.157 * monotonic clock: POSIX clock_gettime
romm-redis | 1:M 21 Sep 2026 21:28:14.158 * Running mode=standalone, port=6379.
romm-redis | 1:M 21 Sep 2026 21:28:14.159 * Server initialized
romm-redis | 1:M 21 Sep 2026 21:28:14.159 * Reading RDB base file on AOF loading...
romm-redis | 1:M 21 Sep 2026 21:28:14.159 * Loading RDB produced by version 7.4.11
romm-redis | 1:M 21 Sep 2026 21:28:14.159 * RDB age 68 seconds
romm-redis | 1:M 21 Sep 2026 21:28:14.159 * RDB memory usage when created 0.90 Mb
romm-redis | 1:M 21 Sep 2026 21:28:14.159 * RDB is base AOF
romm-redis | 1:M 21 Sep 2026 21:28:14.159 * Done loading RDB, keys loaded: 0, keys expired: 0.
romm-redis | 1:M 21 Sep 2026 21:28:14.159 * DB loaded from base file appendonly.aof.1.base.rdb: 0.000 seconds
romm-redis | 1:M 21 Sep 2026 21:28:14.223 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.064 seconds
romm-redis | 1:M 21 Sep 2026 21:28:14.223 * DB loaded from append only file: 0.064 seconds
romm-redis | 1:M 21 Sep 2026 21:28:14.223 * Opening AOF incr file appendonly.aof.1.incr.aof on server start
romm-redis | 1:M 21 Sep 2026 21:28:14.223 * Ready to accept connections tcp
romm-redis | 1:M 21 Sep 2026 21:29:15.065 * 10000 changes in 60 seconds. Saving...
romm-redis | 1:M 21 Sep 2026 21:29:15.066 * Background saving started by pid 52
romm-redis | 52:C 21 Sep 2026 21:29:15.192 * DB saved on disk
romm-redis | 52:C 21 Sep 2026 21:29:15.193 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB
romm-redis | 1:M 21 Sep 2026 21:29:15.268 * Background saving terminated with success
Reference in New Issue View Git Blame Copy Permalink