Files
felhom.eu/documentation/audits/probe-fix-2026-09-22/romm-soak.json
T
admin 8efd2d00df
gates / gates (push) Successful in 26s
romm OOM storm on demo-hp: fixed, measured, closed (R-635)
Found because the operator heard the fans. romm 5.3.0 was promoted that morning; the update read
`done` and the app ran clean for two hours, then OOM-crash-looped for six - 4530 worker SIGKILLs,
~500% CPU, host load 5.2 while otherwise idle, and nothing alarmed.

Raising the limit to 768M was still a guess and fixed nothing (memory.peak hit exactly 768 MiB).
Measured instead: ~216 MiB per warm uvicorn worker, so the image's default of 4 workers needs
~882 MiB. /init reads WEB_SERVER_CONCURRENCY; set to 2.

Proven under load, not just at idle: 26,645 requests over 300 s, memory 416-614 MiB against 768,
trending down, zero SIGKILLs, OOMKilled false. Idle CPU 500% -> 1.64%.

The first soak measured nothing - it was pointed at the scratch-guest subdomain, every request
404'd at traefik in 9 ms, and the counter reported 14,026 successes. Positive and negative controls
are now asserted before any load is driven.

Carried into R-462: `proven` has meant "the update applied and the data survived", not "the new
version runs".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-22 18:04:49 +02:00

137 lines
2.3 KiB
JSON

{
"requests": 26645,
"codes": {
"200": 9687,
"401": 16958
},
"samples": [
{
"t": 2.4,
"stats": [
"102.83%|613.7MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 422
},
{
"t": 25.0,
"stats": [
"203.29%|531.9MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 2240
},
{
"t": 47.3,
"stats": [
"204.17%|422.1MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 4586
},
{
"t": 70.2,
"stats": [
"201.60%|517.4MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 6934
},
{
"t": 93.0,
"stats": [
"196.72%|477.7MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 8879
},
{
"t": 117.0,
"stats": [
"275.33%|574.4MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 10682
},
{
"t": 139.8,
"stats": [
"186.14%|594.1MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 12591
},
{
"t": 162.7,
"stats": [
"183.15%|567.8MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 14456
},
{
"t": 185.4,
"stats": [
"202.06%|525.9MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 16279
},
{
"t": 208.2,
"stats": [
"203.08%|442.4MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 18535
},
{
"t": 231.1,
"stats": [
"192.08%|546.9MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 20929
},
{
"t": 254.0,
"stats": [
"188.73%|504.5MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 22759
},
{
"t": 276.8,
"stats": [
"187.29%|484.4MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 24490
},
{
"t": 299.6,
"stats": [
"203.00%|416.4MiB / 768MiB",
"false|0",
"0"
],
"requests_so_far": 26639
}
],
"duration_s": 300,
"concurrency": 6
}