Security night 2026-11-02: part 3 live proofs (R-1347, R-1348, R-1339, R-1351, Traefik), part 5 walks, big-step walks on the drill (Paperless, n8n 3, SparkyFitness). Evidence only. Docs only.
gates / gates (push) Successful in 2m1s

This commit is contained in:
admin committed 2026-11-03 04:08:23 +01:00
1 parent 03a71b68e6
commit 17820abbe3
114 files changed
+14207

No files matched your search

@@ -0,0 +1,116 @@
[01:30:13] scratch drive folders cleared before FROM (R-656): none existed
[01:30:13] MV-n8n: deploying n8n at FROM {'n8n': 'n8nio/n8n:2.46.4'}
[01:32:09] FROM settled=True in 31.1s :: {"n8n": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}}
[01:32:09] fixture: the BOX walk's own (N8n), through upgrade_boxport
[01:32:11] n8n: /rest/owner/setup http=200
[01:32:12] n8n: login as the seeded owner http=200 ok=True
[01:32:12] C1 (seed reads back BEFORE): True
[01:32:12] MV-n8n: swapping to TO {'n8n': 'n8nio/n8n:3.0.1'}
[01:33:26] TO up -d rc=0
[01:33:57] TO settled=True in 30.9s :: {"n8n": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}}
[01:33:57] migration lines observed: 6
[01:33:57] n8n: login as the seeded owner http=200 ok=True
[01:33:57] RESULT (seed reads back AFTER): True
[01:33:58] memory watch: 600s, 4 callers on 1 path(s) at 172.18.0.2:5678
[01:34:13] + 15s n8n=459M/1536M peak=744M kills=0 rs=0 reqs=300
[01:34:28] + 30s n8n=465M/1536M peak=744M kills=0 rs=0 reqs=600
[01:34:43] + 45s n8n=472M/1536M peak=744M kills=0 rs=0 reqs=896
[01:34:58] + 60s n8n=475M/1536M peak=744M kills=0 rs=0 reqs=1196
[01:35:13] + 76s n8n=486M/1536M peak=744M kills=0 rs=0 reqs=1496
[01:35:28] + 91s n8n=486M/1536M peak=744M kills=0 rs=0 reqs=1796
[01:35:43] + 106s n8n=488M/1536M peak=744M kills=0 rs=0 reqs=2092
[01:35:59] + 121s n8n=490M/1536M peak=744M kills=0 rs=0 reqs=2392
[01:36:14] + 136s n8n=492M/1536M peak=744M kills=0 rs=0 reqs=2692
[01:36:29] + 151s n8n=494M/1536M peak=744M kills=0 rs=0 reqs=2992
[01:36:44] + 166s n8n=499M/1536M peak=744M kills=0 rs=0 reqs=3292
[01:36:59] + 182s n8n=493M/1536M peak=744M kills=0 rs=0 reqs=3588
[01:37:14] + 197s n8n=478M/1536M peak=744M kills=0 rs=0 reqs=3888
[01:37:29] + 212s n8n=474M/1536M peak=744M kills=0 rs=0 reqs=4188
[01:37:44] + 227s n8n=475M/1536M peak=744M kills=0 rs=0 reqs=4488
[01:38:00] + 242s n8n=477M/1536M peak=744M kills=0 rs=0 reqs=4788
[01:38:15] + 257s n8n=482M/1536M peak=744M kills=0 rs=0 reqs=5084
[01:38:30] + 272s n8n=476M/1536M peak=744M kills=0 rs=0 reqs=5384
[01:38:45] + 287s n8n=476M/1536M peak=744M kills=0 rs=0 reqs=5684
[01:39:00] + 302s n8n=486M/1536M peak=744M kills=0 rs=0 reqs=5984
[01:39:15] + 318s n8n=483M/1536M peak=744M kills=0 rs=0 reqs=6284
[01:39:30] + 333s n8n=470M/1536M peak=744M kills=0 rs=0 reqs=6580
[01:39:45] + 348s n8n=473M/1536M peak=744M kills=0 rs=0 reqs=6880
[01:40:01] + 363s n8n=480M/1536M peak=744M kills=0 rs=0 reqs=7180
[01:40:16] + 378s n8n=481M/1536M peak=744M kills=0 rs=0 reqs=7480
[01:40:31] + 393s n8n=482M/1536M peak=744M kills=0 rs=0 reqs=7780
[01:40:46] + 408s n8n=482M/1536M peak=744M kills=0 rs=0 reqs=8076
[01:41:01] + 424s n8n=484M/1536M peak=744M kills=0 rs=0 reqs=8376
[01:41:16] + 439s n8n=488M/1536M peak=744M kills=0 rs=0 reqs=8676
[01:41:31] + 454s n8n=491M/1536M peak=744M kills=0 rs=0 reqs=8976
[01:41:46] + 469s n8n=493M/1536M peak=744M kills=0 rs=0 reqs=9276
[01:42:02] + 484s n8n=490M/1536M peak=744M kills=0 rs=0 reqs=9572
[01:42:17] + 499s n8n=479M/1536M peak=744M kills=0 rs=0 reqs=9872
[01:42:32] + 514s n8n=479M/1536M peak=744M kills=0 rs=0 reqs=10172
[01:42:47] + 529s n8n=481M/1536M peak=744M kills=0 rs=0 reqs=10472
[01:43:02] + 545s n8n=471M/1536M peak=744M kills=0 rs=0 reqs=10772
[01:43:17] + 560s n8n=473M/1536M peak=744M kills=0 rs=0 reqs=11068
[01:43:32] + 575s n8n=472M/1536M peak=744M kills=0 rs=0 reqs=11368
[01:43:47] + 590s n8n=478M/1536M peak=744M kills=0 rs=0 reqs=11668
[01:44:03] + 605s n8n=477M/1536M peak=744M kills=0 rs=0 reqs=11968
[01:44:03] memory watch: killed=False tight=[] requests=11968 codes={'200': 11968}
[01:44:03] MV-n8n: ABORT — putting the FROM images back
[01:44:35] n8n: login as the seeded owner http=200 ok=True
[01:44:35] ABORT: app came back in 30.9s; data present=True
{
"harness_version": 5,
"edge": "MV-n8n",
"app": "n8n",
"note": "night 2026-09-23 within-a-major move: n8n=n8nio/n8n:3.0.1",
"from": {
"n8n": "n8nio/n8n:2.46.4"
},
"to": {
"n8n": "n8nio/n8n:3.0.1"
},
"verdict": "proven",
"seed_read_before": true,
"seed_read_after": true,
"healthy_after": true,
"migration_observed": "\u001b[2Kn8n | Migrations in progress, please do NOT stop the process.",
"abort": "starts-and-serves",
"abort_detail": null,
"engine_state_after": null,
"memory": {
"soak_s": 605.3,
"requested_s": 600,
"requests": 11968,
"codes": {
"200": 11968
},
"first_kill": null,
"containers": {
"n8n": {
"limit": 1610612736,
"peak": 780308480,
"peak_pct": 0.484,
"anon_peak_sampled": 383860736,
"anon_peak_pct": 0.238,
"anon_live_peak_sampled": 383860736,
"anon_live_peak_pct": 0.238,
"swap_peak": 0,
"oom_kills": 0,
"restarts": 0,
"oomkilled_flag": false,
"measured": true
}
},
"unmeasured": [],
"venue_swap_bytes": 0,
"load": "reached"
},
"marks": [],
"bench_overrides": null,
"duration_s": 30.9,
"measured_at": "2026-11-03T01:44:35Z",
"evidence": "evidence/MV-n8n",
"scratch_cleared": [],
"files_changed": [],
"files_changed_detail": [],
"files_ignored": [],
"total_s": 862.1
}
@@ -0,0 +1,8 @@
{
"n8n": {
"status": "running",
"health": "healthy",
"restarts": 0,
"exit": 0
}
}
@@ -0,0 +1,27 @@
n8n | Initializing n8n process
n8n | n8n ready on ::, port 5678
n8n | n8n Task Broker ready on 127.0.0.1, port 5679
n8n | Failed to start Python task runner in internal mode. because Python 3 is missing from this system. Launching a Python runner in internal mode is intended only for debugging and is not recommended for production. Users are encouraged to deploy in external mode. See: https://docs.n8n.io/hosting/configuration/task-runners/#setting-up-external-mode
n8n |
n8n | There are deprecations related to your n8n setup. Please take the recommended actions to update your configuration:
n8n | - WEBHOOK_URL -> Use N8N_WEBHOOK_URL instead, which sets the base URL for both test and production webhooks.
n8n | - N8N_UNVERIFIED_PACKAGES_ENABLED -> The default for this variable will change to `false` in a future version. Set it to `true` explicitly to keep installing unverified community packages.
n8n | - N8N_RUNNERS_MODE -> Internal task runner mode is deprecated and will be removed in a future version. For isolation and scaling, run the task runner launcher as a separate process, set this variable to `external` and share `N8N_RUNNERS_AUTH_TOKEN` with the launcher. See https://docs.n8n.io/deploy/host-n8n/configure-n8n/set-up-task-runners
n8n | - N8N_RUNNERS_TASK_TIMEOUT -> The default for this variable will be reduced from 300 (5 minutes) to 60 (1 minute) in a future version. Set it explicitly to keep your current task timeout.
n8n | - N8N_COMPRESSION_NODE_MAX_DECOMPRESSED_SIZE_BYTES -> The default for this variable will be reduced from 2 GiB to 256 MiB in a future version. Set it explicitly to keep your current limit.
n8n | - N8N_COMPRESSION_NODE_MAX_ZIP_ENTRIES -> The default for this variable will be reduced from 5000 to 1000 in a future version. Set it explicitly to keep your current limit.
n8n |
n8n | [license SDK] Skipping renewal on init: license cert is not initialized
n8n | Instance registered
n8n | Cluster aligned on version 2.46.4
n8n | Discovered 4 cluster checks
n8n | Registered runner "JS Task Runner" (EJQd1WO3JaCc960BPIBpf)
n8n | Recorded version change: 3.0.1 -> 2.46.4
n8n | Sandbox: enabled=false provider=n8n-sandbox (from env)
n8n | Version: 2.46.4
n8n | Building workflow dependency index...
n8n | Scheduler started
n8n | Finished building workflow dependency index. Processed 0 draft workflows, 0 published workflows.
n8n |
n8n | Editor is now accessible via:
n8n | https://auto.gate.invalid
@@ -0,0 +1 @@
null
@@ -0,0 +1,802 @@
[
{
"t": 15.1,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 482000896,
"peak": 780308480,
"anon": 343695360,
"anon_live": 343695360,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 300
},
{
"t": 30.3,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 487768064,
"peak": 780308480,
"anon": 349093888,
"anon_live": 349093888,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 600
},
{
"t": 45.4,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 495058944,
"peak": 780308480,
"anon": 356536320,
"anon_live": 356536320,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 896
},
{
"t": 60.5,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 498343936,
"peak": 780308480,
"anon": 359882752,
"anon_live": 359882752,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 1196
},
{
"t": 75.6,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 510476288,
"peak": 780308480,
"anon": 371408896,
"anon_live": 371408896,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 1496
},
{
"t": 90.7,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 510586880,
"peak": 780308480,
"anon": 371769344,
"anon_live": 371769344,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 1796
},
{
"t": 105.9,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 511909888,
"peak": 780308480,
"anon": 372563968,
"anon_live": 372563968,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 2092
},
{
"t": 121.0,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 514179072,
"peak": 780308480,
"anon": 375066624,
"anon_live": 375066624,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 2392
},
{
"t": 136.1,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 516026368,
"peak": 780308480,
"anon": 376692736,
"anon_live": 376692736,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 2692
},
{
"t": 151.2,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 518434816,
"peak": 780308480,
"anon": 379047936,
"anon_live": 379047936,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 2992
},
{
"t": 166.4,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 523386880,
"peak": 780308480,
"anon": 383860736,
"anon_live": 383860736,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 3292
},
{
"t": 181.5,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 517791744,
"peak": 780308480,
"anon": 378671104,
"anon_live": 378671104,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 3588
},
{
"t": 196.6,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 501731328,
"peak": 780308480,
"anon": 362405888,
"anon_live": 362405888,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 3888
},
{
"t": 211.7,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 497643520,
"peak": 780308480,
"anon": 359198720,
"anon_live": 359198720,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 4188
},
{
"t": 226.9,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 498225152,
"peak": 780308480,
"anon": 359505920,
"anon_live": 359505920,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 4488
},
{
"t": 242.0,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 500215808,
"peak": 780308480,
"anon": 360914944,
"anon_live": 360914944,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 4788
},
{
"t": 257.1,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 505929728,
"peak": 780308480,
"anon": 367034368,
"anon_live": 367034368,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 5084
},
{
"t": 272.3,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 499433472,
"peak": 780308480,
"anon": 360726528,
"anon_live": 360726528,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 5384
},
{
"t": 287.4,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 499589120,
"peak": 780308480,
"anon": 360894464,
"anon_live": 360894464,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 5684
},
{
"t": 302.5,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 509702144,
"peak": 780308480,
"anon": 370745344,
"anon_live": 370745344,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 5984
},
{
"t": 317.6,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 506609664,
"peak": 780308480,
"anon": 367714304,
"anon_live": 367714304,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 6284
},
{
"t": 332.8,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 493318144,
"peak": 780308480,
"anon": 355254272,
"anon_live": 355254272,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 6580
},
{
"t": 347.9,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 496361472,
"peak": 780308480,
"anon": 357564416,
"anon_live": 357564416,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 6880
},
{
"t": 363.0,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 503754752,
"peak": 780308480,
"anon": 365211648,
"anon_live": 365211648,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 7180
},
{
"t": 378.2,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 505040896,
"peak": 780308480,
"anon": 366235648,
"anon_live": 366235648,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 7480
},
{
"t": 393.3,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 505569280,
"peak": 780308480,
"anon": 366882816,
"anon_live": 366882816,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 7780
},
{
"t": 408.4,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 506310656,
"peak": 780308480,
"anon": 367198208,
"anon_live": 367198208,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 8076
},
{
"t": 423.6,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 507871232,
"peak": 780308480,
"anon": 369143808,
"anon_live": 369143808,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 8376
},
{
"t": 438.7,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 512376832,
"peak": 780308480,
"anon": 373366784,
"anon_live": 373366784,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 8676
},
{
"t": 453.8,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 515153920,
"peak": 780308480,
"anon": 375693312,
"anon_live": 375693312,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 8976
},
{
"t": 468.9,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 517971968,
"peak": 780308480,
"anon": 378286080,
"anon_live": 378286080,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 9276
},
{
"t": 484.1,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 513994752,
"peak": 780308480,
"anon": 375099392,
"anon_live": 375099392,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 9572
},
{
"t": 499.2,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 502820864,
"peak": 780308480,
"anon": 364294144,
"anon_live": 364294144,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 9872
},
{
"t": 514.3,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 502829056,
"peak": 780308480,
"anon": 364302336,
"anon_live": 364302336,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 10172
},
{
"t": 529.4,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 504676352,
"peak": 780308480,
"anon": 365989888,
"anon_live": 365989888,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 10472
},
{
"t": 544.6,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 494247936,
"peak": 780308480,
"anon": 355491840,
"anon_live": 355491840,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 10772
},
{
"t": 559.7,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 496680960,
"peak": 780308480,
"anon": 357998592,
"anon_live": 357998592,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 11068
},
{
"t": 574.8,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 495878144,
"peak": 780308480,
"anon": 357322752,
"anon_live": 357322752,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 11368
},
{
"t": 590.0,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 501374976,
"peak": 780308480,
"anon": 362762240,
"anon_live": 362762240,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 11668
},
{
"t": 605.1,
"containers": {
"n8n": {
"limit": 1610612736,
"current": 500539392,
"peak": 780308480,
"anon": 361816064,
"anon_live": 361816064,
"swap": 0,
"swap_peak": 0,
"oom_kill": 0,
"restarts": 0,
"oomkilled_flag": false,
"status": "running",
"cgroup": true
}
},
"requests": 11968
}
]
@@ -0,0 +1,6 @@
n8n | Migrations in progress, please do NOT stop the process.
n8n | Starting migration AddAgentVersionMetadata1793035272968
n8n | Finished migration AddAgentVersionMetadata1793035272968
n8n | Starting migration CreateAgentPublishHistory1793035273653
n8n | Finished migration CreateAgentPublishHistory1793035273653
n8n | Starting migration BackfillAgentDraftVersions1793036689320
@@ -0,0 +1,58 @@
[01:30:13] scratch drive folders cleared before FROM (R-656): none existed
[01:30:13] MV-n8n: deploying n8n at FROM {'n8n': 'n8nio/n8n:2.46.4'}
[01:32:09] FROM settled=True in 31.1s :: {"n8n": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}}
[01:32:09] fixture: the BOX walk's own (N8n), through upgrade_boxport
[01:32:11] n8n: /rest/owner/setup http=200
[01:32:12] n8n: login as the seeded owner http=200 ok=True
[01:32:12] C1 (seed reads back BEFORE): True
[01:32:12] MV-n8n: swapping to TO {'n8n': 'n8nio/n8n:3.0.1'}
[01:33:26] TO up -d rc=0
[01:33:57] TO settled=True in 30.9s :: {"n8n": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}}
[01:33:57] migration lines observed: 6
[01:33:57] n8n: login as the seeded owner http=200 ok=True
[01:33:57] RESULT (seed reads back AFTER): True
[01:33:58] memory watch: 600s, 4 callers on 1 path(s) at 172.18.0.2:5678
[01:34:13] + 15s n8n=459M/1536M peak=744M kills=0 rs=0 reqs=300
[01:34:28] + 30s n8n=465M/1536M peak=744M kills=0 rs=0 reqs=600
[01:34:43] + 45s n8n=472M/1536M peak=744M kills=0 rs=0 reqs=896
[01:34:58] + 60s n8n=475M/1536M peak=744M kills=0 rs=0 reqs=1196
[01:35:13] + 76s n8n=486M/1536M peak=744M kills=0 rs=0 reqs=1496
[01:35:28] + 91s n8n=486M/1536M peak=744M kills=0 rs=0 reqs=1796
[01:35:43] + 106s n8n=488M/1536M peak=744M kills=0 rs=0 reqs=2092
[01:35:59] + 121s n8n=490M/1536M peak=744M kills=0 rs=0 reqs=2392
[01:36:14] + 136s n8n=492M/1536M peak=744M kills=0 rs=0 reqs=2692
[01:36:29] + 151s n8n=494M/1536M peak=744M kills=0 rs=0 reqs=2992
[01:36:44] + 166s n8n=499M/1536M peak=744M kills=0 rs=0 reqs=3292
[01:36:59] + 182s n8n=493M/1536M peak=744M kills=0 rs=0 reqs=3588
[01:37:14] + 197s n8n=478M/1536M peak=744M kills=0 rs=0 reqs=3888
[01:37:29] + 212s n8n=474M/1536M peak=744M kills=0 rs=0 reqs=4188
[01:37:44] + 227s n8n=475M/1536M peak=744M kills=0 rs=0 reqs=4488
[01:38:00] + 242s n8n=477M/1536M peak=744M kills=0 rs=0 reqs=4788
[01:38:15] + 257s n8n=482M/1536M peak=744M kills=0 rs=0 reqs=5084
[01:38:30] + 272s n8n=476M/1536M peak=744M kills=0 rs=0 reqs=5384
[01:38:45] + 287s n8n=476M/1536M peak=744M kills=0 rs=0 reqs=5684
[01:39:00] + 302s n8n=486M/1536M peak=744M kills=0 rs=0 reqs=5984
[01:39:15] + 318s n8n=483M/1536M peak=744M kills=0 rs=0 reqs=6284
[01:39:30] + 333s n8n=470M/1536M peak=744M kills=0 rs=0 reqs=6580
[01:39:45] + 348s n8n=473M/1536M peak=744M kills=0 rs=0 reqs=6880
[01:40:01] + 363s n8n=480M/1536M peak=744M kills=0 rs=0 reqs=7180
[01:40:16] + 378s n8n=481M/1536M peak=744M kills=0 rs=0 reqs=7480
[01:40:31] + 393s n8n=482M/1536M peak=744M kills=0 rs=0 reqs=7780
[01:40:46] + 408s n8n=482M/1536M peak=744M kills=0 rs=0 reqs=8076
[01:41:01] + 424s n8n=484M/1536M peak=744M kills=0 rs=0 reqs=8376
[01:41:16] + 439s n8n=488M/1536M peak=744M kills=0 rs=0 reqs=8676
[01:41:31] + 454s n8n=491M/1536M peak=744M kills=0 rs=0 reqs=8976
[01:41:46] + 469s n8n=493M/1536M peak=744M kills=0 rs=0 reqs=9276
[01:42:02] + 484s n8n=490M/1536M peak=744M kills=0 rs=0 reqs=9572
[01:42:17] + 499s n8n=479M/1536M peak=744M kills=0 rs=0 reqs=9872
[01:42:32] + 514s n8n=479M/1536M peak=744M kills=0 rs=0 reqs=10172
[01:42:47] + 529s n8n=481M/1536M peak=744M kills=0 rs=0 reqs=10472
[01:43:02] + 545s n8n=471M/1536M peak=744M kills=0 rs=0 reqs=10772
[01:43:17] + 560s n8n=473M/1536M peak=744M kills=0 rs=0 reqs=11068
[01:43:32] + 575s n8n=472M/1536M peak=744M kills=0 rs=0 reqs=11368
[01:43:47] + 590s n8n=478M/1536M peak=744M kills=0 rs=0 reqs=11668
[01:44:03] + 605s n8n=477M/1536M peak=744M kills=0 rs=0 reqs=11968
[01:44:03] memory watch: killed=False tight=[] requests=11968 codes={'200': 11968}
[01:44:03] MV-n8n: ABORT — putting the FROM images back
[01:44:35] n8n: login as the seeded owner http=200 ok=True
[01:44:35] ABORT: app came back in 30.9s; data present=True
@@ -0,0 +1,34 @@
n8n | Initializing n8n process
n8n | n8n ready on ::, port 5678
n8n | Migrations in progress, please do NOT stop the process.
n8n | Starting migration AddAgentVersionMetadata1793035272968
n8n | Finished migration AddAgentVersionMetadata1793035272968
n8n | Starting migration CreateAgentPublishHistory1793035273653
n8n | Finished migration CreateAgentPublishHistory1793035273653
n8n | Starting migration BackfillAgentDraftVersions1793036689320
n8n | Finished migration BackfillAgentDraftVersions1793036689320
n8n | Starting migration CreateServiceAccountDetailsTable1793100089202
n8n | Finished migration CreateServiceAccountDetailsTable1793100089202
n8n | Starting migration AddIncludeAllProjectMembersToSelfHealingConfiguration1793106080688
n8n | Finished migration AddIncludeAllProjectMembersToSelfHealingConfiguration1793106080688
n8n | n8n Task Broker ready on 127.0.0.1, port 5679
n8n | Failed to start Python task runner in internal mode. because Python 3 is missing from this system. Launching a Python runner in internal mode is intended only for debugging and is not recommended for production. Users are encouraged to deploy in external mode. See: https://docs.n8n.io/hosting/configuration/task-runners/#setting-up-external-mode
n8n |
n8n | There are deprecations related to your n8n setup. Please take the recommended actions to update your configuration:
n8n | - WEBHOOK_URL -> Use N8N_WEBHOOK_URL instead, which sets the base URL for both test and production webhooks.
n8n | - N8N_RUNNERS_MODE -> Internal task runner mode is deprecated and will be removed in a future version. For isolation and scaling, run the task runner launcher as a separate process, set this variable to `external` and share `N8N_RUNNERS_AUTH_TOKEN` with the launcher. See https://docs.n8n.io/deploy/host-n8n/configure-n8n/set-up-task-runners
n8n |
n8n | [license SDK] Skipping renewal on init: license cert is not initialized
n8n | Instance registered
n8n | Cluster aligned on version 3.0.1
n8n | Discovered 4 cluster checks
n8n | Recorded version change: 2.46.4 -> 3.0.1
n8n | Registered runner "JS Task Runner" (oXz-7Ta6q3JLFzdHcFBoE)
n8n | Sandbox: enabled=false provider=n8n-sandbox (from env)
n8n | Version: 3.0.1
n8n | Building workflow dependency index...
n8n | Scheduler started
n8n | Finished building workflow dependency index. Processed 0 draft workflows, 0 published workflows.
n8n |
n8n | Editor is now accessible via:
n8n | https://auto.gate.invalid
@@ -0,0 +1,8 @@
{
"n8n": {
"status": "running",
"health": "healthy",
"restarts": 0,
"exit": 0
}
}
@@ -0,0 +1,58 @@
{
"harness_version": 5,
"edge": "MV-n8n",
"app": "n8n",
"note": "night 2026-09-23 within-a-major move: n8n=n8nio/n8n:3.0.1",
"from": {
"n8n": "n8nio/n8n:2.46.4"
},
"to": {
"n8n": "n8nio/n8n:3.0.1"
},
"verdict": "proven",
"seed_read_before": true,
"seed_read_after": true,
"healthy_after": true,
"migration_observed": "\u001b[2Kn8n | Migrations in progress, please do NOT stop the process.",
"abort": "starts-and-serves",
"abort_detail": null,
"engine_state_after": null,
"memory": {
"soak_s": 605.3,
"requested_s": 600,
"requests": 11968,
"codes": {
"200": 11968
},
"first_kill": null,
"containers": {
"n8n": {
"limit": 1610612736,
"peak": 780308480,
"peak_pct": 0.484,
"anon_peak_sampled": 383860736,
"anon_peak_pct": 0.238,
"anon_live_peak_sampled": 383860736,
"anon_live_peak_pct": 0.238,
"swap_peak": 0,
"oom_kills": 0,
"restarts": 0,
"oomkilled_flag": false,
"measured": true
}
},
"unmeasured": [],
"venue_swap_bytes": 0,
"load": "reached"
},
"marks": [],
"bench_overrides": null,
"duration_s": 30.9,
"measured_at": "2026-11-03T01:44:35Z",
"evidence": "evidence/MV-n8n",
"scratch_cleared": [],
"files_changed": [],
"files_changed_detail": [],
"files_ignored": [],
"total_s": 862.1
}
@@ -0,0 +1,157 @@
[01:44:39] scratch drive folders cleared before FROM (R-656): none existed
[01:44:39] MV-sparkyfitness: deploying sparkyfitness at FROM {'sparkyfitness-db': 'postgres:18-alpine', 'sparkyfitness-server': 'codewithcj/sparkyfitness_server:v0.17.3', 'sparkyfitness-frontend': 'codewithcj/sparkyfitness:v0.17.3'}
[01:46:17] FROM settled=True in 31.2s :: {"sparkyfitness": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "sparkyfitness-db": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "sparkyfitness-server": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}}
[01:46:17] fixture: the BOX walk's own (Sparkyfitness), through upgrade_boxport
[01:46:17] sparkyfitness: sign-up http=200
[01:46:18] sparkyfitness: check-in http=200
[01:46:18] sparkyfitness: seeded user drill1f7451 with a weight of 72.39 on 2026-09-01
[01:46:18] sparkyfitness: readback of the seeded weight http=200 equal=True
[01:46:18] C1 (seed reads back BEFORE): True
[01:46:18] MV-sparkyfitness: swapping to TO {'sparkyfitness-db': 'postgres:18-alpine', 'sparkyfitness-server': 'codewithcj/sparkyfitness_server:v1.11.0', 'sparkyfitness-frontend': 'codewithcj/sparkyfitness:v1.11.0'}
[01:47:25] TO up -d rc=0
[01:47:57] TO settled=True in 31.2s :: {"sparkyfitness": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "sparkyfitness-db": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "sparkyfitness-server": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}}
[01:47:57] engine state sparkyfitness-db: 18
[01:47:57] migration lines observed: 6
[01:47:57] sparkyfitness: readback of the seeded weight http=200 equal=True
[01:47:57] RESULT (seed reads back AFTER): True
[01:47:58] memory watch: 600s, 4 callers on 1 path(s) at 172.18.0.2:80
[01:48:13] + 15s sparkyfitness=8M/256M peak=9M kills=0 rs=0 sparkyfitness-db=157M/512M peak=194M kills=0 rs=0 sparkyfitness-server=490M/1024M peak=501M kills=0 rs=0 reqs=304
[01:48:28] + 30s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=490M/1024M peak=501M kills=0 rs=0 reqs=604
[01:48:44] + 46s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=490M/1024M peak=501M kills=0 rs=0 reqs=904
[01:48:59] + 61s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=1208
[01:49:14] + 76s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=1508
[01:49:29] + 91s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=1808
[01:49:44] + 106s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=2112
[01:49:59] + 121s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=2412
[01:50:15] + 137s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=154M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=2713
[01:50:30] + 152s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=3016
[01:50:45] + 167s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=3317
[01:51:00] + 182s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=3617
[01:51:15] + 197s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=3921
[01:51:31] + 212s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=4221
[01:51:46] + 228s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=4525
[01:52:01] + 243s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=4825
[01:52:16] + 258s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=5125
[01:52:31] + 273s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=5429
[01:52:46] + 288s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=5729
[01:53:02] + 304s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=6030
[01:53:17] + 319s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=6333
[01:53:32] + 334s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=6634
[01:53:47] + 349s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=6934
[01:54:02] + 364s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=7238
[01:54:17] + 379s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=7538
[01:54:33] + 395s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=7838
[01:54:48] + 410s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=8142
[01:55:03] + 425s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=154M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=8442
[01:55:18] + 440s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=154M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=8742
[01:55:33] + 455s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=9046
[01:55:49] + 470s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=9346
[01:56:04] + 486s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=9646
[01:56:19] + 501s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=9950
[01:56:34] + 516s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=10250
[01:56:49] + 531s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=10550
[01:57:04] + 546s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=10854
[01:57:20] + 562s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=11154
[01:57:35] + 577s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=11458
[01:57:50] + 592s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=11758
[01:58:05] + 607s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=12060
[01:58:06] memory watch: killed=False tight=[] requests=12060 codes={'200': 12060}
[01:58:06] MV-sparkyfitness: ABORT — putting the FROM images back
[02:01:49] ABORT: the app did NOT come back (rc=1, 181.5s)
{
"harness_version": 5,
"edge": "MV-sparkyfitness",
"app": "sparkyfitness",
"note": "night 2026-09-23 within-a-major move: sparkyfitness-server=codewithcj/sparkyfitness_server:v1.11.0, sparkyfitness-frontend=codewithcj/sparkyfitness:v1.11.0",
"from": {
"sparkyfitness-db": "postgres:18-alpine",
"sparkyfitness-server": "codewithcj/sparkyfitness_server:v0.17.3",
"sparkyfitness-frontend": "codewithcj/sparkyfitness:v0.17.3"
},
"to": {
"sparkyfitness-db": "postgres:18-alpine",
"sparkyfitness-server": "codewithcj/sparkyfitness_server:v1.11.0",
"sparkyfitness-frontend": "codewithcj/sparkyfitness:v1.11.0"
},
"verdict": "proven",
"seed_read_before": true,
"seed_read_after": true,
"healthy_after": true,
"migration_observed": "\u001b[2K\u001b[2Ksparkyfitness-server | [2026-11-03T01:46:56.073Z] [INFO] Ensured schema_migrations table exists.",
"abort": "refuses",
"abort_detail": "| detail: undefined,\n\u001b[2Ksparkyfitness-server | hint: undefined,\n\u001b[2Ksparkyfitness-server | position: undefined,\n\u001b[2Ksparkyfitness-server | internalPosition: undefined,\n\u001b[2Ksparkyfitness-server | internalQuery: undefined,\n\u001b[2Ksparkyfitness-server | where: 'SQL statement \"ALTER TABLE public.user_custom_symptom_locations ENABLE ROW LEVEL SECURITY;\"\\n' +\n\u001b[2Ksparkyfitness-server | 'PL/pgSQL function inline_code_block line 86 at EXECUTE',\n\u001b[2Ksparkyfitness-server | schema: undefined,\n\u001b[2Ksparkyfitness-server | table: undefined,\n\u001b[2Ksparkyfitness-server | column: undefined,\n\u001b[2Ksparkyfitness-server | dataType: undefined,\n\u001b[2Ksparkyfitness-server | constraint: undefined,\n\u001b[2Ksparkyfitness-server | file: 'namespace.c',\n\u001b[2Ksparkyfitness-server | line: '631',\n\u001b[2Ksparkyfitness-server | routine: 'RangeVarGetRelidExtended'\n\u001b[2Ksparkyfitness-server | }",
"engine_state_after": {
"sparkyfitness-db": {
"image": "postgres:18-alpine",
"probe": "datadir major version",
"answer": "18",
"probe_rc": 0
}
},
"memory": {
"soak_s": 607.5,
"requested_s": 600,
"requests": 12060,
"codes": {
"200": 12060
},
"first_kill": null,
"containers": {
"sparkyfitness": {
"limit": 268435456,
"peak": 10682368,
"peak_pct": 0.04,
"anon_peak_sampled": 5623808,
"anon_peak_pct": 0.021,
"anon_live_peak_sampled": 5623808,
"anon_live_peak_pct": 0.021,
"swap_peak": 0,
"oom_kills": 0,
"restarts": 0,
"oomkilled_flag": false,
"measured": true
},
"sparkyfitness-db": {
"limit": 536870912,
"peak": 203993088,
"peak_pct": 0.38,
"anon_peak_sampled": 10014720,
"anon_peak_pct": 0.019,
"anon_live_peak_sampled": 10014720,
"anon_live_peak_pct": 0.019,
"swap_peak": 0,
"oom_kills": 0,
"restarts": 0,
"oomkilled_flag": false,
"measured": true
},
"sparkyfitness-server": {
"limit": 1073741824,
"peak": 526200832,
"peak_pct": 0.49,
"anon_peak_sampled": 328564736,
"anon_peak_pct": 0.306,
"anon_live_peak_sampled": 328564736,
"anon_live_peak_pct": 0.306,
"swap_peak": 0,
"oom_kills": 0,
"restarts": 0,
"oomkilled_flag": false,
"measured": true
}
},
"unmeasured": [],
"venue_swap_bytes": 0,
"load": "reached"
},
"marks": [],
"bench_overrides": null,
"duration_s": 31.2,
"measured_at": "2026-11-03T02:01:49Z",
"evidence": "evidence/MV-sparkyfitness",
"scratch_cleared": [],
"files_changed": [],
"files_changed_detail": [],
"files_ignored": [],
"total_s": 1029.9
}
@@ -0,0 +1,80 @@
sparkyfitness-db | (exercise_entry_id IS NOT NULL AND EXISTS (SELECT 1 FROM public.exercise_entries ee WHERE ee.id = exercise_entry_id AND has_diary_access(ee.user_id))) OR
sparkyfitness-db | (exercise_preset_entry_id IS NOT NULL AND EXISTS (SELECT 1 FROM public.exercise_preset_entries epe WHERE epe.id = exercise_preset_entry_id AND has_diary_access(epe.user_id)))
sparkyfitness-db | );
sparkyfitness-db |
sparkyfitness-db | -- Shared View-Only (Tier 2)
sparkyfitness-db | SELECT create_checkin_policy('fasting_logs');
sparkyfitness-db | SELECT create_diary_policy('user_meal_visibilities');
sparkyfitness-db | SELECT create_checkin_policy('sleep_need_calculations');
sparkyfitness-db | SELECT create_checkin_policy('daily_sleep_need');
sparkyfitness-db | -- Day classification is a sleep/wellness (check-in) feature, used only by the
sparkyfitness-db | -- sleep-science service. It must use the check-in policy so check-in delegates
sparkyfitness-db | -- can manage it; the previous diary policy was a misclassification.
sparkyfitness-db | SELECT create_checkin_policy('day_classification_cache');
sparkyfitness-db | -- Onboarding data/status: delegates can read (to avoid repeated onboarding prompts when viewing
sparkyfitness-db | -- another user's diary) but only the account owner can submit or reset their own onboarding.
sparkyfitness-db | CREATE POLICY select_policy ON public.onboarding_data FOR SELECT TO PUBLIC USING (has_profile_read_access(user_id));
sparkyfitness-db | CREATE POLICY modify_policy ON public.onboarding_data FOR ALL TO PUBLIC
sparkyfitness-db | USING (authenticated_user_id() = user_id)
sparkyfitness-db | WITH CHECK (authenticated_user_id() = user_id);
sparkyfitness-db |
sparkyfitness-db | CREATE POLICY select_policy ON public.onboarding_status FOR SELECT TO PUBLIC USING (has_profile_read_access(user_id));
sparkyfitness-db | CREATE POLICY modify_policy ON public.onboarding_status FOR ALL TO PUBLIC
sparkyfitness-db | USING (authenticated_user_id() = user_id)
sparkyfitness-db | WITH CHECK (authenticated_user_id() = user_id);
sparkyfitness-db |
sparkyfitness-db | -- Medications & Symptoms (Tier 3 - Delegate Writable with medications permission)
sparkyfitness-db | SELECT create_medication_policy('medications');
sparkyfitness-db | SELECT create_medication_policy('medication_schedules');
sparkyfitness-db | SELECT create_medication_policy('medication_entries');
sparkyfitness-db | SELECT create_medication_policy('medication_pens');
sparkyfitness-db | SELECT create_medication_policy('injection_entries');
sparkyfitness-db | SELECT create_medication_policy('medication_titration_steps');
sparkyfitness-db | SELECT create_medication_policy('user_custom_symptoms');
sparkyfitness-db | SELECT create_medication_policy('symptom_entries');
sparkyfitness-db | SELECT create_medication_policy('user_custom_symptom_locations');
sparkyfitness-db |
sparkyfitness-db | -- Medications Display Preferences (Tier 2 - Owner-Only Write, Delegate Read)
sparkyfitness-db | CREATE POLICY select_policy ON public.user_medication_display_preferences FOR SELECT TO PUBLIC USING (has_medication_read_access(user_id));
sparkyfitness-db | CREATE POLICY modify_policy ON public.user_medication_display_preferences FOR ALL TO PUBLIC USING (authenticated_user_id() = user_id) WITH CHECK (authenticated_user_id() = user_id);
sparkyfitness-db |
sparkyfitness-server | code: '42P01',
sparkyfitness-server | detail: undefined,
sparkyfitness-server | hint: undefined,
sparkyfitness-server | position: undefined,
sparkyfitness-server | internalPosition: undefined,
sparkyfitness-server | internalQuery: undefined,
sparkyfitness-server | where: 'SQL statement "ALTER TABLE public.user_custom_symptom_locations ENABLE ROW LEVEL SECURITY;"\n' +
sparkyfitness-server | 'PL/pgSQL function inline_code_block line 86 at EXECUTE',
sparkyfitness-server | schema: undefined,
sparkyfitness-server | table: undefined,
sparkyfitness-server | column: undefined,
sparkyfitness-server | dataType: undefined,
sparkyfitness-server | constraint: undefined,
sparkyfitness-server | file: 'namespace.c',
sparkyfitness-server | line: '631',
sparkyfitness-server | routine: 'RangeVarGetRelidExtended'
sparkyfitness-server | }
sparkyfitness-server | Failed to start server: error: relation "public.user_custom_symptom_locations" does not exist
sparkyfitness-server | at /app/SparkyFitnessServer/node_modules/.pnpm/pg@8.20.0/node_modules/pg/lib/client.js:631:17
sparkyfitness-server | at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
sparkyfitness-server | at async applyRlsPolicies (/app/SparkyFitnessServer/utils/applyRlsPolicies.ts:15:5) {
sparkyfitness-server | length: 282,
sparkyfitness-server | severity: 'ERROR',
sparkyfitness-server | code: '42P01',
sparkyfitness-server | detail: undefined,
sparkyfitness-server | hint: undefined,
sparkyfitness-server | position: undefined,
sparkyfitness-server | internalPosition: undefined,
sparkyfitness-server | internalQuery: undefined,
sparkyfitness-server | where: 'SQL statement "ALTER TABLE public.user_custom_symptom_locations ENABLE ROW LEVEL SECURITY;"\n' +
sparkyfitness-server | 'PL/pgSQL function inline_code_block line 86 at EXECUTE',
sparkyfitness-server | schema: undefined,
sparkyfitness-server | table: undefined,
sparkyfitness-server | column: undefined,
sparkyfitness-server | dataType: undefined,
sparkyfitness-server | constraint: undefined,
sparkyfitness-server | file: 'namespace.c',
sparkyfitness-server | line: '631',
sparkyfitness-server | routine: 'RangeVarGetRelidExtended'
sparkyfitness-server | }
@@ -0,0 +1,20 @@
{
"sparkyfitness": {
"status": "created",
"health": null,
"restarts": 0,
"exit": 0
},
"sparkyfitness-db": {
"status": "running",
"health": "healthy",
"restarts": 0,
"exit": 0
},
"sparkyfitness-server": {
"status": "running",
"health": "starting",
"restarts": 0,
"exit": 0
}
}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,8 @@
{
"sparkyfitness-db": {
"image": "postgres:18-alpine",
"probe": "datadir major version",
"answer": "18",
"probe_rc": 0
}
}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,6 @@
sparkyfitness-server | [2026-11-03T01:46:56.073Z] [INFO] Ensured schema_migrations table exists.
sparkyfitness-server | [2026-11-03T01:46:56.079Z] [INFO] Applied migrations: [
sparkyfitness-server | '202508201215_add_garmin_integration_schema.sql',
sparkyfitness-server | '20251019013200_merged_schema_and_data.sql',
sparkyfitness-server | '20251103000000_add_onboarding_schema.sql',
sparkyfitness-server | '20251127194108_migrate_aggregated_food_entries_to_food_entry_meals.sql',
@@ -0,0 +1,60 @@
[01:44:39] scratch drive folders cleared before FROM (R-656): none existed
[01:44:39] MV-sparkyfitness: deploying sparkyfitness at FROM {'sparkyfitness-db': 'postgres:18-alpine', 'sparkyfitness-server': 'codewithcj/sparkyfitness_server:v0.17.3', 'sparkyfitness-frontend': 'codewithcj/sparkyfitness:v0.17.3'}
[01:46:17] FROM settled=True in 31.2s :: {"sparkyfitness": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "sparkyfitness-db": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "sparkyfitness-server": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}}
[01:46:17] fixture: the BOX walk's own (Sparkyfitness), through upgrade_boxport
[01:46:17] sparkyfitness: sign-up http=200
[01:46:18] sparkyfitness: check-in http=200
[01:46:18] sparkyfitness: seeded user drill1f7451 with a weight of 72.39 on 2026-09-01
[01:46:18] sparkyfitness: readback of the seeded weight http=200 equal=True
[01:46:18] C1 (seed reads back BEFORE): True
[01:46:18] MV-sparkyfitness: swapping to TO {'sparkyfitness-db': 'postgres:18-alpine', 'sparkyfitness-server': 'codewithcj/sparkyfitness_server:v1.11.0', 'sparkyfitness-frontend': 'codewithcj/sparkyfitness:v1.11.0'}
[01:47:25] TO up -d rc=0
[01:47:57] TO settled=True in 31.2s :: {"sparkyfitness": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "sparkyfitness-db": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "sparkyfitness-server": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}}
[01:47:57] engine state sparkyfitness-db: 18
[01:47:57] migration lines observed: 6
[01:47:57] sparkyfitness: readback of the seeded weight http=200 equal=True
[01:47:57] RESULT (seed reads back AFTER): True
[01:47:58] memory watch: 600s, 4 callers on 1 path(s) at 172.18.0.2:80
[01:48:13] + 15s sparkyfitness=8M/256M peak=9M kills=0 rs=0 sparkyfitness-db=157M/512M peak=194M kills=0 rs=0 sparkyfitness-server=490M/1024M peak=501M kills=0 rs=0 reqs=304
[01:48:28] + 30s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=490M/1024M peak=501M kills=0 rs=0 reqs=604
[01:48:44] + 46s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=490M/1024M peak=501M kills=0 rs=0 reqs=904
[01:48:59] + 61s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=1208
[01:49:14] + 76s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=1508
[01:49:29] + 91s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=1808
[01:49:44] + 106s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=2112
[01:49:59] + 121s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=2412
[01:50:15] + 137s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=154M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=2713
[01:50:30] + 152s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=378M/1024M peak=501M kills=0 rs=0 reqs=3016
[01:50:45] + 167s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=3317
[01:51:00] + 182s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=3617
[01:51:15] + 197s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=152M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=3921
[01:51:31] + 212s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=4221
[01:51:46] + 228s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=4525
[01:52:01] + 243s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=4825
[01:52:16] + 258s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=5125
[01:52:31] + 273s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=5429
[01:52:46] + 288s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=5729
[01:53:02] + 304s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=374M/1024M peak=501M kills=0 rs=0 reqs=6030
[01:53:17] + 319s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=6333
[01:53:32] + 334s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=6634
[01:53:47] + 349s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=6934
[01:54:02] + 364s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=7238
[01:54:17] + 379s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=7538
[01:54:33] + 395s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=7838
[01:54:48] + 410s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=8142
[01:55:03] + 425s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=154M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=8442
[01:55:18] + 440s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=154M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=8742
[01:55:33] + 455s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=9046
[01:55:49] + 470s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=9346
[01:56:04] + 486s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=375M/1024M peak=501M kills=0 rs=0 reqs=9646
[01:56:19] + 501s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=9950
[01:56:34] + 516s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=10250
[01:56:49] + 531s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=10550
[01:57:04] + 546s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=10854
[01:57:20] + 562s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=11154
[01:57:35] + 577s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=11458
[01:57:50] + 592s sparkyfitness=8M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=11758
[01:58:05] + 607s sparkyfitness=7M/256M peak=10M kills=0 rs=0 sparkyfitness-db=153M/512M peak=194M kills=0 rs=0 sparkyfitness-server=376M/1024M peak=501M kills=0 rs=0 reqs=12060
[01:58:06] memory watch: killed=False tight=[] requests=12060 codes={'200': 12060}
[01:58:06] MV-sparkyfitness: ABORT — putting the FROM images back
[02:01:49] ABORT: the app did NOT come back (rc=1, 181.5s)
@@ -0,0 +1,438 @@
sparkyfitness-server | ◇ injected env (0) from ../.env
sparkyfitness-server | [Secrets] Checking for secret files to load...
sparkyfitness-server | [Secrets] No secrets loaded from files.
sparkyfitness-server | [2026-11-03T01:46:55.909Z] [INFO] Outbound HTTP defaults: 30000ms request timeout, 5000ms per address-family connect attempt.
sparkyfitness | Starting SparkyFitness Frontend as root with environment variables:
sparkyfitness | SPARKY_FITNESS_SERVER_HOST=sparkyfitness-server
sparkyfitness | SPARKY_FITNESS_SERVER_PORT=3010
sparkyfitness | NGINX_RATE_LIMIT=5r/s
sparkyfitness | NGINX_RESOLVER=127.0.0.11
sparkyfitness | NGINX_LISTEN_PORT=80
sparkyfitness | NGINX_ACCESS_LOG=/var/log/nginx/access.log
sparkyfitness-server | [2026-11-03T01:46:55.915Z] [INFO] Environment variable pre-flight checks passed successfully.
sparkyfitness-server | [2026-11-03T01:46:56.049Z] [INFO] Waiting for the database initialization lock...
sparkyfitness-server | [2026-11-03T01:46:56.072Z] [INFO] Role "sparkyapp" already exists.
sparkyfitness-server | [2026-11-03T01:46:56.073Z] [INFO] Ensured schema_migrations table exists.
sparkyfitness-server | [2026-11-03T01:46:56.079Z] [INFO] Applied migrations: [
sparkyfitness-server | '20250703170640_InitialDB.sql',
sparkyfitness-server | '20250710062500_create_goal_presets_table.sql',
sparkyfitness-server | '20250711173824_meal_features.sql',
sparkyfitness-db | The files belonging to this database system will be owned by user "postgres".
sparkyfitness-db | This user must also own the server process.
sparkyfitness-server | '20250712163617_refactor_data_providers_and_add_exercise_source_id.sql',
sparkyfitness-server | '20250713125323_create_oidc_settings_and_update_users_table.sql',
sparkyfitness-server | '20250713135900_add_role_to_users_table.sql',
sparkyfitness-db |
sparkyfitness-db | The database cluster will be initialized with locale "en_US.utf8".
sparkyfitness-db | The default database encoding has accordingly been set to "UTF8".
sparkyfitness-db | The default text search configuration will be set to "english".
sparkyfitness-db |
sparkyfitness-db | Data page checksums are enabled.
sparkyfitness-db |
sparkyfitness-db | fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
sparkyfitness-db | creating subdirectories ... ok
sparkyfitness-db | selecting dynamic shared memory implementation ... posix
sparkyfitness-db | selecting default "max_connections" ... 100
sparkyfitness-db | selecting default "shared_buffers" ... 128MB
sparkyfitness-db | selecting default time zone ... Europe/Budapest
sparkyfitness-db | creating configuration files ... ok
sparkyfitness-db | running bootstrap script ... ok
sparkyfitness-db | sh: locale: not found
sparkyfitness-db | 2026-11-03 02:45:05.335 CET [36] WARNING: no usable system locales were found
sparkyfitness-db | performing post-bootstrap initialization ... ok
sparkyfitness-db | initdb: warning: enabling "trust" authentication for local connections
sparkyfitness-server | '20250713142300_add_oidc_config_fields.sql',
sparkyfitness | NGINX_ERROR_LOG=/var/log/nginx/error.log
sparkyfitness-server | '20250714000000_create_session_table.sql',
sparkyfitness | NGINX_DUMP_CONFIG=false
sparkyfitness-server | '20250714175824_add_api_key_tag_to_ai_service_settings.sql',
sparkyfitness-db | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
sparkyfitness-server | '20250717190700_add_cascading_deletes.sql',
sparkyfitness-db | syncing data to disk ... ok
sparkyfitness-db |
sparkyfitness-db |
sparkyfitness-db | Success. You can now start the database server using:
sparkyfitness-db |
sparkyfitness-db | pg_ctl -D /var/lib/postgresql/data/pgdata -l logfile start
sparkyfitness | SPARKY_FITNESS_FRONTEND_URL=https://sparky.gate.invalid
sparkyfitness | Generating nginx configuration from template...
sparkyfitness | Testing nginx configuration substitution...
sparkyfitness | Validating nginx configuration syntax...
sparkyfitness | nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
sparkyfitness | nginx: configuration file /etc/nginx/nginx.conf test is successful
sparkyfitness | Configuration validated successfully. Starting nginx...
sparkyfitness | Backend will be proxied to: sparkyfitness-server:3010
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: using the "epoll" event method
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: nginx/1.29.3
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: built by gcc 14.2.0 (Alpine 14.2.0)
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: OS: Linux 7.0.14-26-pve
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 524288:524288
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: start worker processes
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: start worker process 22
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: start worker process 23
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: start worker process 24
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: start worker process 25
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: start worker process 26
sparkyfitness | 2026/11/03 02:47:25 [notice] 1#1: start worker process 27
sparkyfitness | 127.0.0.1 - - [03/Nov/2026:02:47:55 +0100] "GET / HTTP/1.1" 200 2967 "-" "Wget"
sparkyfitness-server | '20250717233929_add_is_quick_food_to_foods.sql',
sparkyfitness-server | '20250718190226_add_food_display_limit_to_user_preferences.sql',
sparkyfitness-server | '20250719131500_add_meal_percentages.sql',
sparkyfitness-server | '20250719215400_add_login_management_to_oidc_settings.sql',
sparkyfitness-server | '20250720010000_create_user_nutrient_display_preferences.sql',
sparkyfitness-server | '20250720201800_rename_food_display_limit_to_item_display_limit.sql',
sparkyfitness-server | '20250720220000_add_water_tracking_features.sql',
sparkyfitness-server | '20250720233836_alter_water_goal_ml_to_numeric.sql',
sparkyfitness-server | '20250721000000_add_servings_per_container_to_water_containers.sql',
sparkyfitness-server | '20250721000000_alter_water_container_volume_to_numeric.sql',
sparkyfitness-server | '20250721014900_alter_water_ml_to_numeric.sql',
sparkyfitness-server | '20250819062400_add_mealie_provider_type.sql',
sparkyfitness-server | '20250819083500_add_base_url_to_external_data_providers.sql',
sparkyfitness-server | '202508201215_add_garmin_integration_schema.sql',
sparkyfitness-server | '20250821090000_update_external_data_providers_provider_type_check.sql',
sparkyfitness-server | '20250821091500_add_garth_token_dump_to_external_data_providers.sql',
sparkyfitness-server | '20250821102000_remove_redundant_token_fields_from_external_data_providers.sql',
sparkyfitness-server | '20250831063900_create_mood_entries_table.sql',
sparkyfitness-server | '20250927180257_alter_exercises_table.sql',
sparkyfitness-server | '20250927212000_add_free_exercise_db_to_provider_type_check.sql',
sparkyfitness-server | '20250930002100_add_sets_reps_weight_to_exercise_entries.sql',
sparkyfitness-server | '20251001232600_add_gender_to_users_table.sql',
sparkyfitness-server | '20251002182306_create_backup_settings_table.sql',
sparkyfitness-server | '20251002224500_add_password_reset_to_users_table.sql',
sparkyfitness-server | '20251006021609_add_glycemic_index_to_foods_and_food_variants.sql',
sparkyfitness-server | '20251006220500_create_admin_activity_logs_table.sql',
sparkyfitness-server | '20251008224256_add_workout_features.sql',
sparkyfitness-server | '20251012225400_add_primary_keys_to_workout_tables.sql',
sparkyfitness-server | '20251012225500_add_multiple_sets_to_presets.sql',
sparkyfitness-server | '20251012225900_add_multiple_sets_to_plans.sql',
sparkyfitness-server | '20251012230000_add_multiple_sets_to_exercise_entries.sql',
sparkyfitness-server | '20251015223200_convert_weights_to_kg_and_set_default.sql',
sparkyfitness-server | '202510170132_add_text_measurements.sql',
sparkyfitness-server | '20251017200900_add_multiple_oidc_providers.sql',
sparkyfitness-server | '20251017204200_create_global_settings.sql',
sparkyfitness-server | '20251018001151_add_bmr_and_bodyfat_preferences.sql',
sparkyfitness-server | '20251018223700_add_signing_alg_to_oidc_providers.sql',
sparkyfitness-server | '20251019013200_merged_schema_and_data.sql',
sparkyfitness-server | '20251019013300_merged_rls_policies.sql',
sparkyfitness-server | '20251019154515_update_rls_policies_for_shared_items.sql',
sparkyfitness-server | '20251019203500_fix_rls_permissions_for_public_foods.sql',
sparkyfitness-server | '20251019204000_correct_variant_rls_policy.sql',
sparkyfitness-server | '20251019212500_update_measurement_rls_policies.sql',
sparkyfitness-server | '20251019222100_add_audit_columns_to_measurements.sql',
sparkyfitness-server | '20251020001000_fix_family_access_insert_policy.sql',
sparkyfitness-server | '20251020022000_set_default_shared_with_public.sql',
sparkyfitness-server | '20251020022200_update_rls_policies.sql',
sparkyfitness-server | '20251020022500_fix_food_select_policy.sql',
sparkyfitness-server | '20251020030000_simplify_food_select_policy.sql',
sparkyfitness-server | '20251020031000_update_food_select_policy_with_sharing.sql',
sparkyfitness-server | '20251020032200_add_all_owner_centric_policies.sql',
sparkyfitness-server | '20251020033400_add_food_variants_rls_policy.sql',
sparkyfitness-server | '20251020033700_fix_food_variants_rls_policy.sql',
sparkyfitness-server | '20251020230839_fix_family_access_rls_policies.sql',
sparkyfitness-server | '20251021025423_restrict_family_access_select_policy.sql',
sparkyfitness-server | '20251021034700_add_public_sharing_to_external_providers.sql',
sparkyfitness-server | '20251022201000_fix_exercise_rls_policy.sql',
sparkyfitness-server | '20251022203000_snapshot_exercise_details_in_entries.sql',
sparkyfitness-server | '20251023120500_update_external_providers_select_policy_for_public.sql',
sparkyfitness-server | '20251023221000_update_meals_rls_policy.sql',
sparkyfitness-server | '20251023221500_update_meal_plan_template_assignments_rls_policy.sql',
sparkyfitness-server | '20251023221501_add_meal_id_to_food_entries.sql',
sparkyfitness-server | '20251024230330_add_withings_fields_to_external_data_providers.sql',
sparkyfitness-server | '20251025210900_add_source_to_custom_measurements.sql',
sparkyfitness-server | '20251026125900_add_distance_and_hr_to_exercise_entries_activity_details.sql',
sparkyfitness-server | '202510310132_add_language_to_user_preferences.sql',
sparkyfitness-server | '20251101151100_create_sleep_entries_table.sql',
sparkyfitness-server | '20251103000000_add_onboarding_schema.sql',
sparkyfitness-server | '20251112000000_add_display_name_to_custom_categories.sql',
sparkyfitness-server | '20251114205134_create_exercise_preset_entries_table.sql',
sparkyfitness-server | '20251115003147_add_source_to_exercise_preset_entries.sql',
sparkyfitness-server | '20251115003301_add_exercise_preset_entry_id_to_activity_details.sql',
sparkyfitness-server | '20251115004027_alter_exercise_entry_id_nullable_in_activity_details.sql',
sparkyfitness-server | '20251120011100_add_mfa_magic_link_columns.sql',
sparkyfitness-server | '20251121011500_add_email_mfa_columns_to_users.sql',
sparkyfitness-server | '202511220110_add_default_external_data_providers.sql',
sparkyfitness-server | '202511232110_delete_raw_stress_data_categories.sql',
sparkyfitness-server | '202511270109_add_calorie_goal_adjustment_mode_to_user_preferences.sql',
sparkyfitness-server | '20251127194107_create_food_entry_meals_and_link_food_entries.sql',
sparkyfitness-server | '20251127194108_migrate_aggregated_food_entries_to_food_entry_meals.sql',
sparkyfitness-server | '20251128030115_add_tandoor_provider_type.sql',
sparkyfitness-server | '20251205000000_add_energy_unit_to_user_preferences.sql',
sparkyfitness-server | '20251206000000_add_nutrient_calculation_algorithms.sql',
sparkyfitness-server | '20251206000001_add_serving_size_to_meals.sql',
sparkyfitness-server | '20251207000001_add_default_meal_quantity.sql',
sparkyfitness-server | '20251215005329_add_usda_to_provider_type_check.sql',
sparkyfitness-server | '20251221000000_create_fasting_logs.sql',
sparkyfitness-db |
sparkyfitness-db | waiting for server to start....2026-11-03 02:45:06.638 CET [42] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit
sparkyfitness-db | 2026-11-03 02:45:06.642 CET [42] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
sparkyfitness-db | 2026-11-03 02:45:06.654 CET [48] LOG: database system was shut down at 2026-11-03 02:45:05 CET
sparkyfitness-db | 2026-11-03 02:45:06.663 CET [42] LOG: database system is ready to accept connections
sparkyfitness-db | done
sparkyfitness-db | server started
sparkyfitness-db | CREATE DATABASE
sparkyfitness-db |
sparkyfitness-db |
sparkyfitness-db | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
sparkyfitness-db |
sparkyfitness-db | waiting for server to shut down....2026-11-03 02:45:06.887 CET [42] LOG: received fast shutdown request
sparkyfitness-db | 2026-11-03 02:45:06.892 CET [42] LOG: aborting any active transactions
sparkyfitness-db | 2026-11-03 02:45:06.896 CET [42] LOG: background worker "logical replication launcher" (PID 51) exited with exit code 1
sparkyfitness-db | 2026-11-03 02:45:06.896 CET [46] LOG: shutting down
sparkyfitness-db | 2026-11-03 02:45:06.899 CET [46] LOG: checkpoint starting: shutdown immediate
sparkyfitness-db | 2026-11-03 02:45:07.075 CET [46] LOG: checkpoint complete: wrote 943 buffers (5.8%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 0 recycled; write=0.026 s, sync=0.137 s, total=0.180 s; sync files=303, longest=0.109 s, average=0.001 s; distance=4362 kB, estimate=4362 kB; lsn=0/1BA8868, redo lsn=0/1BA8868
sparkyfitness-db | 2026-11-03 02:45:07.103 CET [42] LOG: database system is shut down
sparkyfitness-db | done
sparkyfitness-db | server stopped
sparkyfitness-db |
sparkyfitness-db | PostgreSQL init process complete; ready for start up.
sparkyfitness-db |
sparkyfitness-db | 2026-11-03 02:45:07.230 CET [1] LOG: starting PostgreSQL 18.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit
sparkyfitness-server | '20251221010000_drop_mood_weight_from_fasting_logs.sql',
sparkyfitness-server | '20251226150700_add_mobile_health_data_features.sql',
sparkyfitness-server | '20251228000000_add_custom_nutrients_tables.sql',
sparkyfitness-server | ... 84 more items
sparkyfitness-server | ]
sparkyfitness-server | [2026-11-03T01:46:56.081Z] [INFO] Applying migration: 20260707000000_add_passkey_registration_tickets.sql
sparkyfitness-server | [2026-11-03T01:46:56.101Z] [INFO] Successfully applied migration: 20260707000000_add_passkey_registration_tickets.sql
sparkyfitness-server | [2026-11-03T01:46:56.102Z] [INFO] Applying migration: 20260707090000_add_provider_verified_to_foods.sql
sparkyfitness-server | [2026-11-03T01:46:56.107Z] [INFO] Successfully applied migration: 20260707090000_add_provider_verified_to_foods.sql
sparkyfitness-server | [2026-11-03T01:46:56.107Z] [INFO] Applying migration: 20260707130000_add_workout_session_columns.sql
sparkyfitness-server | [2026-11-03T01:46:56.114Z] [INFO] Successfully applied migration: 20260707130000_add_workout_session_columns.sql
sparkyfitness-db | 2026-11-03 02:45:07.230 CET [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
sparkyfitness-server | [2026-11-03T01:46:56.114Z] [INFO] Applying migration: 20260709000000_create_food_favorites.sql
sparkyfitness-server | [2026-11-03T01:46:56.130Z] [INFO] Successfully applied migration: 20260709000000_create_food_favorites.sql
sparkyfitness-server | [2026-11-03T01:46:56.130Z] [INFO] Applying migration: 20260710000000_add_supplement_nutrients_to_medications.sql
sparkyfitness-server | [2026-11-03T01:46:56.144Z] [INFO] Successfully applied migration: 20260710000000_add_supplement_nutrients_to_medications.sql
sparkyfitness-server | [2026-11-03T01:46:56.144Z] [INFO] Applying migration: 20260710120000_add_entry_time_and_meal_default_time.sql
sparkyfitness-server | [2026-11-03T01:46:56.150Z] [INFO] Successfully applied migration: 20260710120000_add_entry_time_and_meal_default_time.sql
sparkyfitness-server | [2026-11-03T01:46:56.150Z] [INFO] Applying migration: 20260715120000_add_oura_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.153Z] [INFO] Successfully applied migration: 20260715120000_add_oura_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.154Z] [INFO] Applying migration: 20260716181900_drop_user_preferences_system_prompt.sql
sparkyfitness-server | [2026-11-03T01:46:56.158Z] [INFO] Successfully applied migration: 20260716181900_drop_user_preferences_system_prompt.sql
sparkyfitness-server | [2026-11-03T01:46:56.158Z] [INFO] Applying migration: 20260717120000_create_user_nutrient_goal_preferences.sql
sparkyfitness-server | [2026-11-03T01:46:56.170Z] [INFO] Successfully applied migration: 20260717120000_create_user_nutrient_goal_preferences.sql
sparkyfitness-server | [2026-11-03T01:46:56.170Z] [INFO] Applying migration: 20260723000000_add_spray_medication_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.173Z] [INFO] Successfully applied migration: 20260723000000_add_spray_medication_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.173Z] [INFO] Applying migration: 20260725173500_allow_multiple_active_meal_plans.sql
sparkyfitness-server | [2026-11-03T01:46:56.177Z] [INFO] Successfully applied migration: 20260725173500_allow_multiple_active_meal_plans.sql
sparkyfitness-db | 2026-11-03 02:45:07.230 CET [1] LOG: listening on IPv6 address "::", port 5432
sparkyfitness-db | 2026-11-03 02:45:07.238 CET [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
sparkyfitness-db | 2026-11-03 02:45:07.253 CET [64] LOG: database system was shut down at 2026-11-03 02:45:07 CET
sparkyfitness-db | 2026-11-03 02:45:07.263 CET [1] LOG: database system is ready to accept connections
sparkyfitness-server | [2026-11-03T01:46:56.177Z] [INFO] Applying migration: 20260726000000_guard_serving_size_positive.sql
sparkyfitness-server | [2026-11-03T01:46:56.187Z] [INFO] Successfully applied migration: 20260726000000_guard_serving_size_positive.sql
sparkyfitness-server | [2026-11-03T01:46:56.187Z] [INFO] Applying migration: 20260727100000_set_duration_seconds_modality_distance.sql
sparkyfitness-server | [2026-11-03T01:46:56.261Z] [INFO] Successfully applied migration: 20260727100000_set_duration_seconds_modality_distance.sql
sparkyfitness-server | [2026-11-03T01:46:56.261Z] [INFO] Applying migration: 20260730000000_create_generic_health_and_workout_tables.sql
sparkyfitness-server | [2026-11-03T01:46:56.336Z] [INFO] Successfully applied migration: 20260730000000_create_generic_health_and_workout_tables.sql
sparkyfitness-server | [2026-11-03T01:46:56.336Z] [INFO] Applying migration: 20260730120000_add_time_format_to_user_preferences.sql
sparkyfitness-server | [2026-11-03T01:46:56.343Z] [INFO] Successfully applied migration: 20260730120000_add_time_format_to_user_preferences.sql
sparkyfitness-server | [2026-11-03T01:46:56.343Z] [INFO] Applying migration: 20260730120000_add_workout_preset_set_distance.sql
sparkyfitness-server | [2026-11-03T01:46:56.351Z] [INFO] Successfully applied migration: 20260730120000_add_workout_preset_set_distance.sql
sparkyfitness-server | [2026-11-03T01:46:56.352Z] [INFO] Applying migration: 20260803000000_rename_gps_points_new_constraints.sql
sparkyfitness-server | [2026-11-03T01:46:56.355Z] [INFO] Successfully applied migration: 20260803000000_rename_gps_points_new_constraints.sql
sparkyfitness-server | [2026-11-03T01:46:56.355Z] [INFO] Applying migration: 20260804000000_add_source_id_to_water_intake_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.362Z] [INFO] Successfully applied migration: 20260804000000_add_source_id_to_water_intake_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.362Z] [INFO] Applying migration: 20260807000000_add_record_timezone_to_sleep_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.365Z] [INFO] Successfully applied migration: 20260807000000_add_record_timezone_to_sleep_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.365Z] [INFO] Applying migration: 20260810000000_add_images_to_foods_meals_and_food_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.375Z] [INFO] Successfully applied migration: 20260810000000_add_images_to_foods_meals_and_food_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.375Z] [INFO] Applying migration: 20260814000000_backfill_diary_entry_images_from_parent.sql
sparkyfitness-server | [2026-11-03T01:46:56.382Z] [INFO] Successfully applied migration: 20260814000000_backfill_diary_entry_images_from_parent.sql
sparkyfitness-server | [2026-11-03T01:46:56.382Z] [INFO] Applying migration: 20260816000000_widen_fractional_workout_telemetry_to_numeric.sql
sparkyfitness-server | [2026-11-03T01:46:56.410Z] [INFO] Successfully applied migration: 20260816000000_widen_fractional_workout_telemetry_to_numeric.sql
sparkyfitness-server | [2026-11-03T01:46:56.410Z] [INFO] Applying migration: 20260816173934_flip_goal_mode_custom_percentage_sign.sql
sparkyfitness-server | [2026-11-03T01:46:56.414Z] [INFO] Successfully applied migration: 20260816173934_flip_goal_mode_custom_percentage_sign.sql
sparkyfitness-server | [2026-11-03T01:46:56.414Z] [INFO] Applying migration: 20260816192818_normalize_exercise_json_array_fields.sql
sparkyfitness-server | [2026-11-03T01:46:56.423Z] [INFO] Successfully applied migration: 20260816192818_normalize_exercise_json_array_fields.sql
sparkyfitness-server | [2026-11-03T01:46:56.423Z] [INFO] Applying migration: 20260817000000_add_liftosaur_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.425Z] [INFO] Successfully applied migration: 20260817000000_add_liftosaur_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.425Z] [INFO] Applying migration: 20260818020000_fix_hevy_exercise_entry_distance_units.sql
sparkyfitness-server | [2026-11-03T01:46:56.432Z] [INFO] Successfully applied migration: 20260818020000_fix_hevy_exercise_entry_distance_units.sql
sparkyfitness-server | [2026-11-03T01:46:56.432Z] [INFO] Applying migration: 20260821150000_flatten_nested_exercise_image_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.436Z] [INFO] Successfully applied migration: 20260821150000_flatten_nested_exercise_image_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.436Z] [INFO] Applying migration: 20260821170000_add_food_search_all_providers_default.sql
sparkyfitness-server | [2026-11-03T01:46:56.439Z] [INFO] Successfully applied migration: 20260821170000_add_food_search_all_providers_default.sql
sparkyfitness-server | [2026-11-03T01:46:56.439Z] [INFO] Applying migration: 20260821233000_add_calorie_safety_floor_preferences.sql
sparkyfitness-server | [2026-11-03T01:46:56.445Z] [INFO] Successfully applied migration: 20260821233000_add_calorie_safety_floor_preferences.sql
sparkyfitness-server | [2026-11-03T01:46:56.445Z] [INFO] Applying migration: 20260827180000_add_bmr_to_check_in_measurements.sql
sparkyfitness-server | [2026-11-03T01:46:56.451Z] [INFO] Successfully applied migration: 20260827180000_add_bmr_to_check_in_measurements.sql
sparkyfitness-server | [2026-11-03T01:46:56.451Z] [INFO] Applying migration: 20260829171500_add_total_calories_capture_time.sql
sparkyfitness-server | [2026-11-03T01:46:56.455Z] [INFO] Successfully applied migration: 20260829171500_add_total_calories_capture_time.sql
sparkyfitness-server | [2026-11-03T01:46:56.455Z] [INFO] Applying migration: 20260830120000_add_notes_to_food_tables.sql
sparkyfitness-server | [2026-11-03T01:46:56.459Z] [INFO] Successfully applied migration: 20260830120000_add_notes_to_food_tables.sql
sparkyfitness-server | [2026-11-03T01:46:56.459Z] [INFO] Applying migration: 20260901103000_add_openfoodfacts_automatic_sync.sql
sparkyfitness-server | [2026-11-03T01:46:56.500Z] [INFO] Successfully applied migration: 20260901103000_add_openfoodfacts_automatic_sync.sql
sparkyfitness-server | [2026-11-03T01:46:56.500Z] [INFO] Applying migration: 20260901120000_add_entry_yield_to_food_entry_meals.sql
sparkyfitness-server | [2026-11-03T01:46:56.506Z] [INFO] Successfully applied migration: 20260901120000_add_entry_yield_to_food_entry_meals.sql
sparkyfitness-server | [2026-11-03T01:46:56.506Z] [INFO] Applying migration: 20260901120000_add_progression_fields_to_workout_preset_exercises.sql
sparkyfitness-server | [2026-11-03T01:46:56.513Z] [INFO] Successfully applied migration: 20260901120000_add_progression_fields_to_workout_preset_exercises.sql
sparkyfitness-server | [2026-11-03T01:46:56.514Z] [INFO] Applying migration: 20260904120000_add_chart_scale_mode_preference.sql
sparkyfitness-server | [2026-11-03T01:46:56.518Z] [INFO] Successfully applied migration: 20260904120000_add_chart_scale_mode_preference.sql
sparkyfitness-server | [2026-11-03T01:46:56.518Z] [INFO] Applying migration: 20260905150000_add_caffeine_alcohol_water_and_container_links.sql
sparkyfitness-server | [2026-11-03T01:46:56.572Z] [INFO] Successfully applied migration: 20260905150000_add_caffeine_alcohol_water_and_container_links.sql
sparkyfitness-server | [2026-11-03T01:46:56.572Z] [INFO] Applying migration: 20260908120000_narrow_check_in_bmr_bounds.sql
sparkyfitness-server | [2026-11-03T01:46:56.577Z] [INFO] Successfully applied migration: 20260908120000_narrow_check_in_bmr_bounds.sql
sparkyfitness-server | [2026-11-03T01:46:56.577Z] [INFO] Applying migration: 20260909203000_deduplicate_strava_activity_details.sql
sparkyfitness-server | [2026-11-03T01:46:56.583Z] [INFO] Successfully applied migration: 20260909203000_deduplicate_strava_activity_details.sql
sparkyfitness-server | [2026-11-03T01:46:56.583Z] [INFO] Applying migration: 20260910180000_deduplicate_shared_active_calories.sql
sparkyfitness-server | [2026-11-03T01:46:56.590Z] [INFO] Successfully applied migration: 20260910180000_deduplicate_shared_active_calories.sql
sparkyfitness-server | [2026-11-03T01:46:56.590Z] [INFO] Applying migration: 20260912150000_preserve_data_on_user_and_library_deletes.sql
sparkyfitness-server | [2026-11-03T01:46:56.717Z] [INFO] Successfully applied migration: 20260912150000_preserve_data_on_user_and_library_deletes.sql
sparkyfitness-server | [2026-11-03T01:46:56.717Z] [INFO] Applying migration: 20260912193000_better_auth_1_7_schema.sql
sparkyfitness-server | [2026-11-03T01:46:56.731Z] [INFO] Successfully applied migration: 20260912193000_better_auth_1_7_schema.sql
sparkyfitness-server | [2026-11-03T01:46:56.731Z] [INFO] Applying migration: 20260914210000_cleanup_zero_ml_water_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.734Z] [INFO] Successfully applied migration: 20260914210000_cleanup_zero_ml_water_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.734Z] [INFO] Applying migration: 20260918162032_add_lap_moving_telemetry.sql
sparkyfitness-server | [2026-11-03T01:46:56.738Z] [INFO] Successfully applied migration: 20260918162032_add_lap_moving_telemetry.sql
sparkyfitness-server | [2026-11-03T01:46:56.738Z] [INFO] Applying migration: 20260918170000_backfill_polar_steps_to_check_in.sql
sparkyfitness-server | [2026-11-03T01:46:56.743Z] [INFO] Successfully applied migration: 20260918170000_backfill_polar_steps_to_check_in.sql
sparkyfitness-server | [2026-11-03T01:46:56.743Z] [INFO] Applying migration: 20260918180000_widen_health_metric_samples_for_skin_temperature.sql
sparkyfitness-server | [2026-11-03T01:46:56.747Z] [INFO] Successfully applied migration: 20260918180000_widen_health_metric_samples_for_skin_temperature.sql
sparkyfitness-server | [2026-11-03T01:46:56.747Z] [INFO] Applying migration: 20260918230000_add_admin_system_settings.sql
sparkyfitness-server | [2026-11-03T01:46:56.756Z] [INFO] Successfully applied migration: 20260918230000_add_admin_system_settings.sql
sparkyfitness-server | [2026-11-03T01:46:56.756Z] [INFO] Applying migration: 20260921140000_add_sequential_workout_plan_mode.sql
sparkyfitness-server | [2026-11-03T01:46:56.767Z] [INFO] Successfully applied migration: 20260921140000_add_sequential_workout_plan_mode.sql
sparkyfitness-server | [2026-11-03T01:46:56.767Z] [INFO] Applying migration: 20260921210000_add_watch_telemetry_observed_at.sql
sparkyfitness-server | [2026-11-03T01:46:56.770Z] [INFO] Successfully applied migration: 20260921210000_add_watch_telemetry_observed_at.sql
sparkyfitness-server | [2026-11-03T01:46:56.770Z] [INFO] Applying migration: 20260923170000_add_workout_format_to_presets.sql
sparkyfitness-server | [2026-11-03T01:46:56.775Z] [INFO] Successfully applied migration: 20260923170000_add_workout_format_to_presets.sql
sparkyfitness-server | [2026-11-03T01:46:56.775Z] [INFO] Applying migration: 20260924200000_add_location_and_rir.sql
sparkyfitness-server | [2026-11-03T01:46:56.778Z] [INFO] Successfully applied migration: 20260924200000_add_location_and_rir.sql
sparkyfitness-server | [2026-11-03T01:46:56.779Z] [INFO] Applying migration: 20260925013000_add_watch_duration_minutes.sql
sparkyfitness-server | [2026-11-03T01:46:56.782Z] [INFO] Successfully applied migration: 20260925013000_add_watch_duration_minutes.sql
sparkyfitness-server | [2026-11-03T01:46:56.782Z] [INFO] Applying migration: 20260925180000_add_ramp_increment_to_workout_preset_exercises.sql
sparkyfitness-server | [2026-11-03T01:46:56.785Z] [INFO] Successfully applied migration: 20260925180000_add_ramp_increment_to_workout_preset_exercises.sql
sparkyfitness-server | [2026-11-03T01:46:56.785Z] [INFO] Applying migration: 20260925200000_snapshot_workout_format_on_preset_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.791Z] [INFO] Successfully applied migration: 20260925200000_snapshot_workout_format_on_preset_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.791Z] [INFO] Applying migration: 20260926120000_add_workout_feedback.sql
sparkyfitness-server | [2026-11-03T01:46:56.810Z] [INFO] Successfully applied migration: 20260926120000_add_workout_feedback.sql
sparkyfitness-server | [2026-11-03T01:46:56.810Z] [INFO] Applying migration: 20260926230000_add_record_timezone_to_exercise_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.813Z] [INFO] Successfully applied migration: 20260926230000_add_record_timezone_to_exercise_entries.sql
sparkyfitness-server | [2026-11-03T01:46:56.813Z] [INFO] Applying migration: 20260927180000_add_coros_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.816Z] [INFO] Successfully applied migration: 20260927180000_add_coros_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.816Z] [INFO] Applying migration: 20260929120000_generalize_symptom_tracking.sql
sparkyfitness-server | [2026-11-03T01:46:56.889Z] [INFO] Successfully applied migration: 20260929120000_generalize_symptom_tracking.sql
sparkyfitness-server | [2026-11-03T01:46:56.889Z] [INFO] Applying migration: 20260930120000_add_rate_limit.sql
sparkyfitness-server | [2026-11-03T01:46:56.897Z] [INFO] Successfully applied migration: 20260930120000_add_rate_limit.sql
sparkyfitness-server | [2026-11-03T01:46:56.897Z] [INFO] Applying migration: 20260930174500_add_voice_input_support.sql
sparkyfitness-server | [2026-11-03T01:46:56.905Z] [INFO] Successfully applied migration: 20260930174500_add_voice_input_support.sql
sparkyfitness-server | [2026-11-03T01:46:56.905Z] [INFO] Applying migration: 20260930174501_validate_voice_input_support.sql
sparkyfitness-server | [2026-11-03T01:46:56.910Z] [INFO] Successfully applied migration: 20260930174501_validate_voice_input_support.sql
sparkyfitness-server | [2026-11-03T01:46:56.910Z] [INFO] Applying migration: 20260930190000_add_bodyweight_reps_modality.sql
sparkyfitness-server | [2026-11-03T01:46:56.915Z] [INFO] Successfully applied migration: 20260930190000_add_bodyweight_reps_modality.sql
sparkyfitness-server | [2026-11-03T01:46:56.915Z] [INFO] Applying migration: 20260930190100_validate_bodyweight_reps_modality.sql
sparkyfitness-server | [2026-11-03T01:46:56.918Z] [INFO] Successfully applied migration: 20260930190100_validate_bodyweight_reps_modality.sql
sparkyfitness-server | [2026-11-03T01:46:56.918Z] [INFO] Applying migration: 20261001120000_add_enable_passkey_login.sql
sparkyfitness-server | [2026-11-03T01:46:56.921Z] [INFO] Successfully applied migration: 20261001120000_add_enable_passkey_login.sql
sparkyfitness-server | [2026-11-03T01:46:56.921Z] [INFO] Applying migration: 20261003190000_add_weight_distance_and_duration_modalities.sql
sparkyfitness-server | [2026-11-03T01:46:56.926Z] [INFO] Successfully applied migration: 20261003190000_add_weight_distance_and_duration_modalities.sql
sparkyfitness-server | [2026-11-03T01:46:56.926Z] [INFO] Applying migration: 20261003190001_validate_weight_distance_and_duration_modalities.sql
sparkyfitness-server | [2026-11-03T01:46:56.930Z] [INFO] Successfully applied migration: 20261003190001_validate_weight_distance_and_duration_modalities.sql
sparkyfitness-server | [2026-11-03T01:46:56.930Z] [INFO] Applying migration: 20261004120000_create_user_fasting_preferences.sql
sparkyfitness-server | [2026-11-03T01:46:56.942Z] [INFO] Successfully applied migration: 20261004120000_create_user_fasting_preferences.sql
sparkyfitness-server | [2026-11-03T01:46:56.942Z] [INFO] Applying migration: 20261004180000_add_canadian_nutrient_file_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.953Z] [INFO] Successfully applied migration: 20261004180000_add_canadian_nutrient_file_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.953Z] [INFO] Applying migration: 20261005193000_create_mindfulness_sessions.sql
sparkyfitness-server | [2026-11-03T01:46:56.969Z] [INFO] Successfully applied migration: 20261005193000_create_mindfulness_sessions.sql
sparkyfitness-server | [2026-11-03T01:46:56.969Z] [INFO] Applying migration: 20261005200000_add_sync_started_at_to_external_data_providers.sql
sparkyfitness-server | [2026-11-03T01:46:56.972Z] [INFO] Successfully applied migration: 20261005200000_add_sync_started_at_to_external_data_providers.sql
sparkyfitness-server | [2026-11-03T01:46:56.972Z] [INFO] Applying migration: 20261006180000_unique_sleep_entries_and_custom_categories.sql
sparkyfitness-server | [2026-11-03T01:46:56.983Z] [INFO] Successfully applied migration: 20261006180000_unique_sleep_entries_and_custom_categories.sql
sparkyfitness-server | [2026-11-03T01:46:56.983Z] [INFO] Applying migration: 20261006200000_add_dsld_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.989Z] [INFO] Successfully applied migration: 20261006200000_add_dsld_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:56.989Z] [INFO] Applying migration: 20261008213000_ai_reasoning_effort.sql
sparkyfitness-server | [2026-11-03T01:46:56.994Z] [INFO] Successfully applied migration: 20261008213000_ai_reasoning_effort.sql
sparkyfitness-server | [2026-11-03T01:46:56.994Z] [INFO] Applying migration: 20261008220000_food_assistant_state.sql
sparkyfitness-server | [2026-11-03T01:46:57.031Z] [INFO] Successfully applied migration: 20261008220000_food_assistant_state.sql
sparkyfitness-server | [2026-11-03T01:46:57.031Z] [INFO] Applying migration: 20261010200000_create_run_programs.sql
sparkyfitness-server | [2026-11-03T01:46:57.049Z] [INFO] Successfully applied migration: 20261010200000_create_run_programs.sql
sparkyfitness-server | [2026-11-03T01:46:57.049Z] [INFO] Applying migration: 20261012120000_create_shoes.sql
sparkyfitness-server | [2026-11-03T01:46:57.072Z] [INFO] Successfully applied migration: 20261012120000_create_shoes.sql
sparkyfitness-server | [2026-11-03T01:46:57.072Z] [INFO] Applying migration: 20261013120000_add_exercise_entry_distance_series.sql
sparkyfitness-server | [2026-11-03T01:46:57.084Z] [INFO] Successfully applied migration: 20261013120000_add_exercise_entry_distance_series.sql
sparkyfitness-server | [2026-11-03T01:46:57.084Z] [INFO] Applying migration: 20261013140000_add_custom_run_programs.sql
sparkyfitness-server | [2026-11-03T01:46:57.095Z] [INFO] Successfully applied migration: 20261013140000_add_custom_run_programs.sql
sparkyfitness-server | [2026-11-03T01:46:57.095Z] [INFO] Applying migration: 20261014000000_add_target_steps_to_goals.sql
sparkyfitness-server | [2026-11-03T01:46:57.099Z] [INFO] Successfully applied migration: 20261014000000_add_target_steps_to_goals.sql
sparkyfitness-server | [2026-11-03T01:46:57.099Z] [INFO] Applying migration: 20261014213414_convert_chatbot_mood_scores.sql
sparkyfitness-server | [2026-11-03T01:46:57.102Z] [INFO] Successfully applied migration: 20261014213414_convert_chatbot_mood_scores.sql
sparkyfitness-server | [2026-11-03T01:46:57.102Z] [INFO] Applying migration: 20261014220000_add_livsmedelsverket_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:57.107Z] [INFO] Successfully applied migration: 20261014220000_add_livsmedelsverket_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:57.107Z] [INFO] Applying migration: 20261015120000_create_run_goals.sql
sparkyfitness-server | [2026-11-03T01:46:57.115Z] [INFO] Successfully applied migration: 20261015120000_create_run_goals.sql
sparkyfitness-server | [2026-11-03T01:46:57.115Z] [INFO] Applying migration: 20261018120000_create_fitness_tests.sql
sparkyfitness-server | [2026-11-03T01:46:57.130Z] [INFO] Successfully applied migration: 20261018120000_create_fitness_tests.sql
sparkyfitness-server | [2026-11-03T01:46:57.130Z] [INFO] Applying migration: 20261020000000_add_energy_focus_to_mood_entries.sql
sparkyfitness-server | [2026-11-03T01:46:57.134Z] [INFO] Successfully applied migration: 20261020000000_add_energy_focus_to_mood_entries.sql
sparkyfitness-server | [2026-11-03T01:46:57.134Z] [INFO] Applying migration: 20261020000000_add_mcp_oauth_provider.sql
sparkyfitness-server | [2026-11-03T01:46:57.206Z] [INFO] Successfully applied migration: 20261020000000_add_mcp_oauth_provider.sql
sparkyfitness-server | [2026-11-03T01:46:57.206Z] [INFO] Applying migration: 20261021000000_multiple_mood_checkins.sql
sparkyfitness-server | [2026-11-03T01:46:57.219Z] [INFO] Successfully applied migration: 20261021000000_multiple_mood_checkins.sql
sparkyfitness-server | [2026-11-03T01:46:57.220Z] [INFO] Applying migration: 20261021120000_add_routes.sql
sparkyfitness-server | [2026-11-03T01:46:57.234Z] [INFO] Successfully applied migration: 20261021120000_add_routes.sql
sparkyfitness-server | [2026-11-03T01:46:57.234Z] [INFO] Applying migration: 20261021120100_allow_generated_route_source.sql
sparkyfitness-server | [2026-11-03T01:46:57.238Z] [INFO] Successfully applied migration: 20261021120100_allow_generated_route_source.sql
sparkyfitness-server | [2026-11-03T01:46:57.238Z] [INFO] Applying migration: 20261021120200_add_valhalla_routing_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:57.241Z] [INFO] Successfully applied migration: 20261021120200_add_valhalla_routing_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:57.241Z] [INFO] Applying migration: 20261021130000_add_symptom_impact_timeline.sql
sparkyfitness-server | [2026-11-03T01:46:57.246Z] [INFO] Successfully applied migration: 20261021130000_add_symptom_impact_timeline.sql
sparkyfitness-server | [2026-11-03T01:46:57.246Z] [INFO] Applying migration: 20261021140000_add_manual_fields_to_exercise_entries.sql
sparkyfitness-server | [2026-11-03T01:46:57.249Z] [INFO] Successfully applied migration: 20261021140000_add_manual_fields_to_exercise_entries.sql
sparkyfitness-server | [2026-11-03T01:46:57.249Z] [INFO] Applying migration: 20261022110000_add_adaptive_activity_separation.sql
sparkyfitness-server | [2026-11-03T01:46:57.252Z] [INFO] Successfully applied migration: 20261022110000_add_adaptive_activity_separation.sql
sparkyfitness-server | [2026-11-03T01:46:57.252Z] [INFO] Applying migration: 20261022120000_add_heart_rate_zone_settings.sql
sparkyfitness-server | [2026-11-03T01:46:57.263Z] [INFO] Successfully applied migration: 20261022120000_add_heart_rate_zone_settings.sql
sparkyfitness-server | [2026-11-03T01:46:57.263Z] [INFO] Applying migration: 20261022120000_add_workout_effort_to_exercise_entries.sql
sparkyfitness-server | [2026-11-03T01:46:57.267Z] [INFO] Successfully applied migration: 20261022120000_add_workout_effort_to_exercise_entries.sql
sparkyfitness-server | [2026-11-03T01:46:57.267Z] [INFO] Applying migration: 20261022140000_add_weekly_weight_goal.sql
sparkyfitness-server | [2026-11-03T01:46:57.272Z] [INFO] Successfully applied migration: 20261022140000_add_weekly_weight_goal.sql
sparkyfitness-server | [2026-11-03T01:46:57.272Z] [INFO] Applying migration: 20261023000000_add_user_calorie_rule_history.sql
sparkyfitness-server | [2026-11-03T01:46:57.285Z] [INFO] Successfully applied migration: 20261023000000_add_user_calorie_rule_history.sql
sparkyfitness-server | [2026-11-03T01:46:57.285Z] [INFO] Applying migration: 20261023130000_add_resting_heart_rate_to_checkin_and_vitals.sql
sparkyfitness-server | [2026-11-03T01:46:57.289Z] [INFO] Successfully applied migration: 20261023130000_add_resting_heart_rate_to_checkin_and_vitals.sql
sparkyfitness-server | [2026-11-03T01:46:57.289Z] [INFO] Applying migration: 20261023204500_add_exercise_entry_calorie_link.sql
sparkyfitness-server | [2026-11-03T01:46:57.296Z] [INFO] Successfully applied migration: 20261023204500_add_exercise_entry_calorie_link.sql
sparkyfitness-server | [2026-11-03T01:46:57.296Z] [INFO] Applying migration: 20261024100000_create_plate_inventories.sql
sparkyfitness-server | [2026-11-03T01:46:57.310Z] [INFO] Successfully applied migration: 20261024100000_create_plate_inventories.sql
sparkyfitness-server | [2026-11-03T01:46:57.310Z] [INFO] Applying migration: 20261024120000_add_intervals_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:57.313Z] [INFO] Successfully applied migration: 20261024120000_add_intervals_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:57.313Z] [INFO] Applying migration: 20261025120000_add_hr_zone_display_count.sql
sparkyfitness-server | [2026-11-03T01:46:57.319Z] [INFO] Successfully applied migration: 20261025120000_add_hr_zone_display_count.sql
sparkyfitness-server | [2026-11-03T01:46:57.319Z] [INFO] Applying migration: 20261025120000_add_open_meteo_weather_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:57.321Z] [INFO] Successfully applied migration: 20261025120000_add_open_meteo_weather_provider_type.sql
sparkyfitness-server | [2026-11-03T01:46:57.321Z] [INFO] Applying migration: 20261025170000_add_enable_version_check.sql
sparkyfitness-server | [2026-11-03T01:46:57.326Z] [INFO] Successfully applied migration: 20261025170000_add_enable_version_check.sql
sparkyfitness-server | [2026-11-03T01:46:57.326Z] [INFO] Applying migration: 20261027120000_add_rate_limit_windows.sql
sparkyfitness-server | [2026-11-03T01:46:57.333Z] [INFO] Successfully applied migration: 20261027120000_add_rate_limit_windows.sql
sparkyfitness-server | [2026-11-03T01:46:57.333Z] [INFO] Applying migration: 20261028120000_allow_multiple_sleep_sessions_per_night.sql
sparkyfitness-server | [2026-11-03T01:46:57.341Z] [INFO] Successfully applied migration: 20261028120000_allow_multiple_sleep_sessions_per_night.sql
sparkyfitness-server | [2026-11-03T01:46:57.341Z] [INFO] Applying migration: 20261028130000_add_rls_policy_state.sql
sparkyfitness-server | [2026-11-03T01:46:57.347Z] [INFO] Successfully applied migration: 20261028130000_add_rls_policy_state.sql
sparkyfitness-server | [2026-11-03T01:46:57.347Z] [INFO] Applying migration: 20261029120000_add_plan_assignment_proposals_applied_at.sql
sparkyfitness-server | [2026-11-03T01:46:57.350Z] [INFO] Successfully applied migration: 20261029120000_add_plan_assignment_proposals_applied_at.sql
sparkyfitness-server | [2026-11-03T01:46:57.351Z] [INFO] Applying migration: 20261029120000_create_lab_tables.sql
sparkyfitness-server | [2026-11-03T01:46:57.377Z] [INFO] Successfully applied migration: 20261029120000_create_lab_tables.sql
sparkyfitness-server | [2026-11-03T01:46:57.378Z] [INFO] Applying migration: 20261030010000_add_coros_fit_download_budget.sql
sparkyfitness-server | [2026-11-03T01:46:57.382Z] [INFO] Successfully applied migration: 20261030010000_add_coros_fit_download_budget.sql
sparkyfitness-server | [2026-11-03T01:46:57.382Z] [INFO] Ensuring permissions for role: "sparkyapp"
sparkyfitness-server | [2026-11-03T01:46:57.420Z] [INFO] Successfully ensured permissions for role: "sparkyapp"
sparkyfitness-server | [2026-11-03T01:46:57.420Z] [INFO] Permissions granted to application user.
sparkyfitness-server | [2026-11-03T01:46:57.425Z] [INFO] Applying all RLS policies from rls_policies.sql...
sparkyfitness-server | [2026-11-03T01:46:57.632Z] [INFO] Successfully applied all RLS policies.
sparkyfitness-server | Starting server...
sparkyfitness-server | [2026-11-03T01:47:07.445Z] [INFO] UserProfileRoutes UPLOADS_DIR: /app/SparkyFitnessServer/uploads/avatars
sparkyfitness-server | DEBUG: SPARKY_FITNESS_FRONTEND_URL is: https://sparky.gate.invalid
sparkyfitness-server | [SECURITY] Private network CORS is ENABLED. Ensure this is only on self-hosted/private networks.
sparkyfitness-server | SparkyFitnessServer UPLOADS_BASE_DIR: /app/SparkyFitnessServer/uploads
sparkyfitness-server | [2026-11-03T01:47:07.662Z] [INFO] Ensured temporary upload directory exists: /app/SparkyFitnessServer/temp_uploads/
sparkyfitness-server | [AUTH] Starting Better Auth mounting phase...
sparkyfitness-server | [AUTH] Better Auth handler successfully mounted.
sparkyfitness-server | [AUTH] Synced trusted SSO providers for auto-linking: []
sparkyfitness-server | [2026-11-03T01:47:07.679Z] [INFO] [AUTH] Synced trusted SSO provider origins: []
sparkyfitness-server | [2026-11-03T01:47:07.738Z] [INFO] [CRON] Scheduled backups disabled — skipping schedule
sparkyfitness-server | DEBUG: Server started and listening on port 3010
sparkyfitness-server | [2026-11-03T01:47:07.758Z] [INFO] SparkyFitnessServer listening on port 3010
sparkyfitness-server | View API documentation at: /api/api-docs/swagger
sparkyfitness-server | [2026-11-03T01:47:25.098Z] [INFO] Incoming request: GET /api/health (Path: /api/health)
sparkyfitness-server | [2026-11-03T01:47:55.236Z] [INFO] Incoming request: GET /api/health (Path: /api/health)
@@ -0,0 +1,20 @@
{
"sparkyfitness": {
"status": "running",
"health": "healthy",
"restarts": 0,
"exit": 0
},
"sparkyfitness-db": {
"status": "running",
"health": "healthy",
"restarts": 0,
"exit": 0
},
"sparkyfitness-server": {
"status": "running",
"health": "healthy",
"restarts": 0,
"exit": 0
}
}
@@ -0,0 +1,97 @@
{
"harness_version": 5,
"edge": "MV-sparkyfitness",
"app": "sparkyfitness",
"note": "night 2026-09-23 within-a-major move: sparkyfitness-server=codewithcj/sparkyfitness_server:v1.11.0, sparkyfitness-frontend=codewithcj/sparkyfitness:v1.11.0",
"from": {
"sparkyfitness-db": "postgres:18-alpine",
"sparkyfitness-server": "codewithcj/sparkyfitness_server:v0.17.3",
"sparkyfitness-frontend": "codewithcj/sparkyfitness:v0.17.3"
},
"to": {
"sparkyfitness-db": "postgres:18-alpine",
"sparkyfitness-server": "codewithcj/sparkyfitness_server:v1.11.0",
"sparkyfitness-frontend": "codewithcj/sparkyfitness:v1.11.0"
},
"verdict": "proven",
"seed_read_before": true,
"seed_read_after": true,
"healthy_after": true,
"migration_observed": "\u001b[2K\u001b[2Ksparkyfitness-server | [2026-11-03T01:46:56.073Z] [INFO] Ensured schema_migrations table exists.",
"abort": "refuses",
"abort_detail": "| detail: undefined,\n\u001b[2Ksparkyfitness-server | hint: undefined,\n\u001b[2Ksparkyfitness-server | position: undefined,\n\u001b[2Ksparkyfitness-server | internalPosition: undefined,\n\u001b[2Ksparkyfitness-server | internalQuery: undefined,\n\u001b[2Ksparkyfitness-server | where: 'SQL statement \"ALTER TABLE public.user_custom_symptom_locations ENABLE ROW LEVEL SECURITY;\"\\n' +\n\u001b[2Ksparkyfitness-server | 'PL/pgSQL function inline_code_block line 86 at EXECUTE',\n\u001b[2Ksparkyfitness-server | schema: undefined,\n\u001b[2Ksparkyfitness-server | table: undefined,\n\u001b[2Ksparkyfitness-server | column: undefined,\n\u001b[2Ksparkyfitness-server | dataType: undefined,\n\u001b[2Ksparkyfitness-server | constraint: undefined,\n\u001b[2Ksparkyfitness-server | file: 'namespace.c',\n\u001b[2Ksparkyfitness-server | line: '631',\n\u001b[2Ksparkyfitness-server | routine: 'RangeVarGetRelidExtended'\n\u001b[2Ksparkyfitness-server | }",
"engine_state_after": {
"sparkyfitness-db": {
"image": "postgres:18-alpine",
"probe": "datadir major version",
"answer": "18",
"probe_rc": 0
}
},
"memory": {
"soak_s": 607.5,
"requested_s": 600,
"requests": 12060,
"codes": {
"200": 12060
},
"first_kill": null,
"containers": {
"sparkyfitness": {
"limit": 268435456,
"peak": 10682368,
"peak_pct": 0.04,
"anon_peak_sampled": 5623808,
"anon_peak_pct": 0.021,
"anon_live_peak_sampled": 5623808,
"anon_live_peak_pct": 0.021,
"swap_peak": 0,
"oom_kills": 0,
"restarts": 0,
"oomkilled_flag": false,
"measured": true
},
"sparkyfitness-db": {
"limit": 536870912,
"peak": 203993088,
"peak_pct": 0.38,
"anon_peak_sampled": 10014720,
"anon_peak_pct": 0.019,
"anon_live_peak_sampled": 10014720,
"anon_live_peak_pct": 0.019,
"swap_peak": 0,
"oom_kills": 0,
"restarts": 0,
"oomkilled_flag": false,
"measured": true
},
"sparkyfitness-server": {
"limit": 1073741824,
"peak": 526200832,
"peak_pct": 0.49,
"anon_peak_sampled": 328564736,
"anon_peak_pct": 0.306,
"anon_live_peak_sampled": 328564736,
"anon_live_peak_pct": 0.306,
"swap_peak": 0,
"oom_kills": 0,
"restarts": 0,
"oomkilled_flag": false,
"measured": true
}
},
"unmeasured": [],
"venue_swap_bytes": 0,
"load": "reached"
},
"marks": [],
"bench_overrides": null,
"duration_s": 31.2,
"measured_at": "2026-11-03T02:01:49Z",
"evidence": "evidence/MV-sparkyfitness",
"scratch_cleared": [],
"files_changed": [],
"files_changed_detail": [],
"files_ignored": [],
"total_s": 1029.9
}
@@ -0,0 +1,13 @@
# Part 3 — the live proofs on scratch 9202 (2026-11-03), controller test image of `c4b6844`
The image `0.350.0-rc.1-sec1102fault` was local only: `docker save`/`load`, never pushed, removed after. Driver: `proof1102.py`. Events went to a sink in the guest, not a hub.
| Row | What was staged | What happened | File |
|---|---|---|---|
| R-1347 | an update with its undo copy cut off (finished-marker removed while it verifies; a held volume does not stop the undo, `try1-…`) | hold `update_failed`; the household's restore: „the update hold … is CLEARED", „its update-held flag is forgotten"; six `kill -9`: `crash_loop … STOPPING`, hold `unhealthy_stop`: the watcher watches again | `stdout-r1347.txt`, `stdout-r1347b.txt` |
| R-1348 | a restore cut at „the restore copy is recorded": `docker kill`, the volume held by a created-only container, the controller started | „could NOT be put back", hold `restore_mixed` / `interrupted_put_back`, event `backup_run_failures` with the R-1340 sentence in the sink. It showed a wrong run name (offsite-reconstitute): fixed, `3764445` | `stdout-r1348.txt`, `r1348-after.txt` |
| R-1339 | a guest reboot, before and after | 9 false lines → 0 | `stdout-boot-before.txt`, `stdout-boot-after.txt` |
| R-1351 | reboots: plain; the sync recreating; also the file manager held down so both paths run in one second | 0 Conflict and 0 first-boot WARN on all six, 3 per image. **The race did not fire on the old image either, so these boots do not tell the images apart.** The proof is the test with the lock removed | `stdout-boot-*.txt` |
| Traefik v3.7.17 | three boots | started, dashboard 200, file manager healthy | `stdout-boot-after.txt` |
Left on 9202: one hold in its settings, `radicale` (`restore_mixed`), from R-1348's proof. A brief's fence forbids CC to clear a restore hold; the app itself is removed. To clear it: `pct exec 9202 -- docker exec felhom-controller felhom-controller --clear-restore-hold radicale`. Until then a walk of Radicale on 9202 is refused its start.
@@ -0,0 +1,15 @@
00:38:58 pct reboot 9202 rc=0
00:39:23 login after the reboot: True
00:40:12 images:
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault Up About a minute (healthy)
traefik traefik:v3.7.17 Up About a minute
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
00:40:15 [after] FileBrowser lines of this boot:
f9c43ce65f4e traefik traefik traefik:v3.7.17
2026/11/02 23:39:03 infra.go:401: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/02 23:39:04 handlers.go:3974: [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (11 storage path(s))
00:40:15 [after] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 0
00:40:15 /api/health 200
@@ -0,0 +1,23 @@
00:35:53 pct reboot 9202 rc=0
00:36:18 login after the reboot: True
00:37:07 images:
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
00:37:10 [before] FileBrowser lines of this boot:
e27bdddefe5e traefik traefik traefik:v3.7.16
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/immich not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/calibre-web not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/grimmory not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/metube not mounted — skipping userdata skeleton
2026/11/02 23:35:59 infra.go:378: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/02 23:36:00 handlers.go:3958: [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (11 storage path(s))
00:37:10 [before] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 9
@@ -0,0 +1,17 @@
00:40:38 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
00:41:44 pct reboot 9202 rc=0
00:42:09 login after the reboot: True
00:42:57 images:
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault Up About a minute (healthy)
traefik traefik:v3.7.17 Up About a minute
00:43:01 [after-stage] FileBrowser lines of this boot:
f9c43ce65f4e traefik traefik traefik:v3.7.17
2026/11/02 23:41:49 infra.go:401: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/02 23:41:50 handlers.go:3967: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
00:43:01 [after-stage] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 0
@@ -0,0 +1,26 @@
01:05:36 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
01:06:42 pct reboot 9202 rc=0
01:07:07 login after the reboot: True
01:07:56 images:
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
01:07:59 [before-stage] FileBrowser lines of this boot:
fd1fbea08cb8 traefik traefik traefik:v3.7.16
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/immich not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/calibre-web not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/grimmory not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/metube not mounted — skipping userdata skeleton
2026/11/03 00:06:47 infra.go:378: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/03 00:06:48 handlers.go:3951: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
01:07:59 [before-stage] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 9
@@ -0,0 +1,53 @@
01:05:36 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
01:06:42 pct reboot 9202 rc=0
01:07:07 login after the reboot: True
01:07:56 images:
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
01:07:59 [before-stage] FileBrowser lines of this boot:
fd1fbea08cb8 traefik traefik traefik:v3.7.16
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/immich not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/calibre-web not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/grimmory not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/metube not mounted — skipping userdata skeleton
2026/11/03 00:06:47 infra.go:378: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/03 00:06:48 handlers.go:3951: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
01:07:59 [before-stage] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 9
01:12:54 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
01:13:59 pct reboot 9202 rc=0
01:14:24 login after the reboot: True
01:15:13 images:
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
01:15:16 [before-both-paths] FileBrowser lines of this boot:
fd1fbea08cb8 traefik traefik traefik:v3.7.16
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/immich not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/calibre-web not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/grimmory not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/metube not mounted — skipping userdata skeleton
2026/11/03 00:14:04 infra.go:378: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/03 00:14:04 infra.go:265: [INFO] [infra] filebrowser compose exists — starting without regenerating
2026/11/03 00:14:06 handlers.go:3951: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
01:15:16 [before-both-paths] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 9
@@ -0,0 +1,71 @@
01:05:36 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
01:06:42 pct reboot 9202 rc=0
01:07:07 login after the reboot: True
01:07:56 images:
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
01:07:59 [before-stage] FileBrowser lines of this boot:
fd1fbea08cb8 traefik traefik traefik:v3.7.16
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/immich not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/calibre-web not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/grimmory not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/metube not mounted — skipping userdata skeleton
2026/11/03 00:06:47 infra.go:378: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/03 00:06:48 handlers.go:3951: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
01:07:59 [before-stage] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 9
01:12:54 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
01:13:59 pct reboot 9202 rc=0
01:14:24 login after the reboot: True
01:15:13 images:
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
01:15:16 [before-both-paths] FileBrowser lines of this boot:
fd1fbea08cb8 traefik traefik traefik:v3.7.16
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/immich not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/calibre-web not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/grimmory not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/metube not mounted — skipping userdata skeleton
2026/11/03 00:14:04 infra.go:378: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/03 00:14:04 infra.go:265: [INFO] [infra] filebrowser compose exists — starting without regenerating
2026/11/03 00:14:06 handlers.go:3951: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
01:15:16 [before-both-paths] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 9
01:16:14 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
01:16:29 pct reboot 9202 rc=0
01:16:54 login after the reboot: True
01:17:43 images:
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault Up About a minute (healthy)
traefik traefik:v3.7.17 Up About a minute
01:17:46 [after-both-paths] FileBrowser lines of this boot:
4ce334822229 traefik traefik traefik:v3.7.17
2026/11/03 00:16:34 infra.go:401: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/03 00:16:34 infra.go:285: [INFO] [infra] filebrowser compose exists — starting without regenerating
2026/11/03 00:16:36 handlers.go:3967: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
01:17:46 [after-both-paths] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 0
@@ -0,0 +1,337 @@
#!/usr/bin/env python3
"""proof1102.py <phase> — live proofs of R-1347, R-1348, R-1351, R-1339 and Traefik v3.7.17 on scratch 9202 (2026-11-02 night).
Front doors only where a household has one (deploy, Update, the restore button, Start, Remove). The faults are real docker
state, not switches: the undo copy loses its finished-marker (R-1347), and a created-only container holds the app's volume so the
crash put-back (R-1348) cannot replace it. The cut is `docker kill` of the controller at the log line „the restore copy is recorded".
Phases, in order:
boot-before guest reboot on the controller as found; the FileBrowser lines of that boot
switch load the local-only test image (never pushed), point the box at it and at the event sink
boot-after guest reboot on the test image; the same lines; Traefik's and the file manager's state
r1347 radicale: an update that fails with an undo copy cut off → HOLD; the household's restore
lifts the hold and forgets the flag; 7 kill -9 → the crash-loop watcher stops the app (it watches again)
r1348 the household's Start, then a restore cut at the copy's record, the volume held, the controller started
→ the put-back fails at start-up → the hold AND the operator event in the sink
teardown blocker gone, app removed through the product, controller back on the image it had, yaml byte-identical
Env: SC (0600 scratch with .ctlpw), EV (evidence dir). No secret is printed; the sink stores event bodies only."""
import json, os, subprocess, sys, threading, time
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
os.environ.setdefault("TARGET", "9202")
import box_walk as w
import upgrade_fixtures_box as fb
EV = os.environ.get("EV", ".")
APP, SUB = "radicale", getattr(fb.FIXTURES["radicale"], "sub", "radicale")
VOL = "/var/lib/docker/volumes/felhom-controller-data/_data"
YAML = f"{VOL}/controller.yaml"
SAVE = f"{VOL}/controller.yaml.pre-proof1102"
IMG_FOUND = "gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1"
IMG_TEST = "gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault"
SINK_PORT = 18091
SINK_FILE = "/tmp/sec1102-sink.jsonl"
BLOCKER = "sec1102-blocker"
MARKER = 'felhom-undo-complete'
DRILL = w.DRILL
STATE = os.path.join(os.environ["SC"], ".proof-state.json")
def say(*a):
w.say(*a)
def flush(name):
with open(os.path.join(EV, f"{name}.log"), "a") as f:
f.write("\n".join(w.LOG) + "\n")
def guest(script, timeout=600):
return w.guest(script, timeout=timeout)
def ctl_log(since, rx):
return guest(f"docker logs --since {since} felhom-controller 2>&1 | grep -E '{rx}' | grep -v DEBUG | cut -c1-260 | tail -40")
def now():
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
def wait_login(tries=60):
for _ in range(tries):
time.sleep(5)
try:
w.login()
return True
except SystemExit:
continue
return False
def boot_lines(tag):
"""Reboot the guest and read this boot's FileBrowser and first-boot lines from the controller's log."""
t0 = now()
r = w.sh(w.ssh_args() + [f"pct reboot {w.GUEST}"], timeout=180)
say(f" pct reboot {w.GUEST} rc={r.returncode}")
time.sleep(20)
ok = wait_login(40)
say(f" login after the reboot: {ok}")
time.sleep(45) # the base stack's round and the mounts sync
img = guest("cat /etc/felhom-controller-image; docker ps --format '{{.Names}} {{.Image}} {{.Status}}' | grep -E 'felhom-controller|traefik|filebrowser|cloudflared'")
say(f" images:\n{img}")
lines = guest(f"docker logs --since {t0} felhom-controller 2>&1 | grep -E 'Conflict|first-boot|filebrowser compose exists|FileBrowser mounts synced|FileBrowser sync|not mounted|skipping its folder|could not leave|traefik' | grep -v DEBUG | cut -c1-240")
say(f" [{tag}] FileBrowser lines of this boot:\n{lines}")
say(f" [{tag}] Conflict lines: {lines.count('Conflict')}; first-boot WARN: {lines.count('first-boot base-stack')}; 'not mounted' lines: {lines.count('not mounted')}")
def phase_boot_before():
w.login()
boot_lines("before")
def start_sink():
"""The event sink: a tiny HTTP server in the guest, event bodies only. hub.enabled stays false, so nothing else moves. /tmp is a
tmpfs here, so a guest reboot takes file and process away: it is written again each time."""
sink = f"""
import http.server, json
class H(http.server.BaseHTTPRequestHandler):
def do_POST(self):
n = int(self.headers.get('Content-Length') or 0); b = self.rfile.read(n)
if self.path.endswith('/event'):
open({SINK_FILE!r}, 'ab').write(b + b'\\n')
self.send_response(200); self.send_header('Content-Type', 'application/json'); self.end_headers(); self.wfile.write(b'{{}}')
def do_GET(self):
self.send_response(404); self.end_headers()
def log_message(self, *a): pass
http.server.HTTPServer(('0.0.0.0', {SINK_PORT}), H).serve_forever()
"""
return guest(f"cat > /tmp/sec1102-sink.py <<'EOF'\n{sink}\nEOF\nrm -f {SINK_FILE}; pgrep -f 'sec1102-sin[k]' >/dev/null || (nohup python3 /tmp/sec1102-sink.py >/dev/null 2>&1 &)\nsleep 1; curl -s -o /dev/null -w '%{{http_code}}' -X POST http://127.0.0.1:{SINK_PORT}/ping; echo")
def phase_switch():
say(guest("docker ps --format '{{.Names}} {{.Image}} {{.Status}}'; cat /etc/felhom-controller-image; "
"python3 - <<'PY'\nimport json\ns=json.load(open('/var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json'))\n"
"print('holds as found:', sorted((s.get('restore_holds') or {}).keys()))\nPY"))
tar = os.environ["IMG_TAR"]
w.sh(["scp", "-q", tar, f"{w._T['ssh']}:/tmp/sec1102-ctl.tar.gz"], timeout=300)
w.sh(w.ssh_args() + [f"pct push {w.GUEST} /tmp/sec1102-ctl.tar.gz /tmp/sec1102-ctl.tar.gz && rm -f /tmp/sec1102-ctl.tar.gz"], timeout=300)
say(guest("zcat /tmp/sec1102-ctl.tar.gz | docker load | tail -1; rm -f /tmp/sec1102-ctl.tar.gz", timeout=400))
start_sink()
ip = guest("ip -4 -o addr show eth0 | awk '{print $4}' | cut -d/ -f1").strip()
say(f" sink on {ip}:{SINK_PORT}")
say(guest(f"""set -e
test -f {SAVE} || cp -p {YAML} {SAVE}
python3 - <<'PY'
import re
p = {YAML!r}; s = open(p).read()
s = re.sub(r'(^hub:\\n\\s+api_key: )""', r'\\g<1>"sec1102-test"', s, count=1, flags=re.M)
s = re.sub(r'(^hub:(?:\\n\\s+.*)*?\\n\\s+url: )""', r'\\g<1>"http://{ip}:{SINK_PORT}"', s, count=1, flags=re.M)
open(p, 'w').write(s)
PY
sed -n '/^hub:/,/url:/p' {YAML} | sed -E 's/api_key: .*/api_key: (set)/'
echo {IMG_TEST} > /etc/felhom-controller-image
systemctl restart felhom-controller-bootstrap.service
sleep 8; docker ps --format '{{{{.Names}}}} {{{{.Image}}}} {{{{.Status}}}}' | grep felhom-controller"""))
say(f" login: {wait_login(30)}")
say(guest("docker logs felhom-controller 2>&1 | grep -E 'Notifier enabled|Notifier disabled' | tail -2 | cut -c1-200; docker exec felhom-controller felhom-controller --version | cut -c1-120"))
def phase_boot_stage():
"""The race needs a boot where the sync RECREATES: both boots above found nothing changed, so no race (0 Conflict before too).
A comment appended to FileBrowser's compose makes the file differ from what the sync renders, as after an update."""
w.login()
# Stage 2 (STOP_FB=1): the file manager also stays down through the boot, so the base stack's bring-up starts it too: both paths.
stop = "docker stop filebrowser >/dev/null; " if os.environ.get("STOP_FB") else ""
say(guest(stop + "echo '# sec1102 stage' >> /opt/docker/stacks/filebrowser/docker-compose.yml; tail -1 /opt/docker/stacks/filebrowser/docker-compose.yml; cat /etc/felhom-controller-image"))
boot_lines(os.environ.get("BOOT_TAG", "stage"))
def phase_boot_after():
w.login()
boot_lines("after")
code, d = w.ctl("GET", "/api/health")
say(f" /api/health {code}")
def phase_r1347():
w.login()
drill_imgs = w.images_of(open(f"{DRILL}/templates/{APP}/docker-compose.yml").read())
say(f" drill's {APP}: {drill_imgs}")
svc, frm = next(iter(drill_imgs.items()))
if w.stack(APP).get("deployed"):
w.remove(APP)
w.refresh_box_mirror()
w.sync_rescan()
assert w.deploy(APP, SUB), "the install did not complete"
w.wait_app(SUB, "/", tries=60)
fx = fb.FIXTURES[APP]
tok = fx.seed(w, SUB, say)
say(f" seeded: {bool(tok)}; read back: {tok is not None and fx.verify(w, SUB, tok, say)}")
json.dump({"tok": tok, "generated": getattr(w, "GENERATED", {})}, open(STATE, "w"))
os.chmod(STATE, 0o600)
vols = guest(f"docker volume ls --format '{{{{.Name}}}}' | grep '^{APP}_'").split()
say(f" volumes: {vols}")
guest(f"docker rm -f {BLOCKER} 2>/dev/null")
commit = w.drill_bump(APP, frm, "busybox:1.36.1")
say(f" drill commit {commit}")
say(f" box read TO after {w.sync_rescan(APP, 'busybox:1.36.1', tries=40, delay=10)} s")
t0 = now()
# The undo must fail for a hold to exist. A volume held by a created-only container does not stop it (tried first: the undo
# empties the volume and pours), so the undo copy is cut off, the shape of a helper killed mid-copy: its finished-marker goes
# while the update verifies, and the undo refuses to pour („NOTHING is put back", TestUndo_CutOffCopyIsRefusedBeforeAnythingMoves).
def cut_copy():
for _ in range(120):
time.sleep(3)
names = guest(f"docker volume ls --format '{{{{.Name}}}}' | grep '^{APP}_.*\\.pre-update-'").split()
if names:
time.sleep(15)
say(" the undo copy cut off: " + names[0] + " " + guest(f"docker run --rm -v {names[0]}:/c busybox:1.36.1 sh -c 'rm -f /c/{MARKER} && ls -a /c | tr \"\\n\" \" \"'"))
return
cutter = threading.Thread(target=cut_copy)
cutter.start()
u = w.press_update(APP, poll=2.0, cap_s=1500)
cutter.join(30)
say(f" update: accepted={u.get('accepted')} final={u.get('final_phase')} err={str(u.get('update_error'))[:200]}")
time.sleep(10)
st = w.stack(APP)
say(f" after the update: state={st.get('state')} hold={st.get('hold_reason')!r} phase={st.get('update_phase')} key={st.get('update_error_key')}")
say(" controller log:\n" + ctl_log(t0, f"update {APP}|{APP}: .*(HOLD|hold|undo)|restore hold|Failed to remove"))
flush("r1347-hold")
return phase_r1347b()
def hold_code():
"""The hold's own code, from the box's settings: the API's hold_reason is the household's sentence."""
return guest(f"python3 -c \"import json;s=json.load(open('{VOL}/data/settings.json'));print((s.get('restore_holds') or {{}}).get('{APP}',{{}}).get('reason',''))\"").strip()
def phase_r1347b():
w.login()
code = hold_code()
say(f" the hold in the box's settings: {code!r}")
if code != "update_failed":
say(" STOP: no update hold — the stage did not work; nothing to prove")
return 1
fx = fb.FIXTURES[APP]
tok = None
try:
tok = (json.load(open(STATE)) or {}).get("tok")
except (OSError, ValueError):
pass
t0 = now()
res = w.restore(APP)
say(f" restore: {json.dumps({k: res.get(k) for k in ('ok', 'snapshot_id', 'http', 'seconds', 'state_after', 'hold_after')})}")
time.sleep(10)
st = w.stack(APP)
say(f" after the restore: state={st.get('state')} hold={st.get('hold_reason')!r} phase={st.get('update_phase')} update_error={st.get('update_error')!r} key={st.get('update_error_key')!r}")
say(" controller log:\n" + ctl_log(t0, "update hold|R-1347|update-held|undo cop|restore completed"))
say(f" the seed reads back: {tok is not None and fx.verify(w, SUB, tok, say)}")
subprocess.run(["git", "-C", DRILL, "fetch", "-q", "live"], check=False)
subprocess.run(["git", "-C", DRILL, "reset", "-q", "--hard", "live/main"], check=False)
subprocess.run(["git", "-C", DRILL, "push", "-q", "-f", "origin", "main"], check=False)
w.refresh_box_mirror()
w.sync_rescan()
# the consequence: the crash-loop watcher watches this app again (D53's recipe: seven kill -9, 15 s apart)
t0 = now()
for i in range(10):
pid = guest(f"docker inspect -f '{{{{.State.Pid}}}}' {APP} 2>/dev/null").strip()
if pid and pid != "0":
guest(f"kill -9 {pid}")
say(f" kill -9 #{i + 1} pid {pid}; state {w.stack(APP).get('state')} hold {w.stack(APP).get('hold_reason')!r}")
if w.stack(APP).get("hold_reason"):
break
time.sleep(15)
time.sleep(35)
st = w.stack(APP)
code = hold_code()
say(f" after the kills: state={st.get('state')} hold code={code!r}")
say(" controller log:\n" + ctl_log(t0, "crash_loop|STOPPING|unhealthy"))
return 0 if code == "unhealthy_stop" else 1
def phase_r1348():
w.login()
st0 = json.load(open(STATE))
if getattr(w, "GENERATED", None) is not None:
w.GENERATED.update(st0.get("generated") or {})
st = w.stack(APP)
say(f" before: state={st.get('state')} hold={st.get('hold_reason')!r}")
if hold_code() == "unhealthy_stop":
c, d = w.ctl("POST", f"/api/stacks/{APP}/start") # the household's Start lifts this one (D53)
say(f" Start -> {c} {str(d)[:120]}")
for _ in range(30):
time.sleep(5)
st = w.stack(APP)
if st.get("state") == "running":
break
say(f" after Start: state={st.get('state')} hold={st.get('hold_reason')!r}")
say(" sink answers: " + start_sink())
vols = guest(f"docker volume ls --format '{{{{.Name}}}}' | grep '^{APP}_'").split()
# The cut, inside the guest so it has no ssh delay: at the log line, kill the controller, hold the volume, start again.
cut = (f"timeout 400 sh -c \"docker logs -f --since 2s felhom-controller 2>&1 | grep -m1 'the restore copy is recorded' > /tmp/sec1102-cut.txt\"; "
f"docker kill felhom-controller >/dev/null; docker create --name {BLOCKER} -v {vols[0]}:/x busybox:1.36.1 true >/dev/null; "
f"sleep 2; docker start felhom-controller >/dev/null; cut -c1-200 /tmp/sec1102-cut.txt; date -u +%H:%M:%S; docker ps -a --format '{{{{.Names}}}} {{{{.Status}}}}' | grep -E 'felhom-controller|{BLOCKER}'")
out = {}
th = threading.Thread(target=lambda: out.update(cut=guest(cut, timeout=450)))
t0 = now()
th.start()
time.sleep(4)
code, snaps = w.ctl("GET", f"/api/backup/snapshots?stack={APP}")
snap = w.snapshots(APP)[0]
sid = snap.get("id") or snap.get("snapshot_id") or snap.get("short_id")
sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip()
csrf = open(f"{w.SC}/csrf{os.getpid()}.txt").read().strip()
r = w.sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}", "-X", "POST",
"--data-urlencode", f"_csrf={csrf}", "--data-urlencode", f"stack_name={APP}", "--data-urlencode", f"snapshot_id={sid}",
f"{w.BASE}/backup/restore"], timeout=120)
say(f" restore pressed: {(r.stdout or '').splitlines()[:1]} snapshot {sid}")
th.join(460)
say(f" the cut:\n{out.get('cut')}")
say(f" login after the cut: {wait_login(40)}")
time.sleep(30)
st = w.stack(APP)
code = hold_code()
say(f" after the start: state={st.get('state')} hold code={code!r}")
say(" controller log:\n" + ctl_log(t0, "crash recovery|could NOT be put back|restore hold|HELD|Notifier enabled"))
events = guest(f"cat {SINK_FILE} 2>/dev/null | cut -c1-1400")
say(f" the sink's events:\n{events}")
mailed = [json.loads(l) for l in events.splitlines() if l.strip().startswith("{") and "put back" in l]
say(f" R-1348: operator events with the sentence: {len(mailed)}")
return 0 if code == "restore_mixed" and mailed else 1
def phase_teardown():
w.login()
say(" blocker: " + guest(f"docker rm -f {BLOCKER} 2>&1 | tail -1"))
st = w.stack(APP)
say(f" {APP}: deployed={st.get('deployed')} state={st.get('state')} hold={st.get('hold_reason')!r}")
if st.get("deployed"):
w.remove(APP)
say(guest(f"""cp -p {SAVE} {YAML} && cmp {SAVE} {YAML} && echo YAML-RESTORED-IDENTICAL; rm -f {SAVE}
pkill -f 'sec1102-sin[k]' || true; rm -f /tmp/sec1102-sink.py /tmp/sec1102-cut.txt
cp {SINK_FILE} /tmp/sec1102-sink-keep.jsonl 2>/dev/null; rm -f {SINK_FILE}
echo {IMG_FOUND} > /etc/felhom-controller-image
systemctl restart felhom-controller-bootstrap.service; sleep 10
docker image rm {IMG_TEST} 2>&1 | tail -1 | cut -c1-80
docker ps --format '{{{{.Names}} }} {{{{.Image}}}} {{{{.Status}}}}'
ls /opt/docker/stacks/{APP}/app.yaml 2>&1 | tail -1; docker volume ls --format '{{{{.Name}}}}' | grep '^{APP}_' || echo no-volume
python3 - <<'PY'
import json
s=json.load(open('{VOL}/data/settings.json'))
print('holds left:', sorted((s.get('restore_holds') or {{}}).keys()))
PY"""))
def main():
ph = sys.argv[1]
rc = {"boot-before": phase_boot_before, "switch": phase_switch, "boot-after": phase_boot_after, "boot-stage": phase_boot_stage, "r1347": phase_r1347, "r1347b": phase_r1347b,
"r1348": phase_r1348, "teardown": phase_teardown}[ph]() or 0
flush(ph)
say(f"PHASE {ph} rc={rc}")
return rc
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,55 @@
00:50:13 drill's radicale: {'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:50:14 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
00:50:46 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': [], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt sa
00:50:54 [X] after remove: deployed=False leftovers=''
00:51:03 [5] the box's drill mirror now at c760fca93
00:51:07 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
00:51:07 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
00:51:32 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:51:34 radicale: MKCALENDAR /csalad/felhom-ae0c3b2b/ http=401
00:51:34 seeded: False; read back: False
00:51:37 volumes: ['radicale_radicale_data']
00:51:41 [5] drill commit b10e19151ba8: radicale ghcr.io/kozea/radicale:3.8.1 -> busybox:1.36.1 (push rc=0)
00:51:50 [5] the box's drill mirror now at b10e19151
00:51:50 drill commit b10e19151ba8
00:51:54 box read TO after 4.5 s
00:51:54 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
00:51:54 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None
00:51:56 + 2.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None
00:51:58 + 4.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
00:52:00 + 6.1s phase=verifying label=Működés ellenőrzése… err=None hold=None
00:52:19 the undo copy cut off: radicale_radicale_data.pre-update-20261102T235158Z . .. data
00:57:02 + 307.7s phase=undoing label=Visszaállítás az előző változatra… err=None hold=None
00:57:04 + 309.7s phase=failed label=A frissítés nem sikerült err=A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-11-03 00:51 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza. hold=A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-11-03 00:51 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.
00:57:04 update: accepted=True final=failed err=A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás ne
00:57:14 after the update: state=stopped hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-11-03 00:51 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.' phase=failed key=None
00:57:17 controller log:
2026/11/02 23:51:54 update.go:556: [INFO] [stacks] update radicale: accepted — guarded update started
2026/11/02 23:51:54 update.go:1397: [INFO] [stacks] update radicale: phase checking
2026/11/02 23:51:54 update.go:814: [INFO] [stacks] update radicale: ladder — the installed version radicale=ghcr.io/kozea/radicale:3.8.1 is AT THE HEAD of the update_ladder (1 entries) — the catalog's current definition
2026/11/02 23:51:54 update_guard.go:278: [INFO] [backup] update precondition for radicale: the own unit holds no data yet (no backup run has confirmed it) — not a copy
2026/11/02 23:51:54 update.go:844: [INFO] [stacks] update radicale: no usable copy on any tier — younger than 24h0m0s and not older than this install's deploy (2026-11-02T23:51:07Z) (found: none) — backing up first
2026/11/02 23:51:54 update.go:1397: [INFO] [stacks] update radicale: phase backing-up
2026/11/02 23:51:56 update.go:859: [INFO] [stacks] update radicale: precondition met after the backup — Tier 2 (second drive) copy from 2026-11-02T23:51:55Z
2026/11/02 23:51:56 update.go:1397: [INFO] [stacks] update radicale: phase safety-dump
2026/11/02 23:51:56 update.go:874: [INFO] [stacks] update radicale: safety dump done (0 file(s)) []
2026/11/02 23:51:57 undo.go:287: [INFO] [stacks] update radicale: the undo copy will hold 1 named volume(s), 0.0 MiB
2026/11/02 23:51:57 update.go:1397: [INFO] [stacks] update radicale: phase pinning
2026/11/02 23:51:57 pin.go:385: [INFO] [stacks] update radicale: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/radicale/docker-compose.yml (radicale=busybox:1.36.1)
2026/11/02 23:51:57 update.go:1397: [INFO] [stacks] update radicale: phase pulling
2026/11/02 23:51:58 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:51:58 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:51:59 undo.go:437: [INFO] [stacks] update radicale: copied radicale_radicale_data → radicale_radicale_data.pre-update-20261102T235158Z in 543ms
2026/11/02 23:51:59 update.go:1397: [INFO] [stacks] update radicale: phase starting
2026/11/02 23:51:59 update.go:1397: [INFO] [stacks] update radicale: phase verifying
2026/11/02 23:57:01 update.go:1096: [ERROR] [stacks] update radicale FAILED after the new version was started: not healthy: not healthy within 5m0s (last: state restarting)
2026/11/02 23:57:02 update.go:1088: [INFO] [stacks] update radicale: kept 868 bytes of the app's own log at /opt/docker/stacks/radicale/hold-logs/20261102T235701Z/compose-logs.txt before stopping it (R-621)
2026/11/02 23:57:02 update.go:1397: [INFO] [stacks] update radicale: phase undoing
2026/11/02 23:57:02 undo.go:515: [WARN] [stacks] update radicale: UNDO — putting back the previous version and its 1 volume copy(ies) (reason: not healthy: not healthy within 5m0s (last: state restarting))
2026/11/02 23:57:02 undo.go:521: [ERROR] [stacks] update radicale: the undo copy radicale_radicale_data.pre-update-20261102T235158Z has no finished-marker — it is cut off or missing; NOTHING is put back
2026/11/02 23:57:02 update.go:1108: [ERROR] [stacks] update radicale: the UNDO failed too (untouched) — HOLDING the app; the undo copies are kept: [{radicale_radicale_data radicale_radicale_data.pre-update-20261102T235158Z}]
2026/11/02 23:57:02 settings.go:2101: [WARN] [settings] restore hold SET for radicale — the app stays stopped until it is cleared
2026/11/02 23:57:02 settings.go:2101: [WARN] [settings] restore hold SET for radicale — the app stays stopped until it is cleared
2026/11/02 23:57:02 unattended.go:527: [INFO] [stacks] update radicale: step radicale=busybox:1.36.1 recorded as held — the automatic leg will not press it again until the catalog's ladder changes (ladder 3c58efa0dee4c41c)
2026/11/02 23:57:02 undo.go:639: [INFO] [stacks] update radicale: event app_update_held (hold recorded: true)
@@ -0,0 +1,56 @@
00:50:13 drill's radicale: {'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:50:14 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
00:50:46 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': [], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt sa
00:50:54 [X] after remove: deployed=False leftovers=''
00:51:03 [5] the box's drill mirror now at c760fca93
00:51:07 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
00:51:07 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
00:51:32 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:51:34 radicale: MKCALENDAR /csalad/felhom-ae0c3b2b/ http=401
00:51:34 seeded: False; read back: False
00:51:37 volumes: ['radicale_radicale_data']
00:51:41 [5] drill commit b10e19151ba8: radicale ghcr.io/kozea/radicale:3.8.1 -> busybox:1.36.1 (push rc=0)
00:51:50 [5] the box's drill mirror now at b10e19151
00:51:50 drill commit b10e19151ba8
00:51:54 box read TO after 4.5 s
00:51:54 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
00:51:54 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None
00:51:56 + 2.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None
00:51:58 + 4.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
00:52:00 + 6.1s phase=verifying label=Működés ellenőrzése… err=None hold=None
00:52:19 the undo copy cut off: radicale_radicale_data.pre-update-20261102T235158Z . .. data
00:57:02 + 307.7s phase=undoing label=Visszaállítás az előző változatra… err=None hold=None
00:57:04 + 309.7s phase=failed label=A frissítés nem sikerült err=A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-11-03 00:51 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza. hold=A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-11-03 00:51 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.
00:57:04 update: accepted=True final=failed err=A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás ne
00:57:14 after the update: state=stopped hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-11-03 00:51 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.' phase=failed key=None
00:57:17 controller log:
2026/11/02 23:51:54 update.go:556: [INFO] [stacks] update radicale: accepted — guarded update started
2026/11/02 23:51:54 update.go:1397: [INFO] [stacks] update radicale: phase checking
2026/11/02 23:51:54 update.go:814: [INFO] [stacks] update radicale: ladder — the installed version radicale=ghcr.io/kozea/radicale:3.8.1 is AT THE HEAD of the update_ladder (1 entries) — the catalog's current definition
2026/11/02 23:51:54 update_guard.go:278: [INFO] [backup] update precondition for radicale: the own unit holds no data yet (no backup run has confirmed it) — not a copy
2026/11/02 23:51:54 update.go:844: [INFO] [stacks] update radicale: no usable copy on any tier — younger than 24h0m0s and not older than this install's deploy (2026-11-02T23:51:07Z) (found: none) — backing up first
2026/11/02 23:51:54 update.go:1397: [INFO] [stacks] update radicale: phase backing-up
2026/11/02 23:51:56 update.go:859: [INFO] [stacks] update radicale: precondition met after the backup — Tier 2 (second drive) copy from 2026-11-02T23:51:55Z
2026/11/02 23:51:56 update.go:1397: [INFO] [stacks] update radicale: phase safety-dump
2026/11/02 23:51:56 update.go:874: [INFO] [stacks] update radicale: safety dump done (0 file(s)) []
2026/11/02 23:51:57 undo.go:287: [INFO] [stacks] update radicale: the undo copy will hold 1 named volume(s), 0.0 MiB
2026/11/02 23:51:57 update.go:1397: [INFO] [stacks] update radicale: phase pinning
2026/11/02 23:51:57 pin.go:385: [INFO] [stacks] update radicale: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/radicale/docker-compose.yml (radicale=busybox:1.36.1)
2026/11/02 23:51:57 update.go:1397: [INFO] [stacks] update radicale: phase pulling
2026/11/02 23:51:58 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:51:58 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:51:59 undo.go:437: [INFO] [stacks] update radicale: copied radicale_radicale_data → radicale_radicale_data.pre-update-20261102T235158Z in 543ms
2026/11/02 23:51:59 update.go:1397: [INFO] [stacks] update radicale: phase starting
2026/11/02 23:51:59 update.go:1397: [INFO] [stacks] update radicale: phase verifying
2026/11/02 23:57:01 update.go:1096: [ERROR] [stacks] update radicale FAILED after the new version was started: not healthy: not healthy within 5m0s (last: state restarting)
2026/11/02 23:57:02 update.go:1088: [INFO] [stacks] update radicale: kept 868 bytes of the app's own log at /opt/docker/stacks/radicale/hold-logs/20261102T235701Z/compose-logs.txt before stopping it (R-621)
2026/11/02 23:57:02 update.go:1397: [INFO] [stacks] update radicale: phase undoing
2026/11/02 23:57:02 undo.go:515: [WARN] [stacks] update radicale: UNDO — putting back the previous version and its 1 volume copy(ies) (reason: not healthy: not healthy within 5m0s (last: state restarting))
2026/11/02 23:57:02 undo.go:521: [ERROR] [stacks] update radicale: the undo copy radicale_radicale_data.pre-update-20261102T235158Z has no finished-marker — it is cut off or missing; NOTHING is put back
2026/11/02 23:57:02 update.go:1108: [ERROR] [stacks] update radicale: the UNDO failed too (untouched) — HOLDING the app; the undo copies are kept: [{radicale_radicale_data radicale_radicale_data.pre-update-20261102T235158Z}]
2026/11/02 23:57:02 settings.go:2101: [WARN] [settings] restore hold SET for radicale — the app stays stopped until it is cleared
2026/11/02 23:57:02 settings.go:2101: [WARN] [settings] restore hold SET for radicale — the app stays stopped until it is cleared
2026/11/02 23:57:02 unattended.go:527: [INFO] [stacks] update radicale: step radicale=busybox:1.36.1 recorded as held — the automatic leg will not press it again until the catalog's ladder changes (ladder 3c58efa0dee4c41c)
2026/11/02 23:57:02 undo.go:639: [INFO] [stacks] update radicale: event app_update_held (hold recorded: true)
00:57:17 STOP: no update hold — the stage did not work; nothing to prove
@@ -0,0 +1,29 @@
00:57:45 the hold in the box's settings: 'update_failed'
00:57:45 [R] restoring radicale from snapshot 'helyi' (of 1 offered)
00:57:45 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
00:57:45 + 0.0s restore (True, None, None)
00:58:48 + 62.8s restore (False, None, None)
00:58:48 [R] after restore: state=running hold=None phase=None
00:58:51 restore: {"ok": true, "snapshot_id": "helyi", "http": "HTTP/2 302", "seconds": 62.8, "state_after": "running", "hold_after": null}
00:59:01 after the restore: state=running hold=None phase=None update_error=None key=None
00:59:05 controller log:
2026/11/02 23:57:47 update_guard.go:676: [INFO] [backup] radicale: restore completed — the update hold (set 2026-11-02T23:57:02Z) is CLEARED
2026/11/02 23:57:47 update_guard.go:681: [INFO] [backup] radicale: the restored app is watched and updated again — its update-held flag is forgotten (R-1347)
2026/11/02 23:57:47 update_guard.go:690: [INFO] [backup] radicale: removed 1 undo cop(y/ies) the lifted update hold had kept (R-671)
00:59:05 the seed reads back: False
00:59:14 [5] the box's drill mirror now at c760fca93
00:59:25 kill -9 #1 pid 19447; state running hold None
00:59:46 kill -9 #2 pid 21078; state starting hold None
01:00:09 kill -9 #3 pid 21373; state running hold None
01:00:31 kill -9 #4 pid 21805; state running hold None
01:00:52 kill -9 #5 pid 22187; state running hold None
01:01:14 kill -9 #6 pid 22548; state running hold None
01:01:33 kill -9 #7 pid ; state stopped hold 'Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.'
01:02:11 after the kills: state=stopped hold code='unhealthy_stop'
01:02:15 controller log:
2026/11/03 00:01:19 main.go:4354: [WARN] [deadapp] radicale: crash_loop — 6 in 10m0s; STOPPING it (decision 28)
2026/11/03 00:01:19 update_guard.go:885: [WARN] [backup] radicale is STOPPED by the box: crash_loop (trip 1 within 24h0m0s) — Start gives it one more try (decision 28)
2026/11/03 00:01:25 notifier.go:363: [WARN] Event push failed after 3 attempts (app_stopped_unhealthy/warning): Post "http://192.168.0.158:18091/api/v1/event": dial tcp 192.168.0.158:18091: connect: connection refused
2026/11/03 00:01:49 holdreport.go:169: [INFO] [holdreport] radicale held (unhealthy_stop since 2026-11-03T00:01:19Z) — reported to the hub
@@ -0,0 +1,18 @@
2026/11/03 00:01:19 settings.go:2101: [WARN] [settings] restore hold SET for radicale — the app stays stopped until it is cleared
2026/11/03 00:03:08 restore_crash_putback.go:89: [INFO] [backup] radicale: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195)
2026/11/03 00:03:11 appstop_marker.go:417: [WARN] [appstop] crash recovery: a local restore (op "unit-restore:radicale") was interrupted and left 1 app(s) stopped — restarting them: [radicale]
2026/11/03 00:03:11 appstop_marker.go:444: [ERROR] [appstop] crash recovery: putting radicale back from /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261103T000308Z FAILED: failed to restore 1 volume(s): [radicale_radicale_data] — NOT starting it
2026/11/03 00:03:11 settings.go:2101: [WARN] [settings] restore hold SET for radicale — the app stays stopped until it is cleared
2026/11/03 00:03:11 appstop_putback_wait.go:128: [ERROR] [appstop] radicale: the interrupted restore could NOT be put back (failed to restore 1 volume(s): [radicale_radicale_data]) — the app is HELD for support, the copy stays at /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261103T000308Z (R-1251 H2)
2026/11/03 00:03:11 notifier.go:81: [INFO] Notifier enabled (hub: http://192.168.0.158:18091)
2026/11/03 00:03:11 notifier.go:357: [INFO] Event pushed: backup_run_failures (error) [hu-only] — App "radicale" is HELD STOPPED for support: a restore cut by a stop could not be put back at the next start — putting the interrupted restore's copy back failed: failed to restore 1 volume(s): [radicale_radicale_data]. The app may be on a mix of the backup's and the live data. The copy of the live data is kept at /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261103T000308Z. The app will not start from any path until the hold is cleared (the hub's Clear hold button, or on the box: pct exec <vmid> -- docker exec felhom-controller felhom-controller --clear-restore-hold <app>).
2026/11/03 00:03:26 bootrecon.go:236: [INFO] [bootrecon] "radicale" is a boot orphan by intent but is HELD (held after a restore was interrupted and its live data could not be put back at the next start (2026-11-03T00:03:11Z) — clear the hold to start it) — NOT starting it; whatever is holding it owns its recovery
---
{"customer_id":"demo-hp","event_type":"restore_interrupted","severity":"warning","message":"A(z) radicale visszaállítása megszakadt, mert a doboz újraindult — indítsd el újra.","details":{"op":"restore","stack":"radicale"}}
{"customer_id":"demo-hp","event_type":"backup_failed","severity":"error","message":"Megszakadt egy helyi visszaállítás, mert a vezérlő újraindult. 0 leállva maradt alkalmazás újraindult.","details":{"error":"op=unit-restore:radicale reason=unit_restore started_at=2026-11-03T00:03:07Z restarted=[] put_back_failed_held=[radicale]"}}
{"customer_id":"demo-hp","event_type":"backup_run_failures","severity":"error","message":"App \"radicale\" is HELD STOPPED for support: a restore cut by a stop could not be put back at the next start — putting the interrupted restore's copy back failed: failed to restore 1 volume(s): [radicale_radicale_data]. The app may be on a mix of the backup's and the live data. The copy of the live data is kept at /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261103T000308Z. The app will not start from any path until the hold is cleared (the hub's Clear hold button, or on the box: pct exec \u003cvmid\u003e -- docker exec felhom-controller felhom-controller --clear-restore-hold \u003capp\u003e).","details":{"run_kind":"offsite-reconstitute","failed":1,"attempted":1,"space_known":false,"apps":[{"app":"radicale","leg":"restore-hold-mixed","reason":"interrupted restore: put-back failed"}]}}
{"customer_id":"demo-hp","event_type":"controller_started","severity":"info","message":"Controller elindult (0.350.0-rc.1-sec1102fault)","details":{"selftest_fail":0,"selftest_pass":8,"selftest_warn":0}}
---
{'reason': 'restore_mixed', 'mixed_cause': 'interrupted_put_back', 'failed_step': "putting the interrupted restore's copy back failed: failed to restore 1 volume(s): [radicale_radicale_data]", 'safety_dump': '/mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261103T000308Z', 'at': '2026-11-03T00:03:11Z'}
sec1102-blocker Created
felhom-controller Up 53 seconds (healthy)
@@ -0,0 +1,16 @@
00:38:58 pct reboot 9202 rc=0
00:39:23 login after the reboot: True
00:40:12 images:
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault Up About a minute (healthy)
traefik traefik:v3.7.17 Up About a minute
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
00:40:15 [after] FileBrowser lines of this boot:
f9c43ce65f4e traefik traefik traefik:v3.7.17
2026/11/02 23:39:03 infra.go:401: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/02 23:39:04 handlers.go:3974: [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (11 storage path(s))
00:40:15 [after] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 0
00:40:15 /api/health 200
00:40:15 PHASE boot-after rc=0
@@ -0,0 +1,24 @@
00:35:53 pct reboot 9202 rc=0
00:36:18 login after the reboot: True
00:37:07 images:
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
00:37:10 [before] FileBrowser lines of this boot:
e27bdddefe5e traefik traefik traefik:v3.7.16
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/immich not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/calibre-web not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/grimmory not mounted — skipping userdata skeleton
2026/11/02 23:35:59 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/metube not mounted — skipping userdata skeleton
2026/11/02 23:35:59 infra.go:378: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/02 23:36:00 handlers.go:3958: [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (11 storage path(s))
00:37:10 [before] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 9
00:37:10 PHASE boot-before rc=0
@@ -0,0 +1,19 @@
01:16:14 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
01:16:29 pct reboot 9202 rc=0
01:16:54 login after the reboot: True
01:17:43 images:
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault Up About a minute (healthy)
traefik traefik:v3.7.17 Up About a minute
01:17:46 [after-both-paths] FileBrowser lines of this boot:
4ce334822229 traefik traefik traefik:v3.7.17
2026/11/03 00:16:34 infra.go:401: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/03 00:16:34 infra.go:285: [INFO] [infra] filebrowser compose exists — starting without regenerating
2026/11/03 00:16:36 handlers.go:3967: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
01:17:46 [after-both-paths] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 0
01:17:46 PHASE boot-stage rc=0
@@ -0,0 +1,28 @@
01:12:54 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
01:13:59 pct reboot 9202 rc=0
01:14:24 login after the reboot: True
01:15:13 images:
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
01:15:16 [before-both-paths] FileBrowser lines of this boot:
fd1fbea08cb8 traefik traefik traefik:v3.7.16
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/immich not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/calibre-web not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/grimmory not mounted — skipping userdata skeleton
2026/11/03 00:14:04 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/metube not mounted — skipping userdata skeleton
2026/11/03 00:14:04 infra.go:378: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/03 00:14:04 infra.go:265: [INFO] [infra] filebrowser compose exists — starting without regenerating
2026/11/03 00:14:06 handlers.go:3951: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
01:15:16 [before-both-paths] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 9
01:15:16 PHASE boot-stage rc=0
@@ -0,0 +1,18 @@
00:40:38 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
00:41:44 pct reboot 9202 rc=0
00:42:09 login after the reboot: True
00:42:57 images:
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault Up About a minute (healthy)
traefik traefik:v3.7.17 Up About a minute
00:43:01 [after-stage] FileBrowser lines of this boot:
f9c43ce65f4e traefik traefik traefik:v3.7.17
2026/11/02 23:41:49 infra.go:401: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/02 23:41:50 handlers.go:3967: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
00:43:01 [after-stage] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 0
00:43:01 PHASE boot-stage rc=0
@@ -0,0 +1,27 @@
01:05:36 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
01:06:42 pct reboot 9202 rc=0
01:07:07 login after the reboot: True
01:07:56 images:
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
01:07:59 [before-stage] FileBrowser lines of this boot:
fd1fbea08cb8 traefik traefik traefik:v3.7.16
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/navidrome not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/immich not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/calibre-web not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/grimmory not mounted — skipping userdata skeleton
2026/11/03 00:06:47 handlers.go:4019: [INFO] [web] FileBrowser: drive /mnt/felhom-drives/scratch_hdd/userdata/metube not mounted — skipping userdata skeleton
2026/11/03 00:06:47 infra.go:378: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/03 00:06:48 handlers.go:3951: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
01:07:59 [before-stage] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 9
01:07:59 PHASE boot-stage rc=0
@@ -0,0 +1,19 @@
01:16:14 # sec1102 stage
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
01:16:29 pct reboot 9202 rc=0
01:16:54 login after the reboot: True
01:17:43 images:
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault Up About a minute (healthy)
traefik traefik:v3.7.17 Up About a minute
01:17:46 [after-both-paths] FileBrowser lines of this boot:
4ce334822229 traefik traefik traefik:v3.7.17
2026/11/03 00:16:34 infra.go:401: [INFO] [infra] connected felhom-controller to traefik-public
2026/11/03 00:16:34 infra.go:285: [INFO] [infra] filebrowser compose exists — starting without regenerating
2026/11/03 00:16:36 handlers.go:3967: [INFO] [web] FileBrowser mounts synced (recreated) — 11 storage path(s), config updated
01:17:46 [after-both-paths] Conflict lines: 0; first-boot WARN: 0; 'not mounted' lines: 0
01:17:46 PHASE boot-stage rc=0
@@ -0,0 +1,57 @@
00:50:13 drill's radicale: {'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:50:14 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
00:50:46 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': [], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt sa
00:50:54 [X] after remove: deployed=False leftovers=''
00:51:03 [5] the box's drill mirror now at c760fca93
00:51:07 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
00:51:07 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
00:51:32 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:51:34 radicale: MKCALENDAR /csalad/felhom-ae0c3b2b/ http=401
00:51:34 seeded: False; read back: False
00:51:37 volumes: ['radicale_radicale_data']
00:51:41 [5] drill commit b10e19151ba8: radicale ghcr.io/kozea/radicale:3.8.1 -> busybox:1.36.1 (push rc=0)
00:51:50 [5] the box's drill mirror now at b10e19151
00:51:50 drill commit b10e19151ba8
00:51:54 box read TO after 4.5 s
00:51:54 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
00:51:54 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None
00:51:56 + 2.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None
00:51:58 + 4.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
00:52:00 + 6.1s phase=verifying label=Működés ellenőrzése… err=None hold=None
00:52:19 the undo copy cut off: radicale_radicale_data.pre-update-20261102T235158Z . .. data
00:57:02 + 307.7s phase=undoing label=Visszaállítás az előző változatra… err=None hold=None
00:57:04 + 309.7s phase=failed label=A frissítés nem sikerült err=A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-11-03 00:51 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza. hold=A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-11-03 00:51 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.
00:57:04 update: accepted=True final=failed err=A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás ne
00:57:14 after the update: state=stopped hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) radicale frissítése 2026-11-03 00:57-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: második meghajtó, 2026-11-03 00:51 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.' phase=failed key=None
00:57:17 controller log:
2026/11/02 23:51:54 update.go:556: [INFO] [stacks] update radicale: accepted — guarded update started
2026/11/02 23:51:54 update.go:1397: [INFO] [stacks] update radicale: phase checking
2026/11/02 23:51:54 update.go:814: [INFO] [stacks] update radicale: ladder — the installed version radicale=ghcr.io/kozea/radicale:3.8.1 is AT THE HEAD of the update_ladder (1 entries) — the catalog's current definition
2026/11/02 23:51:54 update_guard.go:278: [INFO] [backup] update precondition for radicale: the own unit holds no data yet (no backup run has confirmed it) — not a copy
2026/11/02 23:51:54 update.go:844: [INFO] [stacks] update radicale: no usable copy on any tier — younger than 24h0m0s and not older than this install's deploy (2026-11-02T23:51:07Z) (found: none) — backing up first
2026/11/02 23:51:54 update.go:1397: [INFO] [stacks] update radicale: phase backing-up
2026/11/02 23:51:56 update.go:859: [INFO] [stacks] update radicale: precondition met after the backup — Tier 2 (second drive) copy from 2026-11-02T23:51:55Z
2026/11/02 23:51:56 update.go:1397: [INFO] [stacks] update radicale: phase safety-dump
2026/11/02 23:51:56 update.go:874: [INFO] [stacks] update radicale: safety dump done (0 file(s)) []
2026/11/02 23:51:57 undo.go:287: [INFO] [stacks] update radicale: the undo copy will hold 1 named volume(s), 0.0 MiB
2026/11/02 23:51:57 update.go:1397: [INFO] [stacks] update radicale: phase pinning
2026/11/02 23:51:57 pin.go:385: [INFO] [stacks] update radicale: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/radicale/docker-compose.yml (radicale=busybox:1.36.1)
2026/11/02 23:51:57 update.go:1397: [INFO] [stacks] update radicale: phase pulling
2026/11/02 23:51:58 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:51:58 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:51:59 undo.go:437: [INFO] [stacks] update radicale: copied radicale_radicale_data → radicale_radicale_data.pre-update-20261102T235158Z in 543ms
2026/11/02 23:51:59 update.go:1397: [INFO] [stacks] update radicale: phase starting
2026/11/02 23:51:59 update.go:1397: [INFO] [stacks] update radicale: phase verifying
2026/11/02 23:57:01 update.go:1096: [ERROR] [stacks] update radicale FAILED after the new version was started: not healthy: not healthy within 5m0s (last: state restarting)
2026/11/02 23:57:02 update.go:1088: [INFO] [stacks] update radicale: kept 868 bytes of the app's own log at /opt/docker/stacks/radicale/hold-logs/20261102T235701Z/compose-logs.txt before stopping it (R-621)
2026/11/02 23:57:02 update.go:1397: [INFO] [stacks] update radicale: phase undoing
2026/11/02 23:57:02 undo.go:515: [WARN] [stacks] update radicale: UNDO — putting back the previous version and its 1 volume copy(ies) (reason: not healthy: not healthy within 5m0s (last: state restarting))
2026/11/02 23:57:02 undo.go:521: [ERROR] [stacks] update radicale: the undo copy radicale_radicale_data.pre-update-20261102T235158Z has no finished-marker — it is cut off or missing; NOTHING is put back
2026/11/02 23:57:02 update.go:1108: [ERROR] [stacks] update radicale: the UNDO failed too (untouched) — HOLDING the app; the undo copies are kept: [{radicale_radicale_data radicale_radicale_data.pre-update-20261102T235158Z}]
2026/11/02 23:57:02 settings.go:2101: [WARN] [settings] restore hold SET for radicale — the app stays stopped until it is cleared
2026/11/02 23:57:02 settings.go:2101: [WARN] [settings] restore hold SET for radicale — the app stays stopped until it is cleared
2026/11/02 23:57:02 unattended.go:527: [INFO] [stacks] update radicale: step radicale=busybox:1.36.1 recorded as held — the automatic leg will not press it again until the catalog's ladder changes (ladder 3c58efa0dee4c41c)
2026/11/02 23:57:02 undo.go:639: [INFO] [stacks] update radicale: event app_update_held (hold recorded: true)
00:57:17 STOP: no update hold — the stage did not work; nothing to prove
00:57:17 PHASE r1347 rc=1
@@ -0,0 +1,30 @@
00:57:45 the hold in the box's settings: 'update_failed'
00:57:45 [R] restoring radicale from snapshot 'helyi' (of 1 offered)
00:57:45 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
00:57:45 + 0.0s restore (True, None, None)
00:58:48 + 62.8s restore (False, None, None)
00:58:48 [R] after restore: state=running hold=None phase=None
00:58:51 restore: {"ok": true, "snapshot_id": "helyi", "http": "HTTP/2 302", "seconds": 62.8, "state_after": "running", "hold_after": null}
00:59:01 after the restore: state=running hold=None phase=None update_error=None key=None
00:59:05 controller log:
2026/11/02 23:57:47 update_guard.go:676: [INFO] [backup] radicale: restore completed — the update hold (set 2026-11-02T23:57:02Z) is CLEARED
2026/11/02 23:57:47 update_guard.go:681: [INFO] [backup] radicale: the restored app is watched and updated again — its update-held flag is forgotten (R-1347)
2026/11/02 23:57:47 update_guard.go:690: [INFO] [backup] radicale: removed 1 undo cop(y/ies) the lifted update hold had kept (R-671)
00:59:05 the seed reads back: False
00:59:14 [5] the box's drill mirror now at c760fca93
00:59:25 kill -9 #1 pid 19447; state running hold None
00:59:46 kill -9 #2 pid 21078; state starting hold None
01:00:09 kill -9 #3 pid 21373; state running hold None
01:00:31 kill -9 #4 pid 21805; state running hold None
01:00:52 kill -9 #5 pid 22187; state running hold None
01:01:14 kill -9 #6 pid 22548; state running hold None
01:01:33 kill -9 #7 pid ; state stopped hold 'Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.'
01:02:11 after the kills: state=stopped hold code='unhealthy_stop'
01:02:15 controller log:
2026/11/03 00:01:19 main.go:4354: [WARN] [deadapp] radicale: crash_loop — 6 in 10m0s; STOPPING it (decision 28)
2026/11/03 00:01:19 update_guard.go:885: [WARN] [backup] radicale is STOPPED by the box: crash_loop (trip 1 within 24h0m0s) — Start gives it one more try (decision 28)
2026/11/03 00:01:25 notifier.go:363: [WARN] Event push failed after 3 attempts (app_stopped_unhealthy/warning): Post "http://192.168.0.158:18091/api/v1/event": dial tcp 192.168.0.158:18091: connect: connection refused
2026/11/03 00:01:49 holdreport.go:169: [INFO] [holdreport] radicale held (unhealthy_stop since 2026-11-03T00:01:19Z) — reported to the hub
01:02:15 PHASE r1347b rc=0
@@ -0,0 +1,50 @@
01:02:52 before: state=running hold=None
01:03:00 sink answers: 200
01:03:07 restore pressed: ['HTTP/2 302 '] snapshot helyi
01:03:12 the cut:
2026/11/03 00:03:08 restore_crash_putback.go:89: [INFO] [backup] radicale: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195)
00:03:11
sec1102-blocker Created
felhom-controller Up Less than a second (health: starting)
01:03:17 login after the cut: True
01:03:51 after the start: state=stopped hold code='restore_mixed'
Traceback (most recent call last):
File "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-2026-11-02/part3/proof1102.py", line 335, in <module>
sys.exit(main())
~~~~^^
File "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-2026-11-02/part3/proof1102.py", line 327, in main
rc = {"boot-before": phase_boot_before, "switch": phase_switch, "boot-after": phase_boot_after, "boot-stage": phase_boot_stage, "r1347": phase_r1347, "r1347b": phase_r1347b,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"r1348": phase_r1348, "teardown": phase_teardown}[ph]() or 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-2026-11-02/part3/proof1102.py", line 295, in phase_r1348
say(" controller log:\n" + ctl_log(t0, "crash recovery|could NOT be put back|restore hold|HELD|Notifier enabled"))
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-2026-11-02/part3/proof1102.py", line 55, in ctl_log
return guest(f"docker logs --since {since} felhom-controller 2>&1 | grep -E '{rx}' | grep -v DEBUG | cut -c1-260 | tail -40")
File "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-2026-11-02/part3/proof1102.py", line 51, in guest
return w.guest(script, timeout=timeout)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts/box_walk.py", line 106, in guest
r = sh(ssh_args() + [
f"export LC_ALL=C; cat > {t}; pct push {GUEST} {t} {t} >/dev/null 2>&1; "
f"pct exec {GUEST} -- bash {t}; pct exec {GUEST} -- rm -f {t}; rm -f {t}"],
timeout=timeout, inp=script)
File "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts/box_walk.py", line 95, in sh
return subprocess.run(args, capture_output=True, text=True, timeout=timeout, input=inp)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/subprocess.py", line 556, in run
stdout, stderr = process.communicate(input, timeout=timeout)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/subprocess.py", line 1222, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/subprocess.py", line 2169, in _communicate
stdout = self._translate_newlines(stdout,
self.stdout.encoding,
self.stdout.errors)
File "/usr/lib/python3.13/subprocess.py", line 1099, in _translate_newlines
data = data.decode(encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 1462-1463: invalid continuation byte
@@ -0,0 +1,24 @@
01:15:34 filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up About a minute (healthy)
paperless-postgres postgres:18-alpine Up About a minute (healthy)
paperless-redis redis:7-alpine Up About a minute (healthy)
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
holds as found: ['radicale']
01:15:41 Loaded image: gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
01:15:48 sink on 192.168.0.158:18091
01:16:01 hub:
api_key: (set)
enabled: false
push_interval: 15m
url: "http://192.168.0.158:18091"
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault Up 8 seconds (healthy)
01:16:06 login: True
01:16:09 2026/11/03 00:15:52 notifier.go:81: [INFO] Notifier enabled (hub: http://192.168.0.158:18091)
felhom-controller 0.350.0-rc.1-sec1102fault (built 2026-11-02T22:24:33Z, commit c4b6844)
01:16:09 PHASE switch rc=0
@@ -0,0 +1,13 @@
01:04:37 blocker: sec1102-blocker
01:04:37 radicale: deployed=True state=stopped hold='a(z) radicale visszaállítása 2026-11-03 00:03-kor félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás biztonsági okból leállítva marad. Ehhez segítség kell: vedd fel velünk a kapcsolatot'
01:04:37 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
01:05:09 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': ['radicale_radicale_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az
01:05:18 [X] after remove: deployed=False leftovers=''
01:05:32 YAML-RESTORED-IDENTICAL
Deleted: sha256:1b2b25755d9c4ced58bfab0a49dd214d395ed9521a08a90d8811c3404407760a
ls: cannot access '/opt/docker/stacks/radicale/app.yaml': No such file or directory
no-volume
holds left: ['radicale']
01:05:32 PHASE teardown rc=0
@@ -0,0 +1,9 @@
01:17:51 blocker:
01:17:51 radicale: deployed=False state=not_deployed hold=None
01:18:05 YAML-RESTORED-IDENTICAL
Deleted: sha256:1b2b25755d9c4ced58bfab0a49dd214d395ed9521a08a90d8811c3404407760a
ls: cannot access '/opt/docker/stacks/radicale/app.yaml': No such file or directory
no-volume
holds left: ['radicale']
01:18:05 PHASE teardown rc=0
@@ -0,0 +1,46 @@
00:37:14 felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up About a minute (healthy)
paperless-postgres postgres:18-alpine Up About a minute (healthy)
paperless-redis redis:7-alpine Up About a minute (healthy)
filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
holds as found: []
00:37:23 Loaded image: gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
00:37:30 sink on 192.168.0.158:18091
00:37:42 hub:
api_key: (set)
enabled: false
push_interval: 15m
url: "http://192.168.0.158:18091"
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault Up 8 seconds (healthy)
00:37:48 login: True
00:37:51 2026/11/02 23:37:34 notifier.go:81: [INFO] Notifier enabled (hub: http://192.168.0.158:18091)
felhom-controller 0.350.0-rc.1-sec1102fault (built 2026-11-02T22:24:33Z, commit c4b6844)
01:15:34 filebrowser gtstef/filebrowser:1.5.8-stable Up About a minute (healthy)
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up About a minute (healthy)
traefik traefik:v3.7.16 Up About a minute
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up About a minute (healthy)
paperless-postgres postgres:18-alpine Up About a minute (healthy)
paperless-redis redis:7-alpine Up About a minute (healthy)
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
holds as found: ['radicale']
01:15:41 Loaded image: gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault
01:15:48 sink on 192.168.0.158:18091
01:16:01 hub:
api_key: (set)
enabled: false
push_interval: 15m
url: "http://192.168.0.158:18091"
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1-sec1102fault Up 8 seconds (healthy)
01:16:06 login: True
01:16:09 2026/11/03 00:15:52 notifier.go:81: [INFO] Notifier enabled (hub: http://192.168.0.158:18091)
felhom-controller 0.350.0-rc.1-sec1102fault (built 2026-11-02T22:24:33Z, commit c4b6844)
@@ -0,0 +1,20 @@
01:04:37 blocker: sec1102-blocker
01:04:37 radicale: deployed=True state=stopped hold='a(z) radicale visszaállítása 2026-11-03 00:03-kor félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás biztonsági okból leállítva marad. Ehhez segítség kell: vedd fel velünk a kapcsolatot'
01:04:37 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
01:05:09 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': ['radicale_radicale_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az
01:05:18 [X] after remove: deployed=False leftovers=''
01:05:32 YAML-RESTORED-IDENTICAL
Deleted: sha256:1b2b25755d9c4ced58bfab0a49dd214d395ed9521a08a90d8811c3404407760a
ls: cannot access '/opt/docker/stacks/radicale/app.yaml': No such file or directory
no-volume
holds left: ['radicale']
01:17:51 blocker:
01:17:51 radicale: deployed=False state=not_deployed hold=None
01:18:05 YAML-RESTORED-IDENTICAL
Deleted: sha256:1b2b25755d9c4ced58bfab0a49dd214d395ed9521a08a90d8811c3404407760a
ls: cannot access '/opt/docker/stacks/radicale/app.yaml': No such file or directory
no-volume
holds left: ['radicale']
@@ -0,0 +1,51 @@
00:43:02 drill's radicale: {'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:43:10 [5] the box's drill mirror now at c760fca93
00:43:15 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
00:43:15 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
00:43:40 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:43:41 radicale: MKCALENDAR /csalad/felhom-0a551a4a/ http=201
00:43:41 radicale: PUT event http=201
00:43:44 radicale: readback http=200 found=True (controls: no auth 401, wrong 401, absent 404)
00:43:44 seeded: True; read back: True
00:43:47 volumes: ['radicale_radicale_data']
00:43:50 blocker: 8ac54e1274ef
00:43:51 [5] drill commit 3017cbb89641: radicale ghcr.io/kozea/radicale:3.8.1 -> busybox:1.36.1 (push rc=0)
00:44:00 [5] the box's drill mirror now at 3017cbb89
00:44:00 drill commit 3017cbb89641
00:44:04 box read TO after 4.5 s
00:44:05 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
00:44:05 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None
00:44:07 + 2.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None
00:44:09 + 4.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
00:44:11 + 6.1s phase=verifying label=Működés ellenőrzése… err=None hold=None
00:49:12 + 307.7s phase=undoing label=Visszaállítás az előző változatra… err=None hold=None
00:49:20 + 315.8s phase=undone label=Visszaállítva az előző változatra err=None hold=None
00:49:20 update: accepted=True final=undone err=None
00:49:31 after the update: state=running hold=None phase=undone key=None
00:49:34 controller log:
2026/11/02 23:44:05 update.go:556: [INFO] [stacks] update radicale: accepted — guarded update started
2026/11/02 23:44:05 update.go:1397: [INFO] [stacks] update radicale: phase checking
2026/11/02 23:44:05 update.go:814: [INFO] [stacks] update radicale: ladder — the installed version radicale=ghcr.io/kozea/radicale:3.8.1 is AT THE HEAD of the update_ladder (1 entries) — the catalog's current definition
2026/11/02 23:44:05 update.go:844: [INFO] [stacks] update radicale: no usable copy on any tier — younger than 24h0m0s and not older than this install's deploy (2026-11-02T23:43:15Z) (found: none) — backing up first
2026/11/02 23:44:05 update.go:1397: [INFO] [stacks] update radicale: phase backing-up
2026/11/02 23:44:07 update.go:859: [INFO] [stacks] update radicale: precondition met after the backup — Tier 2 (second drive) copy from 2026-11-02T23:44:05Z
2026/11/02 23:44:07 update.go:1397: [INFO] [stacks] update radicale: phase safety-dump
2026/11/02 23:44:07 update.go:874: [INFO] [stacks] update radicale: safety dump done (0 file(s)) []
2026/11/02 23:44:07 undo.go:287: [INFO] [stacks] update radicale: the undo copy will hold 1 named volume(s), 0.0 MiB
2026/11/02 23:44:07 update.go:1397: [INFO] [stacks] update radicale: phase pinning
2026/11/02 23:44:07 pin.go:385: [INFO] [stacks] update radicale: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/radicale/docker-compose.yml (radicale=busybox:1.36.1)
2026/11/02 23:44:07 update.go:1397: [INFO] [stacks] update radicale: phase pulling
2026/11/02 23:44:08 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:44:09 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:44:09 undo.go:437: [INFO] [stacks] update radicale: copied radicale_radicale_data → radicale_radicale_data.pre-update-20261102T234409Z in 470ms
2026/11/02 23:44:09 update.go:1397: [INFO] [stacks] update radicale: phase starting
2026/11/02 23:44:10 update.go:1397: [INFO] [stacks] update radicale: phase verifying
2026/11/02 23:49:12 update.go:1096: [ERROR] [stacks] update radicale FAILED after the new version was started: not healthy: not healthy within 5m0s (last: state restarting)
2026/11/02 23:49:12 update.go:1088: [INFO] [stacks] update radicale: kept 868 bytes of the app's own log at /opt/docker/stacks/radicale/hold-logs/20261102T234912Z/compose-logs.txt before stopping it (R-621)
2026/11/02 23:49:12 update.go:1397: [INFO] [stacks] update radicale: phase undoing
2026/11/02 23:49:12 undo.go:515: [WARN] [stacks] update radicale: UNDO — putting back the previous version and its 1 volume copy(ies) (reason: not healthy: not healthy within 5m0s (last: state restarting))
2026/11/02 23:49:18 unattended.go:527: [INFO] [stacks] update radicale: step radicale=busybox:1.36.1 recorded as undone — the automatic leg will not press it again until the catalog's ladder changes (ladder 3c58efa0dee4c41c)
2026/11/02 23:49:19 undo.go:641: [INFO] [stacks] update radicale: event app_update_undone
2026/11/02 23:49:19 undo.go:569: [INFO] [stacks] update radicale: UNDONE in 7s — the previous version is running on the data from before the update (the app's health check passed)
@@ -0,0 +1,52 @@
00:43:02 drill's radicale: {'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:43:10 [5] the box's drill mirror now at c760fca93
00:43:15 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
00:43:15 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
00:43:40 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:43:41 radicale: MKCALENDAR /csalad/felhom-0a551a4a/ http=201
00:43:41 radicale: PUT event http=201
00:43:44 radicale: readback http=200 found=True (controls: no auth 401, wrong 401, absent 404)
00:43:44 seeded: True; read back: True
00:43:47 volumes: ['radicale_radicale_data']
00:43:50 blocker: 8ac54e1274ef
00:43:51 [5] drill commit 3017cbb89641: radicale ghcr.io/kozea/radicale:3.8.1 -> busybox:1.36.1 (push rc=0)
00:44:00 [5] the box's drill mirror now at 3017cbb89
00:44:00 drill commit 3017cbb89641
00:44:04 box read TO after 4.5 s
00:44:05 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
00:44:05 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None
00:44:07 + 2.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None
00:44:09 + 4.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
00:44:11 + 6.1s phase=verifying label=Működés ellenőrzése… err=None hold=None
00:49:12 + 307.7s phase=undoing label=Visszaállítás az előző változatra… err=None hold=None
00:49:20 + 315.8s phase=undone label=Visszaállítva az előző változatra err=None hold=None
00:49:20 update: accepted=True final=undone err=None
00:49:31 after the update: state=running hold=None phase=undone key=None
00:49:34 controller log:
2026/11/02 23:44:05 update.go:556: [INFO] [stacks] update radicale: accepted — guarded update started
2026/11/02 23:44:05 update.go:1397: [INFO] [stacks] update radicale: phase checking
2026/11/02 23:44:05 update.go:814: [INFO] [stacks] update radicale: ladder — the installed version radicale=ghcr.io/kozea/radicale:3.8.1 is AT THE HEAD of the update_ladder (1 entries) — the catalog's current definition
2026/11/02 23:44:05 update.go:844: [INFO] [stacks] update radicale: no usable copy on any tier — younger than 24h0m0s and not older than this install's deploy (2026-11-02T23:43:15Z) (found: none) — backing up first
2026/11/02 23:44:05 update.go:1397: [INFO] [stacks] update radicale: phase backing-up
2026/11/02 23:44:07 update.go:859: [INFO] [stacks] update radicale: precondition met after the backup — Tier 2 (second drive) copy from 2026-11-02T23:44:05Z
2026/11/02 23:44:07 update.go:1397: [INFO] [stacks] update radicale: phase safety-dump
2026/11/02 23:44:07 update.go:874: [INFO] [stacks] update radicale: safety dump done (0 file(s)) []
2026/11/02 23:44:07 undo.go:287: [INFO] [stacks] update radicale: the undo copy will hold 1 named volume(s), 0.0 MiB
2026/11/02 23:44:07 update.go:1397: [INFO] [stacks] update radicale: phase pinning
2026/11/02 23:44:07 pin.go:385: [INFO] [stacks] update radicale: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/radicale/docker-compose.yml (radicale=busybox:1.36.1)
2026/11/02 23:44:07 update.go:1397: [INFO] [stacks] update radicale: phase pulling
2026/11/02 23:44:08 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:44:09 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:44:09 undo.go:437: [INFO] [stacks] update radicale: copied radicale_radicale_data → radicale_radicale_data.pre-update-20261102T234409Z in 470ms
2026/11/02 23:44:09 update.go:1397: [INFO] [stacks] update radicale: phase starting
2026/11/02 23:44:10 update.go:1397: [INFO] [stacks] update radicale: phase verifying
2026/11/02 23:49:12 update.go:1096: [ERROR] [stacks] update radicale FAILED after the new version was started: not healthy: not healthy within 5m0s (last: state restarting)
2026/11/02 23:49:12 update.go:1088: [INFO] [stacks] update radicale: kept 868 bytes of the app's own log at /opt/docker/stacks/radicale/hold-logs/20261102T234912Z/compose-logs.txt before stopping it (R-621)
2026/11/02 23:49:12 update.go:1397: [INFO] [stacks] update radicale: phase undoing
2026/11/02 23:49:12 undo.go:515: [WARN] [stacks] update radicale: UNDO — putting back the previous version and its 1 volume copy(ies) (reason: not healthy: not healthy within 5m0s (last: state restarting))
2026/11/02 23:49:18 unattended.go:527: [INFO] [stacks] update radicale: step radicale=busybox:1.36.1 recorded as undone — the automatic leg will not press it again until the catalog's ladder changes (ladder 3c58efa0dee4c41c)
2026/11/02 23:49:19 undo.go:641: [INFO] [stacks] update radicale: event app_update_undone
2026/11/02 23:49:19 undo.go:569: [INFO] [stacks] update radicale: UNDONE in 7s — the previous version is running on the data from before the update (the app's health check passed)
00:49:34 STOP: no update hold — the stage did not work; nothing to prove
@@ -0,0 +1,53 @@
00:43:02 drill's radicale: {'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:43:10 [5] the box's drill mirror now at c760fca93
00:43:15 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
00:43:15 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
00:43:40 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
00:43:41 radicale: MKCALENDAR /csalad/felhom-0a551a4a/ http=201
00:43:41 radicale: PUT event http=201
00:43:44 radicale: readback http=200 found=True (controls: no auth 401, wrong 401, absent 404)
00:43:44 seeded: True; read back: True
00:43:47 volumes: ['radicale_radicale_data']
00:43:50 blocker: 8ac54e1274ef
00:43:51 [5] drill commit 3017cbb89641: radicale ghcr.io/kozea/radicale:3.8.1 -> busybox:1.36.1 (push rc=0)
00:44:00 [5] the box's drill mirror now at 3017cbb89
00:44:00 drill commit 3017cbb89641
00:44:04 box read TO after 4.5 s
00:44:05 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
00:44:05 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None
00:44:07 + 2.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None
00:44:09 + 4.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
00:44:11 + 6.1s phase=verifying label=Működés ellenőrzése… err=None hold=None
00:49:12 + 307.7s phase=undoing label=Visszaállítás az előző változatra… err=None hold=None
00:49:20 + 315.8s phase=undone label=Visszaállítva az előző változatra err=None hold=None
00:49:20 update: accepted=True final=undone err=None
00:49:31 after the update: state=running hold=None phase=undone key=None
00:49:34 controller log:
2026/11/02 23:44:05 update.go:556: [INFO] [stacks] update radicale: accepted — guarded update started
2026/11/02 23:44:05 update.go:1397: [INFO] [stacks] update radicale: phase checking
2026/11/02 23:44:05 update.go:814: [INFO] [stacks] update radicale: ladder — the installed version radicale=ghcr.io/kozea/radicale:3.8.1 is AT THE HEAD of the update_ladder (1 entries) — the catalog's current definition
2026/11/02 23:44:05 update.go:844: [INFO] [stacks] update radicale: no usable copy on any tier — younger than 24h0m0s and not older than this install's deploy (2026-11-02T23:43:15Z) (found: none) — backing up first
2026/11/02 23:44:05 update.go:1397: [INFO] [stacks] update radicale: phase backing-up
2026/11/02 23:44:07 update.go:859: [INFO] [stacks] update radicale: precondition met after the backup — Tier 2 (second drive) copy from 2026-11-02T23:44:05Z
2026/11/02 23:44:07 update.go:1397: [INFO] [stacks] update radicale: phase safety-dump
2026/11/02 23:44:07 update.go:874: [INFO] [stacks] update radicale: safety dump done (0 file(s)) []
2026/11/02 23:44:07 undo.go:287: [INFO] [stacks] update radicale: the undo copy will hold 1 named volume(s), 0.0 MiB
2026/11/02 23:44:07 update.go:1397: [INFO] [stacks] update radicale: phase pinning
2026/11/02 23:44:07 pin.go:385: [INFO] [stacks] update radicale: pin advanced to /opt/docker/felhom-controller/data/catalog-cache/templates/radicale/docker-compose.yml (radicale=busybox:1.36.1)
2026/11/02 23:44:07 update.go:1397: [INFO] [stacks] update radicale: phase pulling
2026/11/02 23:44:08 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:44:09 update.go:1397: [INFO] [stacks] update radicale: phase copying
2026/11/02 23:44:09 undo.go:437: [INFO] [stacks] update radicale: copied radicale_radicale_data → radicale_radicale_data.pre-update-20261102T234409Z in 470ms
2026/11/02 23:44:09 update.go:1397: [INFO] [stacks] update radicale: phase starting
2026/11/02 23:44:10 update.go:1397: [INFO] [stacks] update radicale: phase verifying
2026/11/02 23:49:12 update.go:1096: [ERROR] [stacks] update radicale FAILED after the new version was started: not healthy: not healthy within 5m0s (last: state restarting)
2026/11/02 23:49:12 update.go:1088: [INFO] [stacks] update radicale: kept 868 bytes of the app's own log at /opt/docker/stacks/radicale/hold-logs/20261102T234912Z/compose-logs.txt before stopping it (R-621)
2026/11/02 23:49:12 update.go:1397: [INFO] [stacks] update radicale: phase undoing
2026/11/02 23:49:12 undo.go:515: [WARN] [stacks] update radicale: UNDO — putting back the previous version and its 1 volume copy(ies) (reason: not healthy: not healthy within 5m0s (last: state restarting))
2026/11/02 23:49:18 unattended.go:527: [INFO] [stacks] update radicale: step radicale=busybox:1.36.1 recorded as undone — the automatic leg will not press it again until the catalog's ladder changes (ladder 3c58efa0dee4c41c)
2026/11/02 23:49:19 undo.go:641: [INFO] [stacks] update radicale: event app_update_undone
2026/11/02 23:49:19 undo.go:569: [INFO] [stacks] update radicale: UNDONE in 7s — the previous version is running on the data from before the update (the app's health check passed)
00:49:34 STOP: no update hold — the stage did not work; nothing to prove
00:49:34 PHASE r1347 rc=1
@@ -0,0 +1,15 @@
# Part 5 — claims walked (2026-11-02/03), on Tester 1, front doors only; mail read in the mailbox
| Claim | Result | File |
|---|---|---|
| fail.app-crash, fault.customer-email, fault.operator-email | walked | `crash-walk*.log`, `mail-*.txt` |
| backup.fill-warning, fail.drive-filling | walked | `fill-walk.log`, `mail-fill-household.txt` |
| backup.restore-proof | walked (journals of three boxes) | `../part4/restore-test-*.txt` |
| backup.offsite | walked (snapshots per day) | `offsite-*.txt` |
| fail.internet-down | walked; slow Backups page, R-1362 | `outage-wan.log` |
| use.lan-fallback | partial; R-1364 | `lan-resolver.txt` |
| fail.hub-down | partial; events lost, R-1361 | `outage-hub.log` |
Tries that taught something, kept: `outage-wan-try1-…` (the hub walk's own safety job lifted the block mid-way), `outage-wan-try2-…` (removing the default route: the agent's self-heal put it back in about 100 s; a good thing, and a wrong way to stage a dead line).
Left on Tester 1: `notifications.email_cleared: true` in its settings (the address was empty when found, typed for the walk, cleared through the page).
@@ -0,0 +1,27 @@
23:42:54 checked as found: []
23:42:54 the group „my apps” was off: switched on, as a household that wants this mail would
23:42:54 save with the address -> HTTP/2 200
23:43:15 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
23:43:15 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
23:43:35 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
23:43:35 installed; kills start 2026-11-02T22:43:35Z
23:43:41 kill -9 #1 pid 869340: state running hold 'None'
23:44:03 kill -9 #2 pid 869695: state running hold 'None'
23:44:25 kill -9 #3 pid 870095: state starting hold 'None'
23:44:47 kill -9 #4 pid 870522: state running hold 'None'
23:45:09 kill -9 #5 pid 870891: state running hold 'None'
23:45:31 kill -9 #6 pid 871229: state running hold 'None'
23:45:53 kill -9 #7 pid 871673: state running hold 'None'
23:46:12 kill -9 #8 pid : state stopped hold 'Az alkalmazást ismét leállítottuk, mert '
23:47:16 controller log:
2026/11/02 22:45:55 [WARN] [deadapp] radicale: crash_loop — 7 in 10m0s; STOPPING it (decision 28)
2026/11/02 22:45:55 [INFO] Event pushed: app_stopped_unhealthy (warning) [hu-only] — A(z) radicale alkalmazást ismét leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája
2026/11/02 22:46:25 [INFO] Event pushed: app_held (info) [hu-only] — App radicale is held: stopped by the box for a crash loop (2026-11-02T22:45:55Z) — the household's Start gives it one more try
23:47:17 Start -> 200
23:47:27 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
23:47:59 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': ['radicale_radicale_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az
23:48:07 [X] after remove: deployed=False leftovers=''
23:48:07 removed: deployed=False
23:48:07 the address cleared -> HTTP/2 200
23:48:12 the page's address now: tester1@felhom.eu
@@ -0,0 +1,31 @@
23:42:54 checked as found: []
23:42:54 the group „my apps” was off: switched on, as a household that wants this mail would
23:42:54 save with the address -> HTTP/2 200
23:43:15 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
23:43:15 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
23:43:35 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
23:43:35 installed; kills start 2026-11-02T22:43:35Z
23:43:41 kill -9 #1 pid 869340: state running hold 'None'
23:44:03 kill -9 #2 pid 869695: state running hold 'None'
23:44:25 kill -9 #3 pid 870095: state starting hold 'None'
23:44:47 kill -9 #4 pid 870522: state running hold 'None'
23:45:09 kill -9 #5 pid 870891: state running hold 'None'
23:45:31 kill -9 #6 pid 871229: state running hold 'None'
23:45:53 kill -9 #7 pid 871673: state running hold 'None'
23:46:12 kill -9 #8 pid : state stopped hold 'Az alkalmazást ismét leállítottuk, mert '
23:47:16 controller log:
2026/11/02 22:45:55 [WARN] [deadapp] radicale: crash_loop — 7 in 10m0s; STOPPING it (decision 28)
2026/11/02 22:45:55 [INFO] Event pushed: app_stopped_unhealthy (warning) [hu-only] — A(z) radicale alkalmazást ismét leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája
2026/11/02 22:46:25 [INFO] Event pushed: app_held (info) [hu-only] — App radicale is held: stopped by the box for a crash loop (2026-11-02T22:45:55Z) — the household's Start gives it one more try
23:47:17 Start -> 200
23:47:27 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
23:47:59 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': ['radicale_radicale_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az
23:48:07 [X] after remove: deployed=False leftovers=''
23:48:07 removed: deployed=False
23:48:07 the address cleared -> HTTP/2 200
23:48:12 the page's address now: tester1@felhom.eu
HTTP/2 200
address now: ''
{'enabled_events': None, 'cooldown_hours': 6, 'email_cleared': True}
@@ -0,0 +1,20 @@
23:31:37 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
23:31:38 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
23:31:58 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
23:31:58 installed running; t0 2026-11-02T22:31:37Z
23:32:04 kill -9 #1 pid 857793: state running hold None
23:32:27 kill -9 #2 pid 858198: state running hold None
23:32:48 kill -9 #3 pid 858665: state running hold None
23:33:11 kill -9 #4 pid 858999: state running hold None
23:33:33 kill -9 #5 pid 859417: state running hold None
23:33:55 kill -9 #6 pid 859823: state starting hold None
23:34:14 kill -9 #7 pid : state stopped hold 'Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.'
23:34:54 after the kills 22:34:54: state=stopped hold='Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.'
23:34:58 controller log:
2026/11/02 22:32:05 [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/11/02 22:33:55 [WARN] [deadapp] radicale: crash_loop — 6 in 10m0s; STOPPING it (decision 28)
2026/11/02 22:34:05 [WARN] [backup] radicale is STOPPED by the box: crash_loop (trip 1 within 24h0m0s) — Start gives it one more try (decision 28)
2026/11/02 22:34:05 [INFO] Event pushed: app_stopped_unhealthy (warning) [hu-only] — A(z) radicale alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.
2026/11/02 22:34:25 [INFO] [holdreport] radicale held (unhealthy_stop since 2026-11-02T22:33:55Z) — reported to the hub
2026/11/02 22:34:25 [INFO] Event pushed: app_held (info) [hu-only] — App radicale is held: stopped by the box for a crash loop (2026-11-02T22:33:55Z) — the household's Start gives it one more try
@@ -0,0 +1,27 @@
23:31:37 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
23:31:38 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
23:31:58 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
23:31:58 installed running; t0 2026-11-02T22:31:37Z
23:32:04 kill -9 #1 pid 857793: state running hold None
23:32:27 kill -9 #2 pid 858198: state running hold None
23:32:48 kill -9 #3 pid 858665: state running hold None
23:33:11 kill -9 #4 pid 858999: state running hold None
23:33:33 kill -9 #5 pid 859417: state running hold None
23:33:55 kill -9 #6 pid 859823: state starting hold None
23:34:14 kill -9 #7 pid : state stopped hold 'Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.'
23:34:54 after the kills 22:34:54: state=stopped hold='Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.'
23:34:58 controller log:
2026/11/02 22:32:05 [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/11/02 22:33:55 [WARN] [deadapp] radicale: crash_loop — 6 in 10m0s; STOPPING it (decision 28)
2026/11/02 22:34:05 [WARN] [backup] radicale is STOPPED by the box: crash_loop (trip 1 within 24h0m0s) — Start gives it one more try (decision 28)
2026/11/02 22:34:05 [INFO] Event pushed: app_stopped_unhealthy (warning) [hu-only] — A(z) radicale alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.
2026/11/02 22:34:25 [INFO] [holdreport] radicale held (unhealthy_stop since 2026-11-02T22:33:55Z) — reported to the hub
2026/11/02 22:34:25 [INFO] Event pushed: app_held (info) [hu-only] — App radicale is held: stopped by the box for a crash loop (2026-11-02T22:33:55Z) — the household's Start gives it one more try
WAITING-FOR-MAIL
23:41:59 the household's Start -> 200 {'ok': True, 'data': {'state': 'starting'}, 'message': 'Stack radicale start requested — state now: starting'}
23:42:09 after Start: state=running hold=None
23:42:09 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
23:42:41 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': ['radicale_radicale_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az
23:42:50 [X] after remove: deployed=False leftovers=''
23:42:53 removed: deployed=False; leftovers: 0 0 ls: cannot access '/opt/docker/stacks/radicale/app.yaml': No such file or directory
@@ -0,0 +1,26 @@
23:31:37 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
23:31:38 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
23:31:58 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
23:31:58 installed running; t0 2026-11-02T22:31:37Z
23:32:04 kill -9 #1 pid 857793: state running hold None
23:32:27 kill -9 #2 pid 858198: state running hold None
23:32:48 kill -9 #3 pid 858665: state running hold None
23:33:11 kill -9 #4 pid 858999: state running hold None
23:33:33 kill -9 #5 pid 859417: state running hold None
23:33:55 kill -9 #6 pid 859823: state starting hold None
23:34:14 kill -9 #7 pid : state stopped hold 'Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.'
23:34:54 after the kills 22:34:54: state=stopped hold='Az alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.'
23:34:58 controller log:
2026/11/02 22:32:05 [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed)
2026/11/02 22:33:55 [WARN] [deadapp] radicale: crash_loop — 6 in 10m0s; STOPPING it (decision 28)
2026/11/02 22:34:05 [WARN] [backup] radicale is STOPPED by the box: crash_loop (trip 1 within 24h0m0s) — Start gives it one more try (decision 28)
2026/11/02 22:34:05 [INFO] Event pushed: app_stopped_unhealthy (warning) [hu-only] — A(z) radicale alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.
2026/11/02 22:34:25 [INFO] [holdreport] radicale held (unhealthy_stop since 2026-11-02T22:33:55Z) — reported to the hub
2026/11/02 22:34:25 [INFO] Event pushed: app_held (info) [hu-only] — App radicale is held: stopped by the box for a crash loop (2026-11-02T22:33:55Z) — the household's Start gives it one more try
23:41:59 the household's Start -> 200 {'ok': True, 'data': {'state': 'starting'}, 'message': 'Stack radicale start requested — state now: starting'}
23:42:09 after Start: state=running hold=None
23:42:09 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
23:42:41 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': ['radicale_radicale_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az
23:42:50 [X] after remove: deployed=False leftovers=''
23:42:53 removed: deployed=False; leftovers: 0 0 ls: cannot access '/opt/docker/stacks/radicale/app.yaml': No such file or directory
@@ -0,0 +1,21 @@
01:59:07 the address saved -> HTTP/2 200
01:59:20 before: 49G 13G 35G 27%
01:59:30 filling with 29 GiB: 42G 5.2G 89%
02:01:40 the watcher, 01:01:40:
2026/11/03 01:01:25 [WARN] [fillwatch] /mnt/felhom-drives/adatlemez ("Adatlemez"): ok warning 89% used, 5.2 GB free of 48.9 GB; notifying the customer
2026/11/03 01:01:25 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 1 notification(s); bands: /mnt/felhom-drives/adatlemez=warning
02:01:43 events:
2026/11/03 01:01:25 [INFO] Event pushed: disk_warning (warning) [hu-only] A(z) Adatlemez trol 89% foglalt 5,2 GB szabad hely maradt. Szabadts fel helyet, mieltt megtelik: trlj felesleges fjlokat, vagy csatlakoztass j meghajtt. H
02:03:17 removing the file: 13G 35G 27%
02:19:20 back to normal, 01:19:20:
02:19:20 the address cleared -> HTTP/2 200
02:19:23 the page's address now: ''
02:19:26 left on the drive: 0
13G 35G 27%
@@ -0,0 +1,21 @@
01:59:07 the address saved -> HTTP/2 200
01:59:20 before: 49G 13G 35G 27%
01:59:30 filling with 29 GiB: 42G 5.2G 89%
02:01:40 the watcher, 01:01:40:
2026/11/03 01:01:25 [WARN] [fillwatch] /mnt/felhom-drives/adatlemez ("Adatlemez"): ok warning 89% used, 5.2 GB free of 48.9 GB; notifying the customer
2026/11/03 01:01:25 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 1 notification(s); bands: /mnt/felhom-drives/adatlemez=warning
02:01:43 events:
2026/11/03 01:01:25 [INFO] Event pushed: disk_warning (warning) [hu-only] A(z) Adatlemez trol 89% foglalt 5,2 GB szabad hely maradt. Szabadts fel helyet, mieltt megtelik: trlj felesleges fjlokat, vagy csatlakoztass j meghajtt. H
02:03:17 removing the file: 13G 35G 27%
02:19:20 back to normal, 01:19:20:
02:19:20 the address cleared -> HTTP/2 200
02:19:23 the page's address now: ''
02:19:26 left on the drive: 0
13G 35G 27%
@@ -0,0 +1,12 @@
== before the cut 02:57:07
dns: 192.168.0.154 app name: 192.168.0.148 other: 104.20.23.154
dashboard by name through the box's own DNS answer: 000
the same at the guest's address: 302
== during the cut 02:57:48
dns: 192.168.0.154 app name: 192.168.0.148 other: 104.20.23.154
dashboard by name through the box's own DNS answer: 000
the same at the guest's address: 302
== demo-hp, for the host-FQDN comparison 02:58:24
box DNS answers: 192.168.0.155 192.168.0.158
dashboard by name at the first answer: 302
Tester 1: /etc/hosts holds 192.168.0.154 felhom.enkicsifelhom.hu (the host's FQDN is the dashboard's name); dnsmasq reads /etc/hosts first, so the dashboard's name answers the HOST, where nothing serves 443.
@@ -0,0 +1,8 @@
Date: 2026-11-02T22:45:56Z To: tester1@felhom.eu From: monitoring@felhom.eu (read through the mailbox, id 1a19e70dafcb18e3)
Subject: [Felhom] Figyelmeztetés: radicale: az alkalmazást leállítottuk, mert újra és újra összeomlott
Kedves Ügyfél! A Felhom rendszered a következő értesítést küldte:
radicale: az alkalmazást leállítottuk, mert újra és újra összeomlott
- Szerver: tester-1 - Időpont: 2026-11-02 23:45 - Szint: Figyelmeztetés - Típus: app_stopped_unhealthy
- Üzenet: A(z) radicale alkalmazást ismét leállítottuk ... A Felhom ügyfélszolgálatát értesítettük.
- Megjegyzés: {"app":"radicale","stack_name":"radicale","kind":"crash_loop","count":7,"window_min":10,"trip":2}
Hub log: 23:45:56 Customer email sent for tester-1/app_stopped_unhealthy; 23:45:55 Operator email suppressed — cooldown (one hour per app).
@@ -0,0 +1,7 @@
Date: 2026-11-02T22:34:06Z To: admin@felhom.eu From: monitoring@felhom.eu (read through the mailbox, id 1a19e660986c516b)
Subject: [Felhom] tester-1: app_stopped_unhealthy
Customer: tester-1
Event: app_stopped_unhealthy
Severity: warning
Time: 2026-11-02 23:34 CET
Message: A(z) radicale alkalmazást leállítottuk, mert újra és újra összeomlott (vagy elfogyott a memóriája). Az Indítás gombbal újra megpróbálhatod.
@@ -0,0 +1,5 @@
Read through the mailbox. id 1a19eecf225a0e00 2026-11-03T01:01:26Z To: tester1@felhom.eu (operator copy id 1a19eecf2a0a4869, To: admin@felhom.eu, same second)
Subject: [Felhom] Figyelmeztetés: A(z) „Adatlemez” tároló 89% foglalt — 5,2 GB szabad hely maradt. Szabadíts fel helyet, mielőtt megtelik: ...
Body: A(z) „Adatlemez” tároló 89% foglalt — 5,2 GB szabad hely maradt. Szabadíts fel helyet, mielőtt megtelik: törölj felesleges fájlokat, vagy csatlakoztass új meghajtót. Ha megtelik, a biztonsági mentések meghiúsulnak.
Részletek: Szerver tester-1; Időpont 2026-11-03 02:01; Szint Figyelmeztetés; Típus disk_warning
Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning","label":"Adatlemez","path":"/mnt/felhom-drives/adatlemez","total_gb":48.914520263671875,"used_percent":88.8099850762859}
@@ -0,0 +1,9 @@
Read through the mailbox 2026-11-03 (another channel than the hub's log). ONE mail per run, every failing app named.
id 1a15a9d97e8fe6e5 2026-10-20T18:40:36Z To: admin@felhom.eu
Subject: [Felhom] Tester4: 3 of 8 apps failed to back up (manual run)
Message: 3 of 8 apps failed to back up in this manual run: immich, nextcloud, paperless-ngx
immich whole app (refused before any write) refused: ... /mnt/felhom-drives/hdd_1: 185.8/195.8 GB used (100%), 0.0 GB free
nextcloud whole app (refused before any write) refused: ...
paperless-ngx whole app (refused before any write) refused: ...
Six more of this subject in 60 days (demo-hp 3 of 3, tester-1 1 of 3, demo-hp 1 of 9, ...).
Household mail seen too: id 1a1703ad1185840f 2026-10-24T23:24:22Z To: tester4@felhom.eu "[Felhom] Hiba: A szerver nem elérhető!" (node_down, Hungarian).
@@ -0,0 +1,14 @@
14 2026-10-18
14 2026-10-21
14 2026-10-22
14 2026-10-23
14 2026-10-24
14 2026-10-25
14 2026-10-26
14 2026-10-27
15 2026-10-28
15 2026-10-29
15 2026-10-30
15 2026-10-31
15 2026-11-01
15 2026-11-02
@@ -0,0 +1,13 @@
3 2026-10-06
7 2026-10-15
6 2026-10-16
6 2026-10-17
4 2026-10-18
4 2026-10-25
4 2026-10-27
4 2026-10-28
25 2026-10-29
4 2026-10-30
4 2026-10-31
4 2026-11-01
4 2026-11-02
@@ -0,0 +1,55 @@
01:17:50 === hub: before
01:17:50 now 00:17:50 UTC
01:17:51 routes host/guest: default via 192.168.0.1 dev vmbr0 proto kernel onlink
default via 192.168.0.1 dev eth0
01:17:55 apps: 10
01:17:55 the public name from outside the box: 302
01:17:56 CUT: ok
01:17:56 now 00:17:56 UTC
01:18:30 from the box: hub 000 internet 200
01:18:30 the dashboard on the home network: login ok, /api/health 200
01:18:30 page /: 32 bytes
01:18:35 page /backups: 50767 bytes
01:18:36 page /settings/notifications: 49636 bytes
01:18:39 apps running: 10
01:18:39 the public name from outside the box: 302
01:18:39 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
01:18:39 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
01:18:59 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
01:18:59 an install during the outage (cached catalog): True, state running
01:22:04 after the kills: state=stopped held=True
01:22:07 controller log:
2026/11/03 00:19:15 [WARN] Event push failed after 3 attempts (app_deploy_started/info): Post "https://hub.felhom.eu/api/v1/event": context deadline exceeded (Client.Time
2026/11/03 00:19:24 [WARN] Event push failed after 3 attempts (app_deployed/info): Post "https://hub.felhom.eu/api/v1/event": context deadline exceeded (Client.Timeout ex
2026/11/03 00:21:25 [WARN] [deadapp] radicale: crash_loop 7 in 10m0s; STOPPING it (decision 28)
2026/11/03 00:22:02 [WARN] Event push failed after 3 attempts (app_stopped_unhealthy/warning): Post "https://hub.felhom.eu/api/v1/event": context deadline exceeded (Clien
01:30:47 dashboard at the end of the outage: 200
01:30:51 the box's own log, warnings since the cut:
1 [WARN] [settings] restore hold SET for radicale the app stays stopped until it is cleared
1 [WARN] [report] wait channel error: read tcp 172.17.0.2:48558->192.168.0.192:443: read: connection timed out backing off (the
1 [WARN] [report] out-of-cycle push failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": context de
1 [WARN] [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": context deadline exceeded (Client.Timeout exceeded while
1 [WARN] [deadapp] radicale: crash_loop 7 in 10m0s; STOPPING it (decision 28)
1 [WARN] [backup] radicale is STOPPED by the box: crash_loop (trip 2 within 24h0m0s) Start gives it one more try (decision 28)
1 [WARN] Event push failed after 3 attempts (app_stopped_unhealthy/warning): Post "https://hub.felhom.eu/api/v1/event": context dead
1 [WARN] Event push failed after 3 attempts (app_held/info): Post "https://hub.felhom.eu/api/v1/event": context deadline exceeded (C
1 [WARN] Event push failed after 3 attempts (app_deployed/info): Post "https://hub.felhom.eu/api/v1/event": context deadline exceede
1 [WARN] Event push failed after 3 attempts (app_deploy_started/info): Post "https://hub.felhom.eu/api/v1/event": context deadline e
01:30:52 RESTORE: ok
01:30:52 now 00:30:52 UTC
01:31:10 back after 15 s: the public name 302; events pushed since 0
01:38:14 events since the restore:
01:38:14 host agent: Nov 03 01:27:29 felhom felhom-agent[2609044]: time=2026-11-03T01:27:29.540+01:00 level=WARN msg="hub: report failed; keeping current interval" err="hub: transport error:
01:38:15 the household's Start -> 200
01:38:25 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
01:38:57 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': ['radicale_radicale_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az
01:39:05 [X] after remove: deployed=False leftovers=''
01:39:06 removed: deployed=False
01:39:07 routes after: default via 192.168.0.1 dev vmbr0 proto kernel onlink
default via 192.168.0.1 dev eth0
@@ -0,0 +1,55 @@
01:17:50 === hub: before
01:17:50 now 00:17:50 UTC
01:17:51 routes host/guest: default via 192.168.0.1 dev vmbr0 proto kernel onlink
default via 192.168.0.1 dev eth0
01:17:55 apps: 10
01:17:55 the public name from outside the box: 302
01:17:56 CUT: ok
01:17:56 now 00:17:56 UTC
01:18:30 from the box: hub 000 internet 200
01:18:30 the dashboard on the home network: login ok, /api/health 200
01:18:30 page /: 32 bytes
01:18:35 page /backups: 50767 bytes
01:18:36 page /settings/notifications: 49636 bytes
01:18:39 apps running: 10
01:18:39 the public name from outside the box: 302
01:18:39 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
01:18:39 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
01:18:59 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
01:18:59 an install during the outage (cached catalog): True, state running
01:22:04 after the kills: state=stopped held=True
01:22:07 controller log:
2026/11/03 00:19:15 [WARN] Event push failed after 3 attempts (app_deploy_started/info): Post "https://hub.felhom.eu/api/v1/event": context deadline exceeded (Client.Time
2026/11/03 00:19:24 [WARN] Event push failed after 3 attempts (app_deployed/info): Post "https://hub.felhom.eu/api/v1/event": context deadline exceeded (Client.Timeout ex
2026/11/03 00:21:25 [WARN] [deadapp] radicale: crash_loop 7 in 10m0s; STOPPING it (decision 28)
2026/11/03 00:22:02 [WARN] Event push failed after 3 attempts (app_stopped_unhealthy/warning): Post "https://hub.felhom.eu/api/v1/event": context deadline exceeded (Clien
01:30:47 dashboard at the end of the outage: 200
01:30:51 the box's own log, warnings since the cut:
1 [WARN] [settings] restore hold SET for radicale the app stays stopped until it is cleared
1 [WARN] [report] wait channel error: read tcp 172.17.0.2:48558->192.168.0.192:443: read: connection timed out backing off (the
1 [WARN] [report] out-of-cycle push failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": context de
1 [WARN] [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": context deadline exceeded (Client.Timeout exceeded while
1 [WARN] [deadapp] radicale: crash_loop 7 in 10m0s; STOPPING it (decision 28)
1 [WARN] [backup] radicale is STOPPED by the box: crash_loop (trip 2 within 24h0m0s) Start gives it one more try (decision 28)
1 [WARN] Event push failed after 3 attempts (app_stopped_unhealthy/warning): Post "https://hub.felhom.eu/api/v1/event": context dead
1 [WARN] Event push failed after 3 attempts (app_held/info): Post "https://hub.felhom.eu/api/v1/event": context deadline exceeded (C
1 [WARN] Event push failed after 3 attempts (app_deployed/info): Post "https://hub.felhom.eu/api/v1/event": context deadline exceede
1 [WARN] Event push failed after 3 attempts (app_deploy_started/info): Post "https://hub.felhom.eu/api/v1/event": context deadline e
01:30:52 RESTORE: ok
01:30:52 now 00:30:52 UTC
01:31:10 back after 15 s: the public name 302; events pushed since 0
01:38:14 events since the restore:
01:38:14 host agent: Nov 03 01:27:29 felhom felhom-agent[2609044]: time=2026-11-03T01:27:29.540+01:00 level=WARN msg="hub: report failed; keeping current interval" err="hub: transport error:
01:38:15 the household's Start -> 200
01:38:25 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
01:38:57 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': ['radicale_radicale_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az
01:39:05 [X] after remove: deployed=False leftovers=''
01:39:06 removed: deployed=False
01:39:07 routes after: default via 192.168.0.1 dev vmbr0 proto kernel onlink
default via 192.168.0.1 dev eth0
@@ -0,0 +1,25 @@
cut 01:39:42
default via 192.168.0.1 dev eth0
169.254.253.0/30 dev eth1 proto kernel scope link src 169.254.253.2
172.16.253.0/29 dev br-35c99f05236d proto kernel scope link src 172.16.253.1
2026-11-03T01:41:50Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=2 event=0 ip=198.41.200.113
2026-11-03T01:41:50Z INF Tunnel connection curve preferences: [X25519MLKEM768 CurveID(65074) CurveP256] connIndex=3 event=0 ip=198.41.192.107
2026-11-03T01:41:51Z INF Registered tunnel connection connIndex=3 connection=4fa183ee-2538-4279-8bd9-d8a87cce4e93 event=0 ip=198.41.192.107 location=bud01 protocol=quic
2026-11-03T01:41:51Z INF Registered tunnel connection connIndex=1 connection=583b8303-a3ee-499f-9d01-4a3b98073b70 event=0 ip=198.41.200.53 location=vie06 protocol=quic
2026-11-03T01:41:51Z INF Registered tunnel connection connIndex=2 connection=7c4e1404-a6fe-421e-94e1-1a8860bd93e7 event=0 ip=198.41.200.113 location=vie06 protocol=quic
agent /health http=404 in 24 ms
Nov 03 02:40:19 felhom felhom-agent[2609044]: time=2026-11-03T02:40:19.717+01:00 level=WARN msg="selfheal: node check FAILED" heal=networking detail="networking.service active=true, default route present=false"
Nov 03 02:40:29 felhom felhom-agent[2609044]: time=2026-11-03T02:40:29.764+01:00 level=ERROR msg="node self-heal: networking still down after start attempt" attempt=1
Nov 03 02:40:59 felhom felhom-agent[2609044]: time=2026-11-03T02:40:59.811+01:00 level=ERROR msg="node self-heal: networking still down after start attempt" attempt=2
Nov 03 02:41:23 felhom felhom-agent[2609044]: time=2026-11-03T02:41:23.610+01:00 level=WARN msg="guestnet: guest network unhealthy — healing" vmid=9201 detail="no default route" bad_probes=2
back 01:42:34
01:40:11
HTTP/2 502 server: cloudflare cf-ray: a547ab62fe285b8a-VIE
01:40:39
HTTP/2 502 server: cloudflare cf-ray: a547ac131b1f5bb4-VIE
01:41:07
HTTP/2 530 server: cloudflare cf-ray: a547acc2b80c5c16-VIE
01:41:35
HTTP/2 302 location: /login server: cloudflare cf-ray: a547ad727d3c5b69-VIE
01:42:03
HTTP/2 302 location: /login server: cloudflare cf-ray: a547ae225b965ad5-VIE
@@ -0,0 +1,35 @@
02:43:01 === wan: before
02:43:01 now 01:43:01 UTC
02:43:02 routes host/guest: default via 192.168.0.1 dev vmbr0 onlink
default via 192.168.0.1 dev eth0
02:43:05 apps: 10
02:43:05 the public name's edge address: 172.67.205.21
02:43:06 the public name through the edge: 302
02:43:27 CUT: ok
02:43:27 now 01:43:27 UTC
02:44:12 from the box: hub 000 internet 000
02:44:12 the dashboard on the home network: login ok, /api/health 200
02:44:12 page /: 32 bytes in 0.0 s
02:46:38 page /backups: 11 bytes in 145.4 s
02:46:38 page /settings/notifications: 49636 bytes in 0.0 s
02:46:42 apps running: 10
02:46:42 the public name through the edge: 530
02:56:22 the public name through the edge, at the end: 530
02:56:22 dashboard at the end of the outage: 200
02:56:26 the box's own log, warnings since the cut:
1 [WARN] [report] wait channel error: context deadline exceeded backing off (the 15-min cycle still reconciles)
1 [WARN] [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": dial tcp 192.168.0.192:443: i/o timeout (Client.Timeout e
1 [ERROR] [sync] Catalog sync failed: git fetch: git fetch --depth 1 origin main: context deadline exceeded (subprocess killed at th
1 [ERROR] [scheduler] Job hub-report failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": dial tcp
02:56:27 RESTORE: ok
02:56:27 now 01:56:27 UTC
02:57:41 back after 60 s: the public name 302; events pushed since 0
03:02:46 events since the restore:
03:02:46 host agent:
03:02:48 routes after: default via 192.168.0.1 dev vmbr0 onlink
default via 192.168.0.1 dev eth0
@@ -0,0 +1,29 @@
01:39:32 === wan: before
01:39:32 now 00:39:32 UTC
01:39:34 routes host/guest: default via 192.168.0.1 dev vmbr0 proto kernel onlink
default via 192.168.0.1 dev eth0
01:39:37 apps: 10
01:39:37 the public name from outside the box: 302
01:39:39 CUT: ok
01:39:39 now 00:39:39 UTC
01:40:13 from the box: hub 302 internet 000
01:40:13 the dashboard on the home network: login ok, /api/health 200
01:40:13 page /: 32 bytes
01:42:18 page /backups: 47895 bytes
01:42:18 page /settings/notifications: 49636 bytes
01:42:21 apps running: 10
01:42:21 the public name from outside the box: 302
01:52:21 dashboard at the end of the outage: 200
01:52:26 the box's own log, warnings since the cut:
01:52:28 RESTORE: ok
01:52:28 now 00:52:28 UTC
01:53:47 back after 60 s: the public name 302; events pushed since 0
01:58:52 events since the restore:
01:58:52 host agent:
01:58:54 routes after: default via 192.168.0.1 dev vmbr0 onlink
default via 192.168.0.1 dev eth0
@@ -0,0 +1,29 @@
01:39:32 === wan: before
01:39:32 now 00:39:32 UTC
01:39:34 routes host/guest: default via 192.168.0.1 dev vmbr0 proto kernel onlink
default via 192.168.0.1 dev eth0
01:39:37 apps: 10
01:39:37 the public name from outside the box: 302
01:39:39 CUT: ok
01:39:39 now 00:39:39 UTC
01:40:13 from the box: hub 302 internet 000
01:40:13 the dashboard on the home network: login ok, /api/health 200
01:40:13 page /: 32 bytes
01:42:18 page /backups: 47895 bytes
01:42:18 page /settings/notifications: 49636 bytes
01:42:21 apps running: 10
01:42:21 the public name from outside the box: 302
01:52:21 dashboard at the end of the outage: 200
01:52:26 the box's own log, warnings since the cut:
01:52:28 RESTORE: ok
01:52:28 now 00:52:28 UTC
01:53:47 back after 60 s: the public name 302; events pushed since 0
01:58:52 events since the restore:
01:58:52 host agent:
01:58:54 routes after: default via 192.168.0.1 dev vmbr0 onlink
default via 192.168.0.1 dev eth0
@@ -0,0 +1,38 @@
02:19:39 === wan: before
02:19:39 now 01:19:39 UTC
02:19:40 routes host/guest: default via 192.168.0.1 dev vmbr0 onlink
default via 192.168.0.1 dev eth0
02:19:44 apps: 10
02:19:44 the public name's edge address: 104.21.22.132
02:19:44 the public name through the edge: 302
02:19:46 CUT: ok
02:19:46 now 01:19:46 UTC
02:20:22 from the box: hub 000 internet 000
02:20:22 the dashboard on the home network: login ok, /api/health 200
02:20:22 page /: 32 bytes in 0.0 s
02:22:38 page /backups: 11 bytes in 135.3 s
02:22:38 page /settings/notifications: 49636 bytes in 0.0 s
02:22:41 apps running: 10
02:22:41 the public name through the edge: 302
02:32:41 the public name through the edge, at the end: 302
02:32:41 dashboard at the end of the outage: 200
02:32:46 the box's own log, warnings since the cut:
1 [WARN] [report] wait channel error: context deadline exceeded backing off (the 15-min cycle still reconciles)
1 [WARN] [monitor] CPU threshold breached: 94% (limit: 90%)
1 [WARN] [backup] same-disk: agent call failed for /mnt/sys_drive / /mnt/felhom-drives/adatlemez: agentapi: POST /disks/same-disk: P
1 [WARN] Event push failed after 3 attempts (health_degraded/warning): Post "https://hub.felhom.eu/api/v1/event": context deadline e
1 [WARN] Event push failed after 3 attempts (db_dump_completed/info): Post "https://hub.felhom.eu/api/v1/event": context deadline ex
1 [ERROR] [quiesce] cycle error: check due: agentapi: GET /backup/tiers: Get "https://169.254.253.1:8443/backup/tiers": context dead
02:32:49 RESTORE: ok
02:32:49 now 01:32:49 UTC
02:34:05 back after 60 s: the public name 302; events pushed since 0
02:39:08 events since the restore:
2026/11/03 01:36:25 [INFO] Event pushed: health_recovered (info) [hu-only] Rendszer llapot helyrellt: ok (volt: warn)
02:39:08 host agent: Nov 03 02:27:45 felhom felhom-agent[2609044]: time=2026-11-03T02:27:45.578+01:00 level=WARN msg="hub: report failed; keeping current interval" err="hub: transport error:
02:39:10 routes after: default via 192.168.0.1 dev vmbr0 onlink
default via 192.168.0.1 dev eth0
@@ -0,0 +1,38 @@
02:19:39 === wan: before
02:19:39 now 01:19:39 UTC
02:19:40 routes host/guest: default via 192.168.0.1 dev vmbr0 onlink
default via 192.168.0.1 dev eth0
02:19:44 apps: 10
02:19:44 the public name's edge address: 104.21.22.132
02:19:44 the public name through the edge: 302
02:19:46 CUT: ok
02:19:46 now 01:19:46 UTC
02:20:22 from the box: hub 000 internet 000
02:20:22 the dashboard on the home network: login ok, /api/health 200
02:20:22 page /: 32 bytes in 0.0 s
02:22:38 page /backups: 11 bytes in 135.3 s
02:22:38 page /settings/notifications: 49636 bytes in 0.0 s
02:22:41 apps running: 10
02:22:41 the public name through the edge: 302
02:32:41 the public name through the edge, at the end: 302
02:32:41 dashboard at the end of the outage: 200
02:32:46 the box's own log, warnings since the cut:
1 [WARN] [report] wait channel error: context deadline exceeded backing off (the 15-min cycle still reconciles)
1 [WARN] [monitor] CPU threshold breached: 94% (limit: 90%)
1 [WARN] [backup] same-disk: agent call failed for /mnt/sys_drive / /mnt/felhom-drives/adatlemez: agentapi: POST /disks/same-disk: P
1 [WARN] Event push failed after 3 attempts (health_degraded/warning): Post "https://hub.felhom.eu/api/v1/event": context deadline e
1 [WARN] Event push failed after 3 attempts (db_dump_completed/info): Post "https://hub.felhom.eu/api/v1/event": context deadline ex
1 [ERROR] [quiesce] cycle error: check due: agentapi: GET /backup/tiers: Get "https://169.254.253.1:8443/backup/tiers": context dead
02:32:49 RESTORE: ok
02:32:49 now 01:32:49 UTC
02:34:05 back after 60 s: the public name 302; events pushed since 0
02:39:08 events since the restore:
2026/11/03 01:36:25 [INFO] Event pushed: health_recovered (info) [hu-only] Rendszer llapot helyrellt: ok (volt: warn)
02:39:08 host agent: Nov 03 02:27:45 felhom felhom-agent[2609044]: time=2026-11-03T02:27:45.578+01:00 level=WARN msg="hub: report failed; keeping current interval" err="hub: transport error:
02:39:10 routes after: default via 192.168.0.1 dev vmbr0 onlink
default via 192.168.0.1 dev eth0
@@ -0,0 +1,35 @@
02:43:01 === wan: before
02:43:01 now 01:43:01 UTC
02:43:02 routes host/guest: default via 192.168.0.1 dev vmbr0 onlink
default via 192.168.0.1 dev eth0
02:43:05 apps: 10
02:43:05 the public name's edge address: 172.67.205.21
02:43:06 the public name through the edge: 302
02:43:27 CUT: ok
02:43:27 now 01:43:27 UTC
02:44:12 from the box: hub 000 internet 000
02:44:12 the dashboard on the home network: login ok, /api/health 200
02:44:12 page /: 32 bytes in 0.0 s
02:46:38 page /backups: 11 bytes in 145.4 s
02:46:38 page /settings/notifications: 49636 bytes in 0.0 s
02:46:42 apps running: 10
02:46:42 the public name through the edge: 530
02:56:22 the public name through the edge, at the end: 530
02:56:22 dashboard at the end of the outage: 200
02:56:26 the box's own log, warnings since the cut:
1 [WARN] [report] wait channel error: context deadline exceeded backing off (the 15-min cycle still reconciles)
1 [WARN] [report] Push failed: Post "https://hub.felhom.eu/api/v1/report": dial tcp 192.168.0.192:443: i/o timeout (Client.Timeout e
1 [ERROR] [sync] Catalog sync failed: git fetch: git fetch --depth 1 origin main: context deadline exceeded (subprocess killed at th
1 [ERROR] [scheduler] Job hub-report failed: hub push failed after 3 attempts: Post "https://hub.felhom.eu/api/v1/report": dial tcp
02:56:27 RESTORE: ok
02:56:27 now 01:56:27 UTC
02:57:41 back after 60 s: the public name 302; events pushed since 0
03:02:46 events since the restore:
03:02:46 host agent:
03:02:48 routes after: default via 192.168.0.1 dev vmbr0 onlink
default via 192.168.0.1 dev eth0
@@ -0,0 +1,49 @@
#!/usr/bin/env python3
"""walk_crash_t1.py — claims fail.app-crash, fault.customer-email, fault.operator-email on Tester 1, front doors only.
A household's app starts crashing. A throwaway Radicale is installed through the product; its main process is killed with
kill -9, 15 s apart (never `docker kill`: the restart policy would mask it). Want: the box stops it (crash loop), holds it, the hub
hears, and a mail reaches the mailbox (read through the mailbox, a different channel). Then the household's Start, then the
product's Remove; nothing left."""
import json, os, sys, time
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
os.environ.setdefault("TARGET", "tester-1")
import box_walk as w
APP, SUB = "radicale", "radicale"
EV = os.environ["EV"]
t0 = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
w.login()
assert not w.stack(APP).get("deployed"), "radicale is already on Tester 1 — not a throwaway"
assert w.deploy(APP, SUB), "install failed"
w.wait_app(SUB, "/", tries=40)
w.say(f" installed {w.stack(APP).get('state')}; t0 {t0}")
start = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
for i in range(12):
pid = w.guest(f"docker inspect -f '{{{{.State.Pid}}}}' {APP} 2>/dev/null").strip()
if pid and pid != "0":
w.guest(f"kill -9 {pid}")
st = w.stack(APP)
w.say(f" kill -9 #{i + 1} pid {pid}: state {st.get('state')} hold {st.get('hold_reason')!r}")
if st.get("hold_reason"):
break
time.sleep(15)
time.sleep(40)
st = w.stack(APP)
w.say(f" after the kills {time.strftime('%H:%M:%S', time.gmtime())}: state={st.get('state')} hold={st.get('hold_reason')!r}")
w.say(" controller log:\n" + w.guest(f"docker logs --since {start} felhom-controller 2>&1 | grep -E 'crash_loop|STOPPING|unhealthy|app_held|app_crash|PushEvent|Event sent' | grep -v DEBUG | cut -c1-240 | tail -14"))
open(os.path.join(EV, "crash-walk-part1.log"), "w").write("\n".join(w.LOG) + "\n")
print("WAITING-FOR-MAIL", flush=True)
# the mail is read by the operator's session from the mailbox; the app stays held until then
time.sleep(int(os.environ.get("HOLD_S", "240")))
c, d = w.ctl("POST", f"/api/stacks/{APP}/start")
w.say(f" the household's Start -> {c} {str(d)[:140]}")
for _ in range(24):
time.sleep(5)
st = w.stack(APP)
if st.get("state") == "running" and not st.get("hold_reason"):
break
w.say(f" after Start: state={st.get('state')} hold={st.get('hold_reason')!r}")
w.remove(APP)
w.say(f" removed: deployed={w.stack(APP).get('deployed')}; leftovers: " + w.guest(f"docker ps -a --format '{{{{.Names}}}}' | grep -c '^{APP}' ; docker volume ls --format '{{{{.Name}}}}' | grep -c '^{APP}_'; ls /opt/docker/stacks/{APP}/app.yaml 2>&1 | tail -1").replace("\n", " "))
open(os.path.join(EV, "crash-walk.log"), "w").write("\n".join(w.LOG) + "\n")
@@ -0,0 +1,75 @@
#!/usr/bin/env python3
"""walk_crash_t1_customer.py — the household's leg of a crash mail (fail.app-crash, fault.customer-email), Tester 1.
Tester 1's notification page has no address (found empty 2026-11-02), so the first walk mailed only the operator. A household
types its address on the page and saves it (the page's own form), then an app crash-loops. Want: the Hungarian mail in the
household's mailbox (the catch-all, read through the mailbox). Then the address is cleared through the same page (its own
„confirm_clear"), the app removed. What stays: the saved list of events = the page's defaults."""
import os, re, sys, time
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
os.environ.setdefault("TARGET", "tester-1")
import box_walk as w
APP, SUB = "radicale", "radicale"
EV, ADDRESS = os.environ["EV"], "tester1@felhom.eu"
def form(fields):
sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip()
csrf = open(f"{w.SC}/csrf{os.getpid()}.txt").read().strip()
args = ["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}", "-X", "POST", "--data-urlencode", f"_csrf={csrf}"]
for k, v in fields:
args += ["--data-urlencode", f"{k}={v}"]
r = w.sh(args + [f"{w.BASE}/settings/notifications"], timeout=60)
return (r.stdout or "").split("\n")[0].strip()
w.login()
page = w.page("/settings/notifications")
page = page if isinstance(page, str) else str(page)
fields = [("notification_email", ADDRESS), ("cooldown_hours", "6")]
for m in re.finditer(r'<input type="hidden" name="(group_[a-z]+_shown)" value="([^"]*)"', page):
fields.append((m.group(1), m.group(2)))
checked = []
for tag in re.findall(r"<input[^>]*type=\"checkbox\"[^>]*>", page):
n = re.search(r'name="([a-z_]+)"', tag)
if n and " checked" in tag:
fields.append((n.group(1), "on"))
checked.append(n.group(1))
w.say(f" checked as found: {checked}")
if not any(c in ("group_apps", "event_app_stopped_unhealthy") for c in checked):
fields.append(("group_apps", "on"))
w.say(" the group „my apps” was off: switched on, as a household that wants this mail would")
w.say(f" save with the address -> {form(fields)}")
time.sleep(20) # the page's save syncs the preferences to the hub
assert not w.stack(APP).get("deployed"), "radicale is on the box — not a throwaway"
assert w.deploy(APP, SUB), "install failed"
w.wait_app(SUB, "/", tries=40)
start = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
w.say(f" installed; kills start {start}")
for i in range(12):
pid = w.guest(f"docker inspect -f '{{{{.State.Pid}}}}' {APP} 2>/dev/null").strip()
if pid and pid != "0":
w.guest(f"kill -9 {pid}")
st = w.stack(APP)
w.say(f" kill -9 #{i + 1} pid {pid}: state {st.get('state')} hold {str(st.get('hold_reason'))[:40]!r}")
if st.get("hold_reason"):
break
time.sleep(15)
time.sleep(60)
w.say(" controller log:\n" + w.guest(f"docker logs --since {start} felhom-controller 2>&1 | grep -E 'STOPPING|Event pushed' | cut -c1-200 | tail -6"))
c, d = w.ctl("POST", f"/api/stacks/{APP}/start")
w.say(f" Start -> {c}")
for _ in range(24):
time.sleep(5)
st = w.stack(APP)
if st.get("state") == "running" and not st.get("hold_reason"):
break
w.remove(APP)
w.say(f" removed: deployed={w.stack(APP).get('deployed')}")
w.say(f" the address cleared -> {form([('notification_email', ''), ('confirm_clear', '1'), ('cooldown_hours', '6')] + [f for f in fields if f[0] not in ('notification_email', 'cooldown_hours')])}")
time.sleep(5)
p2 = w.page("/settings/notifications")
p2 = p2 if isinstance(p2, str) else str(p2)
w.say(" the page's address now: " + (re.search(r'name="notification_email"\s*value="([^"]*)"', p2) or [None, "?"])[1])
open(os.path.join(EV, "crash-walk-customer.log"), "w").write("\n".join(w.LOG) + "\n")
@@ -0,0 +1,64 @@
#!/usr/bin/env python3
"""walk_fill_t1.py — claims backup.fill-warning and fail.drive-filling on Tester 1: a data drive fills; is the household warned BEFORE it is full, in Hungarian,
naming the drive and the free space, once? A test file (fallocate, no write load) takes `adatlemez` over 85 %; the watcher runs every 10 minutes (R-363).
The household's address is typed on the notification page and cleared after. The file goes after the mail; the back-to-normal line is read."""
import os, re, sys, time
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
os.environ.setdefault("TARGET", "tester-1")
import box_walk as w
EV, ADDRESS = os.environ["EV"], "tester1@felhom.eu"
DRIVE = "/mnt/felhom-drives/adatlemez"
FILE = f"{DRIVE}/sec1102-fill.bin"
def form(fields):
sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip()
csrf = open(f"{w.SC}/csrf{os.getpid()}.txt").read().strip()
args = ["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}", "-X", "POST", "--data-urlencode", f"_csrf={csrf}"]
for k, v in fields:
args += ["--data-urlencode", f"{k}={v}"]
return (w.sh(args + [f"{w.BASE}/settings/notifications"], timeout=60).stdout or "").split("\n")[0].strip()
def log_since(t, rx):
return w.guest(f"docker logs --since {t} felhom-controller 2>&1 | grep -aE '{rx}' | grep -av DEBUG | cut -b1-260 | iconv -c -t ascii//TRANSLIT | tail -8")
w.login()
page = w.page("/settings/notifications")
page = page if isinstance(page, str) else str(page)
fields = [("notification_email", ADDRESS), ("cooldown_hours", "6")]
for m in re.finditer(r'<input type="hidden" name="(group_[a-z]+_shown)" value="([^"]*)"', page):
fields.append((m.group(1), m.group(2)))
fields.append(("group_machine", "on"))
w.say(f" the address saved -> {form(fields)}")
time.sleep(10)
w.say(" before: " + w.guest(f"df -h --output=size,used,avail,pcent {DRIVE} | tail -1"))
avail = int(w.guest(f"df -B1G --output=size {DRIVE} | tail -1").strip()) * 86 // 100 - int(w.guest(f"df -B1G --output=used {DRIVE} | tail -1").strip())
w.say(f" filling with {avail} GiB: " + w.guest(f"fallocate -l {avail}G {FILE} && df -h --output=used,avail,pcent {DRIVE} | tail -1"))
t0 = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
for _ in range(50):
time.sleep(15)
got = log_since(t0, "fillwatch.*(notification|warning|critical)")
if re.search(r"[1-9] notification", got):
break
w.say(f" the watcher, {time.strftime('%H:%M:%S', time.gmtime())}:\n{got}")
w.say(" events:\n" + log_since(t0, "Event pushed: (disk_|storage_)"))
time.sleep(int(os.environ.get("MAIL_WAIT_S", "90")))
w.say(" removing the file: " + w.guest(f"rm -f {FILE}; df -h --output=used,avail,pcent {DRIVE} | tail -1"))
t1 = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
for _ in range(50):
time.sleep(15)
got = log_since(t1, "fillwatch.*[1-9] notification|Event pushed: (disk_|storage_)")
if got.strip():
break
w.say(f" back to normal, {time.strftime('%H:%M:%S', time.gmtime())}:\n{got}")
sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip()
w.say(" the address cleared -> " + form([("notification_email", ""), ("confirm_clear", "1"), ("cooldown_hours", "6")]))
time.sleep(3)
p2 = w.page("/settings/notifications")
p2 = p2 if isinstance(p2, str) else str(p2)
w.say(" the page's address now: " + repr((re.search(r'name="notification_email"\s*value="([^"]*)"', p2) or [None, "?"])[1]))
w.say(" left on the drive: " + w.guest(f"ls -la {DRIVE} | grep -c sec1102; df -h --output=used,avail,pcent {DRIVE} | tail -1"))
open(os.path.join(EV, "fill-walk.log"), "w").write("\n".join(w.LOG) + "\n")
@@ -0,0 +1,125 @@
#!/usr/bin/env python3
"""walk_outage_t1.py hub|wan — claims fail.hub-down, fail.internet-down, use.lan-fallback on Tester 1 (VM 341), the household's view.
hub: the hub's address (192.168.0.192 for this LAN) is blackholed on the box's host and guest for MINUTES minutes: from the box's side the
hub (and Gitea, same address) is down. DooPlex is not touched. Want: the dashboard works, the apps run, an app can be installed from
the cached catalog; a crash-loop stop during the outage holds the app; what reaches the hub afterwards is read.
wan: packets to the outside are dropped on host and guest, the hub's LAN address too: the internet is gone, the LAN stays. Want: the dashboard answers
on the home network, the apps run, the public name does not; afterwards the tunnel and the reports come back by themselves.
A safety job on the host puts the routes back after MINUTES+10 minutes whatever happens to this script. Nothing else on the box is changed."""
import json, os, subprocess, sys, time
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
os.environ.setdefault("TARGET", "tester-1")
import box_walk as w
MODE = sys.argv[1]
MIN = int(os.environ.get("MINUTES", "12"))
EV = os.environ["EV"]
APP, SUB = "radicale", "radicale"
HUB = "192.168.0.192"
GW = "192.168.0.1"
def host(cmd, timeout=120):
r = w.sh(w.ssh_args() + [cmd], timeout=timeout)
return ((r.stdout or "") + (r.stderr or "")).strip()
def say_times():
w.say(f" now {time.strftime('%H:%M:%S', time.gmtime())} UTC")
UNDO_HUB = f"ip route del blackhole {HUB}/32 2>/dev/null; pct exec 9201 -- ip route del blackhole {HUB}/32 2>/dev/null"
# Try 1 and 2 removed the default route: the agent's self-heal put it back in about 100 s (guestnet: „no default route ... healing"), so the
# tunnel returned by itself. A dead line keeps its route: packets to the outside are DROPPED instead, the LAN stays, the hub (LAN address here) goes too.
FW = ("iptables -N CUT1102; iptables -A CUT1102 -d {hub}/32 -j DROP; "
"for n in 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 169.254.0.0/16; do iptables -A CUT1102 -d $n -j RETURN; done; iptables -A CUT1102 -j DROP; "
"iptables -I OUTPUT 1 -j CUT1102; iptables -I FORWARD 1 -j CUT1102").format(hub=HUB)
FW_UNDO = "iptables -D OUTPUT -j CUT1102; iptables -D FORWARD -j CUT1102; iptables -F CUT1102; iptables -X CUT1102"
UNDO_WAN = f"{FW_UNDO}; pct exec 9201 -- sh -c '{FW_UNDO}'"
UNDO = UNDO_WAN if MODE == "wan" else UNDO_HUB
w.login()
w.say(f" === {MODE}: before")
say_times()
w.say(" routes host/guest: " + host("ip route | grep default; pct exec 9201 -- ip route | grep default"))
w.say(" apps: " + w.guest("docker ps --format '{{.Names}}' | wc -l"))
pub = "https://felhom.enkicsifelhom.hu/"
# From this LAN the name may not go through Cloudflare (try 1: 302 all through the cut): ask a public resolver and go to the edge's address.
doh = w.sh(["curl", "-s", "-m", "20", "-H", "accept: application/dns-json", "https://cloudflare-dns.com/dns-query?name=felhom.enkicsifelhom.hu&type=A"], timeout=30).stdout or ""
EDGE = (__import__("re").findall(r'"data":"(\d+\.\d+\.\d+\.\d+)"', doh) or [""])[0]
w.say(f" the public name's edge address: {EDGE}")
def public():
r = w.sh(["curl", "-sk", "-m", "25", "-o", "/dev/null", "-w", "%{http_code}", "--resolve", f"felhom.enkicsifelhom.hu:443:{EDGE}", pub], timeout=40)
return r.stdout
w.say(f" the public name through the edge: {public()}")
host(f"nohup sh -c 'sleep {(MIN + 10) * 60}; {UNDO}' >/dev/null 2>&1 &", timeout=20)
t0 = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
cut = f"ip route add blackhole {HUB}/32; pct exec 9201 -- ip route add blackhole {HUB}/32"
if MODE == "wan":
cut = f"{FW}; pct exec 9201 -- sh -c '{FW}'"
w.say(" CUT: " + (host(cut) or "ok"))
say_times()
time.sleep(30)
w.say(" from the box: " + w.guest(f"curl -s -m 6 -o /dev/null -w 'hub %{{http_code}} ' https://hub.felhom.eu/; curl -s -m 6 -o /dev/null -w 'internet %{{http_code}}' https://www.cloudflare.com/ ; echo"))
deadline = time.time() + MIN * 60
try:
w.login()
code, h = w.ctl("GET", "/api/health")
w.say(f" the dashboard on the home network: login ok, /api/health {code}")
for page in ("/", "/backups", "/settings/notifications"):
t = time.time()
p = w.page(page)
w.say(f" page {page}: {len(p if isinstance(p, str) else str(p))} bytes in {time.time() - t:.1f} s")
w.say(" apps running: " + w.guest("docker ps --format '{{.Names}}: {{.Status}}' | grep -vc Exited"))
w.say(f" the public name through the edge: {public()}")
if MODE == "hub":
ok = w.deploy(APP, SUB)
w.say(f" an install during the outage (cached catalog): {ok}, state {w.stack(APP).get('state')}")
for i in range(12):
pid = w.guest(f"docker inspect -f '{{{{.State.Pid}}}}' {APP} 2>/dev/null").strip()
if pid and pid != "0":
w.guest(f"kill -9 {pid}")
if w.stack(APP).get("hold_reason"):
break
time.sleep(15)
time.sleep(25)
st = w.stack(APP)
w.say(f" after the kills: state={st.get('state')} held={bool(st.get('hold_reason'))}")
w.say(" controller log:\n" + w.guest(f"docker logs --since {t0} felhom-controller 2>&1 | grep -aE 'STOPPING|Event push failed|Event pushed|holdreport' | grep -av DEBUG | tail -12 | cut -b1-170 | iconv -c -t ascii"))
while time.time() < deadline:
time.sleep(20)
w.say(f" the public name through the edge, at the end: {public()}")
w.say(" dashboard at the end of the outage: " + str(w.ctl("GET", "/api/health")[0]))
w.say(" the box's own log, warnings since the cut:\n" + w.guest(f"docker logs --since {t0} felhom-controller 2>&1 | grep -aE '\\[(WARN|ERROR)\\]' | grep -av DEBUG | cut -b1-150 | iconv -c -t ascii | sed -E 's/^[0-9/]+ [0-9:]+ //' | sort | uniq -c | sort -rn | head -14"))
finally:
w.say(" RESTORE: " + (host(UNDO) or "ok"))
say_times()
t1 = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
for i in range(40):
time.sleep(15)
tun = public()
got = w.guest(f"docker logs --since {t1} felhom-controller 2>&1 | grep -ac 'Event pushed'").strip()
if tun in ("200", "302", "303", "401", "403") and (MODE == "hub" or i > 2):
w.say(f" back after {(i + 1) * 15} s: the public name {tun}; events pushed since {got}")
break
time.sleep(int(os.environ.get("AFTER_S", "420"))) # the next reports reach the hub
w.say(" events since the restore:\n" + w.guest(f"docker logs --since {t1} felhom-controller 2>&1 | grep -a 'Event pushed' | cut -b1-130 | iconv -c -t ascii | tail -10"))
w.say(" host agent: " + host(f"journalctl -u felhom-agent --since '{t0.replace('T', ' ').replace('Z', '')} UTC' --no-pager 2>/dev/null | grep -aE 'report|hub' | grep -aiE 'fail|ok|sent|error' | tail -4 | cut -c1-170"))
if MODE == "hub":
st = w.stack(APP)
if st.get("hold_reason"):
c, d = w.ctl("POST", f"/api/stacks/{APP}/start")
w.say(f" the household's Start -> {c}")
for _ in range(24):
time.sleep(5)
if w.stack(APP).get("state") == "running":
break
w.remove(APP)
w.say(f" removed: deployed={w.stack(APP).get('deployed')}")
w.say(" routes after: " + host("ip route | grep -E 'default|blackhole'; pct exec 9201 -- ip route | grep -E 'default|blackhole'"))
open(os.path.join(EV, f"outage-{MODE}.log"), "w").write("\n".join(w.LOG) + "\n")
@@ -0,0 +1,24 @@
YAML-IDENTICAL
26: repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
2026/11/03 03:04:56 sync.go:261: [INFO] [sync] catalog moved 87d0254fb2a2 -> c760fca93898 (0 new, 1 stack dir(s) changed)
2026/11/03 03:04:56 sync.go:299: [INFO] [sync] Catalog sync complete
2026/11/03 03:04:56 sync.go:142: [INFO] [sync] Initial sync: Sablonok frisstve frisstve: sparkyfitness
cache HEAD c760fca
https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git
04:05:11 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['PAPERLESS_ADMIN_PASSWORD', 'HDD_PATH']
04:05:12 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
04:07:27 [1] deployed, controller state=running, pinned={'paperless-postgres': 'postgres:18-alpine', 'paperless-redis': 'redis:7-alpine', 'paperless-webserver': 'ghcr.io/paperless-ngx/paperless-ngx:2.20.15'}
paperless installed: True running
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 58 seconds (healthy)
paperless-redis redis:7-alpine Up About a minute (healthy)
paperless-postgres postgres:18-alpine Up About a minute (healthy)
filebrowser gtstef/filebrowser:1.5.8-stable Up 2 minutes (healthy)
traefik traefik:v3.7.16 Up 3 hours
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 Up 2 minutes (healthy)
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1
0
holds: ['radicale']
0
@@ -0,0 +1,30 @@
#!/usr/bin/env python3
"""teardown9202.py — scratch 9202 back as found: Paperless installed again through the product (the walks replaced the throwaway of 2026-09-22; its 90 MB of test
data went with it, the box's own page calls apps here throwaways), the box back on the live catalog, its catalog yaml byte-identical."""
import os, sys, time
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
os.environ.setdefault("TARGET", "9202")
os.environ.setdefault("KEPT", "fresh")
import box_walk as w
VOL = "/var/lib/docker/volumes/felhom-controller-data/_data"
SAVE = f"{VOL}/controller.yaml.pre-sec1102"
w.login()
print(w.guest(f"""set -e
cp -p {SAVE} {VOL}/controller.yaml
cmp {SAVE} {VOL}/controller.yaml && echo YAML-IDENTICAL; grep -n repo_url {VOL}/controller.yaml
rm -f {SAVE}
docker restart felhom-controller >/dev/null"""))
for _ in range(60):
time.sleep(3)
try:
w.login()
break
except SystemExit:
continue
w.sync_rescan()
time.sleep(5)
print(w.guest(f"docker logs --since 2m felhom-controller 2>&1 | grep -a '\\[sync\\]' | grep -av DEBUG | tail -3 | cut -b1-150 | iconv -c -t ascii; git -C {VOL}/data/catalog-cache log -1 --format='cache HEAD %h' 2>/dev/null; git -C {VOL}/data/catalog-cache remote get-url origin 2>/dev/null"))
ok = w.deploy("paperless-ngx", "paperless", None)
print("paperless installed:", ok, w.stack("paperless-ngx").get("state"))
print(w.guest("docker ps --format '{{.Names}} {{.Image}} {{.Status}}'; cat /etc/felhom-controller-image; docker volume ls --format '{{.Name}}' | grep -cE 'pre-update' ; python3 -c \"import json;s=json.load(open('" + VOL + "/data/settings.json'));print('holds:',sorted((s.get('restore_holds') or {}).keys()))\"; rm -f /tmp/sec1102-sink-keep.jsonl; ls /tmp | grep -c sec1102"))
@@ -0,0 +1,133 @@
#!/usr/bin/env python3
"""undo_proof2.py APP MAIN_SVC FROM(svc=ref,…) TO(svc=ref,…) — a big step's UNDO, proven on scratch 9202 through the product's
own guarded Update (generalised from audits/security-app-updates-2026-10-27/part3/tools/undo_proof.py, Jellyfin):
1. the drill's APP must stand at FROM (this tool checks, never guesses); install, seed with the box fixture, read back (C1);
2. a temporary drill commit moves the images to TO AND makes the checks fail (MAIN_SVC's compose healthcheck `false`,
the .felhom.yml probe on a path that does not exist) — so TO starts on the data, runs its migrations, and the
Update fails in its check;
3. press the guarded Update: the final phase must be `undone`; MAIN_SVC's log is saved while TO verifies;
4. after the undo: the running images are FROM and the seed reads back;
5. the temporary drill commit is reverted (the template equal to before), the app removed.
usage: SC=<dir with .ctlpw> TARGET=9202 EV=<dir> python3 undo_proof2.py APP MAIN_SVC FROM TO [DEFINITION_DIR]"""
import json, os, re, subprocess, sys, threading, time
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
import box_walk as w
import upgrade_fixtures_box as fb
import upgrade_fixtures_box28 as f28
APP, MAIN = sys.argv[1], sys.argv[2]
FROM = dict(x.split("=", 1) for x in sys.argv[3].split(","))
TO = dict(x.split("=", 1) for x in sys.argv[4].split(","))
DEF = sys.argv[5] if len(sys.argv) > 5 else None # a definition step: TO is that whole definition (box_walk --to-definition)
FX = dict(f28.FIXTURES28); FX.update(fb.FIXTURES)
fx = FX[APP]
SUB = getattr(fx, "sub", APP)
D = w.DRILL
EV = os.environ["EV"]
os.makedirs(EV, exist_ok=True)
OUT = os.path.join(EV, "undo-proof-%s.json" % APP)
rec = {"app": APP, "from": FROM, "to": TO, "started": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())}
def git(*a):
return subprocess.run(["git", "-C", D] + list(a), capture_output=True, text=True, timeout=120)
def finish(verdict, why):
rec["verdict"], rec["why"] = verdict, why
w.say(f"RESULT {verdict} — {why}")
def failing_to(c, y):
if DEF:
c = open(os.path.join(DEF, "docker-compose.yml")).read()
y = open(os.path.join(DEF, ".felhom.yml")).read()
assert w.images_of(c) == TO, (w.images_of(c), TO)
for svc in ({} if DEF else FROM):
if FROM[svc] != TO[svc]:
assert f"image: {FROM[svc]}" in c, svc
c = c.replace(f"image: {FROM[svc]}", f"image: {TO[svc]}")
# MAIN's own block only (never past the next service), comment lines allowed between healthcheck: and test:
m = re.search(r"(?m)^ %s:\n(?:(?! \S)[^\n]*\n)*? healthcheck:\n(?:\s*#[^\n]*\n)*(\s+)test: [^\n]*\n" % re.escape(MAIN), c)
if m:
c = c[:m.start()] + m.group(0).replace(m.group(0).split("\n")[-2], m.group(1) + 'test: ["CMD", "false"]') + c[m.end():]
else: # no test line of our own (the image's own): add a failing healthcheck under the service
c = re.sub(r"(?m)^( %s:\n)" % re.escape(MAIN), r'\1 healthcheck:\n test: ["CMD", "false"]\n', c, count=1)
# port 1: nothing listens there, so the probe fails on ANY app — an app that answers 200 on every path (a single-page
# app: Termix, 2026-10-28) passed a „no such path" probe and the update ended `done`
y2 = re.sub(r"(?m)^(healthcheck:\n(?: #[^\n]*\n| container: [^\n]*\n)* checks:\n - type: )\w+(\n port: )\d+\n(?: path: [^\n]*\n)?(?: expect:\n status: \d+\n)?",
r'\1api\g<2>1\n path: "/undo-proof-no-such-path"\n expect:\n status: 200\n', y, count=1)
return c, y2
w.login()
git("pull", "-q", "--rebase", "origin", "main")
head0 = git("rev-parse", "HEAD").stdout.strip()
rec["drill_head_before"] = head0
comp, fy = f"{D}/templates/{APP}/docker-compose.yml", f"{D}/templates/{APP}/.felhom.yml"
commit = None
try:
if w.images_of(open(comp).read()) != FROM:
finish("inconclusive", f"the drill's {APP} is not at FROM"); raise SystemExit(1)
if w.stack(APP).get("deployed"):
w.remove(APP)
w.refresh_box_mirror()
w.sync_rescan()
if not w.deploy(APP, SUB):
finish("inconclusive", "the install at FROM did not complete"); raise SystemExit(1)
tok = fx.seed(w, SUB, w.say)
if not tok or not fx.verify(w, SUB, tok, w.say):
finish("inconclusive", "C1: the seed did not read back before the update"); raise SystemExit(1)
c, y = failing_to(open(comp).read(), open(fy).read())
rec["failing_compose_main_healthcheck"] = re.findall(r'test: \["CMD", "false"\]', c)
rec["failing_probe"] = "/undo-proof-no-such-path" in y
open(comp, "w").write(c); open(fy, "w").write(y)
git("add", f"templates/{APP}")
git("commit", "-q", "-m", f"UNDO-PROOF: {APP} at TO whose checks fail (temporary, reverted at the end)")
git("push", "-q", "origin", "main")
commit = git("rev-parse", "--short=12", "HEAD").stdout.strip()
w.say(f" drill commit {commit}: {APP} TO with failing checks")
w.refresh_box_mirror()
main_to = TO[MAIN]
w.sync_rescan(APP, main_to, tries=40, delay=10)
seen = {"lines": []}
stop = threading.Event()
def watch():
while not stop.is_set():
out = w.guest(f"docker inspect -f '{{{{.Config.Image}}}}' {MAIN} 2>/dev/null; docker logs --tail 300 {MAIN} 2>&1 | tail -40")
if main_to.split(":")[-1] in out.split("\n")[0]:
seen["lines"] = out.strip().split("\n")
time.sleep(5)
th = threading.Thread(target=watch, daemon=True)
th.start()
rec["update"] = u = w.press_update(APP, poll=1.0, cap_s=1800)
stop.set(); th.join(timeout=30)
rec["to_log_while_verifying"] = seen["lines"][-40:]
time.sleep(15)
rec["observables_after"] = w.observables(APP)
# by the compose service, not a container of that name: SparkyFitness's frontend service runs in a container called `sparkyfitness`
running = {s: w.guest(f"docker ps --filter label=com.docker.compose.project={APP} --filter label=com.docker.compose.service={s} --format '{{{{.Image}}}}' | head -1").strip() for s in FROM}
rec["running_images_after"] = running
if u.get("final_phase") != "undone":
finish("failed", f"the update ended {u.get('final_phase')!r}, not 'undone'"); raise SystemExit(1)
if any(running[s].split("@")[0] != FROM[s] for s in FROM):
finish("failed", f"after the undo the running images are {running}, not FROM"); raise SystemExit(1)
rec["seed_after_undo"] = ok = fx.verify(w, SUB, tok, w.say)
if not ok:
finish("failed", "the undo put the old images back but the seed does not read back"); raise SystemExit(1)
finish("proven", "TO started on the data and failed its check; the box undid it: FROM runs again and the seed reads back")
finally:
if w.stack(APP).get("deployed"):
w.remove(APP)
if commit:
git("pull", "-q", "--rebase", "origin", "main")
r = git("revert", "--no-edit", commit)
git("push", "-q", "origin", "main")
w.refresh_box_mirror()
w.say(f" drill revert {commit}: rc={r.returncode}")
rec["drill_head_after"] = git("rev-parse", "HEAD").stdout.strip()
rec["drill_tree_equal_to_before"] = git("diff", "--quiet", head0, "HEAD", "--", f"templates/{APP}").returncode == 0
w.say(f" drill templates/{APP} equal to before: {rec['drill_tree_equal_to_before']}")
rec["log"] = w.LOG
json.dump(rec, open(OUT, "w"), indent=2)
@@ -0,0 +1,29 @@
{
"app": "paperless-ngx",
"walk": "security",
"box_walk_version": 1,
"verdict": "inconclusive",
"venue": "box 9202 (drill catalog), the guarded Update",
"from": {
"paperless-webserver": "ghcr.io/paperless-ngx/paperless-ngx:2.20.15",
"paperless-postgres": "postgres:18-alpine",
"paperless-redis": "redis:7-alpine"
},
"to": {
"paperless-webserver": "ghcr.io/paperless-ngx/paperless-ngx:3.4.1",
"paperless-postgres": "postgres:18-alpine",
"paperless-redis": "redis:7-alpine"
},
"why": "the install at FROM did not complete — nothing was tested",
"log": [
"01:46:15 [1] removing the existing paperless-ngx first (scratch box)",
"01:46:22 [X] stop -> 200 {'ok': True, 'message': 'Stack paperless-ngx stop completed'}",
"01:46:27 [X] remove (with drive data) -> 409 {'ok': False, 'error': 'A(z) /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx tárhely jelenleg nem elérhető — az alkalmazás nem távolítható el, amíg a megh",
"01:46:27 [X] refused because the drive path cannot be resolved (R-442, fail-closed and right) — removing the app and KEEPING the drive data instead",
"01:46:54 [X] remove (keeping drive data) -> 200 {'ok': True, 'data': {'removed': 'paperless-ngx', 'volumes_removed': ['paperless-ngx_paperless_data', 'paperless-ngx_paperless_postgres_data', 'paperless-ngx_pa",
"01:47:02 [X] after remove: deployed=False leftovers=''",
"01:47:10 [5] the box's drill mirror now at c760fca93",
"01:47:15 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['PAPERLESS_ADMIN_PASSWORD', 'HDD_PATH']",
"01:47:15 [1] deploy -> 409 {'ok': False, 'data': {'code': 'kept_data_choice', 'title': 'Ennek az alkalmazásnak megvannak a régi adatai', 'body': 'A"
]
}
@@ -0,0 +1,5 @@
01:47:25 [5] the box's drill mirror now at c760fca93
01:47:29 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['PAPERLESS_ADMIN_PASSWORD', 'HDD_PATH']
01:47:29 [1] deploy -> 409 {'ok': False, 'data': {'code': 'kept_data_choice', 'title': 'Ennek az alkalmazásnak megvannak a régi adatai', 'body': 'A
01:47:29 RESULT inconclusive — the install at FROM did not complete
01:47:29 drill templates/paperless-ngx equal to before: True
@@ -0,0 +1,26 @@
{
"app": "paperless-ngx",
"from": {
"paperless-webserver": "ghcr.io/paperless-ngx/paperless-ngx:2.20.15",
"paperless-postgres": "postgres:18-alpine",
"paperless-redis": "redis:7-alpine"
},
"to": {
"paperless-webserver": "ghcr.io/paperless-ngx/paperless-ngx:3.4.1",
"paperless-postgres": "postgres:18-alpine",
"paperless-redis": "redis:7-alpine"
},
"started": "2026-11-03T00:47:16Z",
"drill_head_before": "c760fca938987f643d42963aebca6e27ed2bbea3",
"verdict": "inconclusive",
"why": "the install at FROM did not complete",
"drill_head_after": "c760fca938987f643d42963aebca6e27ed2bbea3",
"drill_tree_equal_to_before": true,
"log": [
"01:47:25 [5] the box's drill mirror now at c760fca93",
"01:47:29 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['PAPERLESS_ADMIN_PASSWORD', 'HDD_PATH']",
"01:47:29 [1] deploy -> 409 {'ok': False, 'data': {'code': 'kept_data_choice', 'title': 'Ennek az alkalmaz\u00e1snak megvannak a r\u00e9gi adatai', 'body': 'A",
"01:47:29 RESULT inconclusive \u2014 the install at FROM did not complete",
"01:47:29 drill templates/paperless-ngx equal to before: True"
]
}
@@ -0,0 +1,11 @@
01:46:15 [1] removing the existing paperless-ngx first (scratch box)
01:46:22 [X] stop -> 200 {'ok': True, 'message': 'Stack paperless-ngx stop completed'}
01:46:27 [X] remove (with drive data) -> 409 {'ok': False, 'error': 'A(z) /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx tárhely jelenleg nem elérhető — az alkalmazás nem távolítható el, amíg a megh
01:46:27 [X] refused because the drive path cannot be resolved (R-442, fail-closed and right) — removing the app and KEEPING the drive data instead
01:46:54 [X] remove (keeping drive data) -> 200 {'ok': True, 'data': {'removed': 'paperless-ngx', 'volumes_removed': ['paperless-ngx_paperless_data', 'paperless-ngx_paperless_postgres_data', 'paperless-ngx_pa
01:47:02 [X] after remove: deployed=False leftovers=''
01:47:10 [5] the box's drill mirror now at c760fca93
01:47:15 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['PAPERLESS_ADMIN_PASSWORD', 'HDD_PATH']
01:47:15 [1] deploy -> 409 {'ok': False, 'data': {'code': 'kept_data_choice', 'title': 'Ennek az alkalmazásnak megvannak a régi adatai', 'body': 'A
01:47:15 [9] verdict inconclusive -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-2026-11-02/walks/paperless-drill/verdict.json
01:47:15 RESULT inconclusive — the install at FROM did not complete — nothing was tested
@@ -0,0 +1,224 @@
{
"app": "mealie",
"walk": "security",
"box_walk_version": 1,
"verdict": "proven",
"venue": "box 9202 (drill catalog), the guarded Update",
"from": {
"mealie": "ghcr.io/mealie-recipes/mealie:v3.29.0"
},
"to": {
"mealie": "ghcr.io/mealie-recipes/mealie:v3.30.0"
},
"observables_before": {
"pinned_images": {
"mealie": "ghcr.io/mealie-recipes/mealie:v3.29.0"
},
"installed_images": {
"mealie": "ghcr.io/mealie-recipes/mealie:v3.29.0"
},
"catalog_images": null,
"live_compose_image_lines": [
"image: ghcr.io/mealie-recipes/mealie:v3.29.0@sha256:78eb28b8776d0bad40c5e605c72b3f4f5976793e491558c7dcbdfc04c835d4bc"
],
"docker_inspect": [
"mealie ghcr.io/mealie-recipes/mealie:v3.29.0@sha256:78eb28b8776d0bad40c5e605c72b3f4f5976793e491558c7dcbdfc04c835d4bc running=true restarts=0"
]
},
"setup_gate_before": {
"browser": {
"status": "302",
"redirect": "https://felhom.enkisfelhom.hu/__gate/start?…"
},
"script": {
"status": "401",
"redirect": null
}
},
"seed": "read_back",
"drill_commit": "c98dffe18656",
"sync_waited_s": 4.5,
"badge_before": {
"hu": [
{
"title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.",
"text": "Frissítés elérhető — ma"
}
],
"en": [
{
"title": "A newer version of this app is available. Select the Update button to start it.",
"text": "Update available — today"
}
]
},
"measured_at": "2026-11-03T00:20:08Z",
"update": {
"accepted": true,
"http": "202",
"phases": [
{
"t": 0.0,
"phase": "backing-up",
"label": "Biztonsági mentés készül a frissítés előtt…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 4.1,
"phase": "safety-dump",
"label": "Adatbázis pillanatkép…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 5.1,
"phase": "pulling",
"label": "Új verzió letöltése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 48.1,
"phase": "copying",
"label": "Az adatok másolása a frissítés előtt…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 53.3,
"phase": "verifying",
"label": "Működés ellenőrzése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 68.6,
"phase": "done",
"label": "Frissítve",
"updating": false,
"error": null,
"hold": null
}
],
"duration_s": 68.7,
"final_phase": "done",
"update_error": null,
"hold_reason": null,
"state": "running"
},
"observables_after": {
"pinned_images": {
"mealie": "ghcr.io/mealie-recipes/mealie:v3.30.0"
},
"installed_images": {
"mealie": "ghcr.io/mealie-recipes/mealie:v3.30.0"
},
"catalog_images": {
"mealie": "ghcr.io/mealie-recipes/mealie:v3.30.0"
},
"live_compose_image_lines": [
"image: ghcr.io/mealie-recipes/mealie:v3.30.0"
],
"docker_inspect": [
"mealie ghcr.io/mealie-recipes/mealie:v3.30.0 running=true restarts=0"
]
},
"badge_after": {
"hu": [
{
"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.",
"text": "Naprakész"
}
],
"en": [
{
"title": "This app is running the newest version available.",
"text": "Up to date"
}
]
},
"state_after": "running",
"hold_after": null,
"to_digests": {
"mealie": "sha256:90f83da6f3ab0a421efc827118a556168573930105f13fa3321768f42163d784"
},
"four_observables": {
"pinned": {
"mealie": "ghcr.io/mealie-recipes/mealie:v3.30.0"
},
"installed": {
"mealie": "ghcr.io/mealie-recipes/mealie:v3.30.0"
},
"catalog": {
"mealie": "ghcr.io/mealie-recipes/mealie:v3.30.0"
},
"live_compose": [
"ghcr.io/mealie-recipes/mealie:v3.30.0"
]
},
"seed_read_after": true,
"setup_gate_after": {
"browser": {
"status": "200",
"redirect": null
},
"script": {
"status": "200",
"redirect": null
}
},
"memory": {
"soak_s": 600,
"interval_s": 10,
"samples": 60,
"containers": {
"mealie": {
"samples": 60,
"anon_peak_bytes": 244592640,
"limit_bytes": 1048576000,
"anon_peak_pct": 23.3,
"anon_live_peak_bytes": 244588544,
"anon_live_peak_pct": 23.3,
"cgroup_peak_pct": 28.1,
"oom_kill_delta": 0,
"oom_killed_flag": false,
"restarts_delta": 0,
"running_all_samples": true,
"cgroup": "/sys/fs/cgroup/system.slice/docker-a1aa37419b5ebbdcbeec0a16fc55cd0dc7eecf72cffc4e4345f400ab48272b95.scope"
}
},
"oom_kills": 0,
"restarts": 0,
"peak_pct": 23.3,
"peak_basis": "anon_live"
},
"why": "the security-fix step ran on the box through the guarded Update; healthy, TO on all four observables, no undo, no hold, a clean soak, seed read back",
"log": [
"01:18:38 [5] the box's drill mirror now at c760fca93",
"01:18:42 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['ADMIN_PASSWORD']",
"01:18:42 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}",
"01:19:47 [1] deployed, controller state=running, pinned={'mealie': 'ghcr.io/mealie-recipes/mealie:v3.29.0'}",
"01:19:47 gate: recipes is gated — passed as the household (cookie set)",
"01:19:53 [2] setup gate as installed: {'browser': {'status': '302', 'redirect': 'https://felhom.enkisfelhom.hu/__gate/start?…'}, 'script': {'status': '401', 'redirect': None}}",
"01:19:53 mealie: create recipe http=201",
"01:19:54 mealie: readback of the seeded recipe http=200 ok=True",
"01:19:55 [5] drill commit c98dffe18656: mealie ghcr.io/mealie-recipes/mealie:v3.29.0 -> ghcr.io/mealie-recipes/mealie:v3.30.0 (push rc=0)",
"01:20:04 [5] the box's drill mirror now at c98dffe18",
"01:20:09 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}",
"01:20:09 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None",
"01:20:13 + 4.1s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None",
"01:20:14 + 5.1s phase=pulling label=Új verzió letöltése… err=None hold=None",
"01:20:57 + 48.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None",
"01:21:02 + 53.3s phase=verifying label=Működés ellenőrzése… err=None hold=None",
"01:21:17 + 68.6s phase=done label=Frissítve err=None hold=None",
"01:21:38 mealie: readback of the seeded recipe http=200 ok=True",
"01:21:38 [7] memory soak 600 s …",
"01:31:44 [7] soak: peak 23.3% of the limit (anon), kills 0, restarts 0"
]
}
@@ -0,0 +1,25 @@
01:18:38 [5] the box's drill mirror now at c760fca93
01:18:42 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['ADMIN_PASSWORD']
01:18:42 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
01:19:47 [1] deployed, controller state=running, pinned={'mealie': 'ghcr.io/mealie-recipes/mealie:v3.29.0'}
01:19:47 gate: recipes is gated — passed as the household (cookie set)
01:19:53 [2] setup gate as installed: {'browser': {'status': '302', 'redirect': 'https://felhom.enkisfelhom.hu/__gate/start?…'}, 'script': {'status': '401', 'redirect': None}}
01:19:53 mealie: create recipe http=201
01:19:54 mealie: readback of the seeded recipe http=200 ok=True
01:19:55 [5] drill commit c98dffe18656: mealie ghcr.io/mealie-recipes/mealie:v3.29.0 -> ghcr.io/mealie-recipes/mealie:v3.30.0 (push rc=0)
01:20:04 [5] the box's drill mirror now at c98dffe18
01:20:09 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
01:20:09 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None
01:20:13 + 4.1s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None
01:20:14 + 5.1s phase=pulling label=Új verzió letöltése… err=None hold=None
01:20:57 + 48.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
01:21:02 + 53.3s phase=verifying label=Működés ellenőrzése… err=None hold=None
01:21:17 + 68.6s phase=done label=Frissítve err=None hold=None
01:21:38 mealie: readback of the seeded recipe http=200 ok=True
01:21:38 [7] memory soak 600 s …
01:31:44 [7] soak: peak 23.3% of the limit (anon), kills 0, restarts 0
01:31:44 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-2026-11-02/walks/mealie-drill/verdict.json
01:31:44 RESULT proven — the security-fix step ran on the box through the guarded Update; healthy, TO on all four observables, no undo, no hold, a clean soak, seed read back
01:31:46 [X] stop -> 200 {'ok': True, 'message': 'Stack mealie stop completed'}
01:32:18 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'mealie', 'volumes_removed': ['mealie_mealie_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalm
01:32:27 [X] after remove: deployed=False leftovers=''
@@ -0,0 +1,232 @@
{
"app": "n8n",
"walk": "security",
"box_walk_version": 1,
"verdict": "proven",
"venue": "box 9202 (drill catalog), the guarded Update",
"from": {
"n8n": "n8nio/n8n:2.46.4"
},
"to": {
"n8n": "n8nio/n8n:3.0.1"
},
"observables_before": {
"pinned_images": {
"n8n": "n8nio/n8n:2.46.4"
},
"installed_images": {
"n8n": "n8nio/n8n:2.46.4"
},
"catalog_images": null,
"live_compose_image_lines": [
"image: n8nio/n8n:2.46.4@sha256:4384b9ff20a78bc1708c35ce7864f84f724611de7801fb36d715998db7c000f1"
],
"docker_inspect": [
"n8n n8nio/n8n:2.46.4@sha256:4384b9ff20a78bc1708c35ce7864f84f724611de7801fb36d715998db7c000f1 running=true restarts=0"
]
},
"setup_gate_before": {
"browser": {
"status": "302",
"redirect": "https://felhom.enkisfelhom.hu/__gate/start?…"
},
"script": {
"status": "401",
"redirect": null
}
},
"seed": "read_back",
"drill_commit": "7a4afc52dede",
"sync_waited_s": 4.4,
"badge_before": {
"hu": [
{
"title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.",
"text": "Frissítés elérhető — ma"
}
],
"en": [
{
"title": "A newer version of this app is available. Select the Update button to start it.",
"text": "Update available — today"
}
]
},
"measured_at": "2026-11-03T02:05:17Z",
"update": {
"accepted": true,
"http": "202",
"phases": [
{
"t": 0.0,
"phase": "checking",
"label": "Ellenőrzés…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 1.0,
"phase": "backing-up",
"label": "Biztonsági mentés készül a frissítés előtt…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 3.1,
"phase": "safety-dump",
"label": "Adatbázis pillanatkép…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 4.1,
"phase": "pulling",
"label": "Új verzió letöltése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 116.9,
"phase": "copying",
"label": "Az adatok másolása a frissítés előtt…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 122.1,
"phase": "verifying",
"label": "Működés ellenőrzése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 138.4,
"phase": "done",
"label": "Frissítve",
"updating": false,
"error": null,
"hold": null
}
],
"duration_s": 138.5,
"final_phase": "done",
"update_error": null,
"hold_reason": null,
"state": "running"
},
"observables_after": {
"pinned_images": {
"n8n": "n8nio/n8n:3.0.1"
},
"installed_images": {
"n8n": "n8nio/n8n:3.0.1"
},
"catalog_images": {
"n8n": "n8nio/n8n:3.0.1"
},
"live_compose_image_lines": [
"image: n8nio/n8n:3.0.1"
],
"docker_inspect": [
"n8n n8nio/n8n:3.0.1 running=true restarts=0"
]
},
"badge_after": {
"hu": [
{
"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.",
"text": "Naprakész"
}
],
"en": [
{
"title": "This app is running the newest version available.",
"text": "Up to date"
}
]
},
"state_after": "running",
"hold_after": null,
"to_digests": {
"n8n": "sha256:4ebd866ae91d222ffa6dd3731eade80ba71b4d13f7470172a7e66a81fe31faa0"
},
"four_observables": {
"pinned": {
"n8n": "n8nio/n8n:3.0.1"
},
"installed": {
"n8n": "n8nio/n8n:3.0.1"
},
"catalog": {
"n8n": "n8nio/n8n:3.0.1"
},
"live_compose": [
"n8nio/n8n:3.0.1"
]
},
"seed_read_after": true,
"setup_gate_after": {
"browser": {
"status": "200",
"redirect": null
},
"script": {
"status": "200",
"redirect": null
}
},
"memory": {
"soak_s": 600,
"interval_s": 10,
"samples": 60,
"containers": {
"n8n": {
"samples": 60,
"anon_peak_bytes": 376991744,
"limit_bytes": 1610612736,
"anon_peak_pct": 23.4,
"anon_live_peak_bytes": 376991744,
"anon_live_peak_pct": 23.4,
"cgroup_peak_pct": 45.2,
"oom_kill_delta": 0,
"oom_killed_flag": false,
"restarts_delta": 0,
"running_all_samples": true,
"cgroup": "/sys/fs/cgroup/system.slice/docker-184ecabf6c41635c544a1b796735060791335e99c200c454b96e1e0d91bec7c6.scope"
}
},
"oom_kills": 0,
"restarts": 0,
"peak_pct": 23.4,
"peak_basis": "anon_live"
},
"why": "the security-fix step ran on the box through the guarded Update; healthy, TO on all four observables, no undo, no hold, a clean soak, seed read back",
"log": [
"03:02:28 [5] the box's drill mirror now at c760fca93",
"03:02:33 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}",
"03:04:59 [1] deployed, controller state=running, pinned={'n8n': 'n8nio/n8n:2.46.4'}",
"03:04:59 gate: auto is gated — passed as the household (cookie set)",
"03:05:02 [2] setup gate as installed: {'browser': {'status': '302', 'redirect': 'https://felhom.enkisfelhom.hu/__gate/start?…'}, 'script': {'status': '401', 'redirect': None}}",
"03:05:03 n8n: /rest/owner/setup http=200",
"03:05:03 n8n: login as the seeded owner http=200 ok=True",
"03:05:04 [5] drill commit 7a4afc52dede: n8n n8nio/n8n:2.46.4 -> n8nio/n8n:3.0.1 (push rc=0)",
"03:05:12 [5] the box's drill mirror now at 7a4afc52d",
"03:05:17 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}",
"03:05:17 + 0.0s phase=checking label=Ellenőrzés… err=None hold=None",
"03:05:18 + 1.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None",
"03:05:20 + 3.1s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None",
"03:05:21 + 4.1s phase=pulling label=Új verzió letöltése… err=None hold=None",
"03:07:14 + 116.9s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None",
"03:07:19 + 122.1s phase=verifying label=Működés ellenőrzése… err=None hold=None",
"03:07:36 + 138.4s phase=done label=Frissítve err=None hold=None",
"03:07:55 n8n: login as the seeded owner http=200 ok=True",
"03:07:55 [7] memory soak 600 s …",
"03:18:01 [7] soak: peak 23.4% of the limit (anon), kills 0, restarts 0"
]
}
@@ -0,0 +1,24 @@
03:18:53 [5] the box's drill mirror now at c760fca93
03:18:58 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
03:21:19 [1] deployed, controller state=running, pinned={'n8n': 'n8nio/n8n:2.46.4'}
03:21:19 gate: auto is gated — passed as the household (cookie set)
03:21:19 n8n: /rest/owner/setup http=200
03:21:20 n8n: login as the seeded owner http=200 ok=True
03:21:20 drill commit 65cc0dfcfa43: n8n TO with failing checks
03:21:29 [5] the box's drill mirror now at 65cc0dfcf
03:21:34 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
03:21:34 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None
03:21:38 + 4.1s phase=pulling label=Új verzió letöltése… err=None hold=None
03:23:31 + 116.8s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
03:23:34 + 119.8s phase=starting label=Indítás az új verzióval… err=None hold=None
03:23:35 + 120.9s phase=verifying label=Működés ellenőrzése… err=None hold=None
03:28:37 + 423.1s phase=undoing label=Visszaállítás az előző változatra… err=None hold=None
03:28:55 + 440.6s phase=undone label=Visszaállítva az előző változatra err=None hold=None
03:29:24 n8n: login as the seeded owner http=200 ok=True
03:29:24 RESULT proven — TO started on the data and failed its check; the box undid it: FROM runs again and the seed reads back
03:29:25 [X] stop -> 200 {'ok': True, 'message': 'Stack n8n stop completed'}
03:29:56 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'n8n', 'volumes_removed': ['n8n_n8n_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem
03:30:05 [X] after remove: deployed=False leftovers=''
03:30:15 [5] the box's drill mirror now at 0fe59cde2
03:30:15 drill revert 65cc0dfcfa43: rc=0
03:30:15 drill templates/n8n equal to before: True
@@ -0,0 +1,170 @@
{
"app": "n8n",
"from": {
"n8n": "n8nio/n8n:2.46.4"
},
"to": {
"n8n": "n8nio/n8n:3.0.1"
},
"started": "2026-11-03T02:18:45Z",
"drill_head_before": "c760fca938987f643d42963aebca6e27ed2bbea3",
"failing_compose_main_healthcheck": [
"test: [\"CMD\", \"false\"]"
],
"failing_probe": true,
"update": {
"accepted": true,
"http": "202",
"phases": [
{
"t": 0.0,
"phase": "backing-up",
"label": "Biztons\u00e1gi ment\u00e9s k\u00e9sz\u00fcl a friss\u00edt\u00e9s el\u0151tt\u2026",
"updating": true,
"error": null,
"hold": null
},
{
"t": 4.1,
"phase": "pulling",
"label": "\u00daj verzi\u00f3 let\u00f6lt\u00e9se\u2026",
"updating": true,
"error": null,
"hold": null
},
{
"t": 116.8,
"phase": "copying",
"label": "Az adatok m\u00e1sol\u00e1sa a friss\u00edt\u00e9s el\u0151tt\u2026",
"updating": true,
"error": null,
"hold": null
},
{
"t": 119.8,
"phase": "starting",
"label": "Ind\u00edt\u00e1s az \u00faj verzi\u00f3val\u2026",
"updating": true,
"error": null,
"hold": null
},
{
"t": 120.9,
"phase": "verifying",
"label": "M\u0171k\u00f6d\u00e9s ellen\u0151rz\u00e9se\u2026",
"updating": true,
"error": null,
"hold": null
},
{
"t": 423.1,
"phase": "undoing",
"label": "Vissza\u00e1ll\u00edt\u00e1s az el\u0151z\u0151 v\u00e1ltozatra\u2026",
"updating": true,
"error": null,
"hold": null
},
{
"t": 440.6,
"phase": "undone",
"label": "Vissza\u00e1ll\u00edtva az el\u0151z\u0151 v\u00e1ltozatra",
"updating": false,
"error": null,
"hold": null
}
],
"duration_s": 440.6,
"final_phase": "undone",
"update_error": null,
"hold_reason": null,
"state": "running"
},
"to_log_while_verifying": [
"n8nio/n8n:3.0.1",
"Initializing n8n process",
"n8n ready on ::, port 5678",
"Migrations in progress, please do NOT stop the process.",
"Starting migration AddAgentVersionMetadata1793035272968",
"Finished migration AddAgentVersionMetadata1793035272968",
"Starting migration CreateAgentPublishHistory1793035273653",
"Finished migration CreateAgentPublishHistory1793035273653",
"Starting migration BackfillAgentDraftVersions1793036689320",
"Finished migration BackfillAgentDraftVersions1793036689320",
"Starting migration CreateServiceAccountDetailsTable1793100089202",
"Finished migration CreateServiceAccountDetailsTable1793100089202",
"Starting migration AddIncludeAllProjectMembersToSelfHealingConfiguration1793106080688",
"Finished migration AddIncludeAllProjectMembersToSelfHealingConfiguration1793106080688",
"n8n Task Broker ready on 127.0.0.1, port 5679",
"Failed to start Python task runner in internal mode. because Python 3 is missing from this system. Launching a Python runner in internal mode is intended only for debugging and is not recommended for production. Users are encouraged to deploy in external mode. See: https://docs.n8n.io/hosting/configuration/task-runners/#setting-up-external-mode",
"",
"There are deprecations related to your n8n setup. Please take the recommended actions to update your configuration:",
" - WEBHOOK_URL -> Use N8N_WEBHOOK_URL instead, which sets the base URL for both test and production webhooks.",
" - N8N_RUNNERS_MODE -> Internal task runner mode is deprecated and will be removed in a future version. For isolation and scaling, run the task runner launcher as a separate process, set this variable to `external` and share `N8N_RUNNERS_AUTH_TOKEN` with the launcher. See https://docs.n8n.io/deploy/host-n8n/configure-n8n/set-up-task-runners",
"",
"[license SDK] Skipping renewal on init: license cert is not initialized",
"Instance registered",
"Cluster aligned on version 3.0.1",
"Discovered 4 cluster checks",
"Recorded version change: 2.46.4 -> 3.0.1",
"Registered runner \"JS Task Runner\" (9-LdMX78uosuaR4-y5vXC) ",
"Sandbox: enabled=false provider=n8n-sandbox (from env)",
"Version: 3.0.1",
"Building workflow dependency index...",
"Scheduler started",
"Finished building workflow dependency index. Processed 0 draft workflows, 0 published workflows.",
"",
"Editor is now accessible via:",
"https://auto.enkisfelhom.hu"
],
"observables_after": {
"pinned_images": {
"n8n": "n8nio/n8n:2.46.4"
},
"installed_images": {
"n8n": "n8nio/n8n:2.46.4"
},
"catalog_images": {
"n8n": "n8nio/n8n:3.0.1"
},
"live_compose_image_lines": [
"image: n8nio/n8n:2.46.4@sha256:4384b9ff20a78bc1708c35ce7864f84f724611de7801fb36d715998db7c000f1"
],
"docker_inspect": [
"n8n n8nio/n8n:2.46.4@sha256:4384b9ff20a78bc1708c35ce7864f84f724611de7801fb36d715998db7c000f1 running=true restarts=0"
]
},
"running_images_after": {
"n8n": "n8nio/n8n:2.46.4@sha256:4384b9ff20a78bc1708c35ce7864f84f724611de7801fb36d715998db7c000f1"
},
"seed_after_undo": true,
"verdict": "proven",
"why": "TO started on the data and failed its check; the box undid it: FROM runs again and the seed reads back",
"drill_head_after": "0fe59cde294a339a4761d4a25ac38282d0ddafc9",
"drill_tree_equal_to_before": true,
"log": [
"03:18:53 [5] the box's drill mirror now at c760fca93",
"03:18:58 [1] deploy -> 202 {'ok': True, 'message': 'Telep\u00edt\u00e9s elind\u00edtva \u2013 az \u00e1llapot a k\u00e1rty\u00e1n k\u00f6vethet\u0151'}",
"03:21:19 [1] deployed, controller state=running, pinned={'n8n': 'n8nio/n8n:2.46.4'}",
"03:21:19 gate: auto is gated \u2014 passed as the household (cookie set)",
"03:21:19 n8n: /rest/owner/setup http=200",
"03:21:20 n8n: login as the seeded owner http=200 ok=True",
"03:21:20 drill commit 65cc0dfcfa43: n8n TO with failing checks",
"03:21:29 [5] the box's drill mirror now at 65cc0dfcf",
"03:21:34 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Friss\u00edt\u00e9s elindult \u2013 az \u00e1llapot a k\u00e1rty\u00e1n k\u00f6vethet\u0151'}",
"03:21:34 + 0.0s phase=backing-up label=Biztons\u00e1gi ment\u00e9s k\u00e9sz\u00fcl a friss\u00edt\u00e9s el\u0151tt\u2026 err=None hold=None",
"03:21:38 + 4.1s phase=pulling label=\u00daj verzi\u00f3 let\u00f6lt\u00e9se\u2026 err=None hold=None",
"03:23:31 + 116.8s phase=copying label=Az adatok m\u00e1sol\u00e1sa a friss\u00edt\u00e9s el\u0151tt\u2026 err=None hold=None",
"03:23:34 + 119.8s phase=starting label=Ind\u00edt\u00e1s az \u00faj verzi\u00f3val\u2026 err=None hold=None",
"03:23:35 + 120.9s phase=verifying label=M\u0171k\u00f6d\u00e9s ellen\u0151rz\u00e9se\u2026 err=None hold=None",
"03:28:37 + 423.1s phase=undoing label=Vissza\u00e1ll\u00edt\u00e1s az el\u0151z\u0151 v\u00e1ltozatra\u2026 err=None hold=None",
"03:28:55 + 440.6s phase=undone label=Vissza\u00e1ll\u00edtva az el\u0151z\u0151 v\u00e1ltozatra err=None hold=None",
"03:29:24 n8n: login as the seeded owner http=200 ok=True",
"03:29:24 RESULT proven \u2014 TO started on the data and failed its check; the box undid it: FROM runs again and the seed reads back",
"03:29:25 [X] stop -> 200 {'ok': True, 'message': 'Stack n8n stop completed'}",
"03:29:56 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'n8n', 'volumes_removed': ['n8n_n8n_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmaz\u00e1s nem ",
"03:30:05 [X] after remove: deployed=False leftovers=''",
"03:30:15 [5] the box's drill mirror now at 0fe59cde2",
"03:30:15 drill revert 65cc0dfcfa43: rc=0",
"03:30:15 drill templates/n8n equal to before: True"
]
}
@@ -0,0 +1,25 @@
03:02:28 [5] the box's drill mirror now at c760fca93
03:02:33 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
03:04:59 [1] deployed, controller state=running, pinned={'n8n': 'n8nio/n8n:2.46.4'}
03:04:59 gate: auto is gated — passed as the household (cookie set)
03:05:02 [2] setup gate as installed: {'browser': {'status': '302', 'redirect': 'https://felhom.enkisfelhom.hu/__gate/start?…'}, 'script': {'status': '401', 'redirect': None}}
03:05:03 n8n: /rest/owner/setup http=200
03:05:03 n8n: login as the seeded owner http=200 ok=True
03:05:04 [5] drill commit 7a4afc52dede: n8n n8nio/n8n:2.46.4 -> n8nio/n8n:3.0.1 (push rc=0)
03:05:12 [5] the box's drill mirror now at 7a4afc52d
03:05:17 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}
03:05:17 + 0.0s phase=checking label=Ellenőrzés… err=None hold=None
03:05:18 + 1.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None
03:05:20 + 3.1s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None
03:05:21 + 4.1s phase=pulling label=Új verzió letöltése… err=None hold=None
03:07:14 + 116.9s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None
03:07:19 + 122.1s phase=verifying label=Működés ellenőrzése… err=None hold=None
03:07:36 + 138.4s phase=done label=Frissítve err=None hold=None
03:07:55 n8n: login as the seeded owner http=200 ok=True
03:07:55 [7] memory soak 600 s …
03:18:01 [7] soak: peak 23.4% of the limit (anon), kills 0, restarts 0
03:18:01 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/security-2026-11-02/walks/n8n3-drill/verdict.json
03:18:01 RESULT proven — the security-fix step ran on the box through the guarded Update; healthy, TO on all four observables, no undo, no hold, a clean soak, seed read back
03:18:02 [X] stop -> 200 {'ok': True, 'message': 'Stack n8n stop completed'}
03:18:34 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'n8n', 'volumes_removed': ['n8n_n8n_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem
03:18:43 [X] after remove: deployed=False leftovers=''
Loaded 100 of 114 files, more files were not shown because too many files have changed in this diff. Show more