diff --git a/CONTEXT.md b/CONTEXT.md index df370f1a..4becc7ba 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -14,6 +14,27 @@ > language, one screen, no identifiers in the prose. Same subjects, different readers; merging them > would make one of the two audiences stop reading. `STATUS.md` is also a **view of `OPEN-ITEMS.md`** > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +## 2026-09-24 (night shift 2026-09-23) — tests off DooPlex's Docker, the test record, 12 published steps, the chaos hour + +**Operator word, `09` §3 decision 21:** tonight the catalog may move every app whose within-a-major edge is +proven on both venues, each with its test record; nothing else moves. **Decided by CC unattended — operator may +reverse** (both recorded in `09` §3): **22** — the memory watch's `memory_tight` reads the app's own memory +(`anon`, sampled), the cgroup peak kept beside it (the file cache read 100 % with 0 kills); **23** — the move +gate asks the registry, for moved refs only (INCONCLUSIVE when it cannot → the push waits). +- controller **v0.267.0** (`80e6ad8c4772`): `internal/dockerexec` (R-650), `CheckDumpComplete` before any + restore mutation (R-640, proven live), R-499's four-branch sentence, R-518's measured stop. **Deployed to 9202 + only; the floor stays 0.266.0** — the brief raises it only if every Part A/B/D proof passed, and Part D found + R-658/R-659 (both pre-existing). Operator question in STATUS. +- catalog: `update_ladder:` (one JSON entry per line; `scripts/ladder.py`), `check-test-record.py` (static, CI) + + `check-test-record-move.py` (history + registry), `upgrade-test.py --write-ladder` the only writer, the + 21-move backfill; harness v3 (box fixtures on the bench via `upgrade_boxport.py`, anon memory, + `files_may_change`). 12 steps published across 11 apps (`audits/night-2026-09-23/C-moves.txt`); wishlist 512M + (R-612) and uptime-kuma's DB wizard skipped (R-613). +- **Found:** R-658 (P1 — a restore drops the compose label, the undo then copies nothing), R-659 (P1 — a held + file-leg app is pointed at a restore that refuses it), R-655 (adventurelog v0.13.0), R-657 (nextcloud reinstall + over kept data), R-651–R-654, R-656, R-660. Closed R-650, R-640, R-499, R-626. Register 330 → 336. +- Record: `documentation/audits/DRILL-night-2026-09-23.md`. + ## Ruling 2026-09-23 (night) — a failed install removes what it started (R-649, controller v0.266.0) Operator: "On failed install, the controller should clean up the containers." `runComposeDeploy`'s failure diff --git a/REPORT.md b/REPORT.md index 90414e50..4c774c20 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,117 +1,34 @@ -# ADDENDUM (same session) — R-649, the operator's ruling: a failed install removes what it started (controller v0.266.0) +# REPORT — night shift 2026-09-23: DooPlex guarded, the test record, 12 published steps, the chaos hour -Ruling: "On failed install, the controller should clean up the containers." Built in `964ae75`: -`runComposeDeploy`'s failure branch runs `compose down` (no `-v` — volumes kept) before recording -not-deployed. Red-proof: the `down` removed → `TestR649_AFailedInstallRemovesWhatItStarted` fails with -`calls=["up -d"]`. Live on 9202 (`audits/r649-2026-09-23/`): outline with a never-healthy redis (drill -template only) failed after 102.7 s → 0 containers left, 3 volumes kept, log line -`the failed deploy's containers were removed (volumes kept) — R-649`. Floor **0.266.0** read back; both demo -boxes on it in 20 s. R-649 closed; **open rows 331 → 330.** Teardown: outline removed, images by name, config -identical to the saved copy, live catalog, drill reset. Nothing else touched. +Full record: `documentation/audits/DRILL-night-2026-09-23.md`; step log `documentation/audits/night-2026-09-23/PROGRESS.md`. +Architecture read first: `09-update-architecture.md` (§3 decisions 11–20, §6.1a, §6.4), `07` §6, `08`. ---- +## Not done, or changed +- **The floor stays 0.266.0** (the brief's rule: Part D found R-658 and R-659). Recommended to raise — the two + P1s are pre-existing; STATUS carries the question. +- adventurelog not moved (R-655); gitea inconclusive (R-624); 11 across-major and 12 no-route/no-fixture apps listed. +- Chaos rounds 2–5 landed their accidents outside the action (the runner waited on the wrong file; fixed from + round 5/6). The event column was rebuilt from the controller's log after the debug ring missed two holds. +- The stop rule fired at round 11 (R-659) after all twelve rounds had run. -# REPORT — clean-up evening: R-634's cause, held apps, the OOM storm (controller v0.265.0, hub v0.121.0) +## What shipped +- felhom-controller **v0.267.0** `80e6ad8c4772` (CI 919): R-650, R-640, R-499, R-518; deployed to 9202 only. +- app-catalog-felhom.eu: the test record + gates `6db08a5` (CI 920), R-612/R-613 `a5a729a`, harness fixes + `1263773`, 12 steps: romm ×2, ghost, wishlist, opengist, navidrome, komga (+768M), n8n, emby, kimai-db, + immich-ML, nextcloud (CI 922–937). +- felhom.eu: evidence, `09` §3 decisions 21–23 and §6.4 parts 4/6, rows R-651..R-660, four closed, capability + map, nightly rotation, STATUS. -2026-09-23 (late evening). Baselines verified live: controller `0a3026180ae8` (v0.264.0), agent -`d9864a94bf62`, felhom.eu `21f17ed32bdc` (hub v0.120.0), catalog `cfcfe5278428` — as the brief said. -Repos touched: **felhom-controller** (v0.265.0 `0054d4b`), **felhom.eu** (hub v0.121.0, manifest, docs, -register, evidence), **admin/app-catalog-drill** (drill commits, reset to live `main`). -**felhom-agent, app-catalog-felhom.eu: untouched.** Architecture read and named: -`09-update-architecture.md` §6.1/§6.1a/§6.4, `08-alarm-ladder.md` §4/§6.2, `02-controller-module-map.md` -(stacks ↔ backup seam). Evidence: `documentation/audits/cleanup-2026-09-23/README.md`. **Method: -endpoint-level** (the endpoints the UI invokes; no browser). +## Red-proofs (each seen failing) +Controller 8 (see `felhom-controller/REPORT.md`); catalog gate 3 (`B2-gate-redproofs.txt`); R-612/R-613 +before/after on 9202 through the product. ---- +## Gates +controller: `go test -count=1 ./...` rc 0, `controller_gates.py` OK. catalog: `catalog_gates.py --fast` OK, +`test_gate_decoys.py` 80 OK, `test_ladder_writer.py` OK, `test_catalog_gates.py` OK. felhom.eu: `repo_gates.py` +(run at commit). `unproven.py --summary`: not walked 35 of 55 (unchanged). -## 1. Not done, or changed - -| item | state | -|---|---| -| Part 1 — R-634 | **cause found in ~25 min (cap 2 h), fixed, proven live.** One design question left → **R-649** (operator) | -| Part 2 — R-625 | **done.** The Update button was ALREADY hidden for a held app on the list (since v0.238.0); the defect was the badge only. Now pinned by a test with a positive control. The badge applies to **both** hold kinds (a failed restore holds the app stopped too; the way back is also a restore) — the brief said `update_failed` only | -| Part 3 — R-636 | **done, and the brief's rule changed shape:** "the same key re-fires ≥ 20 times" cannot work — `OOMKilled` is sticky, so the key re-fires every 30 s after ONE kill (a hiccup would storm after 10 min). The controller now counts real kills (kernel `oom_kill`); the 20-in-30-min threshold is applied to THAT | -| Part 3 — hub live `curl` | **not done:** posting a synthetic event with a real customer's key would put a fabricated alarm in the operator's record; the hub side is proven by unit tests | -| Part 4 — R-647 | **done** | -| Part 5 — R-648 | **done — no per-app endpoint exists**, so the drill presses nothing | -| Part 6 — floor | **raised to 0.265.0** — every live proof passed | -| a mistake of mine | my first draft of the R-634 backup test drove the real dump and **created an empty volume `outline_outline_data` on DooPlex**; my inspection of it pulled `alpine:3.20`. Both verified new and unused, removed by name. Tests rewritten on seams. **R-650** filed (the class is unguarded) | - -**Claims in the brief, checked:** -1. *"sparkyfitness reproduces alone at today's pin"* — **wrong.** Deployed alone in 47.3 s on v0.264.0 - (`10-*`). The 2026-09-22 "alone" walk pressed the whole-box backup itself — the same race. -2. *"a whole-box backup touches a DEPLOYING stack at all (never read)"* — **true, and it is THE cause.** -3. *"20 kills in 30 minutes separates a storm from a hiccup"* — **true only if kills are counted, and they - were not.** RomM's real rate: 4,530 kills in 6 h ≈ **375 per 30 min**; a hiccup is 1–3. 20 kept, applied - to the kernel's counter. Live: 21 kills 2.5 min after start → one storm. -4. *"a per-app backup endpoint exists"* — **wrong.** `router.go` has `/backup/run` and `/backup/tier2` - only; the per-app backup (`RunAppBackupNow`) is reachable only through the guarded update. - -## 2. R-634 — the mechanism, at `file:line` - -1. `stacks/deploy.go:395` — `DeployStack` sets the in-memory `Deployed=true` at ACCEPT (no-stale-button UX). -2. `cmd/controller/main.go:2556` — `ListDeployedStacks` filtered on that flag only → a deploying app is in - every backup leg's list. -3. `backup/backup.go:693` → `DumpAppVolumesSafe` — `StopStack` (`compose down`) in the middle of the deploy, - tar of half-made volumes, `StartStack` (a SECOND `compose up -d`, `backup.go:918`). -4. Both `up` calls fail together; `stacks/deploy.go:421-435` writes `Deployed=false` without asking whether - containers exist. Which `up` wins decides the end: running under „not deployed" (2026-09-22) or - `Created` (tonight, 14:39:48). **No deploy time limit exists** — shape (i) ruled out. - -**Fix:** deploying apps leave the list; the volume leg re-asks right before the stop (SKIP, never FAIL); -`StopStack`/`StartStack` refuse a deploying stack (`ErrStackDeploying`) — the backstop for quiesce, -restore, export, storage and the Stop button, which all had the same exposure. -**Live (`32-*`, `33-*`):** outline's image removed so the deploy must pull, backup at +5 s: the backup ran -15:23:07–15:23:31, stopped gokapi, paperless-ngx and privatebin, **never outline**; `deployed -successfully (took 48.9s)`. The first fix run is kept and labelled NOT A RACE (images cached, 6.3 s). - -## 3. Red-proofs (each seen failing, mutation restored, `=== RUN` checked) - -| # | mutation | test → failure | -|---|---|---| -| 1 | backup skip removed | `TestR634_VolumeLegNeverStopsADeployingApp` — `dumped=[outline]` | -| 2 | StopStack guard removed | `TestR634_StopAndStartRefuseADeployingStack` — `exit code -1` instead of `ErrStackDeploying` | -| 3 | held badge removed (v0.264.0's shape) | `TestR625_AHeldAppSaysStoppedRestoreNeeded` — badge missing, „Frissítés elérhető" present | -| 4 | reader funcs off | `TestR647_AReaderInTheOtherLanguageReadsTheHoldInTheirs` — English hold for the Hungarian reader | -| 5 | Hungarian phrase wired back | `TestR647_HeldEventCarriesTheCopyHoldsKey` | -| 6 | health status passed as severity | `TestR647_DisabledHealthChangeNamesTheSeverity` — `severity warn` | -| 7 | escalation removed | `TestR636_TwentyKillsInThirtyMinutesIsOneStorm` — `kills=20 … storms=0, want 1` | -| 8 | counter never read | `TestR636_ScanReadsTheCounterOfFlaggedContainersOnly` — `Kills:-1` | -| 9 | hub: not operator-only | `TestAppOOMStormIsAllowlistedAndOperatorOnly` — "must be operator-only" | -| 10 | hub: not allow-listed | same test — "must be in allowedEventTypes" | -| 11 | hub: no per-app cooldown | `TestR389_TheAllowListHasExactlyOneMember` | - -Also pinned without a mutation run: 19 kills → no storm, 200 → one, one hiccup over 6 h → none, 20 kills -over 3 h → none, an unreadable counter → never. Gates: controller `go test ./...` rc=0 + -`controller_gates.py` rc=0; hub `go test ./...` rc=0 + `repo_gates.py` rc=0. Parity: `app_info_deployed` -and `stacks_full` regenerated — measured diff **exactly one line each, the new held badge**. - -## 4. Live proofs in both languages (9202) - -- **R-625 + R-647 (1)** (`44-*`), every box/reader pair: hu reader „Megállítva — visszaállítás szükséges", - title „A frissítés nem sikerült, és az automatikus visszaállítás sem."; en reader "Stopped — restore - needed", title "The update did not succeed, and the automatic undo did not either."; with the box in - ENGLISH the Hungarian reader reads Hungarian (page and API). No vikunja Update button; other apps keep - theirs (control). `POST …/update` → **409 `held`**. -- **R-636** (`45-*`, `46-*`): romm at 320M — kills 8, 13, **21 → `OOM STORM — 21 kills in 30 min (limit - 320M, peak 320M)`** + `DROPPED event app_oom_storm (severity error)`; one storm line at 49 kills. -- **R-648** (`43-*`): vikunja's update ran its own `backing-up` phase. - -## 5. Rows - -**Closed (5):** R-634, R-625, R-636, R-647, R-648. **Opened (2):** R-649 (operator question — a deploy that -fails on its own), R-650 (tests that reach real docker act on DooPlex). **Open rows 334 → 331.** -`09` §6.4 parts 8 and 9 marked SHIPPED; `08` §6.2 has the storm rung and the two per-app grains. - -## 6. Floor - -0.265.0 / MinAgent 0.131.0, read back from the hub (`min_controller_version=0.265.0`, -`min_agent=0.131.0`; `managed floor SERVED` for demo-felhom and demo-hp); both on 0.265.0 in 20 s. - -## 7. Teardown — three layers - -- **machine (9202):** all four test apps removed through the product; romm's drive folder by name (R-442 - fence, as always); 0 volumes, 0 undo copies; test images removed by name where unused; config identical to - the saved copy; live catalog; language `hu`; recorders stopped. 9201 not touched. -- **host:** nothing provisioned. **hub:** v0.121.0 + floor (planned). **DooPlex:** the volume and image - above, removed. **drill repo:** reset to live `main`. Password files deleted from the scratchpad. +## Teardown — three layers +Machine: 9202 apps removed through the product (0 containers/volumes/copies), kept drive folders removed by name, +config identical to the saved copy, live catalog; bench 9401 destroyed; 9201 24 containers before/after. +Host: `pct list` as before; template removed. Hub: not touched (no floor change). diff --git a/STATUS.md b/STATUS.md index f4a76645..ac788dd1 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,25 +1,25 @@ # STATUS — what works, what's broken, what's next -**Updated 2026-09-23 (night) — clean-up evening, plus your install ruling. The "runs but not installed" fault has its cause found and fixed. The fleet has the new version.** +**Updated 2026-09-24 (morning) — the night shift. Eleven apps moved to newer versions, each with a written test result. The chaos hour found two serious faults in the undo's way back. The new controller is NOT yet on the demo boxes: one question for you below.** -**Decisions I took on my own: none.** Your install ruling is built (below). +**Decisions I took on my own (you may reverse them):** +1. The memory test now counts only the app's own memory, not the kernel's file cache. The cache made two healthy apps read "100 % full" with no memory kills, and would have forced bigger memory limits for no reason. The old figure is still recorded beside the new one. +2. The catalog's push check now asks the image registry, but only for images a push moves. Without it, a tested image could change before it reaches a box. Other pushes make no network call. -**The top fault: an app could run while the box said "not installed".** Found and fixed. The cause: the "back up now" button backs up every app. It also caught an app that was still installing. It stopped that app in the middle of the install and started it again. The install then failed and wrote "not installed", while the app's containers sometimes kept running. I made this happen on purpose on the scratch machine, twice. With the fix, the backup leaves an installing app alone. I proved that on the same machine: the backup stopped three other apps and did not touch the one installing, and the install finished normally. Sparkyfitness did not fail on its own this time. It installed in 47 seconds. +**What I exercised.** Tests can no longer run real Docker commands on your own machine by accident. A restore now refuses a cut-off database copy before it touches anything (proven with the real restore button). Two page texts now tell the truth: where this box's full backup really goes, and how long the backup button stops the apps (about 8 minutes). Every catalog version move now needs a written test result, and a check refuses a move without one. I moved 11 apps (12 steps), each tested twice: on a test bench with 10 minutes of memory watching, and on the scratch machine through the real Update button. The HP demo box then updated three of its own apps through the real button: all three finished and answer. -**A stopped app now says so.** When an update failed and the app is stopped, the badge says "Stopped — restore needed", in Hungarian or English. There is no Update button. Proven on the scratch machine in both languages. +**What broke, and whether it healed.** +- **After a restore, the automatic undo has nothing to put back.** A restore rebuilds an app's storage without the tag the undo uses to find it. The next failed update is then "undone" with no data copy. The app I tested happened to survive this. Not fixed tonight (one controller release per night). Serious. +- **A stopped app can be pointed at a restore that refuses it.** After a failed update and a failed undo, the page names a backup to restore. For an app with files on a drive, the restore refuses that backup, and on a box with no off-site copy nothing else brings the app back. The stop rule fired here; all chaos rounds had already run. Serious. +- Adventurelog's new version was not moved: our own template checks it the wrong way, and the new version needs an internet download at every start. +- Reinstalling Nextcloud over kept files never finishes, and the box only says "unhealthy". +- Smaller: a stopped app raises a second, extra alarm; the memory test once ran with no real load (fixed and re-run). -**A long memory problem is now louder.** The box now counts real memory kills. If one app is killed 20 times in 30 minutes, you get one louder alarm ("error"), once. Households do not get it. Proven with RomM on the scratch machine: the alarm came at 21 kills and stayed one alarm at 49. +**Rows.** Ten opened, four closed. The list went from 330 to 336. Wishlist and uptime-kuma were fixed in the catalog; their rows stay open, smaller. -**The three small mail and log leftovers are fixed.** A reader who picks the other language now reads a stopped app's message in their own language. +**What needs you — one question.** Should the demo boxes get controller 0.267.0 now? +- **Yes, raise the floor (my recommendation).** The two serious faults are also in the version the boxes run today. 0.267.0 does not cause them, and it makes restores safer. +- **No, wait.** Nothing changes on the boxes. The fixes above stay on the scratch machine only, until you say so. +If you do nothing, the demo boxes stay on 0.266.0. -**The test method is fixed.** My tests no longer press "back up now". An app's update makes its own backup instead. - -**Rows.** Five closed, two new. The list went from 334 to 331. The floor is 0.265.0, and both demo machines updated themselves within 20 seconds. - -**What was not clean.** While I wrote one test, it created an empty storage volume on your own machine. I saw it at once, checked it was new and unused, and removed it. Nothing else was touched. I filed a row so tests cannot do this again. - -**Your ruling is built.** A failed install now removes the containers it started, and keeps the app's data. Proven on the scratch machine: an install that failed left no containers behind and kept its data. The floor is 0.266.0, and both demo machines updated themselves within 20 seconds. The list is now 330 rows. - -**What needs you: nothing.** - -**Nothing on Peti's machine or the off-site box was touched. On your own machine, only the hub was updated, plus the test mistake above. The demo-hp machine was not touched by hand. The scratch machine is back to its three standing apps and the real catalogue.** +**Nothing on Peti's machine, the off-site box or your own machine (beyond normal pushes) was touched. The test bench was deleted. The scratch machine is back on the real catalogue with its standing apps.** diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 8da88294..0bdae469 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -103,7 +103,8 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | `CAMPAIGN-2` T-DEPLOY-SET (7 apps, env config, health-aware); `RERUN-p1p3` (×4 PASS) | Memory-guard FIRING is not live-shown (T-RES-MEMGUARD never fired: ample RAM / auth-walled) — implemented + unit-level only | | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE — the ACTIONS work. NARROWED 2026-09-13: `CAMPAIGN-3` proved `remove` removes the APP, not the DATA — the "delete my data" half was INERT on every box until controller v0.236.0 (R-442). RE-PROVEN 2026-09-13 on demo-hp: data written by the app itself (63 MB) gone after removal and listed; an unresolvable data location is REFUSED (409) with the app kept; an SSD app gets `[]` and a note.** | `CAMPAIGN-2` T-LIFECYCLE (stop/start/restart/update/logs); remove (app only) live in `CAMPAIGN-3`; **remove WITH data: `audits/R442-2026-09-13/`**; **data behaviour: `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01)** | Redeploy-after-remove edge remains open (T-REMOVE-REDEPLOY never cleanly passed — stale dryrun journal); non-pilot-critical | | **Update is GUARDED: it refuses without a restorable backup, backs up first when the copy is stale, and HOLDS an app that does not come up — on ANY backup tier, and the release itself arrives by the managed floor** | controller **v0.237.0 + v0.238.0 + v0.238.1 + v0.239.0**, hub **v0.112.0** | **PROVEN-LIVE (2026-09-13, and again the same afternoon for any tier + floor delivery)** — **afternoon (`audits/rulings-r472-r475-2026-09-13/`):** an undeclared floor above the golden refused with nothing stored (02); a declared floor 0.239.0 / MinAgent 0.129.0 served `from declared` and both demo boxes self-updated in 14 s and 15 s (03); nothing on any tier → backed up first, Tier 1 chosen, done (04); gokapi updated on its Tier-1 unit alone (05); a never-healthy update held naming „saját meghajtó" (07); restored from „helyi", hold cleared (08). **Morning:** scenarios A (real upgrade, success only after health), B (stale copy → backup first), E (pull failure → pin back, app untouched), F (never healthy → held, hold text on API and page), H (start/restart/update and the boot sweep all refuse the held app) and **the restore walk** (Mentések unit restore → back on the old version, hold cleared), on demo-hp with a throwaway app | **`audits/slice4-2026-09-13/`** (live/, redproofs/, gates/); design `architecture/09-update-architecture.md` §6.1 | ~~**Tier-2-only precondition**~~ — superseded by v0.239.0 (any tier, R-475 CLOSED); a Tier-1 route back restores only what the unit holds (R-479); the card keeps the failure sentence after a successful restore (R-480); no automatic rollback, by measurement; a release does not reach the fleet by floor between golden bakes (R-472) **WIDENED 2026-09-21 (the update night) from 3 apps to 21 edges across 19 apps, and NARROWED in one place by the same run.** `audits/DRILL-update-night-2026-09-21.md`. On scratch guest 9202 (controller v0.261.0), against a **private drill catalog** so the live catalog carried no test reference at any point, 21 edges across 19 apps real within-a-major upstream edges were walked through the product's own guarded Update, each app seeded and read back **through its own front door** (R-156) with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive.** **What the PROVEN edges prove, precisely:** the app moved, the four version observables agreed, and the data the app itself was given came back through the app's own interface afterwards. Ten of them printed a verbatim migration line. **What the FAILED edges prove, and they are the more valuable half.** `adventurelog` (a real upstream edge that migrates and then never serves), `tandoor` (an update that SUCCEEDED and was stopped by its own wrong health port), and the PostgreSQL engine major, which refused exactly as predicted. `adventurelog v0.12.1 → v0.13.0` applied **nine database migrations successfully** and then never bound its port; the update held after the full health wait, the hold sentence named the tier, the date and what the copy holds, and the restore the sentence names brought the app back. **That is this row's own promise, exercised on a real upstream edge rather than a staged one.** **AND THE NARROWING, which this row must carry because it is the same mechanism:** the `verifying` phase trusts the `.felhom.yml` probe absolutely, and **two of the 53 templates name a probe the app does not answer** — `tandoor` (port 8080; it listens on 80) and `zipline` (`/api/health`; it answers 404 there, while the compose healthcheck in the same file uses `/api/healthcheck` and is green). For those apps a **successful** update is stopped by its own health wait: tandoor was measured **serving HTTP 200 on the new version at four samples across five minutes**, with docker's own healthcheck green, and was then stopped by `failAndHold` and the household sent to a restore they did not need. **R-618, P1.** No data was lost and the restore works — but "the update is guarded" must not be read as "the guard is right about whether the app came up". **Still true and unchanged:** no automatic rollback (by measurement); the route back is the restore; a multi-major jump ends held honestly. **Not measured on this venue, and named rather than assumed:** every event and every customer mail. Guest 9202 runs `hub.enabled: false` and the notifier returns before it logs (**R-620**), so the whole "who was told" half of `08` was structurally unobservable tonight. **THE NARROWING ABOVE WAS CLOSED THE NEXT DAY, 2026-09-22 — and re-widened the row.** `audits/PROBE-FIX-2026-09-22.md`. All three wrong probes were corrected in the catalog (`app-catalog-felhom.eu@793c4fb`: tandoor `8080→80`, wger `80→8000`, zipline `/api/health→/api/healthcheck`) and **red-proofed live on 9202 through the product in both directions**: at the live pin all three read `Nem egészséges` / `Not healthy` on their own app page while docker reported every container healthy and the front door served a real page; after the real sync all three read `Fut` / `Running` with no redeploy. **tandoor's edge was then re-walked with nothing else changed and ended `done` at +41.1 s**, seed read back, where the identical edge had ended `failed` at +361.9 s with the app stopped — so the tally is now **15 proven, 2 failed, 4 inconclusive**, and all fifteen are on the live catalog. A `--fast` catalog gate (`check-probe-matches-compose.py`) now refuses a probe that does not match the same service's own compose healthcheck, with four red-proofs and ten decoys including the no-PyYAML mode CI actually runs. **WHAT THIS ROW STILL CANNOT CLAIM, and the reason is exactly R-96 rule 3:** the guard is now shown correct for **47 of 53** templates. `paperless-ngx`'s probe has **never run on any box** — no container name matches its stack name, so it is silently skipped and its badge can never go red (**R-630**); and five more cannot be judged statically, one of which (`home-assistant`) is right only because its check type cannot fail (**R-631**). An absent alarm is equally consistent with healthy and with never checked. **And the sweep's ceiling, counted: 28 of the 53 templates have never been deployed by any drill (R-632).** **THAT CEILING WAS REMOVED THE SAME NIGHT, 2026-09-22 — all 28 walked (`audits/DRILL-the-28-2026-09-22.md`), so every template in the catalog has now been attempted at least once.** 26 of 28 deployed, **6 proven**, 5 inconclusive, 14 with no within-a-major edge upstream, 1 failed honestly and 2 undeployable — one of those (`plant-it`) **by design**, refused by the product's lifecycle gate, proven live for the first time. Each app also got the half the update night skipped: a **restore from its own copy, with the seed read back again** — 21 restored, and **2 were correctly REFUSED** with the sentence `07` §6.2 predicts for a class-A app whose local copy holds no file leg. **AND THE NIGHT NARROWED THIS ROW AGAIN, in the place the probe work could not reach.** `paperless-ngx` has no container matching its stack name, so no probe is ever built for it — and `verifying` does not skip: it waits out the full `update.health_timeout` and **HOLDS**, stopping an app whose three containers all read `healthy`. The controller's own words: *`not healthy within 5m0s (last: no probe container) — stopping and HOLDING the app`*, at **+313.0 s**. **R-630, raised to P1.** So "the update is guarded" is now shown correct for 47 of 53 templates, wrong for none, and **actively harmful for the one template that has no probe at all**. **Two further limits on what this row may claim, both about STATE rather than health:** a `remove` sent while a restore is still running reports success and leaves a container restarting with a live public route (**R-633**) — while the product already refuses exactly that clash for `update` and for `restore`, naming the blocking operation; and an app can be **running, healthy and serving while recorded as `deployed: false`**, in which state the product refuses to remove it at all (**R-634**). In both, a person needed a shell to clear what the product could not. **ALL THREE ARE FIXED IN CONTROLLER v0.262.0 (2026-09-22), and the first is PROVEN LIVE.** *The stopped app:* `verifying` no longer loops on a probe that resolves to nothing — it settles on container state, the way an app declaring no check is judged, and says which it did. **Measured on paperless-ngx: the identical Update that ended `failed` at +313.0 s with the app stopped now ends `done` at +53.4 s**, with no `no probe container` warning in the log because the explicit `healthcheck.container` resolved the target. *The ghost:* `RemoveStack` consults the backup side's `Busy` guard — which the product already applied to `update` and to `restore` — and then WATCHES the compose project for 25 s after `down`, removing anything that carries its label and answering `verified: true/false`, because `down` returning 0 is a request rather than a result. *The unremovable app:* the refusal now asks whether anything EXISTS (containers, a compose file, an `app.yaml`) instead of reading a flag. **WHAT THIS ROW STILL MAY NOT CLAIM:** R-634's MECHANISM — why `deployed` goes false while containers run — **is not diagnosed**; only the consequence is fixed. And a probe can be right about the port and still wrong about what a 200 means: `romm` answered 200 from nginx for six hours while its workers were OOM-killed behind it (**R-635**). **"The update is guarded" has never meant "the new version runs".** | -| **A failed update is UNDONE by the box itself — the previous version back with its data from seconds before the update; the app is held only if that undo fails too** | controller **v0.263.2** (the undo), **v0.264.0** + hub **v0.120.0** (the mail) | **PROVEN-LIVE (2026-09-23)** | `audits/undo-live-2026-09-23/README.md` (+ `audits/undo-bakeoff-2026-09-23/` for the method). On scratch guest 9202, through the endpoints the UI invokes: docmost (PostgreSQL), romm (MariaDB) and vikunja (SQLite in a volume), each a real migrating edge failing a deliberately wrong probe, **undone in 30–52 s** with seeds written before the backup, after it and seconds before the press ALL read back through each app's front door, ledgers equal to before; page line in hu and en. Cut-off copy → HOLD (`untouched`), prefix in the box's language; power cut (`pct stop`) during `undoing` → resumed after boot and undone; a person's press after an undo → `done`; removal deletes kept copies. | Folder copy of NAMED volumes only (`09` §3 decision 19); bind folders never touched. **The household is told (2026-09-23, `audits/undo-fleet-2026-09-23/`):** on guest 9201 the household and the operator each received ONE mail per app per outcome — undone and held, in Hungarian (vikunja) and, after one language switch, in English (glance) — with the app named in the subject; `notification_log` rows 937–946 all `sent`. **Not built:** the automatic caller (part 7) — this protects the manual button today. Leftovers: R-647. | +| **A failed update is UNDONE by the box itself — the previous version back with its data from seconds before the update; the app is held only if that undo fails too** | controller **v0.263.2** (the undo), **v0.264.0** + hub **v0.120.0** (the mail) | **PROVEN-LIVE (2026-09-23)** | `audits/undo-live-2026-09-23/README.md` (+ `audits/undo-bakeoff-2026-09-23/` for the method). On scratch guest 9202, through the endpoints the UI invokes: docmost (PostgreSQL), romm (MariaDB) and vikunja (SQLite in a volume), each a real migrating edge failing a deliberately wrong probe, **undone in 30–52 s** with seeds written before the backup, after it and seconds before the press ALL read back through each app's front door, ledgers equal to before; page line in hu and en. Cut-off copy → HOLD (`untouched`), prefix in the box's language; power cut (`pct stop`) during `undoing` → resumed after boot and undone; a person's press after an undo → `done`; removal deletes kept copies. **NARROWED 2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** a controller `kill -9` in `verifying` → resumed after restart → undone, seeds read back (round 9); a 1 GB memory hog and a whole-box backup during `verifying` did not disturb it (rounds 6, 8). **Two holes found:** after ANY restore the app's volumes lose their compose label and the undo copies NOTHING (R-658, P1); and a held FILE-LEG app is told to restore from a copy the restore then refuses, with no other route on a box without an off-site tier (R-659, P1). | Folder copy of NAMED volumes only (`09` §3 decision 19); bind folders never touched. **The household is told (2026-09-23, `audits/undo-fleet-2026-09-23/`):** on guest 9201 the household and the operator each received ONE mail per app per outcome — undone and held, in Hungarian (vikunja) and, after one language switch, in English (glance) — with the app named in the subject; `notification_log` rows 937–946 all `sent`. **Not built:** the automatic caller (part 7) — this protects the manual button today. Leftovers: R-647. | +| **The catalog holds only TESTED steps — an image move without a proven test record (bench + box, digests the registry still serves) is refused at push time** | catalog `6db08a5` (`scripts/check-test-record*.py`, `ladder.py`, `upgrade-test.py --write-ladder`) | **PROVEN (2026-09-23 night)** | `audits/DRILL-night-2026-09-23.md` Part B/C | 16 decoy cases both ways + 3 red-proofs; the gate judged every one of the night's 12 published steps against the live registry and refused a wrong-digest control. 21 earlier moves backfilled from their records. | The BOX does not read the ladder yet (`09` §6.4 part 5): a box two steps behind still jumps (romm on demo-hp took two steps in one press). | | **What `restart` and `update` do to a deployed app whose compose file the catalog already moved** | controller **v0.235.0** | **CHANGED 2026-09-06 — they NO LONGER upgrade it.** The row below records what shipped; this text records what it replaced, because every box under v0.235.0 still behaves the old way. **Up to v0.234.0: PROVEN-LIVE (2026-09-01) — they UPGRADE it.** Every lifecycle action ends in `docker compose up -d`, which makes the container match the file and PULLS the image itself when it is missing (measured: 18.3 s with a pull, 0.5 s without; negative control with an unchanged file did not even recreate the container). This is DELIBERATE on the restart path — `Manager.RestartStack` says so in a comment — but the syncer moves the file under a deployed app on a 15-minute cycle with no deployed check (R-438), and NOTHING tells the customer. | `audits/SPIKE-app-update-2026-09-01.md` §2, §3 | **No safety copy is taken by any of them** — `writeSafetyDump` is DATABASE-ONLY and is not on the update path at all. R-438, R-440, R-443. | | **Whether the box UPGRADES an app by itself, with nobody pressing anything** | controller | **PROVEN-LIVE (2026-09-01) — YES, but only when an app fails to come back.** A plain power cut does NOT upgrade: Docker's `restart: unless-stopped` restores the old containers and the reconciler logs `no boot-orphaned apps (nothing to start)`. When an app does NOT return, `Reconciler.Run` (`bootrecon.go:269`) calls `StartStack` -> `compose up -d` and the app comes back on the NEW version, unattended (measured). **13 non-API call sites across 9 files reach `up -d` this way** — not the five previously believed. | `audits/SPIKE-app-update-2026-09-01.md` §2, §8 | The drive-return gate (`intermediary.go:222`) and `AppStopGuard.Recover` (`appstop_marker.go:283`) call the same function; located by reading, **not exercised live** — stated as such. | | **Whether an app UPGRADE can be undone** | controller + catalog | **PROVEN-LIVE (2026-09-01) — NO, and "rollback" is the wrong word for it.** Once a migration has RUN, putting the old image tag back yields a container that refuses to start: Nextcloud — *"the version of the data (32.0.9.2) is higher than the docker image version (31.0.14.1) and downgrading is not supported"*. A 3-major jump is refused outright (*"only possible to upgrade one major version at a time"*) and IS recoverable, precisely because nothing migrated. Positive control: the data is not destroyed — returning to 32.0.9 restored both seeded markers byte-identical. | `audits/SPIKE-app-update-2026-09-01.md` §7 | The only route back is restoring DATA from a copy taken BEFORE the update — which no update path takes. And a restore's image-level rollback is itself overwritten by the syncer within 15 minutes (R-441). R-40 is confirmed live by the same measurement. | diff --git a/documentation/architecture/09-update-architecture.md b/documentation/architecture/09-update-architecture.md index f17cddad..68be7b56 100644 --- a/documentation/architecture/09-update-architecture.md +++ b/documentation/architecture/09-update-architecture.md @@ -280,6 +280,37 @@ builds them (`audits/update-rulings-2026-09-23/`). full-system backup keeps at least one hour of its [W+2h, W+6h) window. **Built with §6.4 part 7, not before.** +### 2026-09-23 (night) — one operator word, and two decisions taken by CC unattended + +21. **Tonight the catalog may move every app whose within-a-major upstream edge is proven on both + venues, each with its test record; nothing else moves** (operator word, 2026-09-23 evening). The + bar: `proven` on the test bench (seed through the front door, update, read back, ten-minute memory + watch) AND on a scratch box through the real guarded Update; no across-a-major edge, no PostgreSQL + engine major, no `inconclusive`, no app whose seed has no front-door route. **This is decision 13 + applied, not a new rule.** Result: `audits/DRILL-night-2026-09-23.md`. + +22. **The memory watch's `memory_tight` mark reads the app's OWN memory (anon), not the cgroup peak** + — *decided by CC unattended 2026-09-23 night — operator may reverse.* *One sentence:* when the watch + says "tight", should the file cache count? **Options:** (a) the cgroup `memory.peak` as built + (R-635 follow-up); (b) the `anon` figure of `memory.stat`, sampled every 15 s, with the cgroup peak + kept beside it. **Costs:** (a) marks every app that reads files — measured tonight: nextcloud and + immich's PostgreSQL at **100 %** with **0** kernel kills, because the kernel fills the limit with + cache it drops before killing anything — and the gate then demands a raised limit, which inflates + `mem_limit` (a customer-box capacity figure) for no reason; (b) can miss a cache-heavy app whose own + memory is tight only if a kill never happens — and a kill or a restart still fails the edge under + both options. **Why (b):** decision 13's mark exists for "does not fit the memory" (RomM, R-635), and + only the app's own memory decides that. The cgroup peak stays in every record (`memory_cgroup_peak_pct` + in the ladder entry). R-652. + +23. **The push-time test-record gate asks the registry — for moved refs only** — *decided by CC + unattended 2026-09-23 night — operator may reverse.* *One sentence:* may a `--fast` (hook) gate use + the network? **Options:** (a) no — digests compared only in the slow periodic run; (b) yes, but only + for the refs a push MOVES, and an unreachable registry is INCONCLUSIVE (the push is refused until it + can ask). **Costs:** (a) a move whose digest changed between the test and the push is published; + (b) a push that moves an image needs the registry — zero requests for every other push. **Why (b):** + decision 17 says the box pulls the recorded digest; recording one the registry no longer serves makes + every box's pull of that step fail (Scenario E). `app-catalog-felhom.eu/scripts/check-test-record-move.py`. + **RomM follow-ups, operator-agreed the same day:** the test bench watches memory after an update (`upgrade-test.py`, 2026-09-23); a version move checks the memory limit (gate or checklist — §6.4); R-636's louder repeated alarm. @@ -807,7 +838,11 @@ only the live box could show, fixed the same day:** the undo's probe was never a current probe held the app `unhealthy` (v0.263.1), and the "old" `.felhom.yml` taken at update time was already the new one, because `.felhom.yml` flows in on every catalog sync (v0.263.2: the pinned version's file is now recorded in `applied-meta/` whenever a version is pinned). **Residual (R-646):** -an app pinned before v0.263.2 has no such record until its next pin. +an app pinned before v0.263.2 has no such record until its next pin. **Two more, found by the chaos hour of +2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** the undo finds an app's volumes by their compose +label, and a restore recreates them WITHOUT it — so after any restore the undo copies nothing (R-658, P1); and a +held file-leg app is pointed at a restore that refuses a database-only copy, with no route left on a box without +an off-site tier (R-659, P1). A controller kill during `verifying` resumed and undid correctly (round 9). The spike, as it was run by hand before any build: @@ -1045,9 +1080,9 @@ what the part can do to a household's data if it is wrong, not how likely that i | **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold`: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. | | **2** | **SHIPPED — controller v0.264.0 + hub v0.120.0, proven live on 9202 and 9201 2026-09-23** (`audits/undo-fleet-2026-09-23/`). **The update sentences in the household's language** (R-606) and a mail when an automatic update is undone or held: events `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, on by default, mailed in the household's language with the app named in the subject; per-app cooldown on both legs. Leftovers: R-647. | R-606, 15 | **1** | — | none | | **3** | **SHIPPED — controller v0.264.0, proven live on 9202 2026-09-23.** **A disabled notifier says so** (R-620), so the mail of part 2 can be measured on a scratch box at all. | R-620 | **0.5** | — | none | -| **4** | **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only | +| **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only | | **5** | **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape | -| **6** | **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low | +| **6** | **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. **The box half (compare, render `name:tag@sha256`) is not built.** **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low | | **7** | **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has FINISHED (legs are clock-scheduled today, not chained — a completion signal is new), one app at a time (there is no single-flight, §3b Q4), `app_update.unattended` default ON, `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it. **See the one open point below.** | 11, 12 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe | | **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none | | **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none | diff --git a/documentation/audits/DRILL-night-2026-09-23.md b/documentation/audits/DRILL-night-2026-09-23.md index 583371fa..fadc5d7f 100644 --- a/documentation/audits/DRILL-night-2026-09-23.md +++ b/documentation/audits/DRILL-night-2026-09-23.md @@ -7,11 +7,32 @@ the endpoint the UI invokes; pages are fetched as HTML in both languages; no bro ## Not done, or changed - +- **The floor stays at 0.266.0.** The brief raises it only if every live proof of Parts A, B and D passed; Part D + found two P1s (R-658, R-659). Both exist in 0.266.0 as well — v0.267.0 does not cause them and adds R-640's + protection to the same restore path — so the recommendation is to raise it; that is the operator's one + question this morning. +- **Adventurelog v0.13.0 was not moved** (R-655): our own template's healthcheck points at a node path the new + image moved, and the new backend downloads world data from the internet at every start and crash-loops when + the download is cut short. +- **The stop rule fired in round 11** (R-659): all twelve rounds had already run; nothing further was injected. +- **Rounds 2–5 did not land their accidents inside the action** (the instrument, stated in Part D); round 9 is + the clean mid-update kill, round 6 the mid-verify memory hog, rounds 8 and 11 the mid-verify backup and the + concurrent second update. +- **Decisions taken by CC unattended** (operator may reverse, `09` §3): 22 — the memory mark reads the app's own + memory, not the file cache; 23 — the push-time gate asks the registry for moved refs only. +- R-518's per-tier quiesce, R-612's silent-seed half and R-613's template sweep stay open, narrowed. ## Three lines - + +- **Interventions:** none by the operator; 5 by me on my own instruments (a stale-template wait, a dropped + event ring, a killed runner of my own, a broken local edit reverted before any commit, the stop rule on a held + app) — each stated where it happened. No product state was set by hand. +- **Apps moved tonight: 11 of 14 tried (12 steps of 15 edges)**, each with its test record; the 21 moves of + 2026-09-22 backfilled. +- **The result that matters most:** after a restore, an app's volumes lose the label the undo finds them by — + so the next failed update is "undone" with **nothing** put back (R-658, P1); and a held file-leg app can be + pointed at a restore that refuses it (R-659, P1). --- @@ -56,11 +77,87 @@ wrong digest table it REFUSED all three services. **Backfill** — the 21 moves of 2026-09-22, each from the record its commit cited: - +| app | step (the service that moved) | verdict | evidence | note | +|---|---|---|---|---| +| actualbudget | actualbudget: actual-server:26.7.0 → actual-server:26.9.0 | proven | `audits/update-night-2026-09-21/apps/actualbudget/verdict.json` | — | +| audiobookshelf | audiobookshelf: audiobookshelf:2.35.1 → audiobookshelf:2.36.1 | proven | `audits/update-night-2026-09-21/apps/audiobookshelf/verdict.json` | — | +| bookstack | bookstack: bookstack:26.05.2 → bookstack:26.05.5 | proven | `audits/update-night-2026-09-21/apps/bookstack/verdict.json` | — | +| docmost | docmost: docmost:0.95.0 → docmost:0.96.0 | proven | `audits/update-night-2026-09-21/apps/docmost/verdict.json` | — | +| emby | emby: embyserver:4.10.0.20 → embyserver:4.11.0.1 | proven | `audits/the-28-2026-09-22/apps/emby/verdict.json` | — | +| ghost | ghost: ghost:6.53.0-alpine → ghost:6.64.0-alpine | proven | `audits/the-28-2026-09-22/apps/ghost/verdict.json` | — | +| grafana | grafana: grafana:13.1.0 → grafana:13.2.2 | proven | `audits/update-night-2026-09-21/apps/grafana/verdict.json` | — | +| home-assistant | home-assistant: home-assistant:2026.7.2 → home-assistant:2026.9.3 | proven | `audits/update-night-2026-09-21/apps/home-assistant/verdict.json` | — | +| immich | immich-server: immich-server:v3.0.3 → immich-server:v3.2.2 | proven | `audits/the-28-2026-09-22/apps/immich/verdict.json` | — | +| mealie | mealie: mealie:v3.20.1 → mealie:v3.27.0 | proven | `audits/update-night-2026-09-21/apps/mealie/verdict.json` | — | +| n8n | n8n: n8n:2.31.3 → n8n:2.40.5 | proven | `audits/update-night-2026-09-21/apps/n8n/verdict.json` | — | +| navidrome | navidrome: navidrome:0.63.2 → navidrome:0.64.0 | proven | `audits/update-night-2026-09-21/apps/navidrome/verdict.json` | — | +| nextcloud | nextcloud-db: mariadb:11.6 → mariadb:12.3 | proven | `audits/update-night-2026-09-21/apps/nextcloud-engine-mariadb/verdict.json` | commit cited no record; this one named | +| papra | papra: papra:26.6.1-rootless → papra:26.6.2-rootless | proven | `audits/update-night-2026-09-21/apps/papra/verdict.json` | — | +| privatebin | privatebin: pdo:2.0.5 → pdo:2.0.6 | proven | `audits/update-night-2026-09-21/apps/privatebin/verdict.json` | — | +| radarr | radarr: radarr:6.3.0 → radarr:6.4.4 | proven | `audits/the-28-2026-09-22/apps/radarr/verdict.json` | — | +| romm | romm: romm:5.0.0 → romm:5.3.0 | proven | `audits/update-night-2026-09-21/apps/romm/verdict.json` | memory watch 80.9 % (M1) | +| sonarr | sonarr: sonarr:4.0.19 → sonarr:4.0.20 | proven | `audits/the-28-2026-09-22/apps/sonarr/verdict.json` | — | +| tandoor | tandoor: recipes:2.6.13 → recipes:2.6.15 | proven | `audits/update-night-2026-09-21/apps/tandoor/verdict.json` | — | +| termix | termix: termix:2.5.0 → termix:2.8.0 | proven | `audits/the-28-2026-09-22/apps/termix/verdict.json` | — | +| vikunja | vikunja: vikunja:2.3.0 → vikunja:2.6.0 | proven | `audits/update-night-2026-09-21/apps/vikunja/verdict.json` | — | + +**21 of 21 `proven`, 0 `unrecorded`.** Every entry is marked `backfilled`, carries harness v1 (box walk only, no memory watch — except romm's M1 figure), and a digest resolved TONIGHT, which each entry's `note` says. `scripts/ladder_backfill.py`. ## Part C — move every app that proves itself - +**Venues:** the bench = throwaway LXC **9401** on demo-hp (Debian 13, docker 26.1.5, Docker Hub login), +created tonight and destroyed at teardown; the box = guest **9202**, v0.267.0, drill catalog. **Negative +control first:** C3 (`alpine:3.20` as the TO image) → `failed` — the bench measures (`bench-C3/`). +Box walks by `run_edge.py`; bench edges by `upgrade-test.py --move`; each edge's bench evidence copied off +the bench when it ended (`benchq.py`). Evidence per app: `night-2026-09-23/apps//`. + +| app | from → to | class | bench verdict | box verdict | memory: own (cgroup) peak % | migration line | moved? | +|---|---|---|---|---|---|---|---| +| nextcloud | nextcloud 34.0.1 → 34.0.4 | MariaDB + file-leg | proven | proven (done) | 24.2 (100) | nextcloud / Upgrading nextcloud from 34.0.1.2 ... | moved `5d49a11` (files_may_change) | +| romm | romm 5.3.0 → 5.3.1 | MariaDB | proven | proven (done) | n/a — measured before the own-memory sample existed (cgroup 79.5) | romm / INFO: [RomM][init][2026-09-23 21:12:26][0 | moved `7ff4e68` | +| romm-engine | romm-db mariadb 11.4 → 11.8 (own edge) | MariaDB engine | proven | proven (done) | 78.0 (84) | romm / INFO: [RomM][init][2026-09-23 21:47:51][0 | moved `431cdec` | +| kimai | kimai-db mariadb 11.6 → 11.8 (own edge) | MariaDB engine | proven | proven (done) | 37.9 (100) | kimai / [OK] Already at the latest version ("DoctrineMigrations\Version2 | moved `376b2c3` | +| adventurelog | adventurelog v0.12.1 → v0.13.0 (both images) | PostgreSQL (PostGIS) | failed ×2 (frontend never healthy; then backend crash-loop on `download-countries`) | failed (undone at +196 s, drill timeout 90 s) | — | adventurelog / Apply all migrations: account, admin, adventures, aut | **NOT moved** — our template's frontend healthcheck points at `/nodejs/bin/node`, gone in v0.13.0; with it dropped, the backend's start-time internet download crash-looped (R-655) | +| immich | immich-machine-learning v3.0.3 → v3.2.2 | PostgreSQL (server unchanged) | proven | proven (done) | 51.4 (100) | immich-server / [Nest] 7 - 09/23/2026, 9:11:07 PM LO | moved `b82b7c6` | +| n8n | n8n 2.40.5 → 2.41.1 | SQLite | proven | proven (done) | 22.7 (47) | n8n / Migrations in progress, please do NOT stop the process. | moved `04b63db` | +| ghost | ghost 6.64.0 → 6.65.0 | SQLite | proven | proven (done) | 24.9 (43) | ghost / [2026-09-23 21:21:22] WARN Database state requires migration | moved `e6afab9` (first watch had no load — re-run) | +| komga | komga 1.25.0 → 1.27.1 | embedded | proven | proven (done) | 60.0 (77) | komga / 2026-09-23T21:35:41.616+02:00 INFO 1 --- [ main] o.f.core | moved `fc1becc` with 768M (memory_tight at 512M) | +| opengist | opengist 1.13 → 1.15 | SQLite | proven | proven (done) | 77.8 (100) | — | moved `657c83f` (pages under `/-/`, R-654) | +| wishlist | wishlist v0.66.0 → v0.67.1 | SQLite | proven | proven (done) | 35.9 (53) | wishlist / Prisma schema loaded from prisma/schema.prisma. | moved `1545c5f` (after R-612's 512M) | +| navidrome | navidrome 0.64.0 → 0.64.1 | SQLite | proven | proven (done) | 6.3 (9) | navidrome / time="2026-09-23T19:31:04Z" level=info msg="goose: no migrations | moved `76cc1f5` (second ladder step) | +| emby | emby 4.11.0.1 → 4.11.0.3 | embedded | proven | proven (done) | 4.5 (8) | emby / Info SqliteUserRepository: Sqlite compiler options: ATOMIC_INTRINSICS | moved `8d573ad` | +| gitea | gitea 1.27.0 → 1.27.3 | SQLite | inconclusive | inconclusive () | — | — | NOT moved — inconclusive on both: its web installer blocks a headless first account (R-624) | + +**12 steps across 11 apps published**, each written by `upgrade-test.py --write-ladder`, each its own +commit, every push through the hook (the new gate judged each against the registry), CI green (jobs +922–937, matched on head_sha; two commits rode inside a later push and were judged there). Tried: 14 +apps, 15 edges. `night-2026-09-23/C-moves.txt` lists the commits. + +**Listed, not moved:** +- *across a major* (11): sparkyfitness v0.17.3→v1.7.2 (both images), gokapi v1.9.6→v2.2.4, claper 2.5→3.0, + homepage v1.13.2→v2.4.0, paperless-ngx 2.20.15→3.2.1, mariadb →13.0 (romm, kimai, bookstack, + nextcloud), nextcloud 34→35, plex 1.41→1.43. +- *no front-door seed route* (R-624): code-server, outline, rallly; *sign-up closed by design*: vaultwarden, + zipline; *installer*: gitea (tried — inconclusive on both venues). +- *no fixture tonight*: bentopdf, glance, crafty-controller, wger (2.6→2.7), wanderer's meilisearch + (v1.36→v1.54), uptime-kuma (2.4.0→2.5.5 — its first account exists only over socket.io). +- *tried and failed*: adventurelog (R-655, two causes measured). + +**What the night added to the bench:** the box walk's fixtures run on the bench (`upgrade_boxport.py`); a +wishlist fixture; opengist, komga and nextcloud fixtures fixed; the memory watch's load no longer errors on +box-fixture apps (R-653) and samples the app's own memory (decision 22, R-652); `files_may_change` from a +bind-mount tree hash (nextcloud's is the first entry carrying it). + +**R-612 / R-613 — both fixed** (catalog `a5a729a`): wishlist 512M — 128M kills the first-boot seed every +run, sometimes after the Role/Group rows (so the sign-up lie is timing-dependent; it did NOT reproduce on +9202 tonight); 512M completes it (peak 312M). uptime-kuma `UPTIME_KUMA_DB_TYPE=sqlite` — before, the box +read `running` over `setup-database`; after, the real server (`/metrics` 401) and the database in the +volume. `apps/wishlist/r612-*`, `apps/uptime-kuma/r613-*`. + +**A consequence of publishing tonight, seen on demo-hp:** romm now has three steps (the backfilled +5.0.0→5.3.0, 5.3.0→5.3.1, mariadb 11.4→11.8). The box cannot climb one step at a time yet (part 5), so the +press on 9201 took BOTH new steps at once — each tested alone, never together. It ended `done`, front door +200, 0 OOM kills after (`press9201/`). ## Part D — the chaos hour @@ -103,12 +200,90 @@ seed 20260923, drawn in 3 attempt(s) Setup: the six at their FROM pins, each seeded through its front door (A); one whole-box backup on 9202 (the restore rounds need a copy); seed B on docmost, romm and vikunja after it. - +### The twelve rounds — the five things, and the data read back + +Evidence per round: `night-2026-09-23/chaos/round-NN.{json,log}`, `round-NN-run.txt`; rounds 11–12 also +`round-NN-controller.log`. **When the accident landed** is stated, because in rounds 2–5 it did not land +where the schedule meant it to (below). + +| # | action · app × accident | what the household saw (hu / en) | what the box did by itself | steady | events that fired | should have fired, did not | data read back | +|---|---|---|---|---|---|---|---| +| 1 | install · actualbudget × none | installed; badge none | deploy 202 → running | 26.6 s | app_deploy_started, app_deployed; **app_start_failed (warning)** 6 s in — its app cannot be named (see "instrument") | — | A ✓ | +| 2 | good update · docmost 0.95→0.96 (412 spaces) × whole-box backup | „Naprakész" / "Up to date" | safety-dump → pulling → copying (+76.6 s) → starting → verifying → **done 108 s** | 558 s (waited ~7 min for the catalog — instrument) | none | — | A ✓, B ✓ (paged re-read; the first read asked page 1 of 21) | +| 3 | restore · vikunja × controller kill (+28 s) | „Naprakész" / "Up to date" | restore 6 s, running; the kill landed AFTER it; controller back in 0.2 s | 31.5 s | none | — | A ✓, B ✓ | +| 4 | fail health · adventurelog × 1 GB hog | „A(z) adventurelog frissítése … nem sikerült. A doboz automatikusan visszaállította az előző változatot és az adatokat — semmi nem veszett el." / "The update of adventurelog … did not succeed. The box put back the previous version and its data automatically — nothing was lost." | verifying → **undoing +110.9 s → undone +155.1 s** (the hog had ended — instrument) | 605 s | health_change, **app_update_undone (warning)** ✓ | — | A ✓ | +| 5 | install · papra × disk to 3 GB free | installed | deploy → running in 27 s; the fill landed just after | 26.6 s | deploy pair | a disk warning: not measurable (the fill lasted ~20 s, the health job runs every 5 min) | A ✓ | +| 6 | good update · romm 5.3.0→5.3.1 × 1 GB hog (**during verifying**) | „Naprakész" / "Up to date" | … verifying → **done 55.8 s** | 61.7 s | none | — | A ✓ | +| 7 | restore · docmost × **power cut** (+56 s, after the 32 s restore) | „Naprakész" / "Up to date" | restore 32 s; cut; guest back, controller back 12 s, apps up | 103.5 s | controller_started | — | A ✓, B ✓ | +| 8 | cut-off undo copy · navidrome × whole-box backup (+23 s, during verifying) | „Megállítva — visszaállítás szükséges" + the hold sentence naming a restore / "Stopped — restore needed" + the same in English | verifying → undoing +94.9 → **failed 95.4 s**: the copy's marker missing → nothing poured back → HOLD | 101 s | **app_update_held (error)** ✓, then **app_start_failed** 11 s later | — (the second is extra: R-660) | held (A unreadable by design) → **the household's restore brought it back, A ✓** | +| 9 | fail health · vikunja 2.5→2.6 × **controller kill in verifying** | „A(z) vikunja frissítése … nem sikerült. A doboz automatikusan visszaállította …" / "… put back the previous version and its data automatically …" | killed at +58 s → *"update recovery: vikunja was interrupted in verifying … RESUMING the health wait"* → undoing → **undone 157.7 s** — with **`the undo copy will hold 0 named volume(s)`** (R-658) | 163 s | controller_started, **app_update_undone** ✓ | — | A ✓, B ✓ (the old version happens to run on migrated data) | +| 10 | remove · actualbudget × docker restart (+10 s) | the remove press answered **502** | stop done; remove lost to the restart; app stays installed and stopped | cap 1500 s | controller_started; app_start_failed + health_change every 5 min | — | app stopped (A not readable); the household must press again | +| 11 | cut-off undo copy · nextcloud × **a second Update (romm's engine step) at the same moment** | hold as round 8 (both languages); romm „Naprakész" | nextcloud → **failed/HOLD 112.6 s**; romm ran at the same time → **done** (no single-flight, §3b Q4) | 143 s | **app_update_held** ✓, app_start_failed 13 s later | — | romm A ✓; nextcloud held → **the named restore REFUSED → R-659, the stop rule** | +| 12 | remove · papra × none | not installed | stop → remove 200 → verified | 40.9 s | app_removed; health_change (the box-wide job: a held app + a stopped one — true) | — | removed clean (0 containers, 0 volumes) | + +**The instrument, stated rather than hidden.** (1) Rounds 2 and 4 waited ~7 min for the drill commit on +the wrong file (an installed app's stack dir holds the APPLIED compose), so their accidents fired before the +update; round 3's and 5's actions ended before theirs. From round 5 the accident is armed at the press, and +from round 6 the wait reads the badge's own `catalog_images` (seconds). (2) The event column first came from +the controller's debug ring, which overflowed in the long rounds and MISSED both `app_update_held` events; it +was rebuilt from the controller's full log (`E8-events-*`) for rounds 7–12. Rounds 1–6 keep the ring's view, +and round 1's `app_start_failed` cannot be tied to an app: round 7's power cut took the container's earlier +log. From round 11 each round saves the whole controller log. (3) My stop rule first fired on round 8 because +a held app cannot answer; a hold is by design, so recovery — not the readback — is the test. + +### The truth table (`08-alarm-ladder.md`) + +| situation | the ladder says | fired? | +|---|---|---| +| an update undone (rounds 4, 9) | `app_update_undone`, warning, household + operator, one per app per outcome | ✓ both | +| an update held (rounds 8, 11) | `app_update_held`, error | ✓ both — and each was followed by an `app_start_failed` for the same moment (R-660: a held app is stopped by the product and should not ALSO read as down) | +| a controller kill / power cut / docker restart (3, 7, 9, 10) | `controller_started`, info; the boot grace suppresses app alarms for 90 s | ✓ controller_started each time; no app alarm inside the grace | +| a restore that finished (3, 7) | nothing (a refused or interrupted one: `restore_interrupted`) | ✓ nothing | +| a remove (12, teardown) | `app_removed`, info | ✓ | +| a disk filled to 1 GB above the floor (5) | a disk warning from the health job | not measurable — the fill lasted ~20 s | +| the box-wide health job with a held + a stopped app | `health_change`, warning | ✓ every 5 min | + +### demo-hp 9201 — the standing apps Part C moved, pressed through the product + +`night-2026-09-23/press9201/` (live catalog, a hub attached, controller v0.266.0; no whole-box backup pressed). + +| app | from → to | phases | end | front door after | memory after | +|---|---|---|---|---|---| +| opengist | 1.13 → 1.15 | backing-up → … → done | **done in ~16 s** | `/healthcheck` 200 | — | +| kimai | mariadb 11.6 → 11.8 (engine alone) | … → done | **done in ~69 s** | `/en/login` 200 | — | +| romm | 5.3.0 → 5.3.1 **and** mariadb 11.4 → 11.8 in ONE press (no ladder on the box yet) | … → done | **done in ~90 s** | `/api/heartbeat` 200 | 600 MiB / 768, `oom_kill` 0, 0 restarts (checked 10 min later) | + +Every standing app afterwards: 24 containers before, 24 after, all healthy (`E7-9201-after.txt`). Nothing +else on 9201 was touched. ## Teardown - +**Machine — 9202:** every app of the night removed THROUGH THE PRODUCT (docmost, navidrome, vikunja, romm, +nextcloud, adventurelog, actualbudget; papra in round 12): 0 containers, 0 volumes, 0 undo copies left +(`E2-teardown-removes.*`). For the three RESTORED apps the remove answered `volumes_removed: []` while their +volumes did go (compose's `down --volumes` removes them by name) — the report is wrong, not the act (R-658). +The drive folders the product kept (R-442 on 9202, as every night) removed by name: navidrome, nextcloud, romm +(`E3-*`); older folders from earlier nights left as found. 54 test images removed by name before Part D (no +prune): `/var/lib/docker` 86 % → 24 %. `controller.yaml` restored and read back **identical** to +`.pre-night0923`; the catalog cache on 9202 reads live `5d49a11` (`E4-*`, `E5-*`). **9202 stays on controller +v0.267.0** (the scratch guest; self-update off). +**Machine — the bench:** LXC 9401 destroyed (`pct destroy --purge`, its hostname checked first); the Debian +template downloaded for it removed (`E6-*`). +**Machine — 9201:** only the three guarded Updates above; 24 containers before and after, all healthy. +**Host — demo-hp:** `pct list` = 9201, 9202 (as before); `local` 84.49 % before and after; `nvme-scratch` +7.45 % → 7.59 % and `local-lvm` 56.3 % → 57.7 % (the two guests' pulled images) (`00-*`, `E6-*`). +**Drill repo:** reset to live `main` `5d49a11b31fe`, `has_actions: false`, read back (`E5-*`). +**Hub:** not touched. **The floor was NOT raised** — see the three lines. ## Claims in the brief that turned out wrong, or held - +| the brief's claim | what the night found | +|---|---| +| the controller ignores an unknown `.felhom.yml` key (Part B spike) | **held** — v0.266.0 and v0.267.0 synced, deployed, probed and badged navidrome with `update_ladder:` exactly as without it | +| R-633's window closes R-626 | **held, as far as one measurement goes** — 390 s of events + a restart + a reboot: nothing came back. Closed by measurement, not by a named creator | +| 21 backfill records exist | **held** — 21 of 21; one commit (nextcloud's engine move) cited none and its record was found by name | +| wishlist's fault is memory (read from R-612) | **held, and sharpened** — 128M kills the seed every run; whether the sign-up then fails depends on WHEN the kill lands (it did not fail on 9202 tonight) | +| R-518: the page promises „csak néhány másodpercre" | **wrong** — v0.243.0 had already changed it to „általában néhány perc"; tonight states the measured ≈ 8 min | +| `check-image-resolvable.py` already resolves digests | **half** — it asks whether a ref EXISTS and throws the digest away; `image_digest.py` was written | +| one chaos round ≈ one accident inside one action | **not as drawn** — actions of 6–30 s ended before accidents at +28–56 s; four rounds' accidents landed outside the action (stated per round) | +| the chaos schedule's round-11 second update (first draft: adventurelog) | changed BEFORE round 1 to romm's engine step — adventurelog v0.13.0 cannot pass health | diff --git a/documentation/audits/night-2026-09-23/C-moves.txt b/documentation/audits/night-2026-09-23/C-moves.txt index af5d252e..d9957d2c 100644 --- a/documentation/audits/night-2026-09-23/C-moves.txt +++ b/documentation/audits/night-2026-09-23/C-moves.txt @@ -10,3 +10,4 @@ kimai-db mariadb 11.6->11.8 moved in catalog 376b2c3 at 21:59 romm-db mariadb 11.4->11.8 moved in catalog 431cdec at 22:00 immich-machine-learning v3.0.3->v3.2.2 moved in catalog b82b7c6 at 22:01 22:04 9202 images removed by name before Part D (54 test/old images, no prune): /var/lib/docker 86% -> 24% +nextcloud 34.0.1->34.0.4 (files_may_change) moved in catalog 5d49a11 at 22:19 diff --git a/documentation/audits/night-2026-09-23/C-table.md b/documentation/audits/night-2026-09-23/C-table.md new file mode 100644 index 00000000..b3a6b5e7 --- /dev/null +++ b/documentation/audits/night-2026-09-23/C-table.md @@ -0,0 +1,16 @@ +| app | from → to | class | bench verdict | box verdict | memory: own (cgroup) peak % | migration line | moved? | +|---|---|---|---|---|---|---|---| +| nextcloud | nextcloud 34.0.1 → 34.0.4 | MariaDB + file-leg | proven | proven (done) | 24.2 (100) | nextcloud / Upgrading nextcloud from 34.0.1.2 ... | moved `5d49a11` (files_may_change) | +| romm | romm 5.3.0 → 5.3.1 | MariaDB | proven | proven (done) | n/a — measured before the own-memory sample existed (cgroup 79.5) | romm / INFO: [RomM][init][2026-09-23 21:12:26][0 | moved `7ff4e68` | +| romm-engine | romm-db mariadb 11.4 → 11.8 (own edge) | MariaDB engine | proven | proven (done) | 78.0 (84) | romm / INFO: [RomM][init][2026-09-23 21:47:51][0 | moved `431cdec` | +| kimai | kimai-db mariadb 11.6 → 11.8 (own edge) | MariaDB engine | proven | proven (done) | 37.9 (100) | kimai / [OK] Already at the latest version ("DoctrineMigrations\Version2 | moved `376b2c3` | +| adventurelog | adventurelog v0.12.1 → v0.13.0 (both images) | PostgreSQL (PostGIS) | failed ×2 (frontend never healthy; then backend crash-loop on `download-countries`) | failed (undone at +196 s, drill timeout 90 s) | — | adventurelog / Apply all migrations: account, admin, adventures, aut | **NOT moved** — our template's frontend healthcheck points at `/nodejs/bin/node`, gone in v0.13.0; with it dropped, the backend's start-time internet download crash-looped (R-655) | +| immich | immich-machine-learning v3.0.3 → v3.2.2 | PostgreSQL (server unchanged) | proven | proven (done) | 51.4 (100) | immich-server / [Nest] 7 - 09/23/2026, 9:11:07 PM LO | moved `b82b7c6` | +| n8n | n8n 2.40.5 → 2.41.1 | SQLite | proven | proven (done) | 22.7 (47) | n8n / Migrations in progress, please do NOT stop the process. | moved `04b63db` | +| ghost | ghost 6.64.0 → 6.65.0 | SQLite | proven | proven (done) | 24.9 (43) | ghost / [2026-09-23 21:21:22] WARN Database state requires migration | moved `e6afab9` (first watch had no load — re-run) | +| komga | komga 1.25.0 → 1.27.1 | embedded | proven | proven (done) | 60.0 (77) | komga / 2026-09-23T21:35:41.616+02:00 INFO 1 --- [ main] o.f.core | moved `fc1becc` with 768M (memory_tight at 512M) | +| opengist | opengist 1.13 → 1.15 | SQLite | proven | proven (done) | 77.8 (100) | — | moved `657c83f` (pages under `/-/`, R-654) | +| wishlist | wishlist v0.66.0 → v0.67.1 | SQLite | proven | proven (done) | 35.9 (53) | wishlist / Prisma schema loaded from prisma/schema.prisma. | moved `1545c5f` (after R-612's 512M) | +| navidrome | navidrome 0.64.0 → 0.64.1 | SQLite | proven | proven (done) | 6.3 (9) | navidrome / time="2026-09-23T19:31:04Z" level=info msg="goose: no migrations | moved `76cc1f5` (second ladder step) | +| emby | emby 4.11.0.1 → 4.11.0.3 | embedded | proven | proven (done) | 4.5 (8) | emby / Info SqliteUserRepository: Sqlite compiler options: ATOMIC_INTRINSICS | moved `8d573ad` | +| gitea | gitea 1.27.0 → 1.27.3 | SQLite | inconclusive | inconclusive () | — | — | NOT moved — inconclusive on both: its web installer blocks a headless first account (R-624) | diff --git a/documentation/audits/night-2026-09-23/D-rounds.md b/documentation/audits/night-2026-09-23/D-rounds.md new file mode 100644 index 00000000..c02d9c02 --- /dev/null +++ b/documentation/audits/night-2026-09-23/D-rounds.md @@ -0,0 +1,14 @@ +| # | action · app | accident (when) | what the household saw | what the box did by itself | steady | events fired (R-620 witness) | data read back | +|---|---|---|---|---|---|---|---| +| 1 | install · actualbudget | none (armed at start) | hu: — ‖ en: — | deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} | 26.6 s | app_deploy_started (info), app_start_failed (warning), app_deployed (info) | {'A': True} | +| 2 | good_update · docmost | POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'} at 22:30:30 (armed at start) | hu: Naprakész ‖ en: Up to date | safety-dump@0.0s → pulling@2.1s → copying@76.6s → starting@81.4s → verifying@92.4s → done@108.1s | INFO [stacks] update docmost: safety dump done (1 file(s)) [/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260923… | 558.3 s | none | {'A': True, 'B': 'True (re-read, paged)'} | +| 3 | restore · vikunja | kill -9 of the controller's PID: killed 11644 at 22:39:39 (armed at start) | hu: Naprakész ‖ en: Up to date | {'ok': True, 'why': None, 'snapshot_id': 'helyi', 'http': 'HTTP/2 302', 'seconds': 6.1, 'state_after': 'running', 'hold_… | INFO [backup] Recovery unit captured for docmost → /mnt/sys_drive/felhom-data/backups/primary/docmost (images=3, secrets-referenced=2, data_… | 31.5 s | none | {'A': True, 'B': True} | +| 4 | fail_health · adventurelog | 1 GB hog for 5 min: 5cd58c5c3e56c7b2fbb574e9be159f17cc25033c1d3a0e87f0b2692646d8660c chaos-hog 1003MiB / 25.29… (armed at start) | hu: Frissítés elérhető — ma / A(z) adventurelog frissítése 2026-09-23 22:49-kor nem sikerült. A doboz automatikusan visszaállította az előző… ‖ en: Update available — today / The update of adventurelog at 2026-09-23 22:49 did not succeed. The box put back the previous version and its … | safety-dump@0.0s → pulling@2.1s → copying@3.2s → starting@14.2s → verifying@20.5s → undoing@110.9s → undone@155.1s | INFO [stacks] update adventurelog: UNDONE in 44s — the previous version is running on the data from before the update (the app's health chec… | 605.2 s | health_change (warning), app_update_undone (warning) | {'A': True} | +| 5 | install · papra | filled /var/lib/docker to 3 GB free (floor 2 GB + 1 GB): /dev/loop1 69G 62G 3.0G 96% /var/lib/docker at 22:50:… (armed 22:50:17) | hu: — ‖ en: — | deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} | 26.6 s | app_deploy_started (info), app_deployed (info) | {'A': True} | +| 6 | good_update · romm | 1 GB hog for 5 min: 7922307cf3f2b0e1bebb33d30a94d644382b7596a42d0024aec74d1a81d5762f chaos-hog 1004MiB / 25.29… (armed 22:51:04) | hu: Naprakész ‖ en: Up to date | safety-dump@0.0s → pulling@2.1s → copying@3.2s → starting@8.9s → verifying@20.0s → done@55.8s | INFO [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping ensure (held by drive gate) | 61.7 s | none | {'A': True} | +| 7 | restore · docmost | pct stop rc=0, pct start rc=0 at 22:53:24 (armed 22:52:28) | hu: Naprakész ‖ en: Up to date | {'ok': True, 'why': None, 'snapshot_id': 'helyi', 'http': 'HTTP/2 302', 'seconds': 32.5, 'state_after': 'running', 'hold… | INFO [backup] Recovery unit captured for romm → /mnt/felhom-drives/scratch_hdd/userdata/romm/backups/primary/romm (images=3, secrets-referen… | 103.5 s | controller_started (info) | {'A': True, 'B': True} | +| 8 | cutoff · navidrome | POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'} at 22:54:51 (armed 22:54:28) | hu: Megállítva — visszaállítás szükséges / A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotb… ‖ en: Stopped — restore needed / The update did not succeed, and the automatic undo did not either. The data is as the new version left it. The… | safety-dump@0.0s → pulling@1.1s → copying@2.1s → starting@3.2s → verifying@3.7s → undoing@94.9s → failed@95.4s | INFO [backup] Recovery unit captured for papra → /mnt/sys_drive/felhom-data/backups/primary/papra (images=1, secrets-referenced=1, data_keys… | 101.2 s | app_start_failed (warning) | {'A': False} | +| 9 | fail_health · vikunja | kill -9 of the controller's PID: killed 3308 at 23:04:11 (armed 23:03:13) | hu: Frissítés elérhető — ma / A(z) vikunja frissítése 2026-09-23 23:05-kor nem sikerült. A doboz automatikusan visszaállította az előző vált… ‖ en: Update available — today / The update of vikunja at 2026-09-23 23:05 did not succeed. The box put back the previous version and its data … | safety-dump@0.0s → pulling@0.6s → copying@3.7s → verifying@4.2s → undoing@152.0s → undone@157.7s | INFO [stacks] update vikunja: UNDONE in 6s — the previous version is running on the data from before the update (the app's health check pass… | 163.4 s | app_update_undone (warning) | {'A': True, 'B': True} | +| 10 | remove · actualbudget | systemctl restart docker: rc=0 at 23:06:10 (armed 23:06:00) | hu: Naprakész ‖ en: Up to date | remove -> 502 | 1500.4 s | none | {'A': False} | +| 11 | cutoff · nextcloud | a second Update on romm -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissí… (armed 23:40:46) | hu: Megállítva — visszaállítás szükséges / A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotb… ‖ en: Stopped — restore needed / The update did not succeed, and the automatic undo did not either. The data is as the new version left it. The… | safety-dump@0.0s → pulling@2.1s → copying@3.7s → starting@13.2s → verifying@19.5s → undoing@109.9s → failed@112.6s | INFO [backup] Recovery unit captured for romm → /mnt/felhom-drives/scratch_hdd/userdata/romm/backups/primary/romm (images=3, secrets-referen… | 142.7 s | none | {'A': False} | +| 12 | remove · papra | none (armed 23:56:06) | hu: — ‖ en: — | remove -> 200 | 40.9 s | health_change (warning), app_removed (info) | None | diff --git a/documentation/audits/night-2026-09-23/E1-r640-live.json b/documentation/audits/night-2026-09-23/E1-r640-live.json new file mode 100644 index 00000000..bebb1900 --- /dev/null +++ b/documentation/audits/night-2026-09-23/E1-r640-live.json @@ -0,0 +1,62 @@ +{ + "dump": "/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/docmost-postgres.sql", + "cut": "whole=1923884 cut=961942\nur big database filler chaos-hour big database filler chaos-hour big database filler chaos-hour big database filler chaos-hour big database filler chaos-hour big database filler chaos-hour big databas\n0\n", + "refused_restore": { + "result": { + "http": "HTTP/2 302", + "location": [ + "location: /backups/restore?flash=flash.restore.started" + ], + "seconds": 6.1, + "state_after": "running", + "hold_after": null + }, + "status": { + "ok": true, + "data": { + "running": false, + "op": "restore", + "stack": "docmost", + "started_at": "2026-09-23T22:09:04.469721192Z", + "last": { + "op": "restore", + "stack": "docmost", + "ok": false, + "message": "Visszaállítás sikertelen: A(z) docmost adatbázis-másolata csonka: nem ér véget, ahogy egy teljes másolat. A visszaállítás biztonsági okból nem indult el — az alkalmazás és az adatai érintetlenek.", + "finished_at": "2026-09-23T22:09:04.472772813Z" + }, + "last_recent": true + } + } + }, + "page_hu": [], + "page_en": [], + "untouched": true, + "readback_A": true, + "control_restore": { + "result": { + "http": "HTTP/2 302", + "seconds": 32.4, + "state_after": "running", + "hold_after": null + }, + "status": { + "ok": true, + "data": { + "running": false, + "op": "restore", + "stack": "docmost", + "started_at": "2026-09-23T22:09:20.594671114Z", + "last": { + "op": "restore", + "stack": "docmost", + "ok": true, + "message": "A(z) docmost: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", + "finished_at": "2026-09-23T22:09:51.02510124Z" + }, + "last_recent": true + } + } + }, + "control_readback_A": true +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/E1-r640-live.log b/documentation/audits/night-2026-09-23/E1-r640-live.log new file mode 100644 index 00000000..a99addc1 --- /dev/null +++ b/documentation/audits/night-2026-09-23/E1-r640-live.log @@ -0,0 +1,39 @@ +00:08:58 the unit's dump: +total 3748 +drwxr-xr-x 2 root root 4096 Sep 23 20:54 . +drwxr-xr-x 5 root root 4096 Sep 23 20:39 .. +-rw-r--r-- 1 root root 1923884 Sep 23 20:54 docmost-postgres.sql +-rw-r--r-- 1 root root 1904362 Sep 23 20:37 pre-restore-20260923T203715Z-docmost-postgres.sql + +00:09:01 before: +docmost About an hour ago +docmost-postgres About an hour ago +docmost-redis About an hour ago +/docmost 2026-09-23T21:06:20.610992037Z +/docmost-postgres 2026-09-23T21:06:20.683563334Z + +00:09:04 cut: +whole=1923884 cut=961942 +ur big database filler chaos-hour big database filler chaos-hour big database filler chaos-hour big database filler chaos-hour big database filler chaos-hour big database filler chaos-hour big databas +0 + +00:09:04 [R] restoring docmost from snapshot 'helyi' (of 1 offered) +00:09:04 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +00:09:04 + 0.0s restore (False, None, None) +00:09:10 [R] after restore: state=running hold=None phase=None +00:09:13 restore-status after the refused restore: {"ok": true, "data": {"running": false, "op": "restore", "stack": "docmost", "started_at": "2026-09-23T22:09:04.469721192Z", "last": {"op": "restore", "stack": "docmost", "ok": false, "message": "Visszaállítás sikertelen: A(z) docmost adatbázis-másolata csonka: nem ér véget, ahogy egy teljes másolat. A visszaállítás biztonsági okból nem indult el — az alkalmazás és az adatai érintetlenek.", "finished_at": "2026-09-23T22:09:04.472772813Z"}, "last_recent": true}} +00:09:13 /backups (hu) carries the sentence: [] +00:09:13 /backups (en) carries the sentence: [] +00:09:16 after: +/docmost 2026-09-23T21:06:20.610992037Z +/docmost-postgres 2026-09-23T21:06:20.683563334Z + +00:09:17 docmost: login as the seeded user http=200 ok=True +00:09:17 container start times unchanged: True; seed A: True +00:09:20 [R] restoring docmost from snapshot 'helyi' (of 1 offered) +00:09:20 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +00:09:20 + 0.0s restore (True, None, None) +00:09:53 + 32.4s restore (False, None, None) +00:09:53 [R] after restore: state=running hold=None phase=None +00:09:57 docmost: login as the seeded user http=200 ok=True +00:09:57 control (whole copy): status {"ok": true, "data": {"running": false, "op": "restore", "stack": "docmost", "started_at": "2026-09-23T22:09:20.594671114Z", "last": {"op": "restore", "stack": "docmost", "ok": true, "message": "A(z) docmost: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "202; seed A True diff --git a/documentation/audits/night-2026-09-23/E2-teardown-removes.json b/documentation/audits/night-2026-09-23/E2-teardown-removes.json new file mode 100644 index 00000000..7ac8db1b --- /dev/null +++ b/documentation/audits/night-2026-09-23/E2-teardown-removes.json @@ -0,0 +1,13 @@ +{ + "before": "docmost all=3 labeled=0 copies=0\nnavidrome all=2 labeled=0 copies=1\nvikunja all=2 labeled=0 copies=0\nromm all=3 labeled=3 copies=0\nnextcloud all=6 labeled=3 copies=3\nadventurelog all=2 labeled=2 copies=0\nactualbudget all=0 labeled=0 copies=0\n", + "remove": { + "docmost": "200", + "navidrome": "200", + "vikunja": "200", + "romm": "200", + "nextcloud": "200", + "adventurelog": "200", + "actualbudget": "200" + }, + "after": "docmost containers=0 volumes= copies=0\nnavidrome containers=0 volumes= copies=0\nvikunja containers=0 volumes= copies=0\nromm containers=0 volumes= copies=0\nnextcloud containers=0 volumes= copies=0\nadventurelog containers=0 volumes= copies=0\nactualbudget containers=0 volumes= copies=0\n" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/E2-teardown-removes.log b/documentation/audits/night-2026-09-23/E2-teardown-removes.log new file mode 100644 index 00000000..e2570330 --- /dev/null +++ b/documentation/audits/night-2026-09-23/E2-teardown-removes.log @@ -0,0 +1,45 @@ +00:10:27 volumes before the removes: +docmost all=3 labeled=0 copies=0 +navidrome all=2 labeled=0 copies=1 +vikunja all=2 labeled=0 copies=0 +romm all=3 labeled=3 copies=0 +nextcloud all=6 labeled=3 copies=3 +adventurelog all=2 labeled=2 copies=0 +actualbudget all=0 labeled=0 copies=0 + +00:10:28 [X] stop -> 200 {'ok': True, 'message': 'Stack docmost stop completed'} +00:11:00 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'docmost', 'volumes_removed': [], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt saj +00:11:08 [X] after remove: deployed=False leftovers='/opt/docker/stacks/docmost' +00:11:09 [X] stop -> 200 {'ok': True, 'message': 'Stack navidrome stop completed'} +00:11:14 [X] remove (with drive data) -> 409 {'ok': False, 'error': 'A(z) /mnt/felhom-drives/scratch_hdd/userdata/navidrome tárhely jelenleg nem elérhető — az alkalmazás nem távolítható el, amíg a meghajtó +00:11:14 [X] refused because the drive path cannot be resolved (R-442, fail-closed and right) — removing the app and KEEPING the drive data instead +00:11:40 [X] remove (keeping drive data) -> 200 {'ok': True, 'data': {'removed': 'navidrome', 'volumes_removed': [], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'backup_paths_removed': ['/mnt/felhom-d +00:11:49 [X] after remove: deployed=False leftovers='/opt/docker/stacks/navidrome' +00:11:49 [X] stop -> 200 {'ok': True, 'message': 'Stack vikunja stop completed'} +00:12:21 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'vikunja', 'volumes_removed': [], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt saj +00:12:29 [X] after remove: deployed=False leftovers='/opt/docker/stacks/vikunja' +00:12:33 [X] stop -> 200 {'ok': True, 'message': 'Stack romm stop completed'} +00:12:38 [X] remove (with drive data) -> 409 {'ok': False, 'error': 'A(z) /mnt/felhom-drives/scratch_hdd/userdata/romm tárhely jelenleg nem elérhető — az alkalmazás nem távolítható el, amíg a meghajtó viss +00:12:38 [X] refused because the drive path cannot be resolved (R-442, fail-closed and right) — removing the app and KEEPING the drive data instead +00:13:05 [X] remove (keeping drive data) -> 200 {'ok': True, 'data': {'removed': 'romm', 'volumes_removed': ['romm_romm_config', 'romm_romm_db_data', 'romm_romm_redis_data'], 'hdd_paths_removed': [], 'hdd_pat +00:13:13 [X] after remove: deployed=False leftovers='/opt/docker/stacks/romm' +00:13:13 [X] stop -> 200 {'ok': True, 'message': 'Stack nextcloud stop completed'} +00:13:18 [X] remove (with drive data) -> 409 {'ok': False, 'error': 'A(z) /mnt/felhom-drives/scratch_hdd/userdata/nextcloud tárhely jelenleg nem elérhető — az alkalmazás nem távolítható el, amíg a meghajtó +00:13:18 [X] refused because the drive path cannot be resolved (R-442, fail-closed and right) — removing the app and KEEPING the drive data instead +00:13:46 [X] remove (keeping drive data) -> 200 {'ok': True, 'data': {'removed': 'nextcloud', 'volumes_removed': ['nextcloud_nextcloud_db_data', 'nextcloud_nextcloud_html', 'nextcloud_nextcloud_redis_data'], +00:13:55 [X] after remove: deployed=False leftovers='/opt/docker/stacks/nextcloud' +00:13:58 [X] stop -> 200 {'ok': True, 'message': 'Stack adventurelog stop completed'} +00:14:30 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'adventurelog', 'volumes_removed': ['adventurelog_adventurelog_media', 'adventurelog_adventurelog_postgres_data'], 'hdd_paths_r +00:14:39 [X] after remove: deployed=False leftovers='/opt/docker/stacks/adventurelog' +00:14:39 [X] stop -> 200 {'ok': True, 'message': 'Stack actualbudget stop completed'} +00:15:10 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'actualbudget', 'volumes_removed': [], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárol +00:15:19 [X] after remove: deployed=False leftovers='/opt/docker/stacks/actualbudget' +00:15:22 after the removes (the product's own): +docmost containers=0 volumes= copies=0 +navidrome containers=0 volumes= copies=0 +vikunja containers=0 volumes= copies=0 +romm containers=0 volumes= copies=0 +nextcloud containers=0 volumes= copies=0 +adventurelog containers=0 volumes= copies=0 +actualbudget containers=0 volumes= copies=0 + diff --git a/documentation/audits/night-2026-09-23/E3-9202-drive-folders.txt b/documentation/audits/night-2026-09-23/E3-9202-drive-folders.txt new file mode 100644 index 00000000..6a08e316 --- /dev/null +++ b/documentation/audits/night-2026-09-23/E3-9202-drive-folders.txt @@ -0,0 +1,2 @@ +calibre-web documents downloads emby immich jellyfin komga media paperless-ngx plex radarr roms sonarr +169 diff --git a/documentation/audits/night-2026-09-23/E4-9202-repoint-live.txt b/documentation/audits/night-2026-09-23/E4-9202-repoint-live.txt new file mode 100644 index 00000000..58720efb --- /dev/null +++ b/documentation/audits/night-2026-09-23/E4-9202-repoint-live.txt @@ -0,0 +1,10 @@ +git: + branch: main + repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git + sync_interval: 15m + token: + username: "" +hub: +0 + +IDENTICAL-to-pre-night0923 diff --git a/documentation/audits/night-2026-09-23/E5-drill-reset.txt b/documentation/audits/night-2026-09-23/E5-drill-reset.txt new file mode 100644 index 00000000..9dc32d7f --- /dev/null +++ b/documentation/audits/night-2026-09-23/E5-drill-reset.txt @@ -0,0 +1,2 @@ +drill origin/main=5d49a11b31fe live main=5d49a11b31fe has_actions=False (00:16) +5d49a11 nextcloud: 34.0.1 -> 34.0.4 (proven on the bench AND on 9202, with its test record) — files_may_change diff --git a/documentation/audits/night-2026-09-23/E6-demo-hp-after-destroy.txt b/documentation/audits/night-2026-09-23/E6-demo-hp-after-destroy.txt new file mode 100644 index 00000000..a691fcd4 --- /dev/null +++ b/documentation/audits/night-2026-09-23/E6-demo-hp-after-destroy.txt @@ -0,0 +1,9 @@ +purging CT 9401 from related configurations.. +destroyed +template-removed +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +local dir active 40453376 34177084 4189176 84.49% +local-lvm lvmthin active 56487936 32610485 23877450 57.73% +nvme-scratch dir active 983379700 74638248 858714840 7.59% diff --git a/documentation/audits/night-2026-09-23/E6-demo-hp-before-destroy.txt b/documentation/audits/night-2026-09-23/E6-demo-hp-before-destroy.txt new file mode 100644 index 00000000..952874df --- /dev/null +++ b/documentation/audits/night-2026-09-23/E6-demo-hp-before-destroy.txt @@ -0,0 +1,6 @@ +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 running upgrade-harness +local dir active 40453376 34304068 4062192 84.80% +nvme-scratch dir active 983379700 106830612 826522476 10.86% diff --git a/documentation/audits/night-2026-09-23/E7-9201-after.txt b/documentation/audits/night-2026-09-23/E7-9201-after.txt new file mode 100644 index 00000000..1550c823 --- /dev/null +++ b/documentation/audits/night-2026-09-23/E7-9201-after.txt @@ -0,0 +1,24 @@ +romm rommapp/romm:5.3.1 Up 56 minutes (healthy) +romm-db mariadb:11.8 Up 56 minutes (healthy) +kimai-db mariadb:11.8 Up 58 minutes (healthy) +opengist ghcr.io/thomiceli/opengist:1.15 Up 58 minutes (healthy) +felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.266.0 Up 6 hours (healthy) +romm-redis redis:7-alpine Up 56 minutes (healthy) +privatebin privatebin/pdo:2.0.6 Up 10 hours (healthy) +paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 10 hours (healthy) +paperless-postgres postgres:16-alpine Up 10 hours (healthy) +paperless-redis redis:7-alpine Up 10 hours (healthy) +kimai kimai/kimai2:apache-2.57.0 Up 57 minutes (healthy) +docmost docmost/docmost:0.96.0 Up 10 hours (healthy) +docmost-postgres postgres:16-alpine Up 10 hours (healthy) +docmost-redis redis:7-alpine Up 10 hours (healthy) +calibre-web crocodilestick/calibre-web-automated:v4.0.6 Up 10 hours (healthy) +bookstack lscr.io/linuxserver/bookstack:26.05.5 Up 10 hours (healthy) +bookstack-db mariadb:12.3 Up 10 hours (healthy) +adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 Up 10 hours (healthy) +adventurelog-postgres postgis/postgis:16-3.5-alpine Up 10 hours (healthy) +adventurelog-frontend ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 Up 10 hours (healthy) +bentopdf ghcr.io/alam00000/bentopdf:v2.8.6 Up 17 hours (healthy) +filebrowser gtstef/filebrowser:1.3.3-stable Up 3 weeks (healthy) +cloudflared cloudflare/cloudflared:2026.6.0 Up 3 weeks +traefik traefik:v3.6.7 Up 3 weeks diff --git a/documentation/audits/night-2026-09-23/E8-events-from-controller-log.txt b/documentation/audits/night-2026-09-23/E8-events-from-controller-log.txt new file mode 100644 index 00000000..dd870fc5 --- /dev/null +++ b/documentation/audits/night-2026-09-23/E8-events-from-controller-log.txt @@ -0,0 +1,40 @@ +2026/09/23 20:53:44 main.go:335: [INFO] felhom-controller 0.267.0 starting (customer: demo-hp, domain: enkisfelhom.hu) +2026/09/23 20:53:50 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event controller_started (severity info) — further controller_started events are logged at DEBUG only +2026/09/23 20:56:04 undo.go:475: [INFO] [stacks] update navidrome: event app_update_held (hold recorded: true) +2026/09/23 20:56:04 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_update_held (severity error) — further app_update_held events are logged at DEBUG only +2026/09/23 20:56:15 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_start_failed (severity warning) — further app_start_failed events are logged at DEBUG only +2026/09/23 20:58:45 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event health_change (severity warning) — further health_change events are logged at DEBUG only +2026/09/23 21:01:45 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_start_failed (severity warning) +2026/09/23 21:03:45 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:04:19 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event controller_started (severity info) — further controller_started events are logged at DEBUG only +2026/09/23 21:05:51 undo.go:477: [INFO] [stacks] update vikunja: event app_update_undone +2026/09/23 21:05:51 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_update_undone (severity warning) — further app_update_undone events are logged at DEBUG only +2026/09/23 21:06:27 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event controller_started (severity info) — further controller_started events are logged at DEBUG only +2026/09/23 21:11:23 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_start_failed (severity warning) — further app_start_failed events are logged at DEBUG only +2026/09/23 21:11:23 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event health_change (severity warning) — further health_change events are logged at DEBUG only +2026/09/23 21:16:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:21:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:26:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:31:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:36:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:41:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:42:39 undo.go:475: [INFO] [stacks] update nextcloud: event app_update_held (hold recorded: true) +2026/09/23 21:42:39 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_update_held (severity error) — further app_update_held events are logged at DEBUG only +2026/09/23 21:42:52 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_start_failed (severity warning) +2026/09/23 21:46:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:51:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:56:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:56:38 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_removed (severity info) — further app_removed events are logged at DEBUG only +2026/09/23 22:01:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 22:06:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 22:09:22 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_start_failed (severity warning) +2026/09/23 22:11:00 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_removed (severity info) +2026/09/23 22:11:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 22:11:40 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_removed (severity info) +2026/09/23 22:11:53 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_start_failed (severity warning) +2026/09/23 22:12:21 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_removed (severity info) +2026/09/23 22:13:05 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_removed (severity info) +2026/09/23 22:13:46 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_removed (severity info) +2026/09/23 22:14:30 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_removed (severity info) +2026/09/23 22:15:10 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_removed (severity info) +2026/09/23 22:15:53 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event controller_started (severity info) — further controller_started events are logged at DEBUG only diff --git a/documentation/audits/night-2026-09-23/E8-events-per-round.json b/documentation/audits/night-2026-09-23/E8-events-per-round.json new file mode 100644 index 00000000..1771a3e2 --- /dev/null +++ b/documentation/audits/night-2026-09-23/E8-events-per-round.json @@ -0,0 +1,56 @@ +{ + "1": [], + "2": [], + "3": [], + "4": [], + "5": [], + "6": [], + "7": [ + "20:53:50 controller_started (info)" + ], + "8": [ + "20:56:04 app_update_held (error)", + "20:56:15 app_start_failed (warning)", + "20:58:45 health_change (warning)", + "21:01:45 app_start_failed (warning)" + ], + "9": [ + "21:03:45 health_change (warning)", + "21:04:19 controller_started (info)", + "21:05:51 app_update_undone (warning)" + ], + "10": [ + "21:06:27 controller_started (info)", + "21:11:23 app_start_failed (warning)", + "21:11:23 health_change (warning)", + "21:16:23 health_change (warning)", + "21:21:23 health_change (warning)", + "21:26:23 health_change (warning)", + "21:31:23 health_change (warning)", + "21:36:23 health_change (warning)" + ], + "11": [ + "21:41:23 health_change (warning)", + "21:42:39 app_update_held (error)", + "21:42:52 app_start_failed (warning)", + "21:46:23 health_change (warning)", + "21:51:23 health_change (warning)" + ], + "12": [ + "21:56:23 health_change (warning)", + "21:56:38 app_removed (info)", + "22:01:23 health_change (warning)", + "22:06:23 health_change (warning)", + "22:09:22 app_start_failed (warning)", + "22:11:00 app_removed (info)", + "22:11:23 health_change (warning)", + "22:11:40 app_removed (info)", + "22:11:53 app_start_failed (warning)", + "22:12:21 app_removed (info)", + "22:13:05 app_removed (info)", + "22:13:46 app_removed (info)", + "22:14:30 app_removed (info)", + "22:15:10 app_removed (info)", + "22:15:53 controller_started (info)" + ] +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/PROGRESS.md b/documentation/audits/night-2026-09-23/PROGRESS.md index 9074b168..f67db910 100644 --- a/documentation/audits/night-2026-09-23/PROGRESS.md +++ b/documentation/audits/night-2026-09-23/PROGRESS.md @@ -28,3 +28,14 @@ | 21:41 | C move | **wishlist v0.66.0→v0.67.1** + **opengist 1.13→1.15** `657c83f` (CI 925) | apps/wishlist, apps/opengist | | 21:45 | C move | **navidrome 0.64.0→0.64.1** `76cc1f5` — second step on its ladder | apps/navidrome | | 21:40 | C not moved | **adventurelog v0.13.0**: backend + data fine, the NEW frontend's own healthcheck (`/health` → backend `/health/`) never passes in the template's network shape — box UNDID the update (under the 90 s drill timeout), bench never healthy in 420 s | apps/adventurelog | +| 21:58–22:18 | C moves | kimai-db 11.6→11.8 `376b2c3`, romm-db 11.4→11.8 `431cdec` (each its own edge; the engine says "already upgraded to 11.8.9"), immich-ML v3.0.3→v3.2.2 `b82b7c6`, emby `8d573ad`, n8n `04b63db`, komga + 768M `fc1becc`, nextcloud (files_may_change) `5d49a11` — **Part C done: 12 steps, 11 apps** | C-moves.txt | +| 22:05 | D0 | chaos schedule + app table committed BEFORE round 1 (`cf8dec8`); round 11's second update changed to romm's engine step before round 1 | D0-schedule.txt | +| 22:25 | D setup | six deployed + seeded (A), whole-box backup 95 s, seed B (docmost, vikunja; romm's B route 404 on 5.3.0); docmost made big with ONE login (412 spaces, 11 MB — its login is rate-limited); nextcloud rebuilt clean after **R-657** | chaos/00-* | +| 22:29–23:57 | D rounds | 12 rounds run (chaos/round-NN.*); rounds 2–4 accidents landed early (the runner waited on the wrong file for the catalog — fixed: armed at the press); round 8 held → household restore brought it back, A read; round 9 kill mid-verify → resumed → UNDONE, A+B read; round 11 held → **the named restore REFUSED → R-659 (P1), the stop rule** | chaos/, D-rounds.md | +| 23:12 | D finding | restored apps' volumes carry NO compose label → the undo copies nothing (**R-658, P1**), with a control (never-restored apps labeled) | chaos/09-volumes-unlabeled-after-restore.txt | +| 23:21 | D 9201 | guarded Update on the HP box's standing apps moved tonight: opengist → 1.15 `done`, kimai-db → 11.8 `done`, romm → 5.3.1 + 11.8 in ONE press `done`; front doors 200; romm 0 OOM kills after | press9201/ | +| 00:10 | E1 R-640 live | the household's restore refused a half-length docmost copy („…csonka… nem indult el — az alkalmazás és az adatai érintetlenek"); container start times unchanged; A read; the whole copy then restored | E1-r640-live.* | +| 00:15 | E teardown | 9202: all apps removed through the product (0 containers/volumes/copies; restored apps' remove answered `volumes_removed: []` though the volumes went — R-658), kept drive folders removed by name, config identical, live catalog; drill reset `5d49a11`, has_actions false; bench 9401 destroyed + template removed; 9201 24/24 healthy | E2–E7 | +| 00:20 | E events | the debug ring missed both `app_update_held` — rebuilt from the controller's full log (`E8-*`); R-660 (a held app also raises app_start_failed) | E8-* | +| 00:30 | E floor | **NOT raised** (brief's rule: Part D found R-658/R-659, both pre-existing); recommendation to raise → the operator's one question in STATUS | STATUS.md | +| 00:45 | E docs | DRILL record, `09` §3 21–23 + §6.4 + §6.1a, capability map, rotation tick (opengist), register 330 → 336 (R-651..R-660 opened; R-650, R-640, R-499, R-626 closed), CONTEXT, REPORT, STATUS | — | diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/MV-adventurelog.log b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/MV-adventurelog.log new file mode 100644 index 00000000..58e5d854 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/MV-adventurelog.log @@ -0,0 +1,47 @@ +[19:24:31] MV-adventurelog: deploying adventurelog at FROM {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.12.1', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1'} +[19:27:32] FROM settled=True in 124.8s :: {"adventurelog": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[19:27:32] fixture: the BOX walk's own (Django), through upgrade_boxport +[19:27:36] adventurelog: createsuperuser :: Superuser created successfully. +[19:27:39] adventurelog: seeded superuser drill3cc072 +[19:27:43] adventurelog: readback of the seeded account found=True +[19:27:43] C1 (seed reads back BEFORE): True +[19:27:44] MV-adventurelog: swapping to TO {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0'} +[19:28:25] TO up -d rc=0 +[19:35:29] TO settled=False in 423.9s :: {"adventurelog": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": "unhealthy", "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[19:35:29] migration lines observed: 6 +[19:35:34] adventurelog: readback of the seeded account found=True +[19:35:34] RESULT (seed reads back AFTER): True +[19:35:34] MV-adventurelog: ABORT — putting the FROM images back +[19:36:15] adventurelog: readback of the seeded account found=True +[19:36:15] ABORT: app came back in 31.4s; data present=True +{ + "harness_version": 3, + "edge": "MV-adventurelog", + "app": "adventurelog", + "note": "night 2026-09-23 within-a-major move: adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.13.0, adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0", + "from": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1" + }, + "to": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.13.0", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0" + }, + "verdict": "failed", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": false, + "migration_observed": "\u001b[2Kadventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel", + "abort": "starts-and-serves", + "abort_detail": null, + "engine_state_after": null, + "memory": null, + "marks": [], + "duration_s": 423.9, + "measured_at": "2026-09-23T19:36:15Z", + "evidence": "evidence/MV-adventurelog", + "files_changed": [], + "total_s": 703.4 +} diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/abort-states.json b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/abort-states.json new file mode 100644 index 00000000..359b80b4 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/abort-states.json @@ -0,0 +1,20 @@ +{ + "adventurelog": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + }, + "adventurelog-frontend": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + }, + "adventurelog-postgres": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/compose-final.log b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/compose-final.log new file mode 100644 index 00000000..daa8a7b9 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/compose-final.log @@ -0,0 +1,102 @@ +adventurelog-frontend | Listening on http://0.0.0.0:3000 +adventurelog-postgres | The files belonging to this database system will be owned by user "postgres". +adventurelog-postgres | This user must also own the server process. +adventurelog-postgres | +adventurelog-postgres | The database cluster will be initialized with locale "en_US.utf8". +adventurelog-postgres | The default database encoding has accordingly been set to "UTF8". +adventurelog-postgres | The default text search configuration will be set to "english". +adventurelog-postgres | +adventurelog-postgres | Data page checksums are disabled. +adventurelog-postgres | +adventurelog-postgres | fixing permissions on existing directory /var/lib/postgresql/data ... ok +adventurelog-postgres | creating subdirectories ... ok +adventurelog-postgres | selecting dynamic shared memory implementation ... posix +adventurelog-postgres | selecting default max_connections ... 100 +adventurelog-postgres | selecting default shared_buffers ... 128MB +adventurelog-postgres | selecting default time zone ... Europe/Budapest +adventurelog-postgres | creating configuration files ... ok +adventurelog-postgres | running bootstrap script ... ok +adventurelog-postgres | sh: locale: not found +adventurelog-postgres | 2026-09-23 21:25:17.370 CEST [35] WARNING: no usable system locales were found +adventurelog-postgres | performing post-bootstrap initialization ... ok +adventurelog-postgres | syncing data to disk ... ok +adventurelog-postgres | +adventurelog-postgres | +adventurelog-postgres | Success. You can now start the database server using: +adventurelog-postgres | +adventurelog-postgres | pg_ctl -D /var/lib/postgresql/data -l logfile start +adventurelog-postgres | +adventurelog-postgres | initdb: warning: enabling "trust" authentication for local connections +adventurelog-postgres | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. +adventurelog-postgres | waiting for server to start....2026-09-23 21:25:18.984 CEST [41] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +adventurelog-postgres | 2026-09-23 21:25:18.990 CEST [41] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +adventurelog-postgres | 2026-09-23 21:25:19.003 CEST [44] LOG: database system was shut down at 2026-09-23 21:25:18 CEST +adventurelog-postgres | 2026-09-23 21:25:19.015 CEST [41] LOG: database system is ready to accept connections +adventurelog-postgres | done +adventurelog-postgres | server started +adventurelog-postgres | CREATE DATABASE +adventurelog-postgres | +adventurelog-postgres | +adventurelog-postgres | /usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/10_postgis.sh +adventurelog-postgres | CREATE DATABASE +adventurelog-postgres | Loading PostGIS extensions into template_postgis +adventurelog-postgres | CREATE EXTENSION +adventurelog-postgres | CREATE EXTENSION +adventurelog-postgres | You are now connected to database "template_postgis" as user "adventurelog". +adventurelog-postgres | DO +adventurelog-postgres | Loading PostGIS extensions into adventurelog +adventurelog-postgres | CREATE EXTENSION +adventurelog-postgres | CREATE EXTENSION +adventurelog-postgres | You are now connected to database "adventurelog" as user "adventurelog". +adventurelog-postgres | DO +adventurelog-postgres | +adventurelog-postgres | waiting for server to shut down....2026-09-23 21:25:24.162 CEST [41] LOG: received fast shutdown request +adventurelog-postgres | 2026-09-23 21:25:24.168 CEST [41] LOG: aborting any active transactions +adventurelog-postgres | 2026-09-23 21:25:24.172 CEST [41] LOG: background worker "logical replication launcher" (PID 47) exited with exit code 1 +adventurelog-postgres | 2026-09-23 21:25:24.172 CEST [42] LOG: shutting down +adventurelog-postgres | 2026-09-23 21:25:24.184 CEST [42] LOG: checkpoint starting: shutdown immediate +adventurelog-postgres | ..2026-09-23 21:25:26.474 CEST [42] LOG: checkpoint complete: wrote 4465 buffers (27.3%); 0 WAL file(s) added, 0 removed, 2 recycled; write=0.089 s, sync=2.133 s, total=2.302 s; sync files=963, longest=0.026 s, average=0.003 s; distance=34851 kB, estimate=34851 kB; lsn=0/36FFEE8, redo lsn=0/36FFEE8 +adventurelog-postgres | 2026-09-23 21:25:26.501 CEST [41] LOG: database system is shut down +adventurelog-postgres | done +adventurelog-postgres | server stopped +adventurelog-postgres | +adventurelog-postgres | PostgreSQL init process complete; ready for start up. +adventurelog-postgres | +adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 +adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: listening on IPv6 address "::", port 5432 +adventurelog-postgres | 2026-09-23 21:25:26.635 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +adventurelog-postgres | 2026-09-23 21:25:26.655 CEST [65] LOG: database system was shut down at 2026-09-23 21:25:26 CEST +adventurelog-postgres | 2026-09-23 21:25:26.670 CEST [1] LOG: database system is ready to accept connections +adventurelog-postgres | 2026-09-23 21:30:26.401 CEST [63] LOG: checkpoint starting: time +adventurelog-postgres | 2026-09-23 21:34:56.159 CEST [63] LOG: checkpoint complete: wrote 6010 buffers (36.7%); 0 WAL file(s) added, 0 removed, 8 recycled; write=269.655 s, sync=0.064 s, total=269.758 s; sync files=490, longest=0.013 s, average=0.001 s; distance=129719 kB, estimate=129719 kB; lsn=0/F215008, redo lsn=0/B5ADEC0 +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Latest data already imported. +adventurelog | █████╗ ██████╗ ██╗ ██╗███████╗███╗ ██╗████████╗██╗ ██╗██████╗ ███████╗██╗ ██████╗ ██████╗ +adventurelog | ██╔══██╗██╔══██╗██║ ██║██╔════╝████╗ ██║╚══██╔══╝██║ ██║██╔══██╗██╔════╝██║ ██╔═══██╗██╔════╝ +adventurelog | ███████║██║ ██║██║ ██║█████╗ ██╔██╗ ██║ ██║ ██║ ██║██████╔╝█████╗ ██║ ██║ ██║██║ ███╗ +adventurelog | ██╔══██║██║ ██║╚██╗ ██╔╝██╔══╝ ██║╚██╗██║ ██║ ██║ ██║██╔══██╗██╔══╝ ██║ ██║ ██║██║ ██║ +adventurelog | ██║ ██║██████╔╝ ╚████╔╝ ███████╗██║ ╚████║ ██║ ╚██████╔╝██║ ██║███████╗███████╗╚██████╔╝╚██████╔╝ +adventurelog | ╚═╝ ╚═╝╚═════╝ ╚═══╝ ╚══════╝╚═╝ ╚═══╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚══════╝ ╚═════╝ ╚═════╝ +adventurelog | “Go confidently in the direction of your dreams! Live the life you've imagined.” ― Henry David Thoreau +adventurelog | 2026-09-23 19:35:46,046 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. +adventurelog | 2026-09-23 19:35:46,050 INFO supervisord started with pid 1 +adventurelog | 2026-09-23 19:35:47,054 INFO spawned: 'gunicorn' with pid 23 +adventurelog | 2026-09-23 19:35:47,057 INFO spawned: 'memcached' with pid 24 +adventurelog | 2026-09-23 19:35:47,063 INFO spawned: 'nginx' with pid 25 +adventurelog | 2026-09-23 19:35:47,066 INFO spawned: 'sync_visited_regions' with pid 26 +adventurelog | [2026-09-23 19:35:47 +0000] [23] [INFO] Starting gunicorn 23.0.0 +adventurelog | [2026-09-23 19:35:47 +0000] [23] [INFO] Listening at: http://[::]:8000 (23) +adventurelog | [2026-09-23 19:35:47 +0000] [23] [INFO] Using worker: sync +adventurelog | [2026-09-23 19:35:47 +0000] [37] [INFO] Booting worker with pid: 37 +adventurelog | [2026-09-23 19:35:47 +0000] [38] [INFO] Booting worker with pid: 38 +adventurelog | 2026-09-23 19:35:49,009 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog | 2026-09-23 19:35:49,009 INFO success: memcached entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog | 2026-09-23 19:35:49,009 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog | 2026-09-23 19:35:49,009 INFO success: sync_visited_regions entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog | Starting periodic sync worker for midnight background jobs... +adventurelog | Next sync scheduled in 15850s (~4.40h) at UTC midnight diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/engine-state.json b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/engine-state.json new file mode 100644 index 00000000..ec747fa4 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/engine-state.json @@ -0,0 +1 @@ +null \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/files-after.json b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/files-after.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/files-after.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/files-before.json b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/files-before.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/files-before.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/migration-lines.txt b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/migration-lines.txt new file mode 100644 index 00000000..dc4c5e7b --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/migration-lines.txt @@ -0,0 +1,6 @@ +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | Applying adventures.0072_trail_wanderer_author_fields... OK +adventurelog | Applying adventures.0073_pointfield_geography... OK +adventurelog | Applying adventures.0074_contentimage_source_metadata... OK +adventurelog | Applying adventures.0075_make_visit_dates_timezone_aware... OK \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/run.log b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/run.log new file mode 100644 index 00000000..1756748f --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/run.log @@ -0,0 +1,16 @@ +[19:24:31] MV-adventurelog: deploying adventurelog at FROM {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.12.1', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1'} +[19:27:32] FROM settled=True in 124.8s :: {"adventurelog": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[19:27:32] fixture: the BOX walk's own (Django), through upgrade_boxport +[19:27:36] adventurelog: createsuperuser :: Superuser created successfully. +[19:27:39] adventurelog: seeded superuser drill3cc072 +[19:27:43] adventurelog: readback of the seeded account found=True +[19:27:43] C1 (seed reads back BEFORE): True +[19:27:44] MV-adventurelog: swapping to TO {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0'} +[19:28:25] TO up -d rc=0 +[19:35:29] TO settled=False in 423.9s :: {"adventurelog": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": "unhealthy", "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[19:35:29] migration lines observed: 6 +[19:35:34] adventurelog: readback of the seeded account found=True +[19:35:34] RESULT (seed reads back AFTER): True +[19:35:34] MV-adventurelog: ABORT — putting the FROM images back +[19:36:15] adventurelog: readback of the seeded account found=True +[19:36:15] ABORT: app came back in 31.4s; data present=True \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/to-full.log b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/to-full.log new file mode 100644 index 00000000..9b5d73e1 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/to-full.log @@ -0,0 +1,110 @@ +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | Applying adventures.0072_trail_wanderer_author_fields... OK +adventurelog | Applying adventures.0073_pointfield_geography... OK +adventurelog | Applying adventures.0074_contentimage_source_metadata... OK +adventurelog | Applying adventures.0075_make_visit_dates_timezone_aware... OK +adventurelog | Applying adventures.0076_normalize_all_day_visit_end_dates... OK +adventurelog | Applying billing.0001_initial... OK +adventurelog | Applying integrations.0007_wandererintegration_api_key_auth... OK +adventurelog | Applying integrations.0008_endurainintegration... OK +adventurelog | Applying integrations.0009_alter_endurainintegration_auth_method... OK +adventurelog | Applying worldtravel.0019_pointfield_coordinates... OK +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Latest data already imported. +adventurelog | █████╗ ██████╗ ██╗ ██╗███████╗███╗ ██╗████████╗██╗ ██╗██████╗ ███████╗██╗ ██████╗ ██████╗ +adventurelog | ██╔══██╗██╔══██╗██║ ██║██╔════╝████╗ ██║╚══██╔══╝██║ ██║██╔══██╗██╔════╝██║ ██╔═══██╗██╔════╝ +adventurelog | ███████║██║ ██║██║ ██║█████╗ ██╔██╗ ██║ ██║ ██║ ██║██████╔╝█████╗ ██║ ██║ ██║██║ ███╗ +adventurelog | ██╔══██║██║ ██║╚██╗ ██╔╝██╔══╝ ██║╚██╗██║ ██║ ██║ ██║██╔══██╗██╔══╝ ██║ ██║ ██║██║ ██║ +adventurelog | ██║ ██║██████╔╝ ╚████╔╝ ███████╗██║ ╚████║ ██║ ╚██████╔╝██║ ██║███████╗███████╗╚██████╔╝╚██████╔╝ +adventurelog | ╚═╝ ╚═╝╚═════╝ ╚═══╝ ╚══════╝╚═╝ ╚═══╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚══════╝ ╚═════╝ ╚═════╝ +adventurelog | “Go confidently in the direction of your dreams! Live the life you've imagined.” ― Henry David Thoreau +adventurelog | 2026-09-23 19:32:01,030 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. +adventurelog | 2026-09-23 19:32:01,032 INFO supervisord started with pid 1 +adventurelog | 2026-09-23 19:32:02,036 INFO spawned: 'cron' with pid 90 +adventurelog | 2026-09-23 19:32:02,040 INFO spawned: 'gunicorn' with pid 91 +adventurelog | 2026-09-23 19:32:02,044 INFO spawned: 'memcached' with pid 92 +adventurelog | 2026-09-23 19:32:02,047 INFO spawned: 'nginx' with pid 93 +adventurelog | [2026-09-23 19:32:02 +0000] [91] [INFO] Starting gunicorn 23.0.0 +adventurelog | [2026-09-23 19:32:02 +0000] [91] [INFO] Listening at: http://[::]:8000 (91) +adventurelog | [2026-09-23 19:32:02 +0000] [91] [INFO] Using worker: sync +adventurelog | [2026-09-23 19:32:02 +0000] [104] [INFO] Booting worker with pid: 104 +adventurelog | [2026-09-23 19:32:02 +0000] [105] [INFO] Booting worker with pid: 105 +adventurelog | 2026-09-23 19:32:03,706 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog | 2026-09-23 19:32:03,706 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog | 2026-09-23 19:32:03,706 INFO success: memcached entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog | 2026-09-23 19:32:03,706 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog-postgres | The files belonging to this database system will be owned by user "postgres". +adventurelog-postgres | This user must also own the server process. +adventurelog-postgres | +adventurelog-postgres | The database cluster will be initialized with locale "en_US.utf8". +adventurelog-postgres | The default database encoding has accordingly been set to "UTF8". +adventurelog-postgres | The default text search configuration will be set to "english". +adventurelog-postgres | +adventurelog-postgres | Data page checksums are disabled. +adventurelog-postgres | +adventurelog-postgres | fixing permissions on existing directory /var/lib/postgresql/data ... ok +adventurelog-postgres | creating subdirectories ... ok +adventurelog-postgres | selecting dynamic shared memory implementation ... posix +adventurelog-postgres | selecting default max_connections ... 100 +adventurelog-postgres | selecting default shared_buffers ... 128MB +adventurelog-postgres | selecting default time zone ... Europe/Budapest +adventurelog-postgres | creating configuration files ... ok +adventurelog-postgres | running bootstrap script ... ok +adventurelog-postgres | sh: locale: not found +adventurelog-postgres | 2026-09-23 21:25:17.370 CEST [35] WARNING: no usable system locales were found +adventurelog-postgres | performing post-bootstrap initialization ... ok +adventurelog-postgres | syncing data to disk ... ok +adventurelog-postgres | +adventurelog-postgres | +adventurelog-postgres | Success. You can now start the database server using: +adventurelog-postgres | +adventurelog-postgres | pg_ctl -D /var/lib/postgresql/data -l logfile start +adventurelog-postgres | +adventurelog-postgres | initdb: warning: enabling "trust" authentication for local connections +adventurelog-postgres | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. +adventurelog-postgres | waiting for server to start....2026-09-23 21:25:18.984 CEST [41] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +adventurelog-postgres | 2026-09-23 21:25:18.990 CEST [41] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +adventurelog-postgres | 2026-09-23 21:25:19.003 CEST [44] LOG: database system was shut down at 2026-09-23 21:25:18 CEST +adventurelog-postgres | 2026-09-23 21:25:19.015 CEST [41] LOG: database system is ready to accept connections +adventurelog-postgres | done +adventurelog-postgres | server started +adventurelog-postgres | CREATE DATABASE +adventurelog-postgres | +adventurelog-postgres | +adventurelog-postgres | /usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/10_postgis.sh +adventurelog-postgres | CREATE DATABASE +adventurelog-postgres | Loading PostGIS extensions into template_postgis +adventurelog-postgres | CREATE EXTENSION +adventurelog-postgres | CREATE EXTENSION +adventurelog-postgres | You are now connected to database "template_postgis" as user "adventurelog". +adventurelog-postgres | DO +adventurelog-postgres | Loading PostGIS extensions into adventurelog +adventurelog-postgres | CREATE EXTENSION +adventurelog-postgres | CREATE EXTENSION +adventurelog-postgres | You are now connected to database "adventurelog" as user "adventurelog". +adventurelog-frontend | Listening on http://0.0.0.0:3000 +adventurelog-postgres | DO +adventurelog-postgres | +adventurelog-postgres | waiting for server to shut down....2026-09-23 21:25:24.162 CEST [41] LOG: received fast shutdown request +adventurelog-postgres | 2026-09-23 21:25:24.168 CEST [41] LOG: aborting any active transactions +adventurelog-postgres | 2026-09-23 21:25:24.172 CEST [41] LOG: background worker "logical replication launcher" (PID 47) exited with exit code 1 +adventurelog-postgres | 2026-09-23 21:25:24.172 CEST [42] LOG: shutting down +adventurelog-postgres | 2026-09-23 21:25:24.184 CEST [42] LOG: checkpoint starting: shutdown immediate +adventurelog-postgres | ..2026-09-23 21:25:26.474 CEST [42] LOG: checkpoint complete: wrote 4465 buffers (27.3%); 0 WAL file(s) added, 0 removed, 2 recycled; write=0.089 s, sync=2.133 s, total=2.302 s; sync files=963, longest=0.026 s, average=0.003 s; distance=34851 kB, estimate=34851 kB; lsn=0/36FFEE8, redo lsn=0/36FFEE8 +adventurelog-postgres | 2026-09-23 21:25:26.501 CEST [41] LOG: database system is shut down +adventurelog-postgres | done +adventurelog-postgres | server stopped +adventurelog-postgres | +adventurelog-postgres | PostgreSQL init process complete; ready for start up. +adventurelog-postgres | +adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 +adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: listening on IPv6 address "::", port 5432 +adventurelog-postgres | 2026-09-23 21:25:26.635 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +adventurelog-postgres | 2026-09-23 21:25:26.655 CEST [65] LOG: database system was shut down at 2026-09-23 21:25:26 CEST +adventurelog-postgres | 2026-09-23 21:25:26.670 CEST [1] LOG: database system is ready to accept connections +adventurelog-postgres | 2026-09-23 21:30:26.401 CEST [63] LOG: checkpoint starting: time +adventurelog-postgres | 2026-09-23 21:34:56.159 CEST [63] LOG: checkpoint complete: wrote 6010 buffers (36.7%); 0 WAL file(s) added, 0 removed, 8 recycled; write=269.655 s, sync=0.064 s, total=269.758 s; sync files=490, longest=0.013 s, average=0.001 s; distance=129719 kB, estimate=129719 kB; lsn=0/F215008, redo lsn=0/B5ADEC0 diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/to-states.json b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/to-states.json new file mode 100644 index 00000000..5e43de90 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/to-states.json @@ -0,0 +1,20 @@ +{ + "adventurelog": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + }, + "adventurelog-frontend": { + "status": "running", + "health": "unhealthy", + "restarts": 0, + "exit": 0 + }, + "adventurelog-postgres": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/verdict.json b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/verdict.json new file mode 100644 index 00000000..633ee698 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench-override-nodejs/evidence/MV-adventurelog/verdict.json @@ -0,0 +1,31 @@ +{ + "harness_version": 3, + "edge": "MV-adventurelog", + "app": "adventurelog", + "note": "night 2026-09-23 within-a-major move: adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.13.0, adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0", + "from": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1" + }, + "to": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.13.0", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0" + }, + "verdict": "failed", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": false, + "migration_observed": "\u001b[2Kadventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel", + "abort": "starts-and-serves", + "abort_detail": null, + "engine_state_after": null, + "memory": null, + "marks": [], + "duration_s": 423.9, + "measured_at": "2026-09-23T19:36:15Z", + "evidence": "evidence/MV-adventurelog", + "files_changed": [], + "total_s": 703.4 +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/MV-adventurelog.log b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/MV-adventurelog.log index 58e5d854..32b59fd2 100644 --- a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/MV-adventurelog.log +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/MV-adventurelog.log @@ -1,19 +1,19 @@ -[19:24:31] MV-adventurelog: deploying adventurelog at FROM {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.12.1', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1'} -[19:27:32] FROM settled=True in 124.8s :: {"adventurelog": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} -[19:27:32] fixture: the BOX walk's own (Django), through upgrade_boxport -[19:27:36] adventurelog: createsuperuser :: Superuser created successfully. -[19:27:39] adventurelog: seeded superuser drill3cc072 -[19:27:43] adventurelog: readback of the seeded account found=True -[19:27:43] C1 (seed reads back BEFORE): True -[19:27:44] MV-adventurelog: swapping to TO {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0'} -[19:28:25] TO up -d rc=0 -[19:35:29] TO settled=False in 423.9s :: {"adventurelog": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": "unhealthy", "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} -[19:35:29] migration lines observed: 6 -[19:35:34] adventurelog: readback of the seeded account found=True -[19:35:34] RESULT (seed reads back AFTER): True -[19:35:34] MV-adventurelog: ABORT — putting the FROM images back -[19:36:15] adventurelog: readback of the seeded account found=True -[19:36:15] ABORT: app came back in 31.4s; data present=True +[20:39:13] MV-adventurelog: deploying adventurelog at FROM {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.12.1', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1'} +[20:40:17] FROM settled=True in 51.8s :: {"adventurelog": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": null, "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[20:40:17] fixture: the BOX walk's own (Django), through upgrade_boxport +[20:40:20] adventurelog: createsuperuser :: Superuser created successfully. +[20:40:22] adventurelog: seeded superuser drill92a66b +[20:40:26] adventurelog: readback of the seeded account found=True +[20:40:26] C1 (seed reads back BEFORE): True +[20:40:26] MV-adventurelog: swapping to TO {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0'} +[20:40:30] TO up -d rc=0 +[20:47:31] TO settled=False in 420.3s :: {"adventurelog": {"status": "running", "health": "starting", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": "unhealthy", "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[20:47:33] migration lines observed: 6 +[20:55:07] adventurelog: the app never served /admin/login/ +[20:55:07] RESULT (seed reads back AFTER): False +[20:55:07] MV-adventurelog: ABORT — putting the FROM images back +[20:55:52] adventurelog: readback of the seeded account found=True +[20:55:52] ABORT: app came back in 31.2s; data present=True { "harness_version": 3, "edge": "MV-adventurelog", @@ -31,7 +31,7 @@ }, "verdict": "failed", "seed_read_before": true, - "seed_read_after": true, + "seed_read_after": false, "healthy_after": false, "migration_observed": "\u001b[2Kadventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel", "abort": "starts-and-serves", @@ -39,9 +39,9 @@ "engine_state_after": null, "memory": null, "marks": [], - "duration_s": 423.9, - "measured_at": "2026-09-23T19:36:15Z", + "duration_s": 420.3, + "measured_at": "2026-09-23T20:55:52Z", "evidence": "evidence/MV-adventurelog", "files_changed": [], - "total_s": 703.4 + "total_s": 998.6 } diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/abort-states.json b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/abort-states.json index 359b80b4..b53c1e04 100644 --- a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/abort-states.json +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/abort-states.json @@ -7,7 +7,7 @@ }, "adventurelog-frontend": { "status": "running", - "health": "healthy", + "health": null, "restarts": 0, "exit": 0 }, diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/compose-final.log b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/compose-final.log index daa8a7b9..937376f9 100644 --- a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/compose-final.log +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/compose-final.log @@ -1,10 +1,11 @@ -adventurelog-frontend | Listening on http://0.0.0.0:3000 +adventurelog | PostgreSQL is up - continuing... +adventurelog-frontend | Listening on http://0.0.0.0:3000 adventurelog-postgres | The files belonging to this database system will be owned by user "postgres". adventurelog-postgres | This user must also own the server process. adventurelog-postgres | adventurelog-postgres | The database cluster will be initialized with locale "en_US.utf8". adventurelog-postgres | The default database encoding has accordingly been set to "UTF8". -adventurelog-postgres | The default text search configuration will be set to "english". +adventurelog-postgres | The default text search configuration will be set to "english". adventurelog-postgres | adventurelog-postgres | Data page checksums are disabled. adventurelog-postgres | @@ -14,26 +15,65 @@ adventurelog-postgres | selecting default max_connections ... 100 adventurelog-postgres | selecting default shared_buffers ... 128MB adventurelog-postgres | selecting default time zone ... Europe/Budapest +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... adventurelog-postgres | creating configuration files ... ok adventurelog-postgres | running bootstrap script ... ok adventurelog-postgres | sh: locale: not found -adventurelog-postgres | 2026-09-23 21:25:17.370 CEST [35] WARNING: no usable system locales were found +adventurelog-postgres | 2026-09-23 22:39:15.062 CEST [35] WARNING: no usable system locales were found adventurelog-postgres | performing post-bootstrap initialization ... ok +adventurelog-postgres | initdb: warning: enabling "trust" authentication for local connections +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog-postgres | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. +adventurelog | Flag for af already exists adventurelog-postgres | syncing data to disk ... ok +adventurelog | Flag for ax already exists adventurelog-postgres | adventurelog-postgres | adventurelog-postgres | Success. You can now start the database server using: adventurelog-postgres | adventurelog-postgres | pg_ctl -D /var/lib/postgresql/data -l logfile start adventurelog-postgres | -adventurelog-postgres | initdb: warning: enabling "trust" authentication for local connections -adventurelog-postgres | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. -adventurelog-postgres | waiting for server to start....2026-09-23 21:25:18.984 CEST [41] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit -adventurelog-postgres | 2026-09-23 21:25:18.990 CEST [41] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" -adventurelog-postgres | 2026-09-23 21:25:19.003 CEST [44] LOG: database system was shut down at 2026-09-23 21:25:18 CEST -adventurelog-postgres | 2026-09-23 21:25:19.015 CEST [41] LOG: database system is ready to accept connections -adventurelog-postgres | done -adventurelog-postgres | server started +adventurelog-postgres | waiting for server to start....2026-09-23 22:39:16.298 CEST [41] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +adventurelog-postgres | 2026-09-23 22:39:16.302 CEST [41] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +adventurelog-postgres | 2026-09-23 22:39:16.313 CEST [44] LOG: database system was shut down at 2026-09-23 22:39:15 CEST +adventurelog-postgres | 2026-09-23 22:39:16.322 CEST [41] LOG: database system is ready to accept connections +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog-postgres | done +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog-postgres | server started adventurelog-postgres | CREATE DATABASE adventurelog-postgres | adventurelog-postgres | @@ -42,40 +82,115 @@ adventurelog-postgres | Loading PostGIS extensions into template_postgis adventurelog-postgres | CREATE EXTENSION adventurelog-postgres | CREATE EXTENSION +adventurelog | Flag for bn already exists adventurelog-postgres | You are now connected to database "template_postgis" as user "adventurelog". adventurelog-postgres | DO +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists adventurelog-postgres | Loading PostGIS extensions into adventurelog adventurelog-postgres | CREATE EXTENSION adventurelog-postgres | CREATE EXTENSION adventurelog-postgres | You are now connected to database "adventurelog" as user "adventurelog". -adventurelog-postgres | DO +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog-postgres | DO adventurelog-postgres | -adventurelog-postgres | waiting for server to shut down....2026-09-23 21:25:24.162 CEST [41] LOG: received fast shutdown request -adventurelog-postgres | 2026-09-23 21:25:24.168 CEST [41] LOG: aborting any active transactions -adventurelog-postgres | 2026-09-23 21:25:24.172 CEST [41] LOG: background worker "logical replication launcher" (PID 47) exited with exit code 1 -adventurelog-postgres | 2026-09-23 21:25:24.172 CEST [42] LOG: shutting down -adventurelog-postgres | 2026-09-23 21:25:24.184 CEST [42] LOG: checkpoint starting: shutdown immediate -adventurelog-postgres | ..2026-09-23 21:25:26.474 CEST [42] LOG: checkpoint complete: wrote 4465 buffers (27.3%); 0 WAL file(s) added, 0 removed, 2 recycled; write=0.089 s, sync=2.133 s, total=2.302 s; sync files=963, longest=0.026 s, average=0.003 s; distance=34851 kB, estimate=34851 kB; lsn=0/36FFEE8, redo lsn=0/36FFEE8 -adventurelog-postgres | 2026-09-23 21:25:26.501 CEST [41] LOG: database system is shut down +adventurelog-postgres | waiting for server to shut down...2026-09-23 22:39:21.144 CEST [41] LOG: received fast shutdown request +adventurelog-postgres | .2026-09-23 22:39:21.148 CEST [41] LOG: aborting any active transactions +adventurelog-postgres | 2026-09-23 22:39:21.154 CEST [41] LOG: background worker "logical replication launcher" (PID 47) exited with exit code 1 +adventurelog-postgres | 2026-09-23 22:39:21.155 CEST [42] LOG: shutting down +adventurelog-postgres | 2026-09-23 22:39:21.158 CEST [42] LOG: checkpoint starting: shutdown immediate +adventurelog-postgres | 2026-09-23 22:39:21.841 CEST [42] LOG: checkpoint complete: wrote 4465 buffers (27.3%); 0 WAL file(s) added, 0 removed, 2 recycled; write=0.062 s, sync=0.603 s, total=0.686 s; sync files=963, longest=0.473 s, average=0.001 s; distance=34851 kB, estimate=34851 kB; lsn=0/36FFEE8, redo lsn=0/36FFEE8 +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog-postgres | 2026-09-23 22:39:21.867 CEST [41] LOG: database system is shut down adventurelog-postgres | done adventurelog-postgres | server stopped adventurelog-postgres | adventurelog-postgres | PostgreSQL init process complete; ready for start up. adventurelog-postgres | -adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit -adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 -adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: listening on IPv6 address "::", port 5432 -adventurelog-postgres | 2026-09-23 21:25:26.635 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" -adventurelog-postgres | 2026-09-23 21:25:26.655 CEST [65] LOG: database system was shut down at 2026-09-23 21:25:26 CEST -adventurelog-postgres | 2026-09-23 21:25:26.670 CEST [1] LOG: database system is ready to accept connections -adventurelog-postgres | 2026-09-23 21:30:26.401 CEST [63] LOG: checkpoint starting: time -adventurelog-postgres | 2026-09-23 21:34:56.159 CEST [63] LOG: checkpoint complete: wrote 6010 buffers (36.7%); 0 WAL file(s) added, 0 removed, 8 recycled; write=269.655 s, sync=0.064 s, total=269.758 s; sync files=490, longest=0.013 s, average=0.001 s; distance=129719 kB, estimate=129719 kB; lsn=0/F215008, redo lsn=0/B5ADEC0 -adventurelog | PostgreSQL is up - continuing... -adventurelog | Operations to perform: -adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel -adventurelog | Running migrations: -adventurelog | No migrations to apply. -adventurelog | Latest data already imported. +adventurelog-postgres | 2026-09-23 22:39:21.982 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +adventurelog-postgres | 2026-09-23 22:39:21.982 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 +adventurelog-postgres | 2026-09-23 22:39:21.982 CEST [1] LOG: listening on IPv6 address "::", port 5432 +adventurelog-postgres | 2026-09-23 22:39:21.989 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +adventurelog-postgres | 2026-09-23 22:39:22.001 CEST [65] LOG: database system was shut down at 2026-09-23 22:39:21 CEST +adventurelog-postgres | 2026-09-23 22:39:22.012 CEST [1] LOG: database system is ready to accept connections +adventurelog-postgres | 2026-09-23 22:44:22.101 CEST [63] LOG: checkpoint starting: time +adventurelog-postgres | 2026-09-23 22:45:14.598 CEST [63] LOG: checkpoint complete: wrote 526 buffers (3.2%); 0 WAL file(s) added, 0 removed, 1 recycled; write=52.413 s, sync=0.060 s, total=52.498 s; sync files=484, longest=0.009 s, average=0.001 s; distance=11716 kB, estimate=11716 kB; lsn=0/4271090, redo lsn=0/4271058 +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 123, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 148, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content adventurelog | █████╗ ██████╗ ██╗ ██╗███████╗███╗ ██╗████████╗██╗ ██╗██████╗ ███████╗██╗ ██████╗ ██████╗ adventurelog | ██╔══██╗██╔══██╗██║ ██║██╔════╝████╗ ██║╚══██╔══╝██║ ██║██╔══██╗██╔════╝██║ ██╔═══██╗██╔════╝ adventurelog | ███████║██║ ██║██║ ██║█████╗ ██╔██╗ ██║ ██║ ██║ ██║██████╔╝█████╗ ██║ ██║ ██║██║ ███╗ @@ -83,20 +198,20 @@ adventurelog | PostgreSQL is up - continuing... adventurelog | ██║ ██║██████╔╝ ╚████╔╝ ███████╗██║ ╚████║ ██║ ╚██████╔╝██║ ██║███████╗███████╗╚██████╔╝╚██████╔╝ adventurelog | ╚═╝ ╚═╝╚═════╝ ╚═══╝ ╚══════╝╚═╝ ╚═══╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚══════╝ ╚═════╝ ╚═════╝ adventurelog | “Go confidently in the direction of your dreams! Live the life you've imagined.” ― Henry David Thoreau -adventurelog | 2026-09-23 19:35:46,046 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. -adventurelog | 2026-09-23 19:35:46,050 INFO supervisord started with pid 1 -adventurelog | 2026-09-23 19:35:47,054 INFO spawned: 'gunicorn' with pid 23 -adventurelog | 2026-09-23 19:35:47,057 INFO spawned: 'memcached' with pid 24 -adventurelog | 2026-09-23 19:35:47,063 INFO spawned: 'nginx' with pid 25 -adventurelog | 2026-09-23 19:35:47,066 INFO spawned: 'sync_visited_regions' with pid 26 -adventurelog | [2026-09-23 19:35:47 +0000] [23] [INFO] Starting gunicorn 23.0.0 -adventurelog | [2026-09-23 19:35:47 +0000] [23] [INFO] Listening at: http://[::]:8000 (23) -adventurelog | [2026-09-23 19:35:47 +0000] [23] [INFO] Using worker: sync -adventurelog | [2026-09-23 19:35:47 +0000] [37] [INFO] Booting worker with pid: 37 -adventurelog | [2026-09-23 19:35:47 +0000] [38] [INFO] Booting worker with pid: 38 -adventurelog | 2026-09-23 19:35:49,009 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) -adventurelog | 2026-09-23 19:35:49,009 INFO success: memcached entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) -adventurelog | 2026-09-23 19:35:49,009 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) -adventurelog | 2026-09-23 19:35:49,009 INFO success: sync_visited_regions entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog | 2026-09-23 20:55:27,101 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. +adventurelog | 2026-09-23 20:55:27,104 INFO supervisord started with pid 1 +adventurelog | 2026-09-23 20:55:28,107 INFO spawned: 'gunicorn' with pid 23 +adventurelog | 2026-09-23 20:55:28,109 INFO spawned: 'memcached' with pid 24 +adventurelog | 2026-09-23 20:55:28,112 INFO spawned: 'nginx' with pid 25 +adventurelog | 2026-09-23 20:55:28,114 INFO spawned: 'sync_visited_regions' with pid 26 +adventurelog | [2026-09-23 20:55:28 +0000] [23] [INFO] Starting gunicorn 23.0.0 +adventurelog | [2026-09-23 20:55:28 +0000] [23] [INFO] Listening at: http://[::]:8000 (23) +adventurelog | [2026-09-23 20:55:28 +0000] [23] [INFO] Using worker: sync +adventurelog | [2026-09-23 20:55:28 +0000] [37] [INFO] Booting worker with pid: 37 +adventurelog | [2026-09-23 20:55:28 +0000] [38] [INFO] Booting worker with pid: 38 +adventurelog | 2026-09-23 20:55:29,853 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog | 2026-09-23 20:55:29,853 INFO success: memcached entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog | 2026-09-23 20:55:29,853 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog | 2026-09-23 20:55:29,853 INFO success: sync_visited_regions entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) adventurelog | Starting periodic sync worker for midnight background jobs... -adventurelog | Next sync scheduled in 15850s (~4.40h) at UTC midnight +adventurelog | Next sync scheduled in 11069s (~3.07h) at UTC midnight diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/run.log b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/run.log index 1756748f..ab7970f8 100644 --- a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/run.log +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/run.log @@ -1,16 +1,16 @@ -[19:24:31] MV-adventurelog: deploying adventurelog at FROM {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.12.1', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1'} -[19:27:32] FROM settled=True in 124.8s :: {"adventurelog": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} -[19:27:32] fixture: the BOX walk's own (Django), through upgrade_boxport -[19:27:36] adventurelog: createsuperuser :: Superuser created successfully. -[19:27:39] adventurelog: seeded superuser drill3cc072 -[19:27:43] adventurelog: readback of the seeded account found=True -[19:27:43] C1 (seed reads back BEFORE): True -[19:27:44] MV-adventurelog: swapping to TO {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0'} -[19:28:25] TO up -d rc=0 -[19:35:29] TO settled=False in 423.9s :: {"adventurelog": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": "unhealthy", "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} -[19:35:29] migration lines observed: 6 -[19:35:34] adventurelog: readback of the seeded account found=True -[19:35:34] RESULT (seed reads back AFTER): True -[19:35:34] MV-adventurelog: ABORT — putting the FROM images back -[19:36:15] adventurelog: readback of the seeded account found=True -[19:36:15] ABORT: app came back in 31.4s; data present=True \ No newline at end of file +[20:39:13] MV-adventurelog: deploying adventurelog at FROM {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.12.1', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1'} +[20:40:17] FROM settled=True in 51.8s :: {"adventurelog": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": null, "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[20:40:17] fixture: the BOX walk's own (Django), through upgrade_boxport +[20:40:20] adventurelog: createsuperuser :: Superuser created successfully. +[20:40:22] adventurelog: seeded superuser drill92a66b +[20:40:26] adventurelog: readback of the seeded account found=True +[20:40:26] C1 (seed reads back BEFORE): True +[20:40:26] MV-adventurelog: swapping to TO {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0'} +[20:40:30] TO up -d rc=0 +[20:47:31] TO settled=False in 420.3s :: {"adventurelog": {"status": "running", "health": "starting", "restarts": 0, "exit": 0}, "adventurelog-frontend": {"status": "running", "health": "unhealthy", "restarts": 0, "exit": 0}, "adventurelog-postgres": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[20:47:33] migration lines observed: 6 +[20:55:07] adventurelog: the app never served /admin/login/ +[20:55:07] RESULT (seed reads back AFTER): False +[20:55:07] MV-adventurelog: ABORT — putting the FROM images back +[20:55:52] adventurelog: readback of the seeded account found=True +[20:55:52] ABORT: app came back in 31.2s; data present=True \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/to-full.log b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/to-full.log index 9b5d73e1..18b43e0b 100644 --- a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/to-full.log +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/to-full.log @@ -1,41 +1,4406 @@ -adventurelog | PostgreSQL is up - continuing... -adventurelog | Operations to perform: -adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel -adventurelog | Running migrations: -adventurelog | Applying adventures.0072_trail_wanderer_author_fields... OK -adventurelog | Applying adventures.0073_pointfield_geography... OK -adventurelog | Applying adventures.0074_contentimage_source_metadata... OK -adventurelog | Applying adventures.0075_make_visit_dates_timezone_aware... OK -adventurelog | Applying adventures.0076_normalize_all_day_visit_end_dates... OK -adventurelog | Applying billing.0001_initial... OK -adventurelog | Applying integrations.0007_wandererintegration_api_key_auth... OK -adventurelog | Applying integrations.0008_endurainintegration... OK -adventurelog | Applying integrations.0009_alter_endurainintegration_auth_method... OK -adventurelog | Applying worldtravel.0019_pointfield_coordinates... OK -adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... -adventurelog | Latest data already imported. -adventurelog | █████╗ ██████╗ ██╗ ██╗███████╗███╗ ██╗████████╗██╗ ██╗██████╗ ███████╗██╗ ██████╗ ██████╗ -adventurelog | ██╔══██╗██╔══██╗██║ ██║██╔════╝████╗ ██║╚══██╔══╝██║ ██║██╔══██╗██╔════╝██║ ██╔═══██╗██╔════╝ -adventurelog | ███████║██║ ██║██║ ██║█████╗ ██╔██╗ ██║ ██║ ██║ ██║██████╔╝█████╗ ██║ ██║ ██║██║ ███╗ -adventurelog | ██╔══██║██║ ██║╚██╗ ██╔╝██╔══╝ ██║╚██╗██║ ██║ ██║ ██║██╔══██╗██╔══╝ ██║ ██║ ██║██║ ██║ -adventurelog | ██║ ██║██████╔╝ ╚████╔╝ ███████╗██║ ╚████║ ██║ ╚██████╔╝██║ ██║███████╗███████╗╚██████╔╝╚██████╔╝ -adventurelog | ╚═╝ ╚═╝╚═════╝ ╚═══╝ ╚══════╝╚═╝ ╚═══╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝╚══════╝ ╚═════╝ ╚═════╝ -adventurelog | “Go confidently in the direction of your dreams! Live the life you've imagined.” ― Henry David Thoreau -adventurelog | 2026-09-23 19:32:01,030 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. -adventurelog | 2026-09-23 19:32:01,032 INFO supervisord started with pid 1 -adventurelog | 2026-09-23 19:32:02,036 INFO spawned: 'cron' with pid 90 -adventurelog | 2026-09-23 19:32:02,040 INFO spawned: 'gunicorn' with pid 91 -adventurelog | 2026-09-23 19:32:02,044 INFO spawned: 'memcached' with pid 92 -adventurelog | 2026-09-23 19:32:02,047 INFO spawned: 'nginx' with pid 93 -adventurelog | [2026-09-23 19:32:02 +0000] [91] [INFO] Starting gunicorn 23.0.0 -adventurelog | [2026-09-23 19:32:02 +0000] [91] [INFO] Listening at: http://[::]:8000 (91) -adventurelog | [2026-09-23 19:32:02 +0000] [91] [INFO] Using worker: sync -adventurelog | [2026-09-23 19:32:02 +0000] [104] [INFO] Booting worker with pid: 104 -adventurelog | [2026-09-23 19:32:02 +0000] [105] [INFO] Booting worker with pid: 105 -adventurelog | 2026-09-23 19:32:03,706 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) -adventurelog | 2026-09-23 19:32:03,706 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) -adventurelog | 2026-09-23 19:32:03,706 INFO success: memcached entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) -adventurelog | 2026-09-23 19:32:03,706 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) +adventurelog-frontend | Listening on http://0.0.0.0:3000 +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | Applying adventures.0072_trail_wanderer_author_fields... OK +adventurelog | Applying adventures.0073_pointfield_geography... OK +adventurelog | Applying adventures.0074_contentimage_source_metadata... OK +adventurelog | Applying adventures.0075_make_visit_dates_timezone_aware... OK +adventurelog | Applying adventures.0076_normalize_all_day_visit_end_dates... OK +adventurelog | Applying billing.0001_initial... OK +adventurelog | Applying integrations.0007_wandererintegration_api_key_auth... OK +adventurelog | Applying integrations.0008_endurainintegration... OK +adventurelog | Applying integrations.0009_alter_endurainintegration_auth_method... OK +adventurelog | Applying worldtravel.0019_pointfield_coordinates... OK +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... +adventurelog | Operations to perform: +adventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel +adventurelog | Running migrations: +adventurelog | No migrations to apply. +adventurelog | Running download-countries (first boot may take several minutes; ensure ~2GB RAM is available)... +adventurelog | Some data is missing. Re-importing all data. +adventurelog | Starting ultra-memory-efficient import process... +adventurelog | Step 1: Parsing JSON and storing in temporary database... +adventurelog | Flag for af already exists +adventurelog | Flag for ax already exists +adventurelog | Flag for al already exists +adventurelog | Flag for dz already exists +adventurelog | Flag for as already exists +adventurelog | Flag for ad already exists +adventurelog | Flag for ao already exists +adventurelog | Flag for ai already exists +adventurelog | Flag for aq already exists +adventurelog | Flag for ag already exists +adventurelog | Flag for ar already exists +adventurelog | Flag for am already exists +adventurelog | Flag for aw already exists +adventurelog | Flag for au already exists +adventurelog | Flag for at already exists +adventurelog | Flag for az already exists +adventurelog | Flag for bh already exists +adventurelog | Flag for bd already exists +adventurelog | Flag for bb already exists +adventurelog | Flag for by already exists +adventurelog | Flag for be already exists +adventurelog | Flag for bz already exists +adventurelog | Flag for bj already exists +adventurelog | Flag for bm already exists +adventurelog | Flag for bt already exists +adventurelog | Flag for bo already exists +adventurelog | Flag for bq already exists +adventurelog | Flag for ba already exists +adventurelog | Flag for bw already exists +adventurelog | Flag for bv already exists +adventurelog | Flag for br already exists +adventurelog | Flag for io already exists +adventurelog | Flag for bn already exists +adventurelog | Flag for bg already exists +adventurelog | Flag for bf already exists +adventurelog | Flag for bi already exists +adventurelog | Flag for kh already exists +adventurelog | Flag for cm already exists +adventurelog | Flag for ca already exists +adventurelog | Flag for cv already exists +adventurelog | Flag for ky already exists +adventurelog | Flag for cf already exists +adventurelog | Flag for td already exists +adventurelog | Flag for cl already exists +adventurelog | Flag for cn already exists +adventurelog | Flag for cx already exists +adventurelog | Flag for cc already exists +adventurelog | Flag for co already exists +adventurelog | Flag for km already exists +adventurelog | Flag for cg already exists +adventurelog | Flag for ck already exists +adventurelog | Flag for cr already exists +adventurelog | Flag for hr already exists +adventurelog | Flag for cu already exists +adventurelog | Flag for cw already exists +adventurelog | Flag for cy already exists +adventurelog | Flag for cz already exists +adventurelog | Flag for cd already exists +adventurelog | Flag for dk already exists +adventurelog | Flag for dj already exists +adventurelog | Flag for dm already exists +adventurelog | Flag for do already exists +adventurelog | Flag for ec already exists +adventurelog | Flag for eg already exists +adventurelog | Flag for sv already exists +adventurelog | Flag for gq already exists +adventurelog | Flag for er already exists +adventurelog | Flag for ee already exists +adventurelog | Flag for sz already exists +adventurelog | Flag for et already exists +adventurelog | Flag for fk already exists +adventurelog | Flag for fo already exists +adventurelog | Flag for fj already exists +adventurelog | Flag for fi already exists +adventurelog | Flag for fr already exists +adventurelog | Flag for gf already exists +adventurelog | Flag for pf already exists +adventurelog | Flag for tf already exists +adventurelog | Flag for ga already exists +adventurelog | Flag for ge already exists +adventurelog | Traceback (most recent call last): +adventurelog | File "/code/manage.py", line 10, in +adventurelog | execute_from_command_line(sys.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line +adventurelog | utility.execute() +adventurelog | ~~~~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute +adventurelog | self.fetch_command(subcommand).run_from_argv(self.argv) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 420, in run_from_argv +adventurelog | self.execute(*args, **cmd_options) +adventurelog | ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/django/core/management/base.py", line 464, in execute +adventurelog | output = self.handle(*args, **options) +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 122, in handle +adventurelog | self._parse_and_store_temp(countries_json_path, temp_conn) +adventurelog | ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +adventurelog | File "/code/worldtravel/management/commands/download-countries.py", line 147, in _parse_and_store_temp +adventurelog | for country in parser: +adventurelog | ^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/utils.py", line 55, in coros2gen +adventurelog | f.send(value) +adventurelog | ~~~~~~^^^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 46, in utf8_encoder +adventurelog | target.close() +adventurelog | ~~~~~~~~~~~~^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 116, in Lexer +adventurelog | target.send(EOF) +adventurelog | ~~~~~~~~~~~^^^^^ +adventurelog | File "/usr/local/lib/python3.13/site-packages/ijson/backends/python.py", line 161, in parse_value +adventurelog | raise common.IncompleteJSONError('Incomplete JSON content') +adventurelog | ijson.common.IncompleteJSONError: Incomplete JSON content +adventurelog | PostgreSQL is up - continuing... adventurelog-postgres | The files belonging to this database system will be owned by user "postgres". adventurelog-postgres | This user must also own the server process. adventurelog-postgres | @@ -54,8 +4419,10 @@ adventurelog-postgres | creating configuration files ... ok adventurelog-postgres | running bootstrap script ... ok adventurelog-postgres | sh: locale: not found -adventurelog-postgres | 2026-09-23 21:25:17.370 CEST [35] WARNING: no usable system locales were found +adventurelog-postgres | 2026-09-23 22:39:15.062 CEST [35] WARNING: no usable system locales were found adventurelog-postgres | performing post-bootstrap initialization ... ok +adventurelog-postgres | initdb: warning: enabling "trust" authentication for local connections +adventurelog-postgres | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. adventurelog-postgres | syncing data to disk ... ok adventurelog-postgres | adventurelog-postgres | @@ -63,12 +4430,10 @@ adventurelog-postgres | adventurelog-postgres | pg_ctl -D /var/lib/postgresql/data -l logfile start adventurelog-postgres | -adventurelog-postgres | initdb: warning: enabling "trust" authentication for local connections -adventurelog-postgres | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. -adventurelog-postgres | waiting for server to start....2026-09-23 21:25:18.984 CEST [41] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit -adventurelog-postgres | 2026-09-23 21:25:18.990 CEST [41] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" -adventurelog-postgres | 2026-09-23 21:25:19.003 CEST [44] LOG: database system was shut down at 2026-09-23 21:25:18 CEST -adventurelog-postgres | 2026-09-23 21:25:19.015 CEST [41] LOG: database system is ready to accept connections +adventurelog-postgres | waiting for server to start....2026-09-23 22:39:16.298 CEST [41] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +adventurelog-postgres | 2026-09-23 22:39:16.302 CEST [41] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +adventurelog-postgres | 2026-09-23 22:39:16.313 CEST [44] LOG: database system was shut down at 2026-09-23 22:39:15 CEST +adventurelog-postgres | 2026-09-23 22:39:16.322 CEST [41] LOG: database system is ready to accept connections adventurelog-postgres | done adventurelog-postgres | server started adventurelog-postgres | CREATE DATABASE @@ -85,26 +4450,25 @@ adventurelog-postgres | CREATE EXTENSION adventurelog-postgres | CREATE EXTENSION adventurelog-postgres | You are now connected to database "adventurelog" as user "adventurelog". -adventurelog-frontend | Listening on http://0.0.0.0:3000 adventurelog-postgres | DO adventurelog-postgres | -adventurelog-postgres | waiting for server to shut down....2026-09-23 21:25:24.162 CEST [41] LOG: received fast shutdown request -adventurelog-postgres | 2026-09-23 21:25:24.168 CEST [41] LOG: aborting any active transactions -adventurelog-postgres | 2026-09-23 21:25:24.172 CEST [41] LOG: background worker "logical replication launcher" (PID 47) exited with exit code 1 -adventurelog-postgres | 2026-09-23 21:25:24.172 CEST [42] LOG: shutting down -adventurelog-postgres | 2026-09-23 21:25:24.184 CEST [42] LOG: checkpoint starting: shutdown immediate -adventurelog-postgres | ..2026-09-23 21:25:26.474 CEST [42] LOG: checkpoint complete: wrote 4465 buffers (27.3%); 0 WAL file(s) added, 0 removed, 2 recycled; write=0.089 s, sync=2.133 s, total=2.302 s; sync files=963, longest=0.026 s, average=0.003 s; distance=34851 kB, estimate=34851 kB; lsn=0/36FFEE8, redo lsn=0/36FFEE8 -adventurelog-postgres | 2026-09-23 21:25:26.501 CEST [41] LOG: database system is shut down +adventurelog-postgres | waiting for server to shut down...2026-09-23 22:39:21.144 CEST [41] LOG: received fast shutdown request +adventurelog-postgres | .2026-09-23 22:39:21.148 CEST [41] LOG: aborting any active transactions +adventurelog-postgres | 2026-09-23 22:39:21.154 CEST [41] LOG: background worker "logical replication launcher" (PID 47) exited with exit code 1 +adventurelog-postgres | 2026-09-23 22:39:21.155 CEST [42] LOG: shutting down +adventurelog-postgres | 2026-09-23 22:39:21.158 CEST [42] LOG: checkpoint starting: shutdown immediate +adventurelog-postgres | 2026-09-23 22:39:21.841 CEST [42] LOG: checkpoint complete: wrote 4465 buffers (27.3%); 0 WAL file(s) added, 0 removed, 2 recycled; write=0.062 s, sync=0.603 s, total=0.686 s; sync files=963, longest=0.473 s, average=0.001 s; distance=34851 kB, estimate=34851 kB; lsn=0/36FFEE8, redo lsn=0/36FFEE8 +adventurelog-postgres | 2026-09-23 22:39:21.867 CEST [41] LOG: database system is shut down adventurelog-postgres | done adventurelog-postgres | server stopped adventurelog-postgres | adventurelog-postgres | PostgreSQL init process complete; ready for start up. adventurelog-postgres | -adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit -adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 -adventurelog-postgres | 2026-09-23 21:25:26.619 CEST [1] LOG: listening on IPv6 address "::", port 5432 -adventurelog-postgres | 2026-09-23 21:25:26.635 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" -adventurelog-postgres | 2026-09-23 21:25:26.655 CEST [65] LOG: database system was shut down at 2026-09-23 21:25:26 CEST -adventurelog-postgres | 2026-09-23 21:25:26.670 CEST [1] LOG: database system is ready to accept connections -adventurelog-postgres | 2026-09-23 21:30:26.401 CEST [63] LOG: checkpoint starting: time -adventurelog-postgres | 2026-09-23 21:34:56.159 CEST [63] LOG: checkpoint complete: wrote 6010 buffers (36.7%); 0 WAL file(s) added, 0 removed, 8 recycled; write=269.655 s, sync=0.064 s, total=269.758 s; sync files=490, longest=0.013 s, average=0.001 s; distance=129719 kB, estimate=129719 kB; lsn=0/F215008, redo lsn=0/B5ADEC0 +adventurelog-postgres | 2026-09-23 22:39:21.982 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +adventurelog-postgres | 2026-09-23 22:39:21.982 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 +adventurelog-postgres | 2026-09-23 22:39:21.982 CEST [1] LOG: listening on IPv6 address "::", port 5432 +adventurelog-postgres | 2026-09-23 22:39:21.989 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +adventurelog-postgres | 2026-09-23 22:39:22.001 CEST [65] LOG: database system was shut down at 2026-09-23 22:39:21 CEST +adventurelog-postgres | 2026-09-23 22:39:22.012 CEST [1] LOG: database system is ready to accept connections +adventurelog-postgres | 2026-09-23 22:44:22.101 CEST [63] LOG: checkpoint starting: time +adventurelog-postgres | 2026-09-23 22:45:14.598 CEST [63] LOG: checkpoint complete: wrote 526 buffers (3.2%); 0 WAL file(s) added, 0 removed, 1 recycled; write=52.413 s, sync=0.060 s, total=52.498 s; sync files=484, longest=0.009 s, average=0.001 s; distance=11716 kB, estimate=11716 kB; lsn=0/4271090, redo lsn=0/4271058 diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/to-states.json b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/to-states.json index 5e43de90..dc6ff681 100644 --- a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/to-states.json +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/to-states.json @@ -1,7 +1,7 @@ { "adventurelog": { "status": "running", - "health": "healthy", + "health": "starting", "restarts": 0, "exit": 0 }, diff --git a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/verdict.json b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/verdict.json index 633ee698..e2e00fef 100644 --- a/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/verdict.json +++ b/documentation/audits/night-2026-09-23/apps/adventurelog/bench/evidence/MV-adventurelog/verdict.json @@ -15,7 +15,7 @@ }, "verdict": "failed", "seed_read_before": true, - "seed_read_after": true, + "seed_read_after": false, "healthy_after": false, "migration_observed": "\u001b[2Kadventurelog | Apply all migrations: account, admin, adventures, auth, authtoken, billing, contenttypes, integrations, invitations, mfa, sessions, sites, socialaccount, users, worldtravel", "abort": "starts-and-serves", @@ -23,9 +23,9 @@ "engine_state_after": null, "memory": null, "marks": [], - "duration_s": 423.9, - "measured_at": "2026-09-23T19:36:15Z", + "duration_s": 420.3, + "measured_at": "2026-09-23T20:55:52Z", "evidence": "evidence/MV-adventurelog", "files_changed": [], - "total_s": 703.4 + "total_s": 998.6 } \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/MV-nextcloud.log b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/MV-nextcloud.log new file mode 100644 index 00000000..8ef2fbb9 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/MV-nextcloud.log @@ -0,0 +1,145 @@ +[20:01:30] MV-nextcloud: deploying nextcloud at FROM {'nextcloud': 'nextcloud:34.0.1-apache', 'nextcloud-db': 'mariadb:12.3', 'nextcloud-redis': 'redis:7-alpine'} +[20:02:13] FROM settled=True in 31.1s :: {"nextcloud": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "nextcloud-db": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "nextcloud-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[20:02:13] fixture: the BOX walk's own (Nextcloud), through upgrade_boxport +[20:02:15] nextcloud: occ user:add :: The account "drill52d90c" was created successfully Display name set to "drill52d90c" +[20:02:16] nextcloud: seeded user drill52d90c +[20:02:17] nextcloud: readback of the seeded user found=True +[20:02:17] C1 (seed reads back BEFORE): True +[20:02:17] MV-nextcloud: swapping to TO {'nextcloud': 'nextcloud:34.0.4-apache', 'nextcloud-db': 'mariadb:12.3', 'nextcloud-redis': 'redis:7-alpine'} +[20:02:19] TO up -d rc=0 +[20:02:51] TO settled=True in 31.2s :: {"nextcloud": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "nextcloud-db": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "nextcloud-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[20:02:51] engine state nextcloud-db: 12.3.3-MariaDB| This installation of MariaDB is already upgraded to 12.3.3-MariaDB. There is no need to run mariadb-upgrade again. [exit=1] +[20:02:51] migration lines observed: 6 +[20:02:52] nextcloud: readback of the seeded user found=True +[20:02:52] RESULT (seed reads back AFTER): True +[20:02:52] files_may_change: the bind-mounted tree changed under ['/srv/felhom-gate/hdd/appdata/nextcloud'] +[20:02:52] memory watch: 600s, 4 callers on 1 path(s) at 172.18.0.2:80 +[20:03:08] + 15s nextcloud=1023M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=4M/128M peak=6M kills=0 rs=0 reqs=253 +[20:03:23] + 30s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=4M/128M peak=7M kills=0 rs=0 reqs=513 +[20:03:38] + 46s nextcloud=1023M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=6M/128M peak=7M kills=0 rs=0 reqs=774 +[20:03:53] + 61s nextcloud=1023M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=6M/128M peak=8M kills=0 rs=0 reqs=1035 +[20:04:08] + 76s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=7M/128M peak=8M kills=0 rs=0 reqs=1295 +[20:04:23] + 91s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=7M/128M peak=9M kills=0 rs=0 reqs=1559 +[20:04:39] + 106s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=7M/128M peak=9M kills=0 rs=0 reqs=1819 +[20:04:54] + 121s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=7M/128M peak=9M kills=0 rs=0 reqs=2085 +[20:05:09] + 136s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=8M/128M peak=11M kills=0 rs=0 reqs=2346 +[20:05:24] + 152s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=8M/128M peak=11M kills=0 rs=0 reqs=2610 +[20:05:39] + 167s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=9M/128M peak=11M kills=0 rs=0 reqs=2874 +[20:05:54] + 182s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=10M/128M peak=12M kills=0 rs=0 reqs=3135 +[20:06:10] + 197s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=11M/128M peak=13M kills=0 rs=0 reqs=3400 +[20:06:25] + 212s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=11M/128M peak=13M kills=0 rs=0 reqs=3665 +[20:06:40] + 228s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=12M/128M peak=13M kills=0 rs=0 reqs=3928 +[20:06:55] + 243s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=13M/128M peak=14M kills=0 rs=0 reqs=4192 +[20:07:10] + 258s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=14M/128M peak=16M kills=0 rs=0 reqs=4451 +[20:07:25] + 273s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=14M/128M peak=16M kills=0 rs=0 reqs=4709 +[20:07:41] + 288s nextcloud=1023M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=14M/128M peak=16M kills=0 rs=0 reqs=4969 +[20:07:56] + 303s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=15M/128M peak=17M kills=0 rs=0 reqs=5226 +[20:08:11] + 318s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=16M/128M peak=17M kills=0 rs=0 reqs=5482 +[20:08:26] + 334s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=17M/128M peak=20M kills=0 rs=0 reqs=5744 +[20:08:41] + 349s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=18M/128M peak=20M kills=0 rs=0 reqs=6003 +[20:08:56] + 364s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=18M/128M peak=20M kills=0 rs=0 reqs=6263 +[20:09:12] + 379s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=184M/512M peak=187M kills=0 rs=0 nextcloud-redis=18M/128M peak=20M kills=0 rs=0 reqs=6519 +[20:09:27] + 394s nextcloud=999M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=20M/128M peak=24M kills=0 rs=0 reqs=6780 +[20:09:42] + 410s nextcloud=1000M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=20M/128M peak=24M kills=0 rs=0 reqs=7040 +[20:09:57] + 425s nextcloud=999M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=21M/128M peak=24M kills=0 rs=0 reqs=7301 +[20:10:12] + 440s nextcloud=1000M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=21M/128M peak=24M kills=0 rs=0 reqs=7563 +[20:10:27] + 455s nextcloud=1000M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=23M/128M peak=28M kills=0 rs=0 reqs=7826 +[20:10:43] + 470s nextcloud=999M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=24M/128M peak=28M kills=0 rs=0 reqs=8085 +[20:10:58] + 485s nextcloud=1000M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=24M/128M peak=28M kills=0 rs=0 reqs=8345 +[20:11:13] + 501s nextcloud=999M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=24M/128M peak=28M kills=0 rs=0 reqs=8606 +[20:11:28] + 516s nextcloud=1000M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=26M/128M peak=28M kills=0 rs=0 reqs=8862 +[20:11:43] + 531s nextcloud=999M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=26M/128M peak=30M kills=0 rs=0 reqs=9122 +[20:11:58] + 546s nextcloud=992M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=26M/128M peak=30M kills=0 rs=0 reqs=9379 +[20:12:14] + 561s nextcloud=978M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=28M/128M peak=30M kills=0 rs=0 reqs=9639 +[20:12:29] + 576s nextcloud=956M/1024M peak=1024M kills=0 rs=0 nextcloud-db=181M/512M peak=187M kills=0 rs=0 nextcloud-redis=28M/128M peak=30M kills=0 rs=0 reqs=9899 +[20:12:44] + 592s nextcloud=956M/1024M peak=1024M kills=0 rs=0 nextcloud-db=181M/512M peak=187M kills=0 rs=0 nextcloud-redis=30M/128M peak=35M kills=0 rs=0 reqs=10160 +[20:12:59] + 607s nextcloud=930M/1024M peak=1024M kills=0 rs=0 nextcloud-db=181M/512M peak=187M kills=0 rs=0 nextcloud-redis=30M/128M peak=35M kills=0 rs=0 reqs=10423 +[20:12:59] memory watch: killed=False tight=[] requests=10423 codes={'302': 10423} +[20:12:59] MV-nextcloud: ABORT — putting the FROM images back +[20:16:04] ABORT: the app did NOT come back (rc=0, 180.9s) +{ + "harness_version": 3, + "edge": "MV-nextcloud", + "app": "nextcloud", + "note": "night 2026-09-23 within-a-major move: nextcloud=nextcloud:34.0.4-apache", + "from": { + "nextcloud": "nextcloud:34.0.1-apache", + "nextcloud-db": "mariadb:12.3", + "nextcloud-redis": "redis:7-alpine" + }, + "to": { + "nextcloud": "nextcloud:34.0.4-apache", + "nextcloud-db": "mariadb:12.3", + "nextcloud-redis": "redis:7-alpine" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "\u001b[2Knextcloud | Upgrading nextcloud from 34.0.1.2 ...", + "abort": "refuses", + "abort_detail": "Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version?\n\u001b[2Knextcloud | => Configuring PHP session handler...\n\u001b[2Knextcloud | ==> Using Redis as PHP session handler...\n\u001b[2Knextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version?\n\u001b[2Knextcloud | => Configuring PHP session handler...\n\u001b[2Knextcloud | ==> Using Redis as PHP session handler...\n\u001b[2Knextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version?", + "engine_state_after": { + "nextcloud-db": { + "image": "mariadb:12.3", + "probe": "datadir version | the engine's own upgrade verdict", + "answer": "12.3.3-MariaDB| This installation of MariaDB is already upgraded to 12.3.3-MariaDB. There is no need to run mariadb-upgrade again. [exit=1]", + "probe_rc": 0 + } + }, + "memory": { + "soak_s": 607.1, + "requested_s": 600, + "requests": 10423, + "codes": { + "302": 10423 + }, + "first_kill": null, + "containers": { + "nextcloud": { + "limit": 1073741824, + "peak": 1073741824, + "peak_pct": 1.0, + "anon_peak_sampled": 84439040, + "anon_peak_pct": 0.079, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "nextcloud-db": { + "limit": 536870912, + "peak": 196354048, + "peak_pct": 0.366, + "anon_peak_sampled": 130146304, + "anon_peak_pct": 0.242, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "nextcloud-redis": { + "limit": 134217728, + "peak": 37715968, + "peak_pct": 0.281, + "anon_peak_sampled": 9433088, + "anon_peak_pct": 0.07, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + } + }, + "unmeasured": [] + }, + "marks": [ + "files_may_change" + ], + "duration_s": 31.2, + "measured_at": "2026-09-23T20:16:04Z", + "evidence": "evidence/MV-nextcloud", + "files_changed": [ + "/srv/felhom-gate/hdd/appdata/nextcloud" + ], + "total_s": 873.1 +} diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/abort-refusal.txt b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/abort-refusal.txt new file mode 100644 index 00000000..fbaaf3ed --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/abort-refusal.txt @@ -0,0 +1,116 @@ +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Securing system users (equivalent to running mysql_secure_installation) +nextcloud-db | +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Stopping temporary server +nextcloud-db | 2026-09-23 22:01:37 0 [Note] mariadbd (initiated by: unknown): Normal shutdown +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: FTS optimize thread exiting. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Starting shutdown... +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Buffer pool(s) dump completed at 260923 22:01:37 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Shutdown completed; log sequence number 45568; transaction id 15 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] mariadbd: Shutdown complete +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Temporary server stopped +nextcloud-db | +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: MariaDB init process done. Ready for start up. +nextcloud-db | +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Starting MariaDB 12.3.3-MariaDB-ubu2404 source revision 83e909fc2a0dbc394b4b683fb3fa2d7dcf26cc5e server_uid zBCjnNnCwpFijyK/FCLSYWzb8eA= as process 1 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Compressed tables use zlib 1.3 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Number of transaction pools: 1 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions +nextcloud-db | 2026-09-23 22:01:37 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup). +nextcloud-db | create_uring failed: falling back to libaio +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Using Linux native AIO +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: innodb_buffer_pool_size_max=8388608m, innodb_buffer_pool_size=128m +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Completed initialization of buffer pool +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes) +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: End of log at LSN=45568 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Opened 3 undo tablespaces +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ... +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: log sequence number 45568; transaction id 14 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Plugin 'FEEDBACK' is disabled. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Plugin 'wsrep-provider' is disabled. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Buffer pool(s) load completed at 260923 22:01:37 +nextcloud-db | 2026-09-23 22:01:40 0 [Note] Server socket created on IP: '0.0.0.0', port: '3306'. +nextcloud-db | 2026-09-23 22:01:40 0 [Note] Server socket created on IP: '::', port: '3306'. +nextcloud-redis | 1:M 23 Sep 2026 22:05:01.866 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:05:01.867 * Background saving started by pid 137 +nextcloud-redis | 137:C 23 Sep 2026 22:05:01.896 * DB saved on disk +nextcloud-redis | 137:C 23 Sep 2026 22:05:01.896 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:05:01.968 * Background saving terminated with success +nextcloud | => Configuring PHP session handler... +nextcloud-redis | 1:M 23 Sep 2026 22:06:06.001 * 10000 changes in 60 seconds. Saving... +nextcloud-db | 2026-09-23 22:01:40 0 [Note] mariadbd: Event Scheduler: Loaded 0 events +nextcloud-db | 2026-09-23 22:01:40 0 [Note] mariadbd: ready for connections. +nextcloud | ==> Using Redis as PHP session handler... +nextcloud-db | Version: '12.3.3-MariaDB-ubu2404' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud-redis | 1:M 23 Sep 2026 22:06:06.001 * Background saving started by pid 180 +nextcloud-redis | 180:C 23 Sep 2026 22:06:06.038 * DB saved on disk +nextcloud-redis | 180:C 23 Sep 2026 22:06:06.038 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:06:06.102 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:07:10.060 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:07:10.061 * Background saving started by pid 217 +nextcloud-redis | 217:C 23 Sep 2026 22:07:10.103 * DB saved on disk +nextcloud-redis | 217:C 23 Sep 2026 22:07:10.104 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:07:10.162 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:08:15.465 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:08:15.466 * Background saving started by pid 260 +nextcloud-redis | 260:C 23 Sep 2026 22:08:15.511 * DB saved on disk +nextcloud-redis | 260:C 23 Sep 2026 22:08:15.512 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:08:15.568 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:09:20.499 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:09:20.500 * Background saving started by pid 297 +nextcloud-redis | 297:C 23 Sep 2026 22:09:20.555 * DB saved on disk +nextcloud-redis | 297:C 23 Sep 2026 22:09:20.556 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:09:20.601 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:10:25.171 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:10:25.172 * Background saving started by pid 340 +nextcloud-redis | 340:C 23 Sep 2026 22:10:25.240 * DB saved on disk +nextcloud-redis | 340:C 23 Sep 2026 22:10:25.241 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:10:25.273 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:11:30.119 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:11:30.120 * Background saving started by pid 377 +nextcloud-redis | 377:C 23 Sep 2026 22:11:30.186 * DB saved on disk +nextcloud-redis | 377:C 23 Sep 2026 22:11:30.187 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:11:30.220 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:12:35.226 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:12:35.227 * Background saving started by pid 417 +nextcloud-redis | 417:C 23 Sep 2026 22:12:35.314 * DB saved on disk +nextcloud-redis | 417:C 23 Sep 2026 22:12:35.315 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:12:35.329 * Background saving terminated with success +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/abort-states.json b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/abort-states.json new file mode 100644 index 00000000..0a826a35 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/abort-states.json @@ -0,0 +1,20 @@ +{ + "nextcloud": { + "status": "restarting", + "health": "unhealthy", + "restarts": 0, + "exit": 1 + }, + "nextcloud-db": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + }, + "nextcloud-redis": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/compose-final.log b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/compose-final.log new file mode 100644 index 00000000..96cdab64 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/compose-final.log @@ -0,0 +1,167 @@ +nextcloud-redis | 1:C 23 Sep 2026 22:01:31.598 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +nextcloud-redis | 1:C 23 Sep 2026 22:01:31.598 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +nextcloud-redis | 1:C 23 Sep 2026 22:01:31.598 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +nextcloud-redis | 1:C 23 Sep 2026 22:01:31.598 * Configuration loaded +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.598 * monotonic clock: POSIX clock_gettime +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.599 * Running mode=standalone, port=6379. +nextcloud-db | 2026-09-23 22:01:31+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:12.3.3+maria~ubu2404 started. +nextcloud-db | 2026-09-23 22:01:31+02:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB +nextcloud-db | 2026-09-23 22:01:31+02:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' +nextcloud-db | 2026-09-23 22:01:31+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:12.3.3+maria~ubu2404 started. +nextcloud-db | 2026-09-23 22:01:32+02:00 [Note] [Entrypoint]: Initializing database files +nextcloud-db | 2026-09-23 22:01:32 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup). +nextcloud-db | create_uring failed: falling back to libaio +nextcloud-db | 2026-09-23 22:01:34+02:00 [Note] [Entrypoint]: Database files initialized +nextcloud-db | 2026-09-23 22:01:34+02:00 [Note] [Entrypoint]: Starting temporary server +nextcloud-db | 2026-09-23 22:01:34+02:00 [Note] [Entrypoint]: Waiting for server startup +nextcloud-db | 2026-09-23 22:01:34 0 [Note] Starting MariaDB 12.3.3-MariaDB-ubu2404 source revision 83e909fc2a0dbc394b4b683fb3fa2d7dcf26cc5e server_uid zBCjnNnCwpFijyK/FCLSYWzb8eA= as process 93 +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Compressed tables use zlib 1.3 +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Number of transaction pools: 1 +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions +nextcloud-db | 2026-09-23 22:01:34 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup). +nextcloud-db | create_uring failed: falling back to libaio +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Using Linux native AIO +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: innodb_buffer_pool_size_max=8388608m, innodb_buffer_pool_size=128m +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Completed initialization of buffer pool +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes) +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: End of log at LSN=45568 +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Opened 3 undo tablespaces +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active. +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ... +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB. +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.600 * Server initialized +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.603 * Creating AOF base file appendonly.aof.1.base.rdb on server start +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.611 * Creating AOF incr file appendonly.aof.1.incr.aof on server start +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.611 * Ready to accept connections tcp +nextcloud-redis | 1:M 23 Sep 2026 22:03:57.528 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:03:57.529 * Background saving started by pid 100 +nextcloud-redis | 100:C 23 Sep 2026 22:03:57.549 * DB saved on disk +nextcloud-redis | 100:C 23 Sep 2026 22:03:57.550 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:03:57.630 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:05:01.866 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:05:01.867 * Background saving started by pid 137 +nextcloud-redis | 137:C 23 Sep 2026 22:05:01.896 * DB saved on disk +nextcloud-redis | 137:C 23 Sep 2026 22:05:01.896 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:05:01.968 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:06:06.001 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:06:06.001 * Background saving started by pid 180 +nextcloud-redis | 180:C 23 Sep 2026 22:06:06.038 * DB saved on disk +nextcloud-redis | 180:C 23 Sep 2026 22:06:06.038 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:06:06.102 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:07:10.060 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:07:10.061 * Background saving started by pid 217 +nextcloud-redis | 217:C 23 Sep 2026 22:07:10.103 * DB saved on disk +nextcloud-redis | 217:C 23 Sep 2026 22:07:10.104 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:07:10.162 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:08:15.465 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:08:15.466 * Background saving started by pid 260 +nextcloud-redis | 260:C 23 Sep 2026 22:08:15.511 * DB saved on disk +nextcloud-redis | 260:C 23 Sep 2026 22:08:15.512 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:08:15.568 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:09:20.499 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:09:20.500 * Background saving started by pid 297 +nextcloud-redis | 297:C 23 Sep 2026 22:09:20.555 * DB saved on disk +nextcloud-redis | 297:C 23 Sep 2026 22:09:20.556 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:09:20.601 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:10:25.171 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:10:25.172 * Background saving started by pid 340 +nextcloud-redis | 340:C 23 Sep 2026 22:10:25.240 * DB saved on disk +nextcloud-redis | 340:C 23 Sep 2026 22:10:25.241 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:10:25.273 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:11:30.119 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:11:30.120 * Background saving started by pid 377 +nextcloud-redis | 377:C 23 Sep 2026 22:11:30.186 * DB saved on disk +nextcloud-redis | 377:C 23 Sep 2026 22:11:30.187 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:11:30.220 * Background saving terminated with success +nextcloud-redis | 1:M 23 Sep 2026 22:12:35.226 * 10000 changes in 60 seconds. Saving... +nextcloud-redis | 1:M 23 Sep 2026 22:12:35.227 * Background saving started by pid 417 +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud-redis | 417:C 23 Sep 2026 22:12:35.314 * DB saved on disk +nextcloud-redis | 417:C 23 Sep 2026 22:12:35.315 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB +nextcloud-redis | 1:M 23 Sep 2026 22:12:35.329 * Background saving terminated with success +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: log sequence number 45568; transaction id 14 +nextcloud-db | 2026-09-23 22:01:34 0 [Note] Plugin 'FEEDBACK' is disabled. +nextcloud-db | 2026-09-23 22:01:34 0 [Note] Plugin 'wsrep-provider' is disabled. +nextcloud-db | 2026-09-23 22:01:35 0 [Note] mariadbd: Event Scheduler: Loaded 0 events +nextcloud-db | 2026-09-23 22:01:35 0 [Note] Replication not automatically started: --skip-slave-start was specified +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud-db | 2026-09-23 22:01:35 0 [Note] mariadbd: ready for connections. +nextcloud-db | Version: '12.3.3-MariaDB-ubu2404' socket: '/run/mysqld/mysqld.sock' port: 0 mariadb.org binary distribution +nextcloud-db | 2026-09-23 22:01:36+02:00 [Note] [Entrypoint]: Temporary server started. +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Creating database nextcloud +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Creating user nextcloud +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Giving user nextcloud access to schema nextcloud +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Securing system users (equivalent to running mysql_secure_installation) +nextcloud-db | +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Stopping temporary server +nextcloud-db | 2026-09-23 22:01:37 0 [Note] mariadbd (initiated by: unknown): Normal shutdown +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: FTS optimize thread exiting. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Starting shutdown... +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool +nextcloud | => Configuring PHP session handler... +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Buffer pool(s) dump completed at 260923 22:01:37 +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version? +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Shutdown completed; log sequence number 45568; transaction id 15 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] mariadbd: Shutdown complete +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Temporary server stopped +nextcloud-db | +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: MariaDB init process done. Ready for start up. +nextcloud-db | +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Starting MariaDB 12.3.3-MariaDB-ubu2404 source revision 83e909fc2a0dbc394b4b683fb3fa2d7dcf26cc5e server_uid zBCjnNnCwpFijyK/FCLSYWzb8eA= as process 1 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Compressed tables use zlib 1.3 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Number of transaction pools: 1 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions +nextcloud-db | 2026-09-23 22:01:37 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup). +nextcloud-db | create_uring failed: falling back to libaio +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Using Linux native AIO +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: innodb_buffer_pool_size_max=8388608m, innodb_buffer_pool_size=128m +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Completed initialization of buffer pool +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes) +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: End of log at LSN=45568 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Opened 3 undo tablespaces +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ... +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: log sequence number 45568; transaction id 14 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Plugin 'FEEDBACK' is disabled. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Plugin 'wsrep-provider' is disabled. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Buffer pool(s) load completed at 260923 22:01:37 +nextcloud-db | 2026-09-23 22:01:40 0 [Note] Server socket created on IP: '0.0.0.0', port: '3306'. +nextcloud-db | 2026-09-23 22:01:40 0 [Note] Server socket created on IP: '::', port: '3306'. +nextcloud-db | 2026-09-23 22:01:40 0 [Note] mariadbd: Event Scheduler: Loaded 0 events +nextcloud-db | 2026-09-23 22:01:40 0 [Note] mariadbd: ready for connections. +nextcloud-db | Version: '12.3.3-MariaDB-ubu2404' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/engine-state.json b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/engine-state.json new file mode 100644 index 00000000..46fd4c9b --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/engine-state.json @@ -0,0 +1,8 @@ +{ + "nextcloud-db": { + "image": "mariadb:12.3", + "probe": "datadir version | the engine's own upgrade verdict", + "answer": "12.3.3-MariaDB| This installation of MariaDB is already upgraded to 12.3.3-MariaDB. There is no need to run mariadb-upgrade again. [exit=1]", + "probe_rc": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/files-after.json b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/files-after.json new file mode 100644 index 00000000..5d2ef2ba --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/files-after.json @@ -0,0 +1,3 @@ +{ + "/srv/felhom-gate/hdd/appdata/nextcloud": "0b74bfada6b0b4098f84fe456ae07a002528f8d2be77838923a0190c08a591d1" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/files-before.json b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/files-before.json new file mode 100644 index 00000000..8e25502d --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/files-before.json @@ -0,0 +1,3 @@ +{ + "/srv/felhom-gate/hdd/appdata/nextcloud": "b16738917b19ede8b81373358ac77190c8943551c6108562e006f0b2f82080fd" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/memory-samples.json b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/memory-samples.json new file mode 100644 index 00000000..3db54197 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/memory-samples.json @@ -0,0 +1,1562 @@ +[ + { + "t": 15.2, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1073463296, + "peak": 1073741824, + "anon": 75980800, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188903424, + "peak": 193474560, + "anon": 130142208, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 4976640, + "peak": 6905856, + "anon": 3342336, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 253 + }, + { + "t": 30.4, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1072615424, + "peak": 1073741824, + "anon": 76075008, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188870656, + "peak": 193474560, + "anon": 130138112, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 5066752, + "peak": 7360512, + "anon": 3514368, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 513 + }, + { + "t": 45.5, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1072988160, + "peak": 1073741824, + "anon": 76247040, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188870656, + "peak": 193474560, + "anon": 130138112, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 6471680, + "peak": 8298496, + "anon": 3657728, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 774 + }, + { + "t": 60.7, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1072762880, + "peak": 1073741824, + "anon": 76279808, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188907520, + "peak": 193474560, + "anon": 130125824, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 7159808, + "peak": 8744960, + "anon": 3837952, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 1035 + }, + { + "t": 75.9, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1072381952, + "peak": 1073741824, + "anon": 76410880, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 189034496, + "peak": 193474560, + "anon": 130125824, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 7598080, + "peak": 9117696, + "anon": 3969024, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 1295 + }, + { + "t": 91.0, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071833088, + "peak": 1073741824, + "anon": 76345344, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 189034496, + "peak": 193474560, + "anon": 130125824, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 7393280, + "peak": 10133504, + "anon": 4116480, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 1559 + }, + { + "t": 106.2, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071620096, + "peak": 1073741824, + "anon": 76345344, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188776448, + "peak": 193474560, + "anon": 130125824, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 8306688, + "peak": 10133504, + "anon": 4280320, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 1819 + }, + { + "t": 121.4, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071878144, + "peak": 1073741824, + "anon": 76382208, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188776448, + "peak": 193474560, + "anon": 130125824, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 8056832, + "peak": 10133504, + "anon": 4476928, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2085 + }, + { + "t": 136.5, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071407104, + "peak": 1073741824, + "anon": 76455936, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 189030400, + "peak": 193474560, + "anon": 130125824, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 9064448, + "peak": 11710464, + "anon": 4620288, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2346 + }, + { + "t": 151.7, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1070907392, + "peak": 1073741824, + "anon": 76390400, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 189038592, + "peak": 193474560, + "anon": 130134016, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 9265152, + "peak": 11804672, + "anon": 4759552, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2610 + }, + { + "t": 166.9, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071394816, + "peak": 1073741824, + "anon": 76390400, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188784640, + "peak": 193474560, + "anon": 130134016, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 10473472, + "peak": 11927552, + "anon": 4915200, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 2874 + }, + { + "t": 182.1, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1070858240, + "peak": 1073741824, + "anon": 76390400, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188792832, + "peak": 193474560, + "anon": 130138112, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 11112448, + "peak": 13012992, + "anon": 5058560, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 3135 + }, + { + "t": 197.2, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1070899200, + "peak": 1073741824, + "anon": 76390400, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188788736, + "peak": 193474560, + "anon": 130138112, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 11747328, + "peak": 14508032, + "anon": 5206016, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 3400 + }, + { + "t": 212.4, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071067136, + "peak": 1073741824, + "anon": 76316672, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188796928, + "peak": 193474560, + "anon": 130142208, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 12152832, + "peak": 14508032, + "anon": 5349376, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 3665 + }, + { + "t": 227.6, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1070657536, + "peak": 1073741824, + "anon": 76382208, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 189558784, + "peak": 193474560, + "anon": 130142208, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 13053952, + "peak": 14508032, + "anon": 5505024, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 3928 + }, + { + "t": 242.7, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071550464, + "peak": 1073741824, + "anon": 76308480, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188792832, + "peak": 193474560, + "anon": 130142208, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 14180352, + "peak": 15110144, + "anon": 5767168, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 4192 + }, + { + "t": 257.9, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1070862336, + "peak": 1073741824, + "anon": 76349440, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188792832, + "peak": 193474560, + "anon": 130142208, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 14716928, + "peak": 17698816, + "anon": 5910528, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 4451 + }, + { + "t": 273.0, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071931392, + "peak": 1073741824, + "anon": 84180992, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188792832, + "peak": 193474560, + "anon": 130142208, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 15155200, + "peak": 17698816, + "anon": 6057984, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 4709 + }, + { + "t": 288.2, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1073078272, + "peak": 1073741824, + "anon": 84312064, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188792832, + "peak": 193474560, + "anon": 130142208, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 15687680, + "peak": 17698816, + "anon": 6242304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 4969 + }, + { + "t": 303.4, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071325184, + "peak": 1073741824, + "anon": 84353024, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 189046784, + "peak": 193474560, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 16019456, + "peak": 18305024, + "anon": 6369280, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 5226 + }, + { + "t": 318.5, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1070825472, + "peak": 1073741824, + "anon": 84377600, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188796928, + "peak": 193474560, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 16912384, + "peak": 18305024, + "anon": 6516736, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 5482 + }, + { + "t": 333.7, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071808512, + "peak": 1073741824, + "anon": 84340736, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188796928, + "peak": 193474560, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 18112512, + "peak": 21712896, + "anon": 6656000, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 5744 + }, + { + "t": 348.9, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1070882816, + "peak": 1073741824, + "anon": 84422656, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188801024, + "peak": 193474560, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 18972672, + "peak": 21712896, + "anon": 6787072, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 6003 + }, + { + "t": 364.0, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071144960, + "peak": 1073741824, + "anon": 84422656, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 188796928, + "peak": 193474560, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 18944000, + "peak": 21712896, + "anon": 6942720, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 6263 + }, + { + "t": 379.2, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1071620096, + "peak": 1073741824, + "anon": 84422656, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192970752, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 19599360, + "peak": 21712896, + "anon": 7094272, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 6519 + }, + { + "t": 394.4, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1048449024, + "peak": 1073741824, + "anon": 84357120, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192188416, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 21110784, + "peak": 25767936, + "anon": 7241728, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 6780 + }, + { + "t": 409.5, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1049272320, + "peak": 1073741824, + "anon": 84381696, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192188416, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 21905408, + "peak": 25767936, + "anon": 7380992, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 7040 + }, + { + "t": 424.7, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1048432640, + "peak": 1073741824, + "anon": 84316160, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192188416, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 22941696, + "peak": 25767936, + "anon": 7528448, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 7301 + }, + { + "t": 439.9, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1049235456, + "peak": 1073741824, + "anon": 84381696, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192450560, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 23003136, + "peak": 25767936, + "anon": 7659520, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 7563 + }, + { + "t": 455.1, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1048969216, + "peak": 1073741824, + "anon": 84316160, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192188416, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 24616960, + "peak": 29515776, + "anon": 7811072, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 7826 + }, + { + "t": 470.2, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1048436736, + "peak": 1073741824, + "anon": 84316160, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192188416, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 25247744, + "peak": 29515776, + "anon": 7966720, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8085 + }, + { + "t": 485.4, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1048604672, + "peak": 1073741824, + "anon": 84250624, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192188416, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 25239552, + "peak": 29515776, + "anon": 8294400, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8345 + }, + { + "t": 500.6, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1048440832, + "peak": 1073741824, + "anon": 84332544, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192188416, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 26005504, + "peak": 29515776, + "anon": 8409088, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8606 + }, + { + "t": 515.8, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1048776704, + "peak": 1073741824, + "anon": 84414464, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192188416, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 27840512, + "peak": 29892608, + "anon": 8568832, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 8862 + }, + { + "t": 530.9, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1048178688, + "peak": 1073741824, + "anon": 84439040, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192188416, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 27541504, + "peak": 32432128, + "anon": 8691712, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 9122 + }, + { + "t": 546.1, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1040482304, + "peak": 1073741824, + "anon": 84373504, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192188416, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 28295168, + "peak": 32432128, + "anon": 8839168, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 9379 + }, + { + "t": 561.2, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1026338816, + "peak": 1073741824, + "anon": 84398080, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 192188416, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 29364224, + "peak": 32432128, + "anon": 8994816, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 9639 + }, + { + "t": 576.4, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1003057152, + "peak": 1073741824, + "anon": 84393984, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 190226432, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 30355456, + "peak": 32432128, + "anon": 9142272, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 9899 + }, + { + "t": 591.6, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 1002835968, + "peak": 1073741824, + "anon": 84434944, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 190226432, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 31789056, + "peak": 37715968, + "anon": 9302016, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 10160 + }, + { + "t": 606.8, + "containers": { + "nextcloud": { + "limit": 1073741824, + "current": 975462400, + "peak": 1073741824, + "anon": 84410368, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-db": { + "limit": 536870912, + "current": 190226432, + "peak": 196354048, + "anon": 130146304, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + }, + "nextcloud-redis": { + "limit": 134217728, + "current": 31498240, + "peak": 37715968, + "anon": 9433088, + "oom_kill": 0, + "restarts": 0, + "oomkilled_flag": false, + "status": "running", + "cgroup": true + } + }, + "requests": 10423 + } +] \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/migration-lines.txt b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/migration-lines.txt new file mode 100644 index 00000000..f9eb5d31 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/migration-lines.txt @@ -0,0 +1,6 @@ +nextcloud | Upgrading nextcloud from 34.0.1.2 ... +nextcloud | => Searching for hook scripts (*.sh) to run, located in the folder "/docker-entrypoint-hooks.d/pre-upgrade" +nextcloud | ==> Skipped: the "pre-upgrade" folder is empty (or does not exist) +nextcloud | Updating database schema +nextcloud | => Searching for hook scripts (*.sh) to run, located in the folder "/docker-entrypoint-hooks.d/post-upgrade" +nextcloud | ==> Skipped: the "post-upgrade" folder is empty (or does not exist) \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/run.log b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/run.log new file mode 100644 index 00000000..69ed932e --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/run.log @@ -0,0 +1,59 @@ +[20:01:30] MV-nextcloud: deploying nextcloud at FROM {'nextcloud': 'nextcloud:34.0.1-apache', 'nextcloud-db': 'mariadb:12.3', 'nextcloud-redis': 'redis:7-alpine'} +[20:02:13] FROM settled=True in 31.1s :: {"nextcloud": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "nextcloud-db": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "nextcloud-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[20:02:13] fixture: the BOX walk's own (Nextcloud), through upgrade_boxport +[20:02:15] nextcloud: occ user:add :: The account "drill52d90c" was created successfully Display name set to "drill52d90c" +[20:02:16] nextcloud: seeded user drill52d90c +[20:02:17] nextcloud: readback of the seeded user found=True +[20:02:17] C1 (seed reads back BEFORE): True +[20:02:17] MV-nextcloud: swapping to TO {'nextcloud': 'nextcloud:34.0.4-apache', 'nextcloud-db': 'mariadb:12.3', 'nextcloud-redis': 'redis:7-alpine'} +[20:02:19] TO up -d rc=0 +[20:02:51] TO settled=True in 31.2s :: {"nextcloud": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "nextcloud-db": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}, "nextcloud-redis": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[20:02:51] engine state nextcloud-db: 12.3.3-MariaDB| This installation of MariaDB is already upgraded to 12.3.3-MariaDB. There is no need to run mariadb-upgrade again. [exit=1] +[20:02:51] migration lines observed: 6 +[20:02:52] nextcloud: readback of the seeded user found=True +[20:02:52] RESULT (seed reads back AFTER): True +[20:02:52] files_may_change: the bind-mounted tree changed under ['/srv/felhom-gate/hdd/appdata/nextcloud'] +[20:02:52] memory watch: 600s, 4 callers on 1 path(s) at 172.18.0.2:80 +[20:03:08] + 15s nextcloud=1023M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=4M/128M peak=6M kills=0 rs=0 reqs=253 +[20:03:23] + 30s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=4M/128M peak=7M kills=0 rs=0 reqs=513 +[20:03:38] + 46s nextcloud=1023M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=6M/128M peak=7M kills=0 rs=0 reqs=774 +[20:03:53] + 61s nextcloud=1023M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=6M/128M peak=8M kills=0 rs=0 reqs=1035 +[20:04:08] + 76s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=7M/128M peak=8M kills=0 rs=0 reqs=1295 +[20:04:23] + 91s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=7M/128M peak=9M kills=0 rs=0 reqs=1559 +[20:04:39] + 106s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=7M/128M peak=9M kills=0 rs=0 reqs=1819 +[20:04:54] + 121s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=7M/128M peak=9M kills=0 rs=0 reqs=2085 +[20:05:09] + 136s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=8M/128M peak=11M kills=0 rs=0 reqs=2346 +[20:05:24] + 152s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=8M/128M peak=11M kills=0 rs=0 reqs=2610 +[20:05:39] + 167s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=9M/128M peak=11M kills=0 rs=0 reqs=2874 +[20:05:54] + 182s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=10M/128M peak=12M kills=0 rs=0 reqs=3135 +[20:06:10] + 197s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=11M/128M peak=13M kills=0 rs=0 reqs=3400 +[20:06:25] + 212s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=11M/128M peak=13M kills=0 rs=0 reqs=3665 +[20:06:40] + 228s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=12M/128M peak=13M kills=0 rs=0 reqs=3928 +[20:06:55] + 243s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=13M/128M peak=14M kills=0 rs=0 reqs=4192 +[20:07:10] + 258s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=14M/128M peak=16M kills=0 rs=0 reqs=4451 +[20:07:25] + 273s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=14M/128M peak=16M kills=0 rs=0 reqs=4709 +[20:07:41] + 288s nextcloud=1023M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=14M/128M peak=16M kills=0 rs=0 reqs=4969 +[20:07:56] + 303s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=15M/128M peak=17M kills=0 rs=0 reqs=5226 +[20:08:11] + 318s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=16M/128M peak=17M kills=0 rs=0 reqs=5482 +[20:08:26] + 334s nextcloud=1022M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=17M/128M peak=20M kills=0 rs=0 reqs=5744 +[20:08:41] + 349s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=18M/128M peak=20M kills=0 rs=0 reqs=6003 +[20:08:56] + 364s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=180M/512M peak=184M kills=0 rs=0 nextcloud-redis=18M/128M peak=20M kills=0 rs=0 reqs=6263 +[20:09:12] + 379s nextcloud=1021M/1024M peak=1024M kills=0 rs=0 nextcloud-db=184M/512M peak=187M kills=0 rs=0 nextcloud-redis=18M/128M peak=20M kills=0 rs=0 reqs=6519 +[20:09:27] + 394s nextcloud=999M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=20M/128M peak=24M kills=0 rs=0 reqs=6780 +[20:09:42] + 410s nextcloud=1000M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=20M/128M peak=24M kills=0 rs=0 reqs=7040 +[20:09:57] + 425s nextcloud=999M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=21M/128M peak=24M kills=0 rs=0 reqs=7301 +[20:10:12] + 440s nextcloud=1000M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=21M/128M peak=24M kills=0 rs=0 reqs=7563 +[20:10:27] + 455s nextcloud=1000M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=23M/128M peak=28M kills=0 rs=0 reqs=7826 +[20:10:43] + 470s nextcloud=999M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=24M/128M peak=28M kills=0 rs=0 reqs=8085 +[20:10:58] + 485s nextcloud=1000M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=24M/128M peak=28M kills=0 rs=0 reqs=8345 +[20:11:13] + 501s nextcloud=999M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=24M/128M peak=28M kills=0 rs=0 reqs=8606 +[20:11:28] + 516s nextcloud=1000M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=26M/128M peak=28M kills=0 rs=0 reqs=8862 +[20:11:43] + 531s nextcloud=999M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=26M/128M peak=30M kills=0 rs=0 reqs=9122 +[20:11:58] + 546s nextcloud=992M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=26M/128M peak=30M kills=0 rs=0 reqs=9379 +[20:12:14] + 561s nextcloud=978M/1024M peak=1024M kills=0 rs=0 nextcloud-db=183M/512M peak=187M kills=0 rs=0 nextcloud-redis=28M/128M peak=30M kills=0 rs=0 reqs=9639 +[20:12:29] + 576s nextcloud=956M/1024M peak=1024M kills=0 rs=0 nextcloud-db=181M/512M peak=187M kills=0 rs=0 nextcloud-redis=28M/128M peak=30M kills=0 rs=0 reqs=9899 +[20:12:44] + 592s nextcloud=956M/1024M peak=1024M kills=0 rs=0 nextcloud-db=181M/512M peak=187M kills=0 rs=0 nextcloud-redis=30M/128M peak=35M kills=0 rs=0 reqs=10160 +[20:12:59] + 607s nextcloud=930M/1024M peak=1024M kills=0 rs=0 nextcloud-db=181M/512M peak=187M kills=0 rs=0 nextcloud-redis=30M/128M peak=35M kills=0 rs=0 reqs=10423 +[20:12:59] memory watch: killed=False tight=[] requests=10423 codes={'302': 10423} +[20:12:59] MV-nextcloud: ABORT — putting the FROM images back +[20:16:04] ABORT: the app did NOT come back (rc=0, 180.9s) \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/to-full.log b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/to-full.log new file mode 100644 index 00000000..89d3f92c --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/to-full.log @@ -0,0 +1,128 @@ +nextcloud-redis | 1:C 23 Sep 2026 22:01:31.598 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +nextcloud-redis | 1:C 23 Sep 2026 22:01:31.598 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +nextcloud-redis | 1:C 23 Sep 2026 22:01:31.598 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +nextcloud-redis | 1:C 23 Sep 2026 22:01:31.598 * Configuration loaded +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.598 * monotonic clock: POSIX clock_gettime +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.599 * Running mode=standalone, port=6379. +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.600 * Server initialized +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.603 * Creating AOF base file appendonly.aof.1.base.rdb on server start +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.611 * Creating AOF incr file appendonly.aof.1.incr.aof on server start +nextcloud-redis | 1:M 23 Sep 2026 22:01:31.611 * Ready to accept connections tcp +nextcloud-db | 2026-09-23 22:01:31+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:12.3.3+maria~ubu2404 started. +nextcloud-db | 2026-09-23 22:01:31+02:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB +nextcloud-db | 2026-09-23 22:01:31+02:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' +nextcloud-db | 2026-09-23 22:01:31+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:12.3.3+maria~ubu2404 started. +nextcloud-db | 2026-09-23 22:01:32+02:00 [Note] [Entrypoint]: Initializing database files +nextcloud-db | 2026-09-23 22:01:32 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup). +nextcloud-db | create_uring failed: falling back to libaio +nextcloud-db | 2026-09-23 22:01:34+02:00 [Note] [Entrypoint]: Database files initialized +nextcloud-db | 2026-09-23 22:01:34+02:00 [Note] [Entrypoint]: Starting temporary server +nextcloud-db | 2026-09-23 22:01:34+02:00 [Note] [Entrypoint]: Waiting for server startup +nextcloud-db | 2026-09-23 22:01:34 0 [Note] Starting MariaDB 12.3.3-MariaDB-ubu2404 source revision 83e909fc2a0dbc394b4b683fb3fa2d7dcf26cc5e server_uid zBCjnNnCwpFijyK/FCLSYWzb8eA= as process 93 +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Compressed tables use zlib 1.3 +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Number of transaction pools: 1 +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions +nextcloud-db | 2026-09-23 22:01:34 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup). +nextcloud-db | create_uring failed: falling back to libaio +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Using Linux native AIO +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: innodb_buffer_pool_size_max=8388608m, innodb_buffer_pool_size=128m +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Completed initialization of buffer pool +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes) +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: End of log at LSN=45568 +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Opened 3 undo tablespaces +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active. +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ... +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB. +nextcloud-db | 2026-09-23 22:01:34 0 [Note] InnoDB: log sequence number 45568; transaction id 14 +nextcloud-db | 2026-09-23 22:01:34 0 [Note] Plugin 'FEEDBACK' is disabled. +nextcloud-db | 2026-09-23 22:01:34 0 [Note] Plugin 'wsrep-provider' is disabled. +nextcloud-db | 2026-09-23 22:01:35 0 [Note] mariadbd: Event Scheduler: Loaded 0 events +nextcloud-db | 2026-09-23 22:01:35 0 [Note] Replication not automatically started: --skip-slave-start was specified +nextcloud-db | 2026-09-23 22:01:35 0 [Note] mariadbd: ready for connections. +nextcloud-db | Version: '12.3.3-MariaDB-ubu2404' socket: '/run/mysqld/mysqld.sock' port: 0 mariadb.org binary distribution +nextcloud-db | 2026-09-23 22:01:36+02:00 [Note] [Entrypoint]: Temporary server started. +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Creating database nextcloud +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Creating user nextcloud +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Giving user nextcloud access to schema nextcloud +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Securing system users (equivalent to running mysql_secure_installation) +nextcloud-db | +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Stopping temporary server +nextcloud | => Configuring PHP session handler... +nextcloud | ==> Using Redis as PHP session handler... +nextcloud | Initializing nextcloud 34.0.4.1 ... +nextcloud | Upgrading nextcloud from 34.0.1.2 ... +nextcloud | => Searching for hook scripts (*.sh) to run, located in the folder "/docker-entrypoint-hooks.d/pre-upgrade" +nextcloud | ==> Skipped: the "pre-upgrade" folder is empty (or does not exist) +nextcloud | Setting log level to debug +nextcloud | Turned on maintenance mode +nextcloud-db | 2026-09-23 22:01:37 0 [Note] mariadbd (initiated by: unknown): Normal shutdown +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: FTS optimize thread exiting. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Starting shutdown... +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Buffer pool(s) dump completed at 260923 22:01:37 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Shutdown completed; log sequence number 45568; transaction id 15 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] mariadbd: Shutdown complete +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: Temporary server stopped +nextcloud-db | +nextcloud-db | 2026-09-23 22:01:37+02:00 [Note] [Entrypoint]: MariaDB init process done. Ready for start up. +nextcloud-db | +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Starting MariaDB 12.3.3-MariaDB-ubu2404 source revision 83e909fc2a0dbc394b4b683fb3fa2d7dcf26cc5e server_uid zBCjnNnCwpFijyK/FCLSYWzb8eA= as process 1 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Compressed tables use zlib 1.3 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Number of transaction pools: 1 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions +nextcloud-db | 2026-09-23 22:01:37 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup). +nextcloud-db | create_uring failed: falling back to libaio +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Using Linux native AIO +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: innodb_buffer_pool_size_max=8388608m, innodb_buffer_pool_size=128m +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Completed initialization of buffer pool +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes) +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: End of log at LSN=45568 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Opened 3 undo tablespaces +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ... +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: log sequence number 45568; transaction id 14 +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Plugin 'FEEDBACK' is disabled. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] Plugin 'wsrep-provider' is disabled. +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool +nextcloud-db | 2026-09-23 22:01:37 0 [Note] InnoDB: Buffer pool(s) load completed at 260923 22:01:37 +nextcloud-db | 2026-09-23 22:01:40 0 [Note] Server socket created on IP: '0.0.0.0', port: '3306'. +nextcloud-db | 2026-09-23 22:01:40 0 [Note] Server socket created on IP: '::', port: '3306'. +nextcloud-db | 2026-09-23 22:01:40 0 [Note] mariadbd: Event Scheduler: Loaded 0 events +nextcloud-db | 2026-09-23 22:01:40 0 [Note] mariadbd: ready for connections. +nextcloud-db | Version: '12.3.3-MariaDB-ubu2404' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution +nextcloud | Updating database schema +nextcloud | Updated database +nextcloud | Updating ... +nextcloud | Fix broken values of calendar objects +nextcloud | +nextcloud | Starting ... +nextcloud | +nextcloud | Clean up old calendar subscriptions from deleted users that were not cleaned-up +nextcloud | +nextcloud | 0/0 [>---------------------------] 0% Starting ... +nextcloud | +nextcloud | Updated to 1.40.0 +nextcloud | Updating ... +nextcloud | Updated to 34.0.1 +nextcloud | Updating ... +nextcloud | Updated to 5.2.0 +nextcloud | Updating ... +nextcloud | Updated to 7.0.0 +nextcloud | Starting code integrity check... +nextcloud | Finished code integrity check +nextcloud | Update successful +nextcloud | Turned off maintenance mode +nextcloud | Resetting log level +nextcloud | => Searching for hook scripts (*.sh) to run, located in the folder "/docker-entrypoint-hooks.d/post-upgrade" +nextcloud | ==> Skipped: the "post-upgrade" folder is empty (or does not exist) +nextcloud | Initializing finished +nextcloud | Warning: /var/www/html/config/s3.config.php differs from the latest version of this image at /usr/src/nextcloud/config/s3.config.php +nextcloud | => Searching for hook scripts (*.sh) to run, located in the folder "/docker-entrypoint-hooks.d/before-starting" +nextcloud | ==> Skipped: the "before-starting" folder is empty (or does not exist) +nextcloud | 0/0 [->--------------------------] 0%AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.2. Set the 'ServerName' directive globally to suppress this message +nextcloud | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.2. Set the 'ServerName' directive globally to suppress this message +nextcloud | [Wed Sep 23 22:02:46.866497 2026] [mpm_prefork:notice] [pid 1:tid 1] AH00163: Apache/2.4.68 (Debian) PHP/8.5.10 configured -- resuming normal operations +nextcloud | [Wed Sep 23 22:02:46.866539 2026] [core:notice] [pid 1:tid 1] AH00094: Command line: 'apache2 -D FOREGROUND' +nextcloud | 127.0.0.1 - - [23/Sep/2026:22:02:49 +0200] "GET /status.php HTTP/1.1" 200 1064 "-" "curl/8.14.1" diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/to-states.json b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/to-states.json new file mode 100644 index 00000000..516e6dd7 --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/to-states.json @@ -0,0 +1,20 @@ +{ + "nextcloud": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + }, + "nextcloud-db": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + }, + "nextcloud-redis": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/verdict.json b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/verdict.json new file mode 100644 index 00000000..8c0f5e1c --- /dev/null +++ b/documentation/audits/night-2026-09-23/apps/nextcloud/bench/evidence/MV-nextcloud/verdict.json @@ -0,0 +1,86 @@ +{ + "harness_version": 3, + "edge": "MV-nextcloud", + "app": "nextcloud", + "note": "night 2026-09-23 within-a-major move: nextcloud=nextcloud:34.0.4-apache", + "from": { + "nextcloud": "nextcloud:34.0.1-apache", + "nextcloud-db": "mariadb:12.3", + "nextcloud-redis": "redis:7-alpine" + }, + "to": { + "nextcloud": "nextcloud:34.0.4-apache", + "nextcloud-db": "mariadb:12.3", + "nextcloud-redis": "redis:7-alpine" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "\u001b[2Knextcloud | Upgrading nextcloud from 34.0.1.2 ...", + "abort": "refuses", + "abort_detail": "Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version?\n\u001b[2Knextcloud | => Configuring PHP session handler...\n\u001b[2Knextcloud | ==> Using Redis as PHP session handler...\n\u001b[2Knextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version?\n\u001b[2Knextcloud | => Configuring PHP session handler...\n\u001b[2Knextcloud | ==> Using Redis as PHP session handler...\n\u001b[2Knextcloud | Can't start Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have pulled the newest image version?", + "engine_state_after": { + "nextcloud-db": { + "image": "mariadb:12.3", + "probe": "datadir version | the engine's own upgrade verdict", + "answer": "12.3.3-MariaDB| This installation of MariaDB is already upgraded to 12.3.3-MariaDB. There is no need to run mariadb-upgrade again. [exit=1]", + "probe_rc": 0 + } + }, + "memory": { + "soak_s": 607.1, + "requested_s": 600, + "requests": 10423, + "codes": { + "302": 10423 + }, + "first_kill": null, + "containers": { + "nextcloud": { + "limit": 1073741824, + "peak": 1073741824, + "peak_pct": 1.0, + "anon_peak_sampled": 84439040, + "anon_peak_pct": 0.079, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "nextcloud-db": { + "limit": 536870912, + "peak": 196354048, + "peak_pct": 0.366, + "anon_peak_sampled": 130146304, + "anon_peak_pct": 0.242, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + }, + "nextcloud-redis": { + "limit": 134217728, + "peak": 37715968, + "peak_pct": 0.281, + "anon_peak_sampled": 9433088, + "anon_peak_pct": 0.07, + "oom_kills": 0, + "restarts": 0, + "oomkilled_flag": false, + "measured": true + } + }, + "unmeasured": [] + }, + "marks": [ + "files_may_change" + ], + "duration_s": 31.2, + "measured_at": "2026-09-23T20:16:04Z", + "evidence": "evidence/MV-nextcloud", + "files_changed": [ + "/srv/felhom-gate/hdd/appdata/nextcloud" + ], + "total_s": 873.1 +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/bench-C3/C3.log b/documentation/audits/night-2026-09-23/bench-C3/C3.log new file mode 100644 index 00000000..6fe0f9a9 --- /dev/null +++ b/documentation/audits/night-2026-09-23/bench-C3/C3.log @@ -0,0 +1,40 @@ +[18:39:45] C3: deploying privatebin at FROM {'privatebin': 'privatebin/pdo:2.0.5'} +[18:40:16] FROM settled=True in 30.8s :: {"privatebin": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[18:40:16] privatebin: seeded paste id=a7c6f18ad9707051 +[18:40:16] privatebin: readback http=200 marker_present=True +[18:40:16] C1 (seed reads back BEFORE): True +[18:40:16] C3: swapping to TO {'privatebin': 'alpine:3.20'} +[18:40:20] TO up -d rc=0 +[18:47:24] TO settled=False in 424.4s :: {"privatebin": {"status": "restarting", "health": "unhealthy", "restarts": 0, "exit": 0}} +[18:47:24] migration lines observed: 0 +[18:47:24] RESULT (seed reads back AFTER): False +[18:47:25] C3: ABORT — putting the FROM images back +[18:47:56] privatebin: readback http=200 marker_present=True +[18:47:56] ABORT: app came back in 30.8s; data present=True +{ + "harness_version": 3, + "edge": "C3", + "app": "privatebin", + "note": "NEGATIVE control: the TO image starts and exits immediately", + "from": { + "privatebin": "privatebin/pdo:2.0.5" + }, + "to": { + "privatebin": "alpine:3.20" + }, + "verdict": "failed", + "seed_read_before": true, + "seed_read_after": false, + "healthy_after": false, + "migration_observed": null, + "abort": "starts-and-serves", + "abort_detail": null, + "engine_state_after": null, + "memory": null, + "marks": [], + "duration_s": 424.4, + "measured_at": "2026-09-23T18:47:56Z", + "evidence": "evidence/C3", + "files_changed": [], + "total_s": 491.2 +} diff --git a/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/abort-states.json b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/abort-states.json new file mode 100644 index 00000000..f7a71c94 --- /dev/null +++ b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/abort-states.json @@ -0,0 +1,8 @@ +{ + "privatebin": { + "status": "running", + "health": "healthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/compose-final.log b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/compose-final.log new file mode 100644 index 00000000..65467879 --- /dev/null +++ b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/compose-final.log @@ -0,0 +1,5 @@ +privatebin | [23-Sep-2026 20:47:25] NOTICE: fpm is running, pid 11 +privatebin | [23-Sep-2026 20:47:25] NOTICE: ready to handle connections +privatebin | 127.0.0.1 - - [23/Sep/2026:20:47:55 +0200] "GET / HTTP/1.1" 200 19657 "-" "Wget" "-" +privatebin | 172.18.0.1 - - [23/Sep/2026:20:47:55 +0200] "GET / HTTP/1.1" 200 22914 "-" "curl/8.14.1" "-" +privatebin | 172.18.0.1 - - [23/Sep/2026:20:47:55 +0200] "GET /?pasteid=a7c6f18ad9707051 HTTP/1.1" 200 310 "-" "curl/8.14.1" "-" diff --git a/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/engine-state.json b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/engine-state.json new file mode 100644 index 00000000..ec747fa4 --- /dev/null +++ b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/engine-state.json @@ -0,0 +1 @@ +null \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/files-after.json b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/files-after.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/files-after.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/files-before.json b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/files-before.json new file mode 100644 index 00000000..9e26dfee --- /dev/null +++ b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/files-before.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/migration-lines.txt b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/migration-lines.txt new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/run.log b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/run.log new file mode 100644 index 00000000..7723baf4 --- /dev/null +++ b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/run.log @@ -0,0 +1,13 @@ +[18:39:45] C3: deploying privatebin at FROM {'privatebin': 'privatebin/pdo:2.0.5'} +[18:40:16] FROM settled=True in 30.8s :: {"privatebin": {"status": "running", "health": "healthy", "restarts": 0, "exit": 0}} +[18:40:16] privatebin: seeded paste id=a7c6f18ad9707051 +[18:40:16] privatebin: readback http=200 marker_present=True +[18:40:16] C1 (seed reads back BEFORE): True +[18:40:16] C3: swapping to TO {'privatebin': 'alpine:3.20'} +[18:40:20] TO up -d rc=0 +[18:47:24] TO settled=False in 424.4s :: {"privatebin": {"status": "restarting", "health": "unhealthy", "restarts": 0, "exit": 0}} +[18:47:24] migration lines observed: 0 +[18:47:24] RESULT (seed reads back AFTER): False +[18:47:25] C3: ABORT — putting the FROM images back +[18:47:56] privatebin: readback http=200 marker_present=True +[18:47:56] ABORT: app came back in 30.8s; data present=True \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/to-full.log b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/to-full.log new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/to-states.json b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/to-states.json new file mode 100644 index 00000000..04f4a178 --- /dev/null +++ b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/to-states.json @@ -0,0 +1,8 @@ +{ + "privatebin": { + "status": "restarting", + "health": "unhealthy", + "restarts": 0, + "exit": 0 + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/verdict.json b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/verdict.json new file mode 100644 index 00000000..a7d73b9b --- /dev/null +++ b/documentation/audits/night-2026-09-23/bench-C3/evidence/C3/verdict.json @@ -0,0 +1,27 @@ +{ + "harness_version": 3, + "edge": "C3", + "app": "privatebin", + "note": "NEGATIVE control: the TO image starts and exits immediately", + "from": { + "privatebin": "privatebin/pdo:2.0.5" + }, + "to": { + "privatebin": "alpine:3.20" + }, + "verdict": "failed", + "seed_read_before": true, + "seed_read_after": false, + "healthy_after": false, + "migration_observed": null, + "abort": "starts-and-serves", + "abort_detail": null, + "engine_state_after": null, + "memory": null, + "marks": [], + "duration_s": 424.4, + "measured_at": "2026-09-23T18:47:56Z", + "evidence": "evidence/C3", + "files_changed": [], + "total_s": 491.2 +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/benchq-Q11.txt b/documentation/audits/night-2026-09-23/benchq-Q11.txt index 2798c1d7..0bdf0fa9 100644 --- a/documentation/audits/night-2026-09-23/benchq-Q11.txt +++ b/documentation/audits/night-2026-09-23/benchq-Q11.txt @@ -1 +1,3 @@ 22:01:25 START nextcloud ['nextcloud=nextcloud:34.0.4-apache'] +22:16:09 DONE nextcloud rc=0 verdict=proven read_before=True read_after=True abort=refuses marks=['files_may_change'] peaks={'nextcloud': 1.0, 'nextcloud-db': 0.366, 'nextcloud-redis': 0.281} detail='Nextcloud because the version of the data (34.0.4.1) is higher than the docker image version (34.0.1.2) and downgrading is not supported. Are you sure you have ' (884s) +22:16:09 QUEUE Q11 FINISHED diff --git a/documentation/audits/night-2026-09-23/benchq-Q12.txt b/documentation/audits/night-2026-09-23/benchq-Q12.txt new file mode 100644 index 00000000..924fe27b --- /dev/null +++ b/documentation/audits/night-2026-09-23/benchq-Q12.txt @@ -0,0 +1,4 @@ +22:38:24 START adventurelog ['adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0'] +22:39:08 START adventurelog ['adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0'] +22:56:00 DONE adventurelog rc=0 verdict=failed read_before=True read_after=False abort=starts-and-serves marks=[] peaks={} detail='None' (1012s) +22:56:00 QUEUE Q12 FINISHED diff --git a/documentation/audits/night-2026-09-23/bigdb.py b/documentation/audits/night-2026-09-23/bigdb.py new file mode 100644 index 00000000..02ebc04f --- /dev/null +++ b/documentation/audits/night-2026-09-23/bigdb.py @@ -0,0 +1,31 @@ +#!/usr/bin/env python3 +"""Make docmost the BIG database of the chaos hour, through its own API with ONE login (its login is +rate-limited: 290 of 300 per-call logins answered 401 in setup). Spaces with a 4 KB description each, +then the database size asked of PostgreSQL itself.""" +import json, os, sys, time +sys.path.insert(0, ".") +import walk as w +from chaos import load, save, sub_of +s = load(); A = s["apps"]["docmost"]["A"]; sub = sub_of("docmost") +w.login() +jar = f"{w.SC}/big.jar" +rc, code, out = w.app_curl(sub, "/api/auth/login", "-c", jar, "-H", "Content-Type: application/json", + data=json.dumps({"email": A["email"], "password": A["pw"]}), method="POST") +w.say(f"login http={code}") +ok = 0 +desc = ("chaos-hour big database filler " * 140)[:4000] +for i in range(400): + name = f"big{i:03d}{os.urandom(2).hex()}" + rc, code, out = w.app_curl(sub, "/api/spaces/create", "-b", jar, "-H", "Content-Type: application/json", + data=json.dumps({"name": name, "slug": name, "description": desc}), method="POST") + if code in ("200", "201"): + ok += 1 + elif i < 5 or i % 50 == 0: + w.say(f" {i}: http={code} {out[:120]}") +w.say(f"created {ok} spaces with one login") +size = w.guest("docker exec docmost-postgres psql -U docmost -d docmost -Atc \"select pg_size_pretty(pg_database_size('docmost')), (select count(*) from spaces)\"") +w.say(f"docmost database: {size.strip()}") +s["apps"]["docmost"]["big_extra_spaces"] = s["apps"]["docmost"].get("big_extra_spaces", 0) + ok +s["apps"]["docmost"]["db_size"] = size.strip() +save(s) +open("chaos/00-bigdb.log", "w").write("\n".join(w.LOG) + "\n") diff --git a/documentation/audits/night-2026-09-23/chaos-kills.json b/documentation/audits/night-2026-09-23/chaos-kills.json new file mode 100644 index 00000000..ca29034f --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos-kills.json @@ -0,0 +1 @@ +[1790195982.4679723, 1790197454.9976318] \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos-state.json b/documentation/audits/night-2026-09-23/chaos-state.json new file mode 100644 index 00000000..5c59d009 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos-state.json @@ -0,0 +1,88 @@ +{ + "apps": { + "docmost": { + "A": { + "email": "drill-98dd5988@gate.invalid", + "pw": "Drill-229ddff459946cb47bd6" + }, + "big_extra_spaces": 410, + "B": { + "space": "drillBcd9d57" + }, + "db_size": "11 MB|412" + }, + "adventurelog": { + "A": { + "user": "drill3b57cc", + "pw": "Drill-17e741dd879a60b119f5" + } + }, + "romm": { + "A": { + "user": "drill1182a0", + "pw": "Drill-075df1cd6ce6be24d01d" + }, + "B": null + }, + "vikunja": { + "A": { + "user": "drill976cbb", + "pw": "Drill-4d0ece299eca8fcdec71", + "title": "drill-97252f7378", + "pid": 2 + }, + "B": { + "title": "drillB-532c79dd", + "pid": 3, + "tid": 1, + "att": "drill attachment 50c2fcf095d70713" + } + }, + "navidrome": { + "A": { + "user": "drille98865", + "pw": "Drill-02eb15051ebb33619030" + } + }, + "nextcloud": { + "A": { + "uid": "drill4f7de4", + "pw": "Drill-1ede1ff8ca366cc4e6b8" + } + }, + "actualbudget": { + "A": { + "pw": "Drill-29fc431da523254ffc1f" + } + }, + "papra": { + "A": { + "email": "drill-bb835c68@gate.invalid", + "pw": "Drill-3ac01034208d27b2e316" + } + } + }, + "setup_readback": { + "docmost": { + "A": true, + "B": true + }, + "adventurelog": { + "A": true + }, + "romm": { + "A": true + }, + "vikunja": { + "A": true, + "B": true + }, + "navidrome": { + "A": true + }, + "nextcloud": { + "A": null, + "note": "no seed A recorded" + } + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos.py b/documentation/audits/night-2026-09-23/chaos.py index 2250d9db..61fc97c5 100644 --- a/documentation/audits/night-2026-09-23/chaos.py +++ b/documentation/audits/night-2026-09-23/chaos.py @@ -212,11 +212,14 @@ def drill_wait_ref(app, ref): return None -def act(action, app, rnd, note, s): - """Run the action. Returns when the app is steady (or the cap). The accident fires in a thread.""" +def act(action, app, rnd, note, s, arm=lambda: None): + """Run the action. Returns when the app is steady (or the cap). The accident fires in a thread, + ARMED at the moment of the press itself (round 2 showed why: the catalog took ~7 min to reach the + box, and an accident timed from the round's start landed before the update ever began).""" t0 = time.time() if action == "install": vals = w.deploy_values(app, sub_of(app)) + arm() code, d = w.ctl("POST", f"/api/stacks/{app}/deploy", {"values": vals}) note["act"] = f"deploy -> {code} {str(d)[:120]}" secs, st = steady(app, t0) @@ -227,10 +230,12 @@ def act(action, app, rnd, note, s): save(s) return secs, st if action == "remove": + arm() code = w.remove(app) note["act"] = f"remove -> {code}" return steady(app, t0, want_deployed=False) if action == "restore": + arm() res = w.restore(app) note["act"] = {k: res.get(k) for k in ("ok", "why", "snapshot_id", "http", "seconds", "state_after", "hold_after")} return steady(app, t0) @@ -240,8 +245,9 @@ def act(action, app, rnd, note, s): else: # fail_health / cutoff: the real image move + a probe port the app does not answer h = break_edge(app, frm, to, port, 8999 if port != 8999 else 8998) note["drill_commit"] = h - note["badge_catchup_rounds"] = drill_wait_ref(app, to.split(",")[0]) - w.sync_rescan(expect_app=app, expect_ref=to.split(",")[0]) + # For an INSTALLED app the stack dir's compose is the applied one, not the catalog's — waiting on + # it cost ~7 min in rounds 2 and 4. The badge's own input (`catalog_images`) is the right wait. + note["badge_catchup_s"] = w.sync_rescan(expect_app=app, expect_ref=to.split(",")[0], tries=60, delay=5) on_phase = None if action == "cutoff": done = {} @@ -254,6 +260,7 @@ docker run --rm -v $c:/c alpine sh -c 'rm -f /c/felhom-undo-complete; ls /c | he note["cutoff"] = out.strip() w.say(" >>> finished-marker removed from one undo copy:\n" + out) return False + arm() phases, st = live.press(app, poll=0.5, on_phase=on_phase) note["phases"] = phases return steady(app, t0) @@ -352,16 +359,22 @@ def run_round(n): note["before"] = household(app) if w.stack(app).get("deployed") else {"state": "not deployed"} if n in SECOND: # the second app gets its good step in the drill catalog before the round note["second_drill_commit"] = w.drill_bump(SECOND[n], "mariadb:11.4", "mariadb:11.8") - note["second_template_rounds"] = drill_wait_ref(SECOND[n], "mariadb:11.8") + note["second_badge_catchup_s"] = w.sync_rescan(expect_app=SECOND[n], expect_ref="mariadb:11.8", tries=60, delay=5) t0 = time.time() timer = threading.Timer(sched["delay_s"], accident, args=(sched["accident"], n, app, note)) - timer.start() + + def arm(): + note["armed_at"] = time.strftime("%H:%M:%S") + timer.start() try: - secs, st = act(sched["action"], app, n, note, s) + secs, st = act(sched["action"], app, n, note, s, arm) except Exception as e: note["act_exception"] = repr(e) secs, st = None, {} - timer.join() + if timer.is_alive() or note.get("armed_at"): + timer.join() + else: + note["accident"] = "NOT FIRED — the action ended before it was armed" if sched["accident"] in ("power_cut", "docker_restart", "controller_kill"): note["controller_back_s"] = wait_controller() secs2, st = steady(app, t0, want_deployed=(sched["action"] != "remove")) @@ -379,6 +392,10 @@ def run_round(n): note["events"] = events_since(t_iso) note["box_log"] = box_log(t_iso, app) note["undo_copies_left"] = w.guest(f"docker volume ls -q --filter label=felhom.undo-copy-of={app} | wc -l").strip() + # the WHOLE controller log of the round, off the machine now (R-320): round 7's power cut took the + # container's earlier log with it, and round 1's app_start_failed could no longer be tied to an app + open(os.path.join(EVD, f"round-{n:02d}-controller.log"), "w").write( + w.guest(f"docker logs --since {t_iso} felhom-controller 2>&1 | grep -vE 'auth: valid session|router.go:81' | tail -3000")) repair_drill(sched["action"], app) json.dump(note, open(os.path.join(EVD, f"round-{n:02d}.json"), "w"), indent=2, ensure_ascii=False) open(os.path.join(EVD, f"round-{n:02d}.log"), "w").write("\n".join(w.LOG) + "\n") diff --git a/documentation/audits/night-2026-09-23/chaos/00-bigdb.log b/documentation/audits/night-2026-09-23/chaos/00-bigdb.log new file mode 100644 index 00000000..d40217f6 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/00-bigdb.log @@ -0,0 +1,3 @@ +22:26:32 login http=200 +22:26:48 created 400 spaces with one login +22:26:51 docmost database: 11 MB|412 diff --git a/documentation/audits/night-2026-09-23/chaos/00-nextcloud-rebuild.log b/documentation/audits/night-2026-09-23/chaos/00-nextcloud-rebuild.log new file mode 100644 index 00000000..49528e2c --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/00-nextcloud-rebuild.log @@ -0,0 +1,14 @@ +22:27:04 [X] stop -> 200 {'ok': True, 'message': 'Stack nextcloud stop completed'} +22:27:09 [X] remove (with drive data) -> 409 {'ok': False, 'error': 'A(z) /mnt/felhom-drives/scratch_hdd/userdata/nextcloud tárhely jelenleg nem elérhető — az alkalmazás nem távolítható el, amíg a meghajtó +22:27:09 [X] refused because the drive path cannot be resolved (R-442, fail-closed and right) — removing the app and KEEPING the drive data instead +22:27:36 [X] remove (keeping drive data) -> 200 {'ok': True, 'data': {'removed': 'nextcloud', 'volumes_removed': ['nextcloud_nextcloud_db_data', 'nextcloud_nextcloud_html', 'nextcloud_nextcloud_redis_data'], +22:27:45 [X] after remove: deployed=False leftovers='/opt/docker/stacks/nextcloud' +22:27:48 calibre-web documents downloads emby immich jellyfin komga media navidrome paperless-ngx plex radarr romm roms sonarr +22:27:51 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/nextcloud'] +22:27:51 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['NEXTCLOUD_ADMIN_PASSWORD', 'HDD_PATH'] +22:27:51 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:28:36 [1] deployed, controller state=running, pinned={'nextcloud': 'nextcloud:34.0.1-apache', 'nextcloud-db': 'mariadb:12.3', 'nextcloud-redis': 'redis:7-alpine'} +22:28:45 nextcloud: occ user:add :: The account "drill4f7de4" was created successfully Display name set to "drill4f7de4" +22:28:48 nextcloud: seeded user drill4f7de4 +22:28:55 nextcloud: readback of the seeded user found=True +22:28:55 nextcloud seed A: True; readback: True diff --git a/documentation/audits/night-2026-09-23/chaos/00-nextcloud-reinstall-over-kept-data.txt b/documentation/audits/night-2026-09-23/chaos/00-nextcloud-reinstall-over-kept-data.txt new file mode 100644 index 00000000..7d3ee296 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/00-nextcloud-reinstall-over-kept-data.txt @@ -0,0 +1,22 @@ +Installing with MySQL database +=> Searching for hook scripts (*.sh) to run, located in the folder "/docker-entrypoint-hooks.d/pre-installation" +==> Skipped: the "pre-installation" folder is empty (or does not exist) +Starting nextcloud installation +Login is invalid because files already exist for this user +Retrying install... +Login is invalid because files already exist for this user +Retrying install... +Login is invalid because files already exist for this user +Retrying install... +Login is invalid because files already exist for this user +Retrying install... +total 141572 +drwxrwx--- 5 www-data www-data 4096 Sep 23 20:15 . +drwxr-sr-x 3 root root 4096 Sep 23 18:55 .. +-rw-rw-r-- 1 www-data www-data 542 Sep 23 20:15 .htaccess +-rw-rw-r-- 1 www-data www-data 52 Sep 23 18:57 .ncdata +drwxr-sr-x 3 www-data www-data 4096 Sep 23 18:55 admin +drwxr-sr-x 6 www-data www-data 4096 Sep 23 18:57 appdata_oclterted7nn +drwxr-xr-x 2 www-data www-data 4096 Sep 23 18:56 drillf006e5 +-rw-rw-r-- 1 www-data www-data 0 Sep 23 20:15 index.html +-rw-r----- 1 www-data www-data 144934959 Sep 23 18:58 nextcloud.log diff --git a/documentation/audits/night-2026-09-23/chaos/00-setup-run.txt b/documentation/audits/night-2026-09-23/chaos/00-setup-run.txt new file mode 100644 index 00000000..ab1cb4f5 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/00-setup-run.txt @@ -0,0 +1,352 @@ +22:06:47 == setup docmost (PostgreSQL + the BIG database (300 extra spaces)) +22:07:03 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:08:49 [1] deployed, controller state=running, pinned={'docmost': 'docmost/docmost:0.95.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'} +22:08:49 docmost: /api/auth/setup http=200 rc=0 +22:08:49 == setup adventurelog (PostgreSQL (PostGIS)) +22:08:53 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:10:38 [1] deployed, controller state=running, pinned={'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.12.1', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine'} +22:10:44 adventurelog: createsuperuser :: Superuser created successfully. +22:10:50 adventurelog: seeded superuser drill3b57cc +22:10:50 == setup romm (MariaDB) +22:10:56 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/romm'] +22:10:56 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH'] +22:10:56 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:11:56 [1] deployed, controller state=running, pinned={'romm': 'rommapp/romm:5.3.0', 'romm-db': 'mariadb:11.4', 'romm-redis': 'redis:7-alpine'} +22:12:09 romm: POST /api/users http=201 +22:12:09 == setup vikunja (volume-only (SQLite + files in volumes)) +22:12:13 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:12:18 [1] deployed, controller state=running, pinned={'vikunja': 'vikunja/vikunja:2.5.0'} +22:12:18 vikunja: register http=200 +22:12:18 vikunja: create project http=201 +22:12:18 == setup navidrome (volume-only (SQLite) + a drive folder) +22:12:24 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/navidrome'] +22:12:24 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH'] +22:12:24 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:12:40 [1] deployed, controller state=running, pinned={'navidrome': 'deluan/navidrome:0.64.0'} +22:12:40 navidrome: createAdmin http=200 +22:12:40 == setup nextcloud (FILE-LEG (user files on the drive) + MariaDB) +22:12:46 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/nextcloud'] +22:12:46 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['NEXTCLOUD_ADMIN_PASSWORD', 'HDD_PATH'] +22:12:46 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:14:37 [1] deployed, controller state=unhealthy, pinned={'nextcloud': 'nextcloud:34.0.1-apache', 'nextcloud-db': 'mariadb:12.3', 'nextcloud-redis': 'redis:7-alpine'} +22:14:37 [1] NOTE: the controller's own state is 'unhealthy', not 'running' — recorded, not treated as a failure; the fixture's front-door wait is the real gate +22:23:18 nextcloud: occ status never said installed: Nextcloud is not installed - only a limited number of commands are available - installed: false - version: 34.0.1.2 - versionstring: 34.0.1 - edition: - mainten +22:23:19 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-df84-7c98-a311-52eff16d1a5b","name":"drillB90903b","description":"","slug":"drillb90903b","logo":null,"visibility":"private","defaultRol +22:23:19 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e0d7-7610-887d-8149cf47bac5","name":"drillB63ccd4","description":"","slug":"drillb63ccd4","logo":null,"visibility":"private","defaultRol +22:23:19 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e21e-7b48-820b-610638e8d404","name":"drillBd30a5e","description":"","slug":"drillbd30a5e","logo":null,"visibility":"private","defaultRol +22:23:20 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e35c-711f-9e73-76e5c0fc84a1","name":"drillB204493","description":"","slug":"drillb204493","logo":null,"visibility":"private","defaultRol +22:23:20 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e49a-7be3-9188-edb2ce015686","name":"drillBe99e15","description":"","slug":"drillbe99e15","logo":null,"visibility":"private","defaultRol +22:23:20 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e5db-7ad5-9588-6c977acf5dba","name":"drillBffb74c","description":"","slug":"drillbffb74c","logo":null,"visibility":"private","defaultRol +22:23:21 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e71e-78fa-ba8d-5da49b9c0cc7","name":"drillB48b432","description":"","slug":"drillb48b432","logo":null,"visibility":"private","defaultRol +22:23:21 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e85a-7c62-8cd2-a63cf04c8b29","name":"drillBb3fc23","description":"","slug":"drillbb3fc23","logo":null,"visibility":"private","defaultRol +22:23:21 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e996-7e8b-bbb9-d73fd0cac22c","name":"drillBdc4bf4","description":"","slug":"drillbdc4bf4","logo":null,"visibility":"private","defaultRol +22:23:22 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-ead5-73aa-811f-3690fdfe0fc6","name":"drillBcdd5bd","description":"","slug":"drillbcdd5bd","logo":null,"visibility":"private","defaultRol +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:37 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:37 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:37 docmost: 10 extra spaces created (the big database) +22:23:37 == one whole-box backup on 9202 (the restore rounds need a copy) +22:23:37 backup -> 200 {'ok': True, 'message': 'Mentés elindítva'} +22:25:12 backup finished after 95s: {'db_dump': {'count': 5, 'duration': '1m30.406632262s', 'last_run': '2026-09-23T20:25:07.503991541Z', 'success': True}, 'enabled': True, 'running': False} +22:25:13 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cff1-9c46-7b33-802f-cd1da6c34802","name":"drillBcd9d57","description":"","slug":"drillbcd9d57","logo":null,"visibility":"private","defaultRol +22:25:13 seed B for docmost (after the backup): True +22:25:13 romm seed B: POST /api/users (as the admin) http=404 404 page not found + +22:25:13 seed B for romm (after the backup): False +22:25:13 vikunja seed B: project http=200 task=1 attachment upload http=200 {"errors":null,"success":[{"id":1,"task_id":1,"created_by":{"id":1,"name":"","username":"drill976cbb","created":"2026-09 +22:25:13 seed B for vikunja (after the backup): True +22:25:13 docmost: login as the seeded user http=200 ok=True +22:25:14 docmost B: /api/spaces http=200 seeded-space-listed=True (negative control listed=False) +22:25:25 adventurelog: readback of the seeded account found=True +22:25:51 romm: login as the seeded user http=200 ok=True +22:25:52 vikunja: readback of the seeded project http=200 ok=True +22:25:52 vikunja B: project readback=True attachment content readback=True +22:25:52 navidrome: login as the seeded user http=200 ok=True +22:25:52 setup readback: {"docmost": {"A": true, "B": true}, "adventurelog": {"A": true}, "romm": {"A": true}, "vikunja": {"A": true, "B": true}, "navidrome": {"A": true}, "nextcloud": {"A": null, "note": "no seed A recorded"}} diff --git a/documentation/audits/night-2026-09-23/chaos/00-setup.log b/documentation/audits/night-2026-09-23/chaos/00-setup.log new file mode 100644 index 00000000..ab1cb4f5 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/00-setup.log @@ -0,0 +1,352 @@ +22:06:47 == setup docmost (PostgreSQL + the BIG database (300 extra spaces)) +22:07:03 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:08:49 [1] deployed, controller state=running, pinned={'docmost': 'docmost/docmost:0.95.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'} +22:08:49 docmost: /api/auth/setup http=200 rc=0 +22:08:49 == setup adventurelog (PostgreSQL (PostGIS)) +22:08:53 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:10:38 [1] deployed, controller state=running, pinned={'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.12.1', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine'} +22:10:44 adventurelog: createsuperuser :: Superuser created successfully. +22:10:50 adventurelog: seeded superuser drill3b57cc +22:10:50 == setup romm (MariaDB) +22:10:56 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/romm'] +22:10:56 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH'] +22:10:56 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:11:56 [1] deployed, controller state=running, pinned={'romm': 'rommapp/romm:5.3.0', 'romm-db': 'mariadb:11.4', 'romm-redis': 'redis:7-alpine'} +22:12:09 romm: POST /api/users http=201 +22:12:09 == setup vikunja (volume-only (SQLite + files in volumes)) +22:12:13 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:12:18 [1] deployed, controller state=running, pinned={'vikunja': 'vikunja/vikunja:2.5.0'} +22:12:18 vikunja: register http=200 +22:12:18 vikunja: create project http=201 +22:12:18 == setup navidrome (volume-only (SQLite) + a drive folder) +22:12:24 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/navidrome'] +22:12:24 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH'] +22:12:24 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:12:40 [1] deployed, controller state=running, pinned={'navidrome': 'deluan/navidrome:0.64.0'} +22:12:40 navidrome: createAdmin http=200 +22:12:40 == setup nextcloud (FILE-LEG (user files on the drive) + MariaDB) +22:12:46 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/nextcloud'] +22:12:46 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['NEXTCLOUD_ADMIN_PASSWORD', 'HDD_PATH'] +22:12:46 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +22:14:37 [1] deployed, controller state=unhealthy, pinned={'nextcloud': 'nextcloud:34.0.1-apache', 'nextcloud-db': 'mariadb:12.3', 'nextcloud-redis': 'redis:7-alpine'} +22:14:37 [1] NOTE: the controller's own state is 'unhealthy', not 'running' — recorded, not treated as a failure; the fixture's front-door wait is the real gate +22:23:18 nextcloud: occ status never said installed: Nextcloud is not installed - only a limited number of commands are available - installed: false - version: 34.0.1.2 - versionstring: 34.0.1 - edition: - mainten +22:23:19 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-df84-7c98-a311-52eff16d1a5b","name":"drillB90903b","description":"","slug":"drillb90903b","logo":null,"visibility":"private","defaultRol +22:23:19 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e0d7-7610-887d-8149cf47bac5","name":"drillB63ccd4","description":"","slug":"drillb63ccd4","logo":null,"visibility":"private","defaultRol +22:23:19 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e21e-7b48-820b-610638e8d404","name":"drillBd30a5e","description":"","slug":"drillbd30a5e","logo":null,"visibility":"private","defaultRol +22:23:20 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e35c-711f-9e73-76e5c0fc84a1","name":"drillB204493","description":"","slug":"drillb204493","logo":null,"visibility":"private","defaultRol +22:23:20 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e49a-7be3-9188-edb2ce015686","name":"drillBe99e15","description":"","slug":"drillbe99e15","logo":null,"visibility":"private","defaultRol +22:23:20 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e5db-7ad5-9588-6c977acf5dba","name":"drillBffb74c","description":"","slug":"drillbffb74c","logo":null,"visibility":"private","defaultRol +22:23:21 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e71e-78fa-ba8d-5da49b9c0cc7","name":"drillB48b432","description":"","slug":"drillb48b432","logo":null,"visibility":"private","defaultRol +22:23:21 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e85a-7c62-8cd2-a63cf04c8b29","name":"drillBb3fc23","description":"","slug":"drillbb3fc23","logo":null,"visibility":"private","defaultRol +22:23:21 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-e996-7e8b-bbb9-d73fd0cac22c","name":"drillBdc4bf4","description":"","slug":"drillbdc4bf4","logo":null,"visibility":"private","defaultRol +22:23:22 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cfef-ead5-73aa-811f-3690fdfe0fc6","name":"drillBcdd5bd","description":"","slug":"drillbcdd5bd","logo":null,"visibility":"private","defaultRol +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:22 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:23 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:24 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:25 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:26 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:27 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:28 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:29 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:30 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:31 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:32 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:33 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:34 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:35 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:36 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:37 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:37 docmost seed B: /api/spaces/create http=401 {"message":"Unauthorized","statusCode":401} +22:23:37 docmost: 10 extra spaces created (the big database) +22:23:37 == one whole-box backup on 9202 (the restore rounds need a copy) +22:23:37 backup -> 200 {'ok': True, 'message': 'Mentés elindítva'} +22:25:12 backup finished after 95s: {'db_dump': {'count': 5, 'duration': '1m30.406632262s', 'last_run': '2026-09-23T20:25:07.503991541Z', 'success': True}, 'enabled': True, 'running': False} +22:25:13 docmost seed B: /api/spaces/create http=200 {"data":{"id":"01a0cff1-9c46-7b33-802f-cd1da6c34802","name":"drillBcd9d57","description":"","slug":"drillbcd9d57","logo":null,"visibility":"private","defaultRol +22:25:13 seed B for docmost (after the backup): True +22:25:13 romm seed B: POST /api/users (as the admin) http=404 404 page not found + +22:25:13 seed B for romm (after the backup): False +22:25:13 vikunja seed B: project http=200 task=1 attachment upload http=200 {"errors":null,"success":[{"id":1,"task_id":1,"created_by":{"id":1,"name":"","username":"drill976cbb","created":"2026-09 +22:25:13 seed B for vikunja (after the backup): True +22:25:13 docmost: login as the seeded user http=200 ok=True +22:25:14 docmost B: /api/spaces http=200 seeded-space-listed=True (negative control listed=False) +22:25:25 adventurelog: readback of the seeded account found=True +22:25:51 romm: login as the seeded user http=200 ok=True +22:25:52 vikunja: readback of the seeded project http=200 ok=True +22:25:52 vikunja B: project readback=True attachment content readback=True +22:25:52 navidrome: login as the seeded user http=200 ok=True +22:25:52 setup readback: {"docmost": {"A": true, "B": true}, "adventurelog": {"A": true}, "romm": {"A": true}, "vikunja": {"A": true, "B": true}, "navidrome": {"A": true}, "nextcloud": {"A": null, "note": "no seed A recorded"}} diff --git a/documentation/audits/night-2026-09-23/chaos/09-volumes-unlabeled-after-restore.txt b/documentation/audits/night-2026-09-23/chaos/09-volumes-unlabeled-after-restore.txt new file mode 100644 index 00000000..ea674129 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/09-volumes-unlabeled-after-restore.txt @@ -0,0 +1,6 @@ +docmost: all=3 labeled=0 +navidrome: all=2 labeled=0 +vikunja: all=2 labeled=0 +romm: all=3 labeled=3 +nextcloud: all=3 labeled=3 +adventurelog: all=2 labeled=2 diff --git a/documentation/audits/night-2026-09-23/chaos/STOP-round08-held-not-lost.txt b/documentation/audits/night-2026-09-23/chaos/STOP-round08-held-not-lost.txt new file mode 100644 index 00000000..b2ae5b8a --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/STOP-round08-held-not-lost.txt @@ -0,0 +1,2 @@ +round 8: seed A did not read back — the stop rule +Cleared 23:03: round 8's app was HELD (stopped by design after the cut-off undo copy), not lost — the household's restore brought it back and seed A read back (round-08-recovery.*). diff --git a/documentation/audits/night-2026-09-23/chaos/round-01-run.txt b/documentation/audits/night-2026-09-23/chaos/round-01-run.txt new file mode 100644 index 00000000..77d1393c --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-01-run.txt @@ -0,0 +1,8 @@ +22:29:10 ==== ROUND 1: install on actualbudget x none at +11s (2026-09-23T20:29:10Z) +22:29:11 + 0.2s actualbudget: deployed=True state=not_deployed updating=False phase=None hold='None' +22:29:17 + 6.3s actualbudget: deployed=True state=deploying updating=False phase=None hold='None' +22:29:23 + 12.4s actualbudget: deployed=True state=starting updating=False phase=None hold='None' +22:29:37 + 26.6s actualbudget: deployed=True state=running updating=False phase=None hold='None' +22:29:37 actualbudget: /account/bootstrap http=200 :: {"status":"ok","data":{"token":"cbe15b58-92da-4f64-8a4d-04ac8679a894"}} +22:29:41 actualbudget: login with the seeded password http=200 ok=True +22:29:44 ==== ROUND 1 END: steady in 26.6s; readback={'A': True}; events=3 diff --git a/documentation/audits/night-2026-09-23/chaos/round-01.json b/documentation/audits/night-2026-09-23/chaos/round-01.json new file mode 100644 index 00000000..9885c8f7 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-01.json @@ -0,0 +1,70 @@ +{ + "round": 1, + "action": "install", + "accident": "none", + "delay_s": 11, + "app": "actualbudget", + "started": "2026-09-23T20:29:10Z", + "before": { + "state": "not deployed" + }, + "act": "deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}", + "time_to_steady_s": 26.6, + "after": { + "badges": { + "hu": [], + "en": [] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "observables": { + "pinned_images": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "installed_images": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: actualbudget/actual-server:26.9.0" + ], + "docker_inspect": [ + "actualbudget actualbudget/actual-server:26.9.0 running=true restarts=0" + ] + }, + "readback": { + "A": true + }, + "events": [ + "2026-09-23T20:29:10Z app_deploy_started (info)", + "2026-09-23T20:29:16Z app_start_failed (warning)", + "2026-09-23T20:29:22Z app_deployed (info)" + ], + "box_log": [ + "2026-09-23T20:29:10Z INFO [api] Deploy requested for stack: actualbudget", + "2026-09-23T20:29:10Z INFO [stacks] Deploying stack actualbudget with 2 env vars: [DOMAIN, SUBDOMAIN]", + "2026-09-23T20:29:10Z INFO [stacks] Deploying stack actualbudget — checking 1 images...", + "2026-09-23T20:29:21Z INFO [stacks] Stack actualbudget deployed successfully (took 10.9s)", + "2026-09-23T20:29:22Z INFO [stacks] installed-images actualbudget: recorded 1 service(s) (actualbudget=actualbudget/actual-server:26.9.0 (sha256:552beab3dec8…))", + "2026-09-23T20:29:22Z INFO [stacks] pin actualbudget: actualbudget=actualbudget/actual-server:26.9.0", + "2026-09-23T20:29:25Z INFO [stacks] Stack actualbudget post-start status:", + "2026-09-23T20:29:25Z INFO [stacks] actualbudget actualbudget/actual-server:26.9.0 running Up 3 seconds (health: starting)" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-01.log b/documentation/audits/night-2026-09-23/chaos/round-01.log new file mode 100644 index 00000000..f03e4020 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-01.log @@ -0,0 +1,7 @@ +22:29:10 ==== ROUND 1: install on actualbudget x none at +11s (2026-09-23T20:29:10Z) +22:29:11 + 0.2s actualbudget: deployed=True state=not_deployed updating=False phase=None hold='None' +22:29:17 + 6.3s actualbudget: deployed=True state=deploying updating=False phase=None hold='None' +22:29:23 + 12.4s actualbudget: deployed=True state=starting updating=False phase=None hold='None' +22:29:37 + 26.6s actualbudget: deployed=True state=running updating=False phase=None hold='None' +22:29:37 actualbudget: /account/bootstrap http=200 :: {"status":"ok","data":{"token":"cbe15b58-92da-4f64-8a4d-04ac8679a894"}} +22:29:41 actualbudget: login with the seeded password http=200 ok=True diff --git a/documentation/audits/night-2026-09-23/chaos/round-02-run.txt b/documentation/audits/night-2026-09-23/chaos/round-02-run.txt new file mode 100644 index 00000000..28c41c5c --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-02-run.txt @@ -0,0 +1,16 @@ +22:29:44 ==== ROUND 2: good_update on docmost x box_backup at +46s (2026-09-23T20:29:44Z) +22:29:45 [5] drill commit b5b7f194a6b8: docmost docmost/docmost:0.95.0 -> docmost/docmost:0.96.0 (push rc=0) +22:30:30 >>> ACCIDENT box_backup at 22:30:30 +22:30:30 >>> POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'} at 22:30:30 +22:37:14 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +22:37:14 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +22:37:17 + 2.1s phase=pulling label=Új verzió letöltése… +22:38:31 + 76.6s phase=copying label=Az adatok másolása a frissítés előtt… +22:38:36 + 81.4s phase=starting label=Indítás az új verzióval… +22:38:47 + 92.4s phase=verifying label=Működés ellenőrzése… +22:39:03 + 108.1s phase=done label=Frissítve +22:39:03 END phase=done err=None hold=None +22:39:03 + 558.3s docmost: deployed=True state=running updating=False phase=done hold='None' +22:39:07 docmost: login as the seeded user http=200 ok=True +22:39:07 docmost B: /api/spaces http=200 seeded-space-listed=False (negative control listed=False) +22:39:10 ==== ROUND 2 END: steady in 558.3s; readback={'A': True, 'B': False}; events=0 diff --git a/documentation/audits/night-2026-09-23/chaos/round-02.json b/documentation/audits/night-2026-09-23/chaos/round-02.json new file mode 100644 index 00000000..3b19baf7 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-02.json @@ -0,0 +1,199 @@ +{ + "round": 2, + "action": "good_update", + "accident": "POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'} at 22:30:30", + "delay_s": 46, + "app": "docmost", + "started": "2026-09-23T20:29:44Z", + "before": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "drill_commit": "b5b7f194a6b8", + "badge_catchup_rounds": null, + "phases": [ + [ + 0.0, + "safety-dump", + "Adatbázis pillanatkép…" + ], + [ + 2.1, + "pulling", + "Új verzió letöltése…" + ], + [ + 76.6, + "copying", + "Az adatok másolása a frissítés előtt…" + ], + [ + 81.4, + "starting", + "Indítás az új verzióval…" + ], + [ + 92.4, + "verifying", + "Működés ellenőrzése…" + ], + [ + 108.1, + "done", + "Frissítve" + ] + ], + "time_to_steady_s": 558.3, + "after": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": "done", + "hold_reason": null, + "update_error": null + } + }, + "observables": { + "pinned_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "installed_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "catalog_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "live_compose_image_lines": [ + "image: docmost/docmost:0.96.0", + "image: postgres:16-alpine", + "image: redis:7-alpine" + ], + "docker_inspect": [ + "docmost docmost/docmost:0.96.0 running=true restarts=0", + "docmost-postgres postgres:16-alpine running=true restarts=0", + "docmost-redis redis:7-alpine running=true restarts=0" + ] + }, + "readback": { + "A": true, + "B": false + }, + "events": [], + "box_log": [ + "2026-09-23T20:34:12Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:34:24Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:34:36Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:34:46Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:34:49Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:35:01Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:35:13Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:35:25Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:35:37Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:35:50Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:36:02Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:36:14Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:36:26Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:36:39Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:36:46Z INFO [scheduler] Running job: backup-cache", + "2026-09-23T20:36:46Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:36:46Z INFO [backup] Found 5 DB dump files across drives", + "2026-09-23T20:36:46Z INFO [backup] Discovered 5 databases", + "2026-09-23T20:36:46Z INFO [backup] Backup status cache refreshed", + "2026-09-23T20:36:46Z INFO [scheduler] Job backup-cache completed (took 279ms)", + "2026-09-23T20:36:51Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:37:03Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:37:12Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:37:14Z INFO [api] update requested for stack: docmost", + "2026-09-23T20:37:14Z INFO [stacks] update docmost: accepted — guarded update started", + "2026-09-23T20:37:14Z INFO [stacks] update docmost: phase checking", + "2026-09-23T20:37:14Z INFO [stacks] update docmost: precondition met — Tier 1 (own recovery unit) copy from 2026-09-23T20:31:46Z (5m0s old, limit 24h0m0s)", + "2026-09-23T20:37:14Z INFO [stacks] update docmost: phase safety-dump", + "2026-09-23T20:37:15Z INFO [backup] Discovered 5 databases", + "2026-09-23T20:37:15Z INFO [backup] DB dump: docmost-postgres → pre-restore-20260923T203715Z-docmost-postgres.sql (1.8 MB, 331ms, 42 tables)", + "2026-09-23T20:37:15Z INFO [offbox] docmost: pre-restore safety dump written → pre-restore-20260923T203715Z-docmost-postgres.sql (1.8 MB)", + "2026-09-23T20:37:15Z INFO [backup] update safety dump for docmost: 1 file(s) [/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260923T203715Z-docmost-postgres.sql]", + "2026-09-23T20:37:15Z INFO [stacks] update docmost: safety dump done (1 file(s)) [/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/pre-restore-20260923T203715Z-docmost-postgres.sql]", + "2026-09-23T20:37:16Z INFO [stacks] update docmost: the undo copy will hold 3 named volume(s), 65.5 MiB", + "2026-09-23T20:37:16Z INFO [stacks] update docmost: phase pinning", + "2026-09-23T20:37:16Z INFO [stacks] update docmost: pin advanced to the catalog's current definition (docmost=docmost/docmost:0.96.0, docmost-postgres=postgres:16-alpine, docmost-redis=redis:7-alpine)", + "2026-09-23T20:37:16Z INFO [stacks] update docmost: phase pulling", + "2026-09-23T20:37:46Z INFO [deadapp] check alive: 260 scans since boot, 10 deployed app(s) evaluated, 1 currently down", + "2026-09-23T20:38:31Z INFO [stacks] update docmost: phase copying", + "2026-09-23T20:38:33Z INFO [stacks] update docmost: phase copying", + "2026-09-23T20:38:35Z INFO [stacks] update docmost: copied docmost_docmost_postgres_data → docmost_docmost_postgres_data.pre-update-20260923T203833Z in 1.612s", + "2026-09-23T20:38:35Z INFO [stacks] update docmost: phase copying", + "2026-09-23T20:38:35Z INFO [stacks] update docmost: copied docmost_docmost_redis_data → docmost_docmost_redis_data.pre-update-20260923T203833Z in 473ms", + "2026-09-23T20:38:35Z INFO [stacks] update docmost: phase copying", + "2026-09-23T20:38:35Z INFO [stacks] update docmost: copied docmost_docmost_storage → docmost_docmost_storage.pre-update-20260923T203833Z in 443ms", + "2026-09-23T20:38:35Z INFO [stacks] update docmost: phase starting", + "2026-09-23T20:38:46Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:38:47Z INFO [stacks] update docmost: phase verifying", + "2026-09-23T20:39:02Z INFO [stacks] update docmost: healthy after 15s (the app's health check passed)", + "2026-09-23T20:39:02Z INFO [stacks] installed-images docmost: recorded 3 service(s) (docmost=docmost/docmost:0.96.0 (sha256:b56947fcfd08…), docmost-postgres=postgres:16-alpine (sha256:721873c34ceb…), docmost-redis=redis:7-alpine (sha256:858f009f9709…))", + "2026-09-23T20:39:02Z INFO [stacks] update docmost: DONE in 1m48s" + ], + "undo_copies_left": "0", + "readback_B_reread": { + "value": true, + "why": "the first reading asked page 1 only (20 of 412 spaces, sorted by name); B sat pages deep. Row present in PostgreSQL (drillBcd9d57, 22:25:13). Fixture fixed to page through; this is the re-read through the app." + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-02.log b/documentation/audits/night-2026-09-23/chaos/round-02.log new file mode 100644 index 00000000..91fb9357 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-02.log @@ -0,0 +1,15 @@ +22:29:44 ==== ROUND 2: good_update on docmost x box_backup at +46s (2026-09-23T20:29:44Z) +22:29:45 [5] drill commit b5b7f194a6b8: docmost docmost/docmost:0.95.0 -> docmost/docmost:0.96.0 (push rc=0) +22:30:30 >>> ACCIDENT box_backup at 22:30:30 +22:30:30 >>> POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'} at 22:30:30 +22:37:14 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +22:37:14 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +22:37:17 + 2.1s phase=pulling label=Új verzió letöltése… +22:38:31 + 76.6s phase=copying label=Az adatok másolása a frissítés előtt… +22:38:36 + 81.4s phase=starting label=Indítás az új verzióval… +22:38:47 + 92.4s phase=verifying label=Működés ellenőrzése… +22:39:03 + 108.1s phase=done label=Frissítve +22:39:03 END phase=done err=None hold=None +22:39:03 + 558.3s docmost: deployed=True state=running updating=False phase=done hold='None' +22:39:07 docmost: login as the seeded user http=200 ok=True +22:39:07 docmost B: /api/spaces http=200 seeded-space-listed=False (negative control listed=False) diff --git a/documentation/audits/night-2026-09-23/chaos/round-03-run.txt b/documentation/audits/night-2026-09-23/chaos/round-03-run.txt new file mode 100644 index 00000000..817057a0 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-03-run.txt @@ -0,0 +1,13 @@ +22:39:10 ==== ROUND 3: restore on vikunja x controller_kill at +28s (2026-09-23T20:39:10Z) +22:39:11 [R] restoring vikunja from snapshot 'helyi' (of 1 offered) +22:39:11 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +22:39:11 + 0.0s restore (True, None, None) +22:39:17 + 6.1s restore (False, None, None) +22:39:17 [R] after restore: state=running hold=None phase=None +22:39:20 + 9.3s vikunja: deployed=True state=running updating=False phase=None hold='None' +22:39:39 >>> ACCIDENT controller_kill at 22:39:39 +22:39:42 >>> kill -9 of the controller's PID: killed 11644 at 22:39:39 +22:39:42 + 31.5s vikunja: deployed=True state=running updating=False phase=None hold='None' +22:39:46 vikunja: readback of the seeded project http=200 ok=True +22:39:46 vikunja B: project readback=True attachment content readback=True +22:39:49 ==== ROUND 3 END: steady in 31.5s; readback={'A': True, 'B': True}; events=0 diff --git a/documentation/audits/night-2026-09-23/chaos/round-03.json b/documentation/audits/night-2026-09-23/chaos/round-03.json new file mode 100644 index 00000000..ae0ae5ca --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-03.json @@ -0,0 +1,132 @@ +{ + "round": 3, + "action": "restore", + "accident": "kill -9 of the controller's PID: killed 11644 at 22:39:39", + "delay_s": 28, + "app": "vikunja", + "started": "2026-09-23T20:39:10Z", + "before": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "act": { + "ok": true, + "why": null, + "snapshot_id": "helyi", + "http": "HTTP/2 302", + "seconds": 6.1, + "state_after": "running", + "hold_after": null + }, + "controller_back_s": 0.2, + "time_to_steady_s": 31.5, + "after": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "observables": { + "pinned_images": { + "vikunja": "vikunja/vikunja:2.5.0" + }, + "installed_images": { + "vikunja": "vikunja/vikunja:2.5.0" + }, + "catalog_images": { + "vikunja": "vikunja/vikunja:2.5.0" + }, + "live_compose_image_lines": [ + "image: vikunja/vikunja:2.5.0" + ], + "docker_inspect": [ + "vikunja vikunja/vikunja:2.5.0 running=true restarts=0" + ] + }, + "readback": { + "A": true, + "B": true + }, + "events": [], + "box_log": [ + "2026-09-23T20:39:11Z WARN [web] Restore requested (async): stack=vikunja, snapshot=helyi from 172.18.0.2:43380", + "2026-09-23T20:39:11Z INFO [backup] Restoring vikunja from recovery unit /mnt/sys_drive/felhom-data/backups/primary/vikunja: images=1, secrets recovered=1/1, data_keys=0", + "2026-09-23T20:39:11Z INFO [stacks] Stopping stack: vikunja", + "2026-09-23T20:39:11Z INFO [stacks] Stack vikunja stopped successfully (took 0.3s)", + "2026-09-23T20:39:11Z INFO [backup] Restoring Docker volume vikunja_vikunja_data for vikunja", + "2026-09-23T20:39:11Z INFO [backup] Restoring Docker volume vikunja_vikunja_db for vikunja", + "2026-09-23T20:39:12Z INFO [backup] Restored 2 Docker volume(s) for vikunja", + "2026-09-23T20:39:12Z INFO [stacks] Redeploying vikunja from recovery unit with 3 env vars", + "2026-09-23T20:39:12Z INFO [stacks] pin vikunja: vikunja=vikunja/vikunja:2.5.0", + "2026-09-23T20:39:12Z INFO [stacks] Starting stack: vikunja", + "2026-09-23T20:39:12Z INFO [stacks] Stack vikunja started successfully (took 0.4s)", + "2026-09-23T20:39:13Z INFO [stacks] installed-images vikunja: recorded 1 service(s) (vikunja=vikunja/vikunja:2.5.0 (sha256:22df4c1bc884…))", + "2026-09-23T20:39:41Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:39:42Z INFO [stacks] desired state for vikunja recorded as \"running\" (was \"\")", + "2026-09-23T20:39:42Z INFO [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false)", + "2026-09-23T20:39:42Z INFO [scheduler] Registered periodic job: backup-cache (every 5m0s)", + "2026-09-23T20:39:42Z INFO [scheduler] Daily job tier2-backup scheduled for 2026-09-24 03:30 CEST", + "2026-09-23T20:39:42Z INFO [scheduler] Daily job offbox-backup scheduled for 2026-09-24 04:15 CEST", + "2026-09-23T20:39:42Z INFO [backup] Found 6 DB dump files across drives", + "2026-09-23T20:39:42Z INFO [backup] Discovered 5 databases", + "2026-09-23T20:39:42Z INFO [backup] Recovery unit captured for docmost → /mnt/sys_drive/felhom-data/backups/primary/docmost (images=3, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0)", + "2026-09-23T20:39:42Z INFO [backup] Backup status cache refreshed" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-03.log b/documentation/audits/night-2026-09-23/chaos/round-03.log new file mode 100644 index 00000000..857cfa4f --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-03.log @@ -0,0 +1,12 @@ +22:39:10 ==== ROUND 3: restore on vikunja x controller_kill at +28s (2026-09-23T20:39:10Z) +22:39:11 [R] restoring vikunja from snapshot 'helyi' (of 1 offered) +22:39:11 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +22:39:11 + 0.0s restore (True, None, None) +22:39:17 + 6.1s restore (False, None, None) +22:39:17 [R] after restore: state=running hold=None phase=None +22:39:20 + 9.3s vikunja: deployed=True state=running updating=False phase=None hold='None' +22:39:39 >>> ACCIDENT controller_kill at 22:39:39 +22:39:42 >>> kill -9 of the controller's PID: killed 11644 at 22:39:39 +22:39:42 + 31.5s vikunja: deployed=True state=running updating=False phase=None hold='None' +22:39:46 vikunja: readback of the seeded project http=200 ok=True +22:39:46 vikunja B: project readback=True attachment content readback=True diff --git a/documentation/audits/night-2026-09-23/chaos/round-04-run.txt b/documentation/audits/night-2026-09-23/chaos/round-04-run.txt new file mode 100644 index 00000000..dfda3800 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-04-run.txt @@ -0,0 +1,17 @@ +22:39:50 ==== ROUND 4: fail_health on adventurelog x memory_hog at +43s (2026-09-23T20:39:50Z) +22:39:51 [5] drill commit fb8df47449f4: adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1,ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 -> ghcr.io/seanmorley15/adventurelog-backend:v0.13.0,ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0 (push rc=0) +22:40:33 >>> ACCIDENT memory_hog at 22:40:33 +22:40:57 >>> 1 GB hog for 5 min: 5cd58c5c3e56c7b2fbb574e9be159f17cc25033c1d3a0e87f0b2692646d8660c +chaos-hog 1003MiB / 25.29GiB at 22:40:33 +22:47:20 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +22:47:20 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +22:47:22 + 2.1s phase=pulling label=Új verzió letöltése… +22:47:23 + 3.2s phase=copying label=Az adatok másolása a frissítés előtt… +22:47:34 + 14.2s phase=starting label=Indítás az új verzióval… +22:47:40 + 20.5s phase=verifying label=Működés ellenőrzése… +22:49:11 + 110.9s phase=undoing label=Visszaállítás az előző változatra… +22:49:55 + 155.1s phase=undone label=Visszaállítva az előző változatra +22:49:55 END phase=undone err=None hold=None +22:49:55 + 605.2s adventurelog: deployed=True state=running updating=False phase=undone hold='None' +22:50:12 adventurelog: readback of the seeded account found=True +22:50:16 ==== ROUND 4 END: steady in 605.2s; readback={'A': True}; events=2 diff --git a/documentation/audits/night-2026-09-23/chaos/round-04.json b/documentation/audits/night-2026-09-23/chaos/round-04.json new file mode 100644 index 00000000..a067c74c --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-04.json @@ -0,0 +1,208 @@ +{ + "round": 4, + "action": "fail_health", + "accident": "1 GB hog for 5 min: 5cd58c5c3e56c7b2fbb574e9be159f17cc25033c1d3a0e87f0b2692646d8660c\nchaos-hog 1003MiB / 25.29GiB at 22:40:33", + "delay_s": 43, + "app": "adventurelog", + "started": "2026-09-23T20:39:50Z", + "before": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "drill_commit": "fb8df47449f4", + "badge_catchup_rounds": null, + "phases": [ + [ + 0.0, + "safety-dump", + "Adatbázis pillanatkép…" + ], + [ + 2.1, + "pulling", + "Új verzió letöltése…" + ], + [ + 3.2, + "copying", + "Az adatok másolása a frissítés előtt…" + ], + [ + 14.2, + "starting", + "Indítás az új verzióval…" + ], + [ + 20.5, + "verifying", + "Működés ellenőrzése…" + ], + [ + 110.9, + "undoing", + "Visszaállítás az előző változatra…" + ], + [ + 155.1, + "undone", + "Visszaállítva az előző változatra" + ] + ], + "time_to_steady_s": 605.2, + "after": { + "badges": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "lines": { + "hu": { + "undone_line": "A(z) adventurelog frissítése 2026-09-23 22:49-kor nem sikerült. A doboz automatikusan visszaállította az előző változatot és az adatokat — semmi nem veszett el.", + "hold": null + }, + "en": { + "undone_line": "The update of adventurelog at 2026-09-23 22:49 did not succeed. The box put back the previous version and its data automatically — nothing was lost.", + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": "undone", + "hold_reason": null, + "update_error": null + } + }, + "observables": { + "pinned_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "installed_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "catalog_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.13.0", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "image: postgis/postgis:16-3.5-alpine", + "image: ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1" + ], + "docker_inspect": [ + "adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 running=true restarts=0", + "adventurelog-postgres postgis/postgis:16-3.5-alpine running=true restarts=0", + "adventurelog-frontend ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 running=true restarts=0" + ] + }, + "readback": { + "A": true + }, + "events": [ + "2026-09-23T20:49:42Z health_change (warning)", + "2026-09-23T20:49:55Z app_update_undone (warning)" + ], + "box_log": [ + "2026-09-23T20:45:17Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:45:29Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:45:32Z WARN Health probe adventurelog: API GET :8999/api/ → Get \"http://adventurelog:8999/api/\": dial tcp 172.18.0.9:8999: connect: connection refused", + "2026-09-23T20:45:42Z WARN Health probe adventurelog: API GET :8999/api/ → Get \"http://adventurelog:8999/api/\": dial tcp 172.18.0.9:8999: connect: connection refused", + "2026-09-23T20:45:42Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:45:42Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:45:52Z WARN Health probe adventurelog: API GET :8999/api/ → Get \"http://adventurelog:8999/api/\": dial tcp 172.18.0.9:8999: connect: connection refused", + "2026-09-23T20:45:54Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:46:07Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:46:12Z WARN Health probe adventurelog: API GET :8999/api/ → Get \"http://adventurelog:8999/api/\": dial tcp 172.18.0.9:8999: connect: connection refused", + "2026-09-23T20:46:19Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:46:31Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:46:32Z WARN Health probe adventurelog: API GET :8999/api/ → Get \"http://adventurelog:8999/api/\": dial tcp 172.18.0.9:8999: connect: connection refused", + "2026-09-23T20:46:44Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:46:52Z WARN Health probe adventurelog: API GET :8999/api/ → Get \"http://adventurelog:8999/api/\": dial tcp 172.18.0.9:8999: connect: connection refused", + "2026-09-23T20:46:56Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:47:02Z WARN Health probe adventurelog: API GET :8999/api/ → Get \"http://adventurelog:8999/api/\": dial tcp 172.18.0.9:8999: connect: connection refused", + "2026-09-23T20:47:08Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:47:17Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:47:20Z INFO [api] update requested for stack: adventurelog", + "2026-09-23T20:47:20Z INFO [stacks] update adventurelog: accepted — guarded update started", + "2026-09-23T20:47:20Z INFO [stacks] update adventurelog: phase checking", + "2026-09-23T20:47:20Z INFO [stacks] update adventurelog: precondition met — Tier 1 (own recovery unit) copy from 2026-09-23T20:44:42Z (3m0s old, limit 24h0m0s)", + "2026-09-23T20:47:20Z INFO [stacks] update adventurelog: phase safety-dump", + "2026-09-23T20:47:20Z INFO [backup] Discovered 5 databases", + "2026-09-23T20:47:21Z INFO [backup] DB dump: adventurelog-postgres → pre-restore-20260923T204720Z-adventurelog-postgres.sql (7.8 MB, 760ms, 47 tables)", + "2026-09-23T20:47:21Z INFO [offbox] adventurelog: pre-restore safety dump written → pre-restore-20260923T204720Z-adventurelog-postgres.sql (7.8 MB)", + "2026-09-23T20:47:21Z INFO [backup] update safety dump for adventurelog: 1 file(s) [/mnt/sys_drive/felhom-data/backups/primary/adventurelog/db-dumps/pre-restore-20260923T204720Z-adventurelog-postgres.sql]", + "2026-09-23T20:47:21Z INFO [stacks] update adventurelog: safety dump done (1 file(s)) [/mnt/sys_drive/felhom-data/backups/primary/adventurelog/db-dumps/pre-restore-20260923T204720Z-adventurelog-postgres.sql]", + "2026-09-23T20:47:22Z WARN Health probe adventurelog: API GET :8999/api/ → Get \"http://adventurelog:8999/api/\": dial tcp 172.18.0.9:8999: connect: connection refused", + "2026-09-23T20:47:22Z INFO [stacks] update adventurelog: the undo copy will hold 2 named volume(s), 234.6 MiB", + "2026-09-23T20:47:22Z INFO [stacks] update adventurelog: phase pinning", + "2026-09-23T20:47:22Z INFO [stacks] update adventurelog: pin advanced to the catalog's current definition (adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.13.0, adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0, adventurelog-postgres=postgis/postgis:16-3.5-alpine)", + "2026-09-23T20:47:22Z INFO [stacks] update adventurelog: phase pulling", + "2026-09-23T20:47:23Z INFO [stacks] update adventurelog: phase copying", + "2026-09-23T20:47:27Z INFO [stacks] update adventurelog: phase copying", + "2026-09-23T20:47:28Z INFO [stacks] update adventurelog: copied adventurelog_adventurelog_media → adventurelog_adventurelog_media.pre-update-20260923T204727Z in 1.556s", + "2026-09-23T20:47:28Z INFO [stacks] update adventurelog: phase copying", + "2026-09-23T20:47:32Z WARN Health probe adventurelog: API GET :8999/api/ → Get \"http://adventurelog:8999/api/\": dial tcp: lookup adventurelog on 127.0.0.11:53: no such host", + "2026-09-23T20:47:34Z INFO [stacks] update adventurelog: copied adventurelog_adventurelog_postgres_data → adventurelog_adventurelog_postgres_data.pre-update-20260923T204727Z in 5.273s", + "2026-09-23T20:47:34Z INFO [stacks] update adventurelog: phase starting", + "2026-09-23T20:47:40Z INFO [stacks] update adventurelog: phase verifying", + "2026-09-23T20:47:42Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:49:11Z ERROR [stacks] update adventurelog FAILED after the new version was started: not healthy: not healthy within 1m30s (last: state unhealthy)", + "2026-09-23T20:49:11Z INFO [stacks] update adventurelog: kept 4304 bytes of the app's own log at /opt/docker/stacks/adventurelog/hold-logs/20260923T204911Z/compose-logs.txt before stopping it (R-621)", + "2026-09-23T20:49:11Z INFO [stacks] update adventurelog: phase undoing", + "2026-09-23T20:49:11Z WARN [stacks] update adventurelog: UNDO — putting back the previous version and its 2 volume copy(ies) (reason: not healthy: not healthy within 1m30s (last: state unhealthy))", + "2026-09-23T20:49:32Z INFO [stacks] pin adventurelog: adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, adventurelog-postgres=postgis/postgis:16-3.5-alpine", + "2026-09-23T20:49:42Z INFO [scheduler] Running job: backup-cache", + "2026-09-23T20:49:42Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)", + "2026-09-23T20:49:42Z INFO [backup] Found 7 DB dump files across drives", + "2026-09-23T20:49:42Z INFO [backup] Discovered 5 databases", + "2026-09-23T20:49:42Z INFO [backup] Backup status cache refreshed", + "2026-09-23T20:49:42Z INFO [scheduler] Job backup-cache completed (took 400ms)", + "2026-09-23T20:49:55Z INFO [stacks] update adventurelog: event app_update_undone", + "2026-09-23T20:49:55Z WARN notifier disabled (no hub configured): DROPPED event app_update_undone (severity warning) — further app_update_undone events are logged at DEBUG only", + "2026-09-23T20:49:55Z INFO [stacks] update adventurelog: UNDONE in 44s — the previous version is running on the data from before the update (the app's health check passed)" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-04.log b/documentation/audits/night-2026-09-23/chaos/round-04.log new file mode 100644 index 00000000..4a1e102e --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-04.log @@ -0,0 +1,16 @@ +22:39:50 ==== ROUND 4: fail_health on adventurelog x memory_hog at +43s (2026-09-23T20:39:50Z) +22:39:51 [5] drill commit fb8df47449f4: adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1,ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 -> ghcr.io/seanmorley15/adventurelog-backend:v0.13.0,ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0 (push rc=0) +22:40:33 >>> ACCIDENT memory_hog at 22:40:33 +22:40:57 >>> 1 GB hog for 5 min: 5cd58c5c3e56c7b2fbb574e9be159f17cc25033c1d3a0e87f0b2692646d8660c +chaos-hog 1003MiB / 25.29GiB at 22:40:33 +22:47:20 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +22:47:20 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +22:47:22 + 2.1s phase=pulling label=Új verzió letöltése… +22:47:23 + 3.2s phase=copying label=Az adatok másolása a frissítés előtt… +22:47:34 + 14.2s phase=starting label=Indítás az új verzióval… +22:47:40 + 20.5s phase=verifying label=Működés ellenőrzése… +22:49:11 + 110.9s phase=undoing label=Visszaállítás az előző változatra… +22:49:55 + 155.1s phase=undone label=Visszaállítva az előző változatra +22:49:55 END phase=undone err=None hold=None +22:49:55 + 605.2s adventurelog: deployed=True state=running updating=False phase=undone hold='None' +22:50:12 adventurelog: readback of the seeded account found=True diff --git a/documentation/audits/night-2026-09-23/chaos/round-05-run.txt b/documentation/audits/night-2026-09-23/chaos/round-05-run.txt new file mode 100644 index 00000000..0f04af29 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-05-run.txt @@ -0,0 +1,9 @@ +22:50:17 ==== ROUND 5: install on papra x disk_fill at +28s (2026-09-23T20:50:17Z) +22:50:17 + 0.2s papra: deployed=True state=not_deployed updating=False phase=None hold='None' +22:50:19 + 2.3s papra: deployed=True state=starting updating=False phase=None hold='None' +22:50:43 + 26.6s papra: deployed=True state=running updating=False phase=None hold='None' +22:50:44 papra: sign-up http=200 +22:50:45 >>> ACCIDENT disk_fill at 22:50:45 +22:50:48 >>> filled /var/lib/docker to 3 GB free (floor 2 GB + 1 GB): /dev/loop1 69G 62G 3.0G 96% /var/lib/docker at 22:50:45 +22:50:55 papra: sign-in as the seeded account http=200 ok=True +22:50:58 ==== ROUND 5 END: steady in 26.6s; readback={'A': True}; events=2 diff --git a/documentation/audits/night-2026-09-23/chaos/round-05.json b/documentation/audits/night-2026-09-23/chaos/round-05.json new file mode 100644 index 00000000..1bdfb4be --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-05.json @@ -0,0 +1,73 @@ +{ + "round": 5, + "action": "install", + "accident": "filled /var/lib/docker to 3 GB free (floor 2 GB + 1 GB): /dev/loop1 69G 62G 3.0G 96% /var/lib/docker at 22:50:45", + "delay_s": 28, + "app": "papra", + "started": "2026-09-23T20:50:17Z", + "before": { + "state": "not deployed" + }, + "armed_at": "22:50:17", + "act": "deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}", + "time_to_steady_s": 26.6, + "after": { + "badges": { + "hu": [], + "en": [] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "observables": { + "pinned_images": { + "papra": "ghcr.io/papra-hq/papra:26.6.2-rootless" + }, + "installed_images": { + "papra": "ghcr.io/papra-hq/papra:26.6.2-rootless" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: ghcr.io/papra-hq/papra:26.6.2-rootless" + ], + "docker_inspect": [ + "papra ghcr.io/papra-hq/papra:26.6.2-rootless running=true restarts=0" + ] + }, + "readback": { + "A": true + }, + "events": [ + "2026-09-23T20:50:17Z app_deploy_started (info)", + "2026-09-23T20:50:18Z app_deployed (info)" + ], + "box_log": [ + "2026-09-23T20:50:17Z INFO [api] Deploy requested for stack: papra", + "2026-09-23T20:50:17Z INFO [stacks] Deploying stack papra with 3 env vars: [DOMAIN, SUBDOMAIN, AUTH_SECRET]", + "2026-09-23T20:50:17Z INFO [stacks] Deploying stack papra — checking 1 images...", + "2026-09-23T20:50:17Z WARN notifier disabled (no hub configured): DROPPED event app_deploy_started (severity info) — further app_deploy_started events are logged at DEBUG only", + "2026-09-23T20:50:17Z INFO [stacks] Stack papra deployed successfully (took 0.5s)", + "2026-09-23T20:50:18Z INFO [stacks] installed-images papra: recorded 1 service(s) (papra=ghcr.io/papra-hq/papra:26.6.2-rootless (sha256:a281cb44176d…))", + "2026-09-23T20:50:18Z INFO [stacks] pin papra: papra=ghcr.io/papra-hq/papra:26.6.2-rootless", + "2026-09-23T20:50:18Z WARN notifier disabled (no hub configured): DROPPED event app_deployed (severity info) — further app_deployed events are logged at DEBUG only", + "2026-09-23T20:50:21Z INFO [stacks] Stack papra post-start status:", + "2026-09-23T20:50:21Z INFO [stacks] papra ghcr.io/papra-hq/papra:26.6.2-rootless running Up 3 seconds (health: starting)", + "2026-09-23T20:50:42Z INFO [deadapp] check alive: 20 scans since boot, 11 deployed app(s) evaluated, 1 currently down" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-05.log b/documentation/audits/night-2026-09-23/chaos/round-05.log new file mode 100644 index 00000000..469d333e --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-05.log @@ -0,0 +1,8 @@ +22:50:17 ==== ROUND 5: install on papra x disk_fill at +28s (2026-09-23T20:50:17Z) +22:50:17 + 0.2s papra: deployed=True state=not_deployed updating=False phase=None hold='None' +22:50:19 + 2.3s papra: deployed=True state=starting updating=False phase=None hold='None' +22:50:43 + 26.6s papra: deployed=True state=running updating=False phase=None hold='None' +22:50:44 papra: sign-up http=200 +22:50:45 >>> ACCIDENT disk_fill at 22:50:45 +22:50:48 >>> filled /var/lib/docker to 3 GB free (floor 2 GB + 1 GB): /dev/loop1 69G 62G 3.0G 96% /var/lib/docker at 22:50:45 +22:50:55 papra: sign-in as the seeded account http=200 ok=True diff --git a/documentation/audits/night-2026-09-23/chaos/round-06-run.txt b/documentation/audits/night-2026-09-23/chaos/round-06-run.txt new file mode 100644 index 00000000..b2fdbf7f --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-06-run.txt @@ -0,0 +1,16 @@ +22:50:58 ==== ROUND 6: good_update on romm x memory_hog at +46s (2026-09-23T20:50:58Z) +22:50:59 [5] drill commit d2cd01ae0adc: romm rommapp/romm:5.3.0 -> rommapp/romm:5.3.1 (push rc=0) +22:51:04 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +22:51:04 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +22:51:06 + 2.1s phase=pulling label=Új verzió letöltése… +22:51:07 + 3.2s phase=copying label=Az adatok másolása a frissítés előtt… +22:51:13 + 8.9s phase=starting label=Indítás az új verzióval… +22:51:24 + 20.0s phase=verifying label=Működés ellenőrzése… +22:51:50 >>> ACCIDENT memory_hog at 22:51:50 +22:52:00 + 55.8s phase=done label=Frissítve +22:52:00 END phase=done err=None hold=None +22:52:00 + 61.7s romm: deployed=True state=running updating=False phase=done hold='None' +22:52:15 >>> 1 GB hog for 5 min: 7922307cf3f2b0e1bebb33d30a94d644382b7596a42d0024aec74d1a81d5762f +chaos-hog 1004MiB / 25.29GiB at 22:51:50 +22:52:24 romm: login as the seeded user http=200 ok=True +22:52:27 ==== ROUND 6 END: steady in 61.7s; readback={'A': True}; events=0 diff --git a/documentation/audits/night-2026-09-23/chaos/round-06.json b/documentation/audits/night-2026-09-23/chaos/round-06.json new file mode 100644 index 00000000..1552fc42 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-06.json @@ -0,0 +1,174 @@ +{ + "round": 6, + "action": "good_update", + "accident": "1 GB hog for 5 min: 7922307cf3f2b0e1bebb33d30a94d644382b7596a42d0024aec74d1a81d5762f\nchaos-hog 1004MiB / 25.29GiB at 22:51:50", + "delay_s": 46, + "app": "romm", + "started": "2026-09-23T20:50:58Z", + "before": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "drill_commit": "d2cd01ae0adc", + "badge_catchup_s": 4.5, + "armed_at": "22:51:04", + "phases": [ + [ + 0.0, + "safety-dump", + "Adatbázis pillanatkép…" + ], + [ + 2.1, + "pulling", + "Új verzió letöltése…" + ], + [ + 3.2, + "copying", + "Az adatok másolása a frissítés előtt…" + ], + [ + 8.9, + "starting", + "Indítás az új verzióval…" + ], + [ + 20.0, + "verifying", + "Működés ellenőrzése…" + ], + [ + 55.8, + "done", + "Frissítve" + ] + ], + "time_to_steady_s": 61.7, + "after": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": "done", + "hold_reason": null, + "update_error": null + } + }, + "observables": { + "pinned_images": { + "romm": "rommapp/romm:5.3.1", + "romm-db": "mariadb:11.4", + "romm-redis": "redis:7-alpine" + }, + "installed_images": { + "romm": "rommapp/romm:5.3.1", + "romm-db": "mariadb:11.4", + "romm-redis": "redis:7-alpine" + }, + "catalog_images": { + "romm": "rommapp/romm:5.3.1", + "romm-db": "mariadb:11.4", + "romm-redis": "redis:7-alpine" + }, + "live_compose_image_lines": [ + "image: rommapp/romm:5.3.1", + "image: mariadb:11.4", + "image: redis:7-alpine" + ], + "docker_inspect": [ + "romm rommapp/romm:5.3.1 running=true restarts=0", + "romm-db mariadb:11.4 running=true restarts=0", + "romm-redis redis:7-alpine running=true restarts=0" + ] + }, + "readback": { + "A": true + }, + "events": [], + "box_log": [ + "2026-09-23T20:51:00Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T20:51:02Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T20:51:04Z INFO [api] update requested for stack: romm", + "2026-09-23T20:51:04Z INFO [stacks] update romm: accepted — guarded update started", + "2026-09-23T20:51:04Z INFO [stacks] update romm: phase checking", + "2026-09-23T20:51:04Z INFO [stacks] update romm: precondition met — Tier 1 (own recovery unit) copy from 2026-09-23T20:31:55Z (19m0s old, limit 24h0m0s)", + "2026-09-23T20:51:04Z INFO [stacks] update romm: phase safety-dump", + "2026-09-23T20:51:04Z INFO [backup] Discovered 5 databases", + "2026-09-23T20:51:05Z INFO [backup] DB dump: romm-db → pre-restore-20260923T205104Z-romm-mariadb.sql (110.2 KB, 335ms, 36 tables)", + "2026-09-23T20:51:05Z INFO [offbox] romm: pre-restore safety dump written → pre-restore-20260923T205104Z-romm-mariadb.sql (110.2 KB)", + "2026-09-23T20:51:05Z INFO [backup] update safety dump for romm: 1 file(s) [/mnt/felhom-drives/scratch_hdd/userdata/romm/backups/primary/romm/db-dumps/pre-restore-20260923T205104Z-romm-mariadb.sql]", + "2026-09-23T20:51:05Z INFO [stacks] update romm: safety dump done (1 file(s)) [/mnt/felhom-drives/scratch_hdd/userdata/romm/backups/primary/romm/db-dumps/pre-restore-20260923T205104Z-romm-mariadb.sql]", + "2026-09-23T20:51:06Z INFO [stacks] update romm: the undo copy will hold 3 named volume(s), 182.1 MiB", + "2026-09-23T20:51:06Z INFO [stacks] update romm: phase pinning", + "2026-09-23T20:51:06Z INFO [stacks] update romm: pin advanced to the catalog's current definition (romm=rommapp/romm:5.3.1, romm-db=mariadb:11.4, romm-redis=redis:7-alpine)", + "2026-09-23T20:51:06Z INFO [stacks] update romm: phase pulling", + "2026-09-23T20:51:07Z INFO [stacks] update romm: phase copying", + "2026-09-23T20:51:11Z INFO [stacks] update romm: phase copying", + "2026-09-23T20:51:11Z INFO [stacks] update romm: copied romm_romm_config → romm_romm_config.pre-update-20260923T205111Z in 465ms", + "2026-09-23T20:51:11Z INFO [stacks] update romm: phase copying", + "2026-09-23T20:51:12Z INFO [stacks] update romm: copied romm_romm_db_data → romm_romm_db_data.pre-update-20260923T205111Z in 988ms", + "2026-09-23T20:51:12Z INFO [stacks] update romm: phase copying", + "2026-09-23T20:51:13Z INFO [stacks] update romm: copied romm_romm_redis_data → romm_romm_redis_data.pre-update-20260923T205111Z in 501ms", + "2026-09-23T20:51:13Z INFO [stacks] update romm: phase starting", + "2026-09-23T20:51:13Z INFO [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping ensure (held by drive gate)", + "2026-09-23T20:51:24Z INFO [stacks] update romm: phase verifying", + "2026-09-23T20:51:42Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T20:51:59Z INFO [stacks] update romm: healthy after 35s (the app's health check passed)", + "2026-09-23T20:52:00Z INFO [stacks] installed-images romm: recorded 3 service(s) (romm=rommapp/romm:5.3.1 (sha256:0d66b4ea152a…), romm-db=mariadb:11.4 (sha256:70cc072b29b4…), romm-redis=redis:7-alpine (sha256:858f009f9709…))", + "2026-09-23T20:52:00Z INFO [stacks] update romm: DONE in 56s" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-06.log b/documentation/audits/night-2026-09-23/chaos/round-06.log new file mode 100644 index 00000000..4ce76c12 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-06.log @@ -0,0 +1,15 @@ +22:50:58 ==== ROUND 6: good_update on romm x memory_hog at +46s (2026-09-23T20:50:58Z) +22:50:59 [5] drill commit d2cd01ae0adc: romm rommapp/romm:5.3.0 -> rommapp/romm:5.3.1 (push rc=0) +22:51:04 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +22:51:04 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +22:51:06 + 2.1s phase=pulling label=Új verzió letöltése… +22:51:07 + 3.2s phase=copying label=Az adatok másolása a frissítés előtt… +22:51:13 + 8.9s phase=starting label=Indítás az új verzióval… +22:51:24 + 20.0s phase=verifying label=Működés ellenőrzése… +22:51:50 >>> ACCIDENT memory_hog at 22:51:50 +22:52:00 + 55.8s phase=done label=Frissítve +22:52:00 END phase=done err=None hold=None +22:52:00 + 61.7s romm: deployed=True state=running updating=False phase=done hold='None' +22:52:15 >>> 1 GB hog for 5 min: 7922307cf3f2b0e1bebb33d30a94d644382b7596a42d0024aec74d1a81d5762f +chaos-hog 1004MiB / 25.29GiB at 22:51:50 +22:52:24 romm: login as the seeded user http=200 ok=True diff --git a/documentation/audits/night-2026-09-23/chaos/round-07-run.txt b/documentation/audits/night-2026-09-23/chaos/round-07-run.txt new file mode 100644 index 00000000..dd3abb51 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-07-run.txt @@ -0,0 +1,14 @@ +22:52:28 ==== ROUND 7: restore on docmost x power_cut at +56s (2026-09-23T20:52:28Z) +22:52:28 [R] restoring docmost from snapshot 'helyi' (of 1 offered) +22:52:28 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +22:52:28 + 0.0s restore (True, None, None) +22:53:00 + 32.4s restore (False, None, None) +22:53:00 [R] after restore: state=running hold=None phase=None +22:53:04 + 35.8s docmost: deployed=True state=running updating=False phase=None hold='None' +22:53:24 >>> ACCIDENT power_cut at 22:53:24 +22:53:35 >>> pct stop rc=0, pct start rc=0 at 22:53:24 +22:53:47 + 79.1s docmost: deployed=True state=starting updating=False phase=None hold='None' +22:54:11 + 103.5s docmost: deployed=True state=running updating=False phase=None hold='None' +22:54:16 docmost: login as the seeded user http=200 ok=True +22:54:17 docmost B: /api/spaces http=200 seeded-space-listed=True (negative control listed=False) +22:54:22 ==== ROUND 7 END: steady in 103.5s; readback={'A': True, 'B': True}; events=1 diff --git a/documentation/audits/night-2026-09-23/chaos/round-07.json b/documentation/audits/night-2026-09-23/chaos/round-07.json new file mode 100644 index 00000000..a117bf9d --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-07.json @@ -0,0 +1,160 @@ +{ + "round": 7, + "action": "restore", + "accident": "pct stop rc=0, pct start rc=0 at 22:53:24", + "delay_s": 56, + "app": "docmost", + "started": "2026-09-23T20:52:28Z", + "before": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "armed_at": "22:52:28", + "act": { + "ok": true, + "why": null, + "snapshot_id": "helyi", + "http": "HTTP/2 302", + "seconds": 32.5, + "state_after": "running", + "hold_after": null + }, + "controller_back_s": 12.3, + "time_to_steady_s": 103.5, + "after": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "observables": { + "pinned_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "installed_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "catalog_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "live_compose_image_lines": [ + "image: docmost/docmost:0.96.0", + "image: postgres:16-alpine", + "image: redis:7-alpine" + ], + "docker_inspect": [ + "docmost docmost/docmost:0.96.0 running=true restarts=0", + "docmost-redis redis:7-alpine running=true restarts=0", + "docmost-postgres postgres:16-alpine running=true restarts=0" + ] + }, + "readback": { + "A": true, + "B": true + }, + "events": [ + "2026-09-23T20:53:50Z controller_started (info)" + ], + "box_log": [ + "2026-09-23T20:52:28Z WARN [web] Restore requested (async): stack=docmost, snapshot=helyi from 172.18.0.2:56506", + "2026-09-23T20:52:28Z INFO [backup] Restoring docmost from recovery unit /mnt/sys_drive/felhom-data/backups/primary/docmost: images=3, secrets recovered=2/2, data_keys=0", + "2026-09-23T20:52:28Z INFO [stacks] Stopping stack: docmost", + "2026-09-23T20:52:29Z INFO [stacks] Stack docmost stopped successfully (took 1.3s)", + "2026-09-23T20:52:29Z INFO [backup] Restoring Docker volume docmost_docmost_postgres_data for docmost", + "2026-09-23T20:52:30Z INFO [backup] Restoring Docker volume docmost_docmost_redis_data for docmost", + "2026-09-23T20:52:31Z INFO [backup] Restoring Docker volume docmost_docmost_storage for docmost", + "2026-09-23T20:52:31Z INFO [backup] Restored 3 Docker volume(s) for docmost", + "2026-09-23T20:52:31Z INFO [stacks] Redeploying docmost from recovery unit with 4 env vars", + "2026-09-23T20:52:31Z INFO [stacks] pin docmost: docmost=docmost/docmost:0.96.0, docmost-postgres=postgres:16-alpine, docmost-redis=redis:7-alpine", + "2026-09-23T20:52:31Z INFO [stacks] Starting stack docmost services only: [docmost-postgres]", + "2026-09-23T20:52:32Z INFO [stacks] Stack docmost services [docmost-postgres] started (took 0.5s)", + "2026-09-23T20:52:32Z INFO [backup] Discovered 5 databases", + "2026-09-23T20:52:32Z INFO [backup] Restore docmost: replaying DB dump into docmost-postgres (postgres)", + "2026-09-23T20:52:34Z INFO [backup] Imported DB dump docmost-postgres.sql into docmost-postgres (postgres)", + "2026-09-23T20:52:34Z INFO [backup] Restore docmost: replayed 1 DB dump(s)", + "2026-09-23T20:52:34Z INFO [stacks] Starting stack: docmost", + "2026-09-23T20:52:45Z INFO [stacks] Stack docmost started successfully (took 11.2s)", + "2026-09-23T20:52:45Z INFO [stacks] installed-images docmost: recorded 3 service(s) (docmost=docmost/docmost:0.96.0 (sha256:b56947fcfd08…), docmost-postgres=postgres:16-alpine (sha256:721873c34ceb…), docmost-redis=redis:7-alpine (sha256:858f009f9709…))", + "2026-09-23T20:52:48Z INFO [stacks] Stack docmost post-start status:", + "2026-09-23T20:52:48Z INFO [stacks] docmost docmost/docmost:0.96.0 running Up 3 seconds (health: starting)", + "2026-09-23T20:52:48Z INFO [stacks] docmost-postgres postgres:16-alpine running Up 16 seconds (healthy)", + "2026-09-23T20:52:48Z INFO [stacks] docmost-redis redis:7-alpine running Up 14 seconds (healthy)", + "2026-09-23T20:52:58Z INFO [backup] Restore-from-unit completed: docmost — 3 volume(s) of 3 listed, 1 database(s) of 1 listed", + "2026-09-23T20:52:58Z INFO [web] Restore completed (async): stack=docmost in 30.595564222s (volumes 3/3, dbs 1/1)", + "2026-09-23T20:53:44Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T20:53:44Z INFO [stacks] desired state for docmost recorded as \"running\" (was \"\")", + "2026-09-23T20:53:44Z INFO [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false)", + "2026-09-23T20:53:44Z INFO [scheduler] Registered periodic job: backup-cache (every 5m0s)", + "2026-09-23T20:53:44Z INFO [scheduler] Daily job tier2-backup scheduled for 2026-09-24 03:30 CEST", + "2026-09-23T20:53:44Z INFO [scheduler] Daily job offbox-backup scheduled for 2026-09-24 04:15 CEST", + "2026-09-23T20:53:45Z INFO [backup] Found 8 DB dump files across drives", + "2026-09-23T20:53:46Z INFO [backup] Discovered 5 databases", + "2026-09-23T20:53:46Z INFO [backup] Recovery unit captured for adventurelog → /mnt/sys_drive/felhom-data/backups/primary/adventurelog (images=3, secrets-referenced=2, data_keys=1, portable-carried=2/2, withheld=0)", + "2026-09-23T20:53:46Z INFO [backup] Recovery unit captured for papra → /mnt/sys_drive/felhom-data/backups/primary/papra (images=1, secrets-referenced=1, data_keys=1, portable-carried=1/1, withheld=0)", + "2026-09-23T20:53:46Z INFO [backup] Recovery unit captured for romm → /mnt/felhom-drives/scratch_hdd/userdata/romm/backups/primary/romm (images=3, secrets-referenced=3, data_keys=0, portable-carried=3/3, withheld=0)", + "2026-09-23T20:53:46Z INFO [backup] Backup status cache refreshed" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-07.log b/documentation/audits/night-2026-09-23/chaos/round-07.log new file mode 100644 index 00000000..2213536d --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-07.log @@ -0,0 +1,13 @@ +22:52:28 ==== ROUND 7: restore on docmost x power_cut at +56s (2026-09-23T20:52:28Z) +22:52:28 [R] restoring docmost from snapshot 'helyi' (of 1 offered) +22:52:28 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +22:52:28 + 0.0s restore (True, None, None) +22:53:00 + 32.4s restore (False, None, None) +22:53:00 [R] after restore: state=running hold=None phase=None +22:53:04 + 35.8s docmost: deployed=True state=running updating=False phase=None hold='None' +22:53:24 >>> ACCIDENT power_cut at 22:53:24 +22:53:35 >>> pct stop rc=0, pct start rc=0 at 22:53:24 +22:53:47 + 79.1s docmost: deployed=True state=starting updating=False phase=None hold='None' +22:54:11 + 103.5s docmost: deployed=True state=running updating=False phase=None hold='None' +22:54:16 docmost: login as the seeded user http=200 ok=True +22:54:17 docmost B: /api/spaces http=200 seeded-space-listed=True (negative control listed=False) diff --git a/documentation/audits/night-2026-09-23/chaos/round-08-recovery.json b/documentation/audits/night-2026-09-23/chaos/round-08-recovery.json new file mode 100644 index 00000000..39020887 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-08-recovery.json @@ -0,0 +1,36 @@ +{ + "snapshots": [ + { + "time": "2026-09-23T20:31:04Z", + "short_id": "helyi", + "tier": 1, + "drive_label": "Tárhely (navidrome)" + } + ], + "restore": { + "ok": true, + "snapshot_id": "helyi", + "http": "HTTP/2 302", + "seconds": 10.2, + "state_after": "running", + "hold_after": null + }, + "readback_A": true, + "observables": { + "pinned_images": { + "navidrome": "deluan/navidrome:0.64.0" + }, + "installed_images": { + "navidrome": "deluan/navidrome:0.64.0" + }, + "catalog_images": { + "navidrome": "deluan/navidrome:0.64.1" + }, + "live_compose_image_lines": [ + "image: deluan/navidrome:0.64.0" + ], + "docker_inspect": [ + "navidrome deluan/navidrome:0.64.0 running=true restarts=0" + ] + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-08-recovery.log b/documentation/audits/night-2026-09-23/chaos/round-08-recovery.log new file mode 100644 index 00000000..ef69d638 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-08-recovery.log @@ -0,0 +1,9 @@ +23:02:41 restore points offered: [{"time": "2026-09-23T20:31:04Z", "short_id": "helyi", "tier": 1, "drive_label": "Tárhely (navidrome)"}] +23:02:41 [R] restoring navidrome from snapshot 'helyi' (of 1 offered) +23:02:41 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +23:02:41 + 0.0s restore (True, None, None) +23:02:51 + 10.1s restore (False, None, None) +23:02:51 [R] after restore: state=running hold=None phase=None +23:02:55 after restore: state=running hold=None phase=None +23:02:55 navidrome: login as the seeded user http=200 ok=True +23:02:58 seed A after the household's restore: True; pinned {'navidrome': 'deluan/navidrome:0.64.0'} diff --git a/documentation/audits/night-2026-09-23/chaos/round-08-run.txt b/documentation/audits/night-2026-09-23/chaos/round-08-run.txt new file mode 100644 index 00000000..500bb847 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-08-run.txt @@ -0,0 +1,21 @@ +22:54:23 ==== ROUND 8: cutoff on navidrome x box_backup at +23s (2026-09-23T20:54:23Z) +22:54:24 [5] drill commit ce80a4a74a2b: navidrome deluan/navidrome:0.64.0 -> deluan/navidrome:0.64.1 (push rc=0) +22:54:29 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +22:54:29 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +22:54:30 + 1.1s phase=pulling label=Új verzió letöltése… +22:54:31 + 2.1s phase=copying label=Az adatok másolása a frissítés előtt… +22:54:32 + 3.2s phase=starting label=Indítás az új verzióval… +22:54:32 + 3.7s phase=verifying label=Működés ellenőrzése… +22:54:37 >>> finished-marker removed from one undo copy: +copy: navidrome_navidrome_data.pre-update-20260923T205431Z +data + +22:54:51 >>> ACCIDENT box_backup at 22:54:51 +22:54:51 >>> POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'} at 22:54:51 +22:56:04 + 94.9s phase=undoing label=Visszaállítás az előző változatra… +22:56:04 + 95.4s phase=failed label=A frissítés nem sikerült +22:56:04 END phase=failed err='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) navidrome frissítése 2026-09-23 22:56-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:31 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.' hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) navidrome frissítése 2026-09-23 22:56-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:31 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.' +22:56:04 + 101.2s navidrome: deployed=True state=stopped updating=False phase=failed hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új v' +22:59:09 app never answered on music/ (last rc=0 code=404) +23:02:10 app never answered on music/ (last rc=0 code=404) +23:02:15 ==== ROUND 8 END: steady in 101.2s; readback={'A': False}; events=1 diff --git a/documentation/audits/night-2026-09-23/chaos/round-08.json b/documentation/audits/night-2026-09-23/chaos/round-08.json new file mode 100644 index 00000000..ed32cadf --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-08.json @@ -0,0 +1,192 @@ +{ + "round": 8, + "action": "cutoff", + "accident": "POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'} at 22:54:51", + "delay_s": 23, + "app": "navidrome", + "started": "2026-09-23T20:54:23Z", + "before": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "drill_commit": "ce80a4a74a2b", + "badge_catchup_s": 4.7, + "armed_at": "22:54:28", + "cutoff": "copy: navidrome_navidrome_data.pre-update-20260923T205431Z\ndata", + "phases": [ + [ + 0.0, + "safety-dump", + "Adatbázis pillanatkép…" + ], + [ + 1.1, + "pulling", + "Új verzió letöltése…" + ], + [ + 2.1, + "copying", + "Az adatok másolása a frissítés előtt…" + ], + [ + 3.2, + "starting", + "Indítás az új verzióval…" + ], + [ + 3.7, + "verifying", + "Működés ellenőrzése…" + ], + [ + 94.9, + "undoing", + "Visszaállítás az előző változatra…" + ], + [ + 95.4, + "failed", + "A frissítés nem sikerült" + ] + ], + "time_to_steady_s": 101.2, + "after": { + "badges": { + "hu": [ + { + "title": "A frissítés nem sikerült, és az automatikus visszaállítás sem.", + "text": "Megállítva — visszaállítás szükséges" + } + ], + "en": [ + { + "title": "The update did not succeed, and the automatic undo did not either.", + "text": "Stopped — restore needed" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": "A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) navidrome frissítése 2026-09-23 22:56-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:31 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem." + }, + "en": { + "undone_line": null, + "hold": "The update did not succeed, and the automatic undo did not either. The data is as the new version left it. The update of navidrome at 2026-09-23 22:56 did not succeed, and the app did not start on the new version. The app stays stopped for safety, so that its data is not damaged. It can be restored on the Backups page from this backup: own drive, 2026-09-23 22:31 — this copy holds only the settings and the database, not the files." + } + }, + "state": { + "deployed": true, + "state": "stopped", + "update_phase": "failed", + "hold_reason": "A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) navidrome frissítése 2026-09-23 22:56-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:31 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.", + "update_error": "A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) navidrome frissítése 2026-09-23 22:56-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:31 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem." + } + }, + "observables": { + "pinned_images": { + "navidrome": "deluan/navidrome:0.64.1" + }, + "installed_images": { + "navidrome": "deluan/navidrome:0.64.0" + }, + "catalog_images": { + "navidrome": "deluan/navidrome:0.64.1" + }, + "live_compose_image_lines": [ + "image: deluan/navidrome:0.64.1" + ], + "docker_inspect": [] + }, + "readback": { + "A": false + }, + "events": [ + "2026-09-23T21:01:45Z app_start_failed (warning)" + ], + "box_log": [ + "2026-09-23T20:55:22Z WARN [backup] Skipping volume dump for navidrome — the app is HELD stopped (its restore point is preserved)", + "2026-09-23T20:55:22Z INFO [backup] Stopping nextcloud for safe volume dump", + "2026-09-23T20:55:25Z INFO [backup] Volume dump: nextcloud/nextcloud_nextcloud_db_data → 162.7 MB", + "2026-09-23T20:55:25Z WARN Health probe navidrome: API GET :8999/ping → Get \"http://navidrome:8999/ping\": dial tcp 172.18.0.10:8999: connect: connection refused", + "2026-09-23T20:55:32Z INFO [backup] Volume dump: nextcloud/nextcloud_nextcloud_html → 745.1 MB", + "2026-09-23T20:55:33Z INFO [backup] Volume dump: nextcloud/nextcloud_nextcloud_redis_data → 10.0 KB", + "2026-09-23T20:55:33Z INFO [backup] Restarting nextcloud after volume dump", + "2026-09-23T20:55:33Z INFO [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate)", + "2026-09-23T20:55:39Z INFO [backup] Stopping paperless-ngx for safe volume dump", + "2026-09-23T20:55:45Z WARN Health probe navidrome: API GET :8999/ping → Get \"http://navidrome:8999/ping\": dial tcp 172.18.0.10:8999: connect: connection refused", + "2026-09-23T20:55:45Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T20:55:47Z INFO [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 725.0 KB", + "2026-09-23T20:55:48Z INFO [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 67.8 MB", + "2026-09-23T20:55:48Z INFO [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_redis_data → 649.5 KB", + "2026-09-23T20:55:48Z INFO [backup] Restarting paperless-ngx after volume dump", + "2026-09-23T20:55:48Z INFO [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx not mounted — skipping ensure (held by drive gate)", + "2026-09-23T20:56:00Z INFO [backup] Stopping papra for safe volume dump", + "2026-09-23T20:56:01Z INFO [backup] Volume dump: papra/papra_papra_data → 571.0 KB", + "2026-09-23T20:56:01Z INFO [backup] Restarting papra after volume dump", + "2026-09-23T20:56:02Z INFO [backup] Stopping privatebin for safe volume dump", + "2026-09-23T20:56:03Z INFO [backup] Volume dump: privatebin/privatebin_privatebin_data → 1.5 KB", + "2026-09-23T20:56:03Z INFO [backup] Restarting privatebin after volume dump", + "2026-09-23T20:56:03Z ERROR [stacks] update navidrome FAILED after the new version was started: not healthy: not healthy within 1m30s (last: state unhealthy)", + "2026-09-23T20:56:03Z INFO [stacks] update navidrome: kept 4221 bytes of the app's own log at /opt/docker/stacks/navidrome/hold-logs/20260923T205603Z/compose-logs.txt before stopping it (R-621)", + "2026-09-23T20:56:03Z INFO [stacks] update navidrome: phase undoing", + "2026-09-23T20:56:03Z WARN [stacks] update navidrome: UNDO — putting back the previous version and its 1 volume copy(ies) (reason: not healthy: not healthy within 1m30s (last: state unhealthy))", + "2026-09-23T20:56:03Z INFO [backup] Stopping romm for safe volume dump", + "2026-09-23T20:56:04Z ERROR [stacks] update navidrome: the undo copy navidrome_navidrome_data.pre-update-20260923T205431Z has no finished-marker — it is cut off or missing; NOTHING is put back", + "2026-09-23T20:56:04Z ERROR [stacks] update navidrome: the UNDO failed too (untouched) — HOLDING the app; the undo copies are kept: [{navidrome_navidrome_data navidrome_navidrome_data.pre-update-20260923T205431Z}]", + "2026-09-23T20:56:04Z WARN [settings] restore hold SET for navidrome — the app stays stopped until it is cleared", + "2026-09-23T20:56:04Z WARN [backup] navidrome is HELD STOPPED after a failed update (restore point: tier 1 \"saját meghajtó\", 2026-09-23T20:31:04Z; holds: \"csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem\"; undo: \"untouched\")", + "2026-09-23T20:56:04Z INFO [stacks] update navidrome: event app_update_held (hold recorded: true)", + "2026-09-23T20:56:04Z WARN notifier disabled (no hub configured): DROPPED event app_update_held (severity error) — further app_update_held events are logged at DEBUG only", + "2026-09-23T20:56:11Z INFO [backup] Volume dump: romm/romm_romm_config → 2.5 KB", + "2026-09-23T20:56:11Z INFO [backup] Volume dump: romm/romm_romm_db_data → 154.7 MB", + "2026-09-23T20:56:12Z INFO [backup] Volume dump: romm/romm_romm_redis_data → 15.7 MB", + "2026-09-23T20:56:12Z INFO [backup] Restarting romm after volume dump", + "2026-09-23T20:56:12Z INFO [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping ensure (held by drive gate)", + "2026-09-23T20:56:23Z INFO [backup] Stopping vikunja for safe volume dump", + "2026-09-23T20:56:24Z INFO [backup] Volume dump: vikunja/vikunja_vikunja_data → 2.5 KB", + "2026-09-23T20:56:25Z INFO [backup] Volume dump: vikunja/vikunja_vikunja_db → 2.7 MB", + "2026-09-23T20:56:25Z INFO [backup] Restarting vikunja after volume dump", + "2026-09-23T20:56:26Z INFO [backup] App-data backup completed: 5 databases (10.6 MB total), 10 volume dump(s) (1m34.231s)", + "2026-09-23T20:56:26Z INFO [backup] Recovery unit captured for papra → /mnt/sys_drive/felhom-data/backups/primary/papra (images=1, secrets-referenced=1, data_keys=1, portable-carried=1/1, withheld=0)", + "2026-09-23T20:57:45Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T20:58:45Z INFO [scheduler] Running job: backup-cache", + "2026-09-23T20:58:45Z INFO [backup] Found 8 DB dump files across drives", + "2026-09-23T20:58:45Z INFO [backup] Discovered 5 databases", + "2026-09-23T20:58:45Z INFO [backup] Backup status cache refreshed", + "2026-09-23T20:58:45Z INFO [scheduler] Job backup-cache completed (took 207ms)", + "2026-09-23T20:59:45Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:01:45Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)" + ], + "undo_copies_left": "1" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-08.log b/documentation/audits/night-2026-09-23/chaos/round-08.log new file mode 100644 index 00000000..4c347674 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-08.log @@ -0,0 +1,20 @@ +22:54:23 ==== ROUND 8: cutoff on navidrome x box_backup at +23s (2026-09-23T20:54:23Z) +22:54:24 [5] drill commit ce80a4a74a2b: navidrome deluan/navidrome:0.64.0 -> deluan/navidrome:0.64.1 (push rc=0) +22:54:29 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +22:54:29 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +22:54:30 + 1.1s phase=pulling label=Új verzió letöltése… +22:54:31 + 2.1s phase=copying label=Az adatok másolása a frissítés előtt… +22:54:32 + 3.2s phase=starting label=Indítás az új verzióval… +22:54:32 + 3.7s phase=verifying label=Működés ellenőrzése… +22:54:37 >>> finished-marker removed from one undo copy: +copy: navidrome_navidrome_data.pre-update-20260923T205431Z +data + +22:54:51 >>> ACCIDENT box_backup at 22:54:51 +22:54:51 >>> POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'} at 22:54:51 +22:56:04 + 94.9s phase=undoing label=Visszaállítás az előző változatra… +22:56:04 + 95.4s phase=failed label=A frissítés nem sikerült +22:56:04 END phase=failed err='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) navidrome frissítése 2026-09-23 22:56-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:31 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.' hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) navidrome frissítése 2026-09-23 22:56-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:31 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.' +22:56:04 + 101.2s navidrome: deployed=True state=stopped updating=False phase=failed hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új v' +22:59:09 app never answered on music/ (last rc=0 code=404) +23:02:10 app never answered on music/ (last rc=0 code=404) diff --git a/documentation/audits/night-2026-09-23/chaos/round-09-run.txt b/documentation/audits/night-2026-09-23/chaos/round-09-run.txt new file mode 100644 index 00000000..190c9407 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-09-run.txt @@ -0,0 +1,17 @@ +23:03:08 ==== ROUND 9: fail_health on vikunja x controller_kill at +58s (2026-09-23T21:03:08Z) +23:03:09 [5] drill commit 16512923234a: vikunja vikunja/vikunja:2.5.0 -> vikunja/vikunja:2.6.0 (push rc=0) +23:03:13 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +23:03:13 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +23:03:14 + 0.6s phase=pulling label=Új verzió letöltése… +23:03:17 + 3.7s phase=copying label=Az adatok másolása a frissítés előtt… +23:03:17 + 4.2s phase=verifying label=Működés ellenőrzése… +23:04:11 >>> ACCIDENT controller_kill at 23:04:11 +23:04:14 >>> kill -9 of the controller's PID: killed 3308 at 23:04:11 +23:05:45 + 152.0s phase=undoing label=Visszaállítás az előző változatra… +23:05:51 + 157.7s phase=undone label=Visszaállítva az előző változatra +23:05:51 END phase=undone err=None hold=None +23:05:51 + 163.3s vikunja: deployed=True state=running updating=False phase=undone hold='None' +23:05:51 + 163.4s vikunja: deployed=True state=running updating=False phase=undone hold='None' +23:05:55 vikunja: readback of the seeded project http=200 ok=True +23:05:55 vikunja B: project readback=True attachment content readback=True +23:05:59 ==== ROUND 9 END: steady in 163.4s; readback={'A': True, 'B': True}; events=1 diff --git a/documentation/audits/night-2026-09-23/chaos/round-09.json b/documentation/audits/night-2026-09-23/chaos/round-09.json new file mode 100644 index 00000000..db041e0c --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-09.json @@ -0,0 +1,189 @@ +{ + "round": 9, + "action": "fail_health", + "accident": "kill -9 of the controller's PID: killed 3308 at 23:04:11", + "delay_s": 58, + "app": "vikunja", + "started": "2026-09-23T21:03:08Z", + "before": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "drill_commit": "16512923234a", + "badge_catchup_s": 4.4, + "armed_at": "23:03:13", + "phases": [ + [ + 0.0, + "safety-dump", + "Adatbázis pillanatkép…" + ], + [ + 0.6, + "pulling", + "Új verzió letöltése…" + ], + [ + 3.7, + "copying", + "Az adatok másolása a frissítés előtt…" + ], + [ + 4.2, + "verifying", + "Működés ellenőrzése…" + ], + [ + 152.0, + "undoing", + "Visszaállítás az előző változatra…" + ], + [ + 157.7, + "undone", + "Visszaállítva az előző változatra" + ] + ], + "controller_back_s": 0.2, + "time_to_steady_s": 163.4, + "after": { + "badges": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "lines": { + "hu": { + "undone_line": "A(z) vikunja frissítése 2026-09-23 23:05-kor nem sikerült. A doboz automatikusan visszaállította az előző változatot és az adatokat — semmi nem veszett el.", + "hold": null + }, + "en": { + "undone_line": "The update of vikunja at 2026-09-23 23:05 did not succeed. The box put back the previous version and its data automatically — nothing was lost.", + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": "undone", + "hold_reason": null, + "update_error": null + } + }, + "observables": { + "pinned_images": { + "vikunja": "vikunja/vikunja:2.5.0" + }, + "installed_images": { + "vikunja": "vikunja/vikunja:2.5.0" + }, + "catalog_images": { + "vikunja": "vikunja/vikunja:2.6.0" + }, + "live_compose_image_lines": [ + "image: vikunja/vikunja:2.5.0" + ], + "docker_inspect": [ + "vikunja vikunja/vikunja:2.5.0 running=true restarts=0" + ] + }, + "readback": { + "A": true, + "B": true + }, + "events": [ + "2026-09-23T21:05:51Z app_update_undone (warning)" + ], + "box_log": [ + "2026-09-23T21:03:09Z INFO [sync] Updated vikunja/.felhom.yml", + "2026-09-23T21:03:09Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:03:09Z INFO [api] Catalog sync completed: Sablonok frissítve — frissítve: vikunja", + "2026-09-23T21:03:11Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:03:13Z INFO [api] update requested for stack: vikunja", + "2026-09-23T21:03:13Z INFO [stacks] update vikunja: accepted — guarded update started", + "2026-09-23T21:03:13Z INFO [stacks] update vikunja: phase checking", + "2026-09-23T21:03:13Z INFO [stacks] update vikunja: precondition met — Tier 1 (own recovery unit) copy from 2026-09-23T20:56:25Z (7m0s old, limit 24h0m0s)", + "2026-09-23T21:03:13Z INFO [stacks] update vikunja: phase safety-dump", + "2026-09-23T21:03:13Z INFO [backup] Discovered 5 databases", + "2026-09-23T21:03:13Z INFO [backup] update safety dump for vikunja: the app has no database — nothing to copy (no-op)", + "2026-09-23T21:03:13Z INFO [stacks] update vikunja: safety dump done (0 file(s)) []", + "2026-09-23T21:03:13Z INFO [stacks] update vikunja: the undo copy will hold 0 named volume(s), 0.0 MiB", + "2026-09-23T21:03:13Z INFO [stacks] update vikunja: phase pinning", + "2026-09-23T21:03:13Z INFO [stacks] update vikunja: pin advanced to the catalog's current definition (vikunja=vikunja/vikunja:2.6.0)", + "2026-09-23T21:03:13Z INFO [stacks] update vikunja: phase pulling", + "2026-09-23T21:03:16Z INFO [stacks] update vikunja: phase copying", + "2026-09-23T21:03:17Z INFO [stacks] update vikunja: phase starting", + "2026-09-23T21:03:17Z INFO [stacks] update vikunja: phase verifying", + "2026-09-23T21:03:17Z WARN Health probe vikunja: API GET :8999/api/v1/info → Get \"http://vikunja:8999/api/v1/info\": dial tcp 172.18.0.2:8999: connect: connection refused", + "2026-09-23T21:03:35Z WARN Health probe vikunja: API GET :8999/api/v1/info → Get \"http://vikunja:8999/api/v1/info\": dial tcp 172.18.0.2:8999: connect: connection refused", + "2026-09-23T21:03:45Z INFO [scheduler] Running job: backup-cache", + "2026-09-23T21:04:14Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:04:14Z WARN [stacks] update recovery: vikunja was interrupted in verifying (started 2026-09-23T21:03:13Z) — the new version may have run; marking it Updating and RESUMING the health wait", + "2026-09-23T21:04:14Z WARN [update] 1 interrupted update(s) will resume after the backup side is wired: [vikunja]", + "2026-09-23T21:04:14Z INFO [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false)", + "2026-09-23T21:04:14Z INFO [scheduler] Registered periodic job: backup-cache (every 5m0s)", + "2026-09-23T21:04:14Z INFO [scheduler] Daily job tier2-backup scheduled for 2026-09-24 03:30 CEST", + "2026-09-23T21:04:14Z INFO [scheduler] Daily job offbox-backup scheduled for 2026-09-24 04:15 CEST", + "2026-09-23T21:04:14Z INFO [stacks] update vikunja: resuming after a controller restart — `up -d` then the health wait", + "2026-09-23T21:04:14Z INFO [backup] Found 8 DB dump files across drives", + "2026-09-23T21:04:14Z INFO [stacks] update vikunja: phase verifying", + "2026-09-23T21:04:14Z WARN Health probe vikunja: API GET :8999/api/v1/info → Get \"http://vikunja:8999/api/v1/info\": dial tcp 172.18.0.2:8999: connect: connection refused", + "2026-09-23T21:04:14Z INFO [backup] Discovered 5 databases", + "2026-09-23T21:04:14Z INFO [backup] Backup status cache refreshed", + "2026-09-23T21:04:15Z WARN [api] Unauthorized request to /api/stacks/vikunja from 172.18.0.12:57278", + "2026-09-23T21:04:34Z WARN Health probe vikunja: API GET :8999/api/v1/info → Get \"http://vikunja:8999/api/v1/info\": dial tcp 172.18.0.2:8999: connect: connection refused", + "2026-09-23T21:04:44Z WARN Health probe vikunja: API GET :8999/api/v1/info → Get \"http://vikunja:8999/api/v1/info\": dial tcp 172.18.0.2:8999: connect: connection refused", + "2026-09-23T21:04:54Z WARN Health probe vikunja: API GET :8999/api/v1/info → Get \"http://vikunja:8999/api/v1/info\": dial tcp 172.18.0.2:8999: connect: connection refused", + "2026-09-23T21:05:14Z WARN Health probe vikunja: API GET :8999/api/v1/info → Get \"http://vikunja:8999/api/v1/info\": dial tcp 172.18.0.2:8999: connect: connection refused", + "2026-09-23T21:05:24Z WARN Health probe vikunja: API GET :8999/api/v1/info → Get \"http://vikunja:8999/api/v1/info\": dial tcp 172.18.0.2:8999: connect: connection refused", + "2026-09-23T21:05:44Z WARN Health probe vikunja: API GET :8999/api/v1/info → Get \"http://vikunja:8999/api/v1/info\": dial tcp 172.18.0.2:8999: connect: connection refused", + "2026-09-23T21:05:45Z ERROR [stacks] update vikunja FAILED after the new version was started: not healthy: not healthy within 1m30s (last: state unhealthy)", + "2026-09-23T21:05:45Z INFO [stacks] update vikunja: kept 1299 bytes of the app's own log at /opt/docker/stacks/vikunja/hold-logs/20260923T210545Z/compose-logs.txt before stopping it (R-621)", + "2026-09-23T21:05:45Z INFO [stacks] update vikunja: phase undoing", + "2026-09-23T21:05:45Z WARN [stacks] update vikunja: UNDO — putting back the previous version and its 0 volume copy(ies) (reason: not healthy: not healthy within 1m30s (last: state unhealthy))", + "2026-09-23T21:05:45Z INFO [stacks] pin vikunja: vikunja=vikunja/vikunja:2.5.0", + "2026-09-23T21:05:46Z WARN Health probe vikunja: API GET :3456/api/v1/info → Get \"http://vikunja:3456/api/v1/info\": dial tcp 172.18.0.2:3456: connect: connection refused", + "2026-09-23T21:05:51Z INFO [stacks] update vikunja: event app_update_undone", + "2026-09-23T21:05:51Z WARN notifier disabled (no hub configured): DROPPED event app_update_undone (severity warning) — further app_update_undone events are logged at DEBUG only", + "2026-09-23T21:05:51Z INFO [stacks] update vikunja: UNDONE in 6s — the previous version is running on the data from before the update (the app's health check passed)" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-09.log b/documentation/audits/night-2026-09-23/chaos/round-09.log new file mode 100644 index 00000000..a6d1c989 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-09.log @@ -0,0 +1,16 @@ +23:03:08 ==== ROUND 9: fail_health on vikunja x controller_kill at +58s (2026-09-23T21:03:08Z) +23:03:09 [5] drill commit 16512923234a: vikunja vikunja/vikunja:2.5.0 -> vikunja/vikunja:2.6.0 (push rc=0) +23:03:13 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +23:03:13 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +23:03:14 + 0.6s phase=pulling label=Új verzió letöltése… +23:03:17 + 3.7s phase=copying label=Az adatok másolása a frissítés előtt… +23:03:17 + 4.2s phase=verifying label=Működés ellenőrzése… +23:04:11 >>> ACCIDENT controller_kill at 23:04:11 +23:04:14 >>> kill -9 of the controller's PID: killed 3308 at 23:04:11 +23:05:45 + 152.0s phase=undoing label=Visszaállítás az előző változatra… +23:05:51 + 157.7s phase=undone label=Visszaállítva az előző változatra +23:05:51 END phase=undone err=None hold=None +23:05:51 + 163.3s vikunja: deployed=True state=running updating=False phase=undone hold='None' +23:05:51 + 163.4s vikunja: deployed=True state=running updating=False phase=undone hold='None' +23:05:55 vikunja: readback of the seeded project http=200 ok=True +23:05:55 vikunja B: project readback=True attachment content readback=True diff --git a/documentation/audits/night-2026-09-23/chaos/round-10-run.txt b/documentation/audits/night-2026-09-23/chaos/round-10-run.txt new file mode 100644 index 00000000..a39f541a --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-10-run.txt @@ -0,0 +1,10 @@ +23:06:00 ==== ROUND 10: remove on actualbudget x docker_restart at +10s (2026-09-23T21:06:00Z) +23:06:00 [X] stop -> 200 {'ok': True, 'message': 'Stack actualbudget stop completed'} +23:06:10 >>> ACCIDENT docker_restart at 23:06:10 +23:06:12 [X] remove (with drive data) -> 502 {'_raw': 'Bad Gateway'} +23:06:23 >>> systemctl restart docker: rc=0 at 23:06:10 +23:06:23 [X] after remove: deployed=None leftovers='/opt/docker/stacks/actualbudget' +23:06:24 + 24.0s actualbudget: deployed=True state=stopped updating=False phase=None hold='None' +23:34:05 app never answered on budget/ (last rc=0 code=404) +23:40:07 app never answered on budget/ (last rc=0 code=404) +23:40:10 ==== ROUND 10 END: steady in 1500.4s; readback={'A': False}; events=0 diff --git a/documentation/audits/night-2026-09-23/chaos/round-10.json b/documentation/audits/night-2026-09-23/chaos/round-10.json new file mode 100644 index 00000000..76022c57 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-10.json @@ -0,0 +1,126 @@ +{ + "round": 10, + "action": "remove", + "accident": "systemctl restart docker: rc=0 at 23:06:10", + "delay_s": 10, + "app": "actualbudget", + "started": "2026-09-23T21:06:00Z", + "before": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "armed_at": "23:06:00", + "act": "remove -> 502", + "controller_back_s": 0.2, + "time_to_steady_s": 1500.4, + "after": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "stopped", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "observables": { + "pinned_images": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "installed_images": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "catalog_images": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "live_compose_image_lines": [ + "image: actualbudget/actual-server:26.9.0" + ], + "docker_inspect": [] + }, + "readback": { + "A": false + }, + "events": [], + "box_log": [ + "2026-09-23T21:22:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:24:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:26:22Z INFO [scheduler] Running job: backup-cache", + "2026-09-23T21:26:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:26:23Z INFO [backup] Found 8 DB dump files across drives", + "2026-09-23T21:26:23Z INFO [backup] Discovered 5 databases", + "2026-09-23T21:26:23Z INFO [backup] Backup status cache refreshed", + "2026-09-23T21:26:23Z INFO [scheduler] Job backup-cache completed (took 274ms)", + "2026-09-23T21:27:23Z INFO [deadapp] check alive: 40 scans since boot, 11 deployed app(s) evaluated, 1 currently down", + "2026-09-23T21:28:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:30:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:31:22Z INFO [scheduler] Running job: backup-cache", + "2026-09-23T21:31:23Z INFO [backup] Found 8 DB dump files across drives", + "2026-09-23T21:31:23Z INFO [backup] Discovered 5 databases", + "2026-09-23T21:31:23Z INFO [backup] Backup status cache refreshed", + "2026-09-23T21:31:23Z INFO [scheduler] Job backup-cache completed (took 206ms)", + "2026-09-23T21:32:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:34:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:36:22Z INFO [scheduler] Running job: backup-cache", + "2026-09-23T21:36:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:36:23Z INFO [backup] Found 8 DB dump files across drives", + "2026-09-23T21:36:23Z INFO [backup] Discovered 5 databases", + "2026-09-23T21:36:23Z INFO [backup] Backup status cache refreshed", + "2026-09-23T21:36:23Z INFO [scheduler] Job backup-cache completed (took 281ms)", + "2026-09-23T21:37:23Z INFO [deadapp] check alive: 60 scans since boot, 11 deployed app(s) evaluated, 1 currently down", + "2026-09-23T21:38:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-10.log b/documentation/audits/night-2026-09-23/chaos/round-10.log new file mode 100644 index 00000000..6ca6284f --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-10.log @@ -0,0 +1,9 @@ +23:06:00 ==== ROUND 10: remove on actualbudget x docker_restart at +10s (2026-09-23T21:06:00Z) +23:06:00 [X] stop -> 200 {'ok': True, 'message': 'Stack actualbudget stop completed'} +23:06:10 >>> ACCIDENT docker_restart at 23:06:10 +23:06:12 [X] remove (with drive data) -> 502 {'_raw': 'Bad Gateway'} +23:06:23 >>> systemctl restart docker: rc=0 at 23:06:10 +23:06:23 [X] after remove: deployed=None leftovers='/opt/docker/stacks/actualbudget' +23:06:24 + 24.0s actualbudget: deployed=True state=stopped updating=False phase=None hold='None' +23:34:05 app never answered on budget/ (last rc=0 code=404) +23:40:07 app never answered on budget/ (last rc=0 code=404) diff --git a/documentation/audits/night-2026-09-23/chaos/round-11-controller.log b/documentation/audits/night-2026-09-23/chaos/round-11-controller.log new file mode 100644 index 00000000..d887da80 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-11-controller.log @@ -0,0 +1,3000 @@ +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:40:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:40:23 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:40:23 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:40:23 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:40:23 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:40:23 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 91ms) +2026/09/23 21:40:26 router.go:1048: [INFO] [api] Manual catalog sync requested +2026/09/23 21:40:27 router.go:362: [INFO] [api] Manual stack rescan requested +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:40:27 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:40:27 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:40:27 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:40:27 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:40:27 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:40:27 router.go:369: [INFO] [api] Stack rescan completed: 55 stacks found +2026/09/23 21:40:32 router.go:1048: [INFO] [api] Manual catalog sync requested +2026/09/23 21:40:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:40:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (9 skipped not due, 0 skipped no container) +2026/09/23 21:40:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:40:33 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:40:33 router.go:362: [INFO] [api] Manual stack rescan requested +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:40:33 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:40:33 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:40:33 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:40:33 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:40:33 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:40:33 router.go:369: [INFO] [api] Stack rescan completed: 55 stacks found +2026/09/23 21:40:38 router.go:1048: [INFO] [api] Manual catalog sync requested +2026/09/23 21:40:39 router.go:362: [INFO] [api] Manual stack rescan requested +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:40:39 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:40:40 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:40:40 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:40:40 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:40:40 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:40:40 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:40:40 router.go:369: [INFO] [api] Stack rescan completed: 55 stacks found +2026/09/23 21:40:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:40:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:40:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (9 skipped not due, 0 skipped no container) +2026/09/23 21:40:43 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:40:45 router.go:1048: [INFO] [api] Manual catalog sync requested +2026/09/23 21:40:45 sync.go:208: [INFO] [sync] Starting catalog sync +2026/09/23 21:40:45 sync.go:296: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-drill.git (branch: main) +2026/09/23 21:40:45 sync.go:298: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/09/23 21:40:45 sync.go:592: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/09/23 21:40:45 sync.go:592: [DEBUG] [sync] Running: git reset --hard origin/main +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:509: [DEBUG] [sync] nextcloud: catalog has moved past the pin — rendering the stored applied definition +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:396: [INFO] [sync] Updated nextcloud/.felhom.yml +2026/09/23 21:40:45 sync.go:541: [DEBUG] [sync] nextcloud/.felhom.yml: src=665afc1dcb53324c, dst=665afc1dcb53324c (changed) +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:509: [DEBUG] [sync] romm: catalog has moved past the pin — rendering the stored applied definition +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/09/23 21:40:45 sync.go:416: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:40:45 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:40:45 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:40:45 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:40:45 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:40:45 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:40:45 sync.go:244: [DEBUG] [sync] Post-sync hook: triggering missing field injection for 1 stack(s): [nextcloud] +2026/09/23 21:40:45 deploy.go:1104: [DEBUG] [stacks] InjectMissingFields: checking 1 stacks +2026/09/23 21:40:45 deploy.go:1126: [DEBUG] [stacks] InjectMissingFields: checking stack nextcloud — 7 deploy fields, 7 existing env vars +2026/09/23 21:40:45 deploy.go:1189: [INFO] [stacks] InjectMissingFields: processed 1 stacks +2026/09/23 21:40:45 sync.go:263: [INFO] [sync] Catalog sync complete +2026/09/23 21:40:45 router.go:1054: [INFO] [api] Catalog sync completed: Sablonok frissítve — frissítve: nextcloud +2026/09/23 21:40:46 router.go:362: [INFO] [api] Manual stack rescan requested +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:40:46 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:40:46 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:40:46 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:40:46 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:40:46 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:40:46 router.go:369: [INFO] [api] Stack rescan completed: 55 stacks found +2026/09/23 21:40:46 router.go:591: [INFO] [api] update requested for stack: nextcloud +2026/09/23 21:40:46 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6473MB avail≈19425MB +2026/09/23 21:40:46 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23037272 → total=25898MB avail=19425MB used=6473MB (25.0%) +2026/09/23 21:40:46 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1310MB, new_req=256MB, remaining=23948MB +2026/09/23 21:40:46 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=19.1 GB avail=45.8GB (27.9%) +2026/09/23 21:40:46 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6481MB avail≈19417MB +2026/09/23 21:40:46 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23026696 → total=25898MB avail=19417MB used=6481MB (25.0%) +2026/09/23 21:40:46 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1310MB, new_req=256MB, remaining=23948MB +2026/09/23 21:40:46 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=19.1 GB avail=45.8GB (27.9%) +2026/09/23 21:40:46 update.go:488: [INFO] [stacks] update nextcloud: accepted — guarded update started +2026/09/23 21:40:46 update.go:1192: [INFO] [stacks] update nextcloud: phase checking +2026/09/23 21:40:46 update_guard.go:248: [DEBUG] [backup] update precondition for nextcloud: no Tier-2 copy (nincs másodlagos fájlmásolat ehhez az alkalmazáshoz) +2026/09/23 21:40:46 update.go:710: [INFO] [stacks] update nextcloud: precondition met — Tier 1 (own recovery unit) copy from 2026-09-23T20:55:33Z (45m0s old, limit 24h0m0s) +2026/09/23 21:40:46 update.go:1192: [INFO] [stacks] update nextcloud: phase safety-dump +2026/09/23 21:40:46 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/23 21:40:46 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: d6ac44f34971 vikunja vikunja vikunja/vikunja:2.5.0 +8a54f14e8bb0 navidrome navidrome deluan/navidrome:0.64.0 +9c3fff6d8323 romm romm rommapp/romm:5.3.1 +581a28cd0e38 romm-db romm mariadb:11.4 +f428c67086c0 romm-redis romm redis:7-alpine +f12b35ea40d5 privatebin privatebin privatebin/pdo:2.0.6 +20daa65aee34 papra papra ghcr.io/papra-hq/papra:26.6.2-rootless +c916eed5d29a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +c509e115095a paperless-postgres paperless-ngx postgres:1... +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.5.0, not a database) +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.0, not a database) +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/23 21:40:46 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=581a28cd0e38) +2026/09/23 21:40:46 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container papra (image=ghcr.io/papra-hq/papra:26.6.2-rootless, not a database) +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/23 21:40:46 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=c509e115095a) +2026/09/23 21:40:46 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/23 21:40:46 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.1-apache, not a database) +2026/09/23 21:40:46 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=d1237ad5a22f) +2026/09/23 21:40:46 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost (image=docmost/docmost:0.96.0, not a database) +2026/09/23 21:40:46 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: docmost-postgres (id=c981df80d790) +2026/09/23 21:40:46 dbdump.go:198: [DEBUG] DiscoverDatabases: docmost-postgres → stack=docmost, dbUser=docmost, dbName=docmost +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost-redis (image=redis:7-alpine, not a database) +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, not a database) +2026/09/23 21:40:46 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: adventurelog-postgres (id=5abd52db34b8) +2026/09/23 21:40:46 dbdump.go:198: [DEBUG] DiscoverDatabases: adventurelog-postgres → stack=adventurelog, dbUser=adventurelog, dbName=adventurelog +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, not a database) +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.267.0, not a database) +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/23 21:40:46 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/23 21:40:46 dbdump.go:205: [DEBUG] DiscoverDatabases: found 5 database(s), skipped 17 non-DB container(s) +2026/09/23 21:40:46 dbdump.go:208: [INFO] [backup] Discovered 5 databases +2026/09/23 21:40:46 dbdump.go:266: [DEBUG] DumpOne: starting dump for container=nextcloud-db, stack=nextcloud, dbType=mariadb, dest=/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/pre-restore-20260923T214046Z-nextcloud-mariadb.sql +2026/09/23 21:40:46 dbdump.go:321: [DEBUG] DumpOne: mariadb-dump command: docker exec d1237ad5a22f mariadb-dump -u root -p*** --single-transaction --routines --triggers nextcloud +2026/09/23 21:40:47 dbdump.go:405: [DEBUG] DumpOne: completed nextcloud-db → pre-restore-20260923T214046Z-nextcloud-mariadb.sql (size=516.8 KB, valid=true, tables=131, duration=477ms) +2026/09/23 21:40:47 dbdump.go:410: [INFO] [backup] DB dump: nextcloud-db → pre-restore-20260923T214046Z-nextcloud-mariadb.sql (516.8 KB, 477ms, 131 tables) +2026/09/23 21:40:47 offbox_reconstitute.go:255: [INFO] [offbox] nextcloud: pre-restore safety dump written → pre-restore-20260923T214046Z-nextcloud-mariadb.sql (516.8 KB) +2026/09/23 21:40:47 update_guard.go:478: [INFO] [backup] update safety dump for nextcloud: 1 file(s) [/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/pre-restore-20260923T214046Z-nextcloud-mariadb.sql] +2026/09/23 21:40:47 update.go:742: [INFO] [stacks] update nextcloud: safety dump done (1 file(s)) [/mnt/felhom-drives/scratch_hdd/userdata/nextcloud/backups/primary/nextcloud/db-dumps/pre-restore-20260923T214046Z-nextcloud-mariadb.sql] +2026/09/23 21:40:48 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=19.1 GB avail=45.8GB (27.9%) +2026/09/23 21:40:48 undo.go:252: [INFO] [stacks] update nextcloud: the undo copy will hold 3 named volume(s), 910.1 MiB +2026/09/23 21:40:48 update.go:1192: [INFO] [stacks] update nextcloud: phase pinning +2026/09/23 21:40:48 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/nextcloud — 7 env vars, 0 encrypted, 3 sensitive fields +2026/09/23 21:40:48 [INFO] [stacks] SaveAppConfig: saved config for nextcloud +2026/09/23 21:40:48 pin.go:365: [INFO] [stacks] update nextcloud: pin advanced to the catalog's current definition (nextcloud=nextcloud:34.0.4-apache, nextcloud-db=mariadb:12.3, nextcloud-redis=redis:7-alpine) +2026/09/23 21:40:48 update.go:1192: [INFO] [stacks] update nextcloud: phase pulling +2026/09/23 21:40:48 manager.go:1419: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/23 21:40:48 manager.go:1428: [DEBUG] Running: docker compose pull (in /opt/docker/stacks/nextcloud) +2026/09/23 21:40:50 manager.go:1448: [DEBUG] Command completed: docker compose pull (took 1.2s) +2026/09/23 21:40:50 update.go:1192: [INFO] [stacks] update nextcloud: phase copying +2026/09/23 21:40:50 manager.go:1419: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/23 21:40:50 manager.go:1428: [DEBUG] Running: docker compose stop (in /opt/docker/stacks/nextcloud) +2026/09/23 21:40:51 manager.go:1448: [DEBUG] Command completed: docker compose stop (took 1.6s) +2026/09/23 21:40:51 update.go:1192: [INFO] [stacks] update nextcloud: phase copying +2026/09/23 21:40:52 undo.go:279: [INFO] [stacks] update nextcloud: copied nextcloud_nextcloud_db_data → nextcloud_nextcloud_db_data.pre-update-20260923T214051Z in 925ms +2026/09/23 21:40:52 update.go:1192: [INFO] [stacks] update nextcloud: phase copying +2026/09/23 21:40:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:40:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:40:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:40:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:40:53 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:40:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:40:53 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:40:59 undo.go:279: [INFO] [stacks] update nextcloud: copied nextcloud_nextcloud_html → nextcloud_nextcloud_html.pre-update-20260923T214051Z in 6.474s +2026/09/23 21:40:59 update.go:1192: [INFO] [stacks] update nextcloud: phase copying +2026/09/23 21:40:59 undo.go:279: [INFO] [stacks] update nextcloud: copied nextcloud_nextcloud_redis_data → nextcloud_nextcloud_redis_data.pre-update-20260923T214051Z in 442ms +2026/09/23 21:40:59 update.go:1192: [INFO] [stacks] update nextcloud: phase starting +2026/09/23 21:40:59 manager.go:387: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/nextcloud not mounted — skipping ensure (held by drive gate) +2026/09/23 21:40:59 manager.go:1419: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/23 21:40:59 manager.go:1428: [DEBUG] Running: docker compose up -d --remove-orphans (in /opt/docker/stacks/nextcloud) +2026/09/23 21:41:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:41:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:41:03 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:03 manager.go:816: [DEBUG] [stacks] restart-policy of down member "nextcloud" = "unless-stopped" +2026/09/23 21:41:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:03 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (9 skipped not due, 0 skipped no container) +2026/09/23 21:41:05 manager.go:1448: [DEBUG] Command completed: docker compose up -d --remove-orphans (took 6.2s) +2026/09/23 21:41:05 update.go:1192: [INFO] [stacks] update nextcloud: phase verifying +2026/09/23 21:41:05 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:10 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:41:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:41:13 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:13 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:41:15 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:20 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:41:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6448MB avail≈19450MB +2026/09/23 21:41:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23067028 → total=25898MB avail=19450MB used=6448MB (24.9%) +2026/09/23 21:41:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:41:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:41:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.03 1.11 1.11 4/1778 6779" → 1m=2.03 5m=1.11 15m=1.11 +2026/09/23 21:41:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:41:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:41:22 info.go:13: [DEBUG] [system] GetInfo done in 68ms — mem=6448MB/25898MB (24.9%), rootDisk=20.0GB/68.4GB (29.3%), load=2.03/1.11/1.11, temp=49.9°C (hwmon1), cpu=14.5% +2026/09/23 21:41:22 scheduler.go:346: [INFO] [scheduler] Running job: system-health +2026/09/23 21:41:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:41:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:41:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:41:22 scheduler.go:67: [DEBUG] [scheduler] job system-health: execution starting +2026/09/23 21:41:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:41:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:41:22 scheduler.go:346: [INFO] [scheduler] Running job: offsite-credential-retry +2026/09/23 21:41:22 scheduler.go:67: [DEBUG] [scheduler] job offsite-credential-retry: execution starting +2026/09/23 21:41:22 scheduler.go:363: [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/09/23 21:41:22 scheduler.go:346: [INFO] [scheduler] Running job: backup-cache +2026/09/23 21:41:22 scheduler.go:67: [DEBUG] [scheduler] job backup-cache: execution starting +2026/09/23 21:41:23 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:23 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:41:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/navidrome → [navidrome] +2026/09/23 21:41:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/nextcloud → [nextcloud] +2026/09/23 21:41:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx → [paperless-ngx] +2026/09/23 21:41:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/romm → [romm] +2026/09/23 21:41:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [actualbudget, adventurelog, docmost, gokapi, papra, privatebin, vikunja] +2026/09/23 21:41:23 settings.go:815: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (11773 bytes) +2026/09/23 21:41:23 settings.go:818: [INFO] [settings] Settings saved +2026/09/23 21:41:23 backup.go:1117: [INFO] [backup] Found 9 DB dump files across drives +2026/09/23 21:41:23 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6469MB avail≈19429MB +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23041044 → total=25898MB avail=19429MB used=6469MB (25.0%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.95 1.11 1.11 7/1801 6842" → 1m=1.95 5m=1.11 15m=1.11 +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 57.0°C (hwmon2) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetInfo done in 82ms — mem=6469MB/25898MB (25.0%), rootDisk=20.0GB/68.4GB (29.3%), load=1.95/1.11/1.11, temp=57.0°C (hwmon2), cpu=18.3% +2026/09/23 21:41:23 healthcheck.go:86: [DEBUG] [monitor] Raw values: disk=29.3%, hdd=11.0% (configured=true), mem=25.0% (6469MB/25898MB), cpu=18.3%, temp=57.0°C (hwmon2) +2026/09/23 21:41:23 healthcheck.go:118: [DEBUG] [monitor] SSD disk: OK (29%) +2026/09/23 21:41:23 healthcheck.go:151: [DEBUG] [monitor] Memory: OK (25%) +2026/09/23 21:41:23 healthcheck.go:169: [DEBUG] [monitor] CPU: OK (18%) +2026/09/23 21:41:23 healthcheck.go:187: [DEBUG] [monitor] Temperature: OK (57°C) +2026/09/23 21:41:23 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 3381ff86c786 nextcloud nextcloud nextcloud:34.0.4-apache +d6ac44f34971 vikunja vikunja vikunja/vikunja:2.5.0 +8a54f14e8bb0 navidrome navidrome deluan/navidrome:0.64.0 +9c3fff6d8323 romm romm rommapp/romm:5.3.1 +581a28cd0e38 romm-db romm mariadb:11.4 +f428c67086c0 romm-redis romm redis:7-alpine +f12b35ea40d5 privatebin privatebin privatebin/pdo:2.0.6 +20daa65aee34 papra papra ghcr.io/papra-hq/papra:26.6.2-rootless +c916eed5d29a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15... +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.5.0, not a database) +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.0, not a database) +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/23 21:41:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=581a28cd0e38) +2026/09/23 21:41:23 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container papra (image=ghcr.io/papra-hq/papra:26.6.2-rootless, not a database) +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/23 21:41:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=c509e115095a) +2026/09/23 21:41:23 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/23 21:41:23 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/23 21:41:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=d1237ad5a22f) +2026/09/23 21:41:23 healthcheck.go:201: [DEBUG] [monitor] Docker daemon: OK +2026/09/23 21:41:23 healthcheck.go:209: [DEBUG] [monitor] Checking 1 protected containers: [traefik] +2026/09/23 21:41:23 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost (image=docmost/docmost:0.96.0, not a database) +2026/09/23 21:41:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: docmost-postgres (id=c981df80d790) +2026/09/23 21:41:23 healthcheck.go:219: [DEBUG] [monitor] All protected containers running +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:41:23 healthcheck.go:238: [INFO] [monitor] Health check: status=warn +2026/09/23 21:41:23 healthcheck.go:242: [DEBUG] [monitor] Final status: warn (issues=0, warnings=4, info=5) +2026/09/23 21:41:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:41:23 scheduler.go:363: [INFO] [scheduler] Job system-health completed (took 162ms) +2026/09/23 21:41:23 dbdump.go:198: [DEBUG] DiscoverDatabases: docmost-postgres → stack=docmost, dbUser=docmost, dbName=docmost +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost-redis (image=redis:7-alpine, not a database) +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, not a database) +2026/09/23 21:41:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: adventurelog-postgres (id=5abd52db34b8) +2026/09/23 21:41:23 dbdump.go:198: [DEBUG] DiscoverDatabases: adventurelog-postgres → stack=adventurelog, dbUser=adventurelog, dbName=adventurelog +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, not a database) +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.267.0, not a database) +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/23 21:41:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/23 21:41:23 dbdump.go:205: [DEBUG] DiscoverDatabases: found 5 database(s), skipped 17 non-DB container(s) +2026/09/23 21:41:23 dbdump.go:208: [INFO] [backup] Discovered 5 databases +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:41:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:41:23 [INFO] [backup] Discovered app data: 11 apps +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:41:23 infra.go:67: [INFO] [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:41:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:41:23 backup.go:1187: [INFO] [backup] Backup status cache refreshed +2026/09/23 21:41:23 scheduler.go:363: [INFO] [scheduler] Job backup-cache completed (took 234ms) +2026/09/23 21:41:25 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:30 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:41:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:41:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:41:33 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:36 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:36 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.3:8999: connect: connection refused +2026/09/23 21:41:37 router.go:591: [INFO] [api] update requested for stack: romm +2026/09/23 21:41:37 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6439MB avail≈19459MB +2026/09/23 21:41:37 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23077072 → total=25898MB avail=19459MB used=6439MB (24.9%) +2026/09/23 21:41:37 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1266MB, new_req=300MB, remaining=23948MB +2026/09/23 21:41:37 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:41:37 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6445MB avail≈19453MB +2026/09/23 21:41:37 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23070468 → total=25898MB avail=19453MB used=6445MB (24.9%) +2026/09/23 21:41:37 deploy.go:1250: [INFO] [stacks] Memory check: total=25898MB, reserved=384MB, usable=25514MB, committed_used=1266MB, new_req=300MB, remaining=23948MB +2026/09/23 21:41:37 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:41:37 update.go:488: [INFO] [stacks] update romm: accepted — guarded update started +2026/09/23 21:41:37 update.go:1192: [INFO] [stacks] update romm: phase checking +2026/09/23 21:41:37 update_guard.go:248: [DEBUG] [backup] update precondition for romm: no Tier-2 copy (nincs másodlagos fájlmásolat ehhez az alkalmazáshoz) +2026/09/23 21:41:37 update.go:710: [INFO] [stacks] update romm: precondition met — Tier 1 (own recovery unit) copy from 2026-09-23T20:56:12Z (45m0s old, limit 24h0m0s) +2026/09/23 21:41:37 update.go:1192: [INFO] [stacks] update romm: phase safety-dump +2026/09/23 21:41:37 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/23 21:41:37 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 3381ff86c786 nextcloud nextcloud nextcloud:34.0.4-apache +d6ac44f34971 vikunja vikunja vikunja/vikunja:2.5.0 +8a54f14e8bb0 navidrome navidrome deluan/navidrome:0.64.0 +9c3fff6d8323 romm romm rommapp/romm:5.3.1 +581a28cd0e38 romm-db romm mariadb:11.4 +f428c67086c0 romm-redis romm redis:7-alpine +f12b35ea40d5 privatebin privatebin privatebin/pdo:2.0.6 +20daa65aee34 papra papra ghcr.io/papra-hq/papra:26.6.2-rootless +c916eed5d29a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15... +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud (image=nextcloud:34.0.4-apache, not a database) +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.5.0, not a database) +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.0, not a database) +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/23 21:41:37 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=581a28cd0e38) +2026/09/23 21:41:37 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container papra (image=ghcr.io/papra-hq/papra:26.6.2-rootless, not a database) +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/23 21:41:37 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=c509e115095a) +2026/09/23 21:41:37 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/23 21:41:37 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/23 21:41:37 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: nextcloud-db (id=d1237ad5a22f) +2026/09/23 21:41:37 dbdump.go:198: [DEBUG] DiscoverDatabases: nextcloud-db → stack=nextcloud, dbUser=root, dbName=nextcloud +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container nextcloud-redis (image=redis:7-alpine, not a database) +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost (image=docmost/docmost:0.96.0, not a database) +2026/09/23 21:41:37 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: docmost-postgres (id=c981df80d790) +2026/09/23 21:41:37 dbdump.go:198: [DEBUG] DiscoverDatabases: docmost-postgres → stack=docmost, dbUser=docmost, dbName=docmost +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost-redis (image=redis:7-alpine, not a database) +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, not a database) +2026/09/23 21:41:37 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: adventurelog-postgres (id=5abd52db34b8) +2026/09/23 21:41:37 dbdump.go:198: [DEBUG] DiscoverDatabases: adventurelog-postgres → stack=adventurelog, dbUser=adventurelog, dbName=adventurelog +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, not a database) +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.267.0, not a database) +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/23 21:41:37 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/23 21:41:37 dbdump.go:205: [DEBUG] DiscoverDatabases: found 5 database(s), skipped 17 non-DB container(s) +2026/09/23 21:41:37 dbdump.go:208: [INFO] [backup] Discovered 5 databases +2026/09/23 21:41:37 dbdump.go:266: [DEBUG] DumpOne: starting dump for container=romm-db, stack=romm, dbType=mariadb, dest=/mnt/felhom-drives/scratch_hdd/userdata/romm/backups/primary/romm/db-dumps/pre-restore-20260923T214137Z-romm-mariadb.sql +2026/09/23 21:41:37 dbdump.go:321: [DEBUG] DumpOne: mariadb-dump command: docker exec 581a28cd0e38 mariadb-dump -u root -p*** --single-transaction --routines --triggers romm +2026/09/23 21:41:38 dbdump.go:405: [DEBUG] DumpOne: completed romm-db → pre-restore-20260923T214137Z-romm-mariadb.sql (size=110.2 KB, valid=true, tables=36, duration=306ms) +2026/09/23 21:41:38 dbdump.go:410: [INFO] [backup] DB dump: romm-db → pre-restore-20260923T214137Z-romm-mariadb.sql (110.2 KB, 306ms, 36 tables) +2026/09/23 21:41:38 offbox_reconstitute.go:255: [INFO] [offbox] romm: pre-restore safety dump written → pre-restore-20260923T214137Z-romm-mariadb.sql (110.2 KB) +2026/09/23 21:41:38 update_guard.go:478: [INFO] [backup] update safety dump for romm: 1 file(s) [/mnt/felhom-drives/scratch_hdd/userdata/romm/backups/primary/romm/db-dumps/pre-restore-20260923T214137Z-romm-mariadb.sql] +2026/09/23 21:41:38 update.go:742: [INFO] [stacks] update romm: safety dump done (1 file(s)) [/mnt/felhom-drives/scratch_hdd/userdata/romm/backups/primary/romm/db-dumps/pre-restore-20260923T214137Z-romm-mariadb.sql] +2026/09/23 21:41:39 info.go:13: [DEBUG] [system] GetDiskUsage: path="/" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:41:39 undo.go:252: [INFO] [stacks] update romm: the undo copy will hold 3 named volume(s), 183.7 MiB +2026/09/23 21:41:39 update.go:1192: [INFO] [stacks] update romm: phase pinning +2026/09/23 21:41:39 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/romm — 6 env vars, 0 encrypted, 3 sensitive fields +2026/09/23 21:41:39 [INFO] [stacks] SaveAppConfig: saved config for romm +2026/09/23 21:41:39 pin.go:365: [INFO] [stacks] update romm: pin advanced to the catalog's current definition (romm=rommapp/romm:5.3.1, romm-db=mariadb:11.8, romm-redis=redis:7-alpine) +2026/09/23 21:41:39 update.go:1192: [INFO] [stacks] update romm: phase pulling +2026/09/23 21:41:39 manager.go:1419: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, ROMM_AUTH_SECRET_KEY, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (13 app + 0 system) +2026/09/23 21:41:39 manager.go:1428: [DEBUG] Running: docker compose pull (in /opt/docker/stacks/romm) +2026/09/23 21:41:40 manager.go:1448: [DEBUG] Command completed: docker compose pull (took 1.2s) +2026/09/23 21:41:40 update.go:1192: [INFO] [stacks] update romm: phase copying +2026/09/23 21:41:40 manager.go:1419: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, ROMM_AUTH_SECRET_KEY, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (13 app + 0 system) +2026/09/23 21:41:40 manager.go:1428: [DEBUG] Running: docker compose stop (in /opt/docker/stacks/romm) +2026/09/23 21:41:41 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:41:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:41:43 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 7s ago, effective interval 10s, healthy=false +2026/09/23 21:41:43 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (9 skipped not due, 0 skipped no container) +2026/09/23 21:41:46 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:46 manager.go:816: [DEBUG] [stacks] restart-policy of down member "romm" = "unless-stopped" +2026/09/23 21:41:46 manager.go:1448: [DEBUG] Command completed: docker compose stop (took 5.4s) +2026/09/23 21:41:46 update.go:1192: [INFO] [stacks] update romm: phase copying +2026/09/23 21:41:46 undo.go:279: [INFO] [stacks] update romm: copied romm_romm_config → romm_romm_config.pre-update-20260923T214146Z in 446ms +2026/09/23 21:41:46 update.go:1192: [INFO] [stacks] update romm: phase copying +2026/09/23 21:41:47 undo.go:279: [INFO] [stacks] update romm: copied romm_romm_db_data → romm_romm_db_data.pre-update-20260923T214146Z in 990ms +2026/09/23 21:41:47 update.go:1192: [INFO] [stacks] update romm: phase copying +2026/09/23 21:41:48 undo.go:279: [INFO] [stacks] update romm: copied romm_romm_redis_data → romm_romm_redis_data.pre-update-20260923T214146Z in 470ms +2026/09/23 21:41:48 update.go:1192: [INFO] [stacks] update romm: phase starting +2026/09/23 21:41:48 manager.go:387: [INFO] [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping ensure (held by drive gate) +2026/09/23 21:41:48 manager.go:1419: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, ROMM_AUTH_SECRET_KEY, SUBDOMAIN, USERDATA_PATH, IMPORT_PATH] (13 app + 0 system) +2026/09/23 21:41:48 manager.go:1428: [DEBUG] Running: docker compose up -d --remove-orphans (in /opt/docker/stacks/romm) +2026/09/23 21:41:51 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:41:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:41:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:41:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:41:53 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:41:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:41:53 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:41:53 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.3:8999: connect: connection refused +2026/09/23 21:41:53 healthprobe.go:159: [WARN] Health probes: 0 ok, 1 unhealthy (of 1 probed) +2026/09/23 21:41:56 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:01 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:42:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:42:03 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false +2026/09/23 21:42:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:03 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (9 skipped not due, 0 skipped no container) +2026/09/23 21:42:04 manager.go:1448: [DEBUG] Command completed: docker compose up -d --remove-orphans (took 16.2s) +2026/09/23 21:42:04 update.go:1192: [INFO] [stacks] update romm: phase verifying +2026/09/23 21:42:04 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:06 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:09 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:11 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:42:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:42:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:12 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 4 targets (4 skipped not due, 0 skipped no container) +2026/09/23 21:42:12 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.3:8999: connect: connection refused +2026/09/23 21:42:12 healthprobe.go:181: [DEBUG] Health probe navidrome: API GET :4533/ping → 200 (1ms) +2026/09/23 21:42:12 healthprobe.go:181: [DEBUG] Health probe vikunja: API GET :3456/api/v1/info → 200 (2ms) +2026/09/23 21:42:12 healthprobe.go:181: [DEBUG] Health probe privatebin: HTTP GET :8080/ → 200 (4ms) +2026/09/23 21:42:13 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:13 healthprobe.go:159: [WARN] Health probes: 3 ok, 1 unhealthy (of 4 probed) +2026/09/23 21:42:14 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:16 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:19 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:21 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:42:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈5937MB avail≈19961MB +2026/09/23 21:42:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23672816 → total=25898MB avail=19961MB used=5937MB (22.9%) +2026/09/23 21:42:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.2GB avail=44.6GB (29.6%) +2026/09/23 21:42:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:42:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.86 1.24 1.16 5/1754 7639" → 1m=1.86 5m=1.24 15m=1.16 +2026/09/23 21:42:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:42:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 53.5°C (hwmon2) +2026/09/23 21:42:22 info.go:13: [DEBUG] [system] GetInfo done in 67ms — mem=5937MB/25898MB (22.9%), rootDisk=20.2GB/68.4GB (29.6%), load=1.86/1.24/1.16, temp=53.5°C (hwmon2), cpu=19.6% +2026/09/23 21:42:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:42:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:42:22 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/23 21:42:22 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/23 21:42:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:42:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:42:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:42:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:42:23 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:42:23 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:42:23 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:42:23 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:23 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 71ms) +2026/09/23 21:42:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:23 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/23 21:42:23 healthprobe.go:190: [WARN] Health probe nextcloud: API GET :8999/status.php → Get "http://nextcloud:8999/status.php": dial tcp 172.18.0.3:8999: connect: connection refused +2026/09/23 21:42:23 healthprobe.go:181: [DEBUG] Health probe papra: HTTP GET :1221/ → 200 (4ms) +2026/09/23 21:42:23 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:23 healthprobe.go:159: [WARN] Health probes: 1 ok, 1 unhealthy (of 2 probed) +2026/09/23 21:42:24 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:26 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:29 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:31 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:42:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping nextcloud — last check 10s ago, effective interval 10s, healthy=false +2026/09/23 21:42:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:42:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:42:33 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:34 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:36 manager.go:734: [INFO] [stacks] Status refresh: 22 containers across 55 stacks +2026/09/23 21:42:36 update.go:904: [ERROR] [stacks] update nextcloud FAILED after the new version was started: not healthy: not healthy within 1m30s (last: state unhealthy) +2026/09/23 21:42:36 manager.go:1419: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, DB_PASSWORD, DOMAIN, HDD_PATH, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/23 21:42:36 manager.go:1428: [DEBUG] Running: docker compose logs --no-color --tail 400 (in /opt/docker/stacks/nextcloud) +2026/09/23 21:42:36 manager.go:1448: [DEBUG] Command completed: docker compose logs --no-color --tail 400 (took 0.1s) +2026/09/23 21:42:36 update.go:896: [INFO] [stacks] update nextcloud: kept 21737 bytes of the app's own log at /opt/docker/stacks/nextcloud/hold-logs/20260923T214236Z/compose-logs.txt before stopping it (R-621) +2026/09/23 21:42:36 update.go:1192: [INFO] [stacks] update nextcloud: phase undoing +2026/09/23 21:42:36 undo.go:357: [WARN] [stacks] update nextcloud: UNDO — putting back the previous version and its 3 volume copy(ies) (reason: not healthy: not healthy within 1m30s (last: state unhealthy)) +2026/09/23 21:42:36 manager.go:1419: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, HDD_PATH, MYSQL_ROOT_PASSWORD, NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_ADMIN_USER, SUBDOMAIN, DB_PASSWORD, DOMAIN, USERDATA_PATH, IMPORT_PATH] (14 app + 0 system) +2026/09/23 21:42:36 manager.go:1428: [DEBUG] Running: docker compose down (in /opt/docker/stacks/nextcloud) +2026/09/23 21:42:38 manager.go:1448: [DEBUG] Command completed: docker compose down (took 2.2s) +2026/09/23 21:42:39 manager.go:1467: [ERROR] [stacks] execCommand failed: exit status 1 +2026/09/23 21:42:39 undo.go:363: [ERROR] [stacks] update nextcloud: the undo copy nextcloud_nextcloud_db_data.pre-update-20260923T214051Z has no finished-marker — it is cut off or missing; NOTHING is put back +2026/09/23 21:42:39 update.go:916: [ERROR] [stacks] update nextcloud: the UNDO failed too (untouched) — HOLDING the app; the undo copies are kept: [{nextcloud_nextcloud_db_data nextcloud_nextcloud_db_data.pre-update-20260923T214051Z} {nextcloud_nextcloud_html nextcloud_nextcloud_html.pre-update-20260923T214051Z} {nextcloud_nextcloud_redis_data nextcloud_nextcloud_redis_data.pre-update-20260923T214051Z}] +2026/09/23 21:42:39 settings.go:1773: [WARN] [settings] restore hold SET for nextcloud — the app stays stopped until it is cleared +2026/09/23 21:42:39 settings.go:815: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (12113 bytes) +2026/09/23 21:42:39 settings.go:818: [INFO] [settings] Settings saved +2026/09/23 21:42:39 update_guard.go:560: [WARN] [backup] nextcloud is HELD STOPPED after a failed update (restore point: tier 1 "saját meghajtó", 2026-09-23T20:55:33Z; holds: "csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem"; undo: "untouched") +2026/09/23 21:42:39 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:42:39 undo.go:475: [INFO] [stacks] update nextcloud: event app_update_held (hold recorded: true) +2026/09/23 21:42:39 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_update_held (severity error) — further app_update_held events are logged at DEBUG only +2026/09/23 21:42:39 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:42:39 healthprobe.go:181: [DEBUG] Health probe romm: HTTP GET :8080/ → 200 (1ms) +2026/09/23 21:42:39 update.go:863: [INFO] [stacks] update romm: healthy after 35s (the app's health check passed) +2026/09/23 21:42:39 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/romm — 6 env vars, 0 encrypted, 3 sensitive fields +2026/09/23 21:42:39 [INFO] [stacks] SaveAppConfig: saved config for romm +2026/09/23 21:42:39 installed.go:409: [INFO] [stacks] installed-images romm: recorded 3 service(s) (romm=rommapp/romm:5.3.1 (sha256:0d66b4ea152a…), romm-db=mariadb:11.8 (sha256:79d59758afc9…), romm-redis=redis:7-alpine (sha256:858f009f9709…)) +2026/09/23 21:42:39 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:42:39 update.go:871: [INFO] [stacks] update romm: DONE in 1m2s +2026/09/23 21:42:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:42:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:42:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 2 targets (6 skipped not due, 0 skipped no container) +2026/09/23 21:42:42 healthprobe.go:181: [DEBUG] Health probe docmost: HTTP GET :3000/ → 200 (2ms) +2026/09/23 21:42:42 healthprobe.go:181: [DEBUG] Health probe adventurelog: API GET :8000/api/ → 200 (4ms) +2026/09/23 21:42:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:42:43 healthprobe.go:161: [INFO] Health probes: 2 ok (of 2 probed) +2026/09/23 21:42:50 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.9:46156 +2026/09/23 21:42:50 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.9:46156 +2026/09/23 21:42:50 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.9:46156 +2026/09/23 21:42:50 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/nextcloud from 172.18.0.9:46156 +2026/09/23 21:42:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:42:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:42:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:42:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:42:52 notifier.go:1259: [DEBUG] notifier disabled: dropped event app_start_failed (severity warning) +2026/09/23 21:42:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:42:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 13s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:42:53 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:43:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:43:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:43:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:43:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 23s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:03 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:43:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:43:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 33s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:12 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/23 21:43:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:43:12 healthprobe.go:181: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (4ms) +2026/09/23 21:43:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:43:13 healthprobe.go:161: [INFO] Health probes: 1 ok (of 1 probed) +2026/09/23 21:43:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:43:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6232MB avail≈19666MB +2026/09/23 21:43:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23323488 → total=25898MB avail=19666MB used=6232MB (24.1%) +2026/09/23 21:43:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:43:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:43:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.00 1.44 1.23 3/1726 8007" → 1m=2.00 5m=1.44 15m=1.23 +2026/09/23 21:43:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:43:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 48.9°C (hwmon1) +2026/09/23 21:43:22 info.go:13: [DEBUG] [system] GetInfo done in 59ms — mem=6232MB/25898MB (24.1%), rootDisk=20.0GB/68.4GB (29.3%), load=2.00/1.44/1.23, temp=48.9°C (hwmon1), cpu=5.7% +2026/09/23 21:43:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:43:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:43:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:43:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:43:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:43:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 43s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:23 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:43:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:43:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:43:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 53s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:43:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:43:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:43:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m3s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:43:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:43:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:43:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:43:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:43:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:43:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:43:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:43:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m13s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:43:53 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:44:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:44:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m23s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:02 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:44:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:44:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:44:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:44:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:44:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:44:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m33s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:13 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:44:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:44:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6214MB avail≈19684MB +2026/09/23 21:44:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23344400 → total=25898MB avail=19684MB used=6214MB (24.0%) +2026/09/23 21:44:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:44:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:44:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.04 1.27 1.19 2/1723 8146" → 1m=1.04 5m=1.27 15m=1.19 +2026/09/23 21:44:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:44:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 48.9°C (hwmon1) +2026/09/23 21:44:22 info.go:13: [DEBUG] [system] GetInfo done in 60ms — mem=6214MB/25898MB (24.0%), rootDisk=20.0GB/68.4GB (29.3%), load=1.04/1.27/1.19, temp=48.9°C (hwmon1), cpu=5.3% +2026/09/23 21:44:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:44:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:44:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:44:22 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/23 21:44:22 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/23 21:44:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:44:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:44:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m43s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:23 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:44:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:44:23 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:44:23 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:44:23 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:44:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:44:23 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 92ms) +2026/09/23 21:44:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:44:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:44:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m53s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:44:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:44:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:44:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:44:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:44:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m3s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:43 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:44:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:44:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:44:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:44:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:44:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:44:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m13s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:44:53 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:45:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:45:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m23s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:45:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:02 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:45:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:45:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:45:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:45:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:45:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m33s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:13 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:45:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:45:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6234MB avail≈19664MB +2026/09/23 21:45:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23320624 → total=25898MB avail=19664MB used=6234MB (24.1%) +2026/09/23 21:45:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:45:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:45:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.99 1.23 1.18 5/1723 8298" → 1m=0.99 5m=1.23 15m=1.18 +2026/09/23 21:45:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:45:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:45:22 info.go:13: [DEBUG] [system] GetInfo done in 61ms — mem=6234MB/25898MB (24.1%), rootDisk=20.0GB/68.4GB (29.3%), load=0.99/1.23/1.18, temp=49.9°C (hwmon1), cpu=5.8% +2026/09/23 21:45:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:45:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:45:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:45:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:45:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:45:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m43s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:23 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:45:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:45:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:45:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:45:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m53s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:33 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:45:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:45:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:45:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m3s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:45:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:45:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:45:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:45:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:45:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:45:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:45:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m13s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:45:53 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:46:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:46:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:46:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m23s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:02 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:46:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:46:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:46:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:46:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m33s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:12 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:46:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:46:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:46:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6221MB avail≈19677MB +2026/09/23 21:46:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23335044 → total=25898MB avail=19677MB used=6221MB (24.0%) +2026/09/23 21:46:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:46:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:46:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.75 1.12 1.14 2/1713 8437" → 1m=0.75 5m=1.12 15m=1.14 +2026/09/23 21:46:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:46:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:46:22 info.go:13: [DEBUG] [system] GetInfo done in 64ms — mem=6221MB/25898MB (24.0%), rootDisk=20.0GB/68.4GB (29.3%), load=0.75/1.12/1.14, temp=49.9°C (hwmon1), cpu=4.6% +2026/09/23 21:46:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:46:22 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/23 21:46:22 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/23 21:46:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:46:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:46:22 scheduler.go:346: [INFO] [scheduler] Running job: system-health +2026/09/23 21:46:22 scheduler.go:67: [DEBUG] [scheduler] job system-health: execution starting +2026/09/23 21:46:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:46:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:46:22 scheduler.go:346: [INFO] [scheduler] Running job: offsite-credential-retry +2026/09/23 21:46:22 scheduler.go:67: [DEBUG] [scheduler] job offsite-credential-retry: execution starting +2026/09/23 21:46:22 scheduler.go:363: [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:46:22 scheduler.go:346: [INFO] [scheduler] Running job: backup-cache +2026/09/23 21:46:22 scheduler.go:67: [DEBUG] [scheduler] job backup-cache: execution starting +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:46:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:46:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:46:23 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:46:23 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:46:23 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6240MB avail≈19658MB +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23314056 → total=25898MB avail=19658MB used=6240MB (24.1%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.69 1.10 1.14 4/1734 8478" → 1m=0.69 5m=1.10 15m=1.14 +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetInfo done in 68ms — mem=6240MB/25898MB (24.1%), rootDisk=20.0GB/68.4GB (29.3%), load=0.69/1.10/1.14, temp=49.9°C (hwmon1), cpu=4.6% +2026/09/23 21:46:23 healthcheck.go:86: [DEBUG] [monitor] Raw values: disk=29.3%, hdd=11.0% (configured=true), mem=24.1% (6240MB/25898MB), cpu=4.6%, temp=49.9°C (hwmon1) +2026/09/23 21:46:23 healthcheck.go:118: [DEBUG] [monitor] SSD disk: OK (29%) +2026/09/23 21:46:23 healthcheck.go:151: [DEBUG] [monitor] Memory: OK (24%) +2026/09/23 21:46:23 healthcheck.go:169: [DEBUG] [monitor] CPU: OK (5%) +2026/09/23 21:46:23 healthcheck.go:187: [DEBUG] [monitor] Temperature: OK (50°C) +2026/09/23 21:46:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:46:23 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 73ms) +2026/09/23 21:46:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m43s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:23 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:46:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:46:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/nextcloud → [nextcloud] +2026/09/23 21:46:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx → [paperless-ngx] +2026/09/23 21:46:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/romm → [romm] +2026/09/23 21:46:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [actualbudget, adventurelog, docmost, gokapi, papra, privatebin, vikunja] +2026/09/23 21:46:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/navidrome → [navidrome] +2026/09/23 21:46:23 settings.go:815: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (12335 bytes) +2026/09/23 21:46:23 settings.go:818: [INFO] [settings] Settings saved +2026/09/23 21:46:23 backup.go:1117: [INFO] [backup] Found 10 DB dump files across drives +2026/09/23 21:46:23 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/23 21:46:23 healthcheck.go:201: [DEBUG] [monitor] Docker daemon: OK +2026/09/23 21:46:23 healthcheck.go:209: [DEBUG] [monitor] Checking 1 protected containers: [traefik] +2026/09/23 21:46:23 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 16089e1abac5 romm-db romm mariadb:11.8 +d6ac44f34971 vikunja vikunja vikunja/vikunja:2.5.0 +8a54f14e8bb0 navidrome navidrome deluan/navidrome:0.64.0 +9c3fff6d8323 romm romm rommapp/romm:5.3.1 +f428c67086c0 romm-redis romm redis:7-alpine +f12b35ea40d5 privatebin privatebin privatebin/pdo:2.0.6 +20daa65aee34 papra papra ghcr.io/papra-hq/papra:26.6.2-rootless +c916eed5d29a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +c509e115095a paperless-postgres paperless-ngx postgres:1... +2026/09/23 21:46:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=16089e1abac5) +2026/09/23 21:46:23 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.5.0, not a database) +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.0, not a database) +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container papra (image=ghcr.io/papra-hq/papra:26.6.2-rootless, not a database) +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/23 21:46:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=c509e115095a) +2026/09/23 21:46:23 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/23 21:46:23 healthcheck.go:219: [DEBUG] [monitor] All protected containers running +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:46:23 healthcheck.go:238: [INFO] [monitor] Health check: status=warn +2026/09/23 21:46:23 healthcheck.go:242: [DEBUG] [monitor] Final status: warn (issues=0, warnings=4, info=5) +2026/09/23 21:46:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:46:23 scheduler.go:363: [INFO] [scheduler] Job system-health completed (took 160ms) +2026/09/23 21:46:23 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost (image=docmost/docmost:0.96.0, not a database) +2026/09/23 21:46:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: docmost-postgres (id=c981df80d790) +2026/09/23 21:46:23 dbdump.go:198: [DEBUG] DiscoverDatabases: docmost-postgres → stack=docmost, dbUser=docmost, dbName=docmost +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost-redis (image=redis:7-alpine, not a database) +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, not a database) +2026/09/23 21:46:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: adventurelog-postgres (id=5abd52db34b8) +2026/09/23 21:46:23 infra.go:67: [INFO] [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node) +2026/09/23 21:46:23 dbdump.go:198: [DEBUG] DiscoverDatabases: adventurelog-postgres → stack=adventurelog, dbUser=adventurelog, dbName=adventurelog +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, not a database) +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.267.0, not a database) +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/23 21:46:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/23 21:46:23 dbdump.go:205: [DEBUG] DiscoverDatabases: found 4 database(s), skipped 15 non-DB container(s) +2026/09/23 21:46:23 dbdump.go:208: [INFO] [backup] Discovered 4 databases +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:46:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:46:23 [INFO] [backup] Discovered app data: 11 apps +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:46:23 recovery_unit.go:224: [INFO] [backup] Recovery unit captured for romm → /mnt/felhom-drives/scratch_hdd/userdata/romm/backups/primary/romm (images=3, secrets-referenced=3, data_keys=0, portable-carried=3/3, withheld=0) +2026/09/23 21:46:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:46:23 backup.go:1187: [INFO] [backup] Backup status cache refreshed +2026/09/23 21:46:23 scheduler.go:363: [INFO] [scheduler] Job backup-cache completed (took 258ms) +2026/09/23 21:46:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:46:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:46:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:46:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m53s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:33 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:46:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:46:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m3s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:46:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:46:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:46:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:46:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:46:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:46:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:46:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m13s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:46:52 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:46:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:47:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:47:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:47:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:47:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m23s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:03 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:47:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:47:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:47:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m33s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:12 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 3 targets (5 skipped not due, 0 skipped no container) +2026/09/23 21:47:12 healthprobe.go:181: [DEBUG] Health probe navidrome: API GET :4533/ping → 200 (3ms) +2026/09/23 21:47:12 healthprobe.go:181: [DEBUG] Health probe vikunja: API GET :3456/api/v1/info → 200 (3ms) +2026/09/23 21:47:12 healthprobe.go:181: [DEBUG] Health probe privatebin: HTTP GET :8080/ → 200 (5ms) +2026/09/23 21:47:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:47:13 healthprobe.go:161: [INFO] Health probes: 3 ok (of 3 probed) +2026/09/23 21:47:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:47:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6217MB avail≈19681MB +2026/09/23 21:47:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23340796 → total=25898MB avail=19681MB used=6217MB (24.0%) +2026/09/23 21:47:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:47:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:47:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.47 0.98 1.09 5/1730 8733" → 1m=0.47 5m=0.98 15m=1.09 +2026/09/23 21:47:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:47:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:47:22 info.go:13: [DEBUG] [system] GetInfo done in 70ms — mem=6217MB/25898MB (24.0%), rootDisk=20.0GB/68.4GB (29.3%), load=0.47/0.98/1.09, temp=49.9°C (hwmon1), cpu=4.4% +2026/09/23 21:47:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:47:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:47:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m43s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:22 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:47:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:47:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:47:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:47:23 main.go:1934: [INFO] [deadapp] check alive: 80 scans since boot, 11 deployed app(s) evaluated, 2 currently down +2026/09/23 21:47:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:47:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:47:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m53s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/23 21:47:32 healthprobe.go:181: [DEBUG] Health probe papra: HTTP GET :1221/ → 200 (4ms) +2026/09/23 21:47:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:47:33 healthprobe.go:161: [INFO] Health probes: 1 ok (of 1 probed) +2026/09/23 21:47:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:47:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:47:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 3 targets (5 skipped not due, 0 skipped no container) +2026/09/23 21:47:42 healthprobe.go:181: [DEBUG] Health probe romm: HTTP GET :8080/ → 200 (1ms) +2026/09/23 21:47:42 healthprobe.go:181: [DEBUG] Health probe docmost: HTTP GET :3000/ → 200 (2ms) +2026/09/23 21:47:42 healthprobe.go:181: [DEBUG] Health probe adventurelog: API GET :8000/api/ → 200 (5ms) +2026/09/23 21:47:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:47:43 healthprobe.go:161: [INFO] Health probes: 3 ok (of 3 probed) +2026/09/23 21:47:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:47:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:47:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:47:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:47:52 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:47:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:47:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:48:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:48:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:48:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:48:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:03 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:48:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:48:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:48:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:48:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:13 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/23 21:48:13 healthprobe.go:181: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (3ms) +2026/09/23 21:48:13 healthprobe.go:161: [INFO] Health probes: 1 ok (of 1 probed) +2026/09/23 21:48:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:48:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6226MB avail≈19672MB +2026/09/23 21:48:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23330084 → total=25898MB avail=19672MB used=6226MB (24.0%) +2026/09/23 21:48:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:48:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:48:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.44 0.89 1.05 3/1714 8869" → 1m=0.44 5m=0.89 15m=1.05 +2026/09/23 21:48:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:48:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:48:22 info.go:13: [DEBUG] [system] GetInfo done in 60ms — mem=6226MB/25898MB (24.0%), rootDisk=20.0GB/68.4GB (29.3%), load=0.44/0.89/1.05, temp=49.9°C (hwmon1), cpu=4.2% +2026/09/23 21:48:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:48:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:48:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:48:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:22 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:48:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:48:22 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/23 21:48:22 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/23 21:48:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:48:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:48:23 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:48:23 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:48:23 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:48:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:48:23 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 93ms) +2026/09/23 21:48:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:48:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:48:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:48:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:33 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:48:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:48:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:48:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:48:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:48:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:48:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:48:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:48:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:48:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:48:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:48:53 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:49:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:49:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:02 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:49:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:49:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:49:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:49:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:49:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:12 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:49:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:49:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:49:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6234MB avail≈19664MB +2026/09/23 21:49:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23321568 → total=25898MB avail=19664MB used=6234MB (24.1%) +2026/09/23 21:49:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:49:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:49:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.77 0.88 1.04 3/1712 9016" → 1m=0.77 5m=0.88 15m=1.04 +2026/09/23 21:49:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:49:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:49:22 info.go:13: [DEBUG] [system] GetInfo done in 58ms — mem=6234MB/25898MB (24.1%), rootDisk=20.0GB/68.4GB (29.3%), load=0.77/0.88/1.04, temp=49.9°C (hwmon1), cpu=4.3% +2026/09/23 21:49:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:49:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:49:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:49:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:49:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:22 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:49:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:49:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:49:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:49:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:49:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:49:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:49:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:49:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:49:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:49:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:49:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:49:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:49:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:49:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:49:52 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:49:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:50:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:50:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:50:02 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:50:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:03 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:50:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:50:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:50:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:50:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:13 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:50:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:50:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6224MB avail≈19674MB +2026/09/23 21:50:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23332072 → total=25898MB avail=19674MB used=6224MB (24.0%) +2026/09/23 21:50:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:50:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:50:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.66 0.84 1.01 4/1711 9162" → 1m=0.66 5m=0.84 15m=1.01 +2026/09/23 21:50:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:50:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 48.9°C (hwmon1) +2026/09/23 21:50:22 info.go:13: [DEBUG] [system] GetInfo done in 58ms — mem=6224MB/25898MB (24.0%), rootDisk=20.0GB/68.4GB (29.3%), load=0.66/0.84/1.01, temp=48.9°C (hwmon1), cpu=4.5% +2026/09/23 21:50:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:50:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:50:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:50:22 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/23 21:50:22 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/23 21:50:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:22 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:50:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:50:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:50:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:50:23 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:50:23 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:50:23 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:50:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:50:23 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 67ms) +2026/09/23 21:50:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:50:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:50:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:50:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:50:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:33 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:50:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:50:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:50:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:50:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:50:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:50:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:50:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:50:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:50:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:50:52 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:50:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:51:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:51:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:51:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:02 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:51:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:51:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:51:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:51:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:51:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:13 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:13 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:51:22 sync.go:208: [INFO] [sync] Starting catalog sync +2026/09/23 21:51:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:51:22 sync.go:296: [INFO] [sync] Pulling latest from https://gitea.dooplex.hu/admin/app-catalog-drill.git (branch: main) +2026/09/23 21:51:22 sync.go:298: [DEBUG] [sync] git fetch --depth 1 origin main in /opt/docker/felhom-controller/data/catalog-cache +2026/09/23 21:51:22 sync.go:592: [DEBUG] [sync] Running: git fetch --depth 1 origin main +2026/09/23 21:51:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6224MB avail≈19674MB +2026/09/23 21:51:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23331880 → total=25898MB avail=19674MB used=6224MB (24.0%) +2026/09/23 21:51:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:51:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:51:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.64 0.81 0.99 11/1714 9316" → 1m=0.64 5m=0.81 15m=0.99 +2026/09/23 21:51:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:51:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:51:22 info.go:13: [DEBUG] [system] GetInfo done in 73ms — mem=6224MB/25898MB (24.0%), rootDisk=20.0GB/68.4GB (29.3%), load=0.64/0.81/0.99, temp=49.9°C (hwmon1), cpu=5.0% +2026/09/23 21:51:22 sync.go:592: [DEBUG] [sync] Running: git reset --hard origin/main +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] actualbudget/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] actualbudget/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] audiobookshelf/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] audiobookshelf/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] bentopdf/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] bentopdf/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] bookstack/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] bookstack/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] calcom/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] calcom/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] calibre-web/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] calibre-web/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] claper/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] claper/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] code-server/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] code-server/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] crafty-controller/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] crafty-controller/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] docmost/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] docmost/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] emby/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] emby/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] ghost/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] ghost/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] gitea/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] gitea/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] glance/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] glance/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] gokapi/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] gokapi/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] grafana/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] grafana/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] gramps-web/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] gramps-web/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] home-assistant/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] home-assistant/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] homebox/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] homebox/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] homepage/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] homepage/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] immich/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] immich/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] jellyfin/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] jellyfin/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] kimai/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] kimai/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] komga/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] komga/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] mealie/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] mealie/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] n8n/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] n8n/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] navidrome/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] navidrome/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] nextcloud/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] nextcloud/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] onlyoffice/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] onlyoffice/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] opengist/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] opengist/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] outline/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] outline/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] paperless-ngx/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] paperless-ngx/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] papra/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] papra/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] plant-it/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] plant-it/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] privatebin/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] privatebin/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] radarr/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] radarr/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] rallly/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] rallly/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] recipe-importer/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] recipe-importer/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] romm/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] romm/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] seerr/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] seerr/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] sonarr/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] sonarr/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] sparkyfitness/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] sparkyfitness/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] tandoor/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] tandoor/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] termix/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] termix/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] uptime-kuma/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] uptime-kuma/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] vaultwarden/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] vaultwarden/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] vikunja/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] vikunja/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] wanderer/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] wanderer/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] wger/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:416: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/09/23 21:51:22 sync.go:263: [INFO] [sync] Catalog sync complete +2026/09/23 21:51:22 sync.go:135: [INFO] [sync] Periodic sync: Sablonok naprakészek — nincs változás +2026/09/23 21:51:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:51:22 scheduler.go:346: [INFO] [scheduler] Running job: system-health +2026/09/23 21:51:22 scheduler.go:67: [DEBUG] [scheduler] job system-health: execution starting +2026/09/23 21:51:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:51:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:51:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:51:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:51:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:22 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:51:22 scheduler.go:346: [INFO] [scheduler] Running job: offsite-credential-retry +2026/09/23 21:51:22 scheduler.go:67: [DEBUG] [scheduler] job offsite-credential-retry: execution starting +2026/09/23 21:51:22 scheduler.go:363: [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/09/23 21:51:22 scheduler.go:346: [INFO] [scheduler] Running job: backup-cache +2026/09/23 21:51:22 scheduler.go:67: [DEBUG] [scheduler] job backup-cache: execution starting +2026/09/23 21:51:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:51:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/navidrome → [navidrome] +2026/09/23 21:51:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/nextcloud → [nextcloud] +2026/09/23 21:51:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx → [paperless-ngx] +2026/09/23 21:51:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/romm → [romm] +2026/09/23 21:51:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [actualbudget, adventurelog, docmost, gokapi, papra, privatebin, vikunja] +2026/09/23 21:51:23 backup.go:1117: [INFO] [backup] Found 10 DB dump files across drives +2026/09/23 21:51:23 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6241MB avail≈19657MB +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23311640 → total=25898MB avail=19657MB used=6241MB (24.1%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.67 0.81 0.99 6/1733 9384" → 1m=0.67 5m=0.81 15m=0.99 +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetInfo done in 69ms — mem=6241MB/25898MB (24.1%), rootDisk=20.0GB/68.4GB (29.3%), load=0.67/0.81/0.99, temp=49.9°C (hwmon1), cpu=5.0% +2026/09/23 21:51:23 healthcheck.go:86: [DEBUG] [monitor] Raw values: disk=29.3%, hdd=11.0% (configured=true), mem=24.1% (6241MB/25898MB), cpu=5.0%, temp=49.9°C (hwmon1) +2026/09/23 21:51:23 healthcheck.go:118: [DEBUG] [monitor] SSD disk: OK (29%) +2026/09/23 21:51:23 healthcheck.go:151: [DEBUG] [monitor] Memory: OK (24%) +2026/09/23 21:51:23 healthcheck.go:169: [DEBUG] [monitor] CPU: OK (5%) +2026/09/23 21:51:23 healthcheck.go:187: [DEBUG] [monitor] Temperature: OK (50°C) +2026/09/23 21:51:23 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 16089e1abac5 romm-db romm mariadb:11.8 +d6ac44f34971 vikunja vikunja vikunja/vikunja:2.5.0 +8a54f14e8bb0 navidrome navidrome deluan/navidrome:0.64.0 +9c3fff6d8323 romm romm rommapp/romm:5.3.1 +f428c67086c0 romm-redis romm redis:7-alpine +f12b35ea40d5 privatebin privatebin privatebin/pdo:2.0.6 +20daa65aee34 papra papra ghcr.io/papra-hq/papra:26.6.2-rootless +c916eed5d29a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +c509e115095a paperless-postgres paperless-ngx postgres:1... +2026/09/23 21:51:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=16089e1abac5) +2026/09/23 21:51:23 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.5.0, not a database) +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.0, not a database) +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container papra (image=ghcr.io/papra-hq/papra:26.6.2-rootless, not a database) +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/23 21:51:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=c509e115095a) +2026/09/23 21:51:23 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/23 21:51:23 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost (image=docmost/docmost:0.96.0, not a database) +2026/09/23 21:51:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: docmost-postgres (id=c981df80d790) +2026/09/23 21:51:23 healthcheck.go:201: [DEBUG] [monitor] Docker daemon: OK +2026/09/23 21:51:23 healthcheck.go:209: [DEBUG] [monitor] Checking 1 protected containers: [traefik] +2026/09/23 21:51:23 dbdump.go:198: [DEBUG] DiscoverDatabases: docmost-postgres → stack=docmost, dbUser=docmost, dbName=docmost +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost-redis (image=redis:7-alpine, not a database) +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, not a database) +2026/09/23 21:51:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: adventurelog-postgres (id=5abd52db34b8) +2026/09/23 21:51:23 healthcheck.go:219: [DEBUG] [monitor] All protected containers running +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:51:23 healthcheck.go:238: [INFO] [monitor] Health check: status=warn +2026/09/23 21:51:23 healthcheck.go:242: [DEBUG] [monitor] Final status: warn (issues=0, warnings=4, info=5) +2026/09/23 21:51:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:51:23 scheduler.go:363: [INFO] [scheduler] Job system-health completed (took 158ms) +2026/09/23 21:51:23 dbdump.go:198: [DEBUG] DiscoverDatabases: adventurelog-postgres → stack=adventurelog, dbUser=adventurelog, dbName=adventurelog +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, not a database) +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.267.0, not a database) +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/23 21:51:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/23 21:51:23 dbdump.go:205: [DEBUG] DiscoverDatabases: found 4 database(s), skipped 15 non-DB container(s) +2026/09/23 21:51:23 dbdump.go:208: [INFO] [backup] Discovered 4 databases +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:51:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:51:23 [INFO] [backup] Discovered app data: 11 apps +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:51:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:51:23 backup.go:1187: [INFO] [backup] Backup status cache refreshed +2026/09/23 21:51:23 scheduler.go:363: [INFO] [scheduler] Job backup-cache completed (took 211ms) +2026/09/23 21:51:23 infra.go:67: [INFO] [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node) +2026/09/23 21:51:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:51:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:51:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:51:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:33 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:33 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:51:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:51:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:51:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:51:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:43 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:51:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:51:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:51:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:51:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:51:52 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:51:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:51:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:52:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:52:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:02 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:52:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:52:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:52:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:52:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:12 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:52:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:52:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:52:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:52:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6220MB avail≈19678MB +2026/09/23 21:52:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23337180 → total=25898MB avail=19678MB used=6220MB (24.0%) +2026/09/23 21:52:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:52:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:52:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.06 0.93 1.02 6/1711 9603" → 1m=1.06 5m=0.93 15m=1.02 +2026/09/23 21:52:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:52:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:52:22 info.go:13: [DEBUG] [system] GetInfo done in 59ms — mem=6220MB/25898MB (24.0%), rootDisk=20.0GB/68.4GB (29.3%), load=1.06/0.93/1.02, temp=49.9°C (hwmon1), cpu=5.3% +2026/09/23 21:52:22 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/23 21:52:22 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/23 21:52:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:52:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:52:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:52:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:52:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:52:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:52:23 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:52:23 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:52:23 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:52:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:52:23 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 59ms) +2026/09/23 21:52:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:23 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 3 targets (5 skipped not due, 0 skipped no container) +2026/09/23 21:52:23 healthprobe.go:181: [DEBUG] Health probe navidrome: API GET :4533/ping → 200 (1ms) +2026/09/23 21:52:23 healthprobe.go:181: [DEBUG] Health probe vikunja: API GET :3456/api/v1/info → 200 (2ms) +2026/09/23 21:52:23 healthprobe.go:181: [DEBUG] Health probe privatebin: HTTP GET :8080/ → 200 (4ms) +2026/09/23 21:52:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:52:23 healthprobe.go:161: [INFO] Health probes: 3 ok (of 3 probed) +2026/09/23 21:52:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:52:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:52:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:52:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:52:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:52:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:52:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/23 21:52:42 healthprobe.go:181: [DEBUG] Health probe papra: HTTP GET :1221/ → 200 (4ms) +2026/09/23 21:52:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:52:43 healthprobe.go:161: [INFO] Health probes: 1 ok (of 1 probed) +2026/09/23 21:52:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:52:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:52:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:52:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:52:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:52:52 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 3 targets (5 skipped not due, 0 skipped no container) +2026/09/23 21:52:52 healthprobe.go:181: [DEBUG] Health probe romm: HTTP GET :8080/ → 200 (3ms) +2026/09/23 21:52:52 healthprobe.go:181: [DEBUG] Health probe docmost: HTTP GET :3000/ → 200 (4ms) +2026/09/23 21:52:52 healthprobe.go:181: [DEBUG] Health probe adventurelog: API GET :8000/api/ → 200 (5ms) +2026/09/23 21:52:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:52:53 healthprobe.go:161: [INFO] Health probes: 3 ok (of 3 probed) +2026/09/23 21:53:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:53:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:53:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:02 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:53:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:53:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:53:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 5m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:53:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:12 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:53:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:53:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:53:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6234MB avail≈19664MB +2026/09/23 21:53:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23320768 → total=25898MB avail=19664MB used=6234MB (24.1%) +2026/09/23 21:53:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:53:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:53:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.58 0.81 0.97 3/1708 9751" → 1m=0.58 5m=0.81 15m=0.97 +2026/09/23 21:53:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:53:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:53:22 info.go:13: [DEBUG] [system] GetInfo done in 69ms — mem=6234MB/25898MB (24.1%), rootDisk=20.0GB/68.4GB (29.3%), load=0.58/0.81/0.97, temp=49.9°C (hwmon1), cpu=7.3% +2026/09/23 21:53:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:53:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:22 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (7 skipped not due, 0 skipped no container) +2026/09/23 21:53:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:53:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:53:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:53:22 healthprobe.go:181: [DEBUG] Health probe paperless-ngx: HTTP GET :8000/ → 302 (4ms) +2026/09/23 21:53:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:53:23 healthprobe.go:161: [INFO] Health probes: 1 ok (of 1 probed) +2026/09/23 21:53:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:53:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:53:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:53:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:53:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:53:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:53:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:53:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:43 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:53:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:53:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:53:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:53:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:53:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:53:52 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:53:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:54:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:54:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:02 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:54:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:54:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:54:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:54:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:54:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:12 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:54:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:54:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:54:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6243MB avail≈19655MB +2026/09/23 21:54:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23310284 → total=25898MB avail=19655MB used=6243MB (24.1%) +2026/09/23 21:54:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:54:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:54:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.40 0.72 0.93 5/1726 9889" → 1m=0.40 5m=0.72 15m=0.93 +2026/09/23 21:54:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:54:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:54:22 info.go:13: [DEBUG] [system] GetInfo done in 64ms — mem=6243MB/25898MB (24.1%), rootDisk=20.0GB/68.4GB (29.3%), load=0.40/0.72/0.93, temp=49.9°C (hwmon1), cpu=7.8% +2026/09/23 21:54:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:54:22 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/23 21:54:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:22 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/23 21:54:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:22 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:54:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:54:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:54:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:54:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:54:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:54:23 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:54:23 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:54:23 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:54:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:54:23 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 60ms) +2026/09/23 21:54:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:54:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:54:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:54:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:54:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:54:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:54:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:54:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:54:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:54:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:54:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:54:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:54:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:54:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:54:52 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:54:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:55:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:55:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:02 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:02 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:55:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:55:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:55:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:55:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:12 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:55:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:55:13 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:55:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:55:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6226MB avail≈19672MB +2026/09/23 21:55:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23329840 → total=25898MB avail=19672MB used=6226MB (24.0%) +2026/09/23 21:55:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:55:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:55:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.73 0.77 0.93 4/1712 10044" → 1m=0.73 5m=0.77 15m=0.93 +2026/09/23 21:55:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:55:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 49.9°C (hwmon1) +2026/09/23 21:55:22 info.go:13: [DEBUG] [system] GetInfo done in 57ms — mem=6226MB/25898MB (24.0%), rootDisk=20.0GB/68.4GB (29.3%), load=0.73/0.77/0.93, temp=49.9°C (hwmon1), cpu=7.9% +2026/09/23 21:55:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:55:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:55:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:55:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:55:23 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:55:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:23 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:23 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:55:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:55:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:55:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:55:33 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:55:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:55:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:55:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:42 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:42 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:55:43 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:55:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:55:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:55:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:52 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:55:52 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) +2026/09/23 21:55:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:55:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:55:53 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:56:02 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:56:02 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:56:03 manager.go:734: [INFO] [stacks] Status refresh: 19 containers across 55 stacks +2026/09/23 21:56:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping papra — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:03 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:03 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (8 skipped not due, 0 skipped no container) diff --git a/documentation/audits/night-2026-09-23/chaos/round-11-recovery.json b/documentation/audits/night-2026-09-23/chaos/round-11-recovery.json new file mode 100644 index 00000000..28f839fc --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-11-recovery.json @@ -0,0 +1,42 @@ +{ + "snapshots": [ + { + "time": "2026-09-23T21:40:47Z", + "short_id": "helyi", + "tier": 1, + "drive_label": "Tárhely (nextcloud)" + } + ], + "restore": { + "ok": true, + "snapshot_id": "helyi", + "http": "HTTP/2 302", + "seconds": 6.1, + "state_after": "stopped", + "hold_after": "A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-23 23:42-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem." + }, + "readback_A": false, + "observables": { + "pinned_images": { + "nextcloud": "nextcloud:34.0.4-apache", + "nextcloud-db": "mariadb:12.3", + "nextcloud-redis": "redis:7-alpine" + }, + "installed_images": { + "nextcloud": "nextcloud:34.0.1-apache", + "nextcloud-db": "mariadb:12.3", + "nextcloud-redis": "redis:7-alpine" + }, + "catalog_images": { + "nextcloud": "nextcloud:34.0.1-apache", + "nextcloud-db": "mariadb:12.3", + "nextcloud-redis": "redis:7-alpine" + }, + "live_compose_image_lines": [ + "image: nextcloud:34.0.4-apache", + "image: mariadb:12.3", + "image: redis:7-alpine" + ], + "docker_inspect": [] + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-11-recovery.log b/documentation/audits/night-2026-09-23/chaos/round-11-recovery.log new file mode 100644 index 00000000..6e799abb --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-11-recovery.log @@ -0,0 +1,9 @@ +23:57:43 restore points offered: [{"time": "2026-09-23T21:40:47Z", "short_id": "helyi", "tier": 1, "drive_label": "Tárhely (nextcloud)"}] +23:57:43 [R] restoring nextcloud from snapshot 'helyi' (of 1 offered) +23:57:43 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash_error=Ez+a+ment%C3%A9s+nem+tartalmazza+az+alkalmaz%C3%A1s+f%C3%A1jljait%2C+ez%C3%A9rt+nem+%C3%A1ll%C3%ADtjuk+vissza+az+adatb%C3%A1zist+f%C3%B6l%C3%A9j%C3%BCk+%E2%80%94+a+f%C3%A1jlok+%C3%ADgy+a+hely%C3%BCk%C3%B6n+maradnak.+A+f%C3%A1jlok+a+t%C3%A1voli+m%C3%A1solatb%C3%B3l+%C3%A1ll%C3%ADthat%C3%B3k+vissza%3A+Biztons%C3%A1gi+ment%C3%A9s+%E2%86%92+Vissza%C3%A1ll%C3%ADt%C3%A1s%2C+%E2%80%9ETeljes+vissza%C3%A1ll%C3%ADt%C3%A1s+%28f%C3%A1jlok+%2B+adatb%C3%A1zis%29%E2%80%9D.'] +23:57:43 + 0.0s restore (False, None, None) +23:57:49 [R] after restore: state=stopped hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-23 23:42-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.' phase=failed +23:57:52 after restore: state=stopped hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-23 23:42-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.' phase=failed +00:07:55 app never answered on cloud/status.php (last rc=0 code=404) +00:07:55 nextcloud: the app never served /status.php +00:07:58 seed A after the household's restore: False; pinned {'nextcloud': 'nextcloud:34.0.4-apache', 'nextcloud-db': 'mariadb:12.3', 'nextcloud-redis': 'redis:7-alpine'} diff --git a/documentation/audits/night-2026-09-23/chaos/round-11-run.txt b/documentation/audits/night-2026-09-23/chaos/round-11-run.txt new file mode 100644 index 00000000..2a9f153b --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-11-run.txt @@ -0,0 +1,27 @@ +23:40:11 ==== ROUND 11: cutoff on nextcloud x two_updates at +51s (2026-09-23T21:40:10Z) +23:40:12 [5] drill commit 6087baf9cd32: romm mariadb:11.4 -> mariadb:11.8 (push rc=0) +23:40:17 [5] drill commit dfe3309664b2: nextcloud nextcloud:34.0.1-apache -> nextcloud:34.0.4-apache (push rc=0) +23:40:46 [sync] the badge needed 29.0s and 5 sync+rescan rounds to catch up to nextcloud:34.0.4-apache — R-607's window, measured +23:40:46 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +23:40:46 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +23:40:48 + 2.1s phase=pulling label=Új verzió letöltése… +23:40:50 + 3.7s phase=copying label=Az adatok másolása a frissítés előtt… +23:40:59 + 13.2s phase=starting label=Indítás az új verzióval… +23:41:06 + 19.5s phase=verifying label=Működés ellenőrzése… +23:41:11 >>> finished-marker removed from one undo copy: +copy: nextcloud_nextcloud_db_data.pre-update-20260923T214051Z +data + +23:41:37 >>> ACCIDENT two_updates at 23:41:37 +23:41:37 >>> a second Update on romm -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} at 23:41:37 +23:42:36 + 109.9s phase=undoing label=Visszaállítás az előző változatra… +23:42:39 + 112.6s phase=failed label=A frissítés nem sikerült +23:42:39 END phase=failed err='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-23 23:42-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.' hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-23 23:42-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.' +23:42:39 + 142.7s nextcloud: deployed=True state=stopped updating=False phase=failed hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új v' +23:42:39 + 142.7s romm: deployed=True state=running updating=True phase=verifying hold='None' +23:42:41 + 144.7s romm: deployed=True state=running updating=False phase=done hold='None' +23:42:50 romm: login as the seeded user http=200 ok=True +23:45:55 app never answered on cloud/ (last rc=0 code=404) +23:55:58 app never answered on cloud/status.php (last rc=0 code=404) +23:55:58 nextcloud: the app never served /status.php +23:56:05 ==== ROUND 11 END: steady in 142.7s; readback={'A': False}; events=0 diff --git a/documentation/audits/night-2026-09-23/chaos/round-11.json b/documentation/audits/night-2026-09-23/chaos/round-11.json new file mode 100644 index 00000000..6569242f --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-11.json @@ -0,0 +1,190 @@ +{ + "round": 11, + "action": "cutoff", + "accident": "a second Update on romm -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} at 23:41:37", + "delay_s": 51, + "app": "nextcloud", + "started": "2026-09-23T21:40:10Z", + "before": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "second_drill_commit": "6087baf9cd32", + "second_badge_catchup_s": 4.4, + "drill_commit": "dfe3309664b2", + "badge_catchup_s": 29.0, + "armed_at": "23:40:46", + "cutoff": "copy: nextcloud_nextcloud_db_data.pre-update-20260923T214051Z\ndata", + "second_app": "romm", + "phases": [ + [ + 0.0, + "safety-dump", + "Adatbázis pillanatkép…" + ], + [ + 2.1, + "pulling", + "Új verzió letöltése…" + ], + [ + 3.7, + "copying", + "Az adatok másolása a frissítés előtt…" + ], + [ + 13.2, + "starting", + "Indítás az új verzióval…" + ], + [ + 19.5, + "verifying", + "Működés ellenőrzése…" + ], + [ + 109.9, + "undoing", + "Visszaállítás az előző változatra…" + ], + [ + 112.6, + "failed", + "A frissítés nem sikerült" + ] + ], + "time_to_steady_s": 142.7, + "second_app_end": { + "state": "running", + "update_phase": "done", + "hold_reason": null + }, + "second_app_readback": { + "A": true + }, + "after": { + "badges": { + "hu": [ + { + "title": "A frissítés nem sikerült, és az automatikus visszaállítás sem.", + "text": "Megállítva — visszaállítás szükséges" + } + ], + "en": [ + { + "title": "The update did not succeed, and the automatic undo did not either.", + "text": "Stopped — restore needed" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": "A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-23 23:42-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem." + }, + "en": { + "undone_line": null, + "hold": "The update did not succeed, and the automatic undo did not either. The data is as the new version left it. The update of nextcloud at 2026-09-23 23:42 did not succeed, and the app did not start on the new version. The app stays stopped for safety, so that its data is not damaged. It can be restored on the Backups page from this backup: own drive, 2026-09-23 22:55 — this copy holds only the settings and the database, not the files." + } + }, + "state": { + "deployed": true, + "state": "stopped", + "update_phase": "failed", + "hold_reason": "A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-23 23:42-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.", + "update_error": "A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-23 23:42-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem." + } + }, + "observables": { + "pinned_images": { + "nextcloud": "nextcloud:34.0.4-apache", + "nextcloud-db": "mariadb:12.3", + "nextcloud-redis": "redis:7-alpine" + }, + "installed_images": { + "nextcloud": "nextcloud:34.0.1-apache", + "nextcloud-db": "mariadb:12.3", + "nextcloud-redis": "redis:7-alpine" + }, + "catalog_images": { + "nextcloud": "nextcloud:34.0.4-apache", + "nextcloud-db": "mariadb:12.3", + "nextcloud-redis": "redis:7-alpine" + }, + "live_compose_image_lines": [ + "image: nextcloud:34.0.4-apache", + "image: mariadb:12.3", + "image: redis:7-alpine" + ], + "docker_inspect": [] + }, + "readback": { + "A": false + }, + "events": [], + "box_log": [ + "2026-09-23T21:41:48Z INFO [stacks] userdata belt: drive /mnt/felhom-drives/scratch_hdd/userdata/romm not mounted — skipping ensure (held by drive gate)", + "2026-09-23T21:41:53Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.3:8999: connect: connection refused", + "2026-09-23T21:42:12Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.3:8999: connect: connection refused", + "2026-09-23T21:42:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:42:23Z WARN Health probe nextcloud: API GET :8999/status.php → Get \"http://nextcloud:8999/status.php\": dial tcp 172.18.0.3:8999: connect: connection refused", + "2026-09-23T21:42:36Z ERROR [stacks] update nextcloud FAILED after the new version was started: not healthy: not healthy within 1m30s (last: state unhealthy)", + "2026-09-23T21:42:36Z INFO [stacks] update nextcloud: kept 21737 bytes of the app's own log at /opt/docker/stacks/nextcloud/hold-logs/20260923T214236Z/compose-logs.txt before stopping it (R-621)", + "2026-09-23T21:42:36Z INFO [stacks] update nextcloud: phase undoing", + "2026-09-23T21:42:36Z WARN [stacks] update nextcloud: UNDO — putting back the previous version and its 3 volume copy(ies) (reason: not healthy: not healthy within 1m30s (last: state unhealthy))", + "2026-09-23T21:42:39Z ERROR [stacks] update nextcloud: the undo copy nextcloud_nextcloud_db_data.pre-update-20260923T214051Z has no finished-marker — it is cut off or missing; NOTHING is put back", + "2026-09-23T21:42:39Z ERROR [stacks] update nextcloud: the UNDO failed too (untouched) — HOLDING the app; the undo copies are kept: [{nextcloud_nextcloud_db_data nextcloud_nextcloud_db_data.pre-update-20260923T214051Z} {nextcloud_nextcloud_html nextcloud_nextcloud_html.pre-update-20260923T214051Z} {nextcloud_nextcloud_redis_da", + "2026-09-23T21:42:39Z WARN [settings] restore hold SET for nextcloud — the app stays stopped until it is cleared", + "2026-09-23T21:42:39Z WARN [backup] nextcloud is HELD STOPPED after a failed update (restore point: tier 1 \"saját meghajtó\", 2026-09-23T20:55:33Z; holds: \"csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem\"; undo: \"untouched\")", + "2026-09-23T21:42:39Z INFO [stacks] update nextcloud: event app_update_held (hold recorded: true)", + "2026-09-23T21:42:39Z WARN notifier disabled (no hub configured): DROPPED event app_update_held (severity error) — further app_update_held events are logged at DEBUG only", + "2026-09-23T21:44:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:46:22Z INFO [scheduler] Running job: backup-cache", + "2026-09-23T21:46:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:46:23Z INFO [backup] Found 10 DB dump files across drives", + "2026-09-23T21:46:23Z INFO [backup] Discovered 4 databases", + "2026-09-23T21:46:23Z INFO [backup] Recovery unit captured for romm → /mnt/felhom-drives/scratch_hdd/userdata/romm/backups/primary/romm (images=3, secrets-referenced=3, data_keys=0, portable-carried=3/3, withheld=0)", + "2026-09-23T21:46:23Z INFO [backup] Backup status cache refreshed", + "2026-09-23T21:46:23Z INFO [scheduler] Job backup-cache completed (took 258ms)", + "2026-09-23T21:47:23Z INFO [deadapp] check alive: 80 scans since boot, 11 deployed app(s) evaluated, 2 currently down", + "2026-09-23T21:48:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:50:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:51:22Z INFO [scheduler] Running job: backup-cache", + "2026-09-23T21:51:23Z INFO [backup] Found 10 DB dump files across drives", + "2026-09-23T21:51:23Z INFO [backup] Discovered 4 databases", + "2026-09-23T21:51:23Z INFO [backup] Backup status cache refreshed", + "2026-09-23T21:51:23Z INFO [scheduler] Job backup-cache completed (took 211ms)", + "2026-09-23T21:52:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:54:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)" + ], + "undo_copies_left": "3" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-11.log b/documentation/audits/night-2026-09-23/chaos/round-11.log new file mode 100644 index 00000000..979752f4 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-11.log @@ -0,0 +1,26 @@ +23:40:11 ==== ROUND 11: cutoff on nextcloud x two_updates at +51s (2026-09-23T21:40:10Z) +23:40:12 [5] drill commit 6087baf9cd32: romm mariadb:11.4 -> mariadb:11.8 (push rc=0) +23:40:17 [5] drill commit dfe3309664b2: nextcloud nextcloud:34.0.1-apache -> nextcloud:34.0.4-apache (push rc=0) +23:40:46 [sync] the badge needed 29.0s and 5 sync+rescan rounds to catch up to nextcloud:34.0.4-apache — R-607's window, measured +23:40:46 Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +23:40:46 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… +23:40:48 + 2.1s phase=pulling label=Új verzió letöltése… +23:40:50 + 3.7s phase=copying label=Az adatok másolása a frissítés előtt… +23:40:59 + 13.2s phase=starting label=Indítás az új verzióval… +23:41:06 + 19.5s phase=verifying label=Működés ellenőrzése… +23:41:11 >>> finished-marker removed from one undo copy: +copy: nextcloud_nextcloud_db_data.pre-update-20260923T214051Z +data + +23:41:37 >>> ACCIDENT two_updates at 23:41:37 +23:41:37 >>> a second Update on romm -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} at 23:41:37 +23:42:36 + 109.9s phase=undoing label=Visszaállítás az előző változatra… +23:42:39 + 112.6s phase=failed label=A frissítés nem sikerült +23:42:39 END phase=failed err='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-23 23:42-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.' hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új változat által hagyott állapotban vannak. A(z) nextcloud frissítése 2026-09-23 23:42-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem.' +23:42:39 + 142.7s nextcloud: deployed=True state=stopped updating=False phase=failed hold='A frissítés nem sikerült, és az automatikus visszaállítás sem. Az adatok az új v' +23:42:39 + 142.7s romm: deployed=True state=running updating=True phase=verifying hold='None' +23:42:41 + 144.7s romm: deployed=True state=running updating=False phase=done hold='None' +23:42:50 romm: login as the seeded user http=200 ok=True +23:45:55 app never answered on cloud/ (last rc=0 code=404) +23:55:58 app never answered on cloud/status.php (last rc=0 code=404) +23:55:58 nextcloud: the app never served /status.php diff --git a/documentation/audits/night-2026-09-23/chaos/round-12-controller.log b/documentation/audits/night-2026-09-23/chaos/round-12-controller.log new file mode 100644 index 00000000..eefb8dee --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-12-controller.log @@ -0,0 +1,349 @@ +2026/09/23 21:56:06 server.go:545: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.9:43960 +2026/09/23 21:56:06 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/papra from 172.18.0.9:43960 +2026/09/23 21:56:06 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/papra from 172.18.0.9:43960 +2026/09/23 21:56:06 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/papra from 172.18.0.9:43960 +2026/09/23 21:56:06 server.go:545: [DEBUG] [web] ServeHTTP: GET /apps/papra from 172.18.0.9:43960 +2026/09/23 21:56:06 router.go:591: [INFO] [api] stop requested for stack: papra +2026/09/23 21:56:06 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/papra — 3 env vars, 0 encrypted, 1 sensitive fields +2026/09/23 21:56:06 [INFO] [stacks] SaveAppConfig: saved config for papra +2026/09/23 21:56:06 desiredstate.go:81: [INFO] [stacks] desired state for papra recorded as "stopped" (was "running") +2026/09/23 21:56:06 manager.go:1248: [DEBUG] [stacks] StopStack papra: current state=running deployed=true containers=1 +2026/09/23 21:56:06 manager.go:1251: [INFO] [stacks] Stopping stack: papra +2026/09/23 21:56:06 manager.go:1419: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, AUTH_SECRET, DOMAIN, SUBDOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/23 21:56:06 manager.go:1428: [DEBUG] Running: docker compose down (in /opt/docker/stacks/papra) +2026/09/23 21:56:06 manager.go:1448: [DEBUG] Command completed: docker compose down (took 0.7s) +2026/09/23 21:56:06 manager.go:1260: [INFO] [stacks] Stack papra stopped successfully (took 0.7s) +2026/09/23 21:56:06 manager.go:734: [INFO] [stacks] Status refresh: 18 containers across 55 stacks +2026/09/23 21:56:06 lifecycle.go:24: [DEBUG] [integrations] OnStackStop: stack=papra integrationsFound=0 +2026/09/23 21:56:11 router.go:898: [INFO] [api] Remove requested for stack: papra +2026/09/23 21:56:11 delete.go:510: [DEBUG] [stacks] RemoveStack called: name="papra", removeHDDData=true, backupPathsToRemove=2 +2026/09/23 21:56:11 delete.go:524: [DEBUG] [stacks] RemoveStack papra: state=stopped, deployed=true, orphaned=false, deploying=false +2026/09/23 21:56:11 delete.go:581: [INFO] Removing deployed stack: papra (removeHDDData=true, hddDeclared=false, backupPaths=2) +2026/09/23 21:56:11 delete.go:593: [DEBUG] [stacks] RemoveStack papra: found 0 HDD mounts from compose file +2026/09/23 21:56:12 manager.go:1419: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, AUTH_SECRET, DOMAIN, IMPORT_PATH] (9 app + 0 system) +2026/09/23 21:56:12 manager.go:1428: [DEBUG] Running: docker compose down --volumes (in /opt/docker/stacks/papra) +2026/09/23 21:56:12 manager.go:1448: [DEBUG] Command completed: docker compose down --volumes (took 0.1s) +2026/09/23 21:56:12 delete.go:607: [DEBUG] [stacks] RemoveStack papra: compose down output: +2026/09/23 21:56:12 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:56:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:12 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:56:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 2m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:12 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:12 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container) +2026/09/23 21:56:13 manager.go:734: [INFO] [stacks] Status refresh: 18 containers across 55 stacks +2026/09/23 21:56:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:56:22 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6080MB avail≈19818MB +2026/09/23 21:56:22 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23503476 → total=25898MB avail=19818MB used=6080MB (23.5%) +2026/09/23 21:56:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:56:22 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:56:22 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.65 0.76 0.92 4/1655 10334" → 1m=0.65 5m=0.76 15m=0.92 +2026/09/23 21:56:22 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:56:22 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 48.9°C (hwmon1) +2026/09/23 21:56:22 info.go:13: [DEBUG] [system] GetInfo done in 63ms — mem=6080MB/25898MB (23.5%), rootDisk=20.0GB/68.4GB (29.3%), load=0.65/0.76/0.92, temp=48.9°C (hwmon1), cpu=8.0% +2026/09/23 21:56:22 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:56:22 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:56:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:22 scheduler.go:346: [INFO] [scheduler] Running job: system-health +2026/09/23 21:56:22 scheduler.go:67: [DEBUG] [scheduler] job system-health: execution starting +2026/09/23 21:56:22 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true) +2026/09/23 21:56:22 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:56:22 scheduler.go:346: [INFO] [scheduler] Running job: stack-scan +2026/09/23 21:56:22 scheduler.go:67: [DEBUG] [scheduler] job stack-scan: execution starting +2026/09/23 21:56:22 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:56:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:22 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:22 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container) +2026/09/23 21:56:22 scheduler.go:346: [INFO] [scheduler] Running job: offsite-credential-retry +2026/09/23 21:56:22 scheduler.go:67: [DEBUG] [scheduler] job offsite-credential-retry: execution starting +2026/09/23 21:56:22 scheduler.go:363: [INFO] [scheduler] Job offsite-credential-retry completed (took 0s) +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:56:22 scheduler.go:346: [INFO] [scheduler] Running job: backup-cache +2026/09/23 21:56:22 scheduler.go:67: [DEBUG] [scheduler] job backup-cache: execution starting +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:56:22 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=true composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:56:23 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:56:23 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:56:23 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:56:23 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714460KB) → used≈6074MB avail≈19824MB +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714460 availKB=23510524 → total=25898MB avail=19824MB used=6074MB (23.5%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=20.0GB avail=44.8GB (29.3%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=103.1GB avail=787.0GB (11.0%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] readLoadAvg: raw="0.59 0.75 0.91 7/1670 10377" → 1m=0.59 5m=0.75 15m=0.91 +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 48.9°C (hwmon1) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetInfo done in 71ms — mem=6074MB/25898MB (23.5%), rootDisk=20.0GB/68.4GB (29.3%), load=0.59/0.75/0.91, temp=48.9°C (hwmon1), cpu=8.0% +2026/09/23 21:56:23 healthcheck.go:86: [DEBUG] [monitor] Raw values: disk=29.3%, hdd=11.0% (configured=true), mem=23.5% (6074MB/25898MB), cpu=8.0%, temp=48.9°C (hwmon1) +2026/09/23 21:56:23 healthcheck.go:118: [DEBUG] [monitor] SSD disk: OK (29%) +2026/09/23 21:56:23 healthcheck.go:151: [DEBUG] [monitor] Memory: OK (23%) +2026/09/23 21:56:23 healthcheck.go:169: [DEBUG] [monitor] CPU: OK (8%) +2026/09/23 21:56:23 healthcheck.go:187: [DEBUG] [monitor] Temperature: OK (49°C) +2026/09/23 21:56:23 manager.go:734: [INFO] [stacks] Status refresh: 18 containers across 55 stacks +2026/09/23 21:56:23 scheduler.go:363: [INFO] [scheduler] Job stack-scan completed (took 71ms) +2026/09/23 21:56:23 manager.go:734: [INFO] [stacks] Status refresh: 18 containers across 55 stacks +2026/09/23 21:56:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/navidrome → [navidrome] +2026/09/23 21:56:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/nextcloud → [nextcloud] +2026/09/23 21:56:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx → [paperless-ngx] +2026/09/23 21:56:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/scratch_hdd/userdata/romm → [romm] +2026/09/23 21:56:23 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [actualbudget, adventurelog, docmost, gokapi, papra, privatebin, vikunja] +2026/09/23 21:56:23 backup.go:1117: [INFO] [backup] Found 10 DB dump files across drives +2026/09/23 21:56:23 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers +2026/09/23 21:56:23 healthcheck.go:201: [DEBUG] [monitor] Docker daemon: OK +2026/09/23 21:56:23 healthcheck.go:209: [DEBUG] [monitor] Checking 1 protected containers: [traefik] +2026/09/23 21:56:23 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 16089e1abac5 romm-db romm mariadb:11.8 +d6ac44f34971 vikunja vikunja vikunja/vikunja:2.5.0 +8a54f14e8bb0 navidrome navidrome deluan/navidrome:0.64.0 +9c3fff6d8323 romm romm rommapp/romm:5.3.1 +f428c67086c0 romm-redis romm redis:7-alpine +f12b35ea40d5 privatebin privatebin privatebin/pdo:2.0.6 +c916eed5d29a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15 +c509e115095a paperless-postgres paperless-ngx postgres:16-alpine +63a12434f006 paperless-redis paperless-ngx redis:7-alpi... +2026/09/23 21:56:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found mariadb container: romm-db (id=16089e1abac5) +2026/09/23 21:56:23 healthcheck.go:219: [DEBUG] [monitor] All protected containers running +2026/09/23 21:56:23 dbdump.go:198: [DEBUG] DiscoverDatabases: romm-db → stack=romm, dbUser=root, dbName=romm +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container vikunja (image=vikunja/vikunja:2.5.0, not a database) +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container navidrome (image=deluan/navidrome:0.64.0, not a database) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm (image=rommapp/romm:5.3.1, not a database) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/nextcloud" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container romm-redis (image=redis:7-alpine, not a database) +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container privatebin (image=privatebin/pdo:2.0.6, not a database) +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database) +2026/09/23 21:56:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=c509e115095a) +2026/09/23 21:56:23 dbdump.go:896: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless") +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:56:23 healthcheck.go:238: [INFO] [monitor] Health check: status=warn +2026/09/23 21:56:23 healthcheck.go:242: [DEBUG] [monitor] Final status: warn (issues=0, warnings=4, info=5) +2026/09/23 21:56:23 notifier.go:1259: [DEBUG] notifier disabled: dropped event health_change (severity warning) +2026/09/23 21:56:23 scheduler.go:363: [INFO] [scheduler] Job system-health completed (took 162ms) +2026/09/23 21:56:23 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database) +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost (image=docmost/docmost:0.96.0, not a database) +2026/09/23 21:56:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: docmost-postgres (id=c981df80d790) +2026/09/23 21:56:23 dbdump.go:198: [DEBUG] DiscoverDatabases: docmost-postgres → stack=docmost, dbUser=docmost, dbName=docmost +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container docmost-redis (image=redis:7-alpine, not a database) +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, not a database) +2026/09/23 21:56:23 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: adventurelog-postgres (id=5abd52db34b8) +2026/09/23 21:56:23 infra.go:67: [INFO] [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node) +2026/09/23 21:56:23 dbdump.go:198: [DEBUG] DiscoverDatabases: adventurelog-postgres → stack=adventurelog, dbUser=adventurelog, dbName=adventurelog +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, not a database) +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.267.0, not a database) +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.3.3-stable, not a database) +2026/09/23 21:56:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.6.7, not a database) +2026/09/23 21:56:23 dbdump.go:205: [DEBUG] DiscoverDatabases: found 4 database(s), skipped 14 non-DB container(s) +2026/09/23 21:56:23 dbdump.go:208: [INFO] [backup] Discovered 4 databases +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:56:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:56:23 [INFO] [backup] Discovered app data: 11 apps +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/navidrome" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/felhom-drives/scratch_hdd/userdata/romm" total=937.8 GB used=103.1 GB avail=787.0GB (11.0%) +2026/09/23 21:56:23 info.go:13: [DEBUG] [system] GetDiskUsage: path="/mnt/sys_drive" total=68.4 GB used=20.0 GB avail=44.8GB (29.3%) +2026/09/23 21:56:23 backup.go:1187: [INFO] [backup] Backup status cache refreshed +2026/09/23 21:56:23 scheduler.go:363: [INFO] [scheduler] Job backup-cache completed (took 249ms) +2026/09/23 21:56:32 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:56:32 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:56:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m40s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:32 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m10s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:32 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container) +2026/09/23 21:56:33 manager.go:734: [INFO] [stacks] Status refresh: 18 containers across 55 stacks +2026/09/23 21:56:38 delete.go:634: [INFO] [stacks] RemoveStack papra: removed volume(s) [papra_papra_data] +2026/09/23 21:56:38 delete.go:691: [DEBUG] [stacks] RemoveStack papra: processing 2 backup paths for removal (base=/mnt/sys_drive/felhom-data/backups) +2026/09/23 21:56:38 delete.go:708: [INFO] Removed backup data: /mnt/sys_drive/felhom-data/backups/primary/papra (604K) +2026/09/23 21:56:38 delete.go:716: [DEBUG] [stacks] RemoveStack papra: removing app.yaml at /opt/docker/stacks/papra/app.yaml +2026/09/23 21:56:38 delete.go:723: [INFO] Stack papra removed successfully (took 26.5s) +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=true composePath=/opt/docker/stacks/actualbudget/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=true composePath=/opt/docker/stacks/docmost/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=true composePath=/opt/docker/stacks/gokapi/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=false composePath=/opt/docker/stacks/immich/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=true composePath=/opt/docker/stacks/navidrome/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=true composePath=/opt/docker/stacks/nextcloud/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=false composePath=/opt/docker/stacks/rallly/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=true composePath=/opt/docker/stacks/romm/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=true composePath=/opt/docker/stacks/vikunja/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml +2026/09/23 21:56:38 manager.go:585: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml +2026/09/23 21:56:38 manager.go:1655: [DEBUG] [stacks] getCatalogTemplateSlugs: found 53 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates +2026/09/23 21:56:38 manager.go:652: [DEBUG] [stacks] ScanStacks: catalog has 53 template slugs for orphan detection +2026/09/23 21:56:38 manager.go:683: [INFO] [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available) +2026/09/23 21:56:38 manager.go:734: [INFO] [stacks] Status refresh: 18 containers across 55 stacks +2026/09/23 21:56:38 notifier.go:1255: [WARN] notifier disabled (no hub configured): DROPPED event app_removed (severity info) — further app_removed events are logged at DEBUG only +2026/09/23 21:56:38 lifecycle.go:176: [DEBUG] [integrations] OnStackRemove: stack=papra integrationsFound=0 +2026/09/23 21:56:42 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:56:42 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:56:43 manager.go:734: [INFO] [stacks] Status refresh: 18 containers across 55 stacks +2026/09/23 21:56:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 3m50s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:43 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m20s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:43 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container) +2026/09/23 21:56:52 scheduler.go:67: [DEBUG] [scheduler] job ring-spill: execution starting +2026/09/23 21:56:52 scheduler.go:67: [DEBUG] [scheduler] job status-refresh: execution starting +2026/09/23 21:56:52 scheduler.go:67: [DEBUG] [scheduler] job deadapp-check: execution starting +2026/09/23 21:56:52 scheduler.go:67: [DEBUG] [scheduler] job health-probes: execution starting +2026/09/23 21:56:53 manager.go:734: [INFO] [stacks] Status refresh: 18 containers across 55 stacks +2026/09/23 21:56:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping navidrome — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping privatebin — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping romm — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 3m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping docmost — last check 4m0s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:53 healthprobe.go:54: [DEBUG] [stacks] RunHealthProbes: skipping vikunja — last check 4m30s ago, effective interval 5m0s, healthy=true +2026/09/23 21:56:53 healthprobe.go:104: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (7 skipped not due, 0 skipped no container) diff --git a/documentation/audits/night-2026-09-23/chaos/round-12-run.txt b/documentation/audits/night-2026-09-23/chaos/round-12-run.txt new file mode 100644 index 00000000..1dfa2141 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-12-run.txt @@ -0,0 +1,6 @@ +23:56:06 ==== ROUND 12: remove on papra x none at +47s (2026-09-23T21:56:06Z) +23:56:06 [X] stop -> 200 {'ok': True, 'message': 'Stack papra stop completed'} +23:56:38 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'papra', 'volumes_removed': ['papra_papra_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazá +23:56:47 [X] after remove: deployed=False leftovers='/opt/docker/stacks/papra' +23:56:47 + 40.9s papra: deployed=False state=not_deployed updating=False phase=None hold='None' +23:57:02 ==== ROUND 12 END: steady in 40.9s; readback=None; events=2 diff --git a/documentation/audits/night-2026-09-23/chaos/round-12.json b/documentation/audits/night-2026-09-23/chaos/round-12.json new file mode 100644 index 00000000..d8cf7646 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-12.json @@ -0,0 +1,73 @@ +{ + "round": 12, + "action": "remove", + "accident": "none", + "delay_s": 47, + "app": "papra", + "started": "2026-09-23T21:56:06Z", + "before": { + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + }, + "state": { + "deployed": true, + "state": "running", + "update_phase": null, + "hold_reason": null, + "update_error": null + } + }, + "armed_at": "23:56:06", + "act": "remove -> 200", + "time_to_steady_s": 40.9, + "after": { + "state": "not deployed", + "leftovers": "" + }, + "observables": null, + "readback": null, + "events": [ + "2026-09-23T21:56:23Z health_change (warning)", + "2026-09-23T21:56:38Z app_removed (info)" + ], + "box_log": [ + "2026-09-23T21:56:06Z INFO [api] stop requested for stack: papra", + "2026-09-23T21:56:06Z INFO [stacks] desired state for papra recorded as \"stopped\" (was \"running\")", + "2026-09-23T21:56:06Z INFO [stacks] Stopping stack: papra", + "2026-09-23T21:56:06Z INFO [stacks] Stack papra stopped successfully (took 0.7s)", + "2026-09-23T21:56:11Z INFO [api] Remove requested for stack: papra", + "2026-09-23T21:56:11Z INFO Removing deployed stack: papra (removeHDDData=true, hddDeclared=false, backupPaths=2)", + "2026-09-23T21:56:22Z INFO [scheduler] Running job: backup-cache", + "2026-09-23T21:56:23Z INFO [stacks] ScanStacks complete: 55 stacks found (11 deployed, 44 available)", + "2026-09-23T21:56:23Z INFO [backup] Found 10 DB dump files across drives", + "2026-09-23T21:56:23Z INFO [backup] Discovered 4 databases", + "2026-09-23T21:56:23Z INFO [backup] Backup status cache refreshed", + "2026-09-23T21:56:23Z INFO [scheduler] Job backup-cache completed (took 249ms)", + "2026-09-23T21:56:38Z INFO [stacks] RemoveStack papra: removed volume(s) [papra_papra_data]", + "2026-09-23T21:56:38Z INFO Removed backup data: /mnt/sys_drive/felhom-data/backups/primary/papra (604K)", + "2026-09-23T21:56:38Z INFO Stack papra removed successfully (took 26.5s)", + "2026-09-23T21:56:38Z INFO [stacks] ScanStacks complete: 55 stacks found (10 deployed, 45 available)" + ], + "undo_copies_left": "0" +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/chaos/round-12.log b/documentation/audits/night-2026-09-23/chaos/round-12.log new file mode 100644 index 00000000..3228613c --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/round-12.log @@ -0,0 +1,5 @@ +23:56:06 ==== ROUND 12: remove on papra x none at +47s (2026-09-23T21:56:06Z) +23:56:06 [X] stop -> 200 {'ok': True, 'message': 'Stack papra stop completed'} +23:56:38 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'papra', 'volumes_removed': ['papra_papra_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazá +23:56:47 [X] after remove: deployed=False leftovers='/opt/docker/stacks/papra' +23:56:47 + 40.9s papra: deployed=False state=not_deployed updating=False phase=None hold='None' diff --git a/documentation/audits/night-2026-09-23/chaos/run-all.txt b/documentation/audits/night-2026-09-23/chaos/run-all.txt new file mode 100644 index 00000000..cb6eecb3 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos/run-all.txt @@ -0,0 +1,20 @@ +round 1: action=install accident=none steady=26.6 readback={'A': True} events=3 +round 2: action=good_update accident=POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'} at 22:30:30 steady=558.3 readback={'A': True, 'B': False} events=0 +round 3: action=restore accident=kill -9 of the controller's PID: killed 11644 at 22:39:39 steady=31.5 readback={'A': True, 'B': True} events=0 +round 4: action=fail_health accident=1 GB hog for 5 min: 5cd58c5c3e56c7b2fbb574e9be159f17cc25033c1d3a0e87f0b2692646d8660c +chaos-hog 1003MiB / 25.29GiB at 22:40:33 steady=605.2 readback={'A': True} events=2 +round 5: action=install accident=filled /var/lib/docker to 3 GB free (floor 2 GB + 1 GB): /dev/loop1 69G 62G 3.0G 96% /var/lib/docker at 22:50:45 steady=26.6 readback={'A': True} events=2 +round 6: action=good_update accident=1 GB hog for 5 min: 7922307cf3f2b0e1bebb33d30a94d644382b7596a42d0024aec74d1a81d5762f +chaos-hog 1004MiB / 25.29GiB at 22:51:50 steady=61.7 readback={'A': True} events=0 +round 7: action=restore accident=pct stop rc=0, pct start rc=0 at 22:53:24 steady=103.5 readback={'A': True, 'B': True} events=1 +round 8: action=cutoff accident=POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'} at 22:54:51 steady=101.2 readback={'A': False} events=1 +STOP written +STOP file present — not starting round 9 +round 9: action=fail_health accident=kill -9 of the controller's PID: killed 3308 at 23:04:11 steady=163.4 readback={'A': True, 'B': True} events=1 +round 10: action=remove accident=systemctl restart docker: rc=0 at 23:06:10 steady=1500.4 readback={'A': False} events=0 +round 11: action=cutoff accident=a second Update on romm -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} at 23:41:37 steady=142.7 readback={'A': False} events=0 +seed A unreadable because the app is HELD (stopped by design) — recovery is checked by the restore route, not a stop +round 12: action=remove accident=none steady=40.9 readback={} events=2 +Traceback (most recent call last): + File "", line 7, in +AttributeError: 'str' object has no attribute 'get' diff --git a/documentation/audits/night-2026-09-23/chaos_run.sh b/documentation/audits/night-2026-09-23/chaos_run.sh new file mode 100755 index 00000000..d7feb999 --- /dev/null +++ b/documentation/audits/night-2026-09-23/chaos_run.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# run chaos rounds $1..$2 in order; stop on a lost seed A (the brief's stop rule), evidence per round +cd "$(dirname "$0")" +for n in $(seq $1 $2); do + [ -f chaos/STOP ] && { echo "STOP file present — not starting round $n"; exit 0; } + python3 -u chaos.py round $n > chaos/round-$(printf %02d $n)-run.txt 2>&1 + python3 - "$n" <<'PY' +import json, sys +n = int(sys.argv[1]) +r = json.load(open(f"chaos/round-{n:02d}.json")) +rb = r.get("readback") or {} +lost = [k for k, v in rb.items() if k == "A" and v is False] +print(f"round {n}: action={r['action']} accident={r['accident']} steady={r.get('time_to_steady_s')} readback={rb} events={len(r.get('events') or [])}") +held = ((r.get("after") or {}).get("state") or {}).get("hold_reason") +if lost and held: + print("seed A unreadable because the app is HELD (stopped by design) — recovery is checked by the restore route, not a stop") +elif lost and r["action"] not in ("remove",): + open("chaos/STOP", "w").write(f"round {n}: seed A did not read back — the stop rule\n") + print("STOP written") +PY +done diff --git a/documentation/audits/night-2026-09-23/mk_rounds.py b/documentation/audits/night-2026-09-23/mk_rounds.py new file mode 100644 index 00000000..955cbce8 --- /dev/null +++ b/documentation/audits/night-2026-09-23/mk_rounds.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python3 +"""Turn chaos/round-NN.json into the findings doc's round table (the five things + the readback).""" +import glob, json, re + + +def short(x, n=160): + x = re.sub(r"\s+", " ", str(x or "")).strip() + return (x[:n] + "…") if len(x) > n else x + + +rows = [] +for p in sorted(glob.glob("chaos/round-[0-9][0-9].json")): + r = json.load(open(p)) + after = r.get("after") or {} + st = after.get("state") or {} + b = after.get("badges") or {} + lines = after.get("lines") or {} + saw = [] + for lang in ("hu", "en"): + bb = "; ".join(x.get("text", "") for x in (b.get(lang) or [])) + ln = (lines.get(lang) or {}) + extra = ln.get("undone_line") or ln.get("hold") or "" + saw.append(f"{lang}: {bb or '—'}{(' / ' + short(extra, 110)) if extra else ''}") + if not st and after.get("state") == "not deployed": + saw = [f"not installed (leftovers: {after.get('leftovers') or 'none'})"] + phases = " → ".join(f"{ph[1]}@{ph[0]}s" for ph in (r.get("phases") or [])) + boxdid = phases or short(r.get("act"), 120) + keylog = [l for l in (r.get("box_log") or []) if re.search(r"recovery|UNDO|UNDONE|HOLD|held|refus|interrupted|restore", l)] + if keylog: + boxdid += " | " + short(keylog[-1][21:], 140) + rb = r.get("readback") + if r.get("readback_B_reread") is not None: + rb = dict(rb or {}); rb["B"] = f"{r['readback_B_reread']['value']} (re-read, paged)" + rows.append(f"| {r['round']} | {r['action']} · {r['app']} | {short(r.get('accident'), 110)} (armed {r.get('armed_at','at start')}) | " + f"{' ‖ '.join(saw)} | {boxdid} | {r.get('time_to_steady_s')} s | " + f"{', '.join(e.split(' ', 1)[1] for e in (r.get('events') or [])) or 'none'} | {rb} |") +print("| # | action · app | accident (when) | what the household saw | what the box did by itself | steady | events fired (R-620 witness) | data read back |") +print("|---|---|---|---|---|---|---|---|") +print("\n".join(rows)) diff --git a/documentation/audits/night-2026-09-23/nc_rebuild.py b/documentation/audits/night-2026-09-23/nc_rebuild.py new file mode 100644 index 00000000..736f73b4 --- /dev/null +++ b/documentation/audits/night-2026-09-23/nc_rebuild.py @@ -0,0 +1,16 @@ +#!/usr/bin/env python3 +"""Nextcloud for the chaos hour, rebuilt clean after R-657: remove through the product, delete its kept +drive folder BY NAME (the scratch box; the product itself refuses the with-data remove here, R-442), +deploy again, seed A through occ.""" +import sys, time +sys.path.insert(0, ".") +import walk as w +from chaos import load, save, SIX, sub_of +w.login() +w.remove("nextcloud") +w.say(w.guest("rm -rf /mnt/felhom-drives/scratch_hdd/userdata/nextcloud; ls /mnt/felhom-drives/scratch_hdd/userdata/ | tr '\\n' ' '")) +w.deploy("nextcloud", sub_of("nextcloud")) +A = SIX["nextcloud"]["fx"].seed(w, sub_of("nextcloud"), w.say) +s = load(); s["apps"]["nextcloud"]["A"] = A; save(s) +w.say(f"nextcloud seed A: {bool(A)}; readback: {SIX['nextcloud']['fx'].verify(w, sub_of('nextcloud'), A, w.say) if A else None}") +open("chaos/00-nextcloud-rebuild.log", "w").write("\n".join(w.LOG) + "\n") diff --git a/documentation/audits/night-2026-09-23/press9201.py b/documentation/audits/night-2026-09-23/press9201.py new file mode 100644 index 00000000..01c9f563 --- /dev/null +++ b/documentation/audits/night-2026-09-23/press9201.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +"""press9201.py — Part D's second half: on demo-hp guest 9201 (the HP demo box, LIVE catalog, a hub), +press the guarded Update of each STANDING app that Part C moved tonight, one at a time. + + GUEST=9201 python3 press9201.py [ ...] + +Per app: the badge and the app page in BOTH languages before; the guarded Update and every phase with +its time; the four version observables after; the app's front door after; the page in both languages +after. Nothing is seeded (these are the demo box's own standing apps) and nothing is fixed: a failure +here is the undo's first real outing on a standing app — recorded, not repaired. +No whole-box backup is pressed (R-648): the update's own `backing-up` phase backs up the one app. +""" +import json, os, sys, time +os.environ.setdefault("GUEST", "9201") +sys.path.insert(0, ".") +import walk as w +import live + +FRONT = {"romm": ("arcade", "/api/heartbeat"), "opengist": ("gist", "/healthcheck"), + "kimai": ("kimai", "/en/login")} +EV = os.path.join(os.path.dirname(os.path.abspath(__file__)), "press9201") +os.makedirs(EV, exist_ok=True) + + +def sub_for(app): + ac = (w.stack(app).get("app_config") or {}) + env = ac.get("env") or {} + return env.get("SUBDOMAIN") or FRONT[app][0] + + +def main(apps): + assert w.GUEST == "9201", "this script presses on 9201 only" + w.login() + w.say("sync + rescan so the box sees tonight's live catalog") + w.sync_rescan() + for app in apps: + w.LOG.clear() + rec = {"app": app} + sub = sub_for(app) + w.say(f"==== 9201 {app} (sub {sub})") + rec["before"] = {"observables": w.observables(app), "badges": w.badges(app), "lines": live.page_lines(app)} + want = list((rec["before"]["observables"].get("catalog_images") or {}).values()) + w.say(f" catalog now offers {want}; pinned {rec['before']['observables'].get('pinned_images')}") + res = w.press_update(app) + rec["press"] = res + rec["after"] = {"observables": w.observables(app), "badges": w.badges(app), "lines": live.page_lines(app)} + path = FRONT.get(app, (sub, "/"))[1] + rc, code, _ = w.app_curl(sub, path, timeout=30) + rec["front_door"] = {"path": path, "rc": rc, "http": code} + w.say(f" END phase={res.get('final_phase')} hold={res.get('hold_reason')!r} front door {path} -> {code}") + w.say(f" pinned after: {rec['after']['observables'].get('pinned_images')}") + w.say(f" inspect after: {rec['after']['observables'].get('docker_inspect')}") + w.say(f" page hu/en after: {json.dumps(rec['after']['badges'], ensure_ascii=False)[:300]}") + json.dump(rec, open(os.path.join(EV, f"{app}.json"), "w"), indent=2, ensure_ascii=False) + open(os.path.join(EV, f"{app}.log"), "w").write("\n".join(w.LOG) + "\n") + + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/documentation/audits/night-2026-09-23/press9201/kimai.json b/documentation/audits/night-2026-09-23/press9201/kimai.json new file mode 100644 index 00000000..5face3a4 --- /dev/null +++ b/documentation/audits/night-2026-09-23/press9201/kimai.json @@ -0,0 +1,163 @@ +{ + "app": "kimai", + "before": { + "observables": { + "pinned_images": { + "kimai": "kimai/kimai2:apache-2.57.0", + "kimai-db": "mariadb:11.6" + }, + "installed_images": { + "kimai": "kimai/kimai2:apache-2.57.0", + "kimai-db": "mariadb:11.6" + }, + "catalog_images": { + "kimai": "kimai/kimai2:apache-2.57.0", + "kimai-db": "mariadb:11.8" + }, + "live_compose_image_lines": [ + "image: kimai/kimai2:apache-2.57.0", + "image: mariadb:11.6" + ], + "docker_inspect": [ + "kimai kimai/kimai2:apache-2.57.0 running=true restarts=0", + "kimai-db mariadb:11.6 running=true restarts=0" + ] + }, + "badges": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + } + }, + "press": { + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 2.1, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 6.2, + "phase": "copying", + "label": "Az adatok másolása a frissítés előtt…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 18.7, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 34.1, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 64.8, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 64.8, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" + }, + "after": { + "observables": { + "pinned_images": { + "kimai": "kimai/kimai2:apache-2.57.0", + "kimai-db": "mariadb:11.8" + }, + "installed_images": { + "kimai": "kimai/kimai2:apache-2.57.0", + "kimai-db": "mariadb:11.8" + }, + "catalog_images": { + "kimai": "kimai/kimai2:apache-2.57.0", + "kimai-db": "mariadb:11.8" + }, + "live_compose_image_lines": [ + "image: kimai/kimai2:apache-2.57.0", + "image: mariadb:11.8" + ], + "docker_inspect": [ + "kimai-db mariadb:11.8 running=true restarts=0", + "kimai kimai/kimai2:apache-2.57.0 running=true restarts=0" + ] + }, + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + } + }, + "front_door": { + "path": "/en/login", + "rc": 0, + "http": "200" + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/press9201/kimai.log b/documentation/audits/night-2026-09-23/press9201/kimai.log new file mode 100644 index 00000000..342cace0 --- /dev/null +++ b/documentation/audits/night-2026-09-23/press9201/kimai.log @@ -0,0 +1,13 @@ +23:18:32 ==== 9201 kimai (sub time) +23:18:35 catalog now offers ['kimai/kimai2:apache-2.57.0', 'mariadb:11.8']; pinned {'kimai': 'kimai/kimai2:apache-2.57.0', 'kimai-db': 'mariadb:11.6'} +23:18:35 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +23:18:35 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +23:18:38 + 2.1s phase=pulling label=Új verzió letöltése… err=None hold=None +23:18:42 + 6.2s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None +23:18:54 + 18.7s phase=starting label=Indítás az új verzióval… err=None hold=None +23:19:09 + 34.1s phase=verifying label=Működés ellenőrzése… err=None hold=None +23:19:40 + 64.8s phase=done label=Frissítve err=None hold=None +23:19:44 END phase=done hold=None front door /en/login -> 200 +23:19:44 pinned after: {'kimai': 'kimai/kimai2:apache-2.57.0', 'kimai-db': 'mariadb:11.8'} +23:19:44 inspect after: ['kimai-db mariadb:11.8 running=true restarts=0', 'kimai kimai/kimai2:apache-2.57.0 running=true restarts=0'] +23:19:44 page hu/en after: {"hu": [{"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", "text": "Naprakész"}], "en": [{"title": "This app is running the newest version available.", "text": "Up to date"}]} diff --git a/documentation/audits/night-2026-09-23/press9201/opengist.json b/documentation/audits/night-2026-09-23/press9201/opengist.json new file mode 100644 index 00000000..0143afb7 --- /dev/null +++ b/documentation/audits/night-2026-09-23/press9201/opengist.json @@ -0,0 +1,145 @@ +{ + "app": "opengist", + "before": { + "observables": { + "pinned_images": { + "opengist": "ghcr.io/thomiceli/opengist:1.13" + }, + "installed_images": { + "opengist": "ghcr.io/thomiceli/opengist:1.13" + }, + "catalog_images": { + "opengist": "ghcr.io/thomiceli/opengist:1.15" + }, + "live_compose_image_lines": [ + "image: ghcr.io/thomiceli/opengist:1.13" + ], + "docker_inspect": [ + "opengist ghcr.io/thomiceli/opengist:1.13 running=true restarts=0" + ] + }, + "badges": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + } + }, + "press": { + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.0, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 5.1, + "phase": "copying", + "label": "Az adatok másolása a frissítés előtt…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 6.2, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 12.3, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 12.3, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" + }, + "after": { + "observables": { + "pinned_images": { + "opengist": "ghcr.io/thomiceli/opengist:1.15" + }, + "installed_images": { + "opengist": "ghcr.io/thomiceli/opengist:1.15" + }, + "catalog_images": { + "opengist": "ghcr.io/thomiceli/opengist:1.15" + }, + "live_compose_image_lines": [ + "image: ghcr.io/thomiceli/opengist:1.15" + ], + "docker_inspect": [ + "opengist ghcr.io/thomiceli/opengist:1.15 running=true restarts=0" + ] + }, + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + } + }, + "front_door": { + "path": "/healthcheck", + "rc": 0, + "http": "200" + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/press9201/opengist.log b/documentation/audits/night-2026-09-23/press9201/opengist.log new file mode 100644 index 00000000..e0e8b9ec --- /dev/null +++ b/documentation/audits/night-2026-09-23/press9201/opengist.log @@ -0,0 +1,12 @@ +23:18:12 ==== 9201 opengist (sub gist) +23:18:16 catalog now offers ['ghcr.io/thomiceli/opengist:1.15']; pinned {'opengist': 'ghcr.io/thomiceli/opengist:1.13'} +23:18:16 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +23:18:16 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +23:18:17 + 1.0s phase=pulling label=Új verzió letöltése… err=None hold=None +23:18:21 + 5.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None +23:18:22 + 6.2s phase=verifying label=Működés ellenőrzése… err=None hold=None +23:18:28 + 12.3s phase=done label=Frissítve err=None hold=None +23:18:32 END phase=done hold=None front door /healthcheck -> 200 +23:18:32 pinned after: {'opengist': 'ghcr.io/thomiceli/opengist:1.15'} +23:18:32 inspect after: ['opengist ghcr.io/thomiceli/opengist:1.15 running=true restarts=0'] +23:18:32 page hu/en after: {"hu": [{"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", "text": "Naprakész"}], "en": [{"title": "This app is running the newest version available.", "text": "Up to date"}]} diff --git a/documentation/audits/night-2026-09-23/press9201/romm-memory.txt b/documentation/audits/night-2026-09-23/press9201/romm-memory.txt new file mode 100644 index 00000000..2f633d89 --- /dev/null +++ b/documentation/audits/night-2026-09-23/press9201/romm-memory.txt @@ -0,0 +1 @@ +23:23 9201 romm after the two-step press: 599.7MiB/768MiB, oom_kill 0, restarts 0, 0 SIGKILL lines in 10 min diff --git a/documentation/audits/night-2026-09-23/press9201/romm.json b/documentation/audits/night-2026-09-23/press9201/romm.json new file mode 100644 index 00000000..728e5dc3 --- /dev/null +++ b/documentation/audits/night-2026-09-23/press9201/romm.json @@ -0,0 +1,173 @@ +{ + "app": "romm", + "before": { + "observables": { + "pinned_images": { + "romm": "rommapp/romm:5.3.0", + "romm-db": "mariadb:11.4", + "romm-redis": "redis:7-alpine" + }, + "installed_images": { + "romm": "rommapp/romm:5.3.0", + "romm-db": "mariadb:11.4", + "romm-redis": "redis:7-alpine" + }, + "catalog_images": { + "romm": "rommapp/romm:5.3.1", + "romm-db": "mariadb:11.8", + "romm-redis": "redis:7-alpine" + }, + "live_compose_image_lines": [ + "image: rommapp/romm:5.3.0", + "image: mariadb:11.4", + "image: redis:7-alpine" + ], + "docker_inspect": [ + "romm rommapp/romm:5.3.0 running=true restarts=0", + "romm-db mariadb:11.4 running=true restarts=0", + "romm-redis redis:7-alpine running=true restarts=0" + ] + }, + "badges": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + } + }, + "press": { + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 2.1, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 13.4, + "phase": "copying", + "label": "Az adatok másolása a frissítés előtt…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 34.9, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 46.1, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 82.0, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 82.1, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" + }, + "after": { + "observables": { + "pinned_images": { + "romm": "rommapp/romm:5.3.1", + "romm-db": "mariadb:11.8", + "romm-redis": "redis:7-alpine" + }, + "installed_images": { + "romm": "rommapp/romm:5.3.1", + "romm-db": "mariadb:11.8", + "romm-redis": "redis:7-alpine" + }, + "catalog_images": { + "romm": "rommapp/romm:5.3.1", + "romm-db": "mariadb:11.8", + "romm-redis": "redis:7-alpine" + }, + "live_compose_image_lines": [ + "image: rommapp/romm:5.3.1", + "image: mariadb:11.8", + "image: redis:7-alpine" + ], + "docker_inspect": [ + "romm rommapp/romm:5.3.1 running=true restarts=0", + "romm-db mariadb:11.8 running=true restarts=0", + "romm-redis redis:7-alpine running=true restarts=0" + ] + }, + "badges": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "lines": { + "hu": { + "undone_line": null, + "hold": null + }, + "en": { + "undone_line": null, + "hold": null + } + } + }, + "front_door": { + "path": "/api/heartbeat", + "rc": 0, + "http": "200" + } +} \ No newline at end of file diff --git a/documentation/audits/night-2026-09-23/press9201/romm.log b/documentation/audits/night-2026-09-23/press9201/romm.log new file mode 100644 index 00000000..d0129a86 --- /dev/null +++ b/documentation/audits/night-2026-09-23/press9201/romm.log @@ -0,0 +1,13 @@ +23:19:44 ==== 9201 romm (sub jatek) +23:19:47 catalog now offers ['rommapp/romm:5.3.1', 'mariadb:11.8', 'redis:7-alpine']; pinned {'romm': 'rommapp/romm:5.3.0', 'romm-db': 'mariadb:11.4', 'romm-redis': 'redis:7-alpine'} +23:19:47 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +23:19:48 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +23:19:50 + 2.1s phase=pulling label=Új verzió letöltése… err=None hold=None +23:20:01 + 13.4s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None +23:20:22 + 34.9s phase=starting label=Indítás az új verzióval… err=None hold=None +23:20:34 + 46.1s phase=verifying label=Működés ellenőrzése… err=None hold=None +23:21:10 + 82.0s phase=done label=Frissítve err=None hold=None +23:21:17 END phase=done hold=None front door /api/heartbeat -> 200 +23:21:17 pinned after: {'romm': 'rommapp/romm:5.3.1', 'romm-db': 'mariadb:11.8', 'romm-redis': 'redis:7-alpine'} +23:21:17 inspect after: ['romm rommapp/romm:5.3.1 running=true restarts=0', 'romm-db mariadb:11.8 running=true restarts=0', 'romm-redis redis:7-alpine running=true restarts=0'] +23:21:17 page hu/en after: {"hu": [{"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", "text": "Naprakész"}], "en": [{"title": "This app is running the newest version available.", "text": "Up to date"}]} diff --git a/documentation/audits/night-2026-09-23/press9201/run.txt b/documentation/audits/night-2026-09-23/press9201/run.txt new file mode 100644 index 00000000..3070543c --- /dev/null +++ b/documentation/audits/night-2026-09-23/press9201/run.txt @@ -0,0 +1,39 @@ +23:18:08 sync + rescan so the box sees tonight's live catalog +23:18:12 ==== 9201 opengist (sub gist) +23:18:16 catalog now offers ['ghcr.io/thomiceli/opengist:1.15']; pinned {'opengist': 'ghcr.io/thomiceli/opengist:1.13'} +23:18:16 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +23:18:16 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +23:18:17 + 1.0s phase=pulling label=Új verzió letöltése… err=None hold=None +23:18:21 + 5.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None +23:18:22 + 6.2s phase=verifying label=Működés ellenőrzése… err=None hold=None +23:18:28 + 12.3s phase=done label=Frissítve err=None hold=None +23:18:32 END phase=done hold=None front door /healthcheck -> 200 +23:18:32 pinned after: {'opengist': 'ghcr.io/thomiceli/opengist:1.15'} +23:18:32 inspect after: ['opengist ghcr.io/thomiceli/opengist:1.15 running=true restarts=0'] +23:18:32 page hu/en after: {"hu": [{"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", "text": "Naprakész"}], "en": [{"title": "This app is running the newest version available.", "text": "Up to date"}]} +23:18:32 ==== 9201 kimai (sub time) +23:18:35 catalog now offers ['kimai/kimai2:apache-2.57.0', 'mariadb:11.8']; pinned {'kimai': 'kimai/kimai2:apache-2.57.0', 'kimai-db': 'mariadb:11.6'} +23:18:35 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +23:18:35 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +23:18:38 + 2.1s phase=pulling label=Új verzió letöltése… err=None hold=None +23:18:42 + 6.2s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None +23:18:54 + 18.7s phase=starting label=Indítás az új verzióval… err=None hold=None +23:19:09 + 34.1s phase=verifying label=Működés ellenőrzése… err=None hold=None +23:19:40 + 64.8s phase=done label=Frissítve err=None hold=None +23:19:44 END phase=done hold=None front door /en/login -> 200 +23:19:44 pinned after: {'kimai': 'kimai/kimai2:apache-2.57.0', 'kimai-db': 'mariadb:11.8'} +23:19:44 inspect after: ['kimai-db mariadb:11.8 running=true restarts=0', 'kimai kimai/kimai2:apache-2.57.0 running=true restarts=0'] +23:19:44 page hu/en after: {"hu": [{"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", "text": "Naprakész"}], "en": [{"title": "This app is running the newest version available.", "text": "Up to date"}]} +23:19:44 ==== 9201 romm (sub jatek) +23:19:47 catalog now offers ['rommapp/romm:5.3.1', 'mariadb:11.8', 'redis:7-alpine']; pinned {'romm': 'rommapp/romm:5.3.0', 'romm-db': 'mariadb:11.4', 'romm-redis': 'redis:7-alpine'} +23:19:47 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +23:19:48 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +23:19:50 + 2.1s phase=pulling label=Új verzió letöltése… err=None hold=None +23:20:01 + 13.4s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None +23:20:22 + 34.9s phase=starting label=Indítás az új verzióval… err=None hold=None +23:20:34 + 46.1s phase=verifying label=Működés ellenőrzése… err=None hold=None +23:21:10 + 82.0s phase=done label=Frissítve err=None hold=None +23:21:17 END phase=done hold=None front door /api/heartbeat -> 200 +23:21:17 pinned after: {'romm': 'rommapp/romm:5.3.1', 'romm-db': 'mariadb:11.8', 'romm-redis': 'redis:7-alpine'} +23:21:17 inspect after: ['romm rommapp/romm:5.3.1 running=true restarts=0', 'romm-db mariadb:11.8 running=true restarts=0', 'romm-redis redis:7-alpine running=true restarts=0'] +23:21:17 page hu/en after: {"hu": [{"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", "text": "Naprakész"}], "en": [{"title": "This app is running the newest version available.", "text": "Up to date"}]} diff --git a/documentation/audits/night-2026-09-23/r11_recover.py b/documentation/audits/night-2026-09-23/r11_recover.py new file mode 100644 index 00000000..e2ad0c74 --- /dev/null +++ b/documentation/audits/night-2026-09-23/r11_recover.py @@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +"""Round 11 follow-up: the held nextcloud is brought back by the household's OWN route — the restore +the hold sentence names — and seed A is read back. Decides whether round 8 lost data (the stop rule).""" +import json, sys, time +sys.path.insert(0, ".") +import walk as w +from chaos import load, sub_of, SIX +w.login(); s = load() +snaps = w.snapshots("nextcloud") +w.say(f"restore points offered: {json.dumps(snaps, ensure_ascii=False)[:400]}") +res = w.restore("nextcloud") +st = w.stack("nextcloud") +w.say(f"after restore: state={st.get('state')} hold={st.get('hold_reason')!r} phase={st.get('update_phase')}") +A = SIX["nextcloud"]["fx"].verify(w, sub_of("nextcloud"), s["apps"]["nextcloud"]["A"], w.say) +obs = w.observables("nextcloud") +w.say(f"seed A after the household's restore: {A}; pinned {obs.get('pinned_images')}") +json.dump({"snapshots": snaps, "restore": {k: res.get(k) for k in ("ok", "snapshot_id", "http", "seconds", "state_after", "hold_after")}, + "readback_A": A, "observables": obs}, open("chaos/round-11-recovery.json", "w"), indent=2, ensure_ascii=False) +open("chaos/round-11-recovery.log", "w").write("\n".join(w.LOG) + "\n") diff --git a/documentation/audits/night-2026-09-23/r640_live.py b/documentation/audits/night-2026-09-23/r640_live.py new file mode 100644 index 00000000..d92987d4 --- /dev/null +++ b/documentation/audits/night-2026-09-23/r640_live.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 +"""R-640 live on 9202 (v0.267.0): a CUT-OFF database copy in docmost's local unit must be refused by +the household's own restore button, BEFORE anything is touched. + +The copy is cut to half its size (the exact shape measured on 2026-09-23 morning: a copy that ends +inside a COPY block, no completion marker). Positive observables: the restore's own answer carries +the „csonka" sentence; docmost's containers keep their start times (nothing was stopped); seed A still +reads back. Negative control: the whole copy is put back byte for byte and the same button then +restores (so the refusal is about the cut, not about the button).""" +import json, sys, time +sys.path.insert(0, ".") +import walk as w +from chaos import load, sub_of, SIX + +w.login() +s = load() +out = {} +find = w.guest("ls -la /mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/ 2>&1; " + "find / -xdev -path '*backups/primary/docmost*' -name 'docmost-postgres.sql' 2>/dev/null | head -3") +w.say("the unit's dump:\n" + find) +paths = [l for l in find.splitlines() if l.endswith("docmost-postgres.sql") and l.startswith("/")] or \ + ["/mnt/sys_drive/felhom-data/backups/primary/docmost/db-dumps/docmost-postgres.sql"] +if not paths: + sys.exit("no docmost dump found in a local unit") +dump = paths[0] +out["dump"] = dump +starts0 = w.guest("docker ps --filter label=com.docker.compose.project=docmost --format '{{.Names}} {{.RunningFor}}'; " + "for c in docmost docmost-postgres; do docker inspect -f '{{.Name}} {{.State.StartedAt}}' $c; done") +w.say("before:\n" + starts0) +cut = w.guest(f"""set -e +cp -p {dump} /root/r640-whole.sql +n=$(stat -c %s {dump}); head -c $((n/2)) /root/r640-whole.sql > {dump} +echo "whole=$n cut=$(stat -c %s {dump})"; tail -c 200 {dump} | tail -2; echo; grep -c 'dump complete' {dump} || true""") +w.say("cut:\n" + cut) +out["cut"] = cut +res = w.restore("docmost") +code, st = w.ctl("GET", "/api/backup/restore-status") +w.say(f"restore-status after the refused restore: {json.dumps(st, ensure_ascii=False)[:600]}") +out["refused_restore"] = {"result": {k: res.get(k) for k in ("http", "location", "seconds", "state_after", "hold_after")}, + "status": st} +for lang in ("hu", "en"): + h = w.page(f"/backups?lang={lang}") + import re, html + m = re.findall(r"[^<>]{0,160}(?:csonka|cut off)[^<>]{0,200}", html.unescape(h)) + out[f"page_{lang}"] = m[:2] + w.say(f" /backups ({lang}) carries the sentence: {m[:1]}") +starts1 = w.guest("for c in docmost docmost-postgres; do docker inspect -f '{{.Name}} {{.State.StartedAt}}' $c; done") +w.say("after:\n" + starts1) +out["untouched"] = [l for l in starts0.splitlines() if "StartedAt" not in l and l.startswith("/")] == \ + [l for l in starts1.splitlines() if l.startswith("/")] +out["readback_A"] = SIX["docmost"]["fx"].verify(w, sub_of("docmost"), s["apps"]["docmost"]["A"], w.say) +w.say(f"container start times unchanged: {out['untouched']}; seed A: {out['readback_A']}") +# negative control: the whole copy back, the same button restores +w.guest(f"cp -p /root/r640-whole.sql {dump}; rm -f /root/r640-whole.sql; grep -c 'dump complete' {dump}") +res2 = w.restore("docmost") +code, st2 = w.ctl("GET", "/api/backup/restore-status") +out["control_restore"] = {"result": {k: res2.get(k) for k in ("http", "seconds", "state_after", "hold_after")}, + "status": st2} +out["control_readback_A"] = SIX["docmost"]["fx"].verify(w, sub_of("docmost"), s["apps"]["docmost"]["A"], w.say) +w.say(f"control (whole copy): status {json.dumps(st2, ensure_ascii=False)[:300]}; seed A {out['control_readback_A']}") +json.dump(out, open("E1-r640-live.json", "w"), indent=2, ensure_ascii=False) +open("E1-r640-live.log", "w").write("\n".join(w.LOG) + "\n") diff --git a/documentation/audits/night-2026-09-23/r8_recover.py b/documentation/audits/night-2026-09-23/r8_recover.py new file mode 100644 index 00000000..2acdd08f --- /dev/null +++ b/documentation/audits/night-2026-09-23/r8_recover.py @@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +"""Round 8 follow-up: the held navidrome is brought back by the household's OWN route — the restore +the hold sentence names — and seed A is read back. Decides whether round 8 lost data (the stop rule).""" +import json, sys, time +sys.path.insert(0, ".") +import walk as w +from chaos import load, sub_of, SIX +w.login(); s = load() +snaps = w.snapshots("navidrome") +w.say(f"restore points offered: {json.dumps(snaps, ensure_ascii=False)[:400]}") +res = w.restore("navidrome") +st = w.stack("navidrome") +w.say(f"after restore: state={st.get('state')} hold={st.get('hold_reason')!r} phase={st.get('update_phase')}") +A = SIX["navidrome"]["fx"].verify(w, sub_of("navidrome"), s["apps"]["navidrome"]["A"], w.say) +obs = w.observables("navidrome") +w.say(f"seed A after the household's restore: {A}; pinned {obs.get('pinned_images')}") +json.dump({"snapshots": snaps, "restore": {k: res.get(k) for k in ("ok", "snapshot_id", "http", "seconds", "state_after", "hold_after")}, + "readback_A": A, "observables": obs}, open("chaos/round-08-recovery.json", "w"), indent=2, ensure_ascii=False) +open("chaos/round-08-recovery.log", "w").write("\n".join(w.LOG) + "\n") diff --git a/documentation/audits/night-2026-09-23/spike.py b/documentation/audits/night-2026-09-23/spike.py index 369d1f64..c8c71116 100644 --- a/documentation/audits/night-2026-09-23/spike.py +++ b/documentation/audits/night-2026-09-23/spike.py @@ -55,8 +55,24 @@ def docmost_verify_b(sub, A, B): data=json.dumps({"email": A["email"], "password": A["pw"]}), method="POST") if code not in ("200", "201"): w.say(f" docmost B: cannot log in (http={code})"); return False - rc, code, out = w.app_curl(sub, "/api/spaces", "-b", jar, "-H", "Content-Type: application/json", - data="{}", method="POST") + # PAGE THROUGH (night 2026-09-23): the list is 20 per page by name; with the chaos hour's 400 + # "big…" spaces B sits pages deep — the first-page read said False while the row was there. + seen, cursor, code = "", None, None + for _ in range(50): + body = {"limit": 100} + if cursor: + body["cursor"] = cursor + rc, code, out = w.app_curl(sub, "/api/spaces", "-b", jar, "-H", "Content-Type: application/json", + data=json.dumps(body), method="POST") + seen += out or "" + try: + meta = (json.loads(out).get("data") or {}).get("meta") or {} + except Exception: + break + if not meta.get("hasNextPage"): + break + cursor = meta.get("nextCursor") + out = seen ok = code in ("200", "201") and B["space"] in out neg = "drillBnever" in out w.say(f" docmost B: /api/spaces http={code} seeded-space-listed={ok} (negative control listed={neg})") diff --git a/documentation/audits/night-2026-09-23/teardown.py b/documentation/audits/night-2026-09-23/teardown.py new file mode 100644 index 00000000..e4e5c33a --- /dev/null +++ b/documentation/audits/night-2026-09-23/teardown.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +"""Part E teardown on 9202 — every app removed THROUGH THE PRODUCT, then the leftovers read by name. +R-658's second half is measured here: does the remove of a RESTORED app (volumes without the compose +label) leave its volumes behind? docmost, navidrome and vikunja were restored tonight; romm, nextcloud, +adventurelog were not (the control).""" +import json, sys, time +sys.path.insert(0, ".") +import walk as w +w.login() +apps = ["docmost", "navidrome", "vikunja", "romm", "nextcloud", "adventurelog", "actualbudget"] +before = w.guest("for p in " + " ".join(apps) + "; do echo \"$p all=$(docker volume ls -q | grep -c ^${p}_) labeled=$(docker volume ls -q --filter label=com.docker.compose.project=$p | wc -l) copies=$(docker volume ls -q --filter label=felhom.undo-copy-of=$p | wc -l)\"; done") +w.say("volumes before the removes:\n" + before) +res = {} +for a in apps: + st = w.stack(a) + if not st.get("deployed"): + res[a] = "not deployed"; continue + code = w.remove(a) + res[a] = code +after = w.guest("for p in " + " ".join(apps) + "; do echo \"$p containers=$(docker ps -aq --filter label=com.docker.compose.project=$p | wc -l) volumes=$(docker volume ls -q | grep ^${p}_ | tr '\\n' ' ') copies=$(docker volume ls -q --filter label=felhom.undo-copy-of=$p | wc -l)\"; done") +w.say("after the removes (the product's own):\n" + after) +json.dump({"before": before, "remove": res, "after": after}, open("E2-teardown-removes.json", "w"), indent=2) +open("E2-teardown-removes.log", "w").write("\n".join(w.LOG) + "\n") diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 5cb3110b..3b0b240f 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -358,4 +358,7 @@ Compressed here to title, shipping version, evidence, and the sentences that sta | **R-647** | **Three leftovers of the update mail (P3).** v0.265.0: a held update's error is the key `update.error.held`, rendered per reader on both pages and the API; `copy_holds` travels as its key; the two log wordings fixed. | **CLOSED 2026-09-23 — PROVEN-LIVE for (1) (`43-*`, `44-*`); (2)(3) by red-proofed tests** | as above | | **R-648** | **The drill's „Mentés most" was whole-box (P3).** No per-app backup endpoint exists; the harness (`audits/cleanup-2026-09-23/walk.py` `backup_now`) now presses nothing and the guarded update's own `backing-up` phase backs up the throwaway app alone. | **CLOSED 2026-09-23 — PROVEN-LIVE (`43-*`: phase `backing-up` for vikunja only)** | as above | | **R-649** | **A failed install could leave containers under „not deployed" (P2).** Operator ruling 2026-09-23 (option a): the controller cleans up. Closed in **v0.266.0** (`964ae75`): `runComposeDeploy`'s failure branch runs `compose down` (volumes kept) before the record reads not-deployed. | **CLOSED 2026-09-23 — PROVEN-LIVE (`audits/r649-2026-09-23/`: outline with a never-healthy redis — 0 containers left, 3 volumes kept)** | `git show HEAD~1:documentation/backlog/OPEN-ITEMS.md` | - +| **R-650** | **A controller unit test could act on DooPlex's production Docker (P3 → raised to P2 by the night brief).** Closed in **v0.267.0** (`80e6ad8c4772`): `internal/dockerexec` — every docker exec goes through it; under `go test` a real docker is refused with an error naming the command (opt-in `FELHOM_TEST_REAL_DOCKER=1`; a stub under the temp dir allowed). The sweep found 8 `api` tests and the `web`/`stacks`/`backup`/`appexport`/`system` fixtures reaching the real daemon (one `docker-compose down`); `api`/`stacks`/`web` now run under a silent stub. `TestR650_NoBareDockerExec` pins the invariant. | **CLOSED 2026-09-23 — red-proofed twice (guard off → the decoy runs; a bare call → the sweep names it)** | `audits/night-2026-09-23/A1-*` | +| **R-640** | **A truncated PostgreSQL copy loaded with rc 0 into an empty database (P2, narrowed to the restore paths).** Closed in **v0.267.0**: `appbackup.CheckDumpComplete` (the engine's end marker); the unit restore and the off-site restore refuse before the first mutation, every replay checks again before any load. **Rule:** a dump is judged by its END — the header and a `CREATE TABLE` say nothing about whether it finished. | **CLOSED 2026-09-23 — three red-proofs + PROVEN-LIVE on 9202 (the household's restore button refused a half-length docmost copy; containers untouched; the whole copy then restored)** | `audits/night-2026-09-23/A2-*`, `E1-r640-live.*` | +| **R-499** | **Every driveless app was told its data was „already in the full system backup (PBS)" (P2).** Closed in **v0.267.0**: the Tier-2 page's sentence has four branches from the box's own whole-system backup target (own drive / same disk / drive gone / cannot ask); „(PBS)" and „nincs külön teendő" only where true. | **CLOSED 2026-09-23 — two red-proofs; live on 9202 (the `unknown` branch, hu + en, matching `/api/storage/backup-target`)** | `audits/night-2026-09-23/A4-*`, `A7-*` | +| **R-626** | **A removed app came back (P2).** Measured on v0.266.0, NOT reproduced: navidrome removed through the product, 390 s of `docker events` (the remove's destroy seen, no create), a controller restart at +150 s, a guest reboot after → no container, no volume. The two known creators (the restore/remove race R-633, the backup/deploy race R-634) are fixed. **Rule kept from the row:** a check that runs once, immediately, cannot see a thing created just after it — watch a window. Leftover found and filed: R-651. | **CLOSED 2026-09-23 — by measurement (a positive observable: the destroy events)** | `audits/night-2026-09-23/A3-*` | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index b99077ac..b636ec06 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -663,17 +663,17 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-435** | **The snapshot-drop detector is blind to a single-app deletion — the exact shape the box can produce.** `snapshotDropFraction = 0.5` and `snapshotDropFloor = 5` (`hub/internal/monitor/offsite.go`) require a fall of MORE than half the previous count. demo-hp's baseline is **69** across 9 apps, so ~35 snapshots must go before it speaks; **one app's tag is ~9 and is invisible.** `offbox.go:1388` runs `forget --prune` **grouped by host,tags** — a per-tag wipe is precisely the shape a faulty retention or a targeted deletion produces. **This is deliberate, not accidental:** the constant's own comment argues the insensitivity, and *"a detector that cries wolf is switched off within a fortnight"* is a lesson this project paid for. **So this row is NOT a demand to lower the threshold.** It is a demand that the blind spot be written where the operator reads it, because "an unexplained fall is noticed within a day" (R-431, and `STATUS.md`) is true only of falls above half. **Discovered by arithmetic while planning the drill's Phase 2/3 pairing, which could not have worked:** Phase 2 deletes one app and Phase 3 expects the alarm to fire. **LIMITATION NOW WRITTEN INTO THE ALARM'S OWN DOCUMENTATION, hub v0.111.1** — the comment above `snapshotDropFraction` in `hub/internal/monitor/offsite.go` now states what the detector does NOT see, with the demo-hp arithmetic and the `forget --prune` grouping that makes the blind spot sit on the most likely single-app failure. **It also says explicitly that the numbers must NOT be lowered to "fix" this** and that per-app detection needs a SECOND signal keyed on the per-tag count. **The row stays OPEN because documenting a blind spot is not covering it** — and because `STATUS.md` and R-431 both still say "noticed within a day", which is true only of falls above half. | **OPEN — documented in code v0.111.1; the coverage gap itself is unclosed** | | **R-436** | **LEAD, NOT A DEFECT — append-only may be reachable without a new machine, which would make R-95's real prevention far cheaper than the spike concluded.** Hetzner's port-23 restricted shell advertises, in its own `help`, these server-side backends: `borg`, `rsync`, `scp`, `sftp`, **`rclone serve restic --stdio`**. And restic 0.14.0 **recognises the `rclone:` backend** — MEASURED 2026-09-01 with a control: `banana:` → `Fatal: parsing repository location failed: invalid backend`, while `rclone:` → `exec: "rclone": executable file not found in $PATH` (i.e. the backend parsed and it tried to run the helper). rclone is **not** in the controller image today. `rclone serve restic` carries an **`--append-only`** flag. **Why this matters:** the R-95 spike's option 3 was priced at a new always-on service in the recovery path plus either a mount in the hot path or migrating every customer's history — and it was deferred on that price. This route needs neither: the server side already runs at the provider. **THE CAVEAT, STATED FIRST because it may kill the idea:** the **client** supplies the server command line, so a compromised guest could simply omit `--append-only` unless the provider pins it. **NOT ESTABLISHED:** whether Hetzner pins the flag or accepts client-supplied arguments. **That is a vendor question and it is cheap** — it should be asked before any code is written, because if the answer is "client-supplied" this lead is worth nothing. **STRENGTHENED 2026-09-01 (operator supplied the page): the backend is OFFICIALLY DOCUMENTED, not merely advertised in a shell banner.** `docs.hetzner.com/storage/storage-box/access/access-ssh-rsync-borg/#restic` reads: *"Restic is natively supported with the SFTP backend. As another option, we support the restic backend, which is provided by Rclone over SSH."* So the transport exists as a supported product feature and the client half is already proven (restic 0.14.0 parses `rclone:`, measured with a control). **AND THE SAME PAGE SETTLES THAT THE DOCS CANNOT ANSWER THE CAVEAT: neither its Rclone nor its Restic section mentions append-only at all.** That is worth stating because it closes the cheapest alternative to asking — nobody need re-read the documentation hoping for it. **Corroboration, unlooked for:** that page's table of port-23 commands matches, item for item, the `help` output measured live on our own sub-account — independent confirmation that the live measurement was reading the right product's surface. **-- ANSWERED BY HETZNER 2026-09-22 (ticket #2026090103040671), and the lead is GOOD.** Asked directly whether `--append-only` is enforced by Hetzner or taken from what the client sends, support replied: *"You can use --append-only and it would look like this: `command="rclone serve restic --stdio --append-only path/to/repo" `"*, pointing at `fluix.one/blog/hetzner-restic-append-only/`. **The mechanism is an SSH FORCED COMMAND pinned to the key in the Storage Box's `authorized_keys`** - so the flag is enforced on the server side, by a line WE write, and a client asking for a plain `rclone serve restic --stdio` on that key gets the forced one instead. **That is real prevention without a new machine, which is what this lead claimed and could not confirm.** **What it does NOT say, and must not be read as saying:** a key WITHOUT that prefix still gets a deleting server, so this protects a repository only if every key that can reach it carries the forced command - including any key an operator adds later. **NOTHING IS MEASURED YET.** Owed, and cheap: a forced-command key on `u629488`, a restic backup through it that SUCCEEDS, and a `forget --prune` through it that is REFUSED - with the refusal quoted. Until that runs this is a support answer, not a property of our backups. | **OPEN — ask the vendor before building anything** | | **R-437** | **The register compression sweep is OWED, and it was deliberately NOT run inside the 2026-09-01 beta-line session — this row is the record of that choice, not a note.** **The ask:** compress what has closed in `OPEN-ITEMS.md`. **The measurement, taken before deciding:** 181 rows, 316 KB of row text, of which **12 rows / ~25 KB (about 7 %) carry a CLOSED/DECIDED/ANSWERED leading verdict.** So the sweep buys little and touches everything. **Why it was refused as a side-task, and the citation matters:** a compression sweep is the exact operation that has already gone wrong here. The 2026-08-22 sweep (`ef6ac6f`, R-376..R-378) matched a status word ANYWHERE in the row, moved rows that were not closed, and **R-378 caught six in the same session and missed a seventh** — R-87 sat in the wrong register for nine days while the ranking paragraph pointed at nothing (R-405). **That is a session-scale hazard, and running it as the tail end of a session about something else is how it happened the first time.** **WHAT IS OWED, scoped so it can be picked up cold:** (1) classify by the **LEADING VERDICT** of the state cell only — the rule `closed_register_gate.py` already implements and red-proofs, never a whole-row match; (2) move, never rewrite — a compressed row that loses its evidence is worse than a long one; (3) run `closed_register_gate.py` before and after and quote both; (4) re-read the ranking paragraph afterwards, because that is the surface that silently went stale last time. **Not urgent:** the file is 688 lines and every gate reads it in well under a second. | **OPEN — owed; needs its own session, not a tail end** | -| **R-440** | **[P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible.** `compose pull` on a moving tag fetches whatever upstream published that day. **MEASURED 2026-09-01 over `app-catalog-felhom.eu` @ `29edad9c5bf4`: 79 `image:` lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version.** `postgres:16-alpine` (8 apps), `redis:7-alpine` (6), `mariadb:11.6` (2), plus one each of `postgres:15-alpine`, `postgis/postgis:16-3.5-alpine`, `mariadb:11.4`, `mariadb:12.3`, `ghcr.io/claperco/claper:2.5`, `ghcr.io/thomiceli/opengist:1.13`, `wger/server:2.6`. **A 24th is arguable and is recorded rather than rounded away:** `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. **Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists**, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. **MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control.** Running digests on demo-hp compared against what the registry serves for the same tag today: **`mariadb:11.4` MOVED** (`sha256:4f1d8d20...` -> `sha256:611a2fcc...`) and **`mariadb:12.3` MOVED** (`sha256:a02fe89c...` -> `sha256:dd9b303a...`), while `postgres:16-alpine`, `redis:7-alpine`, `mariadb:11.6` and `opengist:1.13` were SAME — **and both fully-pinned CONTROLS (`rommapp/romm:5.0.0`, `privatebin/pdo:2.0.5`) were SAME.** So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under `romm` and `bookstack`, with no catalog change and no record. **Compounding fact found while reading:** the recovery unit records `ImagePins` but the manifest comment says *"image NOT stored - re-pulled on restore"*, so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. **HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): `app.yaml.installed_images` now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade.** What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). **The row therefore stays OPEN and its rank is unchanged** — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. `audits/SPIKE-app-update-2026-09-01.md` | **OPEN — rank P2-MEDIUM; owner: CC** | +| **R-440** | **[P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible.** `compose pull` on a moving tag fetches whatever upstream published that day. **MEASURED 2026-09-01 over `app-catalog-felhom.eu` @ `29edad9c5bf4`: 79 `image:` lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version.** `postgres:16-alpine` (8 apps), `redis:7-alpine` (6), `mariadb:11.6` (2), plus one each of `postgres:15-alpine`, `postgis/postgis:16-3.5-alpine`, `mariadb:11.4`, `mariadb:12.3`, `ghcr.io/claperco/claper:2.5`, `ghcr.io/thomiceli/opengist:1.13`, `wger/server:2.6`. **A 24th is arguable and is recorded rather than rounded away:** `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. **Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists**, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. **MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control.** Running digests on demo-hp compared against what the registry serves for the same tag today: **`mariadb:11.4` MOVED** (`sha256:4f1d8d20...` -> `sha256:611a2fcc...`) and **`mariadb:12.3` MOVED** (`sha256:a02fe89c...` -> `sha256:dd9b303a...`), while `postgres:16-alpine`, `redis:7-alpine`, `mariadb:11.6` and `opengist:1.13` were SAME — **and both fully-pinned CONTROLS (`rommapp/romm:5.0.0`, `privatebin/pdo:2.0.5`) were SAME.** So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under `romm` and `bookstack`, with no catalog change and no record. **Compounding fact found while reading:** the recovery unit records `ImagePins` but the manifest comment says *"image NOT stored - re-pulled on restore"*, so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. **HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): `app.yaml.installed_images` now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade.** What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). **The row therefore stays OPEN and its rank is unchanged** — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. `audits/SPIKE-app-update-2026-09-01.md` **— NIGHT 2026-09-23:** every image moved tonight (and the 21 backfilled moves) carries its resolved digest in the catalog's test record; the floating pins themselves still float until the box pulls by digest (`09` §6.4 part 6). | **OPEN — rank P2-MEDIUM; owner: CC** | | **R-444** | **[P3-LOW] Nothing runs `pct fstrim` on the fleet, and demo-hp's thin pool was carrying ~23.8 GB of blocks the guest had already freed.** MEASURED 2026-09-01 during this spike's teardown: the run itself added ~1.05 GiB that `local-lvm` did not reclaim on delete (68.97% -> 70.91%); `fstrim` INSIDE the unprivileged container is refused (`FITRIM ioctl failed: Operation not permitted`, all three mounts); `pct fstrim 9201` from the PVE host then trimmed **30.2 GiB + 57 GiB** and took `local-lvm` to **26.78%** — **23.8 GB BELOW this run's own starting point**, i.e. the surplus was long-standing, not ours. **Why it is not merely housekeeping:** a thin pool that only ever grows can reach 100% from DELETED data alone, and a full thin pool takes every guest on the host read-only. demo-hp had 16.4 GB free before the trim. **Not urgent, and the row says so** — but the appliance has no periodic trim and no operator surface reports the gap between guest-free and pool-used. Owner: **CC.** `audits/SPIKE-app-update-2026-09-01.md` | **OPEN — rank P3-LOW; owner: CC** | | **R-445** | **[P3-LOW] Hub app telemetry survives the app's removal, so a 15-minute throwaway now sets a FLEET-WIDE memory recommendation.** MEASURED 2026-09-01: this spike's Phase 6 Nextcloud existed for ~15 minutes on demo-hp, spent part of it crash-looping, and was then removed with all volumes. The hub's `/apps/nextcloud` page still reports `Deployments`, `Avg Memory 208 MB`, `P95 Memory 280 MB` and **`Suggested Limit (P95x1.2) = 352 MB`**, plus three MariaDB `io_uring` rows under Known Issues attributed to demo-hp. **The suggested limit is an operator-facing recommendation derived from a sample that no longer exists anywhere** — and Nextcloud is a real catalog app whose limit someone may act on. **RETAINED DELIBERATELY BY THIS RUN, NOT CLEARED, and the reason is part of the row:** the hub offers `POST /apps/nextcloud/reset-telemetry` whose own confirm reads *"Delete all telemetry data for nextcloud? This cannot be undone."* — an irreversible write on the operator's surface, and the operator authorised Phase 6, not this. **The one-line command is recorded in the audit doc so it is a decision, not a task.** The general question is the row: should telemetry for an app with zero live deployments age out, or be excluded from the suggestion? Owner: **VIKTOR rules, CC implements.** `audits/SPIKE-app-update-2026-09-01.md` | **OPEN — rank P3-LOW; owner: VIKTOR rules, CC implements** | -| **R-446** | **[P2-MEDIUM] „Naprakész" can be FALSE, and the badge that says it cannot tell.** Slice 2 (controller v0.233.0, 2026-09-02) compares the RECORDED image reference per compose service against the reference the current template pins, and **queries no registry** — deliberately: a customer's box must not depend on reaching eight upstream registries to render a page (`felhom-controller/controller/internal/web/updatebadge.go`, `compareInstalledToTemplate`). **For the 23 floating pins that comparison is blind by construction:** `postgres:16-alpine`, `mariadb:11.6` and 21 others can carry an identical reference over an image that has moved. **MEASURED, not theorised — spike §5 found `mariadb:11.4` and `mariadb:12.3` had BOTH already moved upstream while two fully-pinned CONTROLS held.** So `romm` and `bookstack` on demo-hp would read „Naprakész" over a database engine build that is not the one the catalog now resolves to. **This is a KNOWN LIMITATION OF A SHIPPED FEATURE, filed the same session rather than left implicit**, and it is stated in the same words in `architecture/09-update-architecture.md` §8.1 and in the controller's `README.md`. The close is a digest comparison against the registry, which needs a network call, a cache and a failure posture — it is not a one-liner and it is not slice 2's job. **Depends on R-440**, whose fix (stop floating) would remove the problem instead of measuring it — take that route first if it is available. `architecture/09-update-architecture.md` **MEASURED 2026-09-21, and the blind spot is not one or two pins.** `audits/UPDATE-ARC-STATE-2026-09-21.md` §3.3: the catalog carries **10 floating pins of 66** (recounted — the old "23" was stale), and **6 of the 7 measurable engine pins have been repushed upstream since the catalog set them** — `postgres:16-alpine` (8 apps), `postgres:15-alpine`, `redis:7-alpine` (6 apps), `mariadb:11.4`, `mariadb:12.3`, `postgis:16-3.5-alpine`; only `mariadb:11.6` has not. The 8th (immich's own ghcr build) is UNMEASURED — ghcr exposes no anonymous last-modified timestamp. **So on demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.** The fix does NOT need the box to query a registry: the catalog can record each pin's digest at push time (`check-image-resolvable.py` already resolves it) and the box compares digests. Put to the operator as `09` §3b **Q6**, recommended YES — the cheapest real improvement on the arc's list. **— UPDATE NIGHT 2026-09-21:** **MEASURED ON A BOX 2026-09-21 (update night, leg B8), and it REFINES the row in two ways rather than merely confirming it.** §8.1's numbers came from a registry sweep on DooPlex; this is the same question asked of a customer-shaped box, where the badge actually renders. On guest 9202, `docmost`'s two floating pins were read as `installed_images` records them and compared against the upstream digests measured the same night: `postgres:16-alpine` → **`sha256:721873c34ceb9…` on the box and `sha256:721873c34ceb9…` upstream**, and `redis:7-alpine` → **`sha256:858f009f9709c…` both sides**. **Identical. So the badge „Naprakész" is TRUE for this box**, and the app reads correctly. **(1) The defect's size is set by INSTALL AGE, not by the catalog.** A floating pin is wrong only for a box that pulled BEFORE the tag moved; a box deployed after the repush holds the current image and its badge is right. R-446's "six repushed pins" measured the tag against the date the CATALOG set it, which is the right measure for *the catalog* and not for *a box*. **(2) The producer Q6 needs ALREADY EXISTS on the box.** `installed_images` records a real `digest` per service (`installed.go` §7.1) — the box knows exactly what it is running. What it cannot do is COMPARE, because the catalog carries no digest to compare against. That is Q6's proposal, and this is a concrete confirmation that only the catalog half is missing. Evidence: `audits/update-night-2026-09-21/23-B8-floating-pin.txt`. **-- RULED 2026-09-23 (`09` §3 decision 17):** YES — the catalog records the image digest of every pin at push time; the box compares against it and, where the catalog carries one, pulls **that exact image**, which makes a floating tag reproducible, not only the badge honest. *Pull-by-digest while the definition names a tag is a claim to verify in the build, not a ruling on mechanism.* **-- 2026-09-23:** pull-by-digest MEASURED on 9202 — Docker and Compose both pull and run `redis:7-alpine@sha256:858f…` and refuse a digest that does not exist. **Build trap, read from source:** `splitImageRef` returns "unorderable" for any ref containing `@` (`stacks/updateorder.go:134`), so a digest-carrying pin must have its digest split off before ordering or every such app reads Unknown. `09` §6.4 part 6. | **READY TO BUILD — owner: CC; `09` §6.4** | -| **R-450** | **[P2-MEDIUM] UPDATE ARC SLICE 6 — a version sequence: automatic WITHIN a major, never ACROSS one, and an engine change gets its OWN edge.** The first half is an operator ruling of 2026-09-02 and its justification is R-449's measurement: a cross-major jump can be refused by the app itself and cannot be undone. **The second half is a rule recorded now, while it is cheap:** an engine change must never be bundled with an app version bump. `bookstack`'s `0b73e5e` moved the application 25.02.2 → 26.05.2 **and** MariaDB 11.6 → 12.3 in one commit — **two migrations behind one edge**, and an unreadable failure when it breaks. Needs a catalog-side convention and, eventually, a gate. `architecture/09-update-architecture.md` §6 **HALF SHIPPED 2026-09-21 (catalog `5ff36d098cbc`): the second half — an engine change gets its OWN edge — is now ENFORCED** by `check-engine-major.py`, which refuses a commit moving a MariaDB major together with any other image move in that template, naming what it was bundled with. The FIRST half (automatic within a major) is Slice 6 and needs four operator answers — `09` §3b **Q1–Q4**, with the shape it would take in `09` §6.2. **The urgency is now measured:** 46 of the catalog's 58 exact pins are behind upstream and **39 of those are within a major** — the population the 2026-09-02 ruling already says may move without a human. **-- RULED 2026-09-23 (operator, `09` §3 decisions 11–15):** Q1–Q4 answered. The update is a leg of the backup chain after off-site and before the full-system backup (11); automatic with a per-box switch ON by default (12); **the TEST decides, not the tag** — the box applies every step the catalog holds because the catalog holds only tested steps, and `CompareImageRefs` moves to the catalog gate (13, REPLACES "never across a major"); a box behind climbs **one tested step at a time** (14); **the box UNDOES a failed update itself** — old definition + the pre-pin safety dump + health check again, HOLD only if the undo fails (15, REPLACES §6.1's no-auto-undo). The undo and the ladder were SPIKED the same day before any build (`audits/update-rulings-2026-09-23/`); build order and costs in `09` §6.4. **-- SPIKED 2026-09-23 (`audits/update-rulings-2026-09-23/`):** the undo works by hand on three real migrating edges and needs eight product additions (R-637..R-642); **the ladder is measured absent** — one press on a box two steps behind jumped vikunja 2.3.0 → 2.5.0 in 9.5 s and 2.4.0 never ran, and the box cannot see intermediate steps at all because its catalog clone is `--depth 1` (`sync.go:283`/`:300`, one commit visible on both demo guests). The ladder's recommended format is an `update_ladder:` list in `.felhom.yml` with each intermediate step's own definition, NOT the git history (romm's image-moving commit is the definition that OOM-looped). The chain's update leg has ≤15 min as ruled (R-643). Build order `09` §6.4. | **READY TO BUILD — owner: CC; `09` §6.4 part by part, each part returns to the operator for go/no-go** | +| **R-446** | **[P2-MEDIUM] „Naprakész" can be FALSE, and the badge that says it cannot tell.** Slice 2 (controller v0.233.0, 2026-09-02) compares the RECORDED image reference per compose service against the reference the current template pins, and **queries no registry** — deliberately: a customer's box must not depend on reaching eight upstream registries to render a page (`felhom-controller/controller/internal/web/updatebadge.go`, `compareInstalledToTemplate`). **For the 23 floating pins that comparison is blind by construction:** `postgres:16-alpine`, `mariadb:11.6` and 21 others can carry an identical reference over an image that has moved. **MEASURED, not theorised — spike §5 found `mariadb:11.4` and `mariadb:12.3` had BOTH already moved upstream while two fully-pinned CONTROLS held.** So `romm` and `bookstack` on demo-hp would read „Naprakész" over a database engine build that is not the one the catalog now resolves to. **This is a KNOWN LIMITATION OF A SHIPPED FEATURE, filed the same session rather than left implicit**, and it is stated in the same words in `architecture/09-update-architecture.md` §8.1 and in the controller's `README.md`. The close is a digest comparison against the registry, which needs a network call, a cache and a failure posture — it is not a one-liner and it is not slice 2's job. **Depends on R-440**, whose fix (stop floating) would remove the problem instead of measuring it — take that route first if it is available. `architecture/09-update-architecture.md` **MEASURED 2026-09-21, and the blind spot is not one or two pins.** `audits/UPDATE-ARC-STATE-2026-09-21.md` §3.3: the catalog carries **10 floating pins of 66** (recounted — the old "23" was stale), and **6 of the 7 measurable engine pins have been repushed upstream since the catalog set them** — `postgres:16-alpine` (8 apps), `postgres:15-alpine`, `redis:7-alpine` (6 apps), `mariadb:11.4`, `mariadb:12.3`, `postgis:16-3.5-alpine`; only `mariadb:11.6` has not. The 8th (immich's own ghcr build) is UNMEASURED — ghcr exposes no anonymous last-modified timestamp. **So on demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.** The fix does NOT need the box to query a registry: the catalog can record each pin's digest at push time (`check-image-resolvable.py` already resolves it) and the box compares digests. Put to the operator as `09` §3b **Q6**, recommended YES — the cheapest real improvement on the arc's list. **— UPDATE NIGHT 2026-09-21:** **MEASURED ON A BOX 2026-09-21 (update night, leg B8), and it REFINES the row in two ways rather than merely confirming it.** §8.1's numbers came from a registry sweep on DooPlex; this is the same question asked of a customer-shaped box, where the badge actually renders. On guest 9202, `docmost`'s two floating pins were read as `installed_images` records them and compared against the upstream digests measured the same night: `postgres:16-alpine` → **`sha256:721873c34ceb9…` on the box and `sha256:721873c34ceb9…` upstream**, and `redis:7-alpine` → **`sha256:858f009f9709c…` both sides**. **Identical. So the badge „Naprakész" is TRUE for this box**, and the app reads correctly. **(1) The defect's size is set by INSTALL AGE, not by the catalog.** A floating pin is wrong only for a box that pulled BEFORE the tag moved; a box deployed after the repush holds the current image and its badge is right. R-446's "six repushed pins" measured the tag against the date the CATALOG set it, which is the right measure for *the catalog* and not for *a box*. **(2) The producer Q6 needs ALREADY EXISTS on the box.** `installed_images` records a real `digest` per service (`installed.go` §7.1) — the box knows exactly what it is running. What it cannot do is COMPARE, because the catalog carries no digest to compare against. That is Q6's proposal, and this is a concrete confirmation that only the catalog half is missing. Evidence: `audits/update-night-2026-09-21/23-B8-floating-pin.txt`. **-- RULED 2026-09-23 (`09` §3 decision 17):** YES — the catalog records the image digest of every pin at push time; the box compares against it and, where the catalog carries one, pulls **that exact image**, which makes a floating tag reproducible, not only the badge honest. *Pull-by-digest while the definition names a tag is a claim to verify in the build, not a ruling on mechanism.* **-- 2026-09-23:** pull-by-digest MEASURED on 9202 — Docker and Compose both pull and run `redis:7-alpine@sha256:858f…` and refuse a digest that does not exist. **Build trap, read from source:** `splitImageRef` returns "unorderable" for any ref containing `@` (`stacks/updateorder.go:134`), so a digest-carrying pin must have its digest split off before ordering or every such app reads Unknown. `09` §6.4 part 6. **— NIGHT 2026-09-23:** the CATALOG half of the close shipped: every ladder entry records the digest the registry served for each `to` ref (`scripts/image_digest.py`), and the move gate refuses a digest the registry no longer serves. The box does not compare it yet (`09` §6.4 part 6, box half). | **READY TO BUILD — owner: CC; `09` §6.4** | +| **R-450** | **[P2-MEDIUM] UPDATE ARC SLICE 6 — a version sequence: automatic WITHIN a major, never ACROSS one, and an engine change gets its OWN edge.** The first half is an operator ruling of 2026-09-02 and its justification is R-449's measurement: a cross-major jump can be refused by the app itself and cannot be undone. **The second half is a rule recorded now, while it is cheap:** an engine change must never be bundled with an app version bump. `bookstack`'s `0b73e5e` moved the application 25.02.2 → 26.05.2 **and** MariaDB 11.6 → 12.3 in one commit — **two migrations behind one edge**, and an unreadable failure when it breaks. Needs a catalog-side convention and, eventually, a gate. `architecture/09-update-architecture.md` §6 **HALF SHIPPED 2026-09-21 (catalog `5ff36d098cbc`): the second half — an engine change gets its OWN edge — is now ENFORCED** by `check-engine-major.py`, which refuses a commit moving a MariaDB major together with any other image move in that template, naming what it was bundled with. The FIRST half (automatic within a major) is Slice 6 and needs four operator answers — `09` §3b **Q1–Q4**, with the shape it would take in `09` §6.2. **The urgency is now measured:** 46 of the catalog's 58 exact pins are behind upstream and **39 of those are within a major** — the population the 2026-09-02 ruling already says may move without a human. **-- RULED 2026-09-23 (operator, `09` §3 decisions 11–15):** Q1–Q4 answered. The update is a leg of the backup chain after off-site and before the full-system backup (11); automatic with a per-box switch ON by default (12); **the TEST decides, not the tag** — the box applies every step the catalog holds because the catalog holds only tested steps, and `CompareImageRefs` moves to the catalog gate (13, REPLACES "never across a major"); a box behind climbs **one tested step at a time** (14); **the box UNDOES a failed update itself** — old definition + the pre-pin safety dump + health check again, HOLD only if the undo fails (15, REPLACES §6.1's no-auto-undo). The undo and the ladder were SPIKED the same day before any build (`audits/update-rulings-2026-09-23/`); build order and costs in `09` §6.4. **-- SPIKED 2026-09-23 (`audits/update-rulings-2026-09-23/`):** the undo works by hand on three real migrating edges and needs eight product additions (R-637..R-642); **the ladder is measured absent** — one press on a box two steps behind jumped vikunja 2.3.0 → 2.5.0 in 9.5 s and 2.4.0 never ran, and the box cannot see intermediate steps at all because its catalog clone is `--depth 1` (`sync.go:283`/`:300`, one commit visible on both demo guests). The ladder's recommended format is an `update_ladder:` list in `.felhom.yml` with each intermediate step's own definition, NOT the git history (romm's image-moving commit is the definition that OOM-looped). The chain's update leg has ≤15 min as ruled (R-643). Build order `09` §6.4. **— NIGHT 2026-09-23: `09` §6.4 part 4 SHIPPED** (catalog `6db08a5`): the test record `update_ladder:` + two gates + the only writer + the 21-move backfill; 12 more steps published with records. Parts 5 (the box climbs), 6-box-half and 7 remain; the romm press on demo-hp showed today's jump live — 5.3.0 → 5.3.1 AND mariadb 11.4 → 11.8 in one press (both tested steps; `done`). | **READY TO BUILD — owner: CC; `09` §6.4 part by part, each part returns to the operator for go/no-go** | | **R-451** | **[P3-LOW] UPDATE ARC SLICE 7 — a fleet sweep: the operator can SEE, and MOVE, how far behind every box is.** Slices 1 and 2 make one box's state visible on that box's own pages. The operator has no fleet view, and **it is not derivable from what is already reported: the hub's report payload carries container name, state, CPU and memory, and NO image field at all** (spike §5, which is why Peti's box could only be recorded UNKNOWN). So this is a hub-side change as well as a controller one. Rank LOW today because the fleet is two enrolled boxes; it rises with the fleet. `architecture/09-update-architecture.md` §6, §8.4 **BOTH SIDES VERIFIED 2026-09-21, and it is cheaper than this row implies.** The controller's payload carries no image (`internal/report/types.go` L98–103) and the hub's `Store.SaveReport` (`hub/internal/store/store.go:965`) denormalises only container **counts** — but **the hub stores the raw report JSON whole**, so a new controller field lands there the day it is sent. What is missing is the denormalisation and the page, not the transport. Shape in `09` §6.2–6.3; the payload question is `09` §3b **Q7**. **-- RULED 2026-09-23 (`09` §3 decision 18):** the report carries, per compose service (database included), the installed reference, the catalog reference and the badge state. **Built later, when the fleet grows** — Q7's recommendation, confirmed. | **RULED — build deferred until the fleet grows; owner: CC** | | **R-454** | **[P3-LOW] Five `internal/web` test files have been `gofmt`-unclean for an unknown length of time, and nothing notices.** MEASURED 2026-09-02: `gofmt -l controller/internal/web/` reports `backups_split_test.go`, `claim_code_naming_test.go`, `disk_health_test.go`, `r400_debug_routes_test.go`, `recovery_test.go` — at the **baseline** commit `960d29b0612c`, i.e. not introduced by v0.233.0 (both files added that day are clean). **`go vet` does not check formatting and `controller_gates.py` has no formatting gate**, so the only thing that would ever surface this is someone running `gofmt -l` by hand, which is how it was found. **Not reformatted in the same session, deliberately** — the minimal-changes rule, and a five-file whitespace commit inside a feature release makes that release's diff unreadable. **Small, and the cost of NOT having the instrument is the row:** the count can only grow, and every future `gofmt -l` run produces noise that hides a real one. Fix is two lines: a `gofmt -l` gate in `controller_gates.py` plus one formatting commit, in that order (the gate first, so the commit is provably complete). Owner: **CC.** | **READY — rank P3-LOW; owner: CC** | | **R-457** | **[P3-LOW] A test that hardcodes a date AND asserts an age derived from it is green on the day it is written and red the next morning — one instance PROVEN, six candidate files named.** MEASURED 2026-09-03: `TestGroupD_BadgeRendersOnBothSurfaces` (shipped the previous day in v0.233.0) pinned a fixture `catalog_since: "2026-07-18"` and asserted the rendered string `"Frissítés elérhető — 46 napja"`. **The pure badge tests inject a clock; the RENDER test does not and cannot** — it goes through the production templates, which call the funcmap entry `updateBadge`, which reads `time.Now()`. The suite was green on 2026-09-02 and **FAILED on 2026-09-03** with *"the behind badge is missing"* on both surfaces, because the true answer had become 47. **Fixed by DERIVING the fixture** — `catalog_since` is computed as *today minus 46 days*, so the test asserts the real number through the real clock and cannot rot. **THE CLASS, which is why this is a row and not just a fix:** a clock-reading test that also carries a date LITERAL is a bomb with a fuse of unknown length, and the suite being green is not evidence it is defused — it is evidence the fuse has not burned down yet. **NAMED AS UNCHECKED CANDIDATES, NOT ACCUSED** — six other test files contain both a `20xx-xx-xx` literal and `time.Now()`: `internal/backup/offbox_test.go`, `internal/web/handler_export_upload_test.go`, `internal/web/r103_tier2_action_test.go`, `internal/web/dashboard_backup_card_test.go`, `internal/web/async_restore_test.go`, `internal/stacks/installed_test.go`. Mixing the two is not itself a defect — it is one only where a literal feeds an assertion evaluated against the real clock — so each needs reading, which is a sweep and not this session. **The instrument that would end the class:** run the suite once under a faked future date in CI and see what turns red. Owner: **CC.** `felhom-controller` v0.234.0 CHANGELOG | **READY — rank P3-LOW; owner: CC** | | **R-458** | **[P3-LOW] `.felhom.yml` keeps flowing to an app whose compose file is FROZEN, so a frozen app can receive a health check written for a version it is not running.** The v0.235.0 render freezes `docker-compose.yml` for a pinned app once the catalog moves past its version, but copies `.felhom.yml` **verbatim in every case** (`Syncer.copyTemplates`). **The asymmetry is deliberate and both directions were considered:** `.felhom.yml` carries no image, and it carries `catalog_since` — the single input the update badge uses to say *„Frissítés elérhető — N napja"* — so freezing it would silently withhold the one number that tells a customer they are behind, i.e. it would break slice 2 to protect slice 3. **What it costs:** the file also carries the controller-side `healthcheck:` block and resource hints, so a template updated for a newer version can hand a frozen app a probe written for software it is not running. **THE FAILURE DIRECTION IS A FALSE ALARM, NEVER DATA LOSS** — the app keeps running; at worst it renders as degraded and, if it persisted, could reach the dead-app alarm path. That is the same class as R-330's false e-mails, which is why this is a row and not a footnote. **Not fixed now, and the reason is that the cheap fix is wrong:** freezing the whole file breaks the badge, and freezing only the `healthcheck:` key means the syncer would have to parse and re-assemble a customer-facing metadata file — new surface on the one path that touches every app on every box every 15 minutes. **What would settle it:** whether any catalog `healthcheck:` has ever been changed in the same commit as an `image:` line (measurable from the catalog's own history, no box needed). If the answer is "never", the exposure is theoretical and the row can be closed by measurement instead of by code. Owner: **CC.** `architecture/09-update-architecture.md` §5.4, §8.5 **— UPDATE NIGHT 2026-09-21:** **MEASURED 2026-09-21 (update night), leg B9, and the row's risk is NARROWER than it states.** A `.felhom.yml`-only change (a health check for a path only a newer version would serve) was pushed to a FROZEN `bentopdf` — installed `v2.8.6`, catalog ahead. §5.4's asymmetry is confirmed live: the new `.felhom.yml` reached the box while the compose `image:` line stayed `v2.8.6`. **But no false alarm was produced**: ten samples over two minutes all read `state=running` with the front door at `200`. The reason is the probe's own semantics, not luck — `healthprobe.go:258-261` treats **any response** as healthy for `type: http`, and the bogus path answers 404, which is a response. **So this row's false-alarm risk exists only for `type: api` probes carrying an `expect` block**, where the status is compared; for every `type: http` template and every `type: api` without `expect`, a newer version's path is invisible to the probe. The row's actual claim — the failure direction is a false alarm, never data loss — stands and is now measured. Evidence: `audits/update-night-2026-09-21/21-B9-frozen-app-newer-felhomyml.md`. | **READY — rank P3-LOW; owner: CC** | | **R-460** | **[P3-LOW] BookStack's FILE half cannot be seeded or verified without a browser, so its upgrades can only ever be auto-proven for the DATABASE.** MEASURED 2026-09-06 while building the R-449 harness. BookStack's API needs a token that is only mintable through its web UI, and its HTTP login is unusable headlessly for a second, independent reason: `APP_URL` comes from the template as `https://${SUBDOMAIN}.${DOMAIN}`, so the app marks its session and XSRF cookies **`secure`**; curl over plain http stores neither and **every login POST returns 419 Page Expired**, which looks exactly like a wrong password. The container serves no TLS. **The database half IS provable** — the harness seeds with `php artisan bookstack:create-admin` and reads back with a DIFFERENT artisan command that must find the record, carrying its own negative control on every call. **What is unprovable is an uploaded image or attachment**, i.e. exactly the half a customer would notice. **THIS IS A FACT ABOUT THE APP, NOT A DEFECT IN THE HARNESS**, and it is recorded because Slice 6 needs to know which apps can be auto-verified and which can only be partly verified — nobody had that list before. **Deliberately NOT worked around:** planting a file in the volume would make the test pass while proving nothing, which is R-156's exact failure. **What would remove it:** a headless token route (upstream), or accepting a browser-driven step for this app alone, which DooPlex cannot run. Owner: **CC.** `audits/SPIKE-upgrade-test-2026-09-06.md` §6 **-- UPDATE NIGHT 2026-09-21:** bookstack's edge was walked again on 2026-09-21 and is again **half-proven**: the database half read back through `php artisan` with its own negative control, the file half untouched. The limitation is unchanged and is now measured on the box as well as on the harness. Two more apps joined the same class tonight for a different reason (R-624). | **READY — rank P3-LOW; owner: CC** | -| **R-462** | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. | **READY — rank P2-MEDIUM; owner: CC (scope already ruled, `09` §3 decision 6)** | +| **R-462** | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). | **READY — rank P2-MEDIUM; owner: CC (scope already ruled, `09` §3 decision 6)** | | **R-463** | **[P2-MEDIUM] The day the catalog moves `postgres:16` to `17`, ELEVEN apps are affected and the container image will NOT perform the conversion — and nothing anywhere records that.** MEASURED 2026-09-06: 11 of the 53 templates carry PostgreSQL — **8 on `postgres:16-alpine`**, 1 on `postgres:15-alpine`, plus `postgis/postgis:16-3.5-alpine` and Immich's own `postgres:16-vectorchord…` build. **A grep of the whole register for `pg_upgrade`, "postgres major" or "postgresql major" returns ZERO** (confirmed this session, and confirmed again before filing). **WHY IT IS NOT THE SAME PROBLEM AS R-459, and this is the point of the row: the two engines fail in OPPOSITE directions.** MariaDB starts anyway and skips the conversion quietly, which is why R-459 went unnoticed until a harness looked. **PostgreSQL REFUSES TO START on a datadir from an older major** — the official image performs no `pg_upgrade` and exits with a message naming both versions. So the Postgres case cannot hide; it will present as eight apps down at once, on the sync after the catalog moves. **DELIBERATELY NOT MEASURED HERE, and saying so is the scope discipline:** R-459's task was scoped to MariaDB, and measuring the Postgres analogue is its own piece of work with its own venue. **This row exists so the gap is a record rather than a sentence in an audit nobody greps.** What would settle it: one edge on the existing harness (`postgres:16-alpine` → `17-alpine`) on a scratch host, which would also exercise the `engine_state_after` field's Postgres probe end to end — it is written but has never run against a real Postgres major. Owner: **CC.** `audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md` §7 **-- UPDATE NIGHT 2026-09-21:** **Measured 2026-09-21, both halves.** (a) What a household sees TODAY: the guarded Update of `postgres:16-alpine` to `17-alpine` on docmost ended **`failed` in 5.1 s**, the app stopped and held, **the pin naming 17 while `installed_images` still said 16 and nothing ran**, the data intact, and the restore the hold sentence names back in **29.1 s**. The engine's refusal line had to be REPRODUCED independently because `failAndHold` destroyed it (R-621): *FATAL: database files are incompatible with server / DETAIL: The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 17.11.* The datadir was still `16` afterwards, and the same copy started under 16 holding 48 tables as the positive control. (b) The conversion **COSTED** on a real seeded 49 MB / 48-table datadir: `pg_dumpall` **2.6 s / 132 201 B**, fresh 17 plus replay **6.5 s / 48 tables restored**, the app up on 17 saying *Database connection successful*, **the seeded account read back**, total **155.9 s of which ~9 s is engine work**. `pg_upgrade` was NOT run: it needs both majors' binaries in one image and no such image exists here. Full paragraph: `audits/update-night-2026-09-21/24-Q5-postgres-conversion-costed.md`. **-- RULED 2026-09-23 (`09` §3 decision 16):** PostgreSQL majors are converted BY THE BOX as a guarded-update step — save everything from the old engine, start the new one empty, load it back, check. Each of the eleven apps is proven on the test bench before the catalog may move it; the engine-major gate stays until then. | **READY TO BUILD — owner: CC; `09` §6.4; the gate stays until all eleven are proven** | | **R-464** | **[P3-LOW] MariaDB's entrypoint prints `MariaDB upgrade not required` on an UNSUPPORTED DOWNGRADE, so that line cannot be used as a soundness signal.** MEASURED 2026-09-06. After converting a datadir to `12.3.3-MariaDB` and then starting **11.6** on it, the entrypoint logs, on every start: **`[Note] [Entrypoint]: MariaDB upgrade not required`**. Asked properly, the same engine answers **`FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported!`** **The entrypoint compares the datadir's recorded version against its own and concludes there is nothing to DO. That is true, and it is not a statement that the state is sound.** **THIS IS THIS PROJECT'S MOST-REPEATED CLASS, in a new costume** — the same shape as `CLAUDE.md`'s "presence is not success" and as R-443's HTTP 200 over a crash-looping app: a reassuring sentence that answers a narrower question than the one a reader will take it for. **Why it is worth a row rather than a footnote: the obvious cheap instrument for R-459 is to grep container logs for that exact line**, and such an instrument would report "fine" for an unsupported downgrade. **The correct probe is `mariadb-upgrade --check-if-upgrade-is-needed`**, which is what `upgrade-test.py`'s `engine_state_after` now uses. **Also recorded, because it nearly produced a wrong answer here: run without credentials that command returns `ERROR 1045 … FATAL ERROR: Upgrade failed` with exit 1** — an authentication failure wearing the shape of a verdict. Owner: **CC.** `audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md` §5.4 | **READY — rank P3-LOW; owner: CC** | | **R-468** | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** | **WATCHING — rank P3-LOW; owner: CC (renew ≤ 14 days or bake); retire at the first external install** | @@ -684,7 +684,6 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-494** | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** | | **R-497** | **[P2-MEDIUM] No product channel ever gives the customer the „Tulajdonosi jelmondat”, yet the self-bind mail says they received it at setup.** MEASURED 2026-09-14: the 5-word phrase is minted at customer creation (`hub/internal/web/configs.go`, `RandomPassphrase(5)`) and shown **only on the operator's customer page**; the self-bind mail (`FormatSelfBindEmail`) says „amelyet a beállításkor kaptál” and the console asks for it; nothing sends it. Day-0 runbook A.2 says the operator must dictate or hand it over — a step that exists only in an operator document. A volunteer whose operator forgets cannot bind the box and is told they already have the phrase. **Fix shape:** the operator's customer-create confirmation says, in one line, to hand this phrase to the customer now; the volunteer instructions say where it comes from (done in `VOLUNTEER-first-hour.md`). **CLOSED 2026-09-14 — hub v0.113.0 deployed (ArgoCD Synced, image 0.113.0). Tests red first (`passphrase_handover_test.go`); live: `tester-1`'s page renders the hand-over sentence 1× (2× with `?flash=created`), negative control 0. The mail change is pinned by test; no mail could be read live (no mailbox).** Evidence: `audits/DOORSTEP-walk-1270-2026-09-14.md` §5. | **CLOSED — hub v0.113.0** | | **R-498** | **[P3-LOW] The „Első lépések" on 52 of 53 app pages tell a customer to open a literal `wiki.DOMAIN` — the placeholder is never filled in.** MEASURED 2026-09-14 on a fresh 0.242.0 box (drill step 5): BookStack's app page renders „Nyisd meg a **wiki.DOMAIN** címet a böngészőben", PrivateBin's „Nyisd meg a **paste.DOMAIN** címet". `grep -rl '\.DOMAIN c[íi]m' app-catalog-felhom.eu/templates/*/.felhom.yml` → **52 of 53** templates carry it in `first_steps`; the controller renders the string as text (`internal/stacks/metadata.go`). A stranger reading their first instruction meets a word that is not an address. **Fix shape:** substitute the stack's real `SUBDOMAIN.DOMAIN` at render time (one place in the controller), with a render test per template that fails on a literal `DOMAIN`. | **READY — rank P3-LOW; owner: CC** | -| **R-499** | **[P2-MEDIUM] Every app without a data drive is told its data is „already in the full system backup (PBS)" and there is „nothing to do" — on a box with no PBS, whose only whole-box copy sits on the same disk.** MEASURED 2026-09-14 on a fresh 0.242.0 box with the DR tier off (drill step 7): `GET /stacks/bookstack/backup` renders „Ennek az alkalmazásnak az adatai a belső rendszerlemezen vannak, amelyek **már szerepelnek a teljes rendszermentésben (PBS)** … ehhez az alkalmazáshoz nincs külön teendő." The sentence sits under `{{if not .IsHDDApp}}` in `controller/internal/web/templates/tier2_config.html:20-26` and consults nothing about where the whole-guest backup goes. On the same box `/backups` says, correctly, „Helyi tároló (local)" and „A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer — így hibás fájlok ellen véd, lemezhiba ellen nem." **Two pages of one product contradict each other, and the reassuring one is the false one.** **Fix shape:** branch the sentence on the box's actual whole-guest target (PBS vs local, same-disk flag the overview already computes); a render test per branch. | **READY — rank P2-MEDIUM; owner: CC** | | **R-500** | **[P3-LOW] The dashboard shows the last backup in UTC while every backup page shows it in local time — two different clock times for one backup.** MEASURED 2026-09-14 on a fresh 0.242.0 box (drill step 12): the same backup (`last_run 2026-09-14T13:41:37Z`) reads „Utolsó mentés: **2026-09-14 13:41**" on `/dashboard` and „Utolsó adatbázis mentés: **2026-09-14 15:41** (most)" on `/backups/apps`. Cause, from source: `controller/internal/web/templates/dashboard.html:154` renders `{{.BackupStatus.LastRun.Format "2006-01-02 15:04"}}` with no conversion to the box's zone, while the backup pages go through the zone-aware helpers in `funcmap.go`. A household comparing the two screens sees a backup two hours apart from itself. **Fix shape:** format through the same zone-aware helper; a render test that pins a non-UTC zone and asserts the local hour. | **READY — rank P3-LOW; owner: CC** | | **R-501** | **[P3-LOW] The documented "confirm your CI run" recipe reads only the LAST page of the jobs list, and that list is not in id order — so it can report a run as missing that exists and passed.** MEASURED 2026-09-14 for felhom.eu commit `38848ff`: `actions/jobs?limit=1` → `total_count 335`; the recipe's page `T/50+1` = page 7 held ids 473…570 and **no match for 8 minutes**; a scan of all seven pages found the job on **page 6** (ids 380…581): `job id=581 name=gates status=completed conclusion=success completed_at 2026-09-14T14:18:23Z`. Pages are not sorted (page 1 ids 1…273, page 2 84…208). `actions/tasks` listed the same run first (`id 582, run_number 335, success`). **Fix shape:** the recipe in `felhom.eu/CLAUDE.md` (end-of-session checklist) scans every page and matches `head_sha`; say so in the same sentence that warns about the id offset (R-417). | **READY — rank P3-LOW; owner: CC** | | **R-502** | **[P3-LOW] The bootstrap regression harness is run by NO gate and NO CI — and it had never exercised the pairing banner.** MEASURED 2026-09-14: `grep -rn bootstrap-modes scripts/*.py .gitea/workflows` → nothing; the harness is run by hand in `felhom-iso-assistant:trixie`. While adding the R-496 checks, its fake hub's `register` reply turned out to carry no `pairing_code`, so `print_pairing_banner` returned early in every run: the banner that a household reads first had no test at all until ISO v1.27.0. **Fix shape:** register the harness in `repo_gates.py` behind a container-available check that reports INCONCLUSIVE (never skip-as-pass) when docker is absent, with a decoy. | **READY — rank P3-LOW; owner: CC** | @@ -697,7 +696,7 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-509** | **[P1-HIGH] A box installed for an EXISTING customer never gets the self-bind e-mail the console tells the volunteer to open.** MEASURED 2026-09-14 (BIGNIGHT, VM 333, ISO 1.27.1): customer `tester-1` now has `tester1@felhom.eu` registered; the box registered as appliance 28 at 17:44:53Z and its console says „Nyisd meg az e-mailben kapott linket"; **ten minutes later the mailbox (read through the Gmail connector) held 0 messages to that address.** Cause, from source: the hub auto-sends the link only at customer creation (`hub/internal/web/configs.go:725`) and at RESET completion (`customer_reset.go:162`); a customer whose e-mail was added later, or whose previous box was destroyed, never receives one unless the operator presses „Send self-bind link". The volunteer guide's operator prerequisites do not list that press. Intervention **I1** of the big night (the operator's button pressed). **Fix shape (for the operator to choose):** send the link when an unclaimed appliance registers and a customer with no host is waiting, or add the press to the guide's operator prerequisites (day-0 A.2). **SHIPPED hub v0.114.0 (2026-09-15), NOT YET PROVEN BY A REAL MAIL:** triggers added — e-mail set/changed on a customer with no box, and host delete — each re-checking no bound host; every send recorded as `selfbind_link_sent` and shown on the Setup tab. Unit-proven with a red-proof (`TestSelfBind_EmailSetOnWaitingCustomerSendsLink`). The live check with the Gmail-read mailbox was NOT run: it needs a throwaway customer, and deleting one runs the RESET cascade (ep0 `deprovision` + Cloudflare), which is fenced without the operator's word. **Closes on:** one real mail from either trigger. | **READY — rank P1-HIGH; owner: CC (hub fix) · operator (which fix shape)** | | **R-511** | **[P2-MEDIUM] A customer whose box is rebuilt keeps its ep0 PBS token, and then the DR tier can be neither provisioned nor re-issued: the hub's error advises the one action that refuses.** MEASURED 2026-09-14 (BIGNIGHT, VM 333, `tester-1`, DR tier ticked): on the new box's WireGuard registration the hub logged `[ERROR] pbsdr auto-provision for tester-1 (WG-registration hook): the endpoint already holds a PBS token for tester-1 but the hub has no descriptor — use the explicit "Re-issue PBS credentials" action — save the customer config to retry`. The operator's `POST /configs/tester-1/pbsdr-reissue` → **400 `No provisioned PBS DR tier for this customer`** (`hub/internal/web/pbsdr.go` ~411). The token was left by the doorstep walk's host delete (a host delete does not deprovision tenancy; only RESET does, which also removes the tunnel). So a box rebuilt for an existing customer — the reinstall journey — has no whole-guest off-site tier and no button that restores it. **Fix shape:** let re-issue adopt an existing endpoint token when the descriptor is absent (the message already assumes it does), or have host delete offer to drop the PBS token. **SHIPPED hub v0.114.0 (2026-09-15):** re-issue ADOPTS the endpoint token when the descriptor is absent and the DR flag is on (re-key + descriptor rebuilt from the endpoint + `pbsdr_adopted` audit row); DR flag off still refuses, endpoint untouched (both pinned, red-proofed). **NOT proven on tester-1's real state:** re-issue needs an enrolled host and tester-1 has none. **ep0 cleanup DONE on the operator's yes (2026-09-15 08:33Z):** the one doorstep snapshot `ns tester-1 / ct/9201/2026-09-14T16:04:53Z` (logical 1 928 820 672 B) forgotten via the local PBS API; namespace lists empty; token `felhom@pbs!tester-1` kept; nothing outside tester-1 read or touched (`D3-*` in `audits/evidence-p1fixes-2026-09-15/`). The token-only release on host delete was NOT built → R-526. **Closes on:** adopt ending in a descriptor the next tester-1 box consumes **CONSEQUENCE MEASURED 2026-09-16:** the code fix shipped for this row is sound and currently INERT - the adopt path runs and the ENDPOINT refuses it for a missing `Datastore.Modify` grant (**R-534**). On the drill box the result is that a fresh install has NO off-site tier at all, and read-only listing of ep0 shows `ns/tester-1/ct` empty before and after the whole drill. This row stays open until R-534's grant is given and a re-issue is seen to succeed. **CLOSED 2026-09-16 — the adopt path is PROVEN LIVE.** The rebuilt-customer case reproduced by itself on a fresh box (the WG-registration hook refused exactly as this row describes and named the Re-issue action), and the adopt then completed: reissue ok → pbsdr ADOPTED (gen 2) → the box consumed the single-use secret two seconds later. The code shipped 2026-09-15 was sound; what made it inert was the ep0 grant (R-534), now given. | **CLOSED 2026-09-16 — proven live on a fresh box** | | **R-516** | **[P3-LOW] English a customer meets on a fresh box and its apps' first screens — enumerated by the big night.** MEASURED 2026-09-14 (BIGNIGHT, VM 333, ISO 1.27.1, controller 0.242.0). Felhom-owned: (1) the dashboard menu item **„Debug"**; (2) the dashboard CPU tile **„Load: 0.29 / 0.39 / 0.37"**; (3) the launcher tile **„Filebrowser"** opens a login in English with no Felhom text (R-513); (4) the storage page mixes formal „Adjon hozzá / Csatlakoztasson" with the product's „te". App first screens a household meets before any Felhom text helps: (5) **Uptime Kuma 2.4 opens on „Which database would you like to use?"** (SQLite / Embedded MariaDB, „Next") — the app card's „Első lépések" does not mention it; (6) PrivateBin, Gokapi, AdventureLog and FileBrowser UIs are English (the apps' own). Already rows: the Proxmox installer screens (R-495, answered by the guide), `wiki.DOMAIN` (R-498). **Fix shape:** rename „Debug"/„Load" (controller); add the Uptime Kuma database step to its card, or pre-seed `db-config.json` for SQLite in the template (catalog). **Added by F4 (20:04:54Z):** (7) the storage page prints the disconnect time as a raw ISO UTC string „Leválasztva: 2026-09-14T19:58:02Z"; (8) the „Meghajtó leválasztva" banner appears twice on every page; (9) „4 telepített alkalmazás nem fut — nézze meg a rendszermonitort" uses the formal form. **Added by F7 (20:50–21:00Z, system disk at 95 %):** (10) a banner on every page in English, „**SSD disk usage high: 90%**"; (11) the dashboard tile reads „Rendszer (/) 61.8 GB / 68.7 GB (**90%**)" while `df` reports **95 %** (reserved blocks ignored), and „(/)" labels the data volume `/mnt/sys_drive`; the deploy page says nothing about free disk. **Added by the i18n spike (2026-09-17, controller v0.247.0):** (12) **six formal („ön") forms in the converted dashboard copy** — „Olvassa be telefonnal" (launcher QR hint), „Biztosan kikapcsolja a megosztást?" (launcher), „Ha újratelepíti … importálnia kell" (the layout's remove-app modal), „Kérjük, vegye fel a kapcsolatot" (backups empty state). They were NOT fixed — a localisation release may not change Hungarian bytes — and are now COUNTED by `controller/scripts/i18n_missing_gate.py` (`HU_FORMAL_CEILING` = 6, a ratchet: a new one convicts, fixing one lowers it). The inventory (`audits/I18N-INVENTORY-2026-09-17.md`) is the list this row closes against in localisation slice 6 (R-561). **Extended 2026-09-17 by slice 1 (controller v0.248.0–v0.250.0):** the converted copy now counts **16** formal forms (`HU_FORMAL_CEILING` 6 → 10 → 12 → 16, each raise stated, Hungarian unchanged by rule) — and the count is an UNDER-count: the gate's stem list sees 4 on the release C pages (storage, drive wizards, debug), while a wider list („írja be”, „adja meg”, „adjon hozzá”, „válassza ki”, „biztosan eltávolítja”, „engedélyezze”, „hozzon létre”, „kattintson”) finds at least **22** keys there alone (login „Adja meg a jelszavát”, the NAS guide, the storage confirms). Widen the stems when this row is worked; the ceiling rises with them. **NARROWED 2026-09-20 by localisation slice 6's walk, item by item** (`audits/i18n-slice6-2026-09-20/R-516-item-by-item.md`): item 2 FIXED (slice 1), items 3 and 6 APP-OWNED (and for an English household item 6 is an advantage), item 11 is a wrong NUMBER rather than a language and needs its own row, item 1 is correct English for an English reader and still open for a Hungarian one. **Items 4, 7, 8, 9 and 10 are NOT DECIDABLE FROM AN ENGLISH WALK** - they are about Hungarian copy quality, a disconnected drive and a full disk, none of which this walk had. **What this row is now waiting for is a HUNGARIAN walk on a box with a second drive**, not another English one; saying it closed would be the kind of closure that makes a register stop meaning anything. | **NARROWED 2026-09-20 - rank P3-LOW; owner: CC; needs a Hungarian walk** | -| **R-518** | **[P2-MEDIUM] „Mentés most" on the whole-system backup stops every app for about eight minutes while the page promises „csak néhány másodpercre".** MEASURED 2026-09-14 (BIGNIGHT, VM 333, 12 apps): the button's call quiesced all 12 stacks at 19:03:23Z (first stopped 19:03:27Z); the local vzdump ran 19:03:49 → 19:09:59Z; the controller then kept the apps stopped for the second (PBS) tier and restarted them at 19:10:09Z after it failed, the last started 19:11:12Z (`phase4/guest-backup-quiesce-log.txt`) — **≈ 7 m 45 s** with every app answering 404. The page under the button: „Pillanatkép-mód: az alkalmazások csak néhány másodpercre állnak le." A household pressing it at dinner loses every app for the length of the dump, and longer on a bigger box. **Fix shape:** state the real expected downtime (it scales with data), or quiesce per tier and not across a second tier's attempt; do not start a tier whose storage is absent (see R-517). **NARROWED 2026-09-15 (controller v0.243.0 + agent v0.131.0):** a tier whose storage the agent reports absent is skipped before anything stops (`backup_tier_skipped`, once per absence; unknown never skipped), and the button copy now says „általában néhány perc, nagyobb adatnál több". Unit-proven with red-proofs. **Still open:** quiesce per tier, so a slow second tier does not keep every app down. | **READY — rank P2-MEDIUM; owner: CC (controller)** | +| **R-518** | **[P2-MEDIUM] „Mentés most" on the whole-system backup stops every app for about eight minutes while the page promises „csak néhány másodpercre".** MEASURED 2026-09-14 (BIGNIGHT, VM 333, 12 apps): the button's call quiesced all 12 stacks at 19:03:23Z (first stopped 19:03:27Z); the local vzdump ran 19:03:49 → 19:09:59Z; the controller then kept the apps stopped for the second (PBS) tier and restarted them at 19:10:09Z after it failed, the last started 19:11:12Z (`phase4/guest-backup-quiesce-log.txt`) — **≈ 7 m 45 s** with every app answering 404. The page under the button: „Pillanatkép-mód: az alkalmazások csak néhány másodpercre állnak le." A household pressing it at dinner loses every app for the length of the dump, and longer on a bigger box. **Fix shape:** state the real expected downtime (it scales with data), or quiesce per tier and not across a second tier's attempt; do not start a tier whose storage is absent (see R-517). **NARROWED 2026-09-15 (controller v0.243.0 + agent v0.131.0):** a tier whose storage the agent reports absent is skipped before anything stops (`backup_tier_skipped`, once per absence; unknown never skipped), and the button copy now says „általában néhány perc, nagyobb adatnál több". Unit-proven with red-proofs. **Still open:** quiesce per tier, so a slow second tier does not keep every app down. **— NIGHT 2026-09-23 (controller v0.267.0):** the copy half is DONE: the page and the confirm now state the measured stop (≈ 8 minutes on a 12-app box), both languages, red-proofed (`audits/night-2026-09-23/A5-*`). The brief's „csak néhány másodpercre" had already gone in v0.243.0. **Still open:** quiesce per tier, so a slow second tier does not keep every app down. | **READY — P2, narrowed to per-tier quiesce; owner: CC (controller)** | | **R-519** | **[P2-MEDIUM] After a backup torn by a power cut, an app's restore point carries the new database dump's time while its files are from the previous run — and no customer screen says the run was interrupted.** MEASURED 2026-09-14 (BIGNIGHT F2, VM 333): „Mentés most" 19:40:03Z; the power was cut 19:40:08Z while adventurelog was stopped for its volume dump. On disk afterwards, `backups/primary/adventurelog`: `db-dumps/adventurelog-postgres.sql` **19:40:07**, `volume-dumps/*.tar` **19:00:35**, `manifest.json created_at 19:02:34Z`; bookstack the same shape (sql 19:40:07, tars 19:00:48). `GET /api/backup/snapshots` for both → `time 2026-09-14T19:40:07Z, helyi` (`phase5/F2/units-on-disk.txt`, `backup-honesty.txt`). `/backups/apps` shows „Utolsó adatbázis mentés 2026-09-14 21:40 · … OK" and every app „Utolsó: 5 perce"; `/backups` and `/dashboard` contain no word of an interruption (fragments `megszakad|sikertelen|nem sikerült` = 0; control „hiba" appears in the standing warning text). The controller itself knew: `[appstop] crash recovery: an app-data backup (volume dump) … was interrupted … restarting them: [adventurelog]` and pushed `backup_failed (error)` to the hub. A household restoring „the 21:40 backup" gets 21:00 files for BookStack's uploads. **Fix shape:** date a point by the oldest part it contains (or mark it partial) and show the interrupted run on the backups page until the next complete one. **F6 (drive unplugged 1 s into a backup, 20:37:59Z) adds three facts:** the run skipped four apps' volume dumps („Skipping volume dump for immich — drive disconnected", also jellyfin, nextcloud, paperless-ngx) and still reported `db_dump {"count":6, "success":true}`; nextcloud's point is dated 20:37:59Z (its SQL finished before the unplug) beside 19:01 volume tars; and a torn `immich-postgres.sql.tmp` (20:38:00) plus F3's `pre-restore-…-nextcloud-mariadb.sql.tmp` are left in the units on the drive. Immich's point correctly stayed at 19:02:34Z (the `.tmp` was not promoted). `/backups/apps` fragments `kihagy|sikertelen|részleges` = 0 (`phase5/F6/`). | **READY — rank P2-MEDIUM; owner: CC (controller)** | | **R-520** | **[P3-LOW] A power cut during a guarded Update leaves no record that an update was running, so whether the journal resumes or aborts honestly is unmeasured for a real version change.** MEASURED 2026-09-14 (BIGNIGHT F3, VM 333): the only Update the catalog allowed after the Phase 4 revert was a same-version one on nextcloud; power was cut 2 s after `update nextcloud: phase pulling` (safety dump written, pin advanced to the unchanged definition). After boot: no log line resumes, aborts or names the interrupted update; `app.yaml` `pinned_images` = `installed_images`, no hold, no verdict; page „Fut · Naprakész". Nothing wrong was produced — and nothing could have been, with identical images. **What it needs:** the same cut on a real bump (a throwaway app with a one-step catalog move on a scratch branch or the scratch guest), asserting the page's message and the pin after boot. **CLOSED 2026-09-21 — measured on a REAL bump, and it behaved.** Scratch guest 9202 (demo-hp), controller v0.260.0, throwaway `uptime-kuma`, real catalog move 2.4.0 -> 2.5.0 (reverted the same session). `pct stop 9202` at 11:04:54.439Z, 3 ms after the poller observed `update_phase: pulling`. **Honest limit on the instrument:** `pct stop` returned 3.8 s later, so the phase at the DECISION is observed and the phase at the freeze is inferred. **After boot the box said so itself — a POSITIVE observable, not an absence:** `update recovery: uptime-kuma was interrupted in pulling (started 2026-09-21T11:04:52Z) — nothing had run; putting the pin back`, then `pin uptime-kuma: …:2.4.0` and `update …: pin and definition PUT BACK to the pre-update version`. `pinned_images` and `installed_images` both 2.4.0, live compose line 2.4.0, no hold (correctly — nothing ran), `update_phase: failed`, app **running and healthy on 2.4.0** 31 s after boot, and the household told: „A frissítés megszakadt, mert a vezérlő újraindult, mielőtt az új verzió elindult volna. Az alkalmazás a korábbi verzióval fut tovább." **THE INSTRUMENT TRAP THIS RUN FOUND IS WORTH MORE THAN THE RESULT.** The first post-crash read, taken from the host against the STOPPED guest, reported the update journal **ABSENT** — which would have made this a defect finding. It was a FALSE NEGATIVE: **`pct mount` maps the guest's ROOTFS ONLY and does not apply the guest's own internal mounts**, and 9202 keeps `/var/lib/docker` on its own ext4 mount under the `mp0` volume, so from the host that path is an EMPTY STUB. The bytes were at `/var/lib/lxc/9202/rootfs/var/lib/felhom/docker/volumes/felhom-controller-data/_data/data/`, and the box's own recovery read them at the next boot. Caught by running `find` over the whole rootfs instead of trusting one constructed path, and by demanding a positive control for the directory searched. **An empty directory is not evidence of an absent file.** **What is NOT measured and does not re-open this row:** the second cut, in the `starting` phase (Scenario F's territory — the migration may have run). Evidence: `audits/update-arc-2026-09-21/` 08-12. **CORRECTED 2026-09-21 (same day): that sentence left the dangerous half of the question in prose and in no row — see R-610, which carries it and CLOSES it with three measurements.** The history above stands as written; only this pointer is added. | **CLOSED 2026-09-21 — measured on a real version change** | | **R-521** | **[P3-LOW] One unplugged drive sends the operator five e-mails and the household none.** MEASURED 2026-09-14 (BIGNIGHT F4, VM 333): `storage_disconnected (error)` at 21:58:02 CEST plus `app_start_failed (warning)` for each of the four apps the drive carries at 21:58:15, each with its own operator mail (hub log: five `Operator email sent`). The customer's mailbox (`tester1@felhom.eu`, read through the connector) received nothing; the household learns of it only on the dashboard, which is honest and says what to do. The apps' stop is a consequence of the drive event, so the four warnings add no information. **Fix shape:** suppress `app_start_failed` for apps stopped by a `storage_disconnected` (the dead-app check already knows the reason — „Hiányzó tárhely"), and decide whether a household gets a mail for a lost drive. **F6, 40 min later, the opposite failure:** a second, separate drive loss (20:38:33Z) produced `storage_disconnected (error)` and four `app_start_failed`, and the hub logged `Operator email suppressed … cooldown` for all five — **no mail at all for the second unplug**; only `health_degraded (warning)` mailed. A per-key cooldown that outlives the recovery (`storage_reconnected` came between them) silences a new incident. **F7:** the system disk at 95 % produced only `health_degraded (warning)`, whose operator mail was **suppressed by the cooldown** left by F6's `health_degraded` 15 minutes earlier; no disk-specific event reached the hub at all — the operator was not told the disk was nearly full. | **READY — rank P3-LOW; owner: CC (controller) · operator (customer mail policy)** | @@ -764,13 +763,13 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-601** | **[P2-MEDIUM] ~~demo-hp is unreachable~~ — WRONG, WITHDRAWN THE SAME DAY. The box was never down; MY ROUTES WERE.** Filed 2026-09-21 morning after `ssh demo-hp`, the hub-vaulted break-glass over the tailnet, `demo-hp-lan`, a ping and `ip neigh` on felhom-pve all failed, and `tailscale status` said *`demo-hp … offline, last seen 30d ago`*. **The operator looked at the hub and said it was ONLINE. It was**: it had reported 13 minutes earlier, and it has been up **4 weeks 2 days**. **Two stale facts, each enough on its own:** (1) `~/.ssh/config` sends `demo-hp` to the tailnet address `100.76.96.79`, and **tailscale is not installed on that box at all** (checked on it: no `tailscaled`, no `tailscale` binary) — so that entry is a dead peer from an earlier build and can never answer; (2) `demo-hp-lan` and `nodes.md` both say `192.168.0.87`, and the box is **statically** on **`192.168.0.104/24`**, bridge-port `nic0` (nodes.md says `enp2s0f0`). **The hub knew the right address the whole time** — every host report carries `addresses: [{iface: vmbr0, cidr: 192.168.0.104/24}, …]`. **What I actually did wrong, and it is the part worth keeping:** I ran `ip neigh` on felhom-pve, and `192.168.0.104 … STALE` was *in that output*, four lines above the `192.168.0.87 … FAILED` I quoted. I searched the output for the address I expected instead of reading it for the address that was there. **The standing rule says a "no access" claim must list what was tried; it does not say the list makes the claim true.** Six failed routes to a stale address are six failures of one assumption, not six pieces of evidence. **FIXED:** both `~/.ssh/config` entries repointed to `.104` (each carrying a comment saying why, including that there is no tailscale on this box), both verified live; `nodes.md` corrected. | **CLOSED 2026-09-21 — withdrawn, the claim was false; the routes are fixed** | | **R-604** | **[P2-MEDIUM] A per-customer controller floor silently excludes that box from every global floor raise, and NOTHING says so — demo-hp missed four of them.** FOUND 2026-09-21 while raising the global floor to 0.259.0 at the operator's request. The raise logged `Global controller-version floor set to "0.259.0"` and then `managed floor SERVED for demo-felhom` — **and nothing at all for demo-hp**, which went on reporting every few minutes and stayed on 0.258.0. Cause: `customer_configs.min_controller_version` for demo-hp held **`0.243.0`**, a per-customer override that wins over the global. It is a **leftover from the 2026-09-16 drill**, whose golden was 0.243.0; **R-343 measured on 2026-08-18 that all five rows were EMPTY and recorded that as a safety property** — it stopped being true and nothing surfaced the change. demo-hp had therefore silently missed the raises to 0.253.0, 0.254.0, 0.257.0 and 0.259.0. **Why it is invisible rather than merely quiet:** `managed floor SERVED` fires **once per CHANGE** (`h.floorNotes`, `api/handler.go:600`), deliberately, because a box reports every few minutes — so a box whose override never changes is silent for ever, and its silence is indistinguishable from the silence of a box that already had the line. A session that raises the floor reads one SERVED line and reasonably concludes the fleet took it. **CLEARED** for demo-hp the same session (rollback line: POST `/customers/demo-hp/floor` with `min_controller_version=0.243.0`, `min_agent=0.131.0`); it then self-updated 0.258.0 → 0.259.0 in **under four minutes**, healthy, `settle-gate: GO — at/above floor 0.259.0`, and its claim page answers **"Wrong or expired code"** in English — the floor delivered the FIX, not a version string, to a box nobody hand-deployed. All five overrides are now empty. **Fix shape (smallest first):** the floor-raise page shows which customers carry an override and would NOT be moved, before the save; or the raise logs one line per customer naming the ones it skipped and why. A raise that quietly reaches half the fleet is worse than one that refuses. | **READY — rank P2-MEDIUM; owner: CC (hub)** | | **R-605** | **[P3-LOW] A catalog gate that REFUSED TO RUN and a gate that ran and could not decide print the same word, so a reader cannot tell which happened.** FOUND 2026-09-21 while answering why the chaos night's update round could not run. On 2026-09-17 `check-image-resolvable` and `check-volume-persistence` both returned INCONCLUSIVE and the drawn `update` action was replaced with `use` (`audits/DRILL-chaos-night-2026-09-17.md:692-695`). **Neither script is defective — they behaved exactly as designed**, and both headers say why: a detector that cannot prove itself must refuse to report rather than guess (`check-image-resolvable.py` cites the 2026-07-21 incident where a Docker Hub throttle read as 24 of 65 pins falsely dead). **What is missing is the DISTINCTION.** `check-volume-persistence.py`'s `self_test` refuses to evaluate ANY app when it cannot build its canary image — a harness-level refusal — while `classify()` returns a per-app UNDETERMINED for an app that wrote nothing; `check-image-resolvable.py` likewise separates a harness-level canary failure (exit 2 at `check()` L180-183) from a per-pin throttle (L121-128). **`catalog_gates.py`'s VERDICT map collapses all of them into one `INCONCLUSIVE` label**, so the operator-facing summary cannot say whether the gate ran at all. **The cost is real and already paid:** no raw stdout of the 2026-09-17 run survives in either evidence directory, so the exact triggering path is INFERRED from the code plus the documented throttle precedent, not observed — a distinct summary line would have recorded it for free. **Fix shape:** have each gate's exit distinguish "the harness refused" from "the result is undetermined" (a third exit code, or a marker line the runner matches), and have `catalog_gates.py` print the two differently. **Ships with a decoy each way (R-421): a run whose canary fails must NOT read as a per-app undetermined, and vice versa.** Small. | **READY — rank P3-LOW; owner: CC (catalog)** | -| **R-607** | **[P3-LOW] A forced catalog sync answers „nincs változás" while the cache DOES change, and `catalog_images` stays stale until a separate rescan — so the update badge can be wrong for a window nobody bounds.** MEASURED 2026-09-21 on scratch guest 9202 (controller v0.260.0) while proving R-524. A real catalog move was pushed, `POST /api/sync` was invoked, and it answered **„nincs változás"** — yet the box's own cache file `/catalog-cache/templates/uptime-kuma/docker-compose.yml` **had moved to the new tag**. `Stack.CatalogImages` as served by the API stayed at the OLD value until a separate `POST /api/stacks/rescan`. **WHY IT MATTERS AND WHY IT IS NOT COSMETIC:** `CatalogImages` is the single input `stacks.CatalogOrder` compares against (v0.260.0), so for that window the badge answers from a stale catalog — it can read „Naprakész" on an app that IS behind, which is the exact failure §5.6 of `09` was written to prevent, arriving by a different door. **It also cost a measurement:** the session that found it nearly recorded a `tag-ok` badge as proof of the R-524 ahead arm when the badge was in fact stale; the honest reading came only after the rescan. **An instrument that can report an old value as a current one is not a measurement.** **TWO SEPARATE QUESTIONS, and the row does not conflate them:** (a) why the sync REPORTS no change when the working tree moved — a wrong sentence, possibly a comparison against the wrong ref; (b) whether `CatalogImages` is refreshed by the sync at all or only by `ScanStacks` on its own timer — if the latter, the staleness window is the scan interval and is bounded but unstated. **Neither was isolated** — this row records the observation, not a diagnosis. **Also observed in the same run, NOT diagnosed and folded in here rather than filed twice:** a removed app leaves `applied-compose.yml` behind in its stack directory. Stated as observed; it was not established whether that is intended. **Fix shape:** first reproduce with a loop that pushes a tag, syncs, and reads `catalog_images` on a timer, so the window is a NUMBER before anything is changed. Evidence: `audits/update-arc-2026-09-21/06-sync-after-bump.txt` and `16-r524-sync-box-ahead.txt`. **— UPDATE NIGHT 2026-09-21:** **Seen again 2026-09-21 (update night), a dozen times in one session, and for the first time with a USER-VISIBLE consequence rather than a measurement one.** On `mealie` the bump was pushed, `POST /api/sync` AND `POST /api/stacks/rescan` were both run, and the badge still read the up-to-date one (HU „Naprakesz", EN "Up to date") — the catalog had not reached `catalog_images` yet. The guarded Update was then pressed and **reported „Frissitve" after 2.1 seconds having moved nothing at all**: pinned, installed, the live compose line and `docker inspect` all still read `v3.20.1`. That is honest given a stale cache — the pin is written from *the catalog's current definition*, which was still the old one — but what the household sees is a button that says it updated them and did not. **A NUMBER, at last, which is what this row asks for:** the night's harness was changed to poll `catalog_images` until the pushed reference appears and to report how long that took; those figures are each edge's `badge_catchup_seconds`, and here they are: **4.4 s, 4.4 s, 4.5 s, 4.5 s — and 29.0 s.** The four fast ones are one sync+rescan round; the 29-second one (`nextcloud`, an engine-sidecar bump) needed **additional** sync+rescan rounds before `catalog_images` carried the pushed reference. **So the window is not a fixed scan interval — it varies by roughly 7x between edges on the same box in the same hour**, which is why a caller (or a household) cannot know when the badge is safe to read. Before tonight this row had no number at all; it now has five, and they disagree with each other, which is itself the most useful thing about them. Every drill-catalog bump of the night was followed by `POST /api/sync` answering „Sablonok naprakészek — nincs változás" while the box's cache HAD moved, with `catalog_images` staying stale until a separate `POST /api/stacks/rescan`. The night's harness therefore rescans unconditionally after every sync, which is a workaround and not a fix. **The window was still never measured as a NUMBER** — that is what the row asks for and what remains owed. | **READY — rank P3-LOW; owner: CC (controller)** | +| **R-607** | **[P3-LOW] A forced catalog sync answers „nincs változás" while the cache DOES change, and `catalog_images` stays stale until a separate rescan — so the update badge can be wrong for a window nobody bounds.** MEASURED 2026-09-21 on scratch guest 9202 (controller v0.260.0) while proving R-524. A real catalog move was pushed, `POST /api/sync` was invoked, and it answered **„nincs változás"** — yet the box's own cache file `/catalog-cache/templates/uptime-kuma/docker-compose.yml` **had moved to the new tag**. `Stack.CatalogImages` as served by the API stayed at the OLD value until a separate `POST /api/stacks/rescan`. **WHY IT MATTERS AND WHY IT IS NOT COSMETIC:** `CatalogImages` is the single input `stacks.CatalogOrder` compares against (v0.260.0), so for that window the badge answers from a stale catalog — it can read „Naprakész" on an app that IS behind, which is the exact failure §5.6 of `09` was written to prevent, arriving by a different door. **It also cost a measurement:** the session that found it nearly recorded a `tag-ok` badge as proof of the R-524 ahead arm when the badge was in fact stale; the honest reading came only after the rescan. **An instrument that can report an old value as a current one is not a measurement.** **TWO SEPARATE QUESTIONS, and the row does not conflate them:** (a) why the sync REPORTS no change when the working tree moved — a wrong sentence, possibly a comparison against the wrong ref; (b) whether `CatalogImages` is refreshed by the sync at all or only by `ScanStacks` on its own timer — if the latter, the staleness window is the scan interval and is bounded but unstated. **Neither was isolated** — this row records the observation, not a diagnosis. **Also observed in the same run, NOT diagnosed and folded in here rather than filed twice:** a removed app leaves `applied-compose.yml` behind in its stack directory. Stated as observed; it was not established whether that is intended. **Fix shape:** first reproduce with a loop that pushes a tag, syncs, and reads `catalog_images` on a timer, so the window is a NUMBER before anything is changed. Evidence: `audits/update-arc-2026-09-21/06-sync-after-bump.txt` and `16-r524-sync-box-ahead.txt`. **— UPDATE NIGHT 2026-09-21:** **Seen again 2026-09-21 (update night), a dozen times in one session, and for the first time with a USER-VISIBLE consequence rather than a measurement one.** On `mealie` the bump was pushed, `POST /api/sync` AND `POST /api/stacks/rescan` were both run, and the badge still read the up-to-date one (HU „Naprakesz", EN "Up to date") — the catalog had not reached `catalog_images` yet. The guarded Update was then pressed and **reported „Frissitve" after 2.1 seconds having moved nothing at all**: pinned, installed, the live compose line and `docker inspect` all still read `v3.20.1`. That is honest given a stale cache — the pin is written from *the catalog's current definition*, which was still the old one — but what the household sees is a button that says it updated them and did not. **A NUMBER, at last, which is what this row asks for:** the night's harness was changed to poll `catalog_images` until the pushed reference appears and to report how long that took; those figures are each edge's `badge_catchup_seconds`, and here they are: **4.4 s, 4.4 s, 4.5 s, 4.5 s — and 29.0 s.** The four fast ones are one sync+rescan round; the 29-second one (`nextcloud`, an engine-sidecar bump) needed **additional** sync+rescan rounds before `catalog_images` carried the pushed reference. **So the window is not a fixed scan interval — it varies by roughly 7x between edges on the same box in the same hour**, which is why a caller (or a household) cannot know when the badge is safe to read. Before tonight this row had no number at all; it now has five, and they disagree with each other, which is itself the most useful thing about them. Every drill-catalog bump of the night was followed by `POST /api/sync` answering „Sablonok naprakészek — nincs változás" while the box's cache HAD moved, with `catalog_images` staying stale until a separate `POST /api/stacks/rescan`. The night's harness therefore rescans unconditionally after every sync, which is a workaround and not a fix. **The window was still never measured as a NUMBER** — that is what the row asks for and what remains owed. **— NIGHT 2026-09-23, measured twice more:** for an INSTALLED app the stack directory's compose is the applied one, so waiting on it for a drill commit took ~7 min (chaos rounds 2 and 4); the badge's own `catalog_images` caught up in seconds. And twice a fresh box walk deployed a STALE template (the uptime-kuma "after" run, the opengist re-walk) because the deploy ran before the sync reached the file — the walks now wait for the box's template to show FROM. | **READY — rank P3-LOW; owner: CC (controller)** | | **R-608** | **[P2-MEDIUM] The controller swaps ITSELF in the middle of a guarded app update, and 04:30 sits inside the window proposed for automatic app updates.** FOUND 2026-09-21 by reading the clock, not by a failure. The controller self-updates daily at `self_update.auto_update_time` — **default 04:30** (`config/config.go` L422, scheduled `cmd/controller/main.go` ~L1365) — and again from `MaybeAutoUpdate` after ANY hub report once a floor sits above the box, so at any hour. The swap restarts the controller container. `09` §3b **Q1** proposes **02:30–05:00** for automatic app updates. **It contains 04:30.** **MEASURED, and the gap was NARROWER than it first looked — which is why the fix is where it is:** the updater's only busy gate was `backupRunning` (`updater.go` L61, read at L487 dry-run, L512 `TriggerUpdate`, L660 `maybeAutoUpdate`), wired in `main.go` L659 to `backupMgr.IsRunning()`. The guarded update's **`backing-up` phase DOES take the backup single-flight** (`RunAppBackupNow` → `acquireRunning`, `backup/update_guard.go:333`), so that ONE phase was already covered. `checking`, `safety-dump`, `pinning`, `pulling`, `starting` and `verifying` were not — and the last two are exactly where the new version may already have touched the customer's data. The reverse was absent too: `UpdatePreflight` never asked whether a swap was running. **CLOSED 2026-09-21 — controller v0.261.0.** `stacks.Manager.AnyUpdating()` → `Updater.SetAppUpdatingCheck`, a deliberate sibling of `SetBackupRunningCheck` consulted in the SAME three places; `Updater.IsUpdateRunning` → `Manager.SetSelfUpdatingCheck`, and `UpdatePreflight` refuses `self_updating`. Both wired in `main.go`, the only place holding both objects — **`stacks` never imports `selfupdate`.** Two sentences born as bundle keys. **THE PROPERTY THAT MATTERS MOST IS THAT THE LOCK DOES NOT LATCH:** `Stack.Updating` is cleared on done, failed AND held, so a HELD app does not block the controller's own updates — including the release that might fix whatever held it. A latching gate would be a worse failure than the one prevented, and a silent one. Pinned by `TestR608_LockReleasesAfterHold`. Four red-proofs, each seen to fail. | **CLOSED 2026-09-21 — controller v0.261.0** | | **R-609** | **[P3-LOW] An update refusal has a machine-readable reason inside the process and none on the wire, so an unattended caller cannot tell "wait" from "never".** `UpdateRefusal.Reason` has existed since v0.237.0 (`busy`, `deploying`, `updating`, `held`, `migrating`, `memory`, `disk`, `no_backup`, `downgrade`, and `self_updating` since v0.261.0) and never left the process: the 409 body carried only the translated sentence. **The distinction is not decorative** — `busy`/`updating`/`deploying`/`migrating`/`self_updating` are TRANSIENT and `held`/`downgrade` are TERMINAL until a person acts. A caller that cannot tell them apart either gives up on a passing backup window or presses a terminally-refused button on every pass for ever. `09` §6.2's unattended caller reads exactly this. **CLOSED 2026-09-21 — controller v0.261.0.** The body gains `data: {"reason": ""}`, ADDITIVELY; the sentence is unchanged and no page moves. Table-driven test over five reachable paths plus a control that a non-refusal carries none. **FOUND WHILE WRITING THE TEST, NOT BY READING — and it was the reason that matters most:** `actionStack` refuses a HELD app on **its own line, BEFORE `UpdatePreflight`** (`api/router.go` ~L601), so `held` would have been the one reason missing from the wire. That line now carries it too. Red-proof seen to fail. | **CLOSED 2026-09-21 — controller v0.261.0** | | **R-610** | **[P2-MEDIUM] The power cut AFTER the new version has started was never measured — R-520 closed on the safe half only.** R-520 (CLOSED 2026-09-21) cut in `pulling`, where **nothing had run**: the pin goes back and that is the easy case. Its own last lines said the `starting` cut was "NOT measured and does not re-open this row", and no row carried it. The dangerous case is the cut AFTER the new version has started and may already have migrated the customer's data — where `RecoverUpdates` (`stacks/update.go:909`) marks the app Updating and RESUMES rather than rolling back. **That behaviour was READ from the source and never observed.** **CLOSED 2026-09-21 — measured THREE times on guest 9202, controller v0.260.0**, with three different apps and two different cut mechanisms: vikunja 2.3.0→2.6.0 and uptime-kuma 2.4.0→2.5.0 by `pct stop` (a real power cut), and wishlist v0.66.0→v0.67.0 by restarting ONLY the controller container (exactly what a self-update does). **All three ended HONEST:** the recovery line appeared, the update resumed, each app came up on the NEW version, and in every case all FOUR version observables agreed — `pinned_images`, `installed_images`, the live compose `image:` line, and `docker inspect` of the running container (digests matched too). No hold, no stuck `Updating`, no surviving journal, no retry loop. The seeded data read back through each app's own front door for the two where a post-cut read-back was taken. **THE DANGEROUS CASE WAS GENUINELY EXERCISED, and the proof is a log line, not an assumption:** vikunja's own log shows `Ran all migrations successfully` and `Vikunja version v2.6.0` at **12:28:26.881 UTC — 0.64 s after the cut decision and ~0.4 s before the guest stopped answering.** The 2.6.0 schema migration had ALREADY been applied to the customer's SQLite database when the power went. Recovery resumed FORWARD, so old-binary-on-migrated-database never happened — **but this branch is one step from it: had the cut landed a second earlier, in `pinning` or `pulling`, the 2.3.0 pin would have been put back onto a 2.6.0 database.** That is not a defect today; it is the reason §4's "no automatic rollback" ruling is right, and it is now evidence rather than argument. **INSTRUMENT LIMIT, stated because it bounds the claim:** `starting` lasts well under a second on this box. Three attempts across two cut mechanisms (`pct stop` returning in 3.0–3.8 s; a controller restart in 1.7 s) ALL landed in `verifying`. No phase was faked. **`RecoverUpdates` handles `starting` and `verifying` in ONE branch, so all three runs exercise the same recovery arm** — the arm under test. A cut that lands inside `starting` itself remains unmeasured and would need an in-process fault injector. Evidence: `audits/update-arc-gaps-2026-09-21/` 04, 05, 07. | **CLOSED 2026-09-21 — measured three times** | | **R-611** | **[P3-LOW] A session reported "everything is done" over a phase it had silently skipped — the process failure, not the missing measurement.** The 2026-09-21 update-arc session's brief contained a Phase 5 spike: one app updated by the box with **nobody pressing anything**, once succeeding and once forced to fail. **It did not run, and nothing said so** — no evidence file, no code, and no sentence in `STATUS.md`, `UPDATE-ARC-STATE-2026-09-21.md`, either `REPORT.md` or `09`. `09` §6.2 was left describing Slice 6 "as it would be built" with no measurement under it, which reads like a considered design rather than an untested one. **Why this is a row and not a grumble:** the missing measurement was recoverable in an afternoon; the missing SENTENCE was not, because the next reader had no way to know it was missing. A skipped phase that is declared costs one line; a skipped phase that is not costs the next session its baseline. **CLOSED 2026-09-21 by the successor session**, which ran it (`audits/update-arc-gaps-2026-09-21/`, scenarios F and G) **and** adopted the standing habit that closes it generally: **the report's FIRST section is "not done", even when empty.** Every part and scenario of a brief is listed there if it was skipped, shortened or changed, with the reason. | **CLOSED 2026-09-21 — run by the successor session; "not done" is now the report's first section** | -| **R-612** | **[P1-HIGH] `wishlist` cannot be signed up to on a fresh Felhom install, the deploy reports SUCCESS, and the error the customer sees is a LIE.** MEASURED 2026-09-21 on guest 9202 while seeding for the power-cut drill. The image's first-boot `pnpm prisma db seed` is **`Killed` — OOM at the catalog's `mem_limit: 128M`**. Without it the `Role` and `Group` rows are absent, so **every** signup fails. **The message the user is shown is `User with username or email already exists`** while the container log says the real cause: `FOREIGN KEY constraint violated`. A household would conclude the account already exists and try to recover a password that was never created. **The controller reports the app running and HEALTHY throughout, and the deploy reported successful** — so nothing on the box says anything is wrong. Repaired on the scratch guest only, to unblock seeding: memory raised to 512 M, the image's own seed re-run, memory put back to 128 M. **The catalog was NOT changed** — the fix is a memory-limit question for the catalog and is deliberately left to a session that can measure the real ceiling rather than guess it. **Needs: the actual peak RSS of that seed, then a `mem_limit` that clears it, plus a check that the seed's failure is not silent.** | **READY — rank P1-HIGH; owner: CC (catalog + a look at whether a failed first-boot seed can ever be visible)** | -| **R-613** | **[P2-MEDIUM] `uptime-kuma` parks on its setup wizard with no login and no monitors, and the box tells the household it is HEALTHY.** MEASURED 2026-09-21 on guest 9202. On first boot uptime-kuma 2.4.0 sits at `SETUP-DATABASE` (`Waiting for user action...`) and its main socket.io server never starts. **The controller's `http :3001` probe sees the wizard's 302 and records the app as running and healthy.** So a monitoring app that cannot be logged into, and is monitoring nothing, is presented to the customer as fine. Passed through its own front door for the drill with `POST /setup-database {"dbConfig":{"type":"sqlite"}}`. **This is the health-check class the catalog skill already warns about — a probe that proves the PORT answers, not that the APP works** — and it is worth a row because the failure direction is a false GREEN, which no alarm will ever catch. **Needs: a healthcheck for this template that fails while the wizard is up** (the catalog `REUSE.md` maps the families), and a sweep for other templates whose probe would pass on a setup wizard. **— UPDATE NIGHT 2026-09-21:** the update night could not seed `uptime-kuma` for the same reason and left it out rather than faking it. | **READY — rank P2-MEDIUM; owner: CC (catalog)** | +| **R-612** | **[P1-HIGH] `wishlist` cannot be signed up to on a fresh Felhom install, the deploy reports SUCCESS, and the error the customer sees is a LIE.** MEASURED 2026-09-21 on guest 9202 while seeding for the power-cut drill. The image's first-boot `pnpm prisma db seed` is **`Killed` — OOM at the catalog's `mem_limit: 128M`**. Without it the `Role` and `Group` rows are absent, so **every** signup fails. **The message the user is shown is `User with username or email already exists`** while the container log says the real cause: `FOREIGN KEY constraint violated`. A household would conclude the account already exists and try to recover a password that was never created. **The controller reports the app running and HEALTHY throughout, and the deploy reported successful** — so nothing on the box says anything is wrong. Repaired on the scratch guest only, to unblock seeding: memory raised to 512 M, the image's own seed re-run, memory put back to 128 M. **The catalog was NOT changed** — the fix is a memory-limit question for the catalog and is deliberately left to a session that can measure the real ceiling rather than guess it. **Needs: the actual peak RSS of that seed, then a `mem_limit` that clears it, plus a check that the seed's failure is not silent.** **— FIXED 2026-09-23 night (catalog `a5a729a`): 512M.** Measured on the bench: the seed peaks at 312–345M and was OOM-killed at 128M on every run (kernel `oom_kill` 3); running, the app sits at ~115M (90 % of the old limit alone). On 9202 at 128M the kill came AFTER the Role/Group rows this time, so the sign-up lie did NOT reproduce — the kill is timing-dependent, the fault is memory (the brief's claim held). At 512M the seed completes (`The seed command has been executed`), and wishlist then moved v0.66.0 → v0.67.1 with its test record. **Still open:** a failed first-boot seed is invisible to the box — nothing reads the seed's exit. | **READY — P3, narrowed to making a failed seed visible; owner: CC (catalog)** | +| **R-613** | **[P2-MEDIUM] `uptime-kuma` parks on its setup wizard with no login and no monitors, and the box tells the household it is HEALTHY.** MEASURED 2026-09-21 on guest 9202. On first boot uptime-kuma 2.4.0 sits at `SETUP-DATABASE` (`Waiting for user action...`) and its main socket.io server never starts. **The controller's `http :3001` probe sees the wizard's 302 and records the app as running and healthy.** So a monitoring app that cannot be logged into, and is monitoring nothing, is presented to the customer as fine. Passed through its own front door for the drill with `POST /setup-database {"dbConfig":{"type":"sqlite"}}`. **This is the health-check class the catalog skill already warns about — a probe that proves the PORT answers, not that the APP works** — and it is worth a row because the failure direction is a false GREEN, which no alarm will ever catch. **Needs: a healthcheck for this template that fails while the wizard is up** (the catalog `REUSE.md` maps the families), and a sweep for other templates whose probe would pass on a setup wizard. **— UPDATE NIGHT 2026-09-21:** the update night could not seed `uptime-kuma` for the same reason and left it out rather than faking it. **— FIXED 2026-09-23 night for uptime-kuma (catalog `a5a729a`):** `UPTIME_KUMA_DB_TYPE=sqlite` — the database choice is made, the wizard never appears, the real server starts (`/api/entry-page` → `entryPage`, `/metrics` 401), the database lands in `/app/data` (the backed-up volume). Red-proofed on 9202 through the product: before, the box read `running` over `setup-database`; after, the real server. (A first "after" run used a stale template — R-607's lag — and is kept.) **Still open:** the sweep for other templates whose probe passes on a setup wizard. | **READY — P3, narrowed to the sweep; owner: CC (catalog)** | | **R-614** | **[P3-LOW] A stale `update_phase` survives a remove and redeploy, so a freshly installed app can read "Frissitve" before it has ever been updated.** OBSERVED 2026-09-21 on guest 9202: a newly deployed `uptime-kuma` read `update_phase=done` / „Frissitve" before any update had been run against it — left in the manager's IN-MEMORY stack state by the previous session's update of a since-removed instance of the same name. It cleared on the next controller restart. **Why it matters beyond the cosmetic:** `update_phase` is one of the fields a person (and, after R-609, an unattended caller) reads to decide whether an update happened. A value that outlives the app it described is the same class as the R-166 "absent means unknown" family — a confident answer about something that no longer exists. **Fix shape:** clear the update fields when a stack is removed, beside wherever `Updating`/`updateHeld` are reset; a test that removes and redeploys and asserts the phase is empty. Small. **FIXED in v0.262.0.** `RemoveStack` calls `ClearUpdateState`, which wipes `Updating`, `UpdatePhase`, `UpdatePhaseLabel`, `UpdateError`, `updateHeld` and `HealthProbe`. The name is the only thing a new install shares with the old one, so the record has to go when the app does. Red-proof seen failing. | **CLOSED 2026-09-22 — v0.262.0: a remove clears the update record** | | **R-602** | **[P3-LOW] The language a signed-in page uses is NOT the language a cookie asks for, and a live probe that forgets this reports a fixed defect as unfixed.** FOUND 2026-09-21 verifying R-598 on guest 9201. `GET /backups` with `felhom_lang=en` returned the **Hungarian** page. That is correct — `langFor` step 2 says a request carrying a session reads the household's saved setting and deliberately ignores the visitor cookie, so a signed-in family never sees a language a previous visitor picked on the sign-in page of the same browser — but it means **the cookie is the right instrument for the anonymous claim/login/bind pages and the wrong one for every page behind auth**, where `?lang=` is. A session that had run only the cookie probe would have concluded R-598 was still open and fixed it a second time. **This is a documentation gap, not a code defect**, and it is the kind that costs a whole session: nothing in `10-localisation.md` §2.2 or in any runbook tells a prober which instrument to use where. **Fix shape:** four lines in `10-localisation.md` §2.2 — a table of surface → language instrument — and a pointer from the live-validation section of the workspace rules. Recorded meanwhile in `audits/i18n-closing-2026-09-21/live/backups-page.md`. | **READY — rank P3-LOW; owner: CC (docs)** | | **R-603** | **[P3-LOW] An English string containing an apostrophe silently never matches on a rendered page, and a `strings.Contains` assertion reads exactly like a missing sentence.** FOUND 2026-09-21 while writing the R-598 render tests. `backup.target.absent` was first written as *"The system backup's drive cannot be reached…"*; `html/template` escapes `'` to `'`, so the page carried the sentence and every assertion for it failed. **The failure mode is the expensive part:** the test said *"the English absent-drive copy never reached the page"*, which is indistinguishable from the handler not being wired — and the obvious next move is to go and re-fix the handler. Reworded to avoid the possessive, and all 23 new English values were then swept for `' " < > &` (zero). **The Hungarian bundle has never hit this** because Hungarian copy uses „quotes" and few apostrophes; **English copy will hit it again.** **Fix shape:** either a bundle gate that refuses an HTML-escapable character in a value destined for a page (and an allow-list for the ones that legitimately need one), or a test helper that compares against `html.EscapeString(want)` so the assertion cannot be fooled. The gate is the better shape — the helper only protects tests that remember to use it. | **READY — rank P3-LOW; owner: CC (controller)** | @@ -791,8 +790,7 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-621** | **[P2-MEDIUM] A held update DESTROYS the evidence of why it failed: `failAndHold` runs `compose down`, the failing containers are removed, and their output is gone before anyone — household, operator or the next session — can read it.** MEASURED 2026-09-21 on guest 9202 on a REAL upstream edge: `adventurelog v0.12.1 → v0.13.0`. The new backend applied **nine Django migrations successfully** and then never listened; the update held after the full 5-minute health wait. **`Manager.failAndHold` (`stacks/update.go:723`) calls `updateCompose(dir, env, "down")`**, which removes the containers rather than stopping them, and nothing captures their logs first. Within seconds the box's own log recorded `Logs result for adventurelog: 0 bytes returned (empty)` and `docker ps -a` held nothing at all. **What survives is the WHAT and not the WHY:** the controller line `update adventurelog FAILED after the new version was started: not healthy: not healthy within 5m0s (last: state unhealthy)` and the household's sentence, both of which say the app did not come up and neither of which says the migrations ran and the server then failed to bind. **This is R-320 ("evidence off the machine before the teardown") as a PRODUCT behaviour rather than a session habit** — the teardown here is the product's own, it is correct to perform (a half-started new version must not keep running), and it happens before anyone can look. **Why it matters beyond a drill:** the hold sentence sends the household to a restore, and after the restore the only remaining question is *should I press Update again?* — which nobody can answer, because the one artefact that would say so no longer exists. It also makes every future held update unreportable to an upstream project. **Fix shape:** capture `compose logs --no-color --tail N` into the stack directory (beside `applied-compose.yml`, which already travels with the stack) IMMEDIATELY before the `down`, and surface it on the app page's hold panel or at least through the existing `/api/stacks//logs` fallback. Bounded size, written once per hold. A test that holds an app and asserts the captured file is non-empty — it fails today. Evidence: `audits/update-night-2026-09-21/apps/adventurelog/why-it-failed.txt`, `state-after-hold.txt`. **FIXED in v0.262.0.** `failAndHold` now writes each service's log into `/hold-logs//compose-logs.txt` (`compose logs --no-color --tail 400`) **before** the `down` that destroys them, best-effort by design: a hold must never fail because its evidence could not be written. This is what `adventurelog` cost — nine migrations ran, the app never bound its port, and the only log that could have said why was gone before anyone looked. | **CLOSED 2026-09-22 — v0.262.0: the hold keeps the app's own log before stopping it** | | **R-622** | **[P2-MEDIUM] `adventurelog v0.13.0` migrates the customer's database and then does not serve — the edge must NOT be promoted, and it is the first real-catalog candidate this project has measured as unsafe.** MEASURED 2026-09-21 on guest 9202 through the product's own guarded Update. `v0.12.1 → v0.13.0` (backend AND frontend together, PostGIS held constant). The backend applied **nine migrations, every one `... OK`** — `adventures.0072_trail_wanderer_author_fields` through `integrations.0009_alter_endurainintegration_auth_method`, plus `billing.0001_initial` — and then the container's own healthcheck failed with `URLError: [Errno 111] Connection refused` on five consecutive checks. The app never bound its port. The update held honestly after the full 5-minute wait. **THE PRODUCT DID EVERYTHING RIGHT AND THAT IS HALF THE FINDING:** the precondition found a Tier-1 copy one minute old, the safety dump was written, the pin advanced BEFORE the pull, the health wait was not short-circuited, the app was stopped rather than left half-running, and the hold sentence named the tier, the date and what the copy holds — „saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza." **This is exactly the case `09` §4 exists for:** the migration RAN, so there is no undo, only a restore — and the restore is the thing slice 4 made sure existed first. **What it needs:** adventurelog stays OFF the promotion list; the cause is not diagnosed here (R-621 is why); and before it is ever promoted the edge should be re-run on the harness with its ABORT step, since an app that migrates and then refuses is the shape most likely to refuse the old image too. Evidence: `audits/update-night-2026-09-21/apps/adventurelog/`. | **READY — rank P2-MEDIUM; owner: CC (catalog); NOT a Felhom defect — an upstream edge that fails** | | **R-623** | **[P3-LOW] The unattended-update caller turned every SUCCESS into a `timeout`, and then refused to press that app again — the instrument, not the box.** FOUND 2026-09-21 (update night) by reading `unattended-caller.py` before relying on it for the Q4 hold measurement. Its `call()` returns the API **envelope** — `{"ok": true, "data": {…}}` — and `follow()` read `update_phase` and `updating` **off the envelope**, where neither exists. Both were therefore always `None`; the end test `not updating and phase in ("done","failed")` could never fire; every followed update ran the full **900-second** timeout and was recorded `timeout`, which the caller treats as terminal and adds to `never_again`. `main()` unwraps `data` for the stack LIST, which is exactly why the within-a-major half of that night worked and this half did not. **The 2026-09-21 run did not catch it because the only pass that reached `follow()` was Scenario F, whose log was lost to a buffering `tail`** — the run's own honestly-recorded instrumentation gap turns out to have hidden a second one underneath it. **This is the R-607 class in the evidence layer rather than the product layer: an instrument that can report a success as a timeout is not a measurement**, and worse, it is a measurement that says the box behaved badly when the box behaved well. **FIXED in the same file 2026-09-21** (unwrap `data`, with the reason written into the docstring so the next reader does not re-derive it), and the fixed caller is what produced tonight's unattended-hold leg. **What it does NOT invalidate:** the G-b no-retry proof, which is entirely in the refusal path and never reached `follow()`. **What it DOES qualify:** any future reading of that night's Scenario F timing — the "51 s – 1 m 26 s" figures come from the ATTENDED scenarios 04/05/07, not from the caller. | **CLOSED 2026-09-21 — fixed in `audits/update-arc-gaps-2026-09-21/unattended-caller.py`** | -| **R-624** | **[P3-LOW] Three of the catalog's apps cannot be seeded by ANY headless route, and for two of them that is a deliberate security decision — so the upgrade harness has a permanent ceiling nobody has written down.** FOUND 2026-09-21 while widening R-462 from 3 apps to . **`vaultwarden` and `zipline` close self-registration ON PURPOSE** — vaultwarden by `SIGNUPS_ALLOWED=false` (R-512, *„a stranger who guesses vault. must not be able to register"*), zipline by answering `E1037: User registration is disabled` — and neither ships a CLI that could make an account instead. So there is no route to a first account without the admin secret, and **that is correct**: the harness must not be the reason a customer-facing app accepts strangers. **`gitea` is a different and fixable case:** the template sets no `INSTALL_LOCK`, so a fresh instance sits in its web-installer state and `gitea admin user create` refuses (`MustInstalled() [F] Unable to load config file for a installed Gitea instance`); POSTing the installer form first would work and was simply not written tonight. **Why this is a row rather than three notes:** `09` §3 decision 6 says the upgrade test goes to **all** apps, and R-462 is costed as if every app is reachable given enough fixture work. It is not. There is a class — *apps whose only account-creating route the catalog deliberately closes* — for which the honest maximum is `inconclusive` unless the harness is given the app's admin secret at deploy time, which is a decision nobody has taken. **Needs:** the class named in R-462's scope so the remaining count is honest; a decision on whether the harness may hold an app's admin secret (it already holds the ones IT generates — see R-619); and, separately and cheaply, a gitea installer-form fixture. Evidence: `audits/update-night-2026-09-21/apps/{vaultwarden,zipline,gitea}/verdict.json`. | **READY — rank P3-LOW; owner: CC (catalog harness)** | -| **R-626** | **[P2-MEDIUM] An app the customer REMOVED came back: the removal returned 200 and deleted the record and the volume, a container was created two seconds later, and Docker's restart policy has kept it running ever since — while the box reports the app as not installed.** FOUND 2026-09-21 during the update night's TEARDOWN, which is the only reason it was found at all. `navidrome` was removed through the product: the `remove_hdd_data:true` call was correctly refused `409` (R-442's fail-closed guard — the drive path could not be resolved on this guest), and the `remove_hdd_data:false` call returned **200** with `volumes_removed: ['navidrome_navidrome_data']`. **Two seconds later a container carrying `com.docker.compose.project=navidrome` was CREATED** (`.Created = 19:12:20Z`), and Docker's `restart: unless-stopped` started it again at the next guest boot (`.StartedAt = 20:19:33Z`, the B5 power cut). Seven hours later: **`app.yaml` absent, `deployed=false`, one volume back, and the controller happily probing it — `Health probe navidrome: API GET :4533/ping → 200`.** **The customer-visible shape is the bad one:** *"I deleted that app and it came back"* — and it came back **blank**, because the volume really was deleted, so it looks installed and is empty. It is also invisible to every sweep that keys on `deployed`, which is exactly why the teardown found it and nothing else did. **WHAT IS NOT ESTABLISHED, and is stated rather than guessed: what created the container.** The controller was restarted several times later in the night and its log no longer reaches that moment — **the second time in one night that a restart destroyed the evidence of the thing that mattered** (see R-621). **Needs:** reproduce with a loop that removes an app and watches `docker events` for 60 s, so the creating path is a NAME and not an inference; then a test that removes an app, reboots, and asserts no container with that compose project exists. **And one instrument lesson worth keeping:** this session's own post-remove check queried the compose-project label and reported clean at 21:12:18 — two seconds before the container appeared. A check that runs once, immediately, cannot see a thing that is created immediately after it. Evidence: `audits/update-night-2026-09-21/26-removed-app-came-back.txt`. | **READY — rank P2-MEDIUM; owner: CC (controller)** | +| **R-624** | **[P3-LOW] Three of the catalog's apps cannot be seeded by ANY headless route, and for two of them that is a deliberate security decision — so the upgrade harness has a permanent ceiling nobody has written down.** FOUND 2026-09-21 while widening R-462 from 3 apps to . **`vaultwarden` and `zipline` close self-registration ON PURPOSE** — vaultwarden by `SIGNUPS_ALLOWED=false` (R-512, *„a stranger who guesses vault. must not be able to register"*), zipline by answering `E1037: User registration is disabled` — and neither ships a CLI that could make an account instead. So there is no route to a first account without the admin secret, and **that is correct**: the harness must not be the reason a customer-facing app accepts strangers. **`gitea` is a different and fixable case:** the template sets no `INSTALL_LOCK`, so a fresh instance sits in its web-installer state and `gitea admin user create` refuses (`MustInstalled() [F] Unable to load config file for a installed Gitea instance`); POSTing the installer form first would work and was simply not written tonight. **Why this is a row rather than three notes:** `09` §3 decision 6 says the upgrade test goes to **all** apps, and R-462 is costed as if every app is reachable given enough fixture work. It is not. There is a class — *apps whose only account-creating route the catalog deliberately closes* — for which the honest maximum is `inconclusive` unless the harness is given the app's admin secret at deploy time, which is a decision nobody has taken. **Needs:** the class named in R-462's scope so the remaining count is honest; a decision on whether the harness may hold an app's admin secret (it already holds the ones IT generates — see R-619); and, separately and cheaply, a gitea installer-form fixture. Evidence: `audits/update-night-2026-09-21/apps/{vaultwarden,zipline,gitea}/verdict.json`. **— NIGHT 2026-09-23:** `gitea` stayed inconclusive on both venues (its installer); `vaultwarden`/`zipline` not attempted (closed sign-up, by design); `code-server`, `outline`, `rallly` have no front-door seed route (listed, not moved); `bentopdf`, `glance`, `crafty-controller`, `wger`, `wanderer` (meilisearch) and `uptime-kuma` have no fixture tonight (listed, not moved). | **READY — rank P3-LOW; owner: CC (catalog harness)** | | **R-628** | **[P2-MEDIUM] An empty search of a mailbox I do not control was turned into a claim about what a THIRD PARTY had done, and it went into the register as fact.** FOUND 2026-09-22; the operator caught it within minutes by producing the thread. The `due_checks_gate` fired R-433 (*have Hetzner answered?*). Two searches of the felhom catch-all came back empty, and the emptiness was written into R-433 as **"Hetzner has not answered"** and **"there is no evidence the tickets were ever opened"**. Both were false: ticket **#2026090103040671** had been opened and answered. **THE PRECISE FAULT IS THE INFERENCE, NOT THE QUERY — and that distinction is the whole value of this row.** Re-run afterwards WITH a positive control: `from:monitoring@felhom.eu` returns **201 threads**, `in:anywhere … includeTrash` reaches SENT, TRASH and mail back to January — **the instrument works.** And the exact ticket number, the exact subject `Storage Box issue`, and `from:hetzner` each still return **nothing**. So the literal finding — *this correspondence is not in this mailbox* — was CORRECT. What was invented was the step from there to *Hetzner has not answered* and *the tickets were never opened*. **A mailbox I can read is not the only place a reply can be**, and the operator's own account is exactly where a support ticket he opened would land. An absent record in ONE place can never answer a question about what SOMEONE ELSE did. **This is R-96 rule 3 in a new surface, one step further out than R-607**: there the instrument reported a stale value as current; here a correct observation was promoted to a conclusion it could not carry. **It is sharper still because the same session, the night before, gave every fixture a negative control and every gate a red-proof — and then reached for a search with neither.** **THE RULE, in one line:** *an empty search may be reported as "absent from the place I looked", never as "it did not happen" — and only after a control query that MUST hit has been seen to hit.* **Done:** the rule is written into the Gmail-access memory, where the next session meets it before it searches rather than after. | **CLOSED 2026-09-22 — rule recorded; R-433 corrected with the real answers** | | **R-627** | **[P2-MEDIUM] Nothing checked that the register is a well-formed table, so an append that ate two rows' state cells went unnoticed until a person read the file — and one row had been broken the same way for 45 days.** FOUND 2026-09-22. The 2026-09-21 update night appended measured results to eight rows with a regex that matched each row's trailing state cell; on **R-446** and **R-458** it consumed the cell and did not restore it, the cell reappeared as a stray FOURTH cell on a DUPLICATED copy of **R-626** and **R-625**, and a blank line was left between each pair. The register then reported **317 rows for 315 findings**, two rows carried no state at all, and two findings existed twice with contradictory state cells. **Nothing caught it:** `one_register_gate.py` compares this file against ROADMAP and `closed_register_gate.py` forbids an id in BOTH files — neither asks whether the file is a well-formed table, and neither notices an id duplicated WITHIN it. **THE RED-PROOF THEN FOUND AN OLDER INSTANCE NOBODY HAD SEEN: R-254 lost its state cell on 2026-08-08 (commit `59527d0`) and had rendered without a State column for 45 days.** **Closed the same day:** `scripts/register_shape_gate.py`, registered in `repo_gates.py` as gate 14 and reached by the pre-push hook, refusing a row that does not end with `|` (an eaten state cell), a duplicated id, or a blank line splitting the table; four decoys in `test_gate_decoys.py` — three convicting on the exact damage shapes and one asserting a healthy register still passes. **TWO THINGS THE RED-PROOF CORRECTED IN THE GATE ITSELF, kept because they are the finding's real content:** a first draft counted CELLS and convicted **125 innocent rows** — register cells carry literal `|` inside prose and shell snippets (`owner: CC | …`), so a row cannot be split on `|`, and a count that cannot be computed is not a check; and it skipped malformed rows before counting ids, reporting **5 duplicates where there were 2**. **Repaired:** both state cells restored from the stray cells that carried them, the two duplicate rows deleted, R-254's verdict sentence given its cell back, and **15 blank lines that split the register into 12 separate markdown tables** removed — every row's text byte-identical afterwards, proven by diff. **And the gate immediately earned itself:** the very next row inserted in this session (R-628) left a blank line behind and the gate refused it. | **CLOSED 2026-09-22 — gate 14, four decoys, register repaired 317→315** | | **R-629** | **[P2-MEDIUM] The drill catalog sent the operator 47 CI-failure alarms in one night, and the drill method that created it did not mention CI at all.** FOUND 2026-09-22 while checking a different mailbox question — which is the only reason it was found. `admin/app-catalog-drill` was created on 2026-09-21 by `POST /api/v1/repos/migrate` from the live catalog, and a migrated repo inherits `has_actions: true`. Every drill push therefore ran the catalog's CI workflow, which failed immediately (the drill repo carries the workflow but the run has no meaningful gate context), and **each failure mailed `admin@felhom.eu`**: *"[felhom CI] gates FAILED in admin/app-catalog-drill"*. **47 runs, 47 alarms, all overnight, all unread and flagged IMPORTANT.** **WHY THIS MATTERS MORE THAN THE NOISE:** that mailbox is the operator's alarm channel, and R-168 made CI mail the thing that notices a bypassed gate. A night of throwaway failures from a repo nobody must ever act on trains the reader to skim exactly the sender that must never be skimmed — and it did it on the night the same mailbox was also carrying real `offsite_snapshots_dropped` and `offsite_proof_empty` alarms. **The drill method wrote down the fences it needed** — private repo, no customer box may follow it, reset at teardown — **and said nothing about CI, because nobody had run a drill repo through a CI-enabled Gitea before.** **FIXED 2026-09-22:** `has_actions` set to **false** on the drill repo (verified by re-reading the repo: `has_actions: False, private: True`), and `09` §6.5 now carries it as a step of creating a drill repo rather than as a thing to notice afterwards. **What is NOT done:** the 47 mails are still in the operator's inbox, unread — deleting another person's mail is not mine to do, and they are named here so they can be cleared in one search: `subject:"gates FAILED in admin/app-catalog-drill"`. | **CLOSED 2026-09-22 — actions disabled, method updated; the 47 mails are the operator's to clear** | @@ -802,11 +800,19 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-633** | **[P2-MEDIUM] A remove sent while a restore is still running reports success, deletes the app's record, and leaves a container restarting forever with a live public route.** MEASURED 2026-09-22 on guest 9202, controller v0.261.0, during the twenty-eight walk. `gokapi` was restored from its own local copy at 11:34:07 and removed at 11:34:22. `POST /backup/restore` answers **302 and works in the background**; the remove tore down what existed and the restore's own `compose up` then RE-CREATED the container at **11:34:24**. **Both calls returned success.** Twenty-five minutes later: `GET /api/stacks/gokapi` reads **`deployed: false`**, and `docker ps -a` shows `gokapi` **`Restarting (1)`** with `RestartCount` climbing, carrying its full traefik label set — including `traefik.http.routers.gokapi.rule: Host(`.enkisfelhom.hu`)`, **a rule with an empty subdomain**, because the deploy values that filled it were deleted with the app. Its own log loops *„Salt for admin password invalid, generating new salt… password does not appear to be a SHA-1 hash"* — the volume holding its config was removed correctly, so the binary can never start. **WHY THIS IS A ROW AND NOT A HARNESS ARTEFACT:** a household can press exactly these two buttons in exactly this order, the product accepted both, and **the remove reported success while leaving the orphan**. Presence of a success message is not evidence of a result. **WHAT IT COSTS:** an app the household believes is gone keeps a container in a restart loop, keeps a route registered on the public reverse proxy, and is invisible to every product surface because the controller no longer records the stack. Nothing in the alarm ladder fires: `08` §4 keys on stacks the controller KNOWS about. **This is R-626's class with the mechanism finally visible** — that row recorded a removed `navidrome` coming back and could not diagnose it because the controller had restarted; here the window is 17 seconds and both halves are in the evidence. **Needs:** the remove path to refuse, or to wait, while a restore for the same stack is in flight — and, either way, to verify the teardown rather than report success without looking. Evidence: `audits/the-28-2026-09-22/apps/gokapi/came-back-evidence.txt`, `apps/gokapi/log.txt`. **FIXED in v0.262.0, both halves.** **(a) The guard the product already had everywhere else:** `RemoveStack` now consults `UpdateGuards.Busy` (backup single-flight, restore status, app-data holds) and `IsUpdating`, and refuses with the app's own sentence in both languages. The product refused exactly this clash for `update` and for `restore` — the restore refusal even NAMES the blocking app — and `remove` was the one door with no lock on it. **(b) `down` returning 0 is a request, not a result:** the compose project is now watched for 25 s after `down`, anything carrying its label is removed BY NAME with its labels logged, and the API answer carries `verified: true/false`. That is the instrument R-626 asked for, in the product rather than in a drill script. **PROVEN LIVE ON 9202, and the live proof found a defect a test had not.** The guard fires: with `privatebin` STOPPED (so the pre-existing "still running" check could not answer first) and a box-wide backup in flight, the remove was refused and the controller said `RemoveStack privatebin REFUSED (busy): a backup or restore is running (single-flight held)`, with the household's own sentence *„Az alkalmazáson mentés vagy visszaállítás fut. Várd meg, amíg befejeződik.”* **But it answered HTTP 500.** `router.go`'s status mapping greps the error TEXT for `not deployed` / `still running` / `not found` / `protected`, and the busy sentence contains none of them, so it fell through to the default. **A 500 tells the UI something broke; this is “wait a moment”.** Fixed in **v0.262.1**: a typed `*stacks.RemoveBusyError` matched with `errors.As`, answered **409**, carrying both the Hungarian bytes and the key — and its test asserts the sentence contains none of the words the text mapping greps for, so the type is load-bearing rather than decorative. **The verification half is proven too:** a permitted remove answered `verified: true` with no reappearance, and no container carrying the project label existed 60 s later. **AND A FIRST ATTEMPT THAT PROVED NOTHING, recorded because it nearly went down as a pass:** the first run refused the remove with `409 still running — stop it first`, which is the PRE-EXISTING running check, not this guard — the restore had already finished by then. A refusal from the wrong rule is not evidence for the new one. | **CLOSED 2026-09-22 — v0.262.0 + v0.262.1; refusal and verification both proven live, and the live proof corrected the status code — **re-proven on v0.262.1: HTTP 409 with the household's sentence**, `RemoveStack privatebin REFUSED (busy)`** | | **R-635** | **[P1-HIGH] `romm 5.3.0` does not fit the memory the template gives it, and the guarded Update called that a success — the app has been OOM-crash-looping on demo-hp for six hours at ~500% CPU.** FOUND 2026-09-22 17:37 **because the operator heard the fans**, which is the only reason it was found at all. `romm` was promoted `5.0.0 -> 5.3.0` on the live catalog that morning (`audits/PROBE-FIX-2026-09-22.md`) after the edge was PROVEN on scratch guest 9202, and the guarded Update was then pressed on demo-hp guest 9201 at **09:08:22Z**, reaching **`done` in 74.8 s** with the app `running`. **It ran clean for two hours.** The first worker kill is at **11:09:20Z**; by 15:38Z there had been **4,530** of them — `Worker (pid:…) was sent SIGKILL! Perhaps out of memory?` — with `docker inspect` reading **`OOMKilled: true`**, the container pinned at **457 MiB of its 512 MiB limit**, and `docker stats` showing **499.51% CPU**. The host's load average sat at **5.2 while otherwise idle**. `rq_cron` is killed and restarted every few seconds in a permanent storm. **THREE THINGS THIS ESTABLISHES, and the third is the one that changes how promotions are judged.** (1) The template's own comment says *`RAM: ~300MB (mem_limit: 1024M total — romm 512M + mariadb 384M + redis 128M)`*; **5.3.0 needs more than 512M and the template was not re-sized when the version moved.** A version move is not only an `image:` line. (2) **The update reported `done` and the app reads `running`**, because nginx answers `GET /` with 200 while the gunicorn workers behind it are being killed — a THIRD variant of the R-618/R-630 theme: the probe is right, the port is right, and the answer is still a false green. (3) **THE ALARM DID FIRE, AND MY FIRST WRITE-UP OF THIS ROW SAID IT DID NOT — CORRECTED 2026-09-22 BY THE OPERATOR, WHO PRODUCED THE MAILS.** The controller HAS an OOM detector (`main.go:821`, `[deadapp] romm: container romm was OOM-killed`, evaluated every 30 s), it emits **`app_oom`** (`notifier.go:726`), the hub allow-lists it (`dispatcher.go:636`) and dispatched it to the OPERATOR channel as **SENT** — `admin@felhom.eu` received **`[Felhom] ⚠️ demo-hp: app_oom`** at **11:09 CEST** and again at **17:48 CEST**, each carrying the app name, the Hungarian sentence and the dashboard link. The CUSTOMER channel is **SKIPPED**, which is correct. **I asserted an absence without looking at the instrument** — the hub's own Events and Notifications tabs show all of it — and I did it by reasoning from a memory note (`lxc-docker-oom-signals-unreliable`, R-528) instead of reading the hub. **That is R-628's shape exactly, four days on, and from the same hand.** **WHAT IS ACTUALLY WRONG, and it is narrower and real:** `notifier.go:715-726` keys the alarm on `container|startedAt` in an `oomSeen` map and emits **once per container lifetime**. So **six hours of continuous thrashing — 4,530 worker kills — produced exactly ONE mail**, severity `warning`, never escalating, while the app went on reading `running`. **A six-hour storm is indistinguishable from a single transient kill.** The hub's App Telemetry panel did carry the magnitude (RomM: **5,023 errors, 632 warnings**, peak 855 MB) but nothing turns that into a second, louder signal. So the fix worth having is not a detector — there is one — but an ESCALATION: a warning that repeats for hours should stop looking like a warning that happened once. **AND THE LESSON FOR R-462's METHOD, which is the real cost:** every `proven` verdict in the update night and in the twenty-eight measures the app for the **minutes of the walk**, not for a day of running. `romm` passed its walk, was seeded, read back and restored — and broke two hours later. **`proven` currently means "the update applied and the data survived", NOT "the new version runs".** Needs: decide between rolling the catalog back to 5.0.0 and raising romm's `mem_limit` (measured, not guessed); and a soak longer than a walk before any future promotion. **FIXED AND MEASURED 2026-09-22, in two steps, and the FIRST step was still a guess.** *Step 1 (operator's choice):* the limit was raised 512M → **768M** (`app-catalog-felhom.eu@886956d`). It slowed the kills from ~12/min to ~7/min and **stopped nothing** — 37 SIGKILLs in five minutes, `OOMKilled` still true, and the cgroup's own `memory.peak` read **exactly 768 MiB**: it hit the new ceiling and died there. *Step 2, from a MEASUREMENT instead:* the per-process RSS inside the container reads **~216 MiB per warm uvicorn worker**, so the image's default of four workers plus the master needs **~882 MiB** before nginx and the job runner — more than any sensible limit for this box. **The lever was in the image all along:** `/init:143` runs `--workers "${WEB_SERVER_CONCURRENCY:-4}"`. **Four workers is a SERVER default on an appliance serving one household.** Setting **`WEB_SERVER_CONCURRENCY=2`** (`app-catalog-felhom.eu@f4eb94f`, limit left at 768M) and applying it through the product's own Update button fixed it. **PROVEN UNDER LOAD, not just at idle:** 6 concurrent callers driven at romm through the household's own route for 300 s — **26,645 requests** (9,687 × 200, 16,958 × 401 on the auth-gated endpoints), CPU a steady **~200%** (exactly two workers saturated, by design), memory oscillating **416–614 MiB against the 768 MiB limit and trending DOWN**, and **zero** SIGKILLs, `OOMKilled: false`, `RestartCount: 0` throughout. At idle afterwards: **1.64% CPU**, 610 MiB, host load falling from 5.2 to 2.4. **AND THE FIRST SOAK MEASURED NOTHING, which is worth more than the second one:** it was pointed at `arcade.enkisfelhom.hu` — the scratch-guest fixture's default subdomain — while this box deployed romm at `jatek`. Every request 404'd at traefik in 9 ms, romm sat idle at 0.64% CPU, and the counter cheerfully reported **14,026 successful requests**. It was caught only because 0.64% CPU under load is not believable. **A positive AND a negative control are now asserted before any load is driven** (`jatek` must not 404; a nonsense host must). R-96 rule 3, in a new surface. **WHAT STAYS OPEN, and it is the part that outlives romm:** 610 MiB of 768 MiB is **79%** — it works with ~158 MiB of headroom and the soak never exceeded 614 MiB, but it is not generous, and nothing watches it. **And the method lesson for R-462:** a version move is not only an `image:` line — the new version's SHAPE (worker counts, per-worker footprint) has to be measured too, and a walk lasting minutes cannot see a ceiling reached in two hours. Every `proven` verdict in the update night and in the twenty-eight means *"the update applied and the data survived"*, **not** *"the new version runs"*. Evidence: `audits/probe-fix-2026-09-22/romm-soak.json`, `romm-soak.out`. | **CLOSED 2026-09-22 — two workers, 768M, proven under 26,645 requests; the 79% headroom and the `proven`-means-minutes lesson are carried into R-462** | | **R-638** | **[P2-MEDIUM] The product's database loader cannot replay a copy over a NEWER schema: on PostgreSQL it FAILS, on MariaDB it leaves the newer version's tables behind.** MEASURED 2026-09-23 on 9202. `ImportDump` (`appbackup/dbdump.go:719`, `psql -v ON_ERROR_STOP=1 --single-transaction`) replays a `pg_dump --clean --if-exists` file over the live database. After docmost 0.95.0 → 0.96.0 migrated, the replay of the pre-update copy was refused in 0.40 s, rc 3: *cannot drop constraint workspaces_pkey on table public.workspaces because other objects depend on it / DETAIL: constraint oauth_clients_workspace_id_fkey …* — the new version created six tables whose foreign keys point at old ones, and `--clean` only drops what the dump knows. Database unchanged (the transaction rolled back). On MariaDB (`mariadb-dump`, `FOREIGN_KEY_CHECKS=0`) the same replay after romm 5.0.0 → 5.3.0 returned rc 0 in 1.25 s and left **12 base tables** of the new version behind; RomM 5.0.0 happened to ignore them. **What worked:** `DROP SCHEMA public CASCADE; CREATE SCHEMA public;` + the dump in ONE transaction — rc 0 in 1.38 s, every table, index and extension back. **Why this is a row of its own and not only part of R-637:** the SAME loader backs shipped paths — `rollbackSafetyDump` (off-site restore's undo) and the dump replay of the restores — so **any restore of a copy taken BEFORE an update that migrated, replayed over the migrated database, may fail the same way. NOT MEASURED:** whether the unit restore the hold sentence names does this (it also carries the data VOLUME tar, which may make the replay moot). That is the measurement owed, on 9202, before anyone relies on it. Evidence: `audits/update-rulings-2026-09-23/README.md` Part 1, `docmost-45`, `romm-44`. **-- NARROWED 2026-09-23:** the undo no longer touches this loader — it copies folders (decision 19, controller v0.263.0). **What stays open is the part about SHIPPED paths:** `rollbackSafetyDump` and the restores' dump replay still replay over whatever schema is live, and whether the unit restore the hold sentence names works after a real schema migration is STILL UNMEASURED. | **OPEN — P2, narrowed to the restore paths; owner: CC; measure the named restore after a real schema migration first** | -| **R-640** | **[P2-MEDIUM] A TRUNCATED PostgreSQL copy loads with exit 0 into an EMPTY database — and `ValidateDump` would accept it.** MEASURED 2026-09-23 on 9202 in a scratch database beside docmost's: the first half of the 135 816-byte undo copy, loaded with `psql -v ON_ERROR_STOP=1 --single-transaction`, returned **rc 0** and left **42 tables and 48 migration rows — and 0 users, 0 spaces, 0 constraints, 0 indexes**; psql treats end-of-file inside a `COPY` as end of data and commits. The whole copy ends with `-- PostgreSQL database dump complete` (plus a `\unrestrict` line); the truncated one does not. `ValidateDump` (`appbackup/dbdump.go:415`, read from source) checks the header and a `CREATE TABLE` only. **An undo or a restore fed a truncated copy would report success, start the app on an empty database, and pass a health check.** MariaDB's truncated load failed (rc 1) but NON-atomically — half the tables already replaced; its copy carries `-- Dump completed`. Fix: check the engine's completion marker before any load. Evidence: `audits/update-rulings-2026-09-23/README.md`, `docmost-46`, `docmost-47`, `romm-44`. **-- NARROWED 2026-09-23:** the undo is not exposed (a folder copy, validated by its own finished-marker). The restore paths still load dumps through `ValidateDump`, which does not check the engine's completion marker. | **OPEN — P2, narrowed to the restore paths; owner: CC** | | **R-643** | **[P2-MEDIUM] The ruled chain leaves the automatic update leg AT MOST 15 MINUTES a night.** FOUND 2026-09-23 while writing the build plan for `09` §3 decision 11 (*updates after the off-site copy, before the full-system backup*). The off-site leg starts at W+105m (`cmd/controller/main.go:961`) and the full-system backup's gate opens at W+2h (`quiesce/quiesce.go:656`, span to W+6h); the legs are clock-scheduled, not chained. One step takes ~1 min when it works and ~2–6 min when it fails and is undone. Options and the recommendation (the full-system backup waits for the leg inside its own window; the leg stops starting steps at W+5h) are in `09` §6.4. **-- RULED 2026-09-23 (`09` §3 decision 20):** the full-system backup waits for the update leg inside its own window; the leg stops starting new steps at W+5h. Built with `09` §6.4 part 7. | **RULED — build with §6.4 part 7; owner: CC** | | **R-644** | **[P3-LOW] `gokapi` on scratch guest 9202 is crash-looping — 329 restarts by 2026-09-23 07:51 UTC, *password does not appear to be a SHA-1 hash* — and the controller still lists it deployed.** OBSERVED at the start of the 2026-09-23 session, not caused by it. The twenty-eight walk's teardown (2026-09-22) removed a `gokapi` container left by R-633 by name; a `gokapi` is running again, recorded `deployed: true`. Not investigated (scope). Likely the R-633/R-634 shape — a restore-then-remove race leaving a record — and a scratch-box fact, not a customer one; filed so the next drill does not read it as its own. | **OPEN — P3; owner: CC; investigate before the next drill on 9202** | | **R-645** | **[P3-LOW] Lifting an update hold by the operator CLI lets the recovery unit be re-captured with the FAILED new definition within seconds — the copy the hold sentence names is overwritten.** MEASURED 2026-09-23 on 9202 during the undo bake-off: docmost was held at 07:59:16Z after a failed 0.95.0 → 0.96.0 update; `--clear-restore-hold docmost` + the controller restart it requires ran at ~07:59:23Z, and at **07:59:26Z** the controller logged *Recovery unit captured for docmost* — the unit's `compose/docker-compose.yml` now named `docmost/docmost:0.96.0`, the version that had just failed. The hold sentence had pointed the household at that unit („saját meghajtó, … 09:55"). The hold is what keeps the nightly legs off a held app (`isHeld`, v0.238.1); once it is lifted by hand, the checksum-gated refresh sees a changed definition and captures it. **Who it hits:** an operator who lifts a hold to inspect or repair, before restoring. With the undo (R-637) a failed update no longer holds unless the undo also fails, so the path is rarer — it does not go away. Candidate shapes, none chosen: the CLI refuses to lift an UPDATE hold (only a restore lifts it); or the lift also puts the pin back; or the capture skips an app whose pin is not what it is running. Evidence: `audits/undo-bakeoff-2026-09-23/docmost-40-undoF.txt` (the invalid run) and README §"Three things". **-- 2026-09-23 (controller v0.263.0):** the undo never reads the unit, so this no longer affects the automatic undo; it still affects an operator who lifts a hold by hand before restoring. | **OPEN — P3; owner: CC** | -| **R-650** | **[P3-LOW] A controller unit test that falls through to the REAL docker acts on the build host — DooPlex.** MEASURED 2026-09-23: my own first draft of `TestR634_VolumeLegNeverStopsADeployingApp` drove the real `DumpAppVolumesSafe` for its control case, and `docker run … tar` CREATED an empty volume `outline_outline_data` on DooPlex (removed by name, verified empty and unused; nothing else touched). The same draft of the stacks test ran a real `docker compose down` from a temp dir named `nextcloud` (no such project existed). Both tests now use seams (`dumpVolumesSafe`; `composeCmd` + an empty PATH). **The class is unguarded:** any test that reaches `exec.Command("docker", …)` on DooPlex acts on production Docker. Fix shape: a test-binary guard in `composeExec`/`execCommand` that refuses a real docker exec when running under `go test` unless an explicit env opt-in is set, plus a decoy test that proves it refuses. | **OPEN — P3; owner: CC (controller)** | +| **R-651** | **[P3-LOW] A removed app leaves `applied-compose.yml` and `applied-meta/` behind in its stack directory.** MEASURED 2026-09-23 night on 9202 (v0.266.0, R-626's run): after `POST /api/stacks/navidrome/remove` answered 200 with `verified: true`, `/opt/docker/stacks/navidrome/` still held `applied-compose.yml` and `applied-meta/` written by the deploy (the directory itself is the catalog mirror and belongs there). Seen again after every remove of the night (`after remove: leftovers='/opt/docker/stacks/'`). **Consequence not measured:** whether a later reinstall of the same app reads a stale `applied-meta/` (the undo's OLD `.felhom.yml`, v0.263.2) before its own deploy overwrites it. **Needs:** the remove deletes both; a test that removes, reinstalls, fails an update and asserts the undo uses the reinstall's file. Evidence: `audits/night-2026-09-23/A3-r626-verdict.json`. | **READY — P3; owner: CC (controller)** | +| **R-652** | **[P3-LOW] The memory watch counted the kernel's file cache as the app's memory.** MEASURED 2026-09-23 night on the bench: nextcloud 34.0.4 read **100 %** of its 1 GiB and immich's PostgreSQL **100 %**, each with **0** kernel `oom_kill`s — `memory.peak` includes page cache, which the kernel drops before it kills anything. Under the watch as built (R-635 follow-up) both would be marked `memory_tight`, and the gate would demand a raised `mem_limit` — a customer-box capacity figure — for cache. **Done the same night (09 §3 decision 22, CC — operator may reverse):** the watch samples the app's own memory (`anon` of `memory.stat`) every 15 s; the mark and the ladder's `memory_peak_pct` read it where measured; the cgroup peak stays beside it (`memory_cgroup_peak_pct`). **Open:** romm's backfilled entry carries M1's 80.9 % cgroup peak (measured before the anon sample existed) — re-measure it on its next step; and decide whether an app whose anon is low but whose cgroup stays pinned at its limit (cache thrash) should be marked at all. Evidence: `audits/night-2026-09-23/apps/nextcloud/bench-1024M/`, `apps/immich/bench-noanon/`. | **READY — P3; owner: CC (catalog harness)** | +| **R-653** | **[P3-LOW] The memory watch's load did not reach box-fixture apps: every request errored.** MEASURED 2026-09-23 night: the load generator followed each app's redirect to `https://.gate.invalid`, which does not resolve, so ghost's and nextcloud's first ten-minute watches recorded **11 797 and 9 427 requests, every one `err`** — memory measured, but idle. Fixed the same night in `upgrade-test.py` (no redirects; the app's own Host header): ghost's re-run served **10 538 × 200**. **Why a row though fixed:** the harness wrote `proven` over a watch with zero effective load and nothing flagged it. **Needs:** the watch reports INCONCLUSIVE when fewer than half its requests reached the app. Evidence: `apps/ghost/bench-noload/`, `apps/ghost/bench/`. | **READY — P3; owner: CC (catalog harness)** | +| **R-654** | **[P3-LOW] opengist 1.15 moved every page under `/-/` — a household's `/login` bookmark answers 404 after the update.** MEASURED 2026-09-23 night: 1.13 serves `/login`, `/register`, `/all`; 1.15.2 answers **404** on all three and serves `/-/login`, `/-/register`, `/-/all`; `/` redirects to `/-/all`. The app, its data and its probe (`/healthcheck`) are fine, and a household arriving at the root lands correctly — only a deep link breaks. 1.15 also marks its session cookie `Secure`. **Needs:** a line in opengist's `app_info` if the operator wants households told; nothing in the product. Evidence: `apps/opengist-oldfixture/`, `apps/opengist/`. | **READY — P3; owner: operator (copy decision) / CC (writes it)** | +| **R-655** | **[P2-MEDIUM] adventurelog v0.13.0 cannot become healthy in the catalog's template — its update is undone on every box.** MEASURED 2026-09-23 night, both venues. v0.13.0's FRONTEND image adds its own `HEALTHCHECK` (`node -e fetch('http://127.0.0.1:3000/health')`), and `/health` answers **503 `{"ok":false,"backend":"unreachable"}`** unless the backend's `/health/` answers OK to the frontend's own request (read from the image's `django-proxy` chunk: `fetch(${getServerEndpoint()}/health/)`). On the bench the backend served `/api/` 200, the seed read back, the migration ran (6 lines) — and the frontend stayed `unhealthy` for 420 s; on 9202 the guarded Update went `verifying` → **`undoing` → `undone`** (under the drill's 90 s timeout). **MEASURED LATER THE SAME NIGHT — two causes, and the first hypothesis was wrong.** (1) The backend's `/health/` answers `200 {"ok": true}` to the frontend (read on a fresh v0.13.0 install); the unhealthy frontend is **our template's own healthcheck override**, `["CMD", "/nodejs/bin/node", …]` — v0.12.1's distroless image keeps node there, v0.13.0 moved it to `/usr/bin/node`, and Docker's health log reads `exec: "/nodejs/bin/node": no such file or directory` 19 times in a row. (2) With the override removed, a second bench run hit a harder wall: v0.13.0's backend runs `download-countries` at EVERY start, fetching world data from the internet, and a cut-short download (`ijson.common.IncompleteJSONError: Incomplete JSON content`) crash-loops the entrypoint — the backend never became healthy in 15 min. The first bench run's download had succeeded. **So v0.13.0's boot depends on an outside download, and whether an update succeeds depends on it too.** **The catalog did NOT move adventurelog.** **Needs:** the override dropped (or pointed at `/usr/bin/node`) in the SAME commit as the image move; and a measured answer on whether `download-countries` can be skipped or pre-seeded (an env switch, or the data in the volume) before the edge is re-proven on both venues. Evidence: `audits/night-2026-09-23/apps/adventurelog/`. | **READY — P2; owner: CC (catalog)** | +| **R-656** | **[P3-LOW] The bench keeps an app's drive folder between runs, so a re-run of the same app starts on the last run's files.** MEASURED 2026-09-23 night: nextcloud's second bench run never installed (`occ status: installed: false` for 5 min) because `/srv/felhom-gate/hdd/appdata/nextcloud` still held the first run's `config/` and data — `compose down -v` removes named volumes, not bind-mounted host folders. Cleared by hand and re-run. Other re-runs tonight (romm's engine step after its app step, komga, immich) reused their drive folders too; their seeds live in databases in named volumes, so their verdicts stand, but the class is unguarded. **Needs:** the bench clears the app's scratch drive folder before FROM (and says so), or runs each edge under a fresh folder. | **READY — P3; owner: CC (catalog harness)** | +| **R-657** | **[P2-MEDIUM] "Remove the app, keep my data", then install it again: nextcloud never installs, and the box only says „unhealthy".** MEASURED 2026-09-23 night on 9202 (v0.267.0): nextcloud was removed through the product keeping its drive folder (the remove with data was refused — R-442's fail-closed guard, as on every drill on 9202 — and the product's keep-data remove taken). An hour later a fresh install of nextcloud on the same box: the template binds `${HDD_PATH}/appdata/nextcloud` to `/var/www/html/data`, the kept folder still holds `admin/`, `appdata_*`, `.ncdata` and a 145 MB `nextcloud.log`, and the image's installer loops **„Login is invalid because files already exist for this user — Retrying install..."**; `occ status` reads `installed: false`. The controller records the deploy as done and the app as `unhealthy`; nothing tells the household that their kept files are what blocks the new install, or what to do. **Why it matters:** keep-data is the choice the product OFFERS a household at remove time — and for nextcloud the kept data makes the app uninstallable. **Needs:** decide the product's promise for a reinstall over kept data, per app class (adopt the data? refuse with a sentence? offer to move it aside?); at minimum a deploy-time refusal or warning when the app's drive folder is not empty. Evidence: `audits/night-2026-09-23/chaos/00-nextcloud-reinstall-over-kept-data.txt`. | **READY — P2; owner: operator (the promise) / CC (the build)** | +| **R-658** | **[P1-HIGH] After a restore, an app's volumes lose their compose labels — so the next update's automatic undo copies NOTHING.** FOUND 2026-09-23 night by the chaos hour on 9202 (v0.267.0). The unit restore recreates each named volume with a bare `docker volume create ` (`internal/backup/restore.go:154`), which carries no `com.docker.compose.project` label; the undo selects the volumes it copies by exactly that label (`internal/stacks/undo.go:156`, `ProjectVolumes`). **Measured, with a control:** the three apps restored in the chaos hour (docmost, navidrome, vikunja) have **0** labeled volumes of 3 / 2 / 2; the three never restored (romm, nextcloud, adventurelog) have all of theirs labeled. **Consequence, seen live in round 9:** vikunja's broken update logged `the undo copy will hold 0 named volume(s), 0.0 MiB`, failed health, and was „UNDONE" — the OLD binary put back on the NEW version's migrated data, with no copy restored. vikunja happens to start on migrated data (the morning spike), so it read back; **an app whose old version refuses migrated data would HOLD, and one that starts but mis-reads it would run on a database the old code does not understand, reported as a successful undo.** The same label filter is used by the remove path (`delete.go:994`, `projectVolumes`) — **measured at teardown:** the remove of the three restored apps DID delete their volumes (compose's `down --volumes` finds them by name) but answered `volumes_removed: []` — the report is wrong, not the act. **Needs (not fixed tonight — one controller release per session):** the restore creates volumes WITH the project's labels (or `compose create` does it); the undo and the remove resolve an app's volumes from its compose file's declared names, not only the label; a test that restores, then fails an update, and asserts the undo copied every declared volume. Evidence: `audits/night-2026-09-23/chaos/round-09.*`, `chaos/09-volumes-unlabeled-after-restore.txt`. | **READY — P1; owner: CC (controller)** | +| **R-659** | **[P1-HIGH] A held app's page names a way back that the restore then REFUSES — and no screen offers another.** MEASURED 2026-09-24 00:00 on 9202 (v0.267.0), chaos round 11: nextcloud's update failed, its undo copy was cut off (the drill removes the finished-marker, `09` §6.1a's measured case), and the product HELD it — honestly, in both languages: *„… Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem."* The household presses exactly that restore (`POST /backup/restore`, snapshot `helyi`, the ONE copy offered) and is refused: *„Ez a mentés nem tartalmazza az alkalmazás fájljait, ezért nem állítjuk vissza az adatbázist föléjük — a fájlok így a helyükön maradnak. A fájlok a távoli másolatból állíthatók vissza: … „Teljes visszaállítás (fájlok + adatbázis)"."* This box has no remote copy. The app stays stopped and held; Update and Start refuse a held app. **So the hold sentence and the restore disagree about the same copy, and on a box without an off-site tier nothing on any page brings the app back.** Both refusals are individually right (the restore will not pour a database over files it cannot also restore; the hold will not start a half-migrated app); together they strand a file-leg app. navidrome, held the same way in round 8, came back through the same button because its copy was accepted. **The brief's stop rule fired here** (a state neither the product nor its screens can recover): Part D had finished all twelve rounds; nothing further was injected. **Needs:** the hold must name only a route the restore will accept for that app (or say plainly that none exists on this box and the operator is needed); and a decision on what a household with no off-site tier may do for a file-leg app whose undo failed. Evidence: `audits/night-2026-09-23/chaos/round-11.*`, `chaos/round-11-recovery.*`. | **READY — P1; owner: operator (the promise) / CC (the build)** | +| **R-660** | **[P3-LOW] A held app also raises `app_start_failed`, seconds after its own `app_update_held`.** MEASURED 2026-09-23 night on 9202 (chaos rounds 8 and 11, from the controller's full log): `app_update_held (error)` at 20:56:04 and 21:42:39, then `app_start_failed (warning)` at 20:56:15 and 21:42:52 — each at the next status refresh after the hold stopped the app. The `DROPPED event` line names no app, so the attribution is by timing, not by name; nothing else changed state at those moments. The hold is a deliberate stop BY THE PRODUCT, a fourth way to stop an app that `classifyRunStates`' three suppressions (`08` §5) do not know — the same class as the nightly volume-dump false alarm (fixed v0.224.0: "a third way to stop an app needs a third suppression set"). The operator gets two alarms for one fact. **Needs:** a held app is not "down" to the app-down ladder; a test that holds an app and asserts no `app_start_failed`. Evidence: `audits/night-2026-09-23/E8-events-*`. | **READY — P3; owner: CC (controller)** |