diff --git a/REPORT-update-night-2026-09-21.md b/REPORT-update-night-2026-09-21.md new file mode 100644 index 00000000..b5fc57a4 --- /dev/null +++ b/REPORT-update-night-2026-09-21.md @@ -0,0 +1,42 @@ +# REPORT — UPDATE NIGHT, 2026-09-21 + +**The full record is `documentation/audits/DRILL-update-night-2026-09-21.md`.** This file is the +session report: what ran, what shipped, what is owed. + +*(Filled at the end of the run. `<…>` are placeholders.)* + +## Not done, or changed from the brief + + + +## What ran + +- **Phase 0** — the fleet floor to **0.261.0** (both demo boxes in **13 s**, hub `SERVED … from + declared`); a private **drill catalog** with a positive and two negative controls; a throwaway + **image store** on the scratch guest; capacity measured; the upstream drift re-run. +- **Phase 1** — real within-a-major upstream edges walked on guest 9202 through the product's + own guarded Update, each seeded and read back through the app's own front door. +- **Phase 2** — the two database engines across a major, through the real Update button. +- **Phase 3** — the bad days, B1–B9. +- **Phase 4** — the morning after. +- **Phase 5** — teardown, three layers, plus Gitea. + +## What shipped + +- `app-catalog-felhom.eu` **@4463243f2e09** — TEST CODE ONLY: four new harness fixtures and seven + new edges (U1–U7). No template changed; no `image:` line moved. Gates green; CI job **830 = + success**. +- `felhom.eu` — this report, the audit, the evidence, the register rows, the architecture updates, + and one correction to `update-arc-gaps-2026-09-21/00-api-recipe.md` (the app page is `/apps/`, + not `/app/`) and one FIX to `unattended-caller.py` (R-623). +- **No controller, agent or hub code was written.** The brief forbade it and none was needed. + +## What is owed + + + +## The live catalog + +**Never touched with a broken, dummy, cross-repo or engine-major reference — not once, not for +thirteen minutes.** Its `main` moved only for the harness commit above, which changes `scripts/` +and zero `image:` lines; the teardown diff proves every `image:` line identical to the drill copy. diff --git a/documentation/audits/DRILL-update-night-2026-09-21.md b/documentation/audits/DRILL-update-night-2026-09-21.md new file mode 100644 index 00000000..af83152f --- /dev/null +++ b/documentation/audits/DRILL-update-night-2026-09-21.md @@ -0,0 +1,174 @@ +# DRILL — UPDATE NIGHT, 2026-09-21 + +**Venue:** scratch guest **9202 `demo-hp-scratch`** on host `demo-hp`, controller **v0.261.0**. +**Catalog:** a private drill copy, `admin/app-catalog-drill`. **The live catalog was not touched.** +**Evidence:** `audits/update-night-2026-09-21/` — `PROGRESS.md` is the step log, `apps//verdict.json` +the per-edge records, `bad-days//` the Phase-3 legs. + +--- + +## Not done, or changed from the brief + +*(Filled at the end of the run. Every phase and every B-leg is listed here if it was skipped, +shortened or altered, with the reason. Empty only if true — R-611.)* + + + +--- + +## The three lines + + + +--- + +## Phase 0 — the two mechanisms, each with its controls + +### 0.1 The floor to 0.261.0 — PROVEN, 13 seconds + +Saved as `min_controller_version=0.261.0` with `min_agent=0.131.0` (read from the release's own +CHANGELOG header, as publish-train rule 1 requires). The hub answered `303 …?flash=floor_set` — not +`floor_needs_min_agent` — and said so itself, twice: + +``` +2026/09/21 20:07:07 [INFO] Global controller-version floor set to "0.261.0" (declared MinAgent "0.131.0") +2026/09/21 20:07:09 [INFO] managed floor SERVED for demo-felhom: floor 0.261.0, agent requirement "0.131.0" from declared (golden 0.258.0) +2026/09/21 20:07:10 [INFO] managed floor SERVED for demo-hp: floor 0.261.0, agent requirement "0.131.0" from declared (golden 0.258.0) +``` + +Both demo boxes were running `0.261.0` **13 seconds after the save** (`20:07:07` → `20:07:20`), each +container `Up … (healthy)`. The vouched golden is `0.258.0`, so this is the declared-MinAgent path of +§3 decision 7 working exactly as R-472 closed it. + +**Which other boxes it reaches:** none. The hub lists three hosts; the third, `drill-r50-0a4f9a`, is +`DOWN` and its agent is `0.129.0`, below the declared `0.131.0`, so the floor is held for it — by +design, and it was left alone. + +Evidence: `01-floor-pre.txt`, `02-floor-save.txt`. + +### 0.2 The drill catalog — PROVEN, with all three controls + +`admin/app-catalog-drill`, private, created from the live catalog's `main` (`f5f6a152b513`). + +**One claim in the brief turned out wrong before a single command was run, and it is the reason the +leg worked at all.** The brief assumed a box follows a second catalog once `git.repo_url` changes. +It does not. `Syncer.gitCloneOrPull` clones **only when `.git` is absent**; otherwise it fetches from +the remote stored in the clone. After the repoint, `git remote -v` in the box's cache still read +`app-catalog-felhom.eu`. The cache directory had to be removed as well. Filed as **R-615**. + +- **Positive control.** A one-step bump committed to the DRILL repo (`uptime-kuma 2.4.0 → 2.5.5`, + a real upstream edge) appeared on 9202 as „**Frissítés elérhető — ma**" / "**Update available — + today**", both languages, with the matching title text. +- **Negative control 1.** The live catalog's `main` is still `f5f6a152b513` and its `uptime-kuma` + pin is still `2.4.0`. +- **Negative control 2.** Both real boxes' catalog caches are still at `f5f6a15` — neither followed + anything. +- **R-607 fired again**, exactly as its row predicts: `POST /api/sync` answered + „Sablonok naprakészek — nincs változás" while the catalog had in fact moved; only + `POST /api/stacks/rescan` made `catalog_images` current. Tonight's line is added to that row. + +**A second brief-claim corrected:** the app page is at **`/apps/`**, not `/app/` as +`update-arc-gaps-2026-09-21/00-api-recipe.md` says. That recipe line is fixed. + +Evidence: `03-drill-repo.txt`, `04-9202-config-pre.txt`, `05-9202-follows-drill.txt`, +`07-positive-control.txt`. + +### 0.3 The throwaway image store — PROVEN, and the comparator claim RUN rather than read + +`registry:2` on 9202 at `127.0.0.1:5000`. Never DooPlex's registry; no real box can reach it. + +| tag | what it is | +|---|---| +| `localhost:5000/drill/glance:1.0.0` | the real `glanceapp/glance:v0.8.6`, retagged — it serves | +| `localhost:5000/drill/glance:1.0.1` | a built image that starts, stays up and **never serves** | +| `localhost:5000/drill/glance:1.0.2` | the FIXED next version, for B6 | +| `localhost:5000/drill/pdf:1.0.0` | the real bentopdf, retagged | +| `localhost:5000/drill/pdf:1.0.1` | **absent from the store** — 404, for the pull-fail leg | + +**The brief's worry about `host:port/` was unfounded, and it was settled by running the comparator, +not by reading it.** `splitImageRef` takes the LAST colon and rejects it only when a `/` follows, so +a registry port is never mistaken for a tag. Four positive cases and one negative control, in the +controller's own package: + +``` +OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","localhost:5000/drill/glance:1.0.1") = (-1,true) +OK CompareImageRefs("localhost:5000/drill/glance:1.0.1","localhost:5000/drill/glance:1.0.0") = ( 1,true) +OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","localhost:5000/drill/glance:1.0.0") = ( 0,true) +OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","localhost:5000/drill/glance:1.0.2") = (-1,true) +OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","glanceapp/glance:1.0.1") = ( 0,false) <- negative control +``` + +The temporary test file was deleted and `git status --porcelain` is empty again. + +Evidence: `09-image-store.txt`. + +### 0.4 Capacity — the brief's claim HOLDS + +9202: **26 GB RAM** (22 free), docker root on the `mp0` volume with **56 GB free**, a 938 GB scratch +drive at 3%. `demo-hp`'s own `/` is at 86% but holds neither the rootfs nor the docker root — both +live on `nvme-scratch`, which is at 2%. Evidence: `08-capacity.txt`. + +### 0.5 The drift re-run — the brief's numbers HOLD EXACTLY + +Re-measured against the live registries at 20:07, catalog `f5f6a152b513`: 53 apps, 66 unique pins, +**46 behind upstream, 39 within a major, 7 across a major** — the same 39 and 7 the brief names. +One pin is unmeasurable tonight (`msdeluise/plant-it` — Docker Hub answered 401 on its tag list) and +one is internal. Evidence: `06-drift-rerun.txt`. + +--- + +## The verdict table + + + +--- + +## Phase 2 — the two database engines + + + +--- + +## Phase 3 — the bad days + + + +--- + +## Phase 4 — the morning after + + + +--- + +## The alarm truth table + +**The event-and-mail half of this night could not be measured, and that is a property of the venue, +not an omission.** Guest 9202 has `hub.enabled: false`; every notifier entry point returns before it +logs anything (`notify/notifier.go:269, :359, :917, :959, :1058`), so no hub event and no customer +mail can be produced or observed there. The hub was **not** enabled on 9202 to get around this: that +would register an unclaimed host at the live hub and could mail a real address, and the brief fences +the hub. Filed as **R-620** (a disabled notifier should at least say which event it dropped). + +So the table below scores the surfaces that DO exist on this box — the app page in both languages, +the dashboard, and the controller's own log — against `08-alarm-ladder.md`. + + + +--- + +## The promotion list for the operator + + + +--- + +## Teardown — three layers, plus Gitea + + + +--- + +## Claims in the brief that turned out wrong + + diff --git a/documentation/audits/update-arc-gaps-2026-09-21/00-api-recipe.md b/documentation/audits/update-arc-gaps-2026-09-21/00-api-recipe.md index db226c2e..61a1f647 100644 --- a/documentation/audits/update-arc-gaps-2026-09-21/00-api-recipe.md +++ b/documentation/audits/update-arc-gaps-2026-09-21/00-api-recipe.md @@ -79,11 +79,17 @@ cost a few minutes. ## 3. Reading the customer's app page, both languages +**The route is `/apps/`, not `/app/`.** This section said `/app/` until 2026-09-21 and +every call it described 404s. `/` redirects to `/launcher`, and the app links live on `/stacks`. + ```bash -curl -sk -H "$H" -H "Cookie: $(cat $S/sess.txt)" "$B/app/vikunja" # Hungarian -curl -sk -H "$H" -H "Cookie: $(cat $S/sess.txt)" "$B/app/vikunja?lang=en" # English +curl -sk -H "$H" -H "Cookie: $(cat $S/sess.txt)" "$B/apps/vikunja" # Hungarian +curl -sk -H "$H" -H "Cookie: $(cat $S/sess.txt)" "$B/apps/vikunja?lang=en" # English ``` +**The session dies whenever the controller restarts** — the store is in memory — so a long run must +be able to log in again mid-flight rather than assuming one login lasts the night. + ## 4. Shell into the guest (for docker / pct only) ```bash diff --git a/documentation/audits/update-arc-gaps-2026-09-21/unattended-caller.py b/documentation/audits/update-arc-gaps-2026-09-21/unattended-caller.py index 8baf72ed..dcdb9742 100644 --- a/documentation/audits/update-arc-gaps-2026-09-21/unattended-caller.py +++ b/documentation/audits/update-arc-gaps-2026-09-21/unattended-caller.py @@ -89,11 +89,24 @@ def edge_is_within_a_major(st): def follow(name, timeout=900): - """Watch one update to its end. Returns done | failed | held | timeout.""" + """Watch one update to its end. Returns done | failed | held | timeout. + + FIXED 2026-09-21 (update night), and the bug is worth keeping written down. `call()` returns + the API ENVELOPE — `{"ok": true, "data": {...}}` — and this function read `update_phase` and + `updating` off the envelope, where they do not exist. Both were therefore ALWAYS `None`, the + end test `not updating and phase in ("done","failed")` could never fire, and **every update + this caller followed ran to the 900-second timeout and was then recorded `timeout` and + `never_again`** — including ones that had succeeded in under two minutes. `main()` unwraps + `data` for the stack LIST, which is why the within-a-major half worked and this half did not. + The 2026-09-21 run never caught it because the only pass that reached `follow()` was the one + whose log was lost to a buffering `tail`. An instrument that turns a success into a timeout is + not a measurement — see R-623. + """ deadline = time.time() + timeout last = None while time.time() < deadline: - st = call("GET", "/api/stacks/%s" % name) + env = call("GET", "/api/stacks/%s" % name) + st = env.get("data") if isinstance(env, dict) and isinstance(env.get("data"), dict) else env phase, updating = st.get("update_phase"), st.get("updating") if phase != last: log(" %s: phase=%s updating=%s" % (name, phase, updating)) diff --git a/documentation/audits/update-night-2026-09-21/01-floor-pre.txt b/documentation/audits/update-night-2026-09-21/01-floor-pre.txt new file mode 100644 index 00000000..d8096403 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/01-floor-pre.txt @@ -0,0 +1,20 @@ +=== floor pre-state 2026-09-21T20:06:43+02:00 +floor: min_controller_version" value="0.260.0" +min_agent: name="min_agent" value="0.131.0" + +=== hub /hosts controller versions BEFORE +Host | Customer | Agent | Status | Guests | CPU | Mem | Disk +demo-felhom-8363b5 | Demo Ügyfél | 0.132.0 floor: declared MinAgent | ONLINE | 1/2 | 0% | 18% | 29% +demo-hp-bb76ea | Demo HP | 0.132.0 floor: declared MinAgent | ONLINE | 1/2 | 0% | 16% | 81% +drill-r50-0a4f9a | drill-r50 | 0.129.0 floor held | DOWN | 1/1 | 20% | 20% | 10% +=== controller images BEFORE floor 2026-09-21T20:06:51+02:00 +--- felhom-pve +VMID Status Lock Name +9201 running demo-felhom +guest 9201: gitea.dooplex.hu/admin/felhom-controller:0.260.0 +--- demo-hp +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +guest 9201: gitea.dooplex.hu/admin/felhom-controller:0.260.0 +guest 9202: gitea.dooplex.hu/admin/felhom-controller:0.261.0 diff --git a/documentation/audits/update-night-2026-09-21/02-floor-save.txt b/documentation/audits/update-night-2026-09-21/02-floor-save.txt new file mode 100644 index 00000000..90daab07 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/02-floor-save.txt @@ -0,0 +1,23 @@ +=== POST /configuration/global-floor 2026-09-21T20:07:07+02:00 +http=303 +Location: /configuration?flash=floor_set +--- floor after save: +min_controller_version" value="0.261.0" +name="min_agent" value="0.131.0" +20:07:20 demo-felhom=gitea.dooplex.hu/admin/felhom-controller:0.261.0 demo-hp=gitea.dooplex.hu/admin/felhom-controller:0.261.0 +BOTH_ARRIVED + +=== hub log: managed floor SERVED 2026-09-21T20:07:26+02:00 +2026/09/21 14:07:02 [INFO] Global controller-version floor set to "0.260.0" (declared MinAgent "0.131.0") +2026/09/21 14:07:04 [INFO] managed floor SERVED for demo-felhom: floor 0.260.0, agent requirement "0.131.0" from declared (golden 0.258.0) +2026/09/21 14:07:05 [INFO] managed floor SERVED for demo-hp: floor 0.260.0, agent requirement "0.131.0" from declared (golden 0.258.0) +2026/09/21 20:07:07 [INFO] Global controller-version floor set to "0.261.0" (declared MinAgent "0.131.0") +2026/09/21 20:07:09 [INFO] managed floor SERVED for demo-felhom: floor 0.261.0, agent requirement "0.131.0" from declared (golden 0.258.0) +2026/09/21 20:07:10 [INFO] managed floor SERVED for demo-hp: floor 0.261.0, agent requirement "0.131.0" from declared (golden 0.258.0) + +=== running controller containers AFTER +--- felhom-pve +guest 9201: gitea.dooplex.hu/admin/felhom-controller:0.261.0 Up 14 seconds (healthy) +--- demo-hp +guest 9201: gitea.dooplex.hu/admin/felhom-controller:0.261.0 Up 14 seconds (healthy) +guest 9202: gitea.dooplex.hu/admin/felhom-controller:0.261.0 Up 5 hours (healthy) diff --git a/documentation/audits/update-night-2026-09-21/03-drill-repo.txt b/documentation/audits/update-night-2026-09-21/03-drill-repo.txt new file mode 100644 index 00000000..8a355e66 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/03-drill-repo.txt @@ -0,0 +1,9 @@ +=== drill repo created 2026-09-21T20:08:46+02:00 +full_name: admin/app-catalog-drill +private: True +default_branch: main +empty: False +--- main HEAD on drill: +f5f6a152b513 | REVERT both drill bumps: all four app pins back to their pre +--- live catalog main (must be unchanged): +f5f6a152b513 diff --git a/documentation/audits/update-night-2026-09-21/04-9202-config-pre.txt b/documentation/audits/update-night-2026-09-21/04-9202-config-pre.txt new file mode 100644 index 00000000..570bce49 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/04-9202-config-pre.txt @@ -0,0 +1,29 @@ +SAVED +-rw------- 1 root root 1944 Sep 21 12:40 /var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml +-rw------- 1 root root 1944 Sep 21 18:09 /var/lib/docker/volumes/felhom-controller-data/_data/controller.yaml.pre-update-night +---git-section--- +git: + branch: main + repo_url: https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git + sync_interval: 15m + token: "" + username: "" +hub: +---update-knobs--- +---cache--- +total 92 +drwxr-xr-x 8 root root 4096 Sep 21 17:55 . +drwxr-xr-x 9 root root 4096 Sep 21 11:04 .. +origin https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (fetch) +origin https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (push) +f5f6a15 REVERT both drill bumps: all four app pins back to their pre-drill images +--- git section now (token redacted): +git: + branch: main + repo_url: https://gitea.dooplex.hu/admin/app-catalog-drill.git + sync_interval: 15m + token: "" + username: "admin" +hub: +--- removing stale catalog cache (origin still pointed at the LIVE repo) +cache removed: ls: cannot access '/var/lib/docker/volumes/felhom-controller-data/_data/data/catalog-cache': No such file or directory diff --git a/documentation/audits/update-night-2026-09-21/05-9202-follows-drill.txt b/documentation/audits/update-night-2026-09-21/05-9202-follows-drill.txt new file mode 100644 index 00000000..a8c1a133 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/05-9202-follows-drill.txt @@ -0,0 +1,14 @@ +gitea.dooplex.hu/admin/felhom-controller:0.261.0 Up 20 seconds (healthy) +---sync-log--- +2026/09/21 18:09:43 sync.go:416: [DEBUG] [sync] wger/.felhom.yml: hash match, skipped +2026/09/21 18:09:43 sync.go:509: [DEBUG] [sync] wishlist: catalog has moved past the pin — rendering the stored applied definition +2026/09/21 18:09:43 sync.go:416: [DEBUG] [sync] wishlist/docker-compose.yml: hash match, skipped +2026/09/21 18:09:43 sync.go:416: [DEBUG] [sync] wishlist/.felhom.yml: hash match, skipped +2026/09/21 18:09:43 sync.go:416: [DEBUG] [sync] zipline/docker-compose.yml: hash match, skipped +2026/09/21 18:09:43 sync.go:416: [DEBUG] [sync] zipline/.felhom.yml: hash match, skipped +2026/09/21 18:09:43 sync.go:263: [INFO] [sync] Catalog sync complete +2026/09/21 18:09:43 sync.go:122: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás +---cache-origin--- +origin https://admin:@gitea.dooplex.hu/admin/app-catalog-drill.git (fetch) +origin https://admin:@gitea.dooplex.hu/admin/app-catalog-drill.git (push) +f104b4c DRILL repo README: what this is and that no customer box may follow it diff --git a/documentation/audits/update-night-2026-09-21/06-drift-rerun.txt b/documentation/audits/update-night-2026-09-21/06-drift-rerun.txt new file mode 100644 index 00000000..4505c9ac --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/06-drift-rerun.txt @@ -0,0 +1,77 @@ +RE-RUN 2026-09-21 20:07 CEST — 66 pins measured + behind upstream : 46 + WITHIN a major : 39 (the brief said 39) + ACROSS a major : 7 (the brief said 7) + other statuses : {'error': 1, 'internal-not-upstream': 1, 'behind': 46, 'no-newer-or-unparsed': 10, 'ok': 8} + +=== WITHIN-A-MAJOR edges available tonight === + actualbudget/actual-server:26.7.0 -> 26.9.0 apps=actualbudget + deluan/navidrome:0.63.2 -> 0.64.0 apps=navidrome + docmost/docmost:0.95.0 -> 0.96.0 apps=docmost + emby/embyserver:4.10.0.20 -> 4.11.0.1 apps=emby + getmeili/meilisearch:v1.36.0 -> v1.54.0 apps=wanderer + ghcr.io/advplyr/audiobookshelf:2.35.1 -> 2.36.1 apps=audiobookshelf + ghcr.io/alam00000/bentopdf:v2.8.6 -> v2.8.8 apps=bentopdf + ghcr.io/cmintey/wishlist:v0.66.0 -> v0.67.0 apps=wishlist + ghcr.io/diced/zipline:4.6.1 -> 4.7.0 apps=zipline + ghcr.io/home-assistant/home-assistant:2026.7.2 -> 2026.9.3 apps=home-assistant + ghcr.io/immich-app/immich-machine-learning:v3.0.3 -> v3.2.2 apps=immich + ghcr.io/immich-app/immich-server:v3.0.3 -> v3.2.2 apps=immich + ghcr.io/lukegus/termix:2.5.0 -> 2.8.0 apps=termix + ghcr.io/mealie-recipes/mealie:v3.20.1 -> v3.27.0 apps=mealie + ghcr.io/papra-hq/papra:26.6.1-rootless -> 26.6.2-rootless apps=papra + ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 -> v0.13.0 apps=adventurelog + ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 -> v0.13.0 apps=adventurelog + ghcr.io/tandoorrecipes/recipes:2.6.13 -> 2.6.15 apps=tandoor + ghcr.io/thomiceli/opengist:1.13 -> 1.15 apps=opengist + ghost:6.53.0-alpine -> 6.64.0-alpine apps=ghost + gitea/gitea:1.27.0 -> 1.27.3 apps=gitea + glanceapp/glance:v0.8.5 -> v0.8.6 apps=glance + gotson/komga:1.25.0 -> 1.27.0 apps=komga + grafana/grafana:13.1.0 -> 13.2.2 apps=grafana + louislam/uptime-kuma:2.4.0 -> 2.5.5 apps=uptime-kuma + lscr.io/linuxserver/bookstack:26.05.2 -> 26.05.5 apps=bookstack + lscr.io/linuxserver/code-server:4.129.0 -> 4.138.0 apps=code-server + lscr.io/linuxserver/radarr:6.3.0 -> 6.4.4 apps=radarr + lscr.io/linuxserver/sonarr:4.0.19 -> 4.0.20 apps=sonarr + lukevella/rallly:4.11.1 -> 4.15.2 apps=rallly + n8nio/n8n:2.31.3 -> 2.40.5 apps=n8n + outlinewiki/outline:1.9.1 -> 1.10.1 apps=outline + plexinc/pms-docker:1.41.4.9463-630c9f557 -> 1.43.4.10903-e5521bd8c apps=plex + privatebin/pdo:2.0.5 -> 2.0.6 apps=privatebin + registry.gitlab.com/crafty-controller/crafty-4:4.10.7 -> 4.11.0 apps=crafty-controller + rommapp/romm:5.0.0 -> 5.3.0 apps=romm + vaultwarden/server:1.36.0-alpine -> 1.37.3-alpine apps=vaultwarden + vikunja/vikunja:2.3.0 -> 2.6.0 apps=vikunja + wger/server:2.6 -> 2.7 apps=wger + +=== ACROSS-A-MAJOR === + codewithcj/sparkyfitness:v0.17.3 -> v1.7.2 apps=sparkyfitness + codewithcj/sparkyfitness_server:v0.17.3 -> v1.7.2 apps=sparkyfitness + f0rc3/gokapi:v1.9.6 -> v2.2.4 apps=gokapi + ghcr.io/claperco/claper:2.5 -> 3.0 apps=claper + ghcr.io/gethomepage/homepage:v1.13.2 -> v2.4.0 apps=homepage + ghcr.io/paperless-ngx/paperless-ngx:2.20.15 -> 3.2.1 apps=paperless-ngx + nextcloud:34.0.1-apache -> 35.0.0-apache apps=nextcloud + +=== errors / not-behind === + [no-newer-or-unparsed] calcom/cal.com:v6.2.0 up to date (no tag newer than current within the same shape) + [no-newer-or-unparsed] crocodilestick/calibre-web-automated:v4.0.6 up to date (no tag newer than current within the same shape) + [no-newer-or-unparsed] fallenbagel/jellyseerr:2.7.3 up to date (no tag newer than current within the same shape) + [no-newer-or-unparsed] flomp/wanderer-db:v0.20.0 up to date (no tag newer than current within the same shape) + [no-newer-or-unparsed] flomp/wanderer-web:v0.20.0 up to date (no tag newer than current within the same shape) + [no-newer-or-unparsed] ghcr.io/gramps-project/grampsweb:v25.6.0 up to date (no tag newer than current within the same shape) + [ok] ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 sha256:1a078b237c1d9 + [no-newer-or-unparsed] ghcr.io/sysadminsmedia/homebox:0.26.2 up to date (no tag newer than current within the same shape) + [internal-not-upstream] gitea.dooplex.hu/admin/recipe-importer:v0.9.11 + [no-newer-or-unparsed] jellyfin/jellyfin:10.11.11 up to date (no tag newer than current within the same shape) + [no-newer-or-unparsed] kimai/kimai2:apache-2.57.0 up to date (no tag newer than current within the same shape) + [ok] mariadb:11.4 sha256:70cc072b29b4a + [ok] mariadb:11.6 sha256:bfb1298c06cd1 + [ok] mariadb:12.3 sha256:805c8e104bd56 + [error] msdeluise/plant-it:0.10.0 HTTP error: 401 Client Error: Unauthorized for url: https://registry-1.docker.io/v2/msdeluise/plant-it/tags/list?n=1000 + [no-newer-or-unparsed] onlyoffice/documentserver:9.4.0 up to date (no tag newer than current within the same shape) + [ok] postgis/postgis:16-3.5-alpine sha256:47e961a569fd5 + [ok] postgres:15-alpine sha256:f7d23353e1b15 + [ok] postgres:16-alpine sha256:721873c34ceb9 + [ok] redis:7-alpine sha256:858f009f9709c diff --git a/documentation/audits/update-night-2026-09-21/07-positive-control.txt b/documentation/audits/update-night-2026-09-21/07-positive-control.txt new file mode 100644 index 00000000..0d3170a5 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/07-positive-control.txt @@ -0,0 +1,27 @@ +=== POSITIVE CONTROL: drill bump reaches 9202 2026-09-21T20:11:32+02:00 +--- POST /api/sync +{"ok":true,"data":{"ok":true,"message":"Sablonok naprakészek — nincs változás"},"message":"Sablonok naprakészek — nincs változás"} + +--- POST /api/stacks/rescan (R-607: always before reading a badge) +{"ok":true,"message":"Rescan completed: 55 stacks found"} + +uptime-kuma state=running deployed=True updating=False phase=None + pinned = {'uptime-kuma': 'louislam/uptime-kuma:2.5.1'} + installed= {'uptime-kuma': 'louislam/uptime-kuma:2.5.1'} + catalog = {'uptime-kuma': 'louislam/uptime-kuma:2.5.5'} + +--- app page badge, HU +--- app page badge, EN + +--- NEGATIVE CONTROL 1: live catalog main unchanged +f5f6a152b513 +11: image: louislam/uptime-kuma:2.4.0 +--- NEGATIVE CONTROL 2: the other two boxes' catalog cache HEAD +felhom-pve 9201: f5f6a15 REVERT both drill bumps: all four app pins back to their pre-drill images +demo-hp 9201: f5f6a15 REVERT both drill bumps: all four app pins back to their pre-drill images + +NOTE: the app page route is /apps/, NOT /app/ as 00-api-recipe.md says. +--- /apps/uptime-kuma +Frissítés elérhető — ma +--- /apps/uptime-kuma?lang=en +Update available — today diff --git a/documentation/audits/update-night-2026-09-21/08-capacity.txt b/documentation/audits/update-night-2026-09-21/08-capacity.txt new file mode 100644 index 00000000..ddea1042 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/08-capacity.txt @@ -0,0 +1,44 @@ +=== CAPACITY 2026-09-21T20:12:53+02:00 +--- demo-hp host + total used free shared buff/cache available +Mem: 29 4 6 0 18 24 + +Filesystem Size Used Avail Use% Mounted on +/dev/mapper/pve-root 39G 32G 5.4G 86% / + +Name Type Status Total (KiB) Used (KiB) Available (KiB) % +felhom-pbs pbs active 0 0 0 0.00% +local dir active 40453376 32791380 5574880 81.06% +local-lvm lvmthin active 56487936 27018179 29469756 47.83% +nvme-scratch dir active 983379700 19190616 914162472 1.95% + +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch + +--- guest 9202 + total used free shared buff/cache available +Mem: 25898 408 22139 8 3358 25489 + +Filesystem Size Used Avail Use% Mounted on +/dev/loop0 32G 1.9G 28G 7% / +/dev/loop0 32G 1.9G 28G 7% /mnt + +TYPE TOTAL ACTIVE SIZE RECLAIMABLE +Images 26 7 9.033GB 5.53GB (61%) +Containers 7 7 53.11MB 0B (0%) +Local Volumes 8 8 16.02MB 0B (0%) +Build Cache 0 0 0B 0B +--- where 9202's disks live +cores: 7 +memory: 25898 +mp0: nvme-scratch:9202/vm-9202-disk-1.raw,mp=/var/lib/felhom,backup=1,size=70G +mp8: /mnt/hdd_1/scratch-drives/scratch_hdd,mp=/mnt/felhom-drives/scratch_hdd +mp9: /var/lib/felhom-agent/guests/9202/bootstrap,mp=/etc/felhom-bootstrap,ro=1 +rootfs: nvme-scratch:9202/vm-9202-disk-0.raw,size=32G + +--- 9202 docker data root + free space +/var/lib/docker +Filesystem Size Used Avail Use% Mounted on +/dev/loop1 69G 9.3G 56G 15% /var/lib/felhom +/dev/nvme0n1 938G 19G 872G 3% /mnt/felhom-drives/scratch_hdd diff --git a/documentation/audits/update-night-2026-09-21/09-SECTION-3b-facts.md b/documentation/audits/update-night-2026-09-21/09-SECTION-3b-facts.md new file mode 100644 index 00000000..9abb09ba --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/09-SECTION-3b-facts.md @@ -0,0 +1,86 @@ +# Draft — facts to add under `09` §3b, Q2–Q6 + +**The questions stay the operator's. No recommendation changes below; where one is strengthened or +weakened by tonight's measurement, that is said in those words and the recommendation itself is left +exactly as written.** Numbers marked `<…>` are filled from the verdict records at the end of the run. + +--- + +## Under Q2 — *May an automatic update run on a bind-data app when no copy holds its FILES?* + +**MEASURED 2026-09-21 (update night).** The hold sentence Q2 turns on was read verbatim off a real +failure, not from source. `adventurelog v0.12.1 → v0.13.0` held and said: + +> „A(z) adventurelog frissítése 2026-09-21 20:53-kor nem sikerült, és az alkalmazás nem indult el az +> új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. +> Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: **saját meghajtó, 2026-09-21 20:47 +> — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.**" + +So the machinery Q2's first option would key on **exists and works**: the sentence names the tier, +the date, **and what the copy holds**, and it does so on a real edge with no prompting. The question +of whether the AUTOMATIC rule should differ from the button's is untouched by this — it is still a +choice, and it is still the operator's. + +**And one thing Q2 did not ask, which tonight makes urgent: after the hold, nobody can find out WHY.** +`failAndHold` removes the containers, so the failing version's own output is gone within seconds +(**R-621**). With a person pressing, they at least watched it happen. With nobody pressing, the only +account of the night is a sentence that says the app did not come up. + +--- + +## Under Q3 — *What counts as "within a major" when the tag is not a version number?* + +**MEASURED 2026-09-21, by running the comparator rather than reading it.** `CompareImageRefs` orders +a reference carrying a `host:port/` prefix correctly — `splitImageRef` takes the LAST colon and +rejects it only when a `/` follows, so a registry port is never mistaken for a tag. Four positive +cases and one negative control (different repositories → not orderable). This matters because it is +what made the whole unattended-hold leg possible: the drill edge `localhost:5000/drill/glance:1.0.0 +→ :1.0.1` **passes** the within-a-major test and still fails, which no real catalog move does. + +**The recommendation is unchanged.** The extension it already names — expose the parsed major from +the same normaliser — is still owed. + +--- + +## Under Q4 — *A held app: who is told, when, and does the box try again?* + +**MEASURED 2026-09-21 (update night), and this is the half that was missing.** + +--- + +## Under Q5 — *PostgreSQL: what has to exist before the catalog may move `postgres:16` to `17`?* + +**MEASURED 2026-09-21, both halves, on a real seeded datadir.** + +**(a) What a household would see today.** + +**(b) The conversion rehearsal, costed.** + +**(c) A fact about the instrument, not the engine.** The harness's own PostgreSQL probe is +`cat /var/lib/postgresql/data/PG_VERSION` **inside the container** (`upgrade-test.py` `ENGINE_PROBES`). + + +**The recommendation is unchanged** — a scripted conversion edge proven on all eleven before the +catalog may move, and the gate stays until it lands. Tonight gives it a price rather than a new +opinion. + +--- + +## Under Q6 — *Should the catalog record each pin's DIGEST at push time?* + +**MEASURED ON A BOX 2026-09-21 (update night), leg B8.** §8.1's numbers came from a registry sweep +run on DooPlex; this is the same question asked of a customer-shaped box, where the badge actually +renders. + +**The recommendation is unchanged** — yes, and it is still the cheapest real improvement on the list. + +--- + +## Not a question, but it belongs beside them + +**The night could not measure a single event or a single customer mail**, because the scratch guest +runs with `hub.enabled: false` and every notifier entry point returns before it logs anything +(**R-620**). Every Q4-shaped question about *who is told* therefore rests tonight on what the +household READS — the app page, the dashboard, the backups page — and not on what is SENT. Recorded +as the limit it is: the venue that is safe enough to break apps on is the one that cannot mail +anybody, and that is not a coincidence to design around silently. diff --git a/documentation/audits/update-night-2026-09-21/09-SECTION-drill-method.md b/documentation/audits/update-night-2026-09-21/09-SECTION-drill-method.md new file mode 100644 index 00000000..085a702b --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/09-SECTION-drill-method.md @@ -0,0 +1,63 @@ +# Draft section for `09-update-architecture.md` — the standing method for update drills + +*(To be inserted after §6.4. Written here first so the audit and the architecture stay in step.)* + +--- + +## 6.5 The drill catalog and the image store — the standing method for update drills + +**Why this section exists.** On 2026-09-21 an afternoon session put a deliberately broken image into +the LIVE catalog for thirteen minutes to produce a failing edge. It was reverted and nothing reached +a customer, but the method was wrong and the brief that asked for it said so. This is the method that +replaces it, proven the same night. + +**The rule, and it has no exception:** *nothing broken, dummy, cross-repo or engine-major ever enters +the live catalog — not as a fallback, not for thirteen minutes. If a leg cannot be done without that, +the leg is skipped and named.* + +### The two mechanisms + +| | what it is | what it makes possible | +|---|---|---| +| **the drill catalog** | `admin/app-catalog-drill` on Gitea — private, a copy of the live catalog's `main` | a scratch box can be pointed at a catalog where a failing edge is *committable*, because it carries none of the live repo's gates | +| **the image store** | a `registry:2` container on the scratch guest at `127.0.0.1:5000` | an edge that **passes the within-a-major test and still fails** — the one shape a real catalog move cannot produce | + +**The image store is not a convenience.** `09` §3b Q4 could not be measured for a year of drills +because the only failing edges available were across-a-major, and the within-a-major rule — correctly +— refuses those before the guarded update is ever reached. *The rule that makes automatic updates +safe is the same rule that refuses the obvious way to break one.* Measuring an unattended HOLD needs +`drill/:X.Y.Z` (the real image, retagged) against `drill/:X.Y.(Z+1)` (a built image that +starts, stays up and never serves) — same repository, same major, plain version tags. A third +flavour, a tag simply **absent** from the store, gives the pull-failure leg. + +`stacks.CompareImageRefs` orders a `host:port/` reference correctly: `splitImageRef` takes the last +colon and rejects it only when a `/` follows, so a registry port is never read as a tag. **Proven by +running it**, four positive cases and a negative control, 2026-09-21. + +### Pointing a box at the drill catalog — the step that is NOT obvious + +**`git.repo_url` alone is inert.** `Syncer.gitCloneOrPull` clones only when the cache has no `.git`; +otherwise it fetches from the remote the clone already stores. The cache directory must be removed as +well, or the box goes on following the live catalog and reports success. Filed as **R-615**; until it +is fixed, the drill procedure is: + +1. save `controller.yaml` as `controller.yaml.pre-update-night`; +2. set `git.repo_url` (and `username`/`token` — the drill repo is private); +3. **remove `/catalog-cache`**; +4. restart the controller, sync, **rescan** (R-607: a sync can answer „nincs változás" while the + catalog has moved, and the badge answers from the stale value until the rescan); +5. **three controls, all quoted in the report** — the drill bump appears on the scratch box; the + other boxes' caches are unchanged; the live catalog's `main` hash is unchanged. + +### What the drill must leave behind + +- `controller.yaml` restored from the saved copy, the controller restarted, and `git.repo_url` **read + back and quoted** as the live catalog. +- The registry container and its volume removed; drill images removed **by name**. Never `prune`. +- The drill repo **kept**, private, reset to the live catalog's `main`, so the next drill starts clean. +- A diff of every `image:` line against the live catalog's `main` — expected: identical. + +### The fence + +Only a scratch guest is ever pointed at the drill catalog. The drill repo's README says so, and no +customer box has credentials for it. The store listens on the guest's loopback only. diff --git a/documentation/audits/update-night-2026-09-21/09-image-store.txt b/documentation/audits/update-night-2026-09-21/09-image-store.txt new file mode 100644 index 00000000..78fa47c3 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/09-image-store.txt @@ -0,0 +1,52 @@ +=== 1. registry:2 on 127.0.0.1:5000 + registry /v2/ http=200 +=== 2. GOOD tag 1.0.0 = the real glance image retagged +localhost:5000/drill/glance:1.0.0 +=== 3. BAD tag 1.0.1 = starts, stays up, never serves +sha256:91c3018c4706e1061b590899adcdb4d92063f82dacf7d662c77472bb878d1f4b +localhost:5000/drill/glance:1.0.1 +=== 4. tags in the store (1.0.2 deliberately ABSENT for the pull-fail leg) +{"name":"drill/glance","tags":["1.0.1","1.0.0"]} + + manifest 1.0.2 (must be 404): 404 + zz_drill_tmp_test.go:23: OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","localhost:5000/drill/glance:1.0.1") = (-1,true) + zz_drill_tmp_test.go:23: OK CompareImageRefs("localhost:5000/drill/glance:1.0.1","localhost:5000/drill/glance:1.0.0") = (1,true) + zz_drill_tmp_test.go:23: OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","localhost:5000/drill/glance:1.0.0") = (0,true) + zz_drill_tmp_test.go:23: OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","localhost:5000/drill/glance:1.0.2") = (-1,true) + zz_drill_tmp_test.go:23: OK CompareImageRefs("localhost:5000/drill/glance:1.0.0","glanceapp/glance:1.0.1") = (0,false) +--- PASS: TestDrillNightCompareLocalRegistryRefs (0.00s) +PASS +ok gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.007s +=== drill/pdf:1.0.0 = the real bentopdf image retagged (for the pull-fail leg B2) +localhost:5000/drill/pdf:1.0.0 +=== drill/glance:1.0.2 = the FIXED next version (for B6: the way out forwards) +localhost:5000/drill/glance:1.0.2 +=== store contents now +drill/glance: {"name":"drill/glance","tags":["1.0.1","1.0.2","1.0.0"]} + +drill/pdf: {"name":"drill/pdf","tags":["1.0.0"]} + +=== drill/pdf:1.0.1 must be ABSENT (the pull-fail tag) + manifest drill/pdf:1.0.1 = 404 (must be 404) +localhost:5000/drill/paste:2.0.0 +localhost:5000/drill/paste:2.0.1 + drill/paste 2.0.0 + 2.0.1 <- privatebin/pdo:2.0.6 +localhost:5000/drill/pdf:2.0.0 +localhost:5000/drill/pdf:2.0.1 + drill/pdf 2.0.0 + 2.0.1 <- ghcr.io/alam00000/bentopdf:v2.8.6 +localhost:5000/drill/wishes:2.0.0 +localhost:5000/drill/wishes:2.0.1 + drill/wishes 2.0.0 + 2.0.1 <- ghcr.io/cmintey/wishlist:v0.67.0 +localhost:5000/drill/status:2.0.0 +localhost:5000/drill/status:2.0.1 + drill/status 2.0.0 + 2.0.1 <- louislam/uptime-kuma:2.5.1 +localhost:5000/drill/gist:2.0.0 +localhost:5000/drill/gist:2.0.1 + drill/gist 2.0.0 + 2.0.1 <- ghcr.io/thomiceli/opengist:1.13 +=== store now: +{"repositories":["drill/gist","drill/glance","drill/paste","drill/pdf","drill/status","drill/wishes"]} + +=== drill/pdf tags (1.0.1 must STILL be absent for B2): +{"name":"drill/pdf","tags":["2.0.0","2.0.1","1.0.0"]} + + drill/pdf:1.0.1 = 404 (must be 404) diff --git a/documentation/audits/update-night-2026-09-21/10-probe-port-sweep.txt b/documentation/audits/update-night-2026-09-21/10-probe-port-sweep.txt new file mode 100644 index 00000000..fcf463f0 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/10-probe-port-sweep.txt @@ -0,0 +1,67 @@ +probe-port sweep — all 53 templates, catalog f5f6a152b513, 2026-09-21 +compares each .felhom.yml http probe port against the compose's traefik loadbalancer port +NOTE: a difference is a CANDIDATE, not a defect — adventurelog differs and is healthy, +because the probe targets the backend container while the traefik ports belong to the +frontend. Only a live measurement decides. + +actualbudget probe=['5006'] traefik=['5006'] services~ 3 ok +adventurelog probe=['8000'] traefik=['80', '3000'] services~ 7 DIFFERS +audiobookshelf probe=['80'] traefik=['80'] services~ 4 ok +bentopdf probe=['8080'] traefik=['8080'] services~ 2 ok +bookstack probe=['80'] traefik=['80'] services~ 6 ok +calcom probe=['3000'] traefik=['3000'] services~ 5 ok +calibre-web probe=['8083'] traefik=['8083'] services~ 3 ok +claper probe=['4000'] traefik=['4000'] services~ 6 ok +code-server probe=['8443'] traefik=['8443'] services~ 3 ok +crafty-controller probe=['8443'] traefik=['8443'] services~ 7 ok +docmost probe=['3000'] traefik=['3000'] services~ 8 ok +emby probe=['8096'] traefik=['8096'] services~ 3 ok +ghost probe=['2368'] traefik=['2368'] services~ 3 ok +gitea probe=['3000'] traefik=['3000'] services~ 3 ok +glance probe=['8080'] traefik=['8080'] services~ 3 ok +gokapi probe=['53842'] traefik=['53842'] services~ 4 ok +grafana probe=['3000'] traefik=['3000'] services~ 3 ok +gramps-web probe=['5000'] traefik=['5000'] services~10 ok +home-assistant probe=['8123'] traefik=['8123'] services~ 3 ok +homebox probe=['7745'] traefik=['7745'] services~ 3 ok +homepage probe=['3000'] traefik=['3000'] services~ 3 ok +immich probe=['2283'] traefik=['2283'] services~ 9 ok +jellyfin probe=['8096'] traefik=['8096'] services~ 4 ok +kimai probe=['8001'] traefik=['8001'] services~ 6 ok +komga probe=['25600'] traefik=['25600'] services~ 3 ok +mealie probe=['9000'] traefik=['9000'] services~ 3 ok +n8n probe=['5678'] traefik=['5678'] services~ 3 ok +navidrome probe=['4533'] traefik=['4533'] services~ 3 ok +nextcloud probe=['80'] traefik=['80'] services~ 8 ok +onlyoffice probe=['80'] traefik=['80'] services~ 4 ok +opengist probe=['6157'] traefik=['6157'] services~ 3 ok +outline probe=['3000'] traefik=['3000'] services~ 8 ok +paperless-ngx probe=['8000'] traefik=['8000'] services~ 8 ok +papra probe=['1221'] traefik=['1221'] services~ 3 ok +plant-it probe=['8080'] traefik=['8080'] services~ 4 ok +plex probe=['32400'] traefik=['32400'] services~ 4 ok +privatebin probe=['8080'] traefik=['8080'] services~ 3 ok +radarr probe=['7878'] traefik=['7878'] services~ 3 ok +rallly probe=['3000'] traefik=['3000'] services~ 5 ok +recipe-importer probe=['8000'] traefik=['8000'] services~ 3 ok +romm probe=['8080'] traefik=['8080'] services~ 8 ok +seerr probe=['5055'] traefik=['5055'] services~ 3 ok +sonarr probe=['8989'] traefik=['8989'] services~ 3 ok +sparkyfitness probe=['80'] traefik=['80'] services~ 8 ok +tandoor probe=['8080'] traefik=['80'] services~ 7 DIFFERS +termix probe=['8080'] traefik=['8080'] services~ 3 ok +uptime-kuma probe=['3001'] traefik=['3001'] services~ 3 ok +vaultwarden probe=['80'] traefik=['80'] services~ 3 ok +vikunja probe=['3456'] traefik=['3456'] services~ 4 ok +wanderer probe=['3000'] traefik=['3000', '8090'] services~ 9 ok +wger probe=['80'] traefik=['8000'] services~ 4 DIFFERS +wishlist probe=['3000'] traefik=['3000'] services~ 4 ok +zipline probe=['3000'] traefik=['3000'] services~ 7 ok + +DIFFERS: ['adventurelog', 'tandoor', 'wger'] + +LIVE MEASUREMENT on guest 9202, 2026-09-21: + tandoor : container listens on 80 ONLY (ss -ltn), docker healthcheck=healthy, + front door /accounts/login/ = 200, controller state = 'unhealthy' -> DEFECT + adventurelog : controller state = 'running' -> NOT a defect + wger : NOT DEPLOYED TONIGHT — suspected, explicitly unmeasured diff --git a/documentation/audits/update-night-2026-09-21/11-notifier-disabled.txt b/documentation/audits/update-night-2026-09-21/11-notifier-disabled.txt new file mode 100644 index 00000000..bb9147f4 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/11-notifier-disabled.txt @@ -0,0 +1,21 @@ +=== 9202 notifier state, 2026-09-21 — why the event/mail half is unmeasurable here + +controller.yaml hub block on guest 9202: + hub: { api_key: "", enabled: false, push_interval: 15m, url: "" } + +the box says so once, at startup, and never again: + notifier.go:75 [INFO] Notifier disabled (hub not configured) + main.go:642 [INFO] [mailrelay] app-email shim unavailable (no hub configured or disabled in config) + selftest.go:68 [INFO] [PASS] Hub connectivity: hub disabled, skipped + +and every event path returns BEFORE it logs anything: + notify/notifier.go:269 Publish -> if !n.enabled { return } + notify/notifier.go:359 NotifyHealthChange -> if !n.enabled { return } + notify/notifier.go:867 SyncPreferences -> refuses with a message (the one that does speak) + notify/notifier.go:917, :959, :1058 the remaining entry points, all silent + +CONSEQUENCE FOR TONIGHT, stated rather than worked around: + the alarm truth table covers the app page, the dashboard and the controller log ONLY. + no hub event and no customer mail could be produced or observed on this venue. + the hub was NOT enabled on 9202 to get around this: that would register an unclaimed + host at the live hub and could mail a real address, and the brief fences the hub. diff --git a/documentation/audits/update-night-2026-09-21/12-probe-vs-compose-healthcheck.txt b/documentation/audits/update-night-2026-09-21/12-probe-vs-compose-healthcheck.txt new file mode 100644 index 00000000..4d0b2bbf --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/12-probe-vs-compose-healthcheck.txt @@ -0,0 +1,64 @@ +PROBE SWEEP 2 — the .felhom.yml controller probe vs the compose's OWN healthcheck +catalog f5f6a152b513, 2026-09-21. Purely static: both live in the SAME template, so where +both exist and disagree on PORT or PATH, one of them is wrong — no runtime needed. + +actualbudget felhom(port=['5006'],path=[]) compose(port=['5006'],path=['/']) -> agree +adventurelog felhom(port=['8000'],path=['/api/']) compose(port=['3000', '8000'],path=['/api/\\']) -> DISAGREE:PATH +audiobookshelf felhom(port=['80'],path=['/healthcheck']) compose(port=['80'],path=['/healthcheck']) -> agree +bentopdf felhom(port=['8080'],path=[]) compose(port=['8080'],path=[]) -> agree +bookstack felhom(port=['80'],path=[]) compose(port=['80'],path=[]) -> agree +calcom felhom(port=['3000'],path=['/api/auth/providers']) compose(port=['3000'],path=['/api/auth/providers']) -> agree +calibre-web felhom(port=['8083'],path=[]) compose(port=['8083'],path=[]) -> agree +claper felhom(port=['4000'],path=[]) compose(port=['4000'],path=[]) -> agree +code-server felhom(port=['8443'],path=['/healthz']) compose(port=['8443'],path=['/healthz']) -> agree +crafty-controller felhom(port=['8443'],path=[]) compose(port=None,path=None) -> compose has no http healthcheck +docmost felhom(port=['3000'],path=[]) compose(port=['3000'],path=['/']) -> agree +emby felhom(port=['8096'],path=['/emby/system/ping']) compose(port=['8096'],path=['/emby/system/ping']) -> agree +ghost felhom(port=['2368'],path=[]) compose(port=['2368'],path=['/']) -> agree +gitea felhom(port=['3000'],path=['/api/healthz']) compose(port=['3000'],path=['/api/healthz']) -> agree +glance felhom(port=['8080'],path=[]) compose(port=['8080'],path=[]) -> agree +gokapi felhom(port=['53842'],path=[]) compose(port=['53842'],path=[]) -> agree +grafana felhom(port=['3000'],path=['/api/health']) compose(port=['3000'],path=['/api/health']) -> agree +gramps-web felhom(port=['5000'],path=[]) compose(port=['5000'],path=[]) -> agree +home-assistant felhom(port=['8123'],path=['/api/']) compose(port=['8123'],path=['/manifest.json']) -> DISAGREE:PATH +homebox felhom(port=['7745'],path=['/api/v1/status']) compose(port=['7745'],path=['/api/v1/status']) -> agree +homepage felhom(port=['3000'],path=[]) compose(port=['3000'],path=[]) -> agree +immich felhom(port=['2283'],path=['/api/server/ping']) compose(port=['2283', '3003'],path=['/api/server/ping', '/ping']) -> agree +jellyfin felhom(port=['8096'],path=['/health']) compose(port=['8096'],path=['/health']) -> agree +kimai felhom(port=['8001'],path=[]) compose(port=['8001'],path=[]) -> agree +komga felhom(port=['25600'],path=['/actuator/health']) compose(port=['25600'],path=['/actuator/health']) -> agree +mealie felhom(port=['9000'],path=[]) compose(port=None,path=None) -> compose has no http healthcheck +n8n felhom(port=['5678'],path=['/healthz']) compose(port=['5678'],path=['/healthz']) -> agree +navidrome felhom(port=['4533'],path=['/ping']) compose(port=['4533'],path=['/ping']) -> agree +nextcloud felhom(port=['80'],path=['/status.php']) compose(port=['80'],path=['/status.php']) -> agree +onlyoffice felhom(port=['80'],path=['/healthcheck']) compose(port=['80'],path=['/healthcheck']) -> agree +opengist felhom(port=['6157'],path=['/healthcheck']) compose(port=['6157'],path=['/healthcheck']) -> agree +outline felhom(port=['3000'],path=['/_health']) compose(port=['3000'],path=['/_health']) -> agree +paperless-ngx felhom(port=['8000'],path=[]) compose(port=['8000'],path=[]) -> agree +papra felhom(port=['1221'],path=[]) compose(port=['1221'],path=[]) -> agree +plant-it felhom(port=['8080'],path=['/api/info']) compose(port=['8080'],path=['/api/info']) -> agree +plex felhom(port=['32400'],path=['/identity']) compose(port=['32400'],path=['/identity']) -> agree +privatebin felhom(port=['8080'],path=[]) compose(port=['8080'],path=[]) -> agree +radarr felhom(port=['7878'],path=['/ping']) compose(port=['7878'],path=['/ping']) -> agree +rallly felhom(port=['3000'],path=[]) compose(port=['3000'],path=[]) -> agree +recipe-importer felhom(port=['8000'],path=['/health']) compose(port=['8000'],path=['/health']) -> agree +romm felhom(port=['8080'],path=[]) compose(port=['8080'],path=['/']) -> agree +seerr felhom(port=['5055'],path=['/api/v1/status']) compose(port=['5055'],path=['/api/v1/status']) -> agree +sonarr felhom(port=['8989'],path=['/ping']) compose(port=['8989'],path=['/ping']) -> agree +sparkyfitness felhom(port=['80'],path=[]) compose(port=['3010', '80'],path=['/', '/api/health']) -> agree +tandoor felhom(port=['8080'],path=['/accounts/login/']) compose(port=['80'],path=['/accounts/login/']) -> DISAGREE:PORT +termix felhom(port=['8080'],path=[]) compose(port=['8080'],path=[]) -> agree +uptime-kuma felhom(port=['3001'],path=[]) compose(port=None,path=None) -> compose has no http healthcheck +vaultwarden felhom(port=['80'],path=['/alive']) compose(port=['80'],path=['/alive']) -> agree +vikunja felhom(port=['3456'],path=['/api/v1/info']) compose(port=None,path=None) -> compose has no http healthcheck +wanderer felhom(port=['3000'],path=[]) compose(port=['3000', '7700', '8090'],path=['/', '/health']) -> agree +wger felhom(port=['80'],path=[]) compose(port=['8000'],path=[]) -> DISAGREE:PORT +wishlist felhom(port=['3000'],path=[]) compose(port=['3000'],path=[]) -> agree +zipline felhom(port=['3000'],path=['/api/health']) compose(port=['3000'],path=['/api/healthcheck']) -> DISAGREE:PATH + +DISAGREEMENTS: 5 + adventurelog DISAGREE:PATH felhom=['8000'],['/api/'] compose=['3000', '8000'],['/api/\\'] + home-assistant DISAGREE:PATH felhom=['8123'],['/api/'] compose=['8123'],['/manifest.json'] + tandoor DISAGREE:PORT felhom=['8080'],['/accounts/login/'] compose=['80'],['/accounts/login/'] + wger DISAGREE:PORT felhom=['80'],[] compose=['8000'],[] + zipline DISAGREE:PATH felhom=['3000'],['/api/health'] compose=['3000'],['/api/healthcheck'] diff --git a/documentation/audits/update-night-2026-09-21/13-probe-sweep-sharpened.txt b/documentation/audits/update-night-2026-09-21/13-probe-sweep-sharpened.txt new file mode 100644 index 00000000..e198905c --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/13-probe-sweep-sharpened.txt @@ -0,0 +1,77 @@ +PROBE SWEEP 3 — the SHARPENED rule, read from healthprobe.go rather than assumed +catalog f5f6a152b513, 2026-09-21, controller v0.261.0 + + type: http -> ANY response is healthy. ONLY THE PORT MATTERS. + type: api with NO expect block -> same as http (healthprobe.go:265). ONLY THE PORT. + type: api WITH expect.status -> the PORT, the PATH and the STATUS all matter. + +So a probe/compose PATH difference is only a candidate when the type is `api` WITH expect. + +actualbudget type=http port=5006 path=None ok +adventurelog type=api port=8000 path=/api/ PATH /api/ not in compose ['/api/\\'] (and expect={'status': 200}) <<< CANDIDATE +audiobookshelf type=api port=80 path=/healthcheck ok +bentopdf type=http port=8080 path=None ok +bookstack type=http port=80 path=None ok +calcom type=api port=3000 path=/api/auth/providers ok +calibre-web type=http port=8083 path=None ok +claper type=http port=4000 path=None ok +code-server type=api port=8443 path=/healthz ok +crafty-controller type=tcp port=8443 path=None ok +docmost type=http port=3000 path=None ok +emby type=api port=8096 path=/emby/system/ping ok +ghost type=http port=2368 path=None ok +gitea type=api port=3000 path=/api/healthz ok +glance type=http port=8080 path=None ok +gokapi type=http port=53842 path=None ok +grafana type=api port=3000 path=/api/health ok +gramps-web type=http port=5000 path=None ok +home-assistant type=api port=8123 path=/api/ ok +homebox type=api port=7745 path=/api/v1/status ok +homepage type=http port=3000 path=None ok +immich type=api port=2283 path=/api/server/ping ok +jellyfin type=api port=8096 path=/health ok +kimai type=http port=8001 path=None ok +komga type=api port=25600 path=/actuator/health ok +mealie type=tcp port=9000 path=None ok +n8n type=api port=5678 path=/healthz ok +navidrome type=api port=4533 path=/ping ok +nextcloud type=api port=80 path=/status.php ok +onlyoffice type=api port=80 path=/healthcheck ok +opengist type=api port=6157 path=/healthcheck ok +outline type=api port=3000 path=/_health ok +paperless-ngx type=http port=8000 path=None ok +papra type=http port=1221 path=None ok +plant-it type=api port=8080 path=/api/info ok +plex type=api port=32400 path=/identity ok +privatebin type=http port=8080 path=None ok +radarr type=api port=7878 path=/ping ok +rallly type=http port=3000 path=None ok +recipe-importer type=api port=8000 path=/health ok +romm type=http port=8080 path=None ok +seerr type=api port=5055 path=/api/v1/status ok +sonarr type=api port=8989 path=/ping ok +sparkyfitness type=http port=80 path=None ok +tandoor type=http port=8080 path=/accounts/login/ PORT 8080 not in compose ['80'] <<< CANDIDATE +termix type=http port=8080 path=None ok +uptime-kuma type=http port=3001 path=None ok +vaultwarden type=api port=80 path=/alive ok +vikunja type=api port=3456 path=/api/v1/info ok +wanderer type=http port=3000 path=None ok +wger type=http port=80 path=None PORT 80 not in compose ['8000'] <<< CANDIDATE +wishlist type=http port=3000 path=None ok +zipline type=api port=3000 path=/api/health PATH /api/health not in compose ['/api/healthcheck'] (and expect={'status': 200}) <<< CANDIDATE + +CANDIDATES: 4 -> ['adventurelog', 'tandoor', 'wger', 'zipline'] + +LIVE RESULTS 2026-09-21 on guest 9202: + tandoor CONFIRMED DEFECT probe type http port 8080; the container listens on 80 ONLY. + Connection refused every time -> the box reads `unhealthy` while the app serves 200. + zipline CONFIRMED DEFECT probe type api + expect 200 on /api/health; zipline 4.6.1 + answers 404 there. The COMPOSE healthcheck in the same file uses /api/healthcheck + and is green. The box reads `unhealthy` while /dashboard answers 200. + wger SUSPECTED, NOT MEASURED. type http, port 80; wger/server serves 8000 and the + template's own traefik label says 8000. Not deployed tonight. + home-assistant NOT a defect, MEASURED: type api with NO expect block, so any response passes — + /api/ answers 401 without a token and that is healthy. Edge PROVEN tonight. + adventurelog NOT a defect, MEASURED: type api + expect 200 on /api/ port 8000, reads `running`. + diff --git a/documentation/audits/update-night-2026-09-21/MORNING-NOTE.md b/documentation/audits/update-night-2026-09-21/MORNING-NOTE.md new file mode 100644 index 00000000..38630558 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/MORNING-NOTE.md @@ -0,0 +1,57 @@ +# Morning note — to go on top of `STATUS.md` + +*(Draft. Numbers marked `<…>` are filled from the verdict records at the end of the run.)* + +--- + +**Updated 2026-09-21 (overnight) — I tested the "update my app" button on as many apps as fit in a +night, on good days and bad ones.

updates are proven safe. One is proven dangerous, and the +machine handled it exactly right.** + +**Decisions I took on my own: none.** Nothing tonight needed a choice you had not already made. + +**The fleet version is now 0.261.0.** You asked for that. Both demo machines took it **thirteen +seconds** after I saved it. The third machine is switched off and will take it when it comes back. + +**What I exercised.** real app updates, each one installed at the version our catalog has today, +filled with real data through the app's own front door, backed up, updated to the newer version that +really exists upstream, and then the data read back. Plus the two database engines, and the bad days. + +**The one that matters most.** *Adventurelog's newer version rewrites the customer's database and then +never starts.* The machine did everything right: it took a backup one minute before, waited the full +five minutes, stopped the app so its data could not be damaged, and told the household in one sentence +where the copy is, when it was made, and **what is inside it**. Then I pressed the restore the sentence +names, and the app came back. **That app must not be moved to the newer version.** + +**What broke, and whether the household could get out.** + + + +**Faults found, all written down, none fixed tonight.** + +1. **Two apps tell the household they are broken when they are fine.** Tandoor's health check asks the + wrong port; Zipline's asks a web address that app does not have — and the *same file* already + contains the right answer in both cases. A cheap check would catch both. +2. **When an update fails, the machine deletes the broken app's log before anyone can read it.** You + get "it did not come up" and nothing else. Nobody can say why, or report it upstream. +3. **Pointing a machine at a different app catalog does not work** — it keeps reading the old one and + says everything is fine. Only matters when you need it, which is the problem. +4. Smaller ones: a setting the machine calls optional and then refuses without; the catalog password + stored in plain text inside the machine's own copy of the catalog. + +**Rows opened and closed.** + +**What needs you.** + +1. **Rotate the Gitea `admin` token.** It was printed into my session log by an ordinary `git remote -v` + — the machine stores it in plain text. *If you do nothing:* the token keeps working and anyone with + my session transcript has it. +2. **The promotion list** — updates proven safe enough to move on the real catalog. Moving a + version is your call, never mine. *If you do nothing:* nothing breaks; those apps stay where they + are and drift further from upstream each month. +3. **The seven open questions** about automatic updates now have more facts beside them, and they are + still yours. *If you do nothing:* the automatic-update work cannot start — every part of it hangs + off the first question, which is simply *may the machine update apps by itself at night?* + +**The live catalog was not touched at all tonight.** Not once, not for thirteen minutes. Everything +ran against a private drill copy on a scratch machine, exactly as you asked. diff --git a/documentation/audits/update-night-2026-09-21/NEW-ROWS.md b/documentation/audits/update-night-2026-09-21/NEW-ROWS.md new file mode 100644 index 00000000..a5a692f4 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/NEW-ROWS.md @@ -0,0 +1,45 @@ +# Rows minted by the update night, 2026-09-21 — staging, to be pasted into OPEN-ITEMS.md + +Highest existing id at the start of the night: **R-614** (304 rows). + +--- + +| **R-615** | **[P3-LOW] Pointing a box at a different app catalog by `git.repo_url` alone is INERT — the box keeps fetching from the repository it first cloned.** FOUND 2026-09-21 by reading `sync.go` **before** running it, which is the only reason the update night's drill catalog worked at all. `Syncer.gitCloneOrPull` (`controller/internal/sync/sync.go:274-306`) clones **only when `/catalog-cache/.git` is absent**; on every later cycle it runs `git fetch --depth 1 origin ` + `git reset --hard origin/` **against the remote stored in the clone**, which `buildRepoURL` wrote at clone time. Changing `git.repo_url` in `controller.yaml` and restarting therefore changes **nothing**: the sync keeps pulling the old catalog and reports success. Measured: after the repoint, `git -C /catalog-cache remote -v` still read `app-catalog-felhom.eu`; the box only followed the drill repo once the cache directory was removed. **Why it matters beyond a drill:** this is the one knob that would move a box to a different or a staged catalog — for a migration, a per-customer catalog, or a rollback of the catalog itself — and it silently does not work. **Nothing is wrong with the CACHING**, which is right; what is missing is that a changed `repo_url` must invalidate the clone. **Fix shape:** on start, compare `git.repo_url` with the clone's `origin` and re-clone when they differ (or `git remote set-url` + a full fetch); log which happened. A test that changes `repo_url` under an existing cache and asserts the next sync reads the NEW repo — it fails today. Evidence: `audits/update-night-2026-09-21/04-9202-config-pre.txt`, `05-9202-follows-drill.txt`. | **READY — rank P3-LOW; owner: CC (controller)** | + +| **R-616** | **[P3-LOW] The catalog credentials are stored in PLAINTEXT in the box's catalog clone and are printed by an ordinary `git remote -v`.** FOUND 2026-09-21 on guest 9202 while pointing it at a private drill catalog. `Syncer.buildRepoURL` injects `username:token` into the HTTPS URL, and `git clone` persists that URL as the clone's `origin`, so `/catalog-cache/.git/config` holds the token in the clear and **any** diagnostic that prints the remote leaks it — which is what happened in this session's own transcript, and is the same shape as R-580 (`curl -w '%{redirect_url}'`). `maskRepoURL` exists and is used for the LOG lines, so the masking intent is already there; the stored remote is the half that was missed. **INERT ON THE FLEET TODAY** — the live catalog is public and `git.token` is empty on every real box — which is exactly why it should be fixed before it is not: the day the catalog goes private, every box carries a readable credential and every support session that runs `git remote -v` prints it. **Fix shape:** store the remote WITHOUT credentials and supply them per-fetch (a credential helper, `http.extraHeader`, or `GIT_ASKPASS`), and a test asserting the clone's stored `origin` contains no `@`. **Operator action from tonight, unrelated to the fix:** the Gitea `admin` token used for the drill repo was printed by that command and must be rotated. Evidence: `audits/update-night-2026-09-21/05-9202-follows-drill.txt` (redacted). | **READY — rank P3-LOW; owner: CC (controller); one operator action (rotate the Gitea admin token)** | + +| **R-617** | **[P3-LOW] The Gitea API token this project uses for pushes cannot create a repository through the documented endpoint, but CAN through `repos/migrate` — so "the token cannot do it" was nearly recorded as a fact when the truth was "one endpoint refuses it".** FOUND 2026-09-21 creating the drill catalog. Both `~/.git-credentials` tokens carry `write:misc,write:notification,write:package,write:issue,write:repository`; `POST /api/v1/user/repos` requires `write:user` and answers **403**, and `POST /api/v1/admin/users//repos` requires `write:admin` and answers 403 too. `POST /api/v1/repos/migrate` with the same token answered **201** and created the private repository. **Why this is a row and not a note:** a session that stopped at the first 403 would have recorded "CC cannot create a Gitea repository" — an unfalsifiable capability claim of exactly the shape the workspace's standing rule 2 forbids — and every later drill would have been designed around a limit that does not exist. **What it needs:** one line in the operations notes saying which endpoint to use, and (optional, operator) a token scoped for the job so the migrate route is not load-bearing. Evidence: `audits/update-night-2026-09-21/03-drill-repo.txt`. | **READY — rank P3-LOW; owner: CC (docs)** | + +| **R-618** | **[P2-MEDIUM] TWO apps are presented to the household as UNHEALTHY while they are working perfectly, and in both cases the SAME template already contains the right answer.** MEASURED 2026-09-21 on guest 9202 (controller v0.261.0, catalog `f5f6a152b513`). Two shapes, one class: **(a) `tandoor` — the WRONG PORT.** `.felhom.yml` probes `port: 8080`; the container listens on **80 and nothing else** (`ss -ltn` inside it), the compose's own traefik label routes to 80, its own docker healthcheck reads `healthy`, and `/accounts/login/` answers **200** through the household's real front door. `GET /api/stacks/tandoor` nevertheless reads `state: "unhealthy"`. **(b) `zipline` — the WRONG PATH.** `.felhom.yml` probes `/api/health`, which zipline 4.6.1 answers **404 `Route GET:/api/health not found`**; **the compose healthcheck in the very same file uses `/api/healthcheck` and is correct and green.** `/dashboard` answers 200. The controller reads `unhealthy`. **This is the MIRROR of R-613** — that is a probe that passes on a broken app (a false GREEN, which no alarm catches); this is a probe that fails on a working app (a false RED). **IT DOES NOT ALARM, AND THAT SETS THE RANK:** `08-alarm-ladder.md` §4 puts `unhealthy` deliberately in the NOT-down set, so no dead-app event and no customer mail follows — the damage is what the household READS, plus anything that gates on `state`. **IT ALREADY COST A MEASUREMENT TONIGHT:** this drill's harness waited for `state == "running"` and hung for its full budget on tandoor, an app that was up the whole time. An instrument waiting for a wrong answer looks exactly like a slow app. **THE GATE THIS WANTS IS CHEAP AND STATIC, AND THAT IS THE FINDING'S REAL VALUE.** Both halves of the answer live in the same template: compare the `.felhom.yml` probe's port and path against the compose's **own** `healthcheck: test:` URL. A sweep of all 53 templates on that rule was run tonight and returns **five** disagreements: `tandoor` (PORT — **CONFIRMED live**), `zipline` (PATH — **CONFIRMED live**), `wger` (PORT, probe 80 vs compose 8000 — **SUSPECTED, NOT MEASURED**, it was not deployed), `home-assistant` (PATH, `/api/` vs `/manifest.json` — **NOT MEASURED**), and `adventurelog` (a FALSE POSITIVE of the sweep's own regex — it reads `running` live). **So the rule finds both real defects, with two candidates and one false positive out of 53** — a good enough signal for a fast gate, provided it reports candidates rather than convictions and a person or a runtime check resolves them. The earlier, cruder rule (probe port vs the *traefik* port) is strictly worse: it clears zipline and convicts adventurelog. **Needs:** fix tandoor's port and zipline's path; measure wger and home-assistant; add the static gate with a decoy each way (R-421) — a template whose probe agrees must not read as a disagreement, and vice versa. **THE GATE'S RULE WAS THEN SHARPENED BY READING `healthprobe.go` RATHER THAN ASSUMING IT, and the sharpening REMOVED a false conviction.** `type: http` treats **any** response as healthy (`healthprobe.go:258-261`), and `type: api` with **no** `expect` block does the same (`:265-268`); only `type: api` WITH `expect.status` cares about the path or the code. So a PATH difference is a candidate only for the third shape, while a PORT difference is a candidate for all of them. Under that rule the 53-template sweep returns **four** candidates — `tandoor` and `zipline` (both CONFIRMED live), `wger` (suspected, unmeasured), and `adventurelog` (a false positive: its compose lists two containers' ports and the probe targets the backend; measured `running`). **`home-assistant` is correctly CLEARED by the sharpened rule** — `type: api`, no `expect`, so its `/api/` answering 401 without a token is healthy, and its edge was PROVEN on the box tonight. The crude rule convicted it; the rule read from the code does not. **That is the gate to build: two of 53 convicted, one suspected, one false positive, and the false positive is resolvable by one live check.** Evidence: `audits/update-night-2026-09-21/10-probe-port-sweep.txt`, `12-probe-vs-compose-healthcheck.txt` and `13-probe-sweep-sharpened.txt`. | **READY — rank P2-MEDIUM; owner: CC (catalog)** | + +| **R-619** | **[P3-LOW] A `type: password` deploy field is MANDATORY however `required` reads, and the `deploy-fields` contract says the opposite — so any caller that trusts it is refused.** MEASURED 2026-09-21 on guest 9202 while widening the update drill. `GET /api/stacks/grafana/deploy-fields` serves `{"env_var":"GF_SECURITY_ADMIN_PASSWORD","type":"password","generate":"password:16","required":false}`; a deploy carrying only the two `required:true` fields is refused **400** „a(z) „Admin jelszó" mező kitöltése kötelező — használja a Generálás gombot…". **The BEHAVIOUR is right and is a decision, not a bug:** `deploy.go:305-312` refuses a `password` field with no caller value on purpose — *"We never silently auto-generate — the user needs to know their password"* — which is the opposite of the `secret` case one branch above, where a generated value the customer never sees is exactly correct. **The defect is the CONTRACT.** `.felhom.yml` declares `required: false`, the API serves that verbatim, and nothing on the wire distinguishes "optional because the box will generate it" (`secret`) from "optional in the template and mandatory in the code" (`password`). A person using the deploy page never meets this because the page renders a Generálás button; **anything that is not that page does**, which now includes this drill harness and would include `09` §6.2's unattended caller the day it deploys anything. **Fix shape (smallest that keeps the decision):** serve `required: true` for `type: password` in the deploy-fields response — one place, derived rather than stored, so templates need no edit — and a test asserting a `password` field always reaches the wire as required. Alternatively state it in the field's `description`, which is weaker because it is prose. Evidence: `audits/update-night-2026-09-21/apps/grafana/log.txt` (the refusal) and `batchA.log`. | **READY — rank P3-LOW; owner: CC (controller)** | + +| **R-620** | **[P3-LOW] A disabled notifier drops every event with NO local trace, so a box whose hub configuration is absent or broken stops telling anyone anything and leaves nothing behind that says so.** FOUND 2026-09-21 on guest 9202 while trying to score the update night's alarm truth table. `hub.enabled: false` there, and `Notifier.Publish` returns at `notify/notifier.go:269` — **before** any log line — as do `NotifyHealthChange` (:359) and four more entry points. Startup says it once (`[INFO] Notifier disabled (hub not configured)`) and then every later event, of every severity up to `critical`, vanishes without a word. **The measurable consequence tonight:** the whole event-and-mail half of the drill was structurally unmeasurable on this venue, and the alarm truth table below covers only the app page, the dashboard and the box's own log. That is a cost this session paid and named; the next one would pay it again. **The consequence on a real box is smaller but not zero:** the fleet's boxes have the hub enabled, and total silence is already caught by the hub's dead-man's-switch (staleness from the LAST REPORT, proven in the 2026-07-22 power-outage audit). What is NOT caught is the in-between — a box that still reports but whose notifier was disabled by a bad config push would go on reporting healthy while dropping every alarm, and the only evidence would be a single INFO line at the last restart. **Fix shape:** one DEBUG (or WARN, once per event type) line on the disabled path naming the event that was dropped, so the absence is visible where it happens rather than inferable from a startup line. Cheap, and it converts an invisible failure into a greppable one — R-96 rule 3 in the place that produces it. Evidence: `audits/update-night-2026-09-21/11-notifier-disabled.txt`. | **READY — rank P3-LOW; owner: CC (controller)** | + +| **R-621** | **[P2-MEDIUM] A held update DESTROYS the evidence of why it failed: `failAndHold` runs `compose down`, the failing containers are removed, and their output is gone before anyone — household, operator or the next session — can read it.** MEASURED 2026-09-21 on guest 9202 on a REAL upstream edge: `adventurelog v0.12.1 → v0.13.0`. The new backend applied **nine Django migrations successfully** and then never listened; the update held after the full 5-minute health wait. **`Manager.failAndHold` (`stacks/update.go:723`) calls `updateCompose(dir, env, "down")`**, which removes the containers rather than stopping them, and nothing captures their logs first. Within seconds the box's own log recorded `Logs result for adventurelog: 0 bytes returned (empty)` and `docker ps -a` held nothing at all. **What survives is the WHAT and not the WHY:** the controller line `update adventurelog FAILED after the new version was started: not healthy: not healthy within 5m0s (last: state unhealthy)` and the household's sentence, both of which say the app did not come up and neither of which says the migrations ran and the server then failed to bind. **This is R-320 ("evidence off the machine before the teardown") as a PRODUCT behaviour rather than a session habit** — the teardown here is the product's own, it is correct to perform (a half-started new version must not keep running), and it happens before anyone can look. **Why it matters beyond a drill:** the hold sentence sends the household to a restore, and after the restore the only remaining question is *should I press Update again?* — which nobody can answer, because the one artefact that would say so no longer exists. It also makes every future held update unreportable to an upstream project. **Fix shape:** capture `compose logs --no-color --tail N` into the stack directory (beside `applied-compose.yml`, which already travels with the stack) IMMEDIATELY before the `down`, and surface it on the app page's hold panel or at least through the existing `/api/stacks//logs` fallback. Bounded size, written once per hold. A test that holds an app and asserts the captured file is non-empty — it fails today. Evidence: `audits/update-night-2026-09-21/apps/adventurelog/why-it-failed.txt`, `state-after-hold.txt`. | **READY — rank P2-MEDIUM; owner: CC (controller)** | + +| **R-622** | **[P2-MEDIUM] `adventurelog v0.13.0` migrates the customer's database and then does not serve — the edge must NOT be promoted, and it is the first real-catalog candidate this project has measured as unsafe.** MEASURED 2026-09-21 on guest 9202 through the product's own guarded Update. `v0.12.1 → v0.13.0` (backend AND frontend together, PostGIS held constant). The backend applied **nine migrations, every one `... OK`** — `adventures.0072_trail_wanderer_author_fields` through `integrations.0009_alter_endurainintegration_auth_method`, plus `billing.0001_initial` — and then the container's own healthcheck failed with `URLError: [Errno 111] Connection refused` on five consecutive checks. The app never bound its port. The update held honestly after the full 5-minute wait. **THE PRODUCT DID EVERYTHING RIGHT AND THAT IS HALF THE FINDING:** the precondition found a Tier-1 copy one minute old, the safety dump was written, the pin advanced BEFORE the pull, the health wait was not short-circuited, the app was stopped rather than left half-running, and the hold sentence named the tier, the date and what the copy holds — „saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza." **This is exactly the case `09` §4 exists for:** the migration RAN, so there is no undo, only a restore — and the restore is the thing slice 4 made sure existed first. **What it needs:** adventurelog stays OFF the promotion list; the cause is not diagnosed here (R-621 is why); and before it is ever promoted the edge should be re-run on the harness with its ABORT step, since an app that migrates and then refuses is the shape most likely to refuse the old image too. Evidence: `audits/update-night-2026-09-21/apps/adventurelog/`. | **READY — rank P2-MEDIUM; owner: CC (catalog); NOT a Felhom defect — an upstream edge that fails** | + +--- + +## Lines to ADD to existing rows (not re-filed) + +**R-607** — append: **Seen again 2026-09-21 (update night), and now a dozen times in one session.** Every drill-catalog bump of the night was followed by `POST /api/sync` answering „Sablonok naprakészek — nincs változás" while the box's cache HAD moved, with `catalog_images` staying stale until a separate `POST /api/stacks/rescan`. The night's harness therefore rescans unconditionally after every sync, which is a workaround and not a fix. **The window was still never measured as a NUMBER** — that is what the row asks for and what remains owed. + +**R-462** — append: **The count moved on 2026-09-21 from 3 apps to .** The update night walked real, within-a-major upstream edges on guest 9202 through the product's own guarded Update, each seeded and read back through the app's own front door: .

proven, failed, inconclusive. Box-side fixtures for apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four of them (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new `EDGES` (U1–U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **Owed:** the harness RUNS for those edges (the code is in; the runs are not), and fixtures for the apps recorded inconclusive tonight. + +**R-463** — append: **Measured 2026-09-21 (update night), both halves.** . And the conversion rehearsal Q5 asks for was costed on a real seeded datadir: . + +**R-469 / R-459** — append: **The MariaDB engine major was pressed through the real Update BUTTON for the first time on 2026-09-21** (it had only ever been run on the harness). . + +**R-446** — append: **Measured on the box 2026-09-21 (update night), leg B8.** . + +**R-458** — append: **Measured 2026-09-21 (update night), leg B9.** . + +**R-613** — append: the update night could not seed `uptime-kuma` for the same reason and left it out rather than faking it. + +**R-460** — append: bookstack's edge was walked again on 2026-09-21 and is again **half-proven** — the database half read back through `php artisan`, the file half untouched. The limitation is unchanged and is now measured on the box as well as on the harness. + +**R-442 (CLOSED)** — append, as a confirmation rather than a reopening: **the fail-closed half was exercised again 2026-09-21** on guest 9202, where `/api/disks` answers `agent not configured`. Three apps deployed with an `HDD_PATH` (navidrome, audiobookshelf, romm) were each REFUSED at „remove with data" — „A(z) …/userdata/ tárhely jelenleg nem elérhető — az alkalmazás nem távolítható el, amíg a meghajtó…" — **with the app kept**, and each was then removed successfully with the data KEPT. So the guard refuses the destructive half and leaves the non-destructive half available, which is exactly the shape the row describes. No change to the row's status. + +| **R-623** | **[P3-LOW] The unattended-update caller turned every SUCCESS into a `timeout`, and then refused to press that app again — the instrument, not the box.** FOUND 2026-09-21 (update night) by reading `unattended-caller.py` before relying on it for the Q4 hold measurement. Its `call()` returns the API **envelope** — `{"ok": true, "data": {…}}` — and `follow()` read `update_phase` and `updating` **off the envelope**, where neither exists. Both were therefore always `None`; the end test `not updating and phase in ("done","failed")` could never fire; every followed update ran the full **900-second** timeout and was recorded `timeout`, which the caller treats as terminal and adds to `never_again`. `main()` unwraps `data` for the stack LIST, which is exactly why the within-a-major half of that night worked and this half did not. **The 2026-09-21 run did not catch it because the only pass that reached `follow()` was Scenario F, whose log was lost to a buffering `tail`** — the run's own honestly-recorded instrumentation gap turns out to have hidden a second one underneath it. **This is the R-607 class in the evidence layer rather than the product layer: an instrument that can report a success as a timeout is not a measurement**, and worse, it is a measurement that says the box behaved badly when the box behaved well. **FIXED in the same file 2026-09-21** (unwrap `data`, with the reason written into the docstring so the next reader does not re-derive it), and the fixed caller is what produced tonight's unattended-hold leg. **What it does NOT invalidate:** the G-b no-retry proof, which is entirely in the refusal path and never reached `follow()`. **What it DOES qualify:** any future reading of that night's Scenario F timing — the "51 s – 1 m 26 s" figures come from the ATTENDED scenarios 04/05/07, not from the caller. | **CLOSED 2026-09-21 — fixed in `audits/update-arc-gaps-2026-09-21/unattended-caller.py`** | diff --git a/documentation/audits/update-night-2026-09-21/PROGRESS.md b/documentation/audits/update-night-2026-09-21/PROGRESS.md new file mode 100644 index 00000000..96137667 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/PROGRESS.md @@ -0,0 +1,28 @@ +# UPDATE NIGHT 2026-09-21 — progress log (one line per finished step) + +Evidence dir: `documentation/audits/update-night-2026-09-21/` +A resuming session reads THIS FILE FIRST and never repeats a finished step. + +| time (CEST) | step | verdict | evidence | +|---|---|---|---| +| 20:07 | P0.1 floor → 0.261.0 (declared MinAgent 0.131.0) | **PROVEN** — both demo boxes arrived in **13 s**; hub `managed floor SERVED … from declared (golden 0.258.0)`; drill-r50 stays held (agent 0.129.0 < 0.131.0, host DOWN) | 01-floor-pre.txt, 02-floor-save.txt | +| 20:08 | P0.2a drill repo `admin/app-catalog-drill` created (private) | **DONE** — Gitea token lacks `write:user`, so `POST /api/v1/repos/migrate` was used instead of `/user/repos`; main = `f5f6a152b513` = live | 03-drill-repo.txt | +| 20:10 | P0.2b 9202 repointed at the drill repo | **PROVEN** — **`git.repo_url` ALONE IS INERT**: `gitCloneOrPull` only clones when `.git` is absent, else fetches from the old `origin`. Cache dir had to be removed too. Config saved as `controller.yaml.pre-update-night` | 04-9202-config-pre.txt, 05-9202-follows-drill.txt | +| 20:12 | P0.2c positive + negative controls | **PROVEN** — drill bump uptime-kuma 2.4.0→2.5.5 shows on 9202 as „Frissítés elérhető — ma" / "Update available — today"; live catalog still `f5f6a152b513` with pin 2.4.0; both real boxes' caches still at `f5f6a15`. R-607 fired again (sync said „nincs változás"). App route is **`/apps/`**, not `/app/` | 06-drift-rerun.txt, 07-positive-control.txt | +| 20:07 | Drift re-run | **CONFIRMED** — 66 pins, 46 behind, **39 within-major, 7 across-major** — the brief's numbers hold exactly | 06-drift-rerun.txt | +| 20:13 | P0.4 capacity measured | **OK** — 9202: 26 GB RAM (22 free), docker root on mp0 with **56 GB free**, drive 872 GB. demo-hp `/` is 86% but holds neither. Brief's capacity claim HOLDS | 08-capacity.txt | +| 20:18 | P0.3 throwaway image store | **PROVEN** — `registry:2` on 9202 `127.0.0.1:5000`; `drill/glance:1.0.0` = real v0.8.6 retagged, `:1.0.1` = starts/stays up/never serves, `:1.0.2` absent (404). **`CompareImageRefs` DOES order `host:port/` refs** — 4 positive + 1 negative control, run not read; tmp test deleted, tree clean | 09-image-store.txt | +| 20:20 | **EDGE privatebin 2.0.5 -> 2.0.6** (file-leg) | **PROVEN** — 15.4 s; seed read back both sides; all four observables agree | apps/privatebin/ | +| 20:25 | **EDGE docmost 0.95.0 -> 0.96.0** (db-postgres, engine constant) | **PROVEN** — 103.5 s; seeded account authenticated after; all four observables agree | apps/docmost/ | +| 20:28 | **EDGE bookstack 26.05.2 -> 26.05.5** (db-mariadb, engine constant) | **PROVEN** — 45.1 s; artisan readback with its own negative control; DATABASE HALF ONLY (R-460) | apps/bookstack/ | +| 20:37 | **FINDING R-618** tandoor probe port | **DEFECT, measured** — probe 8080, app listens on 80 only; docker healthy + front door 200 + controller `unhealthy`. 53-template sweep run: wger suspected, adventurelog cleared | 10-probe-port-sweep.txt | +| 20:38 | **FINDINGS R-615/616/617/619** | filed — repo_url inert; catalog token plaintext in the clone; Gitea migrate endpoint; `type: password` mandatory though the wire says optional | NEW-ROWS.md | +| 20:39 | **EDGE actualbudget 26.7.0 -> 26.9.0** | **PROVEN** — 19.5 s | apps/actualbudget/ | +| 20:44 | **EDGE navidrome 0.63.2 -> 0.64.0** (file-leg, HDD_PATH) | **PROVEN** — 11.3 s | apps/navidrome/ | +| 20:46 | **EDGE audiobookshelf 2.35.1 -> 2.36.1** (file-leg) | **PROVEN** — 23.6 s | apps/audiobookshelf/ | +| 20:54 | **EDGE adventurelog v0.12.1 -> v0.13.0** (db-postgis) | **FAILED — the most valuable result so far.** Nine migrations applied OK, then the app never bound its port; held after the full 5-min wait; the sentence names tier, date and what the copy holds. Rows R-621 (the hold destroys the failure evidence) and R-622 (do not promote this edge) | apps/adventurelog/ | +| 20:59 | **Harness code pushed to the catalog** — 4 fixtures + edges U1..U7 | **DONE, gates green** — live catalog main moves `f5f6a152b513` -> `4463243f2e09`, **`scripts/` ONLY, zero `image:` lines** (the teardown diff still expects every image line identical). Harness RUNS owed | app-catalog-felhom.eu@4463243f2e09 | +| 20:59 | image reclaim BY NAME (no prune) | 34 unused images removed by exact reference; 40 GB -> 55 GB free | reclaim.sh | +| 21:08 | CI check for the catalog push | **GREEN** — job **id=830**, `name='gates'`, `status='completed'`, `conclusion='success'`, matched on `head_sha=4463243f2e09`. Confirms CLAUDE.md's warning: page 1's newest id was 52, the real newest was 830 — job ids are NOT page-ordered | scratchpad/ci.sh | +| 21:12 | **Observation, NOT a new row** — an app with an `HDD_PATH` cannot have its DATA removed on 9202 | R-442's guard working as designed: `/api/disks` answers `agent not configured` on this guest, so the drive path cannot be RESOLVED and the removal is **refused with the app kept** rather than half-deleted. The household's other choice — remove the app, KEEP the data — is accepted. The harness now takes that route and tidies its own directories by name at teardown | apps/navidrome/, apps/romm/ | +| 21:13 | **FINDING R-623** — the unattended caller turned every SUCCESS into a `timeout` | Read before use, not after: `follow()` read `update_phase`/`updating` off the API ENVELOPE, so both were always `None`, every followed update hit the 900 s timeout and was then marked never-press-again. **Fixed in that file** before B1 relied on it. The earlier night missed it because its only `follow()` pass was the one whose log was lost | update-arc-gaps-2026-09-21/unattended-caller.py | diff --git a/documentation/audits/update-night-2026-09-21/apps/actualbudget/app-logs-after.txt b/documentation/audits/update-night-2026-09-21/apps/actualbudget/app-logs-after.txt new file mode 100644 index 00000000..24017f29 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/actualbudget/app-logs-after.txt @@ -0,0 +1,9 @@ +actualbudget | Checking if there are any migrations to run for direction "up"... +actualbudget | Migrations: DONE +actualbudget | Running in production mode - Serving static React app +actualbudget | Listening on :::5006... +actualbudget | Logging in via password +actualbudget | 2026-09-21T18:39:23.916Z info: POST 400 /account/login +actualbudget | Logging in via password +actualbudget | Deleted 0 old sessions +actualbudget | 2026-09-21T18:39:24.258Z info: POST 200 /account/login diff --git a/documentation/audits/update-night-2026-09-21/apps/actualbudget/badges.json b/documentation/audits/update-night-2026-09-21/apps/actualbudget/badges.json new file mode 100644 index 00000000..180bc95a --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/actualbudget/badges.json @@ -0,0 +1,21 @@ +{ + "before": { + "hu": [], + "en": [] + }, + "after": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "drill_commit": "1c47aa5d6192" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/actualbudget/log.txt b/documentation/audits/update-night-2026-09-21/apps/actualbudget/log.txt new file mode 100644 index 00000000..ff127398 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/actualbudget/log.txt @@ -0,0 +1,25 @@ +20:37:49 ==== actualbudget: actualbudget/actual-server:26.7.0 -> actualbudget/actual-server:26.9.0 (sub=budget, class=other) +20:37:49 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +20:38:04 [1] deployed, controller state=running, pinned={'actualbudget': 'actualbudget/actual-server:26.7.0'} +20:38:06 [2] seeding through the app's own front door +20:38:07 actualbudget: /account/bootstrap http=200 :: {"status":"ok","data":{"token":"1164c717-bcfd-4064-9789-cda92a14d07e"}} +20:38:07 [3] control C1 — reading the seed back BEFORE the update +20:38:08 actualbudget: login with the seeded password http=200 ok=True +20:38:08 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +20:38:58 [4] backup idle; last=None +20:38:59 [5] drill commit 1c47aa5d6192: actualbudget actualbudget/actual-server:26.7.0 -> actualbudget/actual-server:26.9.0 (push rc=0) +20:39:03 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +20:39:03 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +20:39:04 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +20:39:04 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +20:39:05 + 1.1s phase=pulling label=Új verzió letöltése… err=None hold=None +20:39:12 + 8.2s phase=starting label=Indítás az új verzióval… err=None hold=None +20:39:13 + 9.3s phase=verifying label=Működés ellenőrzése… err=None hold=None +20:39:23 + 19.5s phase=done label=Frissítve err=None hold=None +20:39:23 [7] reading the seed back AFTER the update +20:39:24 actualbudget: login with the seeded password http=200 ok=True +20:39:26 [8] pinned = {'actualbudget': 'actualbudget/actual-server:26.9.0'} +20:39:26 [8] installed = {'actualbudget': 'actualbudget/actual-server:26.9.0'} +20:39:26 [8] compose = ['image: actualbudget/actual-server:26.9.0'] +20:39:26 [8] inspect = ['actualbudget actualbudget/actual-server:26.9.0 running=true restarts=0'] +20:39:26 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/actualbudget/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/actualbudget/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/actualbudget/observables-before.json new file mode 100644 index 00000000..a0c9d55f --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/actualbudget/observables-before.json @@ -0,0 +1,15 @@ +{ + "pinned_images": { + "actualbudget": "actualbudget/actual-server:26.7.0" + }, + "installed_images": { + "actualbudget": "actualbudget/actual-server:26.7.0" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: actualbudget/actual-server:26.7.0" + ], + "docker_inspect": [ + "actualbudget actualbudget/actual-server:26.7.0 running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/actualbudget/observables.json b/documentation/audits/update-night-2026-09-21/apps/actualbudget/observables.json new file mode 100644 index 00000000..07c43b1d --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/actualbudget/observables.json @@ -0,0 +1,34 @@ +{ + "before": { + "pinned_images": { + "actualbudget": "actualbudget/actual-server:26.7.0" + }, + "installed_images": { + "actualbudget": "actualbudget/actual-server:26.7.0" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: actualbudget/actual-server:26.7.0" + ], + "docker_inspect": [ + "actualbudget actualbudget/actual-server:26.7.0 running=true restarts=0" + ] + }, + "after": { + "pinned_images": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "installed_images": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "catalog_images": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "live_compose_image_lines": [ + "image: actualbudget/actual-server:26.9.0" + ], + "docker_inspect": [ + "actualbudget actualbudget/actual-server:26.9.0 running=true restarts=0" + ] + } +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/actualbudget/phases.json b/documentation/audits/update-night-2026-09-21/apps/actualbudget/phases.json new file mode 100644 index 00000000..18944a5e --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/actualbudget/phases.json @@ -0,0 +1,51 @@ +{ + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.1, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 8.2, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 9.3, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 19.5, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 19.5, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/actualbudget/verdict.json b/documentation/audits/update-night-2026-09-21/apps/actualbudget/verdict.json new file mode 100644 index 00000000..2dcd31df --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/actualbudget/verdict.json @@ -0,0 +1,43 @@ +{ + "harness_version": 1, + "app": "actualbudget", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "other", + "from": { + "actualbudget": "actualbudget/actual-server:26.7.0" + }, + "to": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "actualbudget | Checking if there are any migrations to run for direction \"up\"...", + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 19.5, + "measured_at": "2026-09-21T18:37:49.287537+00:00", + "evidence": "apps/actualbudget/", + "notes": [], + "observables_after": { + "pinned_images": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "installed_images": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "catalog_images": { + "actualbudget": "actualbudget/actual-server:26.9.0" + }, + "live_compose_image_lines": [ + "image: actualbudget/actual-server:26.9.0" + ], + "docker_inspect": [ + "actualbudget actualbudget/actual-server:26.9.0 running=true restarts=0" + ] + }, + "final_phase": "done", + "hold_reason": null, + "update_error": null +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/adventurelog/app-logs-after.txt b/documentation/audits/update-night-2026-09-21/apps/adventurelog/app-logs-after.txt new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/update-night-2026-09-21/apps/adventurelog/badges.json b/documentation/audits/update-night-2026-09-21/apps/adventurelog/badges.json new file mode 100644 index 00000000..7e3a33f2 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/adventurelog/badges.json @@ -0,0 +1,31 @@ +{ + "before": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "after": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "drill_commit": "b4de9025ba9b" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/adventurelog/log.txt b/documentation/audits/update-night-2026-09-21/apps/adventurelog/log.txt new file mode 100644 index 00000000..e4c604ba --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/adventurelog/log.txt @@ -0,0 +1,25 @@ +20:46:54 ==== adventurelog: ghcr.io/seanmorley15/adventurelog-backend:v0.12.1,ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 -> ghcr.io/seanmorley15/adventurelog-backend:v0.13.0,ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0 (sub=travel, class=db-postgis) +20:46:54 [1] adventurelog already deployed — reusing +20:46:56 [2] seeding through the app's own front door +20:47:01 adventurelog: createsuperuser :: Superuser created successfully. +20:47:05 adventurelog: seeded superuser drill0dfc9c +20:47:05 [3] control C1 — reading the seed back BEFORE the update +20:47:13 adventurelog: readback of the seeded account found=True +20:47:13 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +20:48:03 [4] backup idle; last=None +20:48:04 [5] drill commit b4de9025ba9b: adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1,ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 -> ghcr.io/seanmorley15/adventurelog-backend:v0.13.0,ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0 (push rc=0) +20:48:08 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +20:48:08 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +20:48:09 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +20:48:09 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +20:48:10 + 1.0s phase=pulling label=Új verzió letöltése… err=None hold=None +20:48:47 + 37.9s phase=starting label=Indítás az új verzióval… err=None hold=None +20:48:52 + 43.1s phase=verifying label=Működés ellenőrzése… err=None hold=None +20:53:55 + 346.6s phase=failed label=A frissítés nem sikerült err=A(z) adventurelog frissítése 2026-09-21 20:53-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza. hold=A(z) adventurelog frissítése 2026-09-21 20:53-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza. +20:53:55 [7] reading the seed back AFTER the update +20:54:06 adventurelog: READBACK UNUSABLE — a username that cannot exist did not read as absent (None) :: Error response from daemon: No such container: adventurelog +20:54:08 [8] pinned = {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine'} +20:54:08 [8] installed = {'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.12.1', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine'} +20:54:08 [8] compose = ['image: ghcr.io/seanmorley15/adventurelog-backend:v0.13.0', 'image: postgis/postgis:16-3.5-alpine', 'image: ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0'] +20:54:08 [8] inspect = [] +20:54:08 [9] verdict failed -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/adventurelog/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/adventurelog/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/adventurelog/observables-before.json new file mode 100644 index 00000000..f52b0923 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/adventurelog/observables-before.json @@ -0,0 +1,27 @@ +{ + "pinned_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "installed_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "catalog_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "image: postgis/postgis:16-3.5-alpine", + "image: ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1" + ], + "docker_inspect": [ + "adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 running=true restarts=0", + "adventurelog-postgres postgis/postgis:16-3.5-alpine running=true restarts=0", + "adventurelog-frontend ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/adventurelog/observables.json b/documentation/audits/update-night-2026-09-21/apps/adventurelog/observables.json new file mode 100644 index 00000000..7287ad6c --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/adventurelog/observables.json @@ -0,0 +1,52 @@ +{ + "before": { + "pinned_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "installed_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "catalog_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "image: postgis/postgis:16-3.5-alpine", + "image: ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1" + ], + "docker_inspect": [ + "adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 running=true restarts=0", + "adventurelog-postgres postgis/postgis:16-3.5-alpine running=true restarts=0", + "adventurelog-frontend ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 running=true restarts=0" + ] + }, + "after": { + "pinned_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.13.0", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "installed_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "catalog_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.13.0", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/seanmorley15/adventurelog-backend:v0.13.0", + "image: postgis/postgis:16-3.5-alpine", + "image: ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0" + ], + "docker_inspect": [] + } +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/adventurelog/phases.json b/documentation/audits/update-night-2026-09-21/apps/adventurelog/phases.json new file mode 100644 index 00000000..da936944 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/adventurelog/phases.json @@ -0,0 +1,51 @@ +{ + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.0, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 37.9, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 43.1, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 346.6, + "phase": "failed", + "label": "A frissítés nem sikerült", + "updating": false, + "error": "A(z) adventurelog frissítése 2026-09-21 20:53-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "hold": "A(z) adventurelog frissítése 2026-09-21 20:53-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza." + } + ], + "duration_s": 346.6, + "final_phase": "failed", + "update_error": "A(z) adventurelog frissítése 2026-09-21 20:53-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "hold_reason": "A(z) adventurelog frissítése 2026-09-21 20:53-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "state": "stopped" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/adventurelog/state-after-hold.txt b/documentation/audits/update-night-2026-09-21/apps/adventurelog/state-after-hold.txt new file mode 100644 index 00000000..c611dd4e --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/adventurelog/state-after-hold.txt @@ -0,0 +1,26 @@ +=== all adventurelog-ish containers (incl. stopped) +(none at all) + +=== the stack dir +total 32 +drwxr-xr-x 2 root root 4096 Sep 21 18:53 . +drwxr-xr-x 57 root root 4096 Sep 13 20:22 .. +-rw-r--r-- 1 root root 3881 Sep 21 18:48 .felhom.yml +-rw------- 1 root root 1370 Sep 21 18:48 app.yaml +-rw-r--r-- 1 root root 5389 Sep 21 18:48 applied-compose.yml +-rw-r--r-- 1 root root 5389 Sep 21 18:48 docker-compose.yml + +=== the live compose image lines + image: ghcr.io/seanmorley15/adventurelog-backend:v0.13.0 + image: postgis/postgis:16-3.5-alpine + image: ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0 + +=== app.yaml pins +pinned_images: + adventurelog: ghcr.io/seanmorley15/adventurelog-backend:v0.13.0 + adventurelog-frontend: ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0 + adventurelog-postgres: postgis/postgis:16-3.5-alpine + +=== volumes still present +adventurelog_adventurelog_media +adventurelog_adventurelog_postgres_data diff --git a/documentation/audits/update-night-2026-09-21/apps/adventurelog/verdict.json b/documentation/audits/update-night-2026-09-21/apps/adventurelog/verdict.json new file mode 100644 index 00000000..e90c185a --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/adventurelog/verdict.json @@ -0,0 +1,55 @@ +{ + "harness_version": 1, + "app": "adventurelog", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "db-postgis", + "from": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "to": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.13.0", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "verdict": "failed", + "seed_read_before": true, + "seed_read_after": false, + "healthy_after": false, + "migration_observed": null, + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 346.6, + "measured_at": "2026-09-21T18:46:54.260610+00:00", + "evidence": "apps/adventurelog/", + "notes": [ + "the edge ended HELD or failed — this is a RESULT, not an error of the run" + ], + "observables_after": { + "pinned_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.13.0", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "installed_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.12.1", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "catalog_images": { + "adventurelog": "ghcr.io/seanmorley15/adventurelog-backend:v0.13.0", + "adventurelog-frontend": "ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0", + "adventurelog-postgres": "postgis/postgis:16-3.5-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/seanmorley15/adventurelog-backend:v0.13.0", + "image: postgis/postgis:16-3.5-alpine", + "image: ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0" + ], + "docker_inspect": [] + }, + "final_phase": "failed", + "hold_reason": "A(z) adventurelog frissítése 2026-09-21 20:53-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "update_error": "A(z) adventurelog frissítése 2026-09-21 20:53-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza." +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/adventurelog/why-it-failed.txt b/documentation/audits/update-night-2026-09-21/apps/adventurelog/why-it-failed.txt new file mode 100644 index 00000000..070594a7 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/adventurelog/why-it-failed.txt @@ -0,0 +1,42 @@ +=== adventurelog backend, last 40 lines +Error response from daemon: No such container: adventurelog + +=== frontend, last 20 +Error response from daemon: No such container: adventurelog-frontend + +=== container states +2026/09/21 18:53:54 auth.go:142: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog +2026/09/21 18:53:54 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog) +2026/09/21 18:53:55 settings.go:1738: [WARN] [settings] restore hold SET for adventurelog — the app stays stopped until it is cleared +2026/09/21 18:53:55 update_guard.go:531: [WARN] [backup] adventurelog is HELD STOPPED after a failed update (restore point: tier 1 "saját meghajtó", 2026-09-21T18:47:19Z; holds: "a beállításokat, az adatbázist és az adatköteteket tartalmazza") +2026/09/21 18:53:55 auth.go:142: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog +2026/09/21 18:53:55 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog) +2026/09/21 18:53:55 auth.go:142: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog +2026/09/21 18:53:55 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog) +2026/09/21 18:54:03 auth.go:142: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog +2026/09/21 18:54:03 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog) +2026/09/21 18:54:03 auth.go:142: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog +2026/09/21 18:54:03 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog) +2026/09/21 18:54:06 auth.go:142: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog/logs +2026/09/21 18:54:06 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog/logs (path=/stacks/adventurelog/logs) +2026/09/21 18:54:06 manager.go:1290: [INFO] [stacks] Fetching logs for stack adventurelog (tail=500) +2026/09/21 18:54:06 manager.go:1397: [DEBUG] Running: docker compose logs --tail 500 --no-color (in /opt/docker/stacks/adventurelog) +2026/09/21 18:54:06 manager.go:1300: [DEBUG] Logs result for adventurelog: 0 bytes returned (empty) +2026/09/21 18:54:06 auth.go:142: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog +2026/09/21 18:54:06 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog) +2026/09/21 18:54:08 auth.go:142: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog +2026/09/21 18:54:08 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog) +2026/09/21 18:54:43 backup.go:465: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [adventurelog, gitea, glance, tandoor, uptime-kuma, vaultwarden, wishlist, zipline] +2026/09/21 18:54:43 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml +2026/09/21 18:54:43 sync.go:416: [DEBUG] [sync] adventurelog/docker-compose.yml: hash match, skipped +2026/09/21 18:54:43 sync.go:416: [DEBUG] [sync] adventurelog/.felhom.yml: hash match, skipped +2026/09/21 18:48:09 update.go:917: [INFO] [stacks] update adventurelog: phase checking +2026/09/21 18:48:09 update.go:603: [INFO] [stacks] update adventurelog: precondition met — Tier 1 (own recovery unit) copy from 2026-09-21T18:47:19Z (1m0s old, limit 24h0m0s) +2026/09/21 18:48:09 update.go:917: [INFO] [stacks] update adventurelog: phase safety-dump +2026/09/21 18:48:09 update.go:635: [INFO] [stacks] update adventurelog: safety dump done (1 file(s)) [/mnt/sys_drive/felhom-data/backups/primary/adventurelog/db-dumps/pre-restore-20260921T184809Z-adventurelog-postgres.sql] +2026/09/21 18:48:09 update.go:917: [INFO] [stacks] update adventurelog: phase pinning +2026/09/21 18:48:09 pin.go:362: [INFO] [stacks] update adventurelog: pin advanced to the catalog's current definition (adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.13.0, adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.13.0, adventurelog-postgres=postgis/postgis:16-3.5-alpine) +2026/09/21 18:48:09 update.go:917: [INFO] [stacks] update adventurelog: phase pulling +2026/09/21 18:48:46 update.go:917: [INFO] [stacks] update adventurelog: phase starting +2026/09/21 18:48:51 update.go:917: [INFO] [stacks] update adventurelog: phase verifying +2026/09/21 18:53:53 update.go:722: [ERROR] [stacks] update adventurelog FAILED after the new version was started: not healthy: not healthy within 5m0s (last: state unhealthy) — stopping and HOLDING the app; the pin stays on the new version (its migration may have run) diff --git a/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/app-logs-after.txt b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/app-logs-after.txt new file mode 100644 index 00000000..6c4c3e0e --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/app-logs-after.txt @@ -0,0 +1,24 @@ +audiobookshelf | Running in production mode. +audiobookshelf | Options: CONFIG_PATH=/config, METADATA_PATH=/metadata, PORT=80, HOST=undefined, SOURCE=docker, ROUTER_BASE_PATH=/audiobookshelf +audiobookshelf | [2026-09-21 20:46:28.192] INFO: === Starting Server === +audiobookshelf | [2026-09-21 20:46:28.193] INFO: [Server] Init v2.36.1 +audiobookshelf | [2026-09-21 20:46:28.193] INFO: [Server] Node.js Version: v20.20.2 +audiobookshelf | [2026-09-21 20:46:28.193] INFO: [Server] Platform: linux +audiobookshelf | [2026-09-21 20:46:28.194] INFO: [Server] Arch: x64 +audiobookshelf | [2026-09-21 20:46:28.198] INFO: [Database] Initializing db at "/config/absdatabase.sqlite" +audiobookshelf | [2026-09-21 20:46:28.219] INFO: [Database] Loading extension /usr/local/lib/nusqlite3/libnusqlite3.so +audiobookshelf | [2026-09-21 20:46:28.220] INFO: [Database] Successfully loaded extension /usr/local/lib/nusqlite3/libnusqlite3.so +audiobookshelf | [2026-09-21 20:46:28.220] INFO: [Database] Db supports unaccent and unicode foldings +audiobookshelf | [2026-09-21 20:46:28.220] INFO: [Database] Db connection was successful +audiobookshelf | [2026-09-21 20:46:28.280] INFO: [MigrationManager] No migrations to run. +audiobookshelf | [2026-09-21 20:46:28.432] INFO: [Database] Db initialized with models: SequelizeMeta, user, session, apiKey, library, libraryFolder, book, podcast, podcastEpisode, libraryItem, mediaProgress, series, bookSeries, author, bookAuthor, collection, collectionBook, playlist, playlistMediaItem, device, playbackSession, feed, feedEpisode, setting, customMetadataProvider, mediaItemShare +audiobookshelf | [2026-09-21 20:46:28.470] INFO: [Database] Server upgrade detected from 2.35.1 to 2.36.1 +audiobookshelf | [2026-09-21 20:46:28.481] INFO: [Database] running ANALYZE +audiobookshelf | [2026-09-21 20:46:28.493] INFO: [Database] ANALYZE completed +audiobookshelf | [2026-09-21 20:46:28.493] INFO: [Server] Serving from base path "/audiobookshelf" +audiobookshelf | [2026-09-21 20:46:28.495] INFO: [LogManager] Init current daily log filename: 2026-09-21.txt +audiobookshelf | [2026-09-21 20:46:28.502] INFO: [BackupManager] 0 Backups Found +audiobookshelf | [2026-09-21 20:46:28.503] INFO: [BackupManager] Auto Backups are disabled +audiobookshelf | [2026-09-21 20:46:28.524] INFO: Listening on port :80 +audiobookshelf | [2026-09-21 20:46:38.525] ERROR: [LocalAuth] Failed login attempt for username "drill59c654" from ip 192.168.0.180 (Invalid password) +audiobookshelf | [2026-09-21 20:46:38.578] INFO: [LocalAuth] User "drill59c654" logged in from ip 192.168.0.180 diff --git a/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/badges.json b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/badges.json new file mode 100644 index 00000000..ee99fd93 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/badges.json @@ -0,0 +1,31 @@ +{ + "before": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "after": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "drill_commit": "4fbe52bb1955" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/log.txt b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/log.txt new file mode 100644 index 00000000..812a97a0 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/log.txt @@ -0,0 +1,27 @@ +20:44:43 ==== audiobookshelf: ghcr.io/advplyr/audiobookshelf:2.35.1 -> ghcr.io/advplyr/audiobookshelf:2.36.1 (sub=audiobooks, class=file-leg) +20:44:46 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf'] +20:44:46 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH'] +20:44:46 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +20:45:16 [1] deployed, controller state=running, pinned={'audiobookshelf': 'ghcr.io/advplyr/audiobookshelf:2.35.1'} +20:45:18 [2] seeding through the app's own front door +20:45:18 audiobookshelf: /init http=200 +20:45:18 [3] control C1 — reading the seed back BEFORE the update +20:45:18 audiobookshelf: login as the seeded root http=200 ok=True +20:45:18 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +20:46:09 [4] backup idle; last=None +20:46:09 [5] drill commit 4fbe52bb1955: audiobookshelf ghcr.io/advplyr/audiobookshelf:2.35.1 -> ghcr.io/advplyr/audiobookshelf:2.36.1 (push rc=0) +20:46:14 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +20:46:14 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +20:46:14 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +20:46:14 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +20:46:15 + 1.0s phase=pulling label=Új verzió letöltése… err=None hold=None +20:46:27 + 12.3s phase=starting label=Indítás az új verzióval… err=None hold=None +20:46:28 + 13.3s phase=verifying label=Működés ellenőrzése… err=None hold=None +20:46:38 + 23.6s phase=done label=Frissítve err=None hold=None +20:46:38 [7] reading the seed back AFTER the update +20:46:38 audiobookshelf: login as the seeded root http=200 ok=True +20:46:41 [8] pinned = {'audiobookshelf': 'ghcr.io/advplyr/audiobookshelf:2.36.1'} +20:46:41 [8] installed = {'audiobookshelf': 'ghcr.io/advplyr/audiobookshelf:2.36.1'} +20:46:41 [8] compose = ['image: ghcr.io/advplyr/audiobookshelf:2.36.1'] +20:46:41 [8] inspect = ['audiobookshelf ghcr.io/advplyr/audiobookshelf:2.36.1 running=true restarts=0'] +20:46:41 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/observables-before.json new file mode 100644 index 00000000..c50fde18 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/observables-before.json @@ -0,0 +1,15 @@ +{ + "pinned_images": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.35.1" + }, + "installed_images": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.35.1" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: ghcr.io/advplyr/audiobookshelf:2.35.1" + ], + "docker_inspect": [ + "audiobookshelf ghcr.io/advplyr/audiobookshelf:2.35.1 running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/observables.json b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/observables.json new file mode 100644 index 00000000..c2518f23 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/observables.json @@ -0,0 +1,34 @@ +{ + "before": { + "pinned_images": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.35.1" + }, + "installed_images": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.35.1" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: ghcr.io/advplyr/audiobookshelf:2.35.1" + ], + "docker_inspect": [ + "audiobookshelf ghcr.io/advplyr/audiobookshelf:2.35.1 running=true restarts=0" + ] + }, + "after": { + "pinned_images": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.36.1" + }, + "installed_images": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.36.1" + }, + "catalog_images": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.36.1" + }, + "live_compose_image_lines": [ + "image: ghcr.io/advplyr/audiobookshelf:2.36.1" + ], + "docker_inspect": [ + "audiobookshelf ghcr.io/advplyr/audiobookshelf:2.36.1 running=true restarts=0" + ] + } +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/phases.json b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/phases.json new file mode 100644 index 00000000..b53e877b --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/phases.json @@ -0,0 +1,51 @@ +{ + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.0, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 12.3, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 13.3, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 23.6, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 23.6, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/verdict.json b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/verdict.json new file mode 100644 index 00000000..336d381f --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/audiobookshelf/verdict.json @@ -0,0 +1,43 @@ +{ + "harness_version": 1, + "app": "audiobookshelf", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "file-leg", + "from": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.35.1" + }, + "to": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.36.1" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "audiobookshelf | [2026-09-21 20:46:28.280] INFO: [MigrationManager] No migrations to run.", + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 23.6, + "measured_at": "2026-09-21T18:44:43.824636+00:00", + "evidence": "apps/audiobookshelf/", + "notes": [], + "observables_after": { + "pinned_images": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.36.1" + }, + "installed_images": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.36.1" + }, + "catalog_images": { + "audiobookshelf": "ghcr.io/advplyr/audiobookshelf:2.36.1" + }, + "live_compose_image_lines": [ + "image: ghcr.io/advplyr/audiobookshelf:2.36.1" + ], + "docker_inspect": [ + "audiobookshelf ghcr.io/advplyr/audiobookshelf:2.36.1 running=true restarts=0" + ] + }, + "final_phase": "done", + "hold_reason": null, + "update_error": null +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/bookstack/app-logs-after.txt b/documentation/audits/update-night-2026-09-21/apps/bookstack/app-logs-after.txt new file mode 100644 index 00000000..b0eb00a9 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/bookstack/app-logs-after.txt @@ -0,0 +1 @@ +{"ok":true,"data":{"logs":"bookstack-db | 2026-09-21 20:27:01+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:12.3.3+maria~ubu2404 started.\nbookstack-db | 2026-09-21 20:27:01+02:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB\nbookstack-db | 2026-09-21 20:27:01+02:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'\nbookstack-db | 2026-09-21 20:27:01+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:12.3.3+maria~ubu2404 started.\nbookstack-db | 2026-09-21 20:27:01+02:00 [Note] [Entrypoint]: MariaDB upgrade not required\nbookstack-db | 2026-09-21 20:27:01 0 [Note] Starting MariaDB 12.3.3-MariaDB-ubu2404 source revision 83e909fc2a0dbc394b4b683fb3fa2d7dcf26cc5e server_uid 2VA7I5pmSo4da2AgFBUJbumhNvc= as process 1\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: Compressed tables use zlib 1.3\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: Number of transaction pools: 1\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions\nbookstack-db | 2026-09-21 20:27:02 0 [Warning] mariadbd: io_uring_queue_init() failed with EPERM: sysctl kernel.io_uring_disabled has the value 2, or 1 and the user of the process is not a member of sysctl kernel.io_uring_group. (see man 2 io_uring_setup).\nbookstack-db | create_uring failed: falling back to libaio\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: Using Linux native AIO\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: innodb_buffer_pool_size_max=8388608m, innodb_buffer_pool_size=128m\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: Completed initialization of buffer pool\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: End of log at LSN=3356989\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: Opened 3 undo tablespaces\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: log sequence number 3356989; transaction id 3080\nbookstack-db | 2026-09-21 20:27:02 0 [Note] Plugin 'FEEDBACK' is disabled.\nbookstack-db | 2026-09-21 20:27:02 0 [Note] Plugin 'wsrep-provider' is disabled.\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool\nbookstack-db | 2026-09-21 20:27:02 0 [Note] InnoDB: Buffer pool(s) load completed at 260921 20:27:02\nbookstack-db | 2026-09-21 20:27:02 0 [Note] Server socket created on IP: '0.0.0.0', port: '3306'.\nbookstack-db | 2026-09-21 20:27:02 0 [Note] Server socket created on IP: '::', port: '3306'.\nbookstack-db | 2026-09-21 20:27:02 0 [Note] mariadbd: Event Scheduler: Loaded 0 events\nbookstack-db | 2026-09-21 20:27:02 0 [Note] mariadbd: ready for connections.\nbookstack-db | Version: '12.3.3-MariaDB-ubu2404' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution\nbookstack-db | 2026-09-21 20:27:08 5 [Warning] Aborted connection 5 to db: 'unconnected' user: 'unauthenticated' host: '172.19.0.3' (This connection closed normally without authentication)\nbookstack-db | 2026-09-21 20:28:07 21 [Warning] Aborted connection 21 to db: 'unconnected' user: 'unauthenticated' host: '172.19.0.3' (This connection closed normally without authentication)\nbookstack | [migrations] started\nbookstack | [migrations] 01-nginx-site-confs-default: skipped\nbookstack | [migrations] 02-default-location: skipped\nbookstack | [migrations] done\nbookstack | ───────────────────────────────────────\nbookstack | \nbookstack | ██╗ ███████╗██╗ ██████╗\nbookstack | ██║ ██╔════╝██║██╔═══██╗\nbookstack | ██║ ███████╗██║██║ ██║\nbookstack | ██║ ╚════██║██║██║ ██║\nbookstack | ███████╗███████║██║╚██████╔╝\nbookstack | ╚══════╝╚══════╝╚═╝ ╚═════╝\nbookstack | \nbookstack | Brought to you by linuxserver.io\nbookstack | ───────────────────────────────────────\nbookstack | \nbookstack | To support the app dev(s) visit:\nbookstack | Bookstack: https://www.bookstackapp.com/donate/\nbookstack | \nbookstack | To support LSIO projects visit:\nbookstack | https://www.linuxserver.io/donate/\nbookstack | \nbookstack | ───────────────────────────────────────\nbookstack | GID/UID\nbookstack | ───────────────────────────────────────\nbookstack | \nbookstack | User UID: 1000\nbookstack | User GID: 1000\nbookstack | ───────────────────────────────────────\nbookstack | Linuxserver.io version: v26.05.5-ls284\nbookstack | Build-date: 2026-09-14T14:18:17+00:00\nbookstack | ───────────────────────────────────────\nbookstack | \nbookstack | using keys found in /config/keys\nbookstack | Waiting for DB to be available\nbookstack | \nbookstack | INFO Running migrations. \nbookstack | \nbookstack | 2026_07_27_201402_update_users_external_auth_id_collation ..... 35.51ms DONE\nbookstack | \nbookstack | [custom-init] No custom files found, skipping...\nbookstack | [ls.io-init] done.\n"}} diff --git a/documentation/audits/update-night-2026-09-21/apps/bookstack/badges.json b/documentation/audits/update-night-2026-09-21/apps/bookstack/badges.json new file mode 100644 index 00000000..0d1b078f --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/bookstack/badges.json @@ -0,0 +1,21 @@ +{ + "before": { + "hu": [], + "en": [] + }, + "after": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "drill_commit": "5e7c11de0297" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/bookstack/log.txt b/documentation/audits/update-night-2026-09-21/apps/bookstack/log.txt new file mode 100644 index 00000000..19a4dbe7 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/bookstack/log.txt @@ -0,0 +1,25 @@ +20:25:39 ==== bookstack: lscr.io/linuxserver/bookstack:26.05.2 -> lscr.io/linuxserver/bookstack:26.05.5 (sub=wiki, class=db-mariadb) +20:25:39 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +20:26:44 [1] deployed, state=running, pinned={'bookstack': 'lscr.io/linuxserver/bookstack:26.05.2', 'bookstack-db': 'mariadb:12.3'} +20:26:47 [2] seeding through the app's own front door +20:26:50 bookstack: artisan create-admin :: Admin account with email "drill-8c68b2bc@gate.invalid" successfully created! +20:26:50 [3] control C1 — reading the seed back BEFORE the update +20:26:55 bookstack: readback of the seeded account found=True :: This will delete any configure multi-factor authentication methods for user: - ID: 3 - Name: drill-4b3bec - Email: drill-8c68b2bc@gate.inval +20:26:55 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +20:27:30 [4] backup idle; last=None +20:27:31 [5] drill commit 5e7c11de0297: bookstack lscr.io/linuxserver/bookstack:26.05.2 -> lscr.io/linuxserver/bookstack:26.05.5 (push rc=0) +20:27:36 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +20:27:36 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +20:27:36 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +20:27:36 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +20:27:37 + 1.1s phase=pulling label=Új verzió letöltése… err=None hold=None +20:28:00 + 24.6s phase=starting label=Indítás az új verzióval… err=None hold=None +20:28:06 + 29.7s phase=verifying label=Működés ellenőrzése… err=None hold=None +20:28:21 + 45.1s phase=done label=Frissítve err=None hold=None +20:28:21 [7] reading the seed back AFTER the update +20:28:26 bookstack: readback of the seeded account found=True :: This will delete any configure multi-factor authentication methods for user: - ID: 3 - Name: drill-4b3bec - Email: drill-8c68b2bc@gate.inval +20:28:28 [8] pinned = {'bookstack': 'lscr.io/linuxserver/bookstack:26.05.5', 'bookstack-db': 'mariadb:12.3'} +20:28:28 [8] installed = {'bookstack': 'lscr.io/linuxserver/bookstack:26.05.5', 'bookstack-db': 'mariadb:12.3'} +20:28:28 [8] compose = ['image: lscr.io/linuxserver/bookstack:26.05.5', 'image: mariadb:12.3'] +20:28:28 [8] inspect = ['bookstack lscr.io/linuxserver/bookstack:26.05.5 running=true restarts=0', 'bookstack-db mariadb:12.3 running=true restarts=0'] +20:28:28 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/bookstack/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/bookstack/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/bookstack/observables-before.json new file mode 100644 index 00000000..4be5bd0a --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/bookstack/observables-before.json @@ -0,0 +1,19 @@ +{ + "pinned_images": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.2", + "bookstack-db": "mariadb:12.3" + }, + "installed_images": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.2", + "bookstack-db": "mariadb:12.3" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: lscr.io/linuxserver/bookstack:26.05.2", + "image: mariadb:12.3" + ], + "docker_inspect": [ + "bookstack lscr.io/linuxserver/bookstack:26.05.2 running=true restarts=0", + "bookstack-db mariadb:12.3 running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/bookstack/observables.json b/documentation/audits/update-night-2026-09-21/apps/bookstack/observables.json new file mode 100644 index 00000000..40d2a22e --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/bookstack/observables.json @@ -0,0 +1,43 @@ +{ + "before": { + "pinned_images": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.2", + "bookstack-db": "mariadb:12.3" + }, + "installed_images": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.2", + "bookstack-db": "mariadb:12.3" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: lscr.io/linuxserver/bookstack:26.05.2", + "image: mariadb:12.3" + ], + "docker_inspect": [ + "bookstack lscr.io/linuxserver/bookstack:26.05.2 running=true restarts=0", + "bookstack-db mariadb:12.3 running=true restarts=0" + ] + }, + "after": { + "pinned_images": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.5", + "bookstack-db": "mariadb:12.3" + }, + "installed_images": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.5", + "bookstack-db": "mariadb:12.3" + }, + "catalog_images": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.5", + "bookstack-db": "mariadb:12.3" + }, + "live_compose_image_lines": [ + "image: lscr.io/linuxserver/bookstack:26.05.5", + "image: mariadb:12.3" + ], + "docker_inspect": [ + "bookstack lscr.io/linuxserver/bookstack:26.05.5 running=true restarts=0", + "bookstack-db mariadb:12.3 running=true restarts=0" + ] + } +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/bookstack/phases.json b/documentation/audits/update-night-2026-09-21/apps/bookstack/phases.json new file mode 100644 index 00000000..8a025924 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/bookstack/phases.json @@ -0,0 +1,51 @@ +{ + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.1, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 24.6, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 29.7, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 45.1, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 45.1, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/bookstack/verdict.json b/documentation/audits/update-night-2026-09-21/apps/bookstack/verdict.json new file mode 100644 index 00000000..0638c96b --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/bookstack/verdict.json @@ -0,0 +1,50 @@ +{ + "harness_version": 1, + "app": "bookstack", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "db-mariadb", + "from": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.2", + "bookstack-db": "mariadb:12.3" + }, + "to": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.5", + "bookstack-db": "mariadb:12.3" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": null, + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 45.1, + "measured_at": "2026-09-21T18:25:39.453490+00:00", + "evidence": "apps/bookstack/", + "notes": [], + "observables_after": { + "pinned_images": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.5", + "bookstack-db": "mariadb:12.3" + }, + "installed_images": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.5", + "bookstack-db": "mariadb:12.3" + }, + "catalog_images": { + "bookstack": "lscr.io/linuxserver/bookstack:26.05.5", + "bookstack-db": "mariadb:12.3" + }, + "live_compose_image_lines": [ + "image: lscr.io/linuxserver/bookstack:26.05.5", + "image: mariadb:12.3" + ], + "docker_inspect": [ + "bookstack lscr.io/linuxserver/bookstack:26.05.5 running=true restarts=0", + "bookstack-db mariadb:12.3 running=true restarts=0" + ] + }, + "final_phase": "done", + "hold_reason": null, + "update_error": null +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/docmost/app-logs-after.txt b/documentation/audits/update-night-2026-09-21/apps/docmost/app-logs-after.txt new file mode 100644 index 00000000..af9def27 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/docmost/app-logs-after.txt @@ -0,0 +1,41 @@ +docmost-postgres | +docmost-postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization +docmost-postgres | +docmost-postgres | 2026-09-21 20:24:55.198 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +docmost-postgres | 2026-09-21 20:24:55.198 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 +docmost-postgres | 2026-09-21 20:24:55.198 CEST [1] LOG: listening on IPv6 address "::", port 5432 +docmost-postgres | 2026-09-21 20:24:55.203 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +docmost-postgres | 2026-09-21 20:24:55.212 CEST [30] LOG: database system was shut down at 2026-09-21 20:24:54 CEST +docmost-postgres | 2026-09-21 20:24:55.254 CEST [1] LOG: database system is ready to accept connections +docmost-redis | 1:C 21 Sep 2026 20:24:55.117 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. +docmost-redis | 1:C 21 Sep 2026 20:24:55.117 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo +docmost-redis | 1:C 21 Sep 2026 20:24:55.117 * Redis version=7.4.11, bits=64, commit=00000000, modified=0, pid=1, just started +docmost-redis | 1:C 21 Sep 2026 20:24:55.117 * Configuration loaded +docmost-redis | 1:M 21 Sep 2026 20:24:55.118 * Increased maximum number of open files to 10032 (it was originally set to 1024). +docmost-redis | 1:M 21 Sep 2026 20:24:55.118 * monotonic clock: POSIX clock_gettime +docmost-redis | 1:M 21 Sep 2026 20:24:55.119 * Running mode=standalone, port=6379. +docmost-redis | 1:M 21 Sep 2026 20:24:55.119 * Server initialized +docmost-redis | 1:M 21 Sep 2026 20:24:55.119 * Reading RDB base file on AOF loading... +docmost-redis | 1:M 21 Sep 2026 20:24:55.119 * Loading RDB produced by version 7.4.11 +docmost-redis | 1:M 21 Sep 2026 20:24:55.119 * RDB age 146 seconds +docmost-redis | 1:M 21 Sep 2026 20:24:55.119 * RDB memory usage when created 0.90 Mb +docmost-redis | 1:M 21 Sep 2026 20:24:55.119 * RDB is base AOF +docmost-redis | 1:M 21 Sep 2026 20:24:55.119 * Done loading RDB, keys loaded: 0, keys expired: 0. +docmost-redis | 1:M 21 Sep 2026 20:24:55.119 * DB loaded from base file appendonly.aof.1.base.rdb: 0.000 seconds +docmost-redis | 1:M 21 Sep 2026 20:24:55.121 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.002 seconds +docmost-redis | 1:M 21 Sep 2026 20:24:55.121 * DB loaded from append only file: 0.002 seconds +docmost-redis | 1:M 21 Sep 2026 20:24:55.121 * Opening AOF incr file appendonly.aof.1.incr.aof on server start +docmost-redis | 1:M 21 Sep 2026 20:24:55.121 * Ready to accept connections tcp +docmost | $ pnpm --filter ./apps/server run start:prod +docmost | $ cross-env NODE_ENV=production node dist/main +docmost | (node:45) ExperimentalWarning: localStorage is not available because --localstorage-file was not provided. +docmost | (Use `node --trace-warnings ...` to show where the warning was created) +docmost | {"level":"info","time":"2026-09-21T18:25:12.473Z","pid":45,"hostname":"0dc77b73c358","context":"RedisModule","msg":"default: the connection was successfully established"} +docmost | {"level":"info","time":"2026-09-21T18:25:12.728Z","pid":45,"hostname":"0dc77b73c358","context":"DatabaseModule","msg":"Establishing database connection"} +docmost | {"level":"info","time":"2026-09-21T18:25:12.768Z","pid":45,"hostname":"0dc77b73c358","context":"DatabaseModule","msg":"Database connection successful"} +docmost | {"level":"info","time":"2026-09-21T18:25:13.099Z","pid":45,"hostname":"0dc77b73c358","context":"DatabaseMigrationService","msg":"Migration \"20260824T211732-page-title-trgm-index\" executed successfully"} +docmost | {"level":"info","time":"2026-09-21T18:25:13.099Z","pid":45,"hostname":"0dc77b73c358","context":"DatabaseMigrationService","msg":"Migration \"20260825T022612-oauth\" executed successfully"} +docmost | {"level":"info","time":"2026-09-21T18:25:13.099Z","pid":45,"hostname":"0dc77b73c358","context":"DatabaseMigrationService","msg":"Migration \"20260902T121326-siem-destinations\" executed successfully"} +docmost | {"level":"info","time":"2026-09-21T18:25:13.099Z","pid":45,"hostname":"0dc77b73c358","context":"DatabaseMigrationService","msg":"Migration \"20260904T171920-public-spaces\" executed successfully"} +docmost | {"level":"info","time":"2026-09-21T18:25:13.148Z","pid":45,"hostname":"0dc77b73c358","context":"NestApplication","msg":"Nest application successfully started"} +docmost | {"level":"info","time":"2026-09-21T18:25:13.161Z","pid":45,"hostname":"0dc77b73c358","context":"NestApplication","msg":"Listening on http://127.0.0.1:3000 / https://docs.enkisfelhom.hu"} diff --git a/documentation/audits/update-night-2026-09-21/apps/docmost/badges.json b/documentation/audits/update-night-2026-09-21/apps/docmost/badges.json new file mode 100644 index 00000000..a8b77d41 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/docmost/badges.json @@ -0,0 +1,21 @@ +{ + "before": { + "hu": [], + "en": [] + }, + "after": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "drill_commit": "a40ae09be943" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/docmost/log.txt b/documentation/audits/update-night-2026-09-21/apps/docmost/log.txt new file mode 100644 index 00000000..2177368f --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/docmost/log.txt @@ -0,0 +1,25 @@ +20:21:17 ==== docmost: docmost/docmost:0.95.0 -> docmost/docmost:0.96.0 (sub=docs, class=db-postgres) +20:21:18 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +20:22:53 [1] deployed, state=running, pinned={'docmost': 'docmost/docmost:0.95.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'} +20:22:55 [2] seeding through the app's own front door +20:22:56 docmost: /api/auth/setup http=200 rc=0 +20:22:56 [3] control C1 — reading the seed back BEFORE the update +20:22:56 docmost: login as the seeded user http=200 ok=True +20:22:56 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +20:23:32 [4] backup idle; last=None +20:23:33 [5] drill commit a40ae09be943: docmost docmost/docmost:0.95.0 -> docmost/docmost:0.96.0 (push rc=0) +20:23:37 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +20:23:37 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +20:23:37 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +20:23:37 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +20:23:38 + 1.0s phase=pulling label=Új verzió letöltése… err=None hold=None +20:24:54 + 76.9s phase=starting label=Indítás az új verzióval… err=None hold=None +20:25:06 + 88.2s phase=verifying label=Működés ellenőrzése… err=None hold=None +20:25:21 + 103.5s phase=done label=Frissítve err=None hold=None +20:25:21 [7] reading the seed back AFTER the update +20:25:22 docmost: login as the seeded user http=200 ok=True +20:25:24 [8] pinned = {'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'} +20:25:24 [8] installed = {'docmost': 'docmost/docmost:0.96.0', 'docmost-postgres': 'postgres:16-alpine', 'docmost-redis': 'redis:7-alpine'} +20:25:24 [8] compose = ['image: docmost/docmost:0.96.0', 'image: postgres:16-alpine', 'image: redis:7-alpine'] +20:25:24 [8] inspect = ['docmost docmost/docmost:0.96.0 running=true restarts=0', 'docmost-postgres postgres:16-alpine running=true restarts=0', 'docmost-redis redis:7-alpine running=true restarts=0'] +20:25:24 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/docmost/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/docmost/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/docmost/observables-before.json new file mode 100644 index 00000000..f3343e94 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/docmost/observables-before.json @@ -0,0 +1,23 @@ +{ + "pinned_images": { + "docmost": "docmost/docmost:0.95.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "installed_images": { + "docmost": "docmost/docmost:0.95.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: docmost/docmost:0.95.0", + "image: postgres:16-alpine", + "image: redis:7-alpine" + ], + "docker_inspect": [ + "docmost docmost/docmost:0.95.0 running=true restarts=0", + "docmost-postgres postgres:16-alpine running=true restarts=0", + "docmost-redis redis:7-alpine running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/docmost/observables.json b/documentation/audits/update-night-2026-09-21/apps/docmost/observables.json new file mode 100644 index 00000000..cf506a12 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/docmost/observables.json @@ -0,0 +1,52 @@ +{ + "before": { + "pinned_images": { + "docmost": "docmost/docmost:0.95.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "installed_images": { + "docmost": "docmost/docmost:0.95.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: docmost/docmost:0.95.0", + "image: postgres:16-alpine", + "image: redis:7-alpine" + ], + "docker_inspect": [ + "docmost docmost/docmost:0.95.0 running=true restarts=0", + "docmost-postgres postgres:16-alpine running=true restarts=0", + "docmost-redis redis:7-alpine running=true restarts=0" + ] + }, + "after": { + "pinned_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "installed_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "catalog_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "live_compose_image_lines": [ + "image: docmost/docmost:0.96.0", + "image: postgres:16-alpine", + "image: redis:7-alpine" + ], + "docker_inspect": [ + "docmost docmost/docmost:0.96.0 running=true restarts=0", + "docmost-postgres postgres:16-alpine running=true restarts=0", + "docmost-redis redis:7-alpine running=true restarts=0" + ] + } +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/docmost/phases.json b/documentation/audits/update-night-2026-09-21/apps/docmost/phases.json new file mode 100644 index 00000000..f79461b7 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/docmost/phases.json @@ -0,0 +1,51 @@ +{ + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.0, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 76.9, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 88.2, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 103.5, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 103.6, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/docmost/verdict.json b/documentation/audits/update-night-2026-09-21/apps/docmost/verdict.json new file mode 100644 index 00000000..e2210701 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/docmost/verdict.json @@ -0,0 +1,57 @@ +{ + "harness_version": 1, + "app": "docmost", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "db-postgres", + "from": { + "docmost": "docmost/docmost:0.95.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "to": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "docmost | {\"level\":\"info\",\"time\":\"2026-09-21T18:25:13.099Z\",\"pid\":45,\"hostname\":\"0dc77b73c358\",\"context\":\"DatabaseMigrationService\",\"msg\":\"Migration \\\"20260824T211732-page-title-trgm-index\\\" executed successfully\"}", + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 103.6, + "measured_at": "2026-09-21T18:21:18.018992+00:00", + "evidence": "apps/docmost/", + "notes": [], + "observables_after": { + "pinned_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "installed_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "catalog_images": { + "docmost": "docmost/docmost:0.96.0", + "docmost-postgres": "postgres:16-alpine", + "docmost-redis": "redis:7-alpine" + }, + "live_compose_image_lines": [ + "image: docmost/docmost:0.96.0", + "image: postgres:16-alpine", + "image: redis:7-alpine" + ], + "docker_inspect": [ + "docmost docmost/docmost:0.96.0 running=true restarts=0", + "docmost-postgres postgres:16-alpine running=true restarts=0", + "docmost-redis redis:7-alpine running=true restarts=0" + ] + }, + "final_phase": "done", + "hold_reason": null, + "update_error": null +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/gitea/log.txt b/documentation/audits/update-night-2026-09-21/apps/gitea/log.txt new file mode 100644 index 00000000..bd15681c --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/gitea/log.txt @@ -0,0 +1,6 @@ +20:37:09 ==== gitea: gitea/gitea:1.27.0 -> gitea/gitea:1.27.3 (sub=git, class=other) +20:37:10 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +20:37:25 [1] deployed, controller state=running, pinned={'gitea': 'gitea/gitea:1.27.0'} +20:37:27 [2] seeding through the app's own front door +20:37:30 gitea: admin user create :: 2026/09/21 20:37:30 modules/setting/setting.go:106:MustInstalled() [F] Unable to load config file for a installed Gitea instance, you should +20:37:30 [9] verdict inconclusive -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/gitea/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/gitea/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/gitea/observables-before.json new file mode 100644 index 00000000..bc9a383d --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/gitea/observables-before.json @@ -0,0 +1,15 @@ +{ + "pinned_images": { + "gitea": "gitea/gitea:1.27.0" + }, + "installed_images": { + "gitea": "gitea/gitea:1.27.0" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: gitea/gitea:1.27.0" + ], + "docker_inspect": [ + "gitea gitea/gitea:1.27.0 running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/gitea/verdict.json b/documentation/audits/update-night-2026-09-21/apps/gitea/verdict.json new file mode 100644 index 00000000..a3de9a56 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/gitea/verdict.json @@ -0,0 +1,24 @@ +{ + "harness_version": 1, + "app": "gitea", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "other", + "from": { + "gitea": "gitea/gitea:1.27.0" + }, + "to": {}, + "verdict": "inconclusive", + "seed_read_before": false, + "seed_read_after": false, + "healthy_after": false, + "migration_observed": null, + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 20.1, + "measured_at": "2026-09-21T18:37:10.025072+00:00", + "evidence": "apps/gitea/", + "notes": [ + "seed refused through the app's own route — see log.txt for what was tried", + "seed route did not work tonight" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/grafana/app-logs-after.txt b/documentation/audits/update-night-2026-09-21/apps/grafana/app-logs-after.txt new file mode 100644 index 00000000..baf8cf04 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/grafana/app-logs-after.txt @@ -0,0 +1,276 @@ +grafana | logger=settings t=2026-09-21T21:04:46.555328402+02:00 level=info msg="Starting Grafana" version=13.2.2 commit=1bea008f7e4e858b6824c9e364d608bd4d10b13a branch=release-13.2.2#patched compiled=2026-09-14T15:01:48+02:00 +grafana | logger=settings t=2026-09-21T21:04:46.556182975+02:00 level=info msg="Unified migration configs enforced" storage_type=unified target=[all] +grafana | logger=settings t=2026-09-21T21:04:46.556199236+02:00 level=info msg="Enforcing mode 5 for resource in unified storage" resource=folders.folder.grafana.app +grafana | logger=settings t=2026-09-21T21:04:46.556204085+02:00 level=info msg="Enforcing mode 5 for resource in unified storage" resource=dashboards.dashboard.grafana.app +grafana | logger=settings t=2026-09-21T21:04:46.556207893+02:00 level=info msg="Enforcing mode 5 for resource in unified storage" resource=shorturls.shorturl.grafana.app +grafana | logger=settings t=2026-09-21T21:04:46.556213293+02:00 level=info msg="Enforcing mode 5 for resource in unified storage" resource=playlists.playlist.grafana.app +grafana | logger=settings t=2026-09-21T21:04:46.556401198+02:00 level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini +grafana | logger=settings t=2026-09-21T21:04:46.556408261+02:00 level=info msg="Config loaded from" file=/etc/grafana/grafana.ini +grafana | logger=settings t=2026-09-21T21:04:46.556411778+02:00 level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" +grafana | logger=settings t=2026-09-21T21:04:46.556415214+02:00 level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" +grafana | logger=settings t=2026-09-21T21:04:46.556418991+02:00 level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" +grafana | logger=settings t=2026-09-21T21:04:46.556423379+02:00 level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" +grafana | logger=settings t=2026-09-21T21:04:46.556427537+02:00 level=info msg="Config overridden from command line" arg="default.log.mode=console" +grafana | logger=settings t=2026-09-21T21:04:46.556432446+02:00 level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" +grafana | logger=settings t=2026-09-21T21:04:46.556436314+02:00 level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" +grafana | logger=settings t=2026-09-21T21:04:46.55643986+02:00 level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" +grafana | logger=settings t=2026-09-21T21:04:46.556443868+02:00 level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" +grafana | logger=settings t=2026-09-21T21:04:46.556448336+02:00 level=info msg="Config overridden from Environment variable" var="GF_SERVER_ROOT_URL=https://grafana.enkisfelhom.hu" +grafana | logger=settings t=2026-09-21T21:04:46.556452705+02:00 level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********" +grafana | logger=settings t=2026-09-21T21:04:46.556457053+02:00 level=info msg="Config overridden from Environment variable" var="GF_PATHS_HOME=/usr/share/grafana" +grafana | logger=settings t=2026-09-21T21:04:46.556468975+02:00 level=info msg="Config overridden from Environment variable" var="GF_PATHS_CONFIG=/etc/grafana/grafana.ini" +grafana | logger=settings t=2026-09-21T21:04:46.556473634+02:00 level=info msg=Target target=[all] +grafana | logger=settings t=2026-09-21T21:04:46.556479145+02:00 level=info msg="Path Home" path=/usr/share/grafana +grafana | logger=settings t=2026-09-21T21:04:46.556482591+02:00 level=info msg="Path Data" path=/var/lib/grafana +grafana | logger=settings t=2026-09-21T21:04:46.556485947+02:00 level=info msg="Path Logs" path=/var/log/grafana +grafana | logger=settings t=2026-09-21T21:04:46.556489174+02:00 level=info msg="Path Plugins" path="[/var/lib/grafana/plugins /usr/share/grafana/data/plugins-bundled]" +grafana | logger=settings t=2026-09-21T21:04:46.556499212+02:00 level=info msg="Path Provisioning" path=/etc/grafana/provisioning +grafana | logger=settings t=2026-09-21T21:04:46.556503511+02:00 level=info msg="App mode production" +grafana | logger=featuremgmt t=2026-09-21T21:04:46.557422075+02:00 level=info msg=FeatureToggles alertRuleRestore=true alertingBulkActionsInUI=true alertingImportYAMLUI=true alertingListViewV2=true alertingMigrationUI=true alertingNavigationV2=true alertingNotificationsStepMode=true alertingQueryAndExpressionsStepMode=true alertingRulePermanentlyDelete=true alertingRuleRecoverDeleted=true alertingRuleVersionHistoryRestore=true alertingUIOptimizeReducer=true alertingUIUseBackendFilters=true alertingUIUseFullyCompatBackendFilters=true alertingUseNewSimplifiedRoutingHashAlgorithm=true annotationPermissionUpdate=true auditLoggingAppPlatform=true awsAsyncQueryCaching=true awsDatasourcesTempCredentials=true azureMonitorEnableUserAuth=true azureMonitorPrometheusExemplars=true azureResourcePickerUpdates=true cloudWatchCrossAccountQuerying=true cloudWatchNewLabelParsing=true cloudWatchRoundUpEndTime=true dashboardNewLayouts=true dashboardUnifiedDrilldownControls=true disableScriptedDashboards=true enableSCIM=true feedbackButton=true grafana.dashboardSettingsRedesign=true grafana.newPreferencesPage=true grafana.queryVarEditorRedesign=true grafana.scenesFlickeringFix=true grafana.unifiedDataSourcePicker=true grafana.viewPanelPane=true grafanaAssistantInProfilesDrilldown=true improvedExternalSessionHandling=true improvedExternalSessionHandlingSAML=true lokiLabelNamesQueryApi=true lokiQuerySplitting=true newClickhouseConfigPageDesign=true newSavedQueriesExperience=true onlyStoreActionSets=true preferences.rerouteLegacyAPIs=true profilesExemplars=true prometheusAzureOverrideAudience=true prometheusTypeMigration=true provisioning.gitConventions=true provisioning.readmes=true provisioning.userAttribution=true provisioningFolderMetadata=true pyroscopeUTF8LabelNames=true queryLibrary=true react19=true rememberUserOrgForSso=true renderAuthJWT=true restrictedPluginApis=true savedQueriesRBAC=true sqlExpressions=true useKubernetesShortURLsAPI=true useSessionStorageForRedirection=true +grafana | logger=sqlstore t=2026-09-21T21:04:46.557494251+02:00 level=info msg="Connecting to DB" dbtype=sqlite3 +grafana | logger=sqlstore t=2026-09-21T21:04:46.557500644+02:00 level=info msg="Using SQLite driver" driver=modernc.org/sqlite +grafana | logger=migrator t=2026-09-21T21:04:46.558198331+02:00 level=info msg="Starting DB migrations" +grafana | logger=migrator t=2026-09-21T21:04:46.569406406+02:00 level=info msg="Executing migration" id="add manager_kind column to provenance_type table" +grafana | logger=migrator t=2026-09-21T21:04:46.572170886+02:00 level=info msg="Migration successfully executed" id="add manager_kind column to provenance_type table" duration=2.760553ms +grafana | logger=migrator t=2026-09-21T21:04:46.586646823+02:00 level=info msg="Executing migration" id="add manager_identity column to provenance_type table" +grafana | logger=migrator t=2026-09-21T21:04:46.58992308+02:00 level=info msg="Migration successfully executed" id="add manager_identity column to provenance_type table" duration=3.275686ms +grafana | logger=migrator t=2026-09-21T21:04:46.605101522+02:00 level=info msg="Executing migration" id="create table nats_discovery_peers" +grafana | logger=migrator t=2026-09-21T21:04:46.605700894+02:00 level=info msg="Migration successfully executed" id="create table nats_discovery_peers" duration=597.969µs +grafana | logger=migrator t=2026-09-21T21:04:46.623164291+02:00 level=info msg="Executing migration" id="Change key_path collation of nats_discovery_peers in postgres" +grafana | logger=migrator t=2026-09-21T21:04:46.623198095+02:00 level=info msg="Migration successfully executed" id="Change key_path collation of nats_discovery_peers in postgres" duration=36.349µs +grafana | logger=migrator t=2026-09-21T21:04:46.634349463+02:00 level=info msg="Executing migration" id="alter alert_rule_state id column to bigint for postgres" +grafana | logger=migrator t=2026-09-21T21:04:46.634378147+02:00 level=info msg="Migration successfully executed" id="alter alert_rule_state id column to bigint for postgres" duration=31.069µs +grafana | logger=migrator t=2026-09-21T21:04:46.648572432+02:00 level=info msg="Executing migration" id="alter alert_rule_state id sequence to bigint for postgres" +grafana | logger=migrator t=2026-09-21T21:04:46.648600034+02:00 level=info msg="Migration successfully executed" id="alter alert_rule_state id sequence to bigint for postgres" duration=30.127µs +grafana | logger=migrator t=2026-09-21T21:04:46.664103329+02:00 level=info msg="migrations completed" performed=6 skipped=713 duration=95.228456ms +grafana | logger=secrets t=2026-09-21T21:04:46.665044878+02:00 level=info msg="Envelope encryption state" currentprovider=secretKey.v1 +grafana | logger=plugin.angulardetectorsprovider.dynamic t=2026-09-21T21:04:46.745685247+02:00 level=info msg="Restored cache from database" duration=756.258µs +grafana | logger=resource-db t=2026-09-21T21:04:46.746632576+02:00 level=info msg="Using database section" db_type=sqlite3 +grafana | logger=accesscontrol.service t=2026-09-21T21:04:46.746755839+02:00 level=info msg="Starting migration to remove deprecated permissions" migration=removeDeprecatedPermissions +grafana | logger=accesscontrol.service t=2026-09-21T21:04:46.747649035+02:00 level=info msg="Completed migration to remove deprecated permissions" migration=removeDeprecatedPermissions totalRemoved=0 duration=892.775µs +grafana | logger=plugin.store t=2026-09-21T21:04:46.804510311+02:00 level=info msg="Loading plugins..." +grafana | logger=plugins.registration t=2026-09-21T21:04:46.920264939+02:00 level=info msg="Plugin registered" pluginId=grafana-exploretraces-app +grafana | logger=plugins.registration t=2026-09-21T21:04:47.016524944+02:00 level=info msg="Plugin registered" pluginId=grafana-lokiexplore-app +grafana | logger=plugins.registration t=2026-09-21T21:04:47.080112561+02:00 level=info msg="Plugin registered" pluginId=grafana-metricsdrilldown-app +grafana | logger=plugins.registration t=2026-09-21T21:04:47.144369763+02:00 level=info msg="Plugin registered" pluginId=grafana-pyroscope-app +grafana | logger=plugins.registration t=2026-09-21T21:04:47.229460346+02:00 level=info msg="Plugin registered" pluginId=elasticsearch +grafana | logger=plugins.registration t=2026-09-21T21:04:47.30980137+02:00 level=info msg="Plugin registered" pluginId=grafana-postgresql-datasource +grafana | logger=plugins.registration t=2026-09-21T21:04:47.368534029+02:00 level=info msg="Plugin registered" pluginId=grafana-pyroscope-datasource +grafana | logger=plugins.registration t=2026-09-21T21:04:47.458872162+02:00 level=info msg="Plugin registered" pluginId=influxdb +grafana | logger=plugins.registration t=2026-09-21T21:04:47.5081467+02:00 level=info msg="Plugin registered" pluginId=jaeger +grafana | logger=plugins.registration t=2026-09-21T21:04:47.597745116+02:00 level=info msg="Plugin registered" pluginId=loki +grafana | logger=plugins.registration t=2026-09-21T21:04:47.687783262+02:00 level=info msg="Plugin registered" pluginId=mssql +grafana | logger=plugins.registration t=2026-09-21T21:04:47.76455211+02:00 level=info msg="Plugin registered" pluginId=mysql +grafana | logger=plugins.registration t=2026-09-21T21:04:47.809713612+02:00 level=info msg="Plugin registered" pluginId=opentsdb +grafana | logger=plugins.registration t=2026-09-21T21:04:47.919068225+02:00 level=info msg="Plugin registered" pluginId=prometheus +grafana | logger=plugins.registration t=2026-09-21T21:04:47.978599843+02:00 level=info msg="Plugin registered" pluginId=stackdriver +grafana | logger=plugins.registration t=2026-09-21T21:04:48.071302861+02:00 level=info msg="Plugin registered" pluginId=tempo +grafana | logger=plugins.registration t=2026-09-21T21:04:48.116684729+02:00 level=info msg="Plugin registered" pluginId=zipkin +grafana | logger=plugin.store t=2026-09-21T21:04:48.116718973+02:00 level=info msg="Plugins loaded" count=55 duration=1.312209614s +grafana | logger=secret-migrator t=2026-09-21T21:04:48.117357549+02:00 level=info msg="Starting DB migrations" +grafana | logger=secret-migrator t=2026-09-21T21:04:48.118669316+02:00 level=info msg="migrations completed" performed=0 skipped=35 duration=103.084µs +grafana | logger=resource-db t=2026-09-21T21:04:48.121001901+02:00 level=info msg="Initializing Resource DB" db_type=sqlite3 open_conn=0 in_use_conn=0 idle_conn=0 max_open_conn=0 +grafana | logger=resource-migrator t=2026-09-21T21:04:48.121218931+02:00 level=info msg="Starting DB migrations" +grafana | logger=resource-migrator t=2026-09-21T21:04:48.126206219+02:00 level=info msg="Executing migration" id="create table resource_stats_daily" +grafana | logger=resource-migrator t=2026-09-21T21:04:48.126831931+02:00 level=info msg="Migration successfully executed" id="create table resource_stats_daily" duration=623.758µs +grafana | logger=resource-migrator t=2026-09-21T21:04:48.148625685+02:00 level=info msg="Executing migration" id="Change key_path collation of resource_stats_daily in postgres" +grafana | logger=resource-migrator t=2026-09-21T21:04:48.148655953+02:00 level=info msg="Migration successfully executed" id="Change key_path collation of resource_stats_daily in postgres" duration=29.766µs +grafana | logger=resource-migrator t=2026-09-21T21:04:48.157342145+02:00 level=info msg="Executing migration" id="create table resource_stats_aggregates" +grafana | logger=resource-migrator t=2026-09-21T21:04:48.157977495+02:00 level=info msg="Migration successfully executed" id="create table resource_stats_aggregates" duration=635.931µs +grafana | logger=resource-migrator t=2026-09-21T21:04:48.176203994+02:00 level=info msg="Executing migration" id="Change key_path collation of resource_stats_aggregates in postgres" +grafana | logger=resource-migrator t=2026-09-21T21:04:48.176237106+02:00 level=info msg="Migration successfully executed" id="Change key_path collation of resource_stats_aggregates in postgres" duration=35.657µs +grafana | logger=resource-migrator t=2026-09-21T21:04:48.185381985+02:00 level=info msg="Executing migration" id="create table resource_version_policy" +grafana | logger=resource-migrator t=2026-09-21T21:04:48.185971799+02:00 level=info msg="Migration successfully executed" id="create table resource_version_policy" duration=592.359µs +grafana | logger=resource-migrator t=2026-09-21T21:04:48.19738967+02:00 level=info msg="Executing migration" id="Change key_path collation of resource_version_policy in postgres" +grafana | logger=resource-migrator t=2026-09-21T21:04:48.19742125+02:00 level=info msg="Migration successfully executed" id="Change key_path collation of resource_version_policy in postgres" duration=33.404µs +grafana | logger=resource-migrator t=2026-09-21T21:04:48.211113616+02:00 level=info msg="migrations completed" performed=6 skipped=49 duration=85.061248ms +grafana | t=2026-09-21T21:04:48.211665118+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.211641513+02:00 msg="Using channel notifier" logger=sql-resource-server +grafana | logger=bleve-backend namespace=default group=advisor.grafana.app resource=checktypes reason=init t=2026-09-21T21:04:48.213429069+02:00 level=info msg="Building index using memory" +grafana | logger=bleve-backend namespace=default group=advisor.grafana.app resource=checktypes reason=init t=2026-09-21T21:04:48.216799383+02:00 level=info msg="Finished building index" elapsed=3.289532ms listRV=1790017383920012 +grafana | logger=bleve-backend namespace=default group=advisor.grafana.app resource=checktypes reason=init t=2026-09-21T21:04:48.2168507+02:00 level=info msg="Storing index in cache" key="{Namespace:default Group:advisor.grafana.app Resource:checktypes}" expiration=2026-09-21T21:14:48.216848876+02:00 +grafana | logger=resource-search t=2026-09-21T21:04:48.217035679+02:00 level=info msg="search index initialized" duration_secs=0 total_docs=5 +grafana | logger=storage.unified.migrations t=2026-09-21T21:04:48.224381541+02:00 level=info msg="Running migrations for unified storage" +grafana | logger=storage.unified.migrations t=2026-09-21T21:04:48.224467193+02:00 level=info msg="Migration is disabled in config, skipping" migration=snapshots +grafana | logger=storage.unified.migrations t=2026-09-21T21:04:48.224483433+02:00 level=info msg="Migration is disabled in config, skipping" migration=datasource +grafana | logger=storage.unified.migrations t=2026-09-21T21:04:48.224491799+02:00 level=info msg="Migration is disabled in config, skipping" migration=stars +grafana | logger=storage.unified.migrations t=2026-09-21T21:04:48.224499664+02:00 level=info msg="Migration is disabled in config, skipping" migration=preferences +grafana | logger=storage.unified.migrations t=2026-09-21T21:04:48.224511536+02:00 level=info msg="Migration is disabled in config, skipping" migration=querycacheconfigs +grafana | logger=unifiedstorage-migrator t=2026-09-21T21:04:48.224526374+02:00 level=info msg="Starting DB migrations" +grafana | logger=unifiedstorage-migrator t=2026-09-21T21:04:48.224974551+02:00 level=info msg="Executing migration" id="shorturls migration" +grafana | logger=storage.unified.migration_runner.shorturls t=2026-09-21T21:04:48.225114235+02:00 level=info msg="Starting migration for all organizations" org_count=1 resources=[shorturls.shorturl.grafana.app] +grafana | logger=storage.unified.migration_runner.shorturls t=2026-09-21T21:04:48.225155643+02:00 level=info msg="Stored migrator transaction in context for bulk operations (SQLite compatibility)" +grafana | logger=storage.unified.migration_runner.shorturls t=2026-09-21T21:04:48.228583927+02:00 level=info msg="Migrating organization" org_id=1 namespace=default +grafana | logger=storage.unified.migrator t=2026-09-21T21:04:48.228701939+02:00 level=info msg="start migrating legacy resources" namespace=default orgId=1 stackId=0 +grafana | logger=storage.unified.migration_runner.shorturls t=2026-09-21T21:04:48.228827306+02:00 level=info msg="Migration progress" org_id=1 count=-1 message="migrating short URLs..." +grafana | t=2026-09-21T21:04:48.228974184+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.228962892+02:00 msg="Using SQLite transaction from client context" logger=sql-resource-server +grafana | logger=storage.unified.migration_runner.shorturls t=2026-09-21T21:04:48.22944348+02:00 level=info msg="Migration progress" org_id=1 count=0 message="finished reading legacy short URLs from legacy short_url table in 595.615µs (0)" +grafana | logger=storage.unified.migration_runner.shorturls t=2026-09-21T21:04:48.229460512+02:00 level=info msg="Migration progress" org_id=1 count=0 message="finished converting legacy short URLs to unified storage format in 0s (0)" +grafana | logger=storage.unified.migration_runner.shorturls t=2026-09-21T21:04:48.229469509+02:00 level=info msg="Migration progress" org_id=1 count=0 message="finished writing short URLs to unified storage in 0s (0)" +grafana | logger=storage.unified.migration_runner.shorturls t=2026-09-21T21:04:48.229477564+02:00 level=info msg="Migration progress" org_id=1 count=-2 message="finished short URLs... (0)" +grafana | logger=storage.unified.migrator t=2026-09-21T21:04:48.229485389+02:00 level=info msg="finished migrating legacy resources" namespace=default orgId=1 stackId=0 +grafana | t=2026-09-21T21:04:48.229527098+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.229515095+02:00 msg="synchronize collection" key=default/shorturl.grafana.app/shorturls +grafana | t=2026-09-21T21:04:48.23068828+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.230677991+02:00 msg="get stats (still in transaction)" key=default/shorturl.grafana.app/shorturls +grafana | t=2026-09-21T21:04:48.231933542+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.23191672+02:00 msg="successfully locked RV" logger=sql-resource-server nextRV=1790017488231012 key=default/shorturl.grafana.app/shorturls +grafana | t=2026-09-21T21:04:48.232122349+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.232114173+02:00 msg="successfully saved RV" logger=sql-resource-server rv=1790017488231012 key=default/shorturl.grafana.app/shorturls +grafana | logger=storage.unified.migrator namespace=default orgId=1 resources=[shorturls.shorturl.grafana.app] t=2026-09-21T21:04:48.232765683+02:00 level=info msg="start rebuilding index for resources" +grafana | logger=storage.unified.migrator namespace=default orgId=1 resources=[shorturls.shorturl.grafana.app] t=2026-09-21T21:04:48.23350028+02:00 level=info msg="finished rebuilding index for resources" attempts=1 +grafana | logger=storage.unified.migration_runner.shorturls t=2026-09-21T21:04:48.233694547+02:00 level=info msg="Count validation" resource=shorturls.shorturl.grafana.app namespace=default legacy_count=0 unified_count=0 migration_summary_count=0 rejected=0 history=0 +grafana | logger=storage.unified.migration_runner.shorturls t=2026-09-21T21:04:48.233716088+02:00 level=info msg="Migration completed for organization" org_id=1 duration=5.096614ms processed=0 summaries=1 rejected=0 +grafana | logger=storage.unified.migrations t=2026-09-21T21:04:48.234147362+02:00 level=info msg="renaming legacy table" table=short_url newName=short_url_legacy sql="ALTER TABLE `short_url` RENAME TO `short_url_legacy`" +grafana | logger=storage.unified.migration_runner.shorturls t=2026-09-21T21:04:48.243749074+02:00 level=info msg="Migration completed successfully for all organizations" org_count=1 +grafana | logger=unifiedstorage-migrator t=2026-09-21T21:04:48.243781185+02:00 level=info msg="Migration successfully executed" id="shorturls migration" duration=18.805942ms +grafana | logger=unifiedstorage-migrator t=2026-09-21T21:04:48.255062478+02:00 level=info msg="migrations completed" performed=1 skipped=3 duration=30.1216ms +grafana | logger=storage.unified.migrations t=2026-09-21T21:04:48.255422468+02:00 level=info msg="Unified storage migrations completed successfully" +grafana | logger=live.push_http t=2026-09-21T21:04:48.263367501+02:00 level=info msg="Live Push Gateway initialization" +grafana | logger=ngalert.notifier t=2026-09-21T21:04:48.265417222+02:00 level=info component=alertmanager orgID=1 msg="template definitions loaded" mimir=0 grafana=0 total=0 maxTemplateOutput=10485760 +grafana | logger=ngalert.notifier component=alertmanager orgID=1 t=2026-09-21T21:04:48.266609523+02:00 level=info msg="Applying new configuration to Alertmanager" configHash="{12383335334731491449 16454292769479786371 11535544170606148457 1343157387507078592 10278876130067529486 4613426124549341787 5144611013662998819}" +grafana | logger=ngalert.notifier t=2026-09-21T21:04:48.266640652+02:00 level=info component=alertmanager orgID=1 msg="template definitions loaded" mimir=0 grafana=0 total=0 maxTemplateOutput=10485760 +grafana | logger=ngalert.writer t=2026-09-21T21:04:48.276846645+02:00 level=info msg="Setting up remote write using data sources" timeout=30s default_datasource_uid= +grafana | logger=ngalert.state.manager.persist t=2026-09-21T21:04:48.27701899+02:00 level=info msg="Using sync rule state persister" +grafana | logger=query_data t=2026-09-21T21:04:48.278396772+02:00 level=info msg="Query Service initialization" +grafana | logger=annotation.app t=2026-09-21T21:04:48.311216448+02:00 level=info msg="Annotation cleanup disabled (no retention TTL configured)" +grafana | logger=infra.usagestats.collector t=2026-09-21T21:04:48.313458472+02:00 level=info msg="registering usage stat providers" usageStatsProvidersLen=2 +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:48.436042496+02:00 level=info msg=starting module=*sqlstore.SQLStore +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:48.436183703+02:00 level=info msg=starting module=tracing +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:48.436229419+02:00 level=info msg=starting module=grafana-apiserver +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.759424353+02:00 level=info msg="Adding GroupVersion collections.grafana.app v1alpha1 to ResourceManager" +grafana | t=2026-09-21T21:04:48.759610304+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.759598181+02:00 msg="Zanzana is not enabled; skipping folder propagation hooks" +grafana | t=2026-09-21T21:04:48.75964533+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.75964014+02:00 msg="Zanzana is not enabled; skipping folder propagation hooks" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.760949854+02:00 level=info msg="Adding GroupVersion folder.grafana.app v1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.761601845+02:00 level=info msg="Adding GroupVersion folder.grafana.app v1beta1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.762529096+02:00 level=info msg="Adding GroupVersion userstorage.grafana.app v0alpha1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.763773205+02:00 level=info msg="Adding GroupVersion preferences.grafana.app v1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.764244455+02:00 level=info msg="Adding GroupVersion preferences.grafana.app v1alpha1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.769166279+02:00 level=info msg="Adding GroupVersion provisioning.grafana.app v0alpha1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.771456794+02:00 level=info msg="Adding GroupVersion provisioning.grafana.app v1beta1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.771821273+02:00 level=info msg="Adding GroupVersion features.grafana.app v0alpha1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.779350891+02:00 level=info msg="Adding GroupVersion dashboard.grafana.app v2 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.780506574+02:00 level=info msg="Adding GroupVersion dashboard.grafana.app v2beta1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.780919203+02:00 level=info msg="Adding GroupVersion dashboard.grafana.app v2alpha1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.781650774+02:00 level=info msg="Adding GroupVersion dashboard.grafana.app v0alpha1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.782040571+02:00 level=info msg="Adding GroupVersion dashboard.grafana.app v1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.782420178+02:00 level=info msg="Adding GroupVersion dashboard.grafana.app v1beta1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.78357586+02:00 level=info msg="Adding GroupVersion playlist.grafana.app v1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.784141798+02:00 level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" +grafana | t=2026-09-21T21:04:48.784239673+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.784227971+02:00 msg="Installed APIs for app" app=playlist +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.785779782+02:00 level=info msg="Adding GroupVersion plugins.grafana.app v0alpha1 to ResourceManager" +grafana | t=2026-09-21T21:04:48.785904507+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.785878729+02:00 msg="Installed APIs for app" app=plugins +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.787196086+02:00 level=info msg="Adding GroupVersion example.grafana.app v1alpha1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.787751154+02:00 level=info msg="Adding GroupVersion example.grafana.app v0alpha1 to ResourceManager" +grafana | t=2026-09-21T21:04:48.78790765+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.787884245+02:00 msg="Installed APIs for app" app=example +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.787975969+02:00 level=info msg="Skipping API quotas.grafana.app/v0alpha1 because it has no resources." +grafana | t=2026-09-21T21:04:48.78808238+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.78807728+02:00 msg="Installed APIs for app" app=quotas +grafana | logger=storage.unified.migrations t=2026-09-21T21:04:48.788422101+02:00 level=info msg="Resolved storage mode from migration log" resource=shorturls.shorturl.grafana.app mode=unified +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.789355654+02:00 level=info msg="Adding GroupVersion shorturl.grafana.app v1beta1 to ResourceManager" +grafana | t=2026-09-21T21:04:48.789455643+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.789441266+02:00 msg="Installed APIs for app" app=shorturl +grafana | t=2026-09-21T21:04:48.789637075+02:00 level=warn caller=logger.go:224 time=2026-09-21T21:04:48.789629902+02:00 msg="skipped registering status sub-resource that does not support dual writing" resource=alertrules.rules.alerting.grafana.app version=v0alpha1 storagePath=alertrules/status +grafana | t=2026-09-21T21:04:48.789655961+02:00 level=warn caller=logger.go:224 time=2026-09-21T21:04:48.789651493+02:00 msg="skipped registering status sub-resource that does not support dual writing" resource=recordingrules.rules.alerting.grafana.app version=v0alpha1 storagePath=recordingrules/status +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.791788418+02:00 level=info msg="Adding GroupVersion rules.alerting.grafana.app v0alpha1 to ResourceManager" +grafana | t=2026-09-21T21:04:48.791974239+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.791961776+02:00 msg="Installed APIs for app" app=alerting +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.795121452+02:00 level=info msg="Adding GroupVersion notifications.alerting.grafana.app v1beta1 to ResourceManager" +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.79722286+02:00 level=info msg="Adding GroupVersion notifications.alerting.grafana.app v0alpha1 to ResourceManager" +grafana | t=2026-09-21T21:04:48.797404113+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.797393142+02:00 msg="Installed APIs for app" app=alerting-notifications +grafana | logger=grafana-apiserver t=2026-09-21T21:04:48.799002291+02:00 level=info msg="Adding GroupVersion advisor.grafana.app v0alpha1 to ResourceManager" +grafana | t=2026-09-21T21:04:48.799180407+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:48.799163575+02:00 msg="Installed APIs for app" app=advisor +grafana | t=2026-09-21T21:04:49.481475777+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.481453154+02:00 msg="App initialized" app=quotas +grafana | t=2026-09-21T21:04:49.481554475+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.481548865+02:00 msg="App initialized" app=plugins +grafana | t=2026-09-21T21:04:49.481490214+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.481463724+02:00 msg="App initialized" app=example +grafana | t=2026-09-21T21:04:49.481508649+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.481494212+02:00 msg="App initialized" app=shorturl +grafana | t=2026-09-21T21:04:49.481652781+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.481643944+02:00 msg="App initialized" app=playlist +grafana | t=2026-09-21T21:04:49.481721972+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.481714488+02:00 msg="App initialized" app=alerting-notifications +grafana | t=2026-09-21T21:04:49.481744084+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.481736139+02:00 msg="App initialized" app=advisor +grafana | t=2026-09-21T21:04:49.482068045+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.482041235+02:00 msg="start concurrent job driver" logger=concurrent-job-driver num_drivers=3 job_timeout=20m0s lease_renewal_interval=20s +grafana | t=2026-09-21T21:04:49.482175087+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.482154318+02:00 msg="starting job cleanup controller" logger=job-cleanup-controller cleanup_interval=3m0s expiry=1m0s +grafana | t=2026-09-21T21:04:49.489513696+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.489491694+02:00 msg="App initialized" app=alerting +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.48958486+02:00 level=info msg=starting module=plugins.store +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.489637269+02:00 level=info msg=starting module=plugin.backgroundinstaller +grafana | logger=plugin.backgroundinstaller t=2026-09-21T21:04:49.489656515+02:00 level=info msg="Plugins installed" plugins=[] +grafana | logger=plugin.backgroundinstaller t=2026-09-21T21:04:49.489678848+02:00 level=info msg="Installing plugins" plugins="[{ID:elasticsearch Version: URL:} {ID:loki Version: URL:} {ID:jaeger Version: URL:} {ID:zipkin Version: URL:} {ID:stackdriver Version: URL:} {ID:grafana-pyroscope-datasource Version: URL:} {ID:grafana-metricsdrilldown-app Version: URL:} {ID:prometheus Version: URL:} {ID:tempo Version: URL:} {ID:grafana-pyroscope-app Version: URL:} {ID:mssql Version: URL:} {ID:mysql Version: URL:} {ID:opentsdb Version: URL:} {ID:grafana-exploretraces-app Version: URL:} {ID:influxdb Version: URL:} {ID:grafana-advisor-app Version: URL:} {ID:grafana-postgresql-datasource Version: URL:} {ID:grafana-lokiexplore-app Version: URL:}]" +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.489827148+02:00 level=info msg=starting module=provisioning +grafana | logger=provisioning.alerting t=2026-09-21T21:04:49.490802459+02:00 level=info msg="starting to provision alerting" +grafana | logger=provisioning.alerting t=2026-09-21T21:04:49.49083471+02:00 level=info msg="finished to provision alerting" +grafana | logger=bleve-backend namespace=default group=dashboard.grafana.app resource=dashboards reason=search t=2026-09-21T21:04:49.545784746+02:00 level=info msg="Building index using memory" +grafana | logger=bleve-backend namespace=default group=dashboard.grafana.app resource=dashboards reason=search t=2026-09-21T21:04:49.546698702+02:00 level=info msg="Finished building index" elapsed=760.596µs listRV=1790017380769997 +grafana | logger=bleve-backend namespace=default group=dashboard.grafana.app resource=dashboards reason=search t=2026-09-21T21:04:49.546734069+02:00 level=info msg="Storing index in cache" key="{Namespace:default Group:dashboard.grafana.app Resource:dashboards}" expiration=2026-09-21T21:14:49.546732907+02:00 +grafana | logger=provisioning.dashboard t=2026-09-21T21:04:49.560966846+02:00 level=info msg="starting to provision dashboards" +grafana | logger=provisioning.dashboard t=2026-09-21T21:04:49.560996873+02:00 level=info msg="finished to provision dashboards" +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.575587425+02:00 level=info msg=starting module=*updatemanager.GrafanaService +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.575619786+02:00 level=info msg=starting module=*pluginexternal.Service +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.57563225+02:00 level=info msg=starting module=*appregistry.Service +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.57565355+02:00 level=info msg=starting module=*metric.Service +grafana | logger=app-registry t=2026-09-21T21:04:49.575672816+02:00 level=info msg="app registry initialized" +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.575678728+02:00 level=info msg=starting module=*remotecache.RemoteCache +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.575691892+02:00 level=info msg=starting module=*authimpl.UserAuthTokenService +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.575704526+02:00 level=info msg=starting module=*manager.ServiceAccountsService +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.57582819+02:00 level=info msg=starting module=*store.standardStorageService +grafana | logger=grafanaStorageLogger t=2026-09-21T21:04:49.575845092+02:00 level=info msg="Storage starting" +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.575866913+02:00 level=info msg=starting module=*service.DashboardUpdater +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.57599792+02:00 level=info msg=starting module=*migrations.SecretMigrationProviderImpl +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.576042294+02:00 level=info msg=starting module=*cleanup.CleanUpService +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.576056451+02:00 level=info msg=starting module=*api.HTTPServer +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.576158844+02:00 level=info msg=starting module=*live.GrafanaLive +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.576221963+02:00 level=info msg=starting module=*anonimpl.AnonDeviceService +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.576249094+02:00 level=info msg=starting module=*pushhttp.Gateway +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.576289421+02:00 level=info msg=starting module=*ngalert.AlertNG +grafana | logger=ngalert t=2026-09-21T21:04:49.576363591+02:00 level=info msg="Primary node, alert rule evaluation enabled" +grafana | logger=ngalert.state.manager t=2026-09-21T21:04:49.576385081+02:00 level=info msg="Warming state cache for startup" +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.576756292+02:00 level=info msg=starting module=*metrics.InternalMetricsService +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.577201954+02:00 level=info msg=starting module=*service.DashboardServiceImpl +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.577250666+02:00 level=info msg=starting module=*angulardetectorsprovider.Dynamic +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.57734803+02:00 level=info msg=starting module=*notifications.NotificationService +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.577382104+02:00 level=info msg=starting module=*ssosettingsimpl.Service +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.577392183+02:00 level=info msg=starting module=*service.UsageStats +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.57740098+02:00 level=info msg=starting module=*garbagecollectionworker.Worker +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.577479167+02:00 level=info msg=starting module=*authz.EmbeddedZanzanaService +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.577497562+02:00 level=info msg=starting module=*rendering.RenderingService +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.575605309+02:00 level=info msg=starting module=*dualwrite.ZanzanaReconciler +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.576042655+02:00 level=info msg=starting module=*dynamic.KeyRetriever +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.577620725+02:00 level=info msg=starting module=*loginattemptimpl.Service +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.577648577+02:00 level=info msg=starting module=*supportbundlesimpl.Service +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.577751872+02:00 level=info msg=starting module=*updatemanager.PluginsService +grafana | logger=ngalert.multiorg.alertmanager t=2026-09-21T21:04:49.578055305+02:00 level=info msg="Starting MultiOrg Alertmanager" +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.575597314+02:00 level=info msg=starting module=*statscollector.Service +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.578196251+02:00 level=info msg=starting module=*manager.SecretsService +grafana | logger=backgroundsvcs.managerAdapter t=2026-09-21T21:04:49.578646442+02:00 level=info msg="All modules healthy" +grafana | logger=http.server t=2026-09-21T21:04:49.57959891+02:00 level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket= +grafana | logger=ngalert.state.manager t=2026-09-21T21:04:49.64586728+02:00 level=info msg="State cache has been initialized" rules=0 states=0 duration=69.480115ms +grafana | logger=ngalert.scheduler t=2026-09-21T21:04:49.645955326+02:00 level=info msg="Starting scheduler" tickInterval=10s maxAttempts=3 +grafana | logger=ngalert.scheduler t=2026-09-21T21:04:49.645994871+02:00 level=info msg=starting component=ticker first_tick=2026-09-21T21:04:50+02:00 +grafana | logger=plugin.backgroundinstaller t=2026-09-21T21:04:49.673761184+02:00 level=info msg="Installing plugin" pluginId=elasticsearch version= +grafana | t=2026-09-21T21:04:49.683049183+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.68302635+02:00 msg="Starting ConnectionController" logger=provisioning-connection-controller +grafana | t=2026-09-21T21:04:49.683081584+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.683075293+02:00 msg="Starting workers" logger=provisioning-connection-controller count=1 +grafana | t=2026-09-21T21:04:49.683102203+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.683096352+02:00 msg="Started workers" logger=provisioning-connection-controller +grafana | logger=grafana.update.checker t=2026-09-21T21:04:49.710546068+02:00 level=info msg="Update check succeeded" duration=134.915171ms +grafana | logger=plugins.update.checker t=2026-09-21T21:04:49.71175472+02:00 level=info msg="flag evaluation succeeded" flag="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" details="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" +grafana | logger=plugins.update.checker t=2026-09-21T21:04:49.711835663+02:00 level=info msg="flag evaluation succeeded" flag="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" details="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" +grafana | logger=plugins.update.checker t=2026-09-21T21:04:49.711864527+02:00 level=info msg="flag evaluation succeeded" flag="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" details="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" +grafana | logger=plugins.update.checker t=2026-09-21T21:04:49.711910775+02:00 level=info msg="flag evaluation succeeded" flag="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" details="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" +grafana | logger=plugins.update.checker t=2026-09-21T21:04:49.711937215+02:00 level=info msg="flag evaluation succeeded" flag="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" details="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" +grafana | logger=plugins.update.checker t=2026-09-21T21:04:49.711965989+02:00 level=info msg="flag evaluation succeeded" flag="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" details="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" +grafana | logger=plugins.update.checker t=2026-09-21T21:04:49.71199272+02:00 level=info msg="flag evaluation succeeded" flag="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" details="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" +grafana | logger=plugins.update.checker t=2026-09-21T21:04:49.712029088+02:00 level=info msg="Update check succeeded" duration=134.259663ms +grafana | logger=plugins.update.checker t=2026-09-21T21:04:49.712046521+02:00 level=info msg="flag evaluation succeeded" flag="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" details="{Value:false EvaluationDetails:{FlagKey:pluginsAutoUpdate FlagType:bool ResolutionDetail:{Variant:default Reason:STATIC ErrorCode: ErrorMessage: FlagMetadata:map[]}}}" +grafana | logger=plugin.installer t=2026-09-21T21:04:49.746301197+02:00 level=info msg="Updating plugin" pluginId=elasticsearch from=12.8.2 to=12.9.0 +grafana | logger=plugin.elasticsearch t=2026-09-21T21:04:49.749876218+02:00 level=info msg="plugin process exited" plugin=/usr/share/grafana/data/plugins-bundled/elasticsearch/gpx_grafana_elasticsearch_datasource_linux_amd64 id=21 +grafana | t=2026-09-21T21:04:49.78320237+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.783168556+02:00 msg="Starting RepositoryController" logger=provisioning-repository-controller +grafana | t=2026-09-21T21:04:49.78324483+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.783235583+02:00 msg="Starting workers" logger=provisioning-repository-controller count=1 +grafana | t=2026-09-21T21:04:49.783264788+02:00 level=info caller=logger.go:214 time=2026-09-21T21:04:49.783261071+02:00 msg="Started workers" logger=provisioning-repository-controller +grafana | logger=installer.fs t=2026-09-21T21:04:52.147390916+02:00 level=info msg="Downloaded and extracted elasticsearch v12.9.0 zip successfully to /var/lib/grafana/plugins/elasticsearch" +grafana | logger=plugins.registration t=2026-09-21T21:04:52.23333466+02:00 level=info msg="Plugin registered" pluginId=elasticsearch +grafana | logger=plugin.backgroundinstaller t=2026-09-21T21:04:52.233372732+02:00 level=info msg="Plugin successfully installed" pluginId=elasticsearch version= duration=2.559578215s +grafana | logger=plugin.backgroundinstaller t=2026-09-21T21:04:52.421710033+02:00 level=info msg="Installing plugin" pluginId=zipkin version= +grafana | logger=plugin.installer t=2026-09-21T21:04:52.517914534+02:00 level=info msg="Updating plugin" pluginId=zipkin from=12.4.6 to=12.4.8 +grafana | logger=plugin.zipkin t=2026-09-21T21:04:52.521447565+02:00 level=info msg="plugin process exited" plugin=/usr/share/grafana/data/plugins-bundled/zipkin/gpx_grafana-zipkin-datasource_linux_amd64 id=142 +grafana | logger=installer.fs t=2026-09-21T21:04:53.02117033+02:00 level=info msg="Downloaded and extracted zipkin v12.4.8 zip successfully to /var/lib/grafana/plugins/zipkin" +grafana | logger=plugins.registration t=2026-09-21T21:04:53.064778268+02:00 level=info msg="Plugin registered" pluginId=zipkin +grafana | logger=plugin.backgroundinstaller t=2026-09-21T21:04:53.064819566+02:00 level=info msg="Plugin successfully installed" pluginId=zipkin version= duration=643.082081ms +grafana | logger=plugin.backgroundinstaller t=2026-09-21T21:04:53.122700337+02:00 level=info msg="Installing plugin" pluginId=stackdriver version= +grafana | logger=plugin.installer t=2026-09-21T21:04:53.186134153+02:00 level=info msg="Updating plugin" pluginId=stackdriver from=12.6.1 to=12.6.2 +grafana | logger=plugin.stackdriver t=2026-09-21T21:04:53.18946859+02:00 level=info msg="plugin process exited" plugin=/usr/share/grafana/data/plugins-bundled/stackdriver/gpx_grafana_cloudmonitoring_datasource_linux_amd64 id=121 +grafana | logger=installer.fs t=2026-09-21T21:04:56.141607757+02:00 level=info msg="Downloaded and extracted stackdriver v12.6.2 zip successfully to /var/lib/grafana/plugins/stackdriver" +grafana | logger=plugins.registration t=2026-09-21T21:04:56.199249787+02:00 level=info msg="Plugin registered" pluginId=stackdriver +grafana | logger=plugin.backgroundinstaller t=2026-09-21T21:04:56.199457189+02:00 level=info msg="Plugin successfully installed" pluginId=stackdriver version= duration=3.076725293s +grafana | logger=plugin.backgroundinstaller t=2026-09-21T21:04:56.271364026+02:00 level=info msg="Installing plugin" pluginId=grafana-pyroscope-datasource version= +grafana | logger=plugin.installer t=2026-09-21T21:04:56.326267354+02:00 level=info msg="Updating plugin" pluginId=grafana-pyroscope-datasource from=13.0.4 to=13.0.6 +grafana | logger=plugin.grafana-pyroscope-datasource t=2026-09-21T21:04:56.330084022+02:00 level=info msg="plugin process exited" plugin=/usr/share/grafana/data/plugins-bundled/grafana-pyroscope-datasource/gpx_grafana-pyroscope-datasource_linux_amd64 id=42 +grafana | logger=context userId=1 orgId=1 uname=admin t=2026-09-21T21:04:56.758623345+02:00 level=info msg="Request Completed" method=GET path=/api/folders/nope18ec88bd1e status=404 remote_addr=192.168.0.180 time_ms=13 duration=13.051486ms size=51 referer= handler=/api/folders/:uid/ status_source=server diff --git a/documentation/audits/update-night-2026-09-21/apps/grafana/badges.json b/documentation/audits/update-night-2026-09-21/apps/grafana/badges.json new file mode 100644 index 00000000..74b9781a --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/grafana/badges.json @@ -0,0 +1,31 @@ +{ + "before": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "after": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "drill_commit": "f36b5183055e" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/grafana/log.txt b/documentation/audits/update-night-2026-09-21/apps/grafana/log.txt new file mode 100644 index 00000000..08bd17ea --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/grafana/log.txt @@ -0,0 +1,26 @@ +21:02:25 ==== grafana: grafana/grafana:13.1.0 -> grafana/grafana:13.2.2 (sub=grafana, class=other) +21:02:25 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['GF_SECURITY_ADMIN_PASSWORD'] +21:02:25 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +21:03:16 [1] deployed, controller state=running, pinned={'grafana': 'grafana/grafana:13.1.0'} +21:03:18 [2] seeding through the app's own front door +21:03:18 grafana: create folder http=200 +21:03:18 [3] control C1 — reading the seed back BEFORE the update +21:03:18 grafana: readback of the seeded folder http=200 ok=True +21:03:18 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +21:04:19 [4] backup idle; last=None +21:04:20 [5] drill commit f36b5183055e: grafana grafana/grafana:13.1.0 -> grafana/grafana:13.2.2 (push rc=0) +21:04:24 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +21:04:24 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +21:04:24 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +21:04:25 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +21:04:26 + 1.1s phase=pulling label=Új verzió letöltése… err=None hold=None +21:04:44 + 19.5s phase=starting label=Indítás az új verzióval… err=None hold=None +21:04:46 + 21.5s phase=verifying label=Működés ellenőrzése… err=None hold=None +21:04:51 + 26.7s phase=done label=Frissítve err=None hold=None +21:04:51 [7] reading the seed back AFTER the update +21:04:56 grafana: readback of the seeded folder http=200 ok=True +21:04:59 [8] pinned = {'grafana': 'grafana/grafana:13.2.2'} +21:04:59 [8] installed = {'grafana': 'grafana/grafana:13.2.2'} +21:04:59 [8] compose = ['image: grafana/grafana:13.2.2'] +21:04:59 [8] inspect = ['grafana grafana/grafana:13.2.2 running=true restarts=0'] +21:04:59 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/grafana/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/grafana/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/grafana/observables-before.json new file mode 100644 index 00000000..7ec82f29 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/grafana/observables-before.json @@ -0,0 +1,15 @@ +{ + "pinned_images": { + "grafana": "grafana/grafana:13.1.0" + }, + "installed_images": { + "grafana": "grafana/grafana:13.1.0" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: grafana/grafana:13.1.0" + ], + "docker_inspect": [ + "grafana grafana/grafana:13.1.0 running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/grafana/observables.json b/documentation/audits/update-night-2026-09-21/apps/grafana/observables.json new file mode 100644 index 00000000..7736bc2c --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/grafana/observables.json @@ -0,0 +1,34 @@ +{ + "before": { + "pinned_images": { + "grafana": "grafana/grafana:13.1.0" + }, + "installed_images": { + "grafana": "grafana/grafana:13.1.0" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: grafana/grafana:13.1.0" + ], + "docker_inspect": [ + "grafana grafana/grafana:13.1.0 running=true restarts=0" + ] + }, + "after": { + "pinned_images": { + "grafana": "grafana/grafana:13.2.2" + }, + "installed_images": { + "grafana": "grafana/grafana:13.2.2" + }, + "catalog_images": { + "grafana": "grafana/grafana:13.2.2" + }, + "live_compose_image_lines": [ + "image: grafana/grafana:13.2.2" + ], + "docker_inspect": [ + "grafana grafana/grafana:13.2.2 running=true restarts=0" + ] + } +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/grafana/phases.json b/documentation/audits/update-night-2026-09-21/apps/grafana/phases.json new file mode 100644 index 00000000..5371faf8 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/grafana/phases.json @@ -0,0 +1,51 @@ +{ + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.1, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 19.5, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 21.5, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 26.7, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 26.7, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/grafana/verdict.json b/documentation/audits/update-night-2026-09-21/apps/grafana/verdict.json new file mode 100644 index 00000000..070f1890 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/grafana/verdict.json @@ -0,0 +1,43 @@ +{ + "harness_version": 1, + "app": "grafana", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "other", + "from": { + "grafana": "grafana/grafana:13.1.0" + }, + "to": { + "grafana": "grafana/grafana:13.2.2" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "grafana | logger=settings t=2026-09-21T21:04:46.556182975+02:00 level=info msg=\"Unified migration configs enforced\" storage_type=unified target=[all]", + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 26.7, + "measured_at": "2026-09-21T19:02:25.646053+00:00", + "evidence": "apps/grafana/", + "notes": [], + "observables_after": { + "pinned_images": { + "grafana": "grafana/grafana:13.2.2" + }, + "installed_images": { + "grafana": "grafana/grafana:13.2.2" + }, + "catalog_images": { + "grafana": "grafana/grafana:13.2.2" + }, + "live_compose_image_lines": [ + "image: grafana/grafana:13.2.2" + ], + "docker_inspect": [ + "grafana grafana/grafana:13.2.2 running=true restarts=0" + ] + }, + "final_phase": "done", + "hold_reason": null, + "update_error": null +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/home-assistant/app-logs-after.txt b/documentation/audits/update-night-2026-09-21/apps/home-assistant/app-logs-after.txt new file mode 100644 index 00000000..f017974a --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/home-assistant/app-logs-after.txt @@ -0,0 +1,14 @@ +home-assistant | s6-rc: info: service s6rc-oneshot-runner: starting +home-assistant | s6-rc: info: service s6rc-oneshot-runner successfully started +home-assistant | s6-rc: info: service fix-attrs: starting +home-assistant | s6-rc: info: service fix-attrs successfully started +home-assistant | s6-rc: info: service legacy-cont-init: starting +home-assistant | s6-rc: info: service legacy-cont-init successfully started +home-assistant | s6-rc: info: service legacy-services: starting +home-assistant | services-up: info: copying legacy longrun home-assistant (no readiness notification) +home-assistant | s6-rc: info: service legacy-services successfully started +home-assistant | 2026-09-21 21:14:58.394 WARNING (ImportExecutor_0) [py.warnings] /usr/local/lib/python3.14/site-packages/rich/segment.py:547: SyntaxWarning: 'return' in a 'finally' block +home-assistant | return +home-assistant |  +home-assistant | 2026-09-21 21:15:03.281 WARNING (MainThread) [homeassistant.components.http.ban] Login attempt or request with invalid authentication from felhom-controller.traefik-public (172.18.0.7). Requested URL: '/api/'. (Go-http-client/1.1) +home-assistant | 2026-09-21 21:15:04.019 WARNING (MainThread) [homeassistant.components.http.ban] Login attempt or request with invalid authentication from 192.168.0.180 (192.168.0.180). Requested URL: '/auth/login_flow/1da21af37fa875863271de4302276d5d'. (curl/8.14.1) diff --git a/documentation/audits/update-night-2026-09-21/apps/home-assistant/badges.json b/documentation/audits/update-night-2026-09-21/apps/home-assistant/badges.json new file mode 100644 index 00000000..817de170 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/home-assistant/badges.json @@ -0,0 +1,31 @@ +{ + "before": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "after": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "drill_commit": "15a6d080f34a" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/home-assistant/log.txt b/documentation/audits/update-night-2026-09-21/apps/home-assistant/log.txt new file mode 100644 index 00000000..39344049 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/home-assistant/log.txt @@ -0,0 +1,25 @@ +21:10:14 ==== home-assistant: ghcr.io/home-assistant/home-assistant:2026.7.2 -> ghcr.io/home-assistant/home-assistant:2026.9.3 (sub=ha, class=other) +21:10:15 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +21:12:05 [1] deployed, controller state=running, pinned={'home-assistant': 'ghcr.io/home-assistant/home-assistant:2026.7.2'} +21:12:08 [2] seeding through the app's own front door +21:12:08 home-assistant: /api/onboarding/users http=200 +21:12:08 [3] control C1 — reading the seed back BEFORE the update +21:12:09 home-assistant: login as the seeded owner ok=True +21:12:09 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +21:13:14 [4] backup idle; last=None +21:13:15 [5] drill commit 15a6d080f34a: home-assistant ghcr.io/home-assistant/home-assistant:2026.7.2 -> ghcr.io/home-assistant/home-assistant:2026.9.3 (push rc=0) +21:13:19 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +21:13:19 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +21:13:20 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +21:13:20 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +21:13:21 + 1.1s phase=pulling label=Új verzió letöltése… err=None hold=None +21:14:42 + 82.0s phase=starting label=Indítás az új verzióval… err=None hold=None +21:14:48 + 88.2s phase=verifying label=Működés ellenőrzése… err=None hold=None +21:15:03 + 103.6s phase=done label=Frissítve err=None hold=None +21:15:03 [7] reading the seed back AFTER the update +21:15:04 home-assistant: login as the seeded owner ok=True +21:15:06 [8] pinned = {'home-assistant': 'ghcr.io/home-assistant/home-assistant:2026.9.3'} +21:15:06 [8] installed = {'home-assistant': 'ghcr.io/home-assistant/home-assistant:2026.9.3'} +21:15:06 [8] compose = ['image: ghcr.io/home-assistant/home-assistant:2026.9.3'] +21:15:06 [8] inspect = ['home-assistant ghcr.io/home-assistant/home-assistant:2026.9.3 running=true restarts=0'] +21:15:06 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/home-assistant/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/home-assistant/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/home-assistant/observables-before.json new file mode 100644 index 00000000..b2fa4b22 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/home-assistant/observables-before.json @@ -0,0 +1,15 @@ +{ + "pinned_images": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.7.2" + }, + "installed_images": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.7.2" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: ghcr.io/home-assistant/home-assistant:2026.7.2" + ], + "docker_inspect": [ + "home-assistant ghcr.io/home-assistant/home-assistant:2026.7.2 running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/home-assistant/observables.json b/documentation/audits/update-night-2026-09-21/apps/home-assistant/observables.json new file mode 100644 index 00000000..0f89b85b --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/home-assistant/observables.json @@ -0,0 +1,34 @@ +{ + "before": { + "pinned_images": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.7.2" + }, + "installed_images": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.7.2" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: ghcr.io/home-assistant/home-assistant:2026.7.2" + ], + "docker_inspect": [ + "home-assistant ghcr.io/home-assistant/home-assistant:2026.7.2 running=true restarts=0" + ] + }, + "after": { + "pinned_images": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.9.3" + }, + "installed_images": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.9.3" + }, + "catalog_images": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.9.3" + }, + "live_compose_image_lines": [ + "image: ghcr.io/home-assistant/home-assistant:2026.9.3" + ], + "docker_inspect": [ + "home-assistant ghcr.io/home-assistant/home-assistant:2026.9.3 running=true restarts=0" + ] + } +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/home-assistant/phases.json b/documentation/audits/update-night-2026-09-21/apps/home-assistant/phases.json new file mode 100644 index 00000000..65d2d5e9 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/home-assistant/phases.json @@ -0,0 +1,51 @@ +{ + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.1, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 82.0, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 88.2, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 103.6, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 103.6, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/home-assistant/verdict.json b/documentation/audits/update-night-2026-09-21/apps/home-assistant/verdict.json new file mode 100644 index 00000000..bfe3545e --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/home-assistant/verdict.json @@ -0,0 +1,43 @@ +{ + "harness_version": 1, + "app": "home-assistant", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "other", + "from": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.7.2" + }, + "to": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.9.3" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": null, + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 103.6, + "measured_at": "2026-09-21T19:10:14.843396+00:00", + "evidence": "apps/home-assistant/", + "notes": [], + "observables_after": { + "pinned_images": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.9.3" + }, + "installed_images": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.9.3" + }, + "catalog_images": { + "home-assistant": "ghcr.io/home-assistant/home-assistant:2026.9.3" + }, + "live_compose_image_lines": [ + "image: ghcr.io/home-assistant/home-assistant:2026.9.3" + ], + "docker_inspect": [ + "home-assistant ghcr.io/home-assistant/home-assistant:2026.9.3 running=true restarts=0" + ] + }, + "final_phase": "done", + "hold_reason": null, + "update_error": null +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/mealie/log.txt b/documentation/audits/update-night-2026-09-21/apps/mealie/log.txt new file mode 100644 index 00000000..d750b32a --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/mealie/log.txt @@ -0,0 +1,3 @@ +21:02:15 ==== mealie: ghcr.io/mealie-recipes/mealie:v3.20.1 -> ghcr.io/mealie-recipes/mealie:v3.27.0 (sub=recipes, class=db-postgres) +21:02:15 [1] deploy -> 500 {'ok': False, 'error': 'a(z) "recipes" aldomain már használatban van egy másik alkalmazásban'} +21:02:15 [9] verdict inconclusive -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/mealie/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/mealie/verdict.json b/documentation/audits/update-night-2026-09-21/apps/mealie/verdict.json new file mode 100644 index 00000000..d0f1ad08 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/mealie/verdict.json @@ -0,0 +1,21 @@ +{ + "harness_version": 1, + "app": "mealie", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "db-postgres", + "from": {}, + "to": {}, + "verdict": "inconclusive", + "seed_read_before": false, + "seed_read_after": false, + "healthy_after": false, + "migration_observed": null, + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 0.3, + "measured_at": "2026-09-21T19:02:15.239347+00:00", + "evidence": "apps/mealie/", + "notes": [ + "deploy never reached running — nothing else could be measured" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/n8n/app-logs-after.txt b/documentation/audits/update-night-2026-09-21/apps/n8n/app-logs-after.txt new file mode 100644 index 00000000..1be6c8a0 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/n8n/app-logs-after.txt @@ -0,0 +1,161 @@ +n8n | Initializing n8n process +n8n | n8n ready on ::, port 5678 +n8n | Migrations in progress, please do NOT stop the process. +n8n | Starting migration CreateInstanceAiEventsTable1784000000046 +n8n | Finished migration CreateInstanceAiEventsTable1784000000046 +n8n | Starting migration BackfillPreScopingOAuthGrantScopes1784000000047 +n8n | [BackfillPreScopingOAuthGrantScopes1784000000047] Rewrote 0 pre-scoping grants in oauth_refresh_tokens +n8n | [BackfillPreScopingOAuthGrantScopes1784000000047] Rewrote 0 pre-scoping grants in oauth_authorization_codes +n8n | Finished migration BackfillPreScopingOAuthGrantScopes1784000000047 +n8n | Starting migration AddScheduledTaskDispatchedAt1784000000049 +n8n | Finished migration AddScheduledTaskDispatchedAt1784000000049 +n8n | Starting migration AddHostRunIdToInstanceAiCheckpoints1784000000050 +n8n | Finished migration AddHostRunIdToInstanceAiCheckpoints1784000000050 +n8n | Starting migration BackfillInstanceAiEventLog1784000000051 +n8n | Finished migration BackfillInstanceAiEventLog1784000000051 +n8n | Starting migration CreateWorkflowReviewRequestTables1784000000052 +n8n | Finished migration CreateWorkflowReviewRequestTables1784000000052 +n8n | Starting migration AddStoredAtToAgentExecution1784815940110 +n8n | Finished migration AddStoredAtToAgentExecution1784815940110 +n8n | Starting migration AddInstanceCredentials1784815940111 +n8n | Finished migration AddInstanceCredentials1784815940111 +n8n | Starting migration CreateAgentEvalTables1784815940112 +n8n | Finished migration CreateAgentEvalTables1784815940112 +n8n | Starting migration AddAvailableInMcpToAgents1784897791636 +n8n | Finished migration AddAvailableInMcpToAgents1784897791636 +n8n | Starting migration ChangeInstalledNodeVersionType1785162364000 +n8n | Finished migration ChangeInstalledNodeVersionType1785162364000 +n8n | Starting migration AddIsFirstPartyToOAuthClients1785162364001 +n8n | Finished migration AddIsFirstPartyToOAuthClients1785162364001 +n8n | Starting migration AddAgentFileStorageColumns1785186578138 +n8n | Finished migration AddAgentFileStorageColumns1785186578138 +n8n | Starting migration CrashStaleEnqueuedExecutions1785247194306 +n8n | Finished migration CrashStaleEnqueuedExecutions1785247194306 +n8n | Starting migration AddMisfirePolicyToScheduler1785247194307 +n8n | Finished migration AddMisfirePolicyToScheduler1785247194307 +n8n | Starting migration CreateAgentChatAttachmentsTable1785255306000 +n8n | Finished migration CreateAgentChatAttachmentsTable1785255306000 +n8n | Starting migration AddSetupCompletedAtToAgents1785500832626 +n8n | Finished migration AddSetupCompletedAtToAgents1785500832626 +n8n | Starting migration AddAgentExecutionRuntimeState1785828155091 +n8n | Finished migration AddAgentExecutionRuntimeState1785828155091 +n8n | Starting migration CreateAgentCredentialDependencyTable1785828155092 +n8n | Finished migration CreateAgentCredentialDependencyTable1785828155092 +n8n | Starting migration AllowDiscordAgentChatSubscriptions1785840970000 +n8n | Finished migration AllowDiscordAgentChatSubscriptions1785840970000 +n8n | Starting migration CreateWorkflowReviewActivityTablesAndBaseline1785843640527 +n8n | Finished migration CreateWorkflowReviewActivityTablesAndBaseline1785843640527 +n8n | Starting migration CreatePollerStateTable1785926660580 +n8n | Finished migration CreatePollerStateTable1785926660580 +n8n | Starting migration AddReasonToWorkflowPublicationOutbox1786519946974 +n8n | Finished migration AddReasonToWorkflowPublicationOutbox1786519946974 +n8n | Starting migration AddCoalesceOwnerMisfirePolicy1786666615643 +n8n | Finished migration AddCoalesceOwnerMisfirePolicy1786666615643 +n8n | Starting migration AddAgentRevisionColumn1786666615644 +n8n | Finished migration AddAgentRevisionColumn1786666615644 +n8n | Starting migration AddAgentExecutionFailureSummary1787040021605 +n8n | Finished migration AddAgentExecutionFailureSummary1787040021605 +n8n | Starting migration CreateGitConnectionTable1787056876306 +n8n | Finished migration CreateGitConnectionTable1787056876306 +n8n | Starting migration AddTimezoneToAgentTasks1787057050000 +n8n | Finished migration AddTimezoneToAgentTasks1787057050000 +n8n | Starting migration CreateGitConnectionProjectTable1787089039726 +n8n | Finished migration CreateGitConnectionProjectTable1787089039726 +n8n | Starting migration DropWorkflowReviewActivityTypeCheckAndWorkflowIdColumn1787089039727 +n8n | Finished migration DropWorkflowReviewActivityTypeCheckAndWorkflowIdColumn1787089039727 +n8n | Starting migration AddProjectManageMembersScopeToCustomRoles1787140858009 +n8n | Finished migration AddProjectManageMembersScopeToCustomRoles1787140858009 +n8n | Starting migration CreateAgentChannelStatusTable1787213245846 +n8n | Finished migration CreateAgentChannelStatusTable1787213245846 +n8n | Starting migration AddResourceToOAuthRefreshTokens1787739515257 +n8n | Finished migration AddResourceToOAuthRefreshTokens1787739515257 +n8n | Starting migration CreateTypeAvailabilityPolicyTables1787841960965 +n8n | Finished migration CreateTypeAvailabilityPolicyTables1787841960965 +n8n | Starting migration CreateAgentBackgroundJobTable1788191436461 +n8n | Finished migration CreateAgentBackgroundJobTable1788191436461 +n8n | Starting migration DropInstanceAiRunSnapshotsTable1788336311704 +n8n | Finished migration DropInstanceAiRunSnapshotsTable1788336311704 +n8n | Starting migration GeneralizeScheduledJobOwner1788359043381 +n8n | Finished migration GeneralizeScheduledJobOwner1788359043381 +n8n | Starting migration CreateActivityEventTable1788425788714 +n8n | Finished migration CreateActivityEventTable1788425788714 +n8n | Starting migration CreateProjectPoolSettings1788445119183 +n8n | Finished migration CreateProjectPoolSettings1788445119183 +n8n | Starting migration CreateInstanceMonitoringReportTable1788445119184 +n8n | Finished migration CreateInstanceMonitoringReportTable1788445119184 +n8n | Starting migration CreateAgentWorkflowDependencyTable1788522448804 +n8n | Finished migration CreateAgentWorkflowDependencyTable1788522448804 +n8n | Starting migration AddWakeColumnsToAgentBackgroundJob1788527465971 +n8n | Finished migration AddWakeColumnsToAgentBackgroundJob1788527465971 +n8n | Starting migration CreatePromotionsTables1788759351166 +n8n | Finished migration CreatePromotionsTables1788759351166 +n8n | Starting migration DropGitConnectionTables1788780616817 +n8n | Finished migration DropGitConnectionTables1788780616817 +n8n | Starting migration CreateAiPreferenceTable1788882375989 +n8n | Finished migration CreateAiPreferenceTable1788882375989 +n8n | Starting migration CreateAgentMemoryEntryCandidates1789029973536 +n8n | Finished migration CreateAgentMemoryEntryCandidates1789029973536 +n8n | Starting migration AddAgentExecutionAuthorAndMessagesResourceThreadIndex1789121244489 +n8n | Finished migration AddAgentExecutionAuthorAndMessagesResourceThreadIndex1789121244489 +n8n | Starting migration AddInstanceAiThreadHistoryIndex1789131391410 +n8n | Finished migration AddInstanceAiThreadHistoryIndex1789131391410 +n8n | n8n Task Broker ready on 127.0.0.1, port 5679 +n8n | Failed to start Python task runner in internal mode. because Python 3 is missing from this system. Launching a Python runner in internal mode is intended only for debugging and is not recommended for production. Users are encouraged to deploy in external mode. See: https://docs.n8n.io/hosting/configuration/task-runners/#setting-up-external-mode +n8n | +n8n | There are deprecations related to your n8n setup. Please take the recommended actions to update your configuration: +n8n | - WEBHOOK_URL -> Use N8N_WEBHOOK_URL instead, which sets the base URL for both test and production webhooks. +n8n | - N8N_UNVERIFIED_PACKAGES_ENABLED -> The default for this variable will change to `false` in a future version. Set it to `true` explicitly to keep installing unverified community packages. +n8n | - N8N_RUNNERS_MODE -> Internal task runner mode is deprecated and will be removed in a future version. For isolation and scaling, run the task runner launcher as a separate process, set this variable to `external` and share `N8N_RUNNERS_AUTH_TOKEN` with the launcher. See https://docs.n8n.io/deploy/host-n8n/configure-n8n/set-up-task-runners +n8n | - N8N_RUNNERS_TASK_TIMEOUT -> The default for this variable will be reduced from 300 (5 minutes) to 60 (1 minute) in a future version. Set it explicitly to keep your current task timeout. +n8n | - N8N_COMPRESSION_NODE_MAX_DECOMPRESSED_SIZE_BYTES -> The default for this variable will be reduced from 2 GiB to 256 MiB in a future version. Set it explicitly to keep your current limit. +n8n | - N8N_COMPRESSION_NODE_MAX_ZIP_ENTRIES -> The default for this variable will be reduced from 5000 to 1000 in a future version. Set it explicitly to keep your current limit. +n8n | +n8n | [license SDK] Skipping renewal on init: license cert is not initialized +n8n | [runner:js] (node:24) ExperimentalWarning: localStorage is not available because --localstorage-file was not provided. +n8n | (Use `node --trace-warnings ...` to show where the warning was created) +n8n | Instance registered +n8n | Discovered 4 cluster checks +n8n | Registered runner "JS Task Runner" (mOSXw0GusGbIiBQSsHrY2) +n8n | Recorded version change: 2.31.3 -> 2.40.5 +n8n | Sandbox: enabled=false provider=n8n-sandbox (from env) +n8n | Version: 2.40.5 +n8n | Building workflow dependency index... +n8n | +n8n | Editor is now accessible via: +n8n | https://auto.enkisfelhom.hu +n8n | Finished building workflow dependency index. Processed 0 draft workflows, 0 published workflows. +n8n | 2026-09-21T19:01:51.554Z [Rudder] error: Response error code: ECONNREFUSED +n8n | 2026-09-21T19:01:51.773Z [Rudder] error: Response error code: ECONNREFUSED +n8n | 2026-09-21T19:01:52.311Z [Rudder] error: Response error code: ECONNREFUSED +n8n | 2026-09-21T19:01:53.237Z [Rudder] error: Error: ECONNREFUSED +n8n | connect ECONNREFUSED 0.0.0.0:443 +n8n | Error: connect ECONNREFUSED 0.0.0.0:443 +n8n | at AxiosError.from (/usr/local/lib/node_modules/n8n/node_modules/.pnpm/axios@1.18.0_patch_hash=149e256a2a7b632497650b32816716ced972ab02a5ab00fbd8a5158a51722c4_437e4fafb503be805d1d0ae72f0b0deb/node_modules/axios/dist/node/axios.cjs:1355:24) +n8n | at RedirectableRequest.handleRequestError (/usr/local/lib/node_modules/n8n/node_modules/.pnpm/axios@1.18.0_patch_hash=149e256a2a7b632497650b32816716ced972ab02a5ab00fbd8a5158a51722c4_437e4fafb503be805d1d0ae72f0b0deb/node_modules/axios/dist/node/axios.cjs:4026:25) +n8n | at RedirectableRequest.emit (node:events:526:24) +n8n | at ClientRequest.eventHandlers. (/usr/local/lib/node_modules/n8n/node_modules/.pnpm/follow-redirects@1.16.0_debug@4.4.3_supports-color@8.1.1_/node_modules/follow-redirects/index.js:56:24) +n8n | at ClientRequest.emit (node:events:514:20) +n8n | at emitErrorEvent (node:_http_client:114:11) +n8n | at TLSSocket.socketErrorListener (node:_http_client:768:5) +n8n | at TLSSocket.emit (node:events:526:24) +n8n | at emitErrorNT (node:internal/streams/destroy:170:8) +n8n | at emitErrorCloseNT (node:internal/streams/destroy:129:3) +n8n | at Axios.request (/usr/local/lib/node_modules/n8n/node_modules/.pnpm/axios@1.18.0_patch_hash=149e256a2a7b632497650b32816716ced972ab02a5ab00fbd8a5158a51722c4_437e4fafb503be805d1d0ae72f0b0deb/node_modules/axios/dist/node/axios.cjs:5427:41) +n8n | at processTicksAndRejections (node:internal/process/task_queues:104:5) +n8n | at Axios.request (/usr/local/lib/node_modules/n8n/node_modules/.pnpm/axios@1.18.0_patch_hash=149e256a2a7b632497650b32816716ced972ab02a5ab00fbd8a5158a51722c4_437e4fafb503be805d1d0ae72f0b0deb/node_modules/axios/dist/node/axios.cjs:5423:14) +n8n | at Axios.request (/usr/local/lib/node_modules/n8n/node_modules/.pnpm/axios@1.18.0_patch_hash=149e256a2a7b632497650b32816716ced972ab02a5ab00fbd8a5158a51722c4_437e4fafb503be805d1d0ae72f0b0deb/node_modules/axios/dist/node/axios.cjs:5423:14) +n8n | at Axios.request (/usr/local/lib/node_modules/n8n/node_modules/.pnpm/axios@1.18.0_patch_hash=149e256a2a7b632497650b32816716ced972ab02a5ab00fbd8a5158a51722c4_437e4fafb503be805d1d0ae72f0b0deb/node_modules/axios/dist/node/axios.cjs:5423:14) +n8n | +n8n | connect ECONNREFUSED 0.0.0.0:443 +n8n | Error: connect ECONNREFUSED 0.0.0.0:443 +n8n | at TCPConnectWrap.afterConnect [as oncomplete] (node:net:2017:16) +n8n | +n8n | ValidationError: The 'X-Forwarded-For' header is set but the Express 'trust proxy' setting is false (default). This could indicate a misconfiguration which would prevent express-rate-limit from accurately identifying users. See https://express-rate-limit.github.io/ERR_ERL_UNEXPECTED_X_FORWARDED_FOR/ for more information. +n8n | at Object.xForwardedForHeader (/usr/local/lib/node_modules/n8n/node_modules/.pnpm/express-rate-limit@8.2.2_express@5.1.0_supports-color@8.1.1_/node_modules/express-rate-limit/dist/index.cjs:371:13) +n8n | at Object.wrappedValidations. [as xForwardedForHeader] (/usr/local/lib/node_modules/n8n/node_modules/.pnpm/express-rate-limit@8.2.2_express@5.1.0_supports-color@8.1.1_/node_modules/express-rate-limit/dist/index.cjs:685:22) +n8n | at Object.keyGenerator (/usr/local/lib/node_modules/n8n/node_modules/.pnpm/express-rate-limit@8.2.2_express@5.1.0_supports-color@8.1.1_/node_modules/express-rate-limit/dist/index.cjs:788:20) +n8n | at /usr/local/lib/node_modules/n8n/node_modules/.pnpm/express-rate-limit@8.2.2_express@5.1.0_supports-color@8.1.1_/node_modules/express-rate-limit/dist/index.cjs:849:32 +n8n | at /usr/local/lib/node_modules/n8n/node_modules/.pnpm/express-rate-limit@8.2.2_express@5.1.0_supports-color@8.1.1_/node_modules/express-rate-limit/dist/index.cjs:830:5 { +n8n | code: 'ERR_ERL_UNEXPECTED_X_FORWARDED_FOR', +n8n | help: 'https://express-rate-limit.github.io/ERR_ERL_UNEXPECTED_X_FORWARDED_FOR/' +n8n | } diff --git a/documentation/audits/update-night-2026-09-21/apps/n8n/badges.json b/documentation/audits/update-night-2026-09-21/apps/n8n/badges.json new file mode 100644 index 00000000..1914cfaa --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/n8n/badges.json @@ -0,0 +1,31 @@ +{ + "before": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "after": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "drill_commit": "7d42360161d9" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/n8n/log.txt b/documentation/audits/update-night-2026-09-21/apps/n8n/log.txt new file mode 100644 index 00000000..9f91cecd --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/n8n/log.txt @@ -0,0 +1,25 @@ +20:55:28 ==== n8n: n8nio/n8n:2.31.3 -> n8nio/n8n:2.40.5 (sub=auto, class=db-postgres) +20:55:29 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +20:58:34 [1] deployed, controller state=running, pinned={'n8n': 'n8nio/n8n:2.31.3'} +20:58:37 [2] seeding through the app's own front door +20:58:37 n8n: /rest/owner/setup http=200 +20:58:37 [3] control C1 — reading the seed back BEFORE the update +20:58:37 n8n: login as the seeded owner http=200 ok=True +20:58:37 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +20:59:53 [4] backup idle; last=None +20:59:54 [5] drill commit 7d42360161d9: n8n n8nio/n8n:2.31.3 -> n8nio/n8n:2.40.5 (push rc=0) +20:59:58 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +20:59:58 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +20:59:58 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +20:59:58 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +20:59:59 + 1.0s phase=pulling label=Új verzió letöltése… err=None hold=None +21:01:37 + 98.4s phase=starting label=Indítás az új verzióval… err=None hold=None +21:01:41 + 102.5s phase=verifying label=Működés ellenőrzése… err=None hold=None +21:01:56 + 117.9s phase=done label=Frissítve err=None hold=None +21:01:56 [7] reading the seed back AFTER the update +21:01:57 n8n: login as the seeded owner http=200 ok=True +21:01:59 [8] pinned = {'n8n': 'n8nio/n8n:2.40.5'} +21:01:59 [8] installed = {'n8n': 'n8nio/n8n:2.40.5'} +21:01:59 [8] compose = ['image: n8nio/n8n:2.40.5'] +21:01:59 [8] inspect = ['n8n n8nio/n8n:2.40.5 running=true restarts=0'] +21:01:59 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/n8n/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/n8n/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/n8n/observables-before.json new file mode 100644 index 00000000..22856798 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/n8n/observables-before.json @@ -0,0 +1,15 @@ +{ + "pinned_images": { + "n8n": "n8nio/n8n:2.31.3" + }, + "installed_images": { + "n8n": "n8nio/n8n:2.31.3" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: n8nio/n8n:2.31.3" + ], + "docker_inspect": [ + "n8n n8nio/n8n:2.31.3 running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/n8n/observables.json b/documentation/audits/update-night-2026-09-21/apps/n8n/observables.json new file mode 100644 index 00000000..ab73ce70 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/n8n/observables.json @@ -0,0 +1,34 @@ +{ + "before": { + "pinned_images": { + "n8n": "n8nio/n8n:2.31.3" + }, + "installed_images": { + "n8n": "n8nio/n8n:2.31.3" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: n8nio/n8n:2.31.3" + ], + "docker_inspect": [ + "n8n n8nio/n8n:2.31.3 running=true restarts=0" + ] + }, + "after": { + "pinned_images": { + "n8n": "n8nio/n8n:2.40.5" + }, + "installed_images": { + "n8n": "n8nio/n8n:2.40.5" + }, + "catalog_images": { + "n8n": "n8nio/n8n:2.40.5" + }, + "live_compose_image_lines": [ + "image: n8nio/n8n:2.40.5" + ], + "docker_inspect": [ + "n8n n8nio/n8n:2.40.5 running=true restarts=0" + ] + } +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/n8n/phases.json b/documentation/audits/update-night-2026-09-21/apps/n8n/phases.json new file mode 100644 index 00000000..3768ca0f --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/n8n/phases.json @@ -0,0 +1,51 @@ +{ + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.0, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 98.4, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 102.5, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 117.9, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 117.9, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/n8n/verdict.json b/documentation/audits/update-night-2026-09-21/apps/n8n/verdict.json new file mode 100644 index 00000000..cacb5830 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/n8n/verdict.json @@ -0,0 +1,43 @@ +{ + "harness_version": 1, + "app": "n8n", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "db-postgres", + "from": { + "n8n": "n8nio/n8n:2.31.3" + }, + "to": { + "n8n": "n8nio/n8n:2.40.5" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "n8n | Starting migration CreateInstanceAiEventsTable1784000000046", + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 117.9, + "measured_at": "2026-09-21T18:55:28.852396+00:00", + "evidence": "apps/n8n/", + "notes": [], + "observables_after": { + "pinned_images": { + "n8n": "n8nio/n8n:2.40.5" + }, + "installed_images": { + "n8n": "n8nio/n8n:2.40.5" + }, + "catalog_images": { + "n8n": "n8nio/n8n:2.40.5" + }, + "live_compose_image_lines": [ + "image: n8nio/n8n:2.40.5" + ], + "docker_inspect": [ + "n8n n8nio/n8n:2.40.5 running=true restarts=0" + ] + }, + "final_phase": "done", + "hold_reason": null, + "update_error": null +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/navidrome/app-logs-after.txt b/documentation/audits/update-night-2026-09-21/apps/navidrome/app-logs-after.txt new file mode 100644 index 00000000..3715147c --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/navidrome/app-logs-after.txt @@ -0,0 +1,61 @@ +navidrome | _ _ _ _ +navidrome | | \ | | (_) | | +navidrome | | \| | __ ___ ___ __| |_ __ ___ _ __ ___ ___ +navidrome | | . ` |/ _` \ \ / / |/ _` | '__/ _ \| '_ ` _ \ / _ \ +navidrome | | |\ | (_| |\ V /| | (_| | | | (_) | | | | | | __/ +navidrome | \_| \_/\__,_| \_/ |_|\__,_|_| \___/|_| |_| |_|\___| +navidrome | Version: 0.64.0 (1072e9f7) +navidrome | +navidrome | time="2026-09-21T18:44:22Z" level=info msg="No configuration file found. Loaded configuration only from environment variables" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Upgrading DB Schema to latest version" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260712211040_add_primary_key_and_update_index_for_scrobbles.sql (13.3ms)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260714120000_add_playlist_average_rating.sql (2.01ms)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260714123822_add_media_file_title_sort_covering_index.sql (292.94µs)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260719005427_add_album_replaygain.go (4.49ms)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260720015443_uniform_canonical_ids.go (17.66ms)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260722023032_add_artwork_tables.sql (1.57ms)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260810143445_reorder_album_artists_unique_constraint.sql (9.55ms)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260811023338_add_item_genre_tag_indexes.sql (1.33ms)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260815015320_add_user_scrobble_filter.sql (2.71ms)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260816180040_renormalize_album_created_at.sql (615.58µs)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260818002312_add_playlist_imported_hash.sql (2.86ms)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260819204637_add_artwork_trace_columns.sql (7.52ms)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260822062750_add_user_token_epoch.sql (2.84ms)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="OK 20260901225726_normalize_artwork_last_failure.sql (142.21µs)" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="goose: successfully migrated database to version: 20260901225726" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Starting signaler" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Periodic backup is DISABLED" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Starting scheduler" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Scheduling DB analysis check" schedule="@every 30m" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Starting Insight Collector" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Starting artwork worker" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Creating Image cache" maxSize="100 MB" path=/data/cache/images +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Periodic scan is DISABLED" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Finished initializing cache" cache=Image elapsedTime="303.854µs" maxSize=100MB +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Creating Transcoding cache" maxSize="100 MB" path=/data/cache/transcoding +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Finished initializing cache" cache=Transcoding elapsedTime="474.055µs" maxSize=100MB +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Started watcher for library" libraryID=1 name="Music Library" path=/music +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Watcher started for library" absoluteLibPath=/music libraryID=1 name="Music Library" path=/music +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Starting plugin manager" +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Setting Session Timeout" value=48h +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Starting plugin manager" folder=/data/plugins +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Creating new JWT secret" key=JWTSecret +navidrome | time="2026-09-21T18:44:22Z" level=info msg="Login rate limit set" requestLimit=5 windowLength=20s +navidrome | time="2026-09-21T18:44:23Z" level=info msg="Found ffmpeg" path=/usr/bin/ffmpeg +navidrome | time="2026-09-21T18:44:23Z" level=info msg="Mounting Native API routes" path=/api +navidrome | time="2026-09-21T18:44:23Z" level=info msg="Mounting Subsonic API routes" path=/rest +navidrome | time="2026-09-21T18:44:23Z" level=info msg="Mounting Public Endpoints routes" path=/share +navidrome | time="2026-09-21T18:44:23Z" level=info msg="Mounting LastFM Auth routes" path=/api/lastfm +navidrome | time="2026-09-21T18:44:23Z" level=info msg="Mounting ListenBrainz Auth routes" path=/api/listenbrainz +navidrome | time="2026-09-21T18:44:23Z" level=info msg="Mounting Background images routes" path=/backgrounds +navidrome | time="2026-09-21T18:44:23Z" level=info msg="Mounting WebUI routes" path=/app +navidrome | time="2026-09-21T18:44:23Z" level=info msg="Creating backgrounds cache" maxSize="100 MB" path=/data/cache/backgrounds +navidrome | time="2026-09-21T18:44:23Z" level=info msg="Finished initializing cache" cache=backgrounds elapsedTime="293.033µs" maxSize=100MB +navidrome | time="2026-09-21T18:44:23Z" level=info msg="----> Navidrome server is ready!" address="0.0.0.0:4533" startupTime=127.5ms tlsEnabled=false +navidrome | time="2026-09-21T18:44:25Z" level=info msg="Executing initial scan" +navidrome | time="2026-09-21T18:44:25Z" level=info msg="No configuration file found. Loaded configuration only from environment variables" +navidrome | time="2026-09-21T18:44:25Z" level=info msg="Scanner: Starting scan" fullScan=false numLibraries=1 +navidrome | time="2026-09-21T18:44:25Z" level=info msg="Scanner: Finished scanning all libraries" duration=6.4ms +navidrome | time="2026-09-21T18:44:25Z" level=info msg="Scan completed" +navidrome | time="2026-09-21T18:44:28Z" level=warning msg="Unsuccessful login" request="map[Accept:[*/*] Accept-Encoding:[gzip] Content-Length:[61] Content-Type:[application/json] User-Agent:[curl/8.14.1] X-Forwarded-For:[192.168.0.180] X-Forwarded-Host:[music.enkisfelhom.hu] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Server:[2515331bc19e] X-Real-Ip:[192.168.0.180]]" requestId=2430109f9ee0/OGfROO1Xmj-000004 username=drillbf8881 +navidrome | time="2026-09-21T18:44:28Z" level=warning msg="HTTP: POST http://music.enkisfelhom.hu/auth/login" elapsedTime=1.2ms httpStatus=401 remoteAddr="172.18.0.8:39730" requestId=2430109f9ee0/OGfROO1Xmj-000004 responseSize=40 diff --git a/documentation/audits/update-night-2026-09-21/apps/navidrome/badges.json b/documentation/audits/update-night-2026-09-21/apps/navidrome/badges.json new file mode 100644 index 00000000..4c003ecf --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/navidrome/badges.json @@ -0,0 +1,31 @@ +{ + "before": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "after": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "drill_commit": "390395a96ee1" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/navidrome/log.txt b/documentation/audits/update-night-2026-09-21/apps/navidrome/log.txt new file mode 100644 index 00000000..d3eb11f7 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/navidrome/log.txt @@ -0,0 +1,27 @@ +20:42:51 ==== navidrome: deluan/navidrome:0.63.2 -> deluan/navidrome:0.64.0 (sub=music, class=file-leg) +20:42:53 [1] made the drive paths this app requires: ['/mnt/felhom-drives/scratch_hdd/userdata/navidrome'] +20:42:53 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['HDD_PATH'] +20:42:53 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +20:43:13 [1] deployed, controller state=running, pinned={'navidrome': 'deluan/navidrome:0.63.2'} +20:43:15 [2] seeding through the app's own front door +20:43:15 navidrome: createAdmin http=200 +20:43:15 [3] control C1 — reading the seed back BEFORE the update +20:43:16 navidrome: login as the seeded user http=200 ok=True +20:43:16 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +20:44:11 [4] backup idle; last=None +20:44:12 [5] drill commit 390395a96ee1: navidrome deluan/navidrome:0.63.2 -> deluan/navidrome:0.64.0 (push rc=0) +20:44:16 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +20:44:16 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +20:44:17 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +20:44:17 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +20:44:18 + 1.1s phase=pulling label=Új verzió letöltése… err=None hold=None +20:44:22 + 5.2s phase=starting label=Indítás az új verzióval… err=None hold=None +20:44:23 + 6.2s phase=verifying label=Működés ellenőrzése… err=None hold=None +20:44:28 + 11.3s phase=done label=Frissítve err=None hold=None +20:44:28 [7] reading the seed back AFTER the update +20:44:28 navidrome: login as the seeded user http=200 ok=True +20:44:30 [8] pinned = {'navidrome': 'deluan/navidrome:0.64.0'} +20:44:30 [8] installed = {'navidrome': 'deluan/navidrome:0.64.0'} +20:44:30 [8] compose = ['image: deluan/navidrome:0.64.0'] +20:44:30 [8] inspect = ['navidrome deluan/navidrome:0.64.0 running=true restarts=0'] +20:44:30 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/navidrome/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/navidrome/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/navidrome/observables-before.json new file mode 100644 index 00000000..f513c7a2 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/navidrome/observables-before.json @@ -0,0 +1,15 @@ +{ + "pinned_images": { + "navidrome": "deluan/navidrome:0.63.2" + }, + "installed_images": { + "navidrome": "deluan/navidrome:0.63.2" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: deluan/navidrome:0.63.2" + ], + "docker_inspect": [ + "navidrome deluan/navidrome:0.63.2 running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/navidrome/observables.json b/documentation/audits/update-night-2026-09-21/apps/navidrome/observables.json new file mode 100644 index 00000000..aa75ce1e --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/navidrome/observables.json @@ -0,0 +1,34 @@ +{ + "before": { + "pinned_images": { + "navidrome": "deluan/navidrome:0.63.2" + }, + "installed_images": { + "navidrome": "deluan/navidrome:0.63.2" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: deluan/navidrome:0.63.2" + ], + "docker_inspect": [ + "navidrome deluan/navidrome:0.63.2 running=true restarts=0" + ] + }, + "after": { + "pinned_images": { + "navidrome": "deluan/navidrome:0.64.0" + }, + "installed_images": { + "navidrome": "deluan/navidrome:0.64.0" + }, + "catalog_images": { + "navidrome": "deluan/navidrome:0.64.0" + }, + "live_compose_image_lines": [ + "image: deluan/navidrome:0.64.0" + ], + "docker_inspect": [ + "navidrome deluan/navidrome:0.64.0 running=true restarts=0" + ] + } +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/navidrome/phases.json b/documentation/audits/update-night-2026-09-21/apps/navidrome/phases.json new file mode 100644 index 00000000..16a5017c --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/navidrome/phases.json @@ -0,0 +1,51 @@ +{ + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.1, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 5.2, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 6.2, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 11.3, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 11.3, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/navidrome/verdict.json b/documentation/audits/update-night-2026-09-21/apps/navidrome/verdict.json new file mode 100644 index 00000000..5b5cb23d --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/navidrome/verdict.json @@ -0,0 +1,43 @@ +{ + "harness_version": 1, + "app": "navidrome", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "file-leg", + "from": { + "navidrome": "deluan/navidrome:0.63.2" + }, + "to": { + "navidrome": "deluan/navidrome:0.64.0" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": "navidrome | time=\"2026-09-21T18:44:22Z\" level=info msg=\"goose: successfully migrated database to version: 20260901225726\"", + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 11.3, + "measured_at": "2026-09-21T18:42:51.148860+00:00", + "evidence": "apps/navidrome/", + "notes": [], + "observables_after": { + "pinned_images": { + "navidrome": "deluan/navidrome:0.64.0" + }, + "installed_images": { + "navidrome": "deluan/navidrome:0.64.0" + }, + "catalog_images": { + "navidrome": "deluan/navidrome:0.64.0" + }, + "live_compose_image_lines": [ + "image: deluan/navidrome:0.64.0" + ], + "docker_inspect": [ + "navidrome deluan/navidrome:0.64.0 running=true restarts=0" + ] + }, + "final_phase": "done", + "hold_reason": null, + "update_error": null +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/opengist/log.txt b/documentation/audits/update-night-2026-09-21/apps/opengist/log.txt new file mode 100644 index 00000000..2a358dca --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/opengist/log.txt @@ -0,0 +1,18 @@ +21:05:28 ==== opengist: ghcr.io/thomiceli/opengist:1.13 -> ghcr.io/thomiceli/opengist:1.15 (sub=gist, class=other) +21:05:29 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'} +21:05:44 [1] deployed, controller state=running, pinned={'opengist': 'ghcr.io/thomiceli/opengist:1.13'} +21:05:46 [2] seeding through the app's own front door +21:05:46 opengist: /register http=500 +21:05:46 opengist: refused + + + + + + + + + + ", " ", html, flags=re.S) + t = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", t)) + out[lang] = [s.strip() for s in re.split(r"(?<=[.!?]) ", t) + if any(k in s.lower() for k in + ("friss", "update", "vissza", "restore", "ment", "backup", + "masolat", "másolat", "copy", "meghajt", "hib", "error"))][:12] + return out + + +def main(): + app, sub = sys.argv[1], sys.argv[2] + d = os.path.join(HERE, "apps", app) + os.makedirs(d, exist_ok=True) + w.login() + w.say(f"==== FAILWALK {app}: the household's way out of a held update") + + st = w.stack(app) + held = {"state": st.get("state"), "updating": st.get("updating"), + "update_phase": st.get("update_phase"), + "update_phase_label": st.get("update_phase_label"), + "update_error": st.get("update_error"), "hold_reason": st.get("hold_reason"), + "observables": w.observables(app)} + w.say(f" [1] held state: phase={held['update_phase']} hold={held['hold_reason']!r}") + w.say(f" update_error = {held['update_error']!r}") + w.say(f" pinned = {held['observables']['pinned_images']}") + w.say(f" installed = {held['observables']['installed_images']}") + w.say(f" compose = {held['observables']['live_compose_image_lines']}") + w.say(f" inspect = {held['observables']['docker_inspect']}") + + sent = sentences(app) + w.say(f" [2] hold sentence HU: {sent['hu'][:4]}") + w.say(f" [2] hold sentence EN: {sent['en'][:4]}") + + logs = w.app_logs(app, 400) + open(f"{d}/held-app-logs.txt", "w").write(logs) + mig = None + for line in logs.splitlines(): + if re.search(r"Applying .*\.\.\. OK|migrat", line, re.I) and len(line) < 300: + mig = line.strip() + w.say(f" [3] the last migration line the app printed, verbatim: {mig!r}") + + # THE WAY OUT — the button the sentence names + way = w.restore(app) + w.say(f" [4] restore: {way.get('http')} in {way.get('seconds')}s; " + f"state={way.get('state_after')} hold={way.get('hold_after')!r}") + w.say(f" pinned after = {(way.get('observables_after') or {}).get('pinned_images')}") + w.say(f" inspect after = {(way.get('observables_after') or {}).get('docker_inspect')}") + + # did the data come back? asked of the APP + fx = FIXTURES.get(app) + after = {"route": None, "ok": None, + "note": "the pre-update seed token did not survive the failed run's process, so this " + "re-seeds AFTER the restore: it proves the app WORKS again on the old version, " + "not that the specific pre-update row survived. Stated rather than glossed."} + if fx: + tok = fx.seed(w, sub, w.say) + if tok is not None: + after["route"] = "re-seeded after the restore" + after["ok"] = fx.verify(w, sub, tok, w.say) + w.say(f" [5] the app answers its own front door again and holds data: {after['ok']}") + + sent_after = sentences(app) + w.say(f" [6] sentence after the restore HU: {sent_after['hu'][:3]}") + + rec = {"leg": "failwalk", "app": app, "held": held, "hold_sentences": sent, + "last_migration_line": mig, "way_out": way, "data_after_restore": after, + "sentences_after_restore": sent_after} + json.dump(rec, open(f"{d}/failwalk.json", "w"), indent=2, ensure_ascii=False) + open(f"{d}/failwalk-log.txt", "w").write("\n".join(w.LOG) + "\n") + w.say(f" written -> {d}/failwalk.json") + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/update-night-2026-09-21/fixtures.py b/documentation/audits/update-night-2026-09-21/fixtures.py new file mode 100644 index 00000000..281ed54a --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/fixtures.py @@ -0,0 +1,998 @@ +#!/usr/bin/env python3 +"""Box-side seed/verify fixtures for walk.py, guest 9202. + +THE ONE RULE (R-156), carried verbatim from `app-catalog-felhom.eu/scripts/upgrade_fixtures.py`: +*nothing is ever seeded into a volume by hand.* Every seed here goes in through the app's OWN +interface — its HTTP API through the household's real front door (traefik, `Host: .`), +or its own CLI running inside its own container. A raw SQL INSERT or a planted file is never used. + +If an app has no non-browser route, its fixture returns None and the edge is recorded +`inconclusive — no non-browser seed route`, WITH WHAT WAS TRIED. That is a result, not a gap. + +Each fixture: + seed(w, sub, say) -> an opaque token, or None + verify(w, sub, tok, say) -> True / False +verify() must ask the APP, never the filesystem: a migration is supposed to rewrite files. +Where a fixture can prove itself (a negative control that must read as absent) it does so on EVERY +call, so a readback that has broken into always saying "found" fails instead of passing everything. +""" +import base64, json, re, secrets, time + + +def _gx(w, container, *cmd, timeout=240): + """Run a command inside the app's OWN container on 9202 (its own CLI, not our SQL).""" + import shlex + line = " ".join(shlex.quote(c) for c in cmd) + return w.guest(f"docker exec {container} {line} 2>&1", timeout=timeout) + + +# ============================================================================================= +class PrivateBin: + """PrivateBin's own JSON API. A paste is a POST and reading it back is a GET — an + application-level round trip. File-backed, no database: this single seed IS the file half.""" + sub = "paste" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/", want=("200",)): + return None + marker = "upg-" + secrets.token_hex(8) + ct = base64.b64encode(marker.encode()).decode() + body = json.dumps({ + "v": 2, + "adata": [[base64.b64encode(secrets.token_bytes(16)).decode(), + base64.b64encode(secrets.token_bytes(8)).decode(), + 100000, 256, 128, "aes", "gcm", "none"], "plaintext", 0, 0], + "ct": ct, "meta": {"expire": "never"}}) + rc, code, out = w.app_curl(sub, "/", "-H", "X-Requested-With: JSONHttpRequest", + "-H", "Content-Type: application/json", + data=body, method="POST") + try: + j = json.loads(out) + except Exception: + say(f" privatebin: POST returned non-JSON (http {code}): {out[:200]}") + return None + if j.get("status") != 0 or not j.get("id"): + say(f" privatebin: POST refused: {out[:250]}") + return None + say(f" privatebin: seeded paste id={j['id']}") + return {"id": j["id"], "marker": ct} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/", want=("200",), tries=36): + return False + # negative control, every call: a paste id that cannot exist must NOT read back + rc, code, out = w.app_curl(sub, "/?pasteid=" + secrets.token_hex(8), + "-H", "X-Requested-With: JSONHttpRequest") + if t["marker"] in out: + say(" privatebin: READBACK UNUSABLE — a paste id that cannot exist returned the marker") + return False + rc, code, out = w.app_curl(sub, "/?pasteid=" + t["id"], + "-H", "X-Requested-With: JSONHttpRequest") + got = code == "200" and t["marker"] in out + say(f" privatebin: readback http={code} marker_present={got}") + return got + + +# ============================================================================================= +class Docmost: + """Docmost's own REST API: create the first workspace+user, then prove the account survives by + asking the app to AUTHENTICATE it. Login is version-stable across the API churn.""" + sub = "docs" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/", want=("200", "302", "404")): + return None + email = f"drill-{secrets.token_hex(4)}@gate.invalid" + pw = "Drill-" + secrets.token_hex(10) + body = json.dumps({"workspaceName": "drill", "name": "drill", "email": email, "password": pw}) + rc, code, out = w.app_curl(sub, "/api/auth/setup", "-H", "Content-Type: application/json", + data=body, method="POST") + say(f" docmost: /api/auth/setup http={code} rc={rc}") + if code not in ("200", "201"): + say(f" docmost: setup refused: {out[:250]}") + return None + return {"email": email, "pw": pw} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/", want=("200", "302", "404"), tries=36): + return False + # negative control: a password that was never set must NOT authenticate + bad = json.dumps({"email": t["email"], "password": "definitely-" + secrets.token_hex(8)}) + rc, code, _ = w.app_curl(sub, "/api/auth/login", "-H", "Content-Type: application/json", + data=bad, method="POST") + if code in ("200", "201"): + say(" docmost: READBACK UNUSABLE — a wrong password authenticated") + return False + body = json.dumps({"email": t["email"], "password": t["pw"]}) + rc, code, out = w.app_curl(sub, "/api/auth/login", "-H", "Content-Type: application/json", + data=body, method="POST") + ok = code in ("200", "201") + say(f" docmost: login as the seeded user http={code} ok={ok}") + if not ok: + say(f" docmost: login body {out[:200]}") + return ok + + +# ============================================================================================= +class BookStack: + """BookStack mints no API token without a browser, so BOTH halves go through `php artisan` — + BookStack's OWN CLI, inside its own container, against its own User model. + + The exit code carries no information here (`bookstack:reset-mfa` exits 1 for a user it FOUND + and for one it did not), so the discriminator is the OUTPUT: the positive sentence required and + the not-found sentence required absent. The negative control runs on every verify. + + LIMITATION (R-460): this seeds the DATABASE half only. The FILE half needs the API token the + app cannot mint headlessly — so a bookstack edge is at best HALF-proven here. + """ + sub = "wiki" + + def _artisan(self, w, *args): + for path in ("/app/www/artisan", "/var/www/html/artisan"): + out = _gx(w, "bookstack", "php", path, *args) + if "Could not open input file" not in out: + return " ".join(out.split()) + return " ".join(out.split()) + + def _lookup(self, w, email): + out = self._artisan(w, "bookstack:reset-mfa", f"--email={email}") + found = f"Email: {email}" in out + missing = "could not be found" in out + if found == missing: + return None, out + return found, out + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/login", want=("200",), tries=72): + return None + email = f"drill-{secrets.token_hex(4)}@gate.invalid" + pw = "Drill-" + secrets.token_hex(10) + out = self._artisan(w, "bookstack:create-admin", f"--email={email}", + f"--name=drill-{secrets.token_hex(3)}", f"--password={pw}") + say(f" bookstack: artisan create-admin :: {out[:140]}") + if "successfully created" not in out: + return None + return {"email": email, "pw": pw} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/login", want=("200",), tries=72): + say(" bookstack: the app never served /login") + return False + absent, _ = self._lookup(w, f"nobody-{secrets.token_hex(6)}@gate.invalid") + if absent is not False: + say(f" bookstack: READBACK UNUSABLE — an email that cannot exist did not read absent ({absent})") + return False + found, out = self._lookup(w, t["email"]) + say(f" bookstack: readback of the seeded account found={found} :: {out[:140]}") + return found is True + + +# ============================================================================================= +class Gitea: + """Gitea's own admin CLI creates the first user; its own REST API (basic auth) then creates a + repository and reads it back. Both are the app's own interfaces.""" + sub = "git" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/", want=("200", "302")): + return None + user = "drill" + secrets.token_hex(3) + pw = "Drill-" + secrets.token_hex(10) + out = _gx(w, "gitea", "su", "git", "-c", + f"gitea admin user create --username {user} --password {pw} " + f"--email {user}@gate.invalid --admin --must-change-password=false") + say(f" gitea: admin user create :: {' '.join(out.split())[:140]}") + if "has been successfully created" not in out and "successfully created" not in out: + return None + repo = "drillrepo" + secrets.token_hex(3) + rc, code, body = w.app_curl(sub, "/api/v1/user/repos", "-u", f"{user}:{pw}", + "-H", "Content-Type: application/json", + data=json.dumps({"name": repo, "private": True}), method="POST") + say(f" gitea: create repo http={code}") + if code not in ("201", "200"): + say(f" gitea: repo refused {body[:200]}") + return None + return {"user": user, "pw": pw, "repo": repo} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/", want=("200", "302"), tries=36): + return False + rc, code, _ = w.app_curl(sub, f"/api/v1/repos/{t['user']}/nope{secrets.token_hex(4)}", + "-u", f"{t['user']}:{t['pw']}") + if code == "200": + say(" gitea: READBACK UNUSABLE — a repo that cannot exist returned 200") + return False + rc, code, body = w.app_curl(sub, f"/api/v1/repos/{t['user']}/{t['repo']}", + "-u", f"{t['user']}:{t['pw']}") + ok = code == "200" and t["repo"] in body + say(f" gitea: readback of the seeded repo http={code} ok={ok}") + return ok + + +# ============================================================================================= +class Navidrome: + """Navidrome's own REST API: create the first admin through /auth/createAdmin, then prove the + account survives by logging in through the same door.""" + sub = "music" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/", want=("200", "302")): + return None + user = "drill" + secrets.token_hex(3) + pw = "Drill-" + secrets.token_hex(10) + rc, code, out = w.app_curl(sub, "/auth/createAdmin", "-H", "Content-Type: application/json", + data=json.dumps({"username": user, "password": pw}), method="POST") + say(f" navidrome: createAdmin http={code}") + if code not in ("200", "201"): + say(f" navidrome: refused {out[:200]}") + return None + return {"user": user, "pw": pw} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/", want=("200", "302"), tries=36): + return False + bad = json.dumps({"username": t["user"], "password": "wrong-" + secrets.token_hex(6)}) + rc, code, _ = w.app_curl(sub, "/auth/login", "-H", "Content-Type: application/json", + data=bad, method="POST") + if code in ("200", "201"): + say(" navidrome: READBACK UNUSABLE — a wrong password authenticated") + return False + body = json.dumps({"username": t["user"], "password": t["pw"]}) + rc, code, out = w.app_curl(sub, "/auth/login", "-H", "Content-Type: application/json", + data=body, method="POST") + ok = code in ("200", "201") + say(f" navidrome: login as the seeded user http={code} ok={ok}") + return ok + + +# ============================================================================================= +class Vaultwarden: + """Vaultwarden's own account API: register an account, then prove it survives by asking the app + to issue a token for it (its own login endpoint, the household's own route).""" + sub = "vault" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/alive", want=("200",)): + return None + email = f"drill-{secrets.token_hex(4)}@gate.invalid" + # Vaultwarden stores an already-hashed master key; the value is opaque to the server. + key = base64.b64encode(secrets.token_bytes(32)).decode() + body = json.dumps({"email": email, "name": "drill", "masterPasswordHash": key, + "key": "0." + base64.b64encode(secrets.token_bytes(48)).decode(), + "kdf": 0, "kdfIterations": 600000}) + rc, code, out = w.app_curl(sub, "/api/accounts/register", + "-H", "Content-Type: application/json", + data=body, method="POST") + say(f" vaultwarden: register http={code}") + if code not in ("200", "204"): + say(f" vaultwarden: refused {out[:250]}") + return None + return {"email": email, "key": key} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/alive", want=("200",), tries=36): + return False + def login(pwhash): + return w.app_curl(sub, "/identity/connect/token", + "-H", "Content-Type: application/x-www-form-urlencoded", + data=("grant_type=password&scope=api%20offline_access" + f"&client_id=web&deviceType=9&deviceIdentifier=drill" + f"&deviceName=drill&username={t['email']}&password={pwhash}"), + method="POST") + rc, code, _ = login(base64.b64encode(secrets.token_bytes(32)).decode()) + if code == "200": + say(" vaultwarden: READBACK UNUSABLE — a wrong master key authenticated") + return False + rc, code, out = login(t["key"].replace("+", "%2B").replace("=", "%3D").replace("/", "%2F")) + ok = code == "200" and "access_token" in out + say(f" vaultwarden: token for the seeded account http={code} ok={ok}") + if not ok: + say(f" vaultwarden: body {out[:200]}") + return ok + + +# ============================================================================================= +class Django: + """A Django app's OWN management CLI, inside its own container, against its own User model. + + Same category as BookStack's `php artisan`: the app's own code and its own ORM, never a raw SQL + INSERT and never a planted file (R-156). `createsuperuser --noinput` is Django's own documented + non-interactive route, and the readback asks the SAME ORM whether the account exists. + + THE FIXTURE PROVES ITSELF ON EVERY CALL: each verify() also asks for a username that cannot + exist and requires the answer False. A readback that has broken into always saying True + therefore fails instead of passing everything. + + LIMITATION, recorded rather than papered over: this seeds the DATABASE half only. An app whose + data is also FILES (adventurelog's images) has a file half this fixture does not touch. + """ + + def __init__(self, container, sub, ready_path="/", ready=("200", "302", "301", "404"), + python="python", workdir=None): + # `python` and `workdir` are per-app because the image decides them: adventurelog's + # interpreter is on PATH, tandoor ships a VENV and the bare `python` cannot import Django + # at all ("Couldn't import Django. Are you sure it's installed…"). Measured, not guessed. + self.container = container + self.sub = sub + self.ready_path = ready_path + self.ready = ready + self.python = python + self.workdir = workdir + + def _wd(self): + return f"-w {self.workdir} " if self.workdir else "" + + def _manage(self, w, code): + # -c is passed to `manage.py shell`; the app's own shell, its own ORM. + return w.guest( + f"docker exec {self._wd()}{self.container} {self.python} manage.py shell " + f"-c {json.dumps(code)} 2>&1", timeout=300) + + def _exists(self, w, username): + # ONE LINE, semicolon-separated. A `\n` inside a double-quoted shell argument reaches + # python as a literal backslash-n and is a SyntaxError — which is exactly how the first + # adventurelog run read as `inconclusive`. The fixture refused to guess, which is right, + # but the instrument was the thing that was broken. + out = self._manage(w, ( + "from django.contrib.auth import get_user_model; " + f"print('DRILL_ANSWER=' + str(get_user_model().objects.filter(username={username!r}).exists()))" + )) + m = re.search(r"DRILL_ANSWER=(True|False)", out) + return (m.group(1) == "True") if m else None, " ".join(out.split())[-300:] + + def seed(self, w, sub, say): + if not w.wait_app(sub, self.ready_path, want=self.ready, tries=90): + return None + user = "drill" + secrets.token_hex(3) + pw = "Drill-" + secrets.token_hex(10) + out = w.guest( + f"docker exec -e DJANGO_SUPERUSER_PASSWORD={pw} {self._wd()}{self.container} " + f"{self.python} manage.py createsuperuser --noinput " + f"--username {user} --email {user}@gate.invalid 2>&1", timeout=300) + say(f" {self.container}: createsuperuser :: {' '.join(out.split())[:160]}") + got, detail = self._exists(w, user) + if got is not True: + say(f" {self.container}: the account did not appear in the app's own ORM :: {detail[:200]}") + return None + say(f" {self.container}: seeded superuser {user}") + return {"user": user, "pw": pw} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, self.ready_path, want=self.ready, tries=90): + say(f" {self.container}: the app never served {self.ready_path}") + return False + absent, detail = self._exists(w, "nobody" + secrets.token_hex(6)) + if absent is not False: + say(f" {self.container}: READBACK UNUSABLE — a username that cannot exist did not " + f"read as absent ({absent}) :: {detail[:200]}") + return False + found, detail = self._exists(w, t["user"]) + say(f" {self.container}: readback of the seeded account found={found}") + if found is not True: + say(f" {self.container}: :: {detail[:250]}") + return found is True + + +# ============================================================================================= +class Nextcloud: + """Nextcloud's OWN admin CLI, `occ`, inside its own container: its own code, its own user + backend. Not a SQL INSERT and not a planted file (R-156). + + `occ user:info` is the readback, and it PROVES ITSELF on every call: a uid that cannot exist + must answer "user not found". A readback that has broken into always succeeding therefore + fails instead of passing everything. + + This is the app chosen for the MariaDB engine-major edge (`09` §3 decision 5, R-469 lifted): + the app image does NOT move, only the `mariadb:` sidecar, so the edge carries exactly one + migration and a failure is readable. + """ + sub = "cloud" + + def _occ(self, w, *args, timeout=420): + import shlex + line = " ".join(shlex.quote(a) for a in args) + return w.guest(f"docker exec -u www-data nextcloud php occ {line} 2>&1", timeout=timeout) + + def _info(self, w, uid): + out = self._occ(w, "user:info", uid) + flat = " ".join(out.split()) + if "user not found" in flat.lower() or "could not be found" in flat.lower(): + return False, flat + if f"user_id: {uid}" in flat or f"- user_id: {uid}" in flat or f"user_id: {uid}" in out: + return True, flat + return None, flat + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/status.php", want=("200",), tries=120): + return None + uid = "drill" + secrets.token_hex(3) + pw = "Drill-" + secrets.token_hex(10) + out = w.guest( + f"docker exec -u www-data -e OC_PASS={pw} nextcloud php occ user:add " + f"--password-from-env --display-name={uid} {uid} 2>&1", timeout=420) + say(f" nextcloud: occ user:add :: {' '.join(out.split())[:160]}") + got, flat = self._info(w, uid) + if got is not True: + say(f" nextcloud: the account did not appear via occ user:info :: {flat[:220]}") + return None + say(f" nextcloud: seeded user {uid}") + return {"uid": uid, "pw": pw} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/status.php", want=("200",), tries=120): + say(" nextcloud: the app never served /status.php") + return False + absent, flat = self._info(w, "nobody" + secrets.token_hex(6)) + if absent is not False: + say(f" nextcloud: READBACK UNUSABLE — a uid that cannot exist did not read absent " + f"({absent}) :: {flat[:200]}") + return False + found, flat = self._info(w, t["uid"]) + say(f" nextcloud: readback of the seeded user found={found}") + if found is not True: + say(f" nextcloud: :: {flat[:250]}") + return found is True + + +# ============================================================================================= +class Grafana: + """Grafana's own HTTP API as the admin the DEPLOY created. The password is the one the + controller showed the household — read from the app's own `app.yaml`, not invented — and the + data (a folder) goes in and comes back through the app's own REST API.""" + sub = "grafana" + + def _auth(self, w, name="grafana"): + # app.yaml stores this ENCRYPTED (`ENC:…`), so it cannot be read back off the box — which + # is correct, and is why the harness uses the value IT generated for the deploy. + pw = (w.GENERATED.get(name) or {}).get("GF_SECURITY_ADMIN_PASSWORD") or "admin" + return f"admin:{pw}" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/api/health", want=("200",), tries=72): + return None + au = self._auth(w) + title = "drill-" + secrets.token_hex(5) + rc, code, body = w.app_curl(sub, "/api/folders", "-u", au, + "-H", "Content-Type: application/json", + data=json.dumps({"title": title}), method="POST") + say(f" grafana: create folder http={code}") + if code not in ("200", "201"): + say(f" grafana: refused {body[:220]}") + return None + try: + uid = json.loads(body)["uid"] + except Exception: + say(f" grafana: no uid in {body[:200]}") + return None + return {"uid": uid, "title": title} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/api/health", want=("200",), tries=72): + return False + au = self._auth(w) + rc, code, _ = w.app_curl(sub, "/api/folders/nope" + secrets.token_hex(5), "-u", au) + if code == "200": + say(" grafana: READBACK UNUSABLE — a folder uid that cannot exist returned 200") + return False + rc, code, body = w.app_curl(sub, f"/api/folders/{t['uid']}", "-u", au) + ok = code == "200" and t["title"] in body + say(f" grafana: readback of the seeded folder http={code} ok={ok}") + return ok + + +# ============================================================================================= +class AudiobookShelf: + """audiobookshelf's own /init endpoint creates the first root account; its own /login proves + the account survived. Both are the app's own API.""" + sub = "audiobooks" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/status", want=("200",), tries=72): + return None + user = "drill" + secrets.token_hex(3) + pw = "Drill-" + secrets.token_hex(10) + rc, code, body = w.app_curl(sub, "/init", "-H", "Content-Type: application/json", + data=json.dumps({"newRoot": {"username": user, "password": pw}}), + method="POST") + say(f" audiobookshelf: /init http={code}") + if code not in ("200", "204"): + say(f" audiobookshelf: refused {body[:220]}") + return None + return {"user": user, "pw": pw} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/status", want=("200",), tries=72): + return False + bad = json.dumps({"username": t["user"], "password": "wrong-" + secrets.token_hex(6)}) + rc, code, _ = w.app_curl(sub, "/login", "-H", "Content-Type: application/json", + data=bad, method="POST") + if code == "200": + say(" audiobookshelf: READBACK UNUSABLE — a wrong password authenticated") + return False + rc, code, body = w.app_curl(sub, "/login", "-H", "Content-Type: application/json", + data=json.dumps({"username": t["user"], "password": t["pw"]}), + method="POST") + ok = code == "200" and t["user"] in body + say(f" audiobookshelf: login as the seeded root http={code} ok={ok}") + return ok + + +# ============================================================================================= +class ActualBudget: + """Actual's own bootstrap API sets the server password; its own login proves it survived.""" + sub = "budget" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/", want=("200", "302"), tries=72): + return None + pw = "Drill-" + secrets.token_hex(10) + rc, code, body = w.app_curl(sub, "/account/bootstrap", + "-H", "Content-Type: application/json", + data=json.dumps({"password": pw}), method="POST") + say(f" actualbudget: /account/bootstrap http={code} :: {body[:140]}") + if code not in ("200", "201") or '"status":"ok"' not in body: + return None + return {"pw": pw} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/", want=("200", "302"), tries=72): + return False + def login(p): + return w.app_curl(sub, "/account/login", "-H", "Content-Type: application/json", + data=json.dumps({"loginMethod": "password", "password": p}), + method="POST") + rc, code, body = login("wrong-" + secrets.token_hex(6)) + if '"status":"ok"' in body: + say(" actualbudget: READBACK UNUSABLE — a wrong password authenticated") + return False + rc, code, body = login(t["pw"]) + ok = '"status":"ok"' in body + say(f" actualbudget: login with the seeded password http={code} ok={ok}") + if not ok: + say(f" actualbudget: body {body[:200]}") + return ok + + +# ============================================================================================= +class Mealie: + """Mealie ships a documented first-run admin. We log in as it through the app's own OAuth-style + token endpoint, create a recipe through the app's own API, and read the recipe back.""" + sub = "recipes" + + def _token(self, w, sub, pw="MyPassword"): + rc, code, body = w.app_curl( + sub, "/api/auth/token", "-H", "Content-Type: application/x-www-form-urlencoded", + data=f"username=changeme%40example.com&password={pw}", method="POST") + if code != "200": + return None, f"http={code} {body[:200]}" + try: + return json.loads(body)["access_token"], "" + except Exception: + return None, body[:200] + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/api/app/about", want=("200",), tries=90): + return None + tok, why = self._token(w, sub) + if not tok: + say(f" mealie: could not authenticate as the first-run admin :: {why}") + return None + name = "drill-" + secrets.token_hex(5) + rc, code, body = w.app_curl(sub, "/api/recipes", "-H", f"Authorization: Bearer {tok}", + "-H", "Content-Type: application/json", + data=json.dumps({"name": name}), method="POST") + say(f" mealie: create recipe http={code}") + if code not in ("200", "201"): + say(f" mealie: refused {body[:220]}") + return None + slug = body.strip().strip('"') + return {"slug": slug, "name": name} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/api/app/about", want=("200",), tries=90): + return False + tok, why = self._token(w, sub) + if not tok: + say(f" mealie: could not authenticate after the update :: {why}") + return False + rc, code, _ = w.app_curl(sub, "/api/recipes/nope" + secrets.token_hex(5), + "-H", f"Authorization: Bearer {tok}") + if code == "200": + say(" mealie: READBACK UNUSABLE — a slug that cannot exist returned 200") + return False + rc, code, body = w.app_curl(sub, f"/api/recipes/{t['slug']}", + "-H", f"Authorization: Bearer {tok}") + ok = code == "200" and t["name"] in body + say(f" mealie: readback of the seeded recipe http={code} ok={ok}") + return ok + + +# ============================================================================================= +class N8n: + """n8n's own owner-setup API creates the first account; its own login proves it survived.""" + sub = "auto" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/healthz", want=("200",), tries=90): + return None + email = f"drill-{secrets.token_hex(4)}@gate.invalid" + pw = "Drill" + secrets.token_hex(8) + "1" + rc, code, body = w.app_curl(sub, "/rest/owner/setup", "-H", "Content-Type: application/json", + data=json.dumps({"email": email, "firstName": "drill", + "lastName": "drill", "password": pw}), + method="POST") + say(f" n8n: /rest/owner/setup http={code}") + if code not in ("200", "201"): + say(f" n8n: refused {body[:220]}") + return None + return {"email": email, "pw": pw} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/healthz", want=("200",), tries=90): + return False + def login(p): + return w.app_curl(sub, "/rest/login", "-H", "Content-Type: application/json", + data=json.dumps({"emailOrLdapLoginId": t["email"], "password": p}), + method="POST") + rc, code, _ = login("wrong-" + secrets.token_hex(6)) + if code == "200": + say(" n8n: READBACK UNUSABLE — a wrong password authenticated") + return False + rc, code, body = login(t["pw"]) + ok = code == "200" and t["email"] in body + say(f" n8n: login as the seeded owner http={code} ok={ok}") + return ok + + +# ============================================================================================= +class Zipline: + """Zipline's own setup/login API. Zipline 4 creates the first user through its own endpoint.""" + sub = "img" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/api/healthcheck", want=("200",), tries=90): + if not w.wait_app(sub, "/", want=("200", "302", "307"), tries=30): + return None + user = "drill" + secrets.token_hex(3) + pw = "Drill-" + secrets.token_hex(10) + for path in ("/api/auth/register", "/api/auth/setup"): + rc, code, body = w.app_curl(sub, path, "-H", "Content-Type: application/json", + data=json.dumps({"username": user, "password": pw}), + method="POST") + say(f" zipline: {path} http={code} :: {body[:160]}") + if code in ("200", "201"): + return {"user": user, "pw": pw} + say(" zipline: neither register nor setup accepted a first user") + return None + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/", want=("200", "302", "307"), tries=60): + return False + def login(p): + return w.app_curl(sub, "/api/auth/login", "-H", "Content-Type: application/json", + data=json.dumps({"username": t["user"], "password": p}), + method="POST") + rc, code, _ = login("wrong-" + secrets.token_hex(6)) + if code == "200": + say(" zipline: READBACK UNUSABLE — a wrong password authenticated") + return False + rc, code, body = login(t["pw"]) + ok = code == "200" + say(f" zipline: login as the seeded user http={code} ok={ok}") + return ok + + +# ============================================================================================= +class Vikunja: + """Vikunja's own REST API: register a user, log in, create a project, read the project back. + Four calls, all the app's own front door.""" + sub = "tasks" + + def _token(self, w, sub, t, pw=None): + rc, code, body = w.app_curl(sub, "/api/v1/login", "-H", "Content-Type: application/json", + data=json.dumps({"username": t["user"], + "password": pw or t["pw"]}), method="POST") + if code != "200": + return None, f"http={code} {body[:160]}" + try: + return json.loads(body)["token"], "" + except Exception: + return None, body[:160] + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/api/v1/info", want=("200",), tries=72): + return None + user = "drill" + secrets.token_hex(3) + pw = "Drill-" + secrets.token_hex(10) + rc, code, body = w.app_curl(sub, "/api/v1/register", "-H", "Content-Type: application/json", + data=json.dumps({"username": user, "password": pw, + "email": f"{user}@gate.invalid"}), + method="POST") + say(f" vikunja: register http={code}") + if code not in ("200", "201"): + say(f" vikunja: refused {body[:220]}") + return None + t = {"user": user, "pw": pw} + tok, why = self._token(w, sub, t) + if not tok: + say(f" vikunja: could not log in after registering :: {why}") + return None + title = "drill-" + secrets.token_hex(5) + # Vikunja CREATES with PUT, not POST — a POST answers `405 Method Not Allowed`, which + # reads like a broken fixture and is really the wrong verb. Measured 2026-09-21. + rc, code, body = w.app_curl(sub, "/api/v1/projects", "-H", f"Authorization: Bearer {tok}", + "-H", "Content-Type: application/json", + data=json.dumps({"title": title}), method="PUT") + say(f" vikunja: create project http={code}") + if code not in ("200", "201"): + say(f" vikunja: project refused {body[:220]}") + return None + t["title"] = title + t["pid"] = json.loads(body).get("id") + return t + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/api/v1/info", want=("200",), tries=72): + return False + bad, why = self._token(w, sub, t, pw="wrong-" + secrets.token_hex(6)) + if bad: + say(" vikunja: READBACK UNUSABLE — a wrong password authenticated") + return False + tok, why = self._token(w, sub, t) + if not tok: + say(f" vikunja: the seeded account no longer authenticates :: {why}") + return False + rc, code, body = w.app_curl(sub, f"/api/v1/projects/{t['pid']}", + "-H", f"Authorization: Bearer {tok}") + ok = code == "200" and t["title"] in body + say(f" vikunja: readback of the seeded project http={code} ok={ok}") + return ok + + +# ============================================================================================= +class OpenGist: + """Opengist's own signup form, then its own API as that user.""" + sub = "gist" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/", want=("200", "302"), tries=72): + return None + user = "drill" + secrets.token_hex(3) + pw = "Drill-" + secrets.token_hex(10) + # Opengist's sign-up is a CSRF-protected FORM: a bare POST answers 500 with an HTML page, + # which reads like a broken app and is really a missing token. Fetch the form, keep its + # cookie, send its `_csrf` back. Measured 2026-09-21. + jar = f"/tmp/og-{secrets.token_hex(4)}.jar" + rc, code, form = w.app_curl(sub, "/register", "-c", jar) + m = re.search(r'name="_csrf"[^>]*value="([^"]+)"', form or "") + if not m: + say(f" opengist: no _csrf token on /register (http={code})") + return None + rc, code, body = w.app_curl(sub, "/register", "-b", jar, "-c", jar, + "-H", "Content-Type: application/x-www-form-urlencoded", + data=f"_csrf={m.group(1)}&username={user}&password={pw}", + method="POST") + say(f" opengist: /register (with its own _csrf) http={code}") + if code not in ("200", "302", "303"): + say(f" opengist: refused {body[:200]}") + return None + rc, code, body = w.app_curl(sub, "/api/gists", "-u", f"{user}:{pw}") + if code not in ("200", "401", "404"): + say(f" opengist: the API did not answer as that user: http={code}") + name = "drill-" + secrets.token_hex(4) + rc, code, body = w.app_curl(sub, "/api/gists", "-u", f"{user}:{pw}", + "-H", "Content-Type: application/json", + data=json.dumps({"title": name, "visibility": 1, + "files": [{"filename": "a.txt", + "content": name}]}), + method="POST") + say(f" opengist: create gist http={code}") + if code not in ("200", "201"): + say(f" opengist: gist refused {body[:220]}") + return None + return {"user": user, "pw": pw, "name": name} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/", want=("200", "302"), tries=72): + return False + rc, code, body = w.app_curl(sub, "/api/gists", "-u", + f"{t['user']}:wrong-{secrets.token_hex(5)}") + if code == "200": + say(" opengist: READBACK UNUSABLE — a wrong password authenticated") + return False + rc, code, body = w.app_curl(sub, "/api/gists", "-u", f"{t['user']}:{t['pw']}") + ok = code == "200" and t["name"] in body + say(f" opengist: readback of the seeded gist http={code} ok={ok}") + return ok + + +# ============================================================================================= +class Papra: + """Papra's own e-mail sign-up and sign-in endpoints.""" + sub = "papra" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/api/health", want=("200",), tries=72): + if not w.wait_app(sub, "/", want=("200", "302"), tries=30): + return None + email = f"drill-{secrets.token_hex(4)}@gate.invalid" + pw = "Drill-" + secrets.token_hex(10) + rc, code, body = w.app_curl(sub, "/api/auth/sign-up/email", + "-H", "Content-Type: application/json", + data=json.dumps({"email": email, "password": pw, + "name": "drill"}), method="POST") + say(f" papra: sign-up http={code}") + if code not in ("200", "201"): + say(f" papra: refused {body[:220]}") + return None + return {"email": email, "pw": pw} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/", want=("200", "302"), tries=72): + return False + def signin(p): + return w.app_curl(sub, "/api/auth/sign-in/email", + "-H", "Content-Type: application/json", + data=json.dumps({"email": t["email"], "password": p}), method="POST") + rc, code, _ = signin("wrong-" + secrets.token_hex(6)) + if code == "200": + say(" papra: READBACK UNUSABLE — a wrong password authenticated") + return False + rc, code, body = signin(t["pw"]) + ok = code == "200" + say(f" papra: sign-in as the seeded account http={code} ok={ok}") + return ok + + +# ============================================================================================= +class HomeAssistant: + """Home Assistant's own onboarding API creates the owner account and hands back a code the + same API exchanges for a token. Both are the app's own documented non-browser route.""" + sub = "ha" + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/", want=("200", "302"), tries=120): + return None + user = "drill" + secrets.token_hex(3) + pw = "Drill-" + secrets.token_hex(10) + rc, code, body = w.app_curl(sub, "/api/onboarding/users", + "-H", "Content-Type: application/json", + data=json.dumps({"client_id": f"https://{sub}.felhom.invalid/", + "name": "drill", "username": user, + "password": pw, "language": "en"}), + method="POST") + say(f" home-assistant: /api/onboarding/users http={code}") + if code not in ("200", "201"): + say(f" home-assistant: refused {body[:220]}") + return None + return {"user": user, "pw": pw} + + def _login(self, w, sub, user, pw): + """The app's own login flow: start it, then answer it. A 200 with a step_id of + `mfa`/`init` means the credentials were REFUSED; only `create_entry` is a pass.""" + rc, code, body = w.app_curl(sub, "/auth/login_flow", + "-H", "Content-Type: application/json", + data=json.dumps({"client_id": f"https://{sub}.felhom.invalid/", + "handler": ["homeassistant", None], + "redirect_uri": f"https://{sub}.felhom.invalid/", + "type": "authorize"}), method="POST") + if code not in ("200", "201"): + return None, f"flow start http={code} {body[:160]}" + try: + fid = json.loads(body)["flow_id"] + except Exception: + return None, body[:160] + rc, code, body = w.app_curl(sub, f"/auth/login_flow/{fid}", + "-H", "Content-Type: application/json", + data=json.dumps({"client_id": f"https://{sub}.felhom.invalid/", + "username": user, "password": pw}), + method="POST") + try: + j = json.loads(body) + except Exception: + return None, body[:160] + return (j.get("result") if j.get("type") == "create_entry" else None), body[:200] + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/", want=("200", "302"), tries=120): + return False + bad, why = self._login(w, sub, t["user"], "wrong-" + secrets.token_hex(6)) + if bad: + say(" home-assistant: READBACK UNUSABLE — a wrong password authenticated") + return False + good, why = self._login(w, sub, t["user"], t["pw"]) + ok = bool(good) + say(f" home-assistant: login as the seeded owner ok={ok}") + if not ok: + say(f" home-assistant: {why}") + return ok + + +# ============================================================================================= +class Romm: + """RomM's own user API, driven the way RomM's own front end drives it. + + Three things had to be measured rather than guessed, and each one answered a 403 or a 422 that + looked like a different fault: RomM sets a **`romm_csrftoken` cookie** on any GET and requires + it back in an **`x-csrftoken` header** (a bare POST is `403 CSRF token verification failed`, + which reads like an auth problem); the fields go in the **JSON body**, not the query string (a + query-string POST is `422 Field required` for every field it was just given); and `email` is + required alongside username, password and role. + + On a fresh install with no admin the first `POST /api/users` is accepted unauthenticated; + afterwards it is not — which is what makes the readback (`POST /api/login` as that user) a real + authentication rather than a repeat of the seed. + + LIMITATION: this is the DATABASE half. RomM's other half is the ROM library on the drive, which + this does not populate. + """ + sub = "arcade" + + def _csrf(self, w, sub): + jar = f"/tmp/romm-{secrets.token_hex(4)}.jar" + w.app_curl(sub, "/api/heartbeat", "-c", jar) + out = w.sh(["bash", "-lc", f"grep -i csrf {jar} | awk '{{print $7}}'"]).stdout or "" + return jar, out.strip() + + def seed(self, w, sub, say): + if not w.wait_app(sub, "/api/heartbeat", want=("200",), tries=120): + if not w.wait_app(sub, "/", want=("200", "302"), tries=30): + return None + jar, tok = self._csrf(w, sub) + if not tok: + say(" romm: no romm_csrftoken cookie was set on /api/heartbeat") + return None + user = "drill" + secrets.token_hex(3) + pw = "Drill-" + secrets.token_hex(10) + rc, code, body = w.app_curl( + sub, "/api/users", "-b", jar, "-H", f"x-csrftoken: {tok}", + "-H", "Content-Type: application/json", + data=json.dumps({"username": user, "email": f"{user}@gate.invalid", + "password": pw, "role": "admin"}), method="POST") + say(f" romm: POST /api/users http={code}") + if code not in ("200", "201"): + say(f" romm: refused {body[:220]}") + return None + return {"user": user, "pw": pw} + + def verify(self, w, sub, t, say): + if not w.wait_app(sub, "/api/heartbeat", want=("200",), tries=120): + return False + jar, tok = self._csrf(w, sub) + rc, code, _ = w.app_curl(sub, "/api/login", "-b", jar, "-H", f"x-csrftoken: {tok}", + "-u", f"{t['user']}:wrong-{secrets.token_hex(5)}", method="POST") + if code == "200": + say(" romm: READBACK UNUSABLE — a wrong password authenticated") + return False + rc, code, body = w.app_curl(sub, "/api/login", "-b", jar, "-H", f"x-csrftoken: {tok}", + "-u", f"{t['user']}:{t['pw']}", method="POST") + ok = code == "200" + say(f" romm: login as the seeded user http={code} ok={ok}") + if not ok: + say(f" romm: body {body[:200]}") + return ok + + +FIXTURES = { + "home-assistant": HomeAssistant(), + "romm": Romm(), + "vikunja": Vikunja(), + "opengist": OpenGist(), + "papra": Papra(), + "mealie": Mealie(), + "n8n": N8n(), + "zipline": Zipline(), + "grafana": Grafana(), + "audiobookshelf": AudiobookShelf(), + "actualbudget": ActualBudget(), + "nextcloud": Nextcloud(), + "adventurelog": Django("adventurelog", "travel", "/admin/login/"), + "tandoor": Django("tandoor", "recipes", "/accounts/login/", + python="/opt/recipes/venv/bin/python", workdir="/opt/recipes"), + "privatebin": PrivateBin(), + "docmost": Docmost(), + "bookstack": BookStack(), + "gitea": Gitea(), + "navidrome": Navidrome(), + "vaultwarden": Vaultwarden(), +} diff --git a/documentation/audits/update-night-2026-09-21/phase2_pgrehearsal.py b/documentation/audits/update-night-2026-09-21/phase2_pgrehearsal.py new file mode 100644 index 00000000..7c9f8b36 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/phase2_pgrehearsal.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +"""Phase 2.3 — the PostgreSQL conversion rehearsal (`09` §3b Q5). + +A REHEARSAL AND A COSTING, not a procedure for the catalog. It answers one question with numbers: +*what would it actually take to move one of the eleven PostgreSQL apps from 16 to 17, and what +could lose data?* + +Route rehearsed: **logical dump and restore** — dump with 16, fresh 17 datadir, restore, app up, +seed read back. The `pg_upgrade` route is named at the end with what it would need; it is not run +unless time remains, because it needs BOTH majors' binaries in one image and that image does not +exist in this project. + +VENUE, stated because it differs from the brief: this runs on guest 9202 itself, against the same +app the 5.2 leg left on a real 16 datadir with real seeded data — NOT on a separate harness LXC. +The rehearsal is deliberately performed with plain `docker` commands beside the product, never +through the product, because no product path for this exists and inventing one is what this +rehearsal is meant to COST rather than to build. +""" +import json, os, sys, time +from datetime import datetime, timezone + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import walk as w # noqa: E402 +from fixtures import FIXTURES # noqa: E402 + +APP = "docmost" +SUB = "docs" +PG = "docmost-postgres" +OUT = os.path.join(HERE, "bad-days", "P2.3-pg-rehearsal") + + +def step(label, script, timeout=900): + t0 = time.time() + out = w.guest(script, timeout=timeout) + dt = round(time.time() - t0, 1) + w.say(f" [{label}] {dt}s") + for line in out.strip().split("\n")[:14]: + if line.strip(): + w.say(f" {line[:200]}") + return {"label": label, "seconds": dt, "output": out} + + +def main(): + os.makedirs(OUT, exist_ok=True) + w.login() + w.say("==== Phase 2.3: the PostgreSQL 16 -> 17 conversion rehearsal (Q5)") + rec = {"leg": "P2.3", "app": APP, "route": "logical dump and restore", + "measured_at": datetime.now(timezone.utc).isoformat(), "steps": [], "notes": []} + + st = w.stack(APP) + if not st.get("deployed"): + w.say(" docmost is not deployed — the rehearsal needs the 5.2 leg's seeded 16 datadir") + rec["notes"].append("docmost not deployed; rehearsal not run") + json.dump(rec, open(f"{OUT}/result.json", "w"), indent=2, ensure_ascii=False) + return + + # the seed must be readable BEFORE, or nothing after it means anything + fx = FIXTURES[APP] + tok = fx.seed(w, SUB, w.say) + if tok is None: + rec["notes"].append("could not seed before the rehearsal — see log") + json.dump(rec, open(f"{OUT}/result.json", "w"), indent=2, ensure_ascii=False) + return + if not fx.verify(w, SUB, tok, w.say): + rec["notes"].append("C1 failed before the rehearsal") + json.dump(rec, open(f"{OUT}/result.json", "w"), indent=2, ensure_ascii=False) + return + rec["seed_read_before"] = True + + rec["steps"].append(step("00-state-before", f""" +echo "PG_VERSION (the harness's own postgres probe, verbatim):" +docker exec {PG} sh -c 'cat /var/lib/postgresql/data/PG_VERSION 2>&1' +echo "engine version:"; docker exec {PG} postgres --version +echo "datadir size:"; docker exec {PG} sh -c 'du -sh /var/lib/postgresql/data 2>/dev/null' +echo "volume:"; docker inspect {PG} --format '{{{{range .Mounts}}}}{{{{if eq .Destination "/var/lib/postgresql/data"}}}}{{{{.Name}}}}{{{{end}}}}{{{{end}}}}' +""")) + + rec["steps"].append(step("01-stop-the-app-keep-the-engine", f""" +docker stop {APP} >/dev/null 2>&1 && echo "app stopped (the engine stays up to be dumped)" +docker ps --filter name={APP} --format '{{{{.Names}}}} {{{{.Status}}}}' +""")) + + rec["steps"].append(step("02-dump-with-16", f""" +PW=$(docker inspect {PG} --format '{{{{range .Config.Env}}}}{{{{println .}}}}{{{{end}}}}' | sed -n 's/^POSTGRES_PASSWORD=//p') +U=$(docker inspect {PG} --format '{{{{range .Config.Env}}}}{{{{println .}}}}{{{{end}}}}' | sed -n 's/^POSTGRES_USER=//p') +echo "dumping as user=$U" +time docker exec -e PGPASSWORD="$PW" {PG} pg_dumpall -U "$U" > /var/lib/felhom/DRILL-pg16.sql 2>/var/lib/felhom/DRILL-pg16.err +echo "rc=$?" +ls -l --block-size=1 /var/lib/felhom/DRILL-pg16.sql | awk '{{print "dump bytes:", $5}}' +head -3 /var/lib/felhom/DRILL-pg16.err 2>/dev/null +grep -c 'CREATE TABLE' /var/lib/felhom/DRILL-pg16.sql | sed 's/^/CREATE TABLE statements: /' +""")) + + rec["steps"].append(step("03-fresh-17-datadir-and-restore", f""" +PW=$(docker inspect {PG} --format '{{{{range .Config.Env}}}}{{{{println .}}}}{{{{end}}}}' | sed -n 's/^POSTGRES_PASSWORD=//p') +U=$(docker inspect {PG} --format '{{{{range .Config.Env}}}}{{{{println .}}}}{{{{end}}}}' | sed -n 's/^POSTGRES_USER=//p') +DB=$(docker inspect {PG} --format '{{{{range .Config.Env}}}}{{{{println .}}}}{{{{end}}}}' | sed -n 's/^POSTGRES_DB=//p') +NET=$(docker inspect {PG} --format '{{{{range $k,$v := .NetworkSettings.Networks}}}}{{{{$k}}}}{{{{end}}}}' | head -1) +docker rm -f DRILL-pg17 >/dev/null 2>&1; docker volume rm DRILL-pg17-data >/dev/null 2>&1 +docker volume create DRILL-pg17-data >/dev/null +docker run -d --name DRILL-pg17 --network "$NET" \\ + -e POSTGRES_USER="$U" -e POSTGRES_PASSWORD="$PW" -e POSTGRES_DB="$DB" \\ + -v DRILL-pg17-data:/var/lib/postgresql/data postgres:17-alpine >/dev/null +for i in $(seq 1 60); do docker exec DRILL-pg17 pg_isready -U "$U" >/dev/null 2>&1 && break; sleep 2; done +echo "17 up: $(docker exec DRILL-pg17 postgres --version)" +echo "PG_VERSION on the fresh datadir: $(docker exec DRILL-pg17 cat /var/lib/postgresql/data/PG_VERSION)" +time docker exec -i -e PGPASSWORD="$PW" DRILL-pg17 psql -U "$U" -d postgres < /var/lib/felhom/DRILL-pg16.sql > /var/lib/felhom/DRILL-restore.log 2>&1 +echo "restore rc=$?" +grep -ciE '^ERROR' /var/lib/felhom/DRILL-restore.log | sed 's/^/ERROR lines in the restore: /' +grep -iE '^ERROR' /var/lib/felhom/DRILL-restore.log | head -5 +echo "tables restored:"; docker exec -e PGPASSWORD="$PW" DRILL-pg17 psql -U "$U" -d "$DB" -tAc "select count(*) from information_schema.tables where table_schema='public'" +""")) + + rec["steps"].append(step("04-point-the-app-at-17-and-start-it", f""" +NET=$(docker inspect {PG} --format '{{{{range $k,$v := .NetworkSettings.Networks}}}}{{{{$k}}}}{{{{end}}}}' | head -1) +docker stop {PG} >/dev/null 2>&1 +docker network disconnect "$NET" DRILL-pg17 >/dev/null 2>&1 +docker network connect --alias {PG} "$NET" DRILL-pg17 +echo "DRILL-pg17 now answers to the name {PG} on $NET" +docker start {APP} >/dev/null && echo "app started" +for i in $(seq 1 60); do + S=$(docker inspect {APP} --format '{{{{.State.Running}}}}'); [ "$S" = true ] || break; sleep 2 +done +docker ps -a --filter name={APP} --format '{{{{.Names}}}} {{{{.Status}}}}' +docker logs --tail 12 {APP} 2>&1 | tail -12 +""")) + + ok = fx.verify(w, SUB, tok, w.say) + rec["seed_read_after_on_17"] = ok + w.say(f" [05] the seed read back on PostgreSQL 17: {ok}") + + rec["steps"].append(step("06-engine-state-after", f""" +echo "the harness's own postgres probe against the CONVERTED datadir:" +docker exec DRILL-pg17 sh -c 'cat /var/lib/postgresql/data/PG_VERSION 2>&1'; echo "[exit=$?]" +docker exec DRILL-pg17 postgres --version +echo "size of the 17 datadir:"; docker exec DRILL-pg17 sh -c 'du -sh /var/lib/postgresql/data' +""")) + + rec["steps"].append(step("99-put-everything-back", f""" +NET=$(docker inspect {APP} --format '{{{{range $k,$v := .NetworkSettings.Networks}}}}{{{{$k}}}}{{{{end}}}}' | head -1) +docker stop {APP} >/dev/null 2>&1 +docker network disconnect "$NET" DRILL-pg17 >/dev/null 2>&1 +docker rm -f DRILL-pg17 >/dev/null 2>&1 +docker volume rm DRILL-pg17-data >/dev/null 2>&1 +docker start {PG} >/dev/null 2>&1; sleep 5 +docker start {APP} >/dev/null 2>&1; sleep 5 +rm -f /var/lib/felhom/DRILL-pg16.sql /var/lib/felhom/DRILL-pg16.err /var/lib/felhom/DRILL-restore.log +docker ps --filter name={APP} --format '{{{{.Names}}}} {{{{.Image}}}} {{{{.Status}}}}' +docker ps --filter name={PG} --format '{{{{.Names}}}} {{{{.Image}}}} {{{{.Status}}}}' +echo "PG_VERSION back on the original datadir: $(docker exec {PG} cat /var/lib/postgresql/data/PG_VERSION 2>&1)" +""")) + + back = fx.verify(w, SUB, tok, w.say) + rec["seed_read_back_on_16_after_teardown"] = back + w.say(f" [99] the seed still reads on the ORIGINAL 16 datadir after teardown: {back}") + + rec["total_seconds"] = sum(s["seconds"] for s in rec["steps"]) + json.dump(rec, open(f"{OUT}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{OUT}/log.txt", "w").write("\n".join(w.LOG) + "\n") + w.say(f" rehearsal total {rec['total_seconds']}s -> {OUT}/result.json") + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/update-night-2026-09-21/phase3_b1.py b/documentation/audits/update-night-2026-09-21/phase3_b1.py new file mode 100644 index 00000000..fc28c1cf --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/phase3_b1.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 +"""Phase 3 leg B1 — THE UNATTENDED HOLD, the measurement `09` §3b Q4 has never had. + +The 2026-09-21 night could not produce one: the only failing edge available was +`vikunja -> alpine:3.20`, and the within-a-major rule CORRECTLY refused to attempt it. The rule +that makes automatic updates safe is the same rule that refuses the obvious way to break one. + +So this leg builds the edge that rule CANNOT filter out: + + localhost:5000/drill/glance:1.0.0 the real glance image, retagged — it serves + localhost:5000/drill/glance:1.0.1 starts, stays up, and NEVER SERVES — health fails + +Same repository, same major, plain version tags. `stacks.CompareImageRefs` orders them (proven by +run, not by reading, in 09-image-store.txt), so the caller WILL press it — and the health wait in +phase `verifying` must then hold the app. + +Nobody presses anything: `unattended-caller.py` from `audits/update-arc-gaps-2026-09-21/` is used +VERBATIM. It is evidence, not product; it presses the same guarded Update a person presses. + +THE HELD APP IS THEN LEFT ALONE UNTIL PHASE 4. The morning-after look is the measurement, not this. +""" +import json, os, subprocess, sys, time + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import walk as w # noqa: E402 + +APP = "glance" +SUB = "dashboard" +GOOD = "localhost:5000/drill/glance:1.0.0" +BAD = "localhost:5000/drill/glance:1.0.1" +OUT = os.path.join(HERE, "bad-days", "B1-unattended-hold") +CALLER = "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-arc-gaps-2026-09-21/unattended-caller.py" + + +def cur_image(): + return w.guest( + f"grep -E '^\\s+image:' /opt/docker/stacks/{APP}/docker-compose.yml | sed 's/^ *//'").strip() + + +def main(): + os.makedirs(OUT, exist_ok=True) + w.login() + w.say("==== B1: the UNATTENDED HOLD") + + # 0. the box must be clean of other behind-edges, or the caller would press them too + _, d = w.ctl("GET", "/api/stacks") + ss = (d.get("data") or []) + behind = [] + for st in ss: + if not st.get("deployed"): + continue + inst = {k: (v.get("ref") if isinstance(v, dict) else v) + for k, v in ((st.get("app_config") or {}).get("installed_images") or {}).items()} + cat = st.get("catalog_images") or {} + if inst and cat and inst != cat: + behind.append((st["name"], inst, cat)) + w.say(f" [0] deployed apps not level with the catalog BEFORE the leg: " + f"{[b[0] for b in behind]}") + open(f"{OUT}/00-precondition.txt", "w").write(json.dumps(behind, indent=2, ensure_ascii=False)) + + # THE CALLER PRESSES EVERY BEHIND, WITHIN-A-MAJOR APP IT FINDS — that is the whole point of it, + # and it means any OTHER app left behind by Phase 1 would be swept into this leg and muddy it. + # Those are all throwaways, so they are removed through the product first. Recorded, because a + # precondition arranged silently is a precondition nobody can check. + swept = [] + for name, inst, cat in behind: + if name == APP: + continue + w.say(f" [0] removing {name} so the caller has only one app to react to " + f"(installed={inst} catalog={cat})") + w.remove(name) + swept.append(name) + if swept: + w.say(f" [0] swept before the leg: {swept}") + open(f"{OUT}/00-swept.txt", "w").write(json.dumps(swept, indent=2)) + + # 1. put glance on the GOOD drill image, from scratch + st = w.stack(APP) + if st.get("deployed"): + w.say(" [1] removing the existing glance so it is redeployed from the drill image store") + w.remove(APP) + if w.drill_bump(APP, "glanceapp/glance:v0.8.5", GOOD) is None: + # the drill template may already carry a previous drill ref + w.say(" [1] template did not carry the live pin; trying the previous drill ref") + for prev in ("glanceapp/glance:v0.8.6", BAD, "localhost:5000/drill/glance:1.0.2"): + if w.drill_bump(APP, prev, GOOD) is not None: + break + w.sync_rescan() + if not w.deploy(APP, SUB): + w.say(" [1] glance never came up on the GOOD drill image — B1 cannot run") + return + w.say(f" [1] live compose now: {cur_image()}") + open(f"{OUT}/01-deployed-on-good-image.txt", "w").write( + cur_image() + "\n" + json.dumps(w.observables(APP), indent=2, ensure_ascii=False)) + + # 2. a fresh copy, so the update leans on it rather than making one + w.backup_now(APP) + + # 3. the drill catalog publishes the version that starts and never serves + h = w.drill_bump(APP, GOOD, BAD) + w.sync_rescan() + b = w.badges(APP) + w.say(f" [3] badge HU: {b['hu']}") + w.say(f" [3] badge EN: {b['en']}") + json.dump({"drill_commit": h, "badges": b}, + open(f"{OUT}/02-bad-edge-published.json", "w"), indent=2, ensure_ascii=False) + + # 4. NOBODY PRESSES ANYTHING — the caller, verbatim + w.say(" [4] running unattended-caller.py, 3 passes, 90 s apart — nobody presses anything") + t0 = time.time() + with open(f"{OUT}/03-unattended-caller.log", "w") as fh: + p = subprocess.run([sys.executable, CALLER, "--passes", "3", "--every", "90"], + stdout=fh, stderr=subprocess.STDOUT, timeout=2400) + w.say(f" [4] caller exited rc={p.returncode} after {round(time.time()-t0,1)}s") + for line in open(f"{OUT}/03-unattended-caller.log"): + if any(k in line for k in ("BEHIND", "REFUSED", "ENDED", "SKIP", "summary", "phase=")): + w.say(" " + line.rstrip()) + + # 5. what the box says now — the state, and the household's sentences in BOTH languages + st = w.stack(APP) + state = {"state": st.get("state"), "updating": st.get("updating"), + "update_phase": st.get("update_phase"), + "update_phase_label": st.get("update_phase_label"), + "update_error": st.get("update_error"), "hold_reason": st.get("hold_reason"), + "observables": w.observables(APP)} + json.dump(state, open(f"{OUT}/04-state-after.json", "w"), indent=2, ensure_ascii=False) + w.say(f" [5] state={state['state']} phase={state['update_phase']} " + f"error={state['update_error']!r} hold={state['hold_reason']!r}") + w.say(f" [5] pinned={state['observables']['pinned_images']}") + w.say(f" [5] installed={state['observables']['installed_images']}") + w.say(f" [5] inspect={state['observables']['docker_inspect']}") + + for lang, sfx in (("hu", ""), ("en", "?lang=en")): + open(f"{OUT}/05-app-page-{lang}.html", "w").write(w.page(f"/apps/{APP}{sfx}")) + open(f"{OUT}/06-app-logs.txt", "w").write(w.app_logs(APP, 300)) + open(f"{OUT}/log.txt", "w").write("\n".join(w.LOG) + "\n") + w.say(" [6] evidence written. THE APP IS LEFT HELD ON PURPOSE — Phase 4 is the measurement.") + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/update-night-2026-09-21/phase3_b4.py b/documentation/audits/update-night-2026-09-21/phase3_b4.py new file mode 100644 index 00000000..984c6cdd --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/phase3_b4.py @@ -0,0 +1,143 @@ +#!/usr/bin/env python3 +"""Phase 3 leg B4 — two Updates within one second, then five. + +Is there a single-flight, or do they run together? What does memory do? Do they all end honest? + +**Why the images come from the local store.** Each app is put on `drill/:2.0.0` and the edge is +`:2.0.1` — the SAME real image under two tags. The update is real and the pull costs nothing, so +what is measured is CONCURRENCY and not download speed. An edge that could fail on content would +confuse the two. +""" +import json, os, sys, time +import concurrent.futures as cf + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import walk as w # noqa: E402 +from phase3_legs import out # noqa: E402 + +# app -> (subdomain, drill repo, the ref the template carries today) +APPS = { + "privatebin": ("paste", "paste"), + "bentopdf": ("pdf", "pdf"), + "wishlist": ("wishes", "wishes"), + "uptime-kuma": ("status", "status"), + "opengist": ("gist", "gist"), +} +A = "localhost:5000/drill/%s:2.0.0" +B = "localhost:5000/drill/%s:2.0.1" + + +def current_ref(app): + p = f"{w.DRILL}/templates/{app}/docker-compose.yml" + for line in open(p): + s = line.strip() + if s.startswith("image:"): + return s.split("image:", 1)[1].strip() + return None + + +def prep(apps): + """Put each app on the drill A tag and deploy it.""" + w.say(f"==== B4 prep: putting {apps} on the local store's A tag") + for app in apps: + repo = APPS[app][1] + cur = current_ref(app) + if cur != A % repo: + if w.drill_bump(app, cur, A % repo) is None: + w.say(f" [prep] could not point {app} at the drill A tag (current={cur})") + return False + w.sync_rescan() + for app in apps: + st = w.stack(app) + if st.get("deployed"): + inst = {k: (v.get("ref") if isinstance(v, dict) else v) + for k, v in ((st.get("app_config") or {}).get("installed_images") or {}).items()} + if list(inst.values()) != [A % APPS[app][1]]: + w.say(f" [prep] {app} is deployed on {inst} — removing so it comes up on the A tag") + w.remove(app) + if not w.deploy(app, APPS[app][0]): + w.say(f" [prep] {app} never came up on the A tag") + return False + w.backup_now(apps[0]) + return True + + +def publish_b(apps): + w.say(f"==== B4: publishing the B tag for {apps} in ONE drill commit") + for app in apps: + repo = APPS[app][1] + w.drill_bump(app, A % repo, B % repo) + w.sync_rescan() + for app in apps: + st = w.stack(app) + w.say(f" [b] {app}: installed={ {k:(v.get('ref') if isinstance(v,dict) else v) for k,v in ((st.get('app_config') or {}).get('installed_images') or {}).items()} } " + f"catalog={st.get('catalog_images')}") + + +def fire(apps, label): + d = out("B4-concurrent-updates") + w.say(f"==== B4 [{label}]: pressing Update on {apps} at once") + t0 = time.time() + presses = {} + with cf.ThreadPoolExecutor(max_workers=len(apps)) as ex: + fut = {ex.submit(w.ctl, "POST", f"/api/stacks/{a}/update"): a for a in apps} + for f in cf.as_completed(fut): + a = fut[f] + c, b = f.result() + presses[a] = {"http": c, "at_s": round(time.time() - t0, 3), + "reason": (b.get("data") or {}).get("reason") if isinstance(b, dict) else None, + "error": b.get("error") if isinstance(b, dict) else None} + w.say(f" press {a:14} +{presses[a]['at_s']:.3f}s http={c} " + f"reason={presses[a]['reason']!r} :: {(presses[a]['error'] or '')[:90]}") + spread = round(max(p["at_s"] for p in presses.values()), 3) + w.say(f" all {len(apps)} presses issued within {spread}s") + + # were they RUNNING together, or one at a time? sample the phases + samples, mem = [], [] + t1 = time.time() + while time.time() - t1 < 900: + row = {} + for a in apps: + st = w.stack(a) + row[a] = (st.get("updating"), st.get("update_phase")) + samples.append({"t": round(time.time() - t1, 1), "state": row}) + n_updating = sum(1 for v in row.values() if v[0]) + if len(samples) % 4 == 1: + w.say(f" +{samples[-1]['t']:>6.1f}s updating={n_updating} " + + " ".join(f"{a}={row[a][1]}" for a in apps)) + mem.append(w.guest("free -m | sed -n 2p")) + if all(not v[0] for v in row.values()) and time.time() - t1 > 5: + break + time.sleep(2) + + max_concurrent = max(sum(1 for v in s["state"].values() if v[0]) for s in samples) + w.say(f" MOST UPDATES IN FLIGHT AT ONCE: {max_concurrent} of {len(apps)}") + fin = {} + for a in apps: + st = w.stack(a) + fin[a] = {"state": st.get("state"), "update_phase": st.get("update_phase"), + "update_error": st.get("update_error"), "hold_reason": st.get("hold_reason"), + "observables": w.observables(a)} + w.say(f" {a:14} ended phase={fin[a]['update_phase']} err={fin[a]['update_error']!r} " + f"hold={fin[a]['hold_reason']!r} pinned={fin[a]['observables']['pinned_images']}") + rec = {"leg": f"B4-{label}", "apps": apps, "presses": presses, + "press_spread_s": spread, "max_concurrent_updates": max_concurrent, + "samples": samples, "memory_samples": mem, "final": fin, + "elapsed_s": round(time.time() - t1, 1)} + p = f"{d}/{label}.json" + json.dump(rec, open(p, "w"), indent=2, ensure_ascii=False) + open(f"{d}/log.txt", "w").write("\n".join(w.LOG) + "\n") + w.say(f" -> {p}") + + +if __name__ == "__main__": + w.login() + cmd = sys.argv[1] + apps = sys.argv[2].split(",") + if cmd == "prep": + prep(apps) + elif cmd == "publish": + publish_b(apps) + elif cmd == "fire": + fire(apps, sys.argv[3]) diff --git a/documentation/audits/update-night-2026-09-21/phase3_b5.py b/documentation/audits/update-night-2026-09-21/phase3_b5.py new file mode 100644 index 00000000..8739f279 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/phase3_b5.py @@ -0,0 +1,200 @@ +#!/usr/bin/env python3 +"""Phase 3 leg B5 — a power cut in `backing-up`, and in `safety-dump`. + +R-520 cut in `pulling` (nothing had run). R-610 cut after `starting` (the migration had run). The +two phases NOBODY has cut in are the two EARLY ones, and they are the ones that touch the +customer's copy rather than their data: + + backing-up `RunAppBackupNow` is making the copy the update will lean on + safety-dump `WriteUpdateSafetyDump` is writing R-361's undo copy, BEFORE the pin moves + +Source says both should end with *nothing moved, the journal entry dropped, and the interrupted +sentence shown*. THE CLAIM THIS LEG EXISTS TO TEST is narrower and nastier than that: **a backup +artefact that was half-written must not be left looking whole**, because the next update's +precondition will believe it. + +INSTRUMENT LIMITS, STATED UP FRONT because they bound every claim below: + * the poll is 200 ms and `pct stop` returns in 3–4 s, so the phase at the DECISION is observed + and the phase at the FREEZE is inferred. Said every time, never glossed (R-520's own lesson). + * `backing-up` only happens when no tier holds a copy younger than `update.backup_max_age`, so + this leg lowers that knob — an operator-owned setting on `controller.yaml`, restored at teardown. + * `pct mount` shows an EMPTY STUB for the guest's inner mounts, so every post-crash read is taken + from the BOOTED guest, or with `find` over the whole rootfs and a positive control. An empty + directory is not evidence of an absent file. +""" +import json, os, re, subprocess, sys, time +from datetime import datetime + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import walk as w # noqa: E402 +from fixtures import FIXTURES # noqa: E402 +from phase3_legs import out, sentences # noqa: E402 + +V = "/var/lib/docker/volumes/felhom-controller-data/_data" + + +def hostsh(cmd, timeout=300): + return (w.sh(["ssh", "-o", "ConnectTimeout=20", w.HP, cmd], timeout=timeout).stdout or "") + + +def set_knob(key, value): + """Set one operator-owned knob in controller.yaml and restart. Restored at teardown.""" + scr = f""" +python3 - <<'PY' +import re +p="{V}/controller.yaml" +s=open(p).read() +if re.search(r'^update:', s, re.M): + if re.search(r'^(update:\\n(?: .*\\n)*?) {key}:.*$', s, re.M): + s=re.sub(r'^(update:\\n(?: .*\\n)*?) {key}:.*$', r'\\g<1> {key}: {value}', s, count=1, flags=re.M) + else: + s=re.sub(r'^update:\\n', 'update:\\n {key}: {value}\\n', s, count=1, flags=re.M) +else: + s += "\\nupdate:\\n {key}: {value}\\n" +open(p,'w').write(s) +PY +sed -n '/^update:/,/^[a-z]/p' {V}/controller.yaml +systemctl restart felhom-controller-bootstrap.service +sleep 22 +docker ps --filter name=felhom-controller --format '{{{{.Status}}}}' +""" + o = w.guest(scr, timeout=400) + w.say(f" [knob] {key}={value} :: " + " | ".join(x for x in o.split("\n") if x.strip())[:260]) + w.login() + return o + + +def cut_during(app, want_phase, poll=0.2, cap_s=300): + """Press Update, poll at 200 ms, and pull the plug the moment the wanted phase is observed.""" + code, body = w.ctl("POST", f"/api/stacks/{app}/update") + w.say(f" [cut] Update -> {code} {str(body)[:120]}") + if code not in ("202", "200"): + return {"pressed": False, "http": code, "body": body} + seen, t0, decided = [], time.time(), None + while time.time() - t0 < cap_s: + st = w.stack(app) + ph = st.get("update_phase") + if not seen or seen[-1]["phase"] != ph: + seen.append({"t": round(time.time() - t0, 3), "phase": ph}) + w.say(f" +{seen[-1]['t']:>7.3f}s phase={ph}") + if ph == want_phase: + decided = datetime.utcnow().isoformat(timespec="milliseconds") + "Z" + w.say(f" [cut] phase {want_phase!r} OBSERVED at {decided} — pulling the plug NOW") + t1 = time.time() + r = hostsh("pct stop 9202", timeout=180) + w.say(f" [cut] `pct stop 9202` returned after {round(time.time()-t1,2)}s :: {r.strip()[:120]}") + return {"pressed": True, "phases_seen": seen, "cut_decided_at": decided, + "pct_stop_returned_after_s": round(time.time() - t1, 2), + "instrument_limit": "the phase at the DECISION is observed; the phase at the " + "FREEZE is inferred — pct stop is not instantaneous"} + if not st.get("updating") and ph in ("done", "failed"): + w.say(f" [cut] the update ENDED at phase {ph} before {want_phase!r} was ever seen") + return {"pressed": True, "phases_seen": seen, "cut_decided_at": None, + "missed": want_phase, "ended_phase": ph} + time.sleep(poll) + return {"pressed": True, "phases_seen": seen, "cut_decided_at": None, "timeout": True} + + +def boot_and_read(app): + hostsh("pct start 9202", timeout=300) + w.say(" [boot] guest 9202 starting") + for _ in range(90): + time.sleep(5) + o = hostsh("pct exec 9202 -- docker ps --filter name=felhom-controller " + "--format '{{.Status}}' 2>/dev/null") + if "Up" in o: + w.say(f" [boot] controller back: {o.strip()}") + break + time.sleep(20) + w.login() + recovery = w.guest( + "docker logs felhom-controller 2>&1 | grep -iE 'recover|interrupted|journal|pin .*back|" + f"resum' | tail -20; echo '--- journal file:'; ls -la {V}/data/update-journal.json 2>&1") + w.say(" [boot] recovery lines: " + " | ".join( + x for x in recovery.split("\n") if x.strip())[:500]) + st = w.stack(app) + return {"recovery_lines": recovery, "state": st.get("state"), + "update_phase": st.get("update_phase"), "update_error": st.get("update_error"), + "hold_reason": st.get("hold_reason"), "observables": w.observables(app)} + + +def backup_artefacts(app): + """Is a half-written backup artefact left LOOKING WHOLE? The question the leg exists for.""" + return w.guest(f""" +echo "=== the app's own recovery unit + its db dumps, with sizes and times" +find /mnt/sys_drive/felhom-data/backups -path '*{app}*' -type f -printf '%TY-%Tm-%Td %TH:%TM %10s %p\\n' 2>/dev/null | sort | tail -25 +echo "=== any temp/partial names left behind" +find /mnt/sys_drive/felhom-data/backups -path '*{app}*' \\( -name '*.tmp' -o -name '*.part' -o -name '*partial*' -o -name '*.inprogress' \\) 2>/dev/null | head -10 || true +echo "=== the unit manifest, if there is one" +find /mnt/sys_drive/felhom-data/backups -path '*{app}*' -name 'manifest*.json' -exec sh -c 'echo "--- {{}}"; head -c 700 "{{}}"; echo' \\; 2>/dev/null | head -40 +echo "=== ZERO-BYTE files under this app's backups (a half-write that still looks like a file)" +find /mnt/sys_drive/felhom-data/backups -path '*{app}*' -type f -size 0 2>/dev/null | head -10 || echo "(none)" +""", timeout=420) + + +def run(app, sub, phase, label): + d = out(f"B5-{label}") + w.say(f"==== B5: a power cut during `{phase}` on {app}") + rec = {"leg": f"B5-{label}", "app": app, "phase_targeted": phase} + + # B5 needs a PENDING edge or there is nothing for the cut to interrupt. B4 leaves these apps + # level with the catalog, so publish one here — the same image under the next tag, so the pull + # is instant and the cut lands in the EARLY phases this leg is about rather than in `pulling`. + repo = {"privatebin": "paste", "bentopdf": "pdf"}.get(app) + if repo: + cur = None + for line in open(f"{w.DRILL}/templates/{app}/docker-compose.yml"): + if line.strip().startswith("image:"): + cur = line.strip().split("image:", 1)[1].strip() + break + nxt = f"localhost:5000/drill/{repo}:2.0.2" + if cur and cur != nxt: + w.drill_bump(app, cur, nxt) + w.sync_rescan() + st = w.stack(app) + w.say(f" [0] pending edge for the cut: installed=" + f"{ {k:(v.get('ref') if isinstance(v,dict) else v) for k,v in ((st.get('app_config') or {}).get('installed_images') or {}).items()} } " + f"catalog={st.get('catalog_images')}") + + before = w.observables(app) + rec["observables_before"] = before + rec["backup_artefacts_before"] = backup_artefacts(app) + open(f"{d}/00-backups-before.txt", "w").write(rec["backup_artefacts_before"]) + w.say(f" [0] pinned before = {before['pinned_images']}") + + rec["cut"] = cut_during(app, phase) + if not rec["cut"].get("cut_decided_at"): + w.say(" [!] the phase was never observed — the cut did NOT happen. Recorded as a MISS, " + "not as a pass.") + json.dump(rec, open(f"{d}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{d}/log.txt", "w").write("\n".join(w.LOG) + "\n") + return + + rec["after_boot"] = boot_and_read(app) + rec["backup_artefacts_after"] = backup_artefacts(app) + open(f"{d}/01-backups-after.txt", "w").write(rec["backup_artefacts_after"]) + rec["sentences"] = sentences(app) + w.say(f" [2] household sentence HU: {rec['sentences']['hu'][:3]}") + w.say(f" [2] household sentence EN: {rec['sentences']['en'][:3]}") + + fx = FIXTURES.get(app) + if fx: + tok = fx.seed(w, sub, w.say) + rec["app_usable_after"] = bool(tok) and fx.verify(w, sub, tok, w.say) + w.say(f" [3] the app works and holds data after the cut: {rec['app_usable_after']}") + + a, b = rec["observables_before"], rec["after_boot"]["observables"] + rec["pin_moved"] = a["pinned_images"] != b["pinned_images"] + w.say(f" [4] pinned after = {b['pinned_images']} (moved: {rec['pin_moved']})") + json.dump(rec, open(f"{d}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{d}/log.txt", "w").write("\n".join(w.LOG) + "\n") + + +if __name__ == "__main__": + w.login() + cmd = sys.argv[1] + if cmd == "knob": + set_knob(sys.argv[2], sys.argv[3]) + else: + run(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]) diff --git a/documentation/audits/update-night-2026-09-21/phase3_legs.py b/documentation/audits/update-night-2026-09-21/phase3_legs.py new file mode 100644 index 00000000..9faaf064 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/phase3_legs.py @@ -0,0 +1,242 @@ +#!/usr/bin/env python3 +"""Phase 3 — the bad days, legs B2..B8. Usage: phase3_legs.py + +Every leg records the SAME five things (the brief §6): + what the household saw (BOTH languages) · what the box did by itself · time to steady · + which alarm or event fired and whether it was true · which should have fired and did not. + +Nothing here is product code. Every act is a button a person can press. +""" +import json, os, re, subprocess, sys, time +from datetime import datetime + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import walk as w # noqa: E402 + +BAD = os.path.join(HERE, "bad-days") + + +def out(leg): + d = os.path.join(BAD, leg) + os.makedirs(d, exist_ok=True) + return d + + +def sentences(app): + """The household's own sentences on the app page, both languages, ASCII-fragment matched + (R-96 rule 8: Hungarian is searched by ASCII fragment, with a positive and a negative control).""" + res = {} + for lang, sfx in (("hu", ""), ("en", "?lang=en")): + html = w.page(f"/apps/{app}{sfx}") + txt = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", html)) + keep = [s.strip() for s in re.split(r"(?<=[.!?]) ", txt) + if any(k in s.lower() for k in + ("friss", "update", "vissza", "restore", "ment", "backup", + "hib", "error", "megsz", "nem "))] + res[lang] = keep[:12] + return res + + +def events(n=40): + for p in ("/api/events?limit=%d" % n, "/api/debug/dump"): + code, d = w.ctl("GET", p) + if code == "200": + return {"path": p, "body": d} + return {"path": None, "body": None} + + +def snap(app, label): + st = w.stack(app) + return {"label": label, "at": datetime.now().isoformat(timespec="seconds"), + "state": st.get("state"), "updating": st.get("updating"), + "update_phase": st.get("update_phase"), "update_error": st.get("update_error"), + "hold_reason": st.get("hold_reason"), "observables": w.observables(app)} + + +# ---------------------------------------------------------------------------- B2 +def b2(): + """B2 — the new tag cannot be pulled. Phase `pulling` fails; §6.1 says the pin AND the + definition are PUT BACK and nothing ran.""" + APP, SUB = "bentopdf", "pdf" + GOOD, GONE = "localhost:5000/drill/pdf:1.0.0", "localhost:5000/drill/pdf:1.0.1" + d = out("B2-pull-fails") + w.say("==== B2: the new tag cannot be pulled") + st = w.stack(APP) + if st.get("deployed"): + w.remove(APP) + for prev in ("ghcr.io/alam00000/bentopdf:v2.8.6", GONE, "ghcr.io/alam00000/bentopdf:v2.8.8"): + if w.drill_bump(APP, prev, GOOD) is not None: + break + w.sync_rescan() + if not w.deploy(APP, SUB): + w.say(" bentopdf never came up on the drill image — B2 cannot run") + return + w.backup_now(APP) + before = snap(APP, "before") + w.drill_bump(APP, GOOD, GONE) + w.sync_rescan() + bdg = w.badges(APP) + w.say(f" badge HU {bdg['hu']}") + t0 = time.time() + res = w.press_update(APP) + steady = round(time.time() - t0, 1) + after = snap(APP, "after") + # is the OLD version still serving? the household's own door + rc, code, _ = w.app_curl(SUB, "/", timeout=20) + serving = code in ("200", "302") + # what does an unattended caller see on the wire? + code2, ref = w.ctl("POST", f"/api/stacks/{APP}/update") + w.say(f" second press (to read the refusal on the wire): http={code2} {json.dumps(ref, ensure_ascii=False)[:300]}") + rec = {"leg": "B2", "app": APP, "edge": f"{GOOD} -> {GONE} (tag absent from the store)", + "badges": bdg, "phases": res, "before": before, "after": after, + "old_version_still_serving": serving, "time_to_steady_s": steady, + "sentences": sentences(APP), "second_press": {"http": code2, "body": ref}, + "events": events()} + json.dump(rec, open(f"{d}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{d}/log.txt", "w").write("\n".join(w.LOG) + "\n") + w.say(f" phases={[p['phase'] for p in res['phases']]} error={after['update_error']!r} " + f"hold={after['hold_reason']!r} old_serving={serving}") + w.say(f" pinned after = {after['observables']['pinned_images']}") + w.say(f" compose after= {after['observables']['live_compose_image_lines']}") + + +# ---------------------------------------------------------------------------- B3 +def b3(app="bentopdf"): + """B3 — Update pressed WHILE a backup runs. Expect reason `busy`, then a later pass gets in.""" + d = out("B3-busy-during-backup") + w.say("==== B3: Update pressed while a backup is running") + code, msg = w.ctl("POST", "/api/backup/run") + w.say(f" „Mentés most\" -> {code} {str(msg)[:120]}") + time.sleep(1) + tries = [] + for i in range(6): + c, b = w.ctl("POST", f"/api/stacks/{app}/update") + reason = (b.get("data") or {}).get("reason") if isinstance(b, dict) else None + tries.append({"i": i, "http": c, "reason": reason, + "error": b.get("error") if isinstance(b, dict) else None}) + w.say(f" press {i}: http={c} reason={reason!r} :: " + f"{(b.get('error') if isinstance(b,dict) else '') or ''}") + if c != "409": + break + time.sleep(3) + # then wait the backup out and press once more — the caller's "retry next pass" + for _ in range(90): + time.sleep(5) + _, s = w.ctl("GET", "/api/backup/status") + if not (s.get("data") or {}).get("running", False): + break + c, b = w.ctl("POST", f"/api/stacks/{app}/update") + after_reason = (b.get("data") or {}).get("reason") if isinstance(b, dict) else None + w.say(f" after the backup finished: http={c} reason={after_reason!r} " + f":: {(b.get('error') if isinstance(b,dict) else '') or ''}") + # If it was accepted, FOLLOW it to the end. Leaving an update in flight would make the next + # leg's first press read `updating` and measure this leg by accident. + ran = None + if c in ("200", "202"): + t0 = time.time() + while time.time() - t0 < 1200: + st = w.stack(app) + if not st.get("updating") and st.get("update_phase") in ("done", "failed"): + ran = {"final_phase": st.get("update_phase"), + "update_error": st.get("update_error"), + "hold_reason": st.get("hold_reason"), + "seconds": round(time.time() - t0, 1), + "observables": w.observables(app)} + break + time.sleep(2) + w.say(f" the update that got through ended: {ran}") + rec = {"leg": "B3", "app": app, "during_backup": tries, + "after_backup": {"http": c, "reason": after_reason, "body": b}, + "the_update_that_got_through": ran, + "sentences": sentences(app)} + json.dump(rec, open(f"{d}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{d}/log.txt", "w").write("\n".join(w.LOG) + "\n") + + +# ---------------------------------------------------------------------------- B4 +def b4(apps): + """B4 — two Updates within one second, then five. Single-flight, or do they run together?""" + d = out("B4-concurrent-updates") + w.say(f"==== B4: concurrent Updates on {apps}") + import concurrent.futures as cf + results = {} + with cf.ThreadPoolExecutor(max_workers=len(apps)) as ex: + fut = {ex.submit(w.ctl, "POST", f"/api/stacks/{a}/update"): a for a in apps} + for f in cf.as_completed(fut): + a = fut[f] + c, b = f.result() + results[a] = {"http": c, "reason": (b.get("data") or {}).get("reason") + if isinstance(b, dict) else None, "body": b} + w.say(f" {a}: http={c} reason={results[a]['reason']!r}") + mem = w.guest("free -m | head -2; docker stats --no-stream --format " + "'{{.Name}} {{.MemUsage}}' | head -20") + w.say(" memory during: " + " | ".join(mem.split("\n")[:3])) + states = {} + t0 = time.time() + while time.time() - t0 < 1200: + states = {a: w.stack(a) for a in apps} + if all(not s.get("updating") for s in states.values()): + break + time.sleep(3) + fin = {a: {"state": s.get("state"), "update_phase": s.get("update_phase"), + "update_error": s.get("update_error"), "hold_reason": s.get("hold_reason"), + "observables": w.observables(a)} for a, s in states.items()} + for a, v in fin.items(): + w.say(f" {a} ended phase={v['update_phase']} err={v['update_error']!r} hold={v['hold_reason']!r}") + json.dump({"leg": "B4", "apps": apps, "presses": results, "memory_during": mem, + "final": fin, "elapsed_s": round(time.time() - t0, 1)}, + open(f"{d}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{d}/log.txt", "w").write("\n".join(w.LOG) + "\n") + + +# ---------------------------------------------------------------------------- B7 +def b7(app="bentopdf"): + """B7 — the 2 GB disk floor and the memory refusal. Both refuse BEFORE anything is recorded + (§6.1 phase 0), so nothing moves — which is what makes them safe to probe at all.""" + d = out("B7-disk-and-memory-refusals") + w.say("==== B7: the disk floor and the memory refusal") + free_before = w.guest("df -h /var/lib/felhom | tail -1; df -B1 --output=avail /var/lib/felhom | tail -1") + w.say(f" free before: {' '.join(free_before.split())}") + # eat the space down under the 2 GB floor with ONE file, on the scratch guest only + fill = w.guest(""" +AVAIL=$(df -B1 --output=avail /var/lib/felhom | tail -1 | tr -d ' ') +KEEP=1500000000 # leave ~1.5 GB, i.e. UNDER the fixed 2 GB floor +EAT=$((AVAIL - KEEP)) +echo "avail=$AVAIL eat=$EAT" +if [ "$EAT" -gt 0 ]; then fallocate -l "$EAT" /var/lib/felhom/DRILL-FILL.bin && echo filled; fi +df -h /var/lib/felhom | tail -1 +""", timeout=600) + w.say(" " + " | ".join(x for x in fill.split("\n") if x.strip())) + c, b = w.ctl("POST", f"/api/stacks/{app}/update") + reason = (b.get("data") or {}).get("reason") if isinstance(b, dict) else None + w.say(f" Update under the floor: http={c} reason={reason!r} :: " + f"{(b.get('error') if isinstance(b,dict) else '') or ''}") + snap_low = snap(app, "under-the-floor") + sent = sentences(app) + # english refusal too — the 409 body is localised by errText since v0.260.0 + c_en, b_en = w.ctl("POST", f"/api/stacks/{app}/update?lang=en") + w.say(f" refusal EN: http={c_en} :: {(b_en.get('error') if isinstance(b_en,dict) else '') or ''}") + freed = w.guest("rm -f /var/lib/felhom/DRILL-FILL.bin; sync; df -h /var/lib/felhom | tail -1") + w.say(f" freed: {' '.join(freed.split())}") + json.dump({"leg": "B7", "app": app, "free_before": free_before, "fill": fill, + "refusal_hu": {"http": c, "reason": reason, "body": b}, + "refusal_en": {"http": c_en, "body": b_en}, + "state_under_floor": snap_low, "sentences": sent, "freed": freed}, + open(f"{d}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{d}/log.txt", "w").write("\n".join(w.LOG) + "\n") + + +if __name__ == "__main__": + w.login() + leg = sys.argv[1] + if leg == "b2": + b2() + elif leg == "b3": + b3(*sys.argv[2:]) + elif leg == "b4": + b4(sys.argv[2].split(",")) + elif leg == "b7": + b7(*sys.argv[2:]) + else: + sys.exit(f"unknown leg {leg}") diff --git a/documentation/audits/update-night-2026-09-21/phase3_legs2.py b/documentation/audits/update-night-2026-09-21/phase3_legs2.py new file mode 100644 index 00000000..e0a6a79f --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/phase3_legs2.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +"""Phase 3 — legs B6, B8, B9. Usage: phase3_legs2.py [args]""" +import json, os, re, sys, time +from datetime import datetime + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import walk as w # noqa: E402 +from phase3_legs import out, sentences, snap # noqa: E402 + +DRILL = w.DRILL + + +# ---------------------------------------------------------------------------- B6 +def b6(app="glance"): + """B6 — THE WAY OUT, FORWARDS. B1 left this app HELD on an image that never serves. The drill + catalog now publishes a FIXED next version. Does a held app accept the newer Update, or is a + restore the only route? + + WHATEVER IT DOES, the question is whether it matches `09` §6.1 — a design decision is not a + defect (R-370). §6.1 says the hold is `settings.RestoreHold` with `reason: update_failed`, and + that **a successful unit restore lifts an update hold (only that kind)**. It does NOT say a + newer catalog version lifts one. So a refusal here is the DESIGN, and the finding it produces + belongs to Q4 — "is the household stuck until an operator acts?" — not to a bug list. + """ + d = out("B6-way-out-forwards") + FIXED = "localhost:5000/drill/glance:1.0.2" + BADIMG = "localhost:5000/drill/glance:1.0.1" + w.say("==== B6: a held app meets a FIXED newer version") + before = snap(app, "held-before") + w.say(f" before: state={before['state']} phase={before['update_phase']} " + f"hold={before['hold_reason']!r} err={before['update_error']!r}") + h = w.drill_bump(app, BADIMG, FIXED) + w.sync_rescan() + bdg = w.badges(app) + w.say(f" badge HU after the fix is published: {bdg['hu']}") + w.say(f" badge EN after the fix is published: {bdg['en']}") + code, body = w.ctl("POST", f"/api/stacks/{app}/update") + reason = (body.get("data") or {}).get("reason") if isinstance(body, dict) else None + w.say(f" press Update on the HELD app: http={code} reason={reason!r} :: " + f"{(body.get('error') if isinstance(body,dict) else '') or ''}") + res = None + if code in ("200", "202"): + res = w.press_update(app) + w.say(f" it RAN: phases={[p['phase'] for p in res['phases']]} final={res['final_phase']}") + after = snap(app, "after") + rc, hcode, _ = w.app_curl("dashboard", "/", timeout=20) + w.say(f" the household's own door answers http={hcode}") + json.dump({"leg": "B6", "app": app, "fixed_image": FIXED, "drill_commit": h, + "badges": bdg, "press": {"http": code, "reason": reason, "body": body}, + "run": res, "before": before, "after": after, + "front_door_http": hcode, "sentences": sentences(app), + "design_reference": "09 §6.1 — only a successful unit restore lifts an update hold"}, + open(f"{d}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{d}/log.txt", "w").write("\n".join(w.LOG) + "\n") + + +# ---------------------------------------------------------------------------- B8 +def b8(app, service): + """B8 — a FLOATING pin. The badge reads „Naprakész" because the two REFERENCES are equal, + while the image behind the reference has been repushed upstream (R-446 measured six). + + The fact this leg is after is the one Q6 needs: what does `installed_images`' DIGEST say, and + does the product offer to move at all? The box never queries a registry (§8.1), so the + comparison it can make is reference-to-reference — the question is what that costs. + """ + d = out("B8-floating-pin") + w.say(f"==== B8: the floating pin {service} on {app}") + st = w.stack(app) + ac = st.get("app_config") or {} + inst = ac.get("installed_images") or {} + entry = inst.get(service) or {} + local_ref = entry.get("ref") if isinstance(entry, dict) else entry + local_digest = entry.get("digest") if isinstance(entry, dict) else None + cat = (st.get("catalog_images") or {}).get(service) + bdg = w.badges(app) + w.say(f" installed {service} = {local_ref} digest={local_digest}") + w.say(f" catalog {service} = {cat}") + w.say(f" badge HU {bdg['hu']}") + w.say(f" badge EN {bdg['en']}") + + # what the RUNNING container actually is, asked of docker — not of our record + insp = w.guest(f"docker inspect {service} --format " + f"'{{{{.Config.Image}}}} {{{{.Image}}}}' 2>/dev/null; " + f"docker image inspect {local_ref} --format " + f"'{{{{index .RepoDigests 0}}}}' 2>/dev/null") + w.say(f" docker says: {' | '.join(x for x in insp.split(chr(10)) if x.strip())}") + + # the UPSTREAM digest measured by tonight's drift re-run — no box ever asks a registry + up = None + try: + res = json.load(open("/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/" + "d029e2e6-1762-440e-956d-0760c8aea4b3/scratchpad/results.json")) + up = (res.get(local_ref) or {}).get("current_digest") + except Exception as e: + w.say(f" (upstream digest unavailable: {e})") + w.say(f" upstream digest for {local_ref} (measured from DooPlex, never from the box) = {up}") + + code, body = w.ctl("POST", f"/api/stacks/{app}/update") + reason = (body.get("data") or {}).get("reason") if isinstance(body, dict) else None + w.say(f" press Update anyway: http={code} reason={reason!r} :: " + f"{(body.get('error') if isinstance(body,dict) else '') or ''}") + res2 = w.press_update(app) if code in ("200", "202") else None + after = snap(app, "after") + st2 = w.stack(app) + inst2 = ((st2.get("app_config") or {}).get("installed_images") or {}).get(service) or {} + w.say(f" installed {service} AFTER = {inst2}") + json.dump({"leg": "B8", "app": app, "service": service, + "installed_before": entry, "catalog": cat, "badges": bdg, + "docker_inspect": insp, "upstream_digest_measured_on_dooplex": up, + "press": {"http": code, "reason": reason, "body": body}, + "run": res2, "installed_after": inst2, "after": after}, + open(f"{d}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{d}/log.txt", "w").write("\n".join(w.LOG) + "\n") + + +# ---------------------------------------------------------------------------- B9 +def b9(app, sub): + """B9 — a FROZEN app receives a newer `.felhom.yml` (R-458). + + §5.4's deliberate asymmetry: while the catalog is ahead the compose file is frozen, but + `.felhom.yml` KEEPS FLOWING, because it carries `catalog_since` which the badge needs. So a + frozen app can receive a health check written for a version it is not running and read as + degraded. **R-458 says the failure direction is a false alarm, never data loss.** This leg + measures exactly that: the app must stay UP and its data must stay readable while the box + reports it unhealthy. + """ + d = out("B9-frozen-app-newer-felhomyml") + w.say(f"==== B9: a frozen {app} receives a newer .felhom.yml") + st = w.stack(app) + inst = {k: (v.get("ref") if isinstance(v, dict) else v) + for k, v in ((st.get("app_config") or {}).get("installed_images") or {}).items()} + cat = st.get("catalog_images") or {} + frozen = bool(inst) and bool(cat) and inst != cat + w.say(f" installed={inst}") + w.say(f" catalog ={cat}") + w.say(f" FROZEN (catalog ahead of the pin)? {frozen}") + if not frozen: + w.say(" the app is not frozen — B9 needs the catalog AHEAD of the pin. Aborting this leg.") + return + + fy = f"{DRILL}/templates/{app}/.felhom.yml" + orig = open(fy).read() + open(f"{d}/felhomyml-before.txt", "w").write(orig) + before = snap(app, "before") + rc, code0, _ = w.app_curl(sub, "/", timeout=20) + w.say(f" before: state={before['state']} front door http={code0}") + + # a health check written for a NEWER version: a path this version does not serve + probe = "/__drill_only_in_the_newer_version__" + new = re.sub(r"(healthcheck:\s*\n\s*checks:\s*\n)", + r"\1 - type: http\n port: 8080\n path: " + probe + "\n", + orig, count=1) + if new == orig: + new = orig + f"\n# DRILL B9: a health check written for a newer version\nhealthcheck:\n checks:\n - type: http\n port: 8080\n path: {probe}\n" + open(fy, "w").write(new) + w.sh(["git", "-C", DRILL, "add", "-A"]) + w.sh(["git", "-C", DRILL, "commit", "-q", "-m", + f"DRILL B9: {app} .felhom.yml gains a health check for a NEWER version (R-458)"]) + w.sh(["git", "-C", DRILL, "push", "-q", "origin", "main"], timeout=120) + w.say(" pushed a .felhom.yml-only change (no image line touched)") + w.sync_rescan() + time.sleep(20) + + live = w.guest(f"grep -A6 'healthcheck' /opt/docker/stacks/{app}/.felhom.yml | head -12; " + f"echo '---compose image lines---'; " + f"grep -E '^\\s+image:' /opt/docker/stacks/{app}/docker-compose.yml | sed 's/^ *//'") + w.say(" on the box now: " + " | ".join(x for x in live.split("\n") if x.strip())[:400]) + states = [] + for _ in range(10): + s = w.stack(app) + rc, code1, _ = w.app_curl(sub, "/", timeout=15) + states.append({"state": s.get("state"), "health": s.get("health"), + "front_door": code1, "at": datetime.now().isoformat(timespec="seconds")}) + time.sleep(12) + w.say(f" states over 2 minutes: {[ (x['state'], x['health'], x['front_door']) for x in states ]}") + after = snap(app, "after") + sent = sentences(app) + w.say(f" household sentences HU: {sent['hu'][:4]}") + + open(fy, "w").write(orig) + w.sh(["git", "-C", DRILL, "add", "-A"]) + w.sh(["git", "-C", DRILL, "commit", "-q", "-m", f"DRILL B9: revert {app} .felhom.yml"]) + w.sh(["git", "-C", DRILL, "push", "-q", "origin", "main"], timeout=120) + w.sync_rescan() + w.say(" .felhom.yml reverted in the drill catalog") + + json.dump({"leg": "B9", "app": app, "frozen": frozen, "installed": inst, "catalog": cat, + "probe_path": probe, "on_the_box": live, "states_over_2min": states, + "before": before, "after": after, "sentences": sent, + "front_door_before": code0}, + open(f"{d}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{d}/log.txt", "w").write("\n".join(w.LOG) + "\n") + + +if __name__ == "__main__": + w.login() + leg = sys.argv[1] + if leg == "b6": + b6(*sys.argv[2:]) + elif leg == "b8": + b8(sys.argv[2], sys.argv[3]) + elif leg == "b9": + b9(sys.argv[2], sys.argv[3]) + else: + sys.exit(f"unknown leg {leg}") diff --git a/documentation/audits/update-night-2026-09-21/phase4.py b/documentation/audits/update-night-2026-09-21/phase4.py new file mode 100644 index 00000000..038245d7 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/phase4.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python3 +"""Phase 4 — the morning after. + +1. B1's held app AS A HOUSEHOLD WOULD FIND IT AT BREAKFAST: the app page, the dashboard, the + backups page, the event list, the mail — in BOTH languages, quoted. Is there ONE sentence that + says what happened, since when, which copy holds what, and what to press? Scored against + `09` §3b Q4's recommended option. +2. Every app still on 9202 healthy through its own front door; every badge true AFTER a rescan + (R-607 — a badge read before a rescan is not a measurement). +""" +import json, os, re, sys, time + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import walk as w # noqa: E402 + +OUT = os.path.join(HERE, "bad-days", "P4-morning-after") +HELD = "glance" +SUBS = {"glance": "dashboard", "uptime-kuma": "status", "wishlist": "wishes", + "vikunja": "tasks", "bentopdf": "pdf", "privatebin": "paste", "opengist": "gist", + "docmost": "docs", "tandoor": "recipes", "zipline": "img", "gitea": "git", + "vaultwarden": "vault", "romm": "arcade", "mealie": "mealie"} + + +def text_of(html): + t = re.sub(r"", " ", html, flags=re.S) + t = re.sub(r"", " ", t, flags=re.S) + return re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", t)).strip() + + +def quote(page_path, keys, label): + got = {} + for lang, sfx in (("hu", ""), ("en", "?lang=en")): + sep = "&" if "?" in page_path else "?" + p = page_path + (sfx.replace("?", sep) if sfx else "") + html = w.page(p) + t = text_of(html) + hits = [s.strip() for s in re.split(r"(?<=[.!?]) ", t) + if any(k in s.lower() for k in keys)] + got[lang] = hits[:10] + open(f"{OUT}/{label}-{lang}.html", "w").write(html) + return got + + +def main(): + os.makedirs(OUT, exist_ok=True) + w.login() + w.say("==== Phase 4: the morning after") + rec = {} + + st = w.stack(HELD) + rec["held_app_state"] = { + "name": HELD, "state": st.get("state"), "updating": st.get("updating"), + "update_phase": st.get("update_phase"), "update_phase_label": st.get("update_phase_label"), + "update_error": st.get("update_error"), "hold_reason": st.get("hold_reason"), + "observables": w.observables(HELD)} + w.say(f" [1] {HELD}: state={st.get('state')} phase={st.get('update_phase')} " + f"hold={st.get('hold_reason')!r}") + w.say(f" error = {st.get('update_error')!r}") + + KEYS = ("friss", "update", "vissza", "restore", "ment", "backup", "hib", "error", + "megsz", "nem ", "masolat", "másolat", "copy", "meghajt", "drive") + w.say(" [1] the four surfaces a household would look at, both languages") + rec["app_page"] = quote(f"/apps/{HELD}", KEYS, "app-page") + rec["dashboard"] = quote("/dashboard", KEYS, "dashboard") + rec["launcher"] = quote("/", KEYS, "launcher") + rec["backups_page"] = quote("/backups", KEYS, "backups") + rec["backups_apps_page"] = quote("/backups/apps", KEYS, "backups-apps") + for k in ("app_page", "dashboard", "launcher", "backups_page", "backups_apps_page"): + w.say(f" {k:18} HU {rec[k]['hu'][:3]}") + w.say(f" {k:18} EN {rec[k]['en'][:3]}") + + # THE FOUR THINGS Q4's recommended option promises the household are told + page_hu = " ".join(rec["app_page"]["hu"]).lower() + page_en = " ".join(rec["app_page"]["en"]).lower() + rec["q4_score"] = { + "says WHAT happened": any(k in page_hu for k in ("friss", "hib", "megsz")), + "says SINCE WHEN": bool(re.search(r"\d", " ".join(rec["app_page"]["hu"]))), + "says WHICH COPY HOLDS WHAT": any(k in page_hu for k in + ("meghajt", "masolat", "másolat", "tavoli", "távoli", + "sajat", "saját", "helyi")), + "says WHAT TO PRESS": any(k in page_hu for k in ("vissza", "ment", "nyomd", "gomb")), + "the same in ENGLISH": bool(page_en) and any(k in page_en for k in + ("update", "restore", "backup", "copy")), + "IS THE SENTENCE ENGLISH ON THE ENGLISH PAGE (R-606)": None, + } + # R-606: the update sentence is a finished Hungarian string, not a key. Look for Hungarian + # accented words or ASCII Hungarian fragments surviving on the EN page — positive and negative. + hu_frags = ["friss", "megszak", "alkalmaz", "verzio", "verzió", "vissza"] + en_only = ["update", "restore", "backup", "version"] + hu_on_en = [f for f in hu_frags if f in page_en] + en_on_en = [f for f in en_only if f in page_en] + rec["q4_score"]["IS THE SENTENCE ENGLISH ON THE ENGLISH PAGE (R-606)"] = { + "hungarian_fragments_found_on_the_english_page": hu_on_en, + "english_fragments_found (control)": en_on_en, + } + for k, v in rec["q4_score"].items(): + w.say(f" Q4 · {k}: {v}") + + # 2. every app still on the box, through its own front door, and every badge after a rescan + w.say(" [2] every app still on the box — front door and badge, after a rescan") + w.ctl("POST", "/api/sync") + time.sleep(2) + w.ctl("POST", "/api/stacks/rescan") + time.sleep(3) + _, d = w.ctl("GET", "/api/stacks") + apps = [] + for s in (d.get("data") or []): + if not s.get("deployed"): + continue + n = s["name"] + sub = SUBS.get(n, n) + rc, code, _ = w.app_curl(sub, "/", timeout=20) + inst = {k: (v.get("ref") if isinstance(v, dict) else v) + for k, v in ((s.get("app_config") or {}).get("installed_images") or {}).items()} + cat = s.get("catalog_images") or {} + bdg = w.badges(n) + truthful = (inst == cat) == any("naprak" in (x["text"] or "").lower() + for x in bdg["hu"]) if bdg["hu"] else None + apps.append({"name": n, "state": s.get("state"), "front_door_http": code, + "installed": inst, "catalog": cat, "badge_hu": bdg["hu"], + "badge_en": bdg["en"], "badge_matches_the_refs": truthful}) + w.say(f" {n:16} state={str(s.get('state')):10} door={code:4} " + f"badge={[x['text'] for x in bdg['hu']]} refs_equal={inst==cat}") + rec["apps_on_the_box"] = apps + + json.dump(rec, open(f"{OUT}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{OUT}/log.txt", "w").write("\n".join(w.LOG) + "\n") + w.say(f" written -> {OUT}/result.json") + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/update-night-2026-09-21/phase5_teardown.py b/documentation/audits/update-night-2026-09-21/phase5_teardown.py new file mode 100644 index 00000000..e04ef9cf --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/phase5_teardown.py @@ -0,0 +1,186 @@ +#!/usr/bin/env python3 +"""Phase 5 — teardown, three layers, stated. Plus Gitea. + +R-320: evidence is copied off at the end of the phase that produced it, before any revert. By the +time this runs every phase's evidence is already in this directory; this file only tears down, and +it PROVES each layer rather than asserting it. + + MACHINE (9202) every throwaway app removed THROUGH THE PRODUCT with its data; the image store + container and volume gone; drill images removed BY NAME (never `prune`); + controller.yaml restored from the saved copy; the controller restarted; and + `git.repo_url` READ BACK AND QUOTED as the LIVE catalog, with one sync + rescan + showing only the protected infra containers and no badge. + HOST (demo-hp) `pct list` and `pvesm status` before and after; guest 9201 untouched. + HUB nothing provisioned; the floor's state stated. + GITEA the drill repo KEPT, private, RESET to the live catalog's main; and the live + catalog's own main hash plus a diff of every `image:` line — expected: identical. +""" +import json, os, subprocess, sys, time + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import walk as w # noqa: E402 + +OUT = os.path.join(HERE, "teardown") +KEEP = {"traefik", "filebrowser", "felhom-controller"} # the protected infra containers +LIVE_REPO = "https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git" +LIVE_DIR = "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu" +DRILL = w.DRILL +SC = w.SC + + +def sec(name, text): + open(f"{OUT}/{name}", "w").write(text) + w.say(f" -> {name}") + return text + + +def main(): + os.makedirs(OUT, exist_ok=True) + w.login() + w.say("==== Phase 5: teardown, three layers") + rec = {} + + # ---------------------------------------------------------------- BEFORE, host layer + host_before = w.sh(["ssh", "-o", "ConnectTimeout=20", w.HP, + "pct list; echo '--- pvesm'; pvesm status"], timeout=180).stdout or "" + sec("00-host-before.txt", host_before) + rec["host_before"] = host_before + + # ---------------------------------------------------------------- MACHINE: the apps + _, d = w.ctl("GET", "/api/stacks") + deployed = [s["name"] for s in (d.get("data") or []) if s.get("deployed")] + w.say(f" [M] throwaway apps still deployed: {deployed}") + removed = {} + for n in deployed: + if n in KEEP: + continue + removed[n] = w.remove(n) + rec["apps_removed"] = removed + + # ---------------------------------------------------------------- MACHINE: the image store + store = w.guest(""" +echo "=== registry container + volume, removed BY NAME (never a prune)" +docker rm -f drill-registry 2>&1 | head -2 +docker volume rm drill-registry-data 2>&1 | head -2 +echo "=== drill images, removed BY NAME" +for i in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep '^localhost:5000/drill/'); do + echo -n "$i -> "; docker rmi "$i" >/dev/null 2>&1 && echo removed || echo "in use / already gone" +done +docker rmi registry:2 >/dev/null 2>&1 && echo "registry:2 removed" || echo "registry:2 kept/in use" +echo "=== anything left that says drill?" +docker images --format '{{.Repository}}:{{.Tag}}' | grep -i drill || echo "(none)" +docker ps -a --format '{{.Names}}' | grep -i drill || echo "(no drill containers)" +echo "=== NOTHING WAS PRUNED — this is the full image list, for the record" +docker images --format '{{.Repository}}:{{.Tag}} {{.Size}}' | sort | head -40 +""", timeout=900) + sec("01-image-store-removed.txt", store) + + # ---------------------------------------------------------------- MACHINE: the config back + V = "/var/lib/docker/volumes/felhom-controller-data/_data" + back = w.guest(f""" +echo "=== restoring controller.yaml from the pre-update-night copy" +ls -la {V}/controller.yaml {V}/controller.yaml.pre-update-night +cp {V}/controller.yaml.pre-update-night {V}/controller.yaml && echo "restored" +echo "=== the git section, read back (token redacted by this script, not by the box)" +sed -n '/^git:/,/^hub:/p' {V}/controller.yaml | sed 's/token: ".*"/token: ""/' +echo "=== removing the drill catalog cache so the next sync clones the LIVE repo (R-615)" +rm -rf {V}/data/catalog-cache +systemctl restart felhom-controller-bootstrap.service +sleep 25 +docker ps --filter name=felhom-controller --format '{{{{.Image}}}} {{{{.Status}}}}' +echo "=== the cache's origin, READ BACK — this is the quote the brief asks for" +git -C {V}/data/catalog-cache remote -v 2>/dev/null | sed 's#://[^@]*@#://#' +git -C {V}/data/catalog-cache log --oneline -1 2>/dev/null +""", timeout=900) + sec("02-config-restored.txt", back) + rec["repo_url_read_back"] = LIVE_REPO in back + w.say(f" [M] git.repo_url reads back as the LIVE catalog: {rec['repo_url_read_back']}") + + # ---------------------------------------------------------------- MACHINE: sync, rescan, badges + w.login() + w.ctl("POST", "/api/sync") + time.sleep(3) + w.ctl("POST", "/api/stacks/rescan") + time.sleep(3) + _, d = w.ctl("GET", "/api/stacks") + left = [] + for s in (d.get("data") or []): + if not s.get("deployed") and s.get("state") == "not_deployed": + continue + b = w.badges(s["name"]) + left.append({"name": s["name"], "state": s.get("state"), "deployed": s.get("deployed"), + "badge_hu": [x["text"] for x in b["hu"]]}) + rec["still_on_the_box"] = left + for x in left: + w.say(f" [M] {x['name']:20} deployed={x['deployed']} state={x['state']} badge={x['badge_hu']}") + containers = w.guest("docker ps --format '{{.Names}}\t{{.Image}}\t{{.Status}}'") + sec("03-containers-after.txt", containers) + names = {l.split("\t")[0] for l in containers.strip().split("\n") if l.strip()} + rec["only_protected_infra_left"] = names <= KEEP + w.say(f" [M] containers left: {sorted(names)} only protected infra: {rec['only_protected_infra_left']}") + + # ---------------------------------------------------------------- HOST + host_after = w.sh(["ssh", "-o", "ConnectTimeout=20", w.HP, + "pct list; echo '--- pvesm'; pvesm status; echo '--- 9201 untouched:'; " + "pct exec 9201 -- docker ps --format '{{.Names}}' | sort | head -20"], + timeout=180).stdout or "" + sec("04-host-after.txt", host_after) + rec["host_after"] = host_after + w.say(" [H] no harness LXC was created tonight, so none was destroyed — " + "the PostgreSQL rehearsal ran on 9202 itself (stated in the audit)") + + # ---------------------------------------------------------------- HUB + hp = open(f"{SC}/.hubpw").read().strip() + conf = w.sh(["curl", "-s", "-u", f":{hp}", "http://10.43.52.34:8080/configuration"], + timeout=90).stdout or "" + import re + floor = re.search(r'min_controller_version" value="([^"]*)"', conf) + magent = re.search(r'name="min_agent" value="([^"]*)"', conf) + hosts = w.sh(["curl", "-s", "-u", f":{hp}", "http://10.43.52.34:8080/hosts"], + timeout=90).stdout or "" + nhosts = hosts.count("ONLINE") + hosts.count("DOWN") + hubtxt = (f"floor = {floor.group(1) if floor else '?'}\n" + f"min_agent = {magent.group(1) if magent else '?'}\n" + f"host rows seen = {nhosts}\n" + "nothing was provisioned at the hub tonight: no customer, no config, no appliance,\n" + "no binding. The only hub act of the whole night was the floor save in Phase 0.1.\n") + sec("05-hub.txt", hubtxt) + rec["hub"] = hubtxt + w.say(" [U] " + hubtxt.replace("\n", " | ")) + + # ---------------------------------------------------------------- GITEA + w.sh(["git", "-C", LIVE_DIR, "fetch", "-q", "origin"], timeout=180) + live_main = (w.sh(["git", "-C", LIVE_DIR, "rev-parse", "--short=12", "origin/main"]).stdout or "").strip() + w.sh(["git", "-C", DRILL, "fetch", "-q", "origin"], timeout=180) + w.sh(["git", "-C", DRILL, "remote", "remove", "live"], timeout=60) + w.sh(["git", "-C", DRILL, "remote", "add", "live", LIVE_REPO], timeout=60) + w.sh(["git", "-C", DRILL, "fetch", "-q", "live", "main"], timeout=300) + w.sh(["git", "-C", DRILL, "reset", "--hard", "live/main"], timeout=180) + push = w.sh(["git", "-C", DRILL, "push", "--force", "origin", "main"], timeout=300) + drill_main = (w.sh(["git", "-C", DRILL, "rev-parse", "--short=12", "HEAD"]).stdout or "").strip() + + # the diff that matters: every image: line, live vs drill + diff = w.sh(["bash", "-lc", + f"diff <(grep -rhoE '^[[:space:]]+image: .*' {LIVE_DIR}/templates/*/docker-compose.yml | sort) " + f"<(grep -rhoE '^[[:space:]]+image: .*' {DRILL}/templates/*/docker-compose.yml | sort) " + f"&& echo 'IDENTICAL — every image: line matches the live catalog'"], + timeout=180) + gitea = (f"live catalog origin/main : {live_main}\n" + f"drill repo HEAD after reset: {drill_main}\n" + f"reset+force-push rc={push.returncode}\n\n" + f"diff of every `image:` line, live vs drill:\n{diff.stdout}{diff.stderr}\n") + sec("06-gitea.txt", gitea) + rec["live_main"] = live_main + rec["drill_main"] = drill_main + rec["image_lines_identical"] = "IDENTICAL" in diff.stdout + w.say(f" [G] live main={live_main} drill main={drill_main} " + f"image lines identical: {rec['image_lines_identical']}") + + json.dump(rec, open(f"{OUT}/result.json", "w"), indent=2, ensure_ascii=False) + open(f"{OUT}/log.txt", "w").write("\n".join(w.LOG) + "\n") + w.say(f" teardown written -> {OUT}/result.json") + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/update-night-2026-09-21/reclaim.sh b/documentation/audits/update-night-2026-09-21/reclaim.sh new file mode 100755 index 00000000..9ea8c79a --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/reclaim.sh @@ -0,0 +1,20 @@ +#!/bin/bash +# Remove images BY NAME for apps that are no longer deployed. NEVER `docker image prune` and never +# `docker system prune`: a global cleanup on any Felhom host is forbidden, and the scratch guest is +# not an exception worth making. Every removal below names one exact reference and is skipped if any +# container still uses it. +cat <<'SCRIPT' | ssh -o ConnectTimeout=30 demo-hp 'cat > /tmp/rc.sh; pct push 9202 /tmp/rc.sh /tmp/rc.sh >/dev/null 2>&1; pct exec 9202 -- bash /tmp/rc.sh; rm -f /tmp/rc.sh' 2>/dev/null +set -u +echo "free before: $(df -h /var/lib/felhom | tail -1 | awk '{print $4}')" +INUSE=$(docker ps -a --format '{{.Image}}' | sort -u) +N=0 +for i in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep -v '' | sort -u); do + case "$i" in + gitea.dooplex.hu/admin/felhom-controller:*|traefik:*|gtstef/filebrowser:*|registry:2|localhost:5000/drill/*) continue;; + esac + if echo "$INUSE" | grep -qxF "$i"; then continue; fi + if docker rmi "$i" >/dev/null 2>&1; then echo " removed $i"; N=$((N+1)); fi +done +echo "removed $N images BY NAME (no prune was run)" +echo "free after: $(df -h /var/lib/felhom | tail -1 | awk '{print $4}')" +SCRIPT diff --git a/documentation/audits/update-night-2026-09-21/run_edge.py b/documentation/audits/update-night-2026-09-21/run_edge.py new file mode 100644 index 00000000..c76ae2ed --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/run_edge.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +"""run_edge.py [--sub X] [--keep] [--no-remove] + +One app's full walk on guest 9202. Writes everything under +`documentation/audits/update-night-2026-09-21/apps//`: + + log.txt every line this run printed, as it printed it + badges.json the „Frissítés elérhető" badge in BOTH languages, before and after + phases.json every update phase with its timestamp + observables.json the four version observables side by side + verdict.json `09`'s verdict-record shape + +`inconclusive` is never collapsed into `failed`. +""" +import argparse, json, os, sys, time +from datetime import datetime, timezone + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import walk as w # noqa: E402 +from fixtures import FIXTURES # noqa: E402 + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("app") + ap.add_argument("frm") + ap.add_argument("to") + ap.add_argument("--sub", default=None) + ap.add_argument("--no-remove", action="store_true") + ap.add_argument("--class", dest="cls", default="other") + a = ap.parse_args() + + app = a.app + appdir = os.path.join(HERE, "apps", app) + os.makedirs(appdir, exist_ok=True) + fx = FIXTURES.get(app) + sub = a.sub or (fx.sub if fx else app) + + t_start = time.time() + w.say(f"==== {app}: {a.frm} -> {a.to} (sub={sub}, class={a.cls})") + w.login() + + rec = {"harness_version": 1, "app": app, "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": a.cls, "from": {}, "to": {}, "verdict": "inconclusive", + "seed_read_before": False, "seed_read_after": False, "healthy_after": False, + "migration_observed": None, "abort": "not-attempted", "abort_detail": None, + "duration_s": 0, "measured_at": datetime.now(timezone.utc).isoformat(), + "evidence": f"apps/{app}/", "notes": []} + + def bail(why, verdict="inconclusive"): + rec["verdict"] = verdict + rec["notes"].append(why) + rec["duration_s"] = round(time.time() - t_start, 1) + finish(rec, appdir) + sys.exit(0 if verdict != "failed" else 0) + + # ---- 1 deploy at the LIVE pin ------------------------------------------------------- + if not w.deploy(app, sub): + bail("deploy never reached running — nothing else could be measured") + pre = w.observables(app) + rec["from"] = pre["pinned_images"] or {} + json.dump(pre, open(f"{appdir}/observables-before.json", "w"), indent=2, ensure_ascii=False) + + # ---- 2+3 seed and read it back (control C1) ------------------------------------------ + if fx is None: + rec["notes"].append("no fixture: no non-browser seed route was written for this app tonight") + bail("no fixture — recorded inconclusive rather than faked (R-156)") + w.say(" [2] seeding through the app's own front door") + tok = fx.seed(w, sub, w.say) + if tok is None: + rec["notes"].append("seed refused through the app's own route — see log.txt for what was tried") + bail("seed route did not work tonight") + w.say(" [3] control C1 — reading the seed back BEFORE the update") + if not fx.verify(w, sub, tok, w.say): + rec["notes"].append("C1 FAILED: the fixture could not prove itself before the update, " + "so it can prove nothing after") + bail("C1 failed — a fixture that cannot prove itself first proves nothing after") + rec["seed_read_before"] = True + + # ---- 4 „Mentés most" ----------------------------------------------------------------- + w.backup_now(app) + + # ---- 5 the drill bump, sync, rescan, badge ------------------------------------------- + b_before = w.badges(app) + h = w.drill_bump(app, a.frm, a.to) + if h is None: + bail("the drill bump could not be committed — the FROM ref did not match the template") + w.sync_rescan() + b_after = w.badges(app) + json.dump({"before": b_before, "after": b_after, "drill_commit": h}, + open(f"{appdir}/badges.json", "w"), indent=2, ensure_ascii=False) + w.say(f" [5] badge HU: {b_after['hu']}") + w.say(f" [5] badge EN: {b_after['en']}") + st = w.stack(app) + rec["to"] = st.get("catalog_images") or {} + + # ---- 6 the guarded Update ------------------------------------------------------------ + res = w.press_update(app) + json.dump(res, open(f"{appdir}/phases.json", "w"), indent=2, ensure_ascii=False) + rec["duration_s"] = res["duration_s"] + if not res["accepted"]: + rec["notes"].append(f"the Update was REFUSED before anything moved: {json.dumps(res['refusal'], ensure_ascii=False)[:400]}") + bail("refused at the preflight — nothing moved") + + # ---- 7 read the seed back ------------------------------------------------------------ + w.say(" [7] reading the seed back AFTER the update") + after_ok = fx.verify(w, sub, tok, w.say) + rec["seed_read_after"] = after_ok + + # ---- migration line, quoted verbatim, never inferred from timing --------------------- + logs = w.app_logs(app, 500) + open(f"{appdir}/app-logs-after.txt", "w").write(logs) + for pat in ("migrat", "Migrat", "MIGRAT", "upgrade", "Upgrade", "schema"): + for line in logs.splitlines(): + if pat in line and len(line) < 400: + rec["migration_observed"] = line.strip() + break + if rec["migration_observed"]: + break + + # ---- 8 the four observables ---------------------------------------------------------- + post = w.observables(app) + json.dump({"before": pre, "after": post}, + open(f"{appdir}/observables.json", "w"), indent=2, ensure_ascii=False) + w.say(f" [8] pinned = {post['pinned_images']}") + w.say(f" [8] installed = {post['installed_images']}") + w.say(f" [8] compose = {post['live_compose_image_lines']}") + w.say(f" [8] inspect = {post['docker_inspect']}") + rec["observables_after"] = post + + st = w.stack(app) + rec["healthy_after"] = (st.get("state") == "running" and not st.get("hold_reason")) + rec["final_phase"] = res["final_phase"] + rec["hold_reason"] = res.get("hold_reason") + rec["update_error"] = res.get("update_error") + + # ---- 9 the verdict ------------------------------------------------------------------- + moved = post["pinned_images"] != pre["pinned_images"] + if res["final_phase"] == "done" and after_ok and rec["healthy_after"] and moved: + rec["verdict"] = "proven" + elif res.get("hold_reason") or res["final_phase"] == "failed": + rec["verdict"] = "failed" + rec["notes"].append("the edge ended HELD or failed — this is a RESULT, not an error of the run") + elif not after_ok: + rec["verdict"] = "failed" + rec["notes"].append("the app came up but the seeded data did not read back") + else: + rec["verdict"] = "inconclusive" + rec["notes"].append(f"final_phase={res['final_phase']} moved={moved} healthy={rec['healthy_after']}") + + finish(rec, appdir) + + if not a.no_remove and rec["verdict"] != "failed": + w.remove(app) + + +def finish(rec, appdir): + rec["duration_s"] = rec.get("duration_s", 0) + w.write_verdict(rec, appdir) + open(f"{appdir}/log.txt", "w").write("\n".join(w.LOG) + "\n") + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/update-night-2026-09-21/run_rest.sh b/documentation/audits/update-night-2026-09-21/run_rest.sh new file mode 100755 index 00000000..aee6baac --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/run_rest.sh @@ -0,0 +1,67 @@ +#!/bin/bash +# The rest of the night, in DEPENDENCY order. Each step writes its own evidence before the next +# starts (R-320), and each is individually re-runnable. +# +# The order is not arbitrary: +# * B2/B3/B7/B9 all use bentopdf and must run BEFORE B4, which re-points it at the 2.0.x pair. +# * B8 needs a deployed app with a floating engine pin, which Phase 2.2's restore leaves behind. +# * B5 stops and starts the whole guest, so it comes after everything that does not want a reboot. +# * B6 is the LAST thing done to B1's held app, because Phase 4's morning-after look must see the +# app exactly as a household would find it at breakfast — held, and not yet interfered with. +cd /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21 || exit 1 +step() { echo; echo "############################################## $(date +%H:%M:%S) $*"; } + +step "1/15 failwalk adventurelog — the household's way out of tonight's real failed edge" +timeout 1800 python3 failwalk.py adventurelog travel + +step "2/15 Phase 2.1 — MariaDB across a major, through the real Update BUTTON" +timeout 2700 python3 engine_edge.py nextcloud mariadb:11.6 mariadb:12.3 \ + --engine mariadb --container nextcloud-db --sub cloud --restore-after + +step "3/15 Phase 2.2 — PostgreSQL across a major: what a household would see TODAY" +timeout 2700 python3 engine_edge.py docmost postgres:16-alpine postgres:17-alpine \ + --engine postgres --container docmost-postgres --sub docs --restore-after + +step "4/15 Phase 2.3 — the PostgreSQL conversion rehearsal (Q5), costed on a real seeded datadir" +timeout 4500 python3 phase2_pgrehearsal.py + +step "5/15 B1 — the UNATTENDED HOLD (the app is then LEFT HELD until Phase 4)" +timeout 2700 python3 phase3_b1.py + +step "6/15 B2 — the new tag cannot be pulled" +timeout 1800 python3 phase3_legs.py b2 + +step "7/15 B3 — Update pressed while a backup runs" +timeout 1800 python3 phase3_legs.py b3 bentopdf + +step "8/15 B7 — the 2 GB disk floor, and the refusal in both languages" +timeout 1800 python3 phase3_legs.py b7 bentopdf + +step "9/15 B9 — a FROZEN app receives a newer .felhom.yml (R-458)" +timeout 1800 python3 phase3_legs2.py b9 bentopdf pdf + +step "10/15 B8 — a FLOATING pin: the up-to-date badge over an engine image that has moved upstream" +timeout 1800 python3 phase3_legs2.py b8 docmost docmost-postgres + +step "11/15 B4 — two Updates within one second, then five" +timeout 1800 python3 phase3_b4.py prep privatebin,bentopdf +timeout 1800 python3 phase3_b4.py publish privatebin,bentopdf +timeout 1800 python3 phase3_b4.py fire privatebin,bentopdf two +timeout 2700 python3 phase3_b4.py prep privatebin,bentopdf,wishlist,uptime-kuma,opengist +timeout 2700 python3 phase3_b4.py publish privatebin,bentopdf,wishlist,uptime-kuma,opengist +timeout 2700 python3 phase3_b4.py fire privatebin,bentopdf,wishlist,uptime-kuma,opengist five + +step "12/15 B5 — a power cut in backing-up, then in safety-dump (the two phases nobody has cut in)" +timeout 900 python3 phase3_b5.py knob backup_max_age 1m +timeout 2700 python3 phase3_b5.py privatebin paste backing-up backing-up +timeout 2700 python3 phase3_b5.py bentopdf pdf safety-dump safety-dump +timeout 900 python3 phase3_b5.py knob backup_max_age 24h + +step "13/15 Phase 4 — the morning after, with B1's app still held" +timeout 2700 python3 phase4.py + +step "14/15 B6 — the way out FORWARDS: a held app meets a fixed newer version" +timeout 2700 python3 phase3_legs2.py b6 glance + +step "15/15 DONE — teardown is run separately and deliberately (phase5_teardown.py)" +echo "$(date +%H:%M:%S) all steps attempted" diff --git a/documentation/audits/update-night-2026-09-21/summarise.py b/documentation/audits/update-night-2026-09-21/summarise.py new file mode 100644 index 00000000..d604986b --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/summarise.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +"""Build the verdict table, the three lines and the promotion list from the verdict records. + +Nothing here judges anything — it reads what each edge wrote and lays it out. `inconclusive` is +carried through as itself and is never folded into `failed`. +""" +import glob, json, os, sys + +HERE = os.path.dirname(os.path.abspath(__file__)) + + +def short(refs): + if not refs: + return "—" + out = [] + for k, v in refs.items(): + v = str(v) + out.append(v.rsplit("/", 1)[-1]) + return ", ".join(sorted(set(out))) + + +def load(): + recs = [] + for p in sorted(glob.glob(os.path.join(HERE, "apps", "*", "verdict.json"))): + try: + d = json.load(open(p)) + d["_dir"] = os.path.basename(os.path.dirname(p)) + recs.append(d) + except Exception as e: + print(f"skipped {p}: {e}", file=sys.stderr) + return recs + + +def main(): + recs = load() + proven = [r for r in recs if r["verdict"] == "proven"] + failed = [r for r in recs if r["verdict"] == "failed"] + inconc = [r for r in recs if r["verdict"] == "inconclusive"] + + print("## The verdict table\n") + print("One row per edge attempted tonight. `inconclusive` means *we could not measure it*, which") + print("is a different fact from *it does not work* — and only one of them is about the app.\n") + print("| app | from → to | class | box verdict | seed before → after | secs | migration line seen | evidence |") + print("|---|---|---|---|---|---|---|---|") + for r in sorted(recs, key=lambda x: (x["verdict"] != "proven", x["app"])): + mig = r.get("migration_observed") + mig = "yes" if mig else ("—" if r["verdict"] != "proven" else "none printed") + arrow = f"{short(r.get('from'))} → {short(r.get('to'))}" + print(f"| `{r['app']}` | {arrow} | {r.get('class') or r.get('leg') or '—'} " + f"| **{r['verdict']}** | {r.get('seed_read_before')} → {r.get('seed_read_after')} " + f"| {r.get('duration_s')} | {mig} | `apps/{r['_dir']}/` |") + + print(f"\n**{len(proven)} proven · {len(failed)} failed · {len(inconc)} inconclusive " + f"— out of {len(recs)} attempted.**\n") + + if inconc: + print("### Why each inconclusive edge could not be judged\n") + for r in inconc: + why = "; ".join(r.get("notes") or []) or "—" + print(f"- **`{r['app']}`** — {why}") + print() + + if failed: + print("### The edges that failed — the most valuable results of the night\n") + for r in failed: + why = "; ".join(r.get("notes") or []) or "—" + print(f"- **`{r['app']}`** — final phase `{r.get('final_phase')}`, " + f"hold `{r.get('hold_reason')}`, error `{r.get('update_error')}`. {why}") + print() + + print("## The promotion list for the operator\n") + print("**CC promotes nothing.** These are the real, within-a-major edges that ended `proven` on") + print("the box tonight, with the data read back through the app's own front door both before and") + print("after. Moving each of them on the LIVE catalog is the operator's call.\n") + print("| app | the move | what it would mean for a box in the field |") + print("|---|---|---|") + for r in sorted(proven, key=lambda x: x["app"]): + frm, to = short(r.get("from")), short(r.get("to")) + mig = r.get("migration_observed") + note = ("the app runs its own schema migration on the way — proven here, and the update " + "takes a backup first" if mig else + "no migration line printed; the app came up on the new version with its data intact") + print(f"| `{r['app']}` | {frm} → {to} | {note} |") + print() + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/update-night-2026-09-21/walk.py b/documentation/audits/update-night-2026-09-21/walk.py new file mode 100644 index 00000000..46aeea6c --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/walk.py @@ -0,0 +1,466 @@ +#!/usr/bin/env python3 +"""walk.py — ONE app's full update walk on guest 9202, through the product's own endpoints. + +EVIDENCE, NOT PRODUCT. It presses exactly the buttons a person presses: + POST /api/stacks//deploy · POST /api/backup/run · POST /api/sync · POST /api/stacks/rescan + POST /api/stacks//update · POST /api/stacks//remove +and reads GET /api/stacks/. No controller code exists for it. + +The walk, per `09` §6.4 and the update-night brief §4: + 1 deploy from the DRILL catalog at the LIVE pin + 2 seed through the app's OWN front door (R-156: never a volume, never SQL) + 3 read the seed back <- control C1; a fixture that cannot prove itself proves nothing + 4 „Mentés most" + 5 commit the real one-step bump to the DRILL repo, sync, rescan, read the badge in BOTH languages + 6 press the guarded Update, record every phase with timestamps + 7 read the seed back through the front door + 8 the four version observables side by side + 9 write the verdict record in `09`'s JSON shape + +`inconclusive` is a first-class verdict and is NEVER collapsed into `failed`. +""" +import argparse, json, os, re, subprocess, sys, time +from datetime import datetime, timezone + +SC = "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/d029e2e6-1762-440e-956d-0760c8aea4b3/scratchpad" +EV = "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21" +DRILL = "/mnt/5_hdd/felhom.eu/drill/app-catalog-drill" +BASE = "https://192.168.0.114" +HOSTHDR = "Host: felhom.enkisfelhom.hu" +DOMAIN = "enkisfelhom.hu" +HP = "demo-hp" + +LOG = [] + + +def say(*a): + line = " ".join(str(x) for x in a) + ts = datetime.now().strftime("%H:%M:%S") + print(f"{ts} {line}", flush=True) + LOG.append(f"{ts} {line}") + + +def sh(args, timeout=300, inp=None): + try: + return subprocess.run(args, capture_output=True, text=True, timeout=timeout, input=inp) + except (subprocess.TimeoutExpired, OSError) as e: + return subprocess.CompletedProcess(args, 124, "", f"{e}") + + +def guest(script, timeout=600): + """Run a bash script inside guest 9202. Piped as a file — never as an argument (quoting).""" + r = sh(["ssh", "-o", "ConnectTimeout=20", "-o", "StrictHostKeyChecking=accept-new", HP, + "cat > /tmp/w.sh; pct push 9202 /tmp/w.sh /tmp/w.sh >/dev/null 2>&1; " + "pct exec 9202 -- bash /tmp/w.sh; rm -f /tmp/w.sh"], + timeout=timeout, inp=script) + return r.stdout or "" + + +def login(): + pw = open(f"{SC}/.ctlpw").read().strip() + sh(["curl", "-sk", "-D", f"{SC}/hdr.txt", "-o", "/dev/null", "-H", HOSTHDR, + "-X", "POST", "--data-urlencode", f"password={pw}", f"{BASE}/login"]) + h = open(f"{SC}/hdr.txt").read() + m = re.search(r"felhom_session=[A-Za-z0-9._-]+", h, re.I) + if not m: + sys.exit("login failed: no session cookie") + open(f"{SC}/sess.txt", "w").write(m.group(0)) + r = sh(["curl", "-sk", "-L", "-H", HOSTHDR, "-H", f"Cookie: {m.group(0)}", f"{BASE}/"]) + c = re.search(r'/deploy-fields` — instead of assuming DOMAIN+SUBDOMAIN. + + Measured 2026-09-21: three apps in one batch refused at the deploy with a correct 400 because + a required field was absent — `HDD_PATH` (navidrome, audiobookshelf) and an admin password + (grafana). The refusals happen BEFORE anything is created (`deploy.go:324`), which is the only + reason this was safe to discover by running it (live-probes rule). + + A `path` field must name a directory that ALREADY EXISTS (`deploy.go:330`), so one is made on + the scratch drive first — the same act the drive browser performs for a household. + """ + code, d = ctl("GET", f"/api/stacks/{name}/deploy-fields") + fields = (((d.get("data") or {}).get("metadata") or {}).get("deploy_fields")) or [] + values = {"DOMAIN": DOMAIN, "SUBDOMAIN": sub} + made = [] + for f in fields: + ev, ty = f.get("env_var"), f.get("type") + if ev in values: + continue + # `type: password` is MANDATORY whatever `required` says — `deploy.go:305-312` refuses + # when the caller sends none, deliberately ("the user needs to know their password"), + # while `.felhom.yml` declares `required: false` and the API serves that verbatim. A + # caller that trusts the contract gets a 400. Measured tonight on grafana; filed. + if not f.get("required") and ty != "password": + continue # the controller generates the optional secrets itself + if ty == "path": + p = f"{DRIVE}/{name}" + values[ev] = p + made.append(p) + elif ty in ("secret", "password"): + import secrets as _s + values[ev] = "Drill-" + _s.token_hex(12) + GENERATED.setdefault(name, {})[ev] = values[ev] + elif f.get("default"): + values[ev] = f["default"] + else: + values[ev] = f"drill-{name}" + if made: + guest("mkdir -p " + " ".join(made) + "; ls -ld " + " ".join(made)) + say(f" [1] made the drive paths this app requires: {made}") + extra = [k for k in values if k not in ("DOMAIN", "SUBDOMAIN")] + if extra: + say(f" [1] required fields filled beyond DOMAIN/SUBDOMAIN: {extra}") + return values + + +def deploy(name, sub, extra_values=None): + st = stack(name) + if st.get("deployed"): + say(f" [1] {name} already deployed — reusing") + return True + values = deploy_values(name, sub) + if extra_values: + values.update(extra_values) + code, d = ctl("POST", f"/api/stacks/{name}/deploy", {"values": values}) + say(f" [1] deploy -> {code} {str(d)[:120]}") + if code != "202": + return False + # WAIT FOR `deployed`, NOT FOR `running`. Measured 2026-09-21 on tandoor: docker reported the + # container `healthy` while the controller's own state read `unhealthy` — a gate on `running` + # alone therefore times out on an app that is up. The state is RECORDED rather than required; + # the real gate is the fixture's own `wait_app`, which asks whether the APP answers. + seen = None + for _ in range(90): + time.sleep(5) + st = stack(name) + seen = st.get("state") + # `deployed` alone is NOT enough and `state` alone is NOT right. Measured 2026-09-21: + # tandoor reads `unhealthy` while serving (R-618), so gating on "running" hangs; and romm + # read `deployed=True, state=degraded, pinned_images=None` twenty seconds in, i.e. the + # deploy had not finished writing app.yaml. The PIN is the deploy's own completion mark + # (`runComposeDeploy` writes it), so that is what to wait for. + pins = (st.get("app_config") or {}).get("pinned_images") + if st.get("deployed") and pins and seen in ("running", "unhealthy", "degraded"): + say(f" [1] deployed, controller state={seen}, " + f"pinned={(st.get('app_config') or {}).get('pinned_images')}") + if seen != "running": + say(f" [1] NOTE: the controller's own state is {seen!r}, not 'running' — recorded, " + f"not treated as a failure; the fixture's front-door wait is the real gate") + return True + say(f" [1] never became deployed (last controller state={seen!r})") + return False + + +def backup_now(name): + code, d = ctl("POST", "/api/backup/run") + say(f" [4] „Mentés most\" -> {code} {str(d)[:160]}") + for _ in range(90): + time.sleep(5) + c2, s = ctl("GET", "/api/backup/status") + dd = s.get("data") or {} + if not dd.get("running", False): + say(f" [4] backup idle; last={dd.get('last_run') or dd.get('last_db_dump')}") + return True + say(" [4] backup still running after 7.5 min — carrying on") + return False + + +def drill_bump(app, frm, to, service_hint=None): + """Commit the edge to the DRILL repo. catalog_since set by hand (the drill repo has no gates). + + `frm`/`to` may be comma-separated lists of the SAME length: an app whose own version lives in + two images (adventurelog's backend and frontend) moves both in one edge, while its engine + sidecar stays where it is — `09` §3b Q3's rule is per SERVICE, and an app-half edge must move + every service that carries the app's own version and no others. + """ + comp = f"{DRILL}/templates/{app}/docker-compose.yml" + fy = f"{DRILL}/templates/{app}/.felhom.yml" + s = open(comp).read() + froms = [x.strip() for x in frm.split(",") if x.strip()] + tos = [x.strip() for x in to.split(",") if x.strip()] + if len(froms) != len(tos): + say(f" [5] from/to lists differ in length: {froms} vs {tos}") + return None + for f1, t1 in zip(froms, tos): + if f"image: {f1}" not in s: + say(f" [5] FROM ref not found in compose: {f1}") + return None + s = s.replace(f"image: {f1}", f"image: {t1}") + open(comp, "w").write(s) + f = open(fy).read() + today = datetime.now().strftime("%Y-%m-%d") + f = re.sub(r'^catalog_since:.*$', f'catalog_since: "{today}"', f, count=1, flags=re.M) + open(fy, "w").write(f) + sh(["git", "-C", DRILL, "add", "-A"]) + sh(["git", "-C", DRILL, "commit", "-q", "-m", f"DRILL {app}: {frm} -> {to}"]) + r = sh(["git", "-C", DRILL, "push", "-q", "origin", "main"], timeout=120) + h = sh(["git", "-C", DRILL, "rev-parse", "--short=12", "HEAD"]).stdout.strip() + say(f" [5] drill commit {h}: {app} {frm} -> {to} (push rc={r.returncode})") + return h + + +def sync_rescan(): + ctl("POST", "/api/sync") + time.sleep(2) + ctl("POST", "/api/stacks/rescan") # R-607: ALWAYS before reading a badge + time.sleep(2) + + +def badges(name): + out = {} + for lang, suffix in (("hu", ""), ("en", "?lang=en")): + h = page(f"/apps/{name}{suffix}") + m = re.findall(r']*title="([^"]*)"[^>]*>([^<]*)<', h) + out[lang] = [{"title": a.strip(), "text": b.strip()} for a, b in m][:3] + return out + + +def press_update(name, poll=1.0, cap_s=1800): + code, d = ctl("POST", f"/api/stacks/{name}/update") + say(f" [6] Update -> {code} {str(d)[:220]}") + if code not in ("202", "200"): + return {"accepted": False, "http": code, "refusal": d, "phases": [], "duration_s": 0} + phases, seen, t0 = [], None, time.time() + while time.time() - t0 < cap_s: + st = stack(name) + ph = st.get("update_phase") + if ph != seen: + seen = ph + rec = {"t": round(time.time() - t0, 1), "phase": ph, + "label": st.get("update_phase_label"), "updating": st.get("updating"), + "error": st.get("update_error"), "hold": st.get("hold_reason")} + phases.append(rec) + say(f" +{rec['t']:>6.1f}s phase={ph} label={rec['label']} " + f"err={rec['error']} hold={rec['hold']}") + if not st.get("updating") and ph in ("done", "failed", None) and time.time() - t0 > 3: + break + time.sleep(poll) + st = stack(name) + return {"accepted": True, "http": code, "phases": phases, + "duration_s": round(time.time() - t0, 1), + "final_phase": st.get("update_phase"), "update_error": st.get("update_error"), + "hold_reason": st.get("hold_reason"), "state": st.get("state")} + + +def observables(name): + st = stack(name) + ac = st.get("app_config") or {} + live = guest(f""" +grep -E '^\\s+image:' /opt/docker/stacks/{name}/docker-compose.yml 2>/dev/null | sed 's/^ *//' +echo '---inspect---' +for c in $(docker ps -a --filter label=com.docker.compose.project={name} --format '{{{{.Names}}}}'); do + echo -n "$c "; docker inspect "$c" --format '{{{{.Config.Image}}}} running={{{{.State.Running}}}} restarts={{{{.RestartCount}}}}' +done +""") + a, _, b = live.partition("---inspect---") + return { + "pinned_images": ac.get("pinned_images"), + "installed_images": {k: (v.get("ref") if isinstance(v, dict) else v) + for k, v in (ac.get("installed_images") or {}).items()}, + "catalog_images": st.get("catalog_images"), + "live_compose_image_lines": [x for x in a.strip().splitlines() if x.strip()], + "docker_inspect": [x for x in b.strip().splitlines() if x.strip()], + } + + +def app_logs(name, lines=400): + """The app's own container log, DECODED. The endpoint answers a JSON envelope whose `logs` is + one string with escaped newlines — a scan over the envelope sees a single enormous line and + finds nothing, which reads exactly like "the app printed no migration line" and is not. R-96 + rule 3 in a new place: an absent line is not evidence when the instrument cannot see lines.""" + code, d = ctl("GET", f"/api/stacks/{name}/logs?lines={lines}") + if isinstance(d, dict): + data = d.get("data") + if isinstance(data, dict) and isinstance(data.get("logs"), str): + return data["logs"] + if isinstance(d.get("_raw"), str): + return d["_raw"] + return str(d) + + +def write_verdict(rec, appdir): + os.makedirs(appdir, exist_ok=True) + p = os.path.join(appdir, "verdict.json") + json.dump(rec, open(p, "w"), indent=2, ensure_ascii=False) + say(f" [9] verdict {rec['verdict']} -> {p}") + + +def remove(name): + """Remove through the PRODUCT, never `docker rm` (live-probes rule). The remove endpoint + refuses a running stack — `409 still running` — so the stop is part of the act, not a tidy-up.""" + c1, d1 = ctl("POST", f"/api/stacks/{name}/stop") + say(f" [X] stop -> {c1} {str(d1)[:100]}") + for _ in range(24): + time.sleep(5) + if stack(name).get("state") != "running": + break + code, d = ctl("POST", f"/api/stacks/{name}/remove", + {"remove_hdd_data": True, "remove_backups": True}) + say(f" [X] remove (with drive data) -> {code} {str(d)[:160]}") + if code == "409": + # R-442's fail-closed guard: when the storage subsystem cannot RESOLVE the app's drive + # path, the removal is REFUSED and the app is kept rather than half-deleted. On guest 9202 + # `/api/disks` answers `agent not configured`, so every app deployed with an HDD_PATH hits + # this. The household's other choice — remove the app, KEEP the data — is accepted, and the + # harness takes it, then tidies its own directory by name at teardown. + say(" [X] refused because the drive path cannot be resolved (R-442, fail-closed and right)" + " — removing the app and KEEPING the drive data instead") + code, d = ctl("POST", f"/api/stacks/{name}/remove", + {"remove_hdd_data": False, "remove_backups": True}) + say(f" [X] remove (keeping drive data) -> {code} {str(d)[:160]}") + time.sleep(5) + st = stack(name) + left = guest(f"ls -d /opt/docker/stacks/{name} 2>/dev/null; " + f"docker ps -a --filter label=com.docker.compose.project={name} --format '{{{{.Names}}}}'") + say(f" [X] after remove: deployed={st.get('deployed')} leftovers={left.strip()!r}") + return code + + +def app_env(name, key): + """Read one deploy value the CUSTOMER was given (e.g. the generated admin password) from the + app's own `app.yaml`. This is not seeding — it is how the household logs in; the controller + shows them the same value. Data still goes in through the app's own front door.""" + out = guest(f"grep -E '^\\s*{key}:' /opt/docker/stacks/{name}/app.yaml 2>/dev/null | head -1") + if ":" in out: + return out.split(":", 1)[1].strip().strip('"').strip("'") + return "" + + +def snapshots(name): + """The restorable copies the backups page offers for this app.""" + code, d = ctl("GET", f"/api/backup/snapshots?stack={name}") + data = d.get("data") if isinstance(d, dict) else None + if isinstance(data, dict): + for k in ("snapshots", "items", "restore_points"): + if isinstance(data.get(k), list): + return data[k] + return data if isinstance(data, list) else [] + + +def restore(name, snapshot_id=None, wait_s=1200): + """The household's own way out: the „Visszaállítás a mentésből" button on the backups page. + + A FORM post, not an API call — `POST /backup/restore` with `_csrf`, `stack_name`, + `snapshot_id` — because that is the button the sentence tells them to press. + """ + snaps = snapshots(name) + if snapshot_id is None: + if not snaps: + say(f" [R] no restorable copy offered for {name}") + return {"ok": False, "why": "no snapshot offered", "snapshots": snaps} + first = snaps[0] + snapshot_id = first.get("id") or first.get("snapshot_id") or first.get("short_id") + say(f" [R] restoring {name} from snapshot {snapshot_id!r} (of {len(snaps)} offered)") + sess = open(f"{SC}/sess.txt").read().strip() + csrf = open(f"{SC}/csrf.txt").read().strip() + r = sh(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", HOSTHDR, "-H", f"Cookie: {sess}", + "-X", "POST", + "--data-urlencode", f"_csrf={csrf}", + "--data-urlencode", f"stack_name={name}", + "--data-urlencode", f"snapshot_id={snapshot_id}", + f"{BASE}/backup/restore"], timeout=180) + head = (r.stdout or "").split("\n")[0].strip() + loc = [l for l in (r.stdout or "").split("\n") if l.lower().startswith("location:")] + say(f" [R] POST /backup/restore -> {head} {loc[:1]}") + t0 = time.time() + last = None + while time.time() - t0 < wait_s: + code, d = ctl("GET", "/api/backup/restore-status") + dd = d.get("data") or {} + cur = (dd.get("running"), dd.get("phase") or dd.get("state"), dd.get("message")) + if cur != last: + say(f" +{round(time.time()-t0,1):>6.1f}s restore {cur}") + last = cur + if not dd.get("running", False) and time.time() - t0 > 5: + break + time.sleep(2) + st = stack(name) + say(f" [R] after restore: state={st.get('state')} hold={st.get('hold_reason')!r} " + f"phase={st.get('update_phase')}") + return {"ok": True, "snapshot_id": snapshot_id, "snapshots": snaps, + "http": head, "location": loc[:1], "seconds": round(time.time() - t0, 1), + "state_after": st.get("state"), "hold_after": st.get("hold_reason"), + "observables_after": observables(name)}