admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

night 2026-09-13/14: first "be a customer" rotation (adventurelog) — 7 defects found, 13 rows closed
gates / gates (push) Successful in 18s
Details

Browse Source 
New runbooks/nightly-rotation.md; observations_gate.py reads every section
(R-471); target-selection.md names real paths (R-461); R-93 carries the
fact that drill-r50 is gone. Register: R-473/R-474/R-466/R-471/R-453/R-461
and v0.240.0's R-477/R-478/R-480/R-482/R-484/R-485/R-486 closed; R-481,
R-483, R-487, R-488, R-489 opened. 09 §6.1, 07 §6, CONTEXT, STATUS note.
Evidence: audits/nightly-2026-09-13-adventurelog/, audits/v0240-2026-09-13/.
This commit is contained in:
authentik Default Admin
2026-09-13 19:38:00 +02:00
parent 681c3d6a6d
commit 5e8a82c3c4
37 changed files with 1067 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTEXT.md
+3
View File
@@ -31,6 +31,9 @@ on demo-hp — `documentation/audits/slice4-2026-09-13/`, design `09-update-arch
successful unit restore lifts it. successful unit restore lifts it.
- **Anything that writes a restore point skips an app that is held OR updating** (capture, Tier 2, volume - **Anything that writes a restore point skips an app that is held OR updating** (capture, Tier 2, volume
dump) — the updating half was found live (v0.238.1). dump) — the updating half was found live (v0.238.1).
- **Removal keeps the backups AND the Tier-2 record unless the customer asked for the backups to go**
(v0.240.0, R-486/R-474). A removal that forgets the record makes an intact mirror unrestorable — that
was live until v0.239.0. The first nightly rotation (adventurelog, 2026-09-13) found it.
- **A release reaches the fleet by floor between golden bakes when its MinAgent is declared with the - **A release reaches the fleet by floor between golden bakes when its MinAgent is declared with the
floor** (ruling 2026-09-13, hub v0.112.0, R-472 CLOSED). An undeclared floor above the golden is still floor** (ruling 2026-09-13, hub v0.112.0, R-472 CLOSED). An undeclared floor above the golden is still
held, and the forms refuse it. v0.239.0 arrived on both demo boxes this way in ~15 s. Never hand-deploy held, and the forms refuse it. v0.239.0 arrived on both demo boxes this way in ~15 s. Never hand-deploy
REPORT.md
+33 -32
View File
@@ -1,40 +1,41 @@
# REPORT — two rulings: the floor carries a release without a golden, and every backup counts (2026-09-13) # REPORT — the first "be a customer for the night" run: adventurelog on demo-hp (2026-09-13/14)
*Overwritten each session. Nothing durable lives only here — every finding below has a register row. *Overwritten each session. Every finding below has a register row. The controller report is
The controller report is `felhom-controller/REPORT.md`.* `felhom-controller/REPORT.md`, the catalog's `app-catalog-felhom.eu/REPORT.md`.*
> **Both rulings are shipped and proven live.** Hub v0.112.0 serves a floor above the vouched golden > One app walked end to end as a household would; seven product defects found; a controller release
> when the release's MinAgent is declared with it. Controller v0.239.0 reached both demo boxes by that > (v0.240.0) shipped the same night by the managed floor; two catalog fixes; two docs fixes. The
> floor alone, in 14 s and 15 s, and lets an app update on any backup tier. > morning note is the top block of `STATUS.md`.
## Step 0 — the rules file
`.claude/rules/unprompted-work.md` is in none of the three repos and the brief's text block did not
reach this session (searched the workspace, the transcript and memory). Not created — the operator's
rules cannot be invented byte-identically. The brief's own fences were applied instead.
## What changed here ## What changed here
- **Hub v0.112.0** (`f181efd`, deployed by `2f5d3af` + ArgoCD sync; Synced/Healthy, image 0.112.0). - `documentation/runbooks/nightly-rotation.md` — new: the tick list (standing nine first, blocked on
Declared MinAgent stored beside both floors; `ResolveManagedFloor` uses it above the golden; both R-481; then the catalog); `actualbudget` recorded (no non-browser front door), `adventurelog` ticked.
forms refuse a floor above the golden without it (`floor_needs_min_agent`); Hosts badge and a - `scripts/observations_gate.py` — R-471: every observations section is read (`scripts/CHANGELOG.md`).
`managed floor SERVED … from <source>` log. Vouch path and R-120 gate untouched. - `documentation/runbooks/target-selection.md` — R-461: `/mnt/hdd_1` is the NVMe; `drill-r50` exists
- **Runbooks:** `publish-train-rules.md` rule 1; `RUNBOOK-manual-build.md` §3 "Raise the floor to a on neither box (measured), fence kept with the fact beside it; R-93 carries the fact.
release with no golden" and the §4.2 correction resolved. - `09-update-architecture.md` §6.1, `07-backup-architecture.md` §6, `CONTEXT.md` — v0.240.0 notes.
- **Architecture:** `09-update-architecture.md` §3 decisions 7 and 8, §6 and §6.1; `05-hub-architecture.md` - Register: R-473, R-474, R-466, R-471, R-453, R-461, R-477, R-478, R-480, R-482, R-484, R-485, R-486
§5; `07-backup-architecture.md` §6; capability map "Update is GUARDED" row. closed and compressed; R-481, R-482..R-489 opened (R-482 closed the same night). 212 → 207 open,
- **Register:** R-470, R-472, R-475 closed and compressed; R-477..R-480 opened; R-474 re-confirmed. 177 → 188 closed; 432 156 → 426 502 bytes open, 120 598 → 128 499 closed.
- **Evidence:** `documentation/audits/rulings-r472-r475-2026-09-13/` (live 01–10, red-proofs). - Evidence: `documentation/audits/nightly-2026-09-13-adventurelog/` (01–08), `audits/v0240-2026-09-13/`
- `STATUS.md` items 15 and 16 rewritten as done; the cadence correction line fixed. `CONTEXT.md` updated. (release log, floor, validation, red-proofs incl. R-471).
## Numbering note
§3 already held a decision 6 from this morning, so the two new rulings are numbered 7 and 8 rather than 6 and 7.
## Negative control
An undeclared floor above the golden no longer reaches the hold: the form refuses it first (303
`floor_needs_min_agent`, floor still 0.236.0, hub WARN). The hold itself for that shape is proven by
test C and its red-proof.
## Observations ## Observations
1. **The update's Tier-3 lookup takes its full 15 s bound and logs a misleading size WARN.** FILED: R-477 1. **The brief's rules block was not delivered; the file is absent in all repos.** NOT-A-FINDING: an input gap for the operator, recorded in STATUS "needs you", not a product defect.
2. **A leftover unit from a removed install counts as a reinstall's fresh Tier-1 copy.** FILED: R-478 2. **No scratch guest on demo-hp; the standing nine cannot be throwaways on 9201.** FILED: R-481
3. **A Tier-1 route back restores settings only for a bind-data app.** FILED: R-479 3. **adventurelog ran Django DEBUG=True on the public origin.** FILED: R-482
4. **The card keeps the failure sentence after a successful restore.** FILED: R-480 4. **Photo upload fails 500 from every non-browser client (frontend proxy body-length bug).** FILED: R-483
5. **Removal with `remove_backups` left units and prefs again.** FILED: R-474 5. **PostGIS not recognised as a database.** FILED: R-484
6. **The backup card reads dead paths.** FILED: R-485
7. **Removal with backups kept forgot the Tier-2 record; the restore was refused.** FILED: R-486
8. **A removed app is on neither backup page.** FILED: R-487
9. **The backup test package takes 5½ minutes of real waits.** FILED: R-488
10. **`volumes_removed: null` over removed volumes, five times.** FILED: R-489
STATUS.md
+40
View File
@@ -1,5 +1,45 @@
# STATUS — what works, what's broken, what's next # STATUS — what works, what's broken, what's next
**Updated 2026-09-14 (the night of 13→14 — "be a customer for the night", first run). Written in
the order the rules ask for.**
**Decisions I took, so you can undo them.** (1) The rules file your brief named
(`.claude/rules/unprompted-work.md`) exists in no repo, and its text did not reach me, so I could not
create it; I worked to the fences written in the brief itself. (2) The nine standing apps cannot be
the night's throwaway on the same guest (same name), so the rotation starts after them (R-481).
(3) There is no scratch guest, so the restore was done in place: remove the app, keep its backups,
restore, read the data back. (4) One controller release (0.240.0), two catalog changes, no hub change.
**What I exercised.** AdventureLog as a family: install, sign-up, a Balaton trip with three places,
visits, a packing note, an edit and a delete — all through the app's own API; backup now + second
copy; remove with "delete my data"; restore; the guarded Update; remove with "delete backups".
**What broke.** The second-drive restore was refused after a removal that kept the backups (the box
had forgotten the copy existed). The PostGIS database was not treated as a database, so it had no
proper dump. The backup card said "no backups" over 484 MB. A removed app is invisible on both
backup pages. "Delete backups" left everything behind. The app served Django debug pages to the
internet. Photo upload fails from any non-browser client. A glance fresh install crash-looped.
**What I fixed and proved live.** Controller 0.240.0 (delivered by the floor in 16 s and 18 s):
the forgotten copy, the PostGIS dump, the backup card, "delete backups" now deletes, the stale
failure sentence, the slow off-site check, the old-install copy. Catalog: DEBUG off for AdventureLog;
glance now lands healthy on a fresh install. Docs: the observations gate reads every section; the
target-selection runbook names real paths.
**What I filed.** R-481 (scratch guest, your decision), R-483 (photo upload needs a browser check),
R-487 (removed apps invisible on the backup pages), R-488 (a 5-minute test package), R-489 (the
removal reports null over volumes it removed).
**Register.** Before: 432 156 bytes open / 120 598 closed (212 / 177 rows). After: see the top of
`OPEN-ITEMS.md` — 207 open / 188 closed.
**Needs you.** (a) The rules file text — paste it and I create it in all three repos. **If you do
nothing:** nights run on the brief's fences, as tonight. (b) R-481: how to make a scratch guest (a
second enrolled LXC, or an ISO appliance per night). **If you do nothing:** restores stay in-place and
the nine standing apps are never walked. (c) R-483: open AdventureLog in a browser and add a photo
to a place. **If you do nothing:** we do not know whether households can upload photos. (d) R-479
from the afternoon still waits.
**Updated 2026-09-13 (fourth pass) — you decided both things. New releases reach the demo machines **Updated 2026-09-13 (fourth pass) — you decided both things. New releases reach the demo machines
by themselves again, and an app with any backup can now be updated. Both are live and proven on the by themselves again, and an app with any backup can now be updated. Both are live and proven on the
HP. Nothing needs you.** HP. Nothing needs you.**
documentation/architecture/07-backup-architecture.md
+4 -1
View File
@@ -274,7 +274,10 @@ The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what
v0.239.0, R-475): the first fresh copy in the order Tier 2, Tier 1, Tier 3; with none, it backs up v0.239.0, R-475): the first fresh copy in the order Tier 2, Tier 1, Tier 3; with none, it backs up
first. Design: `09-update-architecture.md` §3 decision 8. **What a Tier-1 route back restores is only first. Design: `09-update-architecture.md` §3 decision 8. **What a Tier-1 route back restores is only
what the unit holds** — for an app whose data is a bind mount that is the definition, not the data what the unit holds** — for an app whose data is a bind mount that is the definition, not the data
(R-479). (R-479). **Removal and the tiers (controller v0.240.0):** removing an app with its backups KEPT keeps
the unit, the Tier-2 mirror AND the Tier-2 record, so „Teljes visszaállítás" still works afterwards
(R-486); „Mentési adatok törlése" deletes the unit, every mirror and the app's backup preferences, and
never touches off-site snapshots (R-474). A removed app is listed on neither backup page (R-487, open).
### 6.1 The four tiers, as configured on the live fleet ### 6.1 The four tiers, as configured on the live fleet
documentation/architecture/09-update-architecture.md
+9
View File
@@ -398,6 +398,15 @@ health wait, 53 s before the hold — and wrote the never-started definition int
The Tier-2 mirror the hold names survived only because Tier 2 is daily. `backup.Manager.isHeld` now also The Tier-2 mirror the hold names survived only because Tier 2 is daily. `backup.Manager.isHeld` now also
answers true for an app a guarded update is moving (`SetUpdatingCheck`). answers true for an app a guarded update is moving (`SetUpdatingCheck`).
**v0.240.0 (2026-09-13, evening) — what the afternoon's proof and the first nightly rotation found, fixed.**
Seven rows: removal with backups kept now keeps the Tier-2 RECORD, so the second-drive restore is not
refused over an intact mirror (R-486, P1 — the disaster the second copy exists for); PostGIS/pgvector/
TimescaleDB images are Postgres, so such apps get their logical dump (R-484); "delete backups" deletes
the unit, the mirror(s) and the prefs (R-474/R-466); the backup card sizes them (R-485); a held
update's sentence leaves the card with the hold (R-480); the Tier-3 lookup is one `snapshots` call
(R-477); a unit older than the app's `deployed_at` does not count (R-478). Delivered by the floor in
16 s / 18 s; every row proven live with a throwaway adventurelog. `audits/v0240-2026-09-13/`.
**Proven live on demo-hp, 2026-09-13**, with a throwaway uptime-kuma and real catalog tag changes (each **Proven live on demo-hp, 2026-09-13**, with a throwaway uptime-kuma and real catalog tag changes (each
reverted in the same phase): A (2.3.2→2.4.0, done after health), B (`backup_max_age: 2m` → backup first), reverted in the same phase): A (2.3.2→2.4.0, done after health), B (`backup_max_age: 2m` → backup first),
E (non-existent tag → pin back, container untouched), F (`alpine:3.20` → held), H (three buttons and the E (non-existent tag → pin back, container untouched), F (`alpine:3.20` → held), H (three buttons and the
documentation/audits/nightly-2026-09-13-adventurelog/01-install.txt
+92
View File
@@ -0,0 +1,92 @@
=== NIGHT 2026-09-13 — adventurelog as a customer — step 1: install through POST /api/stacks/adventurelog/deploy — 2026-09-13T16:27:50Z ===
catalog pin: ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 (catalog 48d2003); controller on demo-hp: gitea.dooplex.hu/admin/felhom-controller:0.239.0
BEFORE — containers/volumes/units named adventurelog:
0
0
ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/adventurelog': No such file or directory
ls: cannot access '/mnt/felhom-drives/hdd_1/backups/secondary/adventurelog': No such file or directory
deploy at 2026-09-13T16:27:52Z (secrets generated locally, never printed/stored) -> HTTP 202 {"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"}
after 190s: state=running updating=False phase=None err='' hold='' deploy_error=None
adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 Up About a minute (healthy)
adventurelog-postgres postgis/postgis:16-3.5-alpine Up About a minute (healthy)
adventurelog-frontend ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 Up About a minute (healthy)
--- controller log for the deploy ---
2026/09/13 16:27:52 auth.go:134: [DEBUG] [web] auth: valid session for POST /api/stacks/adventurelog/deploy
2026/09/13 16:27:52 router.go:81: [DEBUG] [api] POST /api/stacks/adventurelog/deploy (path=/stacks/adventurelog/deploy)
2026/09/13 16:27:52 router.go:403: [INFO] [api] Deploy requested for stack: adventurelog
2026/09/13 16:27:52 router.go:81: [DEBUG] [api] deployStack: name=adventurelog contentLength=165
2026/09/13 16:27:52 deploy.go:249: [DEBUG] Deploy adventurelog: received 3 user values
2026/09/13 16:27:52 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/adventurelog — 4 env vars, 2 encrypted, 2 sensitive fields
2026/09/13 16:27:52 [INFO] [stacks] SaveAppConfig: saved config for adventurelog
2026/09/13 16:27:52 deploy.go:379: [INFO] [stacks] Deploying stack adventurelog with 4 env vars: [SECRET_KEY, DB_PASSWORD, DOMAIN, SUBDOMAIN]
2026/09/13 16:27:52 manager.go:1498: [INFO] [stacks] Deploying stack adventurelog — checking 3 images...
2026/09/13 16:27:52 manager.go:1502: [DEBUG] ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 — not found locally, will pull
2026/09/13 16:27:53 manager.go:1502: [DEBUG] ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 — not found locally, will pull
2026/09/13 16:27:53 lifecycle.go:70: [DEBUG] [integrations] OnStackStart: stack=adventurelog, waiting 5s for state refresh
2026/09/13 16:27:53 manager.go:1379: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/adventurelog)
2026/09/13 16:27:53 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:27:53 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:27:58 lifecycle.go:86: [DEBUG] [integrations] OnStackStart: stack=adventurelog integrationsFound=0
2026/09/13 16:28:03 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:28:03 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:28:13 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:28:13 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:28:23 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:28:23 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:28:33 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:28:33 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:28:43 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:28:43 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:28:53 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:28:53 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:29:03 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:29:03 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:29:13 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:29:13 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:29:21 deploy.go:427: [INFO] [stacks] Stack adventurelog deployed successfully (took 88.3s)
2026/09/13 16:29:21 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/adventurelog — 4 env vars, 2 encrypted, 2 sensitive fields
2026/09/13 16:29:21 [INFO] [stacks] SaveAppConfig: saved config for adventurelog
2026/09/13 16:29:21 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/adventurelog — 4 env vars, 0 encrypted, 2 sensitive fields
2026/09/13 16:29:21 [INFO] [stacks] SaveAppConfig: saved config for adventurelog
2026/09/13 16:29:21 installed.go:408: [INFO] [stacks] installed-images adventurelog: recorded 3 service(s) (adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 (sha256:7c759efab147…), adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 (sha256:edd79220f0de…), adventurelog-postgres=postgis/postgis:16-3.5-alpine (sha256:47e961a569fd…))
2026/09/13 16:29:21 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/adventurelog — 4 env vars, 0 encrypted, 2 sensitive fields
2026/09/13 16:29:21 [INFO] [stacks] SaveAppConfig: saved config for adventurelog
2026/09/13 16:29:21 pin.go:93: [INFO] [stacks] pin adventurelog: adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, adventurelog-postgres=postgis/postgis:16-3.5-alpine
2026/09/13 16:29:22 manager.go:549: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/13 16:29:22 backup.go:460: [DEBUG] groupStacksByDrive: /mnt/sys_drive → [adventurelog, bentopdf, bookstack, docmost, kimai, opengist, privatebin]
2026/09/13 16:29:22 dbdump.go:147: [DEBUG] DiscoverDatabases: docker ps output: c9fff76e09c4 adventurelog adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1
0dcf2da87848 adventurelog-postgres adventurelog postgis/postgis:16-3.5-alpine
02c0d5ba11e2 adventurelog-frontend adventurelog ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1
2026/09/13 16:29:22 dbdump.go:169: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, not a database)
2026/09/13 16:29:22 dbdump.go:169: [DEBUG] DiscoverDatabases: skipping container adventurelog-postgres (image=postgis/postgis:16-3.5-alpine, not a database)
2026/09/13 16:29:22 dbdump.go:169: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, not a database)
2026/09/13 16:29:22 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/13 16:29:22 recovery_unit.go:224: [INFO] [backup] Recovery unit captured for adventurelog → /mnt/sys_drive/felhom-data/backups/primary/adventurelog (images=3, secrets-referenced=2, data_keys=1, portable-carried=2/2, withheld=0)
2026/09/13 16:29:23 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:29:23 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:29:24 manager.go:1379: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/adventurelog)
2026/09/13 16:29:24 manager.go:1459: [INFO] [stacks] Stack adventurelog post-start status:
2026/09/13 16:29:24 manager.go:1462: [INFO] [stacks] adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 running Up 3 seconds (health: starting)
2026/09/13 16:29:24 manager.go:1462: [INFO] [stacks] adventurelog-frontend ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 running Up 9 seconds (healthy)
2026/09/13 16:29:24 manager.go:1462: [INFO] [stacks] adventurelog-postgres postgis/postgis:16-3.5-alpine running Up 9 seconds (healthy)
2026/09/13 16:29:33 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:29:33 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:29:43 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:29:43 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:29:53 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:29:53 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:30:03 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:30:03 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:30:13 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:30:13 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:30:23 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:30:23 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:30:33 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:30:33 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:30:43 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:30:43 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:30:53 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:30:53 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:31:03 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:31:03 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
=== step 1 done 2026-09-13T16:31:04Z ===
documentation/audits/nightly-2026-09-13-adventurelog/02-frontdoor-probe.txt
+5
View File
@@ -0,0 +1,5 @@
GET / -> 200 <!doctype html> <html lang="en" data-theme=""> <head> <meta charset="utf-8" /> <link rel="icon" href="./favicon.png
GET /api/ -> 308 /api
GET /auth/browser/v1/config -> 200 {"status": 200, "data": {"account": {"authentication_method": "username", "is_open_for_signup": true}, "socialaccount": {"providers": []}, "mfa": {"supported_types": ["recovery_codes", "totp"]}}}
GET /api/locations/ -> 308 /api/locations
GET /api/collections/ -> 308 /api/collections
documentation/audits/nightly-2026-09-13-adventurelog/03-customer-data.txt
+50
View File
@@ -0,0 +1,50 @@
=== step 2: be a customer — sign up and put a trip in, through the app's own API — 2026-09-13T16:31:49Z ===
signup (password from a 0600 file, never printed) -> 400 {"status": 400, "errors": [{"message": "This field is required.", "code": "required", "param": "first_name"}, {"message": "This field is required.", "code": "required", "param": "last_name"}]}
session -> 401 {"status": 401, "user": null}
API root -> 200 {"locations": "https://travelnight.enkisfelhom.hu/api/locations/?format=json", "collections": "https://travelnight.enkisfelhom.hu/api/collections/?format=json", "transportations": "https://travelnight.enkisfelhom.hu/api/transportations/?format=json", "notes":
collection -> 308
location Tihany -> 308
location Badacsony -> 308
location Keszthely -> 308
note -> 308
photo sha256: 2465d601e4ccefbbde1ceda68c0f06f8c7835f1b6c9cba3f19748ad460b46b95 bytes: 294
extra place -> 308
READ BACK locations: []
READ BACK collections: []
READ BACK notes: []
image urls: []
=== step 2 done 2026-09-13T16:31:49Z ===
=== step 2 (second run: first_name/last_name added, no trailing slashes — the first run is kept above): be a customer — sign up and put a trip in, through the app's own API — 2026-09-13T16:32:02Z ===
signup (password from a 0600 file, never printed) -> 200 {"status": 200, "meta": {"is_authenticated": true}}
session -> 401 {"status": 401, "user": null}
API root -> 200 {"locations": "https://travelnight.enkisfelhom.hu/api/locations/?format=json", "collections": "https://travelnight.enkisfelhom.hu/api/collections/?format=json", "transportations": "https://travelnight.enkisfelhom.hu/api/transportations/?format=json", "notes":
collection -> 500 ValueError at /api/collections/ Cannot assign "<django.contrib.auth.models.AnonymousUser object at 0x704ff05b7950>": "Collection.user" must be a "CustomUser" instance. Request Method: POST Request URL: https://travelnight.enkisfelhom.hu/api/collections/ Djang
location Tihany -> 401 {"detail": "Authentication credentials were not provided."}
location Badacsony -> 401 {"detail": "Authentication credentials were not provided."}
location Keszthely -> 401 {"detail": "Authentication credentials were not provided."}
note -> 401 {"detail": "Authentication credentials were not provided."}
photo sha256: 2465d601e4ccefbbde1ceda68c0f06f8c7835f1b6c9cba3f19748ad460b46b95 bytes: 294
extra place -> 401 {"detail": "Authentication credentials were not provided."}
READ BACK locations: []
READ BACK collections: []
READ BACK notes: []
image urls: []
=== step 2 done 2026-09-13T16:32:02Z ===
=== step 2 (third run: the session cookie now carried by hand, login through the frontend's form; earlier runs kept above): be a customer — sign up and put a trip in, through the app's own API — 2026-09-13T16:34:26Z ===
signup (password from a 0600 file, never printed) -> 400 {"status": 400, "errors": [{"message": "A user is already registered with this email address.", "code": "email_taken", "param": "email"}, {"message": "A user with that username already exists.", "code": "username_taken", "param": "username"}]}
login through the frontend form -> (302, '/')
session -> 200 {"status": 200, "user": "kovacscsalad"}
API root -> 200 {"locations": "https://travelnight.enkisfelhom.hu/api/locations/?format=json", "collections": "https://travelnight.enkisfelhom.hu/api/collections/?format=json", "transportations": "https://travelnight.enkisfelhom.hu/api/transportations/?format=json", "notes":
collection -> 201 {"id": "cb076e3c-28d3-463e-9ef9-e7ce082cc66c", "description": "Családi nyaralás a Balatonnál, két hét, három falu.", "user": "64dac784-a38b-45d9-b989-45945ef8003c", "name": "Balaton 2026 nyár", "is_public": false, "locations": [], "created_at": "2026-09-13T16:
location Tihany -> 400 {"visits": [{"location": ["This field is required."]}]}
location Badacsony -> 400 {"visits": [{"location": ["This field is required."]}]}
location Keszthely -> 400 {"visits": [{"location": ["This field is required."]}]}
note -> 201 {"id": "df9b8dbc-0671-4fb8-b31b-a9d2e7fefd1f", "user": "64dac784-a38b-45d9-b989-45945ef8003c", "name": "Csomaglista", "content": "napkrém, gumimatrac, a nagy kék hűtőtáska, Bence úszógumija", "date": "2026-07-05", "links": null, "is_public": false, "collection
photo sha256: 2465d601e4ccefbbde1ceda68c0f06f8c7835f1b6c9cba3f19748ad460b46b95 bytes: 294
extra place -> 201 {"id": "00f4eac1-95c8-4fb2-bff3-af63903d76b7", "name": "Siófok (tévedés)", "description": "ide nem mentünk", "rating": null, "tags": null, "location": null, "is_public": false, "collections": [], "created_at": "2026-09-13T16:34:27.689164Z", "updated_at": "2026
delete extra place -> 204
READ BACK locations: []
READ BACK collections: [('Balaton 2026 nyár', 0)]
READ BACK notes: []
image urls: []
=== step 2 done 2026-09-13T16:34:28Z ===
documentation/audits/nightly-2026-09-13-adventurelog/03b-login-debug.txt
+17
View File
@@ -0,0 +1,17 @@
cookies in jar before: []
login -> 200 b'{"status": 200, "data": {"user": {"id": 1, "display": "kovacscsalad", "has_usable_password": true, "email": "kovacs.csal'
Set-Cookie: csrftoken=<redacted>; Max-Age=0; Path=/; HttpOnly; Secure; SameSite=Lax
--- ALL response headers of a login through the frontend proxy ---
200
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: https://travelnight.enkisfelhom.hu
Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
Content-Length: 292
Content-Type: application/json
Date: Sun, 13 Sep 2026 16:32:53 GMT
Expires: Sun, 13 Sep 2026 16:32:53 GMT
Server: gunicorn
Set-Cookie: csrftoken=<redacted>; Max-Age=0; Path=/; HttpOnly; Secure; SameSite=Lax
Vary: Cookie, origin
X-Frame-Options: DENY
Connection: close
documentation/audits/nightly-2026-09-13-adventurelog/03c-login-form.txt
+8
View File
@@ -0,0 +1,8 @@
GET /login -> 2409 bytes
POST /login -> 302 location=/
Set-Cookie: csrftoken=<redacted>; Max-Age=0; Path=/; HttpOnly; Secure; SameSite=Lax
Set-Cookie: sessionid=<redacted>; Domain=.enkisfelhom.hu; Path=/; Expires=Sun, 27 Sep 2026 16:33:52 GMT; HttpOnly; Secure; SameSite=Lax
body:
POST /login?/login -> 404 location=None
Set-Cookie: csrftoken=<redacted>; Max-Age=0; Path=/; HttpOnly; Secure; SameSite=Lax
body: <!doctype html> <html lang="en" data-theme=""> <head> <meta charset="utf-8" /> <link rel="icon" href="./favicon.png" /> <meta name="viewport" content="wi
documentation/audits/nightly-2026-09-13-adventurelog/03d-places-and-readback.txt
+14
View File
@@ -0,0 +1,14 @@
=== step 2, places and visits as their own records; the photo; the read-back — 2026-09-13T16:35:04Z ===
login: (302, '/')
collection id found: True
location Tihany -> 201 {"id": "c1fd347e-4cb8-411a-869e-e94b73c6d05e", "name": "Tihany", "collections": ["cb076e3c-28d3-463e-9ef9-e7ce082cc66c"]}
visit -> 201 {"id": "86ed7b45-6c30-4936-ad85-dc55312c7ab6", "start_date": "2026-07-06T00:00:00Z"}
location Badacsony -> 201 {"id": "3f75d187-570e-40b4-9920-cd44020fee39", "name": "Badacsony", "collections": ["cb076e3c-28d3-463e-9ef9-e7ce082cc66c"]}
visit -> 201 {"id": "2e8a7f5f-9e51-452e-ad8f-d47028ec02d2", "start_date": "2026-07-09T00:00:00Z"}
location Keszthely -> 201 {"id": "40070a88-b302-40bc-a5b2-5fc7bcd5ee83", "name": "Keszthely", "collections": ["cb076e3c-28d3-463e-9ef9-e7ce082cc66c"]}
visit -> 201 {"id": "782f2cef-495f-43b5-bb06-498cdc56ded6", "start_date": "2026-07-12T00:00:00Z"}
photo upload (sha256 2465d601e4cc…) -> 500 {"id": null, "image": null}
edit Badacsony -> 200 {"rating": 5.0}
READ BACK: {"locations": [["Badacsony", 5.0, "Bazaltorgonák, szürkebarát a hegyoldalban. Árvíztűrő tükörfúrógép. (javítva)", 0, [["2026-07-09T00:00:00Z", "2026-07-11T00:00:00Z"]]], ["Keszthely", 4.0, "Festetics-kastély, a gyerekek a Balaton Múzeumot szerették.", 0, [["2026-07-12T00:00:00Z", "2026-07-15T00:00:00Z"]]], ["Tihany", 5.0, "Levendulamező és az apátság, ahonnan az egész tó látszik.", 0, [["2026-07-06T00:00:00Z", "2026-07-08T00:00:00Z"]]]], "collections": [["Balaton 2026 nyár", "Családi nyaralás a Balatonnál, két hét, három falu.", 0, []]], "photo": null}
photo byte-identical to the upload: None
=== done 2026-09-13T16:35:06Z ===
documentation/audits/nightly-2026-09-13-adventurelog/03e-photo-500.txt
+8
View File
@@ -0,0 +1,8 @@
POST /api/images -> 500
{"error":"Internal Server Error"}
upload 'tihany-levendula.png' -> 500 {"id": null, "image": null}
upload 'tihany-levendula-árvíztűrő.png' -> 500 {"id": null, "image": null}
--- curl -F upload with header 'Transfer-Encoding: chunked' ---
500
--- curl -F upload with header 'X-none: 1' ---
500
documentation/audits/nightly-2026-09-13-adventurelog/04-backup.txt
+99
View File
@@ -0,0 +1,99 @@
=== step 3: back it up through the backups page's own endpoints — 2026-09-13T16:36:50Z ===
TRUTH before backup: {"locations": [["Badacsony", 5.0, "Bazaltorgonák, szürkebarát a hegyoldalban. Árvíztűrő tükörfúrógép. (javítva)", 0, [["2026-07-09T00:00:00Z", "2026-07-11T00:00:00Z"]]], ["Keszthely", 4.0, "Festetics-kastély, a gyerekek a Balaton Múzeumot szerették.", 0, [["2026-07-12T00:00:00Z", "2026-07-15T00:00:00Z"]]], ["Tihany", 5.0, "Levendulamező és az apátság, ahonnan az egész tó látszik.", 0, [["2026-07-06T00:00:00Z", "2026-07-08T00:00:00Z"]]]], "collections": [["Balaton 2026 nyár", "Családi nyaralás a Balatonnál, két hét, három falu.", 0, []]], "photo": null}
--- units BEFORE ---
/mnt/sys_drive/felhom-data/backups/primary/adventurelog:
2026-09-13T16:29:22.6226313000 1256 manifest.json
2026-09-13T16:29:22.6226313000 452 compose/app.yaml
2026-09-13T16:29:22.6226313000 2623 compose/.felhom.yml
2026-09-13T16:29:22.6226313000 3540 compose/docker-compose.yml
/mnt/felhom-drives/hdd_1/backups/primary/adventurelog: ABSENT
/mnt/felhom-drives/hdd_1/backups/secondary/adventurelog: ABSENT
--- POST /api/backup/run (Mentések: „Mentés most”) at 2026-09-13T16:36:52Z ---
HTTP 200
{"ok":true,"message":"Mentés elindítva"}
backup run idle after 100s; status keys: ['db_dump', 'enabled', 'running']
--- units AFTER the run ---
/mnt/sys_drive/felhom-data/backups/primary/adventurelog:
2026-09-13T16:38:33.2924199360 1350 manifest.json
2026-09-13T16:38:33.2924199360 452 compose/app.yaml
2026-09-13T16:38:33.2924199360 2623 compose/.felhom.yml
2026-09-13T16:38:33.2914199240 3540 compose/docker-compose.yml
2026-09-13T16:36:59.5722645580 200260608 volume-dumps/adventurelog_adventurelog_postgres_data.tar
2026-09-13T16:36:58.3152490620 47497728 volume-dumps/adventurelog_adventurelog_media.tar
/mnt/felhom-drives/hdd_1/backups/primary/adventurelog: ABSENT
/mnt/felhom-drives/hdd_1/backups/secondary/adventurelog: ABSENT
--- POST /api/backup/tier2 (Mentések: 2. mentés) at 2026-09-13T16:38:35Z ---
HTTP 200
{"ok":true,"message":"2. mentés elindítva"}
tier2 idle after 5s
--- units AFTER tier 2 ---
/mnt/sys_drive/felhom-data/backups/primary/adventurelog:
2026-09-13T16:38:33.2924199360 1350 manifest.json
2026-09-13T16:38:33.2924199360 452 compose/app.yaml
2026-09-13T16:38:33.2924199360 2623 compose/.felhom.yml
2026-09-13T16:38:33.2914199240 3540 compose/docker-compose.yml
2026-09-13T16:36:59.5722645580 200260608 volume-dumps/adventurelog_adventurelog_postgres_data.tar
2026-09-13T16:36:58.3152490620 47497728 volume-dumps/adventurelog_adventurelog_media.tar
/mnt/felhom-drives/hdd_1/backups/primary/adventurelog: ABSENT
/mnt/felhom-drives/hdd_1/backups/secondary/adventurelog:
2026-09-13T16:38:33.2924199360 2623 recovery-unit/compose/.felhom.yml
2026-09-13T16:38:33.2914199240 3540 recovery-unit/compose/docker-compose.yml
2026-09-13T16:38:33.2924199360 452 recovery-unit/compose/app.yaml
2026-09-13T16:36:59.5722645580 200260608 recovery-unit/volume-dumps/adventurelog_adventurelog_postgres_data.tar
2026-09-13T16:36:58.3152490620 47497728 recovery-unit/volume-dumps/adventurelog_adventurelog_media.tar
2026-09-13T16:38:33.2924199360 1350 recovery-unit/manifest.json
2026-09-13T16:38:35.6164485870 1 .felhom-tier2-layout
--- the app's backup card (GET /api/stacks/adventurelog/backup-data) ---
HTTP 200 {"ok":true,"data":{"stack":"adventurelog","backup_paths":[{"path":"/mnt/sys_drive/felhom-data/backups/primary/adventurelog/db-dumps","size_bytes":0,"size_human":"","exists":false},{"path":"/mnt/sys_drive/felhom-data/backups/secondary/adventurelog/rsync","size_bytes":0,"size_human":"","exists":false}],"has_backups":false}}
--- controller log, adventurelog backup lines ---
2026/09/13 16:36:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/13 16:36:52 dbdump.go:147: [DEBUG] DiscoverDatabases: docker ps output: c9fff76e09c4 adventurelog adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1
0dcf2da87848 adventurelog-postgres adventurelog postgis/postgis:16-3.5-alpine
02c0d5ba11e2 adventurelog-frontend adventurelog ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1
2026/09/13 16:36:52 dbdump.go:169: [DEBUG] DiscoverDatabases: skipping container adventurelog (image=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, not a database)
2026/09/13 16:36:52 dbdump.go:169: [DEBUG] DiscoverDatabases: skipping container adventurelog-postgres (image=postgis/postgis:16-3.5-alpine, not a database)
2026/09/13 16:36:52 dbdump.go:169: [DEBUG] DiscoverDatabases: skipping container adventurelog-frontend (image=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, not a database)
2026/09/13 16:36:54 backup.go:903: [INFO] [backup] Stopping adventurelog for safe volume dump
2026/09/13 16:36:54 manager.go:1199: [DEBUG] [stacks] StopStack adventurelog: current state=running deployed=true containers=3
2026/09/13 16:36:54 manager.go:1202: [INFO] [stacks] Stopping stack: adventurelog
2026/09/13 16:36:54 manager.go:1379: [DEBUG] Running: docker compose down (in /opt/docker/stacks/adventurelog)
2026/09/13 16:36:58 manager.go:1211: [INFO] [stacks] Stack adventurelog stopped successfully (took 3.3s)
2026/09/13 16:36:58 backup.go:779: [DEBUG] [backup] Dumping volume adventurelog_adventurelog_media for adventurelog
2026/09/13 16:36:58 backup.go:804: [INFO] [backup] Volume dump: adventurelog/adventurelog_adventurelog_media → 45.3 MB
2026/09/13 16:36:58 backup.go:779: [DEBUG] [backup] Dumping volume adventurelog_adventurelog_postgres_data for adventurelog
2026/09/13 16:37:00 backup.go:804: [INFO] [backup] Volume dump: adventurelog/adventurelog_adventurelog_postgres_data → 191.0 MB
2026/09/13 16:37:00 backup.go:913: [INFO] [backup] Restarting adventurelog after volume dump
2026/09/13 16:37:00 manager.go:1117: [DEBUG] [stacks] StartStack adventurelog: current state=stopped deployed=true
2026/09/13 16:37:00 manager.go:1120: [INFO] [stacks] Starting stack: adventurelog
2026/09/13 16:37:00 manager.go:1127: [DEBUG] [stacks] StartStack adventurelog: prepared 10 env vars for compose
2026/09/13 16:37:00 manager.go:1379: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/adventurelog)
2026/09/13 16:37:02 manager.go:780: [DEBUG] [stacks] restart-policy of down member "adventurelog" = "unless-stopped"
2026/09/13 16:37:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/13 16:37:07 manager.go:1135: [INFO] [stacks] Stack adventurelog started successfully (took 6.3s)
2026/09/13 16:37:07 installed.go:397: [DEBUG] [stacks] installed-images adventurelog: unchanged (3 service(s)) — app.yaml not rewritten
2026/09/13 16:37:10 manager.go:1379: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/adventurelog)
2026/09/13 16:37:10 manager.go:1459: [INFO] [stacks] Stack adventurelog post-start status:
2026/09/13 16:37:10 manager.go:1462: [INFO] [stacks] adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 running Up 3 seconds (health: starting)
2026/09/13 16:37:10 manager.go:1462: [INFO] [stacks] adventurelog-frontend ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 running Up 9 seconds (healthy)
2026/09/13 16:37:10 manager.go:1462: [INFO] [stacks] adventurelog-postgres postgis/postgis:16-3.5-alpine running Up 9 seconds (healthy)
2026/09/13 16:37:22 manager.go:549: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=true composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/13 16:37:22 healthprobe.go:153: [DEBUG] Health probe adventurelog: API GET :8000/api/ → 200 (368ms)
2026/09/13 16:37:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 10s ago, effective interval 5m0s, healthy=true
2026/09/13 16:37:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 20s ago, effective interval 5m0s, healthy=true
2026/09/13 16:37:52 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 30s ago, effective interval 5m0s, healthy=true
2026/09/13 16:38:02 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 40s ago, effective interval 5m0s, healthy=true
2026/09/13 16:38:12 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 50s ago, effective interval 5m0s, healthy=true
2026/09/13 16:38:22 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m0s ago, effective interval 5m0s, healthy=true
2026/09/13 16:38:32 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m10s ago, effective interval 5m0s, healthy=true
2026/09/13 16:38:33 recovery_unit.go:224: [INFO] [backup] Recovery unit captured for adventurelog → /mnt/sys_drive/felhom-data/backups/primary/adventurelog (images=3, secrets-referenced=2, data_keys=1, portable-carried=2/2, withheld=0)
2026/09/13 16:38:35 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/13 16:38:35 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/13 16:38:35 tier2.go:424: [INFO] [backup] Tier 2 copied adventurelog → /mnt/felhom-drives/hdd_1/backups/secondary/adventurelog (236.3 MB, 0 leg(s), 0s)
2026/09/13 16:38:35 notifier.go:234: [INFO] Event pushed: crossdrive_completed (info) — Másodlagos mentés elkészült: adventurelog
2026/09/13 16:38:41 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog/backup-data
2026/09/13 16:38:41 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog/backup-data (path=/stacks/adventurelog/backup-data)
2026/09/13 16:38:41 delete.go:631: [DEBUG] [stacks] GetStackBackupData adventurelog: checked path=/mnt/sys_drive/felhom-data/backups/primary/adventurelog/db-dumps exists=false size=
2026/09/13 16:38:41 delete.go:631: [DEBUG] [stacks] GetStackBackupData adventurelog: checked path=/mnt/sys_drive/felhom-data/backups/secondary/adventurelog/rsync exists=false size=
2026/09/13 16:38:41 delete.go:643: [DEBUG] [stacks] GetStackBackupData adventurelog: hasBackups=false
2026/09/13 16:38:42 healthprobe.go:53: [DEBUG] [stacks] RunHealthProbes: skipping adventurelog — last check 1m20s ago, effective interval 5m0s, healthy=true
=== step 3 done 2026-09-13T16:38:43Z ===
documentation/audits/nightly-2026-09-13-adventurelog/05-restore-in-place.txt
+58
View File
@@ -0,0 +1,58 @@
=== step 4: the disaster and the way back — IN PLACE (no scratch guest, R-481) — 2026-09-13T16:40:06Z ===
--- 4a. the customer removes the app with 'delete my data' (backups kept) ---
HTTP 200
{"ok":true,"message":"Stack adventurelog stop completed"}
HTTP 200
{"ok":true,"data":{"removed":"adventurelog","volumes_removed":null,"hdd_paths_removed":[],"hdd_paths_preserved":[],"hdd_note":"Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni."},"message":"Stack adventurelog removed"}
after removal: state=not_deployed updating=False phase=None err='' hold=''
containers: 0 | volumes: 0
the app answers: 404
--- 4b. the way back: POST /backup/tier2/unit-restore (Mentések → „Teljes visszaállítás”, form stack_name) ---
HTTP/2 302
location: /backups/apps?flash_error=nincs+m%C3%A1sodlagos+f%C3%A1jlm%C3%A1solat+ehhez+az+alkalmaz%C3%A1shoz
restore-status after 5s: {"running": false, "op": "restore", "stack": "gokapi", "started_at": "2026-09-13T15:40:16.222440489Z", "last": {"op": "restore", "stack": "gokapi", "ok": true, "message": "A(z) gokapi: a beállítások visszaálltak — az alkalmazás újraindult. FIGYELEM: ez a mentés csak a beállításokat tartalmazta, adatot nem. Az alkalmazás adatai NEM álltak vissza ebből a mentésből.", "finished_at": "2026-09-13T15:40:24.982262899Z"}}
after restore: state=not_deployed updating=False phase=None err='' hold=''
--- 4c. read the data back through the front door ---
login: (404, '')
READ BACK after restore: {"locations": [], "collections": [], "photo": null}
IDENTICAL to the truth recorded before the backup: False
locations: DIFFERENT -> []
collections: DIFFERENT -> []
photo: same
--- controller log: removal + restore ---
2026/09/13 16:40:09 manager.go:1211: [INFO] [stacks] Stack adventurelog stopped successfully (took 3.6s)
2026/09/13 16:40:09 lifecycle.go:24: [DEBUG] [integrations] OnStackStop: stack=adventurelog integrationsFound=0
2026/09/13 16:40:09 auth.go:134: [DEBUG] [web] auth: valid session for POST /api/stacks/adventurelog/remove
2026/09/13 16:40:09 router.go:81: [DEBUG] [api] POST /api/stacks/adventurelog/remove (path=/stacks/adventurelog/remove)
2026/09/13 16:40:09 router.go:838: [INFO] [api] Remove requested for stack: adventurelog
2026/09/13 16:40:09 router.go:81: [DEBUG] [api] removeStack: name=adventurelog
2026/09/13 16:40:09 router.go:81: [DEBUG] [api] removeStack: name=adventurelog removeHDDData=true removeBackups=false
2026/09/13 16:40:09 delete.go:418: [DEBUG] [stacks] RemoveStack called: name="adventurelog", removeHDDData=true, backupPathsToRemove=0
2026/09/13 16:40:09 delete.go:432: [DEBUG] [stacks] RemoveStack adventurelog: state=stopped, deployed=true, orphaned=false, deploying=false
2026/09/13 16:40:09 delete.go:461: [INFO] Removing deployed stack: adventurelog (removeHDDData=true, hddDeclared=false, backupPaths=0)
2026/09/13 16:40:09 delete.go:473: [DEBUG] [stacks] RemoveStack adventurelog: found 0 HDD mounts from compose file
2026/09/13 16:40:09 manager.go:1379: [DEBUG] Running: docker compose down --volumes (in /opt/docker/stacks/adventurelog)
2026/09/13 16:40:10 delete.go:483: [DEBUG] [stacks] RemoveStack adventurelog: compose down output:
2026/09/13 16:40:10 delete.go:552: [DEBUG] [stacks] RemoveStack adventurelog: processing 0 backup paths for removal (base=/mnt/sys_drive/felhom-data/backups)
2026/09/13 16:40:10 delete.go:577: [DEBUG] [stacks] RemoveStack adventurelog: removing app.yaml at /opt/docker/stacks/adventurelog/app.yaml
2026/09/13 16:40:10 delete.go:584: [INFO] Stack adventurelog removed successfully (took 0.2s)
2026/09/13 16:40:10 manager.go:549: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/13 16:40:10 lifecycle.go:176: [DEBUG] [integrations] OnStackRemove: stack=adventurelog integrationsFound=0
2026/09/13 16:40:10 notifier.go:234: [INFO] Event pushed: app_removed (info) — Alkalmazás eltávolítva: adventurelog
2026/09/13 16:40:14 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:40:14 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:40:16 auth.go:134: [DEBUG] [web] auth: valid session for POST /backup/tier2/unit-restore
2026/09/13 16:40:16 server.go:393: [DEBUG] [web] ServeHTTP: POST /backup/tier2/unit-restore from 172.18.0.3:55178
2026/09/13 16:40:16 handlers.go:1871: [WARN] [web] Tier-2 unit restore refused up front: stack=adventurelog: nincs másodlagos fájlmásolat ehhez az alkalmazáshoz — app NOT stopped
2026/09/13 16:40:16 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
2026/09/13 16:40:16 router.go:81: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
2026/09/13 16:40:21 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
2026/09/13 16:40:21 router.go:81: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
2026/09/13 16:40:21 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:40:21 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
2026/09/13 16:41:22 manager.go:549: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/13 16:43:22 manager.go:549: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/13 16:45:22 manager.go:549: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
=== step 4 done 2026-09-13T16:46:02Z ===
documentation/audits/nightly-2026-09-13-adventurelog/05b-restore-tier1.txt
+54
View File
@@ -0,0 +1,54 @@
=== step 4, second route: the app's own copy („helyi”) after the second-drive restore was refused (R-486) — 2026-09-13T16:46:44Z ===
--- what the Mentések apps page shows for adventurelog (removed, backups kept) ---
--- the restore page mentions adventurelog: False | 'helyi' near it: None
--- POST /backup/restore stack_name=adventurelog snapshot_id=helyi at 2026-09-13T16:47:10Z ---
HTTP/2 302
location: /backups/restore?flash=Vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
restore-status after 25s: {"running": false, "op": "restore", "stack": "adventurelog", "started_at": "2026-09-13T16:47:11.027075525Z", "last": {"op": "restore", "stack": "adventurelog", "ok": true, "message": "A(z) adventurelog: 2 adatkötet visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-09-13T16:47:33.529971164Z"}, "last_recent": true}
after: state=running updating=False phase=None err='' hold=''
adventurelog Up 18 seconds (healthy)
adventurelog-frontend Up 24 seconds (healthy)
adventurelog-postgres Up 24 seconds (healthy)
login: (302, '/')
READ BACK after the Tier-1 restore: {"locations": [["Badacsony", 5.0, "Bazaltorgonák, szürkebarát a hegyoldalban. Árvíztűrő tükörfúrógép. (javítva)", 0, [["2026-07-09T00:00:00Z", "2026-07-11T00:00:00Z"]]], ["Keszthely", 4.0, "Festetics-kastély, a gyerekek a Balaton Múzeumot szerették.", 0, [["2026-07-12T00:00:00Z", "2026-07-15T00:00:00Z"]]], ["Tihany", 5.0, "Levendulamező és az apátság, ahonnan az egész tó látszik.", 0, [["2026-07-06T00:00:00Z", "2026-07-08T00:00:00Z"]]]], "collections": [["Balaton 2026 nyár", "Családi nyaralás a Balatonnál, két hét, három falu.", 0, []]], "photo": null}
IDENTICAL to the truth recorded before the backup: False
locations: DIFFERENT
collections: DIFFERENT
photo: same
--- controller log ---
2026/09/13 16:47:11 auth.go:134: [DEBUG] [web] auth: valid session for POST /backup/restore
2026/09/13 16:47:11 server.go:393: [DEBUG] [web] ServeHTTP: POST /backup/restore from 172.18.0.3:60630
2026/09/13 16:47:11 handlers.go:1480: [DEBUG] [web] backupRestoreHandler: stack=adventurelog snapshot=helyi from 172.18.0.3:60630
2026/09/13 16:47:11 handlers.go:1506: [WARN] [web] Restore requested (async): stack=adventurelog, snapshot=helyi from 172.18.0.3:60630
2026/09/13 16:47:11 restore_unit.go:313: [INFO] [backup] Restoring adventurelog from recovery unit /mnt/sys_drive/felhom-data/backups/primary/adventurelog: images=3, secrets recovered=2/2, data_keys=1
2026/09/13 16:47:11 manager.go:1199: [DEBUG] [stacks] StopStack adventurelog: current state=not_deployed deployed=false containers=0
2026/09/13 16:47:11 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
2026/09/13 16:47:11 router.go:81: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
2026/09/13 16:47:11 restore.go:148: [INFO] [backup] Restoring Docker volume adventurelog_adventurelog_media for adventurelog
2026/09/13 16:47:11 restore.go:177: [DEBUG] [backup] Volume adventurelog_adventurelog_media restored successfully
2026/09/13 16:47:11 restore.go:148: [INFO] [backup] Restoring Docker volume adventurelog_adventurelog_postgres_data for adventurelog
2026/09/13 16:47:13 restore.go:177: [DEBUG] [backup] Volume adventurelog_adventurelog_postgres_data restored successfully
2026/09/13 16:47:13 restore.go:182: [INFO] [backup] Restored 2 Docker volume(s) for adventurelog
2026/09/13 16:47:13 pin.go:93: [INFO] [stacks] pin adventurelog: adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, adventurelog-postgres=postgis/postgis:16-3.5-alpine
2026/09/13 16:47:13 manager.go:1117: [DEBUG] [stacks] StartStack adventurelog: current state=not_deployed deployed=true
2026/09/13 16:47:13 manager.go:1127: [DEBUG] [stacks] StartStack adventurelog: prepared 10 env vars for compose
2026/09/13 16:47:16 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
2026/09/13 16:47:16 router.go:81: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
2026/09/13 16:47:20 installed.go:408: [INFO] [stacks] installed-images adventurelog: recorded 3 service(s) (adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 (sha256:7c759efab147…), adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1 (sha256:edd79220f0de…), adventurelog-postgres=postgis/postgis:16-3.5-alpine (sha256:47e961a569fd…))
2026/09/13 16:47:21 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
2026/09/13 16:47:21 router.go:81: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
2026/09/13 16:47:23 restore.go:212: [DEBUG] [backup] Post-restore health check: adventurelog not yet running, waiting...
2026/09/13 16:47:26 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
2026/09/13 16:47:26 router.go:81: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
2026/09/13 16:47:28 restore.go:212: [DEBUG] [backup] Post-restore health check: adventurelog not yet running, waiting...
2026/09/13 16:47:31 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
2026/09/13 16:47:31 router.go:81: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
2026/09/13 16:47:32 healthprobe.go:153: [DEBUG] Health probe adventurelog: API GET :8000/api/ → 200 (320ms)
2026/09/13 16:47:33 restore.go:207: [DEBUG] [backup] Post-restore health check: adventurelog is running
2026/09/13 16:47:33 restore_unit.go:391: [INFO] [backup] Restore-from-unit completed: adventurelog — 2 volume(s) of 2 listed, 0 database(s) of 0 listed
2026/09/13 16:47:33 handlers.go:1518: [INFO] [web] Restore completed (async): stack=adventurelog in 22.502843771s (volumes 2/2, dbs 0/0)
2026/09/13 16:47:36 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
2026/09/13 16:47:36 router.go:81: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
=== done 2026-09-13T16:47:40Z ===
documentation/audits/nightly-2026-09-13-adventurelog/05c-diff.txt
+11
View File
@@ -0,0 +1,11 @@
== locations truth vs after
TRUTH: ["Badacsony", 5.0, "Bazaltorgonák, szürkebarát a hegyoldalban. Árvíztűrő tükörfúrógép. (javítva)", 0, [["2026-07-09T00:00:00Z", "2026-07-11T00:00:00Z"]]]
AFTER: ["Badacsony", 5.0, "Bazaltorgonák, szürkebarát a hegyoldalban. Árvíztűrő tükörfúrógép. (javítva)", 0, [["2026-07-09T00:00:00Z", "2026-07-11T00:00:00Z"]]]
TRUTH: ["Keszthely", 4.0, "Festetics-kastély, a gyerekek a Balaton Múzeumot szerették.", 0, [["2026-07-12T00:00:00Z", "2026-07-15T00:00:00Z"]]]
AFTER: ["Keszthely", 4.0, "Festetics-kastély, a gyerekek a Balaton Múzeumot szerették.", 0, [["2026-07-12T00:00:00Z", "2026-07-15T00:00:00Z"]]]
TRUTH: ["Tihany", 5.0, "Levendulamező és az apátság, ahonnan az egész tó látszik.", 0, [["2026-07-06T00:00:00Z", "2026-07-08T00:00:00Z"]]]
AFTER: ["Tihany", 5.0, "Levendulamező és az apátság, ahonnan az egész tó látszik.", 0, [["2026-07-06T00:00:00Z", "2026-07-08T00:00:00Z"]]]
== collections truth vs after
TRUTH: ["Balaton 2026 nyár", "Családi nyaralás a Balatonnál, két hét, három falu.", 0, []]
AFTER: ["Balaton 2026 nyár", "Családi nyaralás a Balatonnál, két hét, három falu.", 0, []]
EQUAL after JSON normalisation: True
documentation/audits/nightly-2026-09-13-adventurelog/06-update.txt
+45
View File
@@ -0,0 +1,45 @@
=== step 5: the guarded Update — the catalog offers no newer version (pin == catalog v0.12.1; no promotion tonight), so this is the same-version pass — 2026-09-13T16:48:35Z ===
card before: catalog_images={'adventurelog': 'ghcr.io/seanmorley15/adventurelog-backend:v0.12.1', 'adventurelog-frontend': 'ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1', 'adventurelog-postgres': 'postgis/postgis:16-3.5-alpine'} installed=None
units before:
/mnt/sys_drive/felhom-data/backups/primary/adventurelog:
2026-09-13T16:38:33.2924199360 1350 manifest.json
2026-09-13T16:38:33.2924199360 452 compose/app.yaml
2026-09-13T16:38:33.2924199360 2623 compose/.felhom.yml
2026-09-13T16:38:33.2914199240 3540 compose/docker-compose.yml
2026-09-13T16:36:59.5722645580 200260608 volume-dumps/adventurelog_adventurelog_postgres_data.tar
2026-09-13T16:36:58.3152490620 47497728 volume-dumps/adventurelog_adventurelog_media.tar
/mnt/felhom-drives/hdd_1/backups/primary/adventurelog: ABSENT
/mnt/felhom-drives/hdd_1/backups/secondary/adventurelog:
2026-09-13T16:38:33.2924199360 2623 recovery-unit/compose/.felhom.yml
2026-09-13T16:38:33.2914199240 3540 recovery-unit/compose/docker-compose.yml
2026-09-13T16:38:33.2924199360 452 recovery-unit/compose/app.yaml
2026-09-13T16:36:59.5722645580 200260608 recovery-unit/volume-dumps/adventurelog_adventurelog_postgres_data.tar
2026-09-13T16:36:58.3152490620 47497728 recovery-unit/volume-dumps/adventurelog_adventurelog_media.tar
2026-09-13T16:38:33.2924199360 1350 recovery-unit/manifest.json
2026-09-13T16:38:35.6164485870 1 .felhom-tier2-layout
--- POST /api/stacks/adventurelog/update at 2026-09-13T16:48:37Z ---
HTTP 202
{"ok":true,"data":{"accepted":true,"completed":false},"message":"Frissítés elindult – az állapot a kártyán követhető"}
+ 0s phase safety-dump
+ 3s phase done
end (2026-09-13T16:48:40Z, +3s): state=running updating=False phase=done err='' hold=''
--- controller log ---
2026/09/13 16:48:37 update.go:336: [INFO] [stacks] update adventurelog: accepted — guarded update started
2026/09/13 16:48:37 update.go:763: [INFO] [stacks] update adventurelog: phase checking
2026/09/13 16:48:37 update_guard.go:173: [DEBUG] [backup] update precondition for adventurelog: no Tier-2 copy (nincs másodlagos fájlmásolat ehhez az alkalmazáshoz)
2026/09/13 16:48:37 update.go:450: [INFO] [stacks] update adventurelog: precondition met — Tier 1 (own recovery unit) copy from 2026-09-13T16:38:33Z (10m0s old, limit 24h0m0s)
2026/09/13 16:48:37 update.go:763: [INFO] [stacks] update adventurelog: phase safety-dump
2026/09/13 16:48:37 update_guard.go:403: [INFO] [backup] update safety dump for adventurelog: the app has no database — nothing to copy (no-op)
2026/09/13 16:48:37 update.go:482: [INFO] [stacks] update adventurelog: safety dump done (0 file(s)) []
2026/09/13 16:48:37 update.go:763: [INFO] [stacks] update adventurelog: phase pinning
2026/09/13 16:48:37 pin.go:362: [INFO] [stacks] update adventurelog: pin advanced to the catalog's current definition (adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.1, adventurelog-postgres=postgis/postgis:16-3.5-alpine)
2026/09/13 16:48:37 update.go:763: [INFO] [stacks] update adventurelog: phase pulling
2026/09/13 16:48:38 update.go:763: [INFO] [stacks] update adventurelog: phase starting
2026/09/13 16:48:39 update.go:763: [INFO] [stacks] update adventurelog: phase verifying
2026/09/13 16:48:39 healthprobe.go:153: [DEBUG] Health probe adventurelog: API GET :8000/api/ → 200 (3ms)
2026/09/13 16:48:39 update.go:558: [INFO] [stacks] update adventurelog: healthy after 0s (the app's health check passed)
2026/09/13 16:48:39 installed.go:397: [DEBUG] [stacks] installed-images adventurelog: unchanged (3 service(s)) — app.yaml not rewritten
2026/09/13 16:48:39 update.go:564: [INFO] [stacks] update adventurelog: DONE in 2s
login: (302, '/')
READ BACK after the update IDENTICAL to the truth: True
=== step 5 done 2026-09-13T16:48:43Z ===
documentation/audits/nightly-2026-09-13-adventurelog/07-remove-delete-all.txt
+66
View File
@@ -0,0 +1,66 @@
=== step 6: remove with 'delete my data' AND 'delete backups', then is the drive clean? — 2026-09-13T16:49:23Z ===
BEFORE: 3 containers; 2 volumes
/mnt/sys_drive/felhom-data/backups/primary/adventurelog:
2026-09-13T16:38:33.2924199360 1350 manifest.json
2026-09-13T16:38:33.2924199360 452 compose/app.yaml
2026-09-13T16:38:33.2924199360 2623 compose/.felhom.yml
2026-09-13T16:38:33.2914199240 3540 compose/docker-compose.yml
2026-09-13T16:36:59.5722645580 200260608 volume-dumps/adventurelog_adventurelog_postgres_data.tar
2026-09-13T16:36:58.3152490620 47497728 volume-dumps/adventurelog_adventurelog_media.tar
/mnt/felhom-drives/hdd_1/backups/primary/adventurelog: ABSENT
/mnt/felhom-drives/hdd_1/backups/secondary/adventurelog:
2026-09-13T16:38:33.2924199360 2623 recovery-unit/compose/.felhom.yml
2026-09-13T16:38:33.2914199240 3540 recovery-unit/compose/docker-compose.yml
2026-09-13T16:38:33.2924199360 452 recovery-unit/compose/app.yaml
2026-09-13T16:36:59.5722645580 200260608 recovery-unit/volume-dumps/adventurelog_adventurelog_postgres_data.tar
2026-09-13T16:36:58.3152490620 47497728 recovery-unit/volume-dumps/adventurelog_adventurelog_media.tar
2026-09-13T16:38:33.2924199360 1350 recovery-unit/manifest.json
2026-09-13T16:38:35.6164485870 1 .felhom-tier2-layout
HTTP 200
{"ok":true,"message":"Stack adventurelog stop completed"}
HTTP 200
{"ok":true,"data":{"removed":"adventurelog","volumes_removed":null,"hdd_paths_removed":[],"hdd_paths_preserved":[],"hdd_note":"Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni."},"message":"Stack adventurelog removed"}
AFTER: state=not_deployed updating=False phase=done err='' hold=''
containers: 0 | volumes: 0 | app.yaml: ls: cannot access '/opt/docker/stacks/adventurelog/app.yaml': No such file or directory
--- what is left on the drives (must be nothing) ---
/mnt/sys_drive/felhom-data/backups/primary/adventurelog:
2026-09-13T16:38:33.2924199360 1350 manifest.json
2026-09-13T16:38:33.2924199360 452 compose/app.yaml
2026-09-13T16:38:33.2924199360 2623 compose/.felhom.yml
2026-09-13T16:38:33.2914199240 3540 compose/docker-compose.yml
2026-09-13T16:36:59.5722645580 200260608 volume-dumps/adventurelog_adventurelog_postgres_data.tar
2026-09-13T16:36:58.3152490620 47497728 volume-dumps/adventurelog_adventurelog_media.tar
/mnt/felhom-drives/hdd_1/backups/primary/adventurelog: ABSENT
/mnt/felhom-drives/hdd_1/backups/secondary/adventurelog:
2026-09-13T16:38:33.2924199360 2623 recovery-unit/compose/.felhom.yml
2026-09-13T16:38:33.2914199240 3540 recovery-unit/compose/docker-compose.yml
2026-09-13T16:38:33.2924199360 452 recovery-unit/compose/app.yaml
2026-09-13T16:36:59.5722645580 200260608 recovery-unit/volume-dumps/adventurelog_adventurelog_postgres_data.tar
2026-09-13T16:36:58.3152490620 47497728 recovery-unit/volume-dumps/adventurelog_adventurelog_media.tar
2026-09-13T16:38:33.2924199360 1350 recovery-unit/manifest.json
2026-09-13T16:38:35.6164485870 1 .felhom-tier2-layout
images still present (by design? the pull cache): 3
backup prefs left: {'enabled': False}
--- controller log ---
2026/09/13 16:49:31 manager.go:1211: [INFO] [stacks] Stack adventurelog stopped successfully (took 4.3s)
2026/09/13 16:49:31 lifecycle.go:24: [DEBUG] [integrations] OnStackStop: stack=adventurelog integrationsFound=0
2026/09/13 16:49:31 auth.go:134: [DEBUG] [web] auth: valid session for POST /api/stacks/adventurelog/remove
2026/09/13 16:49:31 router.go:81: [DEBUG] [api] POST /api/stacks/adventurelog/remove (path=/stacks/adventurelog/remove)
2026/09/13 16:49:31 router.go:838: [INFO] [api] Remove requested for stack: adventurelog
2026/09/13 16:49:31 router.go:81: [DEBUG] [api] removeStack: name=adventurelog
2026/09/13 16:49:31 router.go:81: [DEBUG] [api] removeStack: name=adventurelog removeHDDData=true removeBackups=true
2026/09/13 16:49:31 delete.go:418: [DEBUG] [stacks] RemoveStack called: name="adventurelog", removeHDDData=true, backupPathsToRemove=1
2026/09/13 16:49:31 delete.go:432: [DEBUG] [stacks] RemoveStack adventurelog: state=stopped, deployed=true, orphaned=false, deploying=false
2026/09/13 16:49:31 delete.go:461: [INFO] Removing deployed stack: adventurelog (removeHDDData=true, hddDeclared=false, backupPaths=1)
2026/09/13 16:49:31 delete.go:473: [DEBUG] [stacks] RemoveStack adventurelog: found 0 HDD mounts from compose file
2026/09/13 16:49:31 manager.go:1379: [DEBUG] Running: docker compose down --volumes (in /opt/docker/stacks/adventurelog)
2026/09/13 16:49:32 delete.go:483: [DEBUG] [stacks] RemoveStack adventurelog: compose down output:
2026/09/13 16:49:32 delete.go:552: [DEBUG] [stacks] RemoveStack adventurelog: processing 1 backup paths for removal (base=/mnt/sys_drive/felhom-data/backups)
2026/09/13 16:49:32 delete.go:577: [DEBUG] [stacks] RemoveStack adventurelog: removing app.yaml at /opt/docker/stacks/adventurelog/app.yaml
2026/09/13 16:49:32 delete.go:584: [INFO] Stack adventurelog removed successfully (took 0.2s)
2026/09/13 16:49:32 manager.go:549: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
2026/09/13 16:49:32 lifecycle.go:176: [DEBUG] [integrations] OnStackRemove: stack=adventurelog integrationsFound=0
2026/09/13 16:49:32 notifier.go:234: [INFO] Event pushed: app_removed (info) — Alkalmazás eltávolítva: adventurelog
2026/09/13 16:49:36 auth.go:134: [DEBUG] [web] auth: valid session for GET /api/stacks/adventurelog
2026/09/13 16:49:36 router.go:81: [DEBUG] [api] GET /api/stacks/adventurelog (path=/stacks/adventurelog)
=== step 6 done 2026-09-13T16:49:43Z ===
documentation/audits/nightly-2026-09-13-adventurelog/08-R473-glance-fresh-install.txt
+17
View File
@@ -0,0 +1,17 @@
=== R-473 live: a fresh glance install must land healthy — 2026-09-13T17:24:12Z ===
sync: HTTP 200 {"ok":true,"data":{"ok":true,"updated":["adventurelog","glance"],"message":"Sablonok frissítve — frissítve: adventurelog, glance"},"message":"Sablonok frissítve — frissítve: adventurelog, glance"}
cache carries the seed: 1
HTTP 202 {"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"}
after 21s: state=running updating=False phase=None err='' hold=''
container: glanceapp/glance:v0.8.5 Up 20 seconds (healthy) restarts=20 seconds ago
restart count: 0
glance log head: [felhom] first boot — seeding a default glance.yml
2026/09/13 19:24:16 Starting server on :8080 (base-url: "", assets-path: "")
config in the volume: 31 /app/config/glance.yml
pages:
- name: Kezdőlap
front door (Host glancenight.enkisfelhom.hu): 200
--- teardown: remove the throwaway with data + backups ---
HTTP 200 {"ok":true,"data":{"removed":"glance","volumes_removed":null,"hdd_paths_removed":[],"hdd_paths_preserved":[],"hdd_note":"Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni."},"message":"Stack glance removed"}
left: 0 containers, 0 volumes; unit: /mnt/sys_drive/felhom-data/backups/primary/glance
=== done 2026-09-13T17:24:46Z ===
documentation/audits/v0240-2026-09-13/00-release.log
+9
View File
@@ -0,0 +1,9 @@
unstaged-left: []
OK [felhom-controller]: 166 cited paths — exact 151, suffix 10, ambiguous 0, cross-repo 5, FAILED 0 (siblings searched: app-catalog-felhom.eu, felhom-agent, felhom.eu)
all controller gates OK
pre-push [felhom-controller]: gates OK - push proceeding.
HEAD=bdcbd50b42bdf7c19942fb2a13de481cfa7f9ff8 origin=bdcbd50b42bdf7c19942fb2a13de481cfa7f9ff8 porcelain=[]
/dev/sda1 9.1T 3.1T 5.5T 37% /mnt/5_hdd
/dev/sdb1 445G 215G 208G 51% /
build-rc=0
0.240.0: digest: sha256:f073cb622669331a69b9bfe02442e1a2b3e12bbc64396ba7e8992ce16b5fcced size: 856
documentation/audits/v0240-2026-09-13/09-floor-0.240.0.txt
+54
View File
@@ -0,0 +1,54 @@
=== FLOOR RAISE: 0.240.0 with min_agent 0.129.0 (the delivery IS the release) ===
T0 POST at 2026-09-13T17:28:00Z
HTTP/1.1 303 See Other
Location: /configuration?flash=floor_set
DB floor + declared: id="global-floor-input" name="min_controller_version" value="0.240.0" | name="min_agent" value="0.129.0" placeholder="MinAgent from |
demo-hp on 0.240.0 at 2026-09-13T17:28:17Z (+16s): gitea.dooplex.hu/admin/felhom-controller:0.240.0 Up 7 seconds (healthy)
demo-felhom on 0.240.0 at 2026-09-13T17:28:18Z (+18s): gitea.dooplex.hu/admin/felhom-controller:0.240.0 Up 10 seconds (healthy)
--- hub log since T0 (managed floor) ---
2026/09/13 19:28:01 [INFO] Global controller-version floor set to "0.240.0" (declared MinAgent "0.129.0")
2026/09/13 19:28:03 [INFO] managed floor SERVED for demo-felhom: floor 0.240.0, agent requirement "0.129.0" from declared (golden 0.236.0)
2026/09/13 19:28:04 [INFO] managed floor SERVED for demo-hp: floor 0.240.0, agent requirement "0.129.0" from declared (golden 0.236.0)
--- demo-hp: controller log (SetFloor / self-update / version) ---
2026/09/13 17:28:10 updater.go:96: [DEBUG] [selfupdate] VerifyStartup: checking update state in /opt/docker/felhom-controller/data
2026/09/13 17:28:10 updater.go:96: [DEBUG] [selfupdate] VerifyStartup: pending update found — target=0.240.0 previous=0.239.0
2026/09/13 17:28:10 updater.go:809: [INFO] [selfupdate] Post-update startup: update successful (0.239.0 → 0.240.0)
2026/09/13 17:28:10 main.go:655: [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30)
2026/09/13 17:28:10 offsiteapply.go:151: [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/09/13 17:28:10 scheduler.go:102: [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s)
2026/09/13 17:28:10 scheduler.go:67: [DEBUG] [scheduler] periodic job registered: name="selfupdate-check" interval=6h0m0s totalJobs=18
2026/09/13 17:28:11 client.go:67: [DEBUG] [agentapi] agent version seen: 0.130.0 (was "")
2026/09/13 17:28:15 builder.go:40: [DEBUG] [report] BuildReport: starting — version=0.240.0, storagePaths=1
2026/09/13 17:28:16 updater.go:96: [DEBUG] [selfupdate] SetFloor: floor "" → "0.240.0"
2026/09/13 17:28:16 updater.go:96: [DEBUG] [selfupdate] maybeAutoUpdate: current 0.240.0 >= floor 0.240.0 — no action
2026/09/13 17:28:20 offsiteapply.go:151: [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.240.0 (we are 0.240.0), no managed update running
--- demo-hp: bootstrap service journal ---
Sep 13 17:28:09 demo-hp systemd[1]: felhom-controller-bootstrap.service: Deactivated successfully.
Sep 13 17:28:09 demo-hp systemd[1]: Stopped felhom-controller-bootstrap.service - Felhom controller bootstrap (deploy the baked controller from the agent-populated config mount).
Sep 13 17:28:09 demo-hp systemd[1]: Stopping felhom-controller-bootstrap.service - Felhom controller bootstrap (deploy the baked controller from the agent-populated config mount)...
Sep 13 17:28:09 demo-hp systemd[1]: Starting felhom-controller-bootstrap.service - Felhom controller bootstrap (deploy the baked controller from the agent-populated config mount)...
Sep 13 17:28:09 demo-hp felhom-controller-bootstrap.sh[2056435]: [ctrl-bootstrap] deploying gitea.dooplex.hu/admin/felhom-controller:0.240.0 from /etc/felhom-bootstrap/bootstrap.json (hostname=demo-hp)
Sep 13 17:28:09 demo-hp felhom-controller-bootstrap.sh[2056503]: 48fe33a5ee1010db5156d9cb29fa6ea466cbc4ec4afd845743a01ac6cd69c302
Sep 13 17:28:09 demo-hp felhom-controller-bootstrap.sh[2056435]: [ctrl-bootstrap] controller started
Sep 13 17:28:09 demo-hp systemd[1]: Finished felhom-controller-bootstrap.service - Felhom controller bootstrap (deploy the baked controller from the agent-populated config mount).
--- demo-felhom: controller log (SetFloor / self-update / version) ---
2026/09/13 17:28:09 [INFO] [selfupdate] Post-update startup: update successful (0.239.0 → 0.240.0)
2026/09/13 17:28:09 [INFO] Self-update enabled (check every 6h, auto-update: false, auto-update time: 04:30)
2026/09/13 17:28:09 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
2026/09/13 17:28:09 [INFO] [scheduler] Registered periodic job: selfupdate-check (every 6h0m0s)
2026/09/13 17:28:19 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.240.0 (we are 0.240.0), no managed update running
--- demo-felhom: bootstrap service journal ---
Sep 13 17:28:07 demo-felhom systemd[1]: felhom-controller-bootstrap.service: Deactivated successfully.
Sep 13 17:28:07 demo-felhom systemd[1]: Stopped felhom-controller-bootstrap.service - Felhom controller bootstrap (deploy the baked controller from the agent-populated config mount).
Sep 13 17:28:07 demo-felhom systemd[1]: Stopping felhom-controller-bootstrap.service - Felhom controller bootstrap (deploy the baked controller from the agent-populated config mount)...
Sep 13 17:28:07 demo-felhom systemd[1]: Starting felhom-controller-bootstrap.service - Felhom controller bootstrap (deploy the baked controller from the agent-populated config mount)...
Sep 13 17:28:07 demo-felhom felhom-controller-bootstrap.sh[3523538]: [ctrl-bootstrap] deploying gitea.dooplex.hu/admin/felhom-controller:0.240.0 from /etc/felhom-bootstrap/bootstrap.json (hostname=demo-felhom)
Sep 13 17:28:07 demo-felhom felhom-controller-bootstrap.sh[3523587]: 47d56d3e7ee979f553f76d9f675780a17a6d90e3b4d0fed22ddf424dd4c60c6c
Sep 13 17:28:08 demo-felhom felhom-controller-bootstrap.sh[3523538]: [ctrl-bootstrap] controller started
Sep 13 17:28:08 demo-felhom systemd[1]: Finished felhom-controller-bootstrap.service - Felhom controller bootstrap (deploy the baked controller from the agent-populated config mount).
SUMMARY T0=2026-09-13T17:28:00Z {'demo-hp': ('2026-09-13T17:28:17Z', 16), 'demo-felhom': ('2026-09-13T17:28:18Z', 18)} not done: []
documentation/audits/v0240-2026-09-13/10-v0240-validation.txt
+62
View File
@@ -0,0 +1,62 @@
=== v0.240.0 live validation on demo-hp — 2026-09-13T17:28:41Z ===
controller: gitea.dooplex.hu/admin/felhom-controller:0.240.0
catalog-cache template carries DEBUG=False: 1
--- deploy ---
HTTP 202 {"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"}
healthy: True
--- R-482: a CSRF failure page must NOT be a debug page ---
login with a bogus CSRF -> 403 | mentions 'DEBUG = True': False | length: 300
--- sign up + a place, then backup now (R-484: a db-dump must appear) ---
signup: 200
login: (302, '/')
location: 201
HTTP 200 {"ok":true,"message":"Mentés elindítva"}
backup idle after 100 s
HTTP 200 {"ok":true,"message":"2. mentés elindítva"}
/mnt/sys_drive/felhom-data/backups/primary/adventurelog:
2026-09-13T17:30:42.0264751580 8170290 db-dumps/adventurelog-postgres.sql
2026-09-13T17:32:19.5796963680 1385 manifest.json
2026-09-13T17:32:19.5796963680 452 compose/app.yaml
2026-09-13T17:32:19.5786963550 2623 compose/.felhom.yml
2026-09-13T17:32:19.5786963550 3844 compose/docker-compose.yml
2026-09-13T17:30:47.5035437220 199777280 volume-dumps/adventurelog_adventurelog_postgres_data.tar
2026-09-13T17:30:46.3705295390 47497728 volume-dumps/adventurelog_adventurelog_media.tar
/mnt/felhom-drives/hdd_1/backups/primary/adventurelog: ABSENT
/mnt/felhom-drives/hdd_1/backups/secondary/adventurelog:
2026-09-13T17:30:42.0264751580 8170290 recovery-unit/db-dumps/adventurelog-postgres.sql
2026-09-13T17:32:19.5786963550 2623 recovery-unit/compose/.felhom.yml
2026-09-13T17:32:19.5786963550 3844 recovery-unit/compose/docker-compose.yml
2026-09-13T17:32:19.5796963680 452 recovery-unit/compose/app.yaml
2026-09-13T17:30:47.5035437220 199777280 recovery-unit/volume-dumps/adventurelog_adventurelog_postgres_data.tar
2026-09-13T17:30:46.3705295390 47497728 recovery-unit/volume-dumps/adventurelog_adventurelog_media.tar
2026-09-13T17:32:19.5796963680 1385 recovery-unit/manifest.json
2026-09-13T17:32:22.5337333470 1 .felhom-tier2-layout
DB dump discovered: 2026/09/13 17:28:47 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned adventurelog-postgres: errors=0 warnings=2 issues=2 (took 27ms)
2026/09/13 17:28:56 installed.go:408: [INFO] [stacks] installed-images adventurelog: recorded 3 service(s) (adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1 (sha256:7c759efab147…), adv
2026/09/13 17:28:56 pin.go:93: [INFO] [stacks] pin adventurelog: adventurelog=ghcr.io/seanmorley15/adventurelog-backend:v0.12.1, adventurelog-frontend=ghcr.io/seanmorley15/adventurelog-frontend:v0.12.
--- R-485: the card ---
HTTP 200 {"ok":true,"data":{"stack":"adventurelog","backup_paths":[{"path":"/mnt/sys_drive/felhom-data/backups/primary/adventurelog","size_bytes":255469986,"size_human":"244M","exists":true},{"path":"/mnt/felhom-drives/hdd_1/backups/secondary/adventurelog","size_bytes":255474083,"size_human":"244M","exists":true}],"has_backups":true}}
truth: [('Tihany', 5.0, 'Levendulamező.')]
--- R-486: remove with backups KEPT, then the second-drive restore must WORK ---
HTTP 200 {"ok":true,"data":{"removed":"adventurelog","volumes_removed":null,"hdd_paths_removed":[],"hdd_paths_preserved":[],"hdd_note":"Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni."},"message":"Stack adventurelog removed"}
volumes after removal: 0
cross_drive record kept: True
HTTP/2 302
location: /backups/apps?flash=Teljes+vissza%C3%A1ll%C3%ADt%C3%A1s+elindult+%E2%80%94+az+%C3%A1llapot+itt+friss%C3%BCl.
restore-status: {"running": false, "op": "tier2-unit-restore", "stack": "adventurelog", "started_at": "2026-09-13T17:32:40.44475981Z", "last": {"op": "tier2-unit-restore", "stack": "adventurelog", "ok": true, "message": "A(z) adventurelog: 2 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult. A visszaállítás forrása a második meghajtón lévő másolat volt (2026-09-13 19:32).", "finished_at": "2026-0
healthy: True
login: (302, '/')
READ BACK after the Tier-2 restore identical: True [('Tihany', 5.0, 'Levendulamező.')]
--- R-480: card after a successful restore (no update failure here, so update_error must be empty) ---
state=running updating=False phase=None err='' hold=''
--- R-474: remove with delete my data AND delete backups → clean ---
HTTP 200 {"ok":true,"data":{"removed":"adventurelog","volumes_removed":null,"hdd_paths_removed":[],"hdd_paths_preserved":[],"hdd_note":"Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni.","backup_paths_removed":["/mnt/sys_drive/felhom-data/backups/primary/adventurelog (244M)","/mnt/felhom-drives/hdd_1/backups/secondary/adventurelog (243.6 MB)"]},"message":"Stack adventurelog removed"}
/mnt/sys_drive/felhom-data/backups/primary/adventurelog: ABSENT
/mnt/felhom-drives/hdd_1/backups/primary/adventurelog: ABSENT
/mnt/felhom-drives/hdd_1/backups/secondary/adventurelog: ABSENT
prefs left: None
--- controller log (removal + mirror deletion) ---
2026/09/13 17:33:12 delete.go:569: [INFO] Removed backup data: /mnt/sys_drive/felhom-data/backups/primary/adventurelog (244M)
2026/09/13 17:33:12 r474_remove_mirrors.go:91: [INFO] [backup] remove adventurelog: Tier-2 mirror deleted: /mnt/felhom-drives/hdd_1/backups/secondary/adventurelog (243.6 MB)
=== done 2026-09-13T17:33:18Z ===
documentation/audits/v0240-2026-09-13/gate-go-test.txt
+37
View File
@@ -0,0 +1,37 @@
ok gitea.dooplex.hu/admin/felhom-controller/cmd/controller 10.379s
ok gitea.dooplex.hu/admin/felhom-controller/internal/agentapi 0.191s
ok gitea.dooplex.hu/admin/felhom-controller/internal/api 0.315s
ok gitea.dooplex.hu/admin/felhom-controller/internal/appbackup 0.023s
ok gitea.dooplex.hu/admin/felhom-controller/internal/appexport 0.974s
? gitea.dooplex.hu/admin/felhom-controller/internal/assets [no test files]
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 318.690s
ok gitea.dooplex.hu/admin/felhom-controller/internal/backupwindow 0.004s
ok gitea.dooplex.hu/admin/felhom-controller/internal/bootrecon 0.008s
ok gitea.dooplex.hu/admin/felhom-controller/internal/bootstrap 14.024s
ok gitea.dooplex.hu/admin/felhom-controller/internal/channelhealth 0.005s
? gitea.dooplex.hu/admin/felhom-controller/internal/cloudflare [no test files]
ok gitea.dooplex.hu/admin/felhom-controller/internal/config 0.005s
? gitea.dooplex.hu/admin/felhom-controller/internal/crypto [no test files]
ok gitea.dooplex.hu/admin/felhom-controller/internal/fillwatch 0.007s
ok gitea.dooplex.hu/admin/felhom-controller/internal/infra 0.010s
? gitea.dooplex.hu/admin/felhom-controller/internal/integrations [no test files]
? gitea.dooplex.hu/admin/felhom-controller/internal/logx [no test files]
ok gitea.dooplex.hu/admin/felhom-controller/internal/mailrelay 0.016s
ok gitea.dooplex.hu/admin/felhom-controller/internal/metrics 0.008s
ok gitea.dooplex.hu/admin/felhom-controller/internal/monitor 0.005s
ok gitea.dooplex.hu/admin/felhom-controller/internal/notify 0.786s
ok gitea.dooplex.hu/admin/felhom-controller/internal/offsiteapply 0.010s
ok gitea.dooplex.hu/admin/felhom-controller/internal/quiesce 0.081s
? gitea.dooplex.hu/admin/felhom-controller/internal/recovery [no test files]
ok gitea.dooplex.hu/admin/felhom-controller/internal/report 2.583s
ok gitea.dooplex.hu/admin/felhom-controller/internal/scheduler 0.155s
? gitea.dooplex.hu/admin/felhom-controller/internal/selftest [no test files]
ok gitea.dooplex.hu/admin/felhom-controller/internal/selfupdate 0.027s
ok gitea.dooplex.hu/admin/felhom-controller/internal/settings 0.011s
ok gitea.dooplex.hu/admin/felhom-controller/internal/setup 0.010s
ok gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.910s
ok gitea.dooplex.hu/admin/felhom-controller/internal/sync 0.018s
ok gitea.dooplex.hu/admin/felhom-controller/internal/system 0.096s
? gitea.dooplex.hu/admin/felhom-controller/internal/util [no test files]
ok gitea.dooplex.hu/admin/felhom-controller/internal/web 29.790s
? gitea.dooplex.hu/admin/felhom-controller/scripts [no test files]
documentation/audits/v0240-2026-09-13/rp-R471.txt
+14
View File
@@ -0,0 +1,14 @@
=== R-471 red-proof — the R-419 decoy shape: a second '## Observations' block appended to a real REPORT.md, holding one unmarked item ===
--- OLD parser (git HEAD 681c3d6, first section only) ---
OK 3. FILED R-479
OK 4. FILED R-480
OK 5. FILED R-474
observations gate OK — every observation is either filed or explicitly declared
old-rc=0 (0 = the unmarked item was NOT seen — LIVE HOLE)
--- NEW parser (every section) ---
item 1: A real finding. It carries no FILED marker and no NOT-A-FINDING marker at all.
neither `FILED: R-NNN` nor `NOT-A-FINDING: <reason>`
An observation that lives only in REPORT.md has a lifetime of ONE SESSION — this file is overwritten every time. That is how the cooldown-grain finding was lost on 2026-08-23 and had to be re-derived the next day.
Fix: add `FILED: R-NNN` naming the row you opened for it, or `NOT-A-FINDING: <why this is not worth a row>`. Opening the row is the default; declaring is the exception and needs its reason stated.
new-rc=1
documentation/audits/v0240-2026-09-13/rp-v240-R474-any-dir-removed.txt
+21
View File
@@ -0,0 +1,21 @@
MUTATION in /mnt/5_hdd/felhom.eu/git/felhom-controller/controller/internal/backup/r474_remove_mirrors.go:
- if !allowed[d] {
+ if false {
=== RUN TestR474_OnlyTheAppsOwnMirrorIsRemoved
[INFO] [settings] No settings.json found, using defaults
[INFO] [settings] Settings saved
[INFO] [settings] Added storage path: /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002
[INFO] [settings] Settings saved
[INFO] [backup] remove gokapi: Tier-2 mirror deleted: /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/gokapi (2 B)
[INFO] [backup] remove gokapi: Tier-2 mirror deleted: /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/kimai (2 B)
[INFO] [backup] remove gokapi: Tier-2 mirror deleted: /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/_shares (2 B)
[INFO] [backup] remove gokapi: Tier-2 mirror deleted: /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/kimai (0 B)
r477_r474_test.go:78: exactly the app's own mirror must be removed, got [/tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/gokapi (2 B) /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/kimai (2 B) /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/_shares (2 B) /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/kimai (0 B)]
r477_r474_test.go:85: /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/kimai must survive: stat /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/kimai: no such file or directory
r477_r474_test.go:85: /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/_shares must survive: stat /tmp/TestR474_OnlyTheAppsOwnMirrorIsRemoved281790753/002/backups/secondary/_shares: no such file or directory
--- FAIL: TestR474_OnlyTheAppsOwnMirrorIsRemoved (0.01s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.012s
FAIL
exit=1
documentation/audits/v0240-2026-09-13/rp-v240-R477-lookup-runs-stats.txt
+20
View File
@@ -0,0 +1,20 @@
MUTATION in /mnt/5_hdd/felhom.eu/git/felhom-controller/controller/internal/backup/update_guard.go:
- times = m.OffsiteSnapshotTimes
+ times = func(c context.Context) (map[string]time.Time, error) {
inv, err := m.OffsiteInventoryList(c)
out := map[string]time.Time{}
for _, a := range inv.Apps {
out[a.App] = a.LatestAt
}
return out, err
}
=== RUN TestR477_TheUpdateOffsiteLookupRunsNoStats
[INFO] [settings] No settings.json found, using defaults
[INFO] [settings] Settings saved
r477_r474_test.go:43: the update lookup must be one snapshots call and no stats; calls=[-r sftp:felhom@nas.local:/srv/repo -o sftp.command=ssh felhom@nas.local -p 22 -oBatchMode=yes -oConnectTimeout=10 -oStrictHostKeyChecking=yes -oUserKnownHostsFile=/tmp/TestR477_TheUpdateOffsiteLookupRunsNoStats2946017940/001/offbox/known_hosts -i /tmp/TestR477_TheUpdateOffsiteLookupRunsNoStats2946017940/001/offbox/ssh_key -s sftp snapshots --json -r sftp:felhom@nas.local:/srv/repo -o sftp.command=ssh felhom@nas.local -p 22 -oBatchMode=yes -oConnectTimeout=10 -oStrictHostKeyChecking=yes -oUserKnownHostsFile=/tmp/TestR477_TheUpdateOffsiteLookupRunsNoStats2946017940/001/offbox/known_hosts -i /tmp/TestR477_TheUpdateOffsiteLookupRunsNoStats2946017940/001/offbox/ssh_key -s sftp stats a1 --json -r sftp:felhom@nas.local:/srv/repo -o sftp.command=ssh felhom@nas.local -p 22 -oBatchMode=yes -oConnectTimeout=10 -oStrictHostKeyChecking=yes -oUserKnownHostsFile=/tmp/TestR477_TheUpdateOffsiteLookupRunsNoStats2946017940/001/offbox/known_hosts -i /tmp/TestR477_TheUpdateOffsiteLookupRunsNoStats2946017940/001/offbox/ssh_key -s sftp stats b2 --json]
--- FAIL: TestR477_TheUpdateOffsiteLookupRunsNoStats (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.007s
FAIL
exit=1
documentation/audits/v0240-2026-09-13/rp-v240-R478-old-install-copy-counts.txt
+11
View File
@@ -0,0 +1,11 @@
MUTATION in /mnt/5_hdd/felhom.eu/git/felhom-controller/controller/internal/stacks/update.go:
- if !deployedAt.IsZero() && p.ProvenAt.Before(deployedAt) {
+ if false {
=== RUN TestR478_ACopyOlderThanThisInstallDoesNotCount
r480_r478_test.go:87: a 2-hour-old unit under a 10-minute-old install belongs to the previous install — back up first; calls=[CanBackUp RestorePoints SafetyDump]
--- FAIL: TestR478_ACopyOlderThanThisInstallDoesNotCount (0.19s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.191s
FAIL
exit=1
documentation/audits/v0240-2026-09-13/rp-v240-R480-sentence-never-cleared.txt
+17
View File
@@ -0,0 +1,17 @@
MUTATION in /mnt/5_hdd/felhom.eu/git/felhom-controller/controller/internal/stacks/update.go:
- if st.updateHeld && !st.Updating && (!st.Deployed || (g != nil && !held)) {
+ if st.updateHeld && false && !st.Updating && (!st.Deployed || (g != nil && !held)) {
=== RUN TestR480_TheFailureSentenceGoesWhenItsHoldIsLifted
r480_r478_test.go:32: GetStack after the hold is lifted: phase="failed" err="HELD-SENTENCE" — the card would say a running app is stopped
r480_r478_test.go:36: GetStacks after the hold is lifted still carries "HELD-SENTENCE"
--- FAIL: TestR480_TheFailureSentenceGoesWhenItsHoldIsLifted (0.01s)
=== RUN TestR480_ARemovedAppShowsNoUpdateOutcome
r480_r478_test.go:47: a removed app must not carry the held update's sentence, got "HELD-SENTENCE"
--- FAIL: TestR480_ARemovedAppShowsNoUpdateOutcome (0.01s)
=== RUN TestR480_AFailureThatHeldNothingKeepsItsSentence
--- PASS: TestR480_AFailureThatHeldNothingKeepsItsSentence (0.01s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.026s
FAIL
exit=1
documentation/audits/v0240-2026-09-13/rp-v240-R484-postgis-not-a-db.txt
+11
View File
@@ -0,0 +1,11 @@
MUTATION in /mnt/5_hdd/felhom.eu/git/felhom-controller/controller/internal/appbackup/dbservices.go:
- , strings.Contains(img, "postgis")
+
=== RUN TestDBTypeForImage
dbservices_test.go:60: dbTypeForImage("postgis/postgis:16-3.5-alpine") = ("", false), want ("postgres", true)
--- FAIL: TestDBTypeForImage (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/appbackup 0.005s
FAIL
exit=1
documentation/audits/v0240-2026-09-13/rp-v240-R485-old-dead-paths.txt
+12
View File
@@ -0,0 +1,12 @@
MUTATION in /mnt/5_hdd/felhom.eu/git/felhom-controller/controller/internal/stacks/delete.go:
- resp.BackupPaths = append(resp.BackupPaths, buildPathInfo(appbackup.RecoveryUnitPath(drivePath, name)))
+ resp.BackupPaths = append(resp.BackupPaths, buildPathInfo(filepath.Join(drivePath, "backups", "primary", name, "db-dumps")))
=== RUN TestR485_BackupCardSeesTheUnitAndTheMirror
r485_backup_card_test.go:46: want the unit and the mirror, got [/tmp/TestR485_BackupCardSeesTheUnitAndTheMirror1453164814/003/backups/secondary/nextcloud]
r485_backup_card_test.go:49: the unit's size must include its volume dumps, got 0
--- FAIL: TestR485_BackupCardSeesTheUnitAndTheMirror (0.01s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/stacks 0.012s
FAIL
exit=1
documentation/audits/v0240-2026-09-13/rp-v240-R486-record-cleared-unconditionally.txt
+14
View File
@@ -0,0 +1,14 @@
MUTATION in /mnt/5_hdd/felhom.eu/git/felhom-controller/controller/internal/api/router.go:
- if r.sett != nil && body.RemoveBackups {
+ if r.sett != nil {
_ = r.sett.SetCrossDriveConfig(name, nil)
}
if r.sett != nil && body.RemoveBackups {
=== RUN TestR486_RemovalKeepsTheTier2RecordUnlessBackupsGo
r474_remove_wiring_test.go:57: removeStack calls SetCrossDriveConfig at line 904 — a removal with backups kept would forget the mirror it kept (R-486)
--- FAIL: TestR486_RemovalKeepsTheTier2RecordUnlessBackupsGo (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-controller/internal/api 0.010s
FAIL
exit=1
documentation/backlog/CLOSED-ITEMS.md
+13
View File
@@ -264,3 +264,16 @@ Compressed here to title, shipping version, evidence, and the sentences that sta
| **R-470** | **Four controller CHANGELOG headers (v0.233.0–v0.236.0) carried no `MinAgent:` line, while the vouch and now the declared floor read it from the header.** Closed 2026-09-13 (`felhom-controller` `f946b0d`): the four headers backfilled with `**MinAgent: 0.129.0** (unchanged)` — v0.232.0's value, proven unchanged (no commit under `internal/agentapi` since 2026-09-01; highest `featureMinAgent` 0.129.0) — and `controller/scripts/minagent_header_gate.py` (fast, blocking) refuses a newest header without the line; a prose or code-span mention does not count (decoy; red-proof F). | **CLOSED 2026-09-13 — GATED** | full text: `git show 2f5d3af:documentation/backlog/OPEN-ITEMS.md` | | **R-470** | **Four controller CHANGELOG headers (v0.233.0–v0.236.0) carried no `MinAgent:` line, while the vouch and now the declared floor read it from the header.** Closed 2026-09-13 (`felhom-controller` `f946b0d`): the four headers backfilled with `**MinAgent: 0.129.0** (unchanged)` — v0.232.0's value, proven unchanged (no commit under `internal/agentapi` since 2026-09-01; highest `featureMinAgent` 0.129.0) — and `controller/scripts/minagent_header_gate.py` (fast, blocking) refuses a newest header without the line; a prose or code-span mention does not count (decoy; red-proof F). | **CLOSED 2026-09-13 — GATED** | full text: `git show 2f5d3af:documentation/backlog/OPEN-ITEMS.md` |
| **R-472** | **The golden cadence ruling and the hub's floor rule contradicted each other: a floor above the vouched golden delivered nothing.** Operator ruling 2026-09-13, hub **v0.112.0** (`f181efd`): a floor saved with the release's declared MinAgent is served above the golden under the same agent comparison; an undeclared one is still held and both forms refuse it (`floor_needs_min_agent`). Proven live: controller 0.239.0 reached demo-hp in 14 s and demo-felhom in 15 s from the save, hub `managed floor SERVED … from declared`. Evidence: `audits/rulings-r472-r475-2026-09-13/` 02, 03. **Reasoning kept:** *the manifest leads the floor inside the golden; above it, the release's own declared MinAgent does* (publish-train rule 1); a declaration binds to its exact floor. | **CLOSED 2026-09-13 — PROVEN-LIVE** | full text: `git show 2f5d3af:documentation/backlog/OPEN-ITEMS.md` | | **R-472** | **The golden cadence ruling and the hub's floor rule contradicted each other: a floor above the vouched golden delivered nothing.** Operator ruling 2026-09-13, hub **v0.112.0** (`f181efd`): a floor saved with the release's declared MinAgent is served above the golden under the same agent comparison; an undeclared one is still held and both forms refuse it (`floor_needs_min_agent`). Proven live: controller 0.239.0 reached demo-hp in 14 s and demo-felhom in 15 s from the save, hub `managed floor SERVED … from declared`. Evidence: `audits/rulings-r472-r475-2026-09-13/` 02, 03. **Reasoning kept:** *the manifest leads the floor inside the golden; above it, the release's own declared MinAgent does* (publish-train rule 1); a declaration binds to its exact floor. | **CLOSED 2026-09-13 — PROVEN-LIVE** | full text: `git show 2f5d3af:documentation/backlog/OPEN-ITEMS.md` |
| **R-475** | **The update precondition was Tier-2-only, so an app with no second-drive copy could not be updated.** Operator ruling 2026-09-13, controller **v0.239.0** (`b93c154`): the first fresh copy in the order Tier 2, Tier 1, Tier 3 (bounded, unreachable = absent + WARN); `backup_max_age` applies to the chosen tier; nothing anywhere → back up first; refused only when no copy and no backup can be taken; the hold names the tier; a Tier-2 failure in the pre-backup is a WARN. Proven live on demo-hp: nothing anywhere → backed up first (04), Tier 1 alone (05), held naming „saját meghajtó” (07), restored from „helyi” (08). Red-proof M (age only on Tier 2) fails. **Reasoning kept:** *first FRESH copy, not first copy* — a stale mirror must not force a backup while the own unit is minutes old. Follow-ups: R-477..R-480. | **CLOSED 2026-09-13 — PROVEN-LIVE** | full text: `git show 2f5d3af:documentation/backlog/OPEN-ITEMS.md` | | **R-475** | **The update precondition was Tier-2-only, so an app with no second-drive copy could not be updated.** Operator ruling 2026-09-13, controller **v0.239.0** (`b93c154`): the first fresh copy in the order Tier 2, Tier 1, Tier 3 (bounded, unreachable = absent + WARN); `backup_max_age` applies to the chosen tier; nothing anywhere → back up first; refused only when no copy and no backup can be taken; the hold names the tier; a Tier-2 failure in the pre-backup is a WARN. Proven live on demo-hp: nothing anywhere → backed up first (04), Tier 1 alone (05), held naming „saját meghajtó” (07), restored from „helyi” (08). Red-proof M (age only on Tier 2) fails. **Reasoning kept:** *first FRESH copy, not first copy* — a stale mirror must not force a backup while the own unit is minutes old. Follow-ups: R-477..R-480. | **CLOSED 2026-09-13 — PROVEN-LIVE** | full text: `git show 2f5d3af:documentation/backlog/OPEN-ITEMS.md` |
| **R-473** | **The `glance` catalog template crash-looped on every fresh install — the image ships no default config.** Closed 2026-09-13 (catalog `50ad286`): an `entrypoint` wrapper seeds a small Hungarian start page on first boot only when `/app/config/glance.yml` is absent, then execs the image's own command (read from the image, not guessed); the seed validated with the image's `config:validate`. Proven live on demo-hp the same evening: a fresh throwaway install healthy in 21 s, restart count 0, front door 200, seed present in the volume (`audits/nightly-2026-09-13-adventurelog/08-R473-glance-fresh-install.txt`). No version moved. | **CLOSED 2026-09-13 — PROVEN-LIVE** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-471** | **`observations_gate.py` read only the FIRST observations section, so an appended second section with an unmarked item passed — and the R-419 decoy had been reading LIVE HOLE at HEAD.** Closed 2026-09-13 (felhom.eu `scripts/observations_gate.py`): `observation_sections` collects every observations heading and `observation_items` pools their items; the heading shown is all of them joined. Cause established: first-heading-wins (the parser `break`s on the first match). Red-proof: the old parser exits 0 on the decoy shape, the new one 1 (`audits/v0240-2026-09-13/rp-R471.txt`); all 12 felhom.eu decoys behave; the felhom.eu and controller REPORTs still pass through the shared script. The "run by hand" half stands: the decoy suite is still not in any runner (R-426's exemptions) — that is a separate row if wanted. | **CLOSED 2026-09-13 — GATED** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-486** | **Removing an app with its backups KEPT forgot its Tier-2 record, so the second-drive restore was refused over an intact mirror.** Closed in controller **v0.240.0** (`bdcbd50`): the record goes only with `remove_backups`. Proven live on demo-hp: remove keeping backups → `cross_drive` record kept → „Teljes visszaállítás" restored 2 volumes and the database, data identical. Red-proof: an unconditional `SetCrossDriveConfig(nil)` fails the wiring test. `audits/v0240-2026-09-13/` (10-v0240-validation.txt; red-proofs rp-v240-*) | **CLOSED 2026-09-13 — PROVEN-LIVE** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-484** | **PostGIS was not a database.** Closed in v0.240.0: `dbTypeForImage` matches `postgis`, `pgvector`, `timescaledb` as Postgres. Proven live: adventurelog's unit now carries `db-dumps/adventurelog-postgres.sql` (8 MB) and the unit restore reports „2 adatkötet és az adatbázis visszaállítva". Red-proof: dropping `postgis` fails the table test. Immich's own image already matched. `audits/v0240-2026-09-13/` (10-v0240-validation.txt; red-proofs rp-v240-*) | **CLOSED 2026-09-13 — PROVEN-LIVE** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-485** | **The backup card read two dead paths and said `has_backups:false` over 484 MB.** Closed in v0.240.0: it sizes the recovery unit and the app's Tier-2 mirror(s). Proven live: `244M` + `244M`, `has_backups:true`. Red-proof: the old paths fail `TestR485`. `audits/v0240-2026-09-13/` (10-v0240-validation.txt; red-proofs rp-v240-*) | **CLOSED 2026-09-13 — PROVEN-LIVE** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-480** | **The card kept a held update's „leállítva marad" sentence after a successful restore and after removal.** Closed in v0.240.0: the stack remembers its last update ended held; `fillHoldReason` hides that outcome once the hold is gone or the app is not deployed; a pull failure keeps its sentence. Red-proof: disabling the block fails three assertions. Live: the card after a restore reads clean. `audits/v0240-2026-09-13/` (10-v0240-validation.txt; red-proofs rp-v240-*) | **CLOSED 2026-09-13 — PROVEN-LIVE** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-477** | **The update's Tier-3 lookup paid the full 15 s bound and blamed another app.** Closed in v0.240.0: `OffsiteSnapshotTimes` — one `snapshots --json`, no per-app `stats`; the inventory page and the update share `offsiteNewestPerTag` (registered read-only for R-408). Red-proof: routing the lookup through the inventory fails `TestR477`. `audits/v0240-2026-09-13/` (10-v0240-validation.txt; red-proofs rp-v240-*) | **CLOSED 2026-09-13 — GATED** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-478** | **A unit left by a removed install counted as the reinstall's fresh copy.** Closed in v0.240.0: `usableRestorePoint` refuses a copy older than the app's `deployed_at`; R-474's fix removes such units anyway. Red-proof: dropping the check fails `TestR478`. `audits/v0240-2026-09-13/` (10-v0240-validation.txt; red-proofs rp-v240-*) | **CLOSED 2026-09-13 — GATED** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-474** | **„Delete backups" deleted only `db-dumps`; the unit and the Tier-2 mirror survived; prefs stayed.** Closed in v0.240.0 for the backups half: the whole unit, every mirror (`Tier2MirrorDirsForApp` / `RemoveTier2Mirrors`, exact-path guarded) and the prefs go; `backup_paths_removed` lists them. Proven live: 244M + 243.6 MB removed, nothing left on either drive, prefs `None`. **The `volumes_removed: null` half is re-filed as R-489.** `audits/v0240-2026-09-13/` (10-v0240-validation.txt; red-proofs rp-v240-*) | **CLOSED 2026-09-13 — PROVEN-LIVE** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-466** | **Removal with „Mentési adatok törlése" left the unit's `compose/` + `manifest.json`.** Subsumed by R-474's fix in v0.240.0 (the whole unit goes). Decided by the same change: the button means the WHOLE unit. `audits/v0240-2026-09-13/` (10-v0240-validation.txt; red-proofs rp-v240-*) | **CLOSED 2026-09-13 — GATED** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-482** | **`adventurelog` ran Django with DEBUG=True on the public origin.** Closed in catalog `ed2c018`: `DEBUG=False` on the backend service. Proven live after the sync: the same CSRF failure renders the 300-byte production page, no debug text. `wger`, `tandoor`, `paperless-ngx` are recorded as unmeasured. `audits/v0240-2026-09-13/` (10-v0240-validation.txt; red-proofs rp-v240-*) | **CLOSED 2026-09-13 — PROVEN-LIVE** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-453** | **`~/.config/credentials` values are single-quoted, and a half-applied strip produced a confidently wrong "password is stale" verdict — twice.** Closed 2026-09-13: the instrument the row asked for exists — `felhom.eu/scripts/read_credential.py KEY <0600-file>` (`66156c6`, 2026-08-31) — and the whole 2026-09-13 night used it for every controller and hub password with zero quoting incidents; the memory `credentials-file-values-are-quoted` now names it. The instrumentation lesson (a discriminator that rules out one alternative does not rule in the rest) stays in the memory. | **CLOSED 2026-09-13 — INSTRUMENTED** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
| **R-461** | **`runbooks/target-selection.md` named a venue that does not exist and fenced a fixture that is gone.** Closed 2026-09-13, both halves checked against both boxes first: (a) no `/mnt/nvme-1tb` on demo-hp or demo-felhom; demo-hp's NVMe is `nvme0n1` at `/mnt/hdd_1` (demo-felhom's `/mnt/hdd_1` is `sdb`) — the runbook now names `/mnt/hdd_1` and says it is the same disk as the data drive; (b) `qm list` is empty on BOTH boxes — `drill-r50` (VM 300) exists nowhere; the fence text stays with the measured absence written beside it, and R-93 carries the fact. | **CLOSED 2026-09-13 — DOCUMENTED** | full text: `git show 681c3d6:documentation/backlog/OPEN-ITEMS.md` |
documentation/backlog/OPEN-ITEMS.md
+6 -10
View File
@@ -319,7 +319,7 @@ unconditional promise, which CAMPAIGN-11 §7 step 7 measured the customer-facing
| **R-124** | **The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""`** and no namespace is literally named `root` — an operator pasting the field into `pct restore --ns root` gets a failure | READY (XS) | — | Pre-existing wire convention (`ToHub` has normalised empty→`"root"` since slice 6), deliberately NOT changed under R-106 so the field's meaning did not shift mid-fix. Documented at `hub.PBSRootNamespace`. Affects only a box with no `namespace` line — **no real customer today**, all three are per-customer. Fix = emit `""` + rely on `namespace_state`, or emit a `--ns`-ready form | CC | | **R-124** | **The recipe spells PBS's root namespace `"root"`, but the PBS API spells it `""`** and no namespace is literally named `root` — an operator pasting the field into `pct restore --ns root` gets a failure | READY (XS) | — | Pre-existing wire convention (`ToHub` has normalised empty→`"root"` since slice 6), deliberately NOT changed under R-106 so the field's meaning did not shift mid-fix. Documented at `hub.PBSRootNamespace`. Affects only a box with no `namespace` line — **no real customer today**, all three are per-customer. Fix = emit `""` + rely on `namespace_state`, or emit a `--ns`-ready form | CC |
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC | | **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
| **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC | | **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
| **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC | | **R-93** | `drill-r50` is both a blocked customer and the only drift fixture **FACT 2026-09-13 (R-461): the fixture is GONE — `qm list` is empty on both demo boxes, so neither option is available and the row's premise no longer holds; the operator decides whether that closes it or reopens it as "build a drift fixture".** | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
| **R-129** | **Every doc says demo-hp has "no baked SSH key"** and needs the G1 break-glass password — but `ssh -o BatchMode=yes demo-hp` authenticated **by key**, first try, 2026-07-31 | READY (XS) | — | Stale in the expensive direction: a session that believes it sends itself to the hub vault for a credential it does not need. Verify who owns the key and when it landed, then correct `CLAUDE.md`, `runbooks/target-selection.md:41-42`, `runbooks/workspace-CLAUDE.md` and `felhom-agent/CLAUDE.md` together — or remove the key if it was not deliberate | CC | | **R-129** | **Every doc says demo-hp has "no baked SSH key"** and needs the G1 break-glass password — but `ssh -o BatchMode=yes demo-hp` authenticated **by key**, first try, 2026-07-31 | READY (XS) | — | Stale in the expensive direction: a session that believes it sends itself to the hub vault for a credential it does not need. Verify who owns the key and when it landed, then correct `CLAUDE.md`, `runbooks/target-selection.md:41-42`, `runbooks/workspace-CLAUDE.md` and `felhom-agent/CLAUDE.md` together — or remove the key if it was not deliberate | CC |
| **R-130** | **A "hard min" that only warns.** A fresh box's `local-lvm` was ~75 GiB against `HARD_MIN_LVM_GIB=120` (`scripts/felhom-host-install.sh`); the installer logged `[WARN] local-lvm free ~75 GiB < hard min 120 GiB` and went on to a **fully successful** install | READY (S) | — | Either the minimum is not hard (rename it and state the real floor) or it is wrong (and 120 GiB is not what a working appliance needs). Leaving it is the R-29 shape: a check that reads as coverage while providing none. Evidence: same audit §8 | CC | | **R-130** | **A "hard min" that only warns.** A fresh box's `local-lvm` was ~75 GiB against `HARD_MIN_LVM_GIB=120` (`scripts/felhom-host-install.sh`); the installer logged `[WARN] local-lvm free ~75 GiB < hard min 120 GiB` and went on to a **fully successful** install | READY (S) | — | Either the minimum is not hard (rename it and state the real floor) or it is wrong (and 120 GiB is not what a working appliance needs). Leaving it is the R-29 shape: a check that reads as coverage while providing none. Evidence: same audit §8 | CC |
| **R-131** | **`sess-f` is a fourth orphaned scratch customer** on the hub ("R-120 golden 0.186.0 proof", DOWN), left by the 2026-07-30 session | READY (XS) | — | After `drill-r50`, `sess-c`, `sess-d` — the accumulation `runbooks/target-selection.md:86-87` and `PROMPT-TEMPLATE.md` §13 both warn about, now on its fourth instance. Delete it (see the recorded command in `audits/tester-gate-golden-0.188.0-2026-07-31.md` §7.1); the recurrence itself argues for a periodic scratch-customer sweep rather than another reminder | CC | | **R-131** | **`sess-f` is a fourth orphaned scratch customer** on the hub ("R-120 golden 0.186.0 proof", DOWN), left by the 2026-07-30 session | READY (XS) | — | After `drill-r50`, `sess-c`, `sess-d` — the accumulation `runbooks/target-selection.md:86-87` and `PROMPT-TEMPLATE.md` §13 both warn about, now on its fourth instance. Delete it (see the recorded command in `audits/tester-gate-golden-0.188.0-2026-07-31.md` §7.1); the recurrence itself argues for a periodic scratch-customer sweep rather than another reminder | CC |
@@ -682,31 +682,27 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
| **R-450** | **[P2-MEDIUM] UPDATE ARC SLICE 6 — a version sequence: automatic WITHIN a major, never ACROSS one, and an engine change gets its OWN edge.** The first half is an operator ruling of 2026-09-02 and its justification is R-449's measurement: a cross-major jump can be refused by the app itself and cannot be undone. **The second half is a rule recorded now, while it is cheap:** an engine change must never be bundled with an app version bump. `bookstack`'s `0b73e5e` moved the application 25.02.2 → 26.05.2 **and** MariaDB 11.6 → 12.3 in one commit — **two migrations behind one edge**, and an unreadable failure when it breaks. Needs a catalog-side convention and, eventually, a gate. `architecture/09-update-architecture.md` §6 | **READY — rank P2-MEDIUM; owner: VIKTOR rules, CC implements** | | **R-450** | **[P2-MEDIUM] UPDATE ARC SLICE 6 — a version sequence: automatic WITHIN a major, never ACROSS one, and an engine change gets its OWN edge.** The first half is an operator ruling of 2026-09-02 and its justification is R-449's measurement: a cross-major jump can be refused by the app itself and cannot be undone. **The second half is a rule recorded now, while it is cheap:** an engine change must never be bundled with an app version bump. `bookstack`'s `0b73e5e` moved the application 25.02.2 → 26.05.2 **and** MariaDB 11.6 → 12.3 in one commit — **two migrations behind one edge**, and an unreadable failure when it breaks. Needs a catalog-side convention and, eventually, a gate. `architecture/09-update-architecture.md` §6 | **READY — rank P2-MEDIUM; owner: VIKTOR rules, CC implements** |
| **R-451** | **[P3-LOW] UPDATE ARC SLICE 7 — a fleet sweep: the operator can SEE, and MOVE, how far behind every box is.** Slices 1 and 2 make one box's state visible on that box's own pages. The operator has no fleet view, and **it is not derivable from what is already reported: the hub's report payload carries container name, state, CPU and memory, and NO image field at all** (spike §5, which is why Peti's box could only be recorded UNKNOWN). So this is a hub-side change as well as a controller one. Rank LOW today because the fleet is two enrolled boxes; it rises with the fleet. `architecture/09-update-architecture.md` §6, §8.4 | **READY — rank P3-LOW; owner: CC** | | **R-451** | **[P3-LOW] UPDATE ARC SLICE 7 — a fleet sweep: the operator can SEE, and MOVE, how far behind every box is.** Slices 1 and 2 make one box's state visible on that box's own pages. The operator has no fleet view, and **it is not derivable from what is already reported: the hub's report payload carries container name, state, CPU and memory, and NO image field at all** (spike §5, which is why Peti's box could only be recorded UNKNOWN). So this is a hub-side change as well as a controller one. Rank LOW today because the fleet is two enrolled boxes; it rises with the fleet. `architecture/09-update-architecture.md` §6, §8.4 | **READY — rank P3-LOW; owner: CC** |
| **R-452** | **[P3-LOW] Nothing enforces `catalog_since`, so the one number the update badge shows can silently under-report.** `app-catalog-felhom.eu` `CLAUDE.md` now states the rule — any commit that changes an `image:` line must set that app's `catalog_since` to the same day — and all 53 apps were backfilled from git history on 2026-09-02 (`69761cf`). **A rule with no instrument is a wish; that is this project's most-repeated finding and this row exists so it is not repeated silently.** A stale `catalog_since` makes „Frissítés elérhető — N napja" under-report N, which is the single number the badge exists to give. **WHY IT WAS NOT BUILT IN THE SAME SESSION, stated rather than implied:** the gate would have to diff an `image:` line against the PARENT commit, and `catalog_gates.py` runs under a runner that fetches at `--depth 1` — there is no parent to diff against. The gate therefore needs a deeper fetch, which is a change to the CI shape and not to a script. **This is the R-421 class in advance: an enumerated gap becomes a row in the same session it is enumerated.** `architecture/09-update-architecture.md` §8.2 | **READY — rank P3-LOW; owner: CC** | | **R-452** | **[P3-LOW] Nothing enforces `catalog_since`, so the one number the update badge shows can silently under-report.** `app-catalog-felhom.eu` `CLAUDE.md` now states the rule — any commit that changes an `image:` line must set that app's `catalog_since` to the same day — and all 53 apps were backfilled from git history on 2026-09-02 (`69761cf`). **A rule with no instrument is a wish; that is this project's most-repeated finding and this row exists so it is not repeated silently.** A stale `catalog_since` makes „Frissítés elérhető — N napja" under-report N, which is the single number the badge exists to give. **WHY IT WAS NOT BUILT IN THE SAME SESSION, stated rather than implied:** the gate would have to diff an `image:` line against the PARENT commit, and `catalog_gates.py` runs under a runner that fetches at `--depth 1` — there is no parent to diff against. The gate therefore needs a deeper fetch, which is a change to the CI shape and not to a script. **This is the R-421 class in advance: an enumerated gap becomes a row in the same session it is enumerated.** `architecture/09-update-architecture.md` §8.2 | **READY — rank P3-LOW; owner: CC** |
| **R-453** | **[P3-LOW] `~/.config/credentials` quotes its values with SINGLE quotes, and a half-applied strip cost a session an hour and produced a WRONG diagnosis that was published before the operator corrected it.** MEASURED 2026-09-02: extracting `PASSWORD` with `sed 's/^"//;s/"$//'` — double quotes only — sent the literal `'` characters as part of the password, and `POST /login` returned **HTTP 200 + `Hibás jelszó`** on BOTH demo boxes. **THE DIAGNOSIS THAT FOLLOWED WAS WRONG AND WAS WRITTEN INTO A REGISTER ROW, A STATUS ITEM AND A MEMORY BEFORE IT WAS CHECKED:** "the vaulted password is stale on both boxes". The operator answered in one line — *the value is in single quotes* — and one retry with `sed "s/^['\\"]//;s/['\\"]$//"` returned **302 + `felhom_session`**. **THE INSTRUMENTATION LESSON, which is the actual finding and outlives the typo:** the controller's own log line `auth.go:176 [WARN] Failed login` was quoted as the discriminator, and it IS a true and useful one — **it separates "wrong password" from "wrong Host header", and that is ALL it separates.** It cannot distinguish a wrong password from wrong password HANDLING, and it was read as if it could. A discriminator that rules out one alternative is not a discriminator that rules in the remaining one. **This is the SECOND time this exact file's quoting has produced a confidently wrong verdict** — the memory `credentials-file-values-are-quoted` was minted for the first (`cut -d=` keeps the quotes → a wrong "the password is stale" diagnosis), and this session applied that memory HALF, stripping one quote character and not the other. **The fix is an instrument, not a resolution to be careful:** one shared helper that extracts a value from that file correctly, used everywhere, so the next session cannot get it half right. Evidence: `tests/VALIDATION-update-slice12-2026-09-02.md` §4.0, which states the correction rather than quietly removing the claim. | **READY — rank P3-LOW; owner: CC** |
| **R-454** | **[P3-LOW] Five `internal/web` test files have been `gofmt`-unclean for an unknown length of time, and nothing notices.** MEASURED 2026-09-02: `gofmt -l controller/internal/web/` reports `backups_split_test.go`, `claim_code_naming_test.go`, `disk_health_test.go`, `r400_debug_routes_test.go`, `recovery_test.go` — at the **baseline** commit `960d29b0612c`, i.e. not introduced by v0.233.0 (both files added that day are clean). **`go vet` does not check formatting and `controller_gates.py` has no formatting gate**, so the only thing that would ever surface this is someone running `gofmt -l` by hand, which is how it was found. **Not reformatted in the same session, deliberately** — the minimal-changes rule, and a five-file whitespace commit inside a feature release makes that release's diff unreadable. **Small, and the cost of NOT having the instrument is the row:** the count can only grow, and every future `gofmt -l` run produces noise that hides a real one. Fix is two lines: a `gofmt -l` gate in `controller_gates.py` plus one formatting commit, in that order (the gate first, so the commit is provably complete). Owner: **CC.** | **READY — rank P3-LOW; owner: CC** | | **R-454** | **[P3-LOW] Five `internal/web` test files have been `gofmt`-unclean for an unknown length of time, and nothing notices.** MEASURED 2026-09-02: `gofmt -l controller/internal/web/` reports `backups_split_test.go`, `claim_code_naming_test.go`, `disk_health_test.go`, `r400_debug_routes_test.go`, `recovery_test.go` — at the **baseline** commit `960d29b0612c`, i.e. not introduced by v0.233.0 (both files added that day are clean). **`go vet` does not check formatting and `controller_gates.py` has no formatting gate**, so the only thing that would ever surface this is someone running `gofmt -l` by hand, which is how it was found. **Not reformatted in the same session, deliberately** — the minimal-changes rule, and a five-file whitespace commit inside a feature release makes that release's diff unreadable. **Small, and the cost of NOT having the instrument is the row:** the count can only grow, and every future `gofmt -l` run produces noise that hides a real one. Fix is two lines: a `gofmt -l` gate in `controller_gates.py` plus one formatting commit, in that order (the gate first, so the commit is provably complete). Owner: **CC.** | **READY — rank P3-LOW; owner: CC** |
| **R-456** | **[P3-LOW] A partly-dead stack is not a boot orphan, and that is written down nowhere.** MEASURED 2026-09-02 on demo-hp while validating v0.233.0: `docker rm -f bookstack` (leaving `bookstack-db` running) then a controller restart produced `Boot reconciliation: 1 boot-orphaned app(s) found: [bentopdf]` — **bookstack was NOT selected**, although the app container was gone and `desired_state: running` was recorded. Removing `bookstack-db` as well made the whole stack orphaned and the very next pass repaired it in 6.3 s. **So `bootrecon.isBootOrphan` requires the stack as a WHOLE to be down; one live member is enough to make it invisible to the reconciler.** **NOT called a defect, and the reason is part of the row:** `StateDegraded` IS in `IsDownState`, and the crash-loop/dead-app alarm path (`classifyRunStates`) does count a degraded stack as down — so the customer IS told; it is the automatic REPAIR that does not fire, and there may be a good reason (repairing half a stack while its DB is live is not obviously safe). **What is certain is that nobody has written the rule down**, so the next session re-derives it the same way this one did — by watching a reconciliation not happen, which is an absent observable and the weakest possible evidence. Either state the rule in `02-controller-module-map.md` with a test pinning it, or change it. Owner: **CC.** `tests/VALIDATION-update-slice12-2026-09-02.md` §2.2 | **READY — rank P3-LOW; owner: CC** | | **R-456** | **[P3-LOW] A partly-dead stack is not a boot orphan, and that is written down nowhere.** MEASURED 2026-09-02 on demo-hp while validating v0.233.0: `docker rm -f bookstack` (leaving `bookstack-db` running) then a controller restart produced `Boot reconciliation: 1 boot-orphaned app(s) found: [bentopdf]` — **bookstack was NOT selected**, although the app container was gone and `desired_state: running` was recorded. Removing `bookstack-db` as well made the whole stack orphaned and the very next pass repaired it in 6.3 s. **So `bootrecon.isBootOrphan` requires the stack as a WHOLE to be down; one live member is enough to make it invisible to the reconciler.** **NOT called a defect, and the reason is part of the row:** `StateDegraded` IS in `IsDownState`, and the crash-loop/dead-app alarm path (`classifyRunStates`) does count a degraded stack as down — so the customer IS told; it is the automatic REPAIR that does not fire, and there may be a good reason (repairing half a stack while its DB is live is not obviously safe). **What is certain is that nobody has written the rule down**, so the next session re-derives it the same way this one did — by watching a reconciliation not happen, which is an absent observable and the weakest possible evidence. Either state the rule in `02-controller-module-map.md` with a test pinning it, or change it. Owner: **CC.** `tests/VALIDATION-update-slice12-2026-09-02.md` §2.2 | **READY — rank P3-LOW; owner: CC** |
| **R-457** | **[P3-LOW] A test that hardcodes a date AND asserts an age derived from it is green on the day it is written and red the next morning — one instance PROVEN, six candidate files named.** MEASURED 2026-09-03: `TestGroupD_BadgeRendersOnBothSurfaces` (shipped the previous day in v0.233.0) pinned a fixture `catalog_since: "2026-07-18"` and asserted the rendered string `"Frissítés elérhető — 46 napja"`. **The pure badge tests inject a clock; the RENDER test does not and cannot** — it goes through the production templates, which call the funcmap entry `updateBadge`, which reads `time.Now()`. The suite was green on 2026-09-02 and **FAILED on 2026-09-03** with *"the behind badge is missing"* on both surfaces, because the true answer had become 47. **Fixed by DERIVING the fixture** — `catalog_since` is computed as *today minus 46 days*, so the test asserts the real number through the real clock and cannot rot. **THE CLASS, which is why this is a row and not just a fix:** a clock-reading test that also carries a date LITERAL is a bomb with a fuse of unknown length, and the suite being green is not evidence it is defused — it is evidence the fuse has not burned down yet. **NAMED AS UNCHECKED CANDIDATES, NOT ACCUSED** — six other test files contain both a `20xx-xx-xx` literal and `time.Now()`: `internal/backup/offbox_test.go`, `internal/web/handler_export_upload_test.go`, `internal/web/r103_tier2_action_test.go`, `internal/web/dashboard_backup_card_test.go`, `internal/web/async_restore_test.go`, `internal/stacks/installed_test.go`. Mixing the two is not itself a defect — it is one only where a literal feeds an assertion evaluated against the real clock — so each needs reading, which is a sweep and not this session. **The instrument that would end the class:** run the suite once under a faked future date in CI and see what turns red. Owner: **CC.** `felhom-controller` v0.234.0 CHANGELOG | **READY — rank P3-LOW; owner: CC** | | **R-457** | **[P3-LOW] A test that hardcodes a date AND asserts an age derived from it is green on the day it is written and red the next morning — one instance PROVEN, six candidate files named.** MEASURED 2026-09-03: `TestGroupD_BadgeRendersOnBothSurfaces` (shipped the previous day in v0.233.0) pinned a fixture `catalog_since: "2026-07-18"` and asserted the rendered string `"Frissítés elérhető — 46 napja"`. **The pure badge tests inject a clock; the RENDER test does not and cannot** — it goes through the production templates, which call the funcmap entry `updateBadge`, which reads `time.Now()`. The suite was green on 2026-09-02 and **FAILED on 2026-09-03** with *"the behind badge is missing"* on both surfaces, because the true answer had become 47. **Fixed by DERIVING the fixture** — `catalog_since` is computed as *today minus 46 days*, so the test asserts the real number through the real clock and cannot rot. **THE CLASS, which is why this is a row and not just a fix:** a clock-reading test that also carries a date LITERAL is a bomb with a fuse of unknown length, and the suite being green is not evidence it is defused — it is evidence the fuse has not burned down yet. **NAMED AS UNCHECKED CANDIDATES, NOT ACCUSED** — six other test files contain both a `20xx-xx-xx` literal and `time.Now()`: `internal/backup/offbox_test.go`, `internal/web/handler_export_upload_test.go`, `internal/web/r103_tier2_action_test.go`, `internal/web/dashboard_backup_card_test.go`, `internal/web/async_restore_test.go`, `internal/stacks/installed_test.go`. Mixing the two is not itself a defect — it is one only where a literal feeds an assertion evaluated against the real clock — so each needs reading, which is a sweep and not this session. **The instrument that would end the class:** run the suite once under a faked future date in CI and see what turns red. Owner: **CC.** `felhom-controller` v0.234.0 CHANGELOG | **READY — rank P3-LOW; owner: CC** |
| **R-458** | **[P3-LOW] `.felhom.yml` keeps flowing to an app whose compose file is FROZEN, so a frozen app can receive a health check written for a version it is not running.** The v0.235.0 render freezes `docker-compose.yml` for a pinned app once the catalog moves past its version, but copies `.felhom.yml` **verbatim in every case** (`Syncer.copyTemplates`). **The asymmetry is deliberate and both directions were considered:** `.felhom.yml` carries no image, and it carries `catalog_since` — the single input the update badge uses to say *„Frissítés elérhető — N napja"* — so freezing it would silently withhold the one number that tells a customer they are behind, i.e. it would break slice 2 to protect slice 3. **What it costs:** the file also carries the controller-side `healthcheck:` block and resource hints, so a template updated for a newer version can hand a frozen app a probe written for software it is not running. **THE FAILURE DIRECTION IS A FALSE ALARM, NEVER DATA LOSS** — the app keeps running; at worst it renders as degraded and, if it persisted, could reach the dead-app alarm path. That is the same class as R-330's false e-mails, which is why this is a row and not a footnote. **Not fixed now, and the reason is that the cheap fix is wrong:** freezing the whole file breaks the badge, and freezing only the `healthcheck:` key means the syncer would have to parse and re-assemble a customer-facing metadata file — new surface on the one path that touches every app on every box every 15 minutes. **What would settle it:** whether any catalog `healthcheck:` has ever been changed in the same commit as an `image:` line (measurable from the catalog's own history, no box needed). If the answer is "never", the exposure is theoretical and the row can be closed by measurement instead of by code. Owner: **CC.** `architecture/09-update-architecture.md` §5.4, §8.5 | **READY — rank P3-LOW; owner: CC** | | **R-458** | **[P3-LOW] `.felhom.yml` keeps flowing to an app whose compose file is FROZEN, so a frozen app can receive a health check written for a version it is not running.** The v0.235.0 render freezes `docker-compose.yml` for a pinned app once the catalog moves past its version, but copies `.felhom.yml` **verbatim in every case** (`Syncer.copyTemplates`). **The asymmetry is deliberate and both directions were considered:** `.felhom.yml` carries no image, and it carries `catalog_since` — the single input the update badge uses to say *„Frissítés elérhető — N napja"* — so freezing it would silently withhold the one number that tells a customer they are behind, i.e. it would break slice 2 to protect slice 3. **What it costs:** the file also carries the controller-side `healthcheck:` block and resource hints, so a template updated for a newer version can hand a frozen app a probe written for software it is not running. **THE FAILURE DIRECTION IS A FALSE ALARM, NEVER DATA LOSS** — the app keeps running; at worst it renders as degraded and, if it persisted, could reach the dead-app alarm path. That is the same class as R-330's false e-mails, which is why this is a row and not a footnote. **Not fixed now, and the reason is that the cheap fix is wrong:** freezing the whole file breaks the badge, and freezing only the `healthcheck:` key means the syncer would have to parse and re-assemble a customer-facing metadata file — new surface on the one path that touches every app on every box every 15 minutes. **What would settle it:** whether any catalog `healthcheck:` has ever been changed in the same commit as an `image:` line (measurable from the catalog's own history, no box needed). If the answer is "never", the exposure is theoretical and the row can be closed by measurement instead of by code. Owner: **CC.** `architecture/09-update-architecture.md` §5.4, §8.5 | **READY — rank P3-LOW; owner: CC** |
| **R-460** | **[P3-LOW] BookStack's FILE half cannot be seeded or verified without a browser, so its upgrades can only ever be auto-proven for the DATABASE.** MEASURED 2026-09-06 while building the R-449 harness. BookStack's API needs a token that is only mintable through its web UI, and its HTTP login is unusable headlessly for a second, independent reason: `APP_URL` comes from the template as `https://${SUBDOMAIN}.${DOMAIN}`, so the app marks its session and XSRF cookies **`secure`**; curl over plain http stores neither and **every login POST returns 419 Page Expired**, which looks exactly like a wrong password. The container serves no TLS. **The database half IS provable** — the harness seeds with `php artisan bookstack:create-admin` and reads back with a DIFFERENT artisan command that must find the record, carrying its own negative control on every call. **What is unprovable is an uploaded image or attachment**, i.e. exactly the half a customer would notice. **THIS IS A FACT ABOUT THE APP, NOT A DEFECT IN THE HARNESS**, and it is recorded because Slice 6 needs to know which apps can be auto-verified and which can only be partly verified — nobody had that list before. **Deliberately NOT worked around:** planting a file in the volume would make the test pass while proving nothing, which is R-156's exact failure. **What would remove it:** a headless token route (upstream), or accepting a browser-driven step for this app alone, which DooPlex cannot run. Owner: **CC.** `audits/SPIKE-upgrade-test-2026-09-06.md` §6 | **READY — rank P3-LOW; owner: CC** | | **R-460** | **[P3-LOW] BookStack's FILE half cannot be seeded or verified without a browser, so its upgrades can only ever be auto-proven for the DATABASE.** MEASURED 2026-09-06 while building the R-449 harness. BookStack's API needs a token that is only mintable through its web UI, and its HTTP login is unusable headlessly for a second, independent reason: `APP_URL` comes from the template as `https://${SUBDOMAIN}.${DOMAIN}`, so the app marks its session and XSRF cookies **`secure`**; curl over plain http stores neither and **every login POST returns 419 Page Expired**, which looks exactly like a wrong password. The container serves no TLS. **The database half IS provable** — the harness seeds with `php artisan bookstack:create-admin` and reads back with a DIFFERENT artisan command that must find the record, carrying its own negative control on every call. **What is unprovable is an uploaded image or attachment**, i.e. exactly the half a customer would notice. **THIS IS A FACT ABOUT THE APP, NOT A DEFECT IN THE HARNESS**, and it is recorded because Slice 6 needs to know which apps can be auto-verified and which can only be partly verified — nobody had that list before. **Deliberately NOT worked around:** planting a file in the volume would make the test pass while proving nothing, which is R-156's exact failure. **What would remove it:** a headless token route (upstream), or accepting a browser-driven step for this app alone, which DooPlex cannot run. Owner: **CC.** `audits/SPIKE-upgrade-test-2026-09-06.md` §6 | **READY — rank P3-LOW; owner: CC** |
| **R-461** | **[P3-LOW] `runbooks/target-selection.md` names a venue that does not exist and fences a fixture that is gone.** MEASURED 2026-09-06 on demo-hp while siting the R-449 guest. (a) The runbook says to put VM disks on a dir storage at **`/mnt/nvme-1tb`, at its root**. **There is no `/mnt/nvme-1tb`** — the 1 TB NVMe is mounted at **`/mnt/hdd_1`**, which is the enrolled user-data drive and the `felhom-backup` target, i.e. the same disk under a different path. The instruction's REASON is still exactly right (`local-lvm` is an over-subscribed thin pool backing the live guest 9201, and this run kept off it — `local-lvm` read **30.50 % before and after**), so the fence held; only its address is stale. (b) The runbook forbids destroying **`drill-r50` (VM 300)**, "the only drift fixture (R-93)". **`qm list` returns nothing on demo-hp** — there are no VMs at all. **The fence currently protects nothing, and R-93's premise that a drift fixture exists is false.** **Why it is a row and not a quiet edit:** a runbook that names a path nobody can find is one a session works around, and working around a safety instruction is how the instruction stops being followed. Both halves need checking against the box before the text is changed — (b) in particular may mean R-93 should be closed or reopened as "the drift fixture is gone", which is a different fact from "do not destroy it". Owner: **CC.** `audits/SPIKE-upgrade-test-2026-09-06.md` §7 | **READY — rank P3-LOW; owner: CC** |
| **R-462** | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 | **READY — rank P2-MEDIUM; owner: VIKTOR rules on scope, CC implements** | | **R-462** | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 | **READY — rank P2-MEDIUM; owner: VIKTOR rules on scope, CC implements** |
| **R-463** | **[P2-MEDIUM] The day the catalog moves `postgres:16` to `17`, ELEVEN apps are affected and the container image will NOT perform the conversion — and nothing anywhere records that.** MEASURED 2026-09-06: 11 of the 53 templates carry PostgreSQL — **8 on `postgres:16-alpine`**, 1 on `postgres:15-alpine`, plus `postgis/postgis:16-3.5-alpine` and Immich's own `postgres:16-vectorchord…` build. **A grep of the whole register for `pg_upgrade`, "postgres major" or "postgresql major" returns ZERO** (confirmed this session, and confirmed again before filing). **WHY IT IS NOT THE SAME PROBLEM AS R-459, and this is the point of the row: the two engines fail in OPPOSITE directions.** MariaDB starts anyway and skips the conversion quietly, which is why R-459 went unnoticed until a harness looked. **PostgreSQL REFUSES TO START on a datadir from an older major** — the official image performs no `pg_upgrade` and exits with a message naming both versions. So the Postgres case cannot hide; it will present as eight apps down at once, on the sync after the catalog moves. **DELIBERATELY NOT MEASURED HERE, and saying so is the scope discipline:** R-459's task was scoped to MariaDB, and measuring the Postgres analogue is its own piece of work with its own venue. **This row exists so the gap is a record rather than a sentence in an audit nobody greps.** What would settle it: one edge on the existing harness (`postgres:16-alpine` → `17-alpine`) on a scratch host, which would also exercise the `engine_state_after` field's Postgres probe end to end — it is written but has never run against a real Postgres major. Owner: **CC.** `audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md` §7 | **READY — rank P2-MEDIUM; owner: CC** | | **R-463** | **[P2-MEDIUM] The day the catalog moves `postgres:16` to `17`, ELEVEN apps are affected and the container image will NOT perform the conversion — and nothing anywhere records that.** MEASURED 2026-09-06: 11 of the 53 templates carry PostgreSQL — **8 on `postgres:16-alpine`**, 1 on `postgres:15-alpine`, plus `postgis/postgis:16-3.5-alpine` and Immich's own `postgres:16-vectorchord…` build. **A grep of the whole register for `pg_upgrade`, "postgres major" or "postgresql major" returns ZERO** (confirmed this session, and confirmed again before filing). **WHY IT IS NOT THE SAME PROBLEM AS R-459, and this is the point of the row: the two engines fail in OPPOSITE directions.** MariaDB starts anyway and skips the conversion quietly, which is why R-459 went unnoticed until a harness looked. **PostgreSQL REFUSES TO START on a datadir from an older major** — the official image performs no `pg_upgrade` and exits with a message naming both versions. So the Postgres case cannot hide; it will present as eight apps down at once, on the sync after the catalog moves. **DELIBERATELY NOT MEASURED HERE, and saying so is the scope discipline:** R-459's task was scoped to MariaDB, and measuring the Postgres analogue is its own piece of work with its own venue. **This row exists so the gap is a record rather than a sentence in an audit nobody greps.** What would settle it: one edge on the existing harness (`postgres:16-alpine` → `17-alpine`) on a scratch host, which would also exercise the `engine_state_after` field's Postgres probe end to end — it is written but has never run against a real Postgres major. Owner: **CC.** `audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md` §7 | **READY — rank P2-MEDIUM; owner: CC** |
| **R-464** | **[P3-LOW] MariaDB's entrypoint prints `MariaDB upgrade not required` on an UNSUPPORTED DOWNGRADE, so that line cannot be used as a soundness signal.** MEASURED 2026-09-06. After converting a datadir to `12.3.3-MariaDB` and then starting **11.6** on it, the entrypoint logs, on every start: **`[Note] [Entrypoint]: MariaDB upgrade not required`**. Asked properly, the same engine answers **`FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported!`** **The entrypoint compares the datadir's recorded version against its own and concludes there is nothing to DO. That is true, and it is not a statement that the state is sound.** **THIS IS THIS PROJECT'S MOST-REPEATED CLASS, in a new costume** — the same shape as `CLAUDE.md`'s "presence is not success" and as R-443's HTTP 200 over a crash-looping app: a reassuring sentence that answers a narrower question than the one a reader will take it for. **Why it is worth a row rather than a footnote: the obvious cheap instrument for R-459 is to grep container logs for that exact line**, and such an instrument would report "fine" for an unsupported downgrade. **The correct probe is `mariadb-upgrade --check-if-upgrade-is-needed`**, which is what `upgrade-test.py`'s `engine_state_after` now uses. **Also recorded, because it nearly produced a wrong answer here: run without credentials that command returns `ERROR 1045 … FATAL ERROR: Upgrade failed` with exit 1** — an authentication failure wearing the shape of a verdict. Owner: **CC.** `audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md` §5.4 | **READY — rank P3-LOW; owner: CC** | | **R-464** | **[P3-LOW] MariaDB's entrypoint prints `MariaDB upgrade not required` on an UNSUPPORTED DOWNGRADE, so that line cannot be used as a soundness signal.** MEASURED 2026-09-06. After converting a datadir to `12.3.3-MariaDB` and then starting **11.6** on it, the entrypoint logs, on every start: **`[Note] [Entrypoint]: MariaDB upgrade not required`**. Asked properly, the same engine answers **`FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported!`** **The entrypoint compares the datadir's recorded version against its own and concludes there is nothing to DO. That is true, and it is not a statement that the state is sound.** **THIS IS THIS PROJECT'S MOST-REPEATED CLASS, in a new costume** — the same shape as `CLAUDE.md`'s "presence is not success" and as R-443's HTTP 200 over a crash-looping app: a reassuring sentence that answers a narrower question than the one a reader will take it for. **Why it is worth a row rather than a footnote: the obvious cheap instrument for R-459 is to grep container logs for that exact line**, and such an instrument would report "fine" for an unsupported downgrade. **The correct probe is `mariadb-upgrade --check-if-upgrade-is-needed`**, which is what `upgrade-test.py`'s `engine_state_after` now uses. **Also recorded, because it nearly produced a wrong answer here: run without credentials that command returns `ERROR 1045 … FATAL ERROR: Upgrade failed` with exit 1** — an authentication failure wearing the shape of a verdict. Owner: **CC.** `audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md` §5.4 | **READY — rank P3-LOW; owner: CC** |
| **R-465** | **[P3-LOW] `cfg.Paths.HDDPath` — the global that R-442 proved is set on NO box (demo-hp AND demo-felhom: 0 `hdd_path`, 0 `FELHOM_PATHS_*`) — still has SIX readers, each reading an always-empty value:** `report/builder.go:69`, `monitor/healthcheck.go:35`, `api/router.go` (system-info), `web/server.go:740`, `cmd/controller/main.go` (auto-discovery seed + metrics HDD path). Removal was silently inert for months on the very same read. Whether any of these is inert the same way — a report field that is always empty, a health check that never fires, a metric never collected — is a one-hour audit: for each reader, name the POSITIVE observable that must appear when it works and check it on the box. R-442 §5 said "do not delete it here"; this row is the audit it deferred. Owner: **CC.** `felhom-controller/REPORT.md` (v0.236.0, Observations 1) | **READY — rank P3-LOW; owner: CC** | | **R-465** | **[P3-LOW] `cfg.Paths.HDDPath` — the global that R-442 proved is set on NO box (demo-hp AND demo-felhom: 0 `hdd_path`, 0 `FELHOM_PATHS_*`) — still has SIX readers, each reading an always-empty value:** `report/builder.go:69`, `monitor/healthcheck.go:35`, `api/router.go` (system-info), `web/server.go:740`, `cmd/controller/main.go` (auto-discovery seed + metrics HDD path). Removal was silently inert for months on the very same read. Whether any of these is inert the same way — a report field that is always empty, a health check that never fires, a metric never collected — is a one-hour audit: for each reader, name the POSITIVE observable that must appear when it works and check it on the box. R-442 §5 said "do not delete it here"; this row is the audit it deferred. Owner: **CC.** `felhom-controller/REPORT.md` (v0.236.0, Observations 1) | **READY — rank P3-LOW; owner: CC** |
| **R-466** | **[P3-LOW] Removing an app with „Mentési adatok törlése" ticked leaves its recovery unit's `compose/` + `manifest.json` on the drive.** The router passes only `backup.AppDBDumpPath(nsRoot, name)` to `RemoveStack`, so `backups/primary/<app>/db-dumps/` goes and the unit root keeps `compose/` (the app's `app.yaml` with the portable secret class at 0600) and `manifest.json`. MEASURED 2026-09-13 on demo-hp after the R-442 Scenario A removal — `audits/R442-2026-09-13/teardown-and-log.txt`, residue check: `backups/primary/nextcloud` still listed with the app gone; removed by hand at teardown. A customer who asked for the backups to go is left with the app's definition and a manifest. **Decide:** the button means the WHOLE unit (pass the unit root, under the same `backups/`-prefix guard) or stays db-dumps-only (then the modal must say so). Owner: **CC.** | **READY — rank P3-LOW; owner: CC** |
| **R-468** | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** | **WATCHING — rank P3-LOW; owner: CC (renew ≤ 14 days or bake); retire at the first external install** | | **R-468** | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** | **WATCHING — rank P3-LOW; owner: CC (renew ≤ 14 days or bake); retire at the first external install** |
| **R-469** | **[P3-LOW] REMOVE THE ENGINE-MAJOR RULE when Slice 4 (R-448) ships — a tracked act, not a lapse.** Since 2026-09-13 `app-catalog-felhom.eu` `CLAUDE.md` rules that *until the Update button takes a verified backup as its precondition, no template may move a database-engine image across a major version* (four MariaDB, eleven PostgreSQL services), and `scripts/check-engine-major.py` (fourth row of `catalog_gates.py`, run by `.githooks/pre-push` with the push range) refuses one, naming the rule and this expiry. **Why the rule:** every `mariadb:` sidecar now carries `MARIADB_AUTO_UPGRADE=1` (R-459), so a MariaDB major move CONVERTS the customer's datadir on the next Update; PostgreSQL converts nothing and refuses to start (R-463). Either way a customer-data event with no backup in front of it. **Honest limit, not re-filed:** the gate needs a parent commit and CI fetches at `--depth 1` — the R-452 gap — so on a shallow clone the runner skips it out loud and only the hook bites. **When R-448 ships:** delete the CLAUDE.md rule, the gate's row and the gate, in one commit that cites this row; then close this. **2026-09-13 — UNBLOCKED, NOT LIFTED.** R-448 shipped in controller v0.237.0/v0.238.0 (slice 4): an update now refuses without a restorable, proven Tier-2 copy, backs up first when it is stale, takes a safety dump, and holds an app that does not come up — the precondition this rule was waiting for. **The rule stays in force until someone deliberately removes it**, which is a separate act (and is worth weighing against R-475: an app with no Tier-2 copy cannot be updated at all, so the guard does not yet cover every app a major engine move would touch). | **READY — unblocked by R-448; rank P3-LOW; owner: CC (removal is a deliberate act)** | | **R-469** | **[P3-LOW] REMOVE THE ENGINE-MAJOR RULE when Slice 4 (R-448) ships — a tracked act, not a lapse.** Since 2026-09-13 `app-catalog-felhom.eu` `CLAUDE.md` rules that *until the Update button takes a verified backup as its precondition, no template may move a database-engine image across a major version* (four MariaDB, eleven PostgreSQL services), and `scripts/check-engine-major.py` (fourth row of `catalog_gates.py`, run by `.githooks/pre-push` with the push range) refuses one, naming the rule and this expiry. **Why the rule:** every `mariadb:` sidecar now carries `MARIADB_AUTO_UPGRADE=1` (R-459), so a MariaDB major move CONVERTS the customer's datadir on the next Update; PostgreSQL converts nothing and refuses to start (R-463). Either way a customer-data event with no backup in front of it. **Honest limit, not re-filed:** the gate needs a parent commit and CI fetches at `--depth 1` — the R-452 gap — so on a shallow clone the runner skips it out loud and only the hook bites. **When R-448 ships:** delete the CLAUDE.md rule, the gate's row and the gate, in one commit that cites this row; then close this. **2026-09-13 — UNBLOCKED, NOT LIFTED.** R-448 shipped in controller v0.237.0/v0.238.0 (slice 4): an update now refuses without a restorable, proven Tier-2 copy, backs up first when it is stale, takes a safety dump, and holds an app that does not come up — the precondition this rule was waiting for. **The rule stays in force until someone deliberately removes it**, which is a separate act (and is worth weighing against R-475: an app with no Tier-2 copy cannot be updated at all, so the guard does not yet cover every app a major engine move would touch). | **READY — unblocked by R-448; rank P3-LOW; owner: CC (removal is a deliberate act)** |
| **R-471** | **[P3-LOW] `observations_gate.py` reads the FIRST `Observations` section of `REPORT.md` and nothing after it, so an appended second section with an unmarked item passes — and the decoy that proves it has been reading "LIVE HOLE" at HEAD, unnoticed, because the decoy suite is run by hand.** MEASURED 2026-09-13 on a clean worktree of `4b2e560`: `python3 scripts/test_gate_decoys.py` → `FAIL: observations/R-419: decoy PASSED - LIVE HOLE (rc=0)`; the gate's own output shows it scanned `## 11. Observations — noticed, documented, NOT acted on` (3 items, all marked) and never reached the appended `## Observations` block the decoy planted. Whether the cause is "first heading wins" or a heading-shape filter is NOT established — only that the planted unmarked item was not seen. **Consequence:** a report with two observation sections gets the second one unchecked. **Two fixes, both owed:** scan every section whose heading contains `Observations`, and put `test_gate_decoys.py` where something runs it (it is the instrument for R-421 and nothing in `repo_gates.py` invokes it). | **READY — rank P3-LOW; owner: CC** |
| **R-473** | **[P2-MEDIUM] The `glance` catalog template crash-loops on EVERY fresh install.** MEASURED 2026-09-13 on demo-hp (deployed as a throwaway for the slice-4 live test): `restarts=13 status=restarting`, the log repeating `parsing config: reading /app/config/glance.yml: open /app/config/glance.yml: no such file or directory`, the `glance_config` volume empty. The image does not create a default config and the template seeds none. The install page reports the deploy as started and the card then shows a restarting app. Not a version problem: the template's current tag v0.8.5 has the same config requirement (the catalog walk that pinned it never did a fresh install). **Fix shape:** seed a minimal `glance.yml` the way `gokapi`'s catalog entry seeds `config.json` (memory `gokapi-headless-setup`), then prove a fresh install lands healthy. Evidence `audits/slice4-2026-09-13/live/02-glance-abandoned.txt`. | **READY — rank P2-MEDIUM; owner: CC** |
| **R-474** | **[P3-LOW] "Remove app" with *also delete backups* deletes only the `db-dumps` directory, and reports `volumes_removed: null` over a volume it DID remove.** MEASURED 2026-09-13 removing the glance throwaway on controller v0.237.0 (`remove_hdd_data:true, remove_backups:true`): response `{"removed":"glance","volumes_removed":null,"hdd_paths_removed":[],…}`; afterwards `docker volume ls` shows no glance volume, while `backups/primary/glance/{compose,manifest.json}` and the stack dir's `applied-compose.yml` remain. The router passes ONLY `backup.AppDBDumpPath(nsRoot, name)` (router.go, beside a comment that disk-tier backup "moved to the host agent" — stale since Tier 2 returned to the controller). So the recovery unit and any Tier-2 copy survive a removal that promised to delete backups, and the answer names no volume. Same class as R-442: the removal's answer does not describe what happened. (Removal also refuses a crash-looping app as "still running — stop it first", which is correct and was observed.) **REPRODUCED a second time the same day** removing the uptime-kuma throwaway: `volumes_removed: null`, and `backups/primary/uptime-kuma/{compose,manifest.json,volume-dumps}` plus `backups/secondary/uptime-kuma/recovery-unit` survived `remove_backups:true` (residue cleared by hand; `audits/slice4-2026-09-13/` `live/15-teardown.txt`). **REPRODUCED a third time 2026-09-13 (afternoon, v0.239.0)** removing the gokapi and actualbudget throwaways with `remove_backups:true`: both `backups/primary/<app>/` units survived (actualbudget's with its 74 KB volume tar), `volumes_removed: null` again, and both `app_backup` prefs remained (`10-teardown.txt`). It also fed R-478. | **READY — rank P3-LOW; owner: CC** |
| **R-476** | **[P3-LOW] The Mentések page names a Tier-2 copy's date from the unit MANIFEST, which moves only when the app's DEFINITION changes — so it can undersell a fresh copy by a day or more.** MEASURED on demo-hp 2026-09-13: bookstack's mirror held `bookstack-mariadb.sql` written 2026-09-13T00:30Z under a manifest dated 2026-09-12T02:15:29Z; the Tier-2 run succeeded at 01:30Z. A capture rewrites the manifest only when checksums, dump NAMES or controller version change, and nightly dumps keep their names. R-403 chose the package date so a PRESERVED package is never shown as fresh — correct — but for a normal run it is the older, flattering-in-reverse date. The update (slice 4) deliberately ages the copy by the last successful copy instead (`Tier2RestorePoint.ProvenCopyTime`), so the two can name different dates. Fix shape: record the unit's DATA time (newest dump mtime) in the manifest, or name `LastSuccess` when the leg was not preserved. | **READY — rank P3-LOW; owner: CC** | | **R-476** | **[P3-LOW] The Mentések page names a Tier-2 copy's date from the unit MANIFEST, which moves only when the app's DEFINITION changes — so it can undersell a fresh copy by a day or more.** MEASURED on demo-hp 2026-09-13: bookstack's mirror held `bookstack-mariadb.sql` written 2026-09-13T00:30Z under a manifest dated 2026-09-12T02:15:29Z; the Tier-2 run succeeded at 01:30Z. A capture rewrites the manifest only when checksums, dump NAMES or controller version change, and nightly dumps keep their names. R-403 chose the package date so a PRESERVED package is never shown as fresh — correct — but for a normal run it is the older, flattering-in-reverse date. The update (slice 4) deliberately ages the copy by the last successful copy instead (`Tier2RestorePoint.ProvenCopyTime`), so the two can name different dates. Fix shape: record the unit's DATA time (newest dump mtime) in the manifest, or name `LastSuccess` when the leg was not preserved. | **READY — rank P3-LOW; owner: CC** |
| **R-477** | **[P3-LOW] The update's Tier-3 lookup pays its full 15 s bound on demo-hp, and the bound shows up as a WARN about a DIFFERENT app.** MEASURED 2026-09-13 (controller v0.239.0, Scenario K): the `checking` phase ran 15:25:45 → 15:26:00, and the only line in the window was `[WARN] [offbox] inventory: size of kimai's newest snapshot unknown: offbox stats b4350226: signal: killed`. Cause: `UpdateRestorePoints` calls `OffsiteInventoryList`, which runs one `snapshots` AND one `stats` per app; the update needs only the snapshot times, and the 15 s context killed a `stats`. **Consequence:** every update with no fresh Tier 2/Tier 1 copy waits up to 15 s before backing up, and the operator log blames kimai for an update of actualbudget. **Fix shape:** a snapshots-only lookup for the update path (no sizes), so the bound is a guard and not the normal duration. `audits/rulings-r472-r475-2026-09-13/04-K-no-unit-anywhere.txt` | **READY — rank P3-LOW; owner: CC** |
| **R-478** | **[P3-LOW] A recovery unit LEFT BEHIND by a removed install counted as the fresh Tier-1 restore point of a reinstall.** MEASURED 2026-09-13 (v0.239.0, Scenario H): gokapi had been removed earlier; its unit (manifest 06:59:17Z) survived (R-474). A new gokapi was deployed at 15:30:46Z and updated at 15:31:10Z: `precondition met — Tier 1 (own recovery unit) copy from 2026-09-13T06:59:17Z (8h32m0s old)`. That unit belonged to the OLD install — a different `app.yaml` (subdomain, generated password). The capture sweep rewrote it only at 15:34:22Z. **Consequence:** in that window a failed update would name, and a restore would apply, the previous install's definition. **Fix shape:** R-474 deleting the unit on removal closes most of it; independently, a unit whose manifest predates the stack's current deploy should not count. `05-H-gokapi-tier1.txt`, `06-F-setup-unit-rewritten.txt` | **READY — rank P3-LOW; owner: CC** |
| **R-479** | **[P2-MEDIUM] For an app whose data is a bind mount, the Tier-1 unit holds settings only — so the route back a Tier-1 hold names restores the definition and NOT the data.** MEASURED 2026-09-13 restoring gokapi from „helyi” after a held update: `a beállítások visszaálltak … FIGYELEM: ez a mentés csak a beállításokat tartalmazta, adatot nem`. The restore message is honest; the HOLD sentence („Visszaállítható … saját meghajtó”) does not say it. The ruling (R-475) accepts any tier, in the order 2, 1, 3 — so for such an app a fresh Tier-1 unit is chosen ahead of an off-site snapshot that WOULD carry the data. **Consequence:** an update whose migration rewrote bind-mounted data has no data route back through the copy it named. **Decision-shaped:** either Tier 1 counts only for apps whose unit carries their data (DB dump / volume tar), or the hold sentence says "settings only" for that case. `07-F-hold-names-own-drive.txt`, `08-restore-from-helyi.txt` | **WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: VIKTOR rules, CC implements** | | **R-479** | **[P2-MEDIUM] For an app whose data is a bind mount, the Tier-1 unit holds settings only — so the route back a Tier-1 hold names restores the definition and NOT the data.** MEASURED 2026-09-13 restoring gokapi from „helyi” after a held update: `a beállítások visszaálltak … FIGYELEM: ez a mentés csak a beállításokat tartalmazta, adatot nem`. The restore message is honest; the HOLD sentence („Visszaállítható … saját meghajtó”) does not say it. The ruling (R-475) accepts any tier, in the order 2, 1, 3 — so for such an app a fresh Tier-1 unit is chosen ahead of an off-site snapshot that WOULD carry the data. **Consequence:** an update whose migration rewrote bind-mounted data has no data route back through the copy it named. **Decision-shaped:** either Tier 1 counts only for apps whose unit carries their data (DB dump / volume tar), or the hold sentence says "settings only" for that case. `07-F-hold-names-own-drive.txt`, `08-restore-from-helyi.txt` | **WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: VIKTOR rules, CC implements** |
| **R-480** | **[P2-MEDIUM] After a successful restore, the app card still shows the failed update's sentence — which says the running app „leállítva marad”.** MEASURED 2026-09-13 on demo-hp (v0.239.0): the restore cleared the hold (`restore hold CLEARED for gokapi`), gokapi ran healthy, `hold_reason` was empty — but `update_phase=failed` and `update_error` stayed, and `GET /stacks` rendered the sentence inside gokapi's card (control: absent from actualbudget's card). It survived the app's removal too (`not_deployed … phase=failed err=…`). Present since v0.238.0's page; this morning's restore walk did not check the card text. **Fix shape:** a successful restore (and a removal) clears the stack's last update outcome. `09-card-after-restore.txt`, `10-teardown.txt` | **READY — rank P2-MEDIUM; owner: CC** | | **R-481** | **[P2-MEDIUM] There is no scratch guest on demo-hp, so the nightly rotation cannot restore a throwaway "into a scratch guest", and the nine standing apps cannot be tonight's throwaway at all.** MEASURED 2026-09-13 (`pct list` / `qm list` on demo-hp: only 9201). The rotation brief needs a second controller guest for two steps — the cross-guest restore, and a throwaway deploy of an app that is already standing on 9201 (the stack name collides, and the standing apps may not be touched). Every earlier cross-guest walk built a whole appliance from the published ISO (VM 323/325, hours each) and enrolled it as a new customer; a `pct clone` of 9201 would carry demo-hp's identity, tunnel and hub enrolment. **Decision-shaped:** which route makes the scratch guest — a persistent second LXC on demo-hp born from the golden template and enrolled as its own customer (`nightly-scratch`), or an ISO-built appliance per night. Until then the rotation restores in place (remove → restore from the unit on the same guest) and skips the standing nine (`runbooks/nightly-rotation.md`). | **WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: VIKTOR rules, CC implements** |
| **R-483** | **[P3-LOW] `adventurelog` v0.12.1: a photo upload through the app's own API proxy fails 500 from every non-browser client — whether a household can upload photos in a browser is UNKNOWN and cannot be measured here.** MEASURED 2026-09-13 on demo-hp (throwaway `travelnight`): `POST /api/images` (multipart: `image`, `location`, `is_primary`) through the public origin returns `{"error":"Internal Server Error"}` from the **frontend** (the backend log shows no request at all); the frontend container logs `RequestContentLengthMismatchError: Request body length does not match content-length header` from its undici forwarder. Same result with an ASCII filename, an accented one, urllib, curl `-F`, and curl with `Transfer-Encoding: chunked`. Everything else on the API — sign-up, the frontend's login form, collections, locations, visits, notes, edit, delete — works. **What is NOT established:** whether the app's own browser page uploads succeed (a browser's FormData body may satisfy the proxy). Per `CLAUDE.md`, that is a manual click-through: open `https://<sub>.<domain>`, add a photo to a place. **Not ours to fix in the template** (a frontend proxy bug); a newer catalog pin is a version promotion, not tonight's. Evidence: `audits/nightly-2026-09-13-adventurelog/03e-photo-500.txt`. | **WAITING-ON-OPERATOR — needs a browser click-through; rank P3-LOW; owner: VIKTOR checks, CC re-files** |
| **R-487** | **[P2-MEDIUM] A removed app whose backups were kept is listed on NEITHER backup page, so the restore that brings it back has no button — the customer's remove-by-mistake route exists only as an endpoint.** MEASURED 2026-09-13 on demo-hp (nightly rotation, `adventurelog` removed with backups kept, unit + mirror on disk): `GET /backups/apps` and `GET /backups/restore` contain the string `adventurelog` zero times; `POST /backup/restore stack_name=adventurelog snapshot_id=helyi` then restored it in 22 s with the data byte-identical. Cause: `buildAppBackupRows` walks `status.AppDataInfo` = `DiscoverAppData` over DEPLOYED stacks only. The off-site list had exactly this defect and was fixed by keying it on the store (R-237, v0.204.0); the local and Tier-2 lists were not. **Fix shape:** list every app with a recovery unit on a registered drive (`ListRestorePoints` over the primary dirs), marking removed ones „eltávolítva — visszaállítható"; the unit restore already reinstalls (R-253). Not a design reversal — the same rule R-237 set. Evidence: `audits/nightly-2026-09-13-adventurelog/05b-restore-tier1.txt`. | **READY — rank P2-MEDIUM; owner: CC** |
| **R-488** | **[P3-LOW] `go test ./internal/backup` takes 5½ minutes: 89 off-site tests wait on real clocks.** MEASURED 2026-09-13 (`-v` timings, run alone: 581 tests, 333 s in total, 89 of them ≥ 1 s — `TestOffbox*`, `TestOffbox3a*`, `TestOffboxRun*`, `TestR4xx*` reconstitute fixtures at 3–8 s each). The controller's per-commit gate is therefore ~6 minutes, most of it sleeping, and two concurrent runs of the package looked like a hang. **Fix shape:** the waits are `waitForHealthy`-style polls and retry back-offs with fixed durations; make them seams the fixtures shorten (the R-457 rule: one clock). Not a correctness defect. | **READY — rank P3-LOW; owner: CC** |
| **R-489** | **[P3-LOW] `POST /api/stacks/{name}/remove` reports `volumes_removed: null` over named volumes it DID remove.** MEASURED 2026-09-13 on demo-hp five times (gokapi, actualbudget, adventurelog ×2, glance): `docker compose down --volumes` removed the app's named volumes (`docker volume ls` count 2 → 0) and the response carried `"volumes_removed":null`. The customer's confirmation dialog therefore cannot say what it deleted. Split out of R-474 (closed in v0.240.0 for the backups half). **Fix shape:** list the volumes before `down --volumes`, diff after, and report the difference (`[]` when none, never `null`). | **READY — rank P3-LOW; owner: CC** |
<!-- DUE-CHECKS-BEGIN — machine-readable. Parsed by scripts/due_checks_gate.py. <!-- DUE-CHECKS-BEGIN — machine-readable. Parsed by scripts/due_checks_gate.py.
One row per dated check. The R-number must have a row above. Dates are UTC. One row per dated check. The R-number must have a row above. Dates are UTC.
documentation/runbooks/nightly-rotation.md
+71
View File
@@ -0,0 +1,71 @@
# Nightly rotation — "be a customer for the night" on demo-hp
One app per night, as a throwaway deploy on demo-hp, through the product's own endpoints: install,
real data through the app's front door (never a volume or database directly — R-156), ten minutes of
household use, backup through the backups page's endpoint, restore and read the data back, the
guarded Update if the catalog offers one, removal with "delete my data", and a clean drive. Every
surprise is a register row before the next step. An app with no non-browser front door is recorded,
not faked. Tick with the date; the next night takes the first unticked app.
**The nine standing apps of demo-hp come first, and they cannot be tonight's throwaway on guest 9201:**
a throwaway deploy shares the stack name with the standing one, and the standing apps may not be
touched. They are ticked only when a scratch guest exists to host the throwaway (R-481).
## Standing on demo-hp (blocked on a scratch guest — R-481)
- [ ] bentopdf
- [ ] bookstack
- [ ] calibre-web
- [ ] docmost
- [ ] kimai
- [ ] opengist
- [ ] paperless-ngx
- [ ] privatebin
- [ ] romm
## The rest of the catalog
- [x] actualbudget — 2026-09-13: no non-browser front door (the client talks a sync protocol, no REST); deploy / update / remove were exercised the same day for R-475 (`audits/rulings-r472-r475-2026-09-13/`). Recorded, not faked.
- [x] adventurelog — 2026-09-13: full walk (install, sign-up + trip data through the API, backup now + Tier 2, remove-with-data-kept → second-drive restore REFUSED (R-486) → own-copy restore byte-identical, same-version guarded Update, remove-with-backups left 484 MB behind (R-474)). Photo upload needs a browser (R-483). Rows: R-482..R-487. `audits/nightly-2026-09-13-adventurelog/`.
- [ ] audiobookshelf
- [ ] calcom
- [ ] claper
- [ ] code-server
- [ ] crafty-controller
- [ ] emby
- [ ] ghost
- [ ] gitea
- [ ] glance
- [ ] gokapi
- [ ] grafana
- [ ] gramps-web
- [ ] home-assistant
- [ ] homebox
- [ ] homepage
- [ ] immich
- [ ] jellyfin
- [ ] komga
- [ ] mealie
- [ ] n8n
- [ ] navidrome
- [ ] nextcloud
- [ ] onlyoffice
- [ ] outline
- [ ] papra
- [ ] plant-it
- [ ] plex
- [ ] radarr
- [ ] rallly
- [ ] recipe-importer
- [ ] seerr
- [ ] sonarr
- [ ] sparkyfitness
- [ ] tandoor
- [ ] termix
- [ ] uptime-kuma
- [ ] vaultwarden
- [ ] vikunja
- [ ] wanderer
- [ ] wger
- [ ] wishlist
- [ ] zipline
documentation/runbooks/target-selection.md
+3 -3
View File
@@ -99,10 +99,10 @@ felhom-pve, and **moved off DooPlex**. This page exists because that ruling sat
**This is the default answer to "where do I run this".** **This is the default answer to "where do I run this".**
- **Care:** `local-lvm` is a **thin pool, over-subscribed** (~144 GiB allocated over ~54 GiB) backing - **Care:** `local-lvm` is a **thin pool, over-subscribed** (~144 GiB allocated over ~54 GiB) backing
live guest 9201 — filling it corrupts every guest. Put VM disks on a dir storage at live guest 9201 — filling it corrupts every guest. Put VM disks on a dir storage at
**`/mnt/nvme-1tb`, at its root** (a subdirectory fails the agent's `exactMount` check → storage reads **`/mnt/hdd_1`, at its root** (R-461, measured 2026-09-13: there is NO `/mnt/nvme-1tb` on either box — demo-hp's 1 TB NVMe is `nvme0n1` mounted at `/mnt/hdd_1`, the enrolled user-data drive and `felhom-backup` target, so "the NVMe" and "the data drive" are one disk) (a subdirectory fails the agent's `exactMount` check → storage reads
`disconnected` forever). **That warning is about one storage, not the box.** `/mnt/nvme-1tb` is also `disconnected` forever). **That warning is about one storage, not the box.** `/mnt/hdd_1` is also
the `felhom-backup` target and the enrolled user-data drive, so remove scratch storages when done. the `felhom-backup` target and the enrolled user-data drive, so remove scratch storages when done.
- **Forbidden:** do not destroy or unblock **`drill-r50` (VM 300)** — the only drift fixture (R-93). - **Forbidden:** do not destroy or unblock **`drill-r50` (VM 300)** — the only drift fixture (R-93). **Measured 2026-09-13 (R-461): `qm list` is EMPTY on both demo-hp and demo-felhom — the VM does not exist anywhere, so the fence currently protects nothing and R-93's premise is gone. The fence stays as written for the day someone rebuilds it; do not read its presence here as evidence the fixture exists.**
(Access: the docs say no baked SSH key and G1 break-glass, but a key authenticated on 2026-07-31 — (Access: the docs say no baked SSH key and G1 break-glass, but a key authenticated on 2026-07-31 —
**R-129**, unresolved.) **R-129**, unresolved.)
scripts/CHANGELOG.md
+10
View File
@@ -1,3 +1,13 @@
## observations_gate.py reads EVERY observations section (2026-09-13, R-471)
`observation_items` returned the first `Observations` heading it found and stopped, so a report
carrying an ordinary `## 11. Observations …` section and, appended below it, a second `## Observations`
block with an unmarked item PASSED. The R-419 decoy plants exactly that shape and had been reading
**LIVE HOLE** at HEAD, because the decoy suite is run by hand. Now every observations section is
read and the items are pooled; the heading reported is all of them joined with ` + `. Red-proof:
the old parser exits 0 on the decoy, the new one refuses it (`documentation/audits/v0240-2026-09-13/rp-R471.txt`).
All 12 decoys behave; the felhom.eu and controller reports still pass.
## golden_currency_gate.py docstring corrected: the floor does NOT carry a release between bakes (2026-09-13, R-472) — NOT A RELEASE ## golden_currency_gate.py docstring corrected: the floor does NOT carry a release between bakes (2026-09-13, R-472) — NOT A RELEASE
Docstring only; the gate's behaviour is unchanged. This morning's entry stated that under the weekly Docstring only; the gate's behaviour is unchanged. This morning's entry stated that under the weekly
scripts/observations_gate.py
+49 -30
View File
@@ -141,40 +141,59 @@ def known_rows(paths):
return rows return rows
def observation_items(text): def observation_sections(text):
"""(items, heading) — each item is (number, its full text). heading is None when absent.""" """Every observations section in the file: [(heading_line, body_lines)], in document order.
R-471 (2026-09-13): this used to return the FIRST matching heading only, so a report carrying an
ordinary `## 11. Observations …` section and, appended below it, a second `## Observations` block
with an unmarked item PASSED — and the R-419 decoy, which plants exactly that shape, had been
reading LIVE HOLE at HEAD without anyone running the decoy suite. Every section is read now.
"""
lines = text.split("\n") lines = text.split("\n")
start = None sections = []
depth = 0 i = 0
for i, line in enumerate(lines): while i < len(lines):
m = HEADING_RE.match(line) m = HEADING_RE.match(lines[i])
if m: if not m:
start, depth = i, len(m.group(1)) i += 1
break continue
if start is None: depth = len(m.group(1))
body = []
j = i + 1
while j < len(lines):
hm = re.match(r"^(#+)\s", lines[j])
if hm and len(hm.group(1)) <= depth:
break
body.append(lines[j])
j += 1
sections.append((lines[i].strip(), body))
i = j
return sections
def observation_items(text):
"""(items, heading) — each item is (number, its full text), over EVERY observations section.
heading is None when there is no section; several headings are joined with " + "."""
sections = observation_sections(text)
if not sections:
return None, None return None, None
body = []
for line in lines[start + 1:]:
hm = re.match(r"^(#+)\s", line)
if hm and len(hm.group(1)) <= depth:
break
body.append(line)
items, cur = [], None items, cur = [], None
for line in body: for _heading, body in sections:
m = ITEM_RE.match(line) for line in body:
if m: m = ITEM_RE.match(line)
if cur: if m:
items.append(cur) if cur:
cur = [m.group(1), m.group(2)] items.append(cur)
elif cur is not None: cur = [m.group(1), m.group(2)]
if line.strip() == "" and cur[1].endswith("\n\n"): elif cur is not None:
continue if line.strip() == "" and cur[1].endswith("\n\n"):
cur[1] += "\n" + line continue
if cur: cur[1] += "\n" + line
items.append(cur) if cur:
return items, lines[start].strip() items.append(cur)
cur = None
return items, " + ".join(h for h, _ in sections)
def main(): def main():