felhom.eu/documentation/audits/update-night-2026-09-21/PROGRESS.md

UPDATE NIGHT 2026-09-21 — progress log (one line per finished step)

Evidence dir: documentation/audits/update-night-2026-09-21/ A resuming session reads THIS FILE FIRST and never repeats a finished step.

time (CEST)	step	verdict	evidence
20:07	P0.1 floor → 0.261.0 (declared MinAgent 0.131.0)	PROVEN — both demo boxes arrived in 13 s; hub managed floor SERVED … from declared (golden 0.258.0); drill-r50 stays held (agent 0.129.0 < 0.131.0, host DOWN)	01-floor-pre.txt, 02-floor-save.txt
20:08	P0.2a drill repo admin/app-catalog-drill created (private)	DONE — Gitea token lacks write:user, so POST /api/v1/repos/migrate was used instead of /user/repos; main = f5f6a152b513 = live	03-drill-repo.txt
20:10	P0.2b 9202 repointed at the drill repo	PROVEN — git.repo_url ALONE IS INERT: gitCloneOrPull only clones when .git is absent, else fetches from the old origin. Cache dir had to be removed too. Config saved as controller.yaml.pre-update-night	04-9202-config-pre.txt, 05-9202-follows-drill.txt
20:12	P0.2c positive + negative controls	PROVEN — drill bump uptime-kuma 2.4.0→2.5.5 shows on 9202 as „Frissítés elérhető — ma" / "Update available — today"; live catalog still f5f6a152b513 with pin 2.4.0; both real boxes' caches still at f5f6a15. R-607 fired again (sync said „nincs változás"). App route is /apps/<n>, not /app/<n>	06-drift-rerun.txt, 07-positive-control.txt
20:07	Drift re-run	CONFIRMED — 66 pins, 46 behind, 39 within-major, 7 across-major — the brief's numbers hold exactly	06-drift-rerun.txt
20:13	P0.4 capacity measured	OK — 9202: 26 GB RAM (22 free), docker root on mp0 with 56 GB free, drive 872 GB. demo-hp / is 86% but holds neither. Brief's capacity claim HOLDS	08-capacity.txt
20:18	P0.3 throwaway image store	PROVEN — registry:2 on 9202 127.0.0.1:5000; drill/glance:1.0.0 = real v0.8.6 retagged, :1.0.1 = starts/stays up/never serves, :1.0.2 absent (404). CompareImageRefs DOES order host:port/ refs — 4 positive + 1 negative control, run not read; tmp test deleted, tree clean	09-image-store.txt
20:20	EDGE privatebin 2.0.5 -> 2.0.6 (file-leg)	PROVEN — 15.4 s; seed read back both sides; all four observables agree	apps/privatebin/
20:25	EDGE docmost 0.95.0 -> 0.96.0 (db-postgres, engine constant)	PROVEN — 103.5 s; seeded account authenticated after; all four observables agree	apps/docmost/
20:28	EDGE bookstack 26.05.2 -> 26.05.5 (db-mariadb, engine constant)	PROVEN — 45.1 s; artisan readback with its own negative control; DATABASE HALF ONLY (R-460)	apps/bookstack/
20:37	FINDING R-618 tandoor probe port	DEFECT, measured — probe 8080, app listens on 80 only; docker healthy + front door 200 + controller unhealthy. 53-template sweep run: wger suspected, adventurelog cleared	10-probe-port-sweep.txt
20:38	FINDINGS R-615/616/617/619	filed — repo_url inert; catalog token plaintext in the clone; Gitea migrate endpoint; type: password mandatory though the wire says optional	NEW-ROWS.md
20:39	EDGE actualbudget 26.7.0 -> 26.9.0	PROVEN — 19.5 s	apps/actualbudget/
20:44	EDGE navidrome 0.63.2 -> 0.64.0 (file-leg, HDD_PATH)	PROVEN — 11.3 s	apps/navidrome/
20:46	EDGE audiobookshelf 2.35.1 -> 2.36.1 (file-leg)	PROVEN — 23.6 s	apps/audiobookshelf/
20:54	EDGE adventurelog v0.12.1 -> v0.13.0 (db-postgis)	FAILED — the most valuable result so far. Nine migrations applied OK, then the app never bound its port; held after the full 5-min wait; the sentence names tier, date and what the copy holds. Rows R-621 (the hold destroys the failure evidence) and R-622 (do not promote this edge)	apps/adventurelog/
20:59	Harness code pushed to the catalog — 4 fixtures + edges U1..U7	DONE, gates green — live catalog main moves f5f6a152b513 -> 4463243f2e09, scripts/ ONLY, zero image: lines (the teardown diff still expects every image line identical). Harness RUNS owed	app-catalog-felhom.eu@4463243f2e09
20:59	image reclaim BY NAME (no prune)	34 unused images removed by exact reference; 40 GB -> 55 GB free	reclaim.sh
21:08	CI check for the catalog push	GREEN — job id=830, name='gates', status='completed', conclusion='success', matched on head_sha=4463243f2e09. Confirms CLAUDE.md's warning: page 1's newest id was 52, the real newest was 830 — job ids are NOT page-ordered	scratchpad/ci.sh
21:12	Observation, NOT a new row — an app with an HDD_PATH cannot have its DATA removed on 9202	R-442's guard working as designed: /api/disks answers agent not configured on this guest, so the drive path cannot be RESOLVED and the removal is refused with the app kept rather than half-deleted. The household's other choice — remove the app, KEEP the data — is accepted. The harness now takes that route and tidies its own directories by name at teardown	apps/navidrome/, apps/romm/
21:13	FINDING R-623 — the unattended caller turned every SUCCESS into a timeout	Read before use, not after: follow() read update_phase/updating off the API ENVELOPE, so both were always None, every followed update hit the 900 s timeout and was then marked never-press-again. Fixed in that file before B1 relied on it. The earlier night missed it because its only follow() pass was the one whose log was lost	update-arc-gaps-2026-09-21/unattended-caller.py
21:18	Interim commit pushed — felhom.eu da20722e76a7	Phase 0 + Phase 1 evidence off the machine before Phases 2-5 (R-320). repo_gates.py --fast: all 15 OK. Includes the two instrument fixes (api-recipe route, unattended-caller follow())	felhom.eu@da20722e76a7
21:18	R-618 RAISED TO P1 by a live measurement	tandoor 2.6.13→2.6.15 was serving HTTP 200 on the NEW version at 21:18:28 while the update sat in verifying, because the probe names port 8080 and the app listens on 80. The health wait then times out and failAndHold STOPS the working app. A wrong port turns every successful update into an outage plus a needless restore	14-tandoor-serving-while-verifying.txt
21:22	EDGE tandoor 2.6.13 -> 2.6.15 (db-postgres)	FAILED — and the failure is OURS, not the app's. The new version was SERVING 200 with a green docker healthcheck at four samples across five minutes; the controller's probe names port 8080 and the app listens on 80, so verifying timed out and failAndHold STOPPED a working app. The four observables disagree exactly as 09 §5.2 predicts. R-618 is now P1	14-tandoor-serving-while-verifying.txt, apps/tandoor/
21:35	failwalk adventurelog — the household's WAY OUT, on a real failed edge	PROVEN — the restore the hold sentence names („helyi", 1 copy offered) ran in 75 s; app back on v0.12.1, all three containers running, hold CLEARED, state running. R-606 CONFIRMED on the hold sentence itself (English page, Hungarian sentence) with positive and negative controls	apps/adventurelog/failwalk.json, 15-r606-*.txt
21:36	Observation after the restore	The hold sentence is GONE from the card (R-480 working) and the app reads „Fut". The badge briefly read „Frissítés elérhető — 65 napja" rather than „ma", because the restore brought back the recovery unit's own .felhom.yml with its pre-drill catalog_since; the syncer copies that file verbatim every cycle, so it heals within one sync. Recorded as an observation, not a row	apps/adventurelog/failwalk.json
21:37	failwalk tandoor — the way out of the WRONG-PROBE hold	PROVEN — restore in 32.4 s, hold cleared, app running, data reads back. Note what this means for R-618: the household's route back works, so nothing is LOST — but they had to walk it for an update that had actually succeeded	apps/tandoor/failwalk.json
21:40	PHASE 2.1 — MariaDB 11.6 -> 12.3 through the REAL Update button	PROVEN, a first. All four SPIKE-r459 observables: datadir 11.6.2->12.3.3; the engine's own check says „already upgraded … no need to run mariadb-upgrade again"; the entrypoint says „Major version upgrade detected … Check required!" then started and finished (NOT the skipped due to $MARIADB_AUTO_UPGRADE line R-459 feared); the engine's own pre-upgrade backup system_mysql_backup_11.6.2-MariaDB.sql.zst 631 905 B. Seeded Nextcloud account read back. All four version observables agree	16-phase2.1-mariadb-major.md, apps/nextcloud-engine-mariadb/
21:45	PHASE 2.2 — PostgreSQL 16 -> 17 through the real Update button	FAILED, exactly as R-463 predicted and nobody had measured. The update ended in 5.1 s, the app was stopped and held, and pinned named postgres:17-alpine while installed still said 16 and nothing was running — the brief's own question answered. The restore then brought it back in 29.1 s (hold CLEARED, health probe 200)	apps/docmost-engine-postgres/
21:45	R-621 demonstrated itself on this leg's KEY artefact	The engine's refusal line was GONE before any probe could read it — failAndHold had removed the container, and the controller log does not carry it either. Reproduced independently instead (R-320)	18-postgres-refusal-reproduced.txt
21:45	My own first reproduction was WRONG, and is kept labelled	The volume lookup returned empty -> the copy was empty -> postgres:17 initialised a FRESH datadir and started happily. A blank PG_VERSION should have stopped the step and did not. Rewritten so every step proves itself first. The accidental result (16 refusing a 17 datadir, verbatim) is real and kept	17-postgres-refusal-reproduced.txt
21:56	B1 — THE UNATTENDED HOLD. MEASURED, and it is the thing 09 Q4 has never had.	The caller pressed ONCE with nobody watching; the drill image started, stayed up and never served; safety-dump -> verifying -> held after 312.9 s; then pass 2 and pass 3 pressed NOTHING — outcomes={'glance': ('held', 312.9)} never_again=['glance']. The hold sentence names the tier, the date and what the copy holds. My follow() fix (R-623) was load-bearing: without it this would have read timeout after 900 s	bad-days/B1-unattended-hold/
21:57	B2 — the new tag cannot be pulled	Exactly §6.1 Scenario E. phase=failed in 1.0 s, hold=None, „Az új verzió letöltése nem sikerült, ezért a frissítés elmaradt. Az alkalmazás a korábbi verzióval fut tovább." Nothing held; the app keeps running the old version	bad-days/B2-pull-fails/
21:57	B3 — Update pressed while a backup runs	409 reason='busy' on six consecutive presses with „A frissítés most nem indítható: mentés/visszaállítás folyamatban. Próbáld újra, ha befejeződött." — the transient reason an unattended caller needs (R-609)	bad-days/B3-busy-during-backup/
21:58	B7 — the 2 GB disk floor	REFUSED correctly: filled to 1.4 GB free, 409 reason='disk', „Nincs elég szabad hely a frissítéshez: 1.4 GB szabad… legalább 2 GB szükséges." Nothing moved. Space released immediately after	bad-days/B7-disk-and-memory-refusals/
21:58	CORRECTION to R-606 — the refusals are NOT English	Three refusals requested with ?lang=en, Hungarian request as control: held, not_deployed and disk all come back identical Hungarian. The pipe exists (langFor honours ?lang=) but the sentences are frozen string constants, so routing them through errText changes nothing	20-refusals-in-english.txt
22:00	B9 — a frozen app gets a newer .felhom.yml	§5.4's asymmetry confirmed live; no false alarm — 10 samples, all running/200. The reason NARROWS R-458: type: http treats any response as healthy, so a wrong path is invisible; the risk is real only for type: api + expect	21-B9-frozen-app-newer-felhomyml.md
22:00	B8 skipped here — docmost was swept by B1's precondition	re-run in phase2_redo.sh with a freshly deployed docmost	phase2_redo.sh
22:02	B4 (two) — two Updates within one second	THERE IS NO SINGLE-FLIGHT. Both accepted 202 within 0.268 s, and 2 of 2 ran in flight at the same time. Both ended done, no error, no hold, both pins advanced. That is a real Slice-6 design input: an unattended caller pressing N apps would run N updates at once	bad-days/B4-concurrent-updates/two.json
22:06	B4 (five) — five Updates within half a second	5 of 5 ran concurrently and ALL ended honest: accepted within 0.448 s, every one done, no error, no hold, every pin advanced, whole batch ~30 s. So the box does not serialise updates at all — and on this box it coped. Memory samples in the record	bad-days/B4-concurrent-updates/five.json
22:08	B5 (backing-up cut) — the phase nobody had cut in	BEHAVED. Cut at backing-up +0.027 s (pct stop returned 9.65 s later — the usual instrument limit, stated). After boot the box said so ITSELF: „crash recovery: an app-data backup (volume dump) … was interrupted and left 1 app(s) stopped — restarting them: [privatebin]" and „update recovery: privatebin was interrupted in backing-up". The pin did NOT move, the app runs, and a fresh paste seeded+read back	bad-days/B5-backing-up/
22:08	B5 (safety-dump cut) — MISSED, recorded as a miss	The phases went backing-up -> pulling -> failed in 0.473 s and safety-dump was never observed, so the plug was never pulled. Recorded as a MISS, not as a pass. To be retried with a genuine pending edge	bad-days/B5-safety-dump/
22:09	Phase 4 — the morning after	Every one of the 10 badges is TRUE (refs equal <-> „Naprakész", refs differ <-> „Frissítés elérhető"). Q4's four promises all scored True on the held app's page. zipline shows the household „Nem egészséges — URL nem elérhető" while running — R-618 in the household's own words	bad-days/P4-morning-after/
22:09	B6 — the way out FORWARDS: REFUSED	A held app met a FIXED newer version. Badge: „Frissítés elérhető — ma" in both languages, button offered; press -> 409 reason='held'. Correct per §6.1 (only a restore lifts a hold) but the page invites what the button refuses — the exact inconsistency R-524 removed for the Ahead case. Filed as R-625	bad-days/B6-way-out-forwards/
22:16	PHASE 2.3 — the PostgreSQL conversion REHEARSAL, costed (Q5)	WORKED end to end. 49 MB / 48 tables: dump with 16 2.6 s / 132 201 B; fresh 17 + replay 6.5 s / 48 tables; the app said „Database connection successful"; the seeded account read back on 17; total 155.9 s, of which ~9 s is engine work. Two benign ERROR lines named. pg_upgrade NOT run — it needs an image that does not exist here	24-Q5-postgres-conversion-costed.md
22:18	wger CONFIRMED — R-618's last candidate closes	probe type: http port: 80; inside the container port 80 refused, port 8000 ANSWERED; docker health green; front door 302; the box says unhealthy. Three confirmed instances now: tandoor, zipline, wger — and the cheap static rule finds all three with one false positive out of 53	22-wger-probe-measured.txt
22:19	B5 (safety-dump cut) — the second early phase, RETRIED and HIT	Cut at safety-dump +0.023 s. After boot: no recovery line and no journal (unlike the backing-up cut, which produced both), the pin did NOT move, the app runs, a fresh paste seeded and read back, and the card shows no interrupted sentence at all. No half-written backup artefact on either cut	25-B5-the-two-early-cuts.md
22:20	Bonus proof across a REAL power cut	The boot sweep met the held glance after an unclean shutdown and deliberately left it alone: „is a boot orphan by intent but is HELD … NOT starting it; whatever is holding it owns its recovery". §6.1's three-unattended-paths claim, proven across a power cut	25-B5-the-two-early-cuts.md
22:22	mealie v3.20.1 -> v3.27.0 (db-postgres)	PROVEN — the edge that failed twice on instrument problems, walked cleanly on the third	apps/mealie/
22:23	PHASE 1+2 COMPLETE	21 edges attempted: 14 proven, 3 failed, 4 inconclusive. Ten of the fourteen printed a verbatim migration line. Up from the three apps this project had ever measured	summarise.py
22:27	PHASE 5 — teardown, three layers plus Gitea	Machine: every throwaway app removed through the product; three with an HDD_PATH refused at „remove with data" (R-442, correct) and removed keeping it; registry, drill images and drill volumes gone BY NAME, never pruned; controller.yaml restored and git.repo_url read back as the LIVE catalog with an empty token; only felhom-controller, filebrowser, traefik left. Host: no harness LXC was created, so none was destroyed; guest 9201 untouched. Hub: nothing provisioned; floor 0.261.0. Gitea: drill repo KEPT, private, reset to live main; every image: line IDENTICAL	teardown/
22:28	The teardown found the night's LAST defect	A navidrome container the product's own removal left behind, resurrected by Docker's restart policy at a power cut, invisible to every sweep keying on deployed. Removed by name. R-626	26-removed-app-came-back.txt
22:4x	Documents, register, CI	12 new rows (register 303 -> 315), 8 existing rows updated, 09 §3b answered for Q2–Q6, §6.4 re-costed, §6.5 (the drill method) added, limitation 8 replaced, capability map widened, rotation 2 -> 16 fully-walked apps, STATUS morning note. CI green by id: catalog job 830, felhom.eu job 870, both success	felhom.eu@8d786f7940b0
—	unproven.py --summary did NOT move	55 claims, 35 not walked — unchanged. Tonight measured the UPDATE arc, which that file tracks as one claim already marked walked; no claim's status changed, so no number moved. Stated because the checklist asks	scripts/unproven.py