diff --git a/REPORT-probe-fix-2026-09-22.md b/REPORT-probe-fix-2026-09-22.md new file mode 100644 index 00000000..f1c8eee5 --- /dev/null +++ b/REPORT-probe-fix-2026-09-22.md @@ -0,0 +1,44 @@ +# REPORT — probe fix, gate, promotion train, 2026-09-22 + +**The full record is `documentation/audits/PROBE-FIX-2026-09-22.md`.** This file is the session +report. The shared `REPORT.md` is deliberately not touched (two sessions in this repo clobber it). + +## Not done, or changed from the brief + +1. **FIFTEEN versions moved, not fourteen** — tandoor was re-walked today and became the fifteenth. +2. **I nearly dropped `nextcloud`'s MariaDB engine move on a wrong assumption.** Running + `check-engine-major.py` against that exact commit ALLOWED it by name under R-469. It moved. + Recorded as a decision the operator may reverse. +3. **The first push of the moves FAILED CI** (job 877, `15d7c2b`): no PyYAML on the runner, the new + gate answered INCONCLUSIVE. Fixed with a degraded mode + five decoys; job **878** = success. +4. **bookstack's phase trace on demo-hp is incomplete** — my own 115 s probe run pressed the Update + and timed out mid-flight. Said plainly rather than presented as a full trace. +5. **The four apps were pressed on ONE demo box, not two.** `demo-felhom` has only `opengist` + installed. I did not install four apps on it to satisfy the instruction. + +**No brief claim turned out wrong.** All three probe faults were verified against the files before +any edit and all three were exactly as stated. + +## What ran + +- **Part 1** — the three probes fixed in one commit; red-proofed live on 9202 in **both** + directions through the product; tandoor's failed edge re-walked and now `done` at +41.1 s; a new + `--fast` catalog gate with four red-proofs and ten decoys; the never-judged templates counted. +- **Part 2** — fifteen moves, one commit per app, `catalog_since` today, all gates green, CI green + by job id; the guarded Update pressed on four apps on demo-hp, all four `done`. + +## What shipped + +- `app-catalog-felhom.eu` **@1ad1f34** — three probe corrections, fifteen version moves, one new + gate, `test_gate_decoys.py` 51 → 56 cases. CI job **878 = success**. +- `felhom.eu` — this report, the audit, the evidence, R-630/R-631/R-632, R-618 CLOSED, `09` §8 + limitation 8, the capability map's guarded-update row, and `STATUS.md`. +- **No controller, agent or hub code.** The brief forbade it and none was needed. + +## What is owed + +- **What `verifying` does for a stack with no probe target** (R-630) — answerable on demo-hp, where + `paperless-ngx` is installed. Not run. +- **A live probe reading for five templates** no static rule can judge (R-631). +- **28 templates never deployed by any drill** (R-632) — the rotation's queue. +- **`vikunja` has no compose healthcheck at all**, so its probe has no oracle in either direction. diff --git a/STATUS.md b/STATUS.md index eb0e6626..fadcbcf7 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,1347 +1,30 @@ # STATUS — what works, what's broken, what's next -**Updated 2026-09-21 (overnight) — I tested the "update my app" button on as many apps as fit in a night, on good days and bad ones. Fourteen updates are proven safe. Three apps are broken in a way that shuts down a working app, and I would fix that first.** +**Updated 2026-09-22 (morning) — I fixed the three apps that shut themselves down after a good update, proved the fix on a real machine, built a check so it cannot happen again, and moved fifteen app versions onto the real catalog. One thing you asked for could not be done, and one thing I nearly got wrong was caught by running the check instead of trusting my reasoning.** -**Decisions I took on my own: none.** Nothing tonight needed a choice you had not already made. +**Decisions I took on my own: one.** I moved Nextcloud's database engine up a version. I had written it down as "dropped — the rules forbid it", then ran the rule instead of believing my memory of it, and the rule **allows** it by name: that permission was granted on 2026-09-21, the template already carries the setting that converts the data, and the move was proven end to end in under four minutes. You forbade PostgreSQL engine moves; this one is MariaDB. **You can reverse it by reverting one commit.** -**The fleet version is 0.261.0.** You asked for that. Both demo machines took it **thirteen seconds** after I saved it. The third machine is switched off and will take it when it comes back. +**The three broken apps are fixed.** Tandoor, Zipline and Wger each had one wrong number or address, so the machine knocked on a door the app does not answer. I fixed all three, then proved it on the scratch machine **in both directions**: before the fix all three showed **„Nem egészséges"** on their own page while the app itself was serving customers normally; after the fix, with no restart and no reinstall, all three showed **„Fut"**. -**What I did.** Twenty-one real updates on a scratch machine, each app installed at the version our catalog has today, filled with real data through the app's own front door, backed up, updated to the newer version that really exists upstream, then the data read back. **Fourteen proven, three failed, four I could not judge.** Ten of the fourteen printed their own "I am rewriting the database" line — so the data really was rewritten, and it still came back. +**And the update that failed now works.** I ran Tandoor's exact same update again — same app, same versions, same button, nothing changed but that one number. Yesterday it ran for **six minutes and shut the app down**. Today it finished in **41 seconds** and the data was still there. That is the whole finding in one line. -**The one I would fix first — three apps tell the machine they are broken when they are fine.** Tandoor, Zipline and Wger each have one wrong number or address in their settings file, so the machine knocks on the wrong door and hears nothing. That alone would only be a wrong label. **But the update also waits on that same check** — so when one of these apps updates *successfully*, the machine waits five minutes, decides it failed, **shuts the working app down**, and tells the household to restore from backup. I watched Tandoor serve customers for five minutes on its new version and then get switched off. Nothing is lost and the restore works, but the household loses their app and does work they did not need to do. **A cheap check would catch all three: each of those files already contains the right answer a few lines further down.** +**There is now a check that catches this before it ships.** The answer was always sitting in the same file, a few lines further down — each app already tells Docker where to knock. The check compares the two. It runs on every push, it refused all three apps before the fix, it passes now, and it is guarded by ten fake-out tests so it cannot quietly stop working. -**What broke, and whether the household could get out.** -- **Adventurelog's newer version rewrites the database and then never starts.** The machine did everything right: backup one minute before, waited the full five minutes, stopped the app so the data could not be hurt, and said in one sentence where the copy is, when it was made and what is inside it. I pressed that restore: **back in 75 seconds.** That app must not be moved to the newer version. -- **Tandoor, as above.** Restored in 32 seconds. -- **PostgreSQL will not jump a version.** Exactly as expected: the database engine refuses to start, the app stops honestly, the data is untouched, and the restore brings it back in 29 seconds. I also rehearsed the conversion that would let those eleven apps ever move: **about nine seconds of database work, under three minutes end to end.** That is a maintenance window, not a project. -- **MariaDB, by contrast, jumps a version cleanly** — and I pressed that through the real button for the first time. The engine converted the data, said so in its own words, and took its own backup first. +**Fifteen versions moved to the real catalog**, one at a time, every check run before each one. Nothing was forced and nothing was dropped. Then I pressed the real Update button on four of them on the demo machine: **all four finished cleanly** — BookStack, Docmost, PrivateBin and RomM are running the new versions. -**The machine also passed every bad day I could invent.** A version that cannot be downloaded: refused in one second, app keeps running. Two updates at once, then five: all ran together and all ended honestly. Power cut in the middle of the backup: the machine recovered itself and said so. Disk nearly full: refused before touching anything. An app left stopped by a failed update stayed stopped after a power cut — *"whatever is holding it owns its recovery."* +**What I could not do.** You asked me to press that button on **both** demo machines. The second machine has only one app installed and none of the four. I did not install them — installing apps on a demo machine is a change, not a test. Both machines did receive the new catalog, and I checked that. -**Four smaller faults, all written down.** A message that says "Updated" when nothing was updated. The failure message shown in Hungarian on the English page — including the sentence that tells a household where their files are. An app the household deleted that **came back by itself**, empty. And when an update fails, the machine deletes the broken app's log before anyone can read why. +**What the check found that nobody was looking for — and it is quieter than the bug it was built for.** +- **Paperless-ngx has never been health-checked at all.** Not "checked wrongly" — never checked. Its containers are named differently from the app, so the machine looks for one and finds nothing, and moves on without a word. A wrong check is loud and we caught it in one night. **A missing check looks exactly like a healthy app.** +- **Five more apps cannot be checked this way.** One of them, Home Assistant, is correct today only by luck: tighten its settings in the obvious way and it breaks the same way Tandoor did. +- **28 of our 53 apps have never been installed by any test.** The overnight run went from 3 apps to 21, which is a lot — but 21 is not 53. **For those 28, we do not know whether updating works.** That list is now the nightly queue. -**Rows opened and closed.** Twelve new, eight existing ones updated with what was measured. The list went from 303 to 315. +**Rows opened and closed.** Three new, one closed. The list went from 318 to 321. **What needs you.** -1. **Rotate the Gitea `admin` token.** The machine stores it in plain text inside its copy of the catalog, so an ordinary diagnostic printed it into my log. *If you do nothing:* the token keeps working and anyone with my session transcript has it. -2. **The promotion list** — fourteen updates proven safe enough to move on the real catalog, and three named that must not move. Moving a version is your call, never mine. *If you do nothing:* nothing breaks; those apps drift further from upstream each month. -3. **The seven questions** about automatic updates now have real facts beside them — including the two that had never been measured: what a stopped app looks like when nobody was watching, and what a database conversion costs. They are still yours. *If you do nothing:* the automatic-update work cannot start, because everything hangs off the first one — *may the machine update apps by itself at night?* - -**The live catalog was never touched with a test change.** Not once, not for thirteen minutes. Everything ran against a private drill copy on a scratch machine. The only change to the real catalog is test code, and I have proved every app's version line is identical to before. - ---- - -**Updated 2026-09-21 (evening) — I cut the power to a machine in the middle of an app update, three times, after the new version had already changed the data. It survived every time.** - -**The fleet version is now 0.260.0.** You approved it. Both demo machines have it. Three machines are -below it — the drill box, the tester's box and Peti's — and all three are switched off or blocked, so -they will take it when they come back. - -**The test the last session skipped without telling you: I ran it.** A machine updated an app with -**nobody pressing anything**, start to finish. It also proved the safer half: when the machine is told -to do something it must not do, it tries **once**, is refused, and then **never asks again**. Four -apps, three rounds, no repeats. - -**The dangerous power cut — the one nobody had measured.** The earlier test cut the power *before* the -new version started, which is the easy case. I cut it *after*. Three times, three apps, two different -ways of pulling the plug. **Every time the machine came back, finished the job, and told the truth.** -The version it thinks it runs, the version written down, and the version actually running all agreed, -every time. One app had already rewritten its own database when the power went — and the test data -read back unchanged. - -**One small lock added.** The machine used to update its own software at 04:30 without checking -whether it was busy updating an app. The window I proposed for automatic app updates covers 04:30, so -the two could have met. Now they queue politely behind each other. I proved it on a real machine: with -an app update running, the machine's own update is refused with a plain sentence; with nothing running, -it is not. **It never gets stuck** — an app that fails does not block the machine's own updates for ever. - -**Three faults found while doing this, all written down, none fixed today.** -1. **Wishlist cannot be signed up to on a fresh machine**, and the error it shows is *wrong* — it says - the account already exists when the real problem is that its first-time setup ran out of memory. - The machine reports the app healthy throughout. This is the one I would fix first. -2. **Uptime-Kuma sits on its setup screen with no login and no monitoring**, and the machine still - reports it healthy. A false green is the kind nothing ever catches. -3. A leftover "updated" label can survive an app being removed and reinstalled. - -**Two honest gaps.** I could never catch the cut in the very first instant of an update — it lasts -under a second, and three attempts with two methods all landed a moment later. And the unattended test -never produced a *stopped* app, because the safety rule correctly refused the broken test case before -it could be tried. Both are written down rather than glossed. - -**A security check flagged one of my test changes** — I briefly pointed an app in the catalog at a -dummy image to make it fail on purpose. It is the standard way to test a failure, no machine of yours -runs that app, and it is now reverted. Flagging it because you should hear it from me. - -**Rows.** Two closed, five opened, one pointer corrected. 307 in total. - -**Needs you — one thing, and it is smaller than last time.** - -**Raise the fleet version again, to 0.261.0?** That carries the lock to the rest of the machines. It is -on the scratch box only today. -**If you do nothing:** nothing breaks. The lock matters most once apps update themselves, which they -still do not. - -The seven questions from this morning are unchanged and still yours. Two of them now have measurements -beside them instead of guesses. - - -## Previous note - - -**Updated 2026-09-21 (afternoon, earlier) — the update feature: a box can no longer be offered an older version as an "update", and I measured how far behind everything actually is.** - -**A decision I took on my own — you can reverse it.** When we move an app back to an older version in -our catalog, a machine that already took the newer one used to show **"Update available"** — and the -button behind it would have put the **older** version back, on top of data the newer one may already -have changed. We cannot undo that. So from today such a machine says **"Up to date"**, and the button -**refuses**, with a sentence saying the version is newer than ours and that going back needs you. It -only refuses when it is certain; in every unclear case it behaves exactly as before. - -**What I measured, and the numbers are not comfortable.** - -- **Your two demo machines are perfectly up to date** — all ten apps, both machines, nothing behind. -- **But our catalog is far behind the world.** Of the app versions we pin, **46 of 58 have a newer - version out there**. **Seven are a big jump** — Nextcloud, Paperless, Claper, Gokapi, Homepage and - SparkyFitness — and all seven have sat still since **18 July, 65 days**. -- **"Up to date" is not always true, and now I know by how much.** Some of our pins name a *line* - rather than an exact version. **Six of the seven I could check have quietly changed underneath us.** - On the HP machine, four apps say "up to date" over a database engine that has moved. The label is - answering a narrower question than the one a household hears. -- **One app's image is gone from the internet.** Plant-it. It was already marked abandoned, so this is - the expected ending, not a fault. A machine already running it keeps running it; it can never be - installed again. - -**A rule I lifted, and half of one I kept.** Since 13 September our catalog has refused to move a -database engine to a new major version, because the Update button took no backup. **It takes one -now**, so I lifted that ban **for MariaDB** — and only when the engine moves **on its own**, never in -the same change as the app itself, because two changes behind one step give an unreadable failure. -**I did not lift it for PostgreSQL.** That engine does not convert your data by itself and simply -refuses to start on the old data — eleven apps would go down at once. A backup is a way back, not a -conversion. - -**I proved it on a real machine, not only in tests.** On the spare machine I installed a throwaway -app, moved our catalog forward, updated it, **cut the power mid-update**, and then moved the catalog -back. Three results: -- **The power cut is handled honestly.** The machine came back, put the app's version back where it - was, ran it, and told the household in plain words that the update was interrupted and the old - version is running. Nothing was left half-done. -- **The new refusal works.** With the machine running the newer version, the page said "Up to date" - and the Update button was refused, in Hungarian and in English. -- **Nothing moves when it refuses.** I pressed it three times; the app was untouched every time. - -**One new fault found while doing it, and I did not paper over it.** On an English page, the sentence -explaining *why* an update stopped is still **Hungarian** — sitting directly under an English label. -Every sentence the update feature shows in that situation has the same problem, including one that -tells a household **whether their files will come back**. I wrote it down as work to do; I did not fix -it today, because one released version per day is the rule and today's was already out. - -**Two things in the brief I was given were wrong, and I want you to know I checked rather than -assumed.** It said the version label was still Hungarian on English pages — it has been fixed since -yesterday and only our note was out of date. And a number describing our catalog, repeated in four -places, had not matched reality for some time. Both corrected. - -**Rows.** Four closed, three now waiting on you, one corrected, three opened. 300 in total. - -**Needs you — two things.** - -1. **Seven questions about updating by itself.** The machine still cannot update an app on its own, - and 39 of the versions we are behind on are small, safe steps that nobody will press a button for - 39 times. I have written each question down with two or three choices, what each costs, and which - one I would pick. They are in the design notes. - **If you do nothing:** nothing breaks, and nothing updates itself either — the gap to the world - keeps widening, quietly. -2. **The fleet version.** Today's fix is on the two demo machines only. Raising the fleet version - would carry it to the rest — including the tester's real machine — and that is your switch, not - mine. - **If you do nothing:** the other machines keep offering a downgrade as an update. That is a label - and a button, not your data at risk, so waiting is cheap. - - -## Previous note - - -**Updated 2026-09-21 (morning) — the one screen that stopped an English speaker is fixed, the floor is raised, and both boxes have it.** - -> **Ready for an English-speaking tester: yes — nothing known now stands in their way.** -> Ready for a Hungarian volunteer: yes, unchanged. - -**What I fixed.** All three things yesterday's walk found. - -1. **The page where you type the code from your e-mail.** It was English; its answers were Hungarian. - Now the whole page answers in your language. Type the code wrong and an English household reads - *"Wrong or expired code"*. Five wrong tries and it says *"Too many attempts — try again in 15 - minutes."* -2. **The code in the e-mail.** An English household now gets **English words** — - four ordinary English words joined by hyphens, instead of `képző-szkítia-ásatás`. You can read it down a - telephone. **It is four words instead of three, on purpose**: the English word list is smaller, so - a fourth word keeps the code at least as hard to guess as the Hungarian one. Never weaker, and a - test does that sum every time rather than trusting me. Hungarian households see no change at all. -3. **The Backup page's warnings** — the ones that say whether your files are really safe — follow - the language now, and the English says exactly what the Hungarian says: it protects against bad - files, **not** against a broken disk. - -**How I know.** I watched the real box answer in both languages, and the two e-mails sit one day -apart in the same inbox — yesterday's Hungarian code and today's English one, same message, same -subject. The Hungarian pages are byte-for-byte what they were; a check proves that on every push and -I proved the check works by breaking one full stop on purpose. - -**One thing proved itself by accident.** My own wrong guesses in Hungarian locked out the English -page too. That is right: the lock counts the person, not the language, so nobody gets extra tries by -switching. - -**Honest limit.** I fixed and proved three things. **I did not re-do the whole first hour as a -stranger on a brand-new machine** — that is the walk that turns this from "nothing stands in the way" -into "someone did it". It is worth doing before you hand a box to a real English tester, and it is -also the walk that would show the two Backup warnings on a screen rather than in a test. - -**The floor is raised — both boxes have the fixes.** You asked, so I did it. Both machines now run -today's version, and the second one **updated itself** in under four minutes with nothing installed -by hand. I checked its screen afterwards: it answers *"Wrong or expired code"* in English. That is -the proof that matters — the fix reached a box I never touched. - -**I was wrong about the HP box, and you were right.** I said it was switched off. It was not: it has -been running for four and a half weeks and reporting all along. **Both of my ways in were pointing at -old addresses.** The box moved to a new address on your network, and the other route goes through a -service that is not even installed on it. I fixed both, checked both, and corrected our notes. The -lesson I am keeping: I tried six ways in and all six failed, but they were six goes at one wrong -assumption, not six pieces of proof. - -**And that hunt found something real.** The HP box had a private version limit set on it during a -test five days ago, and nobody removed it. It quietly **overrode the fleet setting** — so that box -has been missing the last **four** upgrades, and nothing anywhere said so. The alarm only speaks when -something changes, so a box stuck behind a private limit stays silent for ever. I cleared it, and -that is when the box finally took today's version. Worth fixing properly: raising the fleet version -should tell you which machines it will **not** move. - -**Rows.** Three closed, one withdrawn as wrong, four opened. - -**Needs you — nothing.** The floor is done. - -**If you do nothing:** nothing breaks. - - -## Previous note - - -**Updated 2026-09-20 (late) — I installed a box from scratch and used it as an English speaker. It almost works.** - -> **Ready for an English-speaking tester: NOT YET — one screen stops them.** -> Ready for a Hungarian volunteer: yes, unchanged. - -**What I did.** I built a brand-new machine from the installer on our own download page, and then -played a stranger who speaks only English: I followed the English guide and the screens, and nothing -else. Same walk as two weeks ago, in the other language. - -**Almost all of it held.** The download page, the box's own screen, all three e-mails, the page where -you connect the box to your account, the dashboard, both apps, every app description — **all in -English, with no Hungarian left on them at all**. The dashboard came up in English by itself, because -the account says English. Switching between the two languages works both ways. - -**One screen stops them, and it is the worst one to lose.** The page where you type the code from -your e-mail to take control of the box is in English — but its *answers* are in Hungarian. Type the -code wrong and it says „Hibás vagy lejárt kód". An English speaker cannot tell a typo from a dead -code on the one screen standing between them and their machine. That is the blocker. It is about a -day's work, not a rebuild. - -**Two smaller ones.** The code in the e-mail is **three Hungarian words with accents** — you can -paste it, but you could not read it to someone over the phone. And the Backup page's two warnings — -the ones that tell you whether your files are actually protected — are still Hungarian. - -**Something that was broken is now fixed, and I saw it.** Two weeks ago the box's screen kept asking -to be paired forever, even after it was connected. It does not any more: the last thing it shows is -"the box is linked", in both languages. - -**Also today:** the app catalog's English went out to every box this morning, the four smaller -Hungarian leftovers are gone, and a fresh golden image was baked so any new machine starts on today's -version. I also found that our automatic checker had been failing on every catalog push for two hours -and nobody had read the alarm — that is fixed too. - -**Rows.** Nine opened today, two closed, one narrowed. - -**Needs you — one decision, no rush.** Whether to raise the box floor to today's version (0.258.0). -Everything already works without it; it only means every box gets the four small fixes. - -**If you do nothing:** nothing breaks. - - -## Previous note - - - -**Updated 2026-09-18 (night) — the new installer is published, and the box's screen speaks English too.** - -> **Ready for a volunteer: yes.** - -**What changed.** Two screens a person meets before they ever see a dashboard were Hungarian only: -the text on the box's own monitor while it waits to be paired, and the download page. Both are now -bilingual — Hungarian exactly as before, then the same thing in English under it. The boot menu too. - -**`felhom.eu/en/download` is live**, and both download pages now point at the new installer. - -**The new installer is published: version 1.29.0.** I installed from the finished file **twice** — -once from each boot-menu choice — on throwaway machines, let each one boot and then rebooted it, and -checked the screen every time. Both showed the bilingual text, no Proxmox address, and the pairing -code readable in both languages. Then I downloaded the published file back and checked it is -byte-for-byte the one I tested. - -**Something was wrong and I caught it by looking.** The first build's second menu line was too long -for the box GRUB draws, so its English half was chopped off mid-word on the boot screen. I shortened -it and built again; the published image is the fixed one. The Hungarian is what may never change, so -the English is the half that gave way. - -**Two things were already broken before I started.** The installer's own test suite had been failing -for two days and nobody saw — it is not part of any automatic check. And the release checklist still -said every screen must be Hungarian, which would have blocked exactly what you asked for; your -September ruling replaced that, so I rewrote the checklist item rather than skipping it. - -**Rows.** Three opened, one closed, one half-done. - -**Needs you — nothing for the installer.** It is done and live. The old version stays online, so -going back is one edit if anything looks wrong. - -**Still open from earlier today:** raise the box floor to 0.256.1 (so English households get English -alerts everywhere), and whether to change the shared demo password. - - -## Previous note - -**Updated 2026-09-17 (night) — English, step 3 of 3 done: every dashboard page speaks English, and the language switch is there for everyone.** - -> **Ready for a volunteer: yes, unchanged.** A Hungarian household will see one new thing: a small -> „Magyar / English" switch at the bottom of the menu. Nothing else on the Hungarian pages changed. - -**Decisions I took.** None. (The old choice to hide the switch was mine; the plan said to undo it now, -and I did.) - -**What I exercised, on the demo HP box.** The last eleven pages speak English: storage, network -storage, the two drive helpers, sharing, sign-in, the set-password page, the guest launcher, its -password page, the "no such app" page, and the debug page. I saved the Hungarian pages before and after -the update. The only change is the new switch, plus live numbers. The sign-in pages did not change at -all. I switched the box to English and back with the real switch. The hub saw Hungarian, English, -Hungarian. - -**What broke, and whether it is fixed.** -- Six small Hungarian words were still on the English pages („mp", „db", „FIGYELEM"…). No test saw - them; I found them by reading. Fixed, and filed that the test cannot see such words. -- My test cases used made-up page titles. I fixed them to the real ones and re-took those snapshots - from the old code. -- Found, not fixed, filed: three browser-tab titles with an app name stay Hungarian; the drive helper - pages do not light up the Storage menu; the two disks on the dashboard swap places between visits. - -**Rows.** One closed (localisation step 1), four opened. The register went from **260** to **263** open rows. - -**Needs you.** Nothing. **You told me to raise the floor, and I did:** the fleet floor is now -**0.250.0** (declared agent requirement 0.131.0, the release header's). The demo N100 box took it by -itself in about a minute — nobody deployed to it — and its Hungarian launcher now shows the switch. -The HP demo box has its own per-box floor and was already on 0.250.0 by hand. The other three boxes -are down or blocked, so they take it when they come back. **The vouched golden is still 0.246.0, so a -NEW box installed today starts on 0.246.0 and then updates itself to 0.250.0.** - ---- - -## Previous note - -**Updated 2026-09-17 (evening) — English, step 2 of 3 done: the backup pages too.** - -> **Ready for a volunteer: yes, unchanged.** A Hungarian household still sees nothing new. - -**Decisions I took.** None. - -**What I exercised, on the demo HP box.** Ten more pages speak English: the dashboard, the app list, an -app's install and settings page, logs, the system monitor, import, export and the three settings pages. -I saved each page in Hungarian before the update and after it. Only live numbers differ (CPU, memory, -log lines). I switched the box to English and back with the real switch. - -**What broke, and whether it is fixed.** -- One page decides something by reading its own Hungarian word („Fut", running). Translating that word - would quietly break the English page. I left the word Hungarian and filed it. -- A new check found two text pieces on last release's pages that no test ever showed. Fixed. - -- Step 2: the seven backup pages speak English, and the Hungarian pages again did not change. The - check that stops a promise we cannot keep now reads English too — and English showed it had been - missing some Hungarian sentences (verbs split in two). Filed. - -**Rows.** Two opened. The register went from **258** to **260** open rows. - -**Needs you.** Nothing yet. Steps 2 and 3 follow (backups; storage, sharing, sign-in, and the switch -for everyone). - ---- - -## Previous note - -**Updated 2026-09-17 (afternoon) — the dashboard can speak English on three pages, and the Hungarian pages did not change by a single byte.** - -> **Ready for a volunteer: yes, unchanged.** A household sees nothing new. English is switched on per -> box and covers the launcher, the backup overview and an app's page. No recruiting sentence yet: -> English is a proof of the method, not a feature. - -**Decisions I took** (you may reverse either): -1. **How the text is stored.** One word list per language, filled into the page before the page is - built. The other way changes Hungarian characters on the page, so it breaks the „Hungarian must not - change" rule. -2. **The language switch is hidden on Hungarian pages for now.** Only three pages are English, so - showing it to every household would put a half-English dashboard one click away. It appears once the - rest of the pages are done. - -**What I exercised, on the demo box.** -- I saved the three pages in Hungarian before the update and again after it. They are the same, - except the version number and a security token. -- I switched the box to English with the real switch. The pages came up in English. The hub received - „en" in the box's report. I switched it back; the pages are Hungarian again and the hub received „hu". -- What stays Hungarian on the English pages: some messages built by the program, and the app - descriptions from the catalog. Both are later steps of the plan. - -**What broke, and whether it is fixed.** -- Moving the text out of the pages made **one copy check fail and three others blind** to it (for - example the check that stops a promise we cannot keep). Fixed in the same release; each check was - shown to catch a planted mistake in the moved text. -- My text-moving tool **missed eight Hungarian words that have no accents**. I found them by reading - the result, and fixed the tool. -- The hub's field check passed the new field for a wrong reason (the word appears in a comment). Filed, - not fixed. -- The count of all Hungarian text: about **1 900 strings in the pages, 1 100 in the program, 65 in the - hub e-mails, 830 in the app catalog, and a 1 600-word guide**. The prompt's two rough counts were - not reproduced; the difference is named in the inventory. - -**Rows.** Ten opened, none closed. The register went from **248** open rows to **258** (the register -gate's count). Six of the ten are the plan's steps; one is four places where the program decides -something by reading Hungarian words — those must be fixed before any program text is translated. - -**Needs you.** Nothing. You answered both questions: -1. **The console screen and the download page are in scope.** They stay late in the plan. -2. **Screen names translate** („Indítópult" is „Launcher"). Every later step follows this. -3. **For information:** the demo HP box runs the new controller, put there by hand. The other boxes - stay on the previous version until you raise the floor. Nothing changes for them if you do not. - ---- - -## Previous note - -**Updated 2026-09-17 (midday) — the four things chaos night found are fixed, and three of them were proven on the demo box.** - -> **Ready for a volunteer: yes.** Nothing here was blocking; all four made the box more honest or less -> noisy. The recruiting sentence: *if the power fails in the middle of a restore, the box now says so and -> tells you to run it again; and it no longer asks for the recovery code before it can take it.* - -**Decisions I took.** None under the unattended rule. I carried out your three rulings: the quiet-box -alarm waits three report cycles; the restore record is kept on disk; the slow crash-loop warning. I -signed the agent update onto the two demo boxes under yesterday's ruling. - -**What I exercised, on real boxes.** -- The hub now waits **45 minutes** before „the box went quiet" and **90** before „the box is down". The - running hub prints both numbers when it starts. A dead box now pages you 15 minutes later — your - ruling's accepted cost. -- I started a restore on the demo box and killed the controller **two seconds** in. It came back by - itself. The restore page then said „A visszaállítás megszakadt — indítsd el újra", and the hub got the - event. Restoring again cleared the note. -- I killed the demo box's controller **five times, about eight minutes apart**. The agent restarted it - every time, the fast brake never fired, and on the fifth **exactly one** warning mail reached you - (09:29). That was the test — no need to act on it. - -**What broke, and whether it is fixed.** -- Nothing in the product broke. -- One thing I could not prove live: the recovery-code reminder waiting for the box. No demo box is in - that state today. It is proven by tests, and chaos night already measured the 17-minute wait live. -- I found one small gap in my own new code: if a household **removes** an app instead of restoring it - again, its „interrupted" note never goes away. Filed, not fixed — the release was already built. - -**Rows.** Two opened, four closed. The register went from **215** open rows to **213**. - -**Needs you.** Nothing. Both of your „yes" answers are done: -1. **All boxes get the new controller.** The floor is raised to 0.246.0; the N100 took it within seconds and - the HP already had it. **Peti's box is offline** on the hub, so it gets it when it next reports. -2. **Fresh installs get today's fixes.** The hub would only vouch the new agent together with a newer golden, - so I baked golden 0.246.0 (your choice) and vouched both. A brand-new box now installs controller 0.246.0 - and agent 0.132.0. -3. **For your information:** the demo box's slow crash-loop warning stays raised until tomorrow 11:22, - because I caused it on purpose. It cannot mail you again before then. - ---- - -## Previous note - -**Updated 2026-09-17 (morning after chaos night) — twelve rounds of a household under accidents; the box healed itself every time.** - -> **Ready for a volunteer: still yes.** For a night I did random household things on a fresh box -> while random things went wrong: a power cut in the middle of a restore, the reset button four -> seconds into another, a full disk, a dead tunnel, Docker restarting, the internet cut three times, -> and the data drive pulled out of the running machine for twenty minutes. **No customer data was -> lost, and the box put itself back together every time without anyone touching it.** Seventeen -> alarms went off. All seventeen were true, none were missing, and every one reached your mailbox. - -**Decisions I took.** None under the unattended rule. Two judgement calls are logged in the drill -record: one round ran „use" instead of the drawn „update" because the app catalog's own checks could -not vouch for the update; and I did not touch the box at all during the final control round. - -**What I exercised.** A new golden was baked and published. A fresh box installed itself from the -public installer image and bound itself with no press from anyone — both emergency presses I was -allowed stayed unused, and the automatic re-issue after a deleted box was seen working live for the -first time. Then twelve rounds, drawn in advance from a fixed seed and written down before the first -one started. - -**What broke, and whether it is fixed.** Nothing in the product broke. Three things are worth -fixing, all filed, none fixed tonight (no product code was allowed): -- If the machine stops during a restore, **nothing ever tells the household whether it finished.** -- The „this box has gone quiet" alarm allows exactly two report cycles, so **one missed report uses - the whole allowance** — tonight a healthy box came within one second of paging you. -- A disk that fills up and empties again between the daily checks is never mentioned to anyone. -Two smaller ones were filed earlier in the night: the first-hour guide asks for the recovery code -about seventeen minutes before the box can accept it, and a small-disk box keeps retrying a local -backup that can never fit (the off-site copy still worked). -**Most of what broke tonight was my own measuring.** Eleven times a check of mine gave a confident -wrong answer; each is written down with its fix. The worst one delayed the last cleanup step by six -hours. - -**Rows.** Five opened, none closed. The register went from about 212 open rows to about 217 (my own -count this morning reads 215; the difference is how closed rows are counted, not a missing row). - -**What I could not test.** Restoring a single app from the off-site copy. This box was a rebuild of -an existing customer, so its old off-site app backups belong to a key it no longer has — correct and -by design, and the box told you so within seconds. The whole-machine off-site copy **is** there and -intact, but I only listed it; I did not restore from it. - -**Cleanup.** The test machine is gone, its space is back, both demo boxes are still running, and the -off-site backups are untouched. The deleted box's record is gone from the hub; its key is held in -retained custody, as designed, and the customer account is untouched. - -**Needs you.** -1. **Nothing blocking.** A volunteer can start. -2. **The quiet-box alarm margin.** Pick one: wait three report cycles instead of two, or retry a - failed report once straight away. If you do nothing: one network hiccup at the wrong moment pages - you about a box that is fine. -3. **The restore record.** If you do nothing: a household whose power fails mid-restore is never - told whether their restore happened. -4. **Single-app restore from off-site is still unproven on this release.** If you do nothing: it - stays unproven until a box that is not a rebuild is used for a drill. - ---- - -## Previous note - -**Updated 2026-09-16 (late evening) — the box now asks for the recovery code, and the page stops promising a copy that has not run.** - -> **Ready for a volunteer: yes.** The one thing standing in the way this morning is fixed. On a new -> box the off-site copy is switched on but paused until the household writes down their recovery -> code — that pause is deliberate and correct, because that code is the only key and we cannot open -> their copies without it. What was wrong is that nothing asked them. Now every page says so until -> they do it, and the backup page says „would protect" instead of „protects" while it waits. - -**What changed today (this note).** A reminder bar on every page of the dashboard: „the off-site -backup is paused until you create your recovery code", with the button that does it. The sentence -under each app's local backup now tells the truth about the state it is in — protected, waiting, or -no copy at all — instead of promising the same thing in all three. The first-hour guide asks for the -code right after the dashboard password and before the first app, and says plainly that we cannot -get it back for them. The operator step for rebuilding an existing customer's box is written down -where it was missing: normally nothing to press, but one press when the old box was not deleted -through the acknowledged flow. - -**What I proved on real boxes.** On a box whose recovery code exists: no bar anywhere, and the page -says the files are protected. On a box waiting for the code: the bar on every page, an off-site run -refused with „waiting for the key to be placed in escrow" and no copy written, and an app's row -reading „would be protected … paused until you create the recovery code". Both boxes were running -today's build. The throwaway app and the test setup were removed afterwards and checked gone. - -**Decisions I took.** None under the unattended rule. - -**Needs you.** -1. **Nothing blocking.** The installer image you approved is published and live on the download page, - and the recovery-code gap is closed. A volunteer can start. -2. **The slow-crash-loop counter** (the ruling of 2026-09-15) is still owed, and is a job for the - nightly. If you do nothing: a box that keeps crashing slowly is still reported as healthy for - longer than it should be. -3. **One small thing worth knowing, not doing:** there is no button that forgets an off-site - destination once set — only one that disables it. Written down as a low-priority job. If you do - nothing: a household that types the wrong address keeps the old one on the box, switched off. - ---- - -## Previous note - -**Updated 2026-09-16 (drill on a fresh box) — the fixes hold; the backup promise does not.** - -> **Ready for a volunteer: NO — one reason, and it is new.** On a brand-new box with one drive, the -> household's own files are in **no backup at all**, and the backup page says they are. I deleted five -> photos the way a child would, restored from the box's own backup, and the folder came back listing all -> five photos — none of which opens. The bytes had never been copied. The app's own wastebasket still held -> them, and the restore made that unreachable too. - -**What I proved on a fresh box.** The installer downloads and installs; the box lands on the golden this -drill baked (checked by checksum, not by trust); the connect e-mail and the bind page work; the dashboard -opens through the tunnel from outside; the file manager has its own password and „admin/admin" is refused; -four apps installed and were used; the backup page tells the truth per tier; „Mentés most" stopped the apps -for 26 seconds, inside what the button promises. - -**The five faults.** A controller killed during an install: back in 37 seconds. Two reboots a minute apart: -everything back in 124 seconds, and the box did not count the reboots against its own safety brake. Wrong -passwords five times: the app lets you keep trying, the box's own setup code locks for 15 minutes after two -and e-mails you — correctly. Memory pressure: the box still cannot see it (second box, same result). -The deleted photo folder: see above. - -**The automatic connect e-mail: it works.** I deleted the box's record on the hub and the „connect your -Felhom box" e-mail reached the customer **one second later**, naming the reason. That was the last thing -waiting to be proven with a real mailbox. - -**Decisions I took.** None under the unattended rule. - -**Needs you.** -1. **Say whether the backup page may keep promising what it does not hold.** Today, a new box with one drive - backs up its apps' settings and databases — not the household's own files. The page says otherwise, and a - restore then reports success while the files are gone. If you do nothing: the first volunteer can lose - their photos and be told everything is fine. I can fix the wording and the refusal in the controller; the - real protection needs a second drive or the off-site copy switched on. -2. **Grant the off-site server one permission.** The re-issue fails on a missing grant, so a rebuilt or new - box gets no off-site copy at all. If you do nothing: the third backup level stays unavailable for every - new box, and the fix already written stays dead. -3. **Rule on the restart brake.** The box stops retrying after three restarts in fifteen minutes. I measured - that a controller dying every twenty minutes is restarted forever, and the only trace is a note that - e-mails nobody. Options: leave it (the box heals itself and the timeline records it); add a second, - slower counter that raises a warning; or make the fifth restart in a day a warning. My pick: the second - counter — it keeps the healing and ends the silence. If you do nothing: a slowly failing box stays - invisible until someone reads the timeline. - ---- - -## Previous note - -**Updated 2026-09-15 (P1 fixes) — the big night's blockers, fixed and shipped.** - -> **Ready for a volunteer: almost.** The file manager has a real password, the backup page tells the truth, the -> installer is published with its download page, and a dead controller now comes back by itself — proven on the HP. -> Two things still stand in the way: a volunteer's own tunnel is unproven until their box exists, and every box except -> the HP still runs the old agent until you sign its update. - -**Decisions I took.** None under the unattended rule. Three things I did not do, each with its reason: I did not send a -real connect e-mail to test it, because deleting a test customer would touch the off-site server; I did not build a -„release PBS token" button, because the off-site server has no way to remove only the token; the memory warning does not -say „restarted", because nothing restarts the app. - -**What I exercised, on the HP.** Killed the controller: back in 59 seconds. Parked it: it stayed off. Killed it during an -update: the update rolled itself back. After three kills in 13 minutes the box stopped retrying for 30 minutes and you were -e-mailed — that is the safety brake, and it worked. When the brake ended, it started the controller again by itself, and told you that too. -File manager: a new box gets its own password; the HP, where you set one, was left alone. Backup page: correct on the HP. -Vaultwarden refuses strangers. Paperless took 20 documents at once without losing one. - -**What broke, and whether I fixed it.** The backup tile printed „0 B" after an agent restart — fixed, ships next release. -The memory-warning check sees nothing inside our guests, because Docker reports nothing there — not fixed, filed. - -**Rows.** Closed 9, opened 9. Register: 232 before, 232 after. - -**Needs you.** -1. **Sign the agent update for the N100 (and later Peti's box).** If you do nothing: the N100 keeps the old agent, its - dead controller does not restart, and its controller update stays held. -2. **Say where your signing keys live between sessions.** They were readable by other users on DooPlex; I tightened them. - If you do nothing: the keys stay on the build server. -3. **When the next box for „Tester 1" exists, check the tunnel opens from outside.** If you do nothing: the „No TLS Verify" - fix stays unproven. -4. **Allow one test customer to be deleted (it resets on the off-site server), or accept the unit test.** If you do - nothing: the automatic connect e-mail is untested with a real mailbox. - ---- - -## Previous note - -**Updated 2026-09-15 (morning note) — the big night: a household's first month on one fresh box.** - -> **Ready for a volunteer: not yet — three things stop them.** The dashboard link still does not open through -> the tunnel. A box installed for an existing customer gets no bind e-mail. And every box's file manager opens -> with the login „admin" / „admin" — on the HP that login page is reachable from the internet. - -**Decisions I took.** None under the unattended-decision rule. Two things I did not do, each with one reason: I did -not switch on the paid off-site storage for „Tester 1" (it costs money), so this box had no off-site copy and the -off-site restore could not be walked. I stopped injecting faults after the ninth, because the brief's stop rule was met. - -**Interventions (first hour + moving in): 2.** (1) No bind e-mail came for 10 minutes; I pressed the operator's -„send link" button. (2) The tunnel answered 502; I used the home-network address for the rest of the night. - -**Alarm truth table, in five lines.** -1. Every alarm that fired was true. None was false. -2. Missed: the Paperless crash, the broken tunnel, the dead controller, the full disk, and the second drive loss. -3. Three of those were missed because a 30–60 minute mail cooldown silenced a new incident. -4. One drive loss sent five operator mails; the household got no mail for anything all night. -5. The customer's pages were honest about drives, and wrong about backups twice. - -**What broke, and whether the box healed itself.** Power cuts (three, one during a backup, one during an update): healed -in about four minutes, same versions, data intact. Drive pulled and returned: healed in 91 seconds, data intact. Internet -gone: the tunnel came back by itself in 9 seconds. Disk 95 % full: the box kept working. **Controller killed: it did not -heal — no dashboard for 33 minutes, nobody told, only a reboot brought it back. That was the stop.** Also: Paperless -silently lost 20 uploads to memory; the backup page claimed a remote backup that does not exist; the whole-system backup -stopped every app for 8 minutes while promising „a few seconds"; after I reverted a test update in the catalog, the box -offered the downgrade as an update. Nothing lost data that a restore could not bring back. I fixed nothing tonight. - -**Rows.** Opened 16, closed 0. Register: 221 rows before, 237 after. - -**The one sentence for recruiting.** Felhom survived power cuts, a pulled drive and a lost internet by itself tonight, -but do not invite anyone until the tunnel works, the file-manager password is changed on every box, and a dead -controller restarts itself. - -**Needs you.** -1. **Change the file manager's admin password on the HP now** (and on the N100). **If you do nothing:** anyone on - the internet who tries „admin" / „admin" at the HP's files address can read its data drive. -2. **Tick „No TLS Verify" on the „Tester 1" tunnel route in Cloudflare.** **If you do nothing:** no volunteer can open - their dashboard from outside their home. -3. **Decide whether a box installed for an existing customer should get the bind e-mail automatically**, or you press - the button each time. **If you do nothing:** each new install waits for you. -4. **Say whether installer 1.27.1 should be published.** My recommendation: **yes, publish it** — nothing tonight was - the installer's fault; the install, first screen and bind all worked. The three blockers above are on the tunnel, - the controller and the file manager, and they block inviting people, not the installer. **If you do nothing:** the - old installer with the English admin line stays online. -5. **Decide what to do with „Tester 1"'s old off-site data on ep0** (still there, kept tonight) and its stuck DR tier. - **If you do nothing:** the data stays, and every new box for this customer gets a failed whole-system backup alarm. - ---- - -**Updated 2026-09-14 (evening) — the doorstep: installer fixed, walked again, NOT published.** - -> **Ready for a volunteer: not yet — one thing stops them.** On the test customer you chose, the -> dashboard link does not open from our network: the box's Cloudflare tunnel connects but gets no routes -> (12 of 12 tries failed). Your phone reaches it, so something differs between connectors — only you can -> see that in Cloudflare. Everything this task changed works. - -**Decisions you took today.** Keep the installer interactive (a person chooses the disk). Every customer -has their own domain; you create the tunnel. Use the „Tester 1" record for the walk. - -**What I did.** The box's first screen is now Felhom's in Hungarian, from the very first boot — the first -build still showed the English admin line on that boot, so I fixed it again and rebuilt (1.27.1). The -passphrase has one name everywhere, and the hub now tells you, when you create a customer, to hand it -over in person. That hub change is live. The installer was built and checked against the release gate, -then walked end to end: install on a three-disk and a one-disk machine, apps, backup, restore -(byte-identical), removal, power cut, wrong code. Nothing was published. - -**What broke on the way.** The test customer has no e-mail, so no code or link could be sent. The -tunnel has no routes for a new box. I could not drive the graphical installer screen with my tools, -only the text one. One of our install documents wrongly says the controller creates the web addresses. - -**Rows.** Opened 7, closed 1 (the passphrase hand-over). Three more are fixed and close when you publish. - -**Needs you.** (1) **Look at the „Tester 1" tunnel in Cloudflare**: which connectors are there, and does it -have public hostnames? **If you do nothing:** a volunteer on a network like ours cannot open their -dashboard. (2) **Put the volunteer's e-mail on their customer record.** **If you do nothing:** they get no -setup code. (3) **Say yes or no to publishing installer 1.27.1** — only after (1) is sorted, by the rule -you set. **If you do nothing:** the old installer stays online, with the English admin line on screen. -(4) **The test left backup data for „Tester 1" on the off-site server**, written by its DR setting -from the test box (now destroyed); removing it the product's way would also remove the customer's -tunnel. **If you do nothing:** it stays and uses a little space on the off-site server. - ---- - -**Updated 2026-09-14 (afternoon) — the first-hour drill on a fresh box.** - -> **Ready for a volunteer: not yet.** Two things would stop a stranger before their first app: **there -> are no instructions anywhere telling them where the installer is or what to type**, and **the link -> in the "your server started" e-mail does not open for a new customer**, because nobody creates its -> web address automatically. Everything after that point worked. - -**What I exercised.** A brand-new machine on the HP, installed from the public installer, connected -to a new test customer, claimed, two apps installed and used (a family wiki with a Hungarian page and -an attachment, and an encrypted note), a manual backup, one app removed with its data, the other -restored after I deleted its page, a power cut, and a mistyped code. The new golden image was built -first, as the weekly rule says, and the fresh box landed on today's release by itself. - -**What broke.** Nothing lost data: the restore brought the deleted page back byte for byte, and the -power cut brought every app back on the same version with no false alarm. What a stranger would trip -on: no instructions (I wrote a Hungarian draft for you to approve); the dashboard address; the -installer refusing its own default machine name; the box's screen first telling people, in English, to -open the admin page they must never use; the five-word passphrase that no e-mail ever delivers; one -backup page wrongly saying "already covered, nothing to do"; app pages saying "open wiki.DOMAIN" -literally; and the dashboard showing the backup two hours off from the backup page. I reached the -dashboard once in a way a volunteer could not — that is the single intervention. I fixed nothing; this -run only records. - -**Rows.** Opened 9 (eight from the walk, one about how I check the build server), closed 0. Register table rows: 200 before, 209 after. - -**Needs you.** (1) **Decide how a new customer reaches their dashboard**: the hub creates the web -address automatically, or the box offers a home-network address that works with no setup. **If you do -nothing:** every volunteer needs you to create their address by hand before they can log in. -(2) **Read and approve the Hungarian volunteer instructions**, and choose how they are sent. **If you -do nothing:** there is nothing to send a volunteer. (3) **Hand the five-word passphrase to each -volunteer yourself** until an e-mail does it. **If you do nothing:** they cannot connect their box. - ---- - -**Updated 2026-09-14 (morning note, the second night) — the scratch guest is built, one controller release, the rotation restarted.** - -**Decisions I took.** (1) The scratch guest on the HP was built as you ruled: a second guest under -the HP's own customer, on the fast internal disk, sized like the main one, kept on purpose and written -up in all three places. Two properties of it were my call and you may reverse them: it never sees your -real data drive, and it never starts the public tunnel — a second connector would serve the public -domain from a throwaway. (2) A finding I had already listed as fixed turned out half-fixed when -measured live, and the rule is one release per night, so I kept the line open with the exact measurement -instead of shipping a second release. (3) The removed-app listing was a medium-priority line, but it -needed no ruling, touched no customer data and was the rotation's own finding, so I took it into -tonight's release. - -**What I exercised.** The rotation restarted from its first standing app on the scratch guest: front -door, use, backup, second copy, remove-with-data, full restore from the second copy, the guarded -update, remove-everything — all clean. Then a throwaway app for the release proof. - -**What broke, and whether I fixed it.** Six lines fixed and shipped in one controller release, -delivered by the floor in 16 and 17 seconds, proven on the scratch guest: an app you removed while -keeping its backup now shows on the backup pages with a button that reinstalls it (before, the way -back existed only as a hidden endpoint, and a backup kept on a data drive could not be found at all); -a removal now clears the "held after a failed update" mark; the memory card on the monitoring page can -render; the second copy is dated by its data rather than by a file that only moves when the app's -definition changes; and a boot rule is now pinned by a test. Half-fixed: the removal's list of -deleted volumes is right for a freshly installed app and empty for one that came back from a restore, -because the restore recreates the volume without the label the list looks for. Measured, kept open. -One security slip to know about: while building the scratch guest, the HP's retrieval passphrase was -printed once into a tool output here on DooPlex. Nothing left the machine. - -**Rows.** Opened 1 (delete the empty drive-path setting nothing reads any more). Closed 6 (the -scratch guest, the removed-app listing, the hold left behind, the monitoring card, the second-copy -date, the boot rule). Re-scoped 1 (deleted-volume list). Register: 210 open / 194 closed before, -205 open / 200 closed after. - -**Needs you.** (1) Open the backup page once in a browser after removing a throwaway app with its -backup kept, and press the new button — strict screen coverage is yours. **If you do nothing:** the -feature stays proven at the endpoint level only. (2) The passphrase slip: re-issue the HP's retrieval -passphrase from the hub when convenient. **If you do nothing:** the old one stays valid; the exposure -is one line in this session's local record. (3) The scratch guest stays up and idle. **If you do -nothing:** it costs the HP about one and a half gigabytes of memory and nothing else. - ---- - -**Updated 2026-09-13 (evening, before the night) — your four items.** - -**Decisions I took.** (1) The rules file now sits in the workspace root and all three repos, -byte-identical; the gate that checks instruction files required three lines at its top saying it -loads in every session on purpose, so all four copies carry them. (2) The photo fix was applied to -your own test instance on the HP through the Update button, not by hand, and I added one test -account with one photo there. - -**What I did.** The rules file: done. The photo problem: the app's backend serves photos itself, -through a small web server inside its own image; our template sent every request to the front page -server, which has no photos. Fixed in two cuts — the second one because the first spoke to the wrong -port and got empty pictures. Proven without a browser: a photo now comes back as a real image -through the public address, and you confirmed it in your browser at 21:49. The tier-order -ruling: built as controller 0.241.0 — an app whose data lives in mounted folders now prefers the -remote copy over its own-drive copy, and the hold message ends with what the chosen copy holds. -Live on both machines (0.241.0, delivered by the floor in 16 and 18 seconds) and proven on the HP -with a throwaway Nextcloud. - -**What broke on the way.** Nothing standing. One new finding: after an app is removed, its -"held after a failed update" mark stays behind, so a reinstall would start blocked. Cleared by hand, -filed (R-491) for the next release. Register: 206 open / 191 closed. - -**Needs you.** The scratch guest: the ruling as written cannot be built — the hub ties one box to -one customer, and a second enrolled customer on the HP would replace the HP's own enrolment. Two -options: a second guest under the HP's own customer (reversible, tonight-ready, but its reports -would clash with the HP's page unless it stays unenrolled), or a nested appliance VM enrolled as its -own box (fully enrolled, hours to build). My pick: the first, unenrolled. **If you do nothing:** -tonight's rotation keeps restoring in place and skips the standing nine. - -**Updated 2026-09-14 (the night of 13→14 — "be a customer for the night", first run). Written in -the order the rules ask for.** - -**Decisions I took, so you can undo them.** (1) The rules file your brief named -(`.claude/rules/unprompted-work.md`) exists in no repo, and its text did not reach me, so I could not -create it; I worked to the fences written in the brief itself. (2) The nine standing apps cannot be -the night's throwaway on the same guest (same name), so the rotation starts after them (R-481). -(3) There is no scratch guest, so the restore was done in place: remove the app, keep its backups, -restore, read the data back. (4) One controller release (0.240.0), two catalog changes, no hub change. - -**What I exercised.** AdventureLog as a family: install, sign-up, a Balaton trip with three places, -visits, a packing note, an edit and a delete — all through the app's own API; backup now + second -copy; remove with "delete my data"; restore; the guarded Update; remove with "delete backups". - -**What broke.** The second-drive restore was refused after a removal that kept the backups (the box -had forgotten the copy existed). The PostGIS database was not treated as a database, so it had no -proper dump. The backup card said "no backups" over 484 MB. A removed app is invisible on both -backup pages. "Delete backups" left everything behind. The app served Django debug pages to the -internet. Photo upload fails from any non-browser client. A glance fresh install crash-looped. - -**What I fixed and proved live.** Controller 0.240.0 (delivered by the floor in 16 s and 18 s): -the forgotten copy, the PostGIS dump, the backup card, "delete backups" now deletes, the stale -failure sentence, the slow off-site check, the old-install copy. Catalog: DEBUG off for AdventureLog; -glance now lands healthy on a fresh install. Docs and gates: the observations gate reads every section; the -target-selection runbook names real paths; the catalog now refuses an image move that forgets -its `catalog_since` date. - -**What I filed.** R-481 (scratch guest, your decision), R-483 (photo upload needs a browser check), -R-487 (removed apps invisible on the backup pages), R-488 (a 5-minute test package), R-489 (the -removal reports null over volumes it removed), R-490 (the monitoring page's memory card has never -shown, its data call is answered 404). - -**Register.** Before: 432 156 bytes open / 120 598 closed (212 / 177 rows). After: see the top of -`OPEN-ITEMS.md` — 206 open / 190 closed (R-465 audited and closed, R-490 opened). - -**Needs you.** (a) The rules file text — paste it and I create it in all three repos. **If you do -nothing:** nights run on the brief's fences, as tonight. (b) R-481: how to make a scratch guest (a -second enrolled LXC, or an ISO appliance per night). **If you do nothing:** restores stay in-place and -the nine standing apps are never walked. (c) R-483: open AdventureLog in a browser and add a photo -to a place. **If you do nothing:** we do not know whether households can upload photos. (d) R-479 -from the afternoon still waits. - -**Updated 2026-09-13 (fourth pass) — you decided both things. New releases reach the demo machines -by themselves again, and an app with any backup can now be updated. Both are live and proven on the -HP. Nothing needs you.** - -**Updated 2026-09-13 (third pass) — the Update button now takes a backup first and tells the truth. It -is live on both machines and I walked every case on the HP, including putting a broken update back -from its backup. TWO THINGS NEED YOU: item 15 (the demo machines no longer get new releases by -themselves between golden images) and item 16 (apps with no second-drive copy cannot be updated).** Both done the same afternoon. - -**Updated 2026-09-13 (second pass) — you decided both open items. The database engine now finishes -its own conversion on the four MariaDB apps; the upgrade machine proved it and it landed on the HP -without a ripple. Goldens are now weekly and before any install, not per release, and the gate -knows: it reads a dated permission slip that runs out after 14 days. Today's golden (0.236.0) is -baked and live. Nothing needs you.** - -**Updated 2026-09-13 — "delete my data too" now deletes the data, or tells you it could not. Until today the box said it worked and left everything on the drive. Live on both machines (0.236.0), proven on the HP with a throwaway Nextcloud. Nothing needs you. Item 7 is closed: you decided it on 2026-09-02 and the page still listed it open.** - -**Updated 2026-09-06 (third pass) — I chased down the BookStack database problem I found this -morning. Good news: it does not get worse, and fixing it costs seven seconds and loses us nothing. -The catch I expected — that fixing it would stop us being able to go back — turned out not to be -real. TWO THINGS NEED YOU: item 11 (how wide to take the upgrade testing) and item 12 (one small -yes/no on the database setting).** - -**Earlier 2026-09-06 (second pass) — I built a machine that upgrades a real app with real data in it -and then asks the app whether the data is still there. Three apps, five real upgrades: the data -survived every time. It also found a genuine problem in our own BookStack setup.** - -**Earlier 2026-09-06 — a restart no longer changes which version an app runs. Fixes still arrive -every 15 minutes, and a broken app definition still repairs itself. Only the Update button moves a -version now. Live on the HP (0.235.0). Two old items closed: the Hetzner e-mails are answered, and -the Docker Hub login is in place.** - -**Earlier 2026-09-03 — you spotted that OpenGist had no label. You were right, and it was a real gap: -the label only appeared on apps something had restarted. Fixed and live (0.234.0). Every app on both -machines now carries one.** - -**Earlier 2026-09-02 — the box now writes down which version of each app it is running, and shows one -small label saying whether it is up to date: „Naprakész" or „Frissítés elérhető — 52 napja". No version -numbers, and nothing about updating changed.** - -**Earlier 2026-09-01 (third pass) — the backup work is FINISHED for beta, and I have written down -where it stops. One alarm that was telling you something untrue is fixed and live (hub 0.111.1). -ONE THING is waiting on you: two short e-mails to Hetzner, drafted and ready to paste.** - -**Earlier 2026-08-31 — I measured whether the box could test its own remote restore without you, -then built the narrow version you picked. The measurement is why it is 3 seconds a night and -not an evening's work.** - - -> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates -> part of it in plain words, and **nothing may exist only here**. **Items, not paragraphs. One screen.** -> If it does not fit, it belongs in the register instead. - -## Waiting on you - -*This section is allowed to be longer than one screen, and each item says what happens if you do -nothing.* - -1. **Two things are waiting on you: item 11 (how wide to take the upgrade testing) and item 12 (a small yes/no about the database setting — I have measured both costs).** Item 4 (the Hetzner e-mails) is answered and is being handled in a separate session. Item 7 — the safety-copy decision — was ruled on 2026-09-02 and is now marked closed below (it was **not urgent any more**: the thing that made it urgent was that a restart could upgrade an app behind your back, and as of today it cannot. Item 10 is new and needs nothing from you. Item 5's alarm mail can now be ignored for good. Otherwise: Both problems the overnight test found are fixed and proven on - the real machines: - - the background job that could delete a live restore's lock now waits its turn — and the check - that finds the next one like it is a test, not a comment, so it cannot come back quietly; - - the nightly backup check now runs on `demo-felhom`. It said **pass** there this morning, on a - machine where it could not start at all yesterday. - The golden carrying both is baked, vouched, and the floor is raised. `demo-felhom` picked it up - **by itself in about 20 seconds**. - -2. **Whether to keep the test app `bentopdf` on `demo-hp`.** I deployed it last night because it - is the ONLY app of our 53 with neither a database nor stored files — which makes it the only - way to prove the new check does not cry wolf on an app that legitimately has nothing. It - passed silently, which is what we needed to see. **My pick: keep it**, as a permanent control. - **If you do nothing:** it stays, using almost no space. Say the word and I remove it. - -3. **Nothing else about this release.** Everything in 0.232.0 ships in the controller image plus - two register lines in the hub (already live). No customer action, no data migration, no - credential change. - -4. **~~Please send two short e-mails to Hetzner.~~ DONE — you sent them and Hetzner replied (2026-09-06). The reply is being worked in a separate session; nothing about it belongs to the update work.** The background below is kept because it is why the questions were asked. - `felhom.eu/documentation/runbooks/provider-questions-2026-09-01.md` — open it, copy, send. No - password or key is in that file, and none should be added. - - **Why.** Yesterday I told you the snapshots make a wiped backup survivable: lose about a day, copy - the rest back file by file. **The first half is still true. The second half is not, and I found - that out by trying it.** I tried **777,600** snapshot names on the storage, over nine days, in - Hetzner's own naming style. **None of them opened.** Then I found why: your data and the snapshot - door sit on **two different drives** inside the storage, and the door for your data **does not - exist at all**. So there is no way in from the machines. - - **What is still true, and it matters:** a machine that wipes its own backup **still cannot touch - the snapshots of it**. The older copy is there. What we do not have is a way to reach it. - - **The two questions.** One: can the **main** account pull single files out of a snapshot? Two: on - one of Hetzner's own tools, is a "cannot delete" switch forced by them, or chosen by the machine? - **The second one could remove the whole problem** — no new hardware, no moving anyone's data. - - **If you do nothing:** we cannot finish this. The backups keep working and keep being checked; we - simply cannot say what a wiped backup costs, and I would then put this risk back near the top of - your list. **My pick: send both. It is five minutes and it decides an evening's work.** - -5. **You will have received an alarm email from me today about `demo-hp` losing 65 backups. It is a - test and nothing is wrong.** I built the new "someone deleted the backups" alarm and had to fire it - once for real to prove it reaches you. Subject: `[Felhom] 🔴 demo-hp: offsite_snapshots_dropped`. - **No backups were deleted.** **If you do nothing:** nothing — but please do not act on that one - mail. **Closed now:** that was the only such mail, it was a test, and the alarm's wording has since - been corrected (item under *Decided* below). Nothing further is needed from you here. - -6. **Whether to change the hub password** (R-350). I printed it into my own session log on 20 August. - Not in git, not in any saved file — in the log on this machine. **If you do nothing:** it stays as - it is, at the risk you accept by leaving it. I can change it without ever showing you the new one. - -7. **CLOSED 2026-09-13 — you decided this on 2026-09-02, and the page kept listing it open.** The ruling is recorded in `documentation/architecture/09-update-architecture.md` §3: the safety copy is a *verified recent backup as a precondition*, not a new copy made for the update; the guest-snapshot idea is to be spiked before anything is built on it. The text below is kept as the record of what was asked. - - *Original question:* **Where should the safety go before an app updates? This is the one decision from today's - measurement, and it is a design choice, not a bug report.** - - **What I measured.** The box downloads new app versions by itself every 15 minutes and writes them - into the customer's files, whether the app is running or not. Nothing tells the customer. Then the - **Restart** button — not just Update — installs that new version. I watched it download a version - that was not on the machine and swap the app onto it in 18 seconds. **And the box does it on its - own** when an app fails to come back after a crash: nobody pressed anything. - - **One fear is smaller than we thought, and you should have that too.** A plain power cut does - **not** upgrade anything. The apps come back on their old version. It only happens when an app - fails to return. - - **One fear is bigger.** I tested whether we can undo an app update. **We cannot.** Once an app has - moved its data to the new version, putting the old version back gives an app that will not start at - all. So **"rollback" is the wrong word** and I have struck it. The only way back is to restore the - customer's data from a copy taken *before* the update — and today no update takes one. - - **The decision, in one sentence: should the safety copy sit under the Update button only, or under - everything that can install a new version?** - - - **Under the button only.** Cheap and quick. Covers the case a customer causes. **Leaves the - unattended path uncovered** — the one where an app that failed to come back is upgraded with - nobody watching. - - **Under everything.** Covers all of it. Costs more, and it has a hard limit I measured: for a - big app a copy is roughly 30 minutes and about twice the app's size, against a standard box that - ships with 20 GB. **A copy of a large app does not fit.** So this option cannot be built without - also answering where the copy lives. - - **My pick: under everything — but decide the "where does it live" question first**, because the - answer decides whether the rest is even buildable. - - **If you do nothing:** nothing breaks today, and no customer is at risk this week — the fleet is - young and its running versions match the catalog. But the exposure is real and dated: **Peti's box - has a one-major upgrade of `rallly` queued behind its next boot**, from a catalog change made three - days after it went quiet. **Who else is blocked:** nobody can spec the safe-update work until this - is answered, because the two options produce different products. - - Full measurement, with the controls and the quoted output: - `felhom.eu/documentation/audits/SPIKE-app-update-2026-09-01.md`. - -9. **Nothing here needs you. Two things I got wrong, both now fixed.** - - **You found the first one.** OpenGist showed no label. That was not a misunderstanding — the label - only appeared on an app that something had restarted, so an app that simply runs showed **nothing, - possibly for months**. On a quiet machine that is every app, which is exactly the machine we most - want to be able to look at. **Now the box reads what every app is on when it starts up, and writes - it down.** It only looks — it starts nothing and changes no app. Live on both machines: all nine - apps on the HP now carry a label, and so does OpenGist. - - **One thing to expect, so it does not look broken:** the label says **„Naprakész"** when an app is - on the newest version, with **no number at all**. A number only appears when the app is *behind* — - and then it is how long the newer version has been waiting, not how old the running one is. That - was your ruling and I think it is still the right one. - - **The second one was mine and smaller:** a test I wrote pinned a date and an age, so it passed the - day I wrote it and failed the next morning. Fixed, and I have written down that the same trap may - sit in six other test files — named, not accused; someone has to read them. - - The box now writes down which version of each app it is really running, and shows the customer one - small label: **„Naprakész"** or **„Frissítés elérhető — 52 napja"**. No version numbers — a - household cannot act on `26.05.2`. - - **Both halves are proven on the real machine.** I made two apps fail to come back, the box repaired - them itself and wrote down exactly what it installed — one line per container, with the fingerprint - that cannot lie. I checked those fingerprints against the machine independently and they match. Then - I opened the real customer pages and read the labels off them. - - **Where I went wrong, because you should not have to spot it twice.** I told you the saved password - no longer worked on either machine. **It worked fine.** I read it out of the file wrongly — the - value is wrapped in quote marks and I only removed one kind. You caught it in one line. The machine - told me "wrong password", which was true, and I took it to mean the password was wrong when it meant - *what I sent* was wrong. I have filed that as a small job for myself: one shared way of reading that - file, so the next session cannot get it half right. - - **If you do nothing:** nothing. This one is closed. - -10. **Nothing here needs you. The version of an app is now frozen, and only the Update button moves - it.** - - **What used to happen.** The box downloaded the catalog every 15 minutes and wrote each app's new - definition straight over yours — including a new version. From then on, **thirteen** different - things could install that new version: the Update button, the Restart button, or one of eleven - repairs the box performs on its own. Nobody had to press anything. - - **What happens now.** The version is pinned to what you have. Only a deliberate Update moves it. - - **What deliberately did NOT change, because it was worth keeping.** Corrections to an app's - definition — a fixed health check, a memory limit, a new setting — still arrive on the same - 15-minute cycle, and a broken definition still repairs itself. That was a real benefit of the old - behaviour and it is intact. In one sentence: **while the catalog offers the same version you are - running, its fixes reach you; the moment it moves to a newer version, you stay where you are until - you choose to update.** - - **Proven on the real machine**, twice over: I pushed a genuine catalog change with no version in - it and watched it arrive on the normal cycle; then I pushed a version change and watched the app - refuse it. - - **Two things this did NOT do, so they are not read as done.** The Update button is exactly as safe - as it was yesterday — no backup, no undo. That is the next piece of work, and it is item 7. And an - app the box could not confidently pin would have been left behaving exactly as before, loudly; on - the HP that was none of the nine. - - **One rough edge I chose to write down rather than fix:** a frozen app still receives the small - metadata file that carries its health check, so it can be given a check written for a newer - version and look unwell when it is fine. **It cannot lose data — the worst case is a false - alarm.** Freezing that file too would break the „Frissítés elérhető" label, which is a worse trade. - -11. ~~**How wide should I take the upgrade testing?**~~ **DECIDED 2026-09-13: as wide as possible, - through the nightly unattended sessions.** Every app gets its turn as the nightly rotation reaches - it, one hand-written way in per app. The apps that can only be reached through a browser wait for - the sessions that run on your Windows machine with Chrome, where the machine can click. Written - into the update architecture as decision 6. **Nothing to do.** - -12. ~~**Shall I tell the database engine to finish its own conversion?**~~ **DECIDED YES 2026-09-13, - and shipped the same day.** The four MariaDB apps — BookStack, Kimai, Nextcloud, RomM — now carry - one setting that lets the engine convert its own files when it moves to a new major version. - Seven seconds, and it backs itself up first. **Proven before it shipped:** the upgrade machine - re-ran the BookStack edge and this time the engine says, in its own words, that it is already - upgraded — the "skipped" line is gone, and the data read back afterwards. **Watched as it landed:** - the change reached the HP on the normal 15-minute cycle, nothing restarted by itself, and one - deliberate restart came back clean with the app serving. **The rule until the next piece is built:** - the Update button still takes no backup, so no app may move a database engine across a major - version until it does — a gate refuses such a change at push time. **Nothing to do.** - -13. **"Delete my data too" now deletes the data — or tells you it could not.** Until today, when a customer removed an app and ticked the box, the box said it worked and left everything on the drive (128 MB of a Nextcloud on 2026-09-01). The cause: the removal asked one global setting for the drive, and no machine fills that setting in. Every other part of the box already asks the app itself where its data is. Now the removal does too. If the box cannot work out where the data is, it refuses and keeps the app, so you can try again — it never again reports success over data left behind. Proven on the HP with a throwaway Nextcloud: 63 MB the app wrote itself was gone after removal, and the answer listed it; the refusal was shown with the app still in place; an app with no drive data gets a plain "nothing to delete" note. Live on both machines (0.236.0). No standing app was touched. **If you do nothing:** nothing to do. - -14. **The Update button now takes a backup first, and tells the truth.** Nothing needs you. - Before today, pressing Update pulled the new version at once, checked nothing, and said - "done" while the app could already be crashing. Now: - - It refuses first if it must: the app is held, a backup is running, or memory or disk is short. - It also refuses if the app has no backup it could be put back from. - - If the backup is older than a day, it makes a fresh one first. - - It waits for the app to actually be healthy before it says it worked. The button shows each step. - - If the new version does not come up, the app is stopped and held. The page names the backup to - restore it from. The box never puts the old version back by itself, because we measured that - this works for some apps and breaks others. - - **Proven on the HP with a throwaway app:** a real upgrade, a stale backup, a version that does not - exist, a version that never starts, and the restore from the named backup back to the old version. - **One problem showed up during the test and is fixed:** while an update was waiting to see if the - app came up, the regular backup copied the broken version into the app's local backup. It now - leaves an app alone while it is updating. Live as 0.238.1. - -15. **New releases reach the demo machines by themselves again.** You decided this, and it is done. - When I raise the floor for a release, I now also type which agent that release needs. The hub - then moves the machines past the golden image. If I leave that value out, the hub refuses to save. - Release 0.239.0 reached both machines this way in 15 seconds. Nothing needs you. - -16. **Apps with no copy on a second drive can now be updated.** You decided this, and it is done. - The update now uses any backup: the second drive first, then the app's own copy, then the remote - copy. If none is younger than a day, it makes a fresh one first. If the new version fails, the page - names which backup to restore from. Proven on the HP: an app with only its own copy updated, a - broken update was held, and it came back from its own copy. Nothing needs you. - Four small things turned up and are written down: the remote check can take 15 seconds (R-477), - a leftover backup of a removed app counted as a fresh one (R-478), for some apps their own copy - holds only settings (R-479), and the card keeps the old failure sentence after a good restore (R-480). - -8. **`demo-hp`'s network setup does not match our own notes** (R-338) — the machine works, the page is - wrong, or the other way round. **If you do nothing:** the page keeps misleading the next session, - as it misled one by an hour. - -## Decided — and what would reopen each - -- **GOLDENS ARE NOW WEEKLY AND BEFORE ANY INSTALL, NOT PER RELEASE — AND THE GATE KNOWS. DECIDED 2026-09-13.** - In August I baked 25 goldens in 26 days, almost one per release, because the check trips on every - release on purpose. From today: one golden a week, and always before a drill or a fresh install. - **Correction, same day:** I wrote that every release would still reach both demo machines in about - 20 seconds between bakes. **That was wrong at first:** the hub would not move the machines past the - golden image. **It is true again since the afternoon:** the floor now carries the release when I type - which agent it needs (item 15). Release 0.239.0 arrived in 15 seconds. The check now reads a dated permission slip that runs out - after at most 14 days; while it is valid the check warns instead of refusing, and when it runs out - the check is red again until someone bakes or renews. A dated slip cannot be forgotten — it just - expires. Today's golden (0.236.0) is baked, checked three ways, and live. - **Reopens if:** the first outside customer installs (the slip is retired then), or a fresh install - ever lands on a golden older than the week. - -- **THE BACKUP AND RESTORE WORK IS FINISHED FOR BETA. DECIDED 2026-09-01.** - **What is done, and proven on the real machines:** everything **you or a customer** does alone — - getting deleted files back, getting an app's data back, getting a whole app back, and losing a - drive. The restore tells you what it put back, refuses if there is no room, will not accept a - half-copy, and puts your own data back if it fails. - **What is parked until after beta:** everything **only I do, with you** — rebuilding a machine as - itself, losing a whole box, recovering from ransomware, restoring the hub, and losing Hetzner. - **Six of these have never been timed, and the hub has never been restored.** They are written down, - they are real, and **none of them stops a beta customer.** - **Reopens if:** something a customer does for themselves turns out to be broken; **or** Hetzner's - two answers change what the snapshots are worth; **or** a real customer's data is at stake in one - of the parked items. - -- **The alarm that promised too much: FIXED and live (hub 0.111.1). DECIDED 2026-09-01.** - Yesterday's alarm mail said a deleted backup was *"recoverable file-by-file"*. **We now know it is - not.** I removed the promise rather than writing a new one, so the sentence stays true whatever - Hetzner answers. It no longer says the data is lost either — that is still usually untrue. - **Reopens if:** Hetzner's answers give us a real route back; then the alarm can name it. - - -- **The missing-golden warning is now pointed at the people who can act on it (R-404). DECIDED - 2026-09-01, and built the same day.** The warning was aimed at the wrong repository: the one where - a release actually happens never checked at all, while the one that only holds documents was - refused on every push — including the push that RECORDS a golden bake, which is the very act that - clears the warning. So the check was blocking its own cure, and we had skipped it thirteen times. - **What changed:** the release repository now prints a reminder the moment a release is committed - (it never blocks — you cannot bake a golden for a version you have not pushed yet), and the - documents repository still runs the check on every push and still says so loudly, but only refuses - a push that touches real code. **Every other check still blocks everything, always.** Nothing was - silenced and no product code changed. **Reopens if:** a release ever ships without a golden and - nobody noticed — that would mean the reminder is not reaching anyone, and the answer would be to - make the release repository refuse rather than remind. - -- **Getting old backups back yourself: NOT BUILT, deliberately.** **Reopens if:** a real customer - asks. *(R-312)* -- **The unopenable old copy on `demo-felhom`: KEPT as a test fixture** — the only state in existence - where a set-aside store is present and cannot be opened. **Delete when:** that work ships or is - abandoned. *(R-313)* -- **A machine in two kinds of trouble says both things: LEFT AS IT IS.** **Reopens if:** observed - outside a constructed test. *(R-303)* - -## What works - -Both demo machines are home, healthy and reporting — agent **0.130.0** published and running on both. -**Which controller each box runs, and where the floor sits, is item 1 above and is not restated here** — -R-395: this paragraph carried a second copy of those numbers, it went stale by seven releases, and the -page then disagreed with itself about the thing an operator checks first. Ask the hub (`/hosts`, -`/configs`) or the box for what is live; a doc is never the authority on a version. -Off-site is credentialed on `demo-hp` and its store opens with the machine's own key. - -**The fleet, because two summaries have been misread:** five customer records, three machines. -`demo-felhom` and `demo-hp` are ours and disposable; `drill-r50` is a nested drill VM, reverted and -off. **`peti-felhom` is a real machine we have not heard from since 15 July** and has no host record. -**`tester-1` is a record with no machine.** - -## Shipped - -- **Something finally checks that the off-site copies are still there and readable** (R-359 + R-397, - controller 0.227.1, proven on `demo-hp`). Until today **nothing did** — not the box, not the agent. - The whole-machine backups had their own checks; the copies holding your customers' documents and - photos had none, so we would have found a problem at restore time, with a customer waiting. - Now the box checks its own off-site store about once a week and tells you only if something is - wrong. **A pass sends no e-mail, on purpose** — a weekly "everything is fine" is how people stop - reading their alerts. **It also catches itself up:** it asks „has it been more than seven days?", - not „is it Sunday?", so a machine that was switched off on its check day is checked the next day. - **And it never gets in the backup's way** — if a backup or restore is running, the check steps aside - and tries again tomorrow. That is not politeness: the tool would otherwise clear a lock that a live - backup was holding. Proven live, twice over — a real check against your real store (35 seconds), and - a second check fired during the first, which correctly stepped aside without doing anything. - **Read item 2 under „Waiting on you" for what this check does NOT see.** -- **The product stopped claiming a check it never ran.** The monitoring page said an integrity check - ran every Sunday. It did not exist. The e-mail text, the settings checkbox, the hub's side of it and - a debug button were all built and wired to nothing (R-397). They now have the missing piece. - -- **Taking the safety copy no longer destroys the app's own backup** (R-361, controller 0.221.1, - proven on `demo-hp`). Before every restore the machine saves a copy of your live database. To do - that it called the ordinary backup routine — **which always writes to the app's normal backup - filename first** — so the app's real backup was overwritten and then renamed away. Until the next - nightly run the app had **no database backup of its own**, and a local recovery in that window would - have told you the app never had a database. A comment in the code said this could not happen; it - could, and had been happening for four months. Proven fixed the only way it can be: the app's own - backup file is now **byte-identical before and after a restore**, on both database types. -- **A failed database restore now puts your data back by itself** (R-379/R-380, controller 0.220.2, - proven on `demo-hp`). Until today, if a restore of an app's database went wrong, the machine had - already taken a copy of your live database — a good copy — and **nothing in the product could put it - back.** You were shown a filename. On one of the two database types it was worse: part of the - restore applied, part did not, and **the dashboard said the app was healthy**. Now the machine puts - your own copy back automatically and says plainly: the restore failed, your data is as it was, the - app is running. Proven on both database types, byte-identical both times. - **If even that fails**, the app is deliberately **stopped and held** rather than started — a running - app on a half-written database lets you type into it and makes the damage permanent — and you are - told to contact us. That was your ruling this morning. **Two things also stopped:** the error no - longer pastes raw database text at you (it was 615 bytes once, including rows out of your own - database), and the undo copies no longer pile up forever — three per app, and they were being copied - off-site permanently. -- **An empty package can no longer wipe out a good one** (R-403, controller 0.230.0). Yesterday we - wrote this down as *suspected* and said plainly it had not been tested. **We tested it first, and it - was real.** On a demo machine, on yesterday's build: an app's copy on the second drive went from - **120 MB — four database backups and three data archives — to 7 KB, nothing left**, in a single - nightly run, and the run reported success. The cause was that the nightly job only asked *does the - folder exist* before copying over it, and an empty package is a folder that exists. - Now the nightly job refuses to replace a **complete** package with an **empty** one. It keeps what it - has, says so on the app's own backup page, and carries on with everything else. Proven on the same - machine, in the same state: **all seven files still there, byte for byte.** - Two more things came with it. The page no longer calls that copy fresh when the run did not refresh - it — it names the real date of the package instead. And after a restore from the second drive, the - first drive's package is filled back in immediately, so the empty state that started all this cannot - happen again. A copy that legitimately gets smaller still gets smaller; only the one dangerous case - is fenced. -- **The copy on the second drive can now bring an app back** (R-102 + R-103, controller 0.229.0, - proven on `demo-hp`, **and delivered** — golden 0.229.0 is vouched and the fleet floor is raised, so - a machine installed today has it). Every night the box copied each app's whole recovery package onto the second - drive — its settings, its database and its data. It did that for months. **Nothing could open those - copies.** No button, no screen, no command. That mattered most in the one fault the second drive - exists for: if the first drive dies, the package on it dies too, and the copy that survived could - not be read. For **45** of the 53 apps that is everything they own. - Now the same restore that always worked from the first drive can read the copy on the second one, - and the button is on the app's own backup row. **Proved with the first drive's package taken away:** - Docmost came back in 29 seconds — its database, all three data volumes, a file with a Hungarian - accented name back byte for byte, and the app then read its own rows with its own password. Done - again with the app's password file also taken away: the copy carried the passwords too (2 of 2). - The screen that used to say „press that other button on another page" now offers the action itself. - It says plainly that **this one overwrites** what is there — the gentle „Fájlok visszaállítása" - beside it still only adds back missing files — and it names the date of the copy, so nobody puts - last week over today by accident. -- **We counted the apps this affects, and settled it.** Two of our own notes disagreed — 43 or 45. - The answer is **45**, counted with the product's own rule against the live catalogue. The older - count missed **radarr and sonarr**. -- **The off-site restore now works for the other 40 apps** (R-356, controller 0.219.0, proven on - `demo-hp`). It used to refuse before starting, tell the customer a running app „nincs telepítve", - and send them to reinstall it "to the same place" — a place those 40 apps never offer, because they - were never given a drive to choose. It was asking one question to answer two. Proven today on - `privatebin`: data planted through the app itself, backed up, **deleted**, restored — **all 15 files - back byte for byte**, Hungarian accented names included, message „0 fájl és 1 adatkötet - visszaállítva". The 13 apps that do have a drive are unchanged, checked the same way. -- **The off-site restore gives an app's data back at all** (R-354, controller 0.218.0). It used to say - „0 fájl visszaállítva", report success, and the folder was simply not there. It now names what came - back, because a restore that mentions only its file count is how a silent loss reads as a success. -- **Paperless's database is in the backup, and restoring it takes an undo copy first** (R-355, - controller 0.218.0). The dump was landing in a folder named after an app that does not exist. **One - app of 53 was affected**, established with a check first proved able to catch a planted second case. -- **The system tells you when it cannot see the off-site copies** (R-339) — a mail after ~30 minutes, - hourly while it lasts, one all-clear. **Caveat:** it watches whether the machine answers, so it - would *not* have caught the 18 August fault, where one service was wedged and the machine stayed - healthy. -- **The connection leak was ours and is fixed** (R-344). Our agent opened a connection to the off-site - box every 15 minutes and never closed it; the idle timer was switched off. Proved by fixing one - machine and leaving the other: same work, 4 more leaked on the untouched one, none on the fixed one. - The off-site box is back to **17** open connections from **415**. -- **A dated check can no longer be quietly missed** (R-341) — but it speaks on the next push, not on - the day. **A machine we tell to be quiet is no longer reported as dead** (R-321). **One name per - secret** (R-295, R-323). **The hub's own words are under a guard** (R-324). **Removal reverses the - installation** (R-316). **A correct recovery code is no longer called wrong** (R-311). **The drive - can be re-attached after a reinstall** (R-280). - -## Broken, or knowingly incomplete - -- **We do not know what the deep check costs on a BIG store** (R-401). Since 0.228.0 the weekly check - re-reads **all** your stored data, not just the list of it. We had to: a copy was damaged in a way - that left its size unchanged, and the old shallow check said „no errors were found". Only the deep - check caught it. **The cost we measured was four seconds** — 35.0 s before, 39.2 s after — but that - was on a 134 MB store, and it will not stay four seconds. So the box now tells us: any check that - takes longer than five minutes writes a warning naming this item. **If you do nothing:** every - machine re-reads its whole store every week, however large it grows, and the first person to notice - would be a customer whose upload is busy. The warning is there so that does not happen. -- **We ask the off-site box a question about once a second** (R-336) — ~85,000 a day for a box we - write to weekly. The leak that made this dangerous is fixed (R-344); the volume is not. The ceiling - is **under a year** away on the corrected measurement, not two. -- **Peti's machine has no recovery route at all.** A real machine belonging to a real person, silent - since 15 July, no key, no off-site copy, no local backup. **If that drive fails, everything on it is - lost.** First act of any visit: copy the ~3.6 GB off before anything is reinstalled. -- **The agent picks dnsmasq by looking at a file another package owns** (R-317) — one line; LAN name - resolution goes missing quietly. -- **Three facts the machines send still have no reader** (R-264); **the storage page has its own - reason for an empty list** (R-298); **two thirds of the standing picture is unproven** (R-326: - 23 of 55 claims walked — `python3 scripts/unproven.py`); **the picture still describes one defect we - fixed twice** (R-327). - -## Working on next - -The 2026-08-09 batch (R-279 … R-292), still untriaged; the three remaining R-264 readers; R-317 (one -line in the agent); R-327 (decide the naming claim's status); R-359 (nothing reads the off-site store). - +1. **Rotate the Gitea `admin` token** — still open from yesterday. The machine stores it in plain text in its copy of the catalog. *If you do nothing:* the token keeps working and anyone with yesterday's session transcript has it. +2. **Clear 47 alarm e-mails** from yesterday's drill, in one search: `subject:"gates FAILED in admin/app-catalog-drill"`. The cause is fixed. *If you do nothing:* your alarm inbox stays noisy, and that is the inbox that must never be skimmed. +3. **Decide whether the Nextcloud engine move stays.** I explained my reasoning above. *If you do nothing:* it stays, and Nextcloud households will be offered a database upgrade that was proven once on a scratch machine. +4. **The 28 untested apps.** *If you do nothing:* the nightly rotation works through them at a few per night, and the catalog's update promise rests on nothing for those apps until it gets there. +**Nothing on your own machine, the tester's machine, or the off-site box was touched. No product code was written.** diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index dfd401ee..c4a9d9e6 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -102,7 +102,7 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis |---|---|---|---|---| | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | `CAMPAIGN-2` T-DEPLOY-SET (7 apps, env config, health-aware); `RERUN-p1p3` (×4 PASS) | Memory-guard FIRING is not live-shown (T-RES-MEMGUARD never fired: ample RAM / auth-walled) — implemented + unit-level only | | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE — the ACTIONS work. NARROWED 2026-09-13: `CAMPAIGN-3` proved `remove` removes the APP, not the DATA — the "delete my data" half was INERT on every box until controller v0.236.0 (R-442). RE-PROVEN 2026-09-13 on demo-hp: data written by the app itself (63 MB) gone after removal and listed; an unresolvable data location is REFUSED (409) with the app kept; an SSD app gets `[]` and a note.** | `CAMPAIGN-2` T-LIFECYCLE (stop/start/restart/update/logs); remove (app only) live in `CAMPAIGN-3`; **remove WITH data: `audits/R442-2026-09-13/`**; **data behaviour: `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01)** | Redeploy-after-remove edge remains open (T-REMOVE-REDEPLOY never cleanly passed — stale dryrun journal); non-pilot-critical | -| **Update is GUARDED: it refuses without a restorable backup, backs up first when the copy is stale, and HOLDS an app that does not come up — on ANY backup tier, and the release itself arrives by the managed floor** | controller **v0.237.0 + v0.238.0 + v0.238.1 + v0.239.0**, hub **v0.112.0** | **PROVEN-LIVE (2026-09-13, and again the same afternoon for any tier + floor delivery)** — **afternoon (`audits/rulings-r472-r475-2026-09-13/`):** an undeclared floor above the golden refused with nothing stored (02); a declared floor 0.239.0 / MinAgent 0.129.0 served `from declared` and both demo boxes self-updated in 14 s and 15 s (03); nothing on any tier → backed up first, Tier 1 chosen, done (04); gokapi updated on its Tier-1 unit alone (05); a never-healthy update held naming „saját meghajtó" (07); restored from „helyi", hold cleared (08). **Morning:** scenarios A (real upgrade, success only after health), B (stale copy → backup first), E (pull failure → pin back, app untouched), F (never healthy → held, hold text on API and page), H (start/restart/update and the boot sweep all refuse the held app) and **the restore walk** (Mentések unit restore → back on the old version, hold cleared), on demo-hp with a throwaway app | **`audits/slice4-2026-09-13/`** (live/, redproofs/, gates/); design `architecture/09-update-architecture.md` §6.1 | ~~**Tier-2-only precondition**~~ — superseded by v0.239.0 (any tier, R-475 CLOSED); a Tier-1 route back restores only what the unit holds (R-479); the card keeps the failure sentence after a successful restore (R-480); no automatic rollback, by measurement; a release does not reach the fleet by floor between golden bakes (R-472) **WIDENED 2026-09-21 (the update night) from 3 apps to 21 edges across 19 apps, and NARROWED in one place by the same run.** `audits/DRILL-update-night-2026-09-21.md`. On scratch guest 9202 (controller v0.261.0), against a **private drill catalog** so the live catalog carried no test reference at any point, 21 edges across 19 apps real within-a-major upstream edges were walked through the product's own guarded Update, each app seeded and read back **through its own front door** (R-156) with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive.** **What the PROVEN edges prove, precisely:** the app moved, the four version observables agreed, and the data the app itself was given came back through the app's own interface afterwards. Ten of them printed a verbatim migration line. **What the FAILED edges prove, and they are the more valuable half.** `adventurelog` (a real upstream edge that migrates and then never serves), `tandoor` (an update that SUCCEEDED and was stopped by its own wrong health port), and the PostgreSQL engine major, which refused exactly as predicted. `adventurelog v0.12.1 → v0.13.0` applied **nine database migrations successfully** and then never bound its port; the update held after the full health wait, the hold sentence named the tier, the date and what the copy holds, and the restore the sentence names brought the app back. **That is this row's own promise, exercised on a real upstream edge rather than a staged one.** **AND THE NARROWING, which this row must carry because it is the same mechanism:** the `verifying` phase trusts the `.felhom.yml` probe absolutely, and **two of the 53 templates name a probe the app does not answer** — `tandoor` (port 8080; it listens on 80) and `zipline` (`/api/health`; it answers 404 there, while the compose healthcheck in the same file uses `/api/healthcheck` and is green). For those apps a **successful** update is stopped by its own health wait: tandoor was measured **serving HTTP 200 on the new version at four samples across five minutes**, with docker's own healthcheck green, and was then stopped by `failAndHold` and the household sent to a restore they did not need. **R-618, P1.** No data was lost and the restore works — but "the update is guarded" must not be read as "the guard is right about whether the app came up". **Still true and unchanged:** no automatic rollback (by measurement); the route back is the restore; a multi-major jump ends held honestly. **Not measured on this venue, and named rather than assumed:** every event and every customer mail. Guest 9202 runs `hub.enabled: false` and the notifier returns before it logs (**R-620**), so the whole "who was told" half of `08` was structurally unobservable tonight. | +| **Update is GUARDED: it refuses without a restorable backup, backs up first when the copy is stale, and HOLDS an app that does not come up — on ANY backup tier, and the release itself arrives by the managed floor** | controller **v0.237.0 + v0.238.0 + v0.238.1 + v0.239.0**, hub **v0.112.0** | **PROVEN-LIVE (2026-09-13, and again the same afternoon for any tier + floor delivery)** — **afternoon (`audits/rulings-r472-r475-2026-09-13/`):** an undeclared floor above the golden refused with nothing stored (02); a declared floor 0.239.0 / MinAgent 0.129.0 served `from declared` and both demo boxes self-updated in 14 s and 15 s (03); nothing on any tier → backed up first, Tier 1 chosen, done (04); gokapi updated on its Tier-1 unit alone (05); a never-healthy update held naming „saját meghajtó" (07); restored from „helyi", hold cleared (08). **Morning:** scenarios A (real upgrade, success only after health), B (stale copy → backup first), E (pull failure → pin back, app untouched), F (never healthy → held, hold text on API and page), H (start/restart/update and the boot sweep all refuse the held app) and **the restore walk** (Mentések unit restore → back on the old version, hold cleared), on demo-hp with a throwaway app | **`audits/slice4-2026-09-13/`** (live/, redproofs/, gates/); design `architecture/09-update-architecture.md` §6.1 | ~~**Tier-2-only precondition**~~ — superseded by v0.239.0 (any tier, R-475 CLOSED); a Tier-1 route back restores only what the unit holds (R-479); the card keeps the failure sentence after a successful restore (R-480); no automatic rollback, by measurement; a release does not reach the fleet by floor between golden bakes (R-472) **WIDENED 2026-09-21 (the update night) from 3 apps to 21 edges across 19 apps, and NARROWED in one place by the same run.** `audits/DRILL-update-night-2026-09-21.md`. On scratch guest 9202 (controller v0.261.0), against a **private drill catalog** so the live catalog carried no test reference at any point, 21 edges across 19 apps real within-a-major upstream edges were walked through the product's own guarded Update, each app seeded and read back **through its own front door** (R-156) with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive.** **What the PROVEN edges prove, precisely:** the app moved, the four version observables agreed, and the data the app itself was given came back through the app's own interface afterwards. Ten of them printed a verbatim migration line. **What the FAILED edges prove, and they are the more valuable half.** `adventurelog` (a real upstream edge that migrates and then never serves), `tandoor` (an update that SUCCEEDED and was stopped by its own wrong health port), and the PostgreSQL engine major, which refused exactly as predicted. `adventurelog v0.12.1 → v0.13.0` applied **nine database migrations successfully** and then never bound its port; the update held after the full health wait, the hold sentence named the tier, the date and what the copy holds, and the restore the sentence names brought the app back. **That is this row's own promise, exercised on a real upstream edge rather than a staged one.** **AND THE NARROWING, which this row must carry because it is the same mechanism:** the `verifying` phase trusts the `.felhom.yml` probe absolutely, and **two of the 53 templates name a probe the app does not answer** — `tandoor` (port 8080; it listens on 80) and `zipline` (`/api/health`; it answers 404 there, while the compose healthcheck in the same file uses `/api/healthcheck` and is green). For those apps a **successful** update is stopped by its own health wait: tandoor was measured **serving HTTP 200 on the new version at four samples across five minutes**, with docker's own healthcheck green, and was then stopped by `failAndHold` and the household sent to a restore they did not need. **R-618, P1.** No data was lost and the restore works — but "the update is guarded" must not be read as "the guard is right about whether the app came up". **Still true and unchanged:** no automatic rollback (by measurement); the route back is the restore; a multi-major jump ends held honestly. **Not measured on this venue, and named rather than assumed:** every event and every customer mail. Guest 9202 runs `hub.enabled: false` and the notifier returns before it logs (**R-620**), so the whole "who was told" half of `08` was structurally unobservable tonight. **THE NARROWING ABOVE WAS CLOSED THE NEXT DAY, 2026-09-22 — and re-widened the row.** `audits/PROBE-FIX-2026-09-22.md`. All three wrong probes were corrected in the catalog (`app-catalog-felhom.eu@793c4fb`: tandoor `8080→80`, wger `80→8000`, zipline `/api/health→/api/healthcheck`) and **red-proofed live on 9202 through the product in both directions**: at the live pin all three read `Nem egészséges` / `Not healthy` on their own app page while docker reported every container healthy and the front door served a real page; after the real sync all three read `Fut` / `Running` with no redeploy. **tandoor's edge was then re-walked with nothing else changed and ended `done` at +41.1 s**, seed read back, where the identical edge had ended `failed` at +361.9 s with the app stopped — so the tally is now **15 proven, 2 failed, 4 inconclusive**, and all fifteen are on the live catalog. A `--fast` catalog gate (`check-probe-matches-compose.py`) now refuses a probe that does not match the same service's own compose healthcheck, with four red-proofs and ten decoys including the no-PyYAML mode CI actually runs. **WHAT THIS ROW STILL CANNOT CLAIM, and the reason is exactly R-96 rule 3:** the guard is now shown correct for **47 of 53** templates. `paperless-ngx`'s probe has **never run on any box** — no container name matches its stack name, so it is silently skipped and its badge can never go red (**R-630**); and five more cannot be judged statically, one of which (`home-assistant`) is right only because its check type cannot fail (**R-631**). An absent alarm is equally consistent with healthy and with never checked. **And the sweep's ceiling, counted: 28 of the 53 templates have never been deployed by any drill (R-632).** | | **What `restart` and `update` do to a deployed app whose compose file the catalog already moved** | controller **v0.235.0** | **CHANGED 2026-09-06 — they NO LONGER upgrade it.** The row below records what shipped; this text records what it replaced, because every box under v0.235.0 still behaves the old way. **Up to v0.234.0: PROVEN-LIVE (2026-09-01) — they UPGRADE it.** Every lifecycle action ends in `docker compose up -d`, which makes the container match the file and PULLS the image itself when it is missing (measured: 18.3 s with a pull, 0.5 s without; negative control with an unchanged file did not even recreate the container). This is DELIBERATE on the restart path — `Manager.RestartStack` says so in a comment — but the syncer moves the file under a deployed app on a 15-minute cycle with no deployed check (R-438), and NOTHING tells the customer. | `audits/SPIKE-app-update-2026-09-01.md` §2, §3 | **No safety copy is taken by any of them** — `writeSafetyDump` is DATABASE-ONLY and is not on the update path at all. R-438, R-440, R-443. | | **Whether the box UPGRADES an app by itself, with nobody pressing anything** | controller | **PROVEN-LIVE (2026-09-01) — YES, but only when an app fails to come back.** A plain power cut does NOT upgrade: Docker's `restart: unless-stopped` restores the old containers and the reconciler logs `no boot-orphaned apps (nothing to start)`. When an app does NOT return, `Reconciler.Run` (`bootrecon.go:269`) calls `StartStack` -> `compose up -d` and the app comes back on the NEW version, unattended (measured). **13 non-API call sites across 9 files reach `up -d` this way** — not the five previously believed. | `audits/SPIKE-app-update-2026-09-01.md` §2, §8 | The drive-return gate (`intermediary.go:222`) and `AppStopGuard.Recover` (`appstop_marker.go:283`) call the same function; located by reading, **not exercised live** — stated as such. | | **Whether an app UPGRADE can be undone** | controller + catalog | **PROVEN-LIVE (2026-09-01) — NO, and "rollback" is the wrong word for it.** Once a migration has RUN, putting the old image tag back yields a container that refuses to start: Nextcloud — *"the version of the data (32.0.9.2) is higher than the docker image version (31.0.14.1) and downgrading is not supported"*. A 3-major jump is refused outright (*"only possible to upgrade one major version at a time"*) and IS recoverable, precisely because nothing migrated. Positive control: the data is not destroyed — returning to 32.0.9 restored both seeded markers byte-identical. | `audits/SPIKE-app-update-2026-09-01.md` §7 | The only route back is restoring DATA from a copy taken BEFORE the update — which no update path takes. And a restore's image-level rollback is itself overwritten by the syncer within 15 minutes (R-441). R-40 is confirmed live by the same measurement. | diff --git a/documentation/architecture/09-update-architecture.md b/documentation/architecture/09-update-architecture.md index 1a5f2495..66194441 100644 --- a/documentation/architecture/09-update-architecture.md +++ b/documentation/architecture/09-update-architecture.md @@ -1095,6 +1095,21 @@ Version strings stay in the logs, the API and the hub. name a probe the app does not answer**, so a SUCCESSFUL update of those apps ends by STOPPING a working app (**R-618**, P1 — tandoor measured serving HTTP 200 on the new version at four samples across five minutes, then stopped). + **CLOSED 2026-09-22, and the closing changed the numbers above:** the three probes were + corrected (`app-catalog-felhom.eu@793c4fb`), red-proofed live on 9202 in both directions, and a + static gate now refuses a probe that does not match the same service's own compose healthcheck. + **tandoor's edge was re-walked with nothing else changed and ended `done` at +41.1 s** where it + had ended `failed` at +361.9 s — so the tally is **15 proven, 2 failed, 4 inconclusive**, and all + fifteen are on the live catalog since 2026-09-22. + **WHAT THE CLOSING FOUND, and it is the part worth carrying forward:** a wrong probe was the + LOUD failure. Two quiet ones sit beside it. `paperless-ngx` has no container whose name matches + its stack name, so `findProbeContainer` returns nothing and **its probe has never run on any box** + — an absence, with no badge to contradict it (**R-630**). And five more templates cannot be judged + statically at all, one of which (`home-assistant`) is correct only because its check type cannot + fail (**R-631**). **So "the probe is right" is now enforced for 47 of 53 templates and still + unknown for six.** + **AND THE SWEEP'S REAL CEILING, counted rather than felt: 28 of the 53 templates have never been + deployed by any drill** (**R-632**) — the widening above went from 3 apps to 21, and 21 is not 53. 9. **The hub does not record image tags at all.** Its report's container payload carries name, state, CPU and memory, and no image field (spike §5). So the fleet view of §6 slice 7 needs a hub-side change; it is not derivable from what is already reported. diff --git a/documentation/audits/PROBE-FIX-2026-09-22.md b/documentation/audits/PROBE-FIX-2026-09-22.md new file mode 100644 index 00000000..47899884 --- /dev/null +++ b/documentation/audits/PROBE-FIX-2026-09-22.md @@ -0,0 +1,247 @@ +# THE MORNING AFTER — probe fix, gate, and the promotion train, 2026-09-22 + +Evidence: `probe-fix-2026-09-22/`. The night this follows: `DRILL-update-night-2026-09-21.md`. + +--- + +## Not done, or changed from the brief + +**Read this first. Everything else in this document is a claim; this section is where the claims +are bounded.** + +1. **The brief said "the fourteen proven versions". FIFTEEN moved.** The night's fourteen included + `nextcloud`'s MariaDB engine move; tandoor was re-walked today and became the fifteenth. Both are + named below with why. + +2. **I nearly dropped `nextcloud`'s engine move on a wrong assumption, and the gate corrected me.** + I assumed `check-engine-major.py` would refuse `mariadb:11.6 -> 12.3` and had written it up as + "dropped, named". Running the gate against that exact commit instead of assuming it ALLOWED the + move by name, citing **R-469** (Slice 4 shipped; `MARIADB_AUTO_UPGRADE=1` is already in that + template, verified by reading it). The brief forbade **PostgreSQL** engine moves; this is MariaDB + and it was proven end to end in 217.4 s. It moved. **This is the second time in two days that + running the instrument beat reasoning about it** — see R-628. + +3. **The first push of the fifteen moves FAILED CI, and I found it by checking rather than by being + told.** Job **877** on `15d7c2b` read `conclusion: failure`: the CI runner has no PyYAML and the + new gate answered INCONCLUSIVE, which the runner rightly refuses to call a pass. Fixed with a + degraded line-reader mode and five more decoy cases; job **878** on `1ad1f34` is `success`. + **The moves themselves were never in doubt — the gate that shipped beside them was.** + +4. **bookstack's phase trace on demo-hp is INCOMPLETE and the reason is my own instrument.** A + 115-second probe run, meant only to list which apps were installed, pressed the Update as + designed and then hit its timeout mid-update. Its phases are recorded to `+21.6 s starting`; the + rest was read off the box afterwards (`done`, pin and installed both at `26.05.5`). The second, + full run then pressed Update on an already-current app and completed in **3.1 s**, moving + nothing. That second trace is real and is reported, but it is **not** a `26.05.2 -> 26.05.5` + trace. Said here rather than presented as one. + +5. **The brief asked me to "list templates the night's sweep could not judge". The honest answer + needed three categories, not one** — 20 with a verdict record, 4 deployed as props with no edge + walked, 1 deployed only to measure its probe, and **28 never deployed at all**. Filed as R-632 + with the machine-readable list. + +6. **Two things the brief could not have known, both found by the new gate and both left OPEN:** + `paperless-ngx`'s health probe has never run on any box (R-630), and five templates cannot be + judged by any static rule (R-631). Neither is fixed here: R-630's fix is product code or a + container rename on live boxes, and the brief forbade product code. + +**One brief claim that turned out wrong:** none. All three probe faults the brief named were +verified against the files before any edit and all three were exactly as stated — tandoor's compose +line 42 dials `127.0.0.1:80/accounts/login/`, zipline's line 38 dials `/api/healthcheck`, wger's +line 42 dials `127.0.0.1:8000`. + +--- + +## Part 1 — the probes + +### 1.1 The fix + +One commit, `app-catalog-felhom.eu@793c4fb`. No `image:` line moved, so no `catalog_since` moved. + +| app | was | is | the oracle that was already in the file | +|---|---|---|---| +| tandoor | port `8080` | port `80` | `wget --spider -q http://127.0.0.1:80/accounts/login/` | +| zipline | path `/api/health` | path `/api/healthcheck` | `http.get('http://127.0.0.1:3000/api/healthcheck', …)` | +| wger | port `80` | port `8000` | `wget --spider -q http://127.0.0.1:8000` | + +### 1.2 Red-proofed live on 9202, through the product, in both directions + +Deployed at the **live (unfixed) pin** first. Evidence `probe-fix-2026-09-22/probe-before-observe.json`, +`front-door-before.json`, `probe-after.json`. + +| | tandoor | zipline | wger | +|---|---|---|---| +| **BEFORE — controller state** | `unhealthy` | `unhealthy` | `unhealthy` | +| **BEFORE — the app page, HU** | `Nem egészséges` | `Nem egészséges` | `Nem egészséges` | +| **BEFORE — the app page, EN** | `Not healthy` | `Not healthy` | `Not healthy` | +| **BEFORE — docker's own healthcheck** | `healthy, healthy` | `healthy, healthy` | `healthy` | +| **BEFORE — the front door, followed** | **200** `Login / Sign In` | **200** `Zipline` | **200** `wger Workout Manager` | +| **AFTER — controller state** | `running` | `running` | `running` | +| **AFTER — the app page, HU / EN** | `Fut` / `Running` | `Fut` / `Running` | `Fut` / `Running` | + +The fix arrived through the **real sync**: `POST /api/sync` answered +*„Sablonok frissítve — frissítve: tandoor, wger, zipline"*, and all three read `running` at the next +poll — **no redeploy, no container restart**. + +**The badge I quote is the HEALTH word, not the version badge.** The version badge reads +`Naprakész` / `Up to date` in both states and would have proved nothing; a first pass captured it +and it was corrected. The scan strips `|", "", h, flags=re.S) + m = re.findall(r']*title="([^"]*)"[^>]*>([^<]*)<', h) + txt = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", h)) + i = txt.find("←") + return {"tags": [{"title": a.strip(), "text": b.strip()} for a, b in m][:3], + "context": txt[i:i + 200] if i >= 0 else None} + + +def observables(box, app): + st = box.stack(app) + ac = st.get("app_config") or {} + comp = box.guest(f"grep -h 'image:' /opt/docker/stacks/{app}/docker-compose.yml 2>/dev/null") + insp = box.guest("docker ps -a --format '{{.Names}}|{{.Image}}|{{.State}}|{{.Status}}' " + f"| grep '^{app}'") + return {"state": st.get("state"), "pinned_images": ac.get("pinned_images"), + "installed_images": ac.get("installed_images"), + "compose": [l.strip() for l in comp.strip().split("\n") if l.strip()], + "inspect": [l for l in insp.strip().split("\n") if l.strip()]} + + +def press(box, app, cap_s=1800): + code, d = box.ctl("POST", f"/api/stacks/{app}/update") + t0 = time.time() + phases, last = [], None + if code != "202": + return {"accepted": False, "code": code, "answer": str(d)[:300], "phases": []} + while time.time() - t0 < cap_s: + st = box.stack(app) + u = st.get("update") or st.get("update_status") or {} + ph = u.get("phase") or st.get("update_phase") + if ph and ph != last: + phases.append({"t": round(time.time() - t0, 1), "phase": ph, + "label": u.get("label") or u.get("message"), + "error": u.get("error"), "hold": u.get("hold_message")}) + last = ph + print(" +%7.1fs phase=%s %s" % (phases[-1]["t"], ph, phases[-1]["label"]), + flush=True) + if ph in ("done", "failed"): + break + time.sleep(1.0) + return {"accepted": True, "code": code, "answer": str(d)[:200], "phases": phases} + + +def main(): + only = sys.argv[1:] or list(BOXES) + out = {} + for name in only: + print(f"\n######## {name}") + b = Box(name) + print(f" base={b.base} Host={b.host}") + b.login() + code, d = b.ctl("GET", "/api/stacks") + installed = [s["name"] for s in (d.get("data") or []) if s.get("deployed")] + print(" deployed apps:", ", ".join(sorted(installed))) + todo = [a for a in APPS if a in installed] + missing = [a for a in APPS if a not in installed] + print(" will update:", todo, "| NOT installed here:", missing) + code, d = b.ctl("POST", "/api/sync") + print(" sync ->", code, str(d)[:180]) + time.sleep(3) + b.ctl("POST", "/api/stacks/rescan") + time.sleep(3) + rec = {"box": name, "base": b.base, "deployed": sorted(installed), + "not_installed": missing, "apps": {}} + for app in todo: + print(f"\n ==== {app}") + before = observables(b, app) + bhu, ben = badge(b, app), badge(b, app, "?lang=en") + print(" badge HU:", bhu["tags"]) + print(" badge EN:", ben["tags"]) + print(" before:", before["pinned_images"]) + r = press(b, app) + after = observables(b, app) + print(" after :", after["pinned_images"], "| installed:", + {k: (v or {}).get("ref") for k, v in (after["installed_images"] or {}).items()}) + phu, pen = badge(b, app), badge(b, app, "?lang=en") + rec["apps"][app] = {"badge_before_hu": bhu, "badge_before_en": ben, + "before": before, "update": r, "after": after, + "page_after_hu": phu, "page_after_en": pen} + out[name] = rec + json.dump(out, open(os.path.join(HERE, "demo-updates.json"), "w"), + ensure_ascii=False, indent=2) + print("\nwritten demo-updates.json") + + +if __name__ == "__main__": + main() diff --git a/documentation/audits/probe-fix-2026-09-22/front-door-before.json b/documentation/audits/probe-fix-2026-09-22/front-door-before.json new file mode 100644 index 00000000..6cc2610f --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/front-door-before.json @@ -0,0 +1,17 @@ +{ + "tandoor": { + "code": "200", + "bytes": 4981, + "first200chars": "Login Sign In Username * Password * Forgot your password? Remember Me Sign In" + }, + "zipline": { + "code": "200", + "bytes": 6405, + "first200chars": "Zipline" + }, + "wger": { + "code": "200", + "bytes": 52291, + "first200chars": ". --> wger Workout Manager - Features Contact Exercises Mobile app Donate Develop Community Login Register Track your way to your ideal body Join wger, the community-driven free and open-source fitnes" + } +} \ No newline at end of file diff --git a/documentation/audits/probe-fix-2026-09-22/mk_part2.py b/documentation/audits/probe-fix-2026-09-22/mk_part2.py new file mode 100644 index 00000000..3258909a --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/mk_part2.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +"""Generate Part 2 of the report FROM the evidence file, so the table cannot drift from it.""" +import json, io, os +HERE = os.path.dirname(os.path.abspath(__file__)) +d = json.load(open(os.path.join(HERE, "demo-updates.json"), encoding="utf-8")) +mv = json.load(open(os.path.join(HERE, "moves.json"), encoding="utf-8")) + +out = [] +out.append("## Part 2 — the promotion train\n") +out.append("### The moves\n") +out.append("Fifteen commits, one per app, `catalog_since: \"%s\"` on each. Every gate run before " + "each commit; nothing forced, nothing dropped. `check-image-resolvable.py` confirms all " + "18 unique pins still exist upstream.\n" % mv["catalog_since"]) +out.append("| app | move | commit |") +out.append("|---|---|---|") +for app, human, h in mv["moved"]: + out.append("| `%s` | %s | `%s` |" % (app, human, h)) +out.append("| `nextcloud` | **ENGINE** `mariadb:11.6` -> `mariadb:12.3` | `39374d5` |") +out.append("") +out.append("### The guarded Update, pressed on both demo boxes\n") + +for box, rec in d.items(): + out.append("#### `%s` — %s\n" % (box, rec["base"])) + if rec["not_installed"]: + out.append("**Not installed on this box, so not pressed:** %s.\n" + % ", ".join("`%s`" % a for a in rec["not_installed"])) + out.append("| app | badge before (HU / EN) | phases | pin after | installed after | front-door state |") + out.append("|---|---|---|---|---|---|") + for app, a in rec["apps"].items(): + bh = "; ".join(t["text"] for t in a["badge_before_hu"]["tags"]) or "—" + be = "; ".join(t["text"] for t in a["badge_before_en"]["tags"]) or "—" + ph = " → ".join("`%s` +%.1fs" % (p["phase"], p["t"]) for p in a["update"]["phases"]) or "—" + pin = (a["after"]["pinned_images"] or {}).get(app, "—") + ins = ((a["after"]["insta" "lled_images"] or {}).get(app) or {}).get("ref", "—") + st = a["after"]["state"] + out.append("| `%s` | `%s` / `%s` | %s | `%s` | `%s` | `%s` |" % (app, bh, be, ph, pin, ins, st)) + out.append("") + out.append("Page after the update, as the household reads it:\n") + for app, a in rec["apps"].items(): + hu = (a["page_after_hu"]["context"] or "").strip() + en = (a["page_after_en"]["context"] or "").strip() + out.append("- **`%s`** — HU: *%s*" % (app, hu[:170])) + out.append(" EN: *%s*" % en[:170]) + out.append("") + +io.open(os.path.join(HERE, "PART2.md"), "w", encoding="utf-8").write("\n".join(out) + "\n") +print("\n".join(out)[:1500]) diff --git a/documentation/audits/probe-fix-2026-09-22/moves.json b/documentation/audits/probe-fix-2026-09-22/moves.json new file mode 100644 index 00000000..447eed39 --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/moves.json @@ -0,0 +1,76 @@ +{ + "moved": [ + [ + "actualbudget", + "26.7.0 -> 26.9.0", + "2060032" + ], + [ + "audiobookshelf", + "2.35.1 -> 2.36.1", + "6525b8e" + ], + [ + "bookstack", + "26.05.2 -> 26.05.5", + "ac4828f" + ], + [ + "docmost", + "0.95.0 -> 0.96.0", + "6d8cd87" + ], + [ + "grafana", + "13.1.0 -> 13.2.2", + "068f445" + ], + [ + "home-assistant", + "2026.7.2 -> 2026.9.3", + "4405f12" + ], + [ + "mealie", + "v3.20.1 -> v3.27.0", + "008348b" + ], + [ + "n8n", + "2.31.3 -> 2.40.5", + "4132e35" + ], + [ + "navidrome", + "0.63.2 -> 0.64.0", + "7708a04" + ], + [ + "papra", + "26.6.1 -> 26.6.2", + "e96887e" + ], + [ + "privatebin", + "2.0.5 -> 2.0.6", + "f547f16" + ], + [ + "romm", + "5.0.0 -> 5.3.0", + "15f9ebf" + ], + [ + "tandoor", + "2.6.13 -> 2.6.15", + "c3807c7" + ], + [ + "vikunja", + "2.3.0 -> 2.6.0", + "22f598b" + ] + ], + "dropped": [], + "catalog_since": "2026-09-22" +} \ No newline at end of file diff --git a/documentation/audits/probe-fix-2026-09-22/not-judged.json b/documentation/audits/probe-fix-2026-09-22/not-judged.json new file mode 100644 index 00000000..da2af78c --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/not-judged.json @@ -0,0 +1,64 @@ +{ + "total": 53, + "with_verdict_record": [ + "actualbudget", + "adventurelog", + "audiobookshelf", + "bookstack", + "docmost", + "gitea", + "grafana", + "home-assistant", + "mealie", + "n8n", + "navidrome", + "nextcloud", + "opengist", + "papra", + "privatebin", + "romm", + "tandoor", + "vaultwarden", + "vikunja", + "zipline" + ], + "deployed_as_prop_no_edge": [ + "bentopdf", + "glance", + "uptime-kuma", + "wishlist" + ], + "deployed_for_probe_only": [ + "wger" + ], + "never_deployed": [ + "calcom", + "calibre-web", + "claper", + "code-server", + "crafty-controller", + "emby", + "ghost", + "gokapi", + "gramps-web", + "homebox", + "homepage", + "immich", + "jellyfin", + "kimai", + "komga", + "onlyoffice", + "outline", + "paperless-ngx", + "plant-it", + "plex", + "radarr", + "rallly", + "recipe-importer", + "seerr", + "sonarr", + "sparkyfitness", + "termix", + "wanderer" + ] +} \ No newline at end of file diff --git a/documentation/audits/probe-fix-2026-09-22/probe-after.json b/documentation/audits/probe-fix-2026-09-22/probe-after.json new file mode 100644 index 00000000..84154107 --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/probe-after.json @@ -0,0 +1,137 @@ +[ + { + "when": "2026-09-22T08:51:05Z", + "tag": "after", + "app": "tandoor", + "controller_state": "running", + "deployed": true, + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": { + "ref": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "digest": "sha256:f6c58afdea7a721d079ebd6ee5483f2c9da77dd1e709e16d60a82c218e80a451", + "at": "2026-09-22T08:38:25Z" + }, + "tandoor-postgres": { + "ref": "postgres:16-alpine", + "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", + "at": "2026-09-22T08:38:25Z" + } + }, + "front_door_rc": 0, + "front_door_code": "200", + "front_door_bytes": 4981, + "docker_health": "healthy\nhealthy", + "version_badge": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "health_word": { + "hu": "Fut", + "hu_context": "← Alkalmazások Tandoor Recipes Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Receptkezelő és étkezés tervező a családnak ~512M RAM home Pi kompatibilis Áthelyezés másik tárhelyre Ennek az alkalmazásnak az adatai", + "en": "Running", + "en_context": "← Apps Tandoor Recipes Running Up to date Open ↗ Log Export Settings A recipe manager and meal planner for the household ~512M RAM home Runs on Pi Move to another storage You can move this app’s data to another connected" + } + }, + { + "when": "2026-09-22T08:51:08Z", + "tag": "after", + "app": "zipline", + "controller_state": "running", + "deployed": true, + "pinned_images": { + "zipline": "ghcr.io/diced/zipline:4.6.1", + "zipline-postgres": "postgres:16-alpine" + }, + "installed_images": { + "zipline": { + "ref": "ghcr.io/diced/zipline:4.6.1", + "digest": "sha256:b8babbbcf4fef89c0c620e0539a47d908c750c251614309a7662ea4d8a7f501f", + "at": "2026-09-22T08:39:28Z" + }, + "zipline-postgres": { + "ref": "postgres:16-alpine", + "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", + "at": "2026-09-22T08:39:28Z" + } + }, + "front_door_rc": 0, + "front_door_code": "301", + "front_door_bytes": 0, + "docker_health": "healthy\nhealthy", + "version_badge": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "health_word": { + "hu": "Fut", + "hu_context": "← Alkalmazások Zipline Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Screenshot és fájlmegosztó szerver ShareX/Flameshot integrációval ~100M RAM files Csak x86 Áthelyezés másik tárhelyre Ennek az alkalmazásnak a", + "en": "Running", + "en_context": "← Apps Zipline Running Up to date Open ↗ Log Export Settings A screenshot and file sharing server that works with ShareX/Flameshot ~100M RAM files x86 only Move to another storage You can move this app’s data to another " + } + }, + { + "when": "2026-09-22T08:51:10Z", + "tag": "after", + "app": "wger", + "controller_state": "running", + "deployed": true, + "pinned_images": { + "wger": "wger/server:2.6" + }, + "installed_images": { + "wger": { + "ref": "wger/server:2.6", + "digest": "sha256:997ead43aabdcd67d054f933e07d2b23875f01bf43271a267cb7796925ca27c4", + "at": "2026-09-22T08:44:57Z" + } + }, + "front_door_rc": 0, + "front_door_code": "302", + "front_door_bytes": 0, + "docker_health": "healthy", + "version_badge": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "health_word": { + "hu": "Fut", + "hu_context": "← Alkalmazások wger Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Edzésnapló - edzéstervek, haladás követés és testsúly napló ~100M RAM home Pi kompatibilis Áthelyezés másik tárhelyre Ennek az alkalmazásnak az a", + "en": "Running", + "en_context": "← Apps wger Running Up to date Open ↗ Log Export Settings A workout log - training plans, progress and a weight diary ~100M RAM home Runs on Pi Move to another storage You can move this app’s data to another connected st" + } + } +] \ No newline at end of file diff --git a/documentation/audits/probe-fix-2026-09-22/probe-before-observe.json b/documentation/audits/probe-fix-2026-09-22/probe-before-observe.json new file mode 100644 index 00000000..b9f28c3e --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/probe-before-observe.json @@ -0,0 +1,137 @@ +[ + { + "when": "2026-09-22T08:48:49Z", + "tag": "before-observe", + "app": "tandoor", + "controller_state": "unhealthy", + "deployed": true, + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": { + "ref": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "digest": "sha256:f6c58afdea7a721d079ebd6ee5483f2c9da77dd1e709e16d60a82c218e80a451", + "at": "2026-09-22T08:38:25Z" + }, + "tandoor-postgres": { + "ref": "postgres:16-alpine", + "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", + "at": "2026-09-22T08:38:25Z" + } + }, + "front_door_rc": 0, + "front_door_code": "200", + "front_door_bytes": 4981, + "docker_health": "healthy\nhealthy", + "version_badge": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "health_word": { + "hu": "Nem egészséges", + "hu_context": "← Alkalmazások Tandoor Recipes Nem egészséges Naprakész Megnyitás ↗ Napló Exportálás Beállítások Receptkezelő és étkezés tervező a családnak ~512M RAM home Pi kompatibilis Áthelyezés másik tárhelyre Ennek az alkalmazásna", + "en": "Not healthy", + "en_context": "← Apps Tandoor Recipes Not healthy Up to date Open ↗ Log Export Settings A recipe manager and meal planner for the household ~512M RAM home Runs on Pi Move to another storage You can move this app’s data to another conne" + } + }, + { + "when": "2026-09-22T08:48:51Z", + "tag": "before-observe", + "app": "zipline", + "controller_state": "unhealthy", + "deployed": true, + "pinned_images": { + "zipline": "ghcr.io/diced/zipline:4.6.1", + "zipline-postgres": "postgres:16-alpine" + }, + "installed_images": { + "zipline": { + "ref": "ghcr.io/diced/zipline:4.6.1", + "digest": "sha256:b8babbbcf4fef89c0c620e0539a47d908c750c251614309a7662ea4d8a7f501f", + "at": "2026-09-22T08:39:28Z" + }, + "zipline-postgres": { + "ref": "postgres:16-alpine", + "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", + "at": "2026-09-22T08:39:28Z" + } + }, + "front_door_rc": 0, + "front_door_code": "301", + "front_door_bytes": 0, + "docker_health": "healthy\nhealthy", + "version_badge": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "health_word": { + "hu": "Nem egészséges", + "hu_context": "← Alkalmazások Zipline Nem egészséges Naprakész Megnyitás ↗ Napló Exportálás Beállítások Screenshot és fájlmegosztó szerver ShareX/Flameshot integrációval ~100M RAM files Csak x86 Áthelyezés másik tárhelyre Ennek az alka", + "en": "Not healthy", + "en_context": "← Apps Zipline Not healthy Up to date Open ↗ Log Export Settings A screenshot and file sharing server that works with ShareX/Flameshot ~100M RAM files x86 only Move to another storage You can move this app’s data to anot" + } + }, + { + "when": "2026-09-22T08:48:54Z", + "tag": "before-observe", + "app": "wger", + "controller_state": "unhealthy", + "deployed": true, + "pinned_images": { + "wger": "wger/server:2.6" + }, + "installed_images": { + "wger": { + "ref": "wger/server:2.6", + "digest": "sha256:997ead43aabdcd67d054f933e07d2b23875f01bf43271a267cb7796925ca27c4", + "at": "2026-09-22T08:44:57Z" + } + }, + "front_door_rc": 0, + "front_door_code": "302", + "front_door_bytes": 0, + "docker_health": "healthy", + "version_badge": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "health_word": { + "hu": "Nem egészséges", + "hu_context": "← Alkalmazások wger Nem egészséges Naprakész Megnyitás ↗ Napló Exportálás Beállítások Edzésnapló - edzéstervek, haladás követés és testsúly napló ~100M RAM home Pi kompatibilis Áthelyezés másik tárhelyre Ennek az alkalma", + "en": "Not healthy", + "en_context": "← Apps wger Not healthy Up to date Open ↗ Log Export Settings A workout log - training plans, progress and a weight diary ~100M RAM home Runs on Pi Move to another storage You can move this app’s data to another connecte" + } + } +] \ No newline at end of file diff --git a/documentation/audits/probe-fix-2026-09-22/probe-before.json b/documentation/audits/probe-fix-2026-09-22/probe-before.json new file mode 100644 index 00000000..ee3d31e6 --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/probe-before.json @@ -0,0 +1,49 @@ +[ + { + "when": "2026-09-22T08:46:38Z", + "tag": "before", + "app": "tandoor", + "controller_state": "unhealthy", + "deployed": true, + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": { + "ref": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "digest": "sha256:f6c58afdea7a721d079ebd6ee5483f2c9da77dd1e709e16d60a82c218e80a451", + "at": "2026-09-22T08:38:25Z" + }, + "tandoor-postgres": { + "ref": "postgres:16-alpine", + "digest": "sha256:721873c34ceb9f8d8fc265984940dc982404c105f19ad51be9fdc5970a6080ea", + "at": "2026-09-22T08:38:25Z" + } + }, + "front_door_rc": 0, + "front_door_code": "200", + "front_door_bytes": 4981, + "docker_health": "healthy\nhealthy", + "version_badge": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "health_word": { + "hu": "Nem egészséges", + "hu_context": "← Alkalmazások Tandoor Recipes Nem egészséges Naprakész Megnyitás ↗ Napló Exportálás Beállítások Receptkezelő és étkezés tervező a családnak ~512M RAM home Pi kompatibilis Áthelyezés másik tárhelyre Ennek az alkalmazásna", + "en": null, + "en_context": "← Apps Tandoor Recipes Not healthy Up to date Open ↗ Log Export Settings A recipe manager and meal planner for the household ~512M RAM home Runs on Pi Move to another storage You can move this app’s data to another conne" + } + } +] \ No newline at end of file diff --git a/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/app-logs-after.txt b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/app-logs-after.txt new file mode 100644 index 00000000..96755741 --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/app-logs-after.txt @@ -0,0 +1,509 @@ +tandoor-postgres | +tandoor-postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization +tandoor-postgres | +tandoor-postgres | 2026-09-22 10:53:18.859 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +tandoor-postgres | 2026-09-22 10:53:18.859 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 +tandoor-postgres | 2026-09-22 10:53:18.859 CEST [1] LOG: listening on IPv6 address "::", port 5432 +tandoor-postgres | 2026-09-22 10:53:18.867 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +tandoor-postgres | 2026-09-22 10:53:18.880 CEST [29] LOG: database system was shut down at 2026-09-22 10:53:09 CEST +tandoor-postgres | 2026-09-22 10:53:18.895 CEST [1] LOG: database system is ready to accept connections +tandoor | Deleting 'vue3/assets/pt_BR-tNFhL9ci.js.gz' +tandoor | Deleting 'vue3/assets/SupermarketEditor-L_B5nyx2.3796d8a521cf.js' +tandoor | Deleting 'vue3/assets/MealTypeEditor-DAAKjxZB.js' +tandoor | Deleting 'vue3/assets/sl-Nkic6vBb.c6ce1eb24108.js' +tandoor | Deleting 'vue3/assets/nl-DIqabTrY.fcf6d7b2df13.js.gz' +tandoor | Deleting 'vue3/assets/UnitEditor-54HrDOhT.9040eae92234.js.gz' +tandoor | Deleting 'vue3/assets/FoodEditor-BR-tK8Pr.js.gz' +tandoor | Deleting 'vue3/assets/forwardRefs-DVyU53sF.js' +tandoor | Deleting 'vue3/assets/position-PPD1ymuo.js.gz' +tandoor | Deleting 'vue3/assets/ca-C5QQCa9o.ff6e54a70768.js.gz' +tandoor | Deleting 'vue3/assets/KeywordsBar-BwJ0lEDV.js' +tandoor | Deleting 'vue3/assets/number_utils-DFmVVcK0.js' +tandoor | Deleting 'vue3/assets/ja-Cqm3qbrP.js.gz' +tandoor | Deleting 'vue3/assets/bg-_Ardqm8i.c46924a2b9d5.js' +tandoor | Deleting 'vue3/assets/VOverlay-CKBxglwy.b98729efffd0.css.gz' +tandoor | Deleting 'vue3/assets/fr-D1J9AIHe.js' +tandoor | Deleting 'vue3/assets/goto-DubWlX_o.0c25d7a8f5fa.js.gz' +tandoor | Deleting 'vue3/assets/VOverlay-CKBxglwy.css.gz' +tandoor | Deleting 'vue3/assets/lv-BUrQRxBI.8efc7d113546.js' +tandoor | Deleting 'vue3/assets/AccessTokenEditor-BqYrLuQE.4d7347e87897.js.gz' +tandoor | Deleting 'vue3/assets/VSwitch-Djsboh2v.92774fc66b84.css.gz' +tandoor | Deleting 'vue3/assets/PropertyEditor-DK6Qeffb.864b139cbb79.js' +tandoor | Deleting 'vue3/assets/ModelDeletePage-C4id8-tu.1ed7ee64e600.js.gz' +tandoor | Deleting 'vue3/assets/da-DHqdZ4yZ.197be8016aca.js' +tandoor | Deleting 'vue3/assets/ro-CG_JGo0i.js' +tandoor | Deleting 'vue3/assets/ExportDataSettings-BlFk8Tn-.6ccf02326ec4.js.gz' +tandoor | Deleting 'vue3/assets/PropertiesEditor-DCxEsAZO.16f243b74d54.js' +tandoor | Deleting 'vue3/assets/RecipeImportPage-DBKJ6DXr.a23aec8a9383.js' +tandoor | Deleting 'vue3/assets/el-DNs6HoDL.08d7c47f5d2b.js' +tandoor | Deleting 'vue3/assets/RecipeViewPage-BY3ErEFj.457b3d77074e.css' +tandoor | Deleting 'vue3/assets/DeleteConfirmDialog-C1xQIJsS.b663eac25ee4.js.gz' +tandoor | Deleting 'vue3/assets/VTimePicker-DNtQ2GaY.658bf461867a.css' +tandoor | Deleting 'vue3/assets/logo_color-CXE3OqOR.d7c2e31a63b7.svg' +tandoor | Deleting 'vue3/assets/fa-v4compatibility-CCth-dXg.4ed293ceaca9.ttf.gz' +tandoor | Deleting 'vue3/assets/IngredientEditorPage-DF3k1tNH.aa9eb5c590ab.js' +tandoor | Deleting 'vue3/assets/InventoryLocationEditor-DIxi9S-h.5de878ee0a17.js' +tandoor | Deleting 'vue3/assets/ShoppingListView-Dgh1luaN.css' +tandoor | Deleting 'vue3/assets/StartPage-0ugVot93.js.gz' +tandoor | Deleting 'vue3/assets/DeleteConfirmDialog-C1xQIJsS.js' +tandoor | Deleting 'vue3/assets/MealPlanEditor-B5jLcokm.js.gz' +tandoor | Deleting 'vue3/assets/number_utils-Brwft-8t.d570ceeec9ef.css' +tandoor | Deleting 'vue3/assets/MealPlanEditor-B5jLcokm.7c8b48842426.js' +tandoor | Deleting 'vue3/assets/VDivider-BeM1gRhl.css' +tandoor | Deleting 'vue3/assets/AccountSettings-BHfX12gt.js' +tandoor | Deleting 'vue3/assets/CustomFilterEditor-Dc0piaF0.js' +tandoor | Deleting 'vue3/assets/lt-SI6E7sY1.2560d8a9d387.js.gz' +tandoor | Deleting 'vue3/assets/de-5mnKTxw2.js' +tandoor | Deleting 'vue3/assets/SearchSettings-CBkFhMns.js' +tandoor | Deleting 'vue3/assets/logo_color-CXE3OqOR.svg' +tandoor | Deleting 'vue3/assets/fontello-B1X0PDnA.068ca2b316db.ttf' +tandoor | Deleting 'vue3/assets/PropertyEditor-DK6Qeffb.js' +tandoor | Deleting 'vue3/assets/RecipeEditor-CcFlDNPw.css' +tandoor | Deleting 'vue3/assets/RecipeCard-C2Hjv7vL.6b450d3cfcd0.js.gz' +tandoor | Deleting 'vue3/assets/CustomFilterEditor-Dc0piaF0.fe7085369993.js.gz' +tandoor | Deleting 'vue3/assets/VStepper-lk6OG_fY.23af7e185e38.css' +tandoor | Deleting 'vue3/assets/VList-edQgt9_l.2642c051884b.css.gz' +tandoor | Deleting 'vue3/assets/InventoryBookingPage-orELmDxm.js' +tandoor | Deleting 'vue3/assets/PropertyEditorPage-Ti9vpeer.e4b532c48301.js' +tandoor | Deleting 'vue3/assets/pt-CXzOqbUC.js' +tandoor | Deleting 'vue3/assets/BooksPage-Dh8_wlv_.cb4aba43024a.js' +tandoor | Deleting 'vue3/assets/ModelListPage-xoBv_A6i.js.gz' +tandoor | Deleting 'vue3/assets/number_utils-DFmVVcK0.js.gz' +tandoor | Deleting 'vue3/assets/FdcSearchDialog-BvpKKey4.js.gz' +tandoor | Deleting 'vue3/assets/position-PPD1ymuo.7195d47ccd52.js' +tandoor | Deleting 'vue3/assets/model_utils-a_hMB_Qu.js' +tandoor | Deleting 'vue3/assets/useFileApi-BD_kCiAS.js' +tandoor | Deleting 'vue3/assets/he-DxNGVUQd.79ff03f4063a.js' +tandoor | Deleting 'vue3/assets/zh_Hant-CERwsE1Q.js.gz' +tandoor | Deleting 'vue3/assets/SearchPage-BQ1P-4y8.ba1f4c8767b9.js.gz' +tandoor | Deleting 'vue3/assets/InventoryLocationEditor-DIxi9S-h.js' +tandoor | Deleting 'vue3/assets/fi-BLW_HATq.078772422941.js' +tandoor | Deleting 'vue3/assets/nn-CkCilbi1.ccb3acb89917.js.gz' +tandoor | Deleting 'vue3/assets/UserFileEditor-DKOcBuK-.js' +tandoor | Deleting 'vue3/assets/useFileApi-BD_kCiAS.37a20e72aad4.js.gz' +tandoor | Deleting 'vue3/assets/hu-DlZ4REsp.js' +tandoor | Deleting 'vue3/assets/pt-CXzOqbUC.js.gz' +tandoor | Deleting 'vue3/assets/VTimePicker-BwPSjcxJ.a223b07c705e.js' +tandoor | Deleting 'vue3/assets/pt_BR-tNFhL9ci.js' +tandoor | Deleting 'vue3/assets/CustomFilterEditor-Dc0piaF0.js.gz' +tandoor | Deleting 'vue3/assets/ko-ADlXHFgo.js' +tandoor | Deleting 'vue3/assets/VDataTableServer-BYvhcs5d.js' +tandoor | Deleting 'vue3/assets/VBtn-rFOjfrXe.df1b89399b11.css.gz' +tandoor | Deleting 'vue3/assets/SupermarketCategoryEditor-rlIgT3uE.js' +tandoor | Deleting 'vue3/assets/CosmeticSettings-DTudQs61.0308c579b183.css' +tandoor | Deleting 'vue3/assets/RecipeEditor-CcFlDNPw.c63cac250b70.css.gz' +tandoor | Deleting 'vue3/assets/fontello-BJkOxCgW.8d4a4e6f7431.woff2' +tandoor | Deleting 'vue3/assets/MealPlanEditor-B5jLcokm.js' +tandoor | Deleting 'vue3/assets/VAvatar-DYNiWd3v.js' +tandoor | Deleting 'vue3/assets/brand_logo-DHt4CyHb.svg' +tandoor | Deleting 'vue3/assets/ripple-CeGB3d9x.3ba9fc624e34.css.gz' +tandoor | Deleting 'vue3/assets/OpenDataImportSettings-Ds1SQcSg.js.gz' +tandoor | Deleting 'vue3/assets/dimensions-B5x3TxJs.js.gz' +tandoor | Deleting 'vue3/assets/PropertiesEditor-DCxEsAZO.js.gz' +tandoor | Deleting 'vue3/assets/PropertiesEditor-DCxEsAZO.16f243b74d54.js.gz' +tandoor | Deleting 'vue3/assets/VFileUpload-ByFVW7Fp.931d5a48009e.js.gz' +tandoor | Deleting 'vue3/assets/DatabaseModelCol-rKX1F7Em.js.gz' +tandoor | Deleting 'vue3/assets/ConnectorConfigEditor-D06JOZDX.5e0dd3a66e81.js.gz' +tandoor | Deleting 'vue3/assets/IngredientEditorPage-DF3k1tNH.js.gz' +tandoor | Deleting 'vue3/assets/lt-SI6E7sY1.js' +tandoor | Deleting 'vue3/assets/DatabasePage-CY4BXH8T.js.gz' +tandoor | Deleting 'vue3/assets/bg-_Ardqm8i.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListView-C0_akhap.js' +tandoor | Deleting 'vue3/assets/HierarchyEditor-wADOsT_-.css' +tandoor | Deleting 'vue3/assets/fontello-B1X0PDnA.ttf' +tandoor | Deleting 'vue3/assets/VOverlay-BSdT-t2A.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListView-Dgh1luaN.css.gz' +tandoor | Deleting 'vue3/assets/it-DJ2I6vOs.js.gz' +tandoor | Deleting 'vue3/assets/bg-_Ardqm8i.js' +tandoor | Deleting 'vue3/assets/VSwitch-Cs5DQi32.4a7617f1519d.js' +tandoor | Deleting 'vue3/assets/logo_color-CXE3OqOR.svg.gz' +tandoor | Deleting 'vue3/assets/WelcomePage-zPtXSDjc.4dfacc5c8da0.js.gz' +tandoor | Deleting 'vue3/assets/UnitEditor-54HrDOhT.9040eae92234.js' +tandoor | Deleting 'vue3/assets/hr-CMRkObu_.js' +tandoor | Deleting 'vue3/assets/VTextField-j0TpvUy_.42df5f015009.css.gz' +tandoor | Deleting 'vue3/assets/OpenDataImportSettings-Ds1SQcSg.f895a746d306.js.gz' +tandoor | Deleting 'vue3/assets/ja-Cqm3qbrP.d617fa524f97.js' +tandoor | Deleting 'vue3/assets/VAvatar-DYNiWd3v.js.gz' +tandoor | Deleting 'vue3/assets/InventoryBookingPage-orELmDxm.js.gz' +tandoor | Deleting 'vue3/assets/ApiSettings-CzWgpe8R.82be1325ed18.js' +tandoor | Deleting 'vue3/assets/fontello-CnWxryRb.eot' +tandoor | Deleting 'vue3/assets/VList-DL25Fkt0.js.gz' +tandoor | Deleting 'vue3/assets/InventoryEntryLogDialog-Dv6Omj68.d4f65e984493.js.gz' +tandoor | Deleting 'vue3/assets/VSwitch-Cs5DQi32.4a7617f1519d.js.gz' +tandoor | Deleting 'vue3/assets/VColorPicker-DoVATq5z.26d1fdd28765.css.gz' +tandoor | Deleting 'vue3/assets/HouseholdEditor-D5JQGcsU.js' +tandoor | Deleting 'vue3/assets/VList-edQgt9_l.css.gz' +tandoor | Deleting 'vue3/assets/VTabs-DzUjbg1Y.js.gz' +tandoor | Deleting 'vue3/assets/ConnectorConfigEditor-D06JOZDX.js' +tandoor | Deleting 'vue3/assets/BatchDeleteDialog-C9gKVWYX.js.gz' +tandoor | Deleting 'vue3/assets/IngredientEditorPage-DF3k1tNH.aa9eb5c590ab.js.gz' +tandoor | Deleting 'vue3/assets/nn-CkCilbi1.ccb3acb89917.js' +tandoor | Deleting 'vue3/assets/InventoryEntryLogDialog-Dv6Omj68.d4f65e984493.js' +tandoor | Deleting 'vue3/assets/VDivider-BeM1gRhl.727c640d6c8d.css.gz' +tandoor | Deleting 'vue3/assets/ApiSettings-CzWgpe8R.js' +tandoor | Deleting 'vue3/assets/SpaceEditor-BG9pNfzF.9465c6dc1d2f.css' +tandoor | Deleting 'vue3/assets/ripple-CeGB3d9x.css.gz' +tandoor | Deleting 'vue3/assets/AiProviderEditor-C4TWtCwO.e6d60050b025.css.gz' +tandoor | Deleting 'vue3/assets/SupermarketEditor-L_B5nyx2.3796d8a521cf.js.gz' +tandoor | Deleting 'vue3/assets/VDivider-BeM1gRhl.727c640d6c8d.css' +tandoor | Deleting 'vue3/assets/fa-v4compatibility-CCth-dXg.4ed293ceaca9.ttf' +tandoor | Deleting 'vue3/assets/VListItemAction-CXKY8J-P.js.gz' +tandoor | Deleting 'vue3/assets/VColorPicker-DoVATq5z.css.gz' +tandoor | Deleting 'vue3/assets/ModelListPage-xoBv_A6i.4681a991ae27.js.gz' +tandoor | Deleting 'vue3/assets/uk-BClnjIfW.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListPage-Cg_SrILL.f2cae0d44a21.js' +tandoor | Deleting 'vue3/assets/RecipeViewPage-BZECYYUo.94016f72f4cb.js.gz' +tandoor | Deleting 'vue3/assets/number_utils-Brwft-8t.css.gz' +tandoor | Deleting 'vue3/assets/de-5mnKTxw2.js.gz' +tandoor | Deleting 'vue3/assets/VFileUpload-7CWUohzn.d182bb8e4f03.css.gz' +tandoor | Deleting 'vue3/assets/VListItemAction-CXKY8J-P.ffd6a13b9ab5.js' +tandoor | Deleting 'vue3/assets/VAvatar-_OUjgGcO.css.gz' +tandoor | Deleting 'vue3/assets/position-PPD1ymuo.7195d47ccd52.js.gz' +tandoor | Deleting 'vue3/assets/brand_logo-DHt4CyHb.fa64319ab35b.svg.gz' +tandoor | Deleting 'vue3/assets/VChip-D7GkfSeA.js' +tandoor | Deleting 'vue3/assets/ConnectorConfigEditor-D06JOZDX.5e0dd3a66e81.js' +tandoor | Deleting 'vue3/assets/fileFilter-JaX38xV6.js.gz' +tandoor | Deleting 'vue3/assets/MealTypeEditor-DAAKjxZB.js.gz' +tandoor | Deleting 'vue3/assets/pl-B7dc5eNY.js' +tandoor | Deleting 'vue3/assets/MealPlanSettings-BU5yRyFO.js' +tandoor | Deleting 'vue3/assets/VAutocomplete-BnHJd-1b.css.gz' +tandoor | Deleting 'vue3/assets/framework-CKkm3ZT1.0da2482ae88a.js.gz' +tandoor | Deleting 'vue3/assets/AccessTokenEditor-BqYrLuQE.js' +tandoor | Deleting 'vue3/assets/VColorPicker-Bc1o5piM.js.gz' +tandoor | Deleting 'vue3/assets/UserSpaceEditor-BiMFbwI_.7d9049ae96b5.js.gz' +tandoor | Deleting 'vue3/assets/CookLogEditor-wf-1cNFJ.ef3a04991a05.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingSettings-DBfLsRlj.js' +tandoor | Deleting 'vue3/assets/AddToShoppingDialog-CS_S-d2g.eb5504799c65.js' +tandoor | Deleting 'vue3/assets/fa-solid-900-CTAAxXor.4a6591ab5460.woff2' +tandoor | Deleting 'vue3/assets/MealPlanSettings-BU5yRyFO.2f9dbd605868.js.gz' +tandoor | Deleting 'vue3/assets/AiProviderEditor-BYnKvCw_.js.gz' +tandoor | Deleting 'vue3/assets/uk-BClnjIfW.b429b8ed5096.js' +tandoor | Deleting 'vue3/assets/BookViewPage-CHarlNiQ.bdf998b08053.js.gz' +tandoor | Deleting 'vue3/assets/HouseholdPage-BsKZpym2.js' +tandoor | Deleting 'vue3/assets/fi-BLW_HATq.078772422941.js.gz' +tandoor | Deleting 'vue3/assets/fontello-CnWxryRb.e73a0647198c.eot.gz' +tandoor | Deleting 'vue3/assets/InviteLinkEditor-BZ7NMcrL.js' +tandoor | Deleting 'vue3/assets/VChip-D7GkfSeA.4de7c0a8c2e3.js.gz' +tandoor | Deleting 'vue3/assets/VRating-BAibLkoF.css' +tandoor | Deleting 'vue3/assets/RecipeBookEditor-Bha-wuB-.js' +tandoor | Deleting 'vue3/assets/VTextField-WpOvMnND.js' +tandoor | Deleting 'vue3/assets/el-DNs6HoDL.08d7c47f5d2b.js.gz' +tandoor | Deleting 'vue3/assets/AutomationEditor-C6fYAtLL.js.gz' +tandoor | Deleting 'vue3/assets/fontello-CnWxryRb.e73a0647198c.eot' +tandoor | Deleting 'vue3/assets/VDivider-rMXvH15P.085aaed48fa0.js' +tandoor | Deleting 'vue3/assets/ShoppingListView-Dgh1luaN.73de85b4e4df.css.gz' +tandoor | Deleting 'vue3/assets/UserFileEditor-DKOcBuK-.js.gz' +tandoor | Deleting 'vue3/assets/VAutocomplete-CFEp5VcY.js' +tandoor | Deleting 'vue3/assets/SpaceEditor-BG9pNfzF.css' +tandoor | Deleting 'vue3/assets/PropertyEditorPage-Ti9vpeer.js' +tandoor | Deleting 'vue3/assets/ShoppingListPage-Cg_SrILL.js' +tandoor | Deleting 'vue3/assets/integration_utils-D8RSOVmp.js.gz' +tandoor | Deleting 'vue3/assets/nb_NO-CdS1Hktt.294979d98dcb.js' +tandoor | Deleting 'vue3/assets/SearchSettings-CBkFhMns.js.gz' +tandoor | Deleting 'vue3/assets/VChip-Hpg9IuyM.8dc9d3e18964.css.gz' +tandoor | Deleting 'vue3/assets/VAutocomplete-BnHJd-1b.css' +tandoor | Deleting 'vue3/assets/ja-Cqm3qbrP.js' +tandoor | Deleting 'vue3/assets/VOverlay-BSdT-t2A.js' +tandoor | Deleting 'vue3/assets/VList-DL25Fkt0.5373fcf8fbc3.js' +tandoor | Deleting 'vue3/assets/SearchPage-BQ1P-4y8.js' +tandoor | Deleting 'vue3/assets/StartPage-0ugVot93.1c72303e8853.js.gz' +tandoor | Deleting 'vue3/assets/fdc-Byjxh21q.6a8962cb2f19.js.gz' +tandoor | Deleting 'vue3/assets/UserSpaceEditor-BiMFbwI_.7d9049ae96b5.js' +tandoor | Deleting 'vue3/assets/PropertyEditorPage-Ti9vpeer.e4b532c48301.js.gz' +tandoor | Deleting 'vue3/assets/SpaceEditor-CIrx7Wew.44bd56e30e62.js.gz' +tandoor | Deleting 'vue3/assets/CosmeticSettings-DTudQs61.css.gz' +tandoor | Deleting 'vue3/assets/transitions-BVkpOt9D.0916b5adcb73.js.gz' +tandoor | Deleting 'vue3/assets/ModelEditPage-B_U4Berv.js' +tandoor | Deleting 'vue3/assets/dimensions-B5x3TxJs.5c0bb4b51390.js' +tandoor | Deleting 'vue3/assets/AccessTokenEditor-BqYrLuQE.4d7347e87897.js' +tandoor | Deleting 'vue3/assets/VColorPicker-DoVATq5z.26d1fdd28765.css' +tandoor | Deleting 'vue3/assets/AddToShoppingDialog-BXt5jKLm.css' +tandoor | Deleting 'vue3/assets/ru-BXmxlY4u.js.gz' +tandoor | Deleting 'vue3/assets/hu-DlZ4REsp.c1bc941c31dc.js.gz' +tandoor | Deleting 'vue3/assets/VList-edQgt9_l.css' +tandoor | Deleting 'vue3/assets/UnitConversionEditor-BUHvM5Eh.js.gz' +tandoor | Deleting 'vue3/assets/VTextarea-zkkSnzux.js.gz' +tandoor | Deleting 'vue3/assets/ModelEditPage-B_U4Berv.7dc258e43ccf.js.gz' +tandoor | Deleting 'vue3/assets/RecipeCard-C2Hjv7vL.6b450d3cfcd0.js' +tandoor | Deleting 'vue3/assets/MealPlanEditor-DqGGZRzo.css' +tandoor | Deleting 'vue3/assets/VSwitch-Cs5DQi32.js.gz' +tandoor | Deleting 'vue3/assets/TestPage-CPX2qaEd.0b75aa57bbeb.js.gz' +tandoor | Deleting 'vue3/assets/dimensions-B5x3TxJs.5c0bb4b51390.js.gz' +tandoor | Deleting 'vue3/assets/main-nTFEtOSD.js.gz' +tandoor | Deleting 'vue3/assets/UserFileEditor-DKOcBuK-.cc1baa1c02f3.js' +tandoor | Deleting 'vue3/assets/StorageEditor-Up4uWQb6.js' +tandoor | Deleting 'vue3/assets/VBtn-DTdW4rEh.js.gz' +tandoor | Deleting 'vue3/assets/he-DxNGVUQd.js.gz' +tandoor | Deleting 'vue3/assets/VChip-Hpg9IuyM.8dc9d3e18964.css' +tandoor | Deleting 'vue3/assets/VTextarea-dHE66AZf.css' +tandoor | Deleting 'vue3/assets/intersect-siLrmRQw.cc9e340f1351.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListPage-Cg_SrILL.js.gz' +tandoor | Deleting 'vue3/assets/framework-CKkm3ZT1.js' +tandoor | Deleting 'vue3/assets/ca-C5QQCa9o.ff6e54a70768.js' +tandoor | Deleting 'vue3/assets/fa-solid-900-D0aA9rwL.ttf' +tandoor | Deleting 'vue3/assets/VListItemAction-CXKY8J-P.ffd6a13b9ab5.js.gz' +tandoor | Deleting 'vue3/assets/main-nTFEtOSD.4273b900f1bf.js' +tandoor | Deleting 'vue3/assets/transitions-BVkpOt9D.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListEditor-vKmfFTZN.4f5dd4c981a5.js' +tandoor | Deleting 'vue3/assets/ripple-CeGB3d9x.css' +tandoor | Deleting 'vue3/assets/ClosableHelpAlert-BoN8dDrm.js.gz' +tandoor | Deleting 'vue3/assets/position-D7bmUCsu.f212789ebec5.css' +tandoor | Deleting 'vue3/assets/ShoppingListView-C0_akhap.1b6d4259ca8e.js.gz' +tandoor | Deleting 'vue3/assets/hy-HqufUZnw.js' +tandoor | Deleting 'vue3/assets/InventoryBookingPage-orELmDxm.45c2c2f7531a.js' +tandoor | Deleting 'vue3/assets/el-DNs6HoDL.js' +tandoor | Deleting 'vue3/assets/VTabs-DzUjbg1Y.fa06924ae72f.js' +tandoor | Deleting 'vue3/assets/form-CDdumt48.5fcbf914ae94.js' +tandoor | Deleting 'vue3/assets/HouseholdPage-BsKZpym2.a5566a4564d6.js.gz' +tandoor | Deleting 'vue3/assets/forwardRefs-DVyU53sF.64171d66d1df.js.gz' +tandoor | Deleting 'vue3/assets/integration_utils-D8RSOVmp.8006123d3084.js.gz' +tandoor | Deleting 'vue3/assets/SpaceSettings-BPOt7vl3.js.gz' +tandoor | Deleting 'vue3/assets/ModelEditorBase-CZWlxxIQ.cf852babb62d.js' +tandoor | Deleting 'vue3/assets/RecipeBookEditor-Bha-wuB-.js.gz' +tandoor | Deleting 'vue3/assets/ModelEditPage-B5oBrkJw.0c31f8b7052c.css.gz' +tandoor | Deleting 'vue3/assets/VAutocomplete-CFEp5VcY.2a4e88a23ede.js.gz' +tandoor | Deleting 'vue3/assets/FdcSearchDialog-BvpKKey4.js' +tandoor | Deleting 'vue3/assets/transitions-BVkpOt9D.js' +tandoor | Deleting 'vue3/assets/ro-CG_JGo0i.c856bd470eef.js' +tandoor | Deleting 'vue3/assets/RecipeViewPage-BZECYYUo.js.gz' +tandoor | Deleting 'vue3/assets/IngredientsTable-DZblS1P-.js.gz' +tandoor | Deleting 'vue3/assets/RecipeCard-C2Hjv7vL.js.gz' +tandoor | Deleting 'vue3/assets/CosmeticSettings-i2TDHCpS.js' +tandoor | Deleting 'vue3/assets/VBtn-rFOjfrXe.css.gz' +tandoor | Deleting 'vue3/assets/FoodEditor-BR-tK8Pr.24df4c5c0e7e.js.gz' +tandoor | Deleting 'vue3/assets/VDivider-rMXvH15P.js' +tandoor | Deleting 'vue3/assets/cs-DUZpBKTY.js' +tandoor | Deleting 'vue3/assets/TestPage-CPX2qaEd.0b75aa57bbeb.js' +tandoor | Deleting 'vue3/assets/fdc-Byjxh21q.js' +tandoor | Deleting 'vue3/assets/CookLogEditor-wf-1cNFJ.js.gz' +tandoor | Deleting 'vue3/assets/HouseholdEditor-D5JQGcsU.js.gz' +tandoor | Deleting 'vue3/assets/VDivider-rMXvH15P.085aaed48fa0.js.gz' +tandoor | Deleting 'vue3/assets/ripple-Byn3Fc0w.3ad96cecfca7.js' +tandoor | Deleting 'vue3/assets/UserSpaceEditor-BiMFbwI_.js.gz' +tandoor | Deleting 'vue3/assets/fr-D1J9AIHe.a85f6556b6a2.js.gz' +tandoor | Deleting 'vue3/assets/fr-D1J9AIHe.js.gz' +tandoor | Deleting 'vue3/assets/VStepper-lk6OG_fY.css.gz' +tandoor | Deleting 'vue3/assets/ko-ADlXHFgo.js.gz' +tandoor | Deleting 'vue3/assets/WelcomePage-zPtXSDjc.4dfacc5c8da0.js' +tandoor | Deleting 'vue3/assets/VChip-D7GkfSeA.js.gz' +tandoor | Deleting 'vue3/assets/SpaceSetupPage-BSv2ZmPb.js' +tandoor | Deleting 'vue3/assets/zh_Hant-CERwsE1Q.8534ecee0fb0.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListPage-Cg_SrILL.f2cae0d44a21.js.gz' +tandoor | Deleting 'vue3/assets/UserSpaceEditor-BiMFbwI_.js' +tandoor | Deleting 'vue3/assets/VOverlay-BSdT-t2A.1f4f586d7141.js.gz' +tandoor | Deleting 'vue3/assets/RecipeImportPage-DBKJ6DXr.a23aec8a9383.js.gz' +tandoor | Deleting 'vue3/assets/VDataTableServer-BYvhcs5d.d5a7cfdcf126.js' +tandoor | Deleting 'vue3/assets/ar-Ct4DpnJT.js' +tandoor | Deleting 'vue3/assets/VRating-Qn4Hz2ci.119bbc89c24a.js' +tandoor | Deleting 'vue3/assets/number_utils-DFmVVcK0.cfc3e206a4c1.js.gz' +tandoor | Deleting 'vue3/assets/HelpPage-C-xlKPtR.fa54da2fd1cf.js.gz' +tandoor | Deleting 'vue3/assets/HierarchyEditor-wADOsT_-.885aa06e77cd.css.gz' +tandoor | Deleting 'vue3/assets/ripple-Byn3Fc0w.3ad96cecfca7.js.gz' +tandoor | Deleting 'vue3/assets/VChip-Hpg9IuyM.css.gz' +tandoor | Deleting 'vue3/assets/RecipeViewPage-BZECYYUo.94016f72f4cb.js' +tandoor | Deleting 'vue3/assets/PrivateRecipeBadge-BVOXRXwb.dcd01dd00de0.js' +tandoor | Deleting 'vue3/assets/SearchSettings-CBkFhMns.b9bc4a1f032c.js' +tandoor | Deleting 'vue3/assets/fontello-BEgLts9b.a782baa8633b.woff' +tandoor | Deleting 'vue3/assets/nl-DIqabTrY.js.gz' +tandoor | Deleting 'vue3/assets/et-zY9CrDMt.1761c6eb5f48.js' +tandoor | Deleting 'vue3/assets/DeleteConfirmDialog-C1xQIJsS.b663eac25ee4.js' +tandoor | Deleting 'vue3/assets/RecipeEditor-Cu9LLPvL.js.gz' +tandoor | Deleting 'vue3/assets/RecipeBookEditor-Bha-wuB-.e3586751a38a.js' +tandoor | Deleting 'vue3/assets/goto-DubWlX_o.js.gz' +tandoor | Deleting 'vue3/assets/ModelMergeDialog-BCXJZjTO.4753553f721f.js' +tandoor | Deleting 'vue3/assets/AutomationEditor-C6fYAtLL.js' +tandoor | Deleting 'vue3/assets/VTextField-j0TpvUy_.42df5f015009.css' +tandoor | Deleting 'vue3/assets/SupermarketEditor-L_B5nyx2.js' +tandoor | Deleting 'vue3/assets/el-DNs6HoDL.js.gz' +tandoor | Deleting 'vue3/assets/lv-BUrQRxBI.8efc7d113546.js.gz' +tandoor | Deleting 'vue3/assets/MealPlanSettings-BU5yRyFO.2f9dbd605868.js' +tandoor | Deleting 'vue3/assets/NumberScalerDialog-DoOvxgjM.js' +tandoor | Deleting 'vue3/assets/fa-solid-900-D0aA9rwL.ttf.gz' +tandoor | Deleting 'vue3/assets/intersect-siLrmRQw.cc9e340f1351.js' +tandoor | Deleting 'vue3/assets/VTabs-Bz9gKX-q.27a6a10279b7.css' +tandoor | Deleting 'vue3/assets/fa-regular-400-DZaxPHgR.ttf.gz' +tandoor | Deleting 'vue3/assets/fa-v4compatibility-CCth-dXg.ttf.gz' +tandoor | Deleting 'vue3/assets/tr-OeJMQhAK.2c406e95c585.js.gz' +tandoor | Deleting 'vue3/assets/CosmeticSettings-DTudQs61.css' +tandoor | Deleting 'vue3/assets/RecipeViewPage-BY3ErEFj.css.gz' +tandoor | Deleting 'vue3/assets/VSwitch-Djsboh2v.css.gz' +tandoor | Deleting 'vue3/assets/TestPage-CPX2qaEd.js.gz' +tandoor | Deleting 'vue3/assets/model_utils-a_hMB_Qu.js.gz' +tandoor | Deleting 'vue3/assets/ar-Ct4DpnJT.js.gz' +tandoor | Deleting 'vue3/assets/DeleteConfirmDialog-C1xQIJsS.js.gz' +tandoor | Deleting 'vue3/assets/BooksPage-Dh8_wlv_.js.gz' +tandoor | Deleting 'assets/recipe_no_image.c659e2604ad6.svg.gz' +tandoor | Deleting 'assets/logo_black.dc487b18028c.svg.gz' +tandoor | Deleting 'assets/logo_color_shopping.svg.gz' +tandoor | Deleting 'assets/brand_logo.ad2e3aa263da.svg' +tandoor | Deleting 'assets/logo_color_192.c9b9177ff941.png' +tandoor | Deleting 'assets/logo_black.svg' +tandoor | Deleting 'assets/header.svg.gz' +tandoor | Deleting 'assets/brand_logo.svg.gz' +tandoor | Deleting 'assets/logo_color_144.f09537dbf142.png' +tandoor | Deleting 'assets/logo_color_128.png' +tandoor | Deleting 'assets/brand_logo.ad2e3aa263da.svg.gz' +tandoor | Deleting 'assets/logo_color_plan.4ccbb336af9d.svg.gz' +tandoor | Deleting 'assets/spinner.168a09fd2600.svg.gz' +tandoor | Deleting 'assets/logo_color_plan_144.22c6c7678fdb.png' +tandoor | Deleting 'assets/spinner.svg.gz' +tandoor | Deleting 'assets/logo_color_512.png' +tandoor | Deleting 'assets/logo_black.svg.gz' +tandoor | Deleting 'assets/logo_color_svg.d7c2e31a63b7.svg.gz' +tandoor | Deleting 'assets/recipe_no_image.c659e2604ad6.svg' +tandoor | Deleting 'assets/logo_color_plan_144.png' +tandoor | Deleting 'assets/logo_color_32.b7d9707eac91.png' +tandoor | Deleting 'assets/logo_color_512.0a28df8511e3.png' +tandoor | Deleting 'assets/brand_logo_white.svg.gz' +tandoor | Deleting 'assets/logo_black.dc487b18028c.svg' +tandoor | Deleting 'assets/logo_color_plan.svg.gz' +tandoor | Deleting 'assets/logo_color_shopping.fd1e81a4dcab.svg.gz' +tandoor | Deleting 'assets/header.48e8ad9e7540.svg' +tandoor | Deleting 'assets/logo_color_plan_512.png' +tandoor | Deleting 'assets/logo_color_shopping_512.6b2e69402327.png' +tandoor | Deleting 'assets/recipe_no_image.svg' +tandoor | Deleting 'assets/logo_color_128.a90db52449d9.png' +tandoor | Deleting 'assets/logo_color_shopping_512.png' +tandoor | Deleting 'assets/brand_logo.svg' +tandoor | Deleting 'assets/logo_color_plan.svg' +tandoor | Deleting 'assets/logo_color_32.png' +tandoor | Deleting 'assets/logo_color_144.png' +tandoor | Deleting 'assets/logo_color_shopping.fd1e81a4dcab.svg' +tandoor | Deleting 'assets/header.48e8ad9e7540.svg.gz' +tandoor | Deleting 'assets/brand_logo_white.d43daa47dca3.svg.gz' +tandoor | Deleting 'assets/logo_color_180.0fcae5e85a72.png' +tandoor | Deleting 'assets/logo_color_180.png' +tandoor | Deleting 'assets/logo_color_shopping_144.png' +tandoor | Deleting 'assets/logo_color_svg.svg' +tandoor | Deleting 'assets/brand_logo_white.d43daa47dca3.svg' +tandoor | Deleting 'assets/logo_color_plan.4ccbb336af9d.svg' +tandoor | Deleting 'assets/logo_color_plan_512.028b9bcca1eb.png' +tandoor | Deleting 'assets/logo_color_shopping.svg' +tandoor | Deleting 'assets/recipe_no_image.svg.gz' +tandoor | Deleting 'assets/header.svg' +tandoor | Deleting 'assets/logo_color_svg.svg.gz' +tandoor | Deleting 'assets/spinner.168a09fd2600.svg' +tandoor | Deleting 'assets/brand_logo.6ebe02bf6707.png' +tandoor | Deleting 'assets/spinner.svg' +tandoor | Deleting 'assets/logo_color_svg.d7c2e31a63b7.svg' +tandoor | Deleting 'assets/logo_color_shopping_144.6c05e01d07b2.png' +tandoor | Deleting 'assets/logo_color_192.png' +tandoor | Deleting 'assets/brand_logo.png' +tandoor | Deleting 'assets/brand_logo_white.svg' +tandoor | Deleting 'mfa/js/webauthn-json.6a3183313556.js' +tandoor | Deleting 'mfa/js/webauthn-json.6a3183313556.js.gz' +tandoor | Deleting 'mfa/js/webauthn.88878ff60271.js' +tandoor | Deleting 'mfa/js/webauthn.88878ff60271.js.gz' +tandoor | Deleting 'mfa/js/webauthn.js.gz' +tandoor | Deleting 'mfa/js/webauthn-json.js.gz' +tandoor | Deleting 'mfa/js/webauthn-json.js' +tandoor | Deleting 'mfa/js/webauthn.js' +tandoor | Deleting 'themes/tandoor.min.css.gz' +tandoor | Deleting 'themes/tandoor.min.css' +tandoor | Deleting 'themes/tandoor.min.e62f1984b5e4.css' +tandoor | Deleting 'themes/tandoor.min.e62f1984b5e4.css.gz' +tandoor | Deleting 'themes/tandoor_dark.min.3bdb00cbb58c.css.gz' +tandoor | Deleting 'themes/tandoor_dark.min.css' +tandoor | Deleting 'themes/tandoor_dark.min.css.gz' +tandoor | Deleting 'themes/tandoor_dark.min.3bdb00cbb58c.css' +tandoor | Deleting 'custom/css/markdown_blockquote.22de6a06ad0b.css' +tandoor | Deleting 'custom/css/markdown_blockquote.css' +tandoor | Deleting 'custom/css/markdown_blockquote.css.gz' +tandoor | Deleting 'custom/css/markdown_blockquote.22de6a06ad0b.css.gz' +tandoor | Deleting 'webfonts/fa-solid-900.8e4a6dcc692b.eot.gz' +tandoor | Deleting 'webfonts/fa-solid-900.c2801fb415f0.svg' +tandoor | Deleting 'webfonts/fa-brands-400.c5e0f14f88a8.woff' +tandoor | Deleting 'webfonts/fa-regular-400.c4f508e7c4f0.woff' +tandoor | Deleting 'webfonts/poppins_latin_ext_500.d214b888d895.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.ttf' +tandoor | Deleting 'webfonts/fa-solid-900.eot.gz' +tandoor | Deleting 'webfonts/poppins_devanagari_500.fd9b8290076b.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.a9c4bb7348f4.svg' +tandoor | Deleting 'webfonts/poppins_latin_400.9ed361bba848.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.8e4a6dcc692b.eot' +tandoor | Deleting 'webfonts/fa-brands-400.cccc9d29470e.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.0bff33a5fd7e.ttf' +tandoor | Deleting 'webfonts/fa-regular-400.woff' +tandoor | Deleting 'webfonts/fa-regular-400.f5f2566b93e8.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.c1a866ec0e04.eot.gz' +tandoor | Deleting 'webfonts/poppins_devanagari_700.b9f2c13fc361.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.eot' +tandoor | Deleting 'webfonts/fa-brands-400.06147b6cd88c.ttf.gz' +tandoor | Deleting 'webfonts/poppins_devanagari_400.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.svg' +tandoor | Deleting 'webfonts/fa-brands-400.svg.gz' +tandoor | Deleting 'webfonts/fa-regular-400.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.eot' +tandoor | Deleting 'webfonts/poppins_latin_700.woff2' +tandoor | Deleting 'webfonts/poppins_devanagari_700.woff2' +tandoor | Deleting 'webfonts/poppins_devanagari_400.d5e78c53cb07.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.c2801fb415f0.svg.gz' +tandoor | Deleting 'webfonts/fa-solid-900.svg' +tandoor | Deleting 'webfonts/poppins_latin_500.woff2' +tandoor | Deleting 'webfonts/poppins_latin_ext_400.4d1490f32451.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.eot.gz' +tandoor | Deleting 'webfonts/poppins_latin_ext_700.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.7b9568e6389b.svg.gz' +tandoor | Deleting 'webfonts/poppins_latin_400.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.c1a866ec0e04.eot' +tandoor | Deleting 'webfonts/fa-brands-400.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.svg.gz' +tandoor | Deleting 'webfonts/fa-brands-400.a9c4bb7348f4.svg.gz' +tandoor | Deleting 'webfonts/fa-brands-400.5063b105c764.eot.gz' +tandoor | Deleting 'webfonts/fa-regular-400.7b9568e6389b.svg' +tandoor | Deleting 'webfonts/poppins_latin_ext_500.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.woff' +tandoor | Deleting 'webfonts/fa-brands-400.ttf.gz' +tandoor | Deleting 'webfonts/fa-solid-900.44d537ab79f9.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.svg.gz' +tandoor | Deleting 'webfonts/fa-solid-900.eot' +tandoor | Deleting 'webfonts/fa-brands-400.svg' +tandoor | Deleting 'webfonts/poppins_latin_500.84780596e268.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.06147b6cd88c.ttf' +tandoor | Deleting 'webfonts/fa-solid-900.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.333bae208dc3.woff' +tandoor | Deleting 'webfonts/poppins_latin_ext_400.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.65b286af947c.ttf.gz' +tandoor | Deleting 'webfonts/fa-solid-900.ttf.gz' +tandoor | Deleting 'webfonts/fa-brands-400.5063b105c764.eot' +tandoor | Deleting 'webfonts/fa-regular-400.ttf.gz' +tandoor | Deleting 'webfonts/poppins_latin_ext_700.62d7a4c78fc2.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.woff' +tandoor | Deleting 'webfonts/poppins_devanagari_500.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.0bff33a5fd7e.ttf.gz' +tandoor | Deleting 'webfonts/fa-regular-400.eot.gz' +tandoor | Deleting 'webfonts/fa-regular-400.ttf' +tandoor | Deleting 'webfonts/fa-regular-400.65b286af947c.ttf' +tandoor | Deleting 'webfonts/poppins_latin_700.f4f17fd53c7d.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.ttf' +tandoor | 2026/09/22 10:54:37 [crit] 16#16: *9 connect() to unix:/tmp/tandoor.sock failed (2: No such file or directory) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /accounts/login/ HTTP/1.1", upstream: "http://unix:/tmp/tandoor.sock:/accounts/login/", host: "127.0.0.1:80" +tandoor | 127.0.0.1 - - [22/Sep/2026:10:54:37 +0200] "GET /accounts/login/ HTTP/1.1" 502 150 "-" "Wget" "-" +tandoor | +tandoor | 894 static files copied to '/opt/recipes/staticfiles', 2464 post-processed. +tandoor | Done +tandoor | Starting gunicorn +tandoor | [2026-09-22 10:54:37 +0200] [7] [INFO] Starting gunicorn 26.2.0 +tandoor | [2026-09-22 10:54:37 +0200] [7] [INFO] Listening at: unix:/tmp/tandoor.sock (7) +tandoor | [2026-09-22 10:54:37 +0200] [7] [INFO] Using worker: gthread +tandoor | [2026-09-22 10:54:37 +0200] [66] [INFO] Booting worker with pid: 66 +tandoor | [2026-09-22 10:54:37 +0200] [67] [INFO] Booting worker with pid: 67 +tandoor | [2026-09-22 10:54:38 +0200] [68] [INFO] Booting worker with pid: 68 +tandoor | [2026-09-22 10:54:38 +0200] [7] [INFO] Control socket listening at /root/.gunicorn/gunicorn.ctl +tandoor | Running django-vite in production mode (no HMR) +tandoor | Running django-vite in production mode (no HMR) +tandoor | Running django-vite in production mode (no HMR) +tandoor | 127.0.0.1 - - [22/Sep/2026:10:54:45 +0200] "GET /accounts/login/ HTTP/1.1" 200 4981 "-" "Wget" "-" +tandoor | - - [22/Sep/2026:10:54:45 +0200] "GET /accounts/login/ HTTP/1.0" 200 4981 "-" "Wget" +tandoor | ThreadPoolExecutor-0_0 ERROR 2026-09-22 10:54:47,243 django.security.DisallowedHost Invalid HTTP_HOST header: 'tandoor:80'. You may need to add 'tandoor' to ALLOWED_HOSTS. +tandoor | Traceback (most recent call last): +tandoor | File "/opt/recipes/venv/lib/python3.13/site-packages/django/core/handlers/exception.py", line 55, in inner +tandoor | response = get_response(request) +tandoor | File "/opt/recipes/venv/lib/python3.13/site-packages/django/utils/deprecation.py", line 119, in __call__ +tandoor | response = self.process_request(request) +tandoor | File "/opt/recipes/venv/lib/python3.13/site-packages/django/middleware/common.py", line 48, in process_request +tandoor | host = request.get_host() +tandoor | File "/opt/recipes/venv/lib/python3.13/site-packages/django/http/request.py", line 203, in get_host +tandoor | raise DisallowedHost(msg) +tandoor | django.core.exceptions.DisallowedHost: Invalid HTTP_HOST header: 'tandoor:80'. You may need to add 'tandoor' to ALLOWED_HOSTS. +tandoor | - - [22/Sep/2026:10:54:47 +0200] "GET /accounts/login/ HTTP/1.0" 400 143 "-" "Go-http-client/1.1" +tandoor | 172.18.0.2 - - [22/Sep/2026:10:54:47 +0200] "GET /accounts/login/ HTTP/1.1" 400 154 "-" "Go-http-client/1.1" "-" diff --git a/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/badges.json b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/badges.json new file mode 100644 index 00000000..52f6f405 --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/badges.json @@ -0,0 +1,21 @@ +{ + "before": { + "hu": [], + "en": [] + }, + "after": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "drill_commit": "654a28192b38" +} \ No newline at end of file diff --git a/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/log.txt b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/log.txt new file mode 100644 index 00000000..ef9af077 --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/log.txt @@ -0,0 +1,25 @@ +10:52:42 ==== tandoor: ghcr.io/tandoorrecipes/recipes:2.6.13 -> ghcr.io/tandoorrecipes/recipes:2.6.15 (sub=recipes, class=db) +10:52:42 [1] tandoor already deployed — reusing +10:52:45 [2] seeding through the app's own front door +10:52:55 tandoor: createsuperuser :: Running django-vite in production mode (no HMR) Superuser created successfully. +10:52:59 tandoor: seeded superuser drill1d392f +10:52:59 [3] control C1 — reading the seed back BEFORE the update +10:53:05 tandoor: readback of the seeded account found=True +10:53:05 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +10:54:00 [4] backup idle; last=None +10:54:01 [5] drill commit 654a28192b38: tandoor ghcr.io/tandoorrecipes/recipes:2.6.13 -> ghcr.io/tandoorrecipes/recipes:2.6.15 (push rc=0) +10:54:06 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +10:54:06 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +10:54:06 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +10:54:06 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +10:54:07 + 1.0s phase=pulling label=Új verzió letöltése… err=None hold=None +10:54:08 + 2.1s phase=starting label=Indítás az új verzióval… err=None hold=None +10:54:11 + 5.2s phase=verifying label=Működés ellenőrzése… err=None hold=None +10:54:47 + 41.1s phase=done label=Frissítve err=None hold=None +10:54:47 [7] reading the seed back AFTER the update +10:54:54 tandoor: readback of the seeded account found=True +10:54:56 [8] pinned = {'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:16-alpine'} +10:54:56 [8] installed = {'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:16-alpine'} +10:54:56 [8] compose = ['image: ghcr.io/tandoorrecipes/recipes:2.6.15', 'image: postgres:16-alpine'] +10:54:56 [8] inspect = ['tandoor ghcr.io/tandoorrecipes/recipes:2.6.15 running=true restarts=0', 'tandoor-postgres postgres:16-alpine running=true restarts=0'] +10:54:56 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/tandoor/verdict.json diff --git a/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/observables-before.json b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/observables-before.json new file mode 100644 index 00000000..80ee28be --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/observables-before.json @@ -0,0 +1,19 @@ +{ + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: ghcr.io/tandoorrecipes/recipes:2.6.13", + "image: postgres:16-alpine" + ], + "docker_inspect": [ + "tandoor ghcr.io/tandoorrecipes/recipes:2.6.13 running=true restarts=0", + "tandoor-postgres postgres:16-alpine running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/observables.json b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/observables.json new file mode 100644 index 00000000..5d9632cf --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/observables.json @@ -0,0 +1,43 @@ +{ + "before": { + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "catalog_images": null, + "live_compose_image_lines": [ + "image: ghcr.io/tandoorrecipes/recipes:2.6.13", + "image: postgres:16-alpine" + ], + "docker_inspect": [ + "tandoor ghcr.io/tandoorrecipes/recipes:2.6.13 running=true restarts=0", + "tandoor-postgres postgres:16-alpine running=true restarts=0" + ] + }, + "after": { + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "catalog_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/tandoorrecipes/recipes:2.6.15", + "image: postgres:16-alpine" + ], + "docker_inspect": [ + "tandoor ghcr.io/tandoorrecipes/recipes:2.6.15 running=true restarts=0", + "tandoor-postgres postgres:16-alpine running=true restarts=0" + ] + } +} \ No newline at end of file diff --git a/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/phases.json b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/phases.json new file mode 100644 index 00000000..f4206717 --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/phases.json @@ -0,0 +1,51 @@ +{ + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.0, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 2.1, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 5.2, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 41.1, + "phase": "done", + "label": "Frissítve", + "updating": false, + "error": null, + "hold": null + } + ], + "duration_s": 41.1, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" +} \ No newline at end of file diff --git a/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/verdict.json b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/verdict.json new file mode 100644 index 00000000..ce4c472d --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/tandoor-rerun/verdict.json @@ -0,0 +1,51 @@ +{ + "harness_version": 1, + "app": "tandoor", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "db", + "from": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "to": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "verdict": "proven", + "seed_read_before": true, + "seed_read_after": true, + "healthy_after": true, + "migration_observed": null, + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 41.1, + "measured_at": "2026-09-22T08:52:42.724401+00:00", + "evidence": "apps/tandoor/", + "notes": [], + "badge_catchup_seconds": 4.5, + "observables_after": { + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "catalog_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/tandoorrecipes/recipes:2.6.15", + "image: postgres:16-alpine" + ], + "docker_inspect": [ + "tandoor ghcr.io/tandoorrecipes/recipes:2.6.15 running=true restarts=0", + "tandoor-postgres postgres:16-alpine running=true restarts=0" + ] + }, + "final_phase": "done", + "hold_reason": null, + "update_error": null +} \ No newline at end of file diff --git a/documentation/audits/probe-fix-2026-09-22/teardown-catalog-pointer.txt b/documentation/audits/probe-fix-2026-09-22/teardown-catalog-pointer.txt new file mode 100644 index 00000000..4c1e87ad --- /dev/null +++ b/documentation/audits/probe-fix-2026-09-22/teardown-catalog-pointer.txt @@ -0,0 +1,6 @@ +origin https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (fetch) +origin https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git (push) +6c690a1 gates: refuse a health probe the app does not answer (R-618) +13: image: ghcr.io/tandoorrecipes/recipes:2.6.13 +58: image: postgres:16-alpine + diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/app-logs-after.txt b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/app-logs-after.txt new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/badges.json b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/badges.json new file mode 100644 index 00000000..1a6d711c --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/badges.json @@ -0,0 +1,31 @@ +{ + "before": { + "hu": [ + { + "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "text": "Naprakész" + } + ], + "en": [ + { + "title": "This app is running the newest version available.", + "text": "Up to date" + } + ] + }, + "after": { + "hu": [ + { + "title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.", + "text": "Frissítés elérhető — ma" + } + ], + "en": [ + { + "title": "A newer version of this app is available. Select the Update button to start it.", + "text": "Update available — today" + } + ] + }, + "drill_commit": "c7477d1fb921" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/failwalk-log.txt b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/failwalk-log.txt new file mode 100644 index 00000000..769d43a4 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/failwalk-log.txt @@ -0,0 +1,23 @@ +21:36:12 ==== FAILWALK tandoor: the household's way out of a held update +21:36:15 [1] held state: phase=failed hold='A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.' +21:36:15 update_error = 'A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.' +21:36:15 pinned = {'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:16-alpine'} +21:36:15 installed = {'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.13', 'tandoor-postgres': 'postgres:16-alpine'} +21:36:15 compose = ['image: ghcr.io/tandoorrecipes/recipes:2.6.15', 'image: postgres:16-alpine'] +21:36:15 inspect = [] +21:36:15 [2] hold sentence HU: ['Tandoor Recipes — Felhom.eu Indítópult Vezérlőpult Alkalmazások Tárhely Meghajtók Hálózati tárhely Biztonsági mentés Áttekintés Távoli mentés Alkalmazások Visszaállítás Megosztás Hálózati megosztás Rendszermonitor Debug Beállítások Rendszer Értesítések Biztonság és hozzáférés 0.261.0 Magyar English Kijelentkezés ↗ Telepített alkalmazás nem fut: Tandoor Recipes (stopped) Rendszermonitor → Hub kapcsolat kikapcsolva — a központi monitoring nem aktív Rendszermonitor → ← Alkalmazások Tandoor Recipes Leállítva Frissítés elérhető — ma Megnyitás ↗ Napló Exportálás Beállítások A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval.', 'Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.', 'Mentések Receptkezelő és étkezés tervező a családnak ~512M RAM home Pi kompatibilis Áthelyezés másik tárhelyre Ennek az alkalmazásnak az adatait másik csatlakoztatott tárhelyre helyezheted át.', 'Receptek gyűjtése és rendszerezése egy helyen Receptek importálása weboldalakról egy kattintással Heti étkezés tervezés és bevásárlólista generálás Több felhasználó - a család együtt tervezhet Receptek megosztása linkkel családtagokkal, barátokkal Első lépések Nyisd meg a recipes.DOMAIN címet a böngészőben Hozd létre az admin fiókot Importáld az első receptet egy weboldalról (Bookmarklet) Próbáld ki az étkezés tervezőt Hívd meg a családtagokat Dokumentáció Hivatalos dokumentáció ↗'] +21:36:15 [2] hold sentence EN: ['Tandoor Recipes — Felhom.eu Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug Settings System Notifications Security and access 0.261.0 Magyar English Sign out ↗ An installed app is not running: Tandoor Recipes (stopped) System monitor → The hub connection is off — central monitoring is not running System monitor → ← Apps Tandoor Recipes Stopped Update available — today Open ↗ Log Export Settings A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval.', 'Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.', 'Backups A recipe manager and meal planner for the household ~512M RAM home Runs on Pi Move to another storage You can move this app’s data to another connected storage.', 'Keep all your recipes in one place Import a recipe from a web page with one click Plan the week's meals and get a shopping list out of it Several people - the household can plan together Share a recipe by link with family and friends First steps Open recipes.DOMAIN in your browser Create the admin account Import your first recipe from a web page (Bookmarklet) Try the meal planner Invite the household Documentation Official documentation ↗'] +21:36:15 [3] the last migration line the app printed, verbatim: None +21:36:15 [R] restoring tandoor from snapshot 'helyi' (of 1 offered) +21:36:15 [R] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +21:36:15 + 0.0s restore (True, None, None) +21:36:47 + 32.4s restore (False, None, None) +21:36:47 [R] after restore: state=running hold=None phase=None +21:36:50 [4] restore: HTTP/2 302 in 32.4s; state=running hold=None +21:36:50 pinned after = {'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.13', 'tandoor-postgres': 'postgres:16-alpine'} +21:36:50 inspect after = ['tandoor ghcr.io/tandoorrecipes/recipes:2.6.13 running=true restarts=0', 'tandoor-postgres postgres:16-alpine running=true restarts=0'] +21:36:59 tandoor: createsuperuser :: Running django-vite in production mode (no HMR) Superuser created successfully. +21:37:03 tandoor: seeded superuser drill74a269 +21:37:09 tandoor: readback of the seeded account found=True +21:37:09 [5] the app answers its own front door again and holds data: True +21:37:10 [6] sentence after the restore HU: ['Tandoor Recipes — Felhom.eu Indítópult Vezérlőpult Alkalmazások Tárhely Meghajtók Hálózati tárhely Biztonsági mentés Áttekintés Távoli mentés Alkalmazások Visszaállítás Megosztás Hálózati megosztás Rendszermonitor Debug Beállítások Rendszer Értesítések Biztonság és hozzáférés 0.261.0 Magyar English Kijelentkezés ↗ Hub kapcsolat kikapcsolva — a központi monitoring nem aktív Rendszermonitor → ← Alkalmazások Tandoor Recipes Nem egészséges Frissítés elérhető — ma Megnyitás ↗ Napló Exportálás Beállítások Receptkezelő és étkezés tervező a családnak ~512M RAM home Pi kompatibilis Áthelyezés másik tárhelyre Ennek az alkalmazásnak az adatait másik csatlakoztatott tárhelyre helyezheted át.', 'Receptek gyűjtése és rendszerezése egy helyen Receptek importálása weboldalakról egy kattintással Heti étkezés tervezés és bevásárlólista generálás Több felhasználó - a család együtt tervezhet Receptek megosztása linkkel családtagokkal, barátokkal Első lépések Nyisd meg a recipes.DOMAIN címet a böngészőben Hozd létre az admin fiókot Importáld az első receptet egy weboldalról (Bookmarklet) Próbáld ki az étkezés tervezőt Hívd meg a családtagokat Dokumentáció Hivatalos dokumentáció ↗'] diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/failwalk.json b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/failwalk.json new file mode 100644 index 00000000..b3ad6967 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/failwalk.json @@ -0,0 +1,102 @@ +{ + "leg": "failwalk", + "app": "tandoor", + "held": { + "state": "stopped", + "updating": false, + "update_phase": "failed", + "update_phase_label": "A frissítés nem sikerült", + "update_error": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "hold_reason": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "observables": { + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "catalog_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/tandoorrecipes/recipes:2.6.15", + "image: postgres:16-alpine" + ], + "docker_inspect": [] + } + }, + "hold_sentences": { + "hu": [ + "Tandoor Recipes — Felhom.eu Indítópult Vezérlőpult Alkalmazások Tárhely Meghajtók Hálózati tárhely Biztonsági mentés Áttekintés Távoli mentés Alkalmazások Visszaállítás Megosztás Hálózati megosztás Rendszermonitor Debug Beállítások Rendszer Értesítések Biztonság és hozzáférés 0.261.0 Magyar English Kijelentkezés ↗ Telepített alkalmazás nem fut: Tandoor Recipes (stopped) Rendszermonitor → Hub kapcsolat kikapcsolva — a központi monitoring nem aktív Rendszermonitor → ← Alkalmazások Tandoor Recipes Leállítva Frissítés elérhető — ma Megnyitás ↗ Napló Exportálás Beállítások A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval.", + "Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "Mentések Receptkezelő és étkezés tervező a családnak ~512M RAM home Pi kompatibilis Áthelyezés másik tárhelyre Ennek az alkalmazásnak az adatait másik csatlakoztatott tárhelyre helyezheted át.", + "Receptek gyűjtése és rendszerezése egy helyen Receptek importálása weboldalakról egy kattintással Heti étkezés tervezés és bevásárlólista generálás Több felhasználó - a család együtt tervezhet Receptek megosztása linkkel családtagokkal, barátokkal Első lépések Nyisd meg a recipes.DOMAIN címet a böngészőben Hozd létre az admin fiókot Importáld az első receptet egy weboldalról (Bookmarklet) Próbáld ki az étkezés tervezőt Hívd meg a családtagokat Dokumentáció Hivatalos dokumentáció ↗" + ], + "en": [ + "Tandoor Recipes — Felhom.eu Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug Settings System Notifications Security and access 0.261.0 Magyar English Sign out ↗ An installed app is not running: Tandoor Recipes (stopped) System monitor → The hub connection is off — central monitoring is not running System monitor → ← Apps Tandoor Recipes Stopped Update available — today Open ↗ Log Export Settings A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval.", + "Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "Backups A recipe manager and meal planner for the household ~512M RAM home Runs on Pi Move to another storage You can move this app’s data to another connected storage.", + "Keep all your recipes in one place Import a recipe from a web page with one click Plan the week's meals and get a shopping list out of it Several people - the household can plan together Share a recipe by link with family and friends First steps Open recipes.DOMAIN in your browser Create the admin account Import your first recipe from a web page (Bookmarklet) Try the meal planner Invite the household Documentation Official documentation ↗" + ] + }, + "last_migration_line": null, + "way_out": { + "ok": true, + "snapshot_id": "helyi", + "snapshots": [ + { + "time": "2026-09-21T19:16:48Z", + "short_id": "helyi", + "tier": 1, + "drive_label": "Belső SSD (rendszer)" + } + ], + "http": "HTTP/2 302", + "location": [ + "location: /backups/restore?flash=flash.restore.started" + ], + "seconds": 32.4, + "state_after": "running", + "hold_after": null, + "observables_after": { + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "catalog_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/tandoorrecipes/recipes:2.6.13", + "image: postgres:16-alpine" + ], + "docker_inspect": [ + "tandoor ghcr.io/tandoorrecipes/recipes:2.6.13 running=true restarts=0", + "tandoor-postgres postgres:16-alpine running=true restarts=0" + ] + } + }, + "data_after_restore": { + "route": "re-seeded after the restore", + "ok": true, + "note": "the pre-update seed token did not survive the failed run's process, so this re-seeds AFTER the restore: it proves the app WORKS again on the old version, not that the specific pre-update row survived. Stated rather than glossed." + }, + "sentences_after_restore": { + "hu": [ + "Tandoor Recipes — Felhom.eu Indítópult Vezérlőpult Alkalmazások Tárhely Meghajtók Hálózati tárhely Biztonsági mentés Áttekintés Távoli mentés Alkalmazások Visszaállítás Megosztás Hálózati megosztás Rendszermonitor Debug Beállítások Rendszer Értesítések Biztonság és hozzáférés 0.261.0 Magyar English Kijelentkezés ↗ Hub kapcsolat kikapcsolva — a központi monitoring nem aktív Rendszermonitor → ← Alkalmazások Tandoor Recipes Nem egészséges Frissítés elérhető — ma Megnyitás ↗ Napló Exportálás Beállítások Receptkezelő és étkezés tervező a családnak ~512M RAM home Pi kompatibilis Áthelyezés másik tárhelyre Ennek az alkalmazásnak az adatait másik csatlakoztatott tárhelyre helyezheted át.", + "Receptek gyűjtése és rendszerezése egy helyen Receptek importálása weboldalakról egy kattintással Heti étkezés tervezés és bevásárlólista generálás Több felhasználó - a család együtt tervezhet Receptek megosztása linkkel családtagokkal, barátokkal Első lépések Nyisd meg a recipes.DOMAIN címet a böngészőben Hozd létre az admin fiókot Importáld az első receptet egy weboldalról (Bookmarklet) Próbáld ki az étkezés tervezőt Hívd meg a családtagokat Dokumentáció Hivatalos dokumentáció ↗" + ], + "en": [ + "Tandoor Recipes — Felhom.eu Launcher Dashboard Apps Storage Drives Network storage Backup Overview Remote backup Apps Restore Sharing Network sharing System monitor Debug Settings System Notifications Security and access 0.261.0 Magyar English Sign out ↗ The hub connection is off — central monitoring is not running System monitor → ← Apps Tandoor Recipes Not healthy Update available — today Open ↗ Log Export Settings A recipe manager and meal planner for the household ~512M RAM home Runs on Pi Move to another storage You can move this app’s data to another connected storage.", + "Keep all your recipes in one place Import a recipe from a web page with one click Plan the week's meals and get a shopping list out of it Several people - the household can plan together Share a recipe by link with family and friends First steps Open recipes.DOMAIN in your browser Create the admin account Import your first recipe from a web page (Bookmarklet) Try the meal planner Invite the household Documentation Official documentation ↗" + ] + } +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/held-app-logs.txt b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/held-app-logs.txt new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/held-page-en.html b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/held-page-en.html new file mode 100644 index 00000000..4a508cc9 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/held-page-en.html @@ -0,0 +1,654 @@ + + + + + + + + Tandoor Recipes — Felhom.eu + + + + + + + + +
+ + +
+ + +
+ + + + + + +
+ + +
+ + The hub connection is off — central monitoring is not running + System monitor → +
+ + +
+ + + +
+
+ ← Apps +

Tandoor Recipes

+
+
+ + Not healthy + + + Update available — today + Open ↗ + Log + + Export + Settings + + +
+
+ + + + + + +
+ +
+ +

A recipe manager and meal planner for the household

+ +
+ ~512M RAM + home + + Runs on Pi + +
+ +
+
+ + +
+ + + +
+ + + + +
+

Move to another storage

+

You can move this app’s data to another connected storage. The app stops briefly during the move, and the data is deleted from the old place only after the check and a successful restart.

+
+ + +
+ +
+ + + + +
+ +
+

What is it for?

+
    +
  • Keep all your recipes in one place
  • Import a recipe from a web page with one click
  • Plan the week's meals and get a shopping list out of it
  • Several people - the household can plan together
  • Share a recipe by link with family and friends
    • +
+
+ + + +
+

First steps

+
    +
  1. Open recipes.DOMAIN in your browser
  2. Create the admin account
  3. Import your first recipe from a web page (Bookmarklet)
  4. Try the meal planner
  5. Invite the household
    6. +
+
+ + + + + + + + + +
+

Documentation

+

Official documentation ↗

+
+ +
+ + + + + +
+ + + + diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/held-page-hu.html b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/held-page-hu.html new file mode 100644 index 00000000..0008d9c7 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/held-page-hu.html @@ -0,0 +1,654 @@ + + + + + + + + Tandoor Recipes — Felhom.eu + + + + + + + + +
+ + +
+ + +
+ + + + + + +
+ + +
+ + Hub kapcsolat kikapcsolva — a központi monitoring nem aktív + Rendszermonitor → +
+ + +
+ + + +
+
+ ← Alkalmazások +

Tandoor Recipes

+
+
+ + Nem egészséges + + + Frissítés elérhető — ma + Megnyitás ↗ + Napló + + Exportálás + Beállítások + + +
+
+ + + + + + +
+ +
+ +

Receptkezelő és étkezés tervező a családnak

+ +
+ ~512M RAM + home + + Pi kompatibilis + +
+ +
+
+ + +
+ + + +
+ + + + +
+

Áthelyezés másik tárhelyre

+

Ennek az alkalmazásnak az adatait másik csatlakoztatott tárhelyre helyezheted át. Az alkalmazás az áthelyezés alatt rövid időre leáll, az adatok pedig csak az ellenőrzés és a sikeres újraindítás után törlődnek a régi helyről.

+
+ + +
+ +
+ + + + +
+ +
+

Mire használható?

+
    +
  • Receptek gyűjtése és rendszerezése egy helyen
  • Receptek importálása weboldalakról egy kattintással
  • Heti étkezés tervezés és bevásárlólista generálás
  • Több felhasználó - a család együtt tervezhet
  • Receptek megosztása linkkel családtagokkal, barátokkal
    • +
+
+ + + +
+

Első lépések

+
    +
  1. Nyisd meg a recipes.DOMAIN címet a böngészőben
  2. Hozd létre az admin fiókot
  3. Importáld az első receptet egy weboldalról (Bookmarklet)
  4. Próbáld ki az étkezés tervezőt
  5. Hívd meg a családtagokat
    6. +
+
+ + + + + + + + + +
+

Dokumentáció

+

Hivatalos dokumentáció ↗

+
+ +
+ + + + + +
+ + + + diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/log.txt b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/log.txt new file mode 100644 index 00000000..b6074544 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/log.txt @@ -0,0 +1,25 @@ +21:15:25 ==== tandoor: ghcr.io/tandoorrecipes/recipes:2.6.13 -> ghcr.io/tandoorrecipes/recipes:2.6.15 (sub=recipes, class=db-postgres) +21:15:25 [1] tandoor already deployed — reusing +21:15:27 [2] seeding through the app's own front door +21:15:37 tandoor: createsuperuser :: Running django-vite in production mode (no HMR) Superuser created successfully. +21:15:40 tandoor: seeded superuser drill316903 +21:15:40 [3] control C1 — reading the seed back BEFORE the update +21:15:46 tandoor: readback of the seeded account found=True +21:15:46 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +21:16:42 [4] backup idle; last=None +21:16:43 [5] drill commit c7477d1fb921: tandoor ghcr.io/tandoorrecipes/recipes:2.6.13 -> ghcr.io/tandoorrecipes/recipes:2.6.15 (push rc=0) +21:16:47 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +21:16:47 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +21:16:47 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +21:16:47 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +21:16:48 + 1.0s phase=pulling label=Új verzió letöltése… err=None hold=None +21:17:42 + 54.3s phase=starting label=Indítás az új verzióval… err=None hold=None +21:17:46 + 58.4s phase=verifying label=Működés ellenőrzése… err=None hold=None +21:22:49 + 361.9s phase=failed label=A frissítés nem sikerült err=A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza. hold=A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza. +21:22:49 [7] reading the seed back AFTER the update +21:22:52 tandoor: READBACK UNUSABLE — a username that cannot exist did not read as absent (None) :: Error response from daemon: No such container: tandoor +21:22:54 [8] pinned = {'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:16-alpine'} +21:22:54 [8] installed = {'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.13', 'tandoor-postgres': 'postgres:16-alpine'} +21:22:54 [8] compose = ['image: ghcr.io/tandoorrecipes/recipes:2.6.15', 'image: postgres:16-alpine'] +21:22:54 [8] inspect = [] +21:22:54 [9] verdict failed -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/tandoor/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/observables-before.json new file mode 100644 index 00000000..be655443 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/observables-before.json @@ -0,0 +1,22 @@ +{ + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "catalog_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/tandoorrecipes/recipes:2.6.13", + "image: postgres:16-alpine" + ], + "docker_inspect": [ + "tandoor ghcr.io/tandoorrecipes/recipes:2.6.13 running=true restarts=0", + "tandoor-postgres postgres:16-alpine running=true restarts=0" + ] +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/observables.json b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/observables.json new file mode 100644 index 00000000..19e1cc15 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/observables.json @@ -0,0 +1,43 @@ +{ + "before": { + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "catalog_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/tandoorrecipes/recipes:2.6.13", + "image: postgres:16-alpine" + ], + "docker_inspect": [ + "tandoor ghcr.io/tandoorrecipes/recipes:2.6.13 running=true restarts=0", + "tandoor-postgres postgres:16-alpine running=true restarts=0" + ] + }, + "after": { + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "catalog_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/tandoorrecipes/recipes:2.6.15", + "image: postgres:16-alpine" + ], + "docker_inspect": [] + } +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/phases.json b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/phases.json new file mode 100644 index 00000000..67317e2d --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/phases.json @@ -0,0 +1,51 @@ +{ + "accepted": true, + "http": "202", + "phases": [ + { + "t": 0.0, + "phase": "safety-dump", + "label": "Adatbázis pillanatkép…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 1.0, + "phase": "pulling", + "label": "Új verzió letöltése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 54.3, + "phase": "starting", + "label": "Indítás az új verzióval…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 58.4, + "phase": "verifying", + "label": "Működés ellenőrzése…", + "updating": true, + "error": null, + "hold": null + }, + { + "t": 361.9, + "phase": "failed", + "label": "A frissítés nem sikerült", + "updating": false, + "error": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "hold": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza." + } + ], + "duration_s": 361.9, + "final_phase": "failed", + "update_error": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "hold_reason": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "state": "stopped" +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/verdict.json b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/verdict.json new file mode 100644 index 00000000..f7497e30 --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor-before-probe-fix-2026-09-21/verdict.json @@ -0,0 +1,49 @@ +{ + "harness_version": 1, + "app": "tandoor", + "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", + "class": "db-postgres", + "from": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "to": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "verdict": "failed", + "seed_read_before": true, + "seed_read_after": false, + "healthy_after": false, + "migration_observed": null, + "abort": "not-attempted", + "abort_detail": null, + "duration_s": 361.9, + "measured_at": "2026-09-21T19:15:25.308811+00:00", + "evidence": "apps/tandoor/", + "notes": [ + "the edge ended HELD or failed — this is a RESULT, not an error of the run" + ], + "observables_after": { + "pinned_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "installed_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor-postgres": "postgres:16-alpine" + }, + "catalog_images": { + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", + "tandoor-postgres": "postgres:16-alpine" + }, + "live_compose_image_lines": [ + "image: ghcr.io/tandoorrecipes/recipes:2.6.15", + "image: postgres:16-alpine" + ], + "docker_inspect": [] + }, + "final_phase": "failed", + "hold_reason": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", + "update_error": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza." +} \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor/app-logs-after.txt b/documentation/audits/update-night-2026-09-21/apps/tandoor/app-logs-after.txt index e69de29b..96755741 100644 --- a/documentation/audits/update-night-2026-09-21/apps/tandoor/app-logs-after.txt +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor/app-logs-after.txt @@ -0,0 +1,509 @@ +tandoor-postgres | +tandoor-postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization +tandoor-postgres | +tandoor-postgres | 2026-09-22 10:53:18.859 CEST [1] LOG: starting PostgreSQL 16.15 on x86_64-pc-linux-musl, compiled by gcc (Alpine 15.2.0) 15.2.0, 64-bit +tandoor-postgres | 2026-09-22 10:53:18.859 CEST [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 +tandoor-postgres | 2026-09-22 10:53:18.859 CEST [1] LOG: listening on IPv6 address "::", port 5432 +tandoor-postgres | 2026-09-22 10:53:18.867 CEST [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" +tandoor-postgres | 2026-09-22 10:53:18.880 CEST [29] LOG: database system was shut down at 2026-09-22 10:53:09 CEST +tandoor-postgres | 2026-09-22 10:53:18.895 CEST [1] LOG: database system is ready to accept connections +tandoor | Deleting 'vue3/assets/pt_BR-tNFhL9ci.js.gz' +tandoor | Deleting 'vue3/assets/SupermarketEditor-L_B5nyx2.3796d8a521cf.js' +tandoor | Deleting 'vue3/assets/MealTypeEditor-DAAKjxZB.js' +tandoor | Deleting 'vue3/assets/sl-Nkic6vBb.c6ce1eb24108.js' +tandoor | Deleting 'vue3/assets/nl-DIqabTrY.fcf6d7b2df13.js.gz' +tandoor | Deleting 'vue3/assets/UnitEditor-54HrDOhT.9040eae92234.js.gz' +tandoor | Deleting 'vue3/assets/FoodEditor-BR-tK8Pr.js.gz' +tandoor | Deleting 'vue3/assets/forwardRefs-DVyU53sF.js' +tandoor | Deleting 'vue3/assets/position-PPD1ymuo.js.gz' +tandoor | Deleting 'vue3/assets/ca-C5QQCa9o.ff6e54a70768.js.gz' +tandoor | Deleting 'vue3/assets/KeywordsBar-BwJ0lEDV.js' +tandoor | Deleting 'vue3/assets/number_utils-DFmVVcK0.js' +tandoor | Deleting 'vue3/assets/ja-Cqm3qbrP.js.gz' +tandoor | Deleting 'vue3/assets/bg-_Ardqm8i.c46924a2b9d5.js' +tandoor | Deleting 'vue3/assets/VOverlay-CKBxglwy.b98729efffd0.css.gz' +tandoor | Deleting 'vue3/assets/fr-D1J9AIHe.js' +tandoor | Deleting 'vue3/assets/goto-DubWlX_o.0c25d7a8f5fa.js.gz' +tandoor | Deleting 'vue3/assets/VOverlay-CKBxglwy.css.gz' +tandoor | Deleting 'vue3/assets/lv-BUrQRxBI.8efc7d113546.js' +tandoor | Deleting 'vue3/assets/AccessTokenEditor-BqYrLuQE.4d7347e87897.js.gz' +tandoor | Deleting 'vue3/assets/VSwitch-Djsboh2v.92774fc66b84.css.gz' +tandoor | Deleting 'vue3/assets/PropertyEditor-DK6Qeffb.864b139cbb79.js' +tandoor | Deleting 'vue3/assets/ModelDeletePage-C4id8-tu.1ed7ee64e600.js.gz' +tandoor | Deleting 'vue3/assets/da-DHqdZ4yZ.197be8016aca.js' +tandoor | Deleting 'vue3/assets/ro-CG_JGo0i.js' +tandoor | Deleting 'vue3/assets/ExportDataSettings-BlFk8Tn-.6ccf02326ec4.js.gz' +tandoor | Deleting 'vue3/assets/PropertiesEditor-DCxEsAZO.16f243b74d54.js' +tandoor | Deleting 'vue3/assets/RecipeImportPage-DBKJ6DXr.a23aec8a9383.js' +tandoor | Deleting 'vue3/assets/el-DNs6HoDL.08d7c47f5d2b.js' +tandoor | Deleting 'vue3/assets/RecipeViewPage-BY3ErEFj.457b3d77074e.css' +tandoor | Deleting 'vue3/assets/DeleteConfirmDialog-C1xQIJsS.b663eac25ee4.js.gz' +tandoor | Deleting 'vue3/assets/VTimePicker-DNtQ2GaY.658bf461867a.css' +tandoor | Deleting 'vue3/assets/logo_color-CXE3OqOR.d7c2e31a63b7.svg' +tandoor | Deleting 'vue3/assets/fa-v4compatibility-CCth-dXg.4ed293ceaca9.ttf.gz' +tandoor | Deleting 'vue3/assets/IngredientEditorPage-DF3k1tNH.aa9eb5c590ab.js' +tandoor | Deleting 'vue3/assets/InventoryLocationEditor-DIxi9S-h.5de878ee0a17.js' +tandoor | Deleting 'vue3/assets/ShoppingListView-Dgh1luaN.css' +tandoor | Deleting 'vue3/assets/StartPage-0ugVot93.js.gz' +tandoor | Deleting 'vue3/assets/DeleteConfirmDialog-C1xQIJsS.js' +tandoor | Deleting 'vue3/assets/MealPlanEditor-B5jLcokm.js.gz' +tandoor | Deleting 'vue3/assets/number_utils-Brwft-8t.d570ceeec9ef.css' +tandoor | Deleting 'vue3/assets/MealPlanEditor-B5jLcokm.7c8b48842426.js' +tandoor | Deleting 'vue3/assets/VDivider-BeM1gRhl.css' +tandoor | Deleting 'vue3/assets/AccountSettings-BHfX12gt.js' +tandoor | Deleting 'vue3/assets/CustomFilterEditor-Dc0piaF0.js' +tandoor | Deleting 'vue3/assets/lt-SI6E7sY1.2560d8a9d387.js.gz' +tandoor | Deleting 'vue3/assets/de-5mnKTxw2.js' +tandoor | Deleting 'vue3/assets/SearchSettings-CBkFhMns.js' +tandoor | Deleting 'vue3/assets/logo_color-CXE3OqOR.svg' +tandoor | Deleting 'vue3/assets/fontello-B1X0PDnA.068ca2b316db.ttf' +tandoor | Deleting 'vue3/assets/PropertyEditor-DK6Qeffb.js' +tandoor | Deleting 'vue3/assets/RecipeEditor-CcFlDNPw.css' +tandoor | Deleting 'vue3/assets/RecipeCard-C2Hjv7vL.6b450d3cfcd0.js.gz' +tandoor | Deleting 'vue3/assets/CustomFilterEditor-Dc0piaF0.fe7085369993.js.gz' +tandoor | Deleting 'vue3/assets/VStepper-lk6OG_fY.23af7e185e38.css' +tandoor | Deleting 'vue3/assets/VList-edQgt9_l.2642c051884b.css.gz' +tandoor | Deleting 'vue3/assets/InventoryBookingPage-orELmDxm.js' +tandoor | Deleting 'vue3/assets/PropertyEditorPage-Ti9vpeer.e4b532c48301.js' +tandoor | Deleting 'vue3/assets/pt-CXzOqbUC.js' +tandoor | Deleting 'vue3/assets/BooksPage-Dh8_wlv_.cb4aba43024a.js' +tandoor | Deleting 'vue3/assets/ModelListPage-xoBv_A6i.js.gz' +tandoor | Deleting 'vue3/assets/number_utils-DFmVVcK0.js.gz' +tandoor | Deleting 'vue3/assets/FdcSearchDialog-BvpKKey4.js.gz' +tandoor | Deleting 'vue3/assets/position-PPD1ymuo.7195d47ccd52.js' +tandoor | Deleting 'vue3/assets/model_utils-a_hMB_Qu.js' +tandoor | Deleting 'vue3/assets/useFileApi-BD_kCiAS.js' +tandoor | Deleting 'vue3/assets/he-DxNGVUQd.79ff03f4063a.js' +tandoor | Deleting 'vue3/assets/zh_Hant-CERwsE1Q.js.gz' +tandoor | Deleting 'vue3/assets/SearchPage-BQ1P-4y8.ba1f4c8767b9.js.gz' +tandoor | Deleting 'vue3/assets/InventoryLocationEditor-DIxi9S-h.js' +tandoor | Deleting 'vue3/assets/fi-BLW_HATq.078772422941.js' +tandoor | Deleting 'vue3/assets/nn-CkCilbi1.ccb3acb89917.js.gz' +tandoor | Deleting 'vue3/assets/UserFileEditor-DKOcBuK-.js' +tandoor | Deleting 'vue3/assets/useFileApi-BD_kCiAS.37a20e72aad4.js.gz' +tandoor | Deleting 'vue3/assets/hu-DlZ4REsp.js' +tandoor | Deleting 'vue3/assets/pt-CXzOqbUC.js.gz' +tandoor | Deleting 'vue3/assets/VTimePicker-BwPSjcxJ.a223b07c705e.js' +tandoor | Deleting 'vue3/assets/pt_BR-tNFhL9ci.js' +tandoor | Deleting 'vue3/assets/CustomFilterEditor-Dc0piaF0.js.gz' +tandoor | Deleting 'vue3/assets/ko-ADlXHFgo.js' +tandoor | Deleting 'vue3/assets/VDataTableServer-BYvhcs5d.js' +tandoor | Deleting 'vue3/assets/VBtn-rFOjfrXe.df1b89399b11.css.gz' +tandoor | Deleting 'vue3/assets/SupermarketCategoryEditor-rlIgT3uE.js' +tandoor | Deleting 'vue3/assets/CosmeticSettings-DTudQs61.0308c579b183.css' +tandoor | Deleting 'vue3/assets/RecipeEditor-CcFlDNPw.c63cac250b70.css.gz' +tandoor | Deleting 'vue3/assets/fontello-BJkOxCgW.8d4a4e6f7431.woff2' +tandoor | Deleting 'vue3/assets/MealPlanEditor-B5jLcokm.js' +tandoor | Deleting 'vue3/assets/VAvatar-DYNiWd3v.js' +tandoor | Deleting 'vue3/assets/brand_logo-DHt4CyHb.svg' +tandoor | Deleting 'vue3/assets/ripple-CeGB3d9x.3ba9fc624e34.css.gz' +tandoor | Deleting 'vue3/assets/OpenDataImportSettings-Ds1SQcSg.js.gz' +tandoor | Deleting 'vue3/assets/dimensions-B5x3TxJs.js.gz' +tandoor | Deleting 'vue3/assets/PropertiesEditor-DCxEsAZO.js.gz' +tandoor | Deleting 'vue3/assets/PropertiesEditor-DCxEsAZO.16f243b74d54.js.gz' +tandoor | Deleting 'vue3/assets/VFileUpload-ByFVW7Fp.931d5a48009e.js.gz' +tandoor | Deleting 'vue3/assets/DatabaseModelCol-rKX1F7Em.js.gz' +tandoor | Deleting 'vue3/assets/ConnectorConfigEditor-D06JOZDX.5e0dd3a66e81.js.gz' +tandoor | Deleting 'vue3/assets/IngredientEditorPage-DF3k1tNH.js.gz' +tandoor | Deleting 'vue3/assets/lt-SI6E7sY1.js' +tandoor | Deleting 'vue3/assets/DatabasePage-CY4BXH8T.js.gz' +tandoor | Deleting 'vue3/assets/bg-_Ardqm8i.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListView-C0_akhap.js' +tandoor | Deleting 'vue3/assets/HierarchyEditor-wADOsT_-.css' +tandoor | Deleting 'vue3/assets/fontello-B1X0PDnA.ttf' +tandoor | Deleting 'vue3/assets/VOverlay-BSdT-t2A.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListView-Dgh1luaN.css.gz' +tandoor | Deleting 'vue3/assets/it-DJ2I6vOs.js.gz' +tandoor | Deleting 'vue3/assets/bg-_Ardqm8i.js' +tandoor | Deleting 'vue3/assets/VSwitch-Cs5DQi32.4a7617f1519d.js' +tandoor | Deleting 'vue3/assets/logo_color-CXE3OqOR.svg.gz' +tandoor | Deleting 'vue3/assets/WelcomePage-zPtXSDjc.4dfacc5c8da0.js.gz' +tandoor | Deleting 'vue3/assets/UnitEditor-54HrDOhT.9040eae92234.js' +tandoor | Deleting 'vue3/assets/hr-CMRkObu_.js' +tandoor | Deleting 'vue3/assets/VTextField-j0TpvUy_.42df5f015009.css.gz' +tandoor | Deleting 'vue3/assets/OpenDataImportSettings-Ds1SQcSg.f895a746d306.js.gz' +tandoor | Deleting 'vue3/assets/ja-Cqm3qbrP.d617fa524f97.js' +tandoor | Deleting 'vue3/assets/VAvatar-DYNiWd3v.js.gz' +tandoor | Deleting 'vue3/assets/InventoryBookingPage-orELmDxm.js.gz' +tandoor | Deleting 'vue3/assets/ApiSettings-CzWgpe8R.82be1325ed18.js' +tandoor | Deleting 'vue3/assets/fontello-CnWxryRb.eot' +tandoor | Deleting 'vue3/assets/VList-DL25Fkt0.js.gz' +tandoor | Deleting 'vue3/assets/InventoryEntryLogDialog-Dv6Omj68.d4f65e984493.js.gz' +tandoor | Deleting 'vue3/assets/VSwitch-Cs5DQi32.4a7617f1519d.js.gz' +tandoor | Deleting 'vue3/assets/VColorPicker-DoVATq5z.26d1fdd28765.css.gz' +tandoor | Deleting 'vue3/assets/HouseholdEditor-D5JQGcsU.js' +tandoor | Deleting 'vue3/assets/VList-edQgt9_l.css.gz' +tandoor | Deleting 'vue3/assets/VTabs-DzUjbg1Y.js.gz' +tandoor | Deleting 'vue3/assets/ConnectorConfigEditor-D06JOZDX.js' +tandoor | Deleting 'vue3/assets/BatchDeleteDialog-C9gKVWYX.js.gz' +tandoor | Deleting 'vue3/assets/IngredientEditorPage-DF3k1tNH.aa9eb5c590ab.js.gz' +tandoor | Deleting 'vue3/assets/nn-CkCilbi1.ccb3acb89917.js' +tandoor | Deleting 'vue3/assets/InventoryEntryLogDialog-Dv6Omj68.d4f65e984493.js' +tandoor | Deleting 'vue3/assets/VDivider-BeM1gRhl.727c640d6c8d.css.gz' +tandoor | Deleting 'vue3/assets/ApiSettings-CzWgpe8R.js' +tandoor | Deleting 'vue3/assets/SpaceEditor-BG9pNfzF.9465c6dc1d2f.css' +tandoor | Deleting 'vue3/assets/ripple-CeGB3d9x.css.gz' +tandoor | Deleting 'vue3/assets/AiProviderEditor-C4TWtCwO.e6d60050b025.css.gz' +tandoor | Deleting 'vue3/assets/SupermarketEditor-L_B5nyx2.3796d8a521cf.js.gz' +tandoor | Deleting 'vue3/assets/VDivider-BeM1gRhl.727c640d6c8d.css' +tandoor | Deleting 'vue3/assets/fa-v4compatibility-CCth-dXg.4ed293ceaca9.ttf' +tandoor | Deleting 'vue3/assets/VListItemAction-CXKY8J-P.js.gz' +tandoor | Deleting 'vue3/assets/VColorPicker-DoVATq5z.css.gz' +tandoor | Deleting 'vue3/assets/ModelListPage-xoBv_A6i.4681a991ae27.js.gz' +tandoor | Deleting 'vue3/assets/uk-BClnjIfW.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListPage-Cg_SrILL.f2cae0d44a21.js' +tandoor | Deleting 'vue3/assets/RecipeViewPage-BZECYYUo.94016f72f4cb.js.gz' +tandoor | Deleting 'vue3/assets/number_utils-Brwft-8t.css.gz' +tandoor | Deleting 'vue3/assets/de-5mnKTxw2.js.gz' +tandoor | Deleting 'vue3/assets/VFileUpload-7CWUohzn.d182bb8e4f03.css.gz' +tandoor | Deleting 'vue3/assets/VListItemAction-CXKY8J-P.ffd6a13b9ab5.js' +tandoor | Deleting 'vue3/assets/VAvatar-_OUjgGcO.css.gz' +tandoor | Deleting 'vue3/assets/position-PPD1ymuo.7195d47ccd52.js.gz' +tandoor | Deleting 'vue3/assets/brand_logo-DHt4CyHb.fa64319ab35b.svg.gz' +tandoor | Deleting 'vue3/assets/VChip-D7GkfSeA.js' +tandoor | Deleting 'vue3/assets/ConnectorConfigEditor-D06JOZDX.5e0dd3a66e81.js' +tandoor | Deleting 'vue3/assets/fileFilter-JaX38xV6.js.gz' +tandoor | Deleting 'vue3/assets/MealTypeEditor-DAAKjxZB.js.gz' +tandoor | Deleting 'vue3/assets/pl-B7dc5eNY.js' +tandoor | Deleting 'vue3/assets/MealPlanSettings-BU5yRyFO.js' +tandoor | Deleting 'vue3/assets/VAutocomplete-BnHJd-1b.css.gz' +tandoor | Deleting 'vue3/assets/framework-CKkm3ZT1.0da2482ae88a.js.gz' +tandoor | Deleting 'vue3/assets/AccessTokenEditor-BqYrLuQE.js' +tandoor | Deleting 'vue3/assets/VColorPicker-Bc1o5piM.js.gz' +tandoor | Deleting 'vue3/assets/UserSpaceEditor-BiMFbwI_.7d9049ae96b5.js.gz' +tandoor | Deleting 'vue3/assets/CookLogEditor-wf-1cNFJ.ef3a04991a05.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingSettings-DBfLsRlj.js' +tandoor | Deleting 'vue3/assets/AddToShoppingDialog-CS_S-d2g.eb5504799c65.js' +tandoor | Deleting 'vue3/assets/fa-solid-900-CTAAxXor.4a6591ab5460.woff2' +tandoor | Deleting 'vue3/assets/MealPlanSettings-BU5yRyFO.2f9dbd605868.js.gz' +tandoor | Deleting 'vue3/assets/AiProviderEditor-BYnKvCw_.js.gz' +tandoor | Deleting 'vue3/assets/uk-BClnjIfW.b429b8ed5096.js' +tandoor | Deleting 'vue3/assets/BookViewPage-CHarlNiQ.bdf998b08053.js.gz' +tandoor | Deleting 'vue3/assets/HouseholdPage-BsKZpym2.js' +tandoor | Deleting 'vue3/assets/fi-BLW_HATq.078772422941.js.gz' +tandoor | Deleting 'vue3/assets/fontello-CnWxryRb.e73a0647198c.eot.gz' +tandoor | Deleting 'vue3/assets/InviteLinkEditor-BZ7NMcrL.js' +tandoor | Deleting 'vue3/assets/VChip-D7GkfSeA.4de7c0a8c2e3.js.gz' +tandoor | Deleting 'vue3/assets/VRating-BAibLkoF.css' +tandoor | Deleting 'vue3/assets/RecipeBookEditor-Bha-wuB-.js' +tandoor | Deleting 'vue3/assets/VTextField-WpOvMnND.js' +tandoor | Deleting 'vue3/assets/el-DNs6HoDL.08d7c47f5d2b.js.gz' +tandoor | Deleting 'vue3/assets/AutomationEditor-C6fYAtLL.js.gz' +tandoor | Deleting 'vue3/assets/fontello-CnWxryRb.e73a0647198c.eot' +tandoor | Deleting 'vue3/assets/VDivider-rMXvH15P.085aaed48fa0.js' +tandoor | Deleting 'vue3/assets/ShoppingListView-Dgh1luaN.73de85b4e4df.css.gz' +tandoor | Deleting 'vue3/assets/UserFileEditor-DKOcBuK-.js.gz' +tandoor | Deleting 'vue3/assets/VAutocomplete-CFEp5VcY.js' +tandoor | Deleting 'vue3/assets/SpaceEditor-BG9pNfzF.css' +tandoor | Deleting 'vue3/assets/PropertyEditorPage-Ti9vpeer.js' +tandoor | Deleting 'vue3/assets/ShoppingListPage-Cg_SrILL.js' +tandoor | Deleting 'vue3/assets/integration_utils-D8RSOVmp.js.gz' +tandoor | Deleting 'vue3/assets/nb_NO-CdS1Hktt.294979d98dcb.js' +tandoor | Deleting 'vue3/assets/SearchSettings-CBkFhMns.js.gz' +tandoor | Deleting 'vue3/assets/VChip-Hpg9IuyM.8dc9d3e18964.css.gz' +tandoor | Deleting 'vue3/assets/VAutocomplete-BnHJd-1b.css' +tandoor | Deleting 'vue3/assets/ja-Cqm3qbrP.js' +tandoor | Deleting 'vue3/assets/VOverlay-BSdT-t2A.js' +tandoor | Deleting 'vue3/assets/VList-DL25Fkt0.5373fcf8fbc3.js' +tandoor | Deleting 'vue3/assets/SearchPage-BQ1P-4y8.js' +tandoor | Deleting 'vue3/assets/StartPage-0ugVot93.1c72303e8853.js.gz' +tandoor | Deleting 'vue3/assets/fdc-Byjxh21q.6a8962cb2f19.js.gz' +tandoor | Deleting 'vue3/assets/UserSpaceEditor-BiMFbwI_.7d9049ae96b5.js' +tandoor | Deleting 'vue3/assets/PropertyEditorPage-Ti9vpeer.e4b532c48301.js.gz' +tandoor | Deleting 'vue3/assets/SpaceEditor-CIrx7Wew.44bd56e30e62.js.gz' +tandoor | Deleting 'vue3/assets/CosmeticSettings-DTudQs61.css.gz' +tandoor | Deleting 'vue3/assets/transitions-BVkpOt9D.0916b5adcb73.js.gz' +tandoor | Deleting 'vue3/assets/ModelEditPage-B_U4Berv.js' +tandoor | Deleting 'vue3/assets/dimensions-B5x3TxJs.5c0bb4b51390.js' +tandoor | Deleting 'vue3/assets/AccessTokenEditor-BqYrLuQE.4d7347e87897.js' +tandoor | Deleting 'vue3/assets/VColorPicker-DoVATq5z.26d1fdd28765.css' +tandoor | Deleting 'vue3/assets/AddToShoppingDialog-BXt5jKLm.css' +tandoor | Deleting 'vue3/assets/ru-BXmxlY4u.js.gz' +tandoor | Deleting 'vue3/assets/hu-DlZ4REsp.c1bc941c31dc.js.gz' +tandoor | Deleting 'vue3/assets/VList-edQgt9_l.css' +tandoor | Deleting 'vue3/assets/UnitConversionEditor-BUHvM5Eh.js.gz' +tandoor | Deleting 'vue3/assets/VTextarea-zkkSnzux.js.gz' +tandoor | Deleting 'vue3/assets/ModelEditPage-B_U4Berv.7dc258e43ccf.js.gz' +tandoor | Deleting 'vue3/assets/RecipeCard-C2Hjv7vL.6b450d3cfcd0.js' +tandoor | Deleting 'vue3/assets/MealPlanEditor-DqGGZRzo.css' +tandoor | Deleting 'vue3/assets/VSwitch-Cs5DQi32.js.gz' +tandoor | Deleting 'vue3/assets/TestPage-CPX2qaEd.0b75aa57bbeb.js.gz' +tandoor | Deleting 'vue3/assets/dimensions-B5x3TxJs.5c0bb4b51390.js.gz' +tandoor | Deleting 'vue3/assets/main-nTFEtOSD.js.gz' +tandoor | Deleting 'vue3/assets/UserFileEditor-DKOcBuK-.cc1baa1c02f3.js' +tandoor | Deleting 'vue3/assets/StorageEditor-Up4uWQb6.js' +tandoor | Deleting 'vue3/assets/VBtn-DTdW4rEh.js.gz' +tandoor | Deleting 'vue3/assets/he-DxNGVUQd.js.gz' +tandoor | Deleting 'vue3/assets/VChip-Hpg9IuyM.8dc9d3e18964.css' +tandoor | Deleting 'vue3/assets/VTextarea-dHE66AZf.css' +tandoor | Deleting 'vue3/assets/intersect-siLrmRQw.cc9e340f1351.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListPage-Cg_SrILL.js.gz' +tandoor | Deleting 'vue3/assets/framework-CKkm3ZT1.js' +tandoor | Deleting 'vue3/assets/ca-C5QQCa9o.ff6e54a70768.js' +tandoor | Deleting 'vue3/assets/fa-solid-900-D0aA9rwL.ttf' +tandoor | Deleting 'vue3/assets/VListItemAction-CXKY8J-P.ffd6a13b9ab5.js.gz' +tandoor | Deleting 'vue3/assets/main-nTFEtOSD.4273b900f1bf.js' +tandoor | Deleting 'vue3/assets/transitions-BVkpOt9D.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListEditor-vKmfFTZN.4f5dd4c981a5.js' +tandoor | Deleting 'vue3/assets/ripple-CeGB3d9x.css' +tandoor | Deleting 'vue3/assets/ClosableHelpAlert-BoN8dDrm.js.gz' +tandoor | Deleting 'vue3/assets/position-D7bmUCsu.f212789ebec5.css' +tandoor | Deleting 'vue3/assets/ShoppingListView-C0_akhap.1b6d4259ca8e.js.gz' +tandoor | Deleting 'vue3/assets/hy-HqufUZnw.js' +tandoor | Deleting 'vue3/assets/InventoryBookingPage-orELmDxm.45c2c2f7531a.js' +tandoor | Deleting 'vue3/assets/el-DNs6HoDL.js' +tandoor | Deleting 'vue3/assets/VTabs-DzUjbg1Y.fa06924ae72f.js' +tandoor | Deleting 'vue3/assets/form-CDdumt48.5fcbf914ae94.js' +tandoor | Deleting 'vue3/assets/HouseholdPage-BsKZpym2.a5566a4564d6.js.gz' +tandoor | Deleting 'vue3/assets/forwardRefs-DVyU53sF.64171d66d1df.js.gz' +tandoor | Deleting 'vue3/assets/integration_utils-D8RSOVmp.8006123d3084.js.gz' +tandoor | Deleting 'vue3/assets/SpaceSettings-BPOt7vl3.js.gz' +tandoor | Deleting 'vue3/assets/ModelEditorBase-CZWlxxIQ.cf852babb62d.js' +tandoor | Deleting 'vue3/assets/RecipeBookEditor-Bha-wuB-.js.gz' +tandoor | Deleting 'vue3/assets/ModelEditPage-B5oBrkJw.0c31f8b7052c.css.gz' +tandoor | Deleting 'vue3/assets/VAutocomplete-CFEp5VcY.2a4e88a23ede.js.gz' +tandoor | Deleting 'vue3/assets/FdcSearchDialog-BvpKKey4.js' +tandoor | Deleting 'vue3/assets/transitions-BVkpOt9D.js' +tandoor | Deleting 'vue3/assets/ro-CG_JGo0i.c856bd470eef.js' +tandoor | Deleting 'vue3/assets/RecipeViewPage-BZECYYUo.js.gz' +tandoor | Deleting 'vue3/assets/IngredientsTable-DZblS1P-.js.gz' +tandoor | Deleting 'vue3/assets/RecipeCard-C2Hjv7vL.js.gz' +tandoor | Deleting 'vue3/assets/CosmeticSettings-i2TDHCpS.js' +tandoor | Deleting 'vue3/assets/VBtn-rFOjfrXe.css.gz' +tandoor | Deleting 'vue3/assets/FoodEditor-BR-tK8Pr.24df4c5c0e7e.js.gz' +tandoor | Deleting 'vue3/assets/VDivider-rMXvH15P.js' +tandoor | Deleting 'vue3/assets/cs-DUZpBKTY.js' +tandoor | Deleting 'vue3/assets/TestPage-CPX2qaEd.0b75aa57bbeb.js' +tandoor | Deleting 'vue3/assets/fdc-Byjxh21q.js' +tandoor | Deleting 'vue3/assets/CookLogEditor-wf-1cNFJ.js.gz' +tandoor | Deleting 'vue3/assets/HouseholdEditor-D5JQGcsU.js.gz' +tandoor | Deleting 'vue3/assets/VDivider-rMXvH15P.085aaed48fa0.js.gz' +tandoor | Deleting 'vue3/assets/ripple-Byn3Fc0w.3ad96cecfca7.js' +tandoor | Deleting 'vue3/assets/UserSpaceEditor-BiMFbwI_.js.gz' +tandoor | Deleting 'vue3/assets/fr-D1J9AIHe.a85f6556b6a2.js.gz' +tandoor | Deleting 'vue3/assets/fr-D1J9AIHe.js.gz' +tandoor | Deleting 'vue3/assets/VStepper-lk6OG_fY.css.gz' +tandoor | Deleting 'vue3/assets/ko-ADlXHFgo.js.gz' +tandoor | Deleting 'vue3/assets/WelcomePage-zPtXSDjc.4dfacc5c8da0.js' +tandoor | Deleting 'vue3/assets/VChip-D7GkfSeA.js.gz' +tandoor | Deleting 'vue3/assets/SpaceSetupPage-BSv2ZmPb.js' +tandoor | Deleting 'vue3/assets/zh_Hant-CERwsE1Q.8534ecee0fb0.js.gz' +tandoor | Deleting 'vue3/assets/ShoppingListPage-Cg_SrILL.f2cae0d44a21.js.gz' +tandoor | Deleting 'vue3/assets/UserSpaceEditor-BiMFbwI_.js' +tandoor | Deleting 'vue3/assets/VOverlay-BSdT-t2A.1f4f586d7141.js.gz' +tandoor | Deleting 'vue3/assets/RecipeImportPage-DBKJ6DXr.a23aec8a9383.js.gz' +tandoor | Deleting 'vue3/assets/VDataTableServer-BYvhcs5d.d5a7cfdcf126.js' +tandoor | Deleting 'vue3/assets/ar-Ct4DpnJT.js' +tandoor | Deleting 'vue3/assets/VRating-Qn4Hz2ci.119bbc89c24a.js' +tandoor | Deleting 'vue3/assets/number_utils-DFmVVcK0.cfc3e206a4c1.js.gz' +tandoor | Deleting 'vue3/assets/HelpPage-C-xlKPtR.fa54da2fd1cf.js.gz' +tandoor | Deleting 'vue3/assets/HierarchyEditor-wADOsT_-.885aa06e77cd.css.gz' +tandoor | Deleting 'vue3/assets/ripple-Byn3Fc0w.3ad96cecfca7.js.gz' +tandoor | Deleting 'vue3/assets/VChip-Hpg9IuyM.css.gz' +tandoor | Deleting 'vue3/assets/RecipeViewPage-BZECYYUo.94016f72f4cb.js' +tandoor | Deleting 'vue3/assets/PrivateRecipeBadge-BVOXRXwb.dcd01dd00de0.js' +tandoor | Deleting 'vue3/assets/SearchSettings-CBkFhMns.b9bc4a1f032c.js' +tandoor | Deleting 'vue3/assets/fontello-BEgLts9b.a782baa8633b.woff' +tandoor | Deleting 'vue3/assets/nl-DIqabTrY.js.gz' +tandoor | Deleting 'vue3/assets/et-zY9CrDMt.1761c6eb5f48.js' +tandoor | Deleting 'vue3/assets/DeleteConfirmDialog-C1xQIJsS.b663eac25ee4.js' +tandoor | Deleting 'vue3/assets/RecipeEditor-Cu9LLPvL.js.gz' +tandoor | Deleting 'vue3/assets/RecipeBookEditor-Bha-wuB-.e3586751a38a.js' +tandoor | Deleting 'vue3/assets/goto-DubWlX_o.js.gz' +tandoor | Deleting 'vue3/assets/ModelMergeDialog-BCXJZjTO.4753553f721f.js' +tandoor | Deleting 'vue3/assets/AutomationEditor-C6fYAtLL.js' +tandoor | Deleting 'vue3/assets/VTextField-j0TpvUy_.42df5f015009.css' +tandoor | Deleting 'vue3/assets/SupermarketEditor-L_B5nyx2.js' +tandoor | Deleting 'vue3/assets/el-DNs6HoDL.js.gz' +tandoor | Deleting 'vue3/assets/lv-BUrQRxBI.8efc7d113546.js.gz' +tandoor | Deleting 'vue3/assets/MealPlanSettings-BU5yRyFO.2f9dbd605868.js' +tandoor | Deleting 'vue3/assets/NumberScalerDialog-DoOvxgjM.js' +tandoor | Deleting 'vue3/assets/fa-solid-900-D0aA9rwL.ttf.gz' +tandoor | Deleting 'vue3/assets/intersect-siLrmRQw.cc9e340f1351.js' +tandoor | Deleting 'vue3/assets/VTabs-Bz9gKX-q.27a6a10279b7.css' +tandoor | Deleting 'vue3/assets/fa-regular-400-DZaxPHgR.ttf.gz' +tandoor | Deleting 'vue3/assets/fa-v4compatibility-CCth-dXg.ttf.gz' +tandoor | Deleting 'vue3/assets/tr-OeJMQhAK.2c406e95c585.js.gz' +tandoor | Deleting 'vue3/assets/CosmeticSettings-DTudQs61.css' +tandoor | Deleting 'vue3/assets/RecipeViewPage-BY3ErEFj.css.gz' +tandoor | Deleting 'vue3/assets/VSwitch-Djsboh2v.css.gz' +tandoor | Deleting 'vue3/assets/TestPage-CPX2qaEd.js.gz' +tandoor | Deleting 'vue3/assets/model_utils-a_hMB_Qu.js.gz' +tandoor | Deleting 'vue3/assets/ar-Ct4DpnJT.js.gz' +tandoor | Deleting 'vue3/assets/DeleteConfirmDialog-C1xQIJsS.js.gz' +tandoor | Deleting 'vue3/assets/BooksPage-Dh8_wlv_.js.gz' +tandoor | Deleting 'assets/recipe_no_image.c659e2604ad6.svg.gz' +tandoor | Deleting 'assets/logo_black.dc487b18028c.svg.gz' +tandoor | Deleting 'assets/logo_color_shopping.svg.gz' +tandoor | Deleting 'assets/brand_logo.ad2e3aa263da.svg' +tandoor | Deleting 'assets/logo_color_192.c9b9177ff941.png' +tandoor | Deleting 'assets/logo_black.svg' +tandoor | Deleting 'assets/header.svg.gz' +tandoor | Deleting 'assets/brand_logo.svg.gz' +tandoor | Deleting 'assets/logo_color_144.f09537dbf142.png' +tandoor | Deleting 'assets/logo_color_128.png' +tandoor | Deleting 'assets/brand_logo.ad2e3aa263da.svg.gz' +tandoor | Deleting 'assets/logo_color_plan.4ccbb336af9d.svg.gz' +tandoor | Deleting 'assets/spinner.168a09fd2600.svg.gz' +tandoor | Deleting 'assets/logo_color_plan_144.22c6c7678fdb.png' +tandoor | Deleting 'assets/spinner.svg.gz' +tandoor | Deleting 'assets/logo_color_512.png' +tandoor | Deleting 'assets/logo_black.svg.gz' +tandoor | Deleting 'assets/logo_color_svg.d7c2e31a63b7.svg.gz' +tandoor | Deleting 'assets/recipe_no_image.c659e2604ad6.svg' +tandoor | Deleting 'assets/logo_color_plan_144.png' +tandoor | Deleting 'assets/logo_color_32.b7d9707eac91.png' +tandoor | Deleting 'assets/logo_color_512.0a28df8511e3.png' +tandoor | Deleting 'assets/brand_logo_white.svg.gz' +tandoor | Deleting 'assets/logo_black.dc487b18028c.svg' +tandoor | Deleting 'assets/logo_color_plan.svg.gz' +tandoor | Deleting 'assets/logo_color_shopping.fd1e81a4dcab.svg.gz' +tandoor | Deleting 'assets/header.48e8ad9e7540.svg' +tandoor | Deleting 'assets/logo_color_plan_512.png' +tandoor | Deleting 'assets/logo_color_shopping_512.6b2e69402327.png' +tandoor | Deleting 'assets/recipe_no_image.svg' +tandoor | Deleting 'assets/logo_color_128.a90db52449d9.png' +tandoor | Deleting 'assets/logo_color_shopping_512.png' +tandoor | Deleting 'assets/brand_logo.svg' +tandoor | Deleting 'assets/logo_color_plan.svg' +tandoor | Deleting 'assets/logo_color_32.png' +tandoor | Deleting 'assets/logo_color_144.png' +tandoor | Deleting 'assets/logo_color_shopping.fd1e81a4dcab.svg' +tandoor | Deleting 'assets/header.48e8ad9e7540.svg.gz' +tandoor | Deleting 'assets/brand_logo_white.d43daa47dca3.svg.gz' +tandoor | Deleting 'assets/logo_color_180.0fcae5e85a72.png' +tandoor | Deleting 'assets/logo_color_180.png' +tandoor | Deleting 'assets/logo_color_shopping_144.png' +tandoor | Deleting 'assets/logo_color_svg.svg' +tandoor | Deleting 'assets/brand_logo_white.d43daa47dca3.svg' +tandoor | Deleting 'assets/logo_color_plan.4ccbb336af9d.svg' +tandoor | Deleting 'assets/logo_color_plan_512.028b9bcca1eb.png' +tandoor | Deleting 'assets/logo_color_shopping.svg' +tandoor | Deleting 'assets/recipe_no_image.svg.gz' +tandoor | Deleting 'assets/header.svg' +tandoor | Deleting 'assets/logo_color_svg.svg.gz' +tandoor | Deleting 'assets/spinner.168a09fd2600.svg' +tandoor | Deleting 'assets/brand_logo.6ebe02bf6707.png' +tandoor | Deleting 'assets/spinner.svg' +tandoor | Deleting 'assets/logo_color_svg.d7c2e31a63b7.svg' +tandoor | Deleting 'assets/logo_color_shopping_144.6c05e01d07b2.png' +tandoor | Deleting 'assets/logo_color_192.png' +tandoor | Deleting 'assets/brand_logo.png' +tandoor | Deleting 'assets/brand_logo_white.svg' +tandoor | Deleting 'mfa/js/webauthn-json.6a3183313556.js' +tandoor | Deleting 'mfa/js/webauthn-json.6a3183313556.js.gz' +tandoor | Deleting 'mfa/js/webauthn.88878ff60271.js' +tandoor | Deleting 'mfa/js/webauthn.88878ff60271.js.gz' +tandoor | Deleting 'mfa/js/webauthn.js.gz' +tandoor | Deleting 'mfa/js/webauthn-json.js.gz' +tandoor | Deleting 'mfa/js/webauthn-json.js' +tandoor | Deleting 'mfa/js/webauthn.js' +tandoor | Deleting 'themes/tandoor.min.css.gz' +tandoor | Deleting 'themes/tandoor.min.css' +tandoor | Deleting 'themes/tandoor.min.e62f1984b5e4.css' +tandoor | Deleting 'themes/tandoor.min.e62f1984b5e4.css.gz' +tandoor | Deleting 'themes/tandoor_dark.min.3bdb00cbb58c.css.gz' +tandoor | Deleting 'themes/tandoor_dark.min.css' +tandoor | Deleting 'themes/tandoor_dark.min.css.gz' +tandoor | Deleting 'themes/tandoor_dark.min.3bdb00cbb58c.css' +tandoor | Deleting 'custom/css/markdown_blockquote.22de6a06ad0b.css' +tandoor | Deleting 'custom/css/markdown_blockquote.css' +tandoor | Deleting 'custom/css/markdown_blockquote.css.gz' +tandoor | Deleting 'custom/css/markdown_blockquote.22de6a06ad0b.css.gz' +tandoor | Deleting 'webfonts/fa-solid-900.8e4a6dcc692b.eot.gz' +tandoor | Deleting 'webfonts/fa-solid-900.c2801fb415f0.svg' +tandoor | Deleting 'webfonts/fa-brands-400.c5e0f14f88a8.woff' +tandoor | Deleting 'webfonts/fa-regular-400.c4f508e7c4f0.woff' +tandoor | Deleting 'webfonts/poppins_latin_ext_500.d214b888d895.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.ttf' +tandoor | Deleting 'webfonts/fa-solid-900.eot.gz' +tandoor | Deleting 'webfonts/poppins_devanagari_500.fd9b8290076b.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.a9c4bb7348f4.svg' +tandoor | Deleting 'webfonts/poppins_latin_400.9ed361bba848.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.8e4a6dcc692b.eot' +tandoor | Deleting 'webfonts/fa-brands-400.cccc9d29470e.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.0bff33a5fd7e.ttf' +tandoor | Deleting 'webfonts/fa-regular-400.woff' +tandoor | Deleting 'webfonts/fa-regular-400.f5f2566b93e8.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.c1a866ec0e04.eot.gz' +tandoor | Deleting 'webfonts/poppins_devanagari_700.b9f2c13fc361.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.eot' +tandoor | Deleting 'webfonts/fa-brands-400.06147b6cd88c.ttf.gz' +tandoor | Deleting 'webfonts/poppins_devanagari_400.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.svg' +tandoor | Deleting 'webfonts/fa-brands-400.svg.gz' +tandoor | Deleting 'webfonts/fa-regular-400.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.eot' +tandoor | Deleting 'webfonts/poppins_latin_700.woff2' +tandoor | Deleting 'webfonts/poppins_devanagari_700.woff2' +tandoor | Deleting 'webfonts/poppins_devanagari_400.d5e78c53cb07.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.c2801fb415f0.svg.gz' +tandoor | Deleting 'webfonts/fa-solid-900.svg' +tandoor | Deleting 'webfonts/poppins_latin_500.woff2' +tandoor | Deleting 'webfonts/poppins_latin_ext_400.4d1490f32451.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.eot.gz' +tandoor | Deleting 'webfonts/poppins_latin_ext_700.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.7b9568e6389b.svg.gz' +tandoor | Deleting 'webfonts/poppins_latin_400.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.c1a866ec0e04.eot' +tandoor | Deleting 'webfonts/fa-brands-400.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.svg.gz' +tandoor | Deleting 'webfonts/fa-brands-400.a9c4bb7348f4.svg.gz' +tandoor | Deleting 'webfonts/fa-brands-400.5063b105c764.eot.gz' +tandoor | Deleting 'webfonts/fa-regular-400.7b9568e6389b.svg' +tandoor | Deleting 'webfonts/poppins_latin_ext_500.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.woff' +tandoor | Deleting 'webfonts/fa-brands-400.ttf.gz' +tandoor | Deleting 'webfonts/fa-solid-900.44d537ab79f9.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.svg.gz' +tandoor | Deleting 'webfonts/fa-solid-900.eot' +tandoor | Deleting 'webfonts/fa-brands-400.svg' +tandoor | Deleting 'webfonts/poppins_latin_500.84780596e268.woff2' +tandoor | Deleting 'webfonts/fa-brands-400.06147b6cd88c.ttf' +tandoor | Deleting 'webfonts/fa-solid-900.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.333bae208dc3.woff' +tandoor | Deleting 'webfonts/poppins_latin_ext_400.woff2' +tandoor | Deleting 'webfonts/fa-regular-400.65b286af947c.ttf.gz' +tandoor | Deleting 'webfonts/fa-solid-900.ttf.gz' +tandoor | Deleting 'webfonts/fa-brands-400.5063b105c764.eot' +tandoor | Deleting 'webfonts/fa-regular-400.ttf.gz' +tandoor | Deleting 'webfonts/poppins_latin_ext_700.62d7a4c78fc2.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.woff' +tandoor | Deleting 'webfonts/poppins_devanagari_500.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.0bff33a5fd7e.ttf.gz' +tandoor | Deleting 'webfonts/fa-regular-400.eot.gz' +tandoor | Deleting 'webfonts/fa-regular-400.ttf' +tandoor | Deleting 'webfonts/fa-regular-400.65b286af947c.ttf' +tandoor | Deleting 'webfonts/poppins_latin_700.f4f17fd53c7d.woff2' +tandoor | Deleting 'webfonts/fa-solid-900.ttf' +tandoor | 2026/09/22 10:54:37 [crit] 16#16: *9 connect() to unix:/tmp/tandoor.sock failed (2: No such file or directory) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /accounts/login/ HTTP/1.1", upstream: "http://unix:/tmp/tandoor.sock:/accounts/login/", host: "127.0.0.1:80" +tandoor | 127.0.0.1 - - [22/Sep/2026:10:54:37 +0200] "GET /accounts/login/ HTTP/1.1" 502 150 "-" "Wget" "-" +tandoor | +tandoor | 894 static files copied to '/opt/recipes/staticfiles', 2464 post-processed. +tandoor | Done +tandoor | Starting gunicorn +tandoor | [2026-09-22 10:54:37 +0200] [7] [INFO] Starting gunicorn 26.2.0 +tandoor | [2026-09-22 10:54:37 +0200] [7] [INFO] Listening at: unix:/tmp/tandoor.sock (7) +tandoor | [2026-09-22 10:54:37 +0200] [7] [INFO] Using worker: gthread +tandoor | [2026-09-22 10:54:37 +0200] [66] [INFO] Booting worker with pid: 66 +tandoor | [2026-09-22 10:54:37 +0200] [67] [INFO] Booting worker with pid: 67 +tandoor | [2026-09-22 10:54:38 +0200] [68] [INFO] Booting worker with pid: 68 +tandoor | [2026-09-22 10:54:38 +0200] [7] [INFO] Control socket listening at /root/.gunicorn/gunicorn.ctl +tandoor | Running django-vite in production mode (no HMR) +tandoor | Running django-vite in production mode (no HMR) +tandoor | Running django-vite in production mode (no HMR) +tandoor | 127.0.0.1 - - [22/Sep/2026:10:54:45 +0200] "GET /accounts/login/ HTTP/1.1" 200 4981 "-" "Wget" "-" +tandoor | - - [22/Sep/2026:10:54:45 +0200] "GET /accounts/login/ HTTP/1.0" 200 4981 "-" "Wget" +tandoor | ThreadPoolExecutor-0_0 ERROR 2026-09-22 10:54:47,243 django.security.DisallowedHost Invalid HTTP_HOST header: 'tandoor:80'. You may need to add 'tandoor' to ALLOWED_HOSTS. +tandoor | Traceback (most recent call last): +tandoor | File "/opt/recipes/venv/lib/python3.13/site-packages/django/core/handlers/exception.py", line 55, in inner +tandoor | response = get_response(request) +tandoor | File "/opt/recipes/venv/lib/python3.13/site-packages/django/utils/deprecation.py", line 119, in __call__ +tandoor | response = self.process_request(request) +tandoor | File "/opt/recipes/venv/lib/python3.13/site-packages/django/middleware/common.py", line 48, in process_request +tandoor | host = request.get_host() +tandoor | File "/opt/recipes/venv/lib/python3.13/site-packages/django/http/request.py", line 203, in get_host +tandoor | raise DisallowedHost(msg) +tandoor | django.core.exceptions.DisallowedHost: Invalid HTTP_HOST header: 'tandoor:80'. You may need to add 'tandoor' to ALLOWED_HOSTS. +tandoor | - - [22/Sep/2026:10:54:47 +0200] "GET /accounts/login/ HTTP/1.0" 400 143 "-" "Go-http-client/1.1" +tandoor | 172.18.0.2 - - [22/Sep/2026:10:54:47 +0200] "GET /accounts/login/ HTTP/1.1" 400 154 "-" "Go-http-client/1.1" "-" diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor/badges.json b/documentation/audits/update-night-2026-09-21/apps/tandoor/badges.json index 1a6d711c..52f6f405 100644 --- a/documentation/audits/update-night-2026-09-21/apps/tandoor/badges.json +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor/badges.json @@ -1,17 +1,7 @@ { "before": { - "hu": [ - { - "title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", - "text": "Naprakész" - } - ], - "en": [ - { - "title": "This app is running the newest version available.", - "text": "Up to date" - } - ] + "hu": [], + "en": [] }, "after": { "hu": [ @@ -27,5 +17,5 @@ } ] }, - "drill_commit": "c7477d1fb921" + "drill_commit": "654a28192b38" } \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor/log.txt b/documentation/audits/update-night-2026-09-21/apps/tandoor/log.txt index b6074544..ef9af077 100644 --- a/documentation/audits/update-night-2026-09-21/apps/tandoor/log.txt +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor/log.txt @@ -1,25 +1,25 @@ -21:15:25 ==== tandoor: ghcr.io/tandoorrecipes/recipes:2.6.13 -> ghcr.io/tandoorrecipes/recipes:2.6.15 (sub=recipes, class=db-postgres) -21:15:25 [1] tandoor already deployed — reusing -21:15:27 [2] seeding through the app's own front door -21:15:37 tandoor: createsuperuser :: Running django-vite in production mode (no HMR) Superuser created successfully. -21:15:40 tandoor: seeded superuser drill316903 -21:15:40 [3] control C1 — reading the seed back BEFORE the update -21:15:46 tandoor: readback of the seeded account found=True -21:15:46 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} -21:16:42 [4] backup idle; last=None -21:16:43 [5] drill commit c7477d1fb921: tandoor ghcr.io/tandoorrecipes/recipes:2.6.13 -> ghcr.io/tandoorrecipes/recipes:2.6.15 (push rc=0) -21:16:47 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] -21:16:47 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] -21:16:47 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} -21:16:47 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None -21:16:48 + 1.0s phase=pulling label=Új verzió letöltése… err=None hold=None -21:17:42 + 54.3s phase=starting label=Indítás az új verzióval… err=None hold=None -21:17:46 + 58.4s phase=verifying label=Működés ellenőrzése… err=None hold=None -21:22:49 + 361.9s phase=failed label=A frissítés nem sikerült err=A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza. hold=A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza. -21:22:49 [7] reading the seed back AFTER the update -21:22:52 tandoor: READBACK UNUSABLE — a username that cannot exist did not read as absent (None) :: Error response from daemon: No such container: tandoor -21:22:54 [8] pinned = {'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:16-alpine'} -21:22:54 [8] installed = {'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.13', 'tandoor-postgres': 'postgres:16-alpine'} -21:22:54 [8] compose = ['image: ghcr.io/tandoorrecipes/recipes:2.6.15', 'image: postgres:16-alpine'] -21:22:54 [8] inspect = [] -21:22:54 [9] verdict failed -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/tandoor/verdict.json +10:52:42 ==== tandoor: ghcr.io/tandoorrecipes/recipes:2.6.13 -> ghcr.io/tandoorrecipes/recipes:2.6.15 (sub=recipes, class=db) +10:52:42 [1] tandoor already deployed — reusing +10:52:45 [2] seeding through the app's own front door +10:52:55 tandoor: createsuperuser :: Running django-vite in production mode (no HMR) Superuser created successfully. +10:52:59 tandoor: seeded superuser drill1d392f +10:52:59 [3] control C1 — reading the seed back BEFORE the update +10:53:05 tandoor: readback of the seeded account found=True +10:53:05 [4] „Mentés most" -> 200 {'ok': True, 'message': 'Mentés elindítva'} +10:54:00 [4] backup idle; last=None +10:54:01 [5] drill commit 654a28192b38: tandoor ghcr.io/tandoorrecipes/recipes:2.6.13 -> ghcr.io/tandoorrecipes/recipes:2.6.15 (push rc=0) +10:54:06 [5] badge HU: [{'title': 'Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.', 'text': 'Frissítés elérhető — ma'}] +10:54:06 [5] badge EN: [{'title': 'A newer version of this app is available. Select the Update button to start it.', 'text': 'Update available — today'}] +10:54:06 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'} +10:54:06 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None +10:54:07 + 1.0s phase=pulling label=Új verzió letöltése… err=None hold=None +10:54:08 + 2.1s phase=starting label=Indítás az új verzióval… err=None hold=None +10:54:11 + 5.2s phase=verifying label=Működés ellenőrzése… err=None hold=None +10:54:47 + 41.1s phase=done label=Frissítve err=None hold=None +10:54:47 [7] reading the seed back AFTER the update +10:54:54 tandoor: readback of the seeded account found=True +10:54:56 [8] pinned = {'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:16-alpine'} +10:54:56 [8] installed = {'tandoor': 'ghcr.io/tandoorrecipes/recipes:2.6.15', 'tandoor-postgres': 'postgres:16-alpine'} +10:54:56 [8] compose = ['image: ghcr.io/tandoorrecipes/recipes:2.6.15', 'image: postgres:16-alpine'] +10:54:56 [8] inspect = ['tandoor ghcr.io/tandoorrecipes/recipes:2.6.15 running=true restarts=0', 'tandoor-postgres postgres:16-alpine running=true restarts=0'] +10:54:56 [9] verdict proven -> /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/update-night-2026-09-21/apps/tandoor/verdict.json diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor/observables-before.json b/documentation/audits/update-night-2026-09-21/apps/tandoor/observables-before.json index be655443..80ee28be 100644 --- a/documentation/audits/update-night-2026-09-21/apps/tandoor/observables-before.json +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor/observables-before.json @@ -7,10 +7,7 @@ "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", "tandoor-postgres": "postgres:16-alpine" }, - "catalog_images": { - "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", - "tandoor-postgres": "postgres:16-alpine" - }, + "catalog_images": null, "live_compose_image_lines": [ "image: ghcr.io/tandoorrecipes/recipes:2.6.13", "image: postgres:16-alpine" diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor/observables.json b/documentation/audits/update-night-2026-09-21/apps/tandoor/observables.json index 19e1cc15..5d9632cf 100644 --- a/documentation/audits/update-night-2026-09-21/apps/tandoor/observables.json +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor/observables.json @@ -8,10 +8,7 @@ "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", "tandoor-postgres": "postgres:16-alpine" }, - "catalog_images": { - "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", - "tandoor-postgres": "postgres:16-alpine" - }, + "catalog_images": null, "live_compose_image_lines": [ "image: ghcr.io/tandoorrecipes/recipes:2.6.13", "image: postgres:16-alpine" @@ -27,7 +24,7 @@ "tandoor-postgres": "postgres:16-alpine" }, "installed_images": { - "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", "tandoor-postgres": "postgres:16-alpine" }, "catalog_images": { @@ -38,6 +35,9 @@ "image: ghcr.io/tandoorrecipes/recipes:2.6.15", "image: postgres:16-alpine" ], - "docker_inspect": [] + "docker_inspect": [ + "tandoor ghcr.io/tandoorrecipes/recipes:2.6.15 running=true restarts=0", + "tandoor-postgres postgres:16-alpine running=true restarts=0" + ] } } \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor/phases.json b/documentation/audits/update-night-2026-09-21/apps/tandoor/phases.json index 67317e2d..f4206717 100644 --- a/documentation/audits/update-night-2026-09-21/apps/tandoor/phases.json +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor/phases.json @@ -19,7 +19,7 @@ "hold": null }, { - "t": 54.3, + "t": 2.1, "phase": "starting", "label": "Indítás az új verzióval…", "updating": true, @@ -27,7 +27,7 @@ "hold": null }, { - "t": 58.4, + "t": 5.2, "phase": "verifying", "label": "Működés ellenőrzése…", "updating": true, @@ -35,17 +35,17 @@ "hold": null }, { - "t": 361.9, - "phase": "failed", - "label": "A frissítés nem sikerült", + "t": 41.1, + "phase": "done", + "label": "Frissítve", "updating": false, - "error": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", - "hold": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza." + "error": null, + "hold": null } ], - "duration_s": 361.9, - "final_phase": "failed", - "update_error": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", - "hold_reason": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", - "state": "stopped" + "duration_s": 41.1, + "final_phase": "done", + "update_error": null, + "hold_reason": null, + "state": "running" } \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/apps/tandoor/verdict.json b/documentation/audits/update-night-2026-09-21/apps/tandoor/verdict.json index f7497e30..ce4c472d 100644 --- a/documentation/audits/update-night-2026-09-21/apps/tandoor/verdict.json +++ b/documentation/audits/update-night-2026-09-21/apps/tandoor/verdict.json @@ -2,7 +2,7 @@ "harness_version": 1, "app": "tandoor", "venue": "guest 9202 demo-hp-scratch, controller 0.261.0", - "class": "db-postgres", + "class": "db", "from": { "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", "tandoor-postgres": "postgres:16-alpine" @@ -11,26 +11,25 @@ "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", "tandoor-postgres": "postgres:16-alpine" }, - "verdict": "failed", + "verdict": "proven", "seed_read_before": true, - "seed_read_after": false, - "healthy_after": false, + "seed_read_after": true, + "healthy_after": true, "migration_observed": null, "abort": "not-attempted", "abort_detail": null, - "duration_s": 361.9, - "measured_at": "2026-09-21T19:15:25.308811+00:00", + "duration_s": 41.1, + "measured_at": "2026-09-22T08:52:42.724401+00:00", "evidence": "apps/tandoor/", - "notes": [ - "the edge ended HELD or failed — this is a RESULT, not an error of the run" - ], + "notes": [], + "badge_catchup_seconds": 4.5, "observables_after": { "pinned_images": { "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", "tandoor-postgres": "postgres:16-alpine" }, "installed_images": { - "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.13", + "tandoor": "ghcr.io/tandoorrecipes/recipes:2.6.15", "tandoor-postgres": "postgres:16-alpine" }, "catalog_images": { @@ -41,9 +40,12 @@ "image: ghcr.io/tandoorrecipes/recipes:2.6.15", "image: postgres:16-alpine" ], - "docker_inspect": [] + "docker_inspect": [ + "tandoor ghcr.io/tandoorrecipes/recipes:2.6.15 running=true restarts=0", + "tandoor-postgres postgres:16-alpine running=true restarts=0" + ] }, - "final_phase": "failed", - "hold_reason": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza.", - "update_error": "A(z) tandoor frissítése 2026-09-21 21:22-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-21 21:15 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza." + "final_phase": "done", + "hold_reason": null, + "update_error": null } \ No newline at end of file diff --git a/documentation/audits/update-night-2026-09-21/probe_redproof.py b/documentation/audits/update-night-2026-09-21/probe_redproof.py new file mode 100644 index 00000000..41b551ed --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/probe_redproof.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +"""Red-proof for R-618, live on 9202 through the product. + +The claim: three templates name a health probe the app does not answer. The proof must show BOTH +halves at the same moment, or it shows nothing: + (a) the PRODUCT says the app is unhealthy — the controller's state, and the app page's badge; + (b) the APP is fine — its own front door answers 200, and docker's healthcheck reads healthy. +A reading of (a) alone is equally consistent with "the app really is broken", which is exactly the +mistake this whole exercise exists to stop making. +""" +import json, re, sys, time +sys.path.insert(0, '.') +import walk as w + +OUT = "../probe-fix-2026-09-22" +APPS = [("tandoor", "tandoor", "/accounts/login/"), + ("zipline", "zipline", "/"), + ("wger", "wger", "/")] + + +def observe(name, sub, path, tag): + st = w.stack(name) + ac = st.get("app_config") or {} + rc, code, body = w.app_curl(sub, path) + dk = w.guest(f"docker inspect --format '{{{{.State.Health.Status}}}}' " + f"$(docker ps -q --filter name={name}) 2>&1 | head -3").strip() + bg = w.badges(name) + # The health word the HOUSEHOLD reads, in both languages — the version badge is a different + # element and quoting it instead would prove nothing about health (it says "Naprakesz" either + # way). Taken from the rendered page, scripts and styles stripped first: a `|", "", h, flags=re.S) + txt = re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", h)) + m = re.search(r"\u2190 Alkalmaz\u00e1sok .{0,80}?\s(Nem eg\u00e9szs\u00e9ges|Eg\u00e9szs\u00e9ges|Fut|Le\u00e1ll\u00edtva|Indul|Hib\u00e1s)\s", txt) or \ + re.search(r"\u2190 (?:Apps|Alkalmaz\u00e1sok) .{0,80}?\s(Not healthy|Healthy|Running|Stopped|Starting|Failed)\s", txt) + health[lang] = m.group(1) if m else None + health[lang + "_context"] = txt[txt.find("\u2190"):txt.find("\u2190") + 220] if "\u2190" in txt else None + rec = {"when": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "tag": tag, "app": name, + "controller_state": st.get("state"), "deployed": st.get("deployed"), + "pinned_images": ac.get("pinned_images"), "installed_images": ac.get("installed_images"), + "front_door_rc": rc, "front_door_code": code, "front_door_bytes": len(body), + "docker_health": dk, "version_badge": bg, "health_word": health} + print(json.dumps(rec, ensure_ascii=False, indent=2)) + return rec + + +if __name__ == "__main__": + w.login() + tag = sys.argv[1] if len(sys.argv) > 1 else "before" + only = sys.argv[2:] or [a[0] for a in APPS] + recs = [] + for name, sub, path in APPS: + if name not in only: + continue + print(f"\n=== {name} ({tag})") + if tag == "before": + ok = w.deploy(name, sub) + print(f" deploy ok={ok}") + w.wait_app(sub, path) + recs.append(observe(name, sub, path, tag)) + open(f"{OUT}/probe-{tag}.json", "w").write(json.dumps(recs, ensure_ascii=False, indent=2)) + print(f"\nwritten {OUT}/probe-{tag}.json") diff --git a/documentation/audits/update-night-2026-09-21/repoint_drill.py b/documentation/audits/update-night-2026-09-21/repoint_drill.py new file mode 100644 index 00000000..dbe65fcb --- /dev/null +++ b/documentation/audits/update-night-2026-09-21/repoint_drill.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +"""Point guest 9202 at the drill catalog, or back at the live one (09 §6.5, R-615). + +`git.repo_url` alone is INERT: `Syncer.gitCloneOrPull` clones only when the cache has no `.git`, +otherwise it fetches from the remote the existing clone stores. The cache directory must be removed +too, or the box goes on following the live catalog AND REPORTS SUCCESS. That is R-615, and it is why +step 3 exists. +""" +import re, sys, io +sys.path.insert(0, '.') +import walk as w + +VOL = "/var/lib/docker/volumes/felhom-controller-data/_data" +LIVE = "https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git" +DRILL_REPO = "https://gitea.dooplex.hu/admin/app-catalog-drill.git" + + +def creds(): + for l in io.open("/home/kisfenyo/.git-credentials").read().strip().split("\n"): + m = re.match(r'https://(admin):([^@]+)@gitea\.dooplex\.hu', l) + if m: + return m.group(1), m.group(2) + raise SystemExit("no admin credential for gitea.dooplex.hu") + + +def repoint(to_drill): + u, t = creds() if to_drill else ("", "") + url = DRILL_REPO if to_drill else LIVE + script = f""" +set -e +test -f {VOL}/controller.yaml.pre-update-night || cp {VOL}/controller.yaml {VOL}/controller.yaml.pre-update-night +python3 - <<'EOF' +import re +p = "{VOL}/controller.yaml" +s = open(p).read() +s = re.sub(r'(^\\s+repo_url: ).*$', r'\\g<1>{url}', s, count=1, flags=re.M) +s = re.sub(r'(^\\s+token: ).*$', r'\\g<1>"{t}"', s, count=1, flags=re.M) +s = re.sub(r'(^\\s+username: ).*$', r'\\g<1>"{u}"', s, count=1, flags=re.M) +open(p, "w").write(s) +EOF +# R-615: the cache is the thing that actually decides which remote is fetched. +rm -rf {VOL}/catalog-cache {VOL}/data/catalog-cache +docker restart felhom-controller >/dev/null +sleep 12 +grep -A6 '^git:' {VOL}/controller.yaml | sed 's/token:.*/token: /' +""" + print(w.guest(script)) + + +if __name__ == "__main__": + repoint(sys.argv[1] == "drill") diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 6ef4d19e..44fd1326 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -787,7 +787,7 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-615** | **[P3-LOW] Pointing a box at a different app catalog by `git.repo_url` alone is INERT — the box keeps fetching from the repository it first cloned.** FOUND 2026-09-21 by reading `sync.go` **before** running it, which is the only reason the update night's drill catalog worked at all. `Syncer.gitCloneOrPull` (`controller/internal/sync/sync.go:274-306`) clones **only when `/catalog-cache/.git` is absent**; on every later cycle it runs `git fetch --depth 1 origin ` + `git reset --hard origin/` **against the remote stored in the clone**, which `buildRepoURL` wrote at clone time. Changing `git.repo_url` in `controller.yaml` and restarting therefore changes **nothing**: the sync keeps pulling the old catalog and reports success. Measured: after the repoint, `git -C /catalog-cache remote -v` still read `app-catalog-felhom.eu`; the box only followed the drill repo once the cache directory was removed. **Why it matters beyond a drill:** this is the one knob that would move a box to a different or a staged catalog — for a migration, a per-customer catalog, or a rollback of the catalog itself — and it silently does not work. **Nothing is wrong with the CACHING**, which is right; what is missing is that a changed `repo_url` must invalidate the clone. **Fix shape:** on start, compare `git.repo_url` with the clone's `origin` and re-clone when they differ (or `git remote set-url` + a full fetch); log which happened. A test that changes `repo_url` under an existing cache and asserts the next sync reads the NEW repo — it fails today. Evidence: `audits/update-night-2026-09-21/04-9202-config-pre.txt`, `05-9202-follows-drill.txt`. | **READY — rank P3-LOW; owner: CC (controller)** | | **R-616** | **[P3-LOW] The catalog credentials are stored in PLAINTEXT in the box's catalog clone and are printed by an ordinary `git remote -v`.** FOUND 2026-09-21 on guest 9202 while pointing it at a private drill catalog. `Syncer.buildRepoURL` injects `username:token` into the HTTPS URL, and `git clone` persists that URL as the clone's `origin`, so `/catalog-cache/.git/config` holds the token in the clear and **any** diagnostic that prints the remote leaks it — which is what happened in this session's own transcript, and is the same shape as R-580 (`curl -w '%{redirect_url}'`). `maskRepoURL` exists and is used for the LOG lines, so the masking intent is already there; the stored remote is the half that was missed. **INERT ON THE FLEET TODAY** — the live catalog is public and `git.token` is empty on every real box — which is exactly why it should be fixed before it is not: the day the catalog goes private, every box carries a readable credential and every support session that runs `git remote -v` prints it. **Fix shape:** store the remote WITHOUT credentials and supply them per-fetch (a credential helper, `http.extraHeader`, or `GIT_ASKPASS`), and a test asserting the clone's stored `origin` contains no `@`. **Operator action from tonight, unrelated to the fix:** the Gitea `admin` token used for the drill repo was printed by that command and must be rotated. Evidence: `audits/update-night-2026-09-21/05-9202-follows-drill.txt` (redacted). | **READY — rank P3-LOW; owner: CC (controller); one operator action (rotate the Gitea admin token)** | | **R-617** | **[P3-LOW] The Gitea API token this project uses for pushes cannot create a repository through the documented endpoint, but CAN through `repos/migrate` — so "the token cannot do it" was nearly recorded as a fact when the truth was "one endpoint refuses it".** FOUND 2026-09-21 creating the drill catalog. Both `~/.git-credentials` tokens carry `write:misc,write:notification,write:package,write:issue,write:repository`; `POST /api/v1/user/repos` requires `write:user` and answers **403**, and `POST /api/v1/admin/users//repos` requires `write:admin` and answers 403 too. `POST /api/v1/repos/migrate` with the same token answered **201** and created the private repository. **Why this is a row and not a note:** a session that stopped at the first 403 would have recorded "CC cannot create a Gitea repository" — an unfalsifiable capability claim of exactly the shape the workspace's standing rule 2 forbids — and every later drill would have been designed around a limit that does not exist. **What it needs:** one line in the operations notes saying which endpoint to use, and (optional, operator) a token scoped for the job so the migrate route is not load-bearing. Evidence: `audits/update-night-2026-09-21/03-drill-repo.txt`. | **READY — rank P3-LOW; owner: CC (docs)** | -| **R-618** | **[P1-HIGH] THREE apps are presented to the household as UNHEALTHY while they are working perfectly — and because the guarded Update waits on that same probe, a SUCCESSFUL update ends by STOPPING the working app and sending the household to a restore they do not need.** **RANK RAISED FROM P2 TO P1 BY A LIVE MEASUREMENT taken the same night, and the escalation is the whole point:** tandoor's Update 2.6.13 → 2.6.15 was pressed at 21:16:47 and entered `verifying` at 21:17:46. At **21:18:28** the NEW version was `Up 25 seconds` and answering **HTTP 200** on `/accounts/login/` through the household's own front door — while the controller, probing port 8080 where nothing listens, could not see it. `verifying` therefore cannot pass, the full `update.health_timeout` is spent, `Manager.failAndHold` runs `compose down`, and the app is STOPPED. **Nothing is lost** — the data is in the volumes and the restore works — **but one wrong port number in a template converts every successful update of that app into an outage plus an unnecessary restore, for every household running it.** Evidence: `audits/update-night-2026-09-21/14-tandoor-serving-while-verifying.txt`. MEASURED 2026-09-21 on guest 9202 (controller v0.261.0, catalog `f5f6a152b513`). Two shapes, one class: **(a) `tandoor` — the WRONG PORT.** `.felhom.yml` probes `port: 8080`; the container listens on **80 and nothing else** (`ss -ltn` inside it), the compose's own traefik label routes to 80, its own docker healthcheck reads `healthy`, and `/accounts/login/` answers **200** through the household's real front door. `GET /api/stacks/tandoor` nevertheless reads `state: "unhealthy"`. **(b) `zipline` — the WRONG PATH.** `.felhom.yml` probes `/api/health`, which zipline 4.6.1 answers **404 `Route GET:/api/health not found`**; **the compose healthcheck in the very same file uses `/api/healthcheck` and is correct and green.** `/dashboard` answers 200. The controller reads `unhealthy`. **This is the MIRROR of R-613** — that is a probe that passes on a broken app (a false GREEN, which no alarm catches); this is a probe that fails on a working app (a false RED). **IT DOES NOT ALARM, AND THAT SETS THE RANK:** `08-alarm-ladder.md` §4 puts `unhealthy` deliberately in the NOT-down set, so no dead-app event and no customer mail follows — the damage is what the household READS, plus anything that gates on `state`. **IT ALREADY COST A MEASUREMENT TONIGHT:** this drill's harness waited for `state == "running"` and hung for its full budget on tandoor, an app that was up the whole time. An instrument waiting for a wrong answer looks exactly like a slow app. **THE GATE THIS WANTS IS CHEAP AND STATIC, AND THAT IS THE FINDING'S REAL VALUE.** Both halves of the answer live in the same template: compare the `.felhom.yml` probe's port and path against the compose's **own** `healthcheck: test:` URL. A sweep of all 53 templates on that rule was run tonight and returns **five** disagreements: `tandoor` (PORT — **CONFIRMED live**), `zipline` (PATH — **CONFIRMED live**), `wger` (PORT, probe 80 vs compose 8000 — **CONFIRMED live the same night**), `home-assistant` (PATH, `/api/` vs `/manifest.json` — **NOT MEASURED**), and `adventurelog` (a FALSE POSITIVE of the sweep's own regex — it reads `running` live). **So the rule finds both real defects, with two candidates and one false positive out of 53** — a good enough signal for a fast gate, provided it reports candidates rather than convictions and a person or a runtime check resolves them. The earlier, cruder rule (probe port vs the *traefik* port) is strictly worse: it clears zipline and convicts adventurelog. **AND A SECOND FIX SHAPE, ON THE CONTROLLER SIDE, WORTH CONSIDERING BESIDE THE CATALOG ONE:** in both confirmed cases the container's OWN docker healthcheck was **green** the whole time. A `verifying` phase that is about to stop a working app could ask that too — if the compose declares a healthcheck and docker reports `healthy`, the app is alive whatever our probe thinks. That does not excuse a wrong probe, but it turns this failure direction from an outage into a wrong label. It is a design question, not a defect, and is raised here rather than decided. **Needs:** fix tandoor's port (80), zipline's path (`/api/healthcheck`) and wger's port (8000) — **all three are now CONFIRMED live, none is a guess**; add the static gate with a decoy each way (R-421) — a template whose probe agrees must not read as a disagreement, and vice versa. **THE GATE'S RULE WAS THEN SHARPENED BY READING `healthprobe.go` RATHER THAN ASSUMING IT, and the sharpening REMOVED a false conviction.** `type: http` treats **any** response as healthy (`healthprobe.go:258-261`), and `type: api` with **no** `expect` block does the same (`:265-268`); only `type: api` WITH `expect.status` cares about the path or the code. So a PATH difference is a candidate only for the third shape, while a PORT difference is a candidate for all of them. Under that rule the 53-template sweep returns **four** candidates — `tandoor`, `zipline` and **`wger` (all three CONFIRMED live — probe `type: http, port: 80`; inside the container port 80 is `refused` and port 8000 `ANSWERED`; docker's own healthcheck green; front door 302; the box reads `unhealthy`)**, and `adventurelog` (a false positive: its compose lists two containers' ports and the probe targets the backend; measured `running`). **`home-assistant` is correctly CLEARED by the sharpened rule** — `type: api`, no `expect`, so its `/api/` answering 401 without a token is healthy, and its edge was PROVEN on the box tonight. The crude rule convicted it; the rule read from the code does not. **That is the gate to build: two of 53 convicted, one suspected, one false positive, and the false positive is resolvable by one live check.** Evidence: `audits/update-night-2026-09-21/10-probe-port-sweep.txt`, `12-probe-vs-compose-healthcheck.txt` and `13-probe-sweep-sharpened.txt`. | **READY — rank P1-HIGH; owner: CC (catalog)** | +| **R-618** | **[P1-HIGH] THREE apps are presented to the household as UNHEALTHY while they are working perfectly — and because the guarded Update waits on that same probe, a SUCCESSFUL update ends by STOPPING the working app and sending the household to a restore they do not need.** **RANK RAISED FROM P2 TO P1 BY A LIVE MEASUREMENT taken the same night, and the escalation is the whole point:** tandoor's Update 2.6.13 → 2.6.15 was pressed at 21:16:47 and entered `verifying` at 21:17:46. At **21:18:28** the NEW version was `Up 25 seconds` and answering **HTTP 200** on `/accounts/login/` through the household's own front door — while the controller, probing port 8080 where nothing listens, could not see it. `verifying` therefore cannot pass, the full `update.health_timeout` is spent, `Manager.failAndHold` runs `compose down`, and the app is STOPPED. **Nothing is lost** — the data is in the volumes and the restore works — **but one wrong port number in a template converts every successful update of that app into an outage plus an unnecessary restore, for every household running it.** Evidence: `audits/update-night-2026-09-21/14-tandoor-serving-while-verifying.txt`. MEASURED 2026-09-21 on guest 9202 (controller v0.261.0, catalog `f5f6a152b513`). Two shapes, one class: **(a) `tandoor` — the WRONG PORT.** `.felhom.yml` probes `port: 8080`; the container listens on **80 and nothing else** (`ss -ltn` inside it), the compose's own traefik label routes to 80, its own docker healthcheck reads `healthy`, and `/accounts/login/` answers **200** through the household's real front door. `GET /api/stacks/tandoor` nevertheless reads `state: "unhealthy"`. **(b) `zipline` — the WRONG PATH.** `.felhom.yml` probes `/api/health`, which zipline 4.6.1 answers **404 `Route GET:/api/health not found`**; **the compose healthcheck in the very same file uses `/api/healthcheck` and is correct and green.** `/dashboard` answers 200. The controller reads `unhealthy`. **This is the MIRROR of R-613** — that is a probe that passes on a broken app (a false GREEN, which no alarm catches); this is a probe that fails on a working app (a false RED). **IT DOES NOT ALARM, AND THAT SETS THE RANK:** `08-alarm-ladder.md` §4 puts `unhealthy` deliberately in the NOT-down set, so no dead-app event and no customer mail follows — the damage is what the household READS, plus anything that gates on `state`. **IT ALREADY COST A MEASUREMENT TONIGHT:** this drill's harness waited for `state == "running"` and hung for its full budget on tandoor, an app that was up the whole time. An instrument waiting for a wrong answer looks exactly like a slow app. **THE GATE THIS WANTS IS CHEAP AND STATIC, AND THAT IS THE FINDING'S REAL VALUE.** Both halves of the answer live in the same template: compare the `.felhom.yml` probe's port and path against the compose's **own** `healthcheck: test:` URL. A sweep of all 53 templates on that rule was run tonight and returns **five** disagreements: `tandoor` (PORT — **CONFIRMED live**), `zipline` (PATH — **CONFIRMED live**), `wger` (PORT, probe 80 vs compose 8000 — **CONFIRMED live the same night**), `home-assistant` (PATH, `/api/` vs `/manifest.json` — **NOT MEASURED**), and `adventurelog` (a FALSE POSITIVE of the sweep's own regex — it reads `running` live). **So the rule finds both real defects, with two candidates and one false positive out of 53** — a good enough signal for a fast gate, provided it reports candidates rather than convictions and a person or a runtime check resolves them. The earlier, cruder rule (probe port vs the *traefik* port) is strictly worse: it clears zipline and convicts adventurelog. **AND A SECOND FIX SHAPE, ON THE CONTROLLER SIDE, WORTH CONSIDERING BESIDE THE CATALOG ONE:** in both confirmed cases the container's OWN docker healthcheck was **green** the whole time. A `verifying` phase that is about to stop a working app could ask that too — if the compose declares a healthcheck and docker reports `healthy`, the app is alive whatever our probe thinks. That does not excuse a wrong probe, but it turns this failure direction from an outage into a wrong label. It is a design question, not a defect, and is raised here rather than decided. **Needs:** fix tandoor's port (80), zipline's path (`/api/healthcheck`) and wger's port (8000) — **all three are now CONFIRMED live, none is a guess**; add the static gate with a decoy each way (R-421) — a template whose probe agrees must not read as a disagreement, and vice versa. **THE GATE'S RULE WAS THEN SHARPENED BY READING `healthprobe.go` RATHER THAN ASSUMING IT, and the sharpening REMOVED a false conviction.** `type: http` treats **any** response as healthy (`healthprobe.go:258-261`), and `type: api` with **no** `expect` block does the same (`:265-268`); only `type: api` WITH `expect.status` cares about the path or the code. So a PATH difference is a candidate only for the third shape, while a PORT difference is a candidate for all of them. Under that rule the 53-template sweep returns **four** candidates — `tandoor`, `zipline` and **`wger` (all three CONFIRMED live — probe `type: http, port: 80`; inside the container port 80 is `refused` and port 8000 `ANSWERED`; docker's own healthcheck green; front door 302; the box reads `unhealthy`)**, and `adventurelog` (a false positive: its compose lists two containers' ports and the probe targets the backend; measured `running`). **`home-assistant` is correctly CLEARED by the sharpened rule** — `type: api`, no `expect`, so its `/api/` answering 401 without a token is healthy, and its edge was PROVEN on the box tonight. The crude rule convicted it; the rule read from the code does not. **That is the gate to build: two of 53 convicted, one suspected, one false positive, and the false positive is resolvable by one live check.** Evidence: `audits/update-night-2026-09-21/10-probe-port-sweep.txt`, `12-probe-vs-compose-healthcheck.txt` and `13-probe-sweep-sharpened.txt`. **CLOSED 2026-09-22.** All three fixed in one commit (`app-catalog-felhom.eu@793c4fb`): tandoor `8080 -> 80`, wger `80 -> 8000`, zipline `/api/health -> /api/healthcheck`. No `image:` line moved, so no `catalog_since` moved. **RED-PROOFED LIVE ON 9202 THROUGH THE PRODUCT, BOTH DIRECTIONS.** Before the fix, at the LIVE pin, all three read **`Nem egészséges` / `Not healthy`** on their own app page while docker reported every container healthy and each front door served a real page through the household's own route — tandoor 200 `Login / Sign In`, zipline 200 `Zipline`, wger 200 `wger Workout Manager`. The fix was applied through the REAL sync (`POST /api/sync` answered *frissítve: tandoor, wger, zipline*) and all three read **`Fut` / `Running`** at the next poll, with no redeploy and no restart. **AND THE EDGE THAT FAILED WAS RE-WALKED AND PASSED:** tandoor `2.6.13 -> 2.6.15` via the drill catalog ended **`done` at +41.1 s** with the seed read back through tandoor's own front door and both containers running with zero restarts — where the identical edge on 2026-09-21 entered `verifying` at +58.4 s and ended `failed` at **+361.9 s** with the app stopped. **Same app, same versions, same button; the only change is one port number.** tandoor's verdict moved `failed -> proven` and it is now on the live catalog. **THE GATE SHIPPED WITH IT:** `scripts/check-probe-matches-compose.py`, a `--fast` row in `catalog_gates.py`, comparing the probe against the SAME service's own compose healthcheck. The rule was read out of `healthprobe.go` rather than guessed: a wrong PORT refuses for every check type; a wrong PATH refuses only for `type: api` WITH an `expect` block and WARNS otherwise, which is why `home-assistant` is warned about and not convicted. Four red-proofs and five decoys, plus five more with PyYAML shadowed out (R-630's sibling problem — see below). **Residual, filed separately:** R-630 (paperless-ngx's probe can never run at all) and R-631 (five templates the static rule cannot judge). | **CLOSED 2026-09-22 — three probes fixed, red-proofed live both ways, gate shipped with decoys; tandoor re-walked `failed -> proven`** | | **R-619** | **[P3-LOW] A `type: password` deploy field is MANDATORY however `required` reads, and the `deploy-fields` contract says the opposite — so any caller that trusts it is refused.** MEASURED 2026-09-21 on guest 9202 while widening the update drill. `GET /api/stacks/grafana/deploy-fields` serves `{"env_var":"GF_SECURITY_ADMIN_PASSWORD","type":"password","generate":"password:16","required":false}`; a deploy carrying only the two `required:true` fields is refused **400** „a(z) „Admin jelszó" mező kitöltése kötelező — használja a Generálás gombot…". **The BEHAVIOUR is right and is a decision, not a bug:** `deploy.go:305-312` refuses a `password` field with no caller value on purpose — *"We never silently auto-generate — the user needs to know their password"* — which is the opposite of the `secret` case one branch above, where a generated value the customer never sees is exactly correct. **The defect is the CONTRACT.** `.felhom.yml` declares `required: false`, the API serves that verbatim, and nothing on the wire distinguishes "optional because the box will generate it" (`secret`) from "optional in the template and mandatory in the code" (`password`). A person using the deploy page never meets this because the page renders a Generálás button; **anything that is not that page does**, which now includes this drill harness and would include `09` §6.2's unattended caller the day it deploys anything. **Fix shape (smallest that keeps the decision):** serve `required: true` for `type: password` in the deploy-fields response — one place, derived rather than stored, so templates need no edit — and a test asserting a `password` field always reaches the wire as required. Alternatively state it in the field's `description`, which is weaker because it is prose. Evidence: `audits/update-night-2026-09-21/apps/grafana/log.txt` (the refusal) and `batchA.log`. | **READY — rank P3-LOW; owner: CC (controller)** | | **R-620** | **[P3-LOW] A disabled notifier drops every event with NO local trace, so a box whose hub configuration is absent or broken stops telling anyone anything and leaves nothing behind that says so.** FOUND 2026-09-21 on guest 9202 while trying to score the update night's alarm truth table. `hub.enabled: false` there, and `Notifier.Publish` returns at `notify/notifier.go:269` — **before** any log line — as do `NotifyHealthChange` (:359) and four more entry points. Startup says it once (`[INFO] Notifier disabled (hub not configured)`) and then every later event, of every severity up to `critical`, vanishes without a word. **The measurable consequence tonight:** the whole event-and-mail half of the drill was structurally unmeasurable on this venue, and the alarm truth table below covers only the app page, the dashboard and the box's own log. That is a cost this session paid and named; the next one would pay it again. **The consequence on a real box is smaller but not zero:** the fleet's boxes have the hub enabled, and total silence is already caught by the hub's dead-man's-switch (staleness from the LAST REPORT, proven in the 2026-07-22 power-outage audit). What is NOT caught is the in-between — a box that still reports but whose notifier was disabled by a bad config push would go on reporting healthy while dropping every alarm, and the only evidence would be a single INFO line at the last restart. **Fix shape:** one DEBUG (or WARN, once per event type) line on the disabled path naming the event that was dropped, so the absence is visible where it happens rather than inferable from a startup line. Cheap, and it converts an invisible failure into a greppable one — R-96 rule 3 in the place that produces it. Evidence: `audits/update-night-2026-09-21/11-notifier-disabled.txt`. | **READY — rank P3-LOW; owner: CC (controller)** | | **R-621** | **[P2-MEDIUM] A held update DESTROYS the evidence of why it failed: `failAndHold` runs `compose down`, the failing containers are removed, and their output is gone before anyone — household, operator or the next session — can read it.** MEASURED 2026-09-21 on guest 9202 on a REAL upstream edge: `adventurelog v0.12.1 → v0.13.0`. The new backend applied **nine Django migrations successfully** and then never listened; the update held after the full 5-minute health wait. **`Manager.failAndHold` (`stacks/update.go:723`) calls `updateCompose(dir, env, "down")`**, which removes the containers rather than stopping them, and nothing captures their logs first. Within seconds the box's own log recorded `Logs result for adventurelog: 0 bytes returned (empty)` and `docker ps -a` held nothing at all. **What survives is the WHAT and not the WHY:** the controller line `update adventurelog FAILED after the new version was started: not healthy: not healthy within 5m0s (last: state unhealthy)` and the household's sentence, both of which say the app did not come up and neither of which says the migrations ran and the server then failed to bind. **This is R-320 ("evidence off the machine before the teardown") as a PRODUCT behaviour rather than a session habit** — the teardown here is the product's own, it is correct to perform (a half-started new version must not keep running), and it happens before anyone can look. **Why it matters beyond a drill:** the hold sentence sends the household to a restore, and after the restore the only remaining question is *should I press Update again?* — which nobody can answer, because the one artefact that would say so no longer exists. It also makes every future held update unreportable to an upstream project. **Fix shape:** capture `compose logs --no-color --tail N` into the stack directory (beside `applied-compose.yml`, which already travels with the stack) IMMEDIATELY before the `down`, and surface it on the app page's hold panel or at least through the existing `/api/stacks//logs` fallback. Bounded size, written once per hold. A test that holds an app and asserts the captured file is non-empty — it fails today. Evidence: `audits/update-night-2026-09-21/apps/adventurelog/why-it-failed.txt`, `state-after-hold.txt`. | **READY — rank P2-MEDIUM; owner: CC (controller)** | @@ -799,6 +799,9 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-628** | **[P2-MEDIUM] An empty search of a mailbox I do not control was turned into a claim about what a THIRD PARTY had done, and it went into the register as fact.** FOUND 2026-09-22; the operator caught it within minutes by producing the thread. The `due_checks_gate` fired R-433 (*have Hetzner answered?*). Two searches of the felhom catch-all came back empty, and the emptiness was written into R-433 as **"Hetzner has not answered"** and **"there is no evidence the tickets were ever opened"**. Both were false: ticket **#2026090103040671** had been opened and answered. **THE PRECISE FAULT IS THE INFERENCE, NOT THE QUERY — and that distinction is the whole value of this row.** Re-run afterwards WITH a positive control: `from:monitoring@felhom.eu` returns **201 threads**, `in:anywhere … includeTrash` reaches SENT, TRASH and mail back to January — **the instrument works.** And the exact ticket number, the exact subject `Storage Box issue`, and `from:hetzner` each still return **nothing**. So the literal finding — *this correspondence is not in this mailbox* — was CORRECT. What was invented was the step from there to *Hetzner has not answered* and *the tickets were never opened*. **A mailbox I can read is not the only place a reply can be**, and the operator's own account is exactly where a support ticket he opened would land. An absent record in ONE place can never answer a question about what SOMEONE ELSE did. **This is R-96 rule 3 in a new surface, one step further out than R-607**: there the instrument reported a stale value as current; here a correct observation was promoted to a conclusion it could not carry. **It is sharper still because the same session, the night before, gave every fixture a negative control and every gate a red-proof — and then reached for a search with neither.** **THE RULE, in one line:** *an empty search may be reported as "absent from the place I looked", never as "it did not happen" — and only after a control query that MUST hit has been seen to hit.* **Done:** the rule is written into the Gmail-access memory, where the next session meets it before it searches rather than after. | **CLOSED 2026-09-22 — rule recorded; R-433 corrected with the real answers** | | **R-627** | **[P2-MEDIUM] Nothing checked that the register is a well-formed table, so an append that ate two rows' state cells went unnoticed until a person read the file — and one row had been broken the same way for 45 days.** FOUND 2026-09-22. The 2026-09-21 update night appended measured results to eight rows with a regex that matched each row's trailing state cell; on **R-446** and **R-458** it consumed the cell and did not restore it, the cell reappeared as a stray FOURTH cell on a DUPLICATED copy of **R-626** and **R-625**, and a blank line was left between each pair. The register then reported **317 rows for 315 findings**, two rows carried no state at all, and two findings existed twice with contradictory state cells. **Nothing caught it:** `one_register_gate.py` compares this file against ROADMAP and `closed_register_gate.py` forbids an id in BOTH files — neither asks whether the file is a well-formed table, and neither notices an id duplicated WITHIN it. **THE RED-PROOF THEN FOUND AN OLDER INSTANCE NOBODY HAD SEEN: R-254 lost its state cell on 2026-08-08 (commit `59527d0`) and had rendered without a State column for 45 days.** **Closed the same day:** `scripts/register_shape_gate.py`, registered in `repo_gates.py` as gate 14 and reached by the pre-push hook, refusing a row that does not end with `|` (an eaten state cell), a duplicated id, or a blank line splitting the table; four decoys in `test_gate_decoys.py` — three convicting on the exact damage shapes and one asserting a healthy register still passes. **TWO THINGS THE RED-PROOF CORRECTED IN THE GATE ITSELF, kept because they are the finding's real content:** a first draft counted CELLS and convicted **125 innocent rows** — register cells carry literal `|` inside prose and shell snippets (`owner: CC | …`), so a row cannot be split on `|`, and a count that cannot be computed is not a check; and it skipped malformed rows before counting ids, reporting **5 duplicates where there were 2**. **Repaired:** both state cells restored from the stray cells that carried them, the two duplicate rows deleted, R-254's verdict sentence given its cell back, and **15 blank lines that split the register into 12 separate markdown tables** removed — every row's text byte-identical afterwards, proven by diff. **And the gate immediately earned itself:** the very next row inserted in this session (R-628) left a blank line behind and the gate refused it. | **CLOSED 2026-09-22 — gate 14, four decoys, register repaired 317→315** | | **R-629** | **[P2-MEDIUM] The drill catalog sent the operator 47 CI-failure alarms in one night, and the drill method that created it did not mention CI at all.** FOUND 2026-09-22 while checking a different mailbox question — which is the only reason it was found. `admin/app-catalog-drill` was created on 2026-09-21 by `POST /api/v1/repos/migrate` from the live catalog, and a migrated repo inherits `has_actions: true`. Every drill push therefore ran the catalog's CI workflow, which failed immediately (the drill repo carries the workflow but the run has no meaningful gate context), and **each failure mailed `admin@felhom.eu`**: *"[felhom CI] gates FAILED in admin/app-catalog-drill"*. **47 runs, 47 alarms, all overnight, all unread and flagged IMPORTANT.** **WHY THIS MATTERS MORE THAN THE NOISE:** that mailbox is the operator's alarm channel, and R-168 made CI mail the thing that notices a bypassed gate. A night of throwaway failures from a repo nobody must ever act on trains the reader to skim exactly the sender that must never be skimmed — and it did it on the night the same mailbox was also carrying real `offsite_snapshots_dropped` and `offsite_proof_empty` alarms. **The drill method wrote down the fences it needed** — private repo, no customer box may follow it, reset at teardown — **and said nothing about CI, because nobody had run a drill repo through a CI-enabled Gitea before.** **FIXED 2026-09-22:** `has_actions` set to **false** on the drill repo (verified by re-reading the repo: `has_actions: False, private: True`), and `09` §6.5 now carries it as a step of creating a drill repo rather than as a thing to notice afterwards. **What is NOT done:** the 47 mails are still in the operator's inbox, unread — deleting another person's mail is not mine to do, and they are named here so they can be cleared in one search: `subject:"gates FAILED in admin/app-catalog-drill"`. | **CLOSED 2026-09-22 — actions disabled, method updated; the 47 mails are the operator's to clear** | +| **R-630** | **[P2-MEDIUM] `paperless-ngx`'s health probe has never run, on any box, and the badge can never go red.** FOUND 2026-09-22 by the new `probe-matches-compose` gate, which reported it as a WARNING while looking for something else. `findProbeContainer` (`controller/internal/stacks/healthprobe.go:297`) takes the container whose name EQUALS the stack name, else the first whose name has it as a PREFIX; paperless-ngx's containers are `paperless-webserver`, `paperless-postgres` and `paperless-redis`, and **none of them begins with `paperless-ngx`**. The function returns `""`, `:62` counts the stack in `skippedNoContainer` and `continue`s, and no probe is ever built for it. **WHY THIS IS WORSE THAN A WRONG PROBE, WHICH IS WHAT R-618 WAS:** a wrong probe is a FALSE RED — loud, visible, and it stopped an app, which is how it was found within one night. This is a SILENT ABSENCE. The app page shows whatever the container state alone says, nothing contradicts it, and **R-96 rule 3 is the exact shape: an absent alarm is equally consistent with `healthy` and with `never checked`.** **NOT MEASURED, and said so rather than assumed:** what the guarded update's `verifying` phase does for a stack with no probe target — whether it passes immediately or waits out `update.health_timeout` — has not been run. `paperless-ngx` IS installed on demo-hp, so it is answerable on a real box. **Two candidate fixes, neither taken here because both are product code and this session was forbidden it:** give the template a `container_name: paperless-ngx` on its webserver service (catalog-only, one line, but it renames a running container on every existing box); or let the probe fall back to the service the compose declares first, and SAY SO in the health detail. **What is safe to say today:** the gate names it on every push, so it cannot go back to being invisible. | **OPEN — P2; owner: CC; needs the `verifying` behaviour measured on demo-hp before a fix is chosen** | +| **R-631** | **[P3-LOW] Five templates cannot be judged by the probe gate at all, and one is correct only by accident.** FOUND 2026-09-22 when `check-probe-matches-compose.py` was run over all 53. The gate's oracle is the probed service's own compose `healthcheck.test`; where that dials no loopback URL the gate has nothing to compare and reports a WARNING rather than a pass. **`crafty-controller`, `mealie` and `uptime-kuma`** run their healthcheck through a python or script helper (`ssl._create_unverified_context()`, `socket.create_connection`, `extra/healthcheck`), so the port is inside code the gate does not execute. **`vikunja`** has no compose healthcheck on its probed service at all. **`home-assistant`** is the interesting one: its probe path `/api/` differs from the compose's `/manifest.json`, and it is healthy today **only because its check is `type: api` with no `expect` block, which `probeHTTP` treats as "any response is healthy"** — add `expect: {status: 200}` to that template, a change that looks like a tightening, and the app goes permanently unhealthy and every successful update of it starts stopping it. **The gate warns on it for exactly that reason and refuses to call it a pass.** **Needs:** a live probe reading for each of the five on a scratch guest — deploy, read `GET /api/stacks/`, compare with the front door — which is one rotation night's work and closes the last gap R-618 left. | **OPEN — P3; owner: CC; five apps, one live reading each** | +| **R-632** | **[P3-LOW] Twenty-eight of the 53 templates have never been deployed by any update drill, so nothing is known about whether their updates work.** COUNTED 2026-09-22 against the 2026-09-21 sweep, which is the widest one ever run. **20 apps have a verdict record** (14 proven, 3 failed, 3 inconclusive, plus tandoor re-walked to proven on 2026-09-22); **4 more were deployed as props in the bad-days legs with no edge walked** (`bentopdf`, `glance`, `uptime-kuma`, `wishlist`); **1 was deployed only to measure its probe** (`wger`); and **28 have never been deployed at all**: `calcom`, `calibre-web`, `claper`, `code-server`, `crafty-controller`, `emby`, `ghost`, `gokapi`, `gramps-web`, `homebox`, `homepage`, `immich`, `jellyfin`, `kimai`, `komga`, `onlyoffice`, `outline`, `paperless-ngx`, `plant-it`, `plex`, `radarr`, `rallly`, `recipe-importer`, `seerr`, `sonarr`, `sparkyfitness`, `termix`, `wanderer`. **THIS IS NOT A COMPLAINT ABOUT THE SWEEP** — it took one app-catalog-wide night to go from 3 apps ever measured to 21, and a night is the unit available. It is a record of what the catalog's update promise currently rests on: **for 28 of 53 apps, nothing.** **The list is the nightly rotation's queue**, smallest and least stateful first; `paperless-ngx` should be early because R-630 needs a live reading from it anyway, and `crafty-controller`, `mealie` and `uptime-kuma` should be early because R-631 needs one from each. Machine-readable copy: `audits/probe-fix-2026-09-22/not-judged.json`. | **OPEN — P3; owner: CC; the rotation works this list, 28 apps** |