# STATUS — what works, what's broken, what's next **Ready for the first real tester (Tester-2): yes. You confirmed the tunnel route and the connect mails (2026-09-30).** **Updated 2026-10-04 (evening): the HOST's security fixes install themselves too, the hub has a fleet view and four OS alarms, and the tunnel status is true. Both demo boxes run controller 0.292.0 and host agent 0.141.1. Hub 0.131.1. New installs: see the golden line in the section below.** ## Today (2026-10-04, evening): host fixes, the fleet view, a true tunnel status **Decisions I took myself (you may reverse each):** - Only a box the installer itself recorded as "appliance" gets host updates (a record the agent cannot change). - The four alarm times: no update run for 7 days; "reboot needed" for 14 days; the demo boxes approve nothing for 7 days; a box has fixes nobody approved for 14 days. All four are settings. - I released the host agent and the hub a second time today. The first release said "reboot needed" wrongly; the new 14-day alarm would then have mailed you about boxes you had already rebooted. **One decision for you (a safe default if you say nothing) — the Docker engine updates (designed, not built):** 1. **Turn on Docker's "live-restore" on every box.** With it, a Docker engine update restarts no app (measured: 0 restarts). Without it, every app stops for about 30 seconds per engine update. - **A (my pick):** turn it on — in the new-install image and once on existing boxes. It goes on without restarting anything. Cost: it must never be turned off by a plain restart again (that stops every app and starts none). - **B:** leave it off. Every Docker update then means ~30 seconds of every app being down, at night. - **If you say nothing:** nothing changes; Docker updates stay unbuilt. **What I did:** - **The host's Debian fixes now install themselves**, after the guest's, on the same night run, only on appliances, never a kernel or boot package, never a reboot. Proven: demo-felhom installed 108 host fixes and stayed healthy; all 108 came from Debian. A box made "ring 1" for a test installed exactly the one approved host version it lacked. - **A way to put one host package back by hand** is written and proven on demo-hp (and the test taught it two fixes). - **The fleet view** in the hub: one line per box with its updates, "reboot needed since", and the tunnel. - **The tunnel status is now true:** running, not running, or unknown. I blocked demo-hp's tunnel: after two reports (about 30 minutes) you got a "tunnel down" mail; when I unblocked it, "recovered". A simply stopped tunnel heals itself within 5 minutes, before the hub can even see it. - **The night run is fast:** 23–32 seconds when there is nothing to install (target was under 60). - **The kernel test on demo-hp (your two reboots):** Secure Boot works with it, but GRUB's "boot once" does not work on our boxes: the second plain reboot came up on the NEW kernel again. So a new kernel that hangs would stay. That must be solved before kernel updates. demo-hp now runs the newer kernel, healthy. A watchdog chip exists on demo-hp. - **Found and fixed during the live test:** "reboot needed" was wrong in two ways (fixed in the second release). - **Rows:** 2 closed, 2 opened-and-closed the same day, 3 opened. The list went from 332 to 333. **Needs you later (nothing breaks if you wait):** - **A host that crashes does not restart by itself** (Linux's "panic" setting is off). Changing it changes how every box behaves, so it is your call, together with kernel updates. ## Today (2026-10-04, afternoon): the guest's security fixes install themselves **One decision for you (a safe default if you say nothing):** 1. **Undo for a guest update that goes wrong.** I tested it first, as you asked: Proxmox cannot take a snapshot of a customer box at all (the box is linked to the household's drives, and Proxmox refuses). So there is no automatic undo. Today a failed check stops, mails you, and last night's whole-box backup (minutes old) is the undo, by hand. - **A (my pick):** keep it so. Nothing new to build. A restore takes about 1–3 minutes plus losing what apps wrote since the backup. - **B:** build our own disk snapshot under Proxmox. Automatic, but a new mechanism nobody has tested, with a risk to the disk pool. - **If you say nothing:** A stays. **What I did:** - **Your two choices are built.** A returning household's new box sets the old off-site copy aside on night one and starts a new one (nothing deleted). An approved version that Debian already replaced comes from Debian's dated archive. - **Guest security fixes now install themselves.** Each night, after the whole-box backup, the demo boxes install Debian's fixes. When both demo boxes run the same versions healthy for 24 hours and one night, the hub approves that set; every other box then installs exactly those versions. Docker, the host and the kernel are not touched. - Proven live: each demo box installed 53 fixes and stayed healthy. With a 2-minute test wait the hub approved the set; then I put back 24 hours. demo-felhom, made "ring 1" for the test, installed exactly the 3 approved versions it lacked and nothing newer. A run where I stopped an app failed its check and mailed you (you got that mail). - You can switch OS updates off per box in the hub. They are ON by default. The household sees one line in its timeline: "System security fixes installed". - **The tunnel and two other built-in programs are current** (cloudflared was 4 months old). A new monthly check catches them falling behind. The tunnel came back by itself on both demo boxes within 20 seconds. - **Three bugs found and fixed during the live test**, before release. - **Two gaps found, now rows:** existing boxes cannot receive the new update tool through the product (only new installs, or by hand — the demo boxes got it by hand); and the hub's "tunnel status" for every box always reads "inactive" because it checks the wrong place. - **Rows:** 4 closed (one opened and closed the same day), 5 opened. The list went from 331 to 333. ## Today (2026-10-04, day): off-site closed, operating-system updates measured **Two decisions for you — each has a safe default if you say nothing:** 1. **A returning household's first night.** A new box for someone who had a box before makes no off-site copy, because the old copy was made with a key the new box does not have. - **A (my pick):** the new box sets the old copy aside by itself on its first night, then starts a new one. An un-claimed box already does exactly this. Nothing is deleted; you can put the old copy back. Cost: a small box change. A household that wanted to continue the old copy with its recovery code must do that before night one. - **B:** ask the household on the recovery-code evening. Cost: new screens in two languages. If they skip it, the gap stays. - **If you say nothing:** such a household has no off-site copy until someone presses "start a new off-site backup", and you get a mail each time. Nobody is blocked today (no returning household is waiting). 2. **Approved OS updates, when Debian has already replaced the version.** Debian keeps only two versions of a package. - **A (my pick):** the box then fetches the exact approved version from Debian's own dated archive (snapshot.debian.org, signed by Debian). Measured: 2–3 seconds. Cost: one more outside service we rely on — but in 3 months it would have been needed **zero** times for the packages a box has. - **B:** the box waits for the next approval. Cost: no new service; a box can stay unpatched for one cycle. - **If you say nothing:** nothing is blocked now; the first build step can start with B and switch later. **What I did:** - **A restored box is safe by default.** The automatic restore-test was already safe (measured). The agent's disaster restore now refuses to run next to a live original. Restores by hand use a new safe script: no start on boot, no link to the real drives, network off. All proven live on demo-hp. New agent 0.139.0 on both demo boxes. - **The weekly clean-up cannot get stuck any more.** You can allow one bigger clean-up for one box (hub 0.129.0). Proven on a lab copy: 98 backups, the normal limit refused, the bigger one cleaned to 13, and the next week was normal again. - **A dated check for 12 October** looks at the first clean-up that really deletes old backups. - **OS updates, measured on the demo boxes (nothing built for customers):** - The boxes are far behind: 188 updates per host, about 59 per guest. Every guest runs a different Docker version. - A Debian update of the guest took 24 seconds. No app stopped. The undo (restore the guest) took 73 seconds. - A Debian update of demo-hp's host took 60 seconds. The guests kept running. - A broken (killed) update does not fix itself. Two repair commands fix it in 5 seconds. - A Docker update restarts every app (about 30 seconds silent). A Docker setting ("live-restore") avoids that completely — but switching it off again stopped every app and started none. Recorded as a trap. - The new kernel booted fine, and the box fell back to the old kernel on the next restart. **But** if a new kernel hangs, the box keeps trying it. Someone must then switch it off and on. No hardware watchdog is used. - About 40 packages from Proxmox have ordinary names (for example the disk system ZFS and the Secure Boot loader). So "which lane" must follow where a package comes from, not its name. - The internet tunnel program on every box (cloudflared) is 4 months old. Nothing updates it. New row. - **Thank you for being near the box.** demo-hp restarted twice. It now runs the old kernel and starts the new one at its next restart. - **Rows:** 2 closed, 5 opened. The list went from 328 to 331. ## Today (2026-10-04): off-site safety finished - **The weekly clean-up is ON and no longer stops itself.** The fake-backup check now skips young copies that a manual backup replaced the same day, instead of refusing. I ran one window on demo-hp: no refusal, nothing removed (127 → 127), because every candidate was still young. The first real removals come when those copies are older than 8 days — around 11 October. demo-felhom gets its first window at its next night run. - **DooPlex keeps 8 weekly copies of ep0** (your choice). Old copies go weekly; anything ep0 deletes still never reaches the copy by itself. Today's nightly copy ran fine. - **tester-1's 3 old keys are gone.** The daily alarm for tester-1 stopped. (You got one last alarm mail this morning, sent just before I removed them.) - **A restore from the DooPlex copy works.** I restored demo-hp's whole box onto a scratch machine in about 3 minutes, read its data, then deleted it. **One trap found:** a restored box starts automatically and points at the real drives. Run beside the original, that would be two copies of the same box. I switched it off in time; the runbook now warns, and a row asks to make it safe by default. - **"Delete my old set-aside backups" works again.** The hub does it, 7 days after the household's request; the household or you can cancel in that time. Tested on tester-1 with a planted test folder. - **Your answers are recorded**, including that you keep the current Hetzner key for now (a row holds the 3 steps). - **Rows:** 6 closed, 4 opened. The list went from 330 to 328. ## Today (2026-10-03, evening): your choices A and A — built - **A box can no longer delete its off-site backups.** Both demo boxes now use a key that can only add. I tried a delete from each box: refused. Old backups all stayed (demo-felhom 11 → 13, demo-hp 91 → 100). - **No box gets the storage password any more.** The box gives the hub only its public key; the hub puts it in the storage account. I asked for the password with demo-hp's own login: refused. - **The hub keeps the passwords locked (encrypted).** A copy of the hub database no longer reveals them. - **Every day the hub checks each storage account's key file.** It found old unlocked keys from earlier boxes: 4 on demo-felhom's account and 5 on demo-hp's — now removed. tester-1's account still has 3 (its box is gone); you get a daily alarm for it until they go. - **The weekly clean-up window works, but it is switched OFF.** I opened one window on demo-hp by hand: it opened, the fake-backup check refused, and it closed in 3 seconds. The check refused because I had made a manual backup today — it is too strict. I fix that next; until then nothing deletes old backups (there is plenty of room). - **ep0's whole-box backups are copied to DooPlex every night.** First copy: 12 GB in 3 minutes, all 4 backups. DooPlex cannot read them (encrypted per household). A failed copy mails you; the test mail arrived. - **One bug, found and fixed live:** the first new box version misread its backup count as 0, and you got one false alarm mail ("demo-felhom: fell from 11 to 0"). **Ignore that mail.** Fixed 15 minutes later (0.289.1). - **Rows:** 3 closed, 6 opened, 1 opened and closed the same day. The list went from 327 to 330. ## Today (2026-10-03, later): off-site backup safety, step 1 — measured, nothing built - **The "add only" lock works.** I tested it on tester-1's storage account (your choice; no box uses it now). New backups go in. Restore works. Every delete is refused. I put the account back exactly as it was. - **But the lock alone does not protect us yet.** A broken-into box can ask the hub for the storage password. With that password it can log in and remove the lock. First the box must stop getting the password. - **A second trap:** with "add only", an attacker can add fake backups dated in the future. The normal clean-up rule then deletes all the real backups. Any clean-up must check for this. - **The hub keeps every storage password in plain form.** Anyone who reads the hub database can delete every household's off-site backups. New row. - **ep0:** Hetzner cannot snapshot the extra disk at all. One of the three old ideas does not exist. - **Rows:** 2 closed, 3 opened. The list went from 326 to 327 rows. The dated check for 6 October is done. ## Today (2026-10-03): the to-do list is in order — paperwork only, no machine touched - **Finished items left the open list.** It went from 442 rows to 326. Nothing was deleted; each moved row names where its full text is. - **Every open row now has one category and one severity** (P1 now · P2 before the first paying customer · P3 during the first customers · P4 later). **No row is P1.** 27 rows are P2. - **Two new automatic checks** refuse a finished row left in the open list, and a new row without a category or a severity. - **Your four new items are on the roadmap:** security updates for the box's own system; legal pages and business papers; "what if the household leaves Felhom"; a second login step for the dashboard. Two of them are also real findings today: **a box never receives system security updates**, and **the website has no privacy notice, terms or imprint.** - The ranked list and my reasoning: the triage recommendation in the audits folder. **Tester-2 — read only, from the hub.** The customer record exists. Tester-2's box has not registered yet. ## Today (afternoon): every app checked again - **No data-loss fault.** I checked all 58 apps with the fixed check. It made each app really save something, then looked where the data landed. **No app saves data where the backup does not copy it.** - **40 apps: proven correct. 18 apps: not proven either way.** In those 18 the check could not fill every folder (for example an upload folder stays empty because my test saves no upload). Nothing was found outside a backed-up folder. I list them and work on them later. - **papra was broken for new installs, now fixed.** It needs more memory than its limit, so a fresh install crashed in a loop. I measured it and raised the limit. No box runs papra. - **plant-it's program image is gone from Docker Hub.** It is already hidden from new installs. No box runs it. ## Removing an app now tells the truth (your choice A) - When the household removes an app, its own files (books, videos) **stay**. The dialog and the result now say so, and name the folder. Proven on the scratch box: the video was still there after the remove. ## Before the first paying customer Everything here must be done before the first customer who pays: 1. **SparkyFitness:** written permission from the author (the e-mail draft is ready). If none → hidden from new installs. 2. **Tandoor:** written permission from the authors. If none → hidden from new installs. 3. **A lawyer reviews the licence list** (Tandoor, SparkyFitness, Emby, n8n, Plex, the paid "enterprise" parts, Redis). 4. **Agent updates:** I may sign agent updates only until the first paying customer; after that you sign them. Your licence decisions are recorded: Emby, Plex and n8n stay. recipe-importer needs nothing unless we share it. ## Also today - **New version 0.288.0 and a new golden (0.288.0).** - **Rows.** 7 closed, 6 opened. The list went from 436 to 442 rows. ## What needs you 0. **The undo choice at the top of today's section** (A: keep the backup as the undo; B: build our own snapshot). If you say nothing, A stays. Earlier today's two decisions are built. The Hetzner key change stays your call. 1. **plant-it:** keep the hidden template as it is, or remove it entirely (its image no longer exists). **If you say nothing:** it stays hidden; nothing runs it. 2. **Send the SparkyFitness request, and ask the Tandoor authors** (the "Before the first paying customer" list). 3. **Phone test (2 minutes), only if you want it:** say so, and I put MeTube back on demo-hp with a family login. ## Standing steps - **Monthly security re-test: last run 2026-10-01, next due ~2026-11-01.** (You start it with the standing brief.) - **Weekly:** the golden bake (next around 9 October; today's bake was 0.288.0). - **Registry clean-up:** only when the registry disk fills; "show me" mode first, then a person decides.