felhom.eu/STATUS.md

STATUS — what works, what's broken, what's next

Updated 2026-08-04.

A view, not a source. documentation/backlog/OPEN-ITEMS.md is the authority on open work; this page restates part of it in plain words, and nothing may exist only here. Not CONTEXT.md, which is technical state written for Claude Code — keep the two separate. Maintenance: update at the end of every session in which something shipped, broke, or was decided. One screen; cut items rather than extend it.

What works right now

A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer, who sets their own password. They install apps from a catalogue of fifty-three, share files over the home network, and open apps from a launcher or a shared link. Backups run on their own to three places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo box six times, everything returned every time, and an app switched off deliberately stayed off. Proven end to end on real hardware.

What's broken

• Rebuilding a machine throws away its off-site backup HISTORY — and that is worse than we thought yesterday. We spent today establishing what a rebuild actually destroys, and the answer changes the picture. There are three secrets involved. The one we have been talking about — the password to the storage provider — is the one you can re-issue any time, and we did. The one that matters is the key that encrypts the backups themselves. The machine invents that key, and a rebuilt machine invents a brand-new one, so the old backups can no longer be opened by the machine that wrote them. Both demo machines did exactly this, and one of them did it silently. demo-hp lost access to 15 backups (41 MB). demo-felhom — the one we called lucky, because its off-site copy came back on its own in 76 seconds — lost access to 36 backups (1.14 GB), and nothing said a word for thirteen hours. The luck restored the delivery, not the history. Nothing has been deleted, and this was measured without touching either machine or any password. The next scheduled off-site run is tomorrow at 04:15, and it will refuse rather than quietly start a new history — the machine already knows how to recognise this and asks before resetting. There is a decision here for you — see "Waiting on you". (R-193, R-192, and two new items)
• And the safety net we believed was under all of this is not there. We traced the whole path today — from a dead machine to reopened backups — and it does not join up. The important part is not that several steps are manual; it is this: the central hub keeps the old sealed key when a machine re-seals, and we found it keeps the wrong one. It keeps the key for the whole-machine backups and not the key for the off-site file backups — the one this entire problem is about. So the moment a rebuilt machine is asked to make a new recovery code, which is exactly what it asks the customer to do, the last copy of the old key is gone. Both demo machines crossed that line yesterday morning. This means the 51 orphaned backups were beyond reach even if you had kept the recovery codes — which is a relief in one narrow sense and much worse in every other. It also means the customer is currently told, in Hungarian on their own screen, that their old backups "may later be restorable with the matching recovery code". That sentence is not true today. It is a small fix — one missing column — and it must land before anything else here. (R-198)
• Nothing in the recovery path has ever been performed. Making a recovery code is proven, live, with a real customer. Using one is not. No sealed package has ever been handed back to a machine (the hub has the buttons for it; nothing anywhere presses them), no recovered key has ever been put back, no old backup store has ever been reopened, and no file has ever been restored from a recovered key. There is a form missing, a step missing, and a connection missing between two parts of the system. We designed the exercise that would prove it end to end — see "Waiting on you". (R-199, R-200, R-201)
• (fixed 4 Aug) The weekly off-site backup reports FAILED although it worked. It uploaded fine and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer asks — tidying up is the endpoint's job, and that was checked first: the endpoint has been doing it every day since 27 July. Both machines and all future ones. (R-191)
• The off-site copy can be erased by the machine that made it. The credential that writes it can also delete it. A daily snapshot is armed as a stopgap. (R-95, R-87)

What shipped recently

• The daily error email about David is gone. David's record was created when he said he would try Felhom; no machine was ever attached to it, so nothing has ever been expected of it — yet every morning at five the system reported that a database backup had not happened. A customer with no machine ever attached now produces no alarm. A machine that was attached and went quiet still does, because that is a real fault, and there is a test whose whole job is to make sure a future change cannot take that away. David's record was not touched — the record was right; the alarm was wrong. The first quiet morning will be 5 August, so this is not confirmed yet. (R-195)

• A permission the machine needs can now repair itself — and it tells you it had to. Yesterday we found a permission that vanished with no reinstall and no recorded action; the machine now notices, restores it through the same controlled helper that grants it in the first place, and raises an alert saying it was missing and was restored. The alert is the point: a permission that silently fixes itself is one that can be lost over and over and never investigated. Proven twice on a demo machine by taking the permission away by hand. (R-190 — the repair; why it vanished is still open)

• The on-machine backup copy has now been proved to restore — by the machines themselves, both of them, overnight and unasked (84 and 109 seconds). Every restore proof before this was of the off-site copy. (R-86, R-185)

• Three ways the alarm system was misreporting its own work — all fixed. None of them ever risked data. (1) When the machine proved a backup restores, that result could vanish if the agent was restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a day, because the machine correctly refuses to re-prove an archive it has already proven. It is now written to disk with the result and survives. This was caught happening, not predicted: a real 14.5 GB off-site restore passed and left no record at all. (2) Every release had about a fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now published after the binary, and a new check catches the opposite mistake so nothing is traded away. (3) A released binary can now be rebuilt by anyone and checked against the fingerprint you approve — until today, rebuilding produced different bytes. (R-189, R-188, R-186)

What we're working on

• Now: waiting on your answer about the backup key (above). Two small honesty fixes are ready to go the moment you say so — making a changed backup key raise an alarm, and correcting an email that currently states the opposite of what it measured. Neither needs a decision.
• Also tomorrow: confirming what the 04:15 off-site run does on both machines. We expect it to refuse and ask, rather than start fresh — but that is a prediction from the code, not something anyone has watched happen.
• Next: proving the off-site app-data copy can actually be restored — the one tier nothing tests unattended. Most of the machinery it needed arrived with the restore-test change below. (R-87)
• After: the off-site copy that the machine making it can still erase. (R-95)

Waiting on you

• A real decision, and it is the important one on this page. Every machine is going to be reinstalled at some point, and each reinstall throws away the key that opens its off-site backups. There are only two honest answers, and picking either is fine — picking neither is what leaves it where it is. (a) Keep a copy of that key on the Proxmox host, which survives a rebuild. The machinery for this already exists and is barely any work. The price: the key would then sit on a machine you have root on, so "only the customer's recovery code can open these backups" stops being true in the strict sense. It was never quite true anyway — you hold root everywhere — but this makes a real second copy, and copies are exactly what we are already uneasy about elsewhere. (b) Leave the key sealed under the customer's recovery code, accept that every rebuild starts a fresh off-site history, and make sure the system says so loudly on the day it happens instead of never. Old backups are kept, not deleted, but they keep consuming the customer's off-site space. We recommend doing the "says so loudly" part immediately regardless — it is cheap, it costs nothing else, and it is the piece whose absence let a gigabyte of history go quiet for thirteen hours. What we are asking you to choose is (a). (R-193; the full reasoning is in the findings document) Correction to (b), from today: "old backups are kept" is true, but "openable later with the recovery code" is not — see R-198 above. Option (b) is only honest once that is fixed.
• Your ruling about the recovery screen has been priced, and it can be built. You said: a freshly installed machine that finds a sealed package waiting should say so loudly, offer the customer a box to type their recovery code into, and show what would come back before doing anything. All of it is buildable, and one part is already free — the hub is already telling every machine, on every check-in, that a sealed package exists and which key it covers, and the machine currently throws that message away. Showing a preview is also cheap: listing what is in an off-site store reads it without writing to it, so the customer can see how many backups, from when, and for which apps before committing. The real work is one new connection: the unsealing has to happen in the part that runs on the Proxmox host, because the customer-facing part cannot do it — the same crossing your recovery-code ceremony already makes in the other direction. One thing for you to weigh, which we deliberately did not decide: a screen that takes a recovery code and then shows what is in a backup store is reachable by anyone with the household's dashboard password, and the preview reveals backup dates and app names. (R-193)
• Should we run the proof? Nothing about recovery has ever actually been done, so we designed the exercise: on the spare demo machine, make a recovery code and keep it, put a marked file into an app, back it up off-site, wipe the machine, reinstall, recover with the saved code, and check the marked file comes back byte for byte. About half a day, of which under an hour needs you. The failure we are watching for is precise: if the recovered machine reports one backup instead of the ones we put there, it started a fresh history and the proof failed — "the store opened" is not good enough. It costs that machine's contents if it goes wrong, which is acceptable there. Do R-198 first, or the exercise walks a path that is missing a piece. (R-201)
• The orphaned backups still on the storage box. About 1.2 GB across the two demo machines, sitting in set-aside stores nobody can open and nothing prunes, using the 50 GB allowance. We did not touch them. Delete, or leave? (R-193)
• A job, not a decision: the hub password needs changing. A diagnostic command printed it into a session log; nothing suggests anyone else saw it. (R-132)
• One small question, not urgent. The automatic check cannot see which version you have told machines to install, only which ones exist. Closing that needs either a password given to the build server or a check inside the hub itself. (R-184)
• Nothing else.

Changed since last update

• 2026-08-04 (latest) — Traced the whole recovery path end to end and it does not join up. The hub keeps the wrong sealed key when a machine re-seals, so a rebuilt machine's old off-site key is lost the moment the customer is asked for a new recovery code. Nothing in the recovery path has ever been performed. Your ruling about the recovery screen is priced and buildable. The proof exercise is designed and awaiting your go-ahead. (R-198, R-199, R-200, R-201, R-193)
• 2026-08-04 (later) — Established what a machine rebuild actually destroys: not the storage password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost access to their off-site history, one of them silently. A decision is now waiting on you. The daily false alarm about David is fixed. (R-193, R-195)
• 2026-08-04 — Both demo machines proved their on-machine backup restores, on their own, overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the weekly off-site backup reports failure after a successful upload. (R-185, R-191)
• 2026-08-03 — Fixed three ways the alarm system misreported itself: a proof of a working backup that could vanish on a restart (seen happening), a release that emailed a failure for a release that worked, and a released binary nobody could rebuild and check. (R-189, R-188, R-186)