STATUS — what works, what's broken, what's next

Updated 2026-08-04.

A view, not a source. documentation/backlog/OPEN-ITEMS.md is the authority on open work; this page restates part of it in plain words, and nothing may exist only here. Not CONTEXT.md, which is technical state written for Claude Code — keep the two separate. Maintenance: update at the end of every session in which something shipped, broke, or was decided. One screen; cut items rather than extend it.

What works right now

A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer, who sets their own password. They install apps from a catalogue of fifty-three, share files over the home network, and open apps from a launcher or a shared link. Backups run on their own to three places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo box six times, everything returned every time, and an app switched off deliberately stayed off. Proven end to end on real hardware.

What's broken

Rebuilding a machine throws away its off-site backup HISTORY — and that is worse than we thought yesterday. We spent today establishing what a rebuild actually destroys, and the answer changes the picture. There are three secrets involved. The one we have been talking about — the password to the storage provider — is the one you can re-issue any time, and we did. The one that matters is the key that encrypts the backups themselves. The machine invents that key, and a rebuilt machine invents a brand-new one, so the old backups can no longer be opened by the machine that wrote them. Both demo machines did exactly this, and one of them did it silently. demo-hp lost access to 15 backups (41 MB). demo-felhom — the one we called lucky, because its off-site copy came back on its own in 76 seconds — lost access to 36 backups (1.14 GB), and nothing said a word for thirteen hours. The luck restored the delivery, not the history. Nothing has been deleted, and this was measured without touching either machine or any password. The next scheduled off-site run is tomorrow at 04:15, and it will refuse rather than quietly start a new history — the machine already knows how to recognise this and asks before resetting. There is a decision here for you — see "Waiting on you". (R-193, R-192, and two new items)
(fixed 4 Aug) ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer asks — tidying up is the endpoint's job, and that was checked first: the endpoint has been doing it every day since 27 July. Both machines and all future ones. (R-191)
The off-site copy can be erased by the machine that made it. The credential that writes it can also delete it. A daily snapshot is armed as a stopgap. (R-95, R-87)

What shipped recently

The daily error email about David is gone. David's record was created when he said he would try Felhom; no machine was ever attached to it, so nothing has ever been expected of it — yet every morning at five the system reported that a database backup had not happened. A customer with no machine ever attached now produces no alarm. A machine that was attached and went quiet still does, because that is a real fault, and there is a test whose whole job is to make sure a future change cannot take that away. David's record was not touched — the record was right; the alarm was wrong. The first quiet morning will be 5 August, so this is not confirmed yet. (R-195)
A permission the machine needs can now repair itself — and it tells you it had to. Yesterday we found a permission that vanished with no reinstall and no recorded action; the machine now notices, restores it through the same controlled helper that grants it in the first place, and raises an alert saying it was missing and was restored. The alert is the point: a permission that silently fixes itself is one that can be lost over and over and never investigated. Proven twice on a demo machine by taking the permission away by hand. (R-190 — the repair; why it vanished is still open)
The on-machine backup copy has now been proved to restore — by the machines themselves, both of them, overnight and unasked (84 and 109 seconds). Every restore proof before this was of the off-site copy. (R-86, R-185)
Three ways the alarm system was misreporting its own work — all fixed. None of them ever risked data. (1) When the machine proved a backup restores, that result could vanish if the agent was restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a day, because the machine correctly refuses to re-prove an archive it has already proven. It is now written to disk with the result and survives. This was caught happening, not predicted: a real 14.5 GB off-site restore passed and left no record at all. (2) Every release had about a fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now published after the binary, and a new check catches the opposite mistake so nothing is traded away. (3) A released binary can now be rebuilt by anyone and checked against the fingerprint you approve — until today, rebuilding produced different bytes. (R-189, R-188, R-186)

What we're working on

Now: waiting on your answer about the backup key (above). Two small honesty fixes are ready to go the moment you say so — making a changed backup key raise an alarm, and correcting an email that currently states the opposite of what it measured. Neither needs a decision.
Also tomorrow: confirming what the 04:15 off-site run does on both machines. We expect it to refuse and ask, rather than start fresh — but that is a prediction from the code, not something anyone has watched happen.
Next: proving the off-site app-data copy can actually be restored — the one tier nothing tests unattended. Most of the machinery it needed arrived with the restore-test change below. (R-87)
After: the off-site copy that the machine making it can still erase. (R-95)

Waiting on you

A real decision, and it is the important one on this page. Every machine is going to be reinstalled at some point, and each reinstall throws away the key that opens its off-site backups. There are only two honest answers, and picking either is fine — picking neither is what leaves it where it is. (a) Keep a copy of that key on the Proxmox host, which survives a rebuild. The machinery for this already exists and is barely any work. The price: the key would then sit on a machine you have root on, so "only the customer's recovery code can open these backups" stops being true in the strict sense. It was never quite true anyway — you hold root everywhere — but this makes a real second copy, and copies are exactly what we are already uneasy about elsewhere. (b) Leave the key sealed under the customer's recovery code, accept that every rebuild starts a fresh off-site history, and make sure the system says so loudly on the day it happens instead of never. Old backups are kept, not deleted, but they keep consuming the customer's off-site space. We recommend doing the "says so loudly" part immediately regardless — it is cheap, it costs nothing else, and it is the piece whose absence let a gigabyte of history go quiet for thirteen hours. What we are asking you to choose is (a). (R-193; the full reasoning is in the findings document)
A job, not a decision: the hub password needs changing. A diagnostic command printed it into a session log; nothing suggests anyone else saw it. (R-132)
One small question, not urgent. The automatic check cannot see which version you have told machines to install, only which ones exist. Closing that needs either a password given to the build server or a check inside the hub itself. (R-184)
Nothing else.

Changed since last update

2026-08-04 (later) — Established what a machine rebuild actually destroys: not the storage password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost access to their off-site history, one of them silently. A decision is now waiting on you. The daily false alarm about David is fixed. (R-193, R-195)
2026-08-04 — Both demo machines proved their on-machine backup restores, on their own, overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the weekly off-site backup reports failure after a successful upload. (R-185, R-191)
2026-08-03 — Fixed three ways the alarm system misreported itself: a proof of a working backup that could vanish on a restart (seen happening), a release that emailed a failure for a release that worked, and a released binary nobody could rebuild and check. (R-189, R-188, R-186)

9.1 KiB Raw Blame History