128 lines
9.1 KiB
Markdown
128 lines
9.1 KiB
Markdown
# STATUS — what works, what's broken, what's next
|
|
|
|
**Updated 2026-08-04.**
|
|
|
|
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
|
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
|
> which is technical state written for Claude Code — keep the two separate. **Maintenance:** update
|
|
> at the end of every session in which something shipped, broke, or was decided. One screen; cut
|
|
> items rather than extend it.
|
|
|
|
## What works right now
|
|
|
|
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
|
|
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
|
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
|
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
|
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
|
|
box six times, everything returned every time, and an app switched off deliberately stayed off.
|
|
Proven end to end on real hardware.
|
|
|
|
## What's broken
|
|
|
|
- **Rebuilding a machine throws away its off-site backup HISTORY — and that is worse than we thought
|
|
yesterday.** We spent today establishing what a rebuild actually destroys, and the answer changes the
|
|
picture. There are three secrets involved. The one we have been talking about — the password to the
|
|
storage provider — is the one you can re-issue any time, and we did. **The one that matters is the
|
|
key that encrypts the backups themselves. The machine invents that key, and a rebuilt machine
|
|
invents a brand-new one**, so the old backups can no longer be opened by the machine that wrote
|
|
them.
|
|
**Both demo machines did exactly this, and one of them did it silently.** demo-hp lost access to
|
|
15 backups (41 MB). demo-felhom — the one we called lucky, because its off-site copy came back on
|
|
its own in 76 seconds — **lost access to 36 backups (1.14 GB), and nothing said a word for
|
|
thirteen hours.** The luck restored the delivery, not the history.
|
|
Nothing has been deleted, and this was measured without touching either machine or any password.
|
|
**The next scheduled off-site run is tomorrow at 04:15, and it will refuse rather than quietly start
|
|
a new history** — the machine already knows how to recognise this and asks before resetting.
|
|
**There is a decision here for you — see "Waiting on you".** *(R-193, R-192, and two new items)*
|
|
- *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
|
|
and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
|
|
asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
|
|
it every day since 27 July. Both machines and all future ones. *(R-191)*
|
|
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
|
also delete it. A daily snapshot is armed as a stopgap.
|
|
*(R-95, R-87)*
|
|
|
|
## What shipped recently
|
|
|
|
- **The daily error email about David is gone.** David's record was created when he said he would try
|
|
Felhom; no machine was ever attached to it, so nothing has ever been expected of it — yet every
|
|
morning at five the system reported that a database backup had not happened. **A customer with no
|
|
machine ever attached now produces no alarm.** A machine that *was* attached and went quiet still
|
|
does, because that is a real fault, and there is a test whose whole job is to make sure a future
|
|
change cannot take that away. David's record was not touched — the record was right; the alarm was
|
|
wrong. **The first quiet morning will be 5 August, so this is not confirmed yet.** *(R-195)*
|
|
|
|
- **A permission the machine needs can now repair itself — and it tells you it had to.** Yesterday we
|
|
found a permission that vanished with no reinstall and no recorded action; the machine now notices,
|
|
restores it through the same controlled helper that grants it in the first place, and **raises an
|
|
alert saying it was missing and was restored**. The alert is the point: a permission that silently
|
|
fixes itself is one that can be lost over and over and never investigated. Proven twice on a demo
|
|
machine by taking the permission away by hand. *(R-190 — the repair; why it vanished is still open)*
|
|
|
|
- **The on-machine backup copy has now been proved to restore — by the machines themselves**, both of
|
|
them, overnight and unasked (84 and 109 seconds). Every restore proof before this was of the
|
|
off-site copy. *(R-86, R-185)*
|
|
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
|
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
|
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
|
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
|
|
written to disk with the result and survives. This was caught happening, not predicted: a real
|
|
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
|
|
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
|
|
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
|
|
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
|
|
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
|
|
|
|
|
|
## What we're working on
|
|
|
|
- **Now:** waiting on your answer about the backup key (above). Two small honesty fixes are ready to
|
|
go the moment you say so — making a changed backup key raise an alarm, and correcting an email that
|
|
currently states the opposite of what it measured. Neither needs a decision.
|
|
- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
|
|
refuse and ask, rather than start fresh — but that is a prediction from the code, not something
|
|
anyone has watched happen.
|
|
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
|
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
|
*(R-87)*
|
|
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
|
|
|
|
## Waiting on you
|
|
|
|
- **A real decision, and it is the important one on this page.** Every machine is going to be
|
|
reinstalled at some point, and each reinstall throws away the key that opens its off-site backups.
|
|
There are only two honest answers, and picking either is fine — picking neither is what leaves it
|
|
where it is.
|
|
**(a) Keep a copy of that key on the Proxmox host**, which survives a rebuild. The machinery for
|
|
this already exists and is barely any work. The price: the key would then sit on a machine you have
|
|
root on, so "only the customer's recovery code can open these backups" stops being true in the
|
|
strict sense. It was never quite true anyway — you hold root everywhere — but this makes a real
|
|
second copy, and copies are exactly what we are already uneasy about elsewhere.
|
|
**(b) Leave the key sealed under the customer's recovery code**, accept that every rebuild starts a
|
|
fresh off-site history, and make sure the system says so loudly on the day it happens instead of
|
|
never. Old backups are kept, not deleted, but they keep consuming the customer's off-site space.
|
|
**We recommend doing the "says so loudly" part immediately regardless** — it is cheap, it costs
|
|
nothing else, and it is the piece whose absence let a gigabyte of history go quiet for thirteen
|
|
hours. What we are asking you to choose is (a). *(R-193; the full reasoning is in the findings
|
|
document)*
|
|
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
|
session log; nothing suggests anyone else saw it. *(R-132)*
|
|
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
|
machines to install, only which ones exist. Closing that needs either a password given to the build
|
|
server or a check inside the hub itself. *(R-184)*
|
|
- **Nothing else.**
|
|
|
|
## Changed since last update
|
|
|
|
- **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage
|
|
password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost
|
|
access to their off-site history, one of them silently. A decision is now waiting on you. The daily
|
|
false alarm about David is fixed. *(R-193, R-195)*
|
|
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
|
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
|
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
|
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
|
|
that could vanish on a restart (seen happening), a release that emailed a failure for a release
|
|
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
|