admin
d692c46db0
gates / gates (push) Successful in 7s
Four SCHEDULED runs, none triggered by hand: demo-felhom host 83.8s / offsite 540.4s; demo-hp host 109.3s / offsite 300.1s. Each restored into a scratch guest, booted, verified and destroyed itself; zero 990000 guests or volumes afterwards and both local-lvm figures returned to their pre-run values. Both boxes had BOTH tiers due at once, so R-86's ordering was observed live for the first time: never-proven sorted first, each box took its HOST tier, deferred the offsite one, and picked it up on the next evaluation six hours later. The host-tier proofs reached the hub through R-189's merge — demo-felhom's report carries two tiers, and the local one can only have come from disk. The capability map's optimistic half is cashed, with its scope stated: these two boxes, not the fleet. Surfaced and filed rather than fixed: - R-190: a storage ACL that demonstrably worked at 04:44 was gone by 09:24, with a reinstall, any logged pveum activity and any cluster-log entry ruled out. - R-191: every weekly offsite backup uploads successfully and then fails the job on a prune the box is deliberately not allowed to do (R-89 moved it server-side; both boxes still arm keep_last=2). Two corrections to yesterday's record: the R-185 drift DID surface as 403s on the write path (six, with the hub raising whole_guest_backup_failed at the first), and my earlier "no restore_test_* events" was produced by grepping a 404 page.
86 lines
5.5 KiB
Markdown
86 lines
5.5 KiB
Markdown
# STATUS — what works, what's broken, what's next
|
|
|
|
**Updated 2026-08-03.**
|
|
|
|
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
|
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
|
> which is technical state written for Claude Code — keep the two separate. **Maintenance:** update
|
|
> at the end of every session in which something shipped, broke, or was decided. One screen; cut
|
|
> items rather than extend it.
|
|
|
|
## What works right now
|
|
|
|
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
|
|
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
|
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
|
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
|
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
|
|
box six times, everything returned every time, and an app switched off deliberately stayed off.
|
|
Proven end to end on real hardware.
|
|
|
|
## What's broken
|
|
|
|
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
|
|
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
|
|
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
|
|
setting per machine fixes it. *(R-191)*
|
|
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
|
also delete it. A daily snapshot is armed as a stopgap.
|
|
*(R-95, R-87)*
|
|
|
|
## What shipped recently
|
|
|
|
- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
|
|
demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
|
|
checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
|
|
had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
|
|
never been tested on either machine. Both also proved their off-site copy on the same night, one
|
|
after the other rather than at once, which is the machine deciding for itself what to do first.
|
|
*(closes the last open half of R-86/R-185)*
|
|
- **A backup copy the machine was never allowed to read — and could not tell you about.** One demo
|
|
machine kept its whole-machine backups on a dedicated storage area the agent had never been granted
|
|
permission to read. Asked what was there it was told "nothing", while an administrator saw three
|
|
backups. **The permission was one command; the silence was the real fault** — a storage that answers
|
|
"nothing" looks exactly like a brand-new one, which is a normal, healthy state, so that copy had
|
|
never been test-restored and nothing had ever mentioned it. The machine now checks whether it is
|
|
allowed to read each copy it depends on and says so when it is not — the alert reached you by email
|
|
before the permission was granted, which is the whole point. **Both demo machines carried it and
|
|
both are fixed**, and new machines no longer inherit it. *(R-185)*
|
|
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
|
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
|
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
|
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
|
|
written to disk with the result and survives. This was caught happening, not predicted: a real
|
|
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
|
|
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
|
|
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
|
|
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
|
|
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
|
|
|
|
|
|
## What we're working on
|
|
|
|
- **Now:** nothing outstanding.
|
|
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
|
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
|
*(R-87)*
|
|
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
|
|
|
|
## Waiting on you
|
|
|
|
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
|
session log; nothing suggests anyone else saw it. *(R-132)*
|
|
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
|
machines to install, only which ones exist. Closing that needs either a password given to the build
|
|
server or a check inside the hub itself. *(R-184)*
|
|
- **Nothing else.**
|
|
|
|
## Changed since last update
|
|
|
|
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
|
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
|
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
|
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
|
|
that could vanish on a restart (seen happening), a release that emailed a failure for a release
|
|
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
|