admin
3f2b7bc023
gates / gates (push) Successful in 7s
Read-only recon of the escrow -> recovery chain, from a dead node to an open
repository. No production code, no build, no version bump.
Headline: the hub's superseded-escrow retention does NOT retain the offsite
repository password. host_escrow_superseded has no identity_blob column and
demoteCurrentEscrowTx copies only the K-escrow blob, so what survives a
supersession is the PBS datastore key, not the restic repo password. The next
escrow ceremony -- which the system tells a rebuilt box's customer to run --
destroys the last copy. Both demo boxes crossed that line on 2026-08-04.
Also established:
- the hub's blob-serving endpoints (re-enroll / restore-directive) have zero
callers anywhere: agent, hub UI, scripts, runbooks (R-199)
- POST /backup/offbox/inject-password is routed and handled but no template
contains the form (R-200)
- nothing in the recovery path has ever been exercised; the one live
round-trip proof (2026-06-10) predates the ResticRepoPassword field (R-201)
- a fail-closed mint refusal IS implementable: the report ACK already carries
escrow{identity_blob_present, restic_pw_sha256} and the controller discards
it whenever no offbox target exists
Corrections: yesterday's spike annotated (candidate (b) overturned in part --
unattended recovery is impossible, customer-present is not); capability-map
retention claim struck through and replaced with what the code does.
Deliverable: documentation/audits/RECON-offsite-dr-chain-2026-08-04.md
Register: new R-198..R-201; R-193 and R-192 updated; STATUS.md refreshed.
178 lines
14 KiB
Markdown
178 lines
14 KiB
Markdown
# STATUS — what works, what's broken, what's next
|
|
|
|
**Updated 2026-08-04.**
|
|
|
|
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
|
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
|
> which is technical state written for Claude Code — keep the two separate. **Maintenance:** update
|
|
> at the end of every session in which something shipped, broke, or was decided. One screen; cut
|
|
> items rather than extend it.
|
|
|
|
## What works right now
|
|
|
|
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
|
|
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
|
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
|
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
|
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
|
|
box six times, everything returned every time, and an app switched off deliberately stayed off.
|
|
Proven end to end on real hardware.
|
|
|
|
## What's broken
|
|
|
|
- **Rebuilding a machine throws away its off-site backup HISTORY — and that is worse than we thought
|
|
yesterday.** We spent today establishing what a rebuild actually destroys, and the answer changes the
|
|
picture. There are three secrets involved. The one we have been talking about — the password to the
|
|
storage provider — is the one you can re-issue any time, and we did. **The one that matters is the
|
|
key that encrypts the backups themselves. The machine invents that key, and a rebuilt machine
|
|
invents a brand-new one**, so the old backups can no longer be opened by the machine that wrote
|
|
them.
|
|
**Both demo machines did exactly this, and one of them did it silently.** demo-hp lost access to
|
|
15 backups (41 MB). demo-felhom — the one we called lucky, because its off-site copy came back on
|
|
its own in 76 seconds — **lost access to 36 backups (1.14 GB), and nothing said a word for
|
|
thirteen hours.** The luck restored the delivery, not the history.
|
|
Nothing has been deleted, and this was measured without touching either machine or any password.
|
|
**The next scheduled off-site run is tomorrow at 04:15, and it will refuse rather than quietly start
|
|
a new history** — the machine already knows how to recognise this and asks before resetting.
|
|
**There is a decision here for you — see "Waiting on you".** *(R-193, R-192, and two new items)*
|
|
- **And the safety net we believed was under all of this is not there.** We traced the whole path today —
|
|
from a dead machine to reopened backups — and it does not join up. The important part is not that
|
|
several steps are manual; it is this: **the central hub keeps the old sealed key when a machine
|
|
re-seals, and we found it keeps the wrong one.** It keeps the key for the whole-machine backups and
|
|
**not** the key for the off-site file backups — the one this entire problem is about. So the moment
|
|
a rebuilt machine is asked to make a new recovery code, which is exactly what it asks the customer to
|
|
do, **the last copy of the old key is gone.** Both demo machines crossed that line yesterday morning.
|
|
**This means the 51 orphaned backups were beyond reach even if you had kept the recovery codes** —
|
|
which is a relief in one narrow sense and much worse in every other. It also means the customer is
|
|
currently told, in Hungarian on their own screen, that their old backups "may later be restorable
|
|
with the matching recovery code". That sentence is not true today. It is a small fix — one missing
|
|
column — and it must land before anything else here. *(R-198)*
|
|
- **Nothing in the recovery path has ever been performed.** Making a recovery code is proven, live,
|
|
with a real customer. **Using one is not.** No sealed package has ever been handed back to a machine
|
|
(the hub has the buttons for it; nothing anywhere presses them), no recovered key has ever been put
|
|
back, no old backup store has ever been reopened, and no file has ever been restored from a recovered
|
|
key. There is a form missing, a step missing, and a connection missing between two parts of the
|
|
system. We designed the exercise that would prove it end to end — see "Waiting on you".
|
|
*(R-199, R-200, R-201)*
|
|
- *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
|
|
and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
|
|
asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
|
|
it every day since 27 July. Both machines and all future ones. *(R-191)*
|
|
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
|
also delete it. A daily snapshot is armed as a stopgap.
|
|
*(R-95, R-87)*
|
|
|
|
## What shipped recently
|
|
|
|
- **The daily error email about David is gone.** David's record was created when he said he would try
|
|
Felhom; no machine was ever attached to it, so nothing has ever been expected of it — yet every
|
|
morning at five the system reported that a database backup had not happened. **A customer with no
|
|
machine ever attached now produces no alarm.** A machine that *was* attached and went quiet still
|
|
does, because that is a real fault, and there is a test whose whole job is to make sure a future
|
|
change cannot take that away. David's record was not touched — the record was right; the alarm was
|
|
wrong. **The first quiet morning will be 5 August, so this is not confirmed yet.** *(R-195)*
|
|
|
|
- **A permission the machine needs can now repair itself — and it tells you it had to.** Yesterday we
|
|
found a permission that vanished with no reinstall and no recorded action; the machine now notices,
|
|
restores it through the same controlled helper that grants it in the first place, and **raises an
|
|
alert saying it was missing and was restored**. The alert is the point: a permission that silently
|
|
fixes itself is one that can be lost over and over and never investigated. Proven twice on a demo
|
|
machine by taking the permission away by hand. *(R-190 — the repair; why it vanished is still open)*
|
|
|
|
- **The on-machine backup copy has now been proved to restore — by the machines themselves**, both of
|
|
them, overnight and unasked (84 and 109 seconds). Every restore proof before this was of the
|
|
off-site copy. *(R-86, R-185)*
|
|
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
|
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
|
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
|
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
|
|
written to disk with the result and survives. This was caught happening, not predicted: a real
|
|
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
|
|
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
|
|
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
|
|
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
|
|
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
|
|
|
|
|
|
## What we're working on
|
|
|
|
- **Now:** waiting on your answer about the backup key (above). Two small honesty fixes are ready to
|
|
go the moment you say so — making a changed backup key raise an alarm, and correcting an email that
|
|
currently states the opposite of what it measured. Neither needs a decision.
|
|
- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
|
|
refuse and ask, rather than start fresh — but that is a prediction from the code, not something
|
|
anyone has watched happen.
|
|
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
|
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
|
*(R-87)*
|
|
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
|
|
|
|
## Waiting on you
|
|
|
|
- **A real decision, and it is the important one on this page.** Every machine is going to be
|
|
reinstalled at some point, and each reinstall throws away the key that opens its off-site backups.
|
|
There are only two honest answers, and picking either is fine — picking neither is what leaves it
|
|
where it is.
|
|
**(a) Keep a copy of that key on the Proxmox host**, which survives a rebuild. The machinery for
|
|
this already exists and is barely any work. The price: the key would then sit on a machine you have
|
|
root on, so "only the customer's recovery code can open these backups" stops being true in the
|
|
strict sense. It was never quite true anyway — you hold root everywhere — but this makes a real
|
|
second copy, and copies are exactly what we are already uneasy about elsewhere.
|
|
**(b) Leave the key sealed under the customer's recovery code**, accept that every rebuild starts a
|
|
fresh off-site history, and make sure the system says so loudly on the day it happens instead of
|
|
never. Old backups are kept, not deleted, but they keep consuming the customer's off-site space.
|
|
**We recommend doing the "says so loudly" part immediately regardless** — it is cheap, it costs
|
|
nothing else, and it is the piece whose absence let a gigabyte of history go quiet for thirteen
|
|
hours. What we are asking you to choose is (a). *(R-193; the full reasoning is in the findings
|
|
document)*
|
|
**Correction to (b), from today:** "old backups are kept" is true, but "openable later with the
|
|
recovery code" is **not** — see R-198 above. Option (b) is only honest once that is fixed.
|
|
- **Your ruling about the recovery screen has been priced, and it can be built.** You said: a freshly
|
|
installed machine that finds a sealed package waiting should say so loudly, offer the customer a box
|
|
to type their recovery code into, and show what would come back before doing anything. **All of it is
|
|
buildable, and one part is already free** — the hub is *already* telling every machine, on every
|
|
check-in, that a sealed package exists and which key it covers, and the machine currently throws that
|
|
message away. Showing a preview is also cheap: listing what is in an off-site store reads it without
|
|
writing to it, so the customer can see how many backups, from when, and for which apps before
|
|
committing. The real work is one new connection: the unsealing has to happen in the part that runs on
|
|
the Proxmox host, because the customer-facing part cannot do it — the same crossing your recovery-code
|
|
ceremony already makes in the other direction. **One thing for you to weigh, which we deliberately did
|
|
not decide:** a screen that takes a recovery code and then shows what is in a backup store is reachable
|
|
by anyone with the household's dashboard password, and the preview reveals backup dates and app names.
|
|
*(R-193)*
|
|
- **Should we run the proof?** Nothing about recovery has ever actually been done, so we designed the
|
|
exercise: on the spare demo machine, make a recovery code and **keep it**, put a marked file into an
|
|
app, back it up off-site, wipe the machine, reinstall, recover with the saved code, and check the
|
|
marked file comes back byte for byte. About half a day, of which under an hour needs you. The failure
|
|
we are watching for is precise: **if the recovered machine reports one backup instead of the ones we
|
|
put there, it started a fresh history and the proof failed** — "the store opened" is not good enough.
|
|
It costs that machine's contents if it goes wrong, which is acceptable there. **Do R-198 first**, or
|
|
the exercise walks a path that is missing a piece. *(R-201)*
|
|
- **The orphaned backups still on the storage box.** About 1.2 GB across the two demo machines, sitting
|
|
in set-aside stores nobody can open and nothing prunes, using the 50 GB allowance. We did not touch
|
|
them. **Delete, or leave?** *(R-193)*
|
|
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
|
session log; nothing suggests anyone else saw it. *(R-132)*
|
|
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
|
machines to install, only which ones exist. Closing that needs either a password given to the build
|
|
server or a check inside the hub itself. *(R-184)*
|
|
- **Nothing else.**
|
|
|
|
## Changed since last update
|
|
|
|
- **2026-08-04 (latest)** — Traced the whole recovery path end to end and it does not join up. The hub
|
|
keeps the wrong sealed key when a machine re-seals, so a rebuilt machine's old off-site key is lost the
|
|
moment the customer is asked for a new recovery code. Nothing in the recovery path has ever been
|
|
performed. Your ruling about the recovery screen is priced and buildable. The proof exercise is
|
|
designed and awaiting your go-ahead. *(R-198, R-199, R-200, R-201, R-193)*
|
|
- **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage
|
|
password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost
|
|
access to their off-site history, one of them silently. A decision is now waiting on you. The daily
|
|
false alarm about David is fixed. *(R-193, R-195)*
|
|
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
|
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
|
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
|
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
|
|
that could vanish on a restart (seen happening), a release that emailed a failure for a release
|
|
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
|