hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.
- identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
- ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
- CountCurrentEscrowWithIdentity is the census of who the fix protects.
- Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
- Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
the pre-existing retention test stayed green for two months asserting the mechanism.
R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.
R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.
R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.
Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
This commit is contained in:
@@ -20,40 +20,33 @@ Proven end to end on real hardware.
|
||||
|
||||
## What's broken
|
||||
|
||||
- **Rebuilding a machine throws away its off-site backup HISTORY — and that is worse than we thought
|
||||
yesterday.** We spent today establishing what a rebuild actually destroys, and the answer changes the
|
||||
picture. There are three secrets involved. The one we have been talking about — the password to the
|
||||
storage provider — is the one you can re-issue any time, and we did. **The one that matters is the
|
||||
key that encrypts the backups themselves. The machine invents that key, and a rebuilt machine
|
||||
invents a brand-new one**, so the old backups can no longer be opened by the machine that wrote
|
||||
them.
|
||||
**Both demo machines did exactly this, and one of them did it silently.** demo-hp lost access to
|
||||
15 backups (41 MB). demo-felhom — the one we called lucky, because its off-site copy came back on
|
||||
its own in 76 seconds — **lost access to 36 backups (1.14 GB), and nothing said a word for
|
||||
thirteen hours.** The luck restored the delivery, not the history.
|
||||
Nothing has been deleted, and this was measured without touching either machine or any password.
|
||||
**The next scheduled off-site run is tomorrow at 04:15, and it will refuse rather than quietly start
|
||||
a new history** — the machine already knows how to recognise this and asks before resetting.
|
||||
**There is a decision here for you — see "Waiting on you".** *(R-193, R-192, and two new items)*
|
||||
- **And the safety net we believed was under all of this is not there.** We traced the whole path today —
|
||||
from a dead machine to reopened backups — and it does not join up. The important part is not that
|
||||
several steps are manual; it is this: **the central hub keeps the old sealed key when a machine
|
||||
re-seals, and we found it keeps the wrong one.** It keeps the key for the whole-machine backups and
|
||||
**not** the key for the off-site file backups — the one this entire problem is about. So the moment
|
||||
a rebuilt machine is asked to make a new recovery code, which is exactly what it asks the customer to
|
||||
do, **the last copy of the old key is gone.** Both demo machines crossed that line yesterday morning.
|
||||
**This means the 51 orphaned backups were beyond reach even if you had kept the recovery codes** —
|
||||
which is a relief in one narrow sense and much worse in every other. It also means the customer is
|
||||
currently told, in Hungarian on their own screen, that their old backups "may later be restorable
|
||||
with the matching recovery code". That sentence is not true today. It is a small fix — one missing
|
||||
column — and it must land before anything else here. *(R-198)*
|
||||
- **Nothing in the recovery path has ever been performed.** Making a recovery code is proven, live,
|
||||
with a real customer. **Using one is not.** No sealed package has ever been handed back to a machine
|
||||
(the hub has the buttons for it; nothing anywhere presses them), no recovered key has ever been put
|
||||
back, no old backup store has ever been reopened, and no file has ever been restored from a recovered
|
||||
key. There is a form missing, a step missing, and a connection missing between two parts of the
|
||||
system. We designed the exercise that would prove it end to end — see "Waiting on you".
|
||||
*(R-199, R-200, R-201)*
|
||||
- **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key
|
||||
that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one, so the old
|
||||
backups can no longer be opened by the machine that wrote them. Both demo machines did this on 3–4
|
||||
August — demo-hp lost access to 15 backups (41 MB), demo-felhom to 36 (1.14 GB), and **the second one
|
||||
happened in silence for thirteen hours**. From today that silence is fixed (below) and the old key
|
||||
survives the ceremony (below), but **the rebuild itself still starts a fresh history**, and getting
|
||||
the old one back is a path nobody has walked. *(R-193)*
|
||||
- *(fixed 4 Aug)* ~~The safety net we believed was under all of this is not there.~~ **The central hub
|
||||
was keeping the wrong key, and the fix shipped the same day.** When a machine re-seals its backups
|
||||
under a new recovery code, the hub keeps the old sealed package — but it was keeping the one for the
|
||||
whole-machine backups and **not** the one for the off-site file backups, which is the whole point. And
|
||||
because the swap happens in one motion, **the act that destroyed the old key was the recovery-code
|
||||
ceremony itself** — the exact thing a rebuilt machine asks its customer to do, on a screen promising
|
||||
the old backups stay recoverable. Both keys are now kept. **What this cannot undo:** anything
|
||||
superseded before today is gone for good, which includes both demo machines' pre-4-August keys — so
|
||||
those 51 orphaned backups were beyond reach even if the recovery codes had been kept. *(R-198)*
|
||||
- **Nothing in the recovery path has ever been performed, and that has not changed today.** Making a
|
||||
recovery code is proven, live, with a real customer. **Using one is not.** No sealed package has ever
|
||||
been handed back to a machine, no recovered key has ever been put back, no old backup store has ever
|
||||
been reopened. Today's work makes the key **survive**; it does not build the path that hands it back.
|
||||
That is what the proof exercise is for. *(R-199, R-200, R-201)*
|
||||
- **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
|
||||
says the old backups may later be restorable with the matching recovery code. From today that is true
|
||||
for machines that re-seal from now on and **false for anything already orphaned** — and the machine
|
||||
cannot tell which case it is in without the recovery path above. We deliberately did **not** patch the
|
||||
sentence: a conditional promise that can still be wrong is worse on that screen than a vague one.
|
||||
*(R-202)*
|
||||
- *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
|
||||
and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
|
||||
asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
|
||||
@@ -96,9 +89,10 @@ Proven end to end on real hardware.
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** waiting on your answer about the backup key (above). Two small honesty fixes are ready to
|
||||
go the moment you say so — making a changed backup key raise an alarm, and correcting an email that
|
||||
currently states the opposite of what it measured. Neither needs a decision.
|
||||
- **Now:** the proof exercise — wipe a demo machine and recover it with a saved recovery code. It is
|
||||
designed and waiting on your go-ahead. Both honesty fixes shipped today: a changed backup key now
|
||||
raises an alarm on the day, and the email that stated the opposite of what it measured now describes
|
||||
what it actually saw.
|
||||
- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
|
||||
refuse and ask, rather than start fresh — but that is a prediction from the code, not something
|
||||
anyone has watched happen.
|
||||
@@ -109,24 +103,11 @@ Proven end to end on real hardware.
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **A real decision, and it is the important one on this page.** Every machine is going to be
|
||||
reinstalled at some point, and each reinstall throws away the key that opens its off-site backups.
|
||||
There are only two honest answers, and picking either is fine — picking neither is what leaves it
|
||||
where it is.
|
||||
**(a) Keep a copy of that key on the Proxmox host**, which survives a rebuild. The machinery for
|
||||
this already exists and is barely any work. The price: the key would then sit on a machine you have
|
||||
root on, so "only the customer's recovery code can open these backups" stops being true in the
|
||||
strict sense. It was never quite true anyway — you hold root everywhere — but this makes a real
|
||||
second copy, and copies are exactly what we are already uneasy about elsewhere.
|
||||
**(b) Leave the key sealed under the customer's recovery code**, accept that every rebuild starts a
|
||||
fresh off-site history, and make sure the system says so loudly on the day it happens instead of
|
||||
never. Old backups are kept, not deleted, but they keep consuming the customer's off-site space.
|
||||
**We recommend doing the "says so loudly" part immediately regardless** — it is cheap, it costs
|
||||
nothing else, and it is the piece whose absence let a gigabyte of history go quiet for thirteen
|
||||
hours. What we are asking you to choose is (a). *(R-193; the full reasoning is in the findings
|
||||
document)*
|
||||
**Correction to (b), from today:** "old backups are kept" is true, but "openable later with the
|
||||
recovery code" is **not** — see R-198 above. Option (b) is only honest once that is fixed.
|
||||
- *(decided 4 Aug — recorded here until the work it unblocks is done)* You chose **not** to keep a
|
||||
copy of the backup key on the Proxmox host. That is a coherent choice and it has a consequence worth
|
||||
restating: **the customer's own recovery code is now the only route back from a rebuild**, so the
|
||||
sealed package the hub keeps is the single copy. Today's fix is what makes that true rather than
|
||||
wishful — until this morning the ceremony destroyed it. *(R-193)*
|
||||
- **Your ruling about the recovery screen has been priced, and it can be built.** You said: a freshly
|
||||
installed machine that finds a sealed package waiting should say so loudly, offer the customer a box
|
||||
to type their recovery code into, and show what would come back before doing anything. **All of it is
|
||||
@@ -146,11 +127,12 @@ Proven end to end on real hardware.
|
||||
marked file comes back byte for byte. About half a day, of which under an hour needs you. The failure
|
||||
we are watching for is precise: **if the recovered machine reports one backup instead of the ones we
|
||||
put there, it started a fresh history and the proof failed** — "the store opened" is not good enough.
|
||||
It costs that machine's contents if it goes wrong, which is acceptable there. **Do R-198 first**, or
|
||||
the exercise walks a path that is missing a piece. *(R-201)*
|
||||
- **The orphaned backups still on the storage box.** About 1.2 GB across the two demo machines, sitting
|
||||
in set-aside stores nobody can open and nothing prunes, using the 50 GB allowance. We did not touch
|
||||
them. **Delete, or leave?** *(R-193)*
|
||||
It costs that machine's contents if it goes wrong, which is acceptable there. **You said yes, after
|
||||
R-198 — and R-198 shipped this morning, so this is ready to run on your word.** *(R-201)*
|
||||
- **The orphaned backups on the storage box — you said delete, and it is still owed.** About 1.2 GB
|
||||
across the two demo machines, in set-aside stores nobody can open and nothing prunes. Deliberately not
|
||||
done today: erasing data on the storage endpoint is not something that should ride along with a
|
||||
database change, so it wants its own session. *(R-193)*
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
||||
@@ -160,11 +142,11 @@ Proven end to end on real hardware.
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-04 (latest)** — Traced the whole recovery path end to end and it does not join up. The hub
|
||||
keeps the wrong sealed key when a machine re-seals, so a rebuilt machine's old off-site key is lost the
|
||||
moment the customer is asked for a new recovery code. Nothing in the recovery path has ever been
|
||||
performed. Your ruling about the recovery screen is priced and buildable. The proof exercise is
|
||||
designed and awaiting your go-ahead. *(R-198, R-199, R-200, R-201, R-193)*
|
||||
- **2026-08-04 (latest)** — **Fixed the worst of it.** The hub now keeps the off-site backup key when a
|
||||
machine re-seals, instead of only the whole-machine one — four lines of database change that stop the
|
||||
recovery ceremony from destroying the thing it exists to protect. A changed backup key now raises an
|
||||
alarm the same day. The daily email that described the opposite of what it measured now states what it
|
||||
saw, and a refusal to auto-repair leaves a record instead of silence. *(R-198, R-197, R-192, R-196)*
|
||||
- **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage
|
||||
password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost
|
||||
access to their off-site history, one of them silently. A decision is now waiting on you. The daily
|
||||
|
||||
Reference in New Issue
Block a user