docs: R-193 spike report, operator STATUS, and the ROADMAP R-1 scope correction (F3 covers HOST re-enroll, not a guest rebuild)
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# STATUS — what works, what's broken, what's next
|
||||
|
||||
**Updated 2026-08-03.**
|
||||
**Updated 2026-08-04.**
|
||||
|
||||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
||||
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
||||
@@ -20,10 +20,21 @@ Proven end to end on real hardware.
|
||||
|
||||
## What's broken
|
||||
|
||||
- **Rebuilding a machine silently takes away its off-site app-data backup.** Fixed on demo-hp on
|
||||
4 August (re-issued, picked up in 15 seconds, same repository, you escrowed the key), but the
|
||||
underlying fault stands: the password is delivered once and a rebuilt machine cannot ask again.
|
||||
The daily email about it also tells the wrong story. *(R-193, R-192)*
|
||||
- **Rebuilding a machine throws away its off-site backup HISTORY — and that is worse than we thought
|
||||
yesterday.** We spent today establishing what a rebuild actually destroys, and the answer changes the
|
||||
picture. There are three secrets involved. The one we have been talking about — the password to the
|
||||
storage provider — is the one you can re-issue any time, and we did. **The one that matters is the
|
||||
key that encrypts the backups themselves. The machine invents that key, and a rebuilt machine
|
||||
invents a brand-new one**, so the old backups can no longer be opened by the machine that wrote
|
||||
them.
|
||||
**Both demo machines did exactly this, and one of them did it silently.** demo-hp lost access to
|
||||
15 backups (41 MB). demo-felhom — the one we called lucky, because its off-site copy came back on
|
||||
its own in 76 seconds — **lost access to 36 backups (1.14 GB), and nothing said a word for
|
||||
thirteen hours.** The luck restored the delivery, not the history.
|
||||
Nothing has been deleted, and this was measured without touching either machine or any password.
|
||||
**The next scheduled off-site run is tomorrow at 04:15, and it will refuse rather than quietly start
|
||||
a new history** — the machine already knows how to recognise this and asks before resetting.
|
||||
**There is a decision here for you — see "Waiting on you".** *(R-193, R-192, and two new items)*
|
||||
- *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
|
||||
and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
|
||||
asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
|
||||
@@ -34,6 +45,14 @@ Proven end to end on real hardware.
|
||||
|
||||
## What shipped recently
|
||||
|
||||
- **The daily error email about David is gone.** David's record was created when he said he would try
|
||||
Felhom; no machine was ever attached to it, so nothing has ever been expected of it — yet every
|
||||
morning at five the system reported that a database backup had not happened. **A customer with no
|
||||
machine ever attached now produces no alarm.** A machine that *was* attached and went quiet still
|
||||
does, because that is a real fault, and there is a test whose whole job is to make sure a future
|
||||
change cannot take that away. David's record was not touched — the record was right; the alarm was
|
||||
wrong. **The first quiet morning will be 5 August, so this is not confirmed yet.** *(R-195)*
|
||||
|
||||
- **A permission the machine needs can now repair itself — and it tells you it had to.** Yesterday we
|
||||
found a permission that vanished with no reinstall and no recorded action; the machine now notices,
|
||||
restores it through the same controlled helper that grants it in the first place, and **raises an
|
||||
@@ -58,7 +77,12 @@ Proven end to end on real hardware.
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** nothing outstanding.
|
||||
- **Now:** waiting on your answer about the backup key (above). Two small honesty fixes are ready to
|
||||
go the moment you say so — making a changed backup key raise an alarm, and correcting an email that
|
||||
currently states the opposite of what it measured. Neither needs a decision.
|
||||
- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
|
||||
refuse and ask, rather than start fresh — but that is a prediction from the code, not something
|
||||
anyone has watched happen.
|
||||
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
||||
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
||||
*(R-87)*
|
||||
@@ -66,6 +90,22 @@ Proven end to end on real hardware.
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **A real decision, and it is the important one on this page.** Every machine is going to be
|
||||
reinstalled at some point, and each reinstall throws away the key that opens its off-site backups.
|
||||
There are only two honest answers, and picking either is fine — picking neither is what leaves it
|
||||
where it is.
|
||||
**(a) Keep a copy of that key on the Proxmox host**, which survives a rebuild. The machinery for
|
||||
this already exists and is barely any work. The price: the key would then sit on a machine you have
|
||||
root on, so "only the customer's recovery code can open these backups" stops being true in the
|
||||
strict sense. It was never quite true anyway — you hold root everywhere — but this makes a real
|
||||
second copy, and copies are exactly what we are already uneasy about elsewhere.
|
||||
**(b) Leave the key sealed under the customer's recovery code**, accept that every rebuild starts a
|
||||
fresh off-site history, and make sure the system says so loudly on the day it happens instead of
|
||||
never. Old backups are kept, not deleted, but they keep consuming the customer's off-site space.
|
||||
**We recommend doing the "says so loudly" part immediately regardless** — it is cheap, it costs
|
||||
nothing else, and it is the piece whose absence let a gigabyte of history go quiet for thirteen
|
||||
hours. What we are asking you to choose is (a). *(R-193; the full reasoning is in the findings
|
||||
document)*
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
||||
@@ -75,6 +115,10 @@ Proven end to end on real hardware.
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage
|
||||
password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost
|
||||
access to their off-site history, one of them silently. A decision is now waiting on you. The daily
|
||||
false alarm about David is fixed. *(R-193, R-195)*
|
||||
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
||||
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
||||
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
||||
|
||||
Reference in New Issue
Block a user