R-178 CLOSED: both demo boxes reinstalled from the merged golden and proven
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
Two boxes, two DIFFERENT supply paths, so the session proved the disk shape and the delivery route rather than one of them twice. demo-hp (layout proof, --golden <local volid>): mp0 at /var/lib/felhom, backup=1, 70G, no mp1; /var/lib/docker and /mnt/sys_drive both real mounts of its subdirectories via fstab; one df figure and one device id (64519) on all three paths; reboots 3/3 with the binds surviving each. demo-felhom (pipeline proof, --force-gitea-golden): fetch_verify succeeding against the vouched manifest for BOTH artifacts -- 'verified sha256 54e2a4c431daf580... matches the hub manifest' for the golden, a7763d31... for the agent. 250G single volume, grep -c '^mp1:' = 0, reboots 3/3. Journey proven on both, endpoint-level: claim -> deploy -> back up -> restore, with a planted marker returning byte-identical on each box. Ceiling measured gone: 65 GiB and 233 GiB available to a recovery unit, against 19 and 45. R-165 -> IMPLEMENTED, not PROVEN-LIVE, on the operator's ruling. B2, which that row records as the bulkhead's replacement, fired live for the first time and does refuse per app, delete nothing and alert -- but it is checked only in captureAllRecoveryUnits while runVolumeDumps writes the bulk unguarded, and its 'the previous unit is untouched' claim was measured false (182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). -> R-181. New: R-179 (uninstall leaves NAS network-storage units), R-180 (--archive-storage not cross-checked against the ACL grant; 403 at step 8/8 after root@pam is rotated), R-181. Third instance of R-115 recorded (agent 0.120.0 unpublished). No code written, no version bumps -- this was a runbook.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# STATUS — what works, what's broken, what's next
|
||||
|
||||
**Updated 2026-08-02.**
|
||||
**Updated 2026-08-03.**
|
||||
|
||||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
||||
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
||||
@@ -31,9 +31,12 @@ also delete it. A daily snapshot is armed as a stopgap, and we have never restor
|
||||
the data would vanish on the next update. Two are fixed, the third is now clear to fix because it is
|
||||
installed nowhere. *(R-156)*
|
||||
|
||||
**Local backups get 20 GB while apps get 50 GB.** An app that outgrows the smaller space stops being
|
||||
backed up locally — and the off-site copy is made from the local one, so that stops too. Nothing is
|
||||
lost: the last good copy is kept intact. *(R-163)*
|
||||
**The reserve does not guard the step that actually fills the disk.** The rule that refuses a backup
|
||||
when space runs low is checked at the wrong moment: the big write happens first, unchecked, and only
|
||||
the small write after it is refused. So the thing meant to stop a runaway backup is the thing it
|
||||
runs past. Worse, when it does refuse it says *"your last good copy is untouched"* — and we measured
|
||||
that copy being overwritten by the earlier step anyway. Found by deliberately filling a rebuilt demo
|
||||
machine. Nothing was deleted and you were emailed, both correctly. *(R-181)*
|
||||
|
||||
**When that happens, only one page says so** — no email, no alert. The page that answers "is this app
|
||||
backed up?" is the one that stays silent. *(R-158)*
|
||||
@@ -65,9 +68,18 @@ barrier against a runaway backup filling the space the machine needs to run; put
|
||||
first means that when it comes down, the thing watching is already working and already tested.
|
||||
*(R-167, R-158)*
|
||||
|
||||
**The backup partition is gone from the base image.** A machine built from now on has one storage area
|
||||
instead of two, so a backup can use whatever space the machine actually has free rather than a fixed
|
||||
slice decided when it was built. The wall does not move; it stops existing.
|
||||
**The backup partition is gone — and both demo machines now run on the new shape.** Each was wiped
|
||||
and rebuilt from the new base image on 3 August and taken through the whole journey a customer takes:
|
||||
set the machine up, install an app, back it up, restore it. One storage area instead of two, and the
|
||||
space a backup can use went from 19 GB to 65 GB on the small machine and from 45 GB to 233 GB on the
|
||||
big one. The wall does not move; it stops existing. Each machine was rebooted three times over and
|
||||
came back correctly every time. **The two machines were rebuilt deliberately differently** — the
|
||||
first from a copy of the image held locally, the second by the ordinary route a real customer takes,
|
||||
fetching the image and checking it against the fingerprint we publish, so both the disk shape and the
|
||||
delivery route are now proven rather than one twice.
|
||||
|
||||
**Their previous demo apps and data are gone.** That was the point of a wipe and you approved it; the
|
||||
machines now carry a couple of small test apps instead.
|
||||
|
||||
**What replaced the wall.** It was quietly doing a second job — keeping a runaway backup from eating
|
||||
the space the machine needs to keep running. That job is now explicit: if a backup would push the disk
|
||||
@@ -81,32 +93,42 @@ quietly broke: one put your backups inside Docker's own storage, where the norma
|
||||
Docker would wipe them; another exposed all of Docker's internals to the part of the system that
|
||||
manages your drives. The third does neither, and costs one extra line of configuration.
|
||||
|
||||
**Nothing has changed on any existing machine.** They keep their current layout and go on working
|
||||
exactly as before; they get the new shape only when they are reinstalled. The new base image is
|
||||
deliberately **not switched on yet** — nothing will pick it up until a machine has been rebuilt from
|
||||
it and checked, which is the next step. *(R-165, R-163)*
|
||||
**The new base image is now switched on**, so any machine installed from here on gets the new shape.
|
||||
The tester's box is untouched and keeps working exactly as before; it gets the new shape whenever it
|
||||
is reinstalled. *(R-165, R-178)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** the last app whose data was never saved; today's decisions written down.
|
||||
- **Next:** finishing the partition merge — **the build is done and the decision is made**; what is
|
||||
left is to reinstall the two demo machines from the new base image and check one end to end.
|
||||
- **Next:** fixing the reserve so it guards the step that fills the disk, and stopping it from
|
||||
claiming your last good copy is untouched when it is not *(R-181)*. The partition merge itself is
|
||||
done and proven on both machines.
|
||||
- **After:** rebuilding how the machine records whether an app is meant to be running.
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **How a new version reaches a machine.** Pushing the installer publishes it — half a minute later
|
||||
every new machine downloads it, with no staging and no way back but another push. And publishing is
|
||||
a step we remember rather than one the release performs, forgotten twice: a fix can be live here
|
||||
and still not reach a new machine. Nothing is installing today, so this is the cheapest moment to
|
||||
settle both. *(R-110, R-115)*
|
||||
- **How a new version reaches a machine — and it has now been forgotten a third time.** Pushing the
|
||||
installer publishes it: half a minute later every new machine downloads it, with no staging and no
|
||||
way back but another push. And publishing is a step we remember rather than one the release
|
||||
performs. On 3 August the new agent — the half of the partition merge that runs on the machine —
|
||||
turned out to have been built and installed on both demo machines but **never published**, so a
|
||||
rebuild would have quietly put the *old* one back and proved a version nobody ships. Caught before
|
||||
the wipe and fixed in ten minutes, but only because someone happened to look. Nothing is installing
|
||||
today, so this is the cheapest moment to settle it. *(R-110, R-115)*
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **Nothing — both partition-merge questions are answered.** You chose the storage shape and the hard
|
||||
stop; both are built. The tester's box needs no conversion: it will simply be reinstalled. *(R-165)*
|
||||
- **Nothing on the partition merge itself** — the shape is chosen, built, and now proven on both demo
|
||||
machines. The tester's box needs no conversion: it will simply be reinstalled. What is left is the
|
||||
reserve defect above, which is ours to fix, not yours to decide. *(R-165, R-181)*
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-03** — Both demo machines wiped and rebuilt from the new base image, and taken through
|
||||
set-up → install an app → back it up → restore it. The backup space ceiling is gone and measured.
|
||||
The hard stop that replaced the old partition was fired for the first time on real hardware — it
|
||||
refused, it deleted nothing, it emailed you, **and it turned out to be watching the wrong step**;
|
||||
that is now the top thing to fix.
|
||||
|
||||
- **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in
|
||||
a mode where reading a page cannot block a machine's status update; a one-word difference meant that
|
||||
setting had **never taken effect**, for the hub's whole life. Fixed and verified live. **Also found:
|
||||
|
||||
Reference in New Issue
Block a user