R-178 CLOSED: both demo boxes reinstalled from the merged golden and proven
gates / gates (push) Successful in 8s

Two boxes, two DIFFERENT supply paths, so the session proved the disk shape and
the delivery route rather than one of them twice.

demo-hp (layout proof, --golden <local volid>): mp0 at /var/lib/felhom,
backup=1, 70G, no mp1; /var/lib/docker and /mnt/sys_drive both real mounts of
its subdirectories via fstab; one df figure and one device id (64519) on all
three paths; reboots 3/3 with the binds surviving each.

demo-felhom (pipeline proof, --force-gitea-golden): fetch_verify succeeding
against the vouched manifest for BOTH artifacts -- 'verified sha256
54e2a4c431daf580... matches the hub manifest' for the golden, a7763d31... for
the agent. 250G single volume, grep -c '^mp1:' = 0, reboots 3/3.

Journey proven on both, endpoint-level: claim -> deploy -> back up -> restore,
with a planted marker returning byte-identical on each box. Ceiling measured
gone: 65 GiB and 233 GiB available to a recovery unit, against 19 and 45.

R-165 -> IMPLEMENTED, not PROVEN-LIVE, on the operator's ruling. B2, which that
row records as the bulkhead's replacement, fired live for the first time and
does refuse per app, delete nothing and alert -- but it is checked only in
captureAllRecoveryUnits while runVolumeDumps writes the bulk unguarded, and its
'the previous unit is untouched' claim was measured false (182,272 B dump
replaced by 2,147,666,432 B under a manifest still dated 06:34:26). -> R-181.

New: R-179 (uninstall leaves NAS network-storage units), R-180 (--archive-storage
not cross-checked against the ACL grant; 403 at step 8/8 after root@pam is
rotated), R-181. Third instance of R-115 recorded (agent 0.120.0 unpublished).

No code written, no version bumps -- this was a runbook.
This commit is contained in:
2026-08-03 09:34:15 +02:00
parent bdd1a9d130
commit aa62449694
6 changed files with 424 additions and 92 deletions
+42 -20
View File
@@ -1,6 +1,6 @@
# STATUS — what works, what's broken, what's next
**Updated 2026-08-02.**
**Updated 2026-08-03.**
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
@@ -31,9 +31,12 @@ also delete it. A daily snapshot is armed as a stopgap, and we have never restor
the data would vanish on the next update. Two are fixed, the third is now clear to fix because it is
installed nowhere. *(R-156)*
**Local backups get 20 GB while apps get 50 GB.** An app that outgrows the smaller space stops being
backed up locally — and the off-site copy is made from the local one, so that stops too. Nothing is
lost: the last good copy is kept intact. *(R-163)*
**The reserve does not guard the step that actually fills the disk.** The rule that refuses a backup
when space runs low is checked at the wrong moment: the big write happens first, unchecked, and only
the small write after it is refused. So the thing meant to stop a runaway backup is the thing it
runs past. Worse, when it does refuse it says *"your last good copy is untouched"* — and we measured
that copy being overwritten by the earlier step anyway. Found by deliberately filling a rebuilt demo
machine. Nothing was deleted and you were emailed, both correctly. *(R-181)*
**When that happens, only one page says so** — no email, no alert. The page that answers "is this app
backed up?" is the one that stays silent. *(R-158)*
@@ -65,9 +68,18 @@ barrier against a runaway backup filling the space the machine needs to run; put
first means that when it comes down, the thing watching is already working and already tested.
*(R-167, R-158)*
**The backup partition is gone from the base image.** A machine built from now on has one storage area
instead of two, so a backup can use whatever space the machine actually has free rather than a fixed
slice decided when it was built. The wall does not move; it stops existing.
**The backup partition is gone — and both demo machines now run on the new shape.** Each was wiped
and rebuilt from the new base image on 3 August and taken through the whole journey a customer takes:
set the machine up, install an app, back it up, restore it. One storage area instead of two, and the
space a backup can use went from 19 GB to 65 GB on the small machine and from 45 GB to 233 GB on the
big one. The wall does not move; it stops existing. Each machine was rebooted three times over and
came back correctly every time. **The two machines were rebuilt deliberately differently** — the
first from a copy of the image held locally, the second by the ordinary route a real customer takes,
fetching the image and checking it against the fingerprint we publish, so both the disk shape and the
delivery route are now proven rather than one twice.
**Their previous demo apps and data are gone.** That was the point of a wipe and you approved it; the
machines now carry a couple of small test apps instead.
**What replaced the wall.** It was quietly doing a second job — keeping a runaway backup from eating
the space the machine needs to keep running. That job is now explicit: if a backup would push the disk
@@ -81,32 +93,42 @@ quietly broke: one put your backups inside Docker's own storage, where the norma
Docker would wipe them; another exposed all of Docker's internals to the part of the system that
manages your drives. The third does neither, and costs one extra line of configuration.
**Nothing has changed on any existing machine.** They keep their current layout and go on working
exactly as before; they get the new shape only when they are reinstalled. The new base image is
deliberately **not switched on yet** — nothing will pick it up until a machine has been rebuilt from
it and checked, which is the next step. *(R-165, R-163)*
**The new base image is now switched on**, so any machine installed from here on gets the new shape.
The tester's box is untouched and keeps working exactly as before; it gets the new shape whenever it
is reinstalled. *(R-165, R-178)*
## What we're working on
- **Now:** the last app whose data was never saved; today's decisions written down.
- **Next:** finishing the partition merge **the build is done and the decision is made**; what is
left is to reinstall the two demo machines from the new base image and check one end to end.
- **Next:** fixing the reserve so it guards the step that fills the disk, and stopping it from
claiming your last good copy is untouched when it is not *(R-181)*. The partition merge itself is
done and proven on both machines.
- **After:** rebuilding how the machine records whether an app is meant to be running.
## Waiting on you
- **How a new version reaches a machine.** Pushing the installer publishes it — half a minute later
every new machine downloads it, with no staging and no way back but another push. And publishing is
a step we remember rather than one the release performs, forgotten twice: a fix can be live here
and still not reach a new machine. Nothing is installing today, so this is the cheapest moment to
settle both. *(R-110, R-115)*
- **How a new version reaches a machine — and it has now been forgotten a third time.** Pushing the
installer publishes it: half a minute later every new machine downloads it, with no staging and no
way back but another push. And publishing is a step we remember rather than one the release
performs. On 3 August the new agent — the half of the partition merge that runs on the machine —
turned out to have been built and installed on both demo machines but **never published**, so a
rebuild would have quietly put the *old* one back and proved a version nobody ships. Caught before
the wipe and fixed in ten minutes, but only because someone happened to look. Nothing is installing
today, so this is the cheapest moment to settle it. *(R-110, R-115)*
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
session log; nothing suggests anyone else saw it. *(R-132)*
- **Nothing — both partition-merge questions are answered.** You chose the storage shape and the hard
stop; both are built. The tester's box needs no conversion: it will simply be reinstalled. *(R-165)*
- **Nothing on the partition merge itself** — the shape is chosen, built, and now proven on both demo
machines. The tester's box needs no conversion: it will simply be reinstalled. What is left is the
reserve defect above, which is ours to fix, not yours to decide. *(R-165, R-181)*
## Changed since last update
- **2026-08-03** — Both demo machines wiped and rebuilt from the new base image, and taken through
set-up → install an app → back it up → restore it. The backup space ceiling is gone and measured.
The hard stop that replaced the old partition was fired for the first time on real hardware — it
refused, it deleted nothing, it emailed you, **and it turned out to be watching the wrong step**;
that is now the top thing to fix.
- **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in
a mode where reading a page cannot block a machine's status update; a one-word difference meant that
setting had **never taken effect**, for the hub's whole life. Fixed and verified live. **Also found: