docs: P6 EXECUTED — containerd store moved to SSD2; reboot deferred (R-209, R-209a)
gates / gates (push) Successful in 7s

Operator ruled "proceed" after reading the pre-analysis; the storageReserved
condition CC made the move conditional on was applied with it. The pre-analysis
is retained verbatim as §11-bis so the ruling can be audited against what it
was given.

MOVED WITH ZERO LOSS, verified BEFORE the original was touched on four
independent observables: 550,891 = 550,891 filesystem entries; 448 = 448
trusted.overlay xattrs (load-bearing — overlayfs stacking rides them, which is
why rsync -X is not optional); 37,243 = 37,243 hardlinks; byte-identical
meta.db sha256. Verified again after: identical image/tag/volume ID SETS,
cache 2.782 GB / 38 records, ceiling still 30 GiB, pg 4 DBs / 31 tables /
175,135,767 B, redis DBSIZE 2437. End-to-end proof was a real build on the
relocated store, rc=0 — not an inventory count.

k3s was never at risk, and that was established BEFORE stopping anything: it
runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and
~160 pods stayed up. Only the two jarr-* dev containers were affected.

TRAP FOUND WHILE PROVING THE GUARD, and it is the reusable part:
RequiresMountsFor on a path with NO mount unit is a SILENT NO-OP — containerd
started normally against an absent-but-unmounted path. A typo'd guard buys
nothing and says nothing: the built-but-never-wired shape again. The guard was
therefore verified positively at the unit level (Requires= AND
After=mnt-ssd_2.mount on both units), and refusal proven with a genuinely
absent DEVICE via a temporary synthetic .mount unit — /mnt/ssd_2 hosts 12 live
Longhorn replicas and must never be unmounted, and editing fstab on a
production host risks emergency mode at boot. Result:
"Job containerd.service/start failed with result 'dependency'", is-active
inactive. It refused rather than creating an empty store on the root disk.

storageReserved on SSD2 0 -> 80 GB; still Schedulable=True at 76.34%.

NOT PROVEN, AND NOT GLOSSED (R-209a): the operator ruled DO NOT REBOOT, so the
move has not survived a reboot — which is this task's own standard for
validated. The MECHANISM is proven; the CONSEQUENCE (that a real boot mounts
/mnt/ssd_2 before containerd starts, in this host's actual ordering) is not,
and mount-ordering reasoning is exactly the class this project has been burned
by. So: the pre-move tree stays MOVED ASIDE, not deleted (34.3 GB, which is why
/ sits at 54% and not lower), and felhom-store-postboot-check.service — oneshot,
enabled, dry-run PASS at install — validates it automatically at the next boot,
asserting positively that /var/lib/containerd does NOT exist. Read
/var/log/felhom-store-postboot-check.log after the next boot; on PASS the
rollback copy can go and ~34 GB returns.

Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).
This commit is contained in:
2026-08-05 10:27:40 +02:00
parent ee9d9bf203
commit c917251eeb
3 changed files with 198 additions and 12 deletions
+15 -8
View File
@@ -56,6 +56,13 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
quietly around it: the warning meant to catch exactly this **could never fire** (fixed and proved),
and the weekly cleanup is still forbidden from touching the thing that grows (next session).
*(R-205 … R-211)*
- **2026-08-05** — **The build data now lives on the second SSD, moved with nothing lost.** All 345
images, every saved volume and both development databases came through identical — checked before
the original was touched and again afterwards, and confirmed by running a real build on the moved
copy. **The server's own services never went down:** Gitea, the registry, the hub and the backup
system run on a separate system and stayed up throughout. The second SSD now also **reserves 80 GB**
for this, so the storage layer can no longer quietly claim the space and repeat what happened to the
first disk. The old copy is kept as the way back until the machine next restarts. *(R-209)*
- **2026-08-05** — **Three of the four recovery crutches removed.** The reset code works first time;
a credential re-issue no longer blocks off-site backups on a healthy machine; and the default
restore no longer quietly returns the wrong thing. *(R-204 items 13, R-196)*
@@ -79,14 +86,14 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
## Waiting on you
- **Whether to move the build data onto the second SSD at all — and CC's advice is now "probably
not".** You ruled "cap it, then move it". The cap is in and it did the job on its own: the disk sits
at 54% with 199 GB free, and moving is no longer a rescue. The second SSD looks roomy (91% free) and
today the move is safe by a wide margin — but it is the **only** disk of the four with **no space
reserved for itself**, and its volumes are allowed to claim 6.9× more than they currently use. If
they ever grow into that, the move would push it below the same floor that just took the first disk
out of service. **If you want the move, do it together with reserving ~80 GB on that disk; otherwise
leaving it where it is costs nothing now.** *(R-209)*
- **One thing to read after the machine next restarts — and nothing to do until then.** You told me
not to restart DooPlex, so I did not, and the move to the second SSD has therefore never been
through a restart. It works right now and nothing was lost, but a restart is the one test that
matters for this kind of change, and it has not happened. **I made it check itself:** whenever the
machine next starts, for any reason, it writes a plain PASS or FAIL line to
`/var/log/felhom-store-postboot-check.log`. **If it says PASS, the old copy can be deleted and 34 GB
comes back.** Until then I have deliberately kept that old copy, which is the only quick way back —
it is why the disk sits at 54% rather than lower. *(R-209a)*
- **One list to rule on: 193 old images that exist only on this machine.** 131 controller versions and
62 hub versions are not in the registry, so they cannot be re-downloaded — all of them old
(controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing