docs: P6 EXECUTED — containerd store moved to SSD2; reboot deferred (R-209, R-209a)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Operator ruled "proceed" after reading the pre-analysis; the storageReserved condition CC made the move conditional on was applied with it. The pre-analysis is retained verbatim as §11-bis so the ruling can be audited against what it was given. MOVED WITH ZERO LOSS, verified BEFORE the original was touched on four independent observables: 550,891 = 550,891 filesystem entries; 448 = 448 trusted.overlay xattrs (load-bearing — overlayfs stacking rides them, which is why rsync -X is not optional); 37,243 = 37,243 hardlinks; byte-identical meta.db sha256. Verified again after: identical image/tag/volume ID SETS, cache 2.782 GB / 38 records, ceiling still 30 GiB, pg 4 DBs / 31 tables / 175,135,767 B, redis DBSIZE 2437. End-to-end proof was a real build on the relocated store, rc=0 — not an inventory count. k3s was never at risk, and that was established BEFORE stopping anything: it runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and ~160 pods stayed up. Only the two jarr-* dev containers were affected. TRAP FOUND WHILE PROVING THE GUARD, and it is the reusable part: RequiresMountsFor on a path with NO mount unit is a SILENT NO-OP — containerd started normally against an absent-but-unmounted path. A typo'd guard buys nothing and says nothing: the built-but-never-wired shape again. The guard was therefore verified positively at the unit level (Requires= AND After=mnt-ssd_2.mount on both units), and refusal proven with a genuinely absent DEVICE via a temporary synthetic .mount unit — /mnt/ssd_2 hosts 12 live Longhorn replicas and must never be unmounted, and editing fstab on a production host risks emergency mode at boot. Result: "Job containerd.service/start failed with result 'dependency'", is-active inactive. It refused rather than creating an empty store on the root disk. storageReserved on SSD2 0 -> 80 GB; still Schedulable=True at 76.34%. NOT PROVEN, AND NOT GLOSSED (R-209a): the operator ruled DO NOT REBOOT, so the move has not survived a reboot — which is this task's own standard for validated. The MECHANISM is proven; the CONSEQUENCE (that a real boot mounts /mnt/ssd_2 before containerd starts, in this host's actual ordering) is not, and mount-ordering reasoning is exactly the class this project has been burned by. So: the pre-move tree stays MOVED ASIDE, not deleted (34.3 GB, which is why / sits at 54% and not lower), and felhom-store-postboot-check.service — oneshot, enabled, dry-run PASS at install — validates it automatically at the next boot, asserting positively that /var/lib/containerd does NOT exist. Read /var/log/felhom-store-postboot-check.log after the next boot; on PASS the rollback copy can go and ~34 GB returns. Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).
This commit is contained in:
@@ -56,6 +56,13 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
|
||||
quietly around it: the warning meant to catch exactly this **could never fire** (fixed and proved),
|
||||
and the weekly cleanup is still forbidden from touching the thing that grows (next session).
|
||||
*(R-205 … R-211)*
|
||||
- **2026-08-05** — **The build data now lives on the second SSD, moved with nothing lost.** All 345
|
||||
images, every saved volume and both development databases came through identical — checked before
|
||||
the original was touched and again afterwards, and confirmed by running a real build on the moved
|
||||
copy. **The server's own services never went down:** Gitea, the registry, the hub and the backup
|
||||
system run on a separate system and stayed up throughout. The second SSD now also **reserves 80 GB**
|
||||
for this, so the storage layer can no longer quietly claim the space and repeat what happened to the
|
||||
first disk. The old copy is kept as the way back until the machine next restarts. *(R-209)*
|
||||
- **2026-08-05** — **Three of the four recovery crutches removed.** The reset code works first time;
|
||||
a credential re-issue no longer blocks off-site backups on a healthy machine; and the default
|
||||
restore no longer quietly returns the wrong thing. *(R-204 items 1–3, R-196)*
|
||||
@@ -79,14 +86,14 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **Whether to move the build data onto the second SSD at all — and CC's advice is now "probably
|
||||
not".** You ruled "cap it, then move it". The cap is in and it did the job on its own: the disk sits
|
||||
at 54% with 199 GB free, and moving is no longer a rescue. The second SSD looks roomy (91% free) and
|
||||
today the move is safe by a wide margin — but it is the **only** disk of the four with **no space
|
||||
reserved for itself**, and its volumes are allowed to claim 6.9× more than they currently use. If
|
||||
they ever grow into that, the move would push it below the same floor that just took the first disk
|
||||
out of service. **If you want the move, do it together with reserving ~80 GB on that disk; otherwise
|
||||
leaving it where it is costs nothing now.** *(R-209)*
|
||||
- **One thing to read after the machine next restarts — and nothing to do until then.** You told me
|
||||
not to restart DooPlex, so I did not, and the move to the second SSD has therefore never been
|
||||
through a restart. It works right now and nothing was lost, but a restart is the one test that
|
||||
matters for this kind of change, and it has not happened. **I made it check itself:** whenever the
|
||||
machine next starts, for any reason, it writes a plain PASS or FAIL line to
|
||||
`/var/log/felhom-store-postboot-check.log`. **If it says PASS, the old copy can be deleted and 34 GB
|
||||
comes back.** Until then I have deliberately kept that old copy, which is the only quick way back —
|
||||
it is why the disk sits at 54% rather than lower. *(R-209a)*
|
||||
- **One list to rule on: 193 old images that exist only on this machine.** 131 controller versions and
|
||||
62 hub versions are not in the registry, so they cannot be re-downloaded — all of them old
|
||||
(controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing
|
||||
|
||||
Reference in New Issue
Block a user