docs: P6 EXECUTED — containerd store moved to SSD2; reboot deferred (R-209, R-209a)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Operator ruled "proceed" after reading the pre-analysis; the storageReserved condition CC made the move conditional on was applied with it. The pre-analysis is retained verbatim as §11-bis so the ruling can be audited against what it was given. MOVED WITH ZERO LOSS, verified BEFORE the original was touched on four independent observables: 550,891 = 550,891 filesystem entries; 448 = 448 trusted.overlay xattrs (load-bearing — overlayfs stacking rides them, which is why rsync -X is not optional); 37,243 = 37,243 hardlinks; byte-identical meta.db sha256. Verified again after: identical image/tag/volume ID SETS, cache 2.782 GB / 38 records, ceiling still 30 GiB, pg 4 DBs / 31 tables / 175,135,767 B, redis DBSIZE 2437. End-to-end proof was a real build on the relocated store, rc=0 — not an inventory count. k3s was never at risk, and that was established BEFORE stopping anything: it runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and ~160 pods stayed up. Only the two jarr-* dev containers were affected. TRAP FOUND WHILE PROVING THE GUARD, and it is the reusable part: RequiresMountsFor on a path with NO mount unit is a SILENT NO-OP — containerd started normally against an absent-but-unmounted path. A typo'd guard buys nothing and says nothing: the built-but-never-wired shape again. The guard was therefore verified positively at the unit level (Requires= AND After=mnt-ssd_2.mount on both units), and refusal proven with a genuinely absent DEVICE via a temporary synthetic .mount unit — /mnt/ssd_2 hosts 12 live Longhorn replicas and must never be unmounted, and editing fstab on a production host risks emergency mode at boot. Result: "Job containerd.service/start failed with result 'dependency'", is-active inactive. It refused rather than creating an empty store on the root disk. storageReserved on SSD2 0 -> 80 GB; still Schedulable=True at 76.34%. NOT PROVEN, AND NOT GLOSSED (R-209a): the operator ruled DO NOT REBOOT, so the move has not survived a reboot — which is this task's own standard for validated. The MECHANISM is proven; the CONSEQUENCE (that a real boot mounts /mnt/ssd_2 before containerd starts, in this host's actual ordering) is not, and mount-ordering reasoning is exactly the class this project has been burned by. So: the pre-move tree stays MOVED ASIDE, not deleted (34.3 GB, which is why / sits at 54% and not lower), and felhom-store-postboot-check.service — oneshot, enabled, dry-run PASS at install — validates it automatically at the next boot, asserting positively that /var/lib/containerd does NOT exist. Read /var/log/felhom-store-postboot-check.log after the next boot; on PASS the rollback copy can go and ~34 GB returns. Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).
This commit is contained in:
@@ -488,10 +488,188 @@ happened to restart the pod. `Synced/Healthy` was true and beside the point. Fil
|
||||
|
||||
---
|
||||
|
||||
## 11. P6 — the SSD2 move: pre-analysis only. **NOT EXECUTED.**
|
||||
## 11. P6 — the SSD2 move: **EXECUTED 2026-08-05 on operator authorisation**
|
||||
|
||||
**Stopped here as specified. Nothing was moved, no daemon was reconfigured for SSD2, no mount unit was
|
||||
written.** The operator must be present.
|
||||
> **Executed after the operator ruled "proceed", having read the pre-analysis below. The
|
||||
> `storageReserved` condition CC named was applied with it. The operator subsequently ruled
|
||||
> explicitly: DO NOT REBOOT DooPlex — so the reboot validation is DEFERRED, and what that leaves
|
||||
> unproven is stated in §11.9 rather than glossed.**
|
||||
|
||||
### 11.0 Execution record
|
||||
|
||||
**Blast radius, established before anything was stopped:** k3s runs a **separate** containerd
|
||||
(`/run/k3s/containerd/containerd.sock`), so **Gitea, the registry, the hub, PBS, Longhorn and all
|
||||
~160 pods stayed up throughout**. The only workloads affected were the two `jarr-*` dev containers.
|
||||
`ctr -n moby containers list` confirmed exactly those two on the system containerd.
|
||||
|
||||
| # | step | result |
|
||||
|---|---|---|
|
||||
| 1 | stop `docker.service`, `docker.socket`, `containerd.service` | all `inactive` |
|
||||
| 2 | `mkdir /mnt/ssd_2/containerd` (mode 711, root:root) | ok |
|
||||
| 3 | `rsync -aHAX --numeric-ids` | **550,891 files, 33,276,370,041 B, 4 m 58 s** |
|
||||
| 4 | `mv /var/lib/containerd → /var/lib/containerd.pre-move-2026-08-05` | **moved aside, never deleted** |
|
||||
| 5 | `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` (backup kept) | one-line diff |
|
||||
| 6 | `RequiresMountsFor=/mnt/ssd_2` drop-ins on **both** `containerd.service` and `docker.service` | resolved into both units |
|
||||
| 7 | start + verify | see below |
|
||||
| 8 | `storageReserved = 80 GB` on SSD2 | applied |
|
||||
|
||||
**`-X` is load-bearing** — overlayfs snapshots carry `trusted.overlay.*` xattrs, and losing them
|
||||
would silently corrupt layer stacking.
|
||||
|
||||
### 11.1 Copy verified BEFORE the original was touched
|
||||
|
||||
Four independent observables, all matching:
|
||||
|
||||
| check | source | destination |
|
||||
|---|---:|---:|
|
||||
| filesystem entries | 550,891 | **550,891** |
|
||||
| `trusted.overlay` xattrs | 448 | **448** |
|
||||
| files with link count >1 (hardlinks) | 37,243 | **37,243** |
|
||||
| `io.containerd.metadata.v1.bolt/meta.db` sha256 | `c60ecfb4ee134255…` | **`c60ecfb4ee134255…`** |
|
||||
|
||||
(Byte totals differ by +14 MB — ext4 block allocation, not content.)
|
||||
|
||||
### 11.2 Nothing was lost
|
||||
|
||||
| | pre-move | post-move |
|
||||
|---|---:|---:|
|
||||
| unique images | 345 | **345 — identical ID set** |
|
||||
| tag lines | 350 | **350 — identical set** |
|
||||
| docker volumes | 8 | **8 — identical set** |
|
||||
| build cache | 2.782 GB / 38 records | **2.782 GB / 38 records** |
|
||||
| GC ceiling | 30 GiB | **30 GiB, still in force** |
|
||||
| postgres databases | 4 | **4** |
|
||||
| postgres `jarr_dev` bytes | 175,135,767 | **175,135,767** |
|
||||
| postgres public tables | 31 | **31** |
|
||||
| redis `DBSIZE` | 2437 | **2437** |
|
||||
|
||||
**End-to-end proof, not just inventory:** a real `./build.sh` run against the relocated store
|
||||
completed `rc=0`, produced an image, and grew the cache 2.782 → 4.012 GB. The scratch tag was removed
|
||||
and `:latest` restored to the anchor digest afterwards.
|
||||
|
||||
### 11.3 The store is demonstrably on SSD2
|
||||
|
||||
```
|
||||
containerd config dump → root = '/mnt/ssd_2/containerd'
|
||||
du /mnt/ssd_2/containerd → 35,500,183,552 B
|
||||
/var/lib/containerd → does not exist
|
||||
```
|
||||
|
||||
### 11.4 The guard — and a trap found while proving it
|
||||
|
||||
**First attempt did NOT refuse, and that is a finding worth more than the test.** With
|
||||
`RequiresMountsFor=/mnt/absent-guard-test-2026-08-05` — a path with **no mount unit at all** —
|
||||
containerd **started normally**. `RequiresMountsFor` on a path systemd has no mount unit for is a
|
||||
**silent no-op**. So a typo in the guard path buys nothing and says nothing: the same
|
||||
*built-but-never-wired* shape this project has shipped repeatedly.
|
||||
|
||||
**So the guard was verified positively, at the unit level:**
|
||||
|
||||
```
|
||||
containerd.service Requires=mnt-ssd_2.mount After=mnt-ssd_2.mount
|
||||
docker.service Requires=mnt-ssd_2.mount
|
||||
```
|
||||
|
||||
**Then the refusal itself was proven with a genuinely absent DEVICE.** `/mnt/ssd_2` hosts **12 live
|
||||
Longhorn replicas** and must never be unmounted, and editing `fstab` on a production host risks
|
||||
emergency mode at boot — so a **temporary synthetic `.mount` unit** pointing at a nonexistent UUID
|
||||
was used instead, with a temporary `RequiresMountsFor` on the real `containerd.service`:
|
||||
|
||||
```
|
||||
A dependency job for containerd.service failed.
|
||||
systemd: Dependency failed for containerd.service - containerd container runtime.
|
||||
systemd: containerd.service: Job containerd.service/start failed with result 'dependency'.
|
||||
containerd is-active: inactive
|
||||
```
|
||||
|
||||
**containerd REFUSED to start rather than creating an empty store on the root disk** — exactly the
|
||||
required behaviour, and exactly the signature the ep0 runbook describes. All test artifacts were
|
||||
removed; only `10-ssd2-mount.conf` remains on each unit and the guard resolves to `/mnt/ssd_2` alone.
|
||||
|
||||
### 11.5 Longhorn — the condition CC made the move conditional on
|
||||
|
||||
`storageReserved` on SSD2 raised **0 → 80,000,000,000 B**, covering the containerd tree so Longhorn's
|
||||
scheduler can no longer oversubscribe into the space the store needs.
|
||||
|
||||
| disk | available | reserved | `Schedulable` |
|
||||
|---|---:|---:|---|
|
||||
| **Crucial-SSD-240G (SSD2)** | 179,516,211,200 (**76.34%**) | **80,000,000,000** | **True** |
|
||||
| default-disk (SSD1) | 236,978,176,000 (49.64%) | 143,217,954,816 | **True** |
|
||||
| HDD4 | 1,793,798,963,200 (60.82%) | 536,870,912,000 | True |
|
||||
| HDD2 | 2,470,864,486,400 (24.91%) | 536,870,912,000 | **False** — pre-existing, untouched |
|
||||
|
||||
SSD2 remains schedulable with the reserve applied. **Rollback: patch `storageReserved` back to 0.**
|
||||
|
||||
### 11.6 A standing post-boot assertion was installed
|
||||
|
||||
`felhom-store-postboot-check.service` (oneshot, `After=docker.service`, enabled) runs
|
||||
`/usr/local/sbin/felhom-store-postboot-check.sh` at **every** boot and logs to
|
||||
`/var/log/felhom-store-postboot-check.log`. It asserts, with **positive** observables:
|
||||
|
||||
- `/mnt/ssd_2` is mounted;
|
||||
- containerd's effective `root` is `/mnt/ssd_2/containerd`;
|
||||
- **`/var/lib/containerd` does NOT exist** — i.e. no empty store was recreated on the root disk, which
|
||||
is the precise trap the guard exists to prevent;
|
||||
- docker sees ≥100 images;
|
||||
- both dev containers are running.
|
||||
|
||||
Dry-run at install time: **RESULT: PASS** on all six lines. It was proven working *before* being
|
||||
relied upon.
|
||||
|
||||
### 11.7 Disk position after the move
|
||||
|
||||
| | value |
|
||||
|---|---:|
|
||||
| `/` | 224 GB used, **199 GB free, 54%** |
|
||||
| `/mnt/ssd_2` | 52 GB used, **157 GB free, 25%** |
|
||||
| store on SSD2 | 35,500,183,552 B |
|
||||
| **rollback copy retained on `/`** | **34,299,498,496 B** |
|
||||
|
||||
**`/` did not drop below 54% because the rollback copy is deliberately still there.** Reclaiming it
|
||||
returns ~34 GB — see §11.9.
|
||||
|
||||
### 11.8 Rollback, complete and one sequence
|
||||
|
||||
1. `sudo systemctl stop docker docker.socket containerd`
|
||||
2. `sudo mv /var/lib/containerd.pre-move-2026-08-05 /var/lib/containerd`
|
||||
3. `sudo cp /etc/containerd/config.toml.pre-move-2026-08-05 /etc/containerd/config.toml`
|
||||
4. `sudo rm -rf /etc/systemd/system/{containerd,docker}.service.d/10-ssd2-mount.conf`
|
||||
5. `sudo systemctl disable --now felhom-store-postboot-check.service && sudo rm /etc/systemd/system/felhom-store-postboot-check.service /usr/local/sbin/felhom-store-postboot-check.sh`
|
||||
6. `sudo kubectl patch nodes.longhorn.io -n longhorn-system dooplex --type merge -p '{"spec":{"disks":{"Crucial-SSD-240G":{"storageReserved":0}}}}'`
|
||||
7. `sudo systemctl daemon-reload && sudo systemctl start containerd docker`
|
||||
8. `sudo rm -rf /mnt/ssd_2/containerd`
|
||||
|
||||
### 11.9 **NOT PROVEN: the reboot. Stated, not glossed.**
|
||||
|
||||
**The operator ruled explicitly that DooPlex must not be rebooted, so the reboot validation did not
|
||||
happen.** Uptime is unbroken (7 weeks 6 days, since 2026-06-10) — verified, not assumed.
|
||||
|
||||
**By this task's own standard — *"a move that has not survived a reboot has not been validated"* — the
|
||||
move is NOT fully validated,** and it would be dishonest to close it as if it were. Precisely:
|
||||
|
||||
- **PROVEN (mechanism):** the guard is wired into both units (`Requires=` + `After=mnt-ssd_2.mount`),
|
||||
and containerd **refuses to start** when a required mount's device is genuinely absent.
|
||||
- **NOT PROVEN (consequence):** that a real boot sequence mounts `/mnt/ssd_2` before containerd starts
|
||||
*on this host, in the real ordering*. Mount-ordering reasoning is exactly the class this project has
|
||||
been burned by, and the mechanism proof does not substitute for it. `CLAUDE.md` prefers the
|
||||
consequence over the mechanism; here only the mechanism is available.
|
||||
|
||||
**Two consequences follow, and both are deliberate:**
|
||||
|
||||
1. **The rollback copy `/var/lib/containerd.pre-move-2026-08-05` (34.3 GB) STAYS** until a reboot has
|
||||
validated the move. Deleting it now would trade a cheap 34 GB for the only cheap way back.
|
||||
2. **The next reboot — whenever it happens, planned or not — validates this automatically.** The
|
||||
post-boot service writes `RESULT: PASS` or `RESULT: FAIL` to
|
||||
`/var/log/felhom-store-postboot-check.log`. **Read that file after the next boot**; on `PASS`, the
|
||||
rollback copy can be removed and ~34 GB returned to `/`.
|
||||
|
||||
Tracked as **R-209a**.
|
||||
|
||||
---
|
||||
|
||||
## 11-bis. P6 — the pre-analysis the ruling was made on
|
||||
|
||||
*(Retained as written, before execution, so the ruling can be audited against what it was given.)*
|
||||
|
||||
**First, the framing changed underneath this probe.** P3's prune took `/` to 53% used and un-stuck
|
||||
SSD1's Longhorn disk. **The move is no longer a rescue; it is a deliberate choice.** Per standing rule 4,
|
||||
|
||||
Reference in New Issue
Block a user