From c917251eeb024425fb04ea523f7ad05e9b15cb7d Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 5 Aug 2026 10:27:40 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20P6=20EXECUTED=20=E2=80=94=20containerd?= =?UTF-8?q?=20store=20moved=20to=20SSD2;=20reboot=20deferred=20(R-209,=20R?= =?UTF-8?q?-209a)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator ruled "proceed" after reading the pre-analysis; the storageReserved condition CC made the move conditional on was applied with it. The pre-analysis is retained verbatim as §11-bis so the ruling can be audited against what it was given. MOVED WITH ZERO LOSS, verified BEFORE the original was touched on four independent observables: 550,891 = 550,891 filesystem entries; 448 = 448 trusted.overlay xattrs (load-bearing — overlayfs stacking rides them, which is why rsync -X is not optional); 37,243 = 37,243 hardlinks; byte-identical meta.db sha256. Verified again after: identical image/tag/volume ID SETS, cache 2.782 GB / 38 records, ceiling still 30 GiB, pg 4 DBs / 31 tables / 175,135,767 B, redis DBSIZE 2437. End-to-end proof was a real build on the relocated store, rc=0 — not an inventory count. k3s was never at risk, and that was established BEFORE stopping anything: it runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and ~160 pods stayed up. Only the two jarr-* dev containers were affected. TRAP FOUND WHILE PROVING THE GUARD, and it is the reusable part: RequiresMountsFor on a path with NO mount unit is a SILENT NO-OP — containerd started normally against an absent-but-unmounted path. A typo'd guard buys nothing and says nothing: the built-but-never-wired shape again. The guard was therefore verified positively at the unit level (Requires= AND After=mnt-ssd_2.mount on both units), and refusal proven with a genuinely absent DEVICE via a temporary synthetic .mount unit — /mnt/ssd_2 hosts 12 live Longhorn replicas and must never be unmounted, and editing fstab on a production host risks emergency mode at boot. Result: "Job containerd.service/start failed with result 'dependency'", is-active inactive. It refused rather than creating an empty store on the root disk. storageReserved on SSD2 0 -> 80 GB; still Schedulable=True at 76.34%. NOT PROVEN, AND NOT GLOSSED (R-209a): the operator ruled DO NOT REBOOT, so the move has not survived a reboot — which is this task's own standard for validated. The MECHANISM is proven; the CONSEQUENCE (that a real boot mounts /mnt/ssd_2 before containerd starts, in this host's actual ordering) is not, and mount-ordering reasoning is exactly the class this project has been burned by. So: the pre-move tree stays MOVED ASIDE, not deleted (34.3 GB, which is why / sits at 54% and not lower), and felhom-store-postboot-check.service — oneshot, enabled, dry-run PASS at install — validates it automatically at the next boot, asserting positively that /var/lib/containerd does NOT exist. Read /var/log/felhom-store-postboot-check.log after the next boot; on PASS the rollback copy can go and ~34 GB returns. Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit). --- STATUS.md | 23 ++- .../SPIKE-dooplex-buildcache-2026-08-05.md | 184 +++++++++++++++++- documentation/backlog/OPEN-ITEMS.md | 3 +- 3 files changed, 198 insertions(+), 12 deletions(-) diff --git a/STATUS.md b/STATUS.md index 0c2a831..2501b9c 100644 --- a/STATUS.md +++ b/STATUS.md @@ -56,6 +56,13 @@ over; and the file was restored **byte for byte identical**. *(R-201)* quietly around it: the warning meant to catch exactly this **could never fire** (fixed and proved), and the weekly cleanup is still forbidden from touching the thing that grows (next session). *(R-205 … R-211)* +- **2026-08-05** — **The build data now lives on the second SSD, moved with nothing lost.** All 345 + images, every saved volume and both development databases came through identical — checked before + the original was touched and again afterwards, and confirmed by running a real build on the moved + copy. **The server's own services never went down:** Gitea, the registry, the hub and the backup + system run on a separate system and stayed up throughout. The second SSD now also **reserves 80 GB** + for this, so the storage layer can no longer quietly claim the space and repeat what happened to the + first disk. The old copy is kept as the way back until the machine next restarts. *(R-209)* - **2026-08-05** — **Three of the four recovery crutches removed.** The reset code works first time; a credential re-issue no longer blocks off-site backups on a healthy machine; and the default restore no longer quietly returns the wrong thing. *(R-204 items 1–3, R-196)* @@ -79,14 +86,14 @@ over; and the file was restored **byte for byte identical**. *(R-201)* ## Waiting on you -- **Whether to move the build data onto the second SSD at all — and CC's advice is now "probably - not".** You ruled "cap it, then move it". The cap is in and it did the job on its own: the disk sits - at 54% with 199 GB free, and moving is no longer a rescue. The second SSD looks roomy (91% free) and - today the move is safe by a wide margin — but it is the **only** disk of the four with **no space - reserved for itself**, and its volumes are allowed to claim 6.9× more than they currently use. If - they ever grow into that, the move would push it below the same floor that just took the first disk - out of service. **If you want the move, do it together with reserving ~80 GB on that disk; otherwise - leaving it where it is costs nothing now.** *(R-209)* +- **One thing to read after the machine next restarts — and nothing to do until then.** You told me + not to restart DooPlex, so I did not, and the move to the second SSD has therefore never been + through a restart. It works right now and nothing was lost, but a restart is the one test that + matters for this kind of change, and it has not happened. **I made it check itself:** whenever the + machine next starts, for any reason, it writes a plain PASS or FAIL line to + `/var/log/felhom-store-postboot-check.log`. **If it says PASS, the old copy can be deleted and 34 GB + comes back.** Until then I have deliberately kept that old copy, which is the only quick way back — + it is why the disk sits at 54% rather than lower. *(R-209a)* - **One list to rule on: 193 old images that exist only on this machine.** 131 controller versions and 62 hub versions are not in the registry, so they cannot be re-downloaded — all of them old (controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing diff --git a/documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md b/documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md index a18a481..aab13f1 100644 --- a/documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md +++ b/documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md @@ -488,10 +488,188 @@ happened to restart the pod. `Synced/Healthy` was true and beside the point. Fil --- -## 11. P6 — the SSD2 move: pre-analysis only. **NOT EXECUTED.** +## 11. P6 — the SSD2 move: **EXECUTED 2026-08-05 on operator authorisation** -**Stopped here as specified. Nothing was moved, no daemon was reconfigured for SSD2, no mount unit was -written.** The operator must be present. +> **Executed after the operator ruled "proceed", having read the pre-analysis below. The +> `storageReserved` condition CC named was applied with it. The operator subsequently ruled +> explicitly: DO NOT REBOOT DooPlex — so the reboot validation is DEFERRED, and what that leaves +> unproven is stated in §11.9 rather than glossed.** + +### 11.0 Execution record + +**Blast radius, established before anything was stopped:** k3s runs a **separate** containerd +(`/run/k3s/containerd/containerd.sock`), so **Gitea, the registry, the hub, PBS, Longhorn and all +~160 pods stayed up throughout**. The only workloads affected were the two `jarr-*` dev containers. +`ctr -n moby containers list` confirmed exactly those two on the system containerd. + +| # | step | result | +|---|---|---| +| 1 | stop `docker.service`, `docker.socket`, `containerd.service` | all `inactive` | +| 2 | `mkdir /mnt/ssd_2/containerd` (mode 711, root:root) | ok | +| 3 | `rsync -aHAX --numeric-ids` | **550,891 files, 33,276,370,041 B, 4 m 58 s** | +| 4 | `mv /var/lib/containerd → /var/lib/containerd.pre-move-2026-08-05` | **moved aside, never deleted** | +| 5 | `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` (backup kept) | one-line diff | +| 6 | `RequiresMountsFor=/mnt/ssd_2` drop-ins on **both** `containerd.service` and `docker.service` | resolved into both units | +| 7 | start + verify | see below | +| 8 | `storageReserved = 80 GB` on SSD2 | applied | + +**`-X` is load-bearing** — overlayfs snapshots carry `trusted.overlay.*` xattrs, and losing them +would silently corrupt layer stacking. + +### 11.1 Copy verified BEFORE the original was touched + +Four independent observables, all matching: + +| check | source | destination | +|---|---:|---:| +| filesystem entries | 550,891 | **550,891** | +| `trusted.overlay` xattrs | 448 | **448** | +| files with link count >1 (hardlinks) | 37,243 | **37,243** | +| `io.containerd.metadata.v1.bolt/meta.db` sha256 | `c60ecfb4ee134255…` | **`c60ecfb4ee134255…`** | + +(Byte totals differ by +14 MB — ext4 block allocation, not content.) + +### 11.2 Nothing was lost + +| | pre-move | post-move | +|---|---:|---:| +| unique images | 345 | **345 — identical ID set** | +| tag lines | 350 | **350 — identical set** | +| docker volumes | 8 | **8 — identical set** | +| build cache | 2.782 GB / 38 records | **2.782 GB / 38 records** | +| GC ceiling | 30 GiB | **30 GiB, still in force** | +| postgres databases | 4 | **4** | +| postgres `jarr_dev` bytes | 175,135,767 | **175,135,767** | +| postgres public tables | 31 | **31** | +| redis `DBSIZE` | 2437 | **2437** | + +**End-to-end proof, not just inventory:** a real `./build.sh` run against the relocated store +completed `rc=0`, produced an image, and grew the cache 2.782 → 4.012 GB. The scratch tag was removed +and `:latest` restored to the anchor digest afterwards. + +### 11.3 The store is demonstrably on SSD2 + +``` +containerd config dump → root = '/mnt/ssd_2/containerd' +du /mnt/ssd_2/containerd → 35,500,183,552 B +/var/lib/containerd → does not exist +``` + +### 11.4 The guard — and a trap found while proving it + +**First attempt did NOT refuse, and that is a finding worth more than the test.** With +`RequiresMountsFor=/mnt/absent-guard-test-2026-08-05` — a path with **no mount unit at all** — +containerd **started normally**. `RequiresMountsFor` on a path systemd has no mount unit for is a +**silent no-op**. So a typo in the guard path buys nothing and says nothing: the same +*built-but-never-wired* shape this project has shipped repeatedly. + +**So the guard was verified positively, at the unit level:** + +``` +containerd.service Requires=mnt-ssd_2.mount After=mnt-ssd_2.mount +docker.service Requires=mnt-ssd_2.mount +``` + +**Then the refusal itself was proven with a genuinely absent DEVICE.** `/mnt/ssd_2` hosts **12 live +Longhorn replicas** and must never be unmounted, and editing `fstab` on a production host risks +emergency mode at boot — so a **temporary synthetic `.mount` unit** pointing at a nonexistent UUID +was used instead, with a temporary `RequiresMountsFor` on the real `containerd.service`: + +``` +A dependency job for containerd.service failed. +systemd: Dependency failed for containerd.service - containerd container runtime. +systemd: containerd.service: Job containerd.service/start failed with result 'dependency'. +containerd is-active: inactive +``` + +**containerd REFUSED to start rather than creating an empty store on the root disk** — exactly the +required behaviour, and exactly the signature the ep0 runbook describes. All test artifacts were +removed; only `10-ssd2-mount.conf` remains on each unit and the guard resolves to `/mnt/ssd_2` alone. + +### 11.5 Longhorn — the condition CC made the move conditional on + +`storageReserved` on SSD2 raised **0 → 80,000,000,000 B**, covering the containerd tree so Longhorn's +scheduler can no longer oversubscribe into the space the store needs. + +| disk | available | reserved | `Schedulable` | +|---|---:|---:|---| +| **Crucial-SSD-240G (SSD2)** | 179,516,211,200 (**76.34%**) | **80,000,000,000** | **True** | +| default-disk (SSD1) | 236,978,176,000 (49.64%) | 143,217,954,816 | **True** | +| HDD4 | 1,793,798,963,200 (60.82%) | 536,870,912,000 | True | +| HDD2 | 2,470,864,486,400 (24.91%) | 536,870,912,000 | **False** — pre-existing, untouched | + +SSD2 remains schedulable with the reserve applied. **Rollback: patch `storageReserved` back to 0.** + +### 11.6 A standing post-boot assertion was installed + +`felhom-store-postboot-check.service` (oneshot, `After=docker.service`, enabled) runs +`/usr/local/sbin/felhom-store-postboot-check.sh` at **every** boot and logs to +`/var/log/felhom-store-postboot-check.log`. It asserts, with **positive** observables: + +- `/mnt/ssd_2` is mounted; +- containerd's effective `root` is `/mnt/ssd_2/containerd`; +- **`/var/lib/containerd` does NOT exist** — i.e. no empty store was recreated on the root disk, which + is the precise trap the guard exists to prevent; +- docker sees ≥100 images; +- both dev containers are running. + +Dry-run at install time: **RESULT: PASS** on all six lines. It was proven working *before* being +relied upon. + +### 11.7 Disk position after the move + +| | value | +|---|---:| +| `/` | 224 GB used, **199 GB free, 54%** | +| `/mnt/ssd_2` | 52 GB used, **157 GB free, 25%** | +| store on SSD2 | 35,500,183,552 B | +| **rollback copy retained on `/`** | **34,299,498,496 B** | + +**`/` did not drop below 54% because the rollback copy is deliberately still there.** Reclaiming it +returns ~34 GB — see §11.9. + +### 11.8 Rollback, complete and one sequence + +1. `sudo systemctl stop docker docker.socket containerd` +2. `sudo mv /var/lib/containerd.pre-move-2026-08-05 /var/lib/containerd` +3. `sudo cp /etc/containerd/config.toml.pre-move-2026-08-05 /etc/containerd/config.toml` +4. `sudo rm -rf /etc/systemd/system/{containerd,docker}.service.d/10-ssd2-mount.conf` +5. `sudo systemctl disable --now felhom-store-postboot-check.service && sudo rm /etc/systemd/system/felhom-store-postboot-check.service /usr/local/sbin/felhom-store-postboot-check.sh` +6. `sudo kubectl patch nodes.longhorn.io -n longhorn-system dooplex --type merge -p '{"spec":{"disks":{"Crucial-SSD-240G":{"storageReserved":0}}}}'` +7. `sudo systemctl daemon-reload && sudo systemctl start containerd docker` +8. `sudo rm -rf /mnt/ssd_2/containerd` + +### 11.9 **NOT PROVEN: the reboot. Stated, not glossed.** + +**The operator ruled explicitly that DooPlex must not be rebooted, so the reboot validation did not +happen.** Uptime is unbroken (7 weeks 6 days, since 2026-06-10) — verified, not assumed. + +**By this task's own standard — *"a move that has not survived a reboot has not been validated"* — the +move is NOT fully validated,** and it would be dishonest to close it as if it were. Precisely: + +- **PROVEN (mechanism):** the guard is wired into both units (`Requires=` + `After=mnt-ssd_2.mount`), + and containerd **refuses to start** when a required mount's device is genuinely absent. +- **NOT PROVEN (consequence):** that a real boot sequence mounts `/mnt/ssd_2` before containerd starts + *on this host, in the real ordering*. Mount-ordering reasoning is exactly the class this project has + been burned by, and the mechanism proof does not substitute for it. `CLAUDE.md` prefers the + consequence over the mechanism; here only the mechanism is available. + +**Two consequences follow, and both are deliberate:** + +1. **The rollback copy `/var/lib/containerd.pre-move-2026-08-05` (34.3 GB) STAYS** until a reboot has + validated the move. Deleting it now would trade a cheap 34 GB for the only cheap way back. +2. **The next reboot — whenever it happens, planned or not — validates this automatically.** The + post-boot service writes `RESULT: PASS` or `RESULT: FAIL` to + `/var/log/felhom-store-postboot-check.log`. **Read that file after the next boot**; on `PASS`, the + rollback copy can be removed and ~34 GB returned to `/`. + +Tracked as **R-209a**. + +--- + +## 11-bis. P6 — the pre-analysis the ruling was made on + +*(Retained as written, before execution, so the ruling can be audited against what it was given.)* **First, the framing changed underneath this probe.** P3's prune took `/` to 53% used and un-stuck SSD1's Longhorn disk. **The move is no longer a rescue; it is a deliberate choice.** Per standing rule 4, diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index f989927..7e15ec4 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -185,7 +185,8 @@ drill needed — the fourth is R-193, and until it goes the recovery is still op | **R-206** | **The build-cache cap and the weekly prune exist only as a hand-edited `/etc/docker/daemon.json` on DooPlex — not in Ansible, so a rebuild loses them.** The `node_housekeeping` role must also carry the prune, which today it is forbidden to run | **READY (M) — NEW 2026-08-05** | — | **The spike validated the recipe; this row builds it.** Three parts. **(a) Template `/etc/docker/daemon.json`** with the **`policy` array** form — **the flat form (`{"gc":{"reservedSpace":…}}`) is SILENTLY IGNORED**, measured: the daemon starts, logs nothing, and `docker buildx inspect` still reports the built-in defaults. **The oracle is `docker buildx inspect`, never `dockerd --validate`** — the validator returned `configuration OK` for a bogus key AND for the config that then **crashed the daemon** (`filter` takes one value per policy entry, not an array; `error initializing buildkit: filters expect only one value`). **(b) Narrow the role's Docker ban** (`node-housekeeping.sh.j2:10-14`) to permit exactly `docker builder prune -af` and nothing else — the ban's stated premise ("Docker here runs only unrelated jarr-* dev containers") is obsolete: the growth is Felhom Go build cache. **The measured prune is SYNCHRONOUS** (150.35 GB back at t+0, two consecutive polls <1 MB apart within 60 s) — **unlike containerd's image GC, so it needs no `settle_imagefs` equivalent**, but it MUST measure the filesystem rather than trust the command: `prune` claimed **156.9 GB** and the filesystem returned **150.35 GB**, the 6.5 GB gap being layers still shared with images. **(c) A restart-safety note in the role:** a bad `daemon.json` takes the daemon down AND leaves the `unless-stopped` dev containers stopped — they needed a manual `docker start` — so the role must restart-and-verify, not validate-and-assume. Recipe + every measurement: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` | CC | | **R-207** | **`DRY_RUN=1` on `node-housekeeping.sh` is NOT non-mutating — it destroys the metric history it is supposed to let you inspect** | **READY (S) — NEW 2026-08-05** | — | `write_metrics()` (`node-housekeeping.sh.j2:119-150`) has **no `DRY_RUN` guard at all** — `DRY_RUN` appears in it only inside a log line — and it is called from an **unconditional `EXIT` trap** (`:150`). A dry run therefore atomically renames over the live node_exporter textfile, overwriting `node_housekeeping_last_success_timestamp_seconds` with *now* and `reclaimed_bytes` with ~0 — **resetting the staleness clock `HousekeepingStale` watches and erasing the 8-week reclaim history**. Confirmed by reading the source in both the 2026-08-05 audit and this spike; **neither run executed it**, so the history survives. Fix: guard `write_metrics` on `DRY_RUN`, or have the trap skip it. Pairs naturally with R-206 (same file, same role) | CC | | **R-208** | **Every Felhom Go build re-downloads its modules because `ARG VERSION` sits ABOVE the module-download layer — ~440 MB of dead cache per build, 90.5 GB of the 157 GB** | **READY (S) — NEW 2026-08-05, ROOT CAUSE PROVEN** | — | **Measured, not inferred.** All **208** retained `go mod download` records carried **`Usage count: 1`** — not one was ever reused in ~a month of builds. The mechanism was isolated by four controlled builds: an unchanged tree rebuilt with the **same** `--build-arg VERSION` → `RUN go mod download` **CACHED**; the **same** tree with a **new** `VERSION` → **executed**. `COPY go.mod ./` stays CACHED either way, which is the tell: a `COPY`'s key is content-based, while a `RUN`'s key includes the stage **environment**, and `ARG VERSION`/`ARG GIT_COMMIT` are declared *before* the download in `felhom-controller/controller/Dockerfile`. Since every real build passes a fresh version, the layer is invalidated **every single time**. **`felhom.eu/hub/Dockerfile` has the identical defect** (`ARG VERSION`/`ARG BUILD_TIME` above `COPY go.mod go.sum*` → `RUN go mod download`) — and because both Dockerfiles produce byte-identical `buildx du` description strings, the 208 records are a COMBINED count and must not be attributed to one project. **Fix shape (one line each, not applied here):** move the `ARG VERSION`/`ARG GIT_COMMIT`/`ARG BUILD_TIME` declarations down to just above the final `go build`. **Worth more than the cap and the move combined** — the cap bounds the symptom, this removes the source. `build.sh`'s `rm -rf` + `cp -a` and its host-side `go mod tidy` were **ruled out by fingerprinting**: the tree is byte-identical across runs and `tidy` is a no-op | CC | -| **R-209** | **Should the containerd store move to SSD2 at all? — SSD2 is the ONLY Longhorn disk with `storageReserved = 0`, and the move is safe today but breaches the 25% floor if its volumes ever inflate** | **WAITING-ON-OPERATOR** — NEW 2026-08-05 | an operator ruling | **P3's prune already removed the urgency: `/` went 86% → 53% used and SSD1's Longhorn disk went `Schedulable=False (DiskPressure)` → `Schedulable=True` (18.85% → 50.32% available).** The move was ruled "cap then move"; the cap is in and the pressure is gone, so this is now a deliberate choice rather than a rescue. **The numbers, measured (`Crucial-SSD-240G`, `/mnt/ssd_2/data/longhorn`, `storageMaximum` 235,148,750,848):** available today **214,958,080,000 (91.41%)**; 25% floor **58,787,187,712**. Moving the whole containerd tree at steady state (~31.5 GB images + ≤30 GB cache ≈ 65 GB) leaves **63.77%, i.e. +38.8 pp above the floor — comfortably safe as measured.** **But `storageScheduled` on SSD2 is 139,586,437,120 while `df` says only 20,094,939,136 is actually used** — Longhorn has overcommitted 6.9× — and if those volumes ever inflate to their scheduled size, the same disk lands at **13.00%, i.e. 12 pp BELOW the floor → `Schedulable=False`**, which is exactly the failure that just took SSD1 out. **Recommendation: do the move only together with setting `storageReserved` on SSD2 to cover the containerd tree (~80 GB); SSD2 reserving zero while HDD2 and HDD4 each reserve 500 GB is an anomaly in its own right.** Mechanism, if it goes ahead: **containerd's `root` in `/etc/containerd/config.toml`** (the key is present but commented out) — **not** Docker's `data-root`, which would move only 0.62 GB. Guard: `RequiresMountsFor=/mnt/ssd_2` on `containerd.service` **and** `docker.service`, remembering that **`RequiresMountsFor` RE-MOUNTS rather than refusing** ([[ep0-datastore-volume-move-2026-07-27]]) — so it must be tested with a genuinely absent device, and the move is not validated until it has survived a **reboot**. Full pre-analysis: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` §P6 | operator | +| **R-209** | ~~**Should the containerd store move to SSD2 at all?**~~ | **EXECUTED 2026-08-05 on operator ruling — reboot validation DEFERRED → R-209a** | — | **Operator ruled "proceed" having read the pre-analysis; CC's `storageReserved` condition was applied with it.** Moved with **zero loss, verified on four independent observables BEFORE the original was touched** (550,891 entries = 550,891; **448 = 448 `trusted.overlay` xattrs**; 37,243 = 37,243 hardlinks; byte-identical `meta.db` sha256) and again after (identical image/tag/volume ID **sets**, cache 2.782 GB/38 records, pg 4 DBs / 31 tables / 175,135,767 B, redis 2437). **`-X` is load-bearing** — overlayfs stacking rides `trusted.overlay.*`. End-to-end proof was a **real build** on the relocated store, `rc=0`. **k3s was never at risk and this was established before stopping anything:** it runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and ~160 pods stayed up; only the two `jarr-*` dev containers were affected. `storageReserved` on SSD2 **0 → 80 GB**, still `Schedulable=True` at 76.34%. **A TRAP was found while proving the guard, and it is the reusable part: `RequiresMountsFor` on a path with NO mount unit is a SILENT NO-OP** — containerd started normally against an absent-but-unmounted path, so a typo'd guard buys nothing and says nothing (the *built-but-never-wired* shape again). The guard was therefore verified positively at the unit level (`Requires=` **and** `After=mnt-ssd_2.mount` on both units), and refusal was then proven with a genuinely absent **device** — via a temporary synthetic `.mount` unit, because `/mnt/ssd_2` hosts 12 live Longhorn replicas and must never be unmounted, and editing `fstab` on a production host risks emergency mode at boot: `Job containerd.service/start failed with result 'dependency'`, `is-active: inactive`. **Rollback is one documented sequence** (audit §11.8); the pre-move tree is **moved aside, not deleted**. Evidence: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` §11 | — | +| **R-209a** | **The SSD2 move has NOT survived a reboot, so by this project's own standard it is not fully validated** | **WATCHING — NEW 2026-08-05** | the next DooPlex reboot | **Operator ruled explicitly: do NOT reboot DooPlex.** Uptime verified unbroken (7 weeks 6 days, since 2026-06-10). **The distinction is stated rather than glossed: the MECHANISM is proven** — the guard is wired into both units and containerd refuses to start when a required mount's device is absent — **but the CONSEQUENCE is not**: that a real boot mounts `/mnt/ssd_2` before containerd starts, in this host's actual ordering. Mount-ordering reasoning is precisely the class this project has been burned by (`RequiresMountsFor` RE-MOUNTS rather than refusing — the ep0 lesson), and `CLAUDE.md` prefers a consequence assertion over a mechanism one. **Two deliberate consequences: (1)** the rollback copy `/var/lib/containerd.pre-move-2026-08-05` (**34.3 GB on `/`**) **STAYS** until a reboot validates — which is why `/` sits at 54% and not lower; deleting it now would trade a cheap 34 GB for the only cheap way back. **(2)** validation is **automatic and needs no one to remember it**: `felhom-store-postboot-check.service` (oneshot, enabled, dry-run PASS at install) runs at **every** boot and writes `RESULT: PASS`/`FAIL` to `/var/log/felhom-store-postboot-check.log`, asserting positively that `/mnt/ssd_2` is mounted, that containerd's root is on it, that **`/var/lib/containerd` does NOT exist** (the empty-store trap), that ≥100 images are visible and that both dev containers run. **Next action: after the next reboot — planned or not — read that file; on PASS, `rm -rf /var/lib/containerd.pre-move-2026-08-05` returns ~34 GB to `/`** | operator + CC | **P3's prune already removed the urgency: `/` went 86% → 53% used and SSD1's Longhorn disk went `Schedulable=False (DiskPressure)` → `Schedulable=True` (18.85% → 50.32% available).** The move was ruled "cap then move"; the cap is in and the pressure is gone, so this is now a deliberate choice rather than a rescue. **The numbers, measured (`Crucial-SSD-240G`, `/mnt/ssd_2/data/longhorn`, `storageMaximum` 235,148,750,848):** available today **214,958,080,000 (91.41%)**; 25% floor **58,787,187,712**. Moving the whole containerd tree at steady state (~31.5 GB images + ≤30 GB cache ≈ 65 GB) leaves **63.77%, i.e. +38.8 pp above the floor — comfortably safe as measured.** **But `storageScheduled` on SSD2 is 139,586,437,120 while `df` says only 20,094,939,136 is actually used** — Longhorn has overcommitted 6.9× — and if those volumes ever inflate to their scheduled size, the same disk lands at **13.00%, i.e. 12 pp BELOW the floor → `Schedulable=False`**, which is exactly the failure that just took SSD1 out. **Recommendation: do the move only together with setting `storageReserved` on SSD2 to cover the containerd tree (~80 GB); SSD2 reserving zero while HDD2 and HDD4 each reserve 500 GB is an anomaly in its own right.** Mechanism, if it goes ahead: **containerd's `root` in `/etc/containerd/config.toml`** (the key is present but commented out) — **not** Docker's `data-root`, which would move only 0.62 GB. Guard: `RequiresMountsFor=/mnt/ssd_2` on `containerd.service` **and** `docker.service`, remembering that **`RequiresMountsFor` RE-MOUNTS rather than refusing** ([[ep0-datastore-volume-move-2026-07-27]]) — so it must be tested with a genuinely absent device, and the move is not validated until it has survived a **reboot**. Full pre-analysis: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` §P6 | operator | | **R-210** | **Which of the 345 local images may be deleted — 131 controller tags and 62 hub tags exist ONLY on this box and are not recoverable by `docker pull`** | **WAITING-ON-OPERATOR** — NEW 2026-08-05 | an operator ruling | **Nothing was deleted; this is a list, not an action.** The registry was queried directly: `felhom-controller` has **76** tags in Gitea vs **207** locally, `felhom-hub` **45** vs **107**. The **131 + 62 local-only tags are all OLD** — controller `0.39.0`–`0.135.0` plus `v0.35.0`–`v0.39.0`, hub `0.9.0`–`0.57.0` plus `v0.7.2`–`v0.13.0` — while everything from controller `0.136.0` and hub `0.58.0` upward IS in the registry and therefore re-pullable. **Size the prize honestly before spending a decision on it:** per-tag sizes sum to 139.29 GB, but that double-counts shared layers — `docker system df` puts the **real** dedup'd image footprint at **31.02 GB with 27.02 GB reclaimable**, i.e. an order of magnitude less than the build cache P3 already returned. `docker image prune -a` would remove 343 of 345 (only `redis:7-alpine` and `postgres:16-alpine` are held by running containers). **CC's view: not worth doing for the space** — it buys ~27 GB against 199 GB now free, and its only real benefit is dropping unrecoverable clutter | operator | | **R-211** | **Prometheus has no config-reloader — a rules change reaches the pod and is never read** | **READY (S) — NEW 2026-08-05** | — | Found while verifying R-205 rather than by looking for it. The `mon-system/prometheus` Deployment runs **one** container (`prom/prometheus:v3.12.0`) with **no `configmap-reload`/`prometheus-config-reloader` sidecar**. After the ArgoCD sync the updated `node-housekeeping-alerts.yml` was present **inside the pod** (`grep -c "and on(instance)"` → 3 on the mounted symlink) while the Prometheus **rules API still served the old expression** — for **4+ minutes**, with no error anywhere. It only took effect after an explicit `POST /-/reload`. **The consequence is general, not specific to R-205: every rule edit in this repo since the stack was built has silently not applied until something happened to restart the pod** — so "committed and synced" has never meant "in force", and ArgoCD reporting `Synced/Healthy` is true and beside the point. `--web.enable-lifecycle` IS already set, so the fix is small: add a reloader sidecar watching the ConfigMap, or a `checksum/config` pod annotation so a rules change rolls the pod. **Same class as the four *built-but-never-wired* seams** — the control exists, nothing walks it | CC |