ac451f9667e2259c0d72482b124ceb1539577eca
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 admin
|
c917251eeb |
docs: P6 EXECUTED — containerd store moved to SSD2; reboot deferred (R-209, R-209a)
gates / gates (push) Successful in 7s
Operator ruled "proceed" after reading the pre-analysis; the storageReserved condition CC made the move conditional on was applied with it. The pre-analysis is retained verbatim as §11-bis so the ruling can be audited against what it was given. MOVED WITH ZERO LOSS, verified BEFORE the original was touched on four independent observables: 550,891 = 550,891 filesystem entries; 448 = 448 trusted.overlay xattrs (load-bearing — overlayfs stacking rides them, which is why rsync -X is not optional); 37,243 = 37,243 hardlinks; byte-identical meta.db sha256. Verified again after: identical image/tag/volume ID SETS, cache 2.782 GB / 38 records, ceiling still 30 GiB, pg 4 DBs / 31 tables / 175,135,767 B, redis DBSIZE 2437. End-to-end proof was a real build on the relocated store, rc=0 — not an inventory count. k3s was never at risk, and that was established BEFORE stopping anything: it runs a separate containerd, so Gitea, the registry, the hub, PBS, Longhorn and ~160 pods stayed up. Only the two jarr-* dev containers were affected. TRAP FOUND WHILE PROVING THE GUARD, and it is the reusable part: RequiresMountsFor on a path with NO mount unit is a SILENT NO-OP — containerd started normally against an absent-but-unmounted path. A typo'd guard buys nothing and says nothing: the built-but-never-wired shape again. The guard was therefore verified positively at the unit level (Requires= AND After=mnt-ssd_2.mount on both units), and refusal proven with a genuinely absent DEVICE via a temporary synthetic .mount unit — /mnt/ssd_2 hosts 12 live Longhorn replicas and must never be unmounted, and editing fstab on a production host risks emergency mode at boot. Result: "Job containerd.service/start failed with result 'dependency'", is-active inactive. It refused rather than creating an empty store on the root disk. storageReserved on SSD2 0 -> 80 GB; still Schedulable=True at 76.34%. NOT PROVEN, AND NOT GLOSSED (R-209a): the operator ruled DO NOT REBOOT, so the move has not survived a reboot — which is this task's own standard for validated. The MECHANISM is proven; the CONSEQUENCE (that a real boot mounts /mnt/ssd_2 before containerd starts, in this host's actual ordering) is not, and mount-ordering reasoning is exactly the class this project has been burned by. So: the pre-move tree stays MOVED ASIDE, not deleted (34.3 GB, which is why / sits at 54% and not lower), and felhom-store-postboot-check.service — oneshot, enabled, dry-run PASS at install — validates it automatically at the next boot, asserting positively that /var/lib/containerd does NOT exist. Read /var/log/felhom-store-postboot-check.log after the next boot; on PASS the rollback copy can go and ~34 GB returns. Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit). |
||
|
admin
|
ee9d9bf203 |
docs: SPIKE — DooPlex build-cache containment; R-205..R-211 (2026-08-05)
gates / gates (push) Successful in 7s
Spike output only; no production Go code. The one shipped change rides in homelab-manifests 6808a4b (R-205, the monitoring rule). VERDICT: mechanism confirmed, with one correction and one refutation. - CONFIRMED: builder.gc IS honoured under the containerd worker and DOES evict. Proven by naming a 440 MB `go mod download` record present at build N and absent by N+2 — not by absence of an error. - CORRECTED: honoured ONLY in the `policy` array form. The flat form is SILENTLY ignored — daemon starts, logs nothing, keeps its defaults. `dockerd --validate` returned "configuration OK" for a bogus key AND for a config that then crashed the daemon. The oracle is `docker buildx inspect`. - REFUTED: Docker's `data-root` would NOT move the cache — it moves 0.62 GB. The 181.4 GB belongs to the system containerd (`root` in /etc/containerd/config.toml). P3 (operator-approved) executed: prune claimed 156.9 GB, the filesystem returned 150.35 GB (the 6.5 GB gap is layers shared with images), SYNCHRONOUSLY — / went 86% -> 53% used, and Longhorn's default disk went Schedulable=False (DiskPressure) -> Schedulable=True (18.85% -> 50.32%). P7 root-caused the largest item and it is NOT the cap: all 208 `go mod download` records had Usage count 1. Isolated by controlled builds — same VERSION build-arg -> CACHED, new VERSION -> executed, byte-identical tree. `ARG VERSION`/`ARG GIT_COMMIT` sit ABOVE the module-download step, and a RUN's cache key includes the stage environment. Both Dockerfiles have it. One line each to fix -> R-208. P6 NOT EXECUTED — stops at the operator, as specified. Pre-analysis: the move is safe as measured (+38.8 pp above the 25% floor) but SSD2 is the only Longhorn disk with storageReserved=0 and is overcommitted 6.9x; at full inflation the move lands 12 pp BELOW the floor. The prune removed the move's urgency, so CC recommends against it unless ~80 GB is reserved on SSD2 -> R-209. Register: R-205 (CLOSED, shipped), R-206 (Ansible: cap + prune + narrowed Docker ban), R-207 (DRY_RUN guard), R-208 (ARG ordering), R-209/R-210 (operator), R-211 (Prometheus has no config-reloader — rules changes have never applied until something restarted the pod; found while verifying R-205). Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit). |