docs: SPIKE — DooPlex build-cache containment; R-205..R-211 (2026-08-05)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Spike output only; no production Go code. The one shipped change rides in homelab-manifests 6808a4b (R-205, the monitoring rule). VERDICT: mechanism confirmed, with one correction and one refutation. - CONFIRMED: builder.gc IS honoured under the containerd worker and DOES evict. Proven by naming a 440 MB `go mod download` record present at build N and absent by N+2 — not by absence of an error. - CORRECTED: honoured ONLY in the `policy` array form. The flat form is SILENTLY ignored — daemon starts, logs nothing, keeps its defaults. `dockerd --validate` returned "configuration OK" for a bogus key AND for a config that then crashed the daemon. The oracle is `docker buildx inspect`. - REFUTED: Docker's `data-root` would NOT move the cache — it moves 0.62 GB. The 181.4 GB belongs to the system containerd (`root` in /etc/containerd/config.toml). P3 (operator-approved) executed: prune claimed 156.9 GB, the filesystem returned 150.35 GB (the 6.5 GB gap is layers shared with images), SYNCHRONOUSLY — / went 86% -> 53% used, and Longhorn's default disk went Schedulable=False (DiskPressure) -> Schedulable=True (18.85% -> 50.32%). P7 root-caused the largest item and it is NOT the cap: all 208 `go mod download` records had Usage count 1. Isolated by controlled builds — same VERSION build-arg -> CACHED, new VERSION -> executed, byte-identical tree. `ARG VERSION`/`ARG GIT_COMMIT` sit ABOVE the module-download step, and a RUN's cache key includes the stage environment. Both Dockerfiles have it. One line each to fix -> R-208. P6 NOT EXECUTED — stops at the operator, as specified. Pre-analysis: the move is safe as measured (+38.8 pp above the 25% floor) but SSD2 is the only Longhorn disk with storageReserved=0 and is overcommitted 6.9x; at full inflation the move lands 12 pp BELOW the floor. The prune removed the move's urgency, so CC recommends against it unless ~80 GB is reserved on SSD2 -> R-209. Register: R-205 (CLOSED, shipped), R-206 (Ansible: cap + prune + narrowed Docker ban), R-207 (DRY_RUN guard), R-208 (ARG ordering), R-209/R-210 (operator), R-211 (Prometheus has no config-reloader — rules changes have never applied until something restarted the pod; found while verifying R-205). Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).
This commit is contained in:
@@ -47,6 +47,15 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
|
||||
|
||||
## What shipped recently
|
||||
|
||||
- **2026-08-05** — **The DooPlex server's disk is out of danger: 86% full → 54%, and the storage layer
|
||||
is unstuck.** The cause was leftover working data from building our own software — 157 GB of it,
|
||||
growing about 5 GB a day, which nothing was allowed to delete. **148 GB came back in 86 seconds.**
|
||||
The storage layer had already stopped accepting new copies of any volume onto that disk; that is
|
||||
fixed the same day. A **30 GB ceiling** is now in place and was **proved to work by deliberately
|
||||
overfilling it and watching it evict** — not by assuming the setting took. Two things that failed
|
||||
quietly around it: the warning meant to catch exactly this **could never fire** (fixed and proved),
|
||||
and the weekly cleanup is still forbidden from touching the thing that grows (next session).
|
||||
*(R-205 … R-211)*
|
||||
- **2026-08-05** — **Three of the four recovery crutches removed.** The reset code works first time;
|
||||
a credential re-issue no longer blocks off-site backups on a healthy machine; and the default
|
||||
restore no longer quietly returns the wrong thing. *(R-204 items 1–3, R-196)*
|
||||
@@ -70,6 +79,19 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **Whether to move the build data onto the second SSD at all — and CC's advice is now "probably
|
||||
not".** You ruled "cap it, then move it". The cap is in and it did the job on its own: the disk sits
|
||||
at 54% with 199 GB free, and moving is no longer a rescue. The second SSD looks roomy (91% free) and
|
||||
today the move is safe by a wide margin — but it is the **only** disk of the four with **no space
|
||||
reserved for itself**, and its volumes are allowed to claim 6.9× more than they currently use. If
|
||||
they ever grow into that, the move would push it below the same floor that just took the first disk
|
||||
out of service. **If you want the move, do it together with reserving ~80 GB on that disk; otherwise
|
||||
leaving it where it is costs nothing now.** *(R-209)*
|
||||
- **One list to rule on: 193 old images that exist only on this machine.** 131 controller versions and
|
||||
62 hub versions are not in the registry, so they cannot be re-downloaded — all of them old
|
||||
(controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing
|
||||
was deleted.** Worth knowing before you spend time on it: they only account for about 27 GB against
|
||||
199 GB now free, so this is about clutter, not space. *(R-210)*
|
||||
- **The one-shot credential decision — this is now the last thing between a customer and an unaided
|
||||
recovery.** A rebuilt machine has no storage credential of its own, so an operator must press
|
||||
Re-issue. Everything after that point is now self-service. Deciding how a rebuilt machine should get
|
||||
|
||||
Reference in New Issue
Block a user