docs: SPIKE — DooPlex build-cache containment; R-205..R-211 (2026-08-05)
gates / gates (push) Successful in 7s

Spike output only; no production Go code. The one shipped change rides in
homelab-manifests 6808a4b (R-205, the monitoring rule).

VERDICT: mechanism confirmed, with one correction and one refutation.

- CONFIRMED: builder.gc IS honoured under the containerd worker and DOES evict.
  Proven by naming a 440 MB `go mod download` record present at build N and
  absent by N+2 — not by absence of an error.
- CORRECTED: honoured ONLY in the `policy` array form. The flat form is
  SILENTLY ignored — daemon starts, logs nothing, keeps its defaults.
  `dockerd --validate` returned "configuration OK" for a bogus key AND for a
  config that then crashed the daemon. The oracle is `docker buildx inspect`.
- REFUTED: Docker's `data-root` would NOT move the cache — it moves 0.62 GB.
  The 181.4 GB belongs to the system containerd (`root` in
  /etc/containerd/config.toml).

P3 (operator-approved) executed: prune claimed 156.9 GB, the filesystem
returned 150.35 GB (the 6.5 GB gap is layers shared with images), SYNCHRONOUSLY
— / went 86% -> 53% used, and Longhorn's default disk went
Schedulable=False (DiskPressure) -> Schedulable=True (18.85% -> 50.32%).

P7 root-caused the largest item and it is NOT the cap: all 208 `go mod download`
records had Usage count 1. Isolated by controlled builds — same VERSION build-arg
-> CACHED, new VERSION -> executed, byte-identical tree. `ARG VERSION`/`ARG
GIT_COMMIT` sit ABOVE the module-download step, and a RUN's cache key includes
the stage environment. Both Dockerfiles have it. One line each to fix -> R-208.

P6 NOT EXECUTED — stops at the operator, as specified. Pre-analysis: the move is
safe as measured (+38.8 pp above the 25% floor) but SSD2 is the only Longhorn
disk with storageReserved=0 and is overcommitted 6.9x; at full inflation the move
lands 12 pp BELOW the floor. The prune removed the move's urgency, so CC
recommends against it unless ~80 GB is reserved on SSD2 -> R-209.

Register: R-205 (CLOSED, shipped), R-206 (Ansible: cap + prune + narrowed Docker
ban), R-207 (DRY_RUN guard), R-208 (ARG ordering), R-209/R-210 (operator),
R-211 (Prometheus has no config-reloader — rules changes have never applied
until something restarted the pod; found while verifying R-205).

Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).
This commit is contained in:
2026-08-05 10:02:46 +02:00
parent 0dbd954fec
commit ee9d9bf203
3 changed files with 677 additions and 0 deletions
+22
View File
@@ -47,6 +47,15 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
## What shipped recently
- **2026-08-05** — **The DooPlex server's disk is out of danger: 86% full → 54%, and the storage layer
is unstuck.** The cause was leftover working data from building our own software — 157 GB of it,
growing about 5 GB a day, which nothing was allowed to delete. **148 GB came back in 86 seconds.**
The storage layer had already stopped accepting new copies of any volume onto that disk; that is
fixed the same day. A **30 GB ceiling** is now in place and was **proved to work by deliberately
overfilling it and watching it evict** — not by assuming the setting took. Two things that failed
quietly around it: the warning meant to catch exactly this **could never fire** (fixed and proved),
and the weekly cleanup is still forbidden from touching the thing that grows (next session).
*(R-205 … R-211)*
- **2026-08-05** — **Three of the four recovery crutches removed.** The reset code works first time;
a credential re-issue no longer blocks off-site backups on a healthy machine; and the default
restore no longer quietly returns the wrong thing. *(R-204 items 13, R-196)*
@@ -70,6 +79,19 @@ over; and the file was restored **byte for byte identical**. *(R-201)*
## Waiting on you
- **Whether to move the build data onto the second SSD at all — and CC's advice is now "probably
not".** You ruled "cap it, then move it". The cap is in and it did the job on its own: the disk sits
at 54% with 199 GB free, and moving is no longer a rescue. The second SSD looks roomy (91% free) and
today the move is safe by a wide margin — but it is the **only** disk of the four with **no space
reserved for itself**, and its volumes are allowed to claim 6.9× more than they currently use. If
they ever grow into that, the move would push it below the same floor that just took the first disk
out of service. **If you want the move, do it together with reserving ~80 GB on that disk; otherwise
leaving it where it is costs nothing now.** *(R-209)*
- **One list to rule on: 193 old images that exist only on this machine.** 131 controller versions and
62 hub versions are not in the registry, so they cannot be re-downloaded — all of them old
(controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing
was deleted.** Worth knowing before you spend time on it: they only account for about 27 GB against
199 GB now free, so this is about clutter, not space. *(R-210)*
- **The one-shot credential decision — this is now the last thing between a customer and an unaided
recovery.** A rebuilt machine has no storage credential of its own, so an operator must press
Re-issue. Everything after that point is now self-service. Deciding how a rebuilt machine should get