docs: SPIKE — DooPlex build-cache containment; R-205..R-211 (2026-08-05)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
Spike output only; no production Go code. The one shipped change rides in homelab-manifests 6808a4b (R-205, the monitoring rule). VERDICT: mechanism confirmed, with one correction and one refutation. - CONFIRMED: builder.gc IS honoured under the containerd worker and DOES evict. Proven by naming a 440 MB `go mod download` record present at build N and absent by N+2 — not by absence of an error. - CORRECTED: honoured ONLY in the `policy` array form. The flat form is SILENTLY ignored — daemon starts, logs nothing, keeps its defaults. `dockerd --validate` returned "configuration OK" for a bogus key AND for a config that then crashed the daemon. The oracle is `docker buildx inspect`. - REFUTED: Docker's `data-root` would NOT move the cache — it moves 0.62 GB. The 181.4 GB belongs to the system containerd (`root` in /etc/containerd/config.toml). P3 (operator-approved) executed: prune claimed 156.9 GB, the filesystem returned 150.35 GB (the 6.5 GB gap is layers shared with images), SYNCHRONOUSLY — / went 86% -> 53% used, and Longhorn's default disk went Schedulable=False (DiskPressure) -> Schedulable=True (18.85% -> 50.32%). P7 root-caused the largest item and it is NOT the cap: all 208 `go mod download` records had Usage count 1. Isolated by controlled builds — same VERSION build-arg -> CACHED, new VERSION -> executed, byte-identical tree. `ARG VERSION`/`ARG GIT_COMMIT` sit ABOVE the module-download step, and a RUN's cache key includes the stage environment. Both Dockerfiles have it. One line each to fix -> R-208. P6 NOT EXECUTED — stops at the operator, as specified. Pre-analysis: the move is safe as measured (+38.8 pp above the 25% floor) but SSD2 is the only Longhorn disk with storageReserved=0 and is overcommitted 6.9x; at full inflation the move lands 12 pp BELOW the floor. The prune removed the move's urgency, so CC recommends against it unless ~80 GB is reserved on SSD2 -> R-209. Register: R-205 (CLOSED, shipped), R-206 (Ansible: cap + prune + narrowed Docker ban), R-207 (DRY_RUN guard), R-208 (ARG ordering), R-209/R-210 (operator), R-211 (Prometheus has no config-reloader — rules changes have never applied until something restarted the pod; found while verifying R-205). Gates: repo_gates.py --fast — all OK, rc=0 (run separately from this commit).
This commit is contained in:
@@ -0,0 +1,648 @@
|
||||
# SPIKE — DooPlex build-cache containment (cap at 30 GB, relocate to SSD2)
|
||||
|
||||
**Date:** 2026-08-05 · **Host:** DooPlex (192.168.0.180) — **Tier 2, protected**, not a drill target
|
||||
**Class:** Spike. §1, §2, §3, §4, §13-verify, §15 apply; §5–§12 skipped (no production Go code written).
|
||||
**Anchor:** `dooplex_report.md` (2026-08-05 read-only disk audit). Every number in it was treated as a
|
||||
claim to re-measure, not a fact to carry forward.
|
||||
|
||||
> Unit convention: **GB = decimal (10⁹)**, because `df -B1`, `du -B1` and Prometheus emit raw bytes.
|
||||
> `docker buildx inspect` renders **GiB** (2³⁰) — where that matters it is labelled.
|
||||
|
||||
---
|
||||
|
||||
## Verdict
|
||||
|
||||
**The mechanism is confirmed, with one correction and one refutation, and the follow-up is safe to spec.**
|
||||
|
||||
- **Confirmed:** a `builder.gc` policy in `/etc/docker/daemon.json` **is** honoured under the containerd
|
||||
worker, and it **does** evict. Proven by naming a 440 MB cache record present at one build and absent
|
||||
two builds later — not by the absence of an error.
|
||||
- **Corrected:** it is honoured **only in the `policy` array form**. The flat form
|
||||
(`{"gc": {"reservedSpace": …}}`) is **silently ignored** — the daemon starts, logs nothing, and keeps
|
||||
its built-in defaults. `dockerd --validate` is worthless here: it returned `configuration OK` for a
|
||||
bogus key **and** for a config that then crashed the daemon.
|
||||
- **Refuted:** the audit's proposal to change Docker's `data-root` would **not** move the build cache. It
|
||||
moves 0.62 GB. The 181.4 GB belongs to the **system containerd**, relocated via `root` in
|
||||
`/etc/containerd/config.toml`.
|
||||
- **The prune is done and it was the whole rescue.** `/` went **86% → 53% used**, and Longhorn's default
|
||||
disk went **`Schedulable=False (DiskPressure)` → `Schedulable=True`**. The live consequence is cleared.
|
||||
- **The largest single finding is not the cap.** 90.5 GB of the 157 GB was one layer re-downloading on
|
||||
every build, and the root cause is a **one-line Dockerfile ordering defect** — `ARG VERSION` declared
|
||||
above the module-download step. Fixing it removes the source; the cap only bounds the symptom.
|
||||
- **P6 (the SSD2 move) was NOT executed** — it stops at the operator, as specified. The pre-analysis
|
||||
says the move is safe as measured but carries a real Longhorn risk, and the prune removed its urgency.
|
||||
|
||||
---
|
||||
|
||||
## 1. Confirmed baselines
|
||||
|
||||
| Repo | `main` @ commit at spike start | Drift vs the task's stated baseline |
|
||||
|---|---|---|
|
||||
| `homelab-manifests` | `5188058d0e43` | none — matched |
|
||||
| `felhom.eu` | `0dbd954fec90` | none — matched |
|
||||
|
||||
Both trees clean (`git status --porcelain` empty) and `HEAD == origin/main` before any commit.
|
||||
|
||||
**Register:** highest allocated at baseline was **R-204**, re-confirmed in `OPEN-ITEMS.md`. **R-205**
|
||||
onward were free and are allocated here (R-205 … R-211).
|
||||
|
||||
**One path correction:** `homelab-manifests` is **not** in `/mnt/5_hdd/felhom.eu/git/` with the other
|
||||
repos — it lives at **`/home/kisfenyo/git/homelab-manifests`**.
|
||||
|
||||
---
|
||||
|
||||
## 2. P0 — arrival state, expected vs measured
|
||||
|
||||
Measured live before anything was changed. **Values that matched are recorded, not omitted.**
|
||||
|
||||
| Fact | Expected (audit) | **Measured** | |
|
||||
|---|---|---|---|
|
||||
| Root fs size | 477,393,182,720 B | **477,393,182,720 B** | match |
|
||||
| Root fs used | 387.3 GB | **387,332,755,456 B** | match (+10.8 MB in 2 h) |
|
||||
| Root fs avail | 65.7 GB | **65,734,934,528 B** | match |
|
||||
| Root fs percent | 86% | **86%** | match |
|
||||
| Docker server version | 29.6.2 | **29.6.2** (build `3d80467`) | match |
|
||||
| Storage driver | overlayfs, containerd snapshotter | **`overlayfs`, `driver-type: io.containerd.snapshotter.v1`** | match |
|
||||
| Docker Root Dir | `/var/lib/docker`, ~0.6 GB | **`/var/lib/docker`, 618,541,056 B** | match |
|
||||
| containerd root | `/var/lib/containerd`, ~181.4 GB | **181,394,223,104 B** | match |
|
||||
| Build cache total / reclaimable | 157 GB / 147.8 GB | **157 GB / 147.8 GB**, 1446 records | match |
|
||||
| Images / containers | 345 images, 2 running | **345 unique** (350 tag lines), **2 running** | match |
|
||||
| `/etc/docker/daemon.json` | does not exist | **does not exist** (`/etc/docker/` exists, empty) | match |
|
||||
| `/etc/buildkit/` | does not exist | **does not exist** | match |
|
||||
| SSD2 device / mount / fs / free | ~240 GB, ~203 GB free | **`/dev/sde2`, `/mnt/ssd_2`, ext4, 235,148,750,848 B fs, 203,034,353,664 B free** | match |
|
||||
| SSD1 Longhorn | `Schedulable=False (DiskPressure)` | **`Schedulable=False`, `DiskPressure`, 18.85% avail** | match |
|
||||
| HDD2 Longhorn | also False | **`Schedulable=False`, `DiskPressure`, 24.91% avail** | match |
|
||||
| Housekeeping timer | active, last Sun 2026-08-02 | **active**, last **2026-08-02 05:35:10 CEST**, next 08-09, `ExecMainStatus=0` | match |
|
||||
|
||||
**Prohibition honoured:** `node-housekeeping.sh` was **not run**, in any mode. Its state was read from
|
||||
systemd, from the textfile on disk, and from Prometheus. The textfile's mtime is still
|
||||
`2026-08-02 05:35:11` — the 8-week reclaim history is intact.
|
||||
|
||||
**What the audit did not measure — the SSD2 Longhorn record** (`Crucial-SSD-240G`,
|
||||
`/mnt/ssd_2/data/longhorn`):
|
||||
|
||||
| field | value |
|
||||
|---|---:|
|
||||
| `storageMaximum` | 235,148,750,848 |
|
||||
| `storageAvailable` | 214,958,080,000 (**91.41%**) |
|
||||
| `storageScheduled` | 139,586,437,120 |
|
||||
| `storageReserved` | **0** |
|
||||
| condition | **`Schedulable=True`** |
|
||||
| replicas | **12** |
|
||||
|
||||
**Two things stand out and both matter for P6.** SSD2 is the **only** Longhorn disk reserving **zero**
|
||||
(HDD2 and HDD4 each reserve 500 GB). And `storageScheduled` (139.6 GB) exceeds actual usage
|
||||
(`df`: 20,094,939,136 B) by **6.9×** — Longhorn is heavily overcommitted on this disk.
|
||||
|
||||
---
|
||||
|
||||
## 3. P1 — which daemon owns the bytes
|
||||
|
||||
> **The single configuration change that relocates the build cache is `root = "<path>"` in
|
||||
> `/etc/containerd/config.toml`, owned by `containerd.service` — the SYSTEM containerd
|
||||
> (PID 2698677, socket `/run/containerd/containerd.sock`).**
|
||||
|
||||
**The audit's `data-root` proposal is refuted, and the split is measured:**
|
||||
|
||||
| tree | size | moved by `data-root`? |
|
||||
|---|---:|---|
|
||||
| `/var/lib/docker` | **618,541,056 B** (volumes 321 MB, **buildkit metadata 282 MB**, containers 14 MB) | yes |
|
||||
| `/var/lib/containerd` | **181,394,223,104 B** (overlayfs snapshots **170,357,870,592 B**, content 10.98 GB) | **no** |
|
||||
|
||||
`dockerd` runs as `/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock` with **no
|
||||
systemd drop-ins**. BuildKit keeps its *record database* under Docker Root Dir (0.28 GB) but its *layer
|
||||
bytes* in containerd's snapshotter — which is why `data-root` looks plausible and is wrong.
|
||||
|
||||
In `/etc/containerd/config.toml` the `root` key is **present but commented out**, so it defaults to
|
||||
`/var/lib/containerd`. Uncommenting it is the whole change.
|
||||
|
||||
**k3s is unaffected — confirmed, not assumed.** k3s runs a **separate** containerd instance with its own
|
||||
socket (`/run/k3s/containerd/containerd.sock`) and its own root
|
||||
(`/var/lib/rancher/k3s/agent/containerd`, **79,669,473,280 B**). Nothing in this spike touches it.
|
||||
|
||||
---
|
||||
|
||||
## 4. P2 — which GC key schema this release accepts
|
||||
|
||||
**`dockerd --validate` exists on 29.6.2 and is not an oracle for this subtree.** Measured:
|
||||
|
||||
| candidate | `--validate` | reality |
|
||||
|---|---|---|
|
||||
| `defaultKeepStorage` (old flat form) | `configuration OK` | — |
|
||||
| `policy` array | `configuration OK` | **honoured** |
|
||||
| `reservedSpace` / `maxUsedSpace` / `minFreeSpace` flat | `configuration OK` | **silently ignored** |
|
||||
| **`totallyBogusKey`** | **`configuration OK`** | proves the validator does no field checking |
|
||||
| `policy` with `"filter": [three values]` | **`configuration OK`** | **crashed the daemon on restart** |
|
||||
|
||||
> **The real oracle is `docker buildx inspect`, which renders the *effective* policy.** That is an
|
||||
> assertion of effect; `--validate` and "no error in the log" are both assertions of absence, and both
|
||||
> were measured to lie here.
|
||||
|
||||
**The flat form is silently ignored.** With `{"gc":{"enabled":true,"reservedSpace":"1GB","maxUsedSpace":"2GB","minFreeSpace":"10GB"}}`
|
||||
applied and the daemon restarted, `buildx inspect` still reported the built-in defaults, and
|
||||
`journalctl -u docker.service` carried **no warning, no deprecation notice and no error**.
|
||||
|
||||
**The `filter` key takes ONE value per policy entry, not an array.** An array is fatal at startup:
|
||||
|
||||
```
|
||||
error initializing buildkit: error creating buildkit instance: filters expect only one value
|
||||
```
|
||||
|
||||
The daemon then entered a systemd restart loop. **This is also how the two `unless-stopped` dev
|
||||
containers ended up stopped** — and they did **not** come back when the daemon later started cleanly;
|
||||
they needed a manual `docker start`. A follow-up that templates this file must **restart and verify**,
|
||||
never validate-and-assume.
|
||||
|
||||
**Documented statement on containerd-worker applicability:** none found in `dockerd --help` or the
|
||||
installed docs. Recorded as **unknown from documentation**, and settled empirically by P5.
|
||||
|
||||
**The default policy is not "no policy".** Before any change, `buildx inspect` reported built-in rules
|
||||
with **Reserved 41.91 GiB / Max Used 331.6 GiB / Min Free 82.89 GiB**. The audit's "there is no BuildKit
|
||||
GC policy on this host" is *almost* right — there was one, sized so large it could never bind: the cache
|
||||
at 157 GB was less than half the 331.6 GiB ceiling.
|
||||
|
||||
### The validated `daemon.json`, verbatim (final state, left in place deliberately)
|
||||
|
||||
```json
|
||||
{
|
||||
"builder": {
|
||||
"gc": {
|
||||
"enabled": true,
|
||||
"policy": [
|
||||
{ "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" },
|
||||
{ "all": true, "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Effective policy confirmed live (note the **GiB** rendering — `30GB` in JSON becomes a 30 GiB rule):
|
||||
|
||||
```
|
||||
GC Policy rule#0: All: false Reserved Space: 10GiB Max Used Space: 30GiB Min Free Space: 20GiB
|
||||
GC Policy rule#1: All: true Reserved Space: 10GiB Max Used Space: 30GiB Min Free Space: 20GiB
|
||||
```
|
||||
|
||||
**No deprecation warning was emitted** for any accepted form.
|
||||
|
||||
---
|
||||
|
||||
## 5. P3 — the prune: claimed, actual, settle
|
||||
|
||||
Pre-checked: no build in progress, P0 and P7 captured first.
|
||||
|
||||
| | value |
|
||||
|---|---:|
|
||||
| **Claimed by `docker builder prune -af`** | **156.9 GB** (1444 records deleted, 86 s wall-clock) |
|
||||
| **Actual filesystem reclaim** | **150,353,829,888 B (150.35 GB)** |
|
||||
| **Time to settle** | **synchronous — full reclaim visible at t+0** |
|
||||
|
||||
Settle poll (t+0 already showed +150,354,477,056 B; two consecutive readings <50 MB apart at t+60s;
|
||||
the residual −0.4 MB and −0.3 MB steps are ordinary background writes, not lagging reclaim):
|
||||
|
||||
| t | avail | step delta |
|
||||
|---|---:|---:|
|
||||
| 0 s | 215,947,522,048 | — |
|
||||
| 30 s | 215,947,141,120 | −380,928 |
|
||||
| 60 s | 215,946,874,880 | −266,240 |
|
||||
|
||||
**The 6.5 GB gap between claimed and actual is not an error** — it is the layers still shared with
|
||||
retained images (`Shared: 8.6 GB` before the prune). **A weekly job must measure the filesystem, not
|
||||
trust the command's `Total:`.**
|
||||
|
||||
**Unlike containerd's image GC, this returns bytes synchronously**, so the follow-up needs no
|
||||
`settle_imagefs` equivalent — but the settle poll is what *established* that, and it cost 60 seconds.
|
||||
|
||||
| | before | after |
|
||||
|---|---:|---:|
|
||||
| `/` used | 387,474,644,992 B (**86%**) | 240,301,133,824 B (**53%**) |
|
||||
| `/` avail | 65,593,044,992 B | 215,946,874,880 B |
|
||||
| `/var/lib/containerd` | 181,537,714,176 B | 31,160,053,760 B |
|
||||
| build cache | 156.9 GB / 1443 records | **0 B / 0 records** |
|
||||
| images | 345 | **345** (untouched) |
|
||||
| dev containers | 2 healthy | **2 healthy** |
|
||||
|
||||
### The live consequence, cleared
|
||||
|
||||
Longhorn re-evaluated within 60 s:
|
||||
|
||||
| disk | before | after |
|
||||
|---|---|---|
|
||||
| `default-disk-…` (SSD1, `/var/lib/longhorn/`) | `Schedulable=False (DiskPressure)`, 89,967,820,800 B, **18.85%** | **`Schedulable=True`**, 240,228,761,600 B, **50.32%** |
|
||||
| `HDD2` | `Schedulable=False (DiskPressure)`, 24.91% | **unchanged — still False, 24.91%** |
|
||||
|
||||
**HDD2 is untouched by this work and remains unschedulable.** It is a separate item, stated here so it
|
||||
is not read as fixed by association.
|
||||
|
||||
---
|
||||
|
||||
## 6. P5 — proving the ceiling is enforced
|
||||
|
||||
**Rollback stated before the first mutation** and never needed beyond the recovery below:
|
||||
`sudo rm /etc/docker/daemon.json && sudo systemctl restart docker` — the file did not exist, so removal
|
||||
is a complete revert.
|
||||
|
||||
Test ceiling **2 GB** (below one build's working set, deliberately, so eviction is reachable).
|
||||
|
||||
### Build-by-build totals
|
||||
|
||||
| step | build cache total | reclaimable | records |
|
||||
|---|---:|---:|---:|
|
||||
| baseline (post-prune) | 0 B | 0 B | 0 |
|
||||
| controller 1 | 2.583 GB | 2.583 GB | 22 |
|
||||
| hub 1 | **3.703 GB** ← peak | 3.703 GB | 37 |
|
||||
| controller 2 | 3.525 GB | 3.525 GB | 37 |
|
||||
| hub 2 | 3.051 GB | 3.051 GB | 38 |
|
||||
| controller 3 | 2.817 GB | 2.817 GB | 36 |
|
||||
| hub 3 | **2.782 GB** | 2.782 GB | 38 |
|
||||
|
||||
**The total stops rising and trends down** — the cap binds. Records plateau at 36–38.
|
||||
|
||||
### The named eviction — the assertion that matters
|
||||
|
||||
Set-differencing the record IDs, **4 records present at controller-build-1 were absent by
|
||||
controller-build-3**, including the exact class that accumulated 208 copies:
|
||||
|
||||
| ID | size | created | description |
|
||||
|---|---:|---|---|
|
||||
| **`9ak8egnxrg6rkibic2zo44u1`** | **440 MB** | 07:28:34 | **`mount / from exec /bin/sh -c go mod download \|\| true`** |
|
||||
| `kkic530irq4em04kywzwrsjp` | 214.6 MB | 07:28:37 | `… go build -ldflags …` |
|
||||
| `yvityhta4wppwa8cxs525c3g` | 58.83 MB | 07:28:36 | `… go mod tidy` |
|
||||
| `mvttjfrimnt1ttt0ogry75ue` | 49.8 MB | 07:28:35 | `[builder 5/7] COPY . .` |
|
||||
|
||||
Across the whole sequence: `h1→h3` evicted **17** records, `c2→c3` evicted **10**.
|
||||
|
||||
### Tolerance, stated honestly
|
||||
|
||||
Steady state settled at **2.78–2.82 GB against a 2 GiB (2.147 GB) rule — about 30% over**, with a peak
|
||||
of **3.70 GB (~72% over)** mid-sequence.
|
||||
|
||||
**This overshoot is an artefact of the test, not a property of the cap, and should not be extrapolated to
|
||||
30 GB.** A single controller build produces ~2.6 GB of cache in one shot, so a 2 GB ceiling was set
|
||||
*below one build's working set* — GC cannot get under it without deleting the cache it just created. At
|
||||
30 GB the same working set is ~9% of the ceiling and GC has ample room.
|
||||
|
||||
### GC is build-triggered, not timer-driven
|
||||
|
||||
Six minutes idle, polled each minute: **total unchanged at 2.782 GB, records unchanged at 38.** The
|
||||
cache does not shrink on its own. A burst of builds can overshoot and *stay* overshot until the next
|
||||
build — which is the argument for keeping a weekly `docker builder prune` as a backstop (R-206), rather
|
||||
than relying on the cap alone.
|
||||
|
||||
### Then the ruled 30 GB
|
||||
|
||||
Applied, restarted, and confirmed by the effective policy (§4). Both dev containers came back healthy
|
||||
and **their data is intact — asserted positively, not by absence of complaint**:
|
||||
|
||||
| observable | before | after |
|
||||
|---|---:|---:|
|
||||
| postgres `count(*) from pg_database` | 4 | **4** |
|
||||
| postgres `pg_database_size('jarr_dev')` | 175,135,767 | **175,135,767** |
|
||||
| postgres public tables | 31 | **31** |
|
||||
| redis `DBSIZE` | 2437 | **2437** |
|
||||
| docker volumes | 8 | **8, identical names** |
|
||||
|
||||
---
|
||||
|
||||
## 7. P7 — why the module-download layer re-runs
|
||||
|
||||
**Root cause found and proven: `ARG VERSION` / `ARG GIT_COMMIT` are declared in the builder stage
|
||||
*above* the module-download step, so a new version value invalidates every `RUN` beneath them.**
|
||||
|
||||
### The evidence chain
|
||||
|
||||
**1. Every record was used exactly once.** All **208** `go mod download` records (206 at audit time, +2
|
||||
from this spike's builds) carried **`Usage count: 1`**, spanning 2026-07-06 → 2026-08-05, totalling
|
||||
91.36 GB at 438–440 MB each. Not one was ever reused.
|
||||
|
||||
**2. Two builds through `build.sh`, byte-identical tree, both missed.**
|
||||
|
||||
| step | build 1 | build 2 |
|
||||
|---|---|---|
|
||||
| `#8 WORKDIR /build` | CACHED | CACHED |
|
||||
| `#9 COPY go.mod ./` | **CACHED** | **CACHED** |
|
||||
| **`#10 RUN go mod download \|\| true`** | **executed** | **executed** |
|
||||
| `#11 COPY . .` | executed | executed |
|
||||
| `#12 RUN go mod tidy` | executed | executed |
|
||||
| `#13 RUN … go build …` | executed | executed |
|
||||
| `#14/#15/#16` (stage-1 `RUN`s) | CACHED | CACHED |
|
||||
|
||||
`go.mod` was **byte-identical** (`d8ded3f0798c…`) in the repo and in the workspace, mtime unchanged
|
||||
(`2026-07-24 11:54:08`).
|
||||
|
||||
**3. `build.sh`'s churn was ruled out by fingerprinting.** Content-hashing all 737 files across
|
||||
`rm -rf` + `cp -a` + host-side `go mod tidy`: **`tidy` changes nothing**, and the reproduced tree is
|
||||
**content-identical** to the live workspace. `go.mod`'s inode number was even identical across the churn.
|
||||
|
||||
**4. A direct `docker build` on the untouched workspace cached *everything*** — including
|
||||
`RUN go mod download` — proving BuildKit's cache is healthy and the Dockerfile is cacheable.
|
||||
|
||||
**5. The decisive isolation.** Same workspace, same command, only the build-arg value varied:
|
||||
|
||||
| run | `--build-arg VERSION` | `COPY go.mod ./` | **`RUN go mod download`** |
|
||||
|---|---|---|---|
|
||||
| E1 | **new** (`spike-p7e`) | CACHED | **executed** |
|
||||
| E2 | **same** (`spike-p7e`) | CACHED | **CACHED** |
|
||||
|
||||
**The mechanism, and it explains the `#9 CACHED / #10 executed` signature exactly:** a `COPY`'s cache key
|
||||
is content-based and carries no environment, so it hits; a `RUN`'s key includes the stage **environment**,
|
||||
which contains every `ARG` declared above it. Since every real build passes a fresh `VERSION` (and often
|
||||
a fresh `GIT_COMMIT`), the module-download layer is invalidated **on every build, without exception** —
|
||||
206 builds, 206 retained 440 MB snapshots, 90.5 GB.
|
||||
|
||||
**Both Dockerfiles have it.** `felhom.eu/hub/Dockerfile` declares `ARG VERSION` / `ARG BUILD_TIME` above
|
||||
`COPY go.mod go.sum*` → `RUN go mod download`. **And because both produce byte-identical `buildx du`
|
||||
description strings, the 208 records are a COMBINED count** and must not be attributed to one project.
|
||||
|
||||
**Fix shape — not applied here, filed as R-208:** move the `ARG` declarations down to just above the
|
||||
final `go build`. One line per Dockerfile. **This is worth more than the cap and the move combined:** the
|
||||
cap bounds the symptom at 30 GB; this removes the source.
|
||||
|
||||
---
|
||||
|
||||
## 8. P4 — the image store, enumerated (nothing deleted)
|
||||
|
||||
345 unique images across 350 tag lines (5 double-tagged), **0 dangling**. Only two are held by running
|
||||
containers: `redis:7-alpine` and `postgres:16-alpine`. `docker image prune -a` would remove **343**.
|
||||
|
||||
**Size the prize before spending a decision on it.** Per-tag sizes sum to 139.29 GB, but that
|
||||
double-counts shared layers. `docker system df` puts the **real dedup'd footprint at 31.02 GB, of which
|
||||
27.02 GB is reclaimable** — roughly a fifth of what the build-cache prune already returned.
|
||||
|
||||
| GB (per-tag sum) | tags | repository |
|
||||
|---:|---:|---|
|
||||
| 118.44 | **207** | `gitea.dooplex.hu/admin/felhom-controller` |
|
||||
| 7.25 | **107** | `gitea.dooplex.hu/admin/felhom-hub` |
|
||||
| 4.34 | 5 | `gitea.dooplex.hu/admin/jarr` |
|
||||
| 1.69 | 1 | `ghcr.io/seanmorley15/adventurelog-backend` |
|
||||
| 0.94 | 1 | `ghcr.io/papra-hq/papra` |
|
||||
| 0.72 | 1 | `codewithcj/sparkyfitness_server` |
|
||||
| 0.72 | 1 | `postgis/postgis` |
|
||||
| 0.59 | 3 | `gitea.dooplex.hu/admin/sparkyfitness-export` |
|
||||
| (23 more, each ≤0.53 GB) | | |
|
||||
|
||||
### Registry presence — which are recoverable
|
||||
|
||||
Queried against the live Gitea registry v2 API:
|
||||
|
||||
| repo | tags in registry | tags local | **local-only (NOT re-pullable)** |
|
||||
|---|---:|---:|---:|
|
||||
| `felhom-controller` | 76 | 207 | **131** |
|
||||
| `felhom-hub` | 45 | 107 | **62** |
|
||||
|
||||
**The local-only tags are all old.** Controller `0.39.0`–`0.135.0` plus `v0.35.0`–`v0.39.0`; hub
|
||||
`0.9.0`–`0.57.0` plus `v0.7.2`–`v0.13.0`. **Everything from controller `0.136.0` and hub `0.58.0`
|
||||
upward is in the registry** and therefore recoverable by `docker pull`.
|
||||
|
||||
**This is the list the operator still owes a ruling on (R-210).** CC's view: not worth doing for the
|
||||
space — ~27 GB against 199 GB now free. The only real benefit is dropping unrecoverable clutter.
|
||||
|
||||
---
|
||||
|
||||
## 9. P8 — what else writes to Docker
|
||||
|
||||
**The bounded list of build triggers on the host Docker daemon is: interactive/manual runs only.**
|
||||
|
||||
- **18 systemd timers** enumerated — none invokes docker. (`node-housekeeping`, `dooplex-backup`,
|
||||
`apt-daily*`, `sysstat*`, `logrotate`, `fstrim`, `e2scrub_all`, `smart-fancontrol`, …)
|
||||
- **Root crontab: empty** (comments only). **User crontab: one entry**, `sync-hetzner-backups.sh`,
|
||||
writes under `/mnt/5_hdd`.
|
||||
- **`/etc/cron.d`: 4 real files** — anacron, e2scrub_all, sysstat, zfsutils. None touches docker.
|
||||
|
||||
**The hole the audit left open is now closed with positive evidence.** The audit concluded "no CI runner
|
||||
exists on this host" from systemd/process/crontab alone — but a Gitea Actions runner **does** exist as a
|
||||
**cluster pod** (R-168, 2026-08-02). It cannot reach the host daemon:
|
||||
|
||||
- **162 pods swept across all namespaces: `0` mount any `docker`/`containerd` hostPath.**
|
||||
- `act-runner` (`gitea-system`, image `felhom-act-runner:0.1.0`) has one volume — a PVC at `/data`.
|
||||
`privileged: false`, `allowPrivilegeEscalation: false`, **all capabilities dropped**, no docker socket.
|
||||
- Its Dockerfile states the absence is deliberate: *"DELIBERATELY NOT INSTALLED: a Go toolchain, Node, or
|
||||
Docker … There is no container runtime in here on purpose."*
|
||||
- `gates.yml` runs exactly one command — `python3 scripts/<repo>_gates.py --fast` — plus a git fetch and
|
||||
a failure-only Resend alarm. **No `docker build` anywhere.**
|
||||
|
||||
**So the audit's ~7 builds/day attribution to interactive use is confirmed**, and now on a complete
|
||||
enumeration rather than a partial one.
|
||||
|
||||
---
|
||||
|
||||
## 10. P9 — the monitoring rule fix (shipped)
|
||||
|
||||
**File:** `homelab-manifests/mon-system/prometheus-rules.yaml`, alert
|
||||
`RootFsPressureDespiteHousekeeping`. **Commit `6808a4b`.**
|
||||
|
||||
**Change:** `and` → **`and on(instance)`**. A bare `and` requires identical label sets; the left vector
|
||||
carries `{device,fstype,instance,job,mountpoint}` and the right — a node_exporter *textfile* metric —
|
||||
carries only `{instance,job}`. Nothing ever matched.
|
||||
|
||||
### Red-proof (required, and both numbers recorded)
|
||||
|
||||
**Threshold substituted `0.25` → `0.5`, and the reason is stated rather than hidden:** P3's prune had
|
||||
already taken `/` to 44.6% available, so the shipped threshold's left side is now legitimately false. The
|
||||
defect is in **label matching**, not in the threshold, so the proof is run at a threshold where both
|
||||
sides are individually true — which is exactly the condition the original rule was meant to catch.
|
||||
|
||||
| expression | `result_count` |
|
||||
|---|---:|
|
||||
| left side alone (`avail/size < 0.5`) | **1** |
|
||||
| right side alone (`time() − last_success < 1209600`) | **1** |
|
||||
| **as shipped — bare `and`** | **0** |
|
||||
| **fixed — `and on(instance)`** | **1** |
|
||||
|
||||
For completeness, at the shipped 0.25 threshold **both** forms return 0 today — because the left side is
|
||||
false now, not because the fix failed.
|
||||
|
||||
**Annotation rewritten.** It pointed at Longhorn replicas/snapshots (~10% of the growth). It now names
|
||||
`docker buildx du` first, records the 30 GB cap in `/etc/docker/daemon.json`, and notes that a reverted
|
||||
or unapplied cap looks exactly like this alert firing — so the alert text carries its own diagnosis.
|
||||
|
||||
### Deployment — GitOps, no `kubectl set image`, no live ConfigMap edit
|
||||
|
||||
ArgoCD app `monitoring` (auto-sync **off**): hard-refresh → `OutOfSync` at `6808a4be54a7` → deliberate
|
||||
sync → **`Synced` / `Healthy`**, operation phase **`Succeeded`**.
|
||||
|
||||
**Verified in Prometheus, not assumed from ArgoCD:**
|
||||
|
||||
```
|
||||
query : … < 0.25 and on (instance) time() - node_housekeeping_last_success_timestamp_seconds < 1209600
|
||||
state : inactive health : ok lastError : (none)
|
||||
```
|
||||
|
||||
`inactive` is now the **correct** answer — `/` really is fine at 44.6% available.
|
||||
|
||||
> **A near-miss worth recording: I invented a full commit SHA from ArgoCD's truncated display and the
|
||||
> sync failed with `upload-pack: not our ref`.** It failed loudly and cost one retry. It would not have
|
||||
> failed loudly had the fabricated SHA happened to exist.
|
||||
|
||||
### The finding P9 turned up on the way — R-211
|
||||
|
||||
**There is no config-reloader.** The `mon-system/prometheus` Deployment runs **one** container
|
||||
(`prom/prometheus:v3.12.0`) with no reloader sidecar. After the sync the updated rules file was present
|
||||
**inside the pod** (`grep -c "and on(instance)"` → 3 on the mounted symlink) while the Prometheus rules
|
||||
API **still served the old expression for 4+ minutes**, with no error anywhere. It took effect only after
|
||||
an explicit `POST /-/reload` (HTTP 200; `--web.enable-lifecycle` was already set).
|
||||
|
||||
**The consequence is general:** every rule edit in this repo has silently not applied until something
|
||||
happened to restart the pod. `Synced/Healthy` was true and beside the point. Filed as **R-211**.
|
||||
|
||||
---
|
||||
|
||||
## 11. P6 — the SSD2 move: pre-analysis only. **NOT EXECUTED.**
|
||||
|
||||
**Stopped here as specified. Nothing was moved, no daemon was reconfigured for SSD2, no mount unit was
|
||||
written.** The operator must be present.
|
||||
|
||||
**First, the framing changed underneath this probe.** P3's prune took `/` to 53% used and un-stuck
|
||||
SSD1's Longhorn disk. **The move is no longer a rescue; it is a deliberate choice.** Per standing rule 4,
|
||||
CC's disagreement is stated rather than absorbed: **the cap alone resolved the problem the move was meant
|
||||
to solve, and the move carries a Longhorn risk the cap does not.**
|
||||
|
||||
### Mechanism — three candidates
|
||||
|
||||
| | mechanism | survives package upgrade | expressible in Ansible | visible reading the host cold |
|
||||
|---|---|---|---|---|
|
||||
| **(a)** | **`root = "/mnt/ssd_2/…"` in `/etc/containerd/config.toml`** | **yes** — dpkg conffile, local edits preserved | **yes** — a template, the file is already managed-shaped | **yes** — one line in the daemon's own config |
|
||||
| (b) | systemd mount unit / fstab bind onto `/var/lib/containerd` | yes | yes | **weak** — invisible unless you check `findmnt` |
|
||||
| (c) | symlink | fragile | yes | **weak**, and overlayfs mount semantics through a symlink are unproven here |
|
||||
|
||||
**Recommendation: (a).** It is the only one that a person reading the host cold can *see*, and the key
|
||||
already exists in the file (commented out).
|
||||
|
||||
### The boot-order trap, and its guard
|
||||
|
||||
If containerd or Docker starts before `/mnt/ssd_2` is mounted, the daemon **creates an empty store on the
|
||||
root disk** and 345 images appear to have vanished.
|
||||
|
||||
**Guard:** `RequiresMountsFor=/mnt/ssd_2` on **`containerd.service`** and **`docker.service`**. Neither
|
||||
has it today — `containerd.service` carries only `After=network.target dbus.service`.
|
||||
|
||||
> **`RequiresMountsFor` RE-MOUNTS; it does not refuse.** systemd pulls the mount unit in and *starts* it,
|
||||
> so an unmounted-but-available device is silently re-mounted and everything looks fine. It refuses only
|
||||
> when the device is genuinely **unavailable**. **Testing it by unmounting proves nothing** — it must be
|
||||
> tested by pointing at an absent device. This is the ep0 datastore lesson
|
||||
> (`RUNBOOK-ep0-datastore-volume-2026-07-27.md`), earned on this project.
|
||||
|
||||
**And a move that has not survived a reboot has not been validated.** Mount-ordering reasoning is exactly
|
||||
the class this project has been burned by.
|
||||
|
||||
### The Longhorn consequence — the part that decides it
|
||||
|
||||
`Crucial-SSD-240G`, `storageMaximum` 235,148,750,848, **25% floor = 58,787,187,712**:
|
||||
|
||||
| scenario | resulting `storageAvailable` | as % | vs the 25% floor |
|
||||
|---|---:|---:|---:|
|
||||
| today, untouched | 214,958,080,000 | 91.41% | +66.4 pp |
|
||||
| move cache only (30 GB) | 184,958,080,000 | 78.66% | **+53.7 pp** |
|
||||
| move whole containerd tree today (34.3 GB) | 180,658,569,216 | 76.83% | +51.8 pp |
|
||||
| **move at steady state (~31.5 GB images + 30 GB cache ≈ 65 GB)** | 149,958,080,000 | **63.77%** | **+38.8 pp** |
|
||||
|
||||
**As measured, the move is safe by a wide margin — nowhere near the 10-pp caution band.**
|
||||
|
||||
**But the worst case is not comfortable, and it is not hypothetical enough to omit.** `storageScheduled`
|
||||
on SSD2 is **139,586,437,120** while actual usage is **20,094,939,136** — a 6.9× overcommit. If those 12
|
||||
volumes ever inflate to their scheduled size:
|
||||
|
||||
| scenario | available | as % | vs floor |
|
||||
|---|---:|---:|---|
|
||||
| + cache only (30 GB) | 65,562,313,728 | 27.88% | +2.9 pp — inside the caution band |
|
||||
| **+ steady state (65 GB)** | 30,562,313,728 | **13.00%** | **−12.0 pp → `Schedulable=False`** |
|
||||
|
||||
That is precisely the failure that just took SSD1 out of service.
|
||||
|
||||
> **Recommendation: proceed only together with setting Longhorn's `storageReserved` on SSD2 to cover the
|
||||
> containerd tree (~80 GB).** SSD2 reserving **zero** while HDD2 and HDD4 each reserve 500 GB is an
|
||||
> anomaly in its own right, and it is what lets Longhorn oversubscribe into the space the cache needs.
|
||||
> Without that, CC recommends **not moving** — the cap has already done the job.
|
||||
|
||||
### The sequence, prepared and not executed
|
||||
|
||||
Each step with its rollback. **Do not run without the operator.**
|
||||
|
||||
| # | step | rollback |
|
||||
|---|---|---|
|
||||
| 1 | `systemctl stop docker docker.socket containerd` | `systemctl start containerd docker` |
|
||||
| 2 | `mkdir -p /mnt/ssd_2/containerd` | `rmdir` |
|
||||
| 3 | `rsync -aHAX --numeric-ids /var/lib/containerd/ /mnt/ssd_2/containerd/` (small now — ~34 GB, and ~2.8 GB of it is cache) | delete the copy; the original is untouched |
|
||||
| 4 | `mv /var/lib/containerd /var/lib/containerd.pre-move` (**move aside, never delete**) | `mv` back |
|
||||
| 5 | set `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` | restore the commented line |
|
||||
| 6 | drop-in `RequiresMountsFor=/mnt/ssd_2` on `containerd.service` **and** `docker.service`; `systemctl daemon-reload` | delete the drop-ins |
|
||||
| 7 | `systemctl start containerd docker`; **verify `docker images` = 345, both dev containers up, `du` confirms the store's real path** | steps 5→4→1 in reverse |
|
||||
| 8 | set SSD2 `storageReserved` ≈ 80 GB in the Longhorn node CR | set back to 0 |
|
||||
| 9 | **`reboot`, then re-verify everything in step 7** | as above |
|
||||
| 10 | test the guard by pointing at an **absent** device — an unmount alone proves nothing | restore fstab |
|
||||
| 11 | only after a clean reboot + a full build: `rm -rf /var/lib/containerd.pre-move` | — none; do this last |
|
||||
|
||||
---
|
||||
|
||||
## 12. Teardown
|
||||
|
||||
**Nothing was provisioned.** No machine, no VM, no LXC guest, no hub record, no customer, no Longhorn
|
||||
volume, no PVC. There is nothing to decommission.
|
||||
|
||||
**Cleaned up:**
|
||||
|
||||
- The six candidate `daemon.json` files written for P2 (in the session scratch dir, never in
|
||||
`/etc/docker/`) — **removed**.
|
||||
- The scratch workspace fingerprint tree (`ws_sim`) — **removed**.
|
||||
- **Nine scratch image tags** created by the probe builds — `felhom-spike:{p7c,e1,e2}`,
|
||||
`felhom-controller:{spike-p7a,spike-p7b,spike-p7d,evict-c1,evict-c2,evict-c3}`,
|
||||
`felhom-hub:{evict-h1,evict-h2,evict-h3}` — **all removed**.
|
||||
- `felhom-controller:latest` **restored** to `sha256:bdcd4a4ae8e006f1…` (0.198.0), byte-matching the
|
||||
anchor captured before the first build. Image count back to **345**, the P0 baseline.
|
||||
|
||||
**Intentional state left in place, recorded here so it does not read as a leftover:**
|
||||
**`/etc/docker/daemon.json` exists**, with the content in §4. It did not exist before this session.
|
||||
Removing it plus a daemon restart is a complete revert.
|
||||
|
||||
**Not cleaned up, deliberately:** the ~2.78 GB of build cache from the eviction sequence. It is the
|
||||
working cache under a live 30 GB policy, not scratch.
|
||||
|
||||
---
|
||||
|
||||
## 13. Final state
|
||||
|
||||
| | at arrival | **now** |
|
||||
|---|---:|---:|
|
||||
| `/` used | 387,332,755,456 B (**86%**) | **240,301,133,824 B (54%)** |
|
||||
| `/` available | 65,734,934,528 B | **212,766,556,160 B** |
|
||||
| `/var/lib/containerd` | 181,394,223,104 B | **34,299,510,784 B** |
|
||||
| build cache | 157 GB / 1446 records | **2.782 GB / 38 records, capped at 30 GiB** |
|
||||
| images | 345 | **345** |
|
||||
| dev containers | 2 healthy | **2 healthy, data verified intact** |
|
||||
| SSD1 Longhorn | **`Schedulable=False (DiskPressure)`** | **`Schedulable=True`, 50.32%** |
|
||||
| `RootFsPressureDespiteHousekeeping` | **dead — could never fire** | **live, `health=ok`, red-proven** |
|
||||
| growth driver | unbounded, ~5.25 GB/day | **capped; source diagnosed (R-208)** |
|
||||
|
||||
---
|
||||
|
||||
## 14. Observations — noticed, deliberately not acted on
|
||||
|
||||
1. **`build.sh` exists in two copies** — `felhom-controller/controller/build.sh` (in git) and
|
||||
`/mnt/5_hdd/felhom.eu/build/felhom-controller/build.sh` (the operative one). They differ by **one
|
||||
comment line**. The hub solved this with a symlink into the repo; the controller has not. Not fixed —
|
||||
out of scope, but it is a drift channel.
|
||||
2. **`build.sh` tags `:latest` on every build**, including throwaway version tags. A spike build silently
|
||||
moved `latest` off 0.198.0; it was restored from an anchor captured beforehand. A `--no-latest` flag,
|
||||
or tagging `latest` only on `--push`, would remove the trap.
|
||||
3. **The default BuildKit GC policy is derived from disk size**, so a bigger root disk would have raised
|
||||
the 331.6 GiB ceiling too. The cap needs to be explicit no matter how much disk is added.
|
||||
4. **`node-housekeeping.sh`'s own `avail_bytes()` uses `df … | tail -1`** — the truncating-instrument
|
||||
shape this project has been bitten by twice. It is *correct* here (`df` on one path emits exactly one
|
||||
data row), so it was left alone, but it is worth knowing it is there.
|
||||
5. **HDD2 remains `Schedulable=False (DiskPressure)`** at 24.91%, a hair under the 25% floor with 2.47 TB
|
||||
free and its 500 GB reserve satisfied. Untouched by this work, and not fixed by it.
|
||||
6. **The `mon-system` Prometheus has `--web.enable-admin-api`** as well as `--web.enable-lifecycle`. The
|
||||
admin API permits series deletion. Noted, not acted on.
|
||||
7. **Longhorn on SSD2 is overcommitted 6.9×** (139.6 GB scheduled vs 20.1 GB used). That is normal for
|
||||
thin provisioning, but combined with `storageReserved = 0` it is what makes the P6 worst case reach
|
||||
the floor.
|
||||
|
||||
---
|
||||
|
||||
## 15. Register
|
||||
|
||||
Allocated this session: **R-205** (CLOSED — the monitoring rule, shipped), **R-206** (Ansible: cap +
|
||||
prune + narrowed Docker ban), **R-207** (the `DRY_RUN` guard), **R-208** (the `ARG`-ordering cache miss —
|
||||
the largest remaining win), **R-209** (WAITING-ON-OPERATOR — whether to move to SSD2 at all),
|
||||
**R-210** (WAITING-ON-OPERATOR — the image list), **R-211** (no Prometheus config-reloader).
|
||||
Reference in New Issue
Block a user