diff --git a/STATUS.md b/STATUS.md index 2a15a96..0c2a831 100644 --- a/STATUS.md +++ b/STATUS.md @@ -47,6 +47,15 @@ over; and the file was restored **byte for byte identical**. *(R-201)* ## What shipped recently +- **2026-08-05** — **The DooPlex server's disk is out of danger: 86% full → 54%, and the storage layer + is unstuck.** The cause was leftover working data from building our own software — 157 GB of it, + growing about 5 GB a day, which nothing was allowed to delete. **148 GB came back in 86 seconds.** + The storage layer had already stopped accepting new copies of any volume onto that disk; that is + fixed the same day. A **30 GB ceiling** is now in place and was **proved to work by deliberately + overfilling it and watching it evict** — not by assuming the setting took. Two things that failed + quietly around it: the warning meant to catch exactly this **could never fire** (fixed and proved), + and the weekly cleanup is still forbidden from touching the thing that grows (next session). + *(R-205 … R-211)* - **2026-08-05** — **Three of the four recovery crutches removed.** The reset code works first time; a credential re-issue no longer blocks off-site backups on a healthy machine; and the default restore no longer quietly returns the wrong thing. *(R-204 items 1–3, R-196)* @@ -70,6 +79,19 @@ over; and the file was restored **byte for byte identical**. *(R-201)* ## Waiting on you +- **Whether to move the build data onto the second SSD at all — and CC's advice is now "probably + not".** You ruled "cap it, then move it". The cap is in and it did the job on its own: the disk sits + at 54% with 199 GB free, and moving is no longer a rescue. The second SSD looks roomy (91% free) and + today the move is safe by a wide margin — but it is the **only** disk of the four with **no space + reserved for itself**, and its volumes are allowed to claim 6.9× more than they currently use. If + they ever grow into that, the move would push it below the same floor that just took the first disk + out of service. **If you want the move, do it together with reserving ~80 GB on that disk; otherwise + leaving it where it is costs nothing now.** *(R-209)* +- **One list to rule on: 193 old images that exist only on this machine.** 131 controller versions and + 62 hub versions are not in the registry, so they cannot be re-downloaded — all of them old + (controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing + was deleted.** Worth knowing before you spend time on it: they only account for about 27 GB against + 199 GB now free, so this is about clutter, not space. *(R-210)* - **The one-shot credential decision — this is now the last thing between a customer and an unaided recovery.** A rebuilt machine has no storage credential of its own, so an operator must press Re-issue. Everything after that point is now self-service. Deciding how a rebuilt machine should get diff --git a/documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md b/documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md new file mode 100644 index 0000000..a18a481 --- /dev/null +++ b/documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md @@ -0,0 +1,648 @@ +# SPIKE — DooPlex build-cache containment (cap at 30 GB, relocate to SSD2) + +**Date:** 2026-08-05 · **Host:** DooPlex (192.168.0.180) — **Tier 2, protected**, not a drill target +**Class:** Spike. §1, §2, §3, §4, §13-verify, §15 apply; §5–§12 skipped (no production Go code written). +**Anchor:** `dooplex_report.md` (2026-08-05 read-only disk audit). Every number in it was treated as a +claim to re-measure, not a fact to carry forward. + +> Unit convention: **GB = decimal (10⁹)**, because `df -B1`, `du -B1` and Prometheus emit raw bytes. +> `docker buildx inspect` renders **GiB** (2³⁰) — where that matters it is labelled. + +--- + +## Verdict + +**The mechanism is confirmed, with one correction and one refutation, and the follow-up is safe to spec.** + +- **Confirmed:** a `builder.gc` policy in `/etc/docker/daemon.json` **is** honoured under the containerd + worker, and it **does** evict. Proven by naming a 440 MB cache record present at one build and absent + two builds later — not by the absence of an error. +- **Corrected:** it is honoured **only in the `policy` array form**. The flat form + (`{"gc": {"reservedSpace": …}}`) is **silently ignored** — the daemon starts, logs nothing, and keeps + its built-in defaults. `dockerd --validate` is worthless here: it returned `configuration OK` for a + bogus key **and** for a config that then crashed the daemon. +- **Refuted:** the audit's proposal to change Docker's `data-root` would **not** move the build cache. It + moves 0.62 GB. The 181.4 GB belongs to the **system containerd**, relocated via `root` in + `/etc/containerd/config.toml`. +- **The prune is done and it was the whole rescue.** `/` went **86% → 53% used**, and Longhorn's default + disk went **`Schedulable=False (DiskPressure)` → `Schedulable=True`**. The live consequence is cleared. +- **The largest single finding is not the cap.** 90.5 GB of the 157 GB was one layer re-downloading on + every build, and the root cause is a **one-line Dockerfile ordering defect** — `ARG VERSION` declared + above the module-download step. Fixing it removes the source; the cap only bounds the symptom. +- **P6 (the SSD2 move) was NOT executed** — it stops at the operator, as specified. The pre-analysis + says the move is safe as measured but carries a real Longhorn risk, and the prune removed its urgency. + +--- + +## 1. Confirmed baselines + +| Repo | `main` @ commit at spike start | Drift vs the task's stated baseline | +|---|---|---| +| `homelab-manifests` | `5188058d0e43` | none — matched | +| `felhom.eu` | `0dbd954fec90` | none — matched | + +Both trees clean (`git status --porcelain` empty) and `HEAD == origin/main` before any commit. + +**Register:** highest allocated at baseline was **R-204**, re-confirmed in `OPEN-ITEMS.md`. **R-205** +onward were free and are allocated here (R-205 … R-211). + +**One path correction:** `homelab-manifests` is **not** in `/mnt/5_hdd/felhom.eu/git/` with the other +repos — it lives at **`/home/kisfenyo/git/homelab-manifests`**. + +--- + +## 2. P0 — arrival state, expected vs measured + +Measured live before anything was changed. **Values that matched are recorded, not omitted.** + +| Fact | Expected (audit) | **Measured** | | +|---|---|---|---| +| Root fs size | 477,393,182,720 B | **477,393,182,720 B** | match | +| Root fs used | 387.3 GB | **387,332,755,456 B** | match (+10.8 MB in 2 h) | +| Root fs avail | 65.7 GB | **65,734,934,528 B** | match | +| Root fs percent | 86% | **86%** | match | +| Docker server version | 29.6.2 | **29.6.2** (build `3d80467`) | match | +| Storage driver | overlayfs, containerd snapshotter | **`overlayfs`, `driver-type: io.containerd.snapshotter.v1`** | match | +| Docker Root Dir | `/var/lib/docker`, ~0.6 GB | **`/var/lib/docker`, 618,541,056 B** | match | +| containerd root | `/var/lib/containerd`, ~181.4 GB | **181,394,223,104 B** | match | +| Build cache total / reclaimable | 157 GB / 147.8 GB | **157 GB / 147.8 GB**, 1446 records | match | +| Images / containers | 345 images, 2 running | **345 unique** (350 tag lines), **2 running** | match | +| `/etc/docker/daemon.json` | does not exist | **does not exist** (`/etc/docker/` exists, empty) | match | +| `/etc/buildkit/` | does not exist | **does not exist** | match | +| SSD2 device / mount / fs / free | ~240 GB, ~203 GB free | **`/dev/sde2`, `/mnt/ssd_2`, ext4, 235,148,750,848 B fs, 203,034,353,664 B free** | match | +| SSD1 Longhorn | `Schedulable=False (DiskPressure)` | **`Schedulable=False`, `DiskPressure`, 18.85% avail** | match | +| HDD2 Longhorn | also False | **`Schedulable=False`, `DiskPressure`, 24.91% avail** | match | +| Housekeeping timer | active, last Sun 2026-08-02 | **active**, last **2026-08-02 05:35:10 CEST**, next 08-09, `ExecMainStatus=0` | match | + +**Prohibition honoured:** `node-housekeeping.sh` was **not run**, in any mode. Its state was read from +systemd, from the textfile on disk, and from Prometheus. The textfile's mtime is still +`2026-08-02 05:35:11` — the 8-week reclaim history is intact. + +**What the audit did not measure — the SSD2 Longhorn record** (`Crucial-SSD-240G`, +`/mnt/ssd_2/data/longhorn`): + +| field | value | +|---|---:| +| `storageMaximum` | 235,148,750,848 | +| `storageAvailable` | 214,958,080,000 (**91.41%**) | +| `storageScheduled` | 139,586,437,120 | +| `storageReserved` | **0** | +| condition | **`Schedulable=True`** | +| replicas | **12** | + +**Two things stand out and both matter for P6.** SSD2 is the **only** Longhorn disk reserving **zero** +(HDD2 and HDD4 each reserve 500 GB). And `storageScheduled` (139.6 GB) exceeds actual usage +(`df`: 20,094,939,136 B) by **6.9×** — Longhorn is heavily overcommitted on this disk. + +--- + +## 3. P1 — which daemon owns the bytes + +> **The single configuration change that relocates the build cache is `root = ""` in +> `/etc/containerd/config.toml`, owned by `containerd.service` — the SYSTEM containerd +> (PID 2698677, socket `/run/containerd/containerd.sock`).** + +**The audit's `data-root` proposal is refuted, and the split is measured:** + +| tree | size | moved by `data-root`? | +|---|---:|---| +| `/var/lib/docker` | **618,541,056 B** (volumes 321 MB, **buildkit metadata 282 MB**, containers 14 MB) | yes | +| `/var/lib/containerd` | **181,394,223,104 B** (overlayfs snapshots **170,357,870,592 B**, content 10.98 GB) | **no** | + +`dockerd` runs as `/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock` with **no +systemd drop-ins**. BuildKit keeps its *record database* under Docker Root Dir (0.28 GB) but its *layer +bytes* in containerd's snapshotter — which is why `data-root` looks plausible and is wrong. + +In `/etc/containerd/config.toml` the `root` key is **present but commented out**, so it defaults to +`/var/lib/containerd`. Uncommenting it is the whole change. + +**k3s is unaffected — confirmed, not assumed.** k3s runs a **separate** containerd instance with its own +socket (`/run/k3s/containerd/containerd.sock`) and its own root +(`/var/lib/rancher/k3s/agent/containerd`, **79,669,473,280 B**). Nothing in this spike touches it. + +--- + +## 4. P2 — which GC key schema this release accepts + +**`dockerd --validate` exists on 29.6.2 and is not an oracle for this subtree.** Measured: + +| candidate | `--validate` | reality | +|---|---|---| +| `defaultKeepStorage` (old flat form) | `configuration OK` | — | +| `policy` array | `configuration OK` | **honoured** | +| `reservedSpace` / `maxUsedSpace` / `minFreeSpace` flat | `configuration OK` | **silently ignored** | +| **`totallyBogusKey`** | **`configuration OK`** | proves the validator does no field checking | +| `policy` with `"filter": [three values]` | **`configuration OK`** | **crashed the daemon on restart** | + +> **The real oracle is `docker buildx inspect`, which renders the *effective* policy.** That is an +> assertion of effect; `--validate` and "no error in the log" are both assertions of absence, and both +> were measured to lie here. + +**The flat form is silently ignored.** With `{"gc":{"enabled":true,"reservedSpace":"1GB","maxUsedSpace":"2GB","minFreeSpace":"10GB"}}` +applied and the daemon restarted, `buildx inspect` still reported the built-in defaults, and +`journalctl -u docker.service` carried **no warning, no deprecation notice and no error**. + +**The `filter` key takes ONE value per policy entry, not an array.** An array is fatal at startup: + +``` +error initializing buildkit: error creating buildkit instance: filters expect only one value +``` + +The daemon then entered a systemd restart loop. **This is also how the two `unless-stopped` dev +containers ended up stopped** — and they did **not** come back when the daemon later started cleanly; +they needed a manual `docker start`. A follow-up that templates this file must **restart and verify**, +never validate-and-assume. + +**Documented statement on containerd-worker applicability:** none found in `dockerd --help` or the +installed docs. Recorded as **unknown from documentation**, and settled empirically by P5. + +**The default policy is not "no policy".** Before any change, `buildx inspect` reported built-in rules +with **Reserved 41.91 GiB / Max Used 331.6 GiB / Min Free 82.89 GiB**. The audit's "there is no BuildKit +GC policy on this host" is *almost* right — there was one, sized so large it could never bind: the cache +at 157 GB was less than half the 331.6 GiB ceiling. + +### The validated `daemon.json`, verbatim (final state, left in place deliberately) + +```json +{ + "builder": { + "gc": { + "enabled": true, + "policy": [ + { "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" }, + { "all": true, "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" } + ] + } + } +} +``` + +Effective policy confirmed live (note the **GiB** rendering — `30GB` in JSON becomes a 30 GiB rule): + +``` +GC Policy rule#0: All: false Reserved Space: 10GiB Max Used Space: 30GiB Min Free Space: 20GiB +GC Policy rule#1: All: true Reserved Space: 10GiB Max Used Space: 30GiB Min Free Space: 20GiB +``` + +**No deprecation warning was emitted** for any accepted form. + +--- + +## 5. P3 — the prune: claimed, actual, settle + +Pre-checked: no build in progress, P0 and P7 captured first. + +| | value | +|---|---:| +| **Claimed by `docker builder prune -af`** | **156.9 GB** (1444 records deleted, 86 s wall-clock) | +| **Actual filesystem reclaim** | **150,353,829,888 B (150.35 GB)** | +| **Time to settle** | **synchronous — full reclaim visible at t+0** | + +Settle poll (t+0 already showed +150,354,477,056 B; two consecutive readings <50 MB apart at t+60s; +the residual −0.4 MB and −0.3 MB steps are ordinary background writes, not lagging reclaim): + +| t | avail | step delta | +|---|---:|---:| +| 0 s | 215,947,522,048 | — | +| 30 s | 215,947,141,120 | −380,928 | +| 60 s | 215,946,874,880 | −266,240 | + +**The 6.5 GB gap between claimed and actual is not an error** — it is the layers still shared with +retained images (`Shared: 8.6 GB` before the prune). **A weekly job must measure the filesystem, not +trust the command's `Total:`.** + +**Unlike containerd's image GC, this returns bytes synchronously**, so the follow-up needs no +`settle_imagefs` equivalent — but the settle poll is what *established* that, and it cost 60 seconds. + +| | before | after | +|---|---:|---:| +| `/` used | 387,474,644,992 B (**86%**) | 240,301,133,824 B (**53%**) | +| `/` avail | 65,593,044,992 B | 215,946,874,880 B | +| `/var/lib/containerd` | 181,537,714,176 B | 31,160,053,760 B | +| build cache | 156.9 GB / 1443 records | **0 B / 0 records** | +| images | 345 | **345** (untouched) | +| dev containers | 2 healthy | **2 healthy** | + +### The live consequence, cleared + +Longhorn re-evaluated within 60 s: + +| disk | before | after | +|---|---|---| +| `default-disk-…` (SSD1, `/var/lib/longhorn/`) | `Schedulable=False (DiskPressure)`, 89,967,820,800 B, **18.85%** | **`Schedulable=True`**, 240,228,761,600 B, **50.32%** | +| `HDD2` | `Schedulable=False (DiskPressure)`, 24.91% | **unchanged — still False, 24.91%** | + +**HDD2 is untouched by this work and remains unschedulable.** It is a separate item, stated here so it +is not read as fixed by association. + +--- + +## 6. P5 — proving the ceiling is enforced + +**Rollback stated before the first mutation** and never needed beyond the recovery below: +`sudo rm /etc/docker/daemon.json && sudo systemctl restart docker` — the file did not exist, so removal +is a complete revert. + +Test ceiling **2 GB** (below one build's working set, deliberately, so eviction is reachable). + +### Build-by-build totals + +| step | build cache total | reclaimable | records | +|---|---:|---:|---:| +| baseline (post-prune) | 0 B | 0 B | 0 | +| controller 1 | 2.583 GB | 2.583 GB | 22 | +| hub 1 | **3.703 GB** ← peak | 3.703 GB | 37 | +| controller 2 | 3.525 GB | 3.525 GB | 37 | +| hub 2 | 3.051 GB | 3.051 GB | 38 | +| controller 3 | 2.817 GB | 2.817 GB | 36 | +| hub 3 | **2.782 GB** | 2.782 GB | 38 | + +**The total stops rising and trends down** — the cap binds. Records plateau at 36–38. + +### The named eviction — the assertion that matters + +Set-differencing the record IDs, **4 records present at controller-build-1 were absent by +controller-build-3**, including the exact class that accumulated 208 copies: + +| ID | size | created | description | +|---|---:|---|---| +| **`9ak8egnxrg6rkibic2zo44u1`** | **440 MB** | 07:28:34 | **`mount / from exec /bin/sh -c go mod download \|\| true`** | +| `kkic530irq4em04kywzwrsjp` | 214.6 MB | 07:28:37 | `… go build -ldflags …` | +| `yvityhta4wppwa8cxs525c3g` | 58.83 MB | 07:28:36 | `… go mod tidy` | +| `mvttjfrimnt1ttt0ogry75ue` | 49.8 MB | 07:28:35 | `[builder 5/7] COPY . .` | + +Across the whole sequence: `h1→h3` evicted **17** records, `c2→c3` evicted **10**. + +### Tolerance, stated honestly + +Steady state settled at **2.78–2.82 GB against a 2 GiB (2.147 GB) rule — about 30% over**, with a peak +of **3.70 GB (~72% over)** mid-sequence. + +**This overshoot is an artefact of the test, not a property of the cap, and should not be extrapolated to +30 GB.** A single controller build produces ~2.6 GB of cache in one shot, so a 2 GB ceiling was set +*below one build's working set* — GC cannot get under it without deleting the cache it just created. At +30 GB the same working set is ~9% of the ceiling and GC has ample room. + +### GC is build-triggered, not timer-driven + +Six minutes idle, polled each minute: **total unchanged at 2.782 GB, records unchanged at 38.** The +cache does not shrink on its own. A burst of builds can overshoot and *stay* overshot until the next +build — which is the argument for keeping a weekly `docker builder prune` as a backstop (R-206), rather +than relying on the cap alone. + +### Then the ruled 30 GB + +Applied, restarted, and confirmed by the effective policy (§4). Both dev containers came back healthy +and **their data is intact — asserted positively, not by absence of complaint**: + +| observable | before | after | +|---|---:|---:| +| postgres `count(*) from pg_database` | 4 | **4** | +| postgres `pg_database_size('jarr_dev')` | 175,135,767 | **175,135,767** | +| postgres public tables | 31 | **31** | +| redis `DBSIZE` | 2437 | **2437** | +| docker volumes | 8 | **8, identical names** | + +--- + +## 7. P7 — why the module-download layer re-runs + +**Root cause found and proven: `ARG VERSION` / `ARG GIT_COMMIT` are declared in the builder stage +*above* the module-download step, so a new version value invalidates every `RUN` beneath them.** + +### The evidence chain + +**1. Every record was used exactly once.** All **208** `go mod download` records (206 at audit time, +2 +from this spike's builds) carried **`Usage count: 1`**, spanning 2026-07-06 → 2026-08-05, totalling +91.36 GB at 438–440 MB each. Not one was ever reused. + +**2. Two builds through `build.sh`, byte-identical tree, both missed.** + +| step | build 1 | build 2 | +|---|---|---| +| `#8 WORKDIR /build` | CACHED | CACHED | +| `#9 COPY go.mod ./` | **CACHED** | **CACHED** | +| **`#10 RUN go mod download \|\| true`** | **executed** | **executed** | +| `#11 COPY . .` | executed | executed | +| `#12 RUN go mod tidy` | executed | executed | +| `#13 RUN … go build …` | executed | executed | +| `#14/#15/#16` (stage-1 `RUN`s) | CACHED | CACHED | + +`go.mod` was **byte-identical** (`d8ded3f0798c…`) in the repo and in the workspace, mtime unchanged +(`2026-07-24 11:54:08`). + +**3. `build.sh`'s churn was ruled out by fingerprinting.** Content-hashing all 737 files across +`rm -rf` + `cp -a` + host-side `go mod tidy`: **`tidy` changes nothing**, and the reproduced tree is +**content-identical** to the live workspace. `go.mod`'s inode number was even identical across the churn. + +**4. A direct `docker build` on the untouched workspace cached *everything*** — including +`RUN go mod download` — proving BuildKit's cache is healthy and the Dockerfile is cacheable. + +**5. The decisive isolation.** Same workspace, same command, only the build-arg value varied: + +| run | `--build-arg VERSION` | `COPY go.mod ./` | **`RUN go mod download`** | +|---|---|---|---| +| E1 | **new** (`spike-p7e`) | CACHED | **executed** | +| E2 | **same** (`spike-p7e`) | CACHED | **CACHED** | + +**The mechanism, and it explains the `#9 CACHED / #10 executed` signature exactly:** a `COPY`'s cache key +is content-based and carries no environment, so it hits; a `RUN`'s key includes the stage **environment**, +which contains every `ARG` declared above it. Since every real build passes a fresh `VERSION` (and often +a fresh `GIT_COMMIT`), the module-download layer is invalidated **on every build, without exception** — +206 builds, 206 retained 440 MB snapshots, 90.5 GB. + +**Both Dockerfiles have it.** `felhom.eu/hub/Dockerfile` declares `ARG VERSION` / `ARG BUILD_TIME` above +`COPY go.mod go.sum*` → `RUN go mod download`. **And because both produce byte-identical `buildx du` +description strings, the 208 records are a COMBINED count** and must not be attributed to one project. + +**Fix shape — not applied here, filed as R-208:** move the `ARG` declarations down to just above the +final `go build`. One line per Dockerfile. **This is worth more than the cap and the move combined:** the +cap bounds the symptom at 30 GB; this removes the source. + +--- + +## 8. P4 — the image store, enumerated (nothing deleted) + +345 unique images across 350 tag lines (5 double-tagged), **0 dangling**. Only two are held by running +containers: `redis:7-alpine` and `postgres:16-alpine`. `docker image prune -a` would remove **343**. + +**Size the prize before spending a decision on it.** Per-tag sizes sum to 139.29 GB, but that +double-counts shared layers. `docker system df` puts the **real dedup'd footprint at 31.02 GB, of which +27.02 GB is reclaimable** — roughly a fifth of what the build-cache prune already returned. + +| GB (per-tag sum) | tags | repository | +|---:|---:|---| +| 118.44 | **207** | `gitea.dooplex.hu/admin/felhom-controller` | +| 7.25 | **107** | `gitea.dooplex.hu/admin/felhom-hub` | +| 4.34 | 5 | `gitea.dooplex.hu/admin/jarr` | +| 1.69 | 1 | `ghcr.io/seanmorley15/adventurelog-backend` | +| 0.94 | 1 | `ghcr.io/papra-hq/papra` | +| 0.72 | 1 | `codewithcj/sparkyfitness_server` | +| 0.72 | 1 | `postgis/postgis` | +| 0.59 | 3 | `gitea.dooplex.hu/admin/sparkyfitness-export` | +| (23 more, each ≤0.53 GB) | | | + +### Registry presence — which are recoverable + +Queried against the live Gitea registry v2 API: + +| repo | tags in registry | tags local | **local-only (NOT re-pullable)** | +|---|---:|---:|---:| +| `felhom-controller` | 76 | 207 | **131** | +| `felhom-hub` | 45 | 107 | **62** | + +**The local-only tags are all old.** Controller `0.39.0`–`0.135.0` plus `v0.35.0`–`v0.39.0`; hub +`0.9.0`–`0.57.0` plus `v0.7.2`–`v0.13.0`. **Everything from controller `0.136.0` and hub `0.58.0` +upward is in the registry** and therefore recoverable by `docker pull`. + +**This is the list the operator still owes a ruling on (R-210).** CC's view: not worth doing for the +space — ~27 GB against 199 GB now free. The only real benefit is dropping unrecoverable clutter. + +--- + +## 9. P8 — what else writes to Docker + +**The bounded list of build triggers on the host Docker daemon is: interactive/manual runs only.** + +- **18 systemd timers** enumerated — none invokes docker. (`node-housekeeping`, `dooplex-backup`, + `apt-daily*`, `sysstat*`, `logrotate`, `fstrim`, `e2scrub_all`, `smart-fancontrol`, …) +- **Root crontab: empty** (comments only). **User crontab: one entry**, `sync-hetzner-backups.sh`, + writes under `/mnt/5_hdd`. +- **`/etc/cron.d`: 4 real files** — anacron, e2scrub_all, sysstat, zfsutils. None touches docker. + +**The hole the audit left open is now closed with positive evidence.** The audit concluded "no CI runner +exists on this host" from systemd/process/crontab alone — but a Gitea Actions runner **does** exist as a +**cluster pod** (R-168, 2026-08-02). It cannot reach the host daemon: + +- **162 pods swept across all namespaces: `0` mount any `docker`/`containerd` hostPath.** +- `act-runner` (`gitea-system`, image `felhom-act-runner:0.1.0`) has one volume — a PVC at `/data`. + `privileged: false`, `allowPrivilegeEscalation: false`, **all capabilities dropped**, no docker socket. +- Its Dockerfile states the absence is deliberate: *"DELIBERATELY NOT INSTALLED: a Go toolchain, Node, or + Docker … There is no container runtime in here on purpose."* +- `gates.yml` runs exactly one command — `python3 scripts/_gates.py --fast` — plus a git fetch and + a failure-only Resend alarm. **No `docker build` anywhere.** + +**So the audit's ~7 builds/day attribution to interactive use is confirmed**, and now on a complete +enumeration rather than a partial one. + +--- + +## 10. P9 — the monitoring rule fix (shipped) + +**File:** `homelab-manifests/mon-system/prometheus-rules.yaml`, alert +`RootFsPressureDespiteHousekeeping`. **Commit `6808a4b`.** + +**Change:** `and` → **`and on(instance)`**. A bare `and` requires identical label sets; the left vector +carries `{device,fstype,instance,job,mountpoint}` and the right — a node_exporter *textfile* metric — +carries only `{instance,job}`. Nothing ever matched. + +### Red-proof (required, and both numbers recorded) + +**Threshold substituted `0.25` → `0.5`, and the reason is stated rather than hidden:** P3's prune had +already taken `/` to 44.6% available, so the shipped threshold's left side is now legitimately false. The +defect is in **label matching**, not in the threshold, so the proof is run at a threshold where both +sides are individually true — which is exactly the condition the original rule was meant to catch. + +| expression | `result_count` | +|---|---:| +| left side alone (`avail/size < 0.5`) | **1** | +| right side alone (`time() − last_success < 1209600`) | **1** | +| **as shipped — bare `and`** | **0** | +| **fixed — `and on(instance)`** | **1** | + +For completeness, at the shipped 0.25 threshold **both** forms return 0 today — because the left side is +false now, not because the fix failed. + +**Annotation rewritten.** It pointed at Longhorn replicas/snapshots (~10% of the growth). It now names +`docker buildx du` first, records the 30 GB cap in `/etc/docker/daemon.json`, and notes that a reverted +or unapplied cap looks exactly like this alert firing — so the alert text carries its own diagnosis. + +### Deployment — GitOps, no `kubectl set image`, no live ConfigMap edit + +ArgoCD app `monitoring` (auto-sync **off**): hard-refresh → `OutOfSync` at `6808a4be54a7` → deliberate +sync → **`Synced` / `Healthy`**, operation phase **`Succeeded`**. + +**Verified in Prometheus, not assumed from ArgoCD:** + +``` +query : … < 0.25 and on (instance) time() - node_housekeeping_last_success_timestamp_seconds < 1209600 +state : inactive health : ok lastError : (none) +``` + +`inactive` is now the **correct** answer — `/` really is fine at 44.6% available. + +> **A near-miss worth recording: I invented a full commit SHA from ArgoCD's truncated display and the +> sync failed with `upload-pack: not our ref`.** It failed loudly and cost one retry. It would not have +> failed loudly had the fabricated SHA happened to exist. + +### The finding P9 turned up on the way — R-211 + +**There is no config-reloader.** The `mon-system/prometheus` Deployment runs **one** container +(`prom/prometheus:v3.12.0`) with no reloader sidecar. After the sync the updated rules file was present +**inside the pod** (`grep -c "and on(instance)"` → 3 on the mounted symlink) while the Prometheus rules +API **still served the old expression for 4+ minutes**, with no error anywhere. It took effect only after +an explicit `POST /-/reload` (HTTP 200; `--web.enable-lifecycle` was already set). + +**The consequence is general:** every rule edit in this repo has silently not applied until something +happened to restart the pod. `Synced/Healthy` was true and beside the point. Filed as **R-211**. + +--- + +## 11. P6 — the SSD2 move: pre-analysis only. **NOT EXECUTED.** + +**Stopped here as specified. Nothing was moved, no daemon was reconfigured for SSD2, no mount unit was +written.** The operator must be present. + +**First, the framing changed underneath this probe.** P3's prune took `/` to 53% used and un-stuck +SSD1's Longhorn disk. **The move is no longer a rescue; it is a deliberate choice.** Per standing rule 4, +CC's disagreement is stated rather than absorbed: **the cap alone resolved the problem the move was meant +to solve, and the move carries a Longhorn risk the cap does not.** + +### Mechanism — three candidates + +| | mechanism | survives package upgrade | expressible in Ansible | visible reading the host cold | +|---|---|---|---|---| +| **(a)** | **`root = "/mnt/ssd_2/…"` in `/etc/containerd/config.toml`** | **yes** — dpkg conffile, local edits preserved | **yes** — a template, the file is already managed-shaped | **yes** — one line in the daemon's own config | +| (b) | systemd mount unit / fstab bind onto `/var/lib/containerd` | yes | yes | **weak** — invisible unless you check `findmnt` | +| (c) | symlink | fragile | yes | **weak**, and overlayfs mount semantics through a symlink are unproven here | + +**Recommendation: (a).** It is the only one that a person reading the host cold can *see*, and the key +already exists in the file (commented out). + +### The boot-order trap, and its guard + +If containerd or Docker starts before `/mnt/ssd_2` is mounted, the daemon **creates an empty store on the +root disk** and 345 images appear to have vanished. + +**Guard:** `RequiresMountsFor=/mnt/ssd_2` on **`containerd.service`** and **`docker.service`**. Neither +has it today — `containerd.service` carries only `After=network.target dbus.service`. + +> **`RequiresMountsFor` RE-MOUNTS; it does not refuse.** systemd pulls the mount unit in and *starts* it, +> so an unmounted-but-available device is silently re-mounted and everything looks fine. It refuses only +> when the device is genuinely **unavailable**. **Testing it by unmounting proves nothing** — it must be +> tested by pointing at an absent device. This is the ep0 datastore lesson +> (`RUNBOOK-ep0-datastore-volume-2026-07-27.md`), earned on this project. + +**And a move that has not survived a reboot has not been validated.** Mount-ordering reasoning is exactly +the class this project has been burned by. + +### The Longhorn consequence — the part that decides it + +`Crucial-SSD-240G`, `storageMaximum` 235,148,750,848, **25% floor = 58,787,187,712**: + +| scenario | resulting `storageAvailable` | as % | vs the 25% floor | +|---|---:|---:|---:| +| today, untouched | 214,958,080,000 | 91.41% | +66.4 pp | +| move cache only (30 GB) | 184,958,080,000 | 78.66% | **+53.7 pp** | +| move whole containerd tree today (34.3 GB) | 180,658,569,216 | 76.83% | +51.8 pp | +| **move at steady state (~31.5 GB images + 30 GB cache ≈ 65 GB)** | 149,958,080,000 | **63.77%** | **+38.8 pp** | + +**As measured, the move is safe by a wide margin — nowhere near the 10-pp caution band.** + +**But the worst case is not comfortable, and it is not hypothetical enough to omit.** `storageScheduled` +on SSD2 is **139,586,437,120** while actual usage is **20,094,939,136** — a 6.9× overcommit. If those 12 +volumes ever inflate to their scheduled size: + +| scenario | available | as % | vs floor | +|---|---:|---:|---| +| + cache only (30 GB) | 65,562,313,728 | 27.88% | +2.9 pp — inside the caution band | +| **+ steady state (65 GB)** | 30,562,313,728 | **13.00%** | **−12.0 pp → `Schedulable=False`** | + +That is precisely the failure that just took SSD1 out of service. + +> **Recommendation: proceed only together with setting Longhorn's `storageReserved` on SSD2 to cover the +> containerd tree (~80 GB).** SSD2 reserving **zero** while HDD2 and HDD4 each reserve 500 GB is an +> anomaly in its own right, and it is what lets Longhorn oversubscribe into the space the cache needs. +> Without that, CC recommends **not moving** — the cap has already done the job. + +### The sequence, prepared and not executed + +Each step with its rollback. **Do not run without the operator.** + +| # | step | rollback | +|---|---|---| +| 1 | `systemctl stop docker docker.socket containerd` | `systemctl start containerd docker` | +| 2 | `mkdir -p /mnt/ssd_2/containerd` | `rmdir` | +| 3 | `rsync -aHAX --numeric-ids /var/lib/containerd/ /mnt/ssd_2/containerd/` (small now — ~34 GB, and ~2.8 GB of it is cache) | delete the copy; the original is untouched | +| 4 | `mv /var/lib/containerd /var/lib/containerd.pre-move` (**move aside, never delete**) | `mv` back | +| 5 | set `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` | restore the commented line | +| 6 | drop-in `RequiresMountsFor=/mnt/ssd_2` on `containerd.service` **and** `docker.service`; `systemctl daemon-reload` | delete the drop-ins | +| 7 | `systemctl start containerd docker`; **verify `docker images` = 345, both dev containers up, `du` confirms the store's real path** | steps 5→4→1 in reverse | +| 8 | set SSD2 `storageReserved` ≈ 80 GB in the Longhorn node CR | set back to 0 | +| 9 | **`reboot`, then re-verify everything in step 7** | as above | +| 10 | test the guard by pointing at an **absent** device — an unmount alone proves nothing | restore fstab | +| 11 | only after a clean reboot + a full build: `rm -rf /var/lib/containerd.pre-move` | — none; do this last | + +--- + +## 12. Teardown + +**Nothing was provisioned.** No machine, no VM, no LXC guest, no hub record, no customer, no Longhorn +volume, no PVC. There is nothing to decommission. + +**Cleaned up:** + +- The six candidate `daemon.json` files written for P2 (in the session scratch dir, never in + `/etc/docker/`) — **removed**. +- The scratch workspace fingerprint tree (`ws_sim`) — **removed**. +- **Nine scratch image tags** created by the probe builds — `felhom-spike:{p7c,e1,e2}`, + `felhom-controller:{spike-p7a,spike-p7b,spike-p7d,evict-c1,evict-c2,evict-c3}`, + `felhom-hub:{evict-h1,evict-h2,evict-h3}` — **all removed**. +- `felhom-controller:latest` **restored** to `sha256:bdcd4a4ae8e006f1…` (0.198.0), byte-matching the + anchor captured before the first build. Image count back to **345**, the P0 baseline. + +**Intentional state left in place, recorded here so it does not read as a leftover:** +**`/etc/docker/daemon.json` exists**, with the content in §4. It did not exist before this session. +Removing it plus a daemon restart is a complete revert. + +**Not cleaned up, deliberately:** the ~2.78 GB of build cache from the eviction sequence. It is the +working cache under a live 30 GB policy, not scratch. + +--- + +## 13. Final state + +| | at arrival | **now** | +|---|---:|---:| +| `/` used | 387,332,755,456 B (**86%**) | **240,301,133,824 B (54%)** | +| `/` available | 65,734,934,528 B | **212,766,556,160 B** | +| `/var/lib/containerd` | 181,394,223,104 B | **34,299,510,784 B** | +| build cache | 157 GB / 1446 records | **2.782 GB / 38 records, capped at 30 GiB** | +| images | 345 | **345** | +| dev containers | 2 healthy | **2 healthy, data verified intact** | +| SSD1 Longhorn | **`Schedulable=False (DiskPressure)`** | **`Schedulable=True`, 50.32%** | +| `RootFsPressureDespiteHousekeeping` | **dead — could never fire** | **live, `health=ok`, red-proven** | +| growth driver | unbounded, ~5.25 GB/day | **capped; source diagnosed (R-208)** | + +--- + +## 14. Observations — noticed, deliberately not acted on + +1. **`build.sh` exists in two copies** — `felhom-controller/controller/build.sh` (in git) and + `/mnt/5_hdd/felhom.eu/build/felhom-controller/build.sh` (the operative one). They differ by **one + comment line**. The hub solved this with a symlink into the repo; the controller has not. Not fixed — + out of scope, but it is a drift channel. +2. **`build.sh` tags `:latest` on every build**, including throwaway version tags. A spike build silently + moved `latest` off 0.198.0; it was restored from an anchor captured beforehand. A `--no-latest` flag, + or tagging `latest` only on `--push`, would remove the trap. +3. **The default BuildKit GC policy is derived from disk size**, so a bigger root disk would have raised + the 331.6 GiB ceiling too. The cap needs to be explicit no matter how much disk is added. +4. **`node-housekeeping.sh`'s own `avail_bytes()` uses `df … | tail -1`** — the truncating-instrument + shape this project has been bitten by twice. It is *correct* here (`df` on one path emits exactly one + data row), so it was left alone, but it is worth knowing it is there. +5. **HDD2 remains `Schedulable=False (DiskPressure)`** at 24.91%, a hair under the 25% floor with 2.47 TB + free and its 500 GB reserve satisfied. Untouched by this work, and not fixed by it. +6. **The `mon-system` Prometheus has `--web.enable-admin-api`** as well as `--web.enable-lifecycle`. The + admin API permits series deletion. Noted, not acted on. +7. **Longhorn on SSD2 is overcommitted 6.9×** (139.6 GB scheduled vs 20.1 GB used). That is normal for + thin provisioning, but combined with `storageReserved = 0` it is what makes the P6 worst case reach + the floor. + +--- + +## 15. Register + +Allocated this session: **R-205** (CLOSED — the monitoring rule, shipped), **R-206** (Ansible: cap + +prune + narrowed Docker ban), **R-207** (the `DRY_RUN` guard), **R-208** (the `ARG`-ordering cache miss — +the largest remaining win), **R-209** (WAITING-ON-OPERATOR — whether to move to SSD2 at all), +**R-210** (WAITING-ON-OPERATOR — the image list), **R-211** (no Prometheus config-reloader). diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index d39b0e4..f989927 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -181,6 +181,13 @@ drill needed — the fourth is R-193, and until it goes the recovery is still op | **R-167** | ~~**Storage monitoring and backup alerts.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0/.1/.2 + hub v0.89.0, 2026-08-02) | — | **Operator decision D-c. It shipped BEFORE the R-165 merge, not with it** — D-a's condition (2) says the monitoring lands in the same step and never after, and landing it first is strictly better and costs nothing. **Customer half:** new `internal/fillwatch`, per FILESYSTEM (never per app — one full disk holding ten apps would fire ten times). **It emits the PRE-EXISTING `disk_warning`/`disk_critical` pair, which was allowlisted, copy'd, in `DefaultEnabledEvents` and checkbox'd with NO PRODUCER IN ANY REPO** — a complete customer pipeline with no producer, the **sixth** *built-but-never-wired* instance here; minting a new near-duplicate type would have left it inert forever. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range — **proven live: the critical crossing fired on the FREE-BYTE term (1.7 GB) at only 91% used.** Edge-triggered on escalation, state persisted, hysteresis dead zone at 75% / 7 GiB pinned by a test; a nil usage read never warns and never clears one (§8.4). The hub's two generic `customerMessages` entries were **removed** — `FormatCustomerEmail` prefers the entry over the message, so keeping them would discard the drive label and the byte figures. **Operator half: see R-158.** **Live on 9201, all three flows:** `disk_warning` then `disk_critical` both `customer | sent` with the Hungarian rendered, exactly two events across three boots (the edge trigger held on the one between), then a silent clear that re-armed. **v0.191.1** added the once-at-startup run (Daily/Every both wait for their first tick, so a box BOOTING over the line would have stayed silent up to 24 h — the R-100 shape); **v0.191.2** added a per-run positive observable, earned when a quiet run during this session's own validation proved unreadable as evidence. Follow-ups: **R-177** (no run-now path) | — | | **R-168** | ~~CI: no runner exists, and with trunk-based pushes CI can DETECT but not BLOCK~~ | **SHIPPED — and the alarm is DEMONSTRATED** (2026-08-02) | — | **Runner live**: `homelab-manifests/gitea-system/act-runner.yaml`, an unprivileged host-mode `act_runner` in `gitea-system`, one owner-scoped registration serving all four repos (measured: tasks 7-10 all claimed by `felhom-gates-runner`). `.gitea/workflows/gates.yml` in each repo runs that repo's entry point with `--fast` and nothing else; no `uses:` step anywhere. **Six probes, all answered, none STOPped** — `audits/SPIKE-ci-runner-2026-08-02.md`. The two that changed the design: **P2** (stock image has git but NO python3 → custom image `felhom-act-runner:0.1.0`, base pinned, python3 and nothing else) and **P6** (a runner that loses `/data/.runner` re-registers and leaves a dead record behind → the PVC is load-bearing, measured both ways). **P5 is the one that mattered**: a failed run produced NO mail, NO notification row and NO log line from Gitea, so the run now sends its own alarm via Resend and prints the provider's accepted id. **Proven end to end, not asserted**: a deliberately broken commit pushed with `--no-verify` → run #6 `failure` → `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`. Posture shown from the live pod spec: `privileged: false`, all caps dropped, no docker socket, no hostPath, `automountServiceAccountToken: false`, sized at half Gitea's limits so it cannot crowd out the service holding every repository on the same node. **The standing limit stays true and is written into the manifest and every workflow: it DETECTS, it does not BLOCK** — making it block is → R-169 | — | | **R-169** | **CI can only report, because there is no gate in the road.** Every felhom repo pushes straight to `main` with no pull request, so there is no merge for a status check to stand at. R-168's runner therefore notices a broken push *after* it has landed | **WAITING-ON-OPERATOR** (a working-style decision, not a defect) | an operator ruling | Making CI *blocking* requires two things this task deliberately did NOT do, because both change how the operator works and that is not a task's call: **(a)** branch protection on `main`, and **(b)** a pull-request workflow instead of direct-to-`main` pushes. The cost is real — every change would need a PR, which for a single-operator project may be worse than the disease. **The current arrangement is two nets, and it is not nothing**: `.githooks/pre-push` REFUSES locally, and R-168's runner NOTICES when that hook was skipped or was never armed in a clone, and emails. The honest gap is the window between a `--no-verify` push landing and the operator reading the alarm. Decide only if that window ever actually costs something | operator | +| **R-205** | ~~**`RootFsPressureDespiteHousekeeping` can never fire** — a bare PromQL `and` across mismatched label sets~~ | **CLOSED — SHIPPED + RED-PROVEN LIVE** (`homelab-manifests` `6808a4b`, 2026-08-05) | — | **`and` → `and on(instance)`.** Left vector carries `{device,fstype,instance,job,mountpoint}`; the right is a node_exporter *textfile* metric carrying `{instance,job}`. Nothing ever matched, so the alert returned the empty vector unconditionally — `health=ok`, `inactive`, no error, **dead from the day it was written**. It stayed silent through the entire 2026-07 build-cache incident that took `/` to 86% used and put Longhorn's default disk into `Schedulable=False (DiskPressure)`. **Red-proof against the live Prometheus API, both sides individually true** (threshold substituted `0.25`→`0.5` because P3's prune had already cleared the incident; the defect is in label matching, not the threshold): bare `and` → **`result_count 0`**, `and on(instance)` → **`result_count 1`**. **The annotation was also wrong and is rewritten** — it sent the reader to Longhorn replicas/snapshots (~10% of the growth) instead of the unbounded build cache (157 GB, ~95%); it now names `docker buildx du` first and records the 30 GB cap so a reverted cap is diagnosable from the alert text. ArgoCD `monitoring` Synced/Healthy at `6808a4b`; rule live and `health=ok`. Evidence: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` | — | +| **R-206** | **The build-cache cap and the weekly prune exist only as a hand-edited `/etc/docker/daemon.json` on DooPlex — not in Ansible, so a rebuild loses them.** The `node_housekeeping` role must also carry the prune, which today it is forbidden to run | **READY (M) — NEW 2026-08-05** | — | **The spike validated the recipe; this row builds it.** Three parts. **(a) Template `/etc/docker/daemon.json`** with the **`policy` array** form — **the flat form (`{"gc":{"reservedSpace":…}}`) is SILENTLY IGNORED**, measured: the daemon starts, logs nothing, and `docker buildx inspect` still reports the built-in defaults. **The oracle is `docker buildx inspect`, never `dockerd --validate`** — the validator returned `configuration OK` for a bogus key AND for the config that then **crashed the daemon** (`filter` takes one value per policy entry, not an array; `error initializing buildkit: filters expect only one value`). **(b) Narrow the role's Docker ban** (`node-housekeeping.sh.j2:10-14`) to permit exactly `docker builder prune -af` and nothing else — the ban's stated premise ("Docker here runs only unrelated jarr-* dev containers") is obsolete: the growth is Felhom Go build cache. **The measured prune is SYNCHRONOUS** (150.35 GB back at t+0, two consecutive polls <1 MB apart within 60 s) — **unlike containerd's image GC, so it needs no `settle_imagefs` equivalent**, but it MUST measure the filesystem rather than trust the command: `prune` claimed **156.9 GB** and the filesystem returned **150.35 GB**, the 6.5 GB gap being layers still shared with images. **(c) A restart-safety note in the role:** a bad `daemon.json` takes the daemon down AND leaves the `unless-stopped` dev containers stopped — they needed a manual `docker start` — so the role must restart-and-verify, not validate-and-assume. Recipe + every measurement: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` | CC | +| **R-207** | **`DRY_RUN=1` on `node-housekeeping.sh` is NOT non-mutating — it destroys the metric history it is supposed to let you inspect** | **READY (S) — NEW 2026-08-05** | — | `write_metrics()` (`node-housekeeping.sh.j2:119-150`) has **no `DRY_RUN` guard at all** — `DRY_RUN` appears in it only inside a log line — and it is called from an **unconditional `EXIT` trap** (`:150`). A dry run therefore atomically renames over the live node_exporter textfile, overwriting `node_housekeeping_last_success_timestamp_seconds` with *now* and `reclaimed_bytes` with ~0 — **resetting the staleness clock `HousekeepingStale` watches and erasing the 8-week reclaim history**. Confirmed by reading the source in both the 2026-08-05 audit and this spike; **neither run executed it**, so the history survives. Fix: guard `write_metrics` on `DRY_RUN`, or have the trap skip it. Pairs naturally with R-206 (same file, same role) | CC | +| **R-208** | **Every Felhom Go build re-downloads its modules because `ARG VERSION` sits ABOVE the module-download layer — ~440 MB of dead cache per build, 90.5 GB of the 157 GB** | **READY (S) — NEW 2026-08-05, ROOT CAUSE PROVEN** | — | **Measured, not inferred.** All **208** retained `go mod download` records carried **`Usage count: 1`** — not one was ever reused in ~a month of builds. The mechanism was isolated by four controlled builds: an unchanged tree rebuilt with the **same** `--build-arg VERSION` → `RUN go mod download` **CACHED**; the **same** tree with a **new** `VERSION` → **executed**. `COPY go.mod ./` stays CACHED either way, which is the tell: a `COPY`'s key is content-based, while a `RUN`'s key includes the stage **environment**, and `ARG VERSION`/`ARG GIT_COMMIT` are declared *before* the download in `felhom-controller/controller/Dockerfile`. Since every real build passes a fresh version, the layer is invalidated **every single time**. **`felhom.eu/hub/Dockerfile` has the identical defect** (`ARG VERSION`/`ARG BUILD_TIME` above `COPY go.mod go.sum*` → `RUN go mod download`) — and because both Dockerfiles produce byte-identical `buildx du` description strings, the 208 records are a COMBINED count and must not be attributed to one project. **Fix shape (one line each, not applied here):** move the `ARG VERSION`/`ARG GIT_COMMIT`/`ARG BUILD_TIME` declarations down to just above the final `go build`. **Worth more than the cap and the move combined** — the cap bounds the symptom, this removes the source. `build.sh`'s `rm -rf` + `cp -a` and its host-side `go mod tidy` were **ruled out by fingerprinting**: the tree is byte-identical across runs and `tidy` is a no-op | CC | +| **R-209** | **Should the containerd store move to SSD2 at all? — SSD2 is the ONLY Longhorn disk with `storageReserved = 0`, and the move is safe today but breaches the 25% floor if its volumes ever inflate** | **WAITING-ON-OPERATOR** — NEW 2026-08-05 | an operator ruling | **P3's prune already removed the urgency: `/` went 86% → 53% used and SSD1's Longhorn disk went `Schedulable=False (DiskPressure)` → `Schedulable=True` (18.85% → 50.32% available).** The move was ruled "cap then move"; the cap is in and the pressure is gone, so this is now a deliberate choice rather than a rescue. **The numbers, measured (`Crucial-SSD-240G`, `/mnt/ssd_2/data/longhorn`, `storageMaximum` 235,148,750,848):** available today **214,958,080,000 (91.41%)**; 25% floor **58,787,187,712**. Moving the whole containerd tree at steady state (~31.5 GB images + ≤30 GB cache ≈ 65 GB) leaves **63.77%, i.e. +38.8 pp above the floor — comfortably safe as measured.** **But `storageScheduled` on SSD2 is 139,586,437,120 while `df` says only 20,094,939,136 is actually used** — Longhorn has overcommitted 6.9× — and if those volumes ever inflate to their scheduled size, the same disk lands at **13.00%, i.e. 12 pp BELOW the floor → `Schedulable=False`**, which is exactly the failure that just took SSD1 out. **Recommendation: do the move only together with setting `storageReserved` on SSD2 to cover the containerd tree (~80 GB); SSD2 reserving zero while HDD2 and HDD4 each reserve 500 GB is an anomaly in its own right.** Mechanism, if it goes ahead: **containerd's `root` in `/etc/containerd/config.toml`** (the key is present but commented out) — **not** Docker's `data-root`, which would move only 0.62 GB. Guard: `RequiresMountsFor=/mnt/ssd_2` on `containerd.service` **and** `docker.service`, remembering that **`RequiresMountsFor` RE-MOUNTS rather than refusing** ([[ep0-datastore-volume-move-2026-07-27]]) — so it must be tested with a genuinely absent device, and the move is not validated until it has survived a **reboot**. Full pre-analysis: `audits/SPIKE-dooplex-buildcache-2026-08-05.md` §P6 | operator | +| **R-210** | **Which of the 345 local images may be deleted — 131 controller tags and 62 hub tags exist ONLY on this box and are not recoverable by `docker pull`** | **WAITING-ON-OPERATOR** — NEW 2026-08-05 | an operator ruling | **Nothing was deleted; this is a list, not an action.** The registry was queried directly: `felhom-controller` has **76** tags in Gitea vs **207** locally, `felhom-hub` **45** vs **107**. The **131 + 62 local-only tags are all OLD** — controller `0.39.0`–`0.135.0` plus `v0.35.0`–`v0.39.0`, hub `0.9.0`–`0.57.0` plus `v0.7.2`–`v0.13.0` — while everything from controller `0.136.0` and hub `0.58.0` upward IS in the registry and therefore re-pullable. **Size the prize honestly before spending a decision on it:** per-tag sizes sum to 139.29 GB, but that double-counts shared layers — `docker system df` puts the **real** dedup'd image footprint at **31.02 GB with 27.02 GB reclaimable**, i.e. an order of magnitude less than the build cache P3 already returned. `docker image prune -a` would remove 343 of 345 (only `redis:7-alpine` and `postgres:16-alpine` are held by running containers). **CC's view: not worth doing for the space** — it buys ~27 GB against 199 GB now free, and its only real benefit is dropping unrecoverable clutter | operator | +| **R-211** | **Prometheus has no config-reloader — a rules change reaches the pod and is never read** | **READY (S) — NEW 2026-08-05** | — | Found while verifying R-205 rather than by looking for it. The `mon-system/prometheus` Deployment runs **one** container (`prom/prometheus:v3.12.0`) with **no `configmap-reload`/`prometheus-config-reloader` sidecar**. After the ArgoCD sync the updated `node-housekeeping-alerts.yml` was present **inside the pod** (`grep -c "and on(instance)"` → 3 on the mounted symlink) while the Prometheus **rules API still served the old expression** — for **4+ minutes**, with no error anywhere. It only took effect after an explicit `POST /-/reload`. **The consequence is general, not specific to R-205: every rule edit in this repo since the stack was built has silently not applied until something happened to restart the pod** — so "committed and synced" has never meant "in force", and ArgoCD reporting `Synced/Healthy` is true and beside the point. `--web.enable-lifecycle` IS already set, so the fix is small: add a reloader sidecar watching the ConfigMap, or a `checksum/config` pod annotation so a rules change rolls the pod. **Same class as the four *built-but-never-wired* seams** — the control exists, nothing walks it | CC | ## Why the TOP READY rows rank this way