felhom.eu/documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md

# SPIKE — DooPlex build-cache containment (cap at 30 GB, relocate to SSD2)

**Date:** 2026-08-05 · **Host:** DooPlex (192.168.0.180) — **Tier 2, protected**, not a drill target
**Class:** Spike. §1, §2, §3, §4, §13-verify, §15 apply; §5–§12 skipped (no production Go code written).
**Anchor:** `dooplex_report.md` (2026-08-05 read-only disk audit). Every number in it was treated as a
claim to re-measure, not a fact to carry forward.

> Unit convention: **GB = decimal (10⁹)**, because `df -B1`, `du -B1` and Prometheus emit raw bytes.
> `docker buildx inspect` renders **GiB** (2³⁰) — where that matters it is labelled.

---

## Verdict

**The mechanism is confirmed, with one correction and one refutation, and the follow-up is safe to spec.**

- **Confirmed:** a `builder.gc` policy in `/etc/docker/daemon.json` **is** honoured under the containerd
  worker, and it **does** evict. Proven by naming a 440 MB cache record present at one build and absent
  two builds later — not by the absence of an error.
- **Corrected:** it is honoured **only in the `policy` array form**. The flat form
  (`{"gc": {"reservedSpace": …}}`) is **silently ignored** — the daemon starts, logs nothing, and keeps
  its built-in defaults. `dockerd --validate` is worthless here: it returned `configuration OK` for a
  bogus key **and** for a config that then crashed the daemon.
- **Refuted:** the audit's proposal to change Docker's `data-root` would **not** move the build cache. It
  moves 0.62 GB. The 181.4 GB belongs to the **system containerd**, relocated via `root` in
  `/etc/containerd/config.toml`.
- **The prune is done and it was the whole rescue.** `/` went **86% → 53% used**, and Longhorn's default
  disk went **`Schedulable=False (DiskPressure)` → `Schedulable=True`**. The live consequence is cleared.
- **The largest single finding is not the cap.** 90.5 GB of the 157 GB was one layer re-downloading on
  every build, and the root cause is a **one-line Dockerfile ordering defect** — `ARG VERSION` declared
  above the module-download step. Fixing it removes the source; the cap only bounds the symptom.
- **P6 (the SSD2 move) was NOT executed** — it stops at the operator, as specified. The pre-analysis
  says the move is safe as measured but carries a real Longhorn risk, and the prune removed its urgency.

---

## 1. Confirmed baselines

| Repo | `main` @ commit at spike start | Drift vs the task's stated baseline |
|---|---|---|
| `homelab-manifests` | `5188058d0e43` | none — matched |
| `felhom.eu` | `0dbd954fec90` | none — matched |

Both trees clean (`git status --porcelain` empty) and `HEAD == origin/main` before any commit.

**Register:** highest allocated at baseline was **R-204**, re-confirmed in `OPEN-ITEMS.md`. **R-205**
onward were free and are allocated here (R-205 … R-211).

**One path correction:** `homelab-manifests` is **not** in `/mnt/5_hdd/felhom.eu/git/` with the other
repos — it lives at **`/home/kisfenyo/git/homelab-manifests`**.

---

## 2. P0 — arrival state, expected vs measured

Measured live before anything was changed. **Values that matched are recorded, not omitted.**

| Fact | Expected (audit) | **Measured** | |
|---|---|---|---|
| Root fs size | 477,393,182,720 B | **477,393,182,720 B** | match |
| Root fs used | 387.3 GB | **387,332,755,456 B** | match (+10.8 MB in 2 h) |
| Root fs avail | 65.7 GB | **65,734,934,528 B** | match |
| Root fs percent | 86% | **86%** | match |
| Docker server version | 29.6.2 | **29.6.2** (build `3d80467`) | match |
| Storage driver | overlayfs, containerd snapshotter | **`overlayfs`, `driver-type: io.containerd.snapshotter.v1`** | match |
| Docker Root Dir | `/var/lib/docker`, ~0.6 GB | **`/var/lib/docker`, 618,541,056 B** | match |
| containerd root | `/var/lib/containerd`, ~181.4 GB | **181,394,223,104 B** | match |
| Build cache total / reclaimable | 157 GB / 147.8 GB | **157 GB / 147.8 GB**, 1446 records | match |
| Images / containers | 345 images, 2 running | **345 unique** (350 tag lines), **2 running** | match |
| `/etc/docker/daemon.json` | does not exist | **does not exist** (`/etc/docker/` exists, empty) | match |
| `/etc/buildkit/` | does not exist | **does not exist** | match |
| SSD2 device / mount / fs / free | ~240 GB, ~203 GB free | **`/dev/sde2`, `/mnt/ssd_2`, ext4, 235,148,750,848 B fs, 203,034,353,664 B free** | match |
| SSD1 Longhorn | `Schedulable=False (DiskPressure)` | **`Schedulable=False`, `DiskPressure`, 18.85% avail** | match |
| HDD2 Longhorn | also False | **`Schedulable=False`, `DiskPressure`, 24.91% avail** | match |
| Housekeeping timer | active, last Sun 2026-08-02 | **active**, last **2026-08-02 05:35:10 CEST**, next 08-09, `ExecMainStatus=0` | match |

**Prohibition honoured:** `node-housekeeping.sh` was **not run**, in any mode. Its state was read from
systemd, from the textfile on disk, and from Prometheus. The textfile's mtime is still
`2026-08-02 05:35:11` — the 8-week reclaim history is intact.

**What the audit did not measure — the SSD2 Longhorn record** (`Crucial-SSD-240G`,
`/mnt/ssd_2/data/longhorn`):

| field | value |
|---|---:|
| `storageMaximum` | 235,148,750,848 |
| `storageAvailable` | 214,958,080,000 (**91.41%**) |
| `storageScheduled` | 139,586,437,120 |
| `storageReserved` | **0** |
| condition | **`Schedulable=True`** |
| replicas | **12** |

**Two things stand out and both matter for P6.** SSD2 is the **only** Longhorn disk reserving **zero**
(HDD2 and HDD4 each reserve 500 GB). And `storageScheduled` (139.6 GB) exceeds actual usage
(`df`: 20,094,939,136 B) by **6.9×** — Longhorn is heavily overcommitted on this disk.

---

## 3. P1 — which daemon owns the bytes

> **The single configuration change that relocates the build cache is `root = "<path>"` in
> `/etc/containerd/config.toml`, owned by `containerd.service` — the SYSTEM containerd
> (PID 2698677, socket `/run/containerd/containerd.sock`).**

**The audit's `data-root` proposal is refuted, and the split is measured:**

| tree | size | moved by `data-root`? |
|---|---:|---|
| `/var/lib/docker` | **618,541,056 B** (volumes 321 MB, **buildkit metadata 282 MB**, containers 14 MB) | yes |
| `/var/lib/containerd` | **181,394,223,104 B** (overlayfs snapshots **170,357,870,592 B**, content 10.98 GB) | **no** |

`dockerd` runs as `/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock` with **no
systemd drop-ins**. BuildKit keeps its *record database* under Docker Root Dir (0.28 GB) but its *layer
bytes* in containerd's snapshotter — which is why `data-root` looks plausible and is wrong.

In `/etc/containerd/config.toml` the `root` key is **present but commented out**, so it defaults to
`/var/lib/containerd`. Uncommenting it is the whole change.

**k3s is unaffected — confirmed, not assumed.** k3s runs a **separate** containerd instance with its own
socket (`/run/k3s/containerd/containerd.sock`) and its own root
(`/var/lib/rancher/k3s/agent/containerd`, **79,669,473,280 B**). Nothing in this spike touches it.

---

## 4. P2 — which GC key schema this release accepts

**`dockerd --validate` exists on 29.6.2 and is not an oracle for this subtree.** Measured:

| candidate | `--validate` | reality |
|---|---|---|
| `defaultKeepStorage` (old flat form) | `configuration OK` | — |
| `policy` array | `configuration OK` | **honoured** |
| `reservedSpace` / `maxUsedSpace` / `minFreeSpace` flat | `configuration OK` | **silently ignored** |
| **`totallyBogusKey`** | **`configuration OK`** | proves the validator does no field checking |
| `policy` with `"filter": [three values]` | **`configuration OK`** | **crashed the daemon on restart** |

> **The real oracle is `docker buildx inspect`, which renders the *effective* policy.** That is an
> assertion of effect; `--validate` and "no error in the log" are both assertions of absence, and both
> were measured to lie here.

**The flat form is silently ignored.** With `{"gc":{"enabled":true,"reservedSpace":"1GB","maxUsedSpace":"2GB","minFreeSpace":"10GB"}}`
applied and the daemon restarted, `buildx inspect` still reported the built-in defaults, and
`journalctl -u docker.service` carried **no warning, no deprecation notice and no error**.

**The `filter` key takes ONE value per policy entry, not an array.** An array is fatal at startup:

```
error initializing buildkit: error creating buildkit instance: filters expect only one value
```

The daemon then entered a systemd restart loop. **This is also how the two `unless-stopped` dev
containers ended up stopped** — and they did **not** come back when the daemon later started cleanly;
they needed a manual `docker start`. A follow-up that templates this file must **restart and verify**,
never validate-and-assume.

**Documented statement on containerd-worker applicability:** none found in `dockerd --help` or the
installed docs. Recorded as **unknown from documentation**, and settled empirically by P5.

**The default policy is not "no policy".** Before any change, `buildx inspect` reported built-in rules
with **Reserved 41.91 GiB / Max Used 331.6 GiB / Min Free 82.89 GiB**. The audit's "there is no BuildKit
GC policy on this host" is *almost* right — there was one, sized so large it could never bind: the cache
at 157 GB was less than half the 331.6 GiB ceiling.

### The validated `daemon.json`, verbatim (final state, left in place deliberately)

```json
{
  "builder": {
    "gc": {
      "enabled": true,
      "policy": [
        { "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" },
        { "all": true, "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" }
      ]
    }
  }
}
```

Effective policy confirmed live (note the **GiB** rendering — `30GB` in JSON becomes a 30 GiB rule):

```
GC Policy rule#0:  All: false  Reserved Space: 10GiB  Max Used Space: 30GiB  Min Free Space: 20GiB
GC Policy rule#1:  All: true   Reserved Space: 10GiB  Max Used Space: 30GiB  Min Free Space: 20GiB
```

**No deprecation warning was emitted** for any accepted form.

---

## 5. P3 — the prune: claimed, actual, settle

Pre-checked: no build in progress, P0 and P7 captured first.

| | value |
|---|---:|
| **Claimed by `docker builder prune -af`** | **156.9 GB** (1444 records deleted, 86 s wall-clock) |
| **Actual filesystem reclaim** | **150,353,829,888 B (150.35 GB)** |
| **Time to settle** | **synchronous — full reclaim visible at t+0** |

Settle poll (t+0 already showed +150,354,477,056 B; two consecutive readings <50 MB apart at t+60s;
the residual −0.4 MB and −0.3 MB steps are ordinary background writes, not lagging reclaim):

| t | avail | step delta |
|---|---:|---:|
| 0 s | 215,947,522,048 | — |
| 30 s | 215,947,141,120 | −380,928 |
| 60 s | 215,946,874,880 | −266,240 |

**The 6.5 GB gap between claimed and actual is not an error** — it is the layers still shared with
retained images (`Shared: 8.6 GB` before the prune). **A weekly job must measure the filesystem, not
trust the command's `Total:`.**

**Unlike containerd's image GC, this returns bytes synchronously**, so the follow-up needs no
`settle_imagefs` equivalent — but the settle poll is what *established* that, and it cost 60 seconds.

| | before | after |
|---|---:|---:|
| `/` used | 387,474,644,992 B (**86%**) | 240,301,133,824 B (**53%**) |
| `/` avail | 65,593,044,992 B | 215,946,874,880 B |
| `/var/lib/containerd` | 181,537,714,176 B | 31,160,053,760 B |
| build cache | 156.9 GB / 1443 records | **0 B / 0 records** |
| images | 345 | **345** (untouched) |
| dev containers | 2 healthy | **2 healthy** |

### The live consequence, cleared

Longhorn re-evaluated within 60 s:

| disk | before | after |
|---|---|---|
| `default-disk-…` (SSD1, `/var/lib/longhorn/`) | `Schedulable=False (DiskPressure)`, 89,967,820,800 B, **18.85%** | **`Schedulable=True`**, 240,228,761,600 B, **50.32%** |
| `HDD2` | `Schedulable=False (DiskPressure)`, 24.91% | **unchanged — still False, 24.91%** |

**HDD2 is untouched by this work and remains unschedulable.** It is a separate item, stated here so it
is not read as fixed by association.

---

## 6. P5 — proving the ceiling is enforced

**Rollback stated before the first mutation** and never needed beyond the recovery below:
`sudo rm /etc/docker/daemon.json && sudo systemctl restart docker` — the file did not exist, so removal
is a complete revert.

Test ceiling **2 GB** (below one build's working set, deliberately, so eviction is reachable).

### Build-by-build totals

| step | build cache total | reclaimable | records |
|---|---:|---:|---:|
| baseline (post-prune) | 0 B | 0 B | 0 |
| controller 1 | 2.583 GB | 2.583 GB | 22 |
| hub 1 | **3.703 GB** ← peak | 3.703 GB | 37 |
| controller 2 | 3.525 GB | 3.525 GB | 37 |
| hub 2 | 3.051 GB | 3.051 GB | 38 |
| controller 3 | 2.817 GB | 2.817 GB | 36 |
| hub 3 | **2.782 GB** | 2.782 GB | 38 |

**The total stops rising and trends down** — the cap binds. Records plateau at 36–38.

### The named eviction — the assertion that matters

Set-differencing the record IDs, **4 records present at controller-build-1 were absent by
controller-build-3**, including the exact class that accumulated 208 copies:

| ID | size | created | description |
|---|---:|---|---|
| **`9ak8egnxrg6rkibic2zo44u1`** | **440 MB** | 07:28:34 | **`mount / from exec /bin/sh -c go mod download \|\| true`** |
| `kkic530irq4em04kywzwrsjp` | 214.6 MB | 07:28:37 | `… go build -ldflags …` |
| `yvityhta4wppwa8cxs525c3g` | 58.83 MB | 07:28:36 | `… go mod tidy` |
| `mvttjfrimnt1ttt0ogry75ue` | 49.8 MB | 07:28:35 | `[builder 5/7] COPY . .` |

Across the whole sequence: `h1→h3` evicted **17** records, `c2→c3` evicted **10**.

### Tolerance, stated honestly

Steady state settled at **2.78–2.82 GB against a 2 GiB (2.147 GB) rule — about 30% over**, with a peak
of **3.70 GB (~72% over)** mid-sequence.

**This overshoot is an artefact of the test, not a property of the cap, and should not be extrapolated to
30 GB.** A single controller build produces ~2.6 GB of cache in one shot, so a 2 GB ceiling was set
*below one build's working set* — GC cannot get under it without deleting the cache it just created. At
30 GB the same working set is ~9% of the ceiling and GC has ample room.

### GC is build-triggered, not timer-driven

Six minutes idle, polled each minute: **total unchanged at 2.782 GB, records unchanged at 38.** The
cache does not shrink on its own. A burst of builds can overshoot and *stay* overshot until the next
build — which is the argument for keeping a weekly `docker builder prune` as a backstop (R-206), rather
than relying on the cap alone.

### Then the ruled 30 GB

Applied, restarted, and confirmed by the effective policy (§4). Both dev containers came back healthy
and **their data is intact — asserted positively, not by absence of complaint**:

| observable | before | after |
|---|---:|---:|
| postgres `count(*) from pg_database` | 4 | **4** |
| postgres `pg_database_size('jarr_dev')` | 175,135,767 | **175,135,767** |
| postgres public tables | 31 | **31** |
| redis `DBSIZE` | 2437 | **2437** |
| docker volumes | 8 | **8, identical names** |

---

## 7. P7 — why the module-download layer re-runs

**Root cause found and proven: `ARG VERSION` / `ARG GIT_COMMIT` are declared in the builder stage
*above* the module-download step, so a new version value invalidates every `RUN` beneath them.**

### The evidence chain

**1. Every record was used exactly once.** All **208** `go mod download` records (206 at audit time, +2
from this spike's builds) carried **`Usage count: 1`**, spanning 2026-07-06 → 2026-08-05, totalling
91.36 GB at 438–440 MB each. Not one was ever reused.

**2. Two builds through `build.sh`, byte-identical tree, both missed.**

| step | build 1 | build 2 |
|---|---|---|
| `#8 WORKDIR /build` | CACHED | CACHED |
| `#9 COPY go.mod ./` | **CACHED** | **CACHED** |
| **`#10 RUN go mod download \|\| true`** | **executed** | **executed** |
| `#11 COPY . .` | executed | executed |
| `#12 RUN go mod tidy` | executed | executed |
| `#13 RUN … go build …` | executed | executed |
| `#14/#15/#16` (stage-1 `RUN`s) | CACHED | CACHED |

`go.mod` was **byte-identical** (`d8ded3f0798c…`) in the repo and in the workspace, mtime unchanged
(`2026-07-24 11:54:08`).

**3. `build.sh`'s churn was ruled out by fingerprinting.** Content-hashing all 737 files across
`rm -rf` + `cp -a` + host-side `go mod tidy`: **`tidy` changes nothing**, and the reproduced tree is
**content-identical** to the live workspace. `go.mod`'s inode number was even identical across the churn.

**4. A direct `docker build` on the untouched workspace cached *everything*** — including
`RUN go mod download` — proving BuildKit's cache is healthy and the Dockerfile is cacheable.

**5. The decisive isolation.** Same workspace, same command, only the build-arg value varied:

| run | `--build-arg VERSION` | `COPY go.mod ./` | **`RUN go mod download`** |
|---|---|---|---|
| E1 | **new** (`spike-p7e`) | CACHED | **executed** |
| E2 | **same** (`spike-p7e`) | CACHED | **CACHED** |

**The mechanism, and it explains the `#9 CACHED / #10 executed` signature exactly:** a `COPY`'s cache key
is content-based and carries no environment, so it hits; a `RUN`'s key includes the stage **environment**,
which contains every `ARG` declared above it. Since every real build passes a fresh `VERSION` (and often
a fresh `GIT_COMMIT`), the module-download layer is invalidated **on every build, without exception** —
206 builds, 206 retained 440 MB snapshots, 90.5 GB.

**Both Dockerfiles have it.** `felhom.eu/hub/Dockerfile` declares `ARG VERSION` / `ARG BUILD_TIME` above
`COPY go.mod go.sum*` → `RUN go mod download`. **And because both produce byte-identical `buildx du`
description strings, the 208 records are a COMBINED count** and must not be attributed to one project.

**Fix shape — not applied here, filed as R-208:** move the `ARG` declarations down to just above the
final `go build`. One line per Dockerfile. **This is worth more than the cap and the move combined:** the
cap bounds the symptom at 30 GB; this removes the source.

---

## 8. P4 — the image store, enumerated (nothing deleted)

345 unique images across 350 tag lines (5 double-tagged), **0 dangling**. Only two are held by running
containers: `redis:7-alpine` and `postgres:16-alpine`. `docker image prune -a` would remove **343**.

**Size the prize before spending a decision on it.** Per-tag sizes sum to 139.29 GB, but that
double-counts shared layers. `docker system df` puts the **real dedup'd footprint at 31.02 GB, of which
27.02 GB is reclaimable** — roughly a fifth of what the build-cache prune already returned.

| GB (per-tag sum) | tags | repository |
|---:|---:|---|
| 118.44 | **207** | `gitea.dooplex.hu/admin/felhom-controller` |
| 7.25 | **107** | `gitea.dooplex.hu/admin/felhom-hub` |
| 4.34 | 5 | `gitea.dooplex.hu/admin/jarr` |
| 1.69 | 1 | `ghcr.io/seanmorley15/adventurelog-backend` |
| 0.94 | 1 | `ghcr.io/papra-hq/papra` |
| 0.72 | 1 | `codewithcj/sparkyfitness_server` |
| 0.72 | 1 | `postgis/postgis` |
| 0.59 | 3 | `gitea.dooplex.hu/admin/sparkyfitness-export` |
| (23 more, each ≤0.53 GB) | | |

### Registry presence — which are recoverable

Queried against the live Gitea registry v2 API:

| repo | tags in registry | tags local | **local-only (NOT re-pullable)** |
|---|---:|---:|---:|
| `felhom-controller` | 76 | 207 | **131** |
| `felhom-hub` | 45 | 107 | **62** |

**The local-only tags are all old.** Controller `0.39.0`–`0.135.0` plus `v0.35.0`–`v0.39.0`; hub
`0.9.0`–`0.57.0` plus `v0.7.2`–`v0.13.0`. **Everything from controller `0.136.0` and hub `0.58.0`
upward is in the registry** and therefore recoverable by `docker pull`.

**This is the list the operator still owes a ruling on (R-210).** CC's view: not worth doing for the
space — ~27 GB against 199 GB now free. The only real benefit is dropping unrecoverable clutter.

---

## 9. P8 — what else writes to Docker

**The bounded list of build triggers on the host Docker daemon is: interactive/manual runs only.**

- **18 systemd timers** enumerated — none invokes docker. (`node-housekeeping`, `dooplex-backup`,
  `apt-daily*`, `sysstat*`, `logrotate`, `fstrim`, `e2scrub_all`, `smart-fancontrol`, …)
- **Root crontab: empty** (comments only). **User crontab: one entry**, `sync-hetzner-backups.sh`,
  writes under `/mnt/5_hdd`.
- **`/etc/cron.d`: 4 real files** — anacron, e2scrub_all, sysstat, zfsutils. None touches docker.

**The hole the audit left open is now closed with positive evidence.** The audit concluded "no CI runner
exists on this host" from systemd/process/crontab alone — but a Gitea Actions runner **does** exist as a
**cluster pod** (R-168, 2026-08-02). It cannot reach the host daemon:

- **162 pods swept across all namespaces: `0` mount any `docker`/`containerd` hostPath.**
- `act-runner` (`gitea-system`, image `felhom-act-runner:0.1.0`) has one volume — a PVC at `/data`.
  `privileged: false`, `allowPrivilegeEscalation: false`, **all capabilities dropped**, no docker socket.
- Its Dockerfile states the absence is deliberate: *"DELIBERATELY NOT INSTALLED: a Go toolchain, Node, or
  Docker … There is no container runtime in here on purpose."*
- `gates.yml` runs exactly one command — `python3 scripts/<repo>_gates.py --fast` — plus a git fetch and
  a failure-only Resend alarm. **No `docker build` anywhere.**

**So the audit's ~7 builds/day attribution to interactive use is confirmed**, and now on a complete
enumeration rather than a partial one.

---

## 10. P9 — the monitoring rule fix (shipped)

**File:** `homelab-manifests/mon-system/prometheus-rules.yaml`, alert
`RootFsPressureDespiteHousekeeping`. **Commit `6808a4b`.**

**Change:** `and` → **`and on(instance)`**. A bare `and` requires identical label sets; the left vector
carries `{device,fstype,instance,job,mountpoint}` and the right — a node_exporter *textfile* metric —
carries only `{instance,job}`. Nothing ever matched.

### Red-proof (required, and both numbers recorded)

**Threshold substituted `0.25` → `0.5`, and the reason is stated rather than hidden:** P3's prune had
already taken `/` to 44.6% available, so the shipped threshold's left side is now legitimately false. The
defect is in **label matching**, not in the threshold, so the proof is run at a threshold where both
sides are individually true — which is exactly the condition the original rule was meant to catch.

| expression | `result_count` |
|---|---:|
| left side alone (`avail/size < 0.5`) | **1** |
| right side alone (`time() − last_success < 1209600`) | **1** |
| **as shipped — bare `and`** | **0** |
| **fixed — `and on(instance)`** | **1** |

For completeness, at the shipped 0.25 threshold **both** forms return 0 today — because the left side is
false now, not because the fix failed.

**Annotation rewritten.** It pointed at Longhorn replicas/snapshots (~10% of the growth). It now names
`docker buildx du` first, records the 30 GB cap in `/etc/docker/daemon.json`, and notes that a reverted
or unapplied cap looks exactly like this alert firing — so the alert text carries its own diagnosis.

### Deployment — GitOps, no `kubectl set image`, no live ConfigMap edit

ArgoCD app `monitoring` (auto-sync **off**): hard-refresh → `OutOfSync` at `6808a4be54a7` → deliberate
sync → **`Synced` / `Healthy`**, operation phase **`Succeeded`**.

**Verified in Prometheus, not assumed from ArgoCD:**

```
query : … < 0.25 and on (instance) time() - node_housekeeping_last_success_timestamp_seconds < 1209600
state : inactive     health : ok     lastError : (none)
```

`inactive` is now the **correct** answer — `/` really is fine at 44.6% available.

> **A near-miss worth recording: I invented a full commit SHA from ArgoCD's truncated display and the
> sync failed with `upload-pack: not our ref`.** It failed loudly and cost one retry. It would not have
> failed loudly had the fabricated SHA happened to exist.

### The finding P9 turned up on the way — R-211

**There is no config-reloader.** The `mon-system/prometheus` Deployment runs **one** container
(`prom/prometheus:v3.12.0`) with no reloader sidecar. After the sync the updated rules file was present
**inside the pod** (`grep -c "and on(instance)"` → 3 on the mounted symlink) while the Prometheus rules
API **still served the old expression for 4+ minutes**, with no error anywhere. It took effect only after
an explicit `POST /-/reload` (HTTP 200; `--web.enable-lifecycle` was already set).

**The consequence is general:** every rule edit in this repo has silently not applied until something
happened to restart the pod. `Synced/Healthy` was true and beside the point. Filed as **R-211**.

---

## 11. P6 — the SSD2 move: **EXECUTED 2026-08-05 on operator authorisation**

> **Executed after the operator ruled "proceed", having read the pre-analysis below. The
> `storageReserved` condition CC named was applied with it. The operator subsequently ruled
> explicitly: DO NOT REBOOT DooPlex — so the reboot validation is DEFERRED, and what that leaves
> unproven is stated in §11.9 rather than glossed.**

### 11.0 Execution record

**Blast radius, established before anything was stopped:** k3s runs a **separate** containerd
(`/run/k3s/containerd/containerd.sock`), so **Gitea, the registry, the hub, PBS, Longhorn and all
~160 pods stayed up throughout**. The only workloads affected were the two `jarr-*` dev containers.
`ctr -n moby containers list` confirmed exactly those two on the system containerd.

| # | step | result |
|---|---|---|
| 1 | stop `docker.service`, `docker.socket`, `containerd.service` | all `inactive` |
| 2 | `mkdir /mnt/ssd_2/containerd` (mode 711, root:root) | ok |
| 3 | `rsync -aHAX --numeric-ids` | **550,891 files, 33,276,370,041 B, 4 m 58 s** |
| 4 | `mv /var/lib/containerd → /var/lib/containerd.pre-move-2026-08-05` | **moved aside, never deleted** |
| 5 | `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` (backup kept) | one-line diff |
| 6 | `RequiresMountsFor=/mnt/ssd_2` drop-ins on **both** `containerd.service` and `docker.service` | resolved into both units |
| 7 | start + verify | see below |
| 8 | `storageReserved = 80 GB` on SSD2 | applied |

**`-X` is load-bearing** — overlayfs snapshots carry `trusted.overlay.*` xattrs, and losing them
would silently corrupt layer stacking.

### 11.1 Copy verified BEFORE the original was touched

Four independent observables, all matching:

| check | source | destination |
|---|---:|---:|
| filesystem entries | 550,891 | **550,891** |
| `trusted.overlay` xattrs | 448 | **448** |
| files with link count >1 (hardlinks) | 37,243 | **37,243** |
| `io.containerd.metadata.v1.bolt/meta.db` sha256 | `c60ecfb4ee134255…` | **`c60ecfb4ee134255…`** |

(Byte totals differ by +14 MB — ext4 block allocation, not content.)

### 11.2 Nothing was lost

| | pre-move | post-move |
|---|---:|---:|
| unique images | 345 | **345 — identical ID set** |
| tag lines | 350 | **350 — identical set** |
| docker volumes | 8 | **8 — identical set** |
| build cache | 2.782 GB / 38 records | **2.782 GB / 38 records** |
| GC ceiling | 30 GiB | **30 GiB, still in force** |
| postgres databases | 4 | **4** |
| postgres `jarr_dev` bytes | 175,135,767 | **175,135,767** |
| postgres public tables | 31 | **31** |
| redis `DBSIZE` | 2437 | **2437** |

**End-to-end proof, not just inventory:** a real `./build.sh` run against the relocated store
completed `rc=0`, produced an image, and grew the cache 2.782 → 4.012 GB. The scratch tag was removed
and `:latest` restored to the anchor digest afterwards.

### 11.3 The store is demonstrably on SSD2

```
containerd config dump → root = '/mnt/ssd_2/containerd'
du /mnt/ssd_2/containerd → 35,500,183,552 B
/var/lib/containerd     → does not exist
```

### 11.4 The guard — and a trap found while proving it

**First attempt did NOT refuse, and that is a finding worth more than the test.** With
`RequiresMountsFor=/mnt/absent-guard-test-2026-08-05` — a path with **no mount unit at all** —
containerd **started normally**. `RequiresMountsFor` on a path systemd has no mount unit for is a
**silent no-op**. So a typo in the guard path buys nothing and says nothing: the same
*built-but-never-wired* shape this project has shipped repeatedly.

**So the guard was verified positively, at the unit level:**

```
containerd.service  Requires=mnt-ssd_2.mount   After=mnt-ssd_2.mount
docker.service      Requires=mnt-ssd_2.mount
```

**Then the refusal itself was proven with a genuinely absent DEVICE.** `/mnt/ssd_2` hosts **12 live
Longhorn replicas** and must never be unmounted, and editing `fstab` on a production host risks
emergency mode at boot — so a **temporary synthetic `.mount` unit** pointing at a nonexistent UUID
was used instead, with a temporary `RequiresMountsFor` on the real `containerd.service`:

```
A dependency job for containerd.service failed.
systemd: Dependency failed for containerd.service - containerd container runtime.
systemd: containerd.service: Job containerd.service/start failed with result 'dependency'.
containerd is-active: inactive
```

**containerd REFUSED to start rather than creating an empty store on the root disk** — exactly the
required behaviour, and exactly the signature the ep0 runbook describes. All test artifacts were
removed; only `10-ssd2-mount.conf` remains on each unit and the guard resolves to `/mnt/ssd_2` alone.

### 11.5 Longhorn — the condition CC made the move conditional on

`storageReserved` on SSD2 raised **0 → 80,000,000,000 B**, covering the containerd tree so Longhorn's
scheduler can no longer oversubscribe into the space the store needs.

| disk | available | reserved | `Schedulable` |
|---|---:|---:|---|
| **Crucial-SSD-240G (SSD2)** | 179,516,211,200 (**76.34%**) | **80,000,000,000** | **True** |
| default-disk (SSD1) | 236,978,176,000 (49.64%) | 143,217,954,816 | **True** |
| HDD4 | 1,793,798,963,200 (60.82%) | 536,870,912,000 | True |
| HDD2 | 2,470,864,486,400 (24.91%) | 536,870,912,000 | **False** — pre-existing, untouched |

SSD2 remains schedulable with the reserve applied. **Rollback: patch `storageReserved` back to 0.**

### 11.6 A standing post-boot assertion was installed

`felhom-store-postboot-check.service` (oneshot, `After=docker.service`, enabled) runs
`/usr/local/sbin/felhom-store-postboot-check.sh` at **every** boot and logs to
`/var/log/felhom-store-postboot-check.log`. It asserts, with **positive** observables:

- `/mnt/ssd_2` is mounted;
- containerd's effective `root` is `/mnt/ssd_2/containerd`;
- **`/var/lib/containerd` does NOT exist** — i.e. no empty store was recreated on the root disk, which
  is the precise trap the guard exists to prevent;
- docker sees ≥100 images;
- both dev containers are running.

Dry-run at install time: **RESULT: PASS** on all six lines. It was proven working *before* being
relied upon.

### 11.7 Disk position after the move

| | value |
|---|---:|
| `/` | 224 GB used, **199 GB free, 54%** |
| `/mnt/ssd_2` | 52 GB used, **157 GB free, 25%** |
| store on SSD2 | 35,500,183,552 B |
| **rollback copy retained on `/`** | **34,299,498,496 B** |

**`/` did not drop below 54% because the rollback copy is deliberately still there.** Reclaiming it
returns ~34 GB — see §11.9.

### 11.8 Rollback, complete and one sequence

1. `sudo systemctl stop docker docker.socket containerd`
2. `sudo mv /var/lib/containerd.pre-move-2026-08-05 /var/lib/containerd`
3. `sudo cp /etc/containerd/config.toml.pre-move-2026-08-05 /etc/containerd/config.toml`
4. `sudo rm -rf /etc/systemd/system/{containerd,docker}.service.d/10-ssd2-mount.conf`
5. `sudo systemctl disable --now felhom-store-postboot-check.service && sudo rm /etc/systemd/system/felhom-store-postboot-check.service /usr/local/sbin/felhom-store-postboot-check.sh`
6. `sudo kubectl patch nodes.longhorn.io -n longhorn-system dooplex --type merge -p '{"spec":{"disks":{"Crucial-SSD-240G":{"storageReserved":0}}}}'`
7. `sudo systemctl daemon-reload && sudo systemctl start containerd docker`
8. `sudo rm -rf /mnt/ssd_2/containerd`

### 11.9 **NOT PROVEN: the reboot. Stated, not glossed.**

**The operator ruled explicitly that DooPlex must not be rebooted, so the reboot validation did not
happen.** Uptime is unbroken (7 weeks 6 days, since 2026-06-10) — verified, not assumed.

**By this task's own standard — *"a move that has not survived a reboot has not been validated"* — the
move is NOT fully validated,** and it would be dishonest to close it as if it were. Precisely:

- **PROVEN (mechanism):** the guard is wired into both units (`Requires=` + `After=mnt-ssd_2.mount`),
  and containerd **refuses to start** when a required mount's device is genuinely absent.
- **NOT PROVEN (consequence):** that a real boot sequence mounts `/mnt/ssd_2` before containerd starts
  *on this host, in the real ordering*. Mount-ordering reasoning is exactly the class this project has
  been burned by, and the mechanism proof does not substitute for it. `CLAUDE.md` prefers the
  consequence over the mechanism; here only the mechanism is available.

**Two consequences follow, and both are deliberate:**

1. **The rollback copy `/var/lib/containerd.pre-move-2026-08-05` (34.3 GB) STAYS** until a reboot has
   validated the move. Deleting it now would trade a cheap 34 GB for the only cheap way back.
2. **The next reboot — whenever it happens, planned or not — validates this automatically.** The
   post-boot service writes `RESULT: PASS` or `RESULT: FAIL` to
   `/var/log/felhom-store-postboot-check.log`. **Read that file after the next boot**; on `PASS`, the
   rollback copy can be removed and ~34 GB returned to `/`.

Tracked as **R-209a**.

---

## 11-bis. P6 — the pre-analysis the ruling was made on

*(Retained as written, before execution, so the ruling can be audited against what it was given.)*

**First, the framing changed underneath this probe.** P3's prune took `/` to 53% used and un-stuck
SSD1's Longhorn disk. **The move is no longer a rescue; it is a deliberate choice.** Per standing rule 4,
CC's disagreement is stated rather than absorbed: **the cap alone resolved the problem the move was meant
to solve, and the move carries a Longhorn risk the cap does not.**

### Mechanism — three candidates

| | mechanism | survives package upgrade | expressible in Ansible | visible reading the host cold |
|---|---|---|---|---|
| **(a)** | **`root = "/mnt/ssd_2/…"` in `/etc/containerd/config.toml`** | **yes** — dpkg conffile, local edits preserved | **yes** — a template, the file is already managed-shaped | **yes** — one line in the daemon's own config |
| (b) | systemd mount unit / fstab bind onto `/var/lib/containerd` | yes | yes | **weak** — invisible unless you check `findmnt` |
| (c) | symlink | fragile | yes | **weak**, and overlayfs mount semantics through a symlink are unproven here |

**Recommendation: (a).** It is the only one that a person reading the host cold can *see*, and the key
already exists in the file (commented out).

### The boot-order trap, and its guard

If containerd or Docker starts before `/mnt/ssd_2` is mounted, the daemon **creates an empty store on the
root disk** and 345 images appear to have vanished.

**Guard:** `RequiresMountsFor=/mnt/ssd_2` on **`containerd.service`** and **`docker.service`**. Neither
has it today — `containerd.service` carries only `After=network.target dbus.service`.

> **`RequiresMountsFor` RE-MOUNTS; it does not refuse.** systemd pulls the mount unit in and *starts* it,
> so an unmounted-but-available device is silently re-mounted and everything looks fine. It refuses only
> when the device is genuinely **unavailable**. **Testing it by unmounting proves nothing** — it must be
> tested by pointing at an absent device. This is the ep0 datastore lesson
> (`RUNBOOK-ep0-datastore-volume-2026-07-27.md`), earned on this project.

**And a move that has not survived a reboot has not been validated.** Mount-ordering reasoning is exactly
the class this project has been burned by.

### The Longhorn consequence — the part that decides it

`Crucial-SSD-240G`, `storageMaximum` 235,148,750,848, **25% floor = 58,787,187,712**:

| scenario | resulting `storageAvailable` | as % | vs the 25% floor |
|---|---:|---:|---:|
| today, untouched | 214,958,080,000 | 91.41% | +66.4 pp |
| move cache only (30 GB) | 184,958,080,000 | 78.66% | **+53.7 pp** |
| move whole containerd tree today (34.3 GB) | 180,658,569,216 | 76.83% | +51.8 pp |
| **move at steady state (~31.5 GB images + 30 GB cache ≈ 65 GB)** | 149,958,080,000 | **63.77%** | **+38.8 pp** |

**As measured, the move is safe by a wide margin — nowhere near the 10-pp caution band.**

**But the worst case is not comfortable, and it is not hypothetical enough to omit.** `storageScheduled`
on SSD2 is **139,586,437,120** while actual usage is **20,094,939,136** — a 6.9× overcommit. If those 12
volumes ever inflate to their scheduled size:

| scenario | available | as % | vs floor |
|---|---:|---:|---|
| + cache only (30 GB) | 65,562,313,728 | 27.88% | +2.9 pp — inside the caution band |
| **+ steady state (65 GB)** | 30,562,313,728 | **13.00%** | **−12.0 pp → `Schedulable=False`** |

That is precisely the failure that just took SSD1 out of service.

> **Recommendation: proceed only together with setting Longhorn's `storageReserved` on SSD2 to cover the
> containerd tree (~80 GB).** SSD2 reserving **zero** while HDD2 and HDD4 each reserve 500 GB is an
> anomaly in its own right, and it is what lets Longhorn oversubscribe into the space the cache needs.
> Without that, CC recommends **not moving** — the cap has already done the job.

### The sequence, prepared and not executed

Each step with its rollback. **Do not run without the operator.**

| # | step | rollback |
|---|---|---|
| 1 | `systemctl stop docker docker.socket containerd` | `systemctl start containerd docker` |
| 2 | `mkdir -p /mnt/ssd_2/containerd` | `rmdir` |
| 3 | `rsync -aHAX --numeric-ids /var/lib/containerd/ /mnt/ssd_2/containerd/` (small now — ~34 GB, and ~2.8 GB of it is cache) | delete the copy; the original is untouched |
| 4 | `mv /var/lib/containerd /var/lib/containerd.pre-move` (**move aside, never delete**) | `mv` back |
| 5 | set `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` | restore the commented line |
| 6 | drop-in `RequiresMountsFor=/mnt/ssd_2` on `containerd.service` **and** `docker.service`; `systemctl daemon-reload` | delete the drop-ins |
| 7 | `systemctl start containerd docker`; **verify `docker images` = 345, both dev containers up, `du` confirms the store's real path** | steps 5→4→1 in reverse |
| 8 | set SSD2 `storageReserved` ≈ 80 GB in the Longhorn node CR | set back to 0 |
| 9 | **`reboot`, then re-verify everything in step 7** | as above |
| 10 | test the guard by pointing at an **absent** device — an unmount alone proves nothing | restore fstab |
| 11 | only after a clean reboot + a full build: `rm -rf /var/lib/containerd.pre-move` | — none; do this last |

---

## 12. Teardown

**Nothing was provisioned.** No machine, no VM, no LXC guest, no hub record, no customer, no Longhorn
volume, no PVC. There is nothing to decommission.

**Cleaned up:**

- The six candidate `daemon.json` files written for P2 (in the session scratch dir, never in
  `/etc/docker/`) — **removed**.
- The scratch workspace fingerprint tree (`ws_sim`) — **removed**.
- **Nine scratch image tags** created by the probe builds — `felhom-spike:{p7c,e1,e2}`,
  `felhom-controller:{spike-p7a,spike-p7b,spike-p7d,evict-c1,evict-c2,evict-c3}`,
  `felhom-hub:{evict-h1,evict-h2,evict-h3}` — **all removed**.
- `felhom-controller:latest` **restored** to `sha256:bdcd4a4ae8e006f1…` (0.198.0), byte-matching the
  anchor captured before the first build. Image count back to **345**, the P0 baseline.

**Intentional state left in place, recorded here so it does not read as a leftover:**
**`/etc/docker/daemon.json` exists**, with the content in §4. It did not exist before this session.
Removing it plus a daemon restart is a complete revert.

**Not cleaned up, deliberately:** the ~2.78 GB of build cache from the eviction sequence. It is the
working cache under a live 30 GB policy, not scratch.

---

## 13. Final state

| | at arrival | **now** |
|---|---:|---:|
| `/` used | 387,332,755,456 B (**86%**) | **240,301,133,824 B (54%)** |
| `/` available | 65,734,934,528 B | **212,766,556,160 B** |
| `/var/lib/containerd` | 181,394,223,104 B | **34,299,510,784 B** |
| build cache | 157 GB / 1446 records | **2.782 GB / 38 records, capped at 30 GiB** |
| images | 345 | **345** |
| dev containers | 2 healthy | **2 healthy, data verified intact** |
| SSD1 Longhorn | **`Schedulable=False (DiskPressure)`** | **`Schedulable=True`, 50.32%** |
| `RootFsPressureDespiteHousekeeping` | **dead — could never fire** | **live, `health=ok`, red-proven** |
| growth driver | unbounded, ~5.25 GB/day | **capped; source diagnosed (R-208)** |

---

## 14. Observations — noticed, deliberately not acted on

1. **`build.sh` exists in two copies** — `felhom-controller/controller/build.sh` (in git) and
   `/mnt/5_hdd/felhom.eu/build/felhom-controller/build.sh` (the operative one). They differ by **one
   comment line**. The hub solved this with a symlink into the repo; the controller has not. Not fixed —
   out of scope, but it is a drift channel.
2. **`build.sh` tags `:latest` on every build**, including throwaway version tags. A spike build silently
   moved `latest` off 0.198.0; it was restored from an anchor captured beforehand. A `--no-latest` flag,
   or tagging `latest` only on `--push`, would remove the trap.
3. **The default BuildKit GC policy is derived from disk size**, so a bigger root disk would have raised
   the 331.6 GiB ceiling too. The cap needs to be explicit no matter how much disk is added.
4. **`node-housekeeping.sh`'s own `avail_bytes()` uses `df … | tail -1`** — the truncating-instrument
   shape this project has been bitten by twice. It is *correct* here (`df` on one path emits exactly one
   data row), so it was left alone, but it is worth knowing it is there.
5. **HDD2 remains `Schedulable=False (DiskPressure)`** at 24.91%, a hair under the 25% floor with 2.47 TB
   free and its 500 GB reserve satisfied. Untouched by this work, and not fixed by it.
6. **The `mon-system` Prometheus has `--web.enable-admin-api`** as well as `--web.enable-lifecycle`. The
   admin API permits series deletion. Noted, not acted on.
7. **Longhorn on SSD2 is overcommitted 6.9×** (139.6 GB scheduled vs 20.1 GB used). That is normal for
   thin provisioning, but combined with `storageReserved = 0` it is what makes the P6 worst case reach
   the floor.

---

## 15. Register

Allocated this session: **R-205** (CLOSED — the monitoring rule, shipped), **R-206** (Ansible: cap +
prune + narrowed Docker ban), **R-207** (the `DRY_RUN` guard), **R-208** (the `ARG`-ordering cache miss —
the largest remaining win), **R-209** (WAITING-ON-OPERATOR — whether to move to SSD2 at all),
**R-210** (WAITING-ON-OPERATOR — the image list), **R-211** (no Prometheus config-reloader).