felhom.eu/documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md

SPIKE — DooPlex build-cache containment (cap at 30 GB, relocate to SSD2)

Date: 2026-08-05 · Host: DooPlex (192.168.0.180) — Tier 2, protected, not a drill target Class: Spike. §1, §2, §3, §4, §13-verify, §15 apply; §5–§12 skipped (no production Go code written). Anchor: dooplex_report.md (2026-08-05 read-only disk audit). Every number in it was treated as a claim to re-measure, not a fact to carry forward.

Unit convention: GB = decimal (10⁹), because df -B1, du -B1 and Prometheus emit raw bytes. docker buildx inspect renders GiB (2³⁰) — where that matters it is labelled.

---

Verdict

The mechanism is confirmed, with one correction and one refutation, and the follow-up is safe to spec.

• Confirmed: a builder.gc policy in /etc/docker/daemon.json is honoured under the containerd worker, and it does evict. Proven by naming a 440 MB cache record present at one build and absent two builds later — not by the absence of an error.
• Corrected: it is honoured only in the policy array form. The flat form ({"gc": {"reservedSpace": …}}) is silently ignored — the daemon starts, logs nothing, and keeps its built-in defaults. dockerd --validate is worthless here: it returned configuration OK for a bogus key and for a config that then crashed the daemon.
• Refuted: the audit's proposal to change Docker's data-root would not move the build cache. It moves 0.62 GB. The 181.4 GB belongs to the system containerd, relocated via root in /etc/containerd/config.toml.
• The prune is done and it was the whole rescue. / went 86% → 53% used, and Longhorn's default disk went Schedulable=False (DiskPressure) → Schedulable=True. The live consequence is cleared.
• The largest single finding is not the cap. 90.5 GB of the 157 GB was one layer re-downloading on every build, and the root cause is a one-line Dockerfile ordering defect — ARG VERSION declared above the module-download step. Fixing it removes the source; the cap only bounds the symptom.
• P6 (the SSD2 move) was NOT executed — it stops at the operator, as specified. The pre-analysis says the move is safe as measured but carries a real Longhorn risk, and the prune removed its urgency.

---

1. Confirmed baselines

Repo	main @ commit at spike start	Drift vs the task's stated baseline
homelab-manifests	5188058d0e43	none — matched
felhom.eu	0dbd954fec90	none — matched

Both trees clean (git status --porcelain empty) and HEAD == origin/main before any commit.

Register: highest allocated at baseline was R-204, re-confirmed in OPEN-ITEMS.md. R-205 onward were free and are allocated here (R-205 … R-211).

One path correction: homelab-manifests is not in /mnt/5_hdd/felhom.eu/git/ with the other repos — it lives at /home/kisfenyo/git/homelab-manifests.

---

2. P0 — arrival state, expected vs measured

Measured live before anything was changed. Values that matched are recorded, not omitted.

Fact	Expected (audit)	Measured
Root fs size	477,393,182,720 B	477,393,182,720 B	match
Root fs used	387.3 GB	387,332,755,456 B	match (+10.8 MB in 2 h)
Root fs avail	65.7 GB	65,734,934,528 B	match
Root fs percent	86%	86%	match
Docker server version	29.6.2	29.6.2 (build 3d80467)	match
Storage driver	overlayfs, containerd snapshotter	overlayfs, driver-type: io.containerd.snapshotter.v1	match
Docker Root Dir	/var/lib/docker, ~0.6 GB	/var/lib/docker, 618,541,056 B	match
containerd root	/var/lib/containerd, ~181.4 GB	181,394,223,104 B	match
Build cache total / reclaimable	157 GB / 147.8 GB	157 GB / 147.8 GB, 1446 records	match
Images / containers	345 images, 2 running	345 unique (350 tag lines), 2 running	match
/etc/docker/daemon.json	does not exist	does not exist (/etc/docker/ exists, empty)	match
/etc/buildkit/	does not exist	does not exist	match
SSD2 device / mount / fs / free	~240 GB, ~203 GB free	/dev/sde2, /mnt/ssd_2, ext4, 235,148,750,848 B fs, 203,034,353,664 B free	match
SSD1 Longhorn	Schedulable=False (DiskPressure)	Schedulable=False, DiskPressure, 18.85% avail	match
HDD2 Longhorn	also False	Schedulable=False, DiskPressure, 24.91% avail	match
Housekeeping timer	active, last Sun 2026-08-02	active, last 2026-08-02 05:35:10 CEST, next 08-09, ExecMainStatus=0	match

Prohibition honoured: node-housekeeping.sh was not run, in any mode. Its state was read from systemd, from the textfile on disk, and from Prometheus. The textfile's mtime is still 2026-08-02 05:35:11 — the 8-week reclaim history is intact.

What the audit did not measure — the SSD2 Longhorn record (Crucial-SSD-240G, /mnt/ssd_2/data/longhorn):

field	value
storageMaximum	235,148,750,848
storageAvailable	214,958,080,000 (91.41%)
storageScheduled	139,586,437,120
storageReserved	0
condition	Schedulable=True
replicas	12

Two things stand out and both matter for P6. SSD2 is the only Longhorn disk reserving zero (HDD2 and HDD4 each reserve 500 GB). And storageScheduled (139.6 GB) exceeds actual usage (df: 20,094,939,136 B) by 6.9× — Longhorn is heavily overcommitted on this disk.

---

3. P1 — which daemon owns the bytes

The single configuration change that relocates the build cache is root = "<path>" in /etc/containerd/config.toml, owned by containerd.service — the SYSTEM containerd (PID 2698677, socket /run/containerd/containerd.sock).

The audit's data-root proposal is refuted, and the split is measured:

tree	size	moved by data-root?
/var/lib/docker	618,541,056 B (volumes 321 MB, buildkit metadata 282 MB, containers 14 MB)	yes
/var/lib/containerd	181,394,223,104 B (overlayfs snapshots 170,357,870,592 B, content 10.98 GB)	no

dockerd runs as /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock with no systemd drop-ins. BuildKit keeps its record database under Docker Root Dir (0.28 GB) but its layer bytes in containerd's snapshotter — which is why data-root looks plausible and is wrong.

In /etc/containerd/config.toml the root key is present but commented out, so it defaults to /var/lib/containerd. Uncommenting it is the whole change.

k3s is unaffected — confirmed, not assumed. k3s runs a separate containerd instance with its own socket (/run/k3s/containerd/containerd.sock) and its own root (/var/lib/rancher/k3s/agent/containerd, 79,669,473,280 B). Nothing in this spike touches it.

---

4. P2 — which GC key schema this release accepts

dockerd --validate exists on 29.6.2 and is not an oracle for this subtree. Measured:

candidate	--validate	reality
defaultKeepStorage (old flat form)	configuration OK	—
policy array	configuration OK	honoured
reservedSpace / maxUsedSpace / minFreeSpace flat	configuration OK	silently ignored
totallyBogusKey	configuration OK	proves the validator does no field checking
policy with "filter": [three values]	configuration OK	crashed the daemon on restart

The real oracle is docker buildx inspect, which renders the effective policy. That is an assertion of effect; --validate and "no error in the log" are both assertions of absence, and both were measured to lie here.

The flat form is silently ignored. With {"gc":{"enabled":true,"reservedSpace":"1GB","maxUsedSpace":"2GB","minFreeSpace":"10GB"}} applied and the daemon restarted, buildx inspect still reported the built-in defaults, and journalctl -u docker.service carried no warning, no deprecation notice and no error.

The filter key takes ONE value per policy entry, not an array. An array is fatal at startup:

error initializing buildkit: error creating buildkit instance: filters expect only one value

The daemon then entered a systemd restart loop. This is also how the two unless-stopped dev containers ended up stopped — and they did not come back when the daemon later started cleanly; they needed a manual docker start. A follow-up that templates this file must restart and verify, never validate-and-assume.

Documented statement on containerd-worker applicability: none found in dockerd --help or the installed docs. Recorded as unknown from documentation, and settled empirically by P5.

The default policy is not "no policy". Before any change, buildx inspect reported built-in rules with Reserved 41.91 GiB / Max Used 331.6 GiB / Min Free 82.89 GiB. The audit's "there is no BuildKit GC policy on this host" is almost right — there was one, sized so large it could never bind: the cache at 157 GB was less than half the 331.6 GiB ceiling.

The validated daemon.json, verbatim (final state, left in place deliberately)

{
  "builder": {
    "gc": {
      "enabled": true,
      "policy": [
        { "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" },
        { "all": true, "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" }
      ]
    }
  }
}

Effective policy confirmed live (note the GiB rendering — 30GB in JSON becomes a 30 GiB rule):

GC Policy rule#0:  All: false  Reserved Space: 10GiB  Max Used Space: 30GiB  Min Free Space: 20GiB
GC Policy rule#1:  All: true   Reserved Space: 10GiB  Max Used Space: 30GiB  Min Free Space: 20GiB

No deprecation warning was emitted for any accepted form.

---

5. P3 — the prune: claimed, actual, settle

Pre-checked: no build in progress, P0 and P7 captured first.

	value
Claimed by docker builder prune -af	156.9 GB (1444 records deleted, 86 s wall-clock)
Actual filesystem reclaim	150,353,829,888 B (150.35 GB)
Time to settle	synchronous — full reclaim visible at t+0

Settle poll (t+0 already showed +150,354,477,056 B; two consecutive readings <50 MB apart at t+60s; the residual −0.4 MB and −0.3 MB steps are ordinary background writes, not lagging reclaim):

t	avail	step delta
0 s	215,947,522,048	—
30 s	215,947,141,120	−380,928
60 s	215,946,874,880	−266,240

The 6.5 GB gap between claimed and actual is not an error — it is the layers still shared with retained images (Shared: 8.6 GB before the prune). A weekly job must measure the filesystem, not trust the command's Total:.

Unlike containerd's image GC, this returns bytes synchronously, so the follow-up needs no settle_imagefs equivalent — but the settle poll is what established that, and it cost 60 seconds.

	before	after
/ used	387,474,644,992 B (86%)	240,301,133,824 B (53%)
/ avail	65,593,044,992 B	215,946,874,880 B
/var/lib/containerd	181,537,714,176 B	31,160,053,760 B
build cache	156.9 GB / 1443 records	0 B / 0 records
images	345	345 (untouched)
dev containers	2 healthy	2 healthy

The live consequence, cleared

Longhorn re-evaluated within 60 s:

disk	before	after
default-disk-… (SSD1, /var/lib/longhorn/)	Schedulable=False (DiskPressure), 89,967,820,800 B, 18.85%	Schedulable=True, 240,228,761,600 B, 50.32%
HDD2	Schedulable=False (DiskPressure), 24.91%	unchanged — still False, 24.91%

HDD2 is untouched by this work and remains unschedulable. It is a separate item, stated here so it is not read as fixed by association.

---

6. P5 — proving the ceiling is enforced

Rollback stated before the first mutation and never needed beyond the recovery below: sudo rm /etc/docker/daemon.json && sudo systemctl restart docker — the file did not exist, so removal is a complete revert.

Test ceiling 2 GB (below one build's working set, deliberately, so eviction is reachable).

Build-by-build totals

step	build cache total	reclaimable	records
baseline (post-prune)	0 B	0 B	0
controller 1	2.583 GB	2.583 GB	22
hub 1	3.703 GB ← peak	3.703 GB	37
controller 2	3.525 GB	3.525 GB	37
hub 2	3.051 GB	3.051 GB	38
controller 3	2.817 GB	2.817 GB	36
hub 3	2.782 GB	2.782 GB	38

The total stops rising and trends down — the cap binds. Records plateau at 36–38.

The named eviction — the assertion that matters

Set-differencing the record IDs, 4 records present at controller-build-1 were absent by controller-build-3, including the exact class that accumulated 208 copies:

ID	size	created	description
9ak8egnxrg6rkibic2zo44u1	440 MB	07:28:34	mount / from exec /bin/sh -c go mod download || true
kkic530irq4em04kywzwrsjp	214.6 MB	07:28:37	… go build -ldflags …
yvityhta4wppwa8cxs525c3g	58.83 MB	07:28:36	… go mod tidy
mvttjfrimnt1ttt0ogry75ue	49.8 MB	07:28:35	[builder 5/7] COPY . .

Across the whole sequence: h1→h3 evicted 17 records, c2→c3 evicted 10.

Tolerance, stated honestly

Steady state settled at 2.78–2.82 GB against a 2 GiB (2.147 GB) rule — about 30% over, with a peak of 3.70 GB (~72% over) mid-sequence.

This overshoot is an artefact of the test, not a property of the cap, and should not be extrapolated to 30 GB. A single controller build produces ~2.6 GB of cache in one shot, so a 2 GB ceiling was set below one build's working set — GC cannot get under it without deleting the cache it just created. At 30 GB the same working set is ~9% of the ceiling and GC has ample room.

GC is build-triggered, not timer-driven

Six minutes idle, polled each minute: total unchanged at 2.782 GB, records unchanged at 38. The cache does not shrink on its own. A burst of builds can overshoot and stay overshot until the next build — which is the argument for keeping a weekly docker builder prune as a backstop (R-206), rather than relying on the cap alone.

Then the ruled 30 GB

Applied, restarted, and confirmed by the effective policy (§4). Both dev containers came back healthy and their data is intact — asserted positively, not by absence of complaint:

observable	before	after
postgres count(*) from pg_database	4	4
postgres pg_database_size('jarr_dev')	175,135,767	175,135,767
postgres public tables	31	31
redis DBSIZE	2437	2437
docker volumes	8	8, identical names

---

7. P7 — why the module-download layer re-runs

Root cause found and proven: ARG VERSION / ARG GIT_COMMIT are declared in the builder stage above the module-download step, so a new version value invalidates every RUN beneath them.

The evidence chain

1. Every record was used exactly once. All 208 go mod download records (206 at audit time, +2 from this spike's builds) carried Usage count: 1, spanning 2026-07-06 → 2026-08-05, totalling 91.36 GB at 438–440 MB each. Not one was ever reused.

2. Two builds through build.sh, byte-identical tree, both missed.

step	build 1	build 2
#8 WORKDIR /build	CACHED	CACHED
#9 COPY go.mod ./	CACHED	CACHED
#10 RUN go mod download || true	executed	executed
#11 COPY . .	executed	executed
#12 RUN go mod tidy	executed	executed
#13 RUN … go build …	executed	executed
#14/#15/#16 (stage-1 RUNs)	CACHED	CACHED

go.mod was byte-identical (d8ded3f0798c…) in the repo and in the workspace, mtime unchanged (2026-07-24 11:54:08).

3. build.sh's churn was ruled out by fingerprinting. Content-hashing all 737 files across rm -rf + cp -a + host-side go mod tidy: tidy changes nothing, and the reproduced tree is content-identical to the live workspace. go.mod's inode number was even identical across the churn.

4. A direct docker build on the untouched workspace cached everything — including RUN go mod download — proving BuildKit's cache is healthy and the Dockerfile is cacheable.

5. The decisive isolation. Same workspace, same command, only the build-arg value varied:

run	--build-arg VERSION	COPY go.mod ./	RUN go mod download
E1	new (spike-p7e)	CACHED	executed
E2	same (spike-p7e)	CACHED	CACHED

The mechanism, and it explains the #9 CACHED / #10 executed signature exactly: a COPY's cache key is content-based and carries no environment, so it hits; a RUN's key includes the stage environment, which contains every ARG declared above it. Since every real build passes a fresh VERSION (and often a fresh GIT_COMMIT), the module-download layer is invalidated on every build, without exception — 206 builds, 206 retained 440 MB snapshots, 90.5 GB.

Both Dockerfiles have it. felhom.eu/hub/Dockerfile declares ARG VERSION / ARG BUILD_TIME above COPY go.mod go.sum* → RUN go mod download. And because both produce byte-identical buildx du description strings, the 208 records are a COMBINED count and must not be attributed to one project.

Fix shape — not applied here, filed as R-208: move the ARG declarations down to just above the final go build. One line per Dockerfile. This is worth more than the cap and the move combined: the cap bounds the symptom at 30 GB; this removes the source.

---

8. P4 — the image store, enumerated (nothing deleted)

345 unique images across 350 tag lines (5 double-tagged), 0 dangling. Only two are held by running containers: redis:7-alpine and postgres:16-alpine. docker image prune -a would remove 343.

Size the prize before spending a decision on it. Per-tag sizes sum to 139.29 GB, but that double-counts shared layers. docker system df puts the real dedup'd footprint at 31.02 GB, of which 27.02 GB is reclaimable — roughly a fifth of what the build-cache prune already returned.

GB (per-tag sum)	tags	repository
118.44	207	gitea.dooplex.hu/admin/felhom-controller
7.25	107	gitea.dooplex.hu/admin/felhom-hub
4.34	5	gitea.dooplex.hu/admin/jarr
1.69	1	ghcr.io/seanmorley15/adventurelog-backend
0.94	1	ghcr.io/papra-hq/papra
0.72	1	codewithcj/sparkyfitness_server
0.72	1	postgis/postgis
0.59	3	gitea.dooplex.hu/admin/sparkyfitness-export
(23 more, each ≤0.53 GB)

Registry presence — which are recoverable

Queried against the live Gitea registry v2 API:

repo	tags in registry	tags local	local-only (NOT re-pullable)
felhom-controller	76	207	131
felhom-hub	45	107	62

The local-only tags are all old. Controller 0.39.0–0.135.0 plus v0.35.0–v0.39.0; hub 0.9.0–0.57.0 plus v0.7.2–v0.13.0. Everything from controller 0.136.0 and hub 0.58.0 upward is in the registry and therefore recoverable by docker pull.

This is the list the operator still owes a ruling on (R-210). CC's view: not worth doing for the space — ~27 GB against 199 GB now free. The only real benefit is dropping unrecoverable clutter.

---

9. P8 — what else writes to Docker

The bounded list of build triggers on the host Docker daemon is: interactive/manual runs only.

• 18 systemd timers enumerated — none invokes docker. (node-housekeeping, dooplex-backup, apt-daily*, sysstat*, logrotate, fstrim, e2scrub_all, smart-fancontrol, …)
• Root crontab: empty (comments only). User crontab: one entry, sync-hetzner-backups.sh, writes under /mnt/5_hdd.
• /etc/cron.d: 4 real files — anacron, e2scrub_all, sysstat, zfsutils. None touches docker.

The hole the audit left open is now closed with positive evidence. The audit concluded "no CI runner exists on this host" from systemd/process/crontab alone — but a Gitea Actions runner does exist as a cluster pod (R-168, 2026-08-02). It cannot reach the host daemon:

• 162 pods swept across all namespaces: 0 mount any docker/containerd hostPath.
• act-runner (gitea-system, image felhom-act-runner:0.1.0) has one volume — a PVC at /data. privileged: false, allowPrivilegeEscalation: false, all capabilities dropped, no docker socket.
• Its Dockerfile states the absence is deliberate: "DELIBERATELY NOT INSTALLED: a Go toolchain, Node, or Docker … There is no container runtime in here on purpose."
• gates.yml runs exactly one command — python3 scripts/<repo>_gates.py --fast — plus a git fetch and a failure-only Resend alarm. No docker build anywhere.

So the audit's ~7 builds/day attribution to interactive use is confirmed, and now on a complete enumeration rather than a partial one.

---

10. P9 — the monitoring rule fix (shipped)

File: homelab-manifests/mon-system/prometheus-rules.yaml, alert RootFsPressureDespiteHousekeeping. Commit 6808a4b.

Change: and → and on(instance). A bare and requires identical label sets; the left vector carries {device,fstype,instance,job,mountpoint} and the right — a node_exporter textfile metric — carries only {instance,job}. Nothing ever matched.

Red-proof (required, and both numbers recorded)

Threshold substituted 0.25 → 0.5, and the reason is stated rather than hidden: P3's prune had already taken / to 44.6% available, so the shipped threshold's left side is now legitimately false. The defect is in label matching, not in the threshold, so the proof is run at a threshold where both sides are individually true — which is exactly the condition the original rule was meant to catch.

expression	result_count
left side alone (avail/size < 0.5)	1
right side alone (time() − last_success < 1209600)	1
as shipped — bare and	0
fixed — and on(instance)	1

For completeness, at the shipped 0.25 threshold both forms return 0 today — because the left side is false now, not because the fix failed.

Annotation rewritten. It pointed at Longhorn replicas/snapshots (~10% of the growth). It now names docker buildx du first, records the 30 GB cap in /etc/docker/daemon.json, and notes that a reverted or unapplied cap looks exactly like this alert firing — so the alert text carries its own diagnosis.

Deployment — GitOps, no kubectl set image, no live ConfigMap edit

ArgoCD app monitoring (auto-sync off): hard-refresh → OutOfSync at 6808a4be54a7 → deliberate sync → Synced / Healthy, operation phase Succeeded.

Verified in Prometheus, not assumed from ArgoCD:

query : … < 0.25 and on (instance) time() - node_housekeeping_last_success_timestamp_seconds < 1209600
state : inactive     health : ok     lastError : (none)

inactive is now the correct answer — / really is fine at 44.6% available.

A near-miss worth recording: I invented a full commit SHA from ArgoCD's truncated display and the sync failed with upload-pack: not our ref. It failed loudly and cost one retry. It would not have failed loudly had the fabricated SHA happened to exist.

The finding P9 turned up on the way — R-211

There is no config-reloader. The mon-system/prometheus Deployment runs one container (prom/prometheus:v3.12.0) with no reloader sidecar. After the sync the updated rules file was present inside the pod (grep -c "and on(instance)" → 3 on the mounted symlink) while the Prometheus rules API still served the old expression for 4+ minutes, with no error anywhere. It took effect only after an explicit POST /-/reload (HTTP 200; --web.enable-lifecycle was already set).

The consequence is general: every rule edit in this repo has silently not applied until something happened to restart the pod. Synced/Healthy was true and beside the point. Filed as R-211.

---

11. P6 — the SSD2 move: EXECUTED 2026-08-05 on operator authorisation

Executed after the operator ruled "proceed", having read the pre-analysis below. The storageReserved condition CC named was applied with it. The operator subsequently ruled explicitly: DO NOT REBOOT DooPlex — so the reboot validation is DEFERRED, and what that leaves unproven is stated in §11.9 rather than glossed.

11.0 Execution record

Blast radius, established before anything was stopped: k3s runs a separate containerd (/run/k3s/containerd/containerd.sock), so Gitea, the registry, the hub, PBS, Longhorn and all ~160 pods stayed up throughout. The only workloads affected were the two jarr-* dev containers. ctr -n moby containers list confirmed exactly those two on the system containerd.

#	step	result
1	stop docker.service, docker.socket, containerd.service	all inactive
2	mkdir /mnt/ssd_2/containerd (mode 711, root:root)	ok
3	rsync -aHAX --numeric-ids	550,891 files, 33,276,370,041 B, 4 m 58 s
4	mv /var/lib/containerd → /var/lib/containerd.pre-move-2026-08-05	moved aside, never deleted
5	root = '/mnt/ssd_2/containerd' in /etc/containerd/config.toml (backup kept)	one-line diff
6	RequiresMountsFor=/mnt/ssd_2 drop-ins on both containerd.service and docker.service	resolved into both units
7	start + verify	see below
8	storageReserved = 80 GB on SSD2	applied

-X is load-bearing — overlayfs snapshots carry trusted.overlay.* xattrs, and losing them would silently corrupt layer stacking.

11.1 Copy verified BEFORE the original was touched

Four independent observables, all matching:

check	source	destination
filesystem entries	550,891	550,891
trusted.overlay xattrs	448	448
files with link count >1 (hardlinks)	37,243	37,243
io.containerd.metadata.v1.bolt/meta.db sha256	c60ecfb4ee134255…	c60ecfb4ee134255…

(Byte totals differ by +14 MB — ext4 block allocation, not content.)

11.2 Nothing was lost

	pre-move	post-move
unique images	345	345 — identical ID set
tag lines	350	350 — identical set
docker volumes	8	8 — identical set
build cache	2.782 GB / 38 records	2.782 GB / 38 records
GC ceiling	30 GiB	30 GiB, still in force
postgres databases	4	4
postgres jarr_dev bytes	175,135,767	175,135,767
postgres public tables	31	31
redis DBSIZE	2437	2437

End-to-end proof, not just inventory: a real ./build.sh run against the relocated store completed rc=0, produced an image, and grew the cache 2.782 → 4.012 GB. The scratch tag was removed and :latest restored to the anchor digest afterwards.

11.3 The store is demonstrably on SSD2

containerd config dump → root = '/mnt/ssd_2/containerd'
du /mnt/ssd_2/containerd → 35,500,183,552 B
/var/lib/containerd     → does not exist

11.4 The guard — and a trap found while proving it

First attempt did NOT refuse, and that is a finding worth more than the test. With RequiresMountsFor=/mnt/absent-guard-test-2026-08-05 — a path with no mount unit at all — containerd started normally. RequiresMountsFor on a path systemd has no mount unit for is a silent no-op. So a typo in the guard path buys nothing and says nothing: the same built-but-never-wired shape this project has shipped repeatedly.

So the guard was verified positively, at the unit level:

containerd.service  Requires=mnt-ssd_2.mount   After=mnt-ssd_2.mount
docker.service      Requires=mnt-ssd_2.mount

Then the refusal itself was proven with a genuinely absent DEVICE. /mnt/ssd_2 hosts 12 live Longhorn replicas and must never be unmounted, and editing fstab on a production host risks emergency mode at boot — so a temporary synthetic .mount unit pointing at a nonexistent UUID was used instead, with a temporary RequiresMountsFor on the real containerd.service:

A dependency job for containerd.service failed.
systemd: Dependency failed for containerd.service - containerd container runtime.
systemd: containerd.service: Job containerd.service/start failed with result 'dependency'.
containerd is-active: inactive

containerd REFUSED to start rather than creating an empty store on the root disk — exactly the required behaviour, and exactly the signature the ep0 runbook describes. All test artifacts were removed; only 10-ssd2-mount.conf remains on each unit and the guard resolves to /mnt/ssd_2 alone.

11.5 Longhorn — the condition CC made the move conditional on

storageReserved on SSD2 raised 0 → 80,000,000,000 B, covering the containerd tree so Longhorn's scheduler can no longer oversubscribe into the space the store needs.

disk	available	reserved	Schedulable
Crucial-SSD-240G (SSD2)	179,516,211,200 (76.34%)	80,000,000,000	True
default-disk (SSD1)	236,978,176,000 (49.64%)	143,217,954,816	True
HDD4	1,793,798,963,200 (60.82%)	536,870,912,000	True
HDD2	2,470,864,486,400 (24.91%)	536,870,912,000	False — pre-existing, untouched

SSD2 remains schedulable with the reserve applied. Rollback: patch storageReserved back to 0.

11.6 A standing post-boot assertion was installed

felhom-store-postboot-check.service (oneshot, After=docker.service, enabled) runs /usr/local/sbin/felhom-store-postboot-check.sh at every boot and logs to /var/log/felhom-store-postboot-check.log. It asserts, with positive observables:

• /mnt/ssd_2 is mounted;
• containerd's effective root is /mnt/ssd_2/containerd;
• /var/lib/containerd does NOT exist — i.e. no empty store was recreated on the root disk, which is the precise trap the guard exists to prevent;
• docker sees ≥100 images;
• both dev containers are running.

Dry-run at install time: RESULT: PASS on all six lines. It was proven working before being relied upon.

11.7 Disk position after the move

	value
/	224 GB used, 199 GB free, 54%
/mnt/ssd_2	52 GB used, 157 GB free, 25%
store on SSD2	35,500,183,552 B
rollback copy retained on /	34,299,498,496 B

/ did not drop below 54% because the rollback copy is deliberately still there. Reclaiming it returns ~34 GB — see §11.9.

11.8 Rollback, complete and one sequence

1. sudo systemctl stop docker docker.socket containerd
2. sudo mv /var/lib/containerd.pre-move-2026-08-05 /var/lib/containerd
3. sudo cp /etc/containerd/config.toml.pre-move-2026-08-05 /etc/containerd/config.toml
4. sudo rm -rf /etc/systemd/system/{containerd,docker}.service.d/10-ssd2-mount.conf
5. sudo systemctl disable --now felhom-store-postboot-check.service && sudo rm /etc/systemd/system/felhom-store-postboot-check.service /usr/local/sbin/felhom-store-postboot-check.sh
6. sudo kubectl patch nodes.longhorn.io -n longhorn-system dooplex --type merge -p '{"spec":{"disks":{"Crucial-SSD-240G":{"storageReserved":0}}}}'
7. sudo systemctl daemon-reload && sudo systemctl start containerd docker
8. sudo rm -rf /mnt/ssd_2/containerd

11.9 NOT PROVEN: the reboot. Stated, not glossed.

The operator ruled explicitly that DooPlex must not be rebooted, so the reboot validation did not happen. Uptime is unbroken (7 weeks 6 days, since 2026-06-10) — verified, not assumed.

By this task's own standard — "a move that has not survived a reboot has not been validated" — the move is NOT fully validated, and it would be dishonest to close it as if it were. Precisely:

• PROVEN (mechanism): the guard is wired into both units (Requires= + After=mnt-ssd_2.mount), and containerd refuses to start when a required mount's device is genuinely absent.
• NOT PROVEN (consequence): that a real boot sequence mounts /mnt/ssd_2 before containerd starts on this host, in the real ordering. Mount-ordering reasoning is exactly the class this project has been burned by, and the mechanism proof does not substitute for it. CLAUDE.md prefers the consequence over the mechanism; here only the mechanism is available.

Two consequences follow, and both are deliberate:

1. The rollback copy /var/lib/containerd.pre-move-2026-08-05 (34.3 GB) STAYS until a reboot has validated the move. Deleting it now would trade a cheap 34 GB for the only cheap way back.
2. The next reboot — whenever it happens, planned or not — validates this automatically. The post-boot service writes RESULT: PASS or RESULT: FAIL to /var/log/felhom-store-postboot-check.log. Read that file after the next boot; on PASS, the rollback copy can be removed and ~34 GB returned to /.

Tracked as R-209a.

---

11-bis. P6 — the pre-analysis the ruling was made on

(Retained as written, before execution, so the ruling can be audited against what it was given.)

First, the framing changed underneath this probe. P3's prune took / to 53% used and un-stuck SSD1's Longhorn disk. The move is no longer a rescue; it is a deliberate choice. Per standing rule 4, CC's disagreement is stated rather than absorbed: the cap alone resolved the problem the move was meant to solve, and the move carries a Longhorn risk the cap does not.

Mechanism — three candidates

	mechanism	survives package upgrade	expressible in Ansible	visible reading the host cold
(a)	root = "/mnt/ssd_2/…" in /etc/containerd/config.toml	yes — dpkg conffile, local edits preserved	yes — a template, the file is already managed-shaped	yes — one line in the daemon's own config
(b)	systemd mount unit / fstab bind onto /var/lib/containerd	yes	yes	weak — invisible unless you check findmnt
(c)	symlink	fragile	yes	weak, and overlayfs mount semantics through a symlink are unproven here

Recommendation: (a). It is the only one that a person reading the host cold can see, and the key already exists in the file (commented out).

The boot-order trap, and its guard

If containerd or Docker starts before /mnt/ssd_2 is mounted, the daemon creates an empty store on the root disk and 345 images appear to have vanished.

Guard: RequiresMountsFor=/mnt/ssd_2 on containerd.service and docker.service. Neither has it today — containerd.service carries only After=network.target dbus.service.

RequiresMountsFor RE-MOUNTS; it does not refuse. systemd pulls the mount unit in and starts it, so an unmounted-but-available device is silently re-mounted and everything looks fine. It refuses only when the device is genuinely unavailable. Testing it by unmounting proves nothing — it must be tested by pointing at an absent device. This is the ep0 datastore lesson (RUNBOOK-ep0-datastore-volume-2026-07-27.md), earned on this project.

And a move that has not survived a reboot has not been validated. Mount-ordering reasoning is exactly the class this project has been burned by.

The Longhorn consequence — the part that decides it

Crucial-SSD-240G, storageMaximum 235,148,750,848, 25% floor = 58,787,187,712:

scenario	resulting storageAvailable	as %	vs the 25% floor
today, untouched	214,958,080,000	91.41%	+66.4 pp
move cache only (30 GB)	184,958,080,000	78.66%	+53.7 pp
move whole containerd tree today (34.3 GB)	180,658,569,216	76.83%	+51.8 pp
move at steady state (~31.5 GB images + 30 GB cache ≈ 65 GB)	149,958,080,000	63.77%	+38.8 pp

As measured, the move is safe by a wide margin — nowhere near the 10-pp caution band.

But the worst case is not comfortable, and it is not hypothetical enough to omit. storageScheduled on SSD2 is 139,586,437,120 while actual usage is 20,094,939,136 — a 6.9× overcommit. If those 12 volumes ever inflate to their scheduled size:

scenario	available	as %	vs floor
+ cache only (30 GB)	65,562,313,728	27.88%	+2.9 pp — inside the caution band
+ steady state (65 GB)	30,562,313,728	13.00%	−12.0 pp → Schedulable=False

That is precisely the failure that just took SSD1 out of service.

Recommendation: proceed only together with setting Longhorn's storageReserved on SSD2 to cover the containerd tree (~80 GB). SSD2 reserving zero while HDD2 and HDD4 each reserve 500 GB is an anomaly in its own right, and it is what lets Longhorn oversubscribe into the space the cache needs. Without that, CC recommends not moving — the cap has already done the job.

The sequence, prepared and not executed

Each step with its rollback. Do not run without the operator.

#	step	rollback
1	systemctl stop docker docker.socket containerd	systemctl start containerd docker
2	mkdir -p /mnt/ssd_2/containerd	rmdir
3	rsync -aHAX --numeric-ids /var/lib/containerd/ /mnt/ssd_2/containerd/ (small now — ~34 GB, and ~2.8 GB of it is cache)	delete the copy; the original is untouched
4	mv /var/lib/containerd /var/lib/containerd.pre-move (move aside, never delete)	mv back
5	set root = '/mnt/ssd_2/containerd' in /etc/containerd/config.toml	restore the commented line
6	drop-in RequiresMountsFor=/mnt/ssd_2 on containerd.service and docker.service; systemctl daemon-reload	delete the drop-ins
7	systemctl start containerd docker; verify docker images = 345, both dev containers up, du confirms the store's real path	steps 5→4→1 in reverse
8	set SSD2 storageReserved ≈ 80 GB in the Longhorn node CR	set back to 0
9	reboot, then re-verify everything in step 7	as above
10	test the guard by pointing at an absent device — an unmount alone proves nothing	restore fstab
11	only after a clean reboot + a full build: rm -rf /var/lib/containerd.pre-move	— none; do this last

---

12. Teardown

Nothing was provisioned. No machine, no VM, no LXC guest, no hub record, no customer, no Longhorn volume, no PVC. There is nothing to decommission.

Cleaned up:

• The six candidate daemon.json files written for P2 (in the session scratch dir, never in /etc/docker/) — removed.
• The scratch workspace fingerprint tree (ws_sim) — removed.
• Nine scratch image tags created by the probe builds — felhom-spike:{p7c,e1,e2}, felhom-controller:{spike-p7a,spike-p7b,spike-p7d,evict-c1,evict-c2,evict-c3}, felhom-hub:{evict-h1,evict-h2,evict-h3} — all removed.
• felhom-controller:latest restored to sha256:bdcd4a4ae8e006f1… (0.198.0), byte-matching the anchor captured before the first build. Image count back to 345, the P0 baseline.

Intentional state left in place, recorded here so it does not read as a leftover: /etc/docker/daemon.json exists, with the content in §4. It did not exist before this session. Removing it plus a daemon restart is a complete revert.

Not cleaned up, deliberately: the ~2.78 GB of build cache from the eviction sequence. It is the working cache under a live 30 GB policy, not scratch.

---

13. Final state

	at arrival	now
/ used	387,332,755,456 B (86%)	240,301,133,824 B (54%)
/ available	65,734,934,528 B	212,766,556,160 B
/var/lib/containerd	181,394,223,104 B	34,299,510,784 B
build cache	157 GB / 1446 records	2.782 GB / 38 records, capped at 30 GiB
images	345	345
dev containers	2 healthy	2 healthy, data verified intact
SSD1 Longhorn	Schedulable=False (DiskPressure)	Schedulable=True, 50.32%
RootFsPressureDespiteHousekeeping	dead — could never fire	live, health=ok, red-proven
growth driver	unbounded, ~5.25 GB/day	capped; source diagnosed (R-208)

---

14. Observations — noticed, deliberately not acted on

1. build.sh exists in two copies — felhom-controller/controller/build.sh (in git) and /mnt/5_hdd/felhom.eu/build/felhom-controller/build.sh (the operative one). They differ by one comment line. The hub solved this with a symlink into the repo; the controller has not. Not fixed — out of scope, but it is a drift channel.
2. build.sh tags :latest on every build, including throwaway version tags. A spike build silently moved latest off 0.198.0; it was restored from an anchor captured beforehand. A --no-latest flag, or tagging latest only on --push, would remove the trap.
3. The default BuildKit GC policy is derived from disk size, so a bigger root disk would have raised the 331.6 GiB ceiling too. The cap needs to be explicit no matter how much disk is added.
4. node-housekeeping.sh's own avail_bytes() uses df … | tail -1 — the truncating-instrument shape this project has been bitten by twice. It is correct here (df on one path emits exactly one data row), so it was left alone, but it is worth knowing it is there.
5. HDD2 remains Schedulable=False (DiskPressure) at 24.91%, a hair under the 25% floor with 2.47 TB free and its 500 GB reserve satisfied. Untouched by this work, and not fixed by it.
6. The mon-system Prometheus has --web.enable-admin-api as well as --web.enable-lifecycle. The admin API permits series deletion. Noted, not acted on.
7. Longhorn on SSD2 is overcommitted 6.9× (139.6 GB scheduled vs 20.1 GB used). That is normal for thin provisioning, but combined with storageReserved = 0 it is what makes the P6 worst case reach the floor.

---

15. Register

Allocated this session: R-205 (CLOSED — the monitoring rule, shipped), R-206 (Ansible: cap + prune + narrowed Docker ban), R-207 (the DRY_RUN guard), R-208 (the ARG-ordering cache miss — the largest remaining win), R-209 (WAITING-ON-OPERATOR — whether to move to SSD2 at all), R-210 (WAITING-ON-OPERATOR — the image list), R-211 (no Prometheus config-reloader).