Files
felhom.eu/documentation/audits/space-list-2026-11-10
admin b14eb0fad4
gates / gates (push) Successful in 2m0s
hub: a full host disk is the operator's alarm (D68, decision 287) — commit only, not built
host_disk_warning and host_disk_critical join the operator-only register, so a
household that already has the type saved gets no mail. The type leaves the
claim seed list. storage_fill_* is split by storage kind: a drive the household
can free (usb, local-dir, nfs, cifs) still reaches the household; the thin pool,
a PBS store, an unknown kind and a missing kind go to the operator only.

Tests: TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt,
TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree,
TestD68_RegisterAndGate, TestD68_SeedHoldsNoOperatorOnlyType. Red seen three
times (register lines out; gate reading the register only; type back in the seed).

No hub build, no deploy. Also: dated lines on R-1368 and R-1371 for the agent
(776025c8) and controller (844f19f1) commits, and the filesystem-identity
measurement the controller commit cites.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 14:25:41 +01:00
..

The Space list — 2026-11-10 (pilot of the gap review)

What this is. Every place that holds data on a Felhom box, with eight questions for each place. Measured read-only on demo-hp, demo-felhom (N100) and Tester 1. Nothing was changed on any machine. Facts: TABLE.md · marks: MATRIX.md · proof: evidence/ · scripts: tools/. Baselines: felhom.eu a6705ceb04 · controller da060d7e7f · agent 41d5eb8051 · catalog f4c5c797d9.

Wrong claims in the prompt (named first). The prompt calls alpine:3.20, curl and hello-world the product's helper images. The product's helper is alpine with no tag. alpine:3.20 and curl came from test tools. hello-world comes from the template bake. One closed register row is wrong too: R-521 says the guest root disk has a fill watcher. It has none since the two guest volumes became one.

1. Five lines

  1. Places: 45 customer places (18 host, 17 guest, 10 data drive) · 7 test-only · 6 off-box, from documents only.
  2. No document named: 8 of the 45 (H4, H5, H12, H15, G9, G11, G15, G17). 16 more are named only in passing.
  3. Marks, 360 answers: A 294 · D 0 · K 19 · N 47. The 47 N fall into 5 classes = 5 new rows.
  4. Worst N: nobody removes old kernels (H9). A ring-0 box holds 8 kernels = 8.15 GB and gets about one more each week. The update wrapper refuses to remove a package, and nothing else does.
  5. You decide: four things — section 5.

2. The N answers, by class (one register row each; members in MATRIX.md)

  • C1 = R-1368 (10) — host places that only grow: old kernels, the agent's self-update folder (49 binaries, 740 MB), template archives, old agent binary copies, logs of failed backup runs.
  • C2 = R-1369 (24) — a disk is full and there is no written way out. One missing runbook, seen from 21 places.
  • C3 = R-1370 (1) — the host system disk has no space budget.
  • C4 = R-1371 (7) — no space alarm: the guest root disk has no watcher (three places lie on it); four small places. The row also holds two findings that are not N: the host-disk critical mail goes to the household, and one filesystem is watched twice.
  • C5 = R-1372 (5) — small guest places nobody described: helper images, off-site restore copies with no age limit (1.65 GB of two removed apps on the N100), log caps never re-checked, guest log in memory.

3. Things that can grow without a limit, by ruling (not defects)

Undo images: a count per app, no byte limit (decision 53). Kept data of a removed app (decision 40). The household's own files stay on every removal (decision 67). A set-aside unit, until the household removes it (decision 256). A restore's undo copy under a support hold (decision 281). The second-drive copy of a removed app (D25). The backup unit of a removed app that the household kept (07-backup-architecture.md:380). Off-box, from documents only: one image per version for ever in the registry, pruned by hand (decision 62).

4. The four questions

  1. Helper images. The product pulls alpine with no tag at seven call sites (first: felhom-controller/controller/internal/backup/backup.go:988). The bake pulls hello-world (felhom-agent/configs/build-golden.sh:232). None of the three image rules (decisions 53, 56, 285) can delete a helper image (internal/stacks/image_retention.go:244-250). They are small: 8.4 MB, 4 months old on two boxes.
  2. Container log caps. 10 MB × 3 files on all three boxes, for every container; no container escapes the cap. Worst case: demo-hp 780 MB · N100 270 MB · Tester 1 300 MB. The cap is written once at the bake; nothing checks it later.
  3. The host's own disk. Yes, the hub warns: host system disk at 90 % and 95 %; thin pool at 85 % and 90 % (the worse of data and metadata). The mail goes to you; the critical one also to the household. Nothing refuses or repairs. No architecture document names the host-disk alarm.
  4. Kernels and boot. Nothing removes old kernels. /boot is a folder on the system disk: 0.82 / 0.94 / 0.33 GB. The EFI partition is 1 % full on each box. Kernels: 8 / 8 / 3 = 8.15 / 8.15 / 3.05 GB.

System logs, two readings (07:39 and 09:45 UTC, 2 h 6 min apart; evidence/reading-1/, evidence/reading-2/). demo-hp host journal: 873.7 → 898.9 MB = about 12 MB an hour. The journal grows in 8 MiB steps, so two hours give only a rough rate (190–380 MB a day). It was 98 % of all growth on that system disk in the window (+25.7 MB). The N100 and Tester 1 sit at the journal's cap (4.30 GB each); their oldest entries did not move. Lines in the last hour: 13,521 / 13,506 / 11,177, and sudo wrote 95 / 94 / 93 % of them. Guest system log: 8 MB in memory, both readings. Container logs on demo-hp: +5.7 MB (2.7 MB an hour, under the 780 MB worst case). Unnamed volumes: no change in two hours; they come at night (about 29 a day on demo-hp).

5. Decisions for you (also in STATUS.md)

# Question Options and cost CC's pick If you do nothing
D67 May the nightly update remove old kernels by itself? Yes: keep the running one, the one before and the newest; it widens rule R4; one agent release. No: you remove them by hand on each box. Yes, ring 0 first a ring-0 box gains about 1 GB a week
D68 Who gets the "host disk critical" mail? Operator only: one hub change. Keep: the household gets a mail it cannot act on. Operator only the household is mailed about a disk only you can free
D69 Write a short "disk space" section in an architecture document? Yes: about one session; each place gets a limit and a cleaner. No: this table stays the only reference and goes stale. Yes 8 places stay unnamed; new ones appear unseen
D70 Run the next list ("Endings") with this method? Yes, with the three changes below. No: stop the gap review here. Yes the plan stops after its pilot

6. The method's verdict

It found real things nobody had asked about. New: kernels and agent binaries are never removed; the guest root disk has no watcher and a closed row said it had; a household gets the host-disk mail; off-site restore copies have no age limit. Four of the five new rows hold a measured fact, not an opinion. Cost: about 1.5 hours of work and a 1.5-hour wait for the second reading (the plan said 4–5 hours). Noise: the 47 N answers hold about 14 separate gaps. Question 7 gave the same answer 21 times: that is one gap, not 21. Some places are too small to matter (temp folders, the EFI partition). No D answer was found among 45 places: what was decided here was also built. Change three things for "Endings": (1) ask question 7 once for each disk, not for each place; (2) drop places under 100 MB that cannot grow, in the first hour; (3) a helper's answer does not go into the table until the main session has read its pointer — the off-box places here are still candidates for that reason. Weak spot: the off-box places (ep0, DooPlex, the off-site box, the hub database) were not measured, and the Prometheus rules in homelab-manifests were not read. Ruling 2026-11-10 12:22: DooPlex is never measured; ep0 not yet.

For a later session (small fixes, not done today — no code change in this run)

Keep only the newest N files in the agent's self-update folder · watch the guest root disk and de-duplicate fill-watch targets by filesystem (cmd/controller/main.go:2694-2707 has a stale comment) · pin the alpine helper · a stale document, on a protected machine's procedure, so not edited here: runbooks/ep0-datastore-copy.md:85 says "NOT INSTALLED" (closed row R-901: installed 2026-10-09).