From 47e372a5f45f60fce08a0ce781111bb0b30dc4d2 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 10 Nov 2026 10:50:44 +0100 Subject: [PATCH] Space list (pilot of the gap review): 45 places, eight questions each Read-only measurement on demo-hp, demo-felhom and Tester 1, two readings. Audit folder documentation/audits/space-list-2026-11-10/ (README, TABLE, MATRIX, evidence, tools). 360 answers: A 294, D 0, K 19, N 47. Register 225 -> 230: R-1368 (host places that only grow: kernels, agent binaries), R-1369 (no written way out of a full disk), R-1370 (no space budget for the host disk), R-1371 (no space alarm; host-disk mail goes to the household), R-1372 (small guest places). A dated line on nine existing rows. Section headings set to the real counts; R-1268 and R-1312 moved to their category's section. STATUS: new Updated paragraph, decisions D67-D70. N100 diary line. Two stale facts corrected in 05-hub-architecture.md and 11-os-updates.md. Nothing was changed on any machine. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- REPORT-space-list-2026-11-10.md | 91 +++ STATUS.md | 6 +- .../architecture/05-hub-architecture.md | 3 +- documentation/architecture/11-os-updates.md | 2 +- .../audits/space-list-2026-11-10/MATRIX.md | 93 +++ .../audits/space-list-2026-11-10/README.md | 93 +++ .../audits/space-list-2026-11-10/TABLE.md | 729 ++++++++++++++++++ .../evidence/extra/controller-fs-view.txt | 22 + .../extra/controller-state-volume.txt | 31 + .../evidence/extra/hp-9201.txt | 19 + .../evidence/extra/n100-9201.txt | 20 + .../evidence/extra/t1-9201.txt | 21 + .../evidence/os-cleaners/hp-9201.txt | 44 ++ .../evidence/os-cleaners/hp-host.txt | 50 ++ .../evidence/os-cleaners/n100-9201.txt | 44 ++ .../evidence/os-cleaners/n100-host.txt | 48 ++ .../evidence/os-cleaners/t1-9201.txt | 39 + .../evidence/os-cleaners/t1-host.txt | 53 ++ .../evidence/raw/hp-9201.txt | 630 +++++++++++++++ .../evidence/raw/hp-9202.txt | 576 ++++++++++++++ .../evidence/raw/hp-host-followup.txt | 522 +++++++++++++ .../evidence/raw/hp-host.txt | 585 ++++++++++++++ .../evidence/raw/n100-9201.txt | 372 +++++++++ .../evidence/raw/n100-host-followup.txt | 534 +++++++++++++ .../evidence/raw/n100-host.txt | 584 ++++++++++++++ .../evidence/raw/t1-9201.txt | 537 +++++++++++++ .../evidence/raw/t1-host-followup.txt | 449 +++++++++++ .../evidence/raw/t1-host.txt | 526 +++++++++++++ .../evidence/reading-1/hp-9201.tsv | 23 + .../evidence/reading-1/hp-9202.tsv | 23 + .../evidence/reading-1/hp-host.tsv | 25 + .../evidence/reading-1/n100-9201.tsv | 23 + .../evidence/reading-1/n100-host.tsv | 25 + .../evidence/reading-1/t1-9201.tsv | 26 + .../evidence/reading-1/t1-host.tsv | 23 + .../evidence/reading-2/hp-9201.tsv | 23 + .../evidence/reading-2/hp-9202.tsv | 23 + .../evidence/reading-2/hp-host.tsv | 25 + .../evidence/reading-2/n100-9201.tsv | 23 + .../evidence/reading-2/n100-host.tsv | 25 + .../evidence/reading-2/t1-9201.tsv | 26 + .../evidence/reading-2/t1-host.tsv | 23 + .../evidence/searches.txt | 49 ++ .../tools/check-pointers.py | 78 ++ .../tools/guest-baseline.sh | 36 + .../tools/guest-discover.sh | 39 + .../tools/guest-extra.sh | 18 + .../tools/host-baseline.sh | 40 + .../tools/host-discover.sh | 49 ++ .../tools/host-followup.sh | 33 + .../tools/make-matrix.py | 127 +++ .../tools/os-cleaners.sh | 13 + .../space-list-2026-11-10/tools/searches.sh | 66 ++ documentation/backlog/OPEN-ITEMS.md | 39 +- documentation/backlog/STATUS-HISTORY.md | 4 + documentation/lived-in/n100.md | 1 + 56 files changed, 7631 insertions(+), 20 deletions(-) create mode 100644 REPORT-space-list-2026-11-10.md create mode 100644 documentation/audits/space-list-2026-11-10/MATRIX.md create mode 100644 documentation/audits/space-list-2026-11-10/README.md create mode 100644 documentation/audits/space-list-2026-11-10/TABLE.md create mode 100644 documentation/audits/space-list-2026-11-10/evidence/extra/controller-fs-view.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/extra/controller-state-volume.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/extra/hp-9201.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/extra/n100-9201.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/extra/t1-9201.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/os-cleaners/hp-9201.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/os-cleaners/hp-host.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/os-cleaners/n100-9201.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/os-cleaners/n100-host.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/os-cleaners/t1-9201.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/os-cleaners/t1-host.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/raw/hp-9201.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/raw/hp-9202.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/raw/hp-host-followup.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/raw/hp-host.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/raw/n100-9201.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/raw/n100-host-followup.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/raw/n100-host.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/raw/t1-9201.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/raw/t1-host-followup.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/raw/t1-host.txt create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-1/hp-9201.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-1/hp-9202.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-1/hp-host.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-1/n100-9201.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-1/n100-host.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-1/t1-9201.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-1/t1-host.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-2/hp-9201.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-2/hp-9202.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-2/hp-host.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-2/n100-9201.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-2/n100-host.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-2/t1-9201.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/reading-2/t1-host.tsv create mode 100644 documentation/audits/space-list-2026-11-10/evidence/searches.txt create mode 100644 documentation/audits/space-list-2026-11-10/tools/check-pointers.py create mode 100644 documentation/audits/space-list-2026-11-10/tools/guest-baseline.sh create mode 100644 documentation/audits/space-list-2026-11-10/tools/guest-discover.sh create mode 100644 documentation/audits/space-list-2026-11-10/tools/guest-extra.sh create mode 100644 documentation/audits/space-list-2026-11-10/tools/host-baseline.sh create mode 100644 documentation/audits/space-list-2026-11-10/tools/host-discover.sh create mode 100644 documentation/audits/space-list-2026-11-10/tools/host-followup.sh create mode 100644 documentation/audits/space-list-2026-11-10/tools/make-matrix.py create mode 100644 documentation/audits/space-list-2026-11-10/tools/os-cleaners.sh create mode 100644 documentation/audits/space-list-2026-11-10/tools/searches.sh diff --git a/REPORT-space-list-2026-11-10.md b/REPORT-space-list-2026-11-10.md new file mode 100644 index 00000000..b597b61f --- /dev/null +++ b/REPORT-space-list-2026-11-10.md @@ -0,0 +1,91 @@ +# REPORT — the Space list (pilot of the gap review), 2026-11-10 + +Task: every place that holds data on a box, eight questions each. Read-only. Session 07:15–10:00 UTC. +Everything is in `documentation/audits/space-list-2026-11-10/` (`README.md` first). + +## 1. Claims in the prompt or the plan that were wrong + +1. **The helper images.** The prompt names `alpine:3.20`, `curl` and `hello-world` as the product's helper images. + The product pulls `alpine` with **no tag** (seven call sites). `alpine:3.20` and `curl` came from test tools. + `hello-world` comes from the template bake. +2. **Not in the prompt, but found wrong on the way:** closed row R-521 says the guest root disk has a fill watcher. + It has none since the two guest volumes became one (measured on three boxes). Capability-map row C13 says + "per filesystem"; this one filesystem is the exception. Filed in R-1371. The map's status is not changed. +3. **My own number:** an early draft of the table said `sudo` wrote "92 % session lines". R-1367 says 61 %. Both + are right: `sudo` writes three lines for each call; R-1367 counts two of them. The table and the row say so now. +4. The register count in the prompt (225) was right. **The section headings in the register were not:** they + added up to 237. I set all eleven to the real numbers and moved two rows that sat in the wrong section + (R-1268, R-1312 → Backup & restore). No row was changed, closed or re-ranked by that. + +## 2. Five lines + +1. **Places:** 45 customer places (18 host, 17 guest, 10 data drive) · 7 test-only · 6 off-box, from documents only. +2. **No document named:** 8 of the 45. 16 more are named only in passing. +3. **Marks, 360 answers:** A 294 · D 0 · K 19 · N 47. The 47 N are 5 classes = 5 new rows. +4. **Worst N:** nothing removes old kernels from the host. A ring-0 box holds 8 (8.15 GB) and gets about one a week. +5. **The operator decides** four things: D67 (remove old kernels automatically), D68 (who gets the host-disk + critical mail), D69 (write a "disk space" section), D70 (run the next list the same way). + +## 3. The method's verdict + +It worked. It found real gaps nobody had asked about, in about 1.5 hours of work plus a 1.5-hour wait for the +second reading. New and measured: kernels and old agent binaries are never removed; the guest root disk has no +watcher; a household gets the host-disk mail; off-site restore copies have no age limit. +The weak parts: question 7 ("is there a runbook") gave the same answer 21 times — one gap. Some places are too +small to matter. No `D` answer appeared. The off-box places are candidates only: a helper filled them and the main +session did not read each pointer. The Prometheus rules in `homelab-manifests` were not read. +For "Endings": ask the runbook question once for each disk; drop tiny places in the first hour; a helper's answer +enters the table only after the main session read its pointer. + +## 4. Machines, and what was changed on them + +Read: demo-hp host, guests 9201 and 9202 · Tester 1 host and guest 9201 · demo-felhom (N100) host and guest 9201. +Two readings: 07:38–07:39 and 09:44–09:45 UTC. Before each one I checked that no backup, restore or update ran. +**Nothing was changed on any machine.** Provisioned nothing; nothing to tear down (machine, host, hub). +Not read, by the brief: ep0, DooPlex's disks, the off-site box, the hub database. +Evidence was copied to the audit folder at the end of each reading. No secret is in it (scanned); the N100 +evidence holds folder sizes only, no file names. + +## 5. Commits, CI, register + +- Commit and CI: see the last section of this file. +- **Register: 225 before · 230 after · 5 opened · 0 closed.** New: R-1368 (host places that only grow), R-1369 (no + written way out of a full disk), R-1370 (no space budget for the host disk), R-1371 (no space alarm; the host-disk + mail), R-1372 (small guest places). Every N answer belongs to exactly one of them; `MATRIX.md` lists the members. +- A dated line, no other change, on R-548, R-1366, R-1367, R-995, R-1267, R-1268, R-1087, R-1294, R-540. + No rank change is proposed for R-1367: the journal stops at its own cap. +- `STATUS.md`: new "Updated" paragraph (the old one is in `documentation/backlog/STATUS-HISTORY.md`); D67–D70. +- `documentation/lived-in/n100.md`: one dated line with its rows. +- **Facts corrected in place** (facts only; no rule, fence or ruling): + - `documentation/architecture/05-hub-architecture.md:615` — before: "the volume grew from 1 GiB to 2 GiB"; + after: adds "4 GiB since 2026-10-07". Evidence: `manifests/hub.yaml:58` (`storage: 4Gi`). + - `documentation/architecture/11-os-updates.md:107` — before: "json-file log caps. **No `live-restore`.**", + cited `build-golden.sh:138-144`; after: "log caps (10 MB × 3 files), `live-restore: true`", cited `:193-203`. + Evidence: `felhom-agent/configs/build-golden.sh:195-201`; the same document, line 429. + - `documentation/backlog/OPEN-ITEMS.md` — eleven section headings set to the real row counts (see 1.4). +- Capability map, poster, CHANGELOG: no change. The poster facts file: no fact in it changed. +- `python3 scripts/unproven.py --summary`: 23 of 55 not walked. The number did not move. + +## 6. Architecture documents read + +`07-backup-architecture.md` · `09-update-architecture.md` · `11-os-updates.md` · `03-host-agent.md` · +`08-alarm-ladder.md` · `16-storage-physical-identity.md` · `05-hub-architecture.md` · +`02-controller-module-map.md` · `_recovery-inventory-2026-07-28.md` · `00-capability-map.md` (rows A19, C13, C15) · +`felhom-system-poster.facts.md`. **No architecture document covers disk space as a whole** — the absence the task +tests is real: 8 customer places are named nowhere, and `host_disk_*` appears in no architecture document. + +## 7. Observations outside the scope + +- Tester 1 holds 3 kernels, the ring-0 boxes 8. NOT-A-FINDING: Tester 1 is ring 1 and gets fewer kernels. +- Tester 1's drive has four empty folders with no app. NOT-A-FINDING: 0 bytes; the guards refuse a write there. +- Tester 1's host swap use rose from 186 MB to 513 MB between the two readings. NOT-A-FINDING: swap is a fixed + volume and its use moves with memory load; two readings cannot say more. +- `runbooks/ep0-datastore-copy.md:85` says the removal job is "NOT INSTALLED"; closed row R-901 says it was + installed on 2026-10-09. NOT-A-FINDING for this run (a DooPlex procedure; not edited); listed in the README + under "for a later session". +- The comment at `felhom-controller/controller/cmd/controller/main.go:2694-2707` is stale. FILED: R-1371. +- Register section headings were stale; two rows sat in the wrong section. Fixed in this run without a row (small). + +## 8. Commit and CI + +The work commit and its CI result are added here by a second commit. diff --git a/STATUS.md b/STATUS.md index 1077f04c..4f78b649 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-11-09.** demo-hp did not restart, because its local whole-box copy was refused for space again and the night update waited; CC freed space (old images, an old installer file), so tonight's copy fits. Controller 0.351.0 is released to every box (floor 12:18 UTC): it deletes old test builds and old system images by itself. Two new rows: the whole-box copy is mostly app programs (your decision, R-1366), and the host log noise (R-1367). +**Updated 2026-11-10.** CC listed every place that holds data on a box — 45 places, read-only on the three test boxes — and asked eight questions of each: 47 answers were „nobody asked”, now in five new rows (R-1368…R-1372). The worst one: nothing removes old kernels from the host; a ring-0 box holds 8 of them (8 GB) and gets about one more each week. Four decisions wait for you (D67–D70); the list is in `documentation/audits/space-list-2026-11-10/README.md`. ## What works - **The Plex attack did not reach Felhom's secrets** (R-1241). @@ -32,6 +32,10 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| +| D70 | — | Run the next gap list („Endings”: every way a thing stops, is removed or expires) with the same method as the Space list? | **Yes, with three changes** (it found real gaps in about 1.5 hours of work; the changes cut the repeats) | no: the gap review stops after its pilot; the other four lists are never made | no second list is started | — | +| D69 | R-1370 | Write a short „disk space” section in an architecture document: each place, its limit, its cleaner? Today no document covers disk space as a whole. | **Yes** (8 of 45 places are named in no document; about one session) | no: the audit table of 2026-11-10 stays the only reference and goes stale | new places that grow appear unseen, as the kernels and the agent's old binaries did | — | +| D68 | R-1371 | Who gets the „host disk critical” and „storage critical” mail? Today you and, by default, the household. Only you can free a host disk. | **Operator only** (the household cannot act on it; one hub change) | keep: the household gets an alarm it cannot act on and asks you | the household is mailed when the host disk passes 95 % | — | +| D67 | R-1368 | May the nightly update remove old kernels by itself — keep the running one, the one before it and the newest? Today the wrapper refuses every removal (rule R4), so a box keeps every kernel it ever got. | **Yes, ring 0 first** (8 kernels = 8 GB on a ring-0 box, about 1 GB more each week) | no: you remove them by hand on each box, every few months | the host system disk of a ring-0 box grows about 1 GB a week until the 90 % alarm | — | | D66 | R-1365 | Should a security-watch source that stays down mail again every 7 days, as an open notice does? Today it mails once, then only the Monday summary lists it. | **Yes** (a real outage then cannot be forgotten; the seven pins that can never be read no longer count, so it stays quiet when all is well) | no: one line in a long Monday mail stays the only reminder | a source that stays down mails once; the Monday list still shows it | — | | D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | | D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — | diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index 554d08c1..df234b36 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -613,7 +613,8 @@ design; its mechanics hold per slot. Originally: every night at **02:00 Budapest one statement, one point in time, every committed write included — the rows still in `hub.db-wal` too, which a file copy of `hub.db` loses (measured in `internal/dbsnap` tests: a plain copy held 0 of 150 fresh rows). Written as `.tmp`, mode `0600`, then renamed, so a reader never sees half a file. The newest **2** stay (the volume grew from 1 GiB to 2 GiB -for them, operator choice 2026-10-05; one snapshot measured 370 MB). Two runs never overlap (a second gets `ErrBusy`). +for them, operator choice 2026-10-05; one snapshot measured 370 MB; **4 GiB since 2026-10-07**, `manifests/hub.yaml` +`storage: 4Gi` — corrected 2026-11-10). Two runs never overlap (a second gets `ErrBusy`). Each run logs `db snapshot written: (, )`. At start-up the hub runs one when the newest is older than 24 h. **The hub ships nothing itself and holds no ep0 credential:** DooPlex's `felhom-hub-db-backup` unit picks the newest snapshot up at 02:30, checks it (`PRAGMA integrity_check`), encrypts it and pushes it to ep0's `operator` diff --git a/documentation/architecture/11-os-updates.md b/documentation/architecture/11-os-updates.md index 08361c8f..b867f953 100644 --- a/documentation/architecture/11-os-updates.md +++ b/documentation/architecture/11-os-updates.md @@ -104,7 +104,7 @@ boxes installed. Nobody keeps a hand-written list. | Host kernel | The only kernel on the box. The guest has none. | `pve-no-subscription` (`proxmox-kernel-*`) | `[FACT]` LXC shares the host kernel (`01` §2: "one LXC/kernel/Docker daemon") | | Guest | Unprivileged LXC, `nesting=1,keyctl=1`, from `debian-13-standard_13.1-2` | Debian mirrors | `[FACT]` `felhom-agent/configs/build-golden.sh:67,101-103` | | Docker engine | `docker-ce`, `docker-ce-cli`, `containerd.io`, installed at golden BAKE time | `download.docker.com/linux/debian trixie stable` | `[FACT]` `build-golden.sh:108-125` | -| Docker settings | `containerd-snapshotter: false`, json-file log caps. **No `live-restore`.** | baked `daemon.json` | `[FACT]` `build-golden.sh:138-144` | +| Docker settings | `containerd-snapshotter: false`, json-file log caps (10 MB × 3 files), **`live-restore: true`** since the 3.1.0 bake (line 429 below; this cell said „No `live-restore`” until 2026-11-10). | baked `daemon.json` | `[FACT]` `build-golden.sh:193-203` | | Controller | A container in the guest. A Docker engine restart restarts it too. | Felhom registry | `[FACT]` `03` §1 | **[FACT] Nothing updates any of these layers today** (R-812, searched 2026-10-03). The installer diff --git a/documentation/audits/space-list-2026-11-10/MATRIX.md b/documentation/audits/space-list-2026-11-10/MATRIX.md new file mode 100644 index 00000000..f48c4932 --- /dev/null +++ b/documentation/audits/space-list-2026-11-10/MATRIX.md @@ -0,0 +1,93 @@ +# The Space list — the matrix + +Made by `tools/make-matrix.py` from `TABLE.md`. Do not edit it by hand. + +`A` answered · `D` decided, not built · `K` known gap (has a row) · `N` nobody asked. Columns are the eight +questions: 1 content · 2 growth · 3 limit · 4 cleaner · 5 when full · 6 alarm · 7 repair or runbook · 8 how we know. + +| Place | Named | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | N | +|---|---|---|---|---|---|---|---|---|---|---| +| H1 Host system disk, root filesystem `/` (LV `pve/root`) | named | A | A | **N** | K | A | A | **N** | A | 2 | +| H2 Thin pool `pve/data` (data and metadata) | named | A | A | A | A | A | A | **N** | A | 1 | +| H3 Whole-box copies on the current local target | named | A | K | A | A | K | A | K | A | 0 | +| H4 Whole-box copies on a former target | unnamed | A | A | K | K | K | A | K | A | 0 | +| H5 Guest template archives ("golden") in the dump folder | unnamed | A | A | **N** | **N** | A | A | **N** | A | 3 | +| H6 Backup logs and half-written copies in the dump folder | mentioned | A | A | **N** | A | A | A | A | A | 1 | +| H7 Host journal `/var/log/journal` | mentioned | A | K | K | A | K | A | K | A | 0 | +| H8 Other host logs, Proxmox task logs, graph data | mentioned | A | A | A | A | A | A | A | A | 0 | +| H9 Old kernels: `/lib/modules` and `/boot` | mentioned | A | A | **N** | **N** | A | A | **N** | **N** | 4 | +| H10 EFI partition `/boot/efi` | mentioned | A | A | A | A | **N** | **N** | **N** | A | 3 | +| H11 Host package cache and package lists | named | A | A | A | A | A | A | A | A | 0 | +| H12 The agent's self-update folder `/var/lib/felhom-agent/selfupdate` | unnamed | A | A | **N** | **N** | A | A | **N** | A | 3 | +| H13 Old copies of the agent binary (`.prev`, `.bak-*`, `.pre-bundle`) | mentioned | A | A | **N** | **N** | A | A | **N** | A | 3 | +| H14 Update wrapper state and previous signed bundles | mentioned | A | A | A | A | A | A | A | A | 0 | +| H15 Host `/tmp` and `/var/tmp` | unnamed | A | A | A | A | **N** | **N** | A | A | 2 | +| H16 The bare folder under an unmounted drive | named | A | A | A | A | A | A | A | A | 0 | +| H17 The restore-test's scratch guest | named | A | A | A | A | A | A | A | A | 0 | +| H18 Swap (LV `pve/swap`) | named | A | A | A | A | A | A | A | A | 0 | +| G1 Guest root disk (`vm-9201-disk-0`, 32 GiB) | mentioned | A | A | A | A | A | **N** | **N** | A | 2 | +| G2 Guest data volume (`mp0`, `/var/lib/felhom`: Docker and `sys_drive`) | named | A | A | A | A | A | A | **N** | A | 1 | +| G3 App images, with one undo image per app | named | A | A | A | A | A | A | A | A | 0 | +| G4 Old controller images | named | A | A | A | A | A | A | A | A | 0 | +| G5 Infrastructure images (traefik and the other pinned helpers of the box) | named | A | A | A | A | A | A | A | A | 0 | +| G6 Helper images (`alpine`, `hello-world`) | mentioned | A | A | **N** | **N** | A | A | A | A | 2 | +| G7 Named app volumes | named | A | A | A | A | A | A | **N** | A | 1 | +| G8 Unnamed (anonymous) volumes | named | A | K | K | K | K | A | K | A | 0 | +| G9 Container writable layers, with the controller's off-site cache | unnamed | A | A | A | A | A | A | A | A | 0 | +| G10 Container logs | mentioned | A | A | A | A | A | A | **N** | A | 1 | +| G11 Docker build cache | unnamed | A | A | A | A | A | A | A | A | 0 | +| G12 The controller's own state volume | mentioned | A | A | A | A | A | A | **N** | A | 1 | +| G13 App definitions `/opt/docker/stacks` | mentioned | A | A | A | A | A | **N** | **N** | A | 2 | +| G14 Backups of apps with no drive (`sys_drive`) | named | A | A | A | A | A | A | **N** | A | 1 | +| G15 Guest system log | unnamed | A | A | **N** | A | A | **N** | A | A | 2 | +| G16 Guest package cache and lists | mentioned | A | A | A | A | A | **N** | A | A | 1 | +| G17 Guest `/tmp` and `/var/tmp` | unnamed | A | A | A | A | **N** | **N** | A | A | 2 | +| D1 `backups/primary` — the app's own backup unit | named | A | A | A | A | A | A | **N** | A | 1 | +| D2 `backups/secondary` — the copy on a second drive | named | A | A | A | A | A | A | **N** | A | 1 | +| D3 `backups/offsite-restore` — copies fetched from the off-site store | mentioned | A | A | **N** | A | A | A | **N** | A | 2 | +| D4 `backups/offsite-proof` — the nightly proof's scratch | mentioned | A | A | A | A | A | A | A | A | 0 | +| D5 `backups/restore-undo` — what a restore replaced | named | A | A | A | K | A | A | A | A | 0 | +| D6 `kept` — data kept at "remove the app, keep my data" | named | A | A | A | A | A | A | **N** | A | 1 | +| D7 `appdata` — the apps' files on the drive | named | A | A | A | A | A | A | **N** | A | 1 | +| D8 `userdata` — the household's own folders | named | A | A | A | K | A | A | **N** | A | 1 | +| D9 `shares` — folders shared on the home network | mentioned | A | A | A | A | A | A | **N** | A | 1 | +| D10 `backups/set-aside` — the backup of an earlier install of the same app | named | A | A | A | A | A | A | **N** | A | 1 | + +## Totals + +- Places: **45** customer places · named **21** · mentioned only **16** · named by no architecture document **8**. +- Answers: **360** · A **294** · D **0** · K **19** · N **47**. +- Places with no N: **17** of 45. + +| Question | A | D | K | N | +|---|---|---|---|---| +| 1 what is in it | 45 | 0 | 0 | 0 | +| 2 what makes it grow | 42 | 0 | 3 | 0 | +| 3 the limit | 33 | 0 | 3 | 9 | +| 4 who cleans up | 35 | 0 | 5 | 5 | +| 5 what happens when full | 38 | 0 | 4 | 3 | +| 6 who gets an alarm | 38 | 0 | 0 | 7 | +| 7 self-repair or runbook | 19 | 0 | 4 | 22 | +| 8 how do we know | 44 | 0 | 0 | 1 | + +## The N answers, by class row + +| Class | Row | Count | Members (place-question) | +|---|---|---|---| +| C1 | R-1368 | 10 | H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4 | +| C2 | R-1369 | 24 | H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7 | +| C3 | R-1370 | 1 | H1-3 | +| C4 | R-1371 | 7 | H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6 | +| C5 | R-1372 | 5 | G6-3, G6-4, G10-7, G15-3, D3-3 | + +## The K answers, by existing row + +| Row | Count | Members | +|---|---|---| +| R-548 | 3 | H1-4, H3-5, H3-7 | +| R-995 | 3 | G8-4, G8-7, D8-4 | +| R-1087 | 4 | H4-3, H4-4, H4-5, H4-7 | +| R-1267 | 3 | G8-2, G8-3, G8-5 | +| R-1268 | 1 | D5-4 | +| R-1366 | 1 | H3-2 | +| R-1367 | 4 | H7-2, H7-3, H7-5, H7-7 | diff --git a/documentation/audits/space-list-2026-11-10/README.md b/documentation/audits/space-list-2026-11-10/README.md new file mode 100644 index 00000000..5f2237b2 --- /dev/null +++ b/documentation/audits/space-list-2026-11-10/README.md @@ -0,0 +1,93 @@ +# The Space list — 2026-11-10 (pilot of the gap review) + +**What this is.** Every place that holds data on a Felhom box, with eight questions for each place. Measured +read-only on demo-hp, demo-felhom (N100) and Tester 1. Nothing was changed on any machine. +Facts: `TABLE.md` · marks: `MATRIX.md` · proof: `evidence/` · scripts: `tools/`. +Baselines: felhom.eu `a6705ceb04` · controller `da060d7e7f` · agent `41d5eb8051` · catalog `f4c5c797d9`. + +**Wrong claims in the prompt (named first).** The prompt calls `alpine:3.20`, `curl` and `hello-world` the +product's helper images. The product's helper is `alpine` with no tag. `alpine:3.20` and `curl` came from test +tools. `hello-world` comes from the template bake. One closed register row is wrong too: R-521 says the guest +root disk has a fill watcher. It has none since the two guest volumes became one. + +## 1. Five lines + +1. **Places:** 45 customer places (18 host, 17 guest, 10 data drive) · 7 test-only · 6 off-box, from documents only. +2. **No document named:** 8 of the 45 (H4, H5, H12, H15, G9, G11, G15, G17). 16 more are named only in passing. +3. **Marks, 360 answers:** A 294 · D 0 · K 19 · N 47. The 47 N fall into 5 classes = 5 new rows. +4. **Worst N:** nobody removes old kernels (H9). A ring-0 box holds 8 kernels = 8.15 GB and gets about one more + each week. The update wrapper refuses to remove a package, and nothing else does. +5. **You decide:** four things — section 5. + +## 2. The N answers, by class (one register row each; members in `MATRIX.md`) + +- **C1 = R-1368 (10) — host places that only grow:** old kernels, the agent's self-update folder (49 binaries, + 740 MB), template archives, old agent binary copies, logs of failed backup runs. +- **C2 = R-1369 (24) — a disk is full and there is no written way out.** One missing runbook, seen from 21 places. +- **C3 = R-1370 (1) — the host system disk has no space budget.** +- **C4 = R-1371 (7) — no space alarm:** the guest root disk has no watcher (three places lie on it); four small + places. The row also holds two findings that are not N: the host-disk critical mail goes to the household, and + one filesystem is watched twice. +- **C5 = R-1372 (5) — small guest places nobody described:** helper images, off-site restore copies with no age + limit (1.65 GB of two removed apps on the N100), log caps never re-checked, guest log in memory. + +## 3. Things that can grow without a limit, by ruling (not defects) + +Undo images: a count per app, no byte limit (decision 53). Kept data of a removed app (decision 40). The household's +own files stay on every removal (decision 67). A set-aside unit, until the household removes it (decision 256). A +restore's undo copy under a support hold (decision 281). The second-drive copy of a removed app (D25). The backup +unit of a removed app that the household kept (`07-backup-architecture.md:380`). Off-box, from documents only: one image per +version for ever in the registry, pruned by hand (decision 62). + +## 4. The four questions + +1. **Helper images.** The product pulls `alpine` with no tag at seven call sites (first: + `felhom-controller/controller/internal/backup/backup.go:988`). The bake pulls `hello-world` + (`felhom-agent/configs/build-golden.sh:232`). None of the three image rules (decisions 53, 56, 285) can delete + a helper image (`internal/stacks/image_retention.go:244-250`). They are small: 8.4 MB, 4 months old on two boxes. +2. **Container log caps.** 10 MB × 3 files on all three boxes, for every container; no container escapes the cap. + Worst case: demo-hp 780 MB · N100 270 MB · Tester 1 300 MB. The cap is written once at the bake; nothing + checks it later. +3. **The host's own disk.** Yes, the hub warns: host system disk at 90 % and 95 %; thin pool at 85 % and 90 % + (the worse of data and metadata). The mail goes to you; the critical one also to the household. Nothing + refuses or repairs. No architecture document names the host-disk alarm. +4. **Kernels and boot.** Nothing removes old kernels. `/boot` is a folder on the system disk: 0.82 / 0.94 / + 0.33 GB. The EFI partition is 1 % full on each box. Kernels: 8 / 8 / 3 = 8.15 / 8.15 / 3.05 GB. + +**System logs, two readings (07:39 and 09:45 UTC, 2 h 6 min apart; `evidence/reading-1/`, `evidence/reading-2/`).** +demo-hp host journal: 873.7 → 898.9 MB = about 12 MB an hour. The journal grows in 8 MiB steps, so two hours give +only a rough rate (190–380 MB a day). It was 98 % of all growth on that system disk in the window (+25.7 MB). +The N100 and Tester 1 sit at the journal's cap (4.30 GB each); their oldest entries did not move. Lines in the last +hour: 13,521 / 13,506 / 11,177, and `sudo` wrote 95 / 94 / 93 % of them. Guest system log: 8 MB in memory, both +readings. Container logs on demo-hp: +5.7 MB (2.7 MB an hour, under the 780 MB worst case). Unnamed volumes: no +change in two hours; they come at night (about 29 a day on demo-hp). + +## 5. Decisions for you (also in `STATUS.md`) + +| # | Question | Options and cost | CC's pick | If you do nothing | +|---|---|---|---|---| +| D67 | May the nightly update remove old kernels by itself? | **Yes:** keep the running one, the one before and the newest; it widens rule R4; one agent release. **No:** you remove them by hand on each box. | **Yes**, ring 0 first | a ring-0 box gains about 1 GB a week | +| D68 | Who gets the "host disk critical" mail? | **Operator only:** one hub change. **Keep:** the household gets a mail it cannot act on. | **Operator only** | the household is mailed about a disk only you can free | +| D69 | Write a short "disk space" section in an architecture document? | **Yes:** about one session; each place gets a limit and a cleaner. **No:** this table stays the only reference and goes stale. | **Yes** | 8 places stay unnamed; new ones appear unseen | +| D70 | Run the next list ("Endings") with this method? | **Yes, with the three changes below. No:** stop the gap review here. | **Yes** | the plan stops after its pilot | + +## 6. The method's verdict + +**It found real things nobody had asked about.** New: kernels and agent binaries are never removed; the guest root +disk has no watcher and a closed row said it had; a household gets the host-disk mail; off-site restore copies have +no age limit. Four of the five new rows hold a measured fact, not an opinion. +**Cost:** about 1.5 hours of work and a 1.5-hour wait for the second reading (the plan said 4–5 hours). **Noise:** the 47 N answers hold about 14 separate gaps. Question 7 gave the same +answer 21 times: that is one gap, not 21. Some places are too small to matter (temp folders, the EFI partition). +**No `D` answer** was found among 45 places: what was decided here was also built. +**Change three things for "Endings":** (1) ask question 7 once for each disk, not for each place; (2) drop +places under 100 MB that cannot grow, in the first hour; (3) a helper's answer does not go into the table until +the main session has read its pointer — the off-box places here are still candidates for that reason. +**Weak spot:** the off-box places (ep0, DooPlex, the off-site box, the hub database) were not measured, and the +Prometheus rules in `homelab-manifests` were not read. That is the second pass. + +## For a later session (small fixes, not done today — no code change in this run) + +Keep only the newest N files in the agent's self-update folder · watch the guest root disk and de-duplicate +fill-watch targets by filesystem (`cmd/controller/main.go:2694-2707` has a stale comment) · pin the `alpine` +helper · a stale document, on a protected machine's procedure, so not edited here: `runbooks/ep0-datastore-copy.md:85` +says "NOT INSTALLED" (closed row R-901: installed 2026-10-09). diff --git a/documentation/audits/space-list-2026-11-10/TABLE.md b/documentation/audits/space-list-2026-11-10/TABLE.md new file mode 100644 index 00000000..966abe87 --- /dev/null +++ b/documentation/audits/space-list-2026-11-10/TABLE.md @@ -0,0 +1,729 @@ +# The Space list — every place that holds data, eight questions each + +Measured 2026-11-10, first reading 07:38–07:39 UTC, second reading at the end of the session +(`README.md` gives the rates). Read-only. Sizes are allocated bytes; 1 GB = 10⁹ bytes. +Boxes: **hp** = demo-hp (128 GB system disk), **N100** = demo-felhom (512 GB, lived-in), **T1** = Tester 1 +(a VM, 200 GB). Guest = LXC 9201 on each. + +**The eight questions:** 1 what is in it, who owns it · 2 what makes it grow · 3 the limit · 4 who cleans up, +when · 5 what happens at the limit; is household data safe · 6 who gets an alarm, before or after · 7 does it +repair itself; if not, is there a runbook · 8 how do we know. + +**Marks:** `A` answered, with a pointer · `D` decided, not built · `K` known gap, with its row · +`N` nobody asked — it survived the searches in `EV/searches.txt` (S1…S15; controls in the same file). +Every `N` names its class row: **C1** = R-1368 · **C2** = R-1369 · **C3** = R-1370 · **C4** = R-1371 · +**C5** = R-1372. + +**Pointers:** `AG/` = `felhom-agent/` · `CT/` = `felhom-controller/controller/` · `HUB/` = +`felhom.eu/hub/internal/` · `E/` = `felhom.eu/` · `AR/` = `felhom.eu/documentation/architecture/` · `RB/` = +`felhom.eu/documentation/runbooks/` · `EV/` = this folder's `evidence/`. `tools/check-pointers.py` prints the +text behind each one. Source lines are as of the baselines in `README.md`. + +**Named status** (after the place name): `NAMED` = an architecture document describes the place · +`MENTIONED` = a document names it in passing, with no limit or cleaner · `UNNAMED` = no architecture document +names it. + +**A place inside a bigger place** (a folder on the host system disk, on the guest data volume or on a data +drive) takes question 5 and 6 from the bigger place when it has no limit of its own. Question 7 is `N` for +each place that has no cleaner and no runbook — it is ONE missing runbook seen from many places (C2). + +--- + +## Host (Proxmox) — 18 places + +### H1 — Host system disk, root filesystem `/` (LV `pve/root`) · CUSTOMER · NAMED +`AR/07-backup-architecture.md:467`, `AR/16-storage-physical-identity.md:146-150`. Used: hp 23.97 of 41.42 GB (58 %) · +N100 38.09 of 100.86 GB (38 %) · T1 21.64 of 62.90 GB (34 %). Since 2026-10-15: N100 +4.6 GiB, T1 +5.8 GiB (26 days). + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Proxmox, kernels, the host journal, the whole-box copies when the target is `local`, the agent's files. The operator owns it. | A | `EV/reading-1/hp-host.tsv` | +| 2 | H3, H4, H5, H7, H9, H12 below. They hold 72 % (hp), 82 % (N100), 80 % (T1) of the used bytes. | A | `EV/reading-1/n100-host.tsv` | +| 3 | The LV size the Proxmox installer made. No budget says which part may use how much of it. | N | C3 · S5 | +| 4 | Each part by its own rule, or nobody (H5, H9, H12, H13). The whole disk: the operator, by hand, last on 2026-11-07. | K | R-548 | +| 5 | Under 500 MB free the OS update is refused; a local whole-box copy that cannot fit is skipped; LVM's own metadata writes can fail. Household data is not on this disk. | A | `AG/configs/felhom-os-apply:1390-1392`, `AG/internal/backup/runner.go:276-287`, `AG/internal/vzpartial/vzpartial.go:1-12` | +| 6 | The hub: `host_disk_warning` at 90 %, `host_disk_critical` at 95 % — before it is full. To the operator; the critical one also to the household by default. No architecture document names this alarm or its reader (C4, S4; the alarm ladder names only the thin-pool one). | A | `HUB/monitor/host_disk.go:41-42`, `HUB/claim/engine.go:229-243` | +| 7 | No self-repair. No runbook for a full host system disk. | N | C2 · S6 | +| 8 | TEST and MEASURED. | A | `HUB/monitor/host_disk_test.go:40`, `EV/reading-1/t1-host.tsv` | + +### H2 — Thin pool `pve/data` (data and metadata) · CUSTOMER · NAMED +`AR/08-alarm-ladder.md:298-306`, `AR/16-storage-physical-identity.md:126-150`. Data used: hp 40.88 of 57.84 GB (70.67 %) · +N100 3.03 % · T1 13.10 %. Metadata: 2.85 % · 0.58 % · 1.84 %. Promised to the guest: hp 102 GiB on a 53.9 GiB pool · +N100 282 on 349 · T1 102 on 113. hp was 50.75 % on 2026-10-15. + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The guest's root disk and data volume; for a short time the restore-test's scratch guest. | A | `EV/raw/hp-host.txt` | +| 2 | Each block the guest writes. A deleted block comes back only at the weekly trim. | A | `AG/internal/fstrim/fstrim.go:1-12` | +| 3 | The pool size. Auto-extend is off. The sizing rule promises nothing beyond pool − reserve, but on a 128 GB disk the two guest disks alone (32 + 50 or 70 GiB) are larger than the 54 GiB pool. This is written down. | A | `AR/16-storage-physical-identity.md:146-150`, `E/documentation/audits/storage-identity-buildB-2026-10-15/SPACE-CHECK.md:41-42`, `EV/os-cleaners/hp-host.txt` | +| 4 | The agent trims the guest each week (decision 139). A scratch guest that stays is removed by a 10-minute retry. | A | `AG/internal/fstrim/fstrim.go:1-12`, `AG/internal/reconcile/restoretest_retry.go:5-12` | +| 5 | At 100 % the guest's disks go read-only and every app stops. Measured on demo-hp on 2026-09-24. Data already written stays. | A | `AR/03-host-agent.md:513-520` | +| 6 | The hub: `storage_fill_warning` at 85 %, `storage_fill_critical` at 90 %, the worse of data and metadata. Operator; the critical one also to the household by default. On hp the guest's own warning (85 % of 70 GiB) lies above the pool's real size, so this is the only alarm that can come first. | A | `AR/08-alarm-ladder.md:300-307`, `HUB/monitor/storage_fill.go:36-57`, `HUB/claim/engine.go:229-243` | +| 7 | No self-repair. The one pool runbook covers a different case (an orphan snapshot volume). No runbook for a full pool. | N | C2 · S6 · `RB/whole-guest-backup-orphan-thin.md:1-10` | +| 8 | TEST and MEASURED. | A | `HUB/monitor/storage_fill_test.go:34`, `EV/reading-1/hp-host.tsv` | + +### H3 — Whole-box copies on the current local target · CUSTOMER · NAMED +`AR/07-backup-architecture.md:401`. hp: `/var/lib/vz/dump`, 1 archive, 9.10 GB (keep 1) · N100: `/mnt/hdd_1/dump` +(the USB disk), 3 archives, 4.36 GB (keep 3), 2026-11-08…11-10 · T1: `/var/lib/vz/dump`, 3 archives, 9.46 GB (keep 3). + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | One packed copy of the whole guest per night. The agent makes it; Proxmox stores it. | A | `AR/07-backup-architecture.md:401` | +| 2 | The guest's size. Most of it is Docker images that could be pulled again. | K | R-1366 | +| 3 | Keep the newest N (default 3). Proxmox removes the older ones after a good run. | A | `AG/internal/config/config.go:540-550`, `AG/internal/proxmox/mutate.go:99-101` | +| 4 | Proxmox, after each good run. A half-written copy: the agent, at start and before each backup. | A | `AG/internal/backup/runner.go:227-230`, `AG/cmd/felhom-agent/main.go:2227-2238` | +| 5 | A copy that cannot fit is skipped before it starts. On a small system disk the local copy then stops for ever. | K | R-548 | +| 6 | The skipped run is reported to the hub and shown on the box's backup page with its reason. | A | `AG/internal/hub/report.go:620`, `CT/internal/web/backup_handlers.go:149-152` | +| 7 | No self-repair when the disk is too small. | K | R-548 | +| 8 | TEST. | A | `AG/internal/backup/r685_space_test.go:31`, `AG/internal/backup/prune_test.go:24` | + +### H4 — Whole-box copies on a former target · CUSTOMER · UNNAMED +N100 `/var/lib/vz/dump`: 3 copies of 2026-07-28, 17.59 GB (the target moved to the USB disk) · hp `/mnt/hdd_1/dump`: +4 copies of 2026-08-03…08-21, 5.10 GB · T1: none. + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Old whole-box copies on the place that was the target before. | A | `EV/raw/n100-host.txt` | +| 2 | Only a move of the target. After that they stay. | A | `EV/raw/hp-host.txt` | +| 3 | None. Retention looks only at the current target. | K | R-1087 | +| 4 | Nobody. | K | R-1087 | +| 5 | On a small system disk they leave no room for the new copies. | K | R-1087 | +| 6 | The host-disk alarm of H1, when they lie on the system disk. | A | `HUB/monitor/host_disk.go:41-42` | +| 7 | No. | K | R-1087 | +| 8 | MEASURED. Second instance today: demo-hp. | A | `EV/reading-1/hp-host.tsv` | + +### H5 — Guest template archives ("golden") in the dump folder · CUSTOMER · UNNAMED +`/var/lib/vz/dump/vzdump-lxc-9100-*.tar.zst`. hp 1 (0.65 GB, 2026-09-13) · N100 4 (2.56 GB, 2026-07-18…08-03) · +T1 1 (0.65 GB, 2026-10-04). + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The archive the installer downloads to make the guest. Product-owned. | A | `E/scripts/felhom-host-install.sh:3560-3565` | +| 2 | One more at each install or re-install of the box. | A | `EV/raw/n100-host.txt` | +| 3 | None. | N | C1 · S3 | +| 4 | Nobody. Only the uninstall with `--remove-golden` deletes it. | N | C1 · S3 · `E/scripts/felhom-host-install.sh:1520-1524` | +| 5 | It counts against H1. | A | `AG/configs/felhom-os-apply:1390-1392` | +| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` | +| 7 | No self-repair, no runbook. | N | C2 · S6 | +| 8 | MEASURED. | A | `EV/reading-1/n100-host.tsv` | + +### H6 — Backup logs and half-written copies in the dump folder · CUSTOMER · MENTIONED +`AR/03-host-agent.md:518`. Logs: hp 10 (9 of them from failed runs of 09-24 and 10-14, under 2 KB each) · N100 6 · T1 3. +Half-written copies today: none. + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | One text log per backup run; after a power cut a half-written archive. | A | `EV/raw/hp-host.txt` | +| 2 | One log per run. A half-written copy was 12.2 GB on demo-hp on 2026-10-14. | A | `AG/internal/vzpartial/vzpartial.go:1-12` | +| 3 | The log of a failed run has no limit: it has no archive, so nothing removes it. Each is under 2 KB. | N | C1 · S15 | +| 4 | Proxmox removes a good run's log with its archive. The agent removes a half-written copy at start and before each backup. | A | `AG/cmd/felhom-agent/main.go:2227-2238` | +| 5 | A half-written copy made each later backup skip, until the agent learned to remove it. | A | `AG/internal/vzpartial/vzpartial.go:1-12` | +| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` | +| 7 | Yes, for the half-written copy. | A | `AG/cmd/felhom-agent/main.go:2227-2238` | +| 8 | SOURCE and MEASURED. | A | `EV/reading-1/hp-host.tsv` | + +### H7 — Host journal `/var/log/journal` · CUSTOMER · MENTIONED +`AR/03-host-agent.md:72`. hp 0.87 GB, 16 files, oldest 2026-10-04 (cleaned by hand on 11-07) · N100 4.28 GB, 74 files, +oldest 10-04 · T1 4.29 GB, 81 files, oldest 10-13. Lines in one hour: 13,946 / 13,918 / 11,558; lines written by `sudo` for +the agent's calls are 92 % / 92 % / 90 % of them (R-1367's 61 % counts the session pair only; `sudo` writes a third +line, the command, for each call). + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The host's system log. Nobody set it up; it is the Debian default. | A | `EV/os-cleaners/hp-host.txt` | +| 2 | The agent's `sudo` calls: three lines per call, about 12,800 lines an hour. | K | R-1367 | +| 3 | journald's default: 10 % of the disk, at most 4 GB. Two boxes sit at that cap. | K | R-1367 | +| 4 | journald deletes its oldest file when it reaches the cap. | A | `EV/reading-1/n100-host.tsv` | +| 5 | On a small system disk the journal alone takes a tenth of it. | K | R-1367 | +| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` | +| 7 | It stays at its cap by itself. The cap is not ours. | K | R-1367 | +| 8 | MEASURED, two readings. | A | `EV/reading-1/t1-host.tsv` | + +### H8 — Other host logs, Proxmox task logs, graph data · CUSTOMER · MENTIONED +`AR/03-host-agent.md:570`. `/var/log` without the journal: 14.6 / 14.0 / 14.6 MB; task logs 914 / 706 / 877 files; +`/var/lib/rrdcached` 65 / 21 / 5 MB. + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Text logs, one small log per Proxmox task, fixed-size graph files. | A | `EV/reading-1/hp-host.tsv` | +| 2 | Each task the agent starts makes one log. | A | `EV/raw/hp-host.txt` | +| 3 | Rotation: `logrotate`; Proxmox keeps its task index short (oldest task file 2026-10-28). | A | `EV/os-cleaners/hp-host.txt` | +| 4 | `logrotate.timer` and Proxmox itself. | A | `EV/os-cleaners/n100-host.txt` | +| 5 | Small and bounded; it counts against H1. | A | `EV/reading-1/n100-host.tsv` | +| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` | +| 7 | Yes, by rotation. | A | `EV/os-cleaners/t1-host.txt` | +| 8 | MEASURED. First written down today. | A | `EV/reading-1/t1-host.tsv` | + +### H9 — Old kernels: `/lib/modules` and `/boot` · CUSTOMER · MENTIONED +`AR/11-os-updates.md:402`. hp 8 kernels, 8.15 GB + `/boot` 0.82 GB · N100 8, 8.15 GB + 0.94 GB · T1 3, 3.05 GB + 0.33 GB. +`/boot` is a folder on the system disk. The ring-0 boxes installed 7 kernels in 37 days; no kernel was ever removed. + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Each kernel the box ever installed: about 1.02 GB of modules and 80 MB in `/boot`. | A | `EV/raw/hp-host-followup.txt` | +| 2 | Each kernel set the operator approves; ring 0 takes each one first. | A | `AR/11-os-updates.md:164-176` | +| 3 | None. No rule says how many kernels a box keeps. | N | C1 · S1 | +| 4 | Nobody. apt would remove old ones only on `autoremove`, which nothing runs; the update wrapper refuses each plan that removes a package. | N | C1 · S1 · `AG/configs/felhom-os-apply:1712`, `EV/os-cleaners/hp-host.txt` | +| 5 | Under 500 MB free the wrapper refuses the OS update, so security updates stop. After 7 days the hub says the updates are stale. | A | `AG/configs/felhom-os-apply:1390-1392`, `AR/08-alarm-ladder.md:375` | +| 6 | The host-disk alarm of H1; later `os_update_stale` to the operator. | A | `HUB/monitor/host_disk.go:41-42`, `HUB/notify/dispatcher.go:840` | +| 7 | No self-repair, no runbook. | N | C2 · S1 | +| 8 | NOBODY KNOWS how many kernels a 32 GB system disk holds before updates stop. The sizes are MEASURED. | N | C1 · `EV/reading-1/hp-host.tsv` | + +### H10 — EFI partition `/boot/efi` · CUSTOMER · MENTIONED +`AR/11-os-updates.md:532-536`. 9.5 MB of 1.07 GB on each box (1 %). + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The boot loader, and the one-shot flag for a new kernel. | A | `AR/11-os-updates.md:532-536` | +| 2 | A boot-loader update. Kernels are not copied here (GRUB boxes). | A | `AR/11-os-updates.md:402` | +| 3 | The partition: 1.07 GB. | A | `EV/reading-1/hp-host.tsv` | +| 4 | Nothing needs to: 1 % used. | A | `EV/reading-1/n100-host.tsv` | +| 5 | Not written anywhere. | N | C2 · S13 | +| 6 | Nobody: it is its own filesystem and nothing watches it. | N | C4 · S13 | +| 7 | No runbook. | N | C2 · S13 | +| 8 | MEASURED. | A | `EV/reading-1/t1-host.tsv` | + +### H11 — Host package cache and package lists · CUSTOMER · NAMED +`AR/11-os-updates.md:293`. `/var/cache/apt` 92 MB with 0 package files on each box; lists 62 MB. + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | Downloaded packages during an OS update, and the package index. | A | `EV/reading-1/hp-host.tsv` | +| 2 | Each OS update run. | A | `AR/11-os-updates.md:293` | +| 3 | Emptied at the end of each run. | A | `AG/configs/felhom-os-apply:1417` | +| 4 | The update wrapper (`apt-get clean`). | A | `AG/configs/felhom-os-apply:1417` | +| 5 | A run that would not fit is refused before it starts. | A | `AG/configs/felhom-os-apply:1390-1392` | +| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` | +| 7 | Yes: cleaned each run. | A | `AG/configs/felhom-os-apply:1417` | +| 8 | SOURCE and MEASURED (0 package files). | A | `EV/reading-1/t1-host.tsv` | + +### H12 — The agent's self-update folder `/var/lib/felhom-agent/selfupdate` · CUSTOMER · UNNAMED +hp 739.9 MB, 49 binaries · N100 723.9 MB, 48 · T1 509.2 MB, 33. About 15 MB for each agent version the box ever got. + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | One downloaded agent binary per version. The agent owns it. | A | `AG/internal/selfupdate/executor.go:106-128` | +| 2 | Each agent update. | A | `EV/raw/hp-host-followup.txt` | +| 3 | None. | N | C1 · S2 | +| 4 | Nobody. | N | C1 · S2 | +| 5 | It counts against H1. | A | `AG/configs/felhom-os-apply:1390-1392` | +| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` | +| 7 | No self-repair, no runbook. | N | C2 · S2 | +| 8 | MEASURED. | A | `EV/reading-1/hp-host.tsv` | + +### H13 — Old copies of the agent binary (`.prev`, `.bak-*`, `.pre-bundle`) · CUSTOMER · MENTIONED +`AR/03-host-agent.md:893`. `/usr/local/bin/felhom-agent.bak-*`: hp 1 file, 14 MB · N100 11 files, 155 MB · T1 none. + +| # | Answer | Mark | Pointer | +|---|---|---|---| +| 1 | The previous agent binary (`.prev`), and one copy for each re-run of the installer (`.bak-