The close-out of the attended release run of 2026-11-10 (afternoon). - Evidence: documentation/audits/release-space-fixes-2026-11-10/ (agent, controller, hub) and the hub row after the fallback-kernel test. - Rows: dated lines on R-1368, R-1371, R-1301, R-1353. 236 open before and after; opened 0, closed 0. - Capability map: dated notes under C13 and G5; no status changed. - disk-full runbook and 16 §14: the self-update folder is PROVEN as a product path on three test boxes. - STATUS: new "Updated" paragraph; the old one moved to STATUS-HISTORY. - REPORT-release-space-fixes-2026-11-10.md: wrong claims first, one deviation (the hub deploy ran before CI for the manifest-bump commit was read), what stays not proven. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
9.6 KiB
RUNBOOK — a disk on a box is full, or nearly full (R-1369)
When: a space alarm came (host_disk_*, storage_fill_*, the controller's fill warning), a whole-box copy was
skipped for space, or an update was refused for space.
What this is: one short part for each disk. Each part says how to see what is big, what is safe to remove and by
which path, what a hand must never remove, and what the product does by itself. The design is not here: each place
with its limit, cleaner and alarm is architecture/16-storage-physical-identity.md §14. The place ids (H1, G2, D6 …)
are the same there and in audits/space-list-2026-11-10/TABLE.md.
Each hand step carries a mark. PROVEN = a person or CC did exactly this on a box, with the date and the
evidence. NOT PROVEN = nobody did it; it is not a step yet. Every PROVEN step below was proven on demo-hp, a test
box. None was done on a household's box.
Before any hand step, on any disk:
- Read which box it is and what you may do there:
runbooks/target-selection.md. - Check that no backup, restore, restore test or update runs:
pct listshows no lock;ps -ef | grep -E 'vzdump|pct restore|apt'shows nothing. - Save what you see before you remove anything (R-320): the
df, the list, the file's checksum. - On a household's box a hand removes nothing without the operator's word.
1. Host system disk (/)
See what is big (read-only):
df -h /
du -xsh /var/lib/vz/dump /var/lib/vz/template /var/log/journal /lib/modules /boot /var/lib/felhom-agent/selfupdate
journalctl --disk-usage
pvesm list local
dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii
The product does by itself: it empties the package cache in each OS update. It removes old agent binaries from the self-update folder after each good agent update. It keeps the newest N whole-box copies (default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The hub alarms at 90 % and 95 %.
Safe to remove, by which path:
| What | Step | Mark |
|---|---|---|
| Old kernels | ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window |
PROVEN 2026-11-10: 8 → 2 kernels, 6.4 GB, 50 s, one restart (audits/kernel-cleanup-spike-2026-11-10/README.md). The kept older kernel then booted the box to healthy and the box went back (audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/). NOT PROVEN: on a second box |
| The host journal | Save it first, then journalctl --vacuum-size=500M |
PROVEN 2026-11-07: 3.8 GB → 468 MB (audits/demo-check-2026-11-07/README.md; the record is off-git, /mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt). It grows back (R-1367) |
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | pvesm free <volid>, after pct list and qm list show that no guest uses it, and after its checksum matches a copy elsewhere |
PROVEN 2026-10-23 (three ISOs, audits/day-2026-10-23/09-demohp-space.txt), 2026-11-07 (one archive of a removed guest), 2026-11-09 (one ISO). Test boxes only |
| Whole-box copies on a former target (H4) | — | NOT PROVEN. Needs the operator's word (R-1087) |
| Old binaries in the agent's self-update folder (H12) | No hand step. The agent removes them at the end of each good update, and keeps the running version's file | PROVEN 2026-11-10 as a product path on three test boxes: 49 → 1, 48 → 1 and 33 → 1 files; a rollback worked after it (audits/release-space-fixes-2026-11-10/agent/). NOT PROVEN: a removal by hand |
Extra guest template archives (H5); /usr/local/bin/felhom-agent.bak-* (H13) |
— | NOT PROVEN (R-1368) |
Never by hand:
- The running kernel and the one before it. Never a kernel with a plain
apt autoremoveor by a meta-package:aptdid not refuse the running kernel, and one meta-package pulledproxmox-vewith it (audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt). pending.jsonin the self-update folder, and/usr/local/bin/felhom-agent.prev. The agent's rollback needs both.- The newest whole-box copy.
- Anything under
/var/lib/felhom-os-apply/: the wrapper's state and the last config bundles.
2. Thin pool (pve/data)
See what is big (read-only):
lvs -o lv_name,lv_size,data_percent,metadata_percent pve
pct list
The product does by itself: the agent trims each guest once a week (decision 139). It removes a restore test's scratch guest at the end and tries again every 10 minutes when that fails. A restore test starts only with enough free space. The hub alarms at 85 % and 90 % on the worse of data and metadata.
At 100 % the guest's disks go read-only. A write in progress can be cut off: on 2026-09-24 two apps' Redis files
were (architecture/03-host-agent.md §8, the R-672 note; audits/r672-2026-09-24/README.md).
| What | Step | Mark |
|---|---|---|
| Space the guest freed but the pool still holds | pct fstrim <vmid> |
PROVEN 2026-10-06: pool 65.5 % → 33.4 % in 24 s, the box answered all the time (audits/ten-answers-2026-10-06/r444-measure.txt). The product does this weekly |
| A guest that is read-only after a full pool | Free pool space first. Stop the guest. pct fsck <vmid>, then pct fsck <vmid> --device mp0. Start it. Check each app |
PROVEN once, 2026-09-24 (audits/r672-2026-09-24/README.md, Part B; B4-fsck.txt). Two Redis files then needed redis-check-aof --fix, with the operator's yes and a copy kept first |
| A left-over scratch guest or a thin device with no volume | runbooks/whole-guest-backup-orphan-thin.md |
see that runbook |
Never by hand: lvremove of a vm-9201-disk-* volume. lvextend of the pool into the space that belongs to the
internal place. A thin device, except by the runbook above.
3. Guest root disk (32 GiB)
See what is big (read-only):
pct exec <vmid> -- df -h /
pct exec <vmid> -- du -xsh /var /opt /usr /root
The product does by itself: it empties the guest's package cache in each guest OS update. Nothing else. Nobody watches this disk (R-1371). It was 4–5 % used on three boxes on 2026-11-10.
Safe to remove: nothing is proven. NOT PROVEN for every hand step on this disk. When it is full, the agent can grow it; that path is the product's, not a hand's.
Never by hand: /opt/docker/stacks/<app>/app.yaml. It holds the app's settings and its records.
4. Guest data volume (/var/lib/felhom)
See what is big (read-only):
pct exec <vmid> -- df -h /var/lib/felhom
pct exec <vmid> -- docker system df
pct exec <vmid> -- docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}'
Never dump a container's Env.
The product does by itself: it keeps the current image and one undo image for each app, and removes older ones. It keeps the running controller image and the one before; the same for infrastructure images. Docker caps each container's log at 10 MB × 3. It deletes graph rows older than 30 days. It refuses a new app under max(5 GB, 10 %) free. It refuses a backup at 97 % or under 1 GiB, app by app, and deletes nothing. It warns the household at 85 % and 95 %.
Safe to remove, by which path:
| What | Step | Mark |
|---|---|---|
| An app the household does not want | The household removes it on the app's page. That is the only path that frees an app's volume | product path |
| Old controller, infrastructure and helper images that no container uses | docker image rm <name:tag>, one exact name at a time, after docker ps -a shows no container on it |
PROVEN 2026-11-09: 36 images, 3.4 GB (off-git, /mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/cleanup.txt). The controller now removes old controller and infrastructure images by itself. Helper images stay (R-1372) |
| Volumes with no name (G8) | — | NOT PROVEN (R-995, R-1267) |
Never by hand: docker system prune, docker volume prune, docker image prune -a. The undo images are kept on
purpose (decision 53), and volumes hold the apps' data. An app's undo image. A named volume.
5. A data drive
See what is big: sizes of the top folders only. Do not list file names below a household's data.
pct exec <vmid> -- df -h <mount point>
pct exec <vmid> -- du -xsh <mount point>/* <mount point>/backups/*
The product does by itself: the nightly backup replaces each app's one point. The second-drive copy needs max(2 GB, 20 %) free. The controller deletes a restore's undo copy after 30 days without use, and tells the household. The off-site proof cleans its own scratch. The controller warns the household at 85 % and 95 %.
Safe to remove: only what the household removes on the product's pages: the kept data of a removed app (with the app's name typed), a set-aside unit, an off-site restore, its own files. A hand step on a data drive: NOT PROVEN, and not wanted.
Never by hand: userdata, appdata, kept, shares, backups/primary, backups/set-aside,
backups/restore-undo. These grow without a limit by ruling (16 §14.2; decisions 40, 67, 256, 281). Only the
household deletes them.
After any hand step: read df again, save the before and after to the evidence folder, and check the box on the
hub. Then write a dated line on the row that names the place.