Files
felhom.eu/documentation/runbooks/disk-full.md
T

10 KiB
Raw Blame History

RUNBOOK — a disk on a box is full, or nearly full (R-1369)

When: a space alarm came (host_disk_*, storage_fill_*, the controller's fill warning), a whole-box copy was skipped for space, or an update was refused for space.

What this is: one short part for each disk. Each part says how to see what is big, what is safe to remove and by which path, what a hand must never remove, and what the product does by itself. The design is not here: each place with its limit, cleaner and alarm is architecture/16-storage-physical-identity.md §14. The place ids (H1, G2, D6 …) are the same there and in audits/space-list-2026-11-10/TABLE.md.

Each hand step carries a mark. PROVEN = a person or CC did exactly this on a box, with the date and the evidence. NOT PROVEN = nobody did it; it is not a step yet. Every PROVEN step below was proven on demo-hp, a test box. None was done on a household's box.

Before any hand step, on any disk:

  1. Read which box it is and what you may do there: runbooks/target-selection.md.
  2. Check that no backup, restore, restore test or update runs: pct list shows no lock; ps -ef | grep -E 'vzdump|pct restore|apt' shows nothing.
  3. Save what you see before you remove anything (R-320): the df, the list, the file's checksum.
  4. On a household's box a hand removes nothing without the operator's word.

1. Host system disk (/)

See what is big (read-only):

df -h /
du -xsh /var/lib/vz/dump /var/lib/vz/template /var/log/journal /lib/modules /boot /var/lib/felhom-agent/selfupdate
journalctl --disk-usage
pvesm list local
dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii

The product does by itself: it empties the package cache in each OS update. It removes old agent binaries from the self-update folder after each good agent update. It keeps the newest N whole-box copies (default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The hub alarms at 90 % and 95 %.

Safe to remove, by which path:

What Step Mark
Old kernels A ring-0 box removes them by itself at night since agent v0.174.0 (11-os-updates.md §5.11.1). First read what it would do: sudo -u felhom-agent felhom-agent --config /etc/felhom-agent/agent.json --selftest=kernel-cleanup-dry-run (it changes nothing; „would": remove / nothing / refuse / not_run, with the reason). The hand step below is for a box where the program does not run (a ring-1 box, a box that fell back) and for an urgent case: ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window PROVEN 2026-11-10: 8 → 2 kernels, 6.4 GB, 50 s, one restart (audits/kernel-cleanup-spike-2026-11-10/README.md). The kept older kernel then booted the box to healthy and the box went back (audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/). NOT PROVEN: on a second box
The host journal Save it first, then journalctl --vacuum-size=500M PROVEN 2026-11-07: 3.8 GB → 468 MB (audits/demo-check-2026-11-07/README.md; the record is off-git, /mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt). It grows back (R-1367)
An installer ISO that a newer one replaced; a test archive of a guest that is gone pvesm free <volid>, after pct list and qm list show that no guest uses it, and after its checksum matches a copy elsewhere PROVEN 2026-10-23 (three ISOs, audits/day-2026-10-23/09-demohp-space.txt), 2026-11-07 (one archive of a removed guest), 2026-11-09 (one ISO). Test boxes only
Whole-box copies on a former target (H4) — NOT PROVEN. Needs the operator's word (R-1087)
Old binaries in the agent's self-update folder (H12) No hand step. The agent removes them at the end of each good update, and keeps the running version's file PROVEN 2026-11-10 as a product path on three test boxes: 49 → 1, 48 → 1 and 33 → 1 files; a rollback worked after it (audits/release-space-fixes-2026-11-10/agent/). NOT PROVEN: a removal by hand
Extra guest template archives (H5); /usr/local/bin/felhom-agent.bak-* (H13) — NOT PROVEN (R-1368)

Never by hand:

  • The running kernel and the one before it. Never a kernel with a plain apt autoremove or by a meta-package: apt did not refuse the running kernel, and one meta-package pulled proxmox-ve with it (audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt).
  • pending.json in the self-update folder, and /usr/local/bin/felhom-agent.prev. The agent's rollback needs both.
  • The newest whole-box copy.
  • Anything under /var/lib/felhom-os-apply/: the wrapper's state and the last config bundles.

2. Thin pool (pve/data)

See what is big (read-only):

lvs -o lv_name,lv_size,data_percent,metadata_percent pve
pct list

The product does by itself: the agent trims each guest once a week (decision 139). It removes a restore test's scratch guest at the end and tries again every 10 minutes when that fails. A restore test starts only with enough free space. The hub alarms at 85 % and 90 % on the worse of data and metadata.

At 100 % the guest's disks go read-only. A write in progress can be cut off: on 2026-09-24 two apps' Redis files were (architecture/03-host-agent.md §8, the R-672 note; audits/r672-2026-09-24/README.md).

What Step Mark
Space the guest freed but the pool still holds pct fstrim <vmid> PROVEN 2026-10-06: pool 65.5 % → 33.4 % in 24 s, the box answered all the time (audits/ten-answers-2026-10-06/r444-measure.txt). The product does this weekly
A guest that is read-only after a full pool Free pool space first. Stop the guest. pct fsck <vmid>, then pct fsck <vmid> --device mp0. Start it. Check each app PROVEN once, 2026-09-24 (audits/r672-2026-09-24/README.md, Part B; B4-fsck.txt). Two Redis files then needed redis-check-aof --fix, with the operator's yes and a copy kept first
A left-over scratch guest or a thin device with no volume runbooks/whole-guest-backup-orphan-thin.md see that runbook

Never by hand: lvremove of a vm-9201-disk-* volume. lvextend of the pool into the space that belongs to the internal place. A thin device, except by the runbook above.

3. Guest root disk (32 GiB)

See what is big (read-only):

pct exec <vmid> -- df -h /
pct exec <vmid> -- du -xsh /var /opt /usr /root

The product does by itself: it empties the guest's package cache in each guest OS update. Nothing else. Nobody watches this disk (R-1371). It was 4–5 % used on three boxes on 2026-11-10.

Safe to remove: nothing is proven. NOT PROVEN for every hand step on this disk. When it is full, the agent can grow it; that path is the product's, not a hand's.

Never by hand: /opt/docker/stacks/<app>/app.yaml. It holds the app's settings and its records.

4. Guest data volume (/var/lib/felhom)

See what is big (read-only):

pct exec <vmid> -- df -h /var/lib/felhom
pct exec <vmid> -- docker system df
pct exec <vmid> -- docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}'

Never dump a container's Env.

The product does by itself: it keeps the current image and one undo image for each app, and removes older ones. It keeps the running controller image and the one before; the same for infrastructure images. Docker caps each container's log at 10 MB × 3. It deletes graph rows older than 30 days. It refuses a new app under max(5 GB, 10 %) free. It refuses a backup at 97 % or under 1 GiB, app by app, and deletes nothing. It warns the household at 85 % and 95 %.

Safe to remove, by which path:

What Step Mark
An app the household does not want The household removes it on the app's page. That is the only path that frees an app's volume product path
Old controller, infrastructure and helper images that no container uses docker image rm <name:tag>, one exact name at a time, after docker ps -a shows no container on it PROVEN 2026-11-09: 36 images, 3.4 GB (off-git, /mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/cleanup.txt). The controller now removes old controller and infrastructure images by itself. Helper images stay (R-1372)
Volumes with no name (G8) — NOT PROVEN (R-995, R-1267)

Never by hand: docker system prune, docker volume prune, docker image prune -a. The undo images are kept on purpose (decision 53), and volumes hold the apps' data. An app's undo image. A named volume.

5. A data drive

See what is big: sizes of the top folders only. Do not list file names below a household's data.

pct exec <vmid> -- df -h <mount point>
pct exec <vmid> -- du -xsh <mount point>/*  <mount point>/backups/*

The product does by itself: the nightly backup replaces each app's one point. The second-drive copy needs max(2 GB, 20 %) free. The controller deletes a restore's undo copy after 30 days without use, and tells the household. The off-site proof cleans its own scratch. The controller warns the household at 85 % and 95 %.

Safe to remove: only what the household removes on the product's pages: the kept data of a removed app (with the app's name typed), a set-aside unit, an off-site restore, its own files. A hand step on a data drive: NOT PROVEN, and not wanted.

Never by hand: userdata, appdata, kept, shares, backups/primary, backups/set-aside, backups/restore-undo. These grow without a limit by ruling (16 §14.2; decisions 40, 67, 256, 281). Only the household deletes them.


After any hand step: read df again, save the before and after to the evidence folder, and check the box on the hub. Then write a dated line on the row that names the place.