# RUNBOOK — a disk on a box is full, or nearly full (R-1369) **When:** a space alarm came (`host_disk_*`, `storage_fill_*`, the controller's fill warning), a whole-box copy was skipped for space, or an update was refused for space. **What this is:** one short part for each disk. Each part says how to see what is big, what is safe to remove and by which path, what a hand must never remove, and what the product does by itself. **The design is not here**: each place with its limit, cleaner and alarm is `architecture/16-storage-physical-identity.md` §14. The place ids (H1, G2, D6 …) are the same there and in `audits/space-list-2026-11-10/TABLE.md`. **Each hand step carries a mark.** `PROVEN` = a person or CC did exactly this on a box, with the date and the evidence. `NOT PROVEN` = nobody did it; it is not a step yet. **Every `PROVEN` step below was proven on demo-hp, a test box.** None was done on a household's box. **Before any hand step, on any disk:** 1. Read which box it is and what you may do there: `runbooks/target-selection.md`. 2. Check that no backup, restore, restore test or update runs: `pct list` shows no lock; `ps -ef | grep -E 'vzdump|pct restore|apt'` shows nothing. 3. Save what you see before you remove anything (R-320): the `df`, the list, the file's checksum. 4. On a household's box a hand removes nothing without the operator's word. --- ## 1. Host system disk (`/`) **See what is big (read-only):** ```bash df -h / du -xsh /var/lib/vz/dump /var/lib/vz/template /var/log/journal /lib/modules /boot /var/lib/felhom-agent/selfupdate journalctl --disk-usage pvesm list local dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii ``` **The product does by itself:** it empties the package cache in each OS update. It removes old agent binaries from the self-update folder after each good agent update. It keeps the newest N whole-box copies (default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The hub alarms at 90 % and 95 %. **Safe to remove, by which path:** | What | Step | Mark | |---|---|---| | Old kernels | `ssh python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). The kept older kernel then booted the box to healthy and the box went back (`audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **NOT PROVEN:** on a second box | | The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) | | An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free `, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only | | Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) | | Old binaries in the agent's self-update folder (H12) | No hand step. The agent removes them at the end of each good update, and keeps the running version's file | **PROVEN 2026-11-10** as a product path on three test boxes: 49 → 1, 48 → 1 and 33 → 1 files; a rollback worked after it (`audits/release-space-fixes-2026-11-10/agent/`). **NOT PROVEN:** a removal by hand | | Extra guest template archives (H5); `/usr/local/bin/felhom-agent.bak-*` (H13) | — | **NOT PROVEN** (R-1368) | **Never by hand:** - The running kernel and the one before it. Never a kernel with a plain `apt autoremove` or by a meta-package: `apt` did not refuse the running kernel, and one meta-package pulled `proxmox-ve` with it (`audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt`). - `pending.json` in the self-update folder, and `/usr/local/bin/felhom-agent.prev`. The agent's rollback needs both. - The newest whole-box copy. - Anything under `/var/lib/felhom-os-apply/`: the wrapper's state and the last config bundles. ## 2. Thin pool (`pve/data`) **See what is big (read-only):** ```bash lvs -o lv_name,lv_size,data_percent,metadata_percent pve pct list ``` **The product does by itself:** the agent trims each guest once a week (decision 139). It removes a restore test's scratch guest at the end and tries again every 10 minutes when that fails. A restore test starts only with enough free space. The hub alarms at 85 % and 90 % on the worse of data and metadata. **At 100 %** the guest's disks go read-only. A write in progress can be cut off: on 2026-09-24 two apps' Redis files were (`architecture/03-host-agent.md` §8, the R-672 note; `audits/r672-2026-09-24/README.md`). | What | Step | Mark | |---|---|---| | Space the guest freed but the pool still holds | `pct fstrim ` | **PROVEN 2026-10-06**: pool 65.5 % → 33.4 % in 24 s, the box answered all the time (`audits/ten-answers-2026-10-06/r444-measure.txt`). The product does this weekly | | A guest that is read-only after a full pool | Free pool space first. Stop the guest. `pct fsck `, then `pct fsck --device mp0`. Start it. Check each app | **PROVEN once, 2026-09-24** (`audits/r672-2026-09-24/README.md`, Part B; `B4-fsck.txt`). Two Redis files then needed `redis-check-aof --fix`, with the operator's yes and a copy kept first | | A left-over scratch guest or a thin device with no volume | `runbooks/whole-guest-backup-orphan-thin.md` | see that runbook | **Never by hand:** `lvremove` of a `vm-9201-disk-*` volume. `lvextend` of the pool into the space that belongs to the internal place. A thin device, except by the runbook above. ## 3. Guest root disk (32 GiB) **See what is big (read-only):** ```bash pct exec -- df -h / pct exec -- du -xsh /var /opt /usr /root ``` **The product does by itself:** it empties the guest's package cache in each guest OS update. Nothing else. **Nobody watches this disk** (R-1371). It was 4–5 % used on three boxes on 2026-11-10. **Safe to remove:** nothing is proven. **NOT PROVEN** for every hand step on this disk. When it is full, the agent can grow it; that path is the product's, not a hand's. **Never by hand:** `/opt/docker/stacks//app.yaml`. It holds the app's settings and its records. ## 4. Guest data volume (`/var/lib/felhom`) **See what is big (read-only):** ```bash pct exec -- df -h /var/lib/felhom pct exec -- docker system df pct exec -- docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}' ``` Never dump a container's `Env`. **The product does by itself:** it keeps the current image and one undo image for each app, and removes older ones. It keeps the running controller image and the one before; the same for infrastructure images. Docker caps each container's log at 10 MB × 3. It deletes graph rows older than 30 days. It refuses a new app under max(5 GB, 10 %) free. It refuses a backup at 97 % or under 1 GiB, app by app, and deletes nothing. It warns the household at 85 % and 95 %. **Safe to remove, by which path:** | What | Step | Mark | |---|---|---| | An app the household does not want | The household removes it on the app's page. That is the only path that frees an app's volume | product path | | Old controller, infrastructure and helper images that no container uses | `docker image rm `, one exact name at a time, after `docker ps -a` shows no container on it | **PROVEN 2026-11-09**: 36 images, 3.4 GB (off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/cleanup.txt`). The controller now removes old controller and infrastructure images by itself. Helper images stay (R-1372) | | Volumes with no name (G8) | — | **NOT PROVEN** (R-995, R-1267) | **Never by hand:** `docker system prune`, `docker volume prune`, `docker image prune -a`. The undo images are kept on purpose (decision 53), and volumes hold the apps' data. An app's undo image. A named volume. ## 5. A data drive **See what is big:** sizes of the top folders only. Do not list file names below a household's data. ```bash pct exec -- df -h pct exec -- du -xsh /* /backups/* ``` **The product does by itself:** the nightly backup replaces each app's one point. The second-drive copy needs max(2 GB, 20 %) free. The controller deletes a restore's undo copy after 30 days without use, and tells the household. The off-site proof cleans its own scratch. The controller warns the household at 85 % and 95 %. **Safe to remove:** only what the household removes on the product's pages: the kept data of a removed app (with the app's name typed), a set-aside unit, an off-site restore, its own files. **A hand step on a data drive: NOT PROVEN, and not wanted.** **Never by hand:** `userdata`, `appdata`, `kept`, `shares`, `backups/primary`, `backups/set-aside`, `backups/restore-undo`. These grow without a limit by ruling (`16` §14.2; decisions 40, 67, 256, 281). Only the household deletes them. --- **After any hand step:** read `df` again, save the before and after to the evidence folder, and check the box on the hub. Then write a dated line on the row that names the place.