Files
felhom.eu/documentation/runbooks/disk-full.md
T
admin 025a1bcd40
gates / gates (push) Successful in 2m7s
docs: the disk-space section (16 §14), the runbook "a disk is full", the run's report
C4: documentation/runbooks/disk-full.md (R-1369). Five disks; each hand step
is PROVEN with its date and evidence, or NOT PROVEN.
C5: 16-storage-physical-identity.md §14 (D69, decision 288). 45 places, each
with limit, cleaner and alarm; the rulings behind "no limit"; the host disk
budget; the open design points. PROMPT-TEMPLATE §N.5 gets the keep-it-true line.
Close-out: STATUS (D75 new), CONTEXT, dated lines on R-1369, R-1370, R-1371,
three small stale-document fixes, the report of the run.

Documents only. No release, no build, no deploy.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 14:47:25 +01:00

157 lines
9.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# RUNBOOK — a disk on a box is full, or nearly full (R-1369)
**When:** a space alarm came (`host_disk_*`, `storage_fill_*`, the controller's fill warning), a whole-box copy was
skipped for space, or an update was refused for space.
**What this is:** one short part for each disk. Each part says how to see what is big, what is safe to remove and by
which path, what a hand must never remove, and what the product does by itself. **The design is not here**: each place
with its limit, cleaner and alarm is `architecture/16-storage-physical-identity.md` §14. The place ids (H1, G2, D6 …)
are the same there and in `audits/space-list-2026-11-10/TABLE.md`.
**Each hand step carries a mark.** `PROVEN` = a person or CC did exactly this on a box, with the date and the
evidence. `NOT PROVEN` = nobody did it; it is not a step yet. **Every `PROVEN` step below was proven on demo-hp, a test
box.** None was done on a household's box.
**Before any hand step, on any disk:**
1. Read which box it is and what you may do there: `runbooks/target-selection.md`.
2. Check that no backup, restore, restore test or update runs: `pct list` shows no lock; `ps -ef | grep -E
'vzdump|pct restore|apt'` shows nothing.
3. Save what you see before you remove anything (R-320): the `df`, the list, the file's checksum.
4. On a household's box a hand removes nothing without the operator's word.
---
## 1. Host system disk (`/`)
**See what is big (read-only):**
```bash
df -h /
du -xsh /var/lib/vz/dump /var/lib/vz/template /var/log/journal /lib/modules /boot /var/lib/felhom-agent/selfupdate
journalctl --disk-usage
pvesm list local
dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii
```
**The product does by itself:** it empties the package cache in each OS update. It keeps the newest N whole-box copies
(default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS
update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The
hub alarms at 90 % and 95 %.
**Safe to remove, by which path:**
| What | Step | Mark |
|---|---|---|
| Old kernels | `ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). **NOT PROVEN:** on a second box; a boot of the kept older kernel |
| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) |
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free <volid>`, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only |
| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |
| Old binaries in the agent's self-update folder (H12) | — | **NOT PROVEN.** The agent cleans the folder after the release that carries the fix (R-1368) |
| Extra guest template archives (H5); `/usr/local/bin/felhom-agent.bak-*` (H13) | — | **NOT PROVEN** (R-1368) |
**Never by hand:**
- The running kernel and the one before it. Never a kernel with a plain `apt autoremove` or by a meta-package: `apt`
did not refuse the running kernel, and one meta-package pulled `proxmox-ve` with it
(`audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt`).
- `pending.json` in the self-update folder, and `/usr/local/bin/felhom-agent.prev`. The agent's rollback needs both.
- The newest whole-box copy.
- Anything under `/var/lib/felhom-os-apply/`: the wrapper's state and the last config bundles.
## 2. Thin pool (`pve/data`)
**See what is big (read-only):**
```bash
lvs -o lv_name,lv_size,data_percent,metadata_percent pve
pct list
```
**The product does by itself:** the agent trims each guest once a week (decision 139). It removes a restore test's
scratch guest at the end and tries again every 10 minutes when that fails. A restore test starts only with enough free
space. The hub alarms at 85 % and 90 % on the worse of data and metadata.
**At 100 %** the guest's disks go read-only. A write in progress can be cut off: on 2026-09-24 two apps' Redis files
were (`architecture/03-host-agent.md` §8, the R-672 note; `audits/r672-2026-09-24/README.md`).
| What | Step | Mark |
|---|---|---|
| Space the guest freed but the pool still holds | `pct fstrim <vmid>` | **PROVEN 2026-10-06**: pool 65.5 % → 33.4 % in 24 s, the box answered all the time (`audits/ten-answers-2026-10-06/r444-measure.txt`). The product does this weekly |
| A guest that is read-only after a full pool | Free pool space first. Stop the guest. `pct fsck <vmid>`, then `pct fsck <vmid> --device mp0`. Start it. Check each app | **PROVEN once, 2026-09-24** (`audits/r672-2026-09-24/README.md`, Part B; `B4-fsck.txt`). Two Redis files then needed `redis-check-aof --fix`, with the operator's yes and a copy kept first |
| A left-over scratch guest or a thin device with no volume | `runbooks/whole-guest-backup-orphan-thin.md` | see that runbook |
**Never by hand:** `lvremove` of a `vm-9201-disk-*` volume. `lvextend` of the pool into the space that belongs to the
internal place. A thin device, except by the runbook above.
## 3. Guest root disk (32 GiB)
**See what is big (read-only):**
```bash
pct exec <vmid> -- df -h /
pct exec <vmid> -- du -xsh /var /opt /usr /root
```
**The product does by itself:** it empties the guest's package cache in each guest OS update. Nothing else. **Nobody
watches this disk** (R-1371). It was 4–5 % used on three boxes on 2026-11-10.
**Safe to remove:** nothing is proven. **NOT PROVEN** for every hand step on this disk. When it is full, the agent can
grow it; that path is the product's, not a hand's.
**Never by hand:** `/opt/docker/stacks/<app>/app.yaml`. It holds the app's settings and its records.
## 4. Guest data volume (`/var/lib/felhom`)
**See what is big (read-only):**
```bash
pct exec <vmid> -- df -h /var/lib/felhom
pct exec <vmid> -- docker system df
pct exec <vmid> -- docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}'
```
Never dump a container's `Env`.
**The product does by itself:** it keeps the current image and one undo image for each app, and removes older ones. It
keeps the running controller image and the one before; the same for infrastructure images. Docker caps each container's
log at 10 MB × 3. It deletes graph rows older than 30 days. It refuses a new app under max(5 GB, 10 %) free. It refuses
a backup at 97 % or under 1 GiB, app by app, and deletes nothing. It warns the household at 85 % and 95 %.
**Safe to remove, by which path:**
| What | Step | Mark |
|---|---|---|
| An app the household does not want | The household removes it on the app's page. That is the only path that frees an app's volume | product path |
| Old controller, infrastructure and helper images that no container uses | `docker image rm <name:tag>`, one exact name at a time, after `docker ps -a` shows no container on it | **PROVEN 2026-11-09**: 36 images, 3.4 GB (off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/cleanup.txt`). The controller now removes old controller and infrastructure images by itself. Helper images stay (R-1372) |
| Volumes with no name (G8) | — | **NOT PROVEN** (R-995, R-1267) |
**Never by hand:** `docker system prune`, `docker volume prune`, `docker image prune -a`. The undo images are kept on
purpose (decision 53), and volumes hold the apps' data. An app's undo image. A named volume.
## 5. A data drive
**See what is big:** sizes of the top folders only. Do not list file names below a household's data.
```bash
pct exec <vmid> -- df -h <mount point>
pct exec <vmid> -- du -xsh <mount point>/* <mount point>/backups/*
```
**The product does by itself:** the nightly backup replaces each app's one point. The second-drive copy needs
max(2 GB, 20 %) free. The controller deletes a restore's undo copy after 30 days without use, and tells the household.
The off-site proof cleans its own scratch. The controller warns the household at 85 % and 95 %.
**Safe to remove:** only what the household removes on the product's pages: the kept data of a removed app (with the
app's name typed), a set-aside unit, an off-site restore, its own files. **A hand step on a data drive: NOT PROVEN, and
not wanted.**
**Never by hand:** `userdata`, `appdata`, `kept`, `shares`, `backups/primary`, `backups/set-aside`,
`backups/restore-undo`. These grow without a limit by ruling (`16` §14.2; decisions 40, 67, 256, 281). Only the
household deletes them.
---
**After any hand step:** read `df` again, save the before and after to the evidence folder, and check the box on the
hub. Then write a dated line on the row that names the place.