158 lines
10 KiB
Markdown
158 lines
10 KiB
Markdown
# RUNBOOK — a disk on a box is full, or nearly full (R-1369)
|
||
|
||
**When:** a space alarm came (`host_disk_*`, `storage_fill_*`, the controller's fill warning), a whole-box copy was
|
||
skipped for space, or an update was refused for space.
|
||
|
||
**What this is:** one short part for each disk. Each part says how to see what is big, what is safe to remove and by
|
||
which path, what a hand must never remove, and what the product does by itself. **The design is not here**: each place
|
||
with its limit, cleaner and alarm is `architecture/16-storage-physical-identity.md` §14. The place ids (H1, G2, D6 …)
|
||
are the same there and in `audits/space-list-2026-11-10/TABLE.md`.
|
||
|
||
**Each hand step carries a mark.** `PROVEN` = a person or CC did exactly this on a box, with the date and the
|
||
evidence. `NOT PROVEN` = nobody did it; it is not a step yet. **Every `PROVEN` step below was proven on demo-hp, a test
|
||
box.** None was done on a household's box.
|
||
|
||
**Before any hand step, on any disk:**
|
||
|
||
1. Read which box it is and what you may do there: `runbooks/target-selection.md`.
|
||
2. Check that no backup, restore, restore test or update runs: `pct list` shows no lock; `ps -ef | grep -E
|
||
'vzdump|pct restore|apt'` shows nothing.
|
||
3. Save what you see before you remove anything (R-320): the `df`, the list, the file's checksum.
|
||
4. On a household's box a hand removes nothing without the operator's word.
|
||
|
||
---
|
||
|
||
## 1. Host system disk (`/`)
|
||
|
||
**See what is big (read-only):**
|
||
|
||
```bash
|
||
df -h /
|
||
du -xsh /var/lib/vz/dump /var/lib/vz/template /var/log/journal /lib/modules /boot /var/lib/felhom-agent/selfupdate
|
||
journalctl --disk-usage
|
||
pvesm list local
|
||
dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii
|
||
```
|
||
|
||
**The product does by itself:** it empties the package cache in each OS update. It removes old agent binaries from the
|
||
self-update folder after each good agent update. It keeps the newest N whole-box copies
|
||
(default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS
|
||
update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The
|
||
hub alarms at 90 % and 95 %.
|
||
|
||
**Safe to remove, by which path:**
|
||
|
||
| What | Step | Mark |
|
||
|---|---|---|
|
||
| Old kernels | **A ring-0 box removes them by itself at night since agent v0.174.0** (`11-os-updates.md` §5.11.1). First read what it would do: `sudo -u felhom-agent felhom-agent --config /etc/felhom-agent/agent.json --selftest=kernel-cleanup-dry-run` (it changes nothing; „would": remove / nothing / refuse / not_run, with the reason). The hand step below is for a box where the program does not run (a ring-1 box, a box that fell back) and for an urgent case: `ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). The kept older kernel then booted the box to healthy and the box went back (`audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **NOT PROVEN:** on a second box |
|
||
| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) |
|
||
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free <volid>`, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only |
|
||
| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |
|
||
| Old binaries in the agent's self-update folder (H12) | No hand step. The agent removes them at the end of each good update, and keeps the running version's file | **PROVEN 2026-11-10** as a product path on three test boxes: 49 → 1, 48 → 1 and 33 → 1 files; a rollback worked after it (`audits/release-space-fixes-2026-11-10/agent/`). **NOT PROVEN:** a removal by hand |
|
||
| Extra guest template archives (H5); `/usr/local/bin/felhom-agent.bak-*` (H13) | — | **NOT PROVEN** (R-1368) |
|
||
|
||
**Never by hand:**
|
||
|
||
- The running kernel and the one before it. Never a kernel with a plain `apt autoremove` or by a meta-package: `apt`
|
||
did not refuse the running kernel, and one meta-package pulled `proxmox-ve` with it
|
||
(`audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt`).
|
||
- `pending.json` in the self-update folder, and `/usr/local/bin/felhom-agent.prev`. The agent's rollback needs both.
|
||
- The newest whole-box copy.
|
||
- Anything under `/var/lib/felhom-os-apply/`: the wrapper's state and the last config bundles.
|
||
|
||
## 2. Thin pool (`pve/data`)
|
||
|
||
**See what is big (read-only):**
|
||
|
||
```bash
|
||
lvs -o lv_name,lv_size,data_percent,metadata_percent pve
|
||
pct list
|
||
```
|
||
|
||
**The product does by itself:** the agent trims each guest once a week (decision 139). It removes a restore test's
|
||
scratch guest at the end and tries again every 10 minutes when that fails. A restore test starts only with enough free
|
||
space. The hub alarms at 85 % and 90 % on the worse of data and metadata.
|
||
|
||
**At 100 %** the guest's disks go read-only. A write in progress can be cut off: on 2026-09-24 two apps' Redis files
|
||
were (`architecture/03-host-agent.md` §8, the R-672 note; `audits/r672-2026-09-24/README.md`).
|
||
|
||
| What | Step | Mark |
|
||
|---|---|---|
|
||
| Space the guest freed but the pool still holds | `pct fstrim <vmid>` | **PROVEN 2026-10-06**: pool 65.5 % → 33.4 % in 24 s, the box answered all the time (`audits/ten-answers-2026-10-06/r444-measure.txt`). The product does this weekly |
|
||
| A guest that is read-only after a full pool | Free pool space first. Stop the guest. `pct fsck <vmid>`, then `pct fsck <vmid> --device mp0`. Start it. Check each app | **PROVEN once, 2026-09-24** (`audits/r672-2026-09-24/README.md`, Part B; `B4-fsck.txt`). Two Redis files then needed `redis-check-aof --fix`, with the operator's yes and a copy kept first |
|
||
| A left-over scratch guest or a thin device with no volume | `runbooks/whole-guest-backup-orphan-thin.md` | see that runbook |
|
||
|
||
**Never by hand:** `lvremove` of a `vm-9201-disk-*` volume. `lvextend` of the pool into the space that belongs to the
|
||
internal place. A thin device, except by the runbook above.
|
||
|
||
## 3. Guest root disk (32 GiB)
|
||
|
||
**See what is big (read-only):**
|
||
|
||
```bash
|
||
pct exec <vmid> -- df -h /
|
||
pct exec <vmid> -- du -xsh /var /opt /usr /root
|
||
```
|
||
|
||
**The product does by itself:** it empties the guest's package cache in each guest OS update. Nothing else. **Nobody
|
||
watches this disk** (R-1371). It was 4–5 % used on three boxes on 2026-11-10.
|
||
|
||
**Safe to remove:** nothing is proven. **NOT PROVEN** for every hand step on this disk. When it is full, the agent can
|
||
grow it; that path is the product's, not a hand's.
|
||
|
||
**Never by hand:** `/opt/docker/stacks/<app>/app.yaml`. It holds the app's settings and its records.
|
||
|
||
## 4. Guest data volume (`/var/lib/felhom`)
|
||
|
||
**See what is big (read-only):**
|
||
|
||
```bash
|
||
pct exec <vmid> -- df -h /var/lib/felhom
|
||
pct exec <vmid> -- docker system df
|
||
pct exec <vmid> -- docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}'
|
||
```
|
||
|
||
Never dump a container's `Env`.
|
||
|
||
**The product does by itself:** it keeps the current image and one undo image for each app, and removes older ones. It
|
||
keeps the running controller image and the one before; the same for infrastructure images. Docker caps each container's
|
||
log at 10 MB × 3. It deletes graph rows older than 30 days. It refuses a new app under max(5 GB, 10 %) free. It refuses
|
||
a backup at 97 % or under 1 GiB, app by app, and deletes nothing. It warns the household at 85 % and 95 %.
|
||
|
||
**Safe to remove, by which path:**
|
||
|
||
| What | Step | Mark |
|
||
|---|---|---|
|
||
| An app the household does not want | The household removes it on the app's page. That is the only path that frees an app's volume | product path |
|
||
| Old controller, infrastructure and helper images that no container uses | `docker image rm <name:tag>`, one exact name at a time, after `docker ps -a` shows no container on it | **PROVEN 2026-11-09**: 36 images, 3.4 GB (off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/cleanup.txt`). The controller now removes old controller and infrastructure images by itself. Helper images stay (R-1372) |
|
||
| Volumes with no name (G8) | — | **NOT PROVEN** (R-995, R-1267) |
|
||
|
||
**Never by hand:** `docker system prune`, `docker volume prune`, `docker image prune -a`. The undo images are kept on
|
||
purpose (decision 53), and volumes hold the apps' data. An app's undo image. A named volume.
|
||
|
||
## 5. A data drive
|
||
|
||
**See what is big:** sizes of the top folders only. Do not list file names below a household's data.
|
||
|
||
```bash
|
||
pct exec <vmid> -- df -h <mount point>
|
||
pct exec <vmid> -- du -xsh <mount point>/* <mount point>/backups/*
|
||
```
|
||
|
||
**The product does by itself:** the nightly backup replaces each app's one point. The second-drive copy needs
|
||
max(2 GB, 20 %) free. The controller deletes a restore's undo copy after 30 days without use, and tells the household.
|
||
The off-site proof cleans its own scratch. The controller warns the household at 85 % and 95 %.
|
||
|
||
**Safe to remove:** only what the household removes on the product's pages: the kept data of a removed app (with the
|
||
app's name typed), a set-aside unit, an off-site restore, its own files. **A hand step on a data drive: NOT PROVEN, and
|
||
not wanted.**
|
||
|
||
**Never by hand:** `userdata`, `appdata`, `kept`, `shares`, `backups/primary`, `backups/set-aside`,
|
||
`backups/restore-undo`. These grow without a limit by ruling (`16` §14.2; decisions 40, 67, 256, 281). Only the
|
||
household deletes them.
|
||
|
||
---
|
||
|
||
**After any hand step:** read `df` again, save the before and after to the evidence folder, and check the box on the
|
||
hub. Then write a dated line on the row that names the place.
|