docs: the disk-space section (16 §14), the runbook "a disk is full", the run's report
gates / gates (push) Successful in 2m7s

C4: documentation/runbooks/disk-full.md (R-1369). Five disks; each hand step
is PROVEN with its date and evidence, or NOT PROVEN.
C5: 16-storage-physical-identity.md §14 (D69, decision 288). 45 places, each
with limit, cleaner and alarm; the rulings behind "no limit"; the host disk
budget; the open design points. PROMPT-TEMPLATE §N.5 gets the keep-it-true line.
Close-out: STATUS (D75 new), CONTEXT, dated lines on R-1369, R-1370, R-1371,
three small stale-document fixes, the report of the run.

Documents only. No release, no build, no deploy.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-10 14:47:25 +01:00
1 parent b14eb0fad4
commit 025a1bcd40
11 files changed
+504 -8

No files matched your search

+3 -2
View File
@@ -525,8 +525,9 @@ recovery credential).
**Expected remnants** (documented, not residue):
- The **hub host record** (+ its report/guest history) — the hub currently has **no host-delete**;
removing it is a direct-DB operator action (recorded follow-up).
- The **hub host record** (+ its report/guest history) — the operator removes it with the host's delete action in
the hub (`POST /hosts/{id}/delete`; `architecture/05-hub-architecture.md`, the host-delete table). *(Corrected
2026-11-10: this line said the hub had no host delete.)*
- The `sudo` and `dnsmasq` packages (system packages installed as dependencies; inert).
- The **golden vzdump** on the archive storage — pass `--remove-golden` to delete it too.
- System noise: LVM archive metadata, dpkg lists, PVE's own rrd/`/var/lib/lxc` scratch entries.
+156
View File
@@ -0,0 +1,156 @@
# RUNBOOK — a disk on a box is full, or nearly full (R-1369)
**When:** a space alarm came (`host_disk_*`, `storage_fill_*`, the controller's fill warning), a whole-box copy was
skipped for space, or an update was refused for space.
**What this is:** one short part for each disk. Each part says how to see what is big, what is safe to remove and by
which path, what a hand must never remove, and what the product does by itself. **The design is not here**: each place
with its limit, cleaner and alarm is `architecture/16-storage-physical-identity.md` §14. The place ids (H1, G2, D6 …)
are the same there and in `audits/space-list-2026-11-10/TABLE.md`.
**Each hand step carries a mark.** `PROVEN` = a person or CC did exactly this on a box, with the date and the
evidence. `NOT PROVEN` = nobody did it; it is not a step yet. **Every `PROVEN` step below was proven on demo-hp, a test
box.** None was done on a household's box.
**Before any hand step, on any disk:**
1. Read which box it is and what you may do there: `runbooks/target-selection.md`.
2. Check that no backup, restore, restore test or update runs: `pct list` shows no lock; `ps -ef | grep -E
'vzdump|pct restore|apt'` shows nothing.
3. Save what you see before you remove anything (R-320): the `df`, the list, the file's checksum.
4. On a household's box a hand removes nothing without the operator's word.
---
## 1. Host system disk (`/`)
**See what is big (read-only):**
```bash
df -h /
du -xsh /var/lib/vz/dump /var/lib/vz/template /var/log/journal /lib/modules /boot /var/lib/felhom-agent/selfupdate
journalctl --disk-usage
pvesm list local
dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii
```
**The product does by itself:** it empties the package cache in each OS update. It keeps the newest N whole-box copies
(default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS
update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The
hub alarms at 90 % and 95 %.
**Safe to remove, by which path:**
| What | Step | Mark |
|---|---|---|
| Old kernels | `ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). **NOT PROVEN:** on a second box; a boot of the kept older kernel |
| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) |
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free <volid>`, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only |
| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |
| Old binaries in the agent's self-update folder (H12) | — | **NOT PROVEN.** The agent cleans the folder after the release that carries the fix (R-1368) |
| Extra guest template archives (H5); `/usr/local/bin/felhom-agent.bak-*` (H13) | — | **NOT PROVEN** (R-1368) |
**Never by hand:**
- The running kernel and the one before it. Never a kernel with a plain `apt autoremove` or by a meta-package: `apt`
did not refuse the running kernel, and one meta-package pulled `proxmox-ve` with it
(`audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt`).
- `pending.json` in the self-update folder, and `/usr/local/bin/felhom-agent.prev`. The agent's rollback needs both.
- The newest whole-box copy.
- Anything under `/var/lib/felhom-os-apply/`: the wrapper's state and the last config bundles.
## 2. Thin pool (`pve/data`)
**See what is big (read-only):**
```bash
lvs -o lv_name,lv_size,data_percent,metadata_percent pve
pct list
```
**The product does by itself:** the agent trims each guest once a week (decision 139). It removes a restore test's
scratch guest at the end and tries again every 10 minutes when that fails. A restore test starts only with enough free
space. The hub alarms at 85 % and 90 % on the worse of data and metadata.
**At 100 %** the guest's disks go read-only. A write in progress can be cut off: on 2026-09-24 two apps' Redis files
were (`architecture/03-host-agent.md` §8, the R-672 note; `audits/r672-2026-09-24/README.md`).
| What | Step | Mark |
|---|---|---|
| Space the guest freed but the pool still holds | `pct fstrim <vmid>` | **PROVEN 2026-10-06**: pool 65.5 % → 33.4 % in 24 s, the box answered all the time (`audits/ten-answers-2026-10-06/r444-measure.txt`). The product does this weekly |
| A guest that is read-only after a full pool | Free pool space first. Stop the guest. `pct fsck <vmid>`, then `pct fsck <vmid> --device mp0`. Start it. Check each app | **PROVEN once, 2026-09-24** (`audits/r672-2026-09-24/README.md`, Part B; `B4-fsck.txt`). Two Redis files then needed `redis-check-aof --fix`, with the operator's yes and a copy kept first |
| A left-over scratch guest or a thin device with no volume | `runbooks/whole-guest-backup-orphan-thin.md` | see that runbook |
**Never by hand:** `lvremove` of a `vm-9201-disk-*` volume. `lvextend` of the pool into the space that belongs to the
internal place. A thin device, except by the runbook above.
## 3. Guest root disk (32 GiB)
**See what is big (read-only):**
```bash
pct exec <vmid> -- df -h /
pct exec <vmid> -- du -xsh /var /opt /usr /root
```
**The product does by itself:** it empties the guest's package cache in each guest OS update. Nothing else. **Nobody
watches this disk** (R-1371). It was 4–5 % used on three boxes on 2026-11-10.
**Safe to remove:** nothing is proven. **NOT PROVEN** for every hand step on this disk. When it is full, the agent can
grow it; that path is the product's, not a hand's.
**Never by hand:** `/opt/docker/stacks/<app>/app.yaml`. It holds the app's settings and its records.
## 4. Guest data volume (`/var/lib/felhom`)
**See what is big (read-only):**
```bash
pct exec <vmid> -- df -h /var/lib/felhom
pct exec <vmid> -- docker system df
pct exec <vmid> -- docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}'
```
Never dump a container's `Env`.
**The product does by itself:** it keeps the current image and one undo image for each app, and removes older ones. It
keeps the running controller image and the one before; the same for infrastructure images. Docker caps each container's
log at 10 MB × 3. It deletes graph rows older than 30 days. It refuses a new app under max(5 GB, 10 %) free. It refuses
a backup at 97 % or under 1 GiB, app by app, and deletes nothing. It warns the household at 85 % and 95 %.
**Safe to remove, by which path:**
| What | Step | Mark |
|---|---|---|
| An app the household does not want | The household removes it on the app's page. That is the only path that frees an app's volume | product path |
| Old controller, infrastructure and helper images that no container uses | `docker image rm <name:tag>`, one exact name at a time, after `docker ps -a` shows no container on it | **PROVEN 2026-11-09**: 36 images, 3.4 GB (off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/cleanup.txt`). The controller now removes old controller and infrastructure images by itself. Helper images stay (R-1372) |
| Volumes with no name (G8) | — | **NOT PROVEN** (R-995, R-1267) |
**Never by hand:** `docker system prune`, `docker volume prune`, `docker image prune -a`. The undo images are kept on
purpose (decision 53), and volumes hold the apps' data. An app's undo image. A named volume.
## 5. A data drive
**See what is big:** sizes of the top folders only. Do not list file names below a household's data.
```bash
pct exec <vmid> -- df -h <mount point>
pct exec <vmid> -- du -xsh <mount point>/* <mount point>/backups/*
```
**The product does by itself:** the nightly backup replaces each app's one point. The second-drive copy needs
max(2 GB, 20 %) free. The controller deletes a restore's undo copy after 30 days without use, and tells the household.
The off-site proof cleans its own scratch. The controller warns the household at 85 % and 95 %.
**Safe to remove:** only what the household removes on the product's pages: the kept data of a removed app (with the
app's name typed), a set-aside unit, an off-site restore, its own files. **A hand step on a data drive: NOT PROVEN, and
not wanted.**
**Never by hand:** `userdata`, `appdata`, `kept`, `shares`, `backups/primary`, `backups/set-aside`,
`backups/restore-undo`. These grow without a limit by ruling (`16` §14.2; decisions 40, 67, 256, 281). Only the
household deletes them.
---
**After any hand step:** read `df` again, save the before and after to the evidence folder, and check the box on the
hub. Then write a dated line on the row that names the place.
+4 -2
View File
@@ -81,7 +81,7 @@ Provision a new ep0 (06 §5), then on it add DooPlex as a remote and run
Do **not** prune the copy tighter than decision 71 (8 weekly copies); never tighter than ep0's own retention.
## Removing a deleted customer's copy (R-901, `09` §3 decision 181) — WRITTEN 2026-10-08, NOT INSTALLED
## Removing a deleted customer's copy (R-901, `09` §3 decision 181) — WRITTEN 2026-10-08, INSTALLED 2026-10-09
The sync keeps what ep0 removed (`remove-vanished false`), so a deleted customer's namespace stayed here for ever.
Ruling: removed **within 30 days**. The job `scripts/ep0-copy-gc/felhom-ep0-copy-gc` (tests beside it) runs daily at
@@ -92,7 +92,9 @@ nothing; if more than 2 namespaces are absent at once (a partial list, e.g. a re
deletes at most one namespace per run; a HOLD file (`/etc/felhom/ep0-copy-gc/HOLD`) stops it. Without `--apply` it
only prints. Worst case: ≤ 1 day to notice + 7 days grace + 1 day = inside 30 days.
**Not installed. To install (needs the operator's word — it is a change on DooPlex and it deletes data):**
**Installed 2026-10-09 with the operator present (closed R-901: key root-only, a dry run, then the 08:00 timer). Not
re-checked on DooPlex on 2026-11-10, when this heading was corrected from „NOT INSTALLED”. The steps, for an install
from nothing (needs the operator's word — it is a change on DooPlex and it deletes data):**
1. Two tokens, secrets in root-only files under `/etc/felhom/ep0-copy-gc/` (0600), never printed:
- `ep0-reader.secret` — the existing ep0 read token `root@pam!dooplex-sync` (its secret is in `remote.cfg`, base64);
`ep0.fingerprint` — ep0's pinned certificate fingerprint (also in `remote.cfg`).