docs: the disk-space section (16 §14), the runbook "a disk is full", the run's report
gates / gates (push) Successful in 2m7s
gates / gates (push) Successful in 2m7s
C4: documentation/runbooks/disk-full.md (R-1369). Five disks; each hand step is PROVEN with its date and evidence, or NOT PROVEN. C5: 16-storage-physical-identity.md §14 (D69, decision 288). 45 places, each with limit, cleaner and alarm; the rulings behind "no limit"; the host disk budget; the open design points. PROMPT-TEMPLATE §N.5 gets the keep-it-true line. Close-out: STATUS (D75 new), CONTEXT, dated lines on R-1369, R-1370, R-1371, three small stale-document fixes, the report of the run. Documents only. No release, no build, no deploy. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
b14eb0fad4
commit
025a1bcd40
11 files changed
+504
-8
No files matched your search
@@ -525,8 +525,9 @@ recovery credential).
|
||||
|
||||
**Expected remnants** (documented, not residue):
|
||||
|
||||
- The **hub host record** (+ its report/guest history) — the hub currently has **no host-delete**;
|
||||
removing it is a direct-DB operator action (recorded follow-up).
|
||||
- The **hub host record** (+ its report/guest history) — the operator removes it with the host's delete action in
|
||||
the hub (`POST /hosts/{id}/delete`; `architecture/05-hub-architecture.md`, the host-delete table). *(Corrected
|
||||
2026-11-10: this line said the hub had no host delete.)*
|
||||
- The `sudo` and `dnsmasq` packages (system packages installed as dependencies; inert).
|
||||
- The **golden vzdump** on the archive storage — pass `--remove-golden` to delete it too.
|
||||
- System noise: LVM archive metadata, dpkg lists, PVE's own rrd/`/var/lib/lxc` scratch entries.
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
# RUNBOOK — a disk on a box is full, or nearly full (R-1369)
|
||||
|
||||
**When:** a space alarm came (`host_disk_*`, `storage_fill_*`, the controller's fill warning), a whole-box copy was
|
||||
skipped for space, or an update was refused for space.
|
||||
|
||||
**What this is:** one short part for each disk. Each part says how to see what is big, what is safe to remove and by
|
||||
which path, what a hand must never remove, and what the product does by itself. **The design is not here**: each place
|
||||
with its limit, cleaner and alarm is `architecture/16-storage-physical-identity.md` §14. The place ids (H1, G2, D6 …)
|
||||
are the same there and in `audits/space-list-2026-11-10/TABLE.md`.
|
||||
|
||||
**Each hand step carries a mark.** `PROVEN` = a person or CC did exactly this on a box, with the date and the
|
||||
evidence. `NOT PROVEN` = nobody did it; it is not a step yet. **Every `PROVEN` step below was proven on demo-hp, a test
|
||||
box.** None was done on a household's box.
|
||||
|
||||
**Before any hand step, on any disk:**
|
||||
|
||||
1. Read which box it is and what you may do there: `runbooks/target-selection.md`.
|
||||
2. Check that no backup, restore, restore test or update runs: `pct list` shows no lock; `ps -ef | grep -E
|
||||
'vzdump|pct restore|apt'` shows nothing.
|
||||
3. Save what you see before you remove anything (R-320): the `df`, the list, the file's checksum.
|
||||
4. On a household's box a hand removes nothing without the operator's word.
|
||||
|
||||
---
|
||||
|
||||
## 1. Host system disk (`/`)
|
||||
|
||||
**See what is big (read-only):**
|
||||
|
||||
```bash
|
||||
df -h /
|
||||
du -xsh /var/lib/vz/dump /var/lib/vz/template /var/log/journal /lib/modules /boot /var/lib/felhom-agent/selfupdate
|
||||
journalctl --disk-usage
|
||||
pvesm list local
|
||||
dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii
|
||||
```
|
||||
|
||||
**The product does by itself:** it empties the package cache in each OS update. It keeps the newest N whole-box copies
|
||||
(default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS
|
||||
update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The
|
||||
hub alarms at 90 % and 95 %.
|
||||
|
||||
**Safe to remove, by which path:**
|
||||
|
||||
| What | Step | Mark |
|
||||
|---|---|---|
|
||||
| Old kernels | `ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). **NOT PROVEN:** on a second box; a boot of the kept older kernel |
|
||||
| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) |
|
||||
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free <volid>`, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only |
|
||||
| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |
|
||||
| Old binaries in the agent's self-update folder (H12) | — | **NOT PROVEN.** The agent cleans the folder after the release that carries the fix (R-1368) |
|
||||
| Extra guest template archives (H5); `/usr/local/bin/felhom-agent.bak-*` (H13) | — | **NOT PROVEN** (R-1368) |
|
||||
|
||||
**Never by hand:**
|
||||
|
||||
- The running kernel and the one before it. Never a kernel with a plain `apt autoremove` or by a meta-package: `apt`
|
||||
did not refuse the running kernel, and one meta-package pulled `proxmox-ve` with it
|
||||
(`audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt`).
|
||||
- `pending.json` in the self-update folder, and `/usr/local/bin/felhom-agent.prev`. The agent's rollback needs both.
|
||||
- The newest whole-box copy.
|
||||
- Anything under `/var/lib/felhom-os-apply/`: the wrapper's state and the last config bundles.
|
||||
|
||||
## 2. Thin pool (`pve/data`)
|
||||
|
||||
**See what is big (read-only):**
|
||||
|
||||
```bash
|
||||
lvs -o lv_name,lv_size,data_percent,metadata_percent pve
|
||||
pct list
|
||||
```
|
||||
|
||||
**The product does by itself:** the agent trims each guest once a week (decision 139). It removes a restore test's
|
||||
scratch guest at the end and tries again every 10 minutes when that fails. A restore test starts only with enough free
|
||||
space. The hub alarms at 85 % and 90 % on the worse of data and metadata.
|
||||
|
||||
**At 100 %** the guest's disks go read-only. A write in progress can be cut off: on 2026-09-24 two apps' Redis files
|
||||
were (`architecture/03-host-agent.md` §8, the R-672 note; `audits/r672-2026-09-24/README.md`).
|
||||
|
||||
| What | Step | Mark |
|
||||
|---|---|---|
|
||||
| Space the guest freed but the pool still holds | `pct fstrim <vmid>` | **PROVEN 2026-10-06**: pool 65.5 % → 33.4 % in 24 s, the box answered all the time (`audits/ten-answers-2026-10-06/r444-measure.txt`). The product does this weekly |
|
||||
| A guest that is read-only after a full pool | Free pool space first. Stop the guest. `pct fsck <vmid>`, then `pct fsck <vmid> --device mp0`. Start it. Check each app | **PROVEN once, 2026-09-24** (`audits/r672-2026-09-24/README.md`, Part B; `B4-fsck.txt`). Two Redis files then needed `redis-check-aof --fix`, with the operator's yes and a copy kept first |
|
||||
| A left-over scratch guest or a thin device with no volume | `runbooks/whole-guest-backup-orphan-thin.md` | see that runbook |
|
||||
|
||||
**Never by hand:** `lvremove` of a `vm-9201-disk-*` volume. `lvextend` of the pool into the space that belongs to the
|
||||
internal place. A thin device, except by the runbook above.
|
||||
|
||||
## 3. Guest root disk (32 GiB)
|
||||
|
||||
**See what is big (read-only):**
|
||||
|
||||
```bash
|
||||
pct exec <vmid> -- df -h /
|
||||
pct exec <vmid> -- du -xsh /var /opt /usr /root
|
||||
```
|
||||
|
||||
**The product does by itself:** it empties the guest's package cache in each guest OS update. Nothing else. **Nobody
|
||||
watches this disk** (R-1371). It was 4–5 % used on three boxes on 2026-11-10.
|
||||
|
||||
**Safe to remove:** nothing is proven. **NOT PROVEN** for every hand step on this disk. When it is full, the agent can
|
||||
grow it; that path is the product's, not a hand's.
|
||||
|
||||
**Never by hand:** `/opt/docker/stacks/<app>/app.yaml`. It holds the app's settings and its records.
|
||||
|
||||
## 4. Guest data volume (`/var/lib/felhom`)
|
||||
|
||||
**See what is big (read-only):**
|
||||
|
||||
```bash
|
||||
pct exec <vmid> -- df -h /var/lib/felhom
|
||||
pct exec <vmid> -- docker system df
|
||||
pct exec <vmid> -- docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}'
|
||||
```
|
||||
|
||||
Never dump a container's `Env`.
|
||||
|
||||
**The product does by itself:** it keeps the current image and one undo image for each app, and removes older ones. It
|
||||
keeps the running controller image and the one before; the same for infrastructure images. Docker caps each container's
|
||||
log at 10 MB × 3. It deletes graph rows older than 30 days. It refuses a new app under max(5 GB, 10 %) free. It refuses
|
||||
a backup at 97 % or under 1 GiB, app by app, and deletes nothing. It warns the household at 85 % and 95 %.
|
||||
|
||||
**Safe to remove, by which path:**
|
||||
|
||||
| What | Step | Mark |
|
||||
|---|---|---|
|
||||
| An app the household does not want | The household removes it on the app's page. That is the only path that frees an app's volume | product path |
|
||||
| Old controller, infrastructure and helper images that no container uses | `docker image rm <name:tag>`, one exact name at a time, after `docker ps -a` shows no container on it | **PROVEN 2026-11-09**: 36 images, 3.4 GB (off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/cleanup.txt`). The controller now removes old controller and infrastructure images by itself. Helper images stay (R-1372) |
|
||||
| Volumes with no name (G8) | — | **NOT PROVEN** (R-995, R-1267) |
|
||||
|
||||
**Never by hand:** `docker system prune`, `docker volume prune`, `docker image prune -a`. The undo images are kept on
|
||||
purpose (decision 53), and volumes hold the apps' data. An app's undo image. A named volume.
|
||||
|
||||
## 5. A data drive
|
||||
|
||||
**See what is big:** sizes of the top folders only. Do not list file names below a household's data.
|
||||
|
||||
```bash
|
||||
pct exec <vmid> -- df -h <mount point>
|
||||
pct exec <vmid> -- du -xsh <mount point>/* <mount point>/backups/*
|
||||
```
|
||||
|
||||
**The product does by itself:** the nightly backup replaces each app's one point. The second-drive copy needs
|
||||
max(2 GB, 20 %) free. The controller deletes a restore's undo copy after 30 days without use, and tells the household.
|
||||
The off-site proof cleans its own scratch. The controller warns the household at 85 % and 95 %.
|
||||
|
||||
**Safe to remove:** only what the household removes on the product's pages: the kept data of a removed app (with the
|
||||
app's name typed), a set-aside unit, an off-site restore, its own files. **A hand step on a data drive: NOT PROVEN, and
|
||||
not wanted.**
|
||||
|
||||
**Never by hand:** `userdata`, `appdata`, `kept`, `shares`, `backups/primary`, `backups/set-aside`,
|
||||
`backups/restore-undo`. These grow without a limit by ruling (`16` §14.2; decisions 40, 67, 256, 281). Only the
|
||||
household deletes them.
|
||||
|
||||
---
|
||||
|
||||
**After any hand step:** read `df` again, save the before and after to the evidence folder, and check the box on the
|
||||
hub. Then write a dated line on the row that names the place.
|
||||
@@ -81,7 +81,7 @@ Provision a new ep0 (06 §5), then on it add DooPlex as a remote and run
|
||||
|
||||
Do **not** prune the copy tighter than decision 71 (8 weekly copies); never tighter than ep0's own retention.
|
||||
|
||||
## Removing a deleted customer's copy (R-901, `09` §3 decision 181) — WRITTEN 2026-10-08, NOT INSTALLED
|
||||
## Removing a deleted customer's copy (R-901, `09` §3 decision 181) — WRITTEN 2026-10-08, INSTALLED 2026-10-09
|
||||
|
||||
The sync keeps what ep0 removed (`remove-vanished false`), so a deleted customer's namespace stayed here for ever.
|
||||
Ruling: removed **within 30 days**. The job `scripts/ep0-copy-gc/felhom-ep0-copy-gc` (tests beside it) runs daily at
|
||||
@@ -92,7 +92,9 @@ nothing; if more than 2 namespaces are absent at once (a partial list, e.g. a re
|
||||
deletes at most one namespace per run; a HOLD file (`/etc/felhom/ep0-copy-gc/HOLD`) stops it. Without `--apply` it
|
||||
only prints. Worst case: ≤ 1 day to notice + 7 days grace + 1 day = inside 30 days.
|
||||
|
||||
**Not installed. To install (needs the operator's word — it is a change on DooPlex and it deletes data):**
|
||||
**Installed 2026-10-09 with the operator present (closed R-901: key root-only, a dry run, then the 08:00 timer). Not
|
||||
re-checked on DooPlex on 2026-11-10, when this heading was corrected from „NOT INSTALLED”. The steps, for an install
|
||||
from nothing (needs the operator's word — it is a change on DooPlex and it deletes data):**
|
||||
1. Two tokens, secrets in root-only files under `/etc/felhom/ep0-copy-gc/` (0600), never printed:
|
||||
- `ep0-reader.secret` — the existing ep0 read token `root@pam!dooplex-sync` (its secret is in `remote.cfg`, base64);
|
||||
`ep0.fingerprint` — ep0's pinned certificate fingerprint (also in `remote.cfg`).
|
||||
|
||||
Reference in new issue
Block a user