docs: the disk-space section (16 §14), the runbook "a disk is full", the run's report
gates / gates (push) Successful in 2m7s

C4: documentation/runbooks/disk-full.md (R-1369). Five disks; each hand step
is PROVEN with its date and evidence, or NOT PROVEN.
C5: 16-storage-physical-identity.md §14 (D69, decision 288). 45 places, each
with limit, cleaner and alarm; the rulings behind "no limit"; the host disk
budget; the open design points. PROMPT-TEMPLATE §N.5 gets the keep-it-true line.
Close-out: STATUS (D75 new), CONTEXT, dated lines on R-1369, R-1370, R-1371,
three small stale-document fixes, the report of the run.

Documents only. No release, no build, no deploy.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-10 14:47:25 +01:00
1 parent b14eb0fad4
commit 025a1bcd40
11 files changed
+504 -8

No files matched your search

@@ -240,6 +240,179 @@ Also in build B: R-1084 (the whole-box backup offer asks the same-disk question)
offered to attach, by device). A bundle that adds a path needs a step bundle on boxes with an older wrapper (R16, R-880):
0.158.1 was delivered through `0.158.1-step1`.
## 14. Disk space — every place, its limit, its cleaner, its alarm (D69, decision 288)
**What this is.** The design of disk space on a box. One line for each place that holds data: what makes it grow, its
limit, what cleans it, and which alarm fires. Written 2026-11-10 from the space list. **The evidence is not here** — it
is `audits/space-list-2026-11-10/` (`TABLE.md` holds each place with its `file:line` and its measurement; the place ids
below are the pointers into it). Counts on that day: 45 places; 21 have a name in this folder, 16 a mention, 8 none.
**[DESIGN] The rule that keeps this true.** A change that adds a place that holds data, or changes the limit, the
cleaner or the alarm of one, changes this section in the same session (`PROMPT-TEMPLATE.md` §N.5). A place with no
limit is on the list in §14.2 with its ruling, or it has a register row.
### 14.1 The places [FACT — read and measured 2026-11-10, `audits/space-list-2026-11-10/TABLE.md`]
The alarms, by short name:
- **host disk** — the hub's `host_disk_warning` at 90 % and `host_disk_critical` at 95 %. To the operator only (D68,
decision 287, `08-alarm-ladder.md`). That is in the hub's source and not in the hub that runs: until the next hub
release the household can still get the critical one.
- **pool** — the hub's `storage_fill_warning` at 85 % and `storage_fill_critical` at 90 %, on the worse of the pool's
data and metadata. To the operator.
- **data volume** — the controller, every 10 minutes: warning at 85 % or under 5 GiB free, critical at 95 % or under
2 GiB. To the household.
- **drive** — the same watch of the controller, on each registered drive. To the household.
**Host system disk** (`/`, the volume `pve/root`)
| Id | Place | Grows with | Limit | Cleaner | Alarm |
|---|---|---|---|---|---|
| H1 | the system disk as a whole | its parts below (72–82 % of the used bytes) | the size Proxmox gave it; no budget per part (§14.3, R-1370) | each part's own; the whole disk only by the operator's hand (`runbooks/disk-full.md`) | host disk |
| H3 | whole-box copies on the current local target | the guest's size (R-1366) | the newest N (default 3) | Proxmox after each good run; the agent removes half-written copies | a skipped run is reported to the hub and shown on the backup page; host disk |
| H4 | whole-box copies on a former target | a move of the target | none (R-1087) | nobody | host disk |
| H5 | guest template archives | each install and re-install | none (R-1368) | nobody; only an uninstall with `--remove-golden` | host disk |
| H6 | backup logs and half-written copies | each run | a failed run's log: none (R-1368) | Proxmox; the agent | host disk |
| H7 | the host journal | the agent's `sudo` calls (R-1367) | the journald default: 10 % of the disk, at most 4 GB | journald | host disk |
| H8 | other host logs, task logs, graphs | each task | rotation | `logrotate`, Proxmox | host disk |
| H9 | old kernels | each approved kernel | none today. Ruled: keep the running one, the one before it and the newest (decision 286, `11-os-updates.md` §5.11.1). Proven by hand, not built | nobody today | host disk; later `os_update_stale` |
| H10 | the EFI partition | a boot-loader update | 1.07 GB, 1 % used | none needed | none (R-1371) |
| H11 | the host package cache | each OS update | emptied in each run | the update wrapper | host disk |
| H12 | the agent's self-update folder | each agent update | none on a released agent; one binary after the next agent release (in source, R-1368) | the agent, after that release | host disk |
| H13 | old agent binaries | `.prev`: one file. `.bak-*`: each re-run of the installer | none for `.bak-*` (R-1368) | nobody | host disk |
| H14 | wrapper state and previous config bundles | each bundle | the last 3 | the wrapper | host disk |
| H15 | host `/tmp` and `/var/tmp` | nothing that lasts | `/tmp` is in memory: half of memory | systemd, after 10 and 30 days | none for `/tmp` (R-1371) |
| H16 | a bare folder under a drive that is not mounted | a write while the drive is away | the write is refused | nothing to clean | `storage_disconnected`, to the household |
| H18 | swap | fixed | its volume | the kernel | `host_swap_short`, to the operator |
**Thin pool** (`pve/data`)
| Id | Place | Grows with | Limit | Cleaner | Alarm |
|---|---|---|---|---|---|
| H2 | the pool | each block a guest writes | the pool's size; auto-extend is off | the agent's weekly trim (decision 139); a left-over scratch guest: a retry every 10 minutes | pool |
| H17 | the restore test's scratch guest | each restore test | it needs restored size × 1.2 + 5 GiB free, off the tested guest's pool when it can | the agent at the end; a retry every 10 minutes | pool |
**Guest root disk** (32 GiB; 4–5 % used on three boxes)
| Id | Place | Grows with | Limit | Cleaner | Alarm |
|---|---|---|---|---|---|
| G1 | the root disk as a whole | guest OS updates | 32 GiB; the agent can grow it, never shrink it | the update wrapper, `logrotate` | none (R-1371) |
| G13 | app definitions, `/opt/docker/stacks` | the catalog's size | under 2 MB measured | an app removal deletes its `app.yaml` | none (R-1371) |
| G15 | the guest's system log | each line | the journald default; it is in memory (R-1372) | journald; a restart | none |
| G16 | the guest package cache | each guest OS update | emptied in each run | the wrapper | none (R-1371) |
| G17 | guest `/tmp` | nothing that lasts | in memory | systemd; a restart | none |
**Guest data volume** (`mp0`, `/var/lib/felhom`)
| Id | Place | Grows with | Limit | Cleaner | Alarm |
|---|---|---|---|---|---|
| G2 | the data volume as a whole | G3–G12 and G14 | 50 / 100 / 130 GiB (§7; older boxes 70 or 250). A new app is refused under max(5 GB, 10 %) free; a backup is refused at 97 % or under 1 GiB | the three image rules below; the weekly trim | data volume. A box with one volume counts it twice until the next controller release: two warnings for one fill (R-1371) |
| G3 | app images, and one undo image for each app | each install and update | a count: the current and the one before, for each app. No limit in bytes, by ruling (§14.2) | the controller | data volume |
| G4 | old controller images | each controller update | the running one and the one before | the controller | data volume |
| G5 | infrastructure images | a change of the pinned version | the pinned one and the one before | the controller | data volume |
| G6 | helper images (`alpine`, `hello-world`) | pulled once | no version is pinned (R-1372) | nobody | data volume |
| G7 | named app volumes | the app's use | none for each app; the volume's | only an app removal | data volume |
| G8 | volumes with no name | each re-create of a container | none (R-995, R-1267) | nobody | data volume (bytes only) |
| G9 | containers' writable layers; the off-site cache | temporary files; the off-site store's size | none of its own | a re-create drops it | data volume |
| G10 | container logs | each line | 10 MB × 3 for each container | Docker | data volume |
| G11 | the build cache | nothing | empty | — | data volume |
| G12 | the controller's state volume | graph rows; the catalog | rows older than 30 days are deleted | the controller | data volume |
| G14 | backups of apps with no drive (`sys_drive`) | dumps | one point for each app; refused at 97 % or under 1 GiB | the nightly replace; moved units are pruned | data volume |
**A data drive**
| Id | Place | Grows with | Limit | Cleaner | Alarm |
|---|---|---|---|---|---|
| D1 | `backups/primary` | dumps | one point for each app. A removed app's kept unit: none, by ruling (§14.2) | the nightly replace; a removal with "delete the backups too" | drive |
| D2 | `backups/secondary` | the apps' data | it needs max(2 GB, 20 %) free. A removed app's copy: none, by ruling (§14.2) | the mirror; the household, from the backup page | drive |
| D3 | `backups/offsite-restore` | each off-site restore or check | none (R-1372) | the household's delete button | drive |
| D4 | `backups/offsite-proof` | one unit a night, for minutes | a headroom check | the proof itself | drive |
| D5 | `backups/restore-undo` | each restore | 30 days unused; none under a support hold, by ruling (§14.2) | the controller's sweep | drive; the household is told on a delete |
| D6 | `kept` | each "remove, keep my data" | none, by ruling (§14.2) | the household | drive |
| D7 | `appdata` | the app's use | the drive | a removal "with data" | drive |
| D8 | `userdata` | the household | none, by ruling (§14.2) | the household | drive |
| D9 | `shares` | the household's copies | the drive | the household | drive |
| D10 | `backups/set-aside` | each re-install that finds a unit | none, by ruling (§14.2) | the household; a removal with "delete its backups too" | drive |
### 14.2 What can grow without a limit, by ruling [FACT — each line is a ruling, not a defect]
Do not file these as defects (R-370). A change to one is a decision for the operator.
1. Undo images: one for each app, no limit in bytes (decision 53).
2. The kept data of a removed app: only the household deletes it, with the app's name typed (decision 40).
3. The household's own files (decision 67).
4. A set-aside backup unit (decision 256).
5. A restore's undo copy while a support hold is on it (decision 281).
6. The second-drive copy of a removed app (D25).
7. The backup unit of a removed app that the household kept (`07-backup-architecture.md` §6).
8. Images in the registry (decision 62). This is not on a box.
### 14.3 The host system disk — a budget from measured numbers [FACT for the numbers; the answer is OPEN — D75, R-1370]
Measured 2026-11-10 (`audits/space-list-2026-11-10/evidence/reading-1/`, GB = 10⁹ bytes):
| | 128 GB disk (demo-hp) | 200 GB disk (Tester 1) | 512 GB disk (N100) |
|---|---|---|---|
| root volume, as Proxmox made it | 41.4 | 62.9 | 100.9 (the 96 GiB cap) |
| used | 24.0 | 21.6 | 38.1 |
| Proxmox, Debian and all that has no line of its own | 3.3 | 3.2 | 3.4 |
No rule is written here for a size between these three: the Proxmox installer's formula was not read.
**What the root needs when the whole-box copies go to a backup drive:**
| Part | GB | From |
|---|---|---|
| Proxmox, Debian, small parts | 4 | the last line above |
| kernels: the three the ruling keeps, and one in install | 4.4 | one kernel = 1.02 GB of modules + 0.10 GB in `/boot` |
| the host journal at its cap | 4.3 | measured on two boxes (H7) |
| one guest template archive | 0.65 | H5 |
| the agent's files, after the self-update fix | 0.1 | H12 (0.75 before the fix) |
| free space an OS update asks for | 0.5 | `felhom-os-apply`, the free-space check |
| **Sum** | **≈ 14** | |
With the parts nobody cleans today (8 kernels, 49 agent binaries, 4 templates) the same sum is about 21 GB, and it grows
by 1.1 GB with each kernel.
**What local whole-box copies add.** Let C be one packed copy. The copies need keep × C. A new copy also needs
1.25 × C + 1.07 GB free before it starts, because Proxmox prunes only after a good run (A `internal/backup/runner.go`,
the space check). Measured C: 9.1 GB (demo-hp, 26 containers), 3.15 GB (Tester 1), 1.45 GB (N100).
| Keep | C | Copies + the free space the next one needs | With the 14 GB above |
|---|---|---|---|
| 3 | 3.15 GB | 14.5 GB | 28.5 GB |
| 3 | 9.1 GB | 39.7 GB | 53.7 GB |
| 1 | 9.1 GB | 21.5 GB | 35.5 GB |
**What follows from the numbers. This is not a decision.**
- **With a backup drive** the root needs about 14 GB. The 41 GB root of a 128 GB disk holds that. The limit is then the
pool: a 128 GB disk has a 54 GiB pool, and the guest is promised 32 + 50 GiB of it (§7). That is over-promised by
design and it is watched (the pool alarm). The first measured disk whose pool holds 32 + 50 + the 16 GiB reserve is
the 200 GB one (113 GiB).
- **With no backup drive** the root needs 28 to 54 GB at keep 3, by the number of apps. A 128 GB disk holds a small
household. It does not hold demo-hp's app set at keep 3. demo-hp keeps 1, and 35.5 GB in a 41.4 GB root leaves
little room: a copy that cannot fit is skipped before it starts (R-548). The 63 GB root of a 200 GB disk holds the
54 GB case with 9 GB to spare.
- The installer refuses no disk today, and nothing checks this sum at install time.
The question for the operator is D75 in `STATUS.md`: the smallest system disk Felhom accepts, with and without a backup
drive.
### 14.4 Open design points [each has a row]
| Point | Row |
|---|---|
| Nobody watches the guest's root disk (G1, G13, G16). It is 4–5 % used today; app definitions lie on it | R-1371 |
| No alarm for the EFI partition (H10) or for the places in memory (H15, G15, G17) | R-1371 |
| A drive in the backup role reports as `usb` or `local-dir`. So the household gets the `storage_fill` mail for a drive it must not clean by hand | R-1371 |
| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. Not measured on the running hub in the 2026-11-10 run | R-1371 |
| Helper images carry no version; the guest's journal has no stated cap; off-site restores have no limit | R-1372 |
| Old kernels: the keep rule is ruled and proven by hand, not built | `11-os-updates.md` §5.11.1 |
| The parts of the host disk with no limit and no cleaner (H4, H5, H6, H13) | R-1087, R-1368 |
| No budget is enforced; no smallest disk is ruled | R-1370, D75 |
| What to do by hand when a disk is full | `runbooks/disk-full.md` (R-1369) |
## Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
Read at `file:line` on `main` by the main session:
@@ -435,6 +435,8 @@ stayed. **No other customer touched:** the three live boxes' repositories re-lis
**Gap / roadmap (as written):** Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run
**Noted 2026-11-10 (endings list):** „tunnel + zone removed” is not what the code does. No leg of the delete calls Cloudflare; the operator removes the tunnel and the DNS record by hand, guided by the dialog. The operator accepted that on 2026-10-05 (closed R-688). The words above stay as written.
---
<a id="a22"></a>