docs: the disk-space section (16 §14), the runbook "a disk is full", the run's report
gates / gates (push) Successful in 2m7s
gates / gates (push) Successful in 2m7s
C4: documentation/runbooks/disk-full.md (R-1369). Five disks; each hand step is PROVEN with its date and evidence, or NOT PROVEN. C5: 16-storage-physical-identity.md §14 (D69, decision 288). 45 places, each with limit, cleaner and alarm; the rulings behind "no limit"; the host disk budget; the open design points. PROMPT-TEMPLATE §N.5 gets the keep-it-true line. Close-out: STATUS (D75 new), CONTEXT, dated lines on R-1369, R-1370, R-1371, three small stale-document fixes, the report of the run. Documents only. No release, no build, no deploy. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
b14eb0fad4
commit
025a1bcd40
11 files changed
+504
-8
No files matched your search
@@ -375,6 +375,10 @@ Decisions made, architectural state, what's next.
|
||||
> "waiting on the operator" list. **The report names which of the three it did.** The poster is drawn
|
||||
> in Claude Design and nothing here renders it, so `scripts/poster_facts_gate.py` only WARNS when the
|
||||
> facts outrun the drawing — it never fails a push. Same wording in `.claude/rules/unprompted-work.md` §6.
|
||||
> **A place that holds data.** A task that adds a place that holds data on a box, or changes a place's limit, its
|
||||
> cleaner or its alarm, updates the "disk space" section (§14) of `architecture/16-storage-physical-identity.md` in the
|
||||
> same session (D69, `09` §3 decision 288).
|
||||
|
||||
If the task **changed what the platform can do** (new/removed capability, a capability moved status,
|
||||
or **what is open changed**), update **all four** in the SAME session:
|
||||
- **`architecture/00-capability-map.md`** — add/adjust the affected *scenario* row with the correct
|
||||
|
||||
@@ -240,6 +240,179 @@ Also in build B: R-1084 (the whole-box backup offer asks the same-disk question)
|
||||
offered to attach, by device). A bundle that adds a path needs a step bundle on boxes with an older wrapper (R16, R-880):
|
||||
0.158.1 was delivered through `0.158.1-step1`.
|
||||
|
||||
## 14. Disk space — every place, its limit, its cleaner, its alarm (D69, decision 288)
|
||||
|
||||
**What this is.** The design of disk space on a box. One line for each place that holds data: what makes it grow, its
|
||||
limit, what cleans it, and which alarm fires. Written 2026-11-10 from the space list. **The evidence is not here** — it
|
||||
is `audits/space-list-2026-11-10/` (`TABLE.md` holds each place with its `file:line` and its measurement; the place ids
|
||||
below are the pointers into it). Counts on that day: 45 places; 21 have a name in this folder, 16 a mention, 8 none.
|
||||
|
||||
**[DESIGN] The rule that keeps this true.** A change that adds a place that holds data, or changes the limit, the
|
||||
cleaner or the alarm of one, changes this section in the same session (`PROMPT-TEMPLATE.md` §N.5). A place with no
|
||||
limit is on the list in §14.2 with its ruling, or it has a register row.
|
||||
|
||||
### 14.1 The places [FACT — read and measured 2026-11-10, `audits/space-list-2026-11-10/TABLE.md`]
|
||||
|
||||
The alarms, by short name:
|
||||
|
||||
- **host disk** — the hub's `host_disk_warning` at 90 % and `host_disk_critical` at 95 %. To the operator only (D68,
|
||||
decision 287, `08-alarm-ladder.md`). That is in the hub's source and not in the hub that runs: until the next hub
|
||||
release the household can still get the critical one.
|
||||
- **pool** — the hub's `storage_fill_warning` at 85 % and `storage_fill_critical` at 90 %, on the worse of the pool's
|
||||
data and metadata. To the operator.
|
||||
- **data volume** — the controller, every 10 minutes: warning at 85 % or under 5 GiB free, critical at 95 % or under
|
||||
2 GiB. To the household.
|
||||
- **drive** — the same watch of the controller, on each registered drive. To the household.
|
||||
|
||||
**Host system disk** (`/`, the volume `pve/root`)
|
||||
|
||||
| Id | Place | Grows with | Limit | Cleaner | Alarm |
|
||||
|---|---|---|---|---|---|
|
||||
| H1 | the system disk as a whole | its parts below (72–82 % of the used bytes) | the size Proxmox gave it; no budget per part (§14.3, R-1370) | each part's own; the whole disk only by the operator's hand (`runbooks/disk-full.md`) | host disk |
|
||||
| H3 | whole-box copies on the current local target | the guest's size (R-1366) | the newest N (default 3) | Proxmox after each good run; the agent removes half-written copies | a skipped run is reported to the hub and shown on the backup page; host disk |
|
||||
| H4 | whole-box copies on a former target | a move of the target | none (R-1087) | nobody | host disk |
|
||||
| H5 | guest template archives | each install and re-install | none (R-1368) | nobody; only an uninstall with `--remove-golden` | host disk |
|
||||
| H6 | backup logs and half-written copies | each run | a failed run's log: none (R-1368) | Proxmox; the agent | host disk |
|
||||
| H7 | the host journal | the agent's `sudo` calls (R-1367) | the journald default: 10 % of the disk, at most 4 GB | journald | host disk |
|
||||
| H8 | other host logs, task logs, graphs | each task | rotation | `logrotate`, Proxmox | host disk |
|
||||
| H9 | old kernels | each approved kernel | none today. Ruled: keep the running one, the one before it and the newest (decision 286, `11-os-updates.md` §5.11.1). Proven by hand, not built | nobody today | host disk; later `os_update_stale` |
|
||||
| H10 | the EFI partition | a boot-loader update | 1.07 GB, 1 % used | none needed | none (R-1371) |
|
||||
| H11 | the host package cache | each OS update | emptied in each run | the update wrapper | host disk |
|
||||
| H12 | the agent's self-update folder | each agent update | none on a released agent; one binary after the next agent release (in source, R-1368) | the agent, after that release | host disk |
|
||||
| H13 | old agent binaries | `.prev`: one file. `.bak-*`: each re-run of the installer | none for `.bak-*` (R-1368) | nobody | host disk |
|
||||
| H14 | wrapper state and previous config bundles | each bundle | the last 3 | the wrapper | host disk |
|
||||
| H15 | host `/tmp` and `/var/tmp` | nothing that lasts | `/tmp` is in memory: half of memory | systemd, after 10 and 30 days | none for `/tmp` (R-1371) |
|
||||
| H16 | a bare folder under a drive that is not mounted | a write while the drive is away | the write is refused | nothing to clean | `storage_disconnected`, to the household |
|
||||
| H18 | swap | fixed | its volume | the kernel | `host_swap_short`, to the operator |
|
||||
|
||||
**Thin pool** (`pve/data`)
|
||||
|
||||
| Id | Place | Grows with | Limit | Cleaner | Alarm |
|
||||
|---|---|---|---|---|---|
|
||||
| H2 | the pool | each block a guest writes | the pool's size; auto-extend is off | the agent's weekly trim (decision 139); a left-over scratch guest: a retry every 10 minutes | pool |
|
||||
| H17 | the restore test's scratch guest | each restore test | it needs restored size × 1.2 + 5 GiB free, off the tested guest's pool when it can | the agent at the end; a retry every 10 minutes | pool |
|
||||
|
||||
**Guest root disk** (32 GiB; 4–5 % used on three boxes)
|
||||
|
||||
| Id | Place | Grows with | Limit | Cleaner | Alarm |
|
||||
|---|---|---|---|---|---|
|
||||
| G1 | the root disk as a whole | guest OS updates | 32 GiB; the agent can grow it, never shrink it | the update wrapper, `logrotate` | none (R-1371) |
|
||||
| G13 | app definitions, `/opt/docker/stacks` | the catalog's size | under 2 MB measured | an app removal deletes its `app.yaml` | none (R-1371) |
|
||||
| G15 | the guest's system log | each line | the journald default; it is in memory (R-1372) | journald; a restart | none |
|
||||
| G16 | the guest package cache | each guest OS update | emptied in each run | the wrapper | none (R-1371) |
|
||||
| G17 | guest `/tmp` | nothing that lasts | in memory | systemd; a restart | none |
|
||||
|
||||
**Guest data volume** (`mp0`, `/var/lib/felhom`)
|
||||
|
||||
| Id | Place | Grows with | Limit | Cleaner | Alarm |
|
||||
|---|---|---|---|---|---|
|
||||
| G2 | the data volume as a whole | G3–G12 and G14 | 50 / 100 / 130 GiB (§7; older boxes 70 or 250). A new app is refused under max(5 GB, 10 %) free; a backup is refused at 97 % or under 1 GiB | the three image rules below; the weekly trim | data volume. A box with one volume counts it twice until the next controller release: two warnings for one fill (R-1371) |
|
||||
| G3 | app images, and one undo image for each app | each install and update | a count: the current and the one before, for each app. No limit in bytes, by ruling (§14.2) | the controller | data volume |
|
||||
| G4 | old controller images | each controller update | the running one and the one before | the controller | data volume |
|
||||
| G5 | infrastructure images | a change of the pinned version | the pinned one and the one before | the controller | data volume |
|
||||
| G6 | helper images (`alpine`, `hello-world`) | pulled once | no version is pinned (R-1372) | nobody | data volume |
|
||||
| G7 | named app volumes | the app's use | none for each app; the volume's | only an app removal | data volume |
|
||||
| G8 | volumes with no name | each re-create of a container | none (R-995, R-1267) | nobody | data volume (bytes only) |
|
||||
| G9 | containers' writable layers; the off-site cache | temporary files; the off-site store's size | none of its own | a re-create drops it | data volume |
|
||||
| G10 | container logs | each line | 10 MB × 3 for each container | Docker | data volume |
|
||||
| G11 | the build cache | nothing | empty | — | data volume |
|
||||
| G12 | the controller's state volume | graph rows; the catalog | rows older than 30 days are deleted | the controller | data volume |
|
||||
| G14 | backups of apps with no drive (`sys_drive`) | dumps | one point for each app; refused at 97 % or under 1 GiB | the nightly replace; moved units are pruned | data volume |
|
||||
|
||||
**A data drive**
|
||||
|
||||
| Id | Place | Grows with | Limit | Cleaner | Alarm |
|
||||
|---|---|---|---|---|---|
|
||||
| D1 | `backups/primary` | dumps | one point for each app. A removed app's kept unit: none, by ruling (§14.2) | the nightly replace; a removal with "delete the backups too" | drive |
|
||||
| D2 | `backups/secondary` | the apps' data | it needs max(2 GB, 20 %) free. A removed app's copy: none, by ruling (§14.2) | the mirror; the household, from the backup page | drive |
|
||||
| D3 | `backups/offsite-restore` | each off-site restore or check | none (R-1372) | the household's delete button | drive |
|
||||
| D4 | `backups/offsite-proof` | one unit a night, for minutes | a headroom check | the proof itself | drive |
|
||||
| D5 | `backups/restore-undo` | each restore | 30 days unused; none under a support hold, by ruling (§14.2) | the controller's sweep | drive; the household is told on a delete |
|
||||
| D6 | `kept` | each "remove, keep my data" | none, by ruling (§14.2) | the household | drive |
|
||||
| D7 | `appdata` | the app's use | the drive | a removal "with data" | drive |
|
||||
| D8 | `userdata` | the household | none, by ruling (§14.2) | the household | drive |
|
||||
| D9 | `shares` | the household's copies | the drive | the household | drive |
|
||||
| D10 | `backups/set-aside` | each re-install that finds a unit | none, by ruling (§14.2) | the household; a removal with "delete its backups too" | drive |
|
||||
|
||||
### 14.2 What can grow without a limit, by ruling [FACT — each line is a ruling, not a defect]
|
||||
|
||||
Do not file these as defects (R-370). A change to one is a decision for the operator.
|
||||
|
||||
1. Undo images: one for each app, no limit in bytes (decision 53).
|
||||
2. The kept data of a removed app: only the household deletes it, with the app's name typed (decision 40).
|
||||
3. The household's own files (decision 67).
|
||||
4. A set-aside backup unit (decision 256).
|
||||
5. A restore's undo copy while a support hold is on it (decision 281).
|
||||
6. The second-drive copy of a removed app (D25).
|
||||
7. The backup unit of a removed app that the household kept (`07-backup-architecture.md` §6).
|
||||
8. Images in the registry (decision 62). This is not on a box.
|
||||
|
||||
### 14.3 The host system disk — a budget from measured numbers [FACT for the numbers; the answer is OPEN — D75, R-1370]
|
||||
|
||||
Measured 2026-11-10 (`audits/space-list-2026-11-10/evidence/reading-1/`, GB = 10⁹ bytes):
|
||||
|
||||
| | 128 GB disk (demo-hp) | 200 GB disk (Tester 1) | 512 GB disk (N100) |
|
||||
|---|---|---|---|
|
||||
| root volume, as Proxmox made it | 41.4 | 62.9 | 100.9 (the 96 GiB cap) |
|
||||
| used | 24.0 | 21.6 | 38.1 |
|
||||
| Proxmox, Debian and all that has no line of its own | 3.3 | 3.2 | 3.4 |
|
||||
|
||||
No rule is written here for a size between these three: the Proxmox installer's formula was not read.
|
||||
|
||||
**What the root needs when the whole-box copies go to a backup drive:**
|
||||
|
||||
| Part | GB | From |
|
||||
|---|---|---|
|
||||
| Proxmox, Debian, small parts | 4 | the last line above |
|
||||
| kernels: the three the ruling keeps, and one in install | 4.4 | one kernel = 1.02 GB of modules + 0.10 GB in `/boot` |
|
||||
| the host journal at its cap | 4.3 | measured on two boxes (H7) |
|
||||
| one guest template archive | 0.65 | H5 |
|
||||
| the agent's files, after the self-update fix | 0.1 | H12 (0.75 before the fix) |
|
||||
| free space an OS update asks for | 0.5 | `felhom-os-apply`, the free-space check |
|
||||
| **Sum** | **≈ 14** | |
|
||||
|
||||
With the parts nobody cleans today (8 kernels, 49 agent binaries, 4 templates) the same sum is about 21 GB, and it grows
|
||||
by 1.1 GB with each kernel.
|
||||
|
||||
**What local whole-box copies add.** Let C be one packed copy. The copies need keep × C. A new copy also needs
|
||||
1.25 × C + 1.07 GB free before it starts, because Proxmox prunes only after a good run (A `internal/backup/runner.go`,
|
||||
the space check). Measured C: 9.1 GB (demo-hp, 26 containers), 3.15 GB (Tester 1), 1.45 GB (N100).
|
||||
|
||||
| Keep | C | Copies + the free space the next one needs | With the 14 GB above |
|
||||
|---|---|---|---|
|
||||
| 3 | 3.15 GB | 14.5 GB | 28.5 GB |
|
||||
| 3 | 9.1 GB | 39.7 GB | 53.7 GB |
|
||||
| 1 | 9.1 GB | 21.5 GB | 35.5 GB |
|
||||
|
||||
**What follows from the numbers. This is not a decision.**
|
||||
|
||||
- **With a backup drive** the root needs about 14 GB. The 41 GB root of a 128 GB disk holds that. The limit is then the
|
||||
pool: a 128 GB disk has a 54 GiB pool, and the guest is promised 32 + 50 GiB of it (§7). That is over-promised by
|
||||
design and it is watched (the pool alarm). The first measured disk whose pool holds 32 + 50 + the 16 GiB reserve is
|
||||
the 200 GB one (113 GiB).
|
||||
- **With no backup drive** the root needs 28 to 54 GB at keep 3, by the number of apps. A 128 GB disk holds a small
|
||||
household. It does not hold demo-hp's app set at keep 3. demo-hp keeps 1, and 35.5 GB in a 41.4 GB root leaves
|
||||
little room: a copy that cannot fit is skipped before it starts (R-548). The 63 GB root of a 200 GB disk holds the
|
||||
54 GB case with 9 GB to spare.
|
||||
- The installer refuses no disk today, and nothing checks this sum at install time.
|
||||
|
||||
The question for the operator is D75 in `STATUS.md`: the smallest system disk Felhom accepts, with and without a backup
|
||||
drive.
|
||||
|
||||
### 14.4 Open design points [each has a row]
|
||||
|
||||
| Point | Row |
|
||||
|---|---|
|
||||
| Nobody watches the guest's root disk (G1, G13, G16). It is 4–5 % used today; app definitions lie on it | R-1371 |
|
||||
| No alarm for the EFI partition (H10) or for the places in memory (H15, G15, G17) | R-1371 |
|
||||
| A drive in the backup role reports as `usb` or `local-dir`. So the household gets the `storage_fill` mail for a drive it must not clean by hand | R-1371 |
|
||||
| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. Not measured on the running hub in the 2026-11-10 run | R-1371 |
|
||||
| Helper images carry no version; the guest's journal has no stated cap; off-site restores have no limit | R-1372 |
|
||||
| Old kernels: the keep rule is ruled and proven by hand, not built | `11-os-updates.md` §5.11.1 |
|
||||
| The parts of the host disk with no limit and no cleaner (H4, H5, H6, H13) | R-1087, R-1368 |
|
||||
| No budget is enforced; no smallest disk is ruled | R-1370, D75 |
|
||||
| What to do by hand when a disk is full | `runbooks/disk-full.md` (R-1369) |
|
||||
|
||||
## Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
|
||||
|
||||
Read at `file:line` on `main` by the main session:
|
||||
|
||||
@@ -435,6 +435,8 @@ stayed. **No other customer touched:** the three live boxes' repositories re-lis
|
||||
|
||||
**Gap / roadmap (as written):** Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run
|
||||
|
||||
**Noted 2026-11-10 (endings list):** „tunnel + zone removed” is not what the code does. No leg of the delete calls Cloudflare; the operator removes the tunnel and the DNS record by hand, guided by the dialog. The operator accepted that on 2026-10-05 (closed R-688). The words above stay as written.
|
||||
|
||||
---
|
||||
|
||||
<a id="a22"></a>
|
||||
|
||||
@@ -282,8 +282,8 @@ stopping line that lies.
|
||||
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
|
||||
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
|
||||
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
|
||||
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
|
||||
| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. | — | Docs: a short „disk space” section in an architecture document: each part of the system disk with its limit and its cleaner, pointing at the audit table. Then the installer can refuse a disk that cannot hold the sum. | CC |
|
||||
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
|
||||
| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. | — | Docs: a short „disk space” section in an architecture document: each part of the system disk with its limit and its cleaner, pointing at the audit table. Then the installer can refuse a disk that cannot hold the sum. | CC |
|
||||
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
|
||||
| **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC |
|
||||
| **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC |
|
||||
@@ -326,7 +326,7 @@ stopping line that lies.
|
||||
| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC |
|
||||
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
|
||||
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
|
||||
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
|
||||
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
|
||||
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
|
||||
|
||||
## Hub & operator — 16 rows (P2 1, P3 4, P4 11)
|
||||
|
||||
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-11-10 (afternoon)
|
||||
|
||||
**Updated 2026-11-10.** CC listed every place that holds data on a box — 45 places, read-only on the three test boxes — and asked eight questions of each: 47 answers were „nobody asked”, now in five new rows (R-1368…R-1372). The worst one: nothing removes old kernels from the host; a ring-0 box holds 8 of them (8 GB) and gets about one more each week. You ruled on its four questions at 12:22 (D67–D70: yes, operator only, yes, yes — `09` §3 decisions 286–290); the list is in `documentation/audits/space-list-2026-11-10/README.md`.
|
||||
|
||||
## Moved from STATUS on 2026-11-10
|
||||
|
||||
**Updated 2026-11-09.** demo-hp did not restart, because its local whole-box copy was refused for space again and the night update waited; CC freed space (old images, an old installer file), so tonight's copy fits. Controller 0.351.0 is released to every box (floor 12:18 UTC): it deletes old test builds and old system images by itself. Two new rows: the whole-box copy is mostly app programs (your decision, R-1366), and the host log noise (R-1367).
|
||||
|
||||
@@ -525,8 +525,9 @@ recovery credential).
|
||||
|
||||
**Expected remnants** (documented, not residue):
|
||||
|
||||
- The **hub host record** (+ its report/guest history) — the hub currently has **no host-delete**;
|
||||
removing it is a direct-DB operator action (recorded follow-up).
|
||||
- The **hub host record** (+ its report/guest history) — the operator removes it with the host's delete action in
|
||||
the hub (`POST /hosts/{id}/delete`; `architecture/05-hub-architecture.md`, the host-delete table). *(Corrected
|
||||
2026-11-10: this line said the hub had no host delete.)*
|
||||
- The `sudo` and `dnsmasq` packages (system packages installed as dependencies; inert).
|
||||
- The **golden vzdump** on the archive storage — pass `--remove-golden` to delete it too.
|
||||
- System noise: LVM archive metadata, dpkg lists, PVE's own rrd/`/var/lib/lxc` scratch entries.
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
# RUNBOOK — a disk on a box is full, or nearly full (R-1369)
|
||||
|
||||
**When:** a space alarm came (`host_disk_*`, `storage_fill_*`, the controller's fill warning), a whole-box copy was
|
||||
skipped for space, or an update was refused for space.
|
||||
|
||||
**What this is:** one short part for each disk. Each part says how to see what is big, what is safe to remove and by
|
||||
which path, what a hand must never remove, and what the product does by itself. **The design is not here**: each place
|
||||
with its limit, cleaner and alarm is `architecture/16-storage-physical-identity.md` §14. The place ids (H1, G2, D6 …)
|
||||
are the same there and in `audits/space-list-2026-11-10/TABLE.md`.
|
||||
|
||||
**Each hand step carries a mark.** `PROVEN` = a person or CC did exactly this on a box, with the date and the
|
||||
evidence. `NOT PROVEN` = nobody did it; it is not a step yet. **Every `PROVEN` step below was proven on demo-hp, a test
|
||||
box.** None was done on a household's box.
|
||||
|
||||
**Before any hand step, on any disk:**
|
||||
|
||||
1. Read which box it is and what you may do there: `runbooks/target-selection.md`.
|
||||
2. Check that no backup, restore, restore test or update runs: `pct list` shows no lock; `ps -ef | grep -E
|
||||
'vzdump|pct restore|apt'` shows nothing.
|
||||
3. Save what you see before you remove anything (R-320): the `df`, the list, the file's checksum.
|
||||
4. On a household's box a hand removes nothing without the operator's word.
|
||||
|
||||
---
|
||||
|
||||
## 1. Host system disk (`/`)
|
||||
|
||||
**See what is big (read-only):**
|
||||
|
||||
```bash
|
||||
df -h /
|
||||
du -xsh /var/lib/vz/dump /var/lib/vz/template /var/log/journal /lib/modules /boot /var/lib/felhom-agent/selfupdate
|
||||
journalctl --disk-usage
|
||||
pvesm list local
|
||||
dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii
|
||||
```
|
||||
|
||||
**The product does by itself:** it empties the package cache in each OS update. It keeps the newest N whole-box copies
|
||||
(default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS
|
||||
update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The
|
||||
hub alarms at 90 % and 95 %.
|
||||
|
||||
**Safe to remove, by which path:**
|
||||
|
||||
| What | Step | Mark |
|
||||
|---|---|---|
|
||||
| Old kernels | `ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). **NOT PROVEN:** on a second box; a boot of the kept older kernel |
|
||||
| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) |
|
||||
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free <volid>`, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only |
|
||||
| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |
|
||||
| Old binaries in the agent's self-update folder (H12) | — | **NOT PROVEN.** The agent cleans the folder after the release that carries the fix (R-1368) |
|
||||
| Extra guest template archives (H5); `/usr/local/bin/felhom-agent.bak-*` (H13) | — | **NOT PROVEN** (R-1368) |
|
||||
|
||||
**Never by hand:**
|
||||
|
||||
- The running kernel and the one before it. Never a kernel with a plain `apt autoremove` or by a meta-package: `apt`
|
||||
did not refuse the running kernel, and one meta-package pulled `proxmox-ve` with it
|
||||
(`audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt`).
|
||||
- `pending.json` in the self-update folder, and `/usr/local/bin/felhom-agent.prev`. The agent's rollback needs both.
|
||||
- The newest whole-box copy.
|
||||
- Anything under `/var/lib/felhom-os-apply/`: the wrapper's state and the last config bundles.
|
||||
|
||||
## 2. Thin pool (`pve/data`)
|
||||
|
||||
**See what is big (read-only):**
|
||||
|
||||
```bash
|
||||
lvs -o lv_name,lv_size,data_percent,metadata_percent pve
|
||||
pct list
|
||||
```
|
||||
|
||||
**The product does by itself:** the agent trims each guest once a week (decision 139). It removes a restore test's
|
||||
scratch guest at the end and tries again every 10 minutes when that fails. A restore test starts only with enough free
|
||||
space. The hub alarms at 85 % and 90 % on the worse of data and metadata.
|
||||
|
||||
**At 100 %** the guest's disks go read-only. A write in progress can be cut off: on 2026-09-24 two apps' Redis files
|
||||
were (`architecture/03-host-agent.md` §8, the R-672 note; `audits/r672-2026-09-24/README.md`).
|
||||
|
||||
| What | Step | Mark |
|
||||
|---|---|---|
|
||||
| Space the guest freed but the pool still holds | `pct fstrim <vmid>` | **PROVEN 2026-10-06**: pool 65.5 % → 33.4 % in 24 s, the box answered all the time (`audits/ten-answers-2026-10-06/r444-measure.txt`). The product does this weekly |
|
||||
| A guest that is read-only after a full pool | Free pool space first. Stop the guest. `pct fsck <vmid>`, then `pct fsck <vmid> --device mp0`. Start it. Check each app | **PROVEN once, 2026-09-24** (`audits/r672-2026-09-24/README.md`, Part B; `B4-fsck.txt`). Two Redis files then needed `redis-check-aof --fix`, with the operator's yes and a copy kept first |
|
||||
| A left-over scratch guest or a thin device with no volume | `runbooks/whole-guest-backup-orphan-thin.md` | see that runbook |
|
||||
|
||||
**Never by hand:** `lvremove` of a `vm-9201-disk-*` volume. `lvextend` of the pool into the space that belongs to the
|
||||
internal place. A thin device, except by the runbook above.
|
||||
|
||||
## 3. Guest root disk (32 GiB)
|
||||
|
||||
**See what is big (read-only):**
|
||||
|
||||
```bash
|
||||
pct exec <vmid> -- df -h /
|
||||
pct exec <vmid> -- du -xsh /var /opt /usr /root
|
||||
```
|
||||
|
||||
**The product does by itself:** it empties the guest's package cache in each guest OS update. Nothing else. **Nobody
|
||||
watches this disk** (R-1371). It was 4–5 % used on three boxes on 2026-11-10.
|
||||
|
||||
**Safe to remove:** nothing is proven. **NOT PROVEN** for every hand step on this disk. When it is full, the agent can
|
||||
grow it; that path is the product's, not a hand's.
|
||||
|
||||
**Never by hand:** `/opt/docker/stacks/<app>/app.yaml`. It holds the app's settings and its records.
|
||||
|
||||
## 4. Guest data volume (`/var/lib/felhom`)
|
||||
|
||||
**See what is big (read-only):**
|
||||
|
||||
```bash
|
||||
pct exec <vmid> -- df -h /var/lib/felhom
|
||||
pct exec <vmid> -- docker system df
|
||||
pct exec <vmid> -- docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}'
|
||||
```
|
||||
|
||||
Never dump a container's `Env`.
|
||||
|
||||
**The product does by itself:** it keeps the current image and one undo image for each app, and removes older ones. It
|
||||
keeps the running controller image and the one before; the same for infrastructure images. Docker caps each container's
|
||||
log at 10 MB × 3. It deletes graph rows older than 30 days. It refuses a new app under max(5 GB, 10 %) free. It refuses
|
||||
a backup at 97 % or under 1 GiB, app by app, and deletes nothing. It warns the household at 85 % and 95 %.
|
||||
|
||||
**Safe to remove, by which path:**
|
||||
|
||||
| What | Step | Mark |
|
||||
|---|---|---|
|
||||
| An app the household does not want | The household removes it on the app's page. That is the only path that frees an app's volume | product path |
|
||||
| Old controller, infrastructure and helper images that no container uses | `docker image rm <name:tag>`, one exact name at a time, after `docker ps -a` shows no container on it | **PROVEN 2026-11-09**: 36 images, 3.4 GB (off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/cleanup.txt`). The controller now removes old controller and infrastructure images by itself. Helper images stay (R-1372) |
|
||||
| Volumes with no name (G8) | — | **NOT PROVEN** (R-995, R-1267) |
|
||||
|
||||
**Never by hand:** `docker system prune`, `docker volume prune`, `docker image prune -a`. The undo images are kept on
|
||||
purpose (decision 53), and volumes hold the apps' data. An app's undo image. A named volume.
|
||||
|
||||
## 5. A data drive
|
||||
|
||||
**See what is big:** sizes of the top folders only. Do not list file names below a household's data.
|
||||
|
||||
```bash
|
||||
pct exec <vmid> -- df -h <mount point>
|
||||
pct exec <vmid> -- du -xsh <mount point>/* <mount point>/backups/*
|
||||
```
|
||||
|
||||
**The product does by itself:** the nightly backup replaces each app's one point. The second-drive copy needs
|
||||
max(2 GB, 20 %) free. The controller deletes a restore's undo copy after 30 days without use, and tells the household.
|
||||
The off-site proof cleans its own scratch. The controller warns the household at 85 % and 95 %.
|
||||
|
||||
**Safe to remove:** only what the household removes on the product's pages: the kept data of a removed app (with the
|
||||
app's name typed), a set-aside unit, an off-site restore, its own files. **A hand step on a data drive: NOT PROVEN, and
|
||||
not wanted.**
|
||||
|
||||
**Never by hand:** `userdata`, `appdata`, `kept`, `shares`, `backups/primary`, `backups/set-aside`,
|
||||
`backups/restore-undo`. These grow without a limit by ruling (`16` §14.2; decisions 40, 67, 256, 281). Only the
|
||||
household deletes them.
|
||||
|
||||
---
|
||||
|
||||
**After any hand step:** read `df` again, save the before and after to the evidence folder, and check the box on the
|
||||
hub. Then write a dated line on the row that names the place.
|
||||
@@ -81,7 +81,7 @@ Provision a new ep0 (06 §5), then on it add DooPlex as a remote and run
|
||||
|
||||
Do **not** prune the copy tighter than decision 71 (8 weekly copies); never tighter than ep0's own retention.
|
||||
|
||||
## Removing a deleted customer's copy (R-901, `09` §3 decision 181) — WRITTEN 2026-10-08, NOT INSTALLED
|
||||
## Removing a deleted customer's copy (R-901, `09` §3 decision 181) — WRITTEN 2026-10-08, INSTALLED 2026-10-09
|
||||
|
||||
The sync keeps what ep0 removed (`remove-vanished false`), so a deleted customer's namespace stayed here for ever.
|
||||
Ruling: removed **within 30 days**. The job `scripts/ep0-copy-gc/felhom-ep0-copy-gc` (tests beside it) runs daily at
|
||||
@@ -92,7 +92,9 @@ nothing; if more than 2 namespaces are absent at once (a partial list, e.g. a re
|
||||
deletes at most one namespace per run; a HOLD file (`/etc/felhom/ep0-copy-gc/HOLD`) stops it. Without `--apply` it
|
||||
only prints. Worst case: ≤ 1 day to notice + 7 days grace + 1 day = inside 30 days.
|
||||
|
||||
**Not installed. To install (needs the operator's word — it is a change on DooPlex and it deletes data):**
|
||||
**Installed 2026-10-09 with the operator present (closed R-901: key root-only, a dry run, then the 08:00 timer). Not
|
||||
re-checked on DooPlex on 2026-11-10, when this heading was corrected from „NOT INSTALLED”. The steps, for an install
|
||||
from nothing (needs the operator's word — it is a change on DooPlex and it deletes data):**
|
||||
1. Two tokens, secrets in root-only files under `/etc/felhom/ep0-copy-gc/` (0600), never printed:
|
||||
- `ep0-reader.secret` — the existing ep0 read token `root@pam!dooplex-sync` (its secret is in `remote.cfg`, base64);
|
||||
`ep0.fingerprint` — ep0's pinned certificate fingerprint (also in `remote.cfg`).
|
||||
|
||||
Reference in new issue
Block a user