Space list (pilot of the gap review): 45 places, eight questions each
gates / gates (push) Successful in 1m58s

Read-only measurement on demo-hp, demo-felhom and Tester 1, two readings.
Audit folder documentation/audits/space-list-2026-11-10/ (README, TABLE,
MATRIX, evidence, tools). 360 answers: A 294, D 0, K 19, N 47.

Register 225 -> 230: R-1368 (host places that only grow: kernels, agent
binaries), R-1369 (no written way out of a full disk), R-1370 (no space
budget for the host disk), R-1371 (no space alarm; host-disk mail goes to
the household), R-1372 (small guest places). A dated line on nine existing
rows. Section headings set to the real counts; R-1268 and R-1312 moved to
their category's section.

STATUS: new Updated paragraph, decisions D67-D70. N100 diary line.
Two stale facts corrected in 05-hub-architecture.md and 11-os-updates.md.
Nothing was changed on any machine.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-10 10:50:44 +01:00
1 parent a6705ceb04
commit 47e372a5f4
56 files changed
+7631 -20

No files matched your search

@@ -613,7 +613,8 @@ design; its mechanics hold per slot. Originally: every night at **02:00 Budapest
one statement, one point in time, every committed write included — the rows still in `hub.db-wal` too, which a file copy
of `hub.db` loses (measured in `internal/dbsnap` tests: a plain copy held 0 of 150 fresh rows). Written as `.tmp`, mode
`0600`, then renamed, so a reader never sees half a file. The newest **2** stay (the volume grew from 1 GiB to 2 GiB
for them, operator choice 2026-10-05; one snapshot measured 370 MB). Two runs never overlap (a second gets `ErrBusy`).
for them, operator choice 2026-10-05; one snapshot measured 370 MB; **4 GiB since 2026-10-07**, `manifests/hub.yaml`
`storage: 4Gi` — corrected 2026-11-10). Two runs never overlap (a second gets `ErrBusy`).
Each run logs `db snapshot written: <name> (<bytes>, <duration>)`. At start-up the hub runs one when the newest is older
than 24 h. **The hub ships nothing itself and holds no ep0 credential:** DooPlex's `felhom-hub-db-backup` unit picks the
newest snapshot up at 02:30, checks it (`PRAGMA integrity_check`), encrypts it and pushes it to ep0's `operator`
+1 -1
View File
@@ -104,7 +104,7 @@ boxes installed. Nobody keeps a hand-written list.
| Host kernel | The only kernel on the box. The guest has none. | `pve-no-subscription` (`proxmox-kernel-*`) | `[FACT]` LXC shares the host kernel (`01` §2: "one LXC/kernel/Docker daemon") |
| Guest | Unprivileged LXC, `nesting=1,keyctl=1`, from `debian-13-standard_13.1-2` | Debian mirrors | `[FACT]` `felhom-agent/configs/build-golden.sh:67,101-103` |
| Docker engine | `docker-ce`, `docker-ce-cli`, `containerd.io`, installed at golden BAKE time | `download.docker.com/linux/debian trixie stable` | `[FACT]` `build-golden.sh:108-125` |
| Docker settings | `containerd-snapshotter: false`, json-file log caps. **No `live-restore`.** | baked `daemon.json` | `[FACT]` `build-golden.sh:138-144` |
| Docker settings | `containerd-snapshotter: false`, json-file log caps (10 MB × 3 files), **`live-restore: true`** since the 3.1.0 bake (line 429 below; this cell said „No `live-restore`” until 2026-11-10). | baked `daemon.json` | `[FACT]` `build-golden.sh:193-203` |
| Controller | A container in the guest. A Docker engine restart restarts it too. | Felhom registry | `[FACT]` `03` §1 |
**[FACT] Nothing updates any of these layers today** (R-812, searched 2026-10-03). The installer
@@ -0,0 +1,93 @@
# The Space list — the matrix
Made by `tools/make-matrix.py` from `TABLE.md`. Do not edit it by hand.
`A` answered · `D` decided, not built · `K` known gap (has a row) · `N` nobody asked. Columns are the eight
questions: 1 content · 2 growth · 3 limit · 4 cleaner · 5 when full · 6 alarm · 7 repair or runbook · 8 how we know.
| Place | Named | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | N |
|---|---|---|---|---|---|---|---|---|---|---|
| H1 Host system disk, root filesystem `/` (LV `pve/root`) | named | A | A | **N** | K | A | A | **N** | A | 2 |
| H2 Thin pool `pve/data` (data and metadata) | named | A | A | A | A | A | A | **N** | A | 1 |
| H3 Whole-box copies on the current local target | named | A | K | A | A | K | A | K | A | 0 |
| H4 Whole-box copies on a former target | unnamed | A | A | K | K | K | A | K | A | 0 |
| H5 Guest template archives ("golden") in the dump folder | unnamed | A | A | **N** | **N** | A | A | **N** | A | 3 |
| H6 Backup logs and half-written copies in the dump folder | mentioned | A | A | **N** | A | A | A | A | A | 1 |
| H7 Host journal `/var/log/journal` | mentioned | A | K | K | A | K | A | K | A | 0 |
| H8 Other host logs, Proxmox task logs, graph data | mentioned | A | A | A | A | A | A | A | A | 0 |
| H9 Old kernels: `/lib/modules` and `/boot` | mentioned | A | A | **N** | **N** | A | A | **N** | **N** | 4 |
| H10 EFI partition `/boot/efi` | mentioned | A | A | A | A | **N** | **N** | **N** | A | 3 |
| H11 Host package cache and package lists | named | A | A | A | A | A | A | A | A | 0 |
| H12 The agent's self-update folder `/var/lib/felhom-agent/selfupdate` | unnamed | A | A | **N** | **N** | A | A | **N** | A | 3 |
| H13 Old copies of the agent binary (`.prev`, `.bak-*`, `.pre-bundle`) | mentioned | A | A | **N** | **N** | A | A | **N** | A | 3 |
| H14 Update wrapper state and previous signed bundles | mentioned | A | A | A | A | A | A | A | A | 0 |
| H15 Host `/tmp` and `/var/tmp` | unnamed | A | A | A | A | **N** | **N** | A | A | 2 |
| H16 The bare folder under an unmounted drive | named | A | A | A | A | A | A | A | A | 0 |
| H17 The restore-test's scratch guest | named | A | A | A | A | A | A | A | A | 0 |
| H18 Swap (LV `pve/swap`) | named | A | A | A | A | A | A | A | A | 0 |
| G1 Guest root disk (`vm-9201-disk-0`, 32 GiB) | mentioned | A | A | A | A | A | **N** | **N** | A | 2 |
| G2 Guest data volume (`mp0`, `/var/lib/felhom`: Docker and `sys_drive`) | named | A | A | A | A | A | A | **N** | A | 1 |
| G3 App images, with one undo image per app | named | A | A | A | A | A | A | A | A | 0 |
| G4 Old controller images | named | A | A | A | A | A | A | A | A | 0 |
| G5 Infrastructure images (traefik and the other pinned helpers of the box) | named | A | A | A | A | A | A | A | A | 0 |
| G6 Helper images (`alpine`, `hello-world`) | mentioned | A | A | **N** | **N** | A | A | A | A | 2 |
| G7 Named app volumes | named | A | A | A | A | A | A | **N** | A | 1 |
| G8 Unnamed (anonymous) volumes | named | A | K | K | K | K | A | K | A | 0 |
| G9 Container writable layers, with the controller's off-site cache | unnamed | A | A | A | A | A | A | A | A | 0 |
| G10 Container logs | mentioned | A | A | A | A | A | A | **N** | A | 1 |
| G11 Docker build cache | unnamed | A | A | A | A | A | A | A | A | 0 |
| G12 The controller's own state volume | mentioned | A | A | A | A | A | A | **N** | A | 1 |
| G13 App definitions `/opt/docker/stacks` | mentioned | A | A | A | A | A | **N** | **N** | A | 2 |
| G14 Backups of apps with no drive (`sys_drive`) | named | A | A | A | A | A | A | **N** | A | 1 |
| G15 Guest system log | unnamed | A | A | **N** | A | A | **N** | A | A | 2 |
| G16 Guest package cache and lists | mentioned | A | A | A | A | A | **N** | A | A | 1 |
| G17 Guest `/tmp` and `/var/tmp` | unnamed | A | A | A | A | **N** | **N** | A | A | 2 |
| D1 `backups/primary` — the app's own backup unit | named | A | A | A | A | A | A | **N** | A | 1 |
| D2 `backups/secondary` — the copy on a second drive | named | A | A | A | A | A | A | **N** | A | 1 |
| D3 `backups/offsite-restore` — copies fetched from the off-site store | mentioned | A | A | **N** | A | A | A | **N** | A | 2 |
| D4 `backups/offsite-proof` — the nightly proof's scratch | mentioned | A | A | A | A | A | A | A | A | 0 |
| D5 `backups/restore-undo` — what a restore replaced | named | A | A | A | K | A | A | A | A | 0 |
| D6 `kept` — data kept at "remove the app, keep my data" | named | A | A | A | A | A | A | **N** | A | 1 |
| D7 `appdata` — the apps' files on the drive | named | A | A | A | A | A | A | **N** | A | 1 |
| D8 `userdata` — the household's own folders | named | A | A | A | K | A | A | **N** | A | 1 |
| D9 `shares` — folders shared on the home network | mentioned | A | A | A | A | A | A | **N** | A | 1 |
| D10 `backups/set-aside` — the backup of an earlier install of the same app | named | A | A | A | A | A | A | **N** | A | 1 |
## Totals
- Places: **45** customer places · named **21** · mentioned only **16** · named by no architecture document **8**.
- Answers: **360** · A **294** · D **0** · K **19** · N **47**.
- Places with no N: **17** of 45.
| Question | A | D | K | N |
|---|---|---|---|---|
| 1 what is in it | 45 | 0 | 0 | 0 |
| 2 what makes it grow | 42 | 0 | 3 | 0 |
| 3 the limit | 33 | 0 | 3 | 9 |
| 4 who cleans up | 35 | 0 | 5 | 5 |
| 5 what happens when full | 38 | 0 | 4 | 3 |
| 6 who gets an alarm | 38 | 0 | 0 | 7 |
| 7 self-repair or runbook | 19 | 0 | 4 | 22 |
| 8 how do we know | 44 | 0 | 0 | 1 |
## The N answers, by class row
| Class | Row | Count | Members (place-question) |
|---|---|---|---|
| C1 | R-1368 | 10 | H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4 |
| C2 | R-1369 | 24 | H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7 |
| C3 | R-1370 | 1 | H1-3 |
| C4 | R-1371 | 7 | H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6 |
| C5 | R-1372 | 5 | G6-3, G6-4, G10-7, G15-3, D3-3 |
## The K answers, by existing row
| Row | Count | Members |
|---|---|---|
| R-548 | 3 | H1-4, H3-5, H3-7 |
| R-995 | 3 | G8-4, G8-7, D8-4 |
| R-1087 | 4 | H4-3, H4-4, H4-5, H4-7 |
| R-1267 | 3 | G8-2, G8-3, G8-5 |
| R-1268 | 1 | D5-4 |
| R-1366 | 1 | H3-2 |
| R-1367 | 4 | H7-2, H7-3, H7-5, H7-7 |
@@ -0,0 +1,93 @@
# The Space list — 2026-11-10 (pilot of the gap review)
**What this is.** Every place that holds data on a Felhom box, with eight questions for each place. Measured
read-only on demo-hp, demo-felhom (N100) and Tester 1. Nothing was changed on any machine.
Facts: `TABLE.md` · marks: `MATRIX.md` · proof: `evidence/` · scripts: `tools/`.
Baselines: felhom.eu `a6705ceb04` · controller `da060d7e7f` · agent `41d5eb8051` · catalog `f4c5c797d9`.
**Wrong claims in the prompt (named first).** The prompt calls `alpine:3.20`, `curl` and `hello-world` the
product's helper images. The product's helper is `alpine` with no tag. `alpine:3.20` and `curl` came from test
tools. `hello-world` comes from the template bake. One closed register row is wrong too: R-521 says the guest
root disk has a fill watcher. It has none since the two guest volumes became one.
## 1. Five lines
1. **Places:** 45 customer places (18 host, 17 guest, 10 data drive) · 7 test-only · 6 off-box, from documents only.
2. **No document named:** 8 of the 45 (H4, H5, H12, H15, G9, G11, G15, G17). 16 more are named only in passing.
3. **Marks, 360 answers:** A 294 · D 0 · K 19 · N 47. The 47 N fall into 5 classes = 5 new rows.
4. **Worst N:** nobody removes old kernels (H9). A ring-0 box holds 8 kernels = 8.15 GB and gets about one more
each week. The update wrapper refuses to remove a package, and nothing else does.
5. **You decide:** four things — section 5.
## 2. The N answers, by class (one register row each; members in `MATRIX.md`)
- **C1 = R-1368 (10) — host places that only grow:** old kernels, the agent's self-update folder (49 binaries,
740 MB), template archives, old agent binary copies, logs of failed backup runs.
- **C2 = R-1369 (24) — a disk is full and there is no written way out.** One missing runbook, seen from 21 places.
- **C3 = R-1370 (1) — the host system disk has no space budget.**
- **C4 = R-1371 (7) — no space alarm:** the guest root disk has no watcher (three places lie on it); four small
places. The row also holds two findings that are not N: the host-disk critical mail goes to the household, and
one filesystem is watched twice.
- **C5 = R-1372 (5) — small guest places nobody described:** helper images, off-site restore copies with no age
limit (1.65 GB of two removed apps on the N100), log caps never re-checked, guest log in memory.
## 3. Things that can grow without a limit, by ruling (not defects)
Undo images: a count per app, no byte limit (decision 53). Kept data of a removed app (decision 40). The household's
own files stay on every removal (decision 67). A set-aside unit, until the household removes it (decision 256). A
restore's undo copy under a support hold (decision 281). The second-drive copy of a removed app (D25). The backup
unit of a removed app that the household kept (`07-backup-architecture.md:380`). Off-box, from documents only: one image per
version for ever in the registry, pruned by hand (decision 62).
## 4. The four questions
1. **Helper images.** The product pulls `alpine` with no tag at seven call sites (first:
`felhom-controller/controller/internal/backup/backup.go:988`). The bake pulls `hello-world`
(`felhom-agent/configs/build-golden.sh:232`). None of the three image rules (decisions 53, 56, 285) can delete
a helper image (`internal/stacks/image_retention.go:244-250`). They are small: 8.4 MB, 4 months old on two boxes.
2. **Container log caps.** 10 MB × 3 files on all three boxes, for every container; no container escapes the cap.
Worst case: demo-hp 780 MB · N100 270 MB · Tester 1 300 MB. The cap is written once at the bake; nothing
checks it later.
3. **The host's own disk.** Yes, the hub warns: host system disk at 90 % and 95 %; thin pool at 85 % and 90 %
(the worse of data and metadata). The mail goes to you; the critical one also to the household. Nothing
refuses or repairs. No architecture document names the host-disk alarm.
4. **Kernels and boot.** Nothing removes old kernels. `/boot` is a folder on the system disk: 0.82 / 0.94 /
0.33 GB. The EFI partition is 1 % full on each box. Kernels: 8 / 8 / 3 = 8.15 / 8.15 / 3.05 GB.
**System logs, two readings (07:39 and 09:45 UTC, 2 h 6 min apart; `evidence/reading-1/`, `evidence/reading-2/`).**
demo-hp host journal: 873.7 → 898.9 MB = about 12 MB an hour. The journal grows in 8 MiB steps, so two hours give
only a rough rate (190–380 MB a day). It was 98 % of all growth on that system disk in the window (+25.7 MB).
The N100 and Tester 1 sit at the journal's cap (4.30 GB each); their oldest entries did not move. Lines in the last
hour: 13,521 / 13,506 / 11,177, and `sudo` wrote 95 / 94 / 93 % of them. Guest system log: 8 MB in memory, both
readings. Container logs on demo-hp: +5.7 MB (2.7 MB an hour, under the 780 MB worst case). Unnamed volumes: no
change in two hours; they come at night (about 29 a day on demo-hp).
## 5. Decisions for you (also in `STATUS.md`)
| # | Question | Options and cost | CC's pick | If you do nothing |
|---|---|---|---|---|
| D67 | May the nightly update remove old kernels by itself? | **Yes:** keep the running one, the one before and the newest; it widens rule R4; one agent release. **No:** you remove them by hand on each box. | **Yes**, ring 0 first | a ring-0 box gains about 1 GB a week |
| D68 | Who gets the "host disk critical" mail? | **Operator only:** one hub change. **Keep:** the household gets a mail it cannot act on. | **Operator only** | the household is mailed about a disk only you can free |
| D69 | Write a short "disk space" section in an architecture document? | **Yes:** about one session; each place gets a limit and a cleaner. **No:** this table stays the only reference and goes stale. | **Yes** | 8 places stay unnamed; new ones appear unseen |
| D70 | Run the next list ("Endings") with this method? | **Yes, with the three changes below. No:** stop the gap review here. | **Yes** | the plan stops after its pilot |
## 6. The method's verdict
**It found real things nobody had asked about.** New: kernels and agent binaries are never removed; the guest root
disk has no watcher and a closed row said it had; a household gets the host-disk mail; off-site restore copies have
no age limit. Four of the five new rows hold a measured fact, not an opinion.
**Cost:** about 1.5 hours of work and a 1.5-hour wait for the second reading (the plan said 4–5 hours). **Noise:** the 47 N answers hold about 14 separate gaps. Question 7 gave the same
answer 21 times: that is one gap, not 21. Some places are too small to matter (temp folders, the EFI partition).
**No `D` answer** was found among 45 places: what was decided here was also built.
**Change three things for "Endings":** (1) ask question 7 once for each disk, not for each place; (2) drop
places under 100 MB that cannot grow, in the first hour; (3) a helper's answer does not go into the table until
the main session has read its pointer — the off-box places here are still candidates for that reason.
**Weak spot:** the off-box places (ep0, DooPlex, the off-site box, the hub database) were not measured, and the
Prometheus rules in `homelab-manifests` were not read. That is the second pass.
## For a later session (small fixes, not done today — no code change in this run)
Keep only the newest N files in the agent's self-update folder · watch the guest root disk and de-duplicate
fill-watch targets by filesystem (`cmd/controller/main.go:2694-2707` has a stale comment) · pin the `alpine`
helper · a stale document, on a protected machine's procedure, so not edited here: `runbooks/ep0-datastore-copy.md:85`
says "NOT INSTALLED" (closed row R-901: installed 2026-10-09).
@@ -0,0 +1,729 @@
# The Space list — every place that holds data, eight questions each
Measured 2026-11-10, first reading 07:38–07:39 UTC, second reading at the end of the session
(`README.md` gives the rates). Read-only. Sizes are allocated bytes; 1 GB = 10⁹ bytes.
Boxes: **hp** = demo-hp (128 GB system disk), **N100** = demo-felhom (512 GB, lived-in), **T1** = Tester 1
(a VM, 200 GB). Guest = LXC 9201 on each.
**The eight questions:** 1 what is in it, who owns it · 2 what makes it grow · 3 the limit · 4 who cleans up,
when · 5 what happens at the limit; is household data safe · 6 who gets an alarm, before or after · 7 does it
repair itself; if not, is there a runbook · 8 how do we know.
**Marks:** `A` answered, with a pointer · `D` decided, not built · `K` known gap, with its row ·
`N` nobody asked — it survived the searches in `EV/searches.txt` (S1…S15; controls in the same file).
Every `N` names its class row: **C1** = R-1368 · **C2** = R-1369 · **C3** = R-1370 · **C4** = R-1371 ·
**C5** = R-1372.
**Pointers:** `AG/` = `felhom-agent/` · `CT/` = `felhom-controller/controller/` · `HUB/` =
`felhom.eu/hub/internal/` · `E/` = `felhom.eu/` · `AR/` = `felhom.eu/documentation/architecture/` · `RB/` =
`felhom.eu/documentation/runbooks/` · `EV/` = this folder's `evidence/`. `tools/check-pointers.py` prints the
text behind each one. Source lines are as of the baselines in `README.md`.
**Named status** (after the place name): `NAMED` = an architecture document describes the place ·
`MENTIONED` = a document names it in passing, with no limit or cleaner · `UNNAMED` = no architecture document
names it.
**A place inside a bigger place** (a folder on the host system disk, on the guest data volume or on a data
drive) takes question 5 and 6 from the bigger place when it has no limit of its own. Question 7 is `N` for
each place that has no cleaner and no runbook — it is ONE missing runbook seen from many places (C2).
---
## Host (Proxmox) — 18 places
### H1 — Host system disk, root filesystem `/` (LV `pve/root`) · CUSTOMER · NAMED
`AR/07-backup-architecture.md:467`, `AR/16-storage-physical-identity.md:146-150`. Used: hp 23.97 of 41.42 GB (58 %) ·
N100 38.09 of 100.86 GB (38 %) · T1 21.64 of 62.90 GB (34 %). Since 2026-10-15: N100 +4.6 GiB, T1 +5.8 GiB (26 days).
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Proxmox, kernels, the host journal, the whole-box copies when the target is `local`, the agent's files. The operator owns it. | A | `EV/reading-1/hp-host.tsv` |
| 2 | H3, H4, H5, H7, H9, H12 below. They hold 72 % (hp), 82 % (N100), 80 % (T1) of the used bytes. | A | `EV/reading-1/n100-host.tsv` |
| 3 | The LV size the Proxmox installer made. No budget says which part may use how much of it. | N | C3 · S5 |
| 4 | Each part by its own rule, or nobody (H5, H9, H12, H13). The whole disk: the operator, by hand, last on 2026-11-07. | K | R-548 |
| 5 | Under 500 MB free the OS update is refused; a local whole-box copy that cannot fit is skipped; LVM's own metadata writes can fail. Household data is not on this disk. | A | `AG/configs/felhom-os-apply:1390-1392`, `AG/internal/backup/runner.go:276-287`, `AG/internal/vzpartial/vzpartial.go:1-12` |
| 6 | The hub: `host_disk_warning` at 90 %, `host_disk_critical` at 95 % — before it is full. To the operator; the critical one also to the household by default. No architecture document names this alarm or its reader (C4, S4; the alarm ladder names only the thin-pool one). | A | `HUB/monitor/host_disk.go:41-42`, `HUB/claim/engine.go:229-243` |
| 7 | No self-repair. No runbook for a full host system disk. | N | C2 · S6 |
| 8 | TEST and MEASURED. | A | `HUB/monitor/host_disk_test.go:40`, `EV/reading-1/t1-host.tsv` |
### H2 — Thin pool `pve/data` (data and metadata) · CUSTOMER · NAMED
`AR/08-alarm-ladder.md:298-306`, `AR/16-storage-physical-identity.md:126-150`. Data used: hp 40.88 of 57.84 GB (70.67 %) ·
N100 3.03 % · T1 13.10 %. Metadata: 2.85 % · 0.58 % · 1.84 %. Promised to the guest: hp 102 GiB on a 53.9 GiB pool ·
N100 282 on 349 · T1 102 on 113. hp was 50.75 % on 2026-10-15.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The guest's root disk and data volume; for a short time the restore-test's scratch guest. | A | `EV/raw/hp-host.txt` |
| 2 | Each block the guest writes. A deleted block comes back only at the weekly trim. | A | `AG/internal/fstrim/fstrim.go:1-12` |
| 3 | The pool size. Auto-extend is off. The sizing rule promises nothing beyond pool − reserve, but on a 128 GB disk the two guest disks alone (32 + 50 or 70 GiB) are larger than the 54 GiB pool. This is written down. | A | `AR/16-storage-physical-identity.md:146-150`, `E/documentation/audits/storage-identity-buildB-2026-10-15/SPACE-CHECK.md:41-42`, `EV/os-cleaners/hp-host.txt` |
| 4 | The agent trims the guest each week (decision 139). A scratch guest that stays is removed by a 10-minute retry. | A | `AG/internal/fstrim/fstrim.go:1-12`, `AG/internal/reconcile/restoretest_retry.go:5-12` |
| 5 | At 100 % the guest's disks go read-only and every app stops. Measured on demo-hp on 2026-09-24. Data already written stays. | A | `AR/03-host-agent.md:513-520` |
| 6 | The hub: `storage_fill_warning` at 85 %, `storage_fill_critical` at 90 %, the worse of data and metadata. Operator; the critical one also to the household by default. On hp the guest's own warning (85 % of 70 GiB) lies above the pool's real size, so this is the only alarm that can come first. | A | `AR/08-alarm-ladder.md:300-307`, `HUB/monitor/storage_fill.go:36-57`, `HUB/claim/engine.go:229-243` |
| 7 | No self-repair. The one pool runbook covers a different case (an orphan snapshot volume). No runbook for a full pool. | N | C2 · S6 · `RB/whole-guest-backup-orphan-thin.md:1-10` |
| 8 | TEST and MEASURED. | A | `HUB/monitor/storage_fill_test.go:34`, `EV/reading-1/hp-host.tsv` |
### H3 — Whole-box copies on the current local target · CUSTOMER · NAMED
`AR/07-backup-architecture.md:401`. hp: `/var/lib/vz/dump`, 1 archive, 9.10 GB (keep 1) · N100: `/mnt/hdd_1/dump`
(the USB disk), 3 archives, 4.36 GB (keep 3), 2026-11-08…11-10 · T1: `/var/lib/vz/dump`, 3 archives, 9.46 GB (keep 3).
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | One packed copy of the whole guest per night. The agent makes it; Proxmox stores it. | A | `AR/07-backup-architecture.md:401` |
| 2 | The guest's size. Most of it is Docker images that could be pulled again. | K | R-1366 |
| 3 | Keep the newest N (default 3). Proxmox removes the older ones after a good run. | A | `AG/internal/config/config.go:540-550`, `AG/internal/proxmox/mutate.go:99-101` |
| 4 | Proxmox, after each good run. A half-written copy: the agent, at start and before each backup. | A | `AG/internal/backup/runner.go:227-230`, `AG/cmd/felhom-agent/main.go:2227-2238` |
| 5 | A copy that cannot fit is skipped before it starts. On a small system disk the local copy then stops for ever. | K | R-548 |
| 6 | The skipped run is reported to the hub and shown on the box's backup page with its reason. | A | `AG/internal/hub/report.go:620`, `CT/internal/web/backup_handlers.go:149-152` |
| 7 | No self-repair when the disk is too small. | K | R-548 |
| 8 | TEST. | A | `AG/internal/backup/r685_space_test.go:31`, `AG/internal/backup/prune_test.go:24` |
### H4 — Whole-box copies on a former target · CUSTOMER · UNNAMED
N100 `/var/lib/vz/dump`: 3 copies of 2026-07-28, 17.59 GB (the target moved to the USB disk) · hp `/mnt/hdd_1/dump`:
4 copies of 2026-08-03…08-21, 5.10 GB · T1: none.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Old whole-box copies on the place that was the target before. | A | `EV/raw/n100-host.txt` |
| 2 | Only a move of the target. After that they stay. | A | `EV/raw/hp-host.txt` |
| 3 | None. Retention looks only at the current target. | K | R-1087 |
| 4 | Nobody. | K | R-1087 |
| 5 | On a small system disk they leave no room for the new copies. | K | R-1087 |
| 6 | The host-disk alarm of H1, when they lie on the system disk. | A | `HUB/monitor/host_disk.go:41-42` |
| 7 | No. | K | R-1087 |
| 8 | MEASURED. Second instance today: demo-hp. | A | `EV/reading-1/hp-host.tsv` |
### H5 — Guest template archives ("golden") in the dump folder · CUSTOMER · UNNAMED
`/var/lib/vz/dump/vzdump-lxc-9100-*.tar.zst`. hp 1 (0.65 GB, 2026-09-13) · N100 4 (2.56 GB, 2026-07-18…08-03) ·
T1 1 (0.65 GB, 2026-10-04).
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The archive the installer downloads to make the guest. Product-owned. | A | `E/scripts/felhom-host-install.sh:3560-3565` |
| 2 | One more at each install or re-install of the box. | A | `EV/raw/n100-host.txt` |
| 3 | None. | N | C1 · S3 |
| 4 | Nobody. Only the uninstall with `--remove-golden` deletes it. | N | C1 · S3 · `E/scripts/felhom-host-install.sh:1520-1524` |
| 5 | It counts against H1. | A | `AG/configs/felhom-os-apply:1390-1392` |
| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` |
| 7 | No self-repair, no runbook. | N | C2 · S6 |
| 8 | MEASURED. | A | `EV/reading-1/n100-host.tsv` |
### H6 — Backup logs and half-written copies in the dump folder · CUSTOMER · MENTIONED
`AR/03-host-agent.md:518`. Logs: hp 10 (9 of them from failed runs of 09-24 and 10-14, under 2 KB each) · N100 6 · T1 3.
Half-written copies today: none.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | One text log per backup run; after a power cut a half-written archive. | A | `EV/raw/hp-host.txt` |
| 2 | One log per run. A half-written copy was 12.2 GB on demo-hp on 2026-10-14. | A | `AG/internal/vzpartial/vzpartial.go:1-12` |
| 3 | The log of a failed run has no limit: it has no archive, so nothing removes it. Each is under 2 KB. | N | C1 · S15 |
| 4 | Proxmox removes a good run's log with its archive. The agent removes a half-written copy at start and before each backup. | A | `AG/cmd/felhom-agent/main.go:2227-2238` |
| 5 | A half-written copy made each later backup skip, until the agent learned to remove it. | A | `AG/internal/vzpartial/vzpartial.go:1-12` |
| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` |
| 7 | Yes, for the half-written copy. | A | `AG/cmd/felhom-agent/main.go:2227-2238` |
| 8 | SOURCE and MEASURED. | A | `EV/reading-1/hp-host.tsv` |
### H7 — Host journal `/var/log/journal` · CUSTOMER · MENTIONED
`AR/03-host-agent.md:72`. hp 0.87 GB, 16 files, oldest 2026-10-04 (cleaned by hand on 11-07) · N100 4.28 GB, 74 files,
oldest 10-04 · T1 4.29 GB, 81 files, oldest 10-13. Lines in one hour: 13,946 / 13,918 / 11,558; lines written by `sudo` for
the agent's calls are 92 % / 92 % / 90 % of them (R-1367's 61 % counts the session pair only; `sudo` writes a third
line, the command, for each call).
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The host's system log. Nobody set it up; it is the Debian default. | A | `EV/os-cleaners/hp-host.txt` |
| 2 | The agent's `sudo` calls: three lines per call, about 12,800 lines an hour. | K | R-1367 |
| 3 | journald's default: 10 % of the disk, at most 4 GB. Two boxes sit at that cap. | K | R-1367 |
| 4 | journald deletes its oldest file when it reaches the cap. | A | `EV/reading-1/n100-host.tsv` |
| 5 | On a small system disk the journal alone takes a tenth of it. | K | R-1367 |
| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` |
| 7 | It stays at its cap by itself. The cap is not ours. | K | R-1367 |
| 8 | MEASURED, two readings. | A | `EV/reading-1/t1-host.tsv` |
### H8 — Other host logs, Proxmox task logs, graph data · CUSTOMER · MENTIONED
`AR/03-host-agent.md:570`. `/var/log` without the journal: 14.6 / 14.0 / 14.6 MB; task logs 914 / 706 / 877 files;
`/var/lib/rrdcached` 65 / 21 / 5 MB.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Text logs, one small log per Proxmox task, fixed-size graph files. | A | `EV/reading-1/hp-host.tsv` |
| 2 | Each task the agent starts makes one log. | A | `EV/raw/hp-host.txt` |
| 3 | Rotation: `logrotate`; Proxmox keeps its task index short (oldest task file 2026-10-28). | A | `EV/os-cleaners/hp-host.txt` |
| 4 | `logrotate.timer` and Proxmox itself. | A | `EV/os-cleaners/n100-host.txt` |
| 5 | Small and bounded; it counts against H1. | A | `EV/reading-1/n100-host.tsv` |
| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` |
| 7 | Yes, by rotation. | A | `EV/os-cleaners/t1-host.txt` |
| 8 | MEASURED. First written down today. | A | `EV/reading-1/t1-host.tsv` |
### H9 — Old kernels: `/lib/modules` and `/boot` · CUSTOMER · MENTIONED
`AR/11-os-updates.md:402`. hp 8 kernels, 8.15 GB + `/boot` 0.82 GB · N100 8, 8.15 GB + 0.94 GB · T1 3, 3.05 GB + 0.33 GB.
`/boot` is a folder on the system disk. The ring-0 boxes installed 7 kernels in 37 days; no kernel was ever removed.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Each kernel the box ever installed: about 1.02 GB of modules and 80 MB in `/boot`. | A | `EV/raw/hp-host-followup.txt` |
| 2 | Each kernel set the operator approves; ring 0 takes each one first. | A | `AR/11-os-updates.md:164-176` |
| 3 | None. No rule says how many kernels a box keeps. | N | C1 · S1 |
| 4 | Nobody. apt would remove old ones only on `autoremove`, which nothing runs; the update wrapper refuses each plan that removes a package. | N | C1 · S1 · `AG/configs/felhom-os-apply:1712`, `EV/os-cleaners/hp-host.txt` |
| 5 | Under 500 MB free the wrapper refuses the OS update, so security updates stop. After 7 days the hub says the updates are stale. | A | `AG/configs/felhom-os-apply:1390-1392`, `AR/08-alarm-ladder.md:375` |
| 6 | The host-disk alarm of H1; later `os_update_stale` to the operator. | A | `HUB/monitor/host_disk.go:41-42`, `HUB/notify/dispatcher.go:840` |
| 7 | No self-repair, no runbook. | N | C2 · S1 |
| 8 | NOBODY KNOWS how many kernels a 32 GB system disk holds before updates stop. The sizes are MEASURED. | N | C1 · `EV/reading-1/hp-host.tsv` |
### H10 — EFI partition `/boot/efi` · CUSTOMER · MENTIONED
`AR/11-os-updates.md:532-536`. 9.5 MB of 1.07 GB on each box (1 %).
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The boot loader, and the one-shot flag for a new kernel. | A | `AR/11-os-updates.md:532-536` |
| 2 | A boot-loader update. Kernels are not copied here (GRUB boxes). | A | `AR/11-os-updates.md:402` |
| 3 | The partition: 1.07 GB. | A | `EV/reading-1/hp-host.tsv` |
| 4 | Nothing needs to: 1 % used. | A | `EV/reading-1/n100-host.tsv` |
| 5 | Not written anywhere. | N | C2 · S13 |
| 6 | Nobody: it is its own filesystem and nothing watches it. | N | C4 · S13 |
| 7 | No runbook. | N | C2 · S13 |
| 8 | MEASURED. | A | `EV/reading-1/t1-host.tsv` |
### H11 — Host package cache and package lists · CUSTOMER · NAMED
`AR/11-os-updates.md:293`. `/var/cache/apt` 92 MB with 0 package files on each box; lists 62 MB.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Downloaded packages during an OS update, and the package index. | A | `EV/reading-1/hp-host.tsv` |
| 2 | Each OS update run. | A | `AR/11-os-updates.md:293` |
| 3 | Emptied at the end of each run. | A | `AG/configs/felhom-os-apply:1417` |
| 4 | The update wrapper (`apt-get clean`). | A | `AG/configs/felhom-os-apply:1417` |
| 5 | A run that would not fit is refused before it starts. | A | `AG/configs/felhom-os-apply:1390-1392` |
| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` |
| 7 | Yes: cleaned each run. | A | `AG/configs/felhom-os-apply:1417` |
| 8 | SOURCE and MEASURED (0 package files). | A | `EV/reading-1/t1-host.tsv` |
### H12 — The agent's self-update folder `/var/lib/felhom-agent/selfupdate` · CUSTOMER · UNNAMED
hp 739.9 MB, 49 binaries · N100 723.9 MB, 48 · T1 509.2 MB, 33. About 15 MB for each agent version the box ever got.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | One downloaded agent binary per version. The agent owns it. | A | `AG/internal/selfupdate/executor.go:106-128` |
| 2 | Each agent update. | A | `EV/raw/hp-host-followup.txt` |
| 3 | None. | N | C1 · S2 |
| 4 | Nobody. | N | C1 · S2 |
| 5 | It counts against H1. | A | `AG/configs/felhom-os-apply:1390-1392` |
| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` |
| 7 | No self-repair, no runbook. | N | C2 · S2 |
| 8 | MEASURED. | A | `EV/reading-1/hp-host.tsv` |
### H13 — Old copies of the agent binary (`.prev`, `.bak-*`, `.pre-bundle`) · CUSTOMER · MENTIONED
`AR/03-host-agent.md:893`. `/usr/local/bin/felhom-agent.bak-*`: hp 1 file, 14 MB · N100 11 files, 155 MB · T1 none.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The previous agent binary (`.prev`), and one copy for each re-run of the installer (`.bak-<time>`). | A | `AG/configs/felhom-selfupdate-guarded:86`, `E/scripts/felhom-host-install.sh:2675` |
| 2 | `.prev` is one file, replaced each update. `.bak-*` gains a file at each installer re-run. | A | `AG/configs/felhom-selfupdate-guarded:114` |
| 3 | None for `.bak-*`. | N | C1 · S2 |
| 4 | Nobody. | N | C1 · S2 |
| 5 | It counts against H1. | A | `AG/configs/felhom-os-apply:1390-1392` |
| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` |
| 7 | No self-repair, no runbook. | N | C2 · S2 |
| 8 | MEASURED. | A | `EV/reading-1/n100-host.tsv` |
### H14 — Update wrapper state and previous signed bundles · CUSTOMER · MENTIONED
`AR/11-os-updates.md:343`. `/var/lib/felhom-os-apply`: 0.4 MB, 3 previous bundles on each box.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The update wrapper's state and the last signed config bundles. | A | `EV/reading-1/hp-host.tsv` |
| 2 | Each new bundle. | A | `AR/11-os-updates.md:343` |
| 3 | The last 3. | A | `AG/configs/felhom-os-apply:190` |
| 4 | The wrapper, at each bundle. | A | `AG/configs/felhom-os-apply:2315-2318` |
| 5 | Bounded: 0.4 MB. | A | `EV/reading-1/n100-host.tsv` |
| 6 | The host-disk alarm of H1. | A | `HUB/monitor/host_disk.go:41-42` |
| 7 | Yes: pruned each time. | A | `AG/configs/felhom-os-apply:2315-2318` |
| 8 | SOURCE and MEASURED. | A | `EV/reading-1/t1-host.tsv` |
### H15 — Host `/tmp` and `/var/tmp` · CUSTOMER · UNNAMED
`/tmp` is memory (tmpfs), 5–6 entries, 0 MB · `/var/tmp` 1–4 entries, 0 MB.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Short-lived files of the OS and of the agent's wrappers. | A | `EV/reading-1/hp-host.tsv` |
| 2 | Nothing lasting. | A | `EV/reading-1/n100-host.tsv` |
| 3 | `/tmp`: the tmpfs default, half of the memory. | A | `EV/os-cleaners/hp-host.txt` |
| 4 | systemd: 10 days for `/tmp`, 30 days for `/var/tmp`; `/tmp` is empty after each restart. | A | `EV/os-cleaners/hp-host.txt` |
| 5 | A full `/tmp` uses memory, not disk. Not written anywhere what then fails. | N | C2 · S6 |
| 6 | Nobody watches `/tmp`. `/var/tmp` counts against H1. | N | C4 · S6 |
| 7 | Yes: the clean-up timer and each restart. | A | `EV/os-cleaners/t1-host.txt` |
| 8 | MEASURED. | A | `EV/reading-1/t1-host.tsv` |
### H16 — The bare folder under an unmounted drive · CUSTOMER · NAMED
`AR/16-storage-physical-identity.md:209`, `AR/02-controller-module-map.md:179`. Today: 0 bytes. T1's guest shows four
empty, not-mounted folders (`media_a` … `media_d`).
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The empty folder where a drive is mounted. With the drive gone, a write there lands on the system disk. | A | `AG/configs/felhom-backup-target-apply:21-29` |
| 2 | A write while the drive is absent. | A | `AG/internal/localapi/disks.go:414-420` |
| 3 | The guards refuse the write: the backup storage is tied to the mount, and the placeholder folder is owned by host root, which the guest cannot write. | A | `AG/configs/felhom-backup-target-apply:97`, `AG/internal/guesthook/heal.go:1-10` |
| 4 | Nothing to clean: 0 bytes. Nobody removes the empty folder of a drive that left. | A | `EV/raw/t1-9201.txt` |
| 5 | Refused before the write; the drive's size is reported as unknown, never as the system disk's. | A | `AG/internal/localapi/disks.go:414-420` |
| 6 | `storage_disconnected` to the household. | A | `AR/08-alarm-ladder.md:518-524` |
| 7 | Yes: the pre-start hook makes the placeholder so the guest still starts. | A | `AG/internal/guesthook/heal.go:1-10` |
| 8 | TEST (named in the source comment). | A | `AG/internal/localapi/disks.go:414-420` |
### H17 — The restore-test's scratch guest · CUSTOMER · NAMED
`AR/03-host-agent.md:513-520`, capability row A19. None present today on any box.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | A throwaway guest restored from last night's copy, to prove the copy works. | A | `AR/03-host-agent.md:513-520` |
| 2 | Each scheduled restore test: the full unpacked size of the guest. | A | `AR/03-host-agent.md:513-520` |
| 3 | A space check before anything is made: free ≥ restored size × 1.2 + 5 GiB, and off the tested guest's pool when possible. | A | `AR/03-host-agent.md:513-520` |
| 4 | The agent destroys it at the end; a failed destroy is retried each 10 minutes. | A | `AG/internal/reconcile/restoretest.go:476-483`, `AG/internal/reconcile/restoretest_retry.go:5-12` |
| 5 | It once filled the pool and the guest went read-only (2026-09-24); the space check exists since then. | A | `AR/03-host-agent.md:513-520` |
| 6 | The pool alarm of H2. | A | `HUB/monitor/storage_fill.go:36-57` |
| 7 | Yes: the retry timer. | A | `AG/internal/reconcile/restoretest_retry.go:5-12` |
| 8 | DOC with a measured incident, and SOURCE. | A | `AR/03-host-agent.md:513-520` |
### H18 — Swap (LV `pve/swap`) · CUSTOMER · NAMED
`AR/08-alarm-ladder.md:399-402`. 8.59 GB (hp, N100), 8.25 GB (T1); used 0 / 0 / 186 MB.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The host's swap space. | A | `EV/reading-1/hp-host.tsv` |
| 2 | Memory pressure. It is a fixed-size volume. | A | `EV/reading-1/t1-host.tsv` |
| 3 | The volume size. | A | `EV/reading-1/n100-host.tsv` |
| 4 | The kernel. | A | `EV/reading-1/t1-host.tsv` |
| 5 | A memory problem, not a disk problem. | A | `AR/08-alarm-ladder.md:399-402` |
| 6 | `host_swap_short` to the operator, when the swap is smaller than the guests' allowance. | A | `AR/08-alarm-ladder.md:399-402` |
| 7 | Not a space question. | A | `AR/08-alarm-ladder.md:399-402` |
| 8 | DOC and MEASURED. | A | `EV/reading-1/hp-host.tsv` |
---
## Guest 9201 — 17 places
### G1 — Guest root disk (`vm-9201-disk-0`, 32 GiB) · CUSTOMER · MENTIONED
`AR/16-storage-physical-identity.md:134`. Used: hp 1.36 GB · N100 1.19 GB · T1 1.18 GB of 33.5 GB (4–5 %).
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The guest's Debian, the Docker programs, `/opt/docker/stacks`, the package cache. | A | `EV/reading-1/hp-9201.tsv` |
| 2 | Guest OS updates. Very little else. | A | `EV/reading-1/n100-9201.tsv` |
| 3 | 32 GiB. The agent can only make it larger. | A | `AG/internal/reconcile/bringup.go:405-447` |
| 4 | The update wrapper empties the package cache; `logrotate`; the guest journal is in memory (G15). | A | `AG/configs/felhom-os-apply:1417`, `EV/os-cleaners/hp-9201.txt` |
| 5 | The guest's OS update is refused when free space is too low. | A | `AR/11-os-updates.md:302-306` |
| 6 | Nobody. The controller's `/` is the data volume, not this disk — measured on all three boxes. The closed row R-521 says the guest root is watched; that was true only before the two volumes were merged. Capability-map row C13 says "per filesystem": this filesystem is the exception. | N | C4 · S9 · `EV/extra/controller-fs-view.txt`, `CT/internal/system/dockervol.go:14-26` |
| 7 | No self-repair, no runbook. | N | C2 · S6 |
| 8 | MEASURED. | A | `EV/extra/controller-fs-view.txt` |
### G2 — Guest data volume (`mp0`, `/var/lib/felhom`: Docker and `sys_drive`) · CUSTOMER · NAMED
`AR/16-storage-physical-identity.md:126-150`. Used: hp 30.09 of 73.79 GB (41 %) · N100 2.47 of 263.09 GB (1 %) · T1 8.99
of 73.79 GB (12 %). hp was 19.1 GB on 2026-10-15.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Each app's programs (images), each app's own data (volumes), container logs, the backups of apps with no drive. | A | `EV/reading-1/hp-9201.tsv` |
| 2 | G3–G12 and G14 below. On hp the images are 81 % of it. | A | `EV/reading-1/hp-9201.tsv` |
| 3 | The volume: 50 / 100 / 130 GiB by pool size on a new install (older boxes: 70 or 250). A new app is refused below max(5 GB, 10 %) free. A backup is refused at 97 % used or under 1 GiB free. | A | `AR/16-storage-physical-identity.md:146-150`, `CT/internal/system/dockervol.go:14-26`, `CT/internal/backup/recovery_unit.go:383-388` |
| 4 | The three image rules (G3, G4, G5); the weekly trim gives the freed blocks back to the pool. | A | `CT/internal/stacks/image_retention.go:188-193`, `AG/internal/fstrim/fstrim.go:1-12` |
| 5 | New installs are refused; backups refuse app by app and delete nothing; running apps can fail to write. Data already written stays. | A | `CT/internal/api/router.go:477`, `AR/07-backup-architecture.md:1567-1572` |
| 6 | The controller, each 10 minutes: warning at 85 % or 5 GiB free, critical at 95 % or 2 GiB, to the household. Before full. The watch has two targets (`/` and `/mnt/sys_drive`) that are ONE filesystem on all three boxes, and it de-duplicates by path — so one fill can send two warnings (C4; not proved live, a disk may not be filled in this run). | A | `CT/internal/fillwatch/fillwatch.go:40-58`, `CT/cmd/controller/main.go:2745-2749`, `EV/extra/controller-fs-view.txt` |
| 7 | No self-repair. No runbook for a full data volume. | N | C2 · S6 |
| 8 | SOURCE and MEASURED. | A | `CT/internal/fillwatch/fillwatch.go:40-58` |
### G3 — App images, with one undo image per app · CUSTOMER · NAMED
`AR/09-update-architecture.md:687` (decision 53). All images together (`overlay2`): hp 24.28 GB, 22 tagged + 21
untagged · N100 2.18 GB, 11 + 5 · T1 8.30 GB, 12 + 12.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The image each app runs and the image it ran before (for the undo button). | A | `AR/09-update-architecture.md:687` |
| 2 | Each app install; each update adds one image and makes the oldest free to go. | A | `CT/internal/stacks/update.go:1063` |
| 3 | A count, not a size: current + previous for each app. `NONE — chosen` for the total bytes (decision 53). | A | `AR/09-update-architecture.md:687` |
| 4 | The controller, after each update and each removal. | A | `CT/internal/stacks/image_retention.go:188-193`, `CT/internal/stacks/update.go:1063` |
| 5 | As G2. | A | `CT/internal/api/router.go:477` |
| 6 | As G2. | A | `CT/internal/fillwatch/fillwatch.go:40-58` |
| 7 | Yes: the next pass deletes what is no longer kept. | A | `CT/internal/stacks/image_retention.go:370` |
| 8 | TEST. | A | `CT/internal/stacks/image_retention_test.go:111` |
### G4 — Old controller images · CUSTOMER · NAMED
`AR/09-update-architecture.md:710` (decision 56). Bytes are inside G3's figure.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The controller's own image and the one before it. | A | `AR/09-update-architecture.md:710` |
| 2 | Each controller update. | A | `CT/internal/stacks/controller_image_retention.go:13` |
| 3 | Running + previous. | A | `CT/internal/stacks/controller_image_retention.go:25-30` |
| 4 | The controller, after its own update. | A | `CT/internal/stacks/controller_image_retention.go:49-57` |
| 5 | As G2. | A | `CT/internal/api/router.go:477` |
| 6 | As G2. | A | `CT/internal/fillwatch/fillwatch.go:40-58` |
| 7 | Yes: the next pass. | A | `CT/internal/stacks/controller_image_retention.go:49-57` |
| 8 | TEST. | A | `CT/internal/stacks/controller_image_retention_test.go:36` |
### G5 — Infrastructure images (traefik and the other pinned helpers of the box) · CUSTOMER · NAMED
`AR/09-update-architecture.md:1235` (decision 285, 2026-11-09). Bytes are inside G3's figure.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The images of the box's own services. | A | `CT/internal/stacks/infra_image_retention.go:14-16` |
| 2 | Each change of a pinned version. | A | `CT/internal/stacks/infra_image_retention.go:14-16` |
| 3 | The pinned one + the previous. | A | `CT/internal/stacks/infra_image_retention.go:20-25` |
| 4 | The controller. | A | `CT/internal/stacks/infra_image_retention.go:61` |
| 5 | As G2. | A | `CT/internal/api/router.go:477` |
| 6 | As G2. | A | `CT/internal/fillwatch/fillwatch.go:40-58` |
| 7 | Yes: the next pass. | A | `CT/internal/stacks/infra_image_retention.go:61` |
| 8 | TEST. | A | `CT/internal/stacks/infra_image_retention_test.go:42` |
### G6 — Helper images (`alpine`, `hello-world`) · CUSTOMER · MENTIONED
`AR/09-update-architecture.md:2447`. `alpine:latest` 8.42 MB on each box (4 months old on hp and N100, 7 weeks on T1) ·
`hello-world:latest` 10 kB on N100 and T1.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | `alpine`: the small image the controller starts for a backup, a restore, an undo, an export (seven call sites). `hello-world`: left by the template bake. | A | `CT/internal/backup/backup.go:988`, `AG/configs/build-golden.sh:232` |
| 2 | Pulled once, at first use. Never again. | A | `EV/extra/hp-9201.txt` |
| 3 | No version is pinned: the box runs the `alpine` that was newest on the day of its first pull, for ever. | N | C5 · S7 |
| 4 | Nobody. None of the three image rules can reach it: they look only at app, controller and infrastructure repositories. | N | C5 · S7 · `CT/internal/stacks/image_retention.go:244-250` |
| 5 | 8 MB; as G2. | A | `EV/extra/n100-9201.txt` |
| 6 | As G2. | A | `CT/internal/fillwatch/fillwatch.go:40-58` |
| 7 | Not a space question at 8 MB. | A | `EV/extra/t1-9201.txt` |
| 8 | MEASURED and SOURCE. | A | `EV/extra/t1-9201.txt` |
### G7 — Named app volumes · CUSTOMER · NAMED
`AR/07-backup-architecture.md:600-605`. All volumes together: hp 1.61 GB · T1 0.37 GB. Named ones: hp 27 · N100 7 · T1 8.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Each app's own data that is not on a drive: databases, settings. The household's. | A | `AR/07-backup-architecture.md:600-605` |
| 2 | The use of the app. | A | `EV/reading-1/hp-9201.tsv` |
| 3 | No limit per app; the limit is G2's. | A | `CT/internal/system/dockervol.go:14-26` |
| 4 | Only a removal of the app deletes them; a stop keeps them. | A | `CT/internal/stacks/delete.go:763-769`, `CT/internal/stacks/manager.go:1455-1459` |
| 5 | As G2. | A | `CT/internal/api/router.go:477` |
| 6 | As G2. | A | `CT/internal/fillwatch/fillwatch.go:40-58` |
| 7 | No self-repair, no runbook (the same missing runbook as G2). | N | C2 · S6 |
| 8 | SOURCE and MEASURED. | A | `CT/internal/stacks/delete.go:763-769` |
### G8 — Unnamed (anonymous) volumes · CUSTOMER · NAMED
`AR/07-backup-architecture.md:600-605`. hp 1,887 (1,885 unused) · N100 1 · T1 380 (377 unused). hp had 1,157 on
2026-10-15: about 29 more each day. T1 had 132: about 10 a day.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | A volume an image declares by itself and the app definition does not name. Mostly empty. | A | `AR/07-backup-architecture.md:600-605` |
| 2 | Each recreate of such a container makes a new one. | K | R-1267 |
| 3 | None. | K | R-1267 |
| 4 | Nobody; a removal leaves them too. | K | R-995 |
| 5 | They are small today. Nobody measured the effect of many thousands. | K | R-1267 |
| 6 | As G2 (by bytes only; nothing counts them). | A | `CT/internal/fillwatch/fillwatch.go:40-58` |
| 7 | No. | K | R-995 |
| 8 | MEASURED, three readings. | A | `EV/reading-1/hp-9201.tsv` |
### G9 — Container writable layers, with the controller's off-site cache · CUSTOMER · UNNAMED
The controller's own layer: 1.3 MB / 0.3 MB / 0.5 MB. Inside it `/root/.cache/restic`: 3.4 MB / 0.8 MB / 1.0 MB.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | What a container writes outside its volumes. In the controller: the index cache of the off-site backup program. | A | `EV/extra/hp-9201.txt` |
| 2 | The app's own temp files; the off-site cache grows with the size of the off-site store. | A | `EV/extra/n100-9201.txt` |
| 3 | No limit of its own; G2's. No setting places or caps the off-site cache. | A | `EV/searches.txt` |
| 4 | Each recreate of the container drops the layer — so each controller update empties the cache. | A | `EV/extra/t1-9201.txt` |
| 5 | As G2. | A | `CT/internal/api/router.go:477` |
| 6 | As G2. | A | `CT/internal/fillwatch/fillwatch.go:40-58` |
| 7 | Yes: a recreate. | A | `EV/extra/hp-9201.txt` |
| 8 | MEASURED. No document and no source line names the cache (S10). | A | `EV/extra/hp-9201.txt` |
### G10 — Container logs · CUSTOMER · MENTIONED
`AR/11-os-updates.md:107`. hp 26.4 MB, 27 files, 26 containers · N100 9.7 MB, 9 containers · T1 7.3 MB, 10 containers.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | What each container prints. | A | `EV/reading-1/hp-9201.tsv` |
| 2 | Each line an app prints. | A | `EV/reading-1/n100-9201.tsv` |
| 3 | 10 MB × 3 files for each container: at most 780 MB (hp), 270 MB (N100), 300 MB (T1). Each container on each box carries the cap. | A | `AG/configs/build-golden.sh:193-203`, `EV/raw/hp-9201.txt` |
| 4 | Docker rotates the files. | A | `AG/configs/build-golden.sh:193-203` |
| 5 | Bounded; as G2. | A | `CT/internal/api/router.go:477` |
| 6 | As G2. | A | `CT/internal/fillwatch/fillwatch.go:40-58` |
| 7 | The cap is written once, when the template is baked. Nothing checks it on a running box; the one later edit of that file touches another key only. | N | C5 · S12 · `AG/configs/felhom-os-apply:769-773` |
| 8 | MEASURED on each container. | A | `EV/raw/t1-9201.txt` |
### G11 — Docker build cache · CUSTOMER · UNNAMED
hp 0.18 MB; the other two boxes about 0.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | What an image build leaves. A box builds nothing. | A | `EV/reading-1/hp-9201.tsv` |
| 2 | Nothing on a household box. | A | `EV/reading-1/n100-9201.tsv` |
| 3 | No limit; it holds nothing. | A | `EV/reading-1/t1-9201.tsv` |
| 4 | Nothing to clean. | A | `EV/reading-1/hp-9201.tsv` |
| 5 | As G2. | A | `CT/internal/api/router.go:477` |
| 6 | As G2. | A | `CT/internal/fillwatch/fillwatch.go:40-58` |
| 7 | Not needed. | A | `EV/reading-1/n100-9201.tsv` |
| 8 | MEASURED. | A | `EV/reading-1/t1-9201.tsv` |
### G12 — The controller's own state volume · CUSTOMER · MENTIONED
`AR/_recovery-inventory-2026-07-28.md:485`. hp 145 MB · N100 84 MB · T1 80 MB. Largest file `metrics.db`: 120 / 41 / 55 MB.
Catalog cache: 18 / 38 / 19 MB.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The box's settings, the usage graphs' database, the cached app catalog. | A | `EV/extra/controller-state-volume.txt` |
| 2 | One graph row per container per sample; the catalog. | A | `EV/extra/controller-state-volume.txt` |
| 3 | Graph rows older than 30 days are deleted. | A | `CT/cmd/controller/main.go:1755`, `CT/internal/metrics/store.go:311-315` |
| 4 | The controller. | A | `CT/cmd/controller/main.go:1755` |
| 5 | As G2: the install reserve exists so the box's own services can still write. | A | `CT/internal/system/dockervol.go:14-26` |
| 6 | As G2. | A | `CT/internal/fillwatch/fillwatch.go:40-58` |
| 7 | No self-repair, no runbook (the same missing runbook as G2). | N | C2 · S6 |
| 8 | SOURCE and MEASURED. | A | `EV/extra/controller-state-volume.txt` |
### G13 — App definitions `/opt/docker/stacks` · CUSTOMER · MENTIONED
`AR/_recovery-inventory-2026-07-28.md:494`. hp 1.4 MB, 69 folders · T1 1.8 MB, 68. It lies on the guest root disk (G1).
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | One folder per catalog app: the definition files, and `app.yaml` when the app is installed. | A | `CT/internal/stacks/delete.go:532-536` |
| 2 | The catalog's size. | A | `EV/reading-1/hp-9201.tsv` |
| 3 | The catalog: under 2 MB. | A | `EV/reading-1/t1-9201.tsv` |
| 4 | A removal deletes `app.yaml`; the template files stay on purpose. | A | `CT/internal/stacks/delete.go:532-536` |
| 5 | As G1. | A | `AR/11-os-updates.md:302-306` |
| 6 | Nobody, as G1. | N | C4 · S9 |
| 7 | As G1. | N | C2 · S6 |
| 8 | MEASURED. | A | `EV/extra/controller-fs-view.txt` |
### G14 — Backups of apps with no drive (`sys_drive`) · CUSTOMER · NAMED
`AR/07-backup-architecture.md:1567-1572`. hp 1.33 GB, 14 apps · N100 48 MB, 5 · T1 270 MB, 4. On the data volume (G2).
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The nightly backup unit of each app that has no data drive. The household's. | A | `AR/07-backup-architecture.md:1567-1572` |
| 2 | The apps' database and volume dumps. | A | `EV/reading-1/hp-9201.tsv` |
| 3 | One point per app. A backup is refused at 97 % used or under 1 GiB free. | A | `AR/07-backup-architecture.md:398`, `CT/internal/backup/recovery_unit.go:383-388` |
| 4 | Each night's backup replaces the app's unit. A unit left on a place the app moved away from is pruned. | A | `CT/internal/backup/backup.go:1552` |
| 5 | The backup refuses app by app and deletes nothing; the last good unit stays. | A | `CT/internal/backup/recovery_unit.go:412-420`, `AR/07-backup-architecture.md:1567-1572` |
| 6 | As G2 (this is the watch's second target). | A | `CT/cmd/controller/main.go:2745-2749` |
| 7 | No self-repair, no runbook (the same missing runbook as G2). | N | C2 · S6 |
| 8 | SOURCE and MEASURED. | A | `CT/internal/backup/recovery_unit.go:383-388` |
### G15 — Guest system log · CUSTOMER · UNNAMED
The guest has no `/var/log/journal`, so its journal lives in memory (`/run`): 8 MB. It is empty after each guest restart.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The guest's own system log (Docker's service log is in it). | A | `EV/extra/hp-9201.txt` |
| 2 | Each log line of the guest's services. | A | `EV/extra/n100-9201.txt` |
| 3 | journald's default for a log in memory. Nobody chose "in memory": no source line and no document sets it, and a restart loses the log. | N | C5 · S14 |
| 4 | journald; and each restart. | A | `EV/os-cleaners/hp-9201.txt` |
| 5 | The oldest lines go. | A | `EV/os-cleaners/n100-9201.txt` |
| 6 | Nobody. It is memory, not disk. | N | C4 · S9 |
| 7 | It is bounded by itself. | A | `EV/os-cleaners/t1-9201.txt` |
| 8 | MEASURED. | A | `EV/extra/t1-9201.txt` |
### G16 — Guest package cache and lists · CUSTOMER · MENTIONED
`AR/11-os-updates.md:293`. hp 96 MB + 96 MB, 0 package files; the same shape on the other two.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | As H11, inside the guest. | A | `EV/reading-1/hp-9201.tsv` |
| 2 | Each guest OS update. | A | `AR/11-os-updates.md:293` |
| 3 | Emptied at the end of each run. | A | `AG/configs/felhom-os-apply:1417` |
| 4 | The update wrapper. | A | `AG/configs/felhom-os-apply:1417` |
| 5 | As G1. | A | `AR/11-os-updates.md:302-306` |
| 6 | Nobody, as G1. | N | C4 · S9 |
| 7 | Yes: cleaned each run. | A | `AG/configs/felhom-os-apply:1417` |
| 8 | MEASURED (0 package files). | A | `EV/reading-1/t1-9201.tsv` |
### G17 — Guest `/tmp` and `/var/tmp` · CUSTOMER · UNNAMED
`/tmp` is memory. Both near 0 today. An app export is written to a temp folder first.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | Short-lived files. | A | `EV/extra/hp-9201.txt` |
| 2 | Nothing lasting on the guest. | A | `EV/os-cleaners/hp-9201.txt` |
| 3 | The tmpfs default. | A | `EV/os-cleaners/n100-9201.txt` |
| 4 | systemd's clean-up timer; each restart. | A | `EV/extra/hp-9201.txt` |
| 5 | Memory, not disk. Not written anywhere what then fails. | N | C2 · S6 |
| 6 | Nobody. | N | C4 · S9 |
| 7 | Yes: the timer and each restart. | A | `EV/extra/t1-9201.txt` |
| 8 | MEASURED. | A | `EV/extra/n100-9201.txt` |
---
## Data drive (`/mnt/felhom-drives/<drive>/`) — 10 places
Each folder below lies on a registered drive. The drive's alarm is the controller's fill watch
(warning 85 %, critical 95 %, to the household). Drives today: hp `hdd_1` · N100 `hdd_1` (8.14 of 983 GB used) ·
T1 `adatlemez` (13.11 of 52.5 GB used).
### D1 — `backups/primary` — the app's own backup unit · CUSTOMER · NAMED
`AR/07-backup-architecture.md:398`. hp 372 MB, 7 apps · N100 1.39 GB, 5 · T1 100 MB, 1.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The nightly backup unit of each app that has a drive. The household's. | A | `AR/07-backup-architecture.md:398` |
| 2 | The apps' database and volume dumps. | A | `EV/reading-1/n100-9201.tsv` |
| 3 | One point per app. The unit of a removed app, when the household kept it, has no limit: `NONE — chosen`. | A | `AR/07-backup-architecture.md:398`, `AR/07-backup-architecture.md:380` |
| 4 | The nightly backup replaces it; a removal with "delete the backups too" deletes it. | A | `CT/internal/backup/backup.go:1552`, `CT/internal/stacks/delete.go:532-536` |
| 5 | The backup of that app is refused; the last good unit stays byte for byte. | A | `CT/internal/backup/recovery_unit.go:412-420` |
| 6 | The drive's fill watch, to the household, before full. | A | `CT/cmd/controller/main.go:2750-2757` |
| 7 | No self-repair. No runbook for a full data drive. | N | C2 · S6 |
| 8 | SOURCE and MEASURED. | A | `CT/internal/backup/recovery_unit.go:383-388` |
### D2 — `backups/secondary` — the copy on a second drive · CUSTOMER · NAMED
`AR/07-backup-architecture.md:399`. hp 920 MB, 8 apps · N100 294 MB, 4 · T1 13.01 GB, 8 apps (none of the 8 is installed
today; one is 12.49 GB).
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | A full second copy of each app's unit and files, on another drive. | A | `AR/07-backup-architecture.md:399` |
| 2 | The apps' data. | A | `EV/reading-1/t1-9201.tsv` |
| 3 | The copy needs max(2 GB, 20 %) free on the target. A copy whose app is gone is kept with no limit: `NONE — chosen` (D25). | A | `CT/internal/backup/tier2.go:80-84`, `AR/07-backup-architecture.md:1731-1735` |
| 4 | The mirror deletes what the source deleted. A copy of a removed app: only the household, from the backup page. | A | `CT/internal/backup/tier2.go:433`, `CT/internal/backup/removed_units.go:114` |
| 5 | The copy is refused under the headroom. | A | `CT/internal/backup/tier2.go:80-84` |
| 6 | The drive's fill watch. | A | `CT/cmd/controller/main.go:2750-2757` |
| 7 | No self-repair, no runbook (the same missing runbook as D1). | N | C2 · S6 |
| 8 | SOURCE and MEASURED. | A | `EV/raw/t1-9201.txt` |
### D3 — `backups/offsite-restore` — copies fetched from the off-site store · CUSTOMER · MENTIONED
`AR/_recovery-inventory-2026-07-28.md:126`. N100 1.65 GB, 2 apps (1.49 GB + 163 MB; neither app is installed today) ·
hp 0 · T1 0.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The full copy an off-site restore or check downloads before it is used. | A | `CT/internal/backup/offbox_verify_copies.go:10-14` |
| 2 | Each off-site restore or check of an app. | A | `EV/raw/n100-9201.txt` |
| 3 | None: no age, no size. (The restore's undo copy next to it has 30 days — D5.) | N | C5 · S8 |
| 4 | Only the household's delete button. | A | `CT/internal/backup/offbox_verify_copies.go:111-115` |
| 5 | As the drive: a new download is refused when it does not fit. | A | `CT/internal/backup/offbox_proof.go:163-170` |
| 6 | The drive's fill watch. | A | `CT/cmd/controller/main.go:2750-2757` |
| 7 | No self-repair, no runbook (the same missing runbook as D1). | N | C2 · S6 |
| 8 | MEASURED: two copies stayed on the N100 after their apps were removed. | A | `EV/reading-1/n100-9201.tsv` |
### D4 — `backups/offsite-proof` — the nightly proof's scratch · CUSTOMER · MENTIONED
`AR/07-backup-architecture.md:2020`. 0 bytes on each box.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The copy the nightly proof downloads to show the off-site store still holds something. | A | `AR/07-backup-architecture.md:2020` |
| 2 | One app's unit, each night, for minutes. | A | `CT/internal/backup/offbox_proof.go:163-170` |
| 3 | A headroom check runs before the download. | A | `CT/internal/backup/offbox_proof.go:171-183` |
| 4 | The proof itself, at its end; it also removes a stale copy from an interrupted run. | A | `CT/internal/backup/offbox_proof.go:171-183` |
| 5 | The proof is refused before any download. | A | `CT/internal/backup/offbox_proof.go:171-183` |
| 6 | The drive's fill watch. | A | `CT/cmd/controller/main.go:2750-2757` |
| 7 | Yes. | A | `CT/internal/backup/offbox_proof.go:171-183` |
| 8 | SOURCE and MEASURED (empty). | A | `EV/reading-1/hp-9201.tsv` |
### D5 — `backups/restore-undo` — what a restore replaced · CUSTOMER · NAMED
`AR/07-backup-architecture.md:764`, `AR/09-update-architecture.md:1203` (decision 281). The folder does not exist on any of
the three boxes today.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The data a restore replaced, kept so the restore can be undone. | A | `CT/internal/backup/offbox_restore_undo.go:55` |
| 2 | Each restore: a full copy of the app's data. | A | `CT/internal/backup/restore_copies.go:160` |
| 3 | 30 days when unused; no limit while a support hold exists (`NONE — chosen`, decision 281). | A | `CT/internal/backup/restore_copy_sweep.go:27-37` |
| 4 | The controller's sweep. A removal of the app with "delete the backups too" leaves it behind. | K | R-1268 |
| 5 | As the drive. | A | `CT/internal/backup/recovery_unit.go:412-420` |
| 6 | The drive's fill watch; the household is told when a copy is deleted. | A | `CT/internal/notify/notifier.go:1604` |
| 7 | Yes: the sweep. | A | `CT/internal/backup/restore_copy_sweep.go:27-37` |
| 8 | SOURCE. | A | `CT/internal/backup/restore_copy_sweep.go:27-37` |
### D6 — `kept` — data kept at "remove the app, keep my data" · CUSTOMER · NAMED
`AR/07-backup-architecture.md:990-1009`, `AR/09-update-architecture.md:545` (decision 40). hp 208 MB (one app, three dates
of 2026-10-16) · none on the other two.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The removed app's data folder, with its backup unit. The household's. | A | `AR/07-backup-architecture.md:1008` |
| 2 | Each "remove, keep my data". | A | `CT/internal/stacks/kept.go:45` |
| 3 | `NONE — chosen`: kept data is never deleted by the product (decision 40). | A | `AR/09-update-architecture.md:545` |
| 4 | Only the household. | A | `AR/09-update-architecture.md:545` |
| 5 | As the drive. | A | `CT/internal/backup/recovery_unit.go:412-420` |
| 6 | The drive's fill watch. | A | `CT/cmd/controller/main.go:2750-2757` |
| 7 | No self-repair, no runbook (the same missing runbook as D1). | N | C2 · S6 |
| 8 | DOC and MEASURED. | A | `EV/reading-1/hp-9201.tsv` |
### D7 — `appdata` — the apps' files on the drive · CUSTOMER · NAMED
`AR/07-backup-architecture.md:996`. hp 30 MB, 2 apps · N100 441 MB, 4 apps · T1 45 KB, 1.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The files an app keeps on the drive (photos, documents). The household's. | A | `AR/07-backup-architecture.md:996` |
| 2 | The use of the app. | A | `EV/reading-1/n100-9201.tsv` |
| 3 | The drive. No limit per app. | A | `EV/reading-1/n100-9201.tsv` |
| 4 | Only a removal "with data" deletes it. | A | `AR/07-backup-architecture.md:1002-1005` |
| 5 | The app's own write fails. The drive's backups refuse app by app and delete nothing. | A | `AR/07-backup-architecture.md:1567-1572`, `CT/internal/backup/recovery_unit.go:412-420` |
| 6 | The drive's fill watch, before full. | A | `CT/cmd/controller/main.go:2750-2757` |
| 7 | No self-repair, no runbook (the same missing runbook as D1). | N | C2 · S6 |
| 8 | DOC and MEASURED. | A | `EV/reading-1/hp-9201.tsv` |
### D8 — `userdata` — the household's own folders · CUSTOMER · NAMED
`AR/07-backup-architecture.md:1002-1005`, `AR/09-update-architecture.md:810-811` (decision 67). hp 6 MB · N100 0.6 MB ·
T1 0.1 MB (test boxes hold almost nothing here).
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The household's films, music, documents, import folders. | A | `AR/07-backup-architecture.md:1002-1005` |
| 2 | The household. | A | `EV/reading-1/n100-9201.tsv` |
| 3 | `NONE — chosen`: it stays on every removal (decision 67). | A | `AR/09-update-architecture.md:810-811` |
| 4 | Only the household. A removal leaves the app's empty import folder. | K | R-995 |
| 5 | As D7. | A | `CT/internal/backup/recovery_unit.go:412-420` |
| 6 | The drive's fill watch. | A | `CT/cmd/controller/main.go:2750-2757` |
| 7 | No self-repair, no runbook (the same missing runbook as D1). | N | C2 · S6 |
| 8 | DOC and MEASURED. | A | `EV/reading-1/t1-9201.tsv` |
### D9 — `shares` — folders shared on the home network · CUSTOMER · MENTIONED
`AR/_recovery-inventory-2026-07-28.md:432`. N100 one share, 32 KB · none on the other two.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The folders the household shares to its computers. | A | `CT/internal/web/sharing_handlers.go:390-400` |
| 2 | The household's copies. | A | `EV/reading-1/n100-9201.tsv` |
| 3 | The drive. No limit per share. | A | `CT/internal/web/sharing_handlers.go:390-400` |
| 4 | Only the household. | A | `CT/internal/web/sharing_handlers.go:390-400` |
| 5 | As D7. | A | `CT/internal/backup/recovery_unit.go:412-420` |
| 6 | The drive's fill watch. | A | `CT/cmd/controller/main.go:2750-2757` |
| 7 | No self-repair, no runbook (the same missing runbook as D1). | N | C2 · S6 |
| 8 | SOURCE and MEASURED. | A | `EV/reading-1/n100-9201.tsv` |
### D10 — `backups/set-aside` — the backup of an earlier install of the same app · CUSTOMER · NAMED
`AR/07-backup-architecture.md:1793-1800`, `AR/09-update-architecture.md:1344-1346` (decision 256). The folder does not exist
on any of the three boxes today.
| # | Answer | Mark | Pointer |
|---|---|---|---|
| 1 | The backup unit of an earlier install, moved aside when the app is installed again. | A | `AR/07-backup-architecture.md:1793-1800` |
| 2 | Each re-install of an app that still had a unit. | A | `AR/07-backup-architecture.md:1793-1800` |
| 3 | `NONE — chosen`: kept until the household removes it (decision 256). | A | `AR/09-update-architecture.md:1344-1346` |
| 4 | Only the household, or the app's removal with "delete its backups too". | A | `AR/07-backup-architecture.md:1793-1800` |
| 5 | A move that cannot be done refuses the write and leaves the old unit as it was. | A | `AR/07-backup-architecture.md:1793-1800` |
| 6 | The drive's fill watch. | A | `CT/cmd/controller/main.go:2750-2757` |
| 7 | No self-repair, no runbook (the same missing runbook as D1). | N | C2 · S6 |
| 8 | DOC only: no box holds one today. | A | `AR/07-backup-architecture.md:1793-1800` |
---
## TEST-ONLY places — measured, not counted
| Place | Box | Size today | Why it is not a customer place |
|---|---|---|---|
| Scratch guest 9202: data volume, images, volumes | hp | 13.99 GB; 12.79 GB images; 368 volumes (363 unnamed, all unused) | the test guest (`EV/reading-1/hp-9202.tsv`) |
| Drive `scratch_hdd` | hp | — | the scratch guest's drive |
| Test VM disks and installer images on `/mnt/hdd_1` | hp | the filesystem holds 322.7 GB | Tester 1 and the other test VMs live here |
| Proxmox template cache `/var/lib/vz/template` | hp, N100 | 130 MB each (12 KB on T1) | only a box that bakes the guest template downloads one |
| Sampler scripts in `/var/tmp` | hp | 4 small files of 2026-10-14 | left by a test session |
| `traefik/whoami` image | T1 | 6.6 MB | a test tool |
| `alpine:3.20`, `curlimages/curl` images | hp | removed by hand on 2026-11-09 | test tools, not the product (see `README.md`, question 1) |
---
## DOC-ONLY places — second pass
Not measured in this run (no login to ep0, no inventory of DooPlex, no read of the hub database). The marks are
candidates from documents and source; they are NOT in `MATRIX.md`. A helper filled them; the main session did not
read each pointer.
| Place | What the documents say | Candidate gaps |
|---|---|---|
| O1 — the shared off-site box (Hetzner Storage Box, restic) | keep 7 daily, 4 weekly, 6 monthly; hub warns at 80 % / 90 % of the box (`AR/07-backup-architecture.md:400`) | limit and "what when it fills": R-540; a weekly clean-up refused by its own guard: R-1294 |
| O2 — ep0's backup store (PBS datastore) | a 100 GB volume, 13 % used on 2026-07-27; the server prunes | what happens when it is full; a runbook for that |
| O3 — ep0's own system disk | no document found | grow, limit, full, alarm, runbook — all open; clean-up: R-91 |
| O4 — DooPlex's copy of ep0's store | 8 weekly copies (decision 71) | what happens when full; who is alarmed |
| O5 — the hub's database and its snapshots | rows 90 days; the newest 2 snapshots; a 4 GiB volume (`E/manifests/hub.yaml:58`) | what happens when full; who is alarmed |
| O6 — the image registry and the release copy | one image per version for ever; pruned only by hand (decision 62 — decided, not built as a job) | what happens when full; who is alarmed; R-1235, R-1259 |
@@ -0,0 +1,22 @@
# what the controller container sees — read 2026-11-10T07:58:12Z with: pct exec 9201 -- docker exec felhom-controller df -B1 / /opt/docker/stacks /mnt/sys_drive
== demo-hp
Filesystem 1B-blocks Used Available Use% Mounted on
overlay 73793978368 30091509760 39927607296 43% /
/dev/mapper/pve-vm--9201--disk--0 33501757440 1357545472 30409449472 5% /opt/docker/stacks
/dev/mapper/pve-vm--9201--disk--1 73793978368 30091509760 39927607296 43% /mnt/sys_drive
guest-root:
/dev/mapper/pve-vm--9201--disk--0 33501757440 1357545472 30409449472 5% /
== n100
Filesystem 1B-blocks Used Available Use% Mounted on
overlay 263086084096 2472120320 247175413760 1% /
/dev/mapper/pve-vm--9201--disk--2 33501757440 1188171776 30578823168 4% /opt/docker/stacks
/dev/mapper/pve-vm--9201--disk--3 263086084096 2472120320 247175413760 1% /mnt/sys_drive
guest-root:
/dev/mapper/pve-vm--9201--disk--2 33501757440 1188171776 30578823168 4% /
== tester1
Filesystem 1B-blocks Used Available Use% Mounted on
overlay 73793978368 8987238400 61031878656 13% /
/dev/mapper/pve-vm--9201--disk--0 33501757440 1182224384 30584770560 4% /opt/docker/stacks
/dev/mapper/pve-vm--9201--disk--1 73793978368 8987238400 61031878656 13% /mnt/sys_drive
guest-root:
/dev/mapper/pve-vm--9201--disk--0 33501757440 1182224384 30584770560 4% /
@@ -0,0 +1,31 @@
== hp 2026-11-10T07:56:18Z
145383424 /var/lib/docker/volumes/felhom-controller-data/_data
145367040 /var/lib/docker/volumes/felhom-controller-data/_data/data
== n100 2026-11-10T07:56:19Z
84152320 /var/lib/docker/volumes/felhom-controller-data/_data
84144128 /var/lib/docker/volumes/felhom-controller-data/_data/data
== t1 2026-11-10T07:56:20Z
79626240 /var/lib/docker/volumes/felhom-controller-data/_data
79613952 /var/lib/docker/volumes/felhom-controller-data/_data/data
== hp 2026-11-10T07:56:26Z
145367040 .
17801216 ./catalog-cache
20480 ./offbox
4096 ./ctl
23
119607296 2026-11-10 metrics.db
6843352 2026-11-10 metrics.db-wal
== n100 2026-11-10T07:56:27Z
84144128 .
38461440 ./catalog-cache
20480 ./offbox
4096 ./ctl
19
40714240 2026-11-10 metrics.db
== t1 2026-11-10T07:56:28Z
79613952 .
19161088 ./catalog-cache
20480 ./offbox
4096 ./ctl
21
55234560 2026-11-10 metrics.db
@@ -0,0 +1,19 @@
== utc 2026-11-10T07:46:08Z
== controller container size (writable layer / virtual)
felhom-controller 1.3MB (virtual 350MB)
== restic cache folders inside the controller container (size, entries)
/root/.cache/restic 3366912 2
/tmp 4096 0
/var/tmp 4096 0
HOME=/root
XDG_CACHE_HOME=unset
RESTIC_CACHE_DIR=unset
== controller state volume top level (sizes only)
== guest journal
/var/log/journal absent
Archived and active journals take up 8M in the file system.
/run 6290305024 10915840
active active inactive active
tmpfs
== images with no container, by repository (name only)
alpine:latest 8.42MB 4 months ago
@@ -0,0 +1,20 @@
== utc 2026-11-10T07:45:36Z
== controller container size (writable layer / virtual)
felhom-controller 298kB (virtual 349MB)
== restic cache folders inside the controller container (size, entries)
/root/.cache/restic 843776 2
/tmp 4096 0
/var/tmp 4096 0
HOME=/root
XDG_CACHE_HOME=unset
RESTIC_CACHE_DIR=unset
== controller state volume top level (sizes only)
== guest journal
/var/log/journal absent
Archived and active journals take up 8M in the file system.
/run 3307651072 9334784
active active inactive active
tmpfs
== images with no container, by repository (name only)
alpine:latest 8.42MB 4 months ago
hello-world:latest 10.1kB 7 months ago
@@ -0,0 +1,21 @@
== utc 2026-11-10T07:46:11Z
== controller container size (writable layer / virtual)
felhom-controller 468kB (virtual 349MB)
== restic cache folders inside the controller container (size, entries)
/root/.cache/restic 1040384 2
/tmp 4096 0
/var/tmp 4096 0
HOME=/root
XDG_CACHE_HOME=unset
RESTIC_CACHE_DIR=unset
== controller state volume top level (sizes only)
== guest journal
/var/log/journal absent
Archived and active journals take up 8M in the file system.
/run 1650966528 9433088
active active inactive active
tmpfs
== images with no container, by repository (name only)
alpine:latest 8.42MB 7 weeks ago
hello-world:latest 10.1kB 7 months ago
traefik/whoami:v1.10.3 6.58MB 2 years ago
@@ -0,0 +1,44 @@
### demo-hp 2026-11-10T07:41:44Z
## /tmp fs
tmpfs 14.6G 0
## tmpfiles rules for /tmp and /var/tmp
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
## timers
systemd-tmpfiles-clean.timer enabled=static active=active last=Tue 2026-11-10 02:09:53 UTC
logrotate.timer enabled=enabled active=active last=Tue 2026-11-10 00:47:41 UTC
apt-daily.timer enabled=enabled active=active last=Mon 2026-11-09 18:40:41 UTC
apt-daily-upgrade.timer enabled=enabled active=active last=Tue 2026-11-10 06:31:53 UTC
fstrim.timer enabled=enabled active=inactive last=
## logrotate.d
alternatives apt btmp dpkg wtmp wtmpdb
## journald effective
(all defaults)
## apt periodic
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
## pve task index
## lvm autoextend
## docker daemon.json keys (guest)
{
"features": {
"containerd-snapshotter": false
},
"live-restore": true,
"log-driver": "json-file",
"log-opts": {
"max-file": "3",
"max-size": "10m"
}
}
## oldest files in /var/tmp and /tmp (dates only)
2026-11-10 /tmp/.ICE-unix
2026-11-10 /tmp/.X11-unix
2026-11-10 /tmp/.XIM-unix
2026-11-10 /tmp/.font-unix
2026-11-10 /tmp/systemd-private-09206f2068524a2e881d5328e0bbc345-systemd-logind.service-g8WCve
2026-11-10 /var/tmp/network
2026-11-10 /var/tmp/systemd-private-09206f2068524a2e881d5328e0bbc345-systemd-logind.service-Q0avKv
@@ -0,0 +1,50 @@
### demo-hp 2026-11-10T07:41:42Z
## /tmp fs
tmpfs 14.6G 0
## tmpfiles rules for /tmp and /var/tmp
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
## timers
systemd-tmpfiles-clean.timer enabled=static active=active last=Tue 2026-11-10 03:09:02 CET
logrotate.timer enabled=enabled active=active last=Tue 2026-11-10 00:27:01 CET
apt-daily.timer enabled=enabled active=active last=Tue 2026-11-10 04:06:22 CET
apt-daily-upgrade.timer enabled=enabled active=active last=Tue 2026-11-10 06:55:09 CET
fstrim.timer enabled=enabled active=active last=Mon 2026-11-09 01:25:39 CET
## logrotate.d
alternatives apt btmp ceph-common chrony corosync dpkg frr pve pve-firewall wtmp wtmpdb
## journald effective
(all defaults)
## apt periodic
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve-signed$";
## pve task index
260 /var/log/pve/tasks/index
650 /var/log/pve/tasks/index.1
910 total
2026-10-28
2026-11-10
## lvm autoextend
thin_pool_autoextend_threshold=100
thin_pool_autoextend_percent=20
## docker daemon.json keys (guest)
## oldest files in /var/tmp and /tmp (dates only)
2026-10-14 /var/tmp/guest_sampler.sh
2026-10-14 /var/tmp/host_sampler.sh
2026-11-10 /tmp/.font-unix
2026-11-10 /tmp/.ICE-unix
2026-11-10 /tmp/systemd-private-b6e0edea2b5647dfa36f5bb3825adc3a-chrony.service-Tzw1xO
2026-11-10 /tmp/systemd-private-b6e0edea2b5647dfa36f5bb3825adc3a-systemd-logind.service-AqauM3
2026-11-10 /tmp/.X11-unix
2026-11-10 /tmp/.XIM-unix
@@ -0,0 +1,44 @@
### demo-felhom 2026-11-10T07:41:13Z
## /tmp fs
tmpfs 7.7G 0
## tmpfiles rules for /tmp and /var/tmp
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
## timers
systemd-tmpfiles-clean.timer enabled=static active=active last=Tue 2026-11-10 04:51:34 UTC
logrotate.timer enabled=enabled active=active last=Tue 2026-11-10 00:54:14 UTC
apt-daily.timer enabled=enabled active=active last=Tue 2026-11-10 06:49:16 UTC
apt-daily-upgrade.timer enabled=enabled active=active last=Tue 2026-11-10 06:32:16 UTC
fstrim.timer enabled=enabled active=inactive last=
## logrotate.d
alternatives apt btmp dpkg wtmp wtmpdb
## journald effective
(all defaults)
## apt periodic
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
## pve task index
## lvm autoextend
## docker daemon.json keys (guest)
{
"features": {
"containerd-snapshotter": false
},
"live-restore": true,
"log-driver": "json-file",
"log-opts": {
"max-file": "3",
"max-size": "10m"
}
}
## oldest files in /var/tmp and /tmp (dates only)
2026-11-08 /tmp/.ICE-unix
2026-11-08 /tmp/.X11-unix
2026-11-08 /tmp/.XIM-unix
2026-11-08 /tmp/.font-unix
2026-11-08 /tmp/systemd-private-84664f34e28e46b5ac5b6370374b6a84-systemd-logind.service-cjq9uj
2026-11-08 /var/tmp/network
2026-11-08 /var/tmp/systemd-private-84664f34e28e46b5ac5b6370374b6a84-systemd-logind.service-QWfar9
@@ -0,0 +1,48 @@
### demo-felhom 2026-11-10T07:41:12Z
## /tmp fs
tmpfs 7.7G 0
## tmpfiles rules for /tmp and /var/tmp
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
## timers
systemd-tmpfiles-clean.timer enabled=static active=active last=Tue 2026-11-10 05:50:32 CET
logrotate.timer enabled=enabled active=active last=Tue 2026-11-10 00:21:32 CET
apt-daily.timer enabled=enabled active=active last=Tue 2026-11-10 00:20:45 CET
apt-daily-upgrade.timer enabled=enabled active=active last=Tue 2026-11-10 06:03:00 CET
fstrim.timer enabled=enabled active=active last=Mon 2026-11-09 01:32:30 CET
## logrotate.d
alternatives apt btmp ceph-common chrony corosync dpkg frr pve pve-firewall wtmp wtmpdb
## journald effective
(all defaults)
## apt periodic
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve-signed$";
## pve task index
114 /var/log/pve/tasks/index
588 /var/log/pve/tasks/index.1
702 total
2026-09-12
2026-11-10
## lvm autoextend
thin_pool_autoextend_threshold=100
thin_pool_autoextend_percent=20
## docker daemon.json keys (guest)
## oldest files in /var/tmp and /tmp (dates only)
2026-11-08 /tmp/.font-unix
2026-11-08 /tmp/.ICE-unix
2026-11-08 /tmp/systemd-private-72899d2625244e069e8e8099775d1e51-systemd-logind.service-2RHCoj
2026-11-08 /tmp/.X11-unix
2026-11-08 /tmp/.XIM-unix
2026-11-08 /var/tmp/systemd-private-72899d2625244e069e8e8099775d1e51-systemd-logind.service-p1H9e5
@@ -0,0 +1,39 @@
### tester-1 2026-11-10T07:41:47Z
## /tmp fs
tmpfs 3.8G 0
## tmpfiles rules for /tmp and /var/tmp
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
## timers
systemd-tmpfiles-clean.timer enabled=static active=active last=Tue 2026-11-10 02:10:36 UTC
logrotate.timer enabled=enabled active=active last=Tue 2026-11-10 00:58:57 UTC
apt-daily.timer enabled=enabled active=active last=Mon 2026-11-09 18:35:38 UTC
apt-daily-upgrade.timer enabled=enabled active=active last=Tue 2026-11-10 06:14:46 UTC
fstrim.timer enabled=enabled active=inactive last=
## logrotate.d
alternatives apt btmp dpkg wtmp wtmpdb
## journald effective
(all defaults)
## apt periodic
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
## pve task index
## lvm autoextend
## docker daemon.json keys (guest)
{
"features": { "containerd-snapshotter": false },
"live-restore": true,
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "3" }
}
## oldest files in /var/tmp and /tmp (dates only)
2026-11-10 /tmp/.ICE-unix
2026-11-10 /tmp/.X11-unix
2026-11-10 /tmp/.XIM-unix
2026-11-10 /tmp/.font-unix
2026-11-10 /tmp/systemd-private-9c0247fd2659403fbafc3e8edacc25e7-systemd-logind.service-IVpm4X
2026-11-10 /var/tmp/network
2026-11-10 /var/tmp/systemd-private-9c0247fd2659403fbafc3e8edacc25e7-systemd-logind.service-64Jtca
@@ -0,0 +1,53 @@
### felhom 2026-11-10T07:41:46Z
## /tmp fs
tmpfs 3.8G 0
## tmpfiles rules for /tmp and /var/tmp
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
## timers
systemd-tmpfiles-clean.timer enabled=static active=active last=Tue 2026-11-10 03:09:30 CET
logrotate.timer enabled=enabled active=active last=Tue 2026-11-10 00:42:19 CET
apt-daily.timer enabled=enabled active=active last=Tue 2026-11-10 07:10:39 CET
apt-daily-upgrade.timer enabled=enabled active=active last=Tue 2026-11-10 06:06:30 CET
fstrim.timer enabled=enabled active=active last=Mon 2026-11-09 01:13:20 CET
## logrotate.d
alternatives apt btmp ceph-common chrony corosync dpkg frr pve pve-firewall wtmp wtmpdb
## journald effective
(all defaults)
## apt periodic
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-20-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-20-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-20-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-22-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-22-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-22-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.2-6-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.2-6-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.2-6-pve-signed$";
## pve task index
218 /var/log/pve/tasks/index
655 /var/log/pve/tasks/index.1
873 total
2026-10-26
2026-11-10
## lvm autoextend
thin_pool_autoextend_threshold=100
thin_pool_autoextend_percent=20
## docker daemon.json keys (guest)
## oldest files in /var/tmp and /tmp (dates only)
2026-11-10 /tmp/.font-unix
2026-11-10 /tmp/.ICE-unix
2026-11-10 /tmp/systemd-private-bf6ac6018a8e4819beed5a725e3967f2-chrony.service-riOfpD
2026-11-10 /tmp/systemd-private-bf6ac6018a8e4819beed5a725e3967f2-systemd-logind.service-7mVQXv
2026-11-10 /tmp/.X11-unix
2026-11-10 /tmp/.XIM-unix
2026-11-10 /var/tmp/systemd-private-bf6ac6018a8e4819beed5a725e3967f2-chrony.service-PseIro
2026-11-10 /var/tmp/systemd-private-bf6ac6018a8e4819beed5a725e3967f2-systemd-logind.service-h85dy4
@@ -0,0 +1,630 @@
### identity
demo-hp
2026-11-10T07:26:54Z
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
### df bytes
Filesystem Type 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-vm--9201--disk--0 ext4 33501757440 1357545472 30409449472 5% /
/dev/mapper/pve-vm--9201--disk--1 ext4 73793978368 30090113024 39929004032 43% /var/lib/felhom
/dev/mapper/pve-root ext4 41424257024 23968079872 15318970368 62% /mnt/felhom-drives
/dev/nvme0n1 ext4 1006980812800 322734333952 633019228160 34% /mnt/felhom-drives/hdd_1
### df inodes
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pve-vm--9201--disk--0 2097152 25024 2072128 2% /
/dev/mapper/pve-vm--9201--disk--1 4587520 1030944 3556576 23% /var/lib/felhom
/dev/mapper/pve-root 2588672 120141 2468531 5% /mnt/felhom-drives
/dev/nvme0n1 62513152 796 62512356 1% /mnt/felhom-drives/hdd_1
### docker info
driver=overlay2 root=/var/lib/docker logdriver=json-file images=43 containers=26 running=26 server=29.9.3
### daemon.json
{
"features": {
"containerd-snapshotter": false
},
"live-restore": true,
"log-driver": "json-file",
"log-opts": {
"max-file": "3",
"max-size": "10m"
}
}
total 12
drwxr-xr-x 2 root root 4096 2026-08-18 10:51 .
drwxr-xr-x 73 root root 4096 2026-11-10 06:41 ..
-rw-r--r-- 1 root root 177 2026-10-04 14:09 daemon.json
### docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 43 23 24.15GB 10.38GB (42%)
Containers 26 26 128.7MB 0B (0%)
Local Volumes 1914 29 1.531GB 108.4MB (7%)
Build Cache 0 0 0B 0B
### containers (name image state logdriver logopts)
/adventurelog ghcr.io/seanmorley15/adventurelog-backend:v0.13.1@sha256:8c45e21f95875ccd35a6e68f7da72504df9e837d6ee703379512cb8de6a67082 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:15:12.266135097Z
/adventurelog-frontend ghcr.io/seanmorley15/adventurelog-frontend:v0.13.1@sha256:568b1be1c42f456df066a8bb55be8ce0046b2c6fc96b1650db5d2162eb4f1ce9 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:15:12.049621947Z
/adventurelog-postgres postgis/postgis:16-3.5-alpine@sha256:47e961a569fd52ff31f0fe205ed91eeab17d9f5fff6722e6d7ea6b588748b293 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:15:12.049901044Z
/bentopdf ghcr.io/alam00000/bentopdf:v2.8.10@sha256:039239902a01e58a5a0f52e29039fafa18ffc2a481bb8acd807599f1c6dde7ef running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T01:35:50.652437385Z
/bookstack lscr.io/linuxserver/bookstack:26.09.2@sha256:2fdeeb1a2028ce7cb1274696237e32b9f849e16b6d77314d08c78356a6016fca running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:15:27.414180295Z
/bookstack-db mariadb:12.3@sha256:d46e102fa8b9b72bfcdeb56f9dd538742092119dc8b11f31fe662605cd234cd9 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:15:27.353891135Z
/calibre-web crocodilestick/calibre-web-automated:v4.0.8@sha256:5e00373854247750cc3e4479b492ae09293ff5e06ed10177f226634d97888679 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:15:38.335063304Z
/cloudflared cloudflare/cloudflared:2026.10.4 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-10-27T15:41:55.342428957Z
/docmost docmost/docmost:0.97.0@sha256:40c454d74ffdd9ecb15833e4d96f3a630d59aad862a1869381ae6ee086454419 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:15:46.32825246Z
/docmost-postgres postgres:18-alpine@sha256:10a5c69a83035ab5239f5620d708520e8b28f5aa1f2b20143e6024532edadfb5 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:15:46.17595913Z
/docmost-redis redis:7-alpine@sha256:6ba19c50b56777f5c11c58eff8ddb69d0d5f27ba131cffb63f180ccfd43c66af running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:15:46.175998855Z
/felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.351.0 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T01:54:28.879820485Z
/filebrowser gtstef/filebrowser:1.5.8-stable running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-10-27T15:41:42.297239117Z
/home-assistant ghcr.io/home-assistant/home-assistant:2026.10.4@sha256:23f4dfbb69ce76315dbc4de7915504f1c2dcecd7ca2648113a493a385972e133 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:16:08.759436825Z
/kimai kimai/kimai2:2.67.0@sha256:3084f1e5ecdc10afafc193911039a06cdabda4473f3a48d62edbe460e3c154ee running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:16:14.968458478Z
/kimai-db mariadb:11.8@sha256:79d59758afc91b89b120b0a8904d637f5a3b3e1c4900f29b740d6d46c72fef68 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:16:14.784144041Z
/opengist ghcr.io/thomiceli/opengist:1.15 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:16:23.016150231Z
/paperless-postgres postgres:18-alpine@sha256:10a5c69a83035ab5239f5620d708520e8b28f5aa1f2b20143e6024532edadfb5 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:16:34.625997254Z
/paperless-redis redis:7-alpine@sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:16:34.62586807Z
/paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:3.4.1@sha256:1a08d8dfc2b51a08de09652dd86ad4d0c611c362382f944533ad38b73125fbc4 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:16:34.666335933Z
/plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:16:57.27660352Z
/privatebin privatebin/pdo:2.1.0@sha256:3400b0a4714d62161727a710c82a3e319cfea364ac6e591aed653f5bb6bb8055 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:16:58.550098967Z
/romm rommapp/romm:5.4.1@sha256:8677fb7b81b1d3db20b849e989a1dadc2a11159e51eb34a2acdcb01102274edf running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:17:11.005693511Z
/romm-db mariadb:11.8@sha256:c4124d1afa875e47f2892edf26b482fa54f796e2f342d3717a04ca091a22a9bd running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:17:10.958253807Z
/romm-redis redis:7-alpine@sha256:6ba19c50b56777f5c11c58eff8ddb69d0d5f27ba131cffb63f180ccfd43c66af running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T03:17:10.958292761Z
/traefik traefik:v3.7.17 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-03T07:04:40.937691489Z
### container log files (bytes path) and total
10000042 2026-11-10 /var/lib/docker/containers/c797bb7dfdba57ee05a4d687cb715b172c14998c012db9671650f71884775216/c797bb7dfdba57ee05a4d687cb715b172c14998c012db9671650f71884775216-json.log.1
6291478 2026-11-10 /var/lib/docker/containers/f6446630708007c3b2f97a8d62043193c6516a7a62daf31037e42989a51c9393/f6446630708007c3b2f97a8d62043193c6516a7a62daf31037e42989a51c9393-json.log
4097540 2026-11-10 /var/lib/docker/containers/c797bb7dfdba57ee05a4d687cb715b172c14998c012db9671650f71884775216/c797bb7dfdba57ee05a4d687cb715b172c14998c012db9671650f71884775216-json.log
3095011 2026-11-10 /var/lib/docker/containers/fc0bfe5b53c6a7711d5d742b8ba9c4b3fbdbf96da7270311ffcf4e23f28817c8/fc0bfe5b53c6a7711d5d742b8ba9c4b3fbdbf96da7270311ffcf4e23f28817c8-json.log
1640394 2026-11-10 /var/lib/docker/containers/549f296e6ef9f6594f5e8789ecb86579f0c6ee45091b38bafdca69b50377d27a/549f296e6ef9f6594f5e8789ecb86579f0c6ee45091b38bafdca69b50377d27a-json.log
279496 2026-11-10 /var/lib/docker/containers/b053ad939c00b7e23519efd9ea129fe0a5e26517be28852dfd6a1398c6efb332/b053ad939c00b7e23519efd9ea129fe0a5e26517be28852dfd6a1398c6efb332-json.log
124103 2026-11-10 /var/lib/docker/containers/9db9aba521daf9f19352a64511f4f84872ade957b643c0db40dd510837143cba/9db9aba521daf9f19352a64511f4f84872ade957b643c0db40dd510837143cba-json.log
104953 2026-11-10 /var/lib/docker/containers/968c315833e121d6608bddcd948180426de6d2e0888db6d8f4905fad399bc238/968c315833e121d6608bddcd948180426de6d2e0888db6d8f4905fad399bc238-json.log
90170 2026-11-10 /var/lib/docker/containers/8322e02d1221f8794c7b8a8104ea2cb66806d8700af1125b447741403eccc56f/8322e02d1221f8794c7b8a8104ea2cb66806d8700af1125b447741403eccc56f-json.log
37417 2026-11-10 /var/lib/docker/containers/d336700d026f007079bcd4ee6340c13d0a49e1205d5373215617d6599110e60f/d336700d026f007079bcd4ee6340c13d0a49e1205d5373215617d6599110e60f-json.log
36854 2026-11-10 /var/lib/docker/containers/187f15c8a63373481dbb0cf0114423c752bbe77b0462339f978bc1de96ba5e35/187f15c8a63373481dbb0cf0114423c752bbe77b0462339f978bc1de96ba5e35-json.log
36719 2026-11-10 /var/lib/docker/containers/447c20f03f192c72465aaaf3f2fca4e64e092aa49f1aa9b8953f729059bb39ef/447c20f03f192c72465aaaf3f2fca4e64e092aa49f1aa9b8953f729059bb39ef-json.log
28120 2026-11-10 /var/lib/docker/containers/c97d99b7db538e808f914fb0f42bacccdf3ef83e47164afde3c4f780874e97af/c97d99b7db538e808f914fb0f42bacccdf3ef83e47164afde3c4f780874e97af-json.log
17718 2026-11-10 /var/lib/docker/containers/2e5cbfc47ac7bd7807b0bb753c8b3591d78238ea64a90757b7e725bc2ccde5c7/2e5cbfc47ac7bd7807b0bb753c8b3591d78238ea64a90757b7e725bc2ccde5c7-json.log
14783 2026-11-10 /var/lib/docker/containers/858bf53bc3e466c511c97b2c8a43b5aae4e89d446444ccfabc000eefaf5f02b3/858bf53bc3e466c511c97b2c8a43b5aae4e89d446444ccfabc000eefaf5f02b3-json.log
6566 2026-11-10 /var/lib/docker/containers/828c2184abd99e9f39a814c6e6aa8131af71512d8ad385d05b194d5fd6a9630c/828c2184abd99e9f39a814c6e6aa8131af71512d8ad385d05b194d5fd6a9630c-json.log
6392 2026-11-10 /var/lib/docker/containers/c84eb2df107ab9aa8314855244ff58f820619ef451c207589191c8dbbc419fc5/c84eb2df107ab9aa8314855244ff58f820619ef451c207589191c8dbbc419fc5-json.log
5531 2026-11-10 /var/lib/docker/containers/a40c3fa7da67ebd86319ba097b5a101c045c08642aa7e211d692ccd1daef7f75/a40c3fa7da67ebd86319ba097b5a101c045c08642aa7e211d692ccd1daef7f75-json.log
5104 2026-11-10 /var/lib/docker/containers/4d277d98bedad2e3ae9384a87556e70abdcdc092a17951de0dad744b856ac043/4d277d98bedad2e3ae9384a87556e70abdcdc092a17951de0dad744b856ac043-json.log
4858 2026-11-10 /var/lib/docker/containers/d7782ccb24b778787378d3cd054d8ca8759734eec5992dee08cfec7db21e1146/d7782ccb24b778787378d3cd054d8ca8759734eec5992dee08cfec7db21e1146-json.log
4598 2026-11-10 /var/lib/docker/containers/f60105af0a82c0af7a1d785978700c558bce19260918e02cf12a98b67b3c6cd2/f60105af0a82c0af7a1d785978700c558bce19260918e02cf12a98b67b3c6cd2-json.log
2089 2026-11-10 /var/lib/docker/containers/1285eb73346d5bc1fa459e37199bac73d8991ed8d040a257c16e6d4366cfd0c6/1285eb73346d5bc1fa459e37199bac73d8991ed8d040a257c16e6d4366cfd0c6-json.log
1959 2026-11-10 /var/lib/docker/containers/b64621bdb89baa6c1b17876faab6dada5eefd43468478efff58716b570f31b3a/b64621bdb89baa6c1b17876faab6dada5eefd43468478efff58716b570f31b3a-json.log
1810 2026-11-10 /var/lib/docker/containers/85f7cbec620ce39a46aa3a0f0b1797996f269b2aca27bcb0957c66e7f36df75f/85f7cbec620ce39a46aa3a0f0b1797996f269b2aca27bcb0957c66e7f36df75f-json.log
1794 2026-11-10 /var/lib/docker/containers/84695db927f589bd56cc172040471feb7e04ecf77a6ad00d452cf9338f3019eb/84695db927f589bd56cc172040471feb7e04ecf77a6ad00d452cf9338f3019eb-json.log
332 2026-11-10 /var/lib/docker/containers/010e86f8bbab26b8fe8646009f39e49b9c46f12e95e8fe8d93bc29872f6bdf3b/010e86f8bbab26b8fe8646009f39e49b9c46f12e95e8fe8d93bc29872f6bdf3b-json.log
103 2026-11-10 /var/lib/docker/containers/4a555f9dd8b0141c144591c445972c834e7713e8dbd170e33c7edb04f2f0db2a/4a555f9dd8b0141c144591c445972c834e7713e8dbd170e33c7edb04f2f0db2a-json.log
TOTAL bytes=25935934 files=27
### container writable layers (docker ps -as)
adventurelog 298kB (virtual 852MB)
adventurelog-frontend 0B (virtual 285MB)
adventurelog-postgres 63B (virtual 524MB)
bentopdf 1.08kB (virtual 321MB)
bookstack 28.3kB (virtual 454MB)
bookstack-db 2B (virtual 334MB)
calibre-web 37MB (virtual 1.76GB)
cloudflared 0B (virtual 64.6MB)
docmost 8.23kB (virtual 760MB)
docmost-postgres 68B (virtual 304MB)
docmost-redis 0B (virtual 39.1MB)
felhom-controller 1.3MB (virtual 350MB)
filebrowser 92.5kB (virtual 195MB)
home-assistant 41.9MB (virtual 2.39GB)
kimai 75.8kB (virtual 798MB)
kimai-db 2B (virtual 329MB)
opengist 9.57kB (virtual 94.4MB)
paperless-postgres 68B (virtual 304MB)
paperless-redis 0B (virtual 39.1MB)
paperless-webserver 48MB (virtual 2.53GB)
plex 9.88kB (virtual 357MB)
privatebin 0B (virtual 79.7MB)
romm 5.8kB (virtual 838MB)
romm-db 2B (virtual 329MB)
romm-redis 0B (virtual 39.1MB)
traefik 0B (virtual 191MB)
### images
alpine:latest d529dd0c6e55 2026-06-16 00:01:29 +0000 UTC 8.42MB
cloudflare/cloudflared:2026.10.3 1475624a6d23 2026-10-21 13:26:38 +0000 UTC 64.6MB
cloudflare/cloudflared:2026.10.4 472341c39b02 2026-10-26 11:27:24 +0000 UTC 64.6MB
crocodilestick/calibre-web-automated:<none> 43505abb1b0e 2026-09-28 15:27:40 +0000 UTC 1.72GB
crocodilestick/calibre-web-automated:v4.0.6 5ef521449e62 2026-02-04 23:42:57 +0000 UTC 1.72GB
docmost/docmost:0.96.0 0f146d72f36f 2026-09-08 15:49:25 +0000 UTC 753MB
docmost/docmost:<none> 3d1b72da191d 2026-10-25 13:47:04 +0000 UTC 760MB
ghcr.io/alam00000/bentopdf:<none> 5bcdb20ee8f0 2026-10-19 10:09:21 +0000 UTC 321MB
ghcr.io/alam00000/bentopdf:v2.8.6 4baadf01bd68 2026-06-28 06:30:01 +0000 UTC 287MB
ghcr.io/home-assistant/home-assistant:<none> 8e7d93e48bed 2026-10-24 20:09:10 +0000 UTC 2.35GB
ghcr.io/home-assistant/home-assistant:<none> cd74b0e02cee 2026-09-27 09:57:02 +0000 UTC 2.34GB
ghcr.io/paperless-ngx/paperless-ngx:2.20.15 623d64064a8d 2026-04-27 02:45:05 +0000 UTC 1.42GB
ghcr.io/paperless-ngx/paperless-ngx:<none> 354dc6ebeb86 2026-10-31 05:27:36 +0000 UTC 2.48GB
ghcr.io/seanmorley15/adventurelog-backend:<none> 58a19bf88da4 2026-10-27 20:28:54 +0000 UTC 852MB
ghcr.io/seanmorley15/adventurelog-backend:<none> 5b94cc2b3a36 2026-08-12 00:37:32 +0000 UTC 821MB
ghcr.io/seanmorley15/adventurelog-frontend:<none> 4951bea7f9f0 2026-10-27 20:42:25 +0000 UTC 285MB
ghcr.io/seanmorley15/adventurelog-frontend:<none> dfb732cc96ca 2026-08-11 22:59:14 +0000 UTC 296MB
ghcr.io/thomiceli/opengist:1.15 0599ba5a0357 2026-08-30 02:07:48 +0000 UTC 94.4MB
gitea.dooplex.hu/admin/felhom-controller:0.350.0 1f32a260789a 2026-11-03 07:04:04 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.351.0 ed7f096c3ff7 2026-11-09 12:14:14 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-samba:1.2.0 fc741cdd77e4 2026-10-16 18:25:34 +0000 UTC 107MB
gtstef/filebrowser:1.5.6-stable 96e26edeefc3 2026-09-04 21:33:58 +0000 UTC 195MB
gtstef/filebrowser:1.5.8-stable 0019084d4a6f 2026-10-04 21:15:40 +0000 UTC 195MB
kimai/kimai2:<none> 4175fc5e54be 2026-09-13 20:43:55 +0000 UTC 798MB
lscr.io/linuxserver/bookstack:<none> b274bee3f111 2026-09-30 06:56:35 +0000 UTC 416MB
lscr.io/linuxserver/bookstack:<none> f6f94cac70ef 2026-10-26 22:30:17 +0000 UTC 453MB
mariadb:11.8 cc7f5fdd7c8f 2026-09-16 03:26:24 +0000 UTC 329MB
mariadb:12.3 dfff46ef3f9d 2026-09-16 03:26:24 +0000 UTC 334MB
mariadb:<none> 0f2506b4b989 2026-10-26 23:24:57 +0000 UTC 329MB
mariadb:<none> 2215e9af9663 2026-10-26 23:24:58 +0000 UTC 334MB
plexinc/pms-docker:1.41.4.9463-630c9f557 0186be6d1305 2025-02-07 10:23:03 +0000 UTC 351MB
plexinc/pms-docker:1.43.5.11029-6bf8af3f0 ad86d2872084 2026-10-13 07:57:12 +0000 UTC 357MB
postgis/postgis:16-3.5-alpine 0dbac35d4dae 2026-08-31 11:36:26 +0000 UTC 524MB
postgres:<none> 57b23a0326ee 2026-10-21 22:56:40 +0000 UTC 304MB
postgres:<none> c293117fcecd 2026-09-17 21:29:50 +0000 UTC 304MB
privatebin/pdo:2.0.6 7aa03d985415 2026-08-08 06:24:54 +0000 UTC 73.2MB
privatebin/pdo:<none> d108bf546a18 2026-10-21 21:13:05 +0000 UTC 79.7MB
redis:7-alpine f84b0c467801 2026-09-17 21:32:46 +0000 UTC 39.1MB
redis:<none> 98c6ca8f397f 2026-10-26 16:40:19 +0000 UTC 39.1MB
rommapp/romm:<none> eab97d88d9d7 2026-09-23 03:08:21 +0000 UTC 776MB
rommapp/romm:<none> eb5e6a168d0d 2026-10-22 22:52:28 +0000 UTC 838MB
traefik:v3.7.16 3733df53e74c 2026-10-21 22:55:46 +0000 UTC 194MB
traefik:v3.7.17 a72cb2f4a210 2026-10-30 20:17:26 +0000 UTC 191MB
### images count / dangling
22
21
### images in use by a container
1 cloudflare/cloudflared:2026.10.4
1 crocodilestick/calibre-web-automated:v4.0.8
1 docmost/docmost:0.97.0
1 ghcr.io/alam00000/bentopdf:v2.8.10
1 ghcr.io/home-assistant/home-assistant:2026.10.4
1 ghcr.io/paperless-ngx/paperless-ngx:3.4.1
1 ghcr.io/seanmorley15/adventurelog-backend:v0.13.1
1 ghcr.io/seanmorley15/adventurelog-frontend:v0.13.1
1 ghcr.io/thomiceli/opengist:1.15
1 gitea.dooplex.hu/admin/felhom-controller:0.351.0
1 gtstef/filebrowser:1.5.8-stable
1 kimai/kimai2:2.67.0
1 lscr.io/linuxserver/bookstack:26.09.2
2 mariadb:11.8
1 mariadb:12.3
1 plexinc/pms-docker:1.43.5.11029-6bf8af3f0
1 postgis/postgis:16-3.5-alpine
2 postgres:18-alpine
1 privatebin/pdo:2.1.0
3 redis:7-alpine
1 rommapp/romm:5.4.1
1 traefik:v3.7.17
### volumes (count, anonymous count, dangling count)
1914
1887
1885
### volumes by size (system df -v, Local Volumes block)
Local Volumes space usage:
VOLUME NAME LINKS SIZE
03f49885654f407215ddb3c58cde91693f7687adef28fb47c87af4b10e5bd095 0 0B
1232a067ff189567ef6a16b34bdec4e5d1e4c403b10fc3446565cf7b584f550a 0 0B
74de5ace7242b0f56561d7c8506086c6dd0a1fed53806f8dcacc0cb6db870a98 0 0B
7904dd7ba84638d7d167fb4dd3de121e5cb061309e32f3b7ef309ea3dcb6d55c 0 0B
e1df2945fd7d48d0b7aa301d4ef277dda23a219b4b0d15b98afd9978530b230f 0 0B
3e189f460299cbe4ee15853afed4959fa5acec6f440467989448511d822f956e 0 0B
48ca5870749adda5a207baf4ff9d5cdc414fdc430a960eea29d56bc5bf5172a0 0 0B
e27edefe11edbd285405c171bb2432abbf32270edaf4f06d2b179aec38aae06b 0 0B
a6812a23d9ec7fb37ddaf6ccc20fdff9ab04643c4297353e9b3f5dd8a9f18f5a 0 0B
db2f86e3d8afe9529fb3fcd4f1c5078a71268af835520008f8d8a84113eb5d8d 0 0B
294f21a947a881e5880a55a3b9ed07c0915867f29d7f93fe3c91212d90ca5e8c 0 0B
231155085e48f73acf3f1cc4209ff7e137d59d8872f5c2c94651cd57397e16ba 0 0B
95f0dd2891731bfb2a4d17dbac857edb7b183a5367048530341d1c1b8bc98d88 0 340B
979743f08e565b497e38f123831cda56df679bd300514b7e542cea1b79c90a4b 0 0B
c83a5d777a79c0cf77429f4d31d7e9ca4dc813d9c824a2c7c109c922f4c2aab9 0 0B
e4bc38eeaa2c9c9526e7bb96297f8d84660a8a5bec3886149d5f881147ce7901 0 0B
f1813b8b800591c3ab7c8f53c9b29c7450d2fbd2df9977f8a14b469e640f5117 0 0B
2653a3772b013fd42f2966f874d92ffa488f488cc09c234228fa431cfeb493e3 0 0B
5076ffed7f06eb28d943449241a2c40801975a83f3219973c892888ca33373f7 0 340B
d4dd9cda36f3fcf62f55f97e5ab661ba7b3fedeabf530fbf197264c92c174a8f 0 340B
e10c70ecc3e20d36da59d4a222d10d86a6f2afdb08527651627397d25369e49b 0 340B
8fd13a1ae4d1999926d7043d587b4c58832842c652a2d39583526879d8829733 0 0B
b4dbb3cb8e4d89d4bc7ac16515ae28d438c282c80e09362c2103ecdba0d06e47 0 0B
0addbb76a767dc23520c3656e89457ad53b3c308c113c1b32fe902372b33f619 0 0B
33a5887af40ce9d6b5ad30f431db09dae55f82e7abf15e2d8189dccee7212930 0 340B
ac0d73f902a3959fb38475d75eb9c2a9209c4b573613037406c13b664b26fdb7 0 0B
dc7a46b3244a0864ae6b89f638b13c4180e6dcfbc21d08caf3c70857736f78c3 0 0B
ef208802f3d712a63a6ffcd635e271c7f7103d39f185b9eb69669ed1365404b2 0 0B
3f1aa2693ddfa5824829c8d875e9bbc89025932974bd76f8689359a836535b06 0 0B
66db5fbe09bfa9169bd4de912f63aad411a4be4b49f2d3e4f390e7de52feba47 0 0B
96b8738b41e354965e0204479f06b6eae4d08062cf5df73aaa150981f5096e69 0 340B
filebrowser_filebrowser_data 1 135.6kB
e0509819ce8118d65b9d67b8de3510f9198a6bb9cc10afeb148c88b06dbcc58c 0 0B
3d744888a10c5ce52927f37fd4c1e26f8c58ed6bb73955d45b41e86586cd9221 0 0B
aea91708c67f085851c0e0fc819abd8ce7e7fa439d98ee428842994e390971a9 0 0B
6445a3c2c18f80fdeda1d2de2652be8872e5795b2f2a764d967f2b283ab7f4b9 0 0B
d285cec811cd6ccfc98ef5bf8dc7b530efc62c523df3057912d7f7e3ea8f39f6 0 0B
f2b5603dcddacc1223dd58ddac49e94c4e5ecd5506b8a24025aba15935679545 0 0B
2b8c965407a42bca225b507cb4e43aced6d07b8dd6f29f365a9c082c305abaf2 0 0B
459a42b6307f5eae7892874d5688dff03dea515d2d82287a0ae5e22d6fc2bd90 0 340B
58d29c2b737cc3235615693e50070efc33377b61e03bfc9686c261cf14a89415 0 0B
67c9f6eae61e4a49244c63e8a4e5027ff86b02e5908f7b67d8618049b2bd2b98 0 0B
aefd819c5739bad666281c4f94f29eb89286fdd2595f318ae0feb85998c6becb 0 340B
d81755320ce6ddd8188f8d949c30f9a482defe01154336c58b93df3fde6d37aa 0 0B
22d2081e29878d3281b5693d9f08900f514e8bd8c544c418634ea54816cd7871 0 0B
91926f9cbcbb8606b0b73f08e7fcd4573a7f20c9b16d6943bbdeb8004b82b7e7 0 0B
e8e64c9463aa3b9c2b7841e1a4ddf106238692648d24fedd10a1280c437d7c70 0 0B
fea1595ce86f9815969a1072806dad3d855522530da83111bffb81d992ad998b 0 0B
30fdd1a604264ec7c1695504f51bf7c7bfca30e60fd35841df6170ae1aa04cb3 0 0B
39feb1fe61e110a54fe65151bc1f3e4fe0bfd256df44b0b92f2ee13809e31d98 0 0B
54dcff1ab56ddf788ba41226fe94e8b3019ecd1ddc6bdde877838c1365ca1ba6 0 340B
73133cf3e0b8c85111a76d38d8525b383f565cd48a78a02067130386536dd57d 0 0B
bcf9e434c796c50406ff1160220ee22f8caa2f4c8742f38a0459dde7d1e8213f 0 0B
4f1bbf0673fa77fb687ee749005b74aed57ee1386731edf9f19cd8c1729e269d 0 0B
6616b4fcdd42ae5115a75284de27081ac78b3eb66b1ef81a10d7b7e433f5f56b 0 0B
d21cbf8a753f724f8030c94d163cc7ce0c5f22d233f64f4842caf4533355f4df 0 0B
0f9051c4da8c562072e97a3ff57d36dadd9377699effea716b3eb8a9be4bac93 0 0B
1847c5a9615c016abd38b7882cda9f6093c7264a8bb70a42c3e7d42b2c19f9d1 0 0B
9e82f19c92f34b7271cbf7e23d8b3726d88dddeaf0207afe3920abbf4265d0f6 0 340B
cc40a6a2e50213a5f03588f3e57cc5c148e5a03922a930cf71eae41e1607c8ab 0 0B
feb4b6601fe43f8f1b0628817a03181c117b8649aa3f0ed5f9ac19ea713079f1 0 340B
1324df64a3999c5d2f4c1d4e1e3dfda696ad47fa74e9c18c66320cc908e6fca4 0 0B
4dc4f9dceb3bd114f3e0009fba5b85f35bb48df6a140a75ed713b1f980b5c732 0 0B
5260886abbbbee9907ff5ed4aea23a281bdf09e3ef54db4dd9496841244511e0 0 0B
af623d7f811f81ac856fbdaf86e7f0baee0c7644b114662d4f4bc4f0d8938f68 0 340B
b0a5a369cff18d4a340ae36924bf2a7a8b5890985cb6d992caaabe033b1510f5 0 0B
0ce6f0d7485bd5a326d8cd7ad69c69153ae1223b64d7b6dd071d55da5d7901c0 0 0B
3ad438f70a2b9671198b3f07bbdcb1de9e407bd94e0579f7bbc237db9dd789da 0 0B
912e3eca4c7f172cd13cb1649d4c21ce556c26de108ce7823dc6a41ad72b652f 0 340B
1fac7ed4c5f5b5cd4b23b82dac19061c17e9dc68ea175f61f8edad5cad42f074 0 340B
38c3660e480cebbd6ab966a3a3719dac16f91224e2a6e2fe3101e55d88dc808d 0 0B
4ec98f69a656cc56f9f9376d307b290b92be2592a597272e68ef15f76082b5c8 0 0B
6add3d01023ae155aaf518559aac961fde363d1d7432c3727c08d5a1141a2807 0 0B
f1e605c01376db60047bcbecf49502f22566cafaa37a4cf347d9f714a66b81d4 0 340B
31ed5d79dee55adaab1c1ba15557d1f62833a8e57f47712c038404d14f2abe4e 0 0B
71cc9dad25018d6cc20bcde962f53b01cfc76637bbdfbe7e68d119d4cf9412aa 0 0B
9bfba0cabe6ca9bde6f701a01abb196b2a2c04fae7b6c92701443a9f20e4385c 0 340B
aca1b634d9c2cb38c08a0ecced171d7316af9856656d1169bb354fae4ce5489d 0 0B
aca308235da196f3abc9115289059b822951d563e0748240d7a02414436abc75 0 0B
e45f55d7df0c0c08be448329377b5a120ab170a7731058dab744ffa9d9a17f66 0 0B
f434ba2f013b0924559cce20cc0dc857e9af62ef9b2743d2c2f362eecc6d5a12 0 340B
60e0beb8f0d48b4aa89ab4f5df49bfb92b394a2995658144be9d06a63a76713d 0 0B
ae88a898f65f55632beae0323bb720202ff5b677cb44c447b2421a254e94d16f 0 340B
02fbda77c99ef3830eeeaf66c120e2bc25c13b2524f40db8838402c425612e06 0 340B
033cb48511fd1790620a6a84830d623f1314b9e3207fde0f47760dd8fcd61e66 0 340B
470aca854918eb1cba3bde5dc376289779d3f6989324794e93abe89419d8915d 0 0B
5fd0f2208dcec79722c6d8f89463cddb8bb7aa07a54a05cde1b9106b7d8cf318 0 340B
ef5177437ce4b33dbac505560d7b99b984028bdb264eacfe03819f85fec86f85 0 0B
2a77ee0d81a92f79b8b227203d67f7aa2bce4d2e32de87ebee184611ff3ea8cf 0 340B
4e6a69609f47f6ca951fd7d3c80ca14dab45082cbb2c63aa907e493dfd713c53 0 0B
af75ffc72dff0f0d3b6b0f845bc47b5304fa04b5a276b118ff45ebd884c8e7e5 0 0B
48295a6e6f2f378f3d4b86c9d5cc86515a7a3ee02d1988a29f97e69ef98b5e65 0 340B
59fff5fb409864742a8b96cc2fde464629f72f477cdac2932c332676a1d75011 0 0B
821c94f2ae83bc919acfa39f01397f8285cf9c33d933b835044308ff7dc91722 0 0B
2d72f0c01860559ce2736f02b1addc2fe3084ba2570a2f8cd3e843c7f0429575 0 0B
2edaf5bdf89877ca38ebebc142a4c074a3db1e3a281aa9f2ca7663e7c204b229 0 0B
417fb94c6813acd7f5839687f4af46b0b5cb95ef62f3235b00d93bc6518e79af 0 0B
121ad02c3e154c310269e9d57060052d5572de0037b21a75803b613cc532b66b 0 340B
305a8d06b96479abeecc56c74883544a811a21e1edae0ef855af436526a33a61 0 0B
51e97b37a01ecdbe0f09c4367af6824c91ea7f06bbe98f3fd8c9a0bfd486bc51 0 0B
7bf1e12320001df7dbf87f644973e411f72771daa195e250b6ea9fdbab6809d7 0 0B
a23034c3e84efc4704b4b36053082251f9a66b03e21d43b77a8fdae7742903c4 0 0B
d9eb6e73011d8409f9c706850ce3a9957c08cd71dafaf5c1ff5ecb739e06305a 0 0B
31159b1a5b04d2c0a20252088713411f1df4c92828f8a01f11b0819aa1df1018 0 0B
3207e0a27b348dec75fef9314c04e04584bf2b212ef4618d545d79c1f0fa80b3 0 0B
34107d4db68738b155a083b6713b62a807dd82bf2be6801da0939e662d3d06ab 0 0B
0f3b441fb48507974c4e90239b51c26bbdfeb5fbf9cc01c35b63709c8f866b85 0 0B
a1211006fa90cdf3ab4a48a83220f20bee4ed86537ae0648bcfa415bf75b54a7 0 0B
319b08088ad174ac2d695737961f18e8dda1734e7a58831e32deb1798d7a2d7f 0 0B
a570c4e28bda2f42e2b6972c39982aab373ec766b5e395a631c1b28395e2e257 0 0B
b085a082847f0c5d445a3ddd713e688d463a54bdf354a381da5dbffe23610a31 0 0B
ba265b21b108b64219cfcfcae7fb04f7592c60a471837dc56df128c9b230ed4a 0 0B
ee2dba800b34748d31874a23e9d68da283baacb89274c4c35d9e33c0658d2265 0 340B
f4a1080fd071fe3092c9f8d544fe214bc4ebb009dc7211e402c8a0c465530956 0 0B
162a6da644801ec4ca17a7da59bd8e9dc7b708c6f8ba4c91c16b1482edff0add 0 0B
083854a8e4eb76b0f80942436b940b9be99ddf293147c66b0f5d7d36e4c8a499 0 340B
0d2fa91f98c68f8c79488c5581abadd3b34258f18f935d00b438319c73d918f0 0 0B
19b7c7346fe4f15f2fe8700dcbc3fcec8cfdfd05d3f771de4727363572739558 0 0B
1db0be547f2db99a6e48f7cfe6676e177b8a52ca3d4668e51081ea17bcf03851 0 340B
294cb19bd7082537b6eff5648d351361789c7a94b91b3cf0be9a36b8b89ffdf2 0 0B
08712730d388cdb5b2a2fef5a13de1b8ee0b9480e1162de390a96c7d34fa4ec6 0 340B
0bbcdd396035d0970a2cbaa07294ed10f7883e4a34774d5eac1a780cc33feba4 0 0B
6f6876a7e2ef722eb432c5551472733903ef9b30342e8ffc9f52e70796b55242 0 0B
899f39dc848b0a2301509229b16df8dd5eac23af1846fe68bff61d8d86d0252a 0 0B
b9bb8df6f543b2b522f16700b48a28c215b03b5d7c40d17a0d6d03dcf68947a8 0 0B
2ec6db93de97b7171e27cec42167cb624b56172323c471369d3fd9dbf17d4a69 0 0B
8e3c592d7da1d5a063828eb3619dd3e8f3feb3ef513717cd8934b2975c8687c5 0 0B
9546774301e987fb2df197692b38282a35485e671a7f9cf9f9bbcc693953cf3f 0 0B
cab0662e7a743d022eb05f3b548bb2754c47ea0e032b7a15ec842f9805d9d416 0 0B
e3f4eef6fea7967694719b24315839090df4aec70f2702596bbcb6349065744a 0 0B
1a448629c8bf17f247c73b77506ea0b2ffff662e9345103d4b3d319d2485e0d6 0 0B
26d89393802f5985b66a9fb2863082247c863f8f353b3e2d14e0a9ff9b320bbc 0 0B
371ad2914f89b084ce10ae2ad282c36b70eb1311284b6668cdc04f6f33ef0cb8 0 0B
5a96ac05c4f1170e34ba0c85d26545ebbdf4ec8e4cfbc27049f6fc670477c02e 0 0B
a0816becc766f0a035b2e38982cbbed0a93dffe14339b87240e922b7d79fb9be 0 0B
cd1c9fc214658c418a9bf29b4f00c8147632c6fdac3d7a40c97b02a57e77318d 0 0B
f96ff174cf2942660fd6bb2bf409923b6b7c483b1e904de00df75d9e812b6a51 0 0B
170ee3fcee462651254f6b3976e76f6bc85c8499537bcdf5946654558b8ef650 0 340B
354071bc530510f70e5fc39a5f2f3330875bda12ee7123c089fd050ab6673a0e 0 0B
69f5a6e3618e9769ec58abe33721f8debce3ea349ae2a0896c5efe3bc376bb82 0 0B
b1109abaa9981ffc100100438e2dbc1d3c5590a35dffa95de6a7053d7501eb25 0 340B
e9b5b098efc8880ad3b1a7e933c25227d15d5f202c738b503879a852171ba24d 0 0B
100684df87807c764b18dbef1b3a832a70a23351335bffbb92d1ed7f5b57551c 0 340B
38af2f419e8de3a87a07a640ded9b5b62fc0411f858e0846cc2cca02c441db1e 0 0B
ac3f3f667d094c16227b9f2e4b69fa7220a6cdfd846b159d9952b0025bb29430 0 0B
af58754319138086c192dc8253e471c2240366c1390cc4dd72e3b27956ed633d 0 340B
4f0b42dd1dd5e6b6069316ff11ea95ea022b2fe7ced163556b9fad7620bb27d3 0 0B
### build cache
awk: write failure (Broken pipe)
awk: close failed on file "/dev/stdout" (Broken pipe)
Build cache usage: 0B
CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED
### docker root du depth 1
28760637440 /var/lib/docker
27008151552 /var/lib/docker/overlay2
1607688192 /var/lib/docker/volumes
117231616 /var/lib/docker/image
27115520 /var/lib/docker/containers
245760 /var/lib/docker/network
167936 /var/lib/docker/buildkit
12288 /var/lib/docker/plugins
8192 /var/lib/docker/tmp
4096 /var/lib/docker/swarm
4096 /var/lib/docker/runtimes
### journald
Archived and active journals take up 8M in the file system.
[Journal]
Nov 10 01:54:17 demo-hp systemd-journald[46]: Runtime Journal (/run/log/journal/a2481f753f3c48ebb179852799b57627) is 8M, max 599.8M, 591.8M free.
Nov 10 01:54:17 demo-hp systemd-journald[46]: Runtime Journal (/run/log/journal/a2481f753f3c48ebb179852799b57627) is 8M, max 599.8M, 591.8M free.
### /var/log
253952 /var/log
45056 /var/log/apt
4096 /var/log/private
### apt
95723520 /var/cache/apt
96104448 /var/lib/apt/lists
0
### tmp
0 /tmp
20480 /var/tmp
2026-11-10 40 /tmp/.ICE-unix
2026-11-10 40 /tmp/.X11-unix
2026-11-10 40 /tmp/.XIM-unix
2026-11-10 40 /tmp/.font-unix
2026-11-10 4096 /var/tmp/network
2026-11-10 4096 /var/tmp/systemd-private-09206f2068524a2e881d5328e0bbc345-systemd-logind.service-Q0avKv
2026-11-10 60 /tmp/systemd-private-09206f2068524a2e881d5328e0bbc345-systemd-logind.service-g8WCve
entries=7
### felhom dirs
/etc/felhom-bootstrap
/etc/felhom-controller-image
/opt/docker
/var/lib/felhom
### root filesystem top 45 to depth 4
1355317248 /
913195008 /usr
336863232 /usr/bin
268349440 /var
266133504 /usr/lib
182734848 /usr/share
167092224 /var/lib
166846464 /mnt
166842368 /mnt/sizing-bench
166756352 /mnt/sizing-bench/backups
166752256 /mnt/sizing-bench/backups/secondary
154406912 /usr/lib/x86_64-linux-gnu
110944256 /usr/libexec
110768128 /usr/libexec/docker
110018560 /usr/libexec/docker/cli-plugins
99504128 /var/cache
96387072 /var/lib/apt
96104448 /var/lib/apt/lists
95723520 /var/cache/apt
67096576 /var/lib/dpkg
66502656 /usr/share/locale
30691328 /usr/share/doc
28700672 /usr/lib/python3.13
26943488 /usr/lib/x86_64-linux-gnu/perl
25632768 /usr/lib/git-core
23773184 /usr/share/perl
23769088 /usr/share/perl/5.40.1
23347200 /usr/lib/udev
17895424 /usr/share/man
16842752 /usr/share/i18n
16412672 /usr/sbin
13266944 /usr/share/i18n/locales
10358784 /usr/lib/file
9621504 /var/lib/dpkg/info
9428992 /usr/lib/udev/hwdb.d
9031680 /usr/lib/python3
9027584 /usr/lib/python3/dist-packages
8601600 /usr/lib/x86_64-linux-gnu/gconv
7770112 /usr/share/terminfo
6914048 /usr/lib/x86_64-linux-gnu/systemd
6008832 /usr/lib/systemd
5705728 /usr/lib/x86_64-linux-gnu/perl-base
5439488 /usr/lib/python3.13/__pycache__
5259264 /usr/share/doc/git
5156864 /usr/share/man/man1
sort: write failed: 'standard output': Broken pipe
sort: write error
### /var/lib/felhom depth 3 (data volume)
30087921664 /var/lib/felhom
28760657920 /var/lib/felhom/docker
27008151552 /var/lib/felhom/docker/overlay2
1607692288 /var/lib/felhom/docker/volumes
1327243264 /var/lib/felhom/sys_drive
1327239168 /var/lib/felhom/sys_drive/felhom-data
1327214592 /var/lib/felhom/sys_drive/felhom-data/backups
117231616 /var/lib/felhom/docker/image
27131904 /var/lib/felhom/docker/containers
245760 /var/lib/felhom/docker/network
241664 /var/lib/felhom/docker/network/files
167936 /var/lib/felhom/docker/buildkit
20480 /var/lib/felhom/sys_drive/felhom-data/userdata
16384 /var/lib/felhom/lost+found
12288 /var/lib/felhom/docker/plugins
8192 /var/lib/felhom/docker/tmp
4096 /var/lib/felhom/docker/tmp/moby-detect-rro1448369834
4096 /var/lib/felhom/docker/swarm
4096 /var/lib/felhom/docker/runtimes
4096 /var/lib/felhom/docker/plugins/tmp
4096 /var/lib/felhom/docker/plugins/storage
### /opt/docker depth 2
2179072 /opt/docker
2174976 /opt/docker/stacks
77824 /opt/docker/stacks/romm
69632 /opt/docker/stacks/traefik
69632 /opt/docker/stacks/paperless-ngx
61440 /opt/docker/stacks/adventurelog
57344 /opt/docker/stacks/vikunja
57344 /opt/docker/stacks/plex
53248 /opt/docker/stacks/home-assistant
53248 /opt/docker/stacks/glance
53248 /opt/docker/stacks/docmost
53248 /opt/docker/stacks/bookstack
45056 /opt/docker/stacks/privatebin
45056 /opt/docker/stacks/kimai
45056 /opt/docker/stacks/calibre-web
45056 /opt/docker/stacks/bentopdf
40960 /opt/docker/stacks/immich
36864 /opt/docker/stacks/sabnzbd
36864 /opt/docker/stacks/opengist
32768 /opt/docker/stacks/wger
32768 /opt/docker/stacks/nextcloud
32768 /opt/docker/stacks/grocy
28672 /opt/docker/stacks/wishlist
28672 /opt/docker/stacks/wanderer
28672 /opt/docker/stacks/transmission
28672 /opt/docker/stacks/sparkyfitness
28672 /opt/docker/stacks/qbittorrent
28672 /opt/docker/stacks/nzbget
28672 /opt/docker/stacks/lubelogger
28672 /opt/docker/stacks/karakeep
28672 /opt/docker/stacks/homebox
28672 /opt/docker/stacks/dawarich
28672 /opt/docker/stacks/actualbudget
24576 /opt/docker/stacks/zipline
24576 /opt/docker/stacks/vaultwarden
24576 /opt/docker/stacks/uptime-kuma
24576 /opt/docker/stacks/termix
24576 /opt/docker/stacks/tandoor
24576 /opt/docker/stacks/sonarr
24576 /opt/docker/stacks/recipe-importer
24576 /opt/docker/stacks/rallly
24576 /opt/docker/stacks/radicale
24576 /opt/docker/stacks/radarr
24576 /opt/docker/stacks/papra
24576 /opt/docker/stacks/outline
24576 /opt/docker/stacks/navidrome
24576 /opt/docker/stacks/n8n
24576 /opt/docker/stacks/metube
24576 /opt/docker/stacks/mealie
24576 /opt/docker/stacks/komga
### drives under /mnt/felhom-drives (depth 3 from each drive)
-- /mnt/felhom-drives/hdd_1
/dev/nvme0n1 1006980812800 322734333952 633019228160 34% /mnt/felhom-drives/hdd_1
1536958464 /mnt/felhom-drives/hdd_1
1291870208 /mnt/felhom-drives/hdd_1/backups
919998464 /mnt/felhom-drives/hdd_1/backups/secondary
423739392 /mnt/felhom-drives/hdd_1/backups/secondary/adventurelog
371859456 /mnt/felhom-drives/hdd_1/backups/primary
227741696 /mnt/felhom-drives/hdd_1/backups/primary/romm
208642048 /mnt/felhom-drives/hdd_1/kept
208637952 /mnt/felhom-drives/hdd_1/kept/romm
208494592 /mnt/felhom-drives/hdd_1/kept/romm/2026-10-16_113539
181469184 /mnt/felhom-drives/hdd_1/backups/secondary/kimai
167829504 /mnt/felhom-drives/hdd_1/backups/secondary/bookstack
132833280 /mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx
125968384 /mnt/felhom-drives/hdd_1/backups/secondary/docmost
30277632 /mnt/felhom-drives/hdd_1/appdata
30261248 /mnt/felhom-drives/hdd_1/appdata/paperless
30253056 /mnt/felhom-drives/hdd_1/appdata/paperless/media
18546688 /mnt/felhom-drives/hdd_1/backups/secondary/home-assistant
10199040 /mnt/felhom-drives/hdd_1/backups/primary/plex
6164480 /mnt/felhom-drives/hdd_1/userdata
5054464 /mnt/felhom-drives/hdd_1/userdata/media
4972544 /mnt/felhom-drives/hdd_1/userdata/media/books
2170880 /mnt/felhom-drives/hdd_1/backups/secondary/privatebin
1081344 /mnt/felhom-drives/hdd_1/userdata/roms
1073152 /mnt/felhom-drives/hdd_1/userdata/roms/DRILL-2026-08-21
983040 /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
229376 /mnt/felhom-drives/hdd_1/backups/secondary/opengist
69632 /mnt/felhom-drives/hdd_1/kept/romm/2026-10-16_152659
69632 /mnt/felhom-drives/hdd_1/kept/romm/2026-10-16_120728
40960 /mnt/felhom-drives/hdd_1/backups/secondary/bentopdf
40960 /mnt/felhom-drives/hdd_1/backups/primary/qbittorrent
28672 /mnt/felhom-drives/hdd_1/backups/primary/sonarr
28672 /mnt/felhom-drives/hdd_1/backups/primary/radarr
16384 /mnt/felhom-drives/hdd_1/userdata/media/transmission
16384 /mnt/felhom-drives/hdd_1/userdata/media/torrents
12288 /mnt/felhom-drives/hdd_1/userdata/import
12288 /mnt/felhom-drives/hdd_1/appdata/romm
8192 /mnt/felhom-drives/hdd_1/appdata/romm/resources
4096 /mnt/felhom-drives/hdd_1/userdata/roms/roms
4096 /mnt/felhom-drives/hdd_1/userdata/roms-bios
4096 /mnt/felhom-drives/hdd_1/userdata/media/usenet
4096 /mnt/felhom-drives/hdd_1/userdata/media/tv
4096 /mnt/felhom-drives/hdd_1/userdata/media/podcasts
4096 /mnt/felhom-drives/hdd_1/userdata/media/photos
4096 /mnt/felhom-drives/hdd_1/userdata/media/nzbget
4096 /mnt/felhom-drives/hdd_1/userdata/media/music
4096 /mnt/felhom-drives/hdd_1/userdata/media/movies
4096 /mnt/felhom-drives/hdd_1/userdata/media/metube
4096 /mnt/felhom-drives/hdd_1/userdata/media/grimmory
4096 /mnt/felhom-drives/hdd_1/userdata/media/comics
4096 /mnt/felhom-drives/hdd_1/userdata/media/audiobooks
4096 /mnt/felhom-drives/hdd_1/userdata/import/paperless
4096 /mnt/felhom-drives/hdd_1/userdata/import/calibre
4096 /mnt/felhom-drives/hdd_1/userdata/downloads
4096 /mnt/felhom-drives/hdd_1/userdata/documents
4096 /mnt/felhom-drives/hdd_1/backups/offsite-restore
4096 /mnt/felhom-drives/hdd_1/backups/offsite-proof
4096 /mnt/felhom-drives/hdd_1/appdata/paperless/export
### timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2026-11-10 09:51:30 UTC 2h 23min Mon 2026-11-09 18:40:41 UTC - apt-daily.timer apt-daily.service
Wed 2026-11-11 00:00:00 UTC 16h Tue 2026-11-10 00:00:01 UTC - dpkg-db-backup.timer dpkg-db-backup.service
Wed 2026-11-11 00:33:45 UTC 17h Tue 2026-11-10 00:47:41 UTC - logrotate.timer logrotate.service
Wed 2026-11-11 02:09:53 UTC 18h Tue 2026-11-10 02:09:53 UTC 5h 17min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Wed 2026-11-11 06:14:05 UTC 22h Tue 2026-11-10 06:31:53 UTC 55min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Wed 2026-11-11 09:16:04 UTC 1 day 1h Tue 2026-11-10 00:30:40 UTC - man-db.timer man-db.service
Sun 2026-11-15 03:10:10 UTC 4 days Sun 2026-11-08 03:10:50 UTC - e2scrub_all.timer e2scrub_all.service
- - - - fstrim.timer fstrim.service
8 timers listed.
### cron
/etc/cron.d:
e2scrub_all
/etc/cron.daily:
apt-compat
dpkg
logrotate
man-db
/etc/cron.weekly:
man-db
### end
2026-11-10T07:27:50Z
@@ -0,0 +1,576 @@
### identity
demo-hp-scratch
2026-11-10T07:27:52Z
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
SCRATCH GUEST — R-481, operator ruling 2026-09-13 (option 1)
The nightly rotation throwaway host under the demo-hp customer. Hub OFF, tunnel OFF, agent OFF, off-site OFF.
Persists across nights on purpose. Apps here are throwaways; nothing here is a customer promise.
### df bytes
Filesystem Type 1B-blocks Used Available Use% Mounted on
/dev/loop0 ext4 33501757440 1920802816 29846192128 7% /
/dev/loop1 ext4 73391005696 13990232064 55625900032 21% /var/lib/felhom
/dev/nvme0n1 ext4 1006980812800 322734333952 633019228160 34% /mnt/felhom-drives/scratch_hdd
/dev/mapper/pve-root ext4 41424257024 23968083968 15318966272 62% /etc/felhom-bootstrap
### df inodes
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/loop0 2097152 33510 2063642 2% /
/dev/loop1 4587520 340608 4246912 8% /var/lib/felhom
/dev/nvme0n1 62513152 796 62512356 1% /mnt/felhom-drives/scratch_hdd
/dev/mapper/pve-root 2588672 120141 2468531 5% /etc/felhom-bootstrap
### docker info
driver=overlay2 root=/var/lib/docker logdriver=json-file images=48 containers=6 running=6 server=29.8.2
### daemon.json
{
"features": {
"containerd-snapshotter": false
},
"live-restore": true,
"log-driver": "json-file",
"log-opts": {
"max-file": "3",
"max-size": "10m"
}
}
total 12
drwxr-xr-x 2 root root 4096 2026-09-13 07:49 .
drwxr-xr-x 73 root root 4096 2026-11-10 07:19 ..
-rw-r--r-- 1 root root 177 2026-10-04 14:08 daemon.json
### docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 48 6 12.73GB 8.811GB (69%)
Containers 6 6 52.76MB 0B (0%)
Local Volumes 368 5 235.2MB 5.564MB (2%)
Build Cache 4 0 262B 262B
### containers (name image state logdriver logopts)
/felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.350.0 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T01:54:21.255658958Z
/filebrowser gtstef/filebrowser:1.5.8-stable running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-03T03:04:56.668897081Z
/paperless-postgres postgres:18-alpine@sha256:10a5c69a83035ab5239f5620d708520e8b28f5aa1f2b20143e6024532edadfb5 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T01:30:11.044904019Z
/paperless-redis redis:7-alpine@sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T01:30:11.044776057Z
/paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:3.4.1@sha256:1a08d8dfc2b51a08de09652dd86ad4d0c611c362382f944533ad38b73125fbc4 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T01:30:11.116314798Z
/traefik traefik:v3.7.17 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-03T06:55:34.357310781Z
### container log files (bytes path) and total
7719887 2026-11-10 /var/lib/docker/containers/c45534b518cc4026e44d45cc704cca2efc04a51e025d4c03042d4e3cf79714b9/c45534b518cc4026e44d45cc704cca2efc04a51e025d4c03042d4e3cf79714b9-json.log
5704181 2026-11-10 /var/lib/docker/containers/6ae0e68d9c7fa59c92e396afffcd63f6b8f1e2a679cde71ae0384a23d5bac9a4/6ae0e68d9c7fa59c92e396afffcd63f6b8f1e2a679cde71ae0384a23d5bac9a4-json.log
3845616 2026-11-10 /var/lib/docker/containers/a1b2f5b1305ecbb7cbebc52cad5ab0cefc41e7e9973418bfc6dc6199c5de4e62/a1b2f5b1305ecbb7cbebc52cad5ab0cefc41e7e9973418bfc6dc6199c5de4e62-json.log
60250 2026-11-10 /var/lib/docker/containers/8925c3304170ca7c37a335f5d4cfbc3c4c88d0e634dafc7b08425ca664be3d07/8925c3304170ca7c37a335f5d4cfbc3c4c88d0e634dafc7b08425ca664be3d07-json.log
24304 2026-11-10 /var/lib/docker/containers/ea499a71393e76f9208b33251bc6cd28e7b729c21a9fdca4eeaf7de5632678bd/ea499a71393e76f9208b33251bc6cd28e7b729c21a9fdca4eeaf7de5632678bd-json.log
10339 2026-11-10 /var/lib/docker/containers/c115a3f36a0f6bfde56c312360eedd970323387f72f5b304ddb434a22c769679/c115a3f36a0f6bfde56c312360eedd970323387f72f5b304ddb434a22c769679-json.log
TOTAL bytes=17364577 files=6
### container writable layers (docker ps -as)
felhom-controller 0B (virtual 348MB)
filebrowser 4.72MB (virtual 200MB)
paperless-postgres 68B (virtual 304MB)
paperless-redis 0B (virtual 39.1MB)
paperless-webserver 48MB (virtual 2.53GB)
traefik 0B (virtual 191MB)
### images
<none>:<none> 1979d2941666 2026-06-29 19:02:35 +0000 UTC 457MB
<none>:<none> a6f184081d44 2026-10-06 13:24:13 +0000 UTC 410MB
alpine:latest 320994c3b997 2026-09-17 20:37:20 +0000 UTC 8.42MB
busybox:1.36.1 b116e1550744 2023-05-18 22:34:17 +0000 UTC 4.42MB
curlimages/curl:8.11.1 7551dbeefe0d 2024-12-11 08:46:58 +0000 UTC 21.8MB
ghcr.io/paperless-ngx/paperless-ngx:<none> 354dc6ebeb86 2026-10-31 05:27:36 +0000 UTC 2.48GB
ghcr.io/paperless-ngx/paperless-ngx:<none> 623d64064a8d 2026-04-27 02:45:05 +0000 UTC 1.42GB
ghcr.io/puppeteer/puppeteer:latest b8b7cf1b6cd2 2026-09-23 12:55:08 +0000 UTC 2.03GB
ghcr.io/seerr-team/seerr:v3.5.1 61f6fa0cf472 2026-10-21 15:40:13 +0000 UTC 890MB
gitea.dooplex.hu/admin/felhom-controller:0.314.0-rc.1 80b7b25c4433 2026-10-15 12:35:37 +0000 UTC 411MB
gitea.dooplex.hu/admin/felhom-controller:0.326.0-rc.1 fe5356752637 2026-10-17 09:19:26 +0000 UTC 347MB
gitea.dooplex.hu/admin/felhom-controller:0.326.0-rc.2 a8dda4a2c4b3 2026-10-17 09:41:59 +0000 UTC 347MB
gitea.dooplex.hu/admin/felhom-controller:0.327.0-rc.2 c6f3294cf70f 2026-10-17 11:00:21 +0000 UTC 347MB
gitea.dooplex.hu/admin/felhom-controller:0.327.0-rc.3 cd60b77ad708 2026-10-17 11:27:31 +0000 UTC 347MB
gitea.dooplex.hu/admin/felhom-controller:0.327.0-rc.4 e369239d8017 2026-10-17 11:38:32 +0000 UTC 347MB
gitea.dooplex.hu/admin/felhom-controller:0.327.1-rc.1 bdbde2ded625 2026-10-17 11:57:23 +0000 UTC 347MB
gitea.dooplex.hu/admin/felhom-controller:0.331.0-rc.1 88738376ade2 2026-10-18 19:06:00 +0000 UTC 347MB
gitea.dooplex.hu/admin/felhom-controller:0.337.0-rc.1 9793c3d374dc 2026-10-21 13:47:28 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.338.0-rc.1 f6697ab01e86 2026-10-21 19:21:38 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.338.0-rc.2 430c9209c027 2026-10-21 21:42:53 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.339.0-rc.1 f98ab2595036 2026-10-22 10:32:32 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.340.0-rc.1 47c02842b8a7 2026-10-22 14:30:15 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.340.0-rc.2 48fd2c580327 2026-10-22 15:29:39 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.340.0-rc.3 c5d1d90b6205 2026-10-22 15:48:57 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.340.0-rc.4 58d9dc8ab96f 2026-10-22 15:52:41 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.340.0-rc.5 c2cc5b41f531 2026-10-22 18:22:18 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.340.0-rc.8 8df3526c564e 2026-10-22 21:00:20 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.341.0-rc.1 bcbf73d8e476 2026-10-23 11:11:07 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.346.1-rc.1 0445c43fd2e3 2026-10-27 14:51:02 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.347.0 fb19f3f6881a 2026-10-29 08:57:22 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.1 1e0e09bce4b5 2026-10-29 12:08:23 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.348.0-rc.2 2f31d8611e2b 2026-10-29 13:37:38 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.349.0-rc.1 712055a90134 2026-10-29 16:11:39 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.350.0 1f32a260789a 2026-11-03 07:04:04 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.350.0-rc.1 9e6047ed5880 2026-11-03 06:55:14 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-samba:1.2.0 fc741cdd77e4 2026-10-16 18:25:34 +0000 UTC 107MB
gtstef/filebrowser:1.3.3-stable 095fd20d87be 2026-05-18 22:10:20 +0000 UTC 215MB
gtstef/filebrowser:1.5.6-stable 96e26edeefc3 2026-09-04 21:33:58 +0000 UTC 195MB
gtstef/filebrowser:1.5.8-stable 0019084d4a6f 2026-10-04 21:15:40 +0000 UTC 195MB
jellyfin/jellyfin:10.11.11 d57d4a0c18b3 2026-06-06 15:43:54 +0000 UTC 1.56GB
postgres:<none> 57b23a0326ee 2026-10-21 22:56:40 +0000 UTC 304MB
postgres:<none> c293117fcecd 2026-09-17 21:29:50 +0000 UTC 304MB
redis:7-alpine 98c6ca8f397f 2026-10-26 16:40:19 +0000 UTC 39.1MB
redis:7.4-alpine 98c6ca8f397f 2026-10-26 16:40:19 +0000 UTC 39.1MB
redis:<none> f84b0c467801 2026-09-17 21:32:46 +0000 UTC 39.1MB
traefik:v3.6.7 91528df1690f 2026-01-28 02:37:11 +0000 UTC 186MB
traefik:v3.7.13 f9309349d2c1 2026-09-17 21:33:14 +0000 UTC 195MB
traefik:v3.7.16 3733df53e74c 2026-10-21 22:55:46 +0000 UTC 194MB
traefik:v3.7.17 a72cb2f4a210 2026-10-30 20:17:26 +0000 UTC 191MB
### images count / dangling
41
7
### images in use by a container
1 ghcr.io/paperless-ngx/paperless-ngx:3.4.1
1 gitea.dooplex.hu/admin/felhom-controller:0.350.0
1 gtstef/filebrowser:1.5.8-stable
1 postgres:18-alpine
1 redis:7-alpine
1 traefik:v3.7.17
### volumes (count, anonymous count, dangling count)
368
363
363
### volumes by size (system df -v, Local Volumes block)
Local Volumes space usage:
VOLUME NAME LINKS SIZE
1f5120663962c7b03539bb993ab736be4800096e3d1f7147eb922d44b308ae6a 0 0B
49d0cebfc3199cef395ae27a6ffa479340c1c2f69da9248a777d7bff9b6b9ca9 0 340B
67eff4c5c60d493dc2a9f7841f7f138b610dc389c32793a27a62ddc13d417a1c 0 0B
99f7aaac2646c66dd0eeeebc157b8563b56299c27ab4bf64e13920d089ca7156 0 340B
ac78e0a5ac10c7d0239e0e1ba3b604cd6fd418d87a6f32b66f3ce98c674164d8 0 0B
b8dd8697ad52f98dc44ed3c940752aa52e16d6ab7563d36b1c9e221db22d1e11 0 340B
eb420ae07f9c459abc101c1264fa8cf779002ef20748ef97abe7c3764c24f032 0 0B
f1b01ee5efb2117033ddcad5f5eafe8e6f65ad6fc3e933902bfc494c13a7d3c0 0 0B
01feef94eb7e38a3679921b66cfe206e08adfd9b360ed4a60414cde606de1643 0 340B
122fe66bacc9f04b3f3e4f6d0a3b15abdef8321eed36637a1284e3bd145463b8 0 340B
3461d48cd9e88caf5363b0e4d96e799f6759ab8b8bc6ba4c148492a1637c01f0 0 0B
36571aa758bc335acc463c3ea074422a5123c309a37e1a7fc818a2069941c493 0 0B
4e052b4ef5602a68eddcea049bbc4e79b306012cf8b7f0c9cfc3e6c39af64fce 0 0B
63e2655474549ef03aaf7d9d741502287d2f01170c5ca008d7b09b8e00b1be66 0 0B
645d70ab53013564fd243eae61db209e3793b826a9eeffd77395b18e27458789 0 0B
8678479fa55ea3871598eb6914eb8428ba599da4c619c62f231e6931f0cb1f54 0 340B
0443572c778a5932495c24e6b8b7adae1942474eac484e797538038bed6e21cc 0 0B
2bc95b104191ffd31bdfdbe1e038d284d0624c04735094494c2c77d957e8e905 0 0B
82f8964decc14b9c760ca151ddf5ddf0c68b5258718cbae8aa41d71b1985c32a 0 0B
aa5cf2d08c68b8d3dfe3330861935fd97ff8fb9169645f6c632d970ad71b575c 0 0B
f287b03a768126d72af210dceab12e329573779e7aad56c123ac0dbb14fc4f4e 0 0B
f347e779fb8c09daba2a757dfc3ee7c415f8f5a2eaace19535d363cc3ab3171a 0 0B
fbc407de7b12a4975b60dac518ac287f076659213cce708871162342419ab383 0 0B
paperless-ngx_paperless_data 1 3.129MB
2aaab84bf843817d9e6f32b673277fd1e0c2c8c2d038b8126df44927b1b41006 0 0B
81aaa941ada415c3fc585a4abd540a6f606c464c2fa21857f843bb7c94dff7ed 0 340B
8a155792d26111203749f0188ed3ab8c89ef34cdc2459dec14fe0ef0ad13ee44 0 0B
9c59fdcefb7ed258e615f8cb688334a69aebdb8e38fd1075a60d6fd98167065a 0 340B
ccc8692427c942a9e326bb89716d388caedeaf0253e2ba1d092f49352277e6f2 0 0B
f2ec30f9f836ed26cb13fdb80064d1dcabf953d1952fea2d85bb6410f4cb2271 0 340B
fe503d3e9a34b113d4c56b59c5986e13c44b6e401b6e6681eb62712d24368329 0 0B
2dcb4bd2c3c618d71f582d33d5b1243e6ab075d3b00abc238d4430668a2b6f55 0 0B
4098005900d38c71b152787ce4bdedfb8e2b2b8b061f87645b2b4a14e0004335 0 0B
556c805068afd3f6784bda2eb59642d7dc03756580e959e1c75262711a30d4d9 0 340B
8097f97f4ae00b654b1c0b4408ad585f5bfce7999e482f244bd698268e1f6061 0 0B
b21ec6b73a9f1f51623355f2bf8229b5f6f9da1484e1ce105cf8e0363475a5b8 0 0B
6623edab4a2d76a6b476bf0162a0e44749731b4684f5f814ad253868d9015235 0 340B
93c2b9d0a1f54c2e6355d206d93bf2309f81076628aae9d868406c288b0f6bf4 0 0B
b06f996b7de17a5a10b370b046fefca3ae5c59a6fa9301eff9bdce3630b9d411 0 0B
c67a888239bc5e11ed44ca31307f3b46f7f76698d706ffc764024110d1a69ceb 0 0B
dd1ce11b2dbbd070e00a4cdb9c9c6795b4f74ad2d92ada9bc2ffbe9408c33faa 0 0B
e48031df0e75f6e1ceda86624fa39a2823f1bf7c434cbb008348e485ccae93b5 0 0B
fd453dd47dc82ff7625bd4093c62140fe14a348814f094b43b519f7f59293af7 0 0B
07402fbe22b572302f3317958f8cc4c6ca30317e0ee5bbbe9ff51c60b3da5a05 0 340B
2788da1d8f917a9d5b8acab2c7fe7336b6a7810f1c7edab63ed327a6022ceddf 0 0B
3d9c1f23abbcef771a4c38a75d75865a12d76a4359fe83c1828652dba3f91710 0 0B
4d3a9f9bb88bec79b8f37d92658cd08fd4b5a9a2934acd882c1ddc81d9bb38f8 0 0B
5c7dd57f445d87a39eea1e7b299e1a0acbbc946d6bfc92ba165c4af8933b322a 0 0B
67711784cb23efbc0762e83f03365b837ec5216c38f3ba682a82eaff6a7863db 0 0B
13175cc97740c834e2404c11ba9fa9418d0c390d27a87e6e869977d6a86c20c0 0 0B
219118eb01fa7be03fcce2124b74708cacafe66aea617fd2bafa32f0b2d4fd80 0 0B
748cce6aa136bdc0b3111c73a7c9434508200f8c231933b925d42b37b9cca17c 0 340B
7fc41ddc47424ea11d4940f7062cdae69b92af645e6759f8759f47b689a19cb7 0 0B
2ac49fc9cc2839f55c8abae76039950d15a5c301ea5b3918fa3e5d8715a15f42 0 0B
c424d546895a59e0f88e2296a665c03f4618ac304ef19742b6f1c543665b9bb3 0 0B
ea35c704202ee36415d4f7a0873bcb254d30268450d7c9ed2dc611c1e2f4d5d1 0 0B
2f08e882c5d23373e2df5486dce28d4728330f9f0131a2735ebc938bfb4d7b3a 0 0B
35d80e556f979788c122b4b7fee88755b78de02123a746f5e502bb2adf7802dc 0 0B
426ccc08685def1c9025c94d343de0e59713073984634ee226e69634d6a18fb8 0 0B
4559b91854060f83ea3110e72e58993df3ba4d449c910ef2e334d54fbacfff8b 0 0B
4c54a4cda3c5280d5b13facf3089f8a0710c4570876aaeb679025aabf92df99f 0 0B
552a4b340eb6be03c188cb17628aee0f916b544142465f8e52e7e9d2325206ab 0 0B
9abdd18e6a7b07800067237c93d48448d49d9ad2388a91d35653323c2bed4389 0 340B
ad79ebffc7d8225cfbee6ad5fec4092ba7376f38dca912d6df8ea932f1eba20b 0 0B
3000c7863b2e80a72329ca1fa0dd03d2fd3994a2391e82450a8c92199cbb56b6 0 0B
7ef24d6bf9fb190db2572505b489e45731a326c5c9c4ae6e36ffd7874560bbab 0 0B
8158fe1f757d26cadb484ed9690ad4d20d4cb382eeec7c9b0627b365afd6576f 0 0B
ba0d1cf9b0562eb78e4d755913a57390a62f4c8ccd905a81ee6c81c32721f3cf 0 0B
e86b5072ee382465c27a91762d19182e11fa150371f40cae0f04e82c39bf00e0 0 0B
fa453f6fb451a564d31e704bbca0bcbcc9ad1109a25a8e5ad5c704789a424c4b 0 340B
104e16e4796c52fe212c04b06417161e0bab20fc01632bfcf71421b73b979fbc 0 0B
158c590779b033e48b5efcf895fd0bad4c6f277303d72b963cd0802ecac1f440 0 0B
56c17b46ed2c6a8cc1a7fc281b3e6a080f6bcc839d8be68cefa5b4d4973c76d4 0 0B
c56e78b468dfe3991fc787ab7fe30b4282a73d7fd2e79e4dfc9305bb064685ea 0 0B
ec30ca07d0b0af16afe60f217dedf15573045ed9adc45e21c3592ee6716fbe39 0 0B
fdc05b3bfda3101bec7d66b5e0cfb589e06f1f4b4d6b2f9d84dac470b0abf53b 0 0B
075b39bc7ae8c56a6d99923e4938ddbbb4174df6f0e653abf9c99b420b401a31 0 0B
3d9a93769342e83a3c512d517b99cd64f8e794eb305c009e25c73a6f6627ad29 0 0B
8c4d45c90f336c0ffd5d619765d8d2a45e802d619548f187bb3f2672b3534969 0 340B
aadb8f0be78890f1e08630b7a762a82853dc1402326bd16cd2a2844fa63fef1d 0 0B
d82518d66e35ec7f6a4ba6d9dd687d09457c1b83961e4197fac7ec05a23e940f 0 0B
13694424a3895e8e73cc1e06af29aae0c3179c3cb5587b9daadeadcdb29b7cc7 0 0B
283e9d53742a13c471e5f59f53a57c179d461f0017ad5f8db27881cdd994cd04 0 0B
346eaddf98e91d6d6e7b8c526074c27847e4a18be01b41027bec93abc2b7b2fb 0 340B
3ac9820f84d0a50f233c63a62ee1b63b92acac026d68c60a8227f370b0be00b6 0 0B
43d6928bc4bb7a1e097122424e35f22393a0ccf2e03863d7dc3cce44163c8ae8 0 340B
74c6b39b7b926dd56db16f175026c2bf9e80a912916f273c122227672f181d03 0 0B
b676fcf4f5272015b942c8589135a18fa16204c5aec457f1d0898ce6c84638eb 0 0B
bfb91f5666855fd57228801ed0a4294981bd2a6665a93d1093bf3c4a0e89cb2c 0 0B
04762469942ae24c353f929fced777f245bb9f2f888a9f2044ff27fc45933665 0 0B
991a5d344112325fb2c40700cd0ee96a784ce2150cba0a7b0be3ed6eb2411878 0 0B
9ee6f261789b97ea0887a6a905bf5bba61a7070cc91ff64cda0091923d9cdf25 0 0B
c52417cfa3d923ce7168b7c2b082eb3ab31bc1750ff3f3edfc2c1cd59fb74f3d 0 0B
cb10732ef556d77ec0a0a5d7b29c0789b47a9757b67bdb3fa9d5be22809debd0 0 0B
e7945baeafe0ac6d9a1e6761786c35fddad76c81d86e544d369188c186a0389b 0 0B
edcc77bb3d279e3dfe7d687fe30e22f95f5d9391b9908e61b2206a6cf309ea91 0 340B
1177a1c33ceac30ff2084a227655962dc4ba14f1701d4012920e07052309b525 0 0B
24da0da42dc2eb9b6d35dcb7cd513b1e5c7d2ee38b6d59d9a1d51f0d01f6f920 0 0B
4a9ed2aed1fe75a0b02f688e29a20acc2bf7e081ba7518a7d75451be30fdc4e1 0 0B
60496cd3a1e1ad1dcaed30427b70a57b8b14a122bae7f2cd882f8154a4549978 0 0B
c35e0b961694a882dcc4c7bdb8e657d4b0685687cb5c4469559ce5078c95302a 0 0B
dbcfeebe1162c457fdf6f7cbfff66c971dfaca8a1a8ae2fb5472fb9007f83ddc 0 0B
218a186ab13a38d514cf9f523e5798452ae4e5ec76ecf26fe1a46ae22b47a508 0 0B
338f7535ae9ef91754e76604238ca7652273ffce9e6f0a1300de0b1edb61da77 0 0B
3fa030697dbaffb6809991bb69a335dc4e9accfafd4ed5ee953ed676673912e6 0 0B
5b84763ca63a28ae79c1393d700c534e7dd6e610f890f540981e49972e3854e9 0 340B
981c658ac4002ab2fd89de7e7739b40b37a7c131f39ba8503104feb5a4c75c39 0 0B
c5708e0a13ababf73203b1e297b3b73229cd6370e3fec1653b7cf1dc143a0d0a 0 0B
ed6559d50a88df220a64702ea2638c1c2c038274d2f1d1712294026c1326c26c 0 0B
96983f5525c94ceea8647c8c671334a1650a202122b1b47424ecb09613de80f0 0 0B
cb613ceec1bf86b718059ef3e864dce7ed60e5b6a7bd5ab9f7c633350df48f3a 0 0B
19029cc077643dc7cf917efbf2d0fbf271a31c1a9d777cf853821b6bda710dca 0 0B
1a8fe48409be3defe01713e2f621091630b965e1805cbb5872d1738043efd35c 0 0B
2a71e17dfa1e930ffe0c213cd1615c8d1adcbbf8d61615a38064d331c265c02e 0 0B
4bbaae5ae8cc8cc3330a73bb872b16e1420805197fa9bad903f7db9a1f358041 0 0B
77d29d141866c8e567f3e50016dad4fb374da4254eca82b232a7db84fd16502f 0 340B
8cae3c834dd9284d1d0069de042c2d50e49a138a78d93a59eb752272e4a5e4f4 0 0B
921d92a1c692ea0fd42cbe500bbdb1f3defcd5fb9b51df216451aa260d95ad45 0 0B
a74b9b2333ee936c17edd0525b49fec1337768ccab68f5576a39486c875ff968 0 0B
5ec44fe3223976727132521b3055187e88783716a1f5fd07d1625205acff9888 0 0B
b1591e2a2604b6404d0d18e4b2b7599aafbcfb2bbacba06b5de5ff14abf14be3 0 0B
d6a74fc39944848c79012bb06d915c609b92c1099881b90b4cda59074bd1124f 0 0B
paperless-ngx_paperless_redis_data 1 2.563kB
3a3d7bc33a1ba21ae4a39c8cb345254cf66cd1dc60afae64e2299513b094b93b 0 0B
abadcab5791a7ee3fbd00e1cb1be565a386d593163df8b9850b0e29a85c8ad4d 0 0B
e5f9a9a8f7e4b6d097aef1b0493180a547dbf19006920433e7b80aa8aac0026b 0 0B
2a302ae451a03b90150d0419eab6c45feba05e79aca413c61965d6f1a242438c 0 0B
2e4c79db25c47436b44245376b1f6ff1991208ba081fddce1938c3897fc63c07 0 0B
4e3a990001b8f067ce7df40350894d1e3ed7104521a5f7bd4c0bc99e70b164cf 0 0B
5dd86f849e4df5fac5c42b9277b942eac37f493f715e371717e11fcca47a1ce7 0 0B
134995217d29dc47fb0ca1a274ceb5ea74225d36a98c4bbe6e07bf8c8d6898c7 0 340B
670e6447721f667df5222cb06a68c2682eb92803184a44faead8aea0dff44df6 0 0B
ad4bdc89e440b4eab0d9ca8c504471ba2669cc3082799c3309030b538fda3a94 0 0B
b404dd875a61a454c670aaa474b0b160d3d725113e99d099cab3f5af3daeb45a 0 0B
b5344b17b0a5c47f5216b0fd64bac157d8f4a54caa90341ea61767fae6e2843e 0 0B
4f2266bb0b8a9b902eaa439ae9225963d6fe4359fe30569c2d8afa926b582585 0 0B
4f2ebb779645de5cf9d24e93c6fbd9d95d358b390dec94eed6f0223016785e42 0 0B
63ddb5c60e6cf39dea412903cf1d6c0164a1146bcf4ab98ee94c9b1f9e45c6bb 0 0B
71cb6156444948894aec343c9a2c0c7674aca84d292d090c7f786d13146709ed 0 0B
86022b0cd1043facf7b1c933c3b6a6ff36277a76d50bdb2da6a5690d57381b9e 0 0B
9e9fa98cb2344c5eda9bae7417557f2bc2503de1eed8219fdc2d2d378e1883fe 0 0B
b9328a708e9a302e2b74b4afac0cd86ea2fbd66b03e7ed2cf3907ce1dfbcbe2b 0 0B
eea963e8499939d36d09e42d8d5cd86798e81e264c4d4c012191e42926e508b8 0 0B
09ea67268ce8c0442357ba316f6431cd96143da6bb54008d3c342be8b6ddb169 0 0B
1083038e9f32c8d880d37f3b7d789d94f3151bd1dd5f8435ff84ab2251a64019 0 0B
479c57a0efc3843a2e93c905451f53ae413cfec5d5399f46ca5380f2387c3cd7 0 0B
7bfe063b0155297f5bca22c7ed02ae7ef9ce1b968670607415f7720e1e8649f0 0 0B
### build cache
awk: write failure (Broken pipe)
awk: close failed on file "/dev/stdout" (Broken pipe)
Build cache usage: 262B
CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED
u3iufu2kbct2 source.local 182B 7 weeks ago 7 weeks ago 1 false
kyegjip696x3 source.local 0B 7 weeks ago 7 weeks ago 1 false
5hleco144phk regular 0B 7 weeks ago 7 weeks ago 1 false
pcrdms1hdxmk regular 80B 7 weeks ago 7 weeks ago 1 false
### docker root du depth 1
13988020224 /var/lib/docker
13676625920 /var/lib/docker/overlay2
247779328 /var/lib/docker/volumes
45240320 /var/lib/docker/image
17645568 /var/lib/docker/containers
495616 /var/lib/docker/buildkit
196608 /var/lib/docker/network
12288 /var/lib/docker/plugins
8192 /var/lib/docker/tmp
4096 /var/lib/docker/swarm
4096 /var/lib/docker/runtimes
### journald
Archived and active journals take up 8M in the file system.
[Journal]
Nov 10 01:54:16 demo-hp-scratch systemd-journald[46]: Runtime Journal (/run/log/journal/2ebfece84e7a4ef7adae80ee5738cf21) is 8M, max 599.8M, 591.8M free.
Nov 10 01:54:16 demo-hp-scratch systemd-journald[46]: Runtime Journal (/run/log/journal/2ebfece84e7a4ef7adae80ee5738cf21) is 8M, max 599.8M, 591.8M free.
### /var/log
102400 /var/log
24576 /var/log/apt
4096 /var/log/private
### apt
273600512 /var/cache/apt
95510528 /var/lib/apt/lists
10
### tmp
0 /tmp
20480 /var/tmp
2026-11-10 40 /tmp/.ICE-unix
2026-11-10 40 /tmp/.X11-unix
2026-11-10 40 /tmp/.XIM-unix
2026-11-10 40 /tmp/.font-unix
2026-11-10 4096 /var/tmp/network
2026-11-10 4096 /var/tmp/systemd-private-125ecfc1db3047fbbf4e763de09f2c08-systemd-logind.service-Ow3BCE
2026-11-10 60 /tmp/systemd-private-125ecfc1db3047fbbf4e763de09f2c08-systemd-logind.service-NJqsTs
entries=7
### felhom dirs
/etc/felhom-bootstrap
/etc/felhom-controller-image
/etc/felhom-controller-image.bak-0.301.0
/etc/felhom-controller-image.pre-designbuild
/etc/felhom-controller-image.pre-morning1006
/etc/felhom-controller-image.pre-undo-live
/etc/felhom-scratch-disposition
/opt/docker
/srv/felhom-gate
/var/lib/felhom
### root filesystem top 45 to depth 4
1918656512 /
906305536 /usr
444448768 /var
413372416 /mnt
413364224 /mnt/sizing-bench
335740928 /mnt/sizing-bench/appdata
335736832 /mnt/sizing-bench/appdata/immich
334995456 /usr/bin
277381120 /var/cache
273600512 /var/cache/apt
265764864 /usr/lib
182714368 /usr/share
178462720 /var/cache/apt/archives
165945344 /var/lib
154312704 /usr/lib/x86_64-linux-gnu
147255296 /root
146108416 /root/sizing
146104320 /root/sizing/content
146087936 /root/sizing/content/photos
106311680 /usr/libexec
106135552 /usr/libexec/docker
105385984 /usr/libexec/docker/cli-plugins
95793152 /var/lib/apt
95510528 /var/lib/apt/lists
77533184 /mnt/sizing-bench/backups
77524992 /mnt/sizing-bench/backups/secondary
67100672 /var/lib/dpkg
66502656 /usr/share/locale
30691328 /usr/share/doc
28688384 /usr/lib/python3.13
26943488 /usr/lib/x86_64-linux-gnu/perl
25632768 /usr/lib/git-core
23773184 /usr/share/perl
23769088 /usr/share/perl/5.40.1
23347200 /usr/lib/udev
17899520 /usr/share/man
16842752 /usr/share/i18n
16412672 /usr/sbin
13266944 /usr/share/i18n/locales
10358784 /usr/lib/file
9625600 /var/lib/dpkg/info
9428992 /usr/lib/udev/hwdb.d
9031680 /usr/lib/python3
9027584 /usr/lib/python3/dist-packages
8601600 /usr/lib/x86_64-linux-gnu/gconv
sort: write failed: 'standard output': Broken pipe
sort: write error
### /var/lib/felhom depth 3 (data volume)
13988098048 /var/lib/felhom
13988020224 /var/lib/felhom/docker
13676625920 /var/lib/felhom/docker/overlay2
247779328 /var/lib/felhom/docker/volumes
45240320 /var/lib/felhom/docker/image
17645568 /var/lib/felhom/docker/containers
495616 /var/lib/felhom/docker/buildkit
196608 /var/lib/felhom/docker/network
192512 /var/lib/felhom/docker/network/files
57344 /var/lib/felhom/sys_drive
49152 /var/lib/felhom/sys_drive/felhom-data
24576 /var/lib/felhom/sys_drive/felhom-data/userdata
20480 /var/lib/felhom/sys_drive/felhom-data/backups
16384 /var/lib/felhom/lost+found
12288 /var/lib/felhom/docker/plugins
8192 /var/lib/felhom/docker/tmp
4096 /var/lib/felhom/sys_drive/appdata
4096 /var/lib/felhom/docker/tmp/moby-detect-rro3963682943
4096 /var/lib/felhom/docker/swarm
4096 /var/lib/felhom/docker/runtimes
4096 /var/lib/felhom/docker/plugins/tmp
4096 /var/lib/felhom/docker/plugins/storage
### /opt/docker depth 2
2994176 /opt/docker
2990080 /opt/docker/stacks
237568 /opt/docker/stacks/romm
176128 /opt/docker/stacks/docmost
163840 /opt/docker/stacks/nextcloud
159744 /opt/docker/stacks/sparkyfitness
126976 /opt/docker/stacks/vikunja
102400 /opt/docker/stacks/wanderer
94208 /opt/docker/stacks/paperless-ngx
86016 /opt/docker/stacks/termix
77824 /opt/docker/stacks/navidrome
69632 /opt/docker/stacks/adventurelog
65536 /opt/docker/stacks/jellyfin
57344 /opt/docker/stacks/zipline
53248 /opt/docker/stacks/radicale
49152 /opt/docker/stacks/wishlist
40960 /opt/docker/stacks/rallly
40960 /opt/docker/stacks/outline
40960 /opt/docker/stacks/immich
40960 /opt/docker/stacks/gitea
36864 /opt/docker/stacks/traefik
36864 /opt/docker/stacks/sabnzbd
36864 /opt/docker/stacks/privatebin
36864 /opt/docker/stacks/plex
36864 /opt/docker/stacks/n8n
32768 /opt/docker/stacks/wger
32768 /opt/docker/stacks/seerr
32768 /opt/docker/stacks/homepage
32768 /opt/docker/stacks/grocy
28672 /opt/docker/stacks/transmission
28672 /opt/docker/stacks/qbittorrent
28672 /opt/docker/stacks/nzbget
28672 /opt/docker/stacks/lubelogger
28672 /opt/docker/stacks/karakeep
28672 /opt/docker/stacks/dawarich
24576 /opt/docker/stacks/vaultwarden
24576 /opt/docker/stacks/uptime-kuma
24576 /opt/docker/stacks/tandoor
24576 /opt/docker/stacks/sonarr
24576 /opt/docker/stacks/recipe-importer
24576 /opt/docker/stacks/radarr
24576 /opt/docker/stacks/papra
24576 /opt/docker/stacks/metube
24576 /opt/docker/stacks/mealie
24576 /opt/docker/stacks/komga
24576 /opt/docker/stacks/kimai
24576 /opt/docker/stacks/homebox
24576 /opt/docker/stacks/home-assistant
24576 /opt/docker/stacks/grimmory
24576 /opt/docker/stacks/grafana
### drives under /mnt/felhom-drives (depth 3 from each drive)
-- /mnt/felhom-drives/scratch_hdd
/dev/nvme0n1 1006980812800 322734333952 633019228160 34% /mnt/felhom-drives/scratch_hdd
111988736 /mnt/felhom-drives/scratch_hdd
77508608 /mnt/felhom-drives/scratch_hdd/backups
77488128 /mnt/felhom-drives/scratch_hdd/backups/primary
77484032 /mnt/felhom-drives/scratch_hdd/backups/primary/paperless-ngx
33083392 /mnt/felhom-drives/scratch_hdd/userdata
30248960 /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx
30236672 /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx/appdata
1523712 /mnt/felhom-drives/scratch_hdd/userdata/media
1355776 /mnt/felhom-drives/scratch_hdd/kept
1351680 /mnt/felhom-drives/scratch_hdd/kept/paperless-ngx
1347584 /mnt/felhom-drives/scratch_hdd/kept/paperless-ngx/2026-11-03_014826
995328 /mnt/felhom-drives/scratch_hdd/userdata/media/metube
593920 /mnt/felhom-drives/scratch_hdd/userdata/calibre-web
581632 /mnt/felhom-drives/scratch_hdd/userdata/calibre-web/userdata
450560 /mnt/felhom-drives/scratch_hdd/userdata/media/books
417792 /mnt/felhom-drives/scratch_hdd/userdata/immich
335872 /mnt/felhom-drives/scratch_hdd/userdata/immich/kept
73728 /mnt/felhom-drives/scratch_hdd/userdata/romm
57344 /mnt/felhom-drives/scratch_hdd/userdata/immich/appdata
32768 /mnt/felhom-drives/scratch_hdd/userdata/romm/kept
32768 /mnt/felhom-drives/scratch_hdd/appdata
28672 /mnt/felhom-drives/scratch_hdd/userdata/sonarr
28672 /mnt/felhom-drives/scratch_hdd/userdata/radarr
28672 /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf
24576 /mnt/felhom-drives/scratch_hdd/userdata/navidrome
24576 /mnt/felhom-drives/scratch_hdd/userdata/komga
24576 /mnt/felhom-drives/scratch_hdd/appdata/paperless
20480 /mnt/felhom-drives/scratch_hdd/userdata/plex
20480 /mnt/felhom-drives/scratch_hdd/userdata/jellyfin
20480 /mnt/felhom-drives/scratch_hdd/userdata/emby
16384 /mnt/felhom-drives/scratch_hdd/userdata/sonarr/userdata
16384 /mnt/felhom-drives/scratch_hdd/userdata/romm/appdata
16384 /mnt/felhom-drives/scratch_hdd/userdata/radarr/userdata
16384 /mnt/felhom-drives/scratch_hdd/userdata/media/transmission
16384 /mnt/felhom-drives/scratch_hdd/userdata/media/torrents
16384 /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf/userdata
16384 /mnt/felhom-drives/scratch_hdd/appdata/paperless/media
12288 /mnt/felhom-drives/scratch_hdd/userdata/romm/userdata
12288 /mnt/felhom-drives/scratch_hdd/userdata/navidrome/userdata
12288 /mnt/felhom-drives/scratch_hdd/userdata/komga/userdata
12288 /mnt/felhom-drives/scratch_hdd/userdata/immich/userdata
8192 /mnt/felhom-drives/scratch_hdd/userdata/sonarr/backups
8192 /mnt/felhom-drives/scratch_hdd/userdata/roms
8192 /mnt/felhom-drives/scratch_hdd/userdata/romm/backups
8192 /mnt/felhom-drives/scratch_hdd/userdata/radarr/backups
8192 /mnt/felhom-drives/scratch_hdd/userdata/plex/userdata
8192 /mnt/felhom-drives/scratch_hdd/userdata/plex/backups
8192 /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx/backups
8192 /mnt/felhom-drives/scratch_hdd/userdata/navidrome/backups
8192 /mnt/felhom-drives/scratch_hdd/userdata/komga/backups
8192 /mnt/felhom-drives/scratch_hdd/userdata/jellyfin/userdata
8192 /mnt/felhom-drives/scratch_hdd/userdata/jellyfin/backups
8192 /mnt/felhom-drives/scratch_hdd/userdata/immich/backups
8192 /mnt/felhom-drives/scratch_hdd/userdata/emby/userdata
8192 /mnt/felhom-drives/scratch_hdd/userdata/emby/backups
8192 /mnt/felhom-drives/scratch_hdd/userdata/calibre-web/backups
8192 /mnt/felhom-drives/scratch_hdd/userdata/audiobookshelf/backups
4096 /mnt/felhom-drives/scratch_hdd/userdata/roms/roms
4096 /mnt/felhom-drives/scratch_hdd/userdata/roms-bios
4096 /mnt/felhom-drives/scratch_hdd/userdata/nextcloud
### timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2026-11-10 16:15:06 UTC 8h Mon 2026-11-09 23:31:47 UTC - apt-daily.timer apt-daily.service
Wed 2026-11-11 00:00:00 UTC 16h Tue 2026-11-10 00:00:36 UTC - dpkg-db-backup.timer dpkg-db-backup.service
Wed 2026-11-11 00:31:48 UTC 17h Tue 2026-11-10 00:21:47 UTC - logrotate.timer logrotate.service
Wed 2026-11-11 02:09:55 UTC 18h Tue 2026-11-10 02:09:55 UTC 5h 18min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Wed 2026-11-11 06:10:13 UTC 22h Tue 2026-11-10 06:00:55 UTC 1h 27min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Wed 2026-11-11 11:11:14 UTC 1 day 3h Tue 2026-11-10 02:51:55 UTC 4h 36min ago man-db.timer man-db.service
Sun 2026-11-15 03:10:17 UTC 4 days Sun 2026-11-08 03:10:47 UTC - e2scrub_all.timer e2scrub_all.service
- - - - fstrim.timer fstrim.service
8 timers listed.
### cron
/etc/cron.d:
e2scrub_all
/etc/cron.daily:
apt-compat
dpkg
logrotate
man-db
/etc/cron.weekly:
man-db
### end
2026-11-10T07:28:04Z
@@ -0,0 +1,522 @@
### identity
demo-hp
2026-11-10T07:33:37Z
### in flight
1160517 /usr/bin/proxmox-backup-client status --output-format=json --repository felhom@pbs!demo-hp@10.77.0.1:felhom-offsite
(end in flight)
### agent state dir (depth 2, bytes)
740171776 /var/lib/felhom-agent
739938304 /var/lib/felhom-agent/selfupdate
32768 /var/lib/felhom-agent/os
32768 /var/lib/felhom-agent/guests
20480 /var/lib/felhom-agent/wg
16384 /var/lib/felhom-agent/guests/9201
16384 /var/lib/felhom-agent/felhom-sshd
12288 /var/lib/felhom-agent/units
12288 /var/lib/felhom-agent/pbsdr
12288 /var/lib/felhom-agent/guests/9202
4096 /var/lib/felhom-agent/escrow-stage
### selfupdate listing
total 722600
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-07 23:13 .
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10 02:46 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 14190233 2026-09-15 10:44 felhom-agent-0.131.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14195690 2026-09-17 10:29 felhom-agent-0.132.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14262881 2026-09-24 21:48 felhom-agent-0.133.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14271901 2026-09-24 22:57 felhom-agent-0.134.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14273488 2026-09-27 12:27 felhom-agent-0.135.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14274024 2026-09-27 13:58 felhom-agent-0.136.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14273888 2026-09-27 14:13 felhom-agent-0.137.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14278858 2026-09-30 10:40 felhom-agent-0.138.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14284239 2026-10-04 08:55 felhom-agent-0.139.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14355058 2026-10-04 11:37 felhom-agent-0.140.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14376457 2026-10-04 13:22 felhom-agent-0.141.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14376105 2026-10-04 14:22 felhom-agent-0.141.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14427309 2026-10-04 16:24 felhom-agent-0.142.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14427261 2026-10-04 18:34 felhom-agent-0.142.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14461055 2026-10-04 19:49 felhom-agent-0.143.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14493513 2026-10-05 07:27 felhom-agent-0.144.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14494141 2026-10-05 08:00 felhom-agent-0.144.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14494291 2026-10-05 09:29 felhom-agent-0.145.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14504200 2026-10-05 12:27 felhom-agent-0.146.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14504898 2026-10-05 19:12 felhom-agent-0.147.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14515292 2026-10-06 00:42 felhom-agent-0.148.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14569321 2026-10-06 11:51 felhom-agent-0.149.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14590727 2026-10-07 09:06 felhom-agent-0.150.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14631775 2026-10-07 13:20 felhom-agent-0.151.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14715119 2026-10-07 17:31 felhom-agent-0.152.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14719868 2026-10-07 19:01 felhom-agent-0.153.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14731041 2026-10-09 07:16 felhom-agent-0.154.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14783135 2026-10-13 20:46 felhom-agent-0.155.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14820843 2026-10-14 13:00 felhom-agent-0.156.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14874944 2026-10-15 21:11 felhom-agent-0.157.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14914581 2026-10-16 04:11 felhom-agent-0.158.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14919180 2026-10-16 04:26 felhom-agent-0.158.1
-rwxr-xr-x 1 felhom-agent felhom-agent 15999736 2026-10-17 15:37 felhom-agent-0.159.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16007680 2026-10-17 16:53 felhom-agent-0.159.1
-rwxr-xr-x 1 felhom-agent felhom-agent 16007752 2026-10-18 13:08 felhom-agent-0.159.2
-rwxr-xr-x 1 felhom-agent felhom-agent 16071264 2026-10-19 03:23 felhom-agent-0.160.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16082682 2026-10-20 08:38 felhom-agent-0.161.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16094507 2026-10-22 09:49 felhom-agent-0.162.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16183935 2026-10-23 08:49 felhom-agent-0.163.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16272301 2026-10-24 09:32 felhom-agent-0.164.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16286780 2026-10-25 17:02 felhom-agent-0.165.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16291150 2026-10-26 09:03 felhom-agent-0.166.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16296021 2026-10-27 13:48 felhom-agent-0.167.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16320123 2026-10-29 09:55 felhom-agent-0.168.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16350287 2026-10-29 12:55 felhom-agent-0.169.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16379051 2026-10-29 17:11 felhom-agent-0.170.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16389260 2026-10-30 10:11 felhom-agent-0.171.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16393888 2026-11-03 08:56 felhom-agent-0.172.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16393888 2026-11-07 23:11 felhom-agent-0.172.1
### selfupdate subdirs listing
### agent state top-level listing (names, sizes, dates only)
total 136
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10 02:46 .
drwxr-xr-x 46 root root 4096 2026-10-10 04:42 ..
-rw------- 1 felhom-agent felhom-agent 593 2026-11-10 02:46 backup-success-state.json
-rw------- 1 felhom-agent felhom-agent 290 2026-11-09 13:14 controller-swap-9201.json
-rw------- 1 felhom-agent felhom-agent 126 2026-10-17 16:23 drive-intents.json
drwx------ 2 felhom-agent felhom-agent 4096 2026-10-19 03:48 escrow-stage
drwx------ 2 felhom-agent felhom-agent 4096 2026-08-21 17:59 felhom-sshd
-rw------- 1 felhom-agent felhom-agent 301 2026-10-19 03:49 format-job.json
-rw------- 1 felhom-agent felhom-agent 67 2026-10-17 17:08 guest-binds.json
-rw------- 1 felhom-agent felhom-agent 240 2026-11-04 10:01 guest-disk-trim.json
drwx------ 4 felhom-agent felhom-agent 4096 2026-09-13 22:16 guests
-rw------- 1 felhom-agent felhom-agent 35762 2026-11-09 05:59 journal.log
-rw------- 1 felhom-agent felhom-agent 302 2026-11-03 11:37 line-test.json
-rw-r--r-- 1 felhom-agent felhom-agent 591 2026-08-21 17:59 local-api.crt
-rw------- 1 felhom-agent felhom-agent 227 2026-08-21 17:59 local-api.key
-rw------- 1 felhom-agent felhom-agent 82 2026-08-21 18:00 local-tokens.log
-rw------- 1 felhom-agent felhom-agent 6052 2026-11-07 23:27 nonces.log
drwx------ 2 felhom-agent felhom-agent 4096 2026-11-10 08:24 os
drwx------ 2 felhom-agent felhom-agent 4096 2026-08-21 18:09 pbsdr
-rw------- 1 felhom-agent felhom-agent 367 2026-11-09 05:59 restore-test-state.json
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-07 23:13 selfupdate
-rw------- 1 felhom-agent felhom-agent 32 2026-11-03 08:56 store-grant-seen.json
drwx------ 2 felhom-agent felhom-agent 4096 2026-10-17 16:11 units
drwx------ 2 felhom-agent felhom-agent 4096 2026-08-21 17:59 wg
### os-apply state
376832 /var/lib/felhom-os-apply
364544 /var/lib/felhom-os-apply/bundle-prev
159744 /var/lib/felhom-os-apply/bundle-prev/20261107T222713Z-before-0.172.1
159744 /var/lib/felhom-os-apply/bundle-prev/20261026T080307Z-before-0.166.0
40960 /var/lib/felhom-os-apply/bundle-prev/20261029T161106Z-before-0.170.0
4096 /var/lib/felhom-os-apply/agent-update
/var/lib/felhom-os-apply:
total 20
drwx------ 4 root root 4096 2026-11-07 23:27 .
drwxr-xr-x 46 root root 4096 2026-10-10 04:42 ..
drwxr-xr-x 2 root root 4096 2026-11-07 23:11 agent-update
drwxr-xr-x 5 root root 4096 2026-11-07 23:27 bundle-prev
-rw-r--r-- 1 root root 96 2026-11-07 23:27 nonces.json
/var/lib/felhom-os-apply/bundle-prev:
total 20
drwxr-xr-x 5 root root 4096 2026-11-07 23:27 .
drwx------ 4 root root 4096 2026-11-07 23:27 ..
drwxr-xr-x 3 root root 4096 2026-10-26 09:03 20261026T080307Z-before-0.166.0
drwxr-xr-x 3 root root 4096 2026-10-29 17:11 20261029T161106Z-before-0.170.0
drwxr-xr-x 3 root root 4096 2026-11-07 23:27 20261107T222713Z-before-0.172.1
### usr local bin + sbin
/usr/local/bin:
total 45984
drwxr-xr-x 2 root root 4096 2026-11-07 23:11 .
drwxr-xr-x 11 root root 4096 2026-05-21 13:31 ..
-rwxr-xr-x 1 root root 16393888 2026-11-07 23:11 felhom-agent
-rwxr-xr-x 1 root root 14284239 2026-10-04 11:02 felhom-agent.bak-0.139.0
-rwxr-xr-x 1 root root 16393888 2026-11-03 08:56 felhom-agent.prev
/usr/local/sbin:
total 336
drwxr-xr-x 3 root root 4096 2026-11-07 23:27 .
drwxr-xr-x 11 root root 4096 2026-05-21 13:31 ..
-rwxr-xr-x 1 root root 6654 2026-10-04 20:34 felhom-backup-target-apply
-rwxr-xr-x 1 root root 27518 2026-07-31 19:03 felhom-bootstrap.sh
-rwxr-xr-x 1 root root 8687 2026-10-04 16:08 felhom-crash-guard
-rwxr-xr-x 1 root root 6816 2026-10-16 04:41 felhom-internal-place
-rwxr-xr-x 1 root root 3393 2026-08-21 17:59 felhom-mgmt-watchdog
-rwxr-xr-x 1 root root 5439 2026-10-16 04:41 felhom-mkfs-guarded
-rwxr-xr-x 1 root root 142720 2026-11-07 23:27 felhom-os-apply
-rwxr-xr-x 1 root root 50933 2026-10-04 18:00 felhom-os-apply.pre-bundle
-rwxr-xr-x 1 root root 9087 2026-08-21 17:59 felhom-pbs-apply
-rwxr-xr-x 1 root root 26062 2026-10-17 16:08 felhom-priv-apply
-rwxr-xr-x 1 root root 7806 2026-10-05 12:57 felhom-selfupdate-guarded
-rwxr-xr-x 1 root root 943 2026-08-21 17:59 felhom-shared-parent.sh
-rwxr-xr-x 1 root root 3844 2026-10-17 16:08 felhom-thin-check
-rwxr-xr-x 1 root root 5537 2026-10-14 13:30 felhom-vzdump-partial-clean
drwxr-xr-x 2 root root 4096 2026-10-05 07:44 __pycache__
### dumps on drives
-- /mnt/hdd_1/dump
total 4980164
drwxr-xr-x 2 root root 4096 2026-08-21 05:17 .
drwxr-xr-x 9 root root 4096 2026-10-17 16:24 ..
-rw-r--r-- 1 root root 649547835 2026-08-03 07:33 vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst
-rw-r--r-- 1 root root 1504 2026-08-19 05:07 vzdump-lxc-9201-2026_08_19-05_06_40.log
-rw-r--r-- 1 root root 1483114465 2026-08-19 05:07 vzdump-lxc-9201-2026_08_19-05_06_40.tar.zst
-rw-r--r-- 1 root root 16 2026-08-19 05:07 vzdump-lxc-9201-2026_08_19-05_06_40.tar.zst.notes
-rw-r--r-- 1 root root 1504 2026-08-20 05:12 vzdump-lxc-9201-2026_08_20-05_11_40.log
-rw-r--r-- 1 root root 1483358426 2026-08-20 05:12 vzdump-lxc-9201-2026_08_20-05_11_40.tar.zst
-rw-r--r-- 1 root root 16 2026-08-20 05:12 vzdump-lxc-9201-2026_08_20-05_11_40.tar.zst.notes
-rw-r--r-- 1 root root 1504 2026-08-21 05:17 vzdump-lxc-9201-2026_08_21-05_16_40.log
-rw-r--r-- 1 root root 1483606500 2026-08-21 05:17 vzdump-lxc-9201-2026_08_21-05_16_40.tar.zst
-rw-r--r-- 1 root root 16 2026-08-21 05:17 vzdump-lxc-9201-2026_08_21-05_16_40.tar.zst.notes
-- /var/lib/vz/dump
total 9521616
drwxr-xr-x 2 root root 4096 2026-11-10 02:46 .
drwxr-xr-x 7 root root 4096 2026-08-21 18:00 ..
-rw-r--r-- 1 root root 654115664 2026-09-13 22:16 felhom-golden-0.236.0.tar.zst
-rw-r--r-- 1 root root 1874 2026-09-24 07:05 vzdump-lxc-9201-2026_09_24-06_59_28.log
-rw-r--r-- 1 root root 1874 2026-09-24 07:24 vzdump-lxc-9201-2026_09_24-07_17_53.log
-rw-r--r-- 1 root root 1874 2026-09-24 07:49 vzdump-lxc-9201-2026_09_24-07_42_54.log
-rw-r--r-- 1 root root 187 2026-09-24 08:22 vzdump-lxc-9201-2026_09_24-08_22_55.log
-rw-r--r-- 1 root root 1780 2026-09-24 22:45 vzdump-lxc-9201-2026_09_24-22_40_21.log
-rw-r--r-- 1 root root 1563 2026-10-14 07:28 vzdump-lxc-9201-2026_10_14-07_17_45.log
-rw-r--r-- 1 root root 1942 2026-10-14 11:06 vzdump-lxc-9201-2026_10_14-11_02_50.log
-rw-r--r-- 1 root root 994 2026-10-14 13:42 vzdump-lxc-9201-2026_10_14-13_42_48.log
-rw-r--r-- 1 root root 1495 2026-11-10 02:46 vzdump-lxc-9201-2026_11_10-02_35_34.log
-rw-r--r-- 1 root root 9095951125 2026-11-10 02:46 vzdump-lxc-9201-2026_11_10-02_35_34.tar.zst
-rw-r--r-- 1 root root 16 2026-11-10 02:46 vzdump-lxc-9201-2026_11_10-02_35_34.tar.zst.notes
-rw-r--r-- 1 root root 2234 2026-10-14 11:02 vzdump-lxc-9202-2026_10_14-11_01_36.log
### template cache
/var/lib/vz/template/cache:
total 126924
drwxr-xr-x 2 root root 4096 2026-10-01 16:14 .
drwxr-xr-x 4 root root 4096 2026-08-21 17:42 ..
-rw-r--r-- 1 root root 129954319 2026-07-14 21:37 debian-13-standard_13.6-1_amd64.tar.zst
/var/lib/vz/template/iso:
total 8
drwxr-xr-x 2 root root 4096 2026-11-09 09:51 .
drwxr-xr-x 4 root root 4096 2026-08-21 17:42 ..
### journald disk usage + limits
Archived and active journals take up 833.2M in the file system.
Nov 10 02:53:57 demo-hp systemd-journald[351]: Runtime Journal (/run/log/journal/9756a908fef046fda6a56428d3fefe3c) is 8M, max 299.9M, 291.9M free.
Nov 10 02:53:57 demo-hp systemd-journald[351]: System Journal (/var/log/journal/9756a908fef046fda6a56428d3fefe3c) is 801.2M, max 3.8G, 3G free.
### journal oldest + newest entry
2026-11-10T08:33:37+01:00 demo-hp systemd[1]: Started session-40.scope - Session 40 of User root.
2026-10-04T21:42:08+02:00 demo-hp systemd[779137]: Queued start job for default target default.target.
2026-10-04T21:42:08+02:00 demo-hp systemd[779137]: Created slice app.slice - User Application Slice.
### journal files (count, oldest, newest)
files=16 bytes=1107296256 oldest=2026-10-04 newest=2026-11-10
### journal bytes exact
873717760 /var/log/journal
### journal last hour: lines by identifier (top 12)
12825 "SYSLOG_IDENTIFIER":"sudo"
442 "SYSLOG_IDENTIFIER":"felhom-agent"
368 "SYSLOG_IDENTIFIER":"systemd"
31 "SYSLOG_IDENTIFIER":"sshd-session"
23 "SYSLOG_IDENTIFIER":"systemd-logind"
6 "SYSLOG_IDENTIFIER":"postfix/smtp"
4 "SYSLOG_IDENTIFIER":"(systemd)"
3 "SYSLOG_IDENTIFIER":"(sd-pam)"
3 "SYSLOG_IDENTIFIER":"postfix/qmgr"
3 "SYSLOG_IDENTIFIER":"CRON"
### journal last hour: total lines
13708
### journald conf files
[Journal]
### thin-check script
-- /usr/local/sbin/felhom-thin-check
#!/bin/bash
# felhom-thin-check — list the thin device ids a thin pool holds next to the ones LVM knows (R-1012). READ-ONLY: it
# names an orphan, it never removes one. Installed as /usr/local/sbin/felhom-thin-check (config bundle).
#
# felhom-thin-check <vg> <pool>
#
# A full host root while vzdump takes its snapshot can leave a thin snapshot inside the pool that LVM no longer knows
# (demo-hp 2026-10-14: `thin_ls` device 4, LVM only 1 and 2); from then on every snapshot backup of that guest fails
# („Creation of new snapshot 4 of device 2 failed") and nothing said why. The repair (`dmsetup message … delete <id>`)
# stays a HUMAN step, decided by the operator — this helper only reads.
#
# Reading the live pool's device list needs a metadata snapshot: the kernel's documented way to read metadata while
# the pool is in use (`reserve_metadata_snap`, `thin_ls --metadata-snap`, `release_metadata_snap`). It changes no data
# and no mapping; it is held for the length of one `thin_ls -o DEV` (well under a second on these pools). Rules:
# 1. the arguments are an LVM name pair; <vg>/<pool> is a thin pool (lv_attr starts with `t`);
# 2. only one run at a time (a lock file); a metadata snapshot that someone ELSE holds is never touched — the run
# refuses (exit 3) and does not release it;
# 3. LVM's list is read before AND after the pool's, so a volume created in between is not reported as an orphan
# (the caller treats an id as an orphan only when it is in the pool and in neither LVM list).
# Output, one record per line: lvm-before <id> <lv> | pool <id> | lvm-after <id> <lv> | ok
# Exit: 0 read, 1 refused (arguments / not a thin pool), 3 a metadata snapshot is already held or cannot be reserved,
# 4 the pool could not be read. Pinned by configs/test_felhom_thin_check.py.
set -u
LVS=/usr/sbin/lvs
DMSETUP=/usr/sbin/dmsetup
THIN_LS=/usr/sbin/thin_ls
LOCK=/run/lock/felhom-thin-check.lock
NAME_RE='^[A-Za-z0-9_+][A-Za-z0-9_.+-]{0,63}$'
die() { echo "felhom-thin-check: REFUSED: $1" >&2; exit "${2:-1}"; } # $1 the reason, $2 the exit code
lvm_ids() { # $1 tag → "<tag> <id> <lv>" for every thin volume of this pool
"$LVS" --noheadings --separator , -o lv_name,thin_id --select "pool_lv=$POOL" -- "$VG" 2>/dev/null |
awk -F, -v tag="$1" '{ gsub(/ /, ""); if ($2 ~ /^[0-9]+$/) print tag, $2, $1 }'
}
release() { "$DMSETUP" message "$TPOOL" 0 release_metadata_snap >/dev/null 2>&1; }
main() {
[[ $# -eq 2 ]] || die "usage: felhom-thin-check <vg> <pool>"
VG=$1 POOL=$2
local attr held ids
[[ "$VG" =~ $NAME_RE && "$POOL" =~ $NAME_RE ]] || die "not an LVM name pair"
attr=$("$LVS" --noheadings -o lv_attr -- "$VG/$POOL" 2>/dev/null | tr -d ' ')
[[ "$attr" == t* ]] || die "$VG/$POOL is not a thin pool"
DM="${VG//-/--}-${POOL//-/--}"
TPOOL="$DM-tpool"
TMETA="/dev/mapper/${DM}_tmeta"
exec 9>"$LOCK" || die "cannot open the lock file" 4
flock -w 30 9 || die "another check holds the lock" 3
# Field 7 of the thin-pool status is the held metadata root: "-" when no metadata snapshot exists.
held=$("$DMSETUP" status "$TPOOL" 2>/dev/null | awk '$3 == "thin-pool" { print $7 }')
[[ -n "$held" ]] || die "cannot read the status of $TPOOL" 4
[[ "$held" == "-" ]] || die "a metadata snapshot of $TPOOL is already held (by someone else) — not touched" 3
lvm_ids lvm-before
"$DMSETUP" message "$TPOOL" 0 reserve_metadata_snap >/dev/null 2>&1 || die "could not reserve a metadata snapshot of $TPOOL" 3
trap release EXIT
trap 'release; trap - EXIT; exit 4' INT TERM HUP
if ! ids=$("$THIN_LS" --metadata-snap --no-headers -o DEV "$TMETA" 2>/dev/null); then
die "thin_ls could not read $TMETA" 4
fi
release
trap - EXIT INT TERM HUP
awk '$1 ~ /^[0-9]+$/ { print "pool", $1 }' <<<"$ids"
lvm_ids lvm-after
echo ok
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then main "$@"; fi
-- /usr/local/sbin/felhom-vzdump-partial-clean
#!/bin/bash
# felhom-vzdump-partial-clean — remove ONE dead, half-written vzdump backup (R-1006). The only delete the agent's
# sudoers allows for backup files. Installed as /usr/local/sbin/felhom-vzdump-partial-clean (config bundle).
#
# felhom-vzdump-partial-clean <storage> <vmid> <name>
#
# A power cut during the night backup leaves `vzdump-lxc-<vmid>-<date>.tar.dat` (+ its `.tmp/` directory) on the
# storage; nothing removes it, every later backup is then skipped „not enough space", and a full host root breaks
# LVM's own metadata writes (power-cut-fixes-2026-10-14 C4). This helper removes such a leftover — and NOTHING else.
# It trusts nothing the agent says; every rule is checked here, as root, and any doubt REFUSES (exit 1):
# 1. the arguments: a storage id, a vmid, and a name that is exactly
# vzdump-(lxc|qemu)-<THAT vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp) — never a finished archive
# (.tar.zst, .vma.zst, …), never a .log/.notes, never another vmid;
# 2. the storage: its section in /etc/pve/storage.cfg is `dir: <storage>` with a `path`; the target is
# <path>/dump/<name>, not a symlink, and its real directory IS <path>/dump;
# 3. not in use: no vzdump runs on the host at all (a task `…:vzdump:…` or a `vzdump` command — a job for several
# guests carries no vmid in its task name), and no
# process holds the target (or, for the .tmp directory, anything under it) open or as its working directory
# (C2: after a killed worker, tar and zstd kept writing into the .tar.dat for 4 min 40 s, none named „vzdump");
# 4. dead: last modified before this boot (`btime` in /proc/stat), or untouched for at least QUIET_S seconds.
# Prints one line on success: „removed <name> <bytes> B (<why>)". Pinned by configs/test_felhom_vzdump_partial_clean.py.
set -u
STORAGE_CFG=/etc/pve/storage.cfg
PROC=/proc
QUIET_S=120
die() { echo "felhom-vzdump-partial-clean: REFUSED: $*" >&2; exit 1; }
storage_path() { # $1 storage id → prints the dir storage's path, or nothing
awk -v want="$1" '
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (id == want); next }
in_s && /^[ \t]+path[ \t]/ { if (type == "dir") { sub(/^[ \t]+path[ \t]+/, ""); sub(/[ \t]+$/, ""); print; exit } }
' "$STORAGE_CFG"
}
storage_type() {
awk -v want="$1" '/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); id = h[2]; sub(/[ \t]+$/, "", id); if (id == want) { print h[1]; exit } }' "$STORAGE_CFG"
}
boot_time() { awk '$1 == "btime" { print $2 }' "$PROC/stat"; }
newest_mtime() { # a file's mtime, or the newest mtime of anything in a directory (the dir itself included)
if [[ -d "$1" ]]; then find "$1" -xdev -printf '%T@\n' 2>/dev/null | sort -n | tail -1 | cut -d. -f1
else stat -c %Y "$1"; fi
}
busy_reason() { # $1 vmid, $2 target → prints why it is in use, or nothing
local vmid=$1 target=$2 p link cmd
for p in "$PROC"/[0-9]*; do
[[ -r "$p/cmdline" ]] || continue
cmd=$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)
# ANY vzdump on the host refuses: a job for several guests runs as `task UPID:…:vzdump::…` (no vmid in it), and
# its tar may be between this guest's files. Waiting costs one backup run; deleting a live one costs the backup.
if [[ "$cmd" == *":vzdump:"* ]] || [[ "$cmd" =~ (^|/)vzdump([[:space:]]|$) ]]; then
echo "a vzdump runs on this host (pid ${p##*/})"; return
fi
for link in "$p/cwd" "$p"/fd/*; do
local to
to=$(readlink "$link" 2>/dev/null) || continue
if [[ "$to" == "$target" || "$to" == "$target/"* ]]; then
echo "pid ${p##*/} has it open"; return
fi
done
done
}
main() {
local storage=${1:-} vmid=${2:-} name=${3:-}
[[ $# -eq 3 ]] || die "usage: felhom-vzdump-partial-clean <storage> <vmid> <name>"
[[ "$storage" =~ ^[A-Za-z][A-Za-z0-9_-]{0,63}$ ]] || die "bad storage id"
[[ "$vmid" =~ ^[1-9][0-9]{2,8}$ ]] || die "bad vmid"
[[ "$name" =~ ^vzdump-(lxc|qemu)-${vmid}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$ ]] \
|| die "name is not a half-written vzdump of guest $vmid: $name"
[[ "$(storage_type "$storage")" == "dir" ]] || die "storage $storage is not a dir storage"
local path; path=$(storage_path "$storage")
[[ "$path" == /* ]] || die "storage $storage has no path"
local dump="$path/dump" target="$path/dump/$name"
[[ -e "$target" ]] || die "no such entry: $target"
[[ ! -L "$target" ]] || die "a symlink: $target"
[[ "$(realpath -e "$dump")" == "$(realpath -e "$(dirname "$target")")" ]] || die "not directly in $dump"
if [[ "$name" == *.tmp ]]; then [[ -d "$target" ]] || die ".tmp is not a directory"
else [[ -f "$target" ]] || die "not a regular file"; fi
local busy; busy=$(busy_reason "$vmid" "$target")
[[ -z "$busy" ]] || die "in use: $busy"
local bt mt now why
bt=$(boot_time); mt=$(newest_mtime "$target"); now=$(date +%s)
[[ "$bt" =~ ^[0-9]+$ && "$mt" =~ ^[0-9]+$ ]] || die "cannot read the boot time or the modification time"
if (( mt < bt )); then why="written before this boot"
elif (( now - mt >= QUIET_S )); then why="untouched for $(( now - mt )) s, no vzdump for $vmid, nothing holds it"
else die "modified $(( now - mt )) s ago — may still be written"; fi
local size; size=$(du -sb "$target" 2>/dev/null | cut -f1)
if [[ -d "$target" ]]; then rm -rf --one-file-system -- "$target"; else rm -f -- "$target"; fi
[[ ! -e "$target" ]] || die "remove failed: $target"
echo "removed $name ${size:-?} B ($why)"
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
main "$@"
fi
### units mentioning thin-check / partial-clean
/etc/sudoers.d/felhom-agent
### lvm thin settings
thin_pool_autoextend_threshold=100
thin_pool_autoextend_percent=20
monitoring=1
active
### agent config: selected non-secret keys
-- /etc/felhom-agent/agent.json
"local_backup_target": "local"
retention": 1
"keep_last": 0
### vzdump.conf + storage prune settings
1:dir: local
2: path /var/lib/vz
3: content vztmpl,import,backup,iso
5:lvmthin: local-lvm
8: content images,rootdir
10:pbs: felhom-pbs
13: content backup
19:dir: nvme-scratch
20: path /mnt/hdd_1
21: content images,rootdir
22: is_mountpoint yes
### kernels
proxmox-kernel-7.0.14-20-pve un
proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20 1006525 ii
proxmox-kernel-7.0.14-22-pve un
proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22 1006522 ii
proxmox-kernel-7.0.14-23-pve un
proxmox-kernel-7.0.14-23-pve-signed 7.0.14-23 1006525 ii
proxmox-kernel-7.0.14-24-pve un
proxmox-kernel-7.0.14-24-pve-signed 7.0.14-24 1006528 ii
proxmox-kernel-7.0.14-25-pve un
proxmox-kernel-7.0.14-25-pve-signed 7.0.14-25 1007819 ii
proxmox-kernel-7.0.14-26-pve un
proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26 1007815 ii
proxmox-kernel-7.0.14-28-pve un
proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28 1007817 ii
proxmox-kernel-7.0.2-6-pve un
proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6 1005219 ii
proxmox-kernel-helper 9.1.0+fde2 109 ii
7.0.14-28-pve
### apt history: kernel installs/removes
Start-Date: 2026-11-10 02:50:13
Start-Date: 2026-10-10 04:41:43
Start-Date: 2026-10-11 04:42:11
Start-Date: 2026-10-16 04:48:58
Start-Date: 2026-10-21 04:45:13
Start-Date: 2026-10-29 05:06:42
Start-Date: 2026-10-04 09:42:20
### apt autoremove config
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve-signed$";
### boot
Filesystem 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 41424257024 23968104448 15318945792 62% /
/dev/sda2 1071624192 9469952 1062154240 1% /boot/efi
total 788032
drwxr-xr-x 5 root root 4096 2026-11-10 02:50 .
drwxr-xr-x 19 root root 4096 2026-08-21 17:43 ..
-rw-r--r-- 1 root root 308363 2026-09-24 12:43 config-7.0.14-20-pve
-rw-r--r-- 1 root root 308363 2026-10-06 07:41 config-7.0.14-22-pve
-rw-r--r-- 1 root root 308363 2026-10-07 22:25 config-7.0.14-23-pve
-rw-r--r-- 1 root root 308363 2026-10-09 13:32 config-7.0.14-24-pve
-rw-r--r-- 1 root root 308500 2026-10-17 22:42 config-7.0.14-25-pve
-rw-r--r-- 1 root root 308500 2026-10-22 20:45 config-7.0.14-26-pve
-rw-r--r-- 1 root root 308500 2026-11-02 12:54 config-7.0.14-28-pve
-rw-r--r-- 1 root root 308342 2026-05-20 10:55 config-7.0.2-6-pve
drwxr-xr-x 3 root root 4096 1970-01-01 01:00 efi
drwxr-xr-x 6 root root 4096 2026-11-10 02:56 grub
-rw-r--r-- 1 root root 73094554 2026-10-10 04:41 initrd.img-7.0.14-20-pve
-rw-r--r-- 1 root root 73101983 2026-10-10 04:42 initrd.img-7.0.14-22-pve
-rw-r--r-- 1 root root 73131609 2026-10-11 04:42 initrd.img-7.0.14-23-pve
-rw-r--r-- 1 root root 73115537 2026-10-16 04:49 initrd.img-7.0.14-24-pve
-rw-r--r-- 1 root root 73286572 2026-10-21 04:45 initrd.img-7.0.14-25-pve
-rw-r--r-- 1 root root 73492957 2026-10-29 05:07 initrd.img-7.0.14-26-pve
-rw-r--r-- 1 root root 73547547 2026-11-10 02:50 initrd.img-7.0.14-28-pve
-rw-r--r-- 1 root root 77353337 2026-10-04 09:32 initrd.img-7.0.2-6-pve
-rw-r--r-- 1 root root 151020 2024-11-17 17:43 memtest86+ia32.bin
-rw-r--r-- 1 root root 152064 2024-11-17 17:43 memtest86+ia32.efi
-rw-r--r-- 1 root root 155992 2024-11-17 17:43 memtest86+x64.bin
-rw-r--r-- 1 root root 157184 2024-11-17 17:43 memtest86+x64.efi
drwxr-xr-x 2 root root 4096 2026-11-10 02:50 pve
-rw-r--r-- 1 root root 10483512 2026-09-24 12:43 System.map-7.0.14-20-pve
-rw-r--r-- 1 root root 10483429 2026-10-06 07:41 System.map-7.0.14-22-pve
-rw-r--r-- 1 root root 10483429 2026-10-07 22:25 System.map-7.0.14-23-pve
-rw-r--r-- 1 root root 10483455 2026-10-09 13:32 System.map-7.0.14-24-pve
-rw-r--r-- 1 root root 10341401 2026-10-17 22:42 System.map-7.0.14-25-pve
-rw-r--r-- 1 root root 10341401 2026-10-22 20:45 System.map-7.0.14-26-pve
-rw-r--r-- 1 root root 10341916 2026-11-02 12:54 System.map-7.0.14-28-pve
-rw-r--r-- 1 root root 10452992 2026-05-20 10:55 System.map-7.0.2-6-pve
-rw-r--r-- 1 root root 16284776 2026-09-24 12:43 vmlinuz-7.0.14-20-pve
-rw-r--r-- 1 root root 16288872 2026-10-06 07:41 vmlinuz-7.0.14-22-pve
-rw-r--r-- 1 root root 16288872 2026-10-07 22:25 vmlinuz-7.0.14-23-pve
-rw-r--r-- 1 root root 16288872 2026-10-09 13:32 vmlinuz-7.0.14-24-pve
-rw-r--r-- 1 root root 16284776 2026-10-17 22:42 vmlinuz-7.0.14-25-pve
-rw-r--r-- 1 root root 16280680 2026-10-22 20:45 vmlinuz-7.0.14-26-pve
### pve tasks / rrd
4173824 /var/log/pve
64802816 /var/lib/rrdcached
914
### root dir extras
176128 /root
16384 /root/.ssh
12288 /root/.cache
### df now
Filesystem 1B-blocks Used Avail Use% Mounted on
/dev/mapper/pve-root 41424257024 23968104448 15318945792 62% /
data 57843646464B 70.67 2.85
root 42408607744B
swap 8589934592B
vm-9201-disk-0 34359738368B 7.46
vm-9201-disk-1 75161927680B 50.98
### end
2026-11-10T07:33:38Z
@@ -0,0 +1,585 @@
### identity
demo-hp
2026-11-10T07:25:27Z
pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve)
felhom-agent 0.172.1
### in-flight (must be empty)
### lsblk
NAME SIZE TYPE FSTYPE MOUNTPOINT MODEL TRAN
loop0 34359738368 loop ext4
loop1 75161927680 loop ext4
sda 128035676160 disk SanDisk X600 M.2 2280 SATA 128GB sata
|-sda1 1031168 part
|-sda2 1073741824 part vfat /boot/efi
`-sda3 126700487168 part LVM2_member
|-pve-swap 8589934592 lvm swap [SWAP]
|-pve-root 42408607744 lvm ext4 /
|-pve-data_tmeta 1073741824 lvm
| `-pve-data-tpool 57843646464 lvm
| |-pve-data 57843646464 lvm
| |-pve-vm--9201--disk--0 34359738368 lvm ext4
| `-pve-vm--9201--disk--1 75161927680 lvm ext4
`-pve-data_tdata 57843646464 lvm
`-pve-data-tpool 57843646464 lvm
|-pve-data 57843646464 lvm
|-pve-vm--9201--disk--0 34359738368 lvm ext4
`-pve-vm--9201--disk--1 75161927680 lvm ext4
nvme0n1 1024209543168 disk ext4 /mnt/hdd_1 KXG50PNV1T02 NVMe TOSHIBA 1024GB nvme
### df bytes
Filesystem Type 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-root ext4 41424257024 23968075776 15318974464 62% /
/dev/sda2 vfat 1071624192 9469952 1062154240 1% /boot/efi
/dev/nvme0n1 ext4 1006980812800 322734333952 633019228160 34% /mnt/hdd_1
/dev/fuse fuse 134217728 20480 134197248 1% /etc/pve
### df inodes
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pve-root 2588672 120141 2468531 5% /
/dev/sda2 0 0 0 - /boot/efi
/dev/nvme0n1 62513152 796 62512356 1% /mnt/hdd_1
/dev/fuse 262144 40 262104 1% /etc/pve
### vgs
VG VSize VFree
pve 126697340928 15707668480
### lvs
LV VG Attr LSize Pool Data% Meta% MSize
data pve twi-aotz-- 57843646464 70.67 2.85 1073741824
root pve -wi-ao---- 42408607744
swap pve -wi-ao---- 8589934592
vm-9201-disk-0 pve Vwi-aotz-- 34359738368 data 7.46
vm-9201-disk-1 pve Vwi-aotz-- 75161927680 data 50.98
### thin pool autoextend + monitoring
Configuration node activation/thin_pool_autoextend_threshold not found
Configuration node activation/thin_pool_autoextend_percent not found
Configuration node activation/monitoring not found
### pvesm status
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
felhom-pbs pbs active 0 0 0 0.00%
local dir active 40453376 23406324 14959936 57.86%
local-lvm lvmthin active 56487936 39920024 16567911 70.67%
nvme-scratch dir active 983379700 315170248 618182840 32.05%
### storage.cfg (no secrets live here; passwords are in /etc/pve/priv)
dir: local
path /var/lib/vz
content vztmpl,import,backup,iso
lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir
pbs: felhom-pbs
datastore felhom-offsite
server <redacted>
content backup
encryption-key <redacted>
fingerprint <redacted>
namespace demo-hp
username <redacted>
dir: nvme-scratch
path /mnt/hdd_1
content images,rootdir
is_mountpoint yes
### pvesm list per storage (count, bytes, oldest, newest)
felhom-pbs total entries=2 bytes=61372481541
felhom-pbs type=backup entries=2 bytes=61372481541
local total entries=3 bytes=9880021108
local type=backup entries=2 bytes=9750066789
local type=vztmpl entries=1 bytes=129954319
local-lvm total entries=2 bytes=109521666048
local-lvm type=rootdir entries=2 bytes=109521666048
nvme-scratch total entries=6 bytes=442382172160
nvme-scratch type=rootdir entries=3 bytes=173946175488
nvme-scratch type=images entries=3 bytes=268435996672
### guest configs (disk + mountpoint lines only)
-- /etc/pve/lxc/9201.conf
memory: 25898
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
rootfs: local-lvm:vm-9201-disk-0,size=32G
swap: 25898
-- /etc/pve/lxc/9202.conf
memory: 25898
mp0: nvme-scratch:9202/vm-9202-disk-3.raw,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/hdd_1/scratch-drives/scratch_hdd,mp=/mnt/felhom-drives/scratch_hdd
mp9: /var/lib/felhom-agent/guests/9202/bootstrap,mp=/etc/felhom-bootstrap,ro=1
rootfs: nvme-scratch:9202/vm-9202-disk-2.raw,size=32G
swap: 512
-- /etc/pve/lxc/9401.conf
memory: 10240
rootfs: nvme-scratch:9401/vm-9401-disk-0.raw,size=60G
swap: 0
-- /etc/pve/qemu-server/341.conf
efidisk0: nvme-scratch:341/vm-341-disk-0.raw,efitype=4m,pre-enrolled-keys=0,size=528K
memory: 8192
scsi0: nvme-scratch:341/vm-341-disk-1.raw,size=200G
scsi1: nvme-scratch:341/vm-341-disk-2.raw,size=50G
### vzdump.conf + jobs (keep rules)
### journald
Archived and active journals take up 833.2M in the file system.
[Journal]
18
IDX BOOT ID FIRST ENTRY LAST ENTRY
0 b6e0edea2b5647dfa36f5bb3825adc3a Tue 2026-11-10 02:53:57 CET Tue 2026-11-10 08:25:34 CET
### /var/log top
888266752 /var/log
873717760 /var/log/journal
8613888 /var/log/pveproxy
4173824 /var/log/pve
1277952 /var/log/ifupdown2
90112 /var/log/apt
12288 /var/log/vzdump
12288 /var/log/runit
4096 /var/log/samba
4096 /var/log/private
4096 /var/log/lxc
4096 /var/log/frr
4096 /var/log/corosync
4096 /var/log/chrony
4096 /var/log/ceph
### /var/log big files
75497472 2026-11-10 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-00000000019863ac-00065fdabd48e216.journal
75497472 2026-11-09 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-0000000001967e6f-00065fd31201f8c0.journal
75497472 2026-11-09 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-000000000194a2a7-00065fcbbff95473.journal
75497472 2026-11-08 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-000000000192b8f8-00065fc3f29378b3.journal
75497472 2026-11-08 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-000000000190d02c-00065fbc2c5ae667.journal
75497472 2026-11-08 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-00000000018efb31-00065fb51fc1c0ca.journal
75497472 2026-11-07 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-00000000018d1917-00065fad7fea7f05.journal
75497472 2026-11-07 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-00000000018b39d0-00065fa5fd749421.journal
75497472 2026-11-07 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-0000000001895b19-00065f9e7a9b2feb.journal
75497472 2026-11-06 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-0000000001877a73-00065f96e9075040.journal
75497472 2026-11-06 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-0000000001859e94-00065f8f821cf41b.journal
75497472 2026-11-06 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-000000000183be11-00065f87f10c819a.journal
75497472 2026-11-05 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-000000000181dbcb-00065f805073dc6b.journal
75497472 2026-11-05 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system@a52f7862246a4b6880a56bd9b6b23930-00000000017fffd0-00065f78e90691c4.journal
41943040 2026-11-10 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/system.journal
8388608 2026-10-04 /var/log/journal/9756a908fef046fda6a56428d3fefe3c/user-1000.journal
5692044 2026-11-10 /var/log/pveproxy/access.log.1
### logrotate configs present
alternatives
apt
btmp
ceph-common
chrony
corosync
dpkg
frr
pve
pve-firewall
wtmp
wtmpdb
### kernels installed
linux-image-7.0.14-20-pve-amd64 un
linux-image-7.0.14-22-pve-amd64 un
linux-image-7.0.14-23-pve-amd64 un
linux-image-7.0.14-24-pve-amd64 un
linux-image-7.0.14-25-pve-amd64 un
linux-image-7.0.14-26-pve-amd64 un
linux-image-7.0.14-28-pve-amd64 un
linux-image-7.0.2-6-pve-amd64 un
linux-image-amd64 un
linux-image-generic un
proxmox-kernel-7.0 7.0.14-28 37 ii
proxmox-kernel-7.0.14-20-pve un
proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20 1006525 ii
proxmox-kernel-7.0.14-22-pve un
proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22 1006522 ii
proxmox-kernel-7.0.14-23-pve un
proxmox-kernel-7.0.14-23-pve-signed 7.0.14-23 1006525 ii
proxmox-kernel-7.0.14-24-pve un
proxmox-kernel-7.0.14-24-pve-signed 7.0.14-24 1006528 ii
proxmox-kernel-7.0.14-25-pve un
proxmox-kernel-7.0.14-25-pve-signed 7.0.14-25 1007819 ii
proxmox-kernel-7.0.14-26-pve un
proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26 1007815 ii
proxmox-kernel-7.0.14-28-pve un
proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28 1007817 ii
proxmox-kernel-7.0.2-6-pve un
proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6 1005219 ii
proxmox-kernel-helper 9.1.0+fde2 109 ii
pve-kernel-7.0 un
pve-kernel-helper un
7.0.14-28-pve
### proxmox-boot-tool
Manually selected kernels:
None.
Automatically selected kernels:
7.0.14-26-pve
7.0.14-28-pve
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
E: /etc/kernel/proxmox-boot-uuids does not exist.
### /boot
Filesystem 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 41424257024 23968075776 15318974464 62% /
/dev/sda2 1071624192 9469952 1062154240 1% /boot/efi
823369728 /boot
16437248 /boot/grub
4096 /boot/pve
77353337 Oct 4 09:32 initrd.img-7.0.2-6-pve
73547547 Nov 10 02:50 initrd.img-7.0.14-28-pve
73492957 Oct 29 05:07 initrd.img-7.0.14-26-pve
73286572 Oct 21 04:45 initrd.img-7.0.14-25-pve
73131609 Oct 11 04:42 initrd.img-7.0.14-23-pve
73115537 Oct 16 04:49 initrd.img-7.0.14-24-pve
73101983 Oct 10 04:42 initrd.img-7.0.14-22-pve
73094554 Oct 10 04:41 initrd.img-7.0.14-20-pve
16288872 Oct 9 13:32 vmlinuz-7.0.14-24-pve
16288872 Oct 7 22:25 vmlinuz-7.0.14-23-pve
16288872 Oct 6 07:41 vmlinuz-7.0.14-22-pve
16284776 Sep 24 12:43 vmlinuz-7.0.14-20-pve
16284776 Oct 17 22:42 vmlinuz-7.0.14-25-pve
16280680 Oct 22 20:45 vmlinuz-7.0.14-26-pve
16280680 Nov 2 12:54 vmlinuz-7.0.14-28-pve
16227432 May 20 10:55 vmlinuz-7.0.2-6-pve
10483512 Sep 24 12:43 System.map-7.0.14-20-pve
10483455 Oct 9 13:32 System.map-7.0.14-24-pve
10483429 Oct 7 22:25 System.map-7.0.14-23-pve
10483429 Oct 6 07:41 System.map-7.0.14-22-pve
10452992 May 20 10:55 System.map-7.0.2-6-pve
10341916 Nov 2 12:54 System.map-7.0.14-28-pve
10341401 Oct 22 20:45 System.map-7.0.14-26-pve
10341401 Oct 17 22:42 System.map-7.0.14-25-pve
308500 Oct 22 20:45 config-7.0.14-26-pve
308500 Oct 17 22:42 config-7.0.14-25-pve
308500 Nov 2 12:54 config-7.0.14-28-pve
308363 Sep 24 12:43 config-7.0.14-20-pve
308363 Oct 9 13:32 config-7.0.14-24-pve
308363 Oct 7 22:25 config-7.0.14-23-pve
### /lib/modules
8148783104 /lib/modules
1019707392 /lib/modules/7.0.14-28-pve
1019707392 /lib/modules/7.0.14-26-pve
1019707392 /lib/modules/7.0.14-25-pve
1018175488 /lib/modules/7.0.14-24-pve
1018171392 /lib/modules/7.0.14-23-pve
1018171392 /lib/modules/7.0.14-22-pve
1018171392 /lib/modules/7.0.14-20-pve
1016967168 /lib/modules/7.0.2-6-pve
### apt
92004352 /var/cache/apt
61833216 /var/lib/apt/lists
0
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve-signed$";
01autoremove
10pveapthook
20listchanges
70debconf
76pveconf
76pveproxy
99mmdebstrap
### apt autoremove candidates (dry read)
1
### tmp
0 /tmp
28672 /var/tmp
2026-10-14 1266 /var/tmp/guest_sampler.sh
2026-10-14 1340 /var/tmp/host_sampler.sh
2026-11-10 40 /tmp/.ICE-unix
2026-11-10 40 /tmp/.X11-unix
2026-11-10 40 /tmp/.XIM-unix
2026-11-10 60 /tmp/systemd-private-b6e0edea2b5647dfa36f5bb3825adc3a-systemd-logind.service-AqauM3
10
### systemd-tmpfiles rules touching tmp
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
### /var/lib/vz
9880125440 /var/lib/vz
9750130688 /var/lib/vz/dump
129974272 /var/lib/vz/template
129966080 /var/lib/vz/template/cache
8192 /var/lib/vz/snippets
4096 /var/lib/vz/template/iso
4096 /var/lib/vz/import
4096 /var/lib/vz/images
-- /var/lib/vz/dump
654115664 2026-09-13 felhom-golden-0.236.0.tar.zst
1874 2026-09-24 vzdump-lxc-9201-2026_09_24-06_59_28.log
1874 2026-09-24 vzdump-lxc-9201-2026_09_24-07_17_53.log
1874 2026-09-24 vzdump-lxc-9201-2026_09_24-07_42_54.log
187 2026-09-24 vzdump-lxc-9201-2026_09_24-08_22_55.log
1780 2026-09-24 vzdump-lxc-9201-2026_09_24-22_40_21.log
1563 2026-10-14 vzdump-lxc-9201-2026_10_14-07_17_45.log
1942 2026-10-14 vzdump-lxc-9201-2026_10_14-11_02_50.log
994 2026-10-14 vzdump-lxc-9201-2026_10_14-13_42_48.log
2234 2026-10-14 vzdump-lxc-9202-2026_10_14-11_01_36.log
1495 2026-11-10 vzdump-lxc-9201-2026_11_10-02_35_34.log
9095951125 2026-11-10 vzdump-lxc-9201-2026_11_10-02_35_34.tar.zst
16 2026-11-10 vzdump-lxc-9201-2026_11_10-02_35_34.tar.zst.notes
-- /var/lib/vz/template/cache
129954319 2026-07-14 debian-13-standard_13.6-1_amd64.tar.zst
-- /var/lib/vz/template/iso
### felhom dirs (candidates)
/etc/felhom
/etc/felhom-agent
/etc/felhom-oob.nft
/etc/felhom-sshd
/run/felhom-sshd.pid
/var/lib/felhom-agent
/var/lib/felhom-crash-guard
/var/lib/felhom-install
/var/lib/felhom-kernel
/var/lib/felhom-os-apply
/var/log/felhom-bootstrap-postinst.log
-- /etc/felhom
24576 /etc/felhom
files=6 oldest=2026-08-21 newest=2026-11-07
-- /etc/felhom-agent
24576 /etc/felhom-agent
files=5 oldest=2026-08-21 newest=2026-09-28
-- /etc/felhom-oob.nft
4096 /etc/felhom-oob.nft
files=1 oldest=2026-08-21 newest=2026-08-21
-- /etc/felhom-sshd
24576 /etc/felhom-sshd
8192 /etc/felhom-sshd/authorized_keys
files=4 oldest=2026-08-21 newest=2026-10-05
-- /var/lib/felhom-agent
740171776 /var/lib/felhom-agent
739938304 /var/lib/felhom-agent/selfupdate
32768 /var/lib/felhom-agent/os
32768 /var/lib/felhom-agent/guests
20480 /var/lib/felhom-agent/wg
16384 /var/lib/felhom-agent/guests/9201
16384 /var/lib/felhom-agent/felhom-sshd
12288 /var/lib/felhom-agent/units
12288 /var/lib/felhom-agent/pbsdr
12288 /var/lib/felhom-agent/guests/9202
4096 /var/lib/felhom-agent/escrow-stage
files=86 oldest=2026-08-21 newest=2026-11-10
-- /var/lib/felhom-crash-guard
8192 /var/lib/felhom-crash-guard
files=1 oldest=2026-11-10 newest=2026-11-10
-- /var/lib/felhom-install
8192 /var/lib/felhom-install
files=1 oldest=2026-10-18 newest=2026-10-18
-- /var/lib/felhom-kernel
12288 /var/lib/felhom-kernel
files=1 oldest=2026-11-10 newest=2026-11-10
-- /var/lib/felhom-os-apply
376832 /var/lib/felhom-os-apply
364544 /var/lib/felhom-os-apply/bundle-prev
159744 /var/lib/felhom-os-apply/bundle-prev/20261107T222713Z-before-0.172.1
159744 /var/lib/felhom-os-apply/bundle-prev/20261026T080307Z-before-0.166.0
40960 /var/lib/felhom-os-apply/bundle-prev/20261029T161106Z-before-0.170.0
4096 /var/lib/felhom-os-apply/agent-update
files=4 oldest=2026-10-26 newest=2026-11-07
-- /var/log/felhom-bootstrap-postinst.log
4096 /var/log/felhom-bootstrap-postinst.log
files=1 oldest=2026-08-21 newest=2026-08-21
### felhom binaries + slots
4096 Nov 7 23:11 .
4096 May 21 13:31 ..
16393888 Nov 7 23:11 felhom-agent
14284239 Oct 4 11:02 felhom-agent.bak-0.139.0
16393888 Nov 3 08:56 felhom-agent.prev
4096 Nov 7 23:27 .
4096 May 21 13:31 ..
4096 Oct 5 07:44 __pycache__
6654 Oct 4 20:34 felhom-backup-target-apply
27518 Jul 31 19:03 felhom-bootstrap.sh
8687 Oct 4 16:08 felhom-crash-guard
6816 Oct 16 04:41 felhom-internal-place
3393 Aug 21 17:59 felhom-mgmt-watchdog
5439 Oct 16 04:41 felhom-mkfs-guarded
142720 Nov 7 23:27 felhom-os-apply
50933 Oct 4 18:00 felhom-os-apply.pre-bundle
9087 Aug 21 17:59 felhom-pbs-apply
26062 Oct 17 16:08 felhom-priv-apply
7806 Oct 5 12:57 felhom-selfupdate-guarded
943 Aug 21 17:59 felhom-shared-parent.sh
3844 Oct 17 16:08 felhom-thin-check
5537 Oct 14 13:30 felhom-vzdump-partial-clean
### root filesystem, top 70 dirs to depth 4
23965904896 /
11856850944 /var
11276845056 /usr
10793242624 /var/lib
9880125440 /var/lib/vz
9750130688 /var/lib/vz/dump
9661677568 /usr/lib
8148783104 /usr/lib/modules
1165066240 /usr/share
1019707392 /usr/lib/modules/7.0.14-28-pve
1019707392 /usr/lib/modules/7.0.14-26-pve
1019707392 /usr/lib/modules/7.0.14-25-pve
1018175488 /usr/lib/modules/7.0.14-24-pve
1018171392 /usr/lib/modules/7.0.14-23-pve
1018171392 /usr/lib/modules/7.0.14-22-pve
1018171392 /usr/lib/modules/7.0.14-20-pve
1016967168 /usr/lib/modules/7.0.2-6-pve
888266752 /var/log
873717760 /var/log/journal
873709568 /var/log/journal/9756a908fef046fda6a56428d3fefe3c
823369728 /boot
740171776 /var/lib/felhom-agent
739938304 /var/lib/felhom-agent/selfupdate
596193280 /usr/lib/x86_64-linux-gnu
561770496 /usr/lib/firmware
365461504 /usr/share/pve-edk2-firmware
340119552 /usr/bin
327876608 /usr/share/kvm
171753472 /var/cache
160776192 /usr/lib/firmware/nvidia
129974272 /var/lib/vz/template
118800384 /usr/share/locale
113082368 /usr/lib/firmware/amdgpu
92004352 /var/cache/apt
79261696 /usr/share/doc
73621504 /usr/lib/ceph
65826816 /usr/lib/x86_64-linux-gnu/proxmox-backup
64802816 /var/lib/rrdcached
64794624 /var/lib/rrdcached/db
62689280 /var/lib/apt
61833216 /var/lib/apt/lists
58044416 /var/cache/proxmox-backup
56102912 /usr/lib/python3
56098816 /usr/lib/python3/dist-packages
51273728 /usr/lib/ceph/denc
49410048 /usr/lib/x86_64-linux-gnu/perl5
47620096 /usr/local
47083520 /usr/local/bin
46153728 /usr/lib/firmware/intel
45576192 /usr/lib/firmware/ath11k
43646976 /usr/sbin
38559744 /usr/share/pve-docs
36179968 /var/lib/dpkg
34566144 /var/lib/dpkg/info
34213888 /usr/lib/firmware/ath12k
28688384 /usr/lib/python3.13
27885568 /usr/lib/firmware/mediatek
27856896 /usr/share/javascript
26943488 /usr/lib/x86_64-linux-gnu/perl
26587136 /usr/lib/grub
25767936 /usr/share/man
25546752 /usr/share/javascript/extjs
23773184 /usr/share/perl
23769088 /usr/share/perl/5.40.1
23724032 /usr/share/nmap
23490560 /usr/lib/udev
22081536 /usr/share/doc/pve-qemu-kvm
18653184 /usr/libexec
17694720 /usr/lib/frr
17686528 /usr/lib/podman
### root fs: file count per top dir
5053 /var
93482 /usr
20 /root
4 /home
0 /opt
1064 /etc
0 /srv
0 /tmp
### root home
176128 /root
16384 /root/.ssh
12288 /root/.cache
### other mounted filesystems, depth 2
-- /mnt/hdd_1
322732228608 /mnt/hdd_1
315983560704 /mnt/hdd_1/images
211935100928 /mnt/hdd_1/images/341
64135630848 /mnt/hdd_1/images/9202
39912824832 /mnt/hdd_1/images/9401
5099683840 /mnt/hdd_1/dump
1536958464 /mnt/hdd_1/felhom-data
1291870208 /mnt/hdd_1/felhom-data/backups
208642048 /mnt/hdd_1/felhom-data/kept
111992832 /mnt/hdd_1/scratch-drives
111988736 /mnt/hdd_1/scratch-drives/scratch_hdd
30277632 /mnt/hdd_1/felhom-data/appdata
6164480 /mnt/hdd_1/felhom-data/userdata
16384 /mnt/hdd_1/lost+found
8192 /mnt/hdd_1/e2d-images
4096 /mnt/hdd_1/private
4096 /mnt/hdd_1/e2d-images/images
### pve task logs + rrd
4173824 /var/log/pve
64802816 /var/lib/rrdcached
4222976 /var/lib/pve-cluster
106496 /var/lib/pve-manager
914
2026-08-21
2026-11-10
### core dumps
4096 /var/lib/systemd/coredump
0
### swap
NAME TYPE SIZE USED PRIO
/dev/dm-0 partition 8589930496 0 -1
### timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2026-11-10 08:26:02 CET 21s Tue 2026-11-10 08:25:02 CET 38s ago felhom-mgmt-watchdog.timer felhom-mgmt-watchdog.service
Tue 2026-11-10 09:09:09 CET 43min Tue 2026-11-10 08:09:09 CET 16min ago felhom-crash-guard-check.timer felhom-crash-guard-check.service
Tue 2026-11-10 15:51:50 CET 7h Tue 2026-11-10 04:06:22 CET 4h 19min ago apt-daily.timer apt-daily.service
Wed 2026-11-11 00:00:00 CET 15h Tue 2026-11-10 00:00:01 CET - dpkg-db-backup.timer dpkg-db-backup.service
Wed 2026-11-11 00:52:38 CET 16h Tue 2026-11-10 00:27:01 CET - logrotate.timer logrotate.service
Wed 2026-11-11 03:09:02 CET 18h Tue 2026-11-10 03:09:02 CET 5h 16min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Wed 2026-11-11 04:54:05 CET 20h Tue 2026-11-10 03:05:09 CET 5h 20min ago pve-daily-update.timer pve-daily-update.service
Wed 2026-11-11 05:44:16 CET 21h Tue 2026-11-10 00:25:01 CET - man-db.timer man-db.service
Wed 2026-11-11 06:38:33 CET 22h Tue 2026-11-10 06:55:09 CET 1h 30min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Sun 2026-11-15 03:10:36 CET 4 days Sun 2026-11-08 03:10:49 CET - xfs_scrub_all.timer xfs_scrub_all.service
Sun 2026-11-15 03:10:36 CET 4 days Sun 2026-11-08 03:10:49 CET - e2scrub_all.timer e2scrub_all.service
Mon 2026-11-16 00:34:32 CET 5 days Mon 2026-11-09 01:25:39 CET - fstrim.timer fstrim.service
12 timers listed.
### end
2026-11-10T07:25:41Z
@@ -0,0 +1,372 @@
### identity
demo-felhom
2026-11-10T07:26:21Z
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
### df bytes
Filesystem Type 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-vm--9201--disk--2 ext4 33501757440 1188171776 30578823168 4% /
/dev/mapper/pve-vm--9201--disk--3 ext4 263086084096 2469072896 247178461184 1% /var/lib/felhom
/dev/mapper/pve-root ext4 100861726720 38093578240 57597411328 40% /mnt/felhom-drives
/dev/sdb ext4 983351140352 8143593472 925180526592 1% /mnt/felhom-drives/hdd_1
### df inodes
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pve-vm--9201--disk--2 2097152 24736 2072416 2% /
/dev/mapper/pve-vm--9201--disk--3 16384000 39913 16344087 1% /var/lib/felhom
/dev/mapper/pve-root 6291456 120797 6170659 2% /mnt/felhom-drives
/dev/sdb 61054976 731 61054245 1% /mnt/felhom-drives/hdd_1
### docker info
driver=overlay2 root=/var/lib/docker logdriver=json-file images=16 containers=9 running=9 server=29.9.3
### daemon.json
{
"features": {
"containerd-snapshotter": false
},
"live-restore": true,
"log-driver": "json-file",
"log-opts": {
"max-file": "3",
"max-size": "10m"
}
}
total 12
drwxr-xr-x 2 root root 4096 2026-08-03 04:56 .
drwxr-xr-x 73 root root 4096 2026-11-10 07:13 ..
-rw-r--r-- 1 root root 177 2026-10-04 14:13 daemon.json
### docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 16 9 2.183GB 555.2MB (25%)
Containers 9 9 597.2kB 0B (0%)
Local Volumes 8 7 122.9MB 0B (0%)
Build Cache 0 0 0B 0B
### containers (name image state logdriver logopts)
/cloudflared cloudflare/cloudflared:2026.10.4 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-10-27T15:41:36.214523712Z
/felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.351.0 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-09T12:14:03.719130271Z
/filebrowser gtstef/filebrowser:1.5.8-stable running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-10-29T03:48:47.349042768Z
/opengist ghcr.io/thomiceli/opengist:1.15@sha256:7edc91273ee7d10a17406da8a08c803158baaff984c39b789c1313473d068f21 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T04:39:35.815075613Z
/prowlarr lscr.io/linuxserver/prowlarr:version-2.6.5.5623@sha256:f9151e5bc1025c6d0a630d503210cdcb6bb55a7cc098562609d96a408d838902 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T04:39:36.457250981Z
/qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1@sha256:92bfd78d731e254ba64f62b77b0696add6447a33642ee5feda106a26b285aa7f running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T04:39:36.934982736Z
/radarr lscr.io/linuxserver/radarr:6.4.4@sha256:adb6c09d6b729ea5e642c99cea35af72702ef476bf4763f153299ac5db9f0b4f running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T04:39:37.534816564Z
/sonarr lscr.io/linuxserver/sonarr:4.0.20@sha256:f247545d23ba8b233d6604575347e48a623fe6ad75dda02348bf81917f3b5c06 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T04:39:38.103004033Z
/traefik traefik:v3.7.17 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-03T07:04:16.118212587Z
### container log files (bytes path) and total
5598390 2026-11-10 /var/lib/docker/containers/23e70794a73f747ca9fa386de6db5b067c1dd1a1154d0b95a3008fc43cc89543/23e70794a73f747ca9fa386de6db5b067c1dd1a1154d0b95a3008fc43cc89543-json.log
3049245 2026-11-10 /var/lib/docker/containers/2d129ef6038009dbdc9cd609d923c1518e00915079b64174446fa699010bdad5/2d129ef6038009dbdc9cd609d923c1518e00915079b64174446fa699010bdad5-json.log
862215 2026-11-10 /var/lib/docker/containers/f5a34ac4ca039685708b1c0f2aeb77fbc07b36a66d39d610741c535bd0a0c33c/f5a34ac4ca039685708b1c0f2aeb77fbc07b36a66d39d610741c535bd0a0c33c-json.log
107002 2026-11-10 /var/lib/docker/containers/372fb8861b8465781dd368079d5e49753e4fe9435f062efa363bf5dd0f60e493/372fb8861b8465781dd368079d5e49753e4fe9435f062efa363bf5dd0f60e493-json.log
14504 2026-11-10 /var/lib/docker/containers/0ae65ebe58bbb0b820694835986645296d2d5c54c835f3b302fab8b2ccc8b6ca/0ae65ebe58bbb0b820694835986645296d2d5c54c835f3b302fab8b2ccc8b6ca-json.log
11323 2026-11-10 /var/lib/docker/containers/e0ad39296075caeb8c9057b26767fc77b2b8cb802ad75e376205e5a2e780cd92/e0ad39296075caeb8c9057b26767fc77b2b8cb802ad75e376205e5a2e780cd92-json.log
7999 2026-11-10 /var/lib/docker/containers/e8be61b6ba6a0131ad107e12efbcf09bc56bb9461427b283e8aa6402d5a6e7f8/e8be61b6ba6a0131ad107e12efbcf09bc56bb9461427b283e8aa6402d5a6e7f8-json.log
451 2026-11-10 /var/lib/docker/containers/fe22e143bf13dcea2205d56e3080de04675594ccc2e3fd72ad130407c89b8892/fe22e143bf13dcea2205d56e3080de04675594ccc2e3fd72ad130407c89b8892-json.log
333 2026-11-10 /var/lib/docker/containers/d88482fad678b1b5188f6f56ec0849b8c4628f1e64862b3fca892e401c224465/d88482fad678b1b5188f6f56ec0849b8c4628f1e64862b3fca892e401c224465-json.log
TOTAL bytes=9651462 files=9
### container writable layers (docker ps -as)
cloudflared 0B (virtual 64.6MB)
felhom-controller 298kB (virtual 349MB)
filebrowser 219kB (virtual 195MB)
opengist 9.57kB (virtual 94.4MB)
prowlarr 23.4kB (virtual 185MB)
qbittorrent 0B (virtual 172MB)
radarr 23.4kB (virtual 211MB)
sonarr 23.4kB (virtual 203MB)
traefik 0B (virtual 191MB)
### images
alpine:latest d529dd0c6e55 2026-06-16 00:01:29 +0000 UTC 8.42MB
cloudflare/cloudflared:2026.10.3 1475624a6d23 2026-10-21 13:26:38 +0000 UTC 64.6MB
cloudflare/cloudflared:2026.10.4 472341c39b02 2026-10-26 11:27:24 +0000 UTC 64.6MB
ghcr.io/thomiceli/opengist:<none> 0599ba5a0357 2026-08-30 02:07:48 +0000 UTC 94.4MB
gitea.dooplex.hu/admin/felhom-controller:0.350.0 1f32a260789a 2026-11-03 07:04:04 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.351.0 ed7f096c3ff7 2026-11-09 12:14:14 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-samba:1.1.0 8e21163a1646 2026-07-20 11:36:16 +0000 UTC 103MB
gtstef/filebrowser:1.5.6-stable 96e26edeefc3 2026-09-04 21:33:58 +0000 UTC 195MB
gtstef/filebrowser:1.5.8-stable 0019084d4a6f 2026-10-04 21:15:40 +0000 UTC 195MB
hello-world:latest e2ac70e7319a 2026-03-23 21:33:59 +0000 UTC 10.1kB
lscr.io/linuxserver/prowlarr:<none> cc10ec0691c7 2026-10-07 10:43:17 +0000 UTC 185MB
lscr.io/linuxserver/radarr:<none> 17292112a802 2026-09-20 19:11:07 +0000 UTC 211MB
lscr.io/linuxserver/sonarr:<none> 1a11b87590c6 2026-09-26 01:46:53 +0000 UTC 203MB
qbittorrentofficial/qbittorrent-nox:<none> dd9d91f629c2 2026-09-28 20:14:56 +0000 UTC 172MB
traefik:v3.7.16 3733df53e74c 2026-10-21 22:55:46 +0000 UTC 194MB
traefik:v3.7.17 a72cb2f4a210 2026-10-30 20:17:26 +0000 UTC 191MB
### images count / dangling
11
5
### images in use by a container
1 cloudflare/cloudflared:2026.10.4
1 ghcr.io/thomiceli/opengist:1.15
1 gitea.dooplex.hu/admin/felhom-controller:0.351.0
1 gtstef/filebrowser:1.5.8-stable
1 lscr.io/linuxserver/prowlarr:version-2.6.5.5623
1 lscr.io/linuxserver/radarr:6.4.4
1 lscr.io/linuxserver/sonarr:4.0.20
1 qbittorrentofficial/qbittorrent-nox:5.2.4-1
1 traefik:v3.7.17
### volumes (count, anonymous count, dangling count)
8
1
1
### volumes by size (system df -v, Local Volumes block)
Local Volumes space usage:
VOLUME NAME LINKS SIZE
prowlarr_prowlarr_config 1 7.175MB
qbittorrent_qbittorrent_config 1 10.35MB
radarr_radarr_config 1 8.607MB
sonarr_sonarr_config 1 22.81MB
b8a3eb602356720f6ff7b34b1a1c46d9964e031d26b0f7313ec1b6812b28b562 0 0B
felhom-controller-data 1 73.65MB
filebrowser_filebrowser_data 1 135.6kB
opengist_opengist_data 1 172.8kB
### build cache
Build cache usage: 0B
CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED
### docker root du depth 1
2419159040 /var/lib/docker
2258448384 /var/lib/docker/overlay2
135462912 /var/lib/docker/volumes
14815232 /var/lib/docker/image
10063872 /var/lib/docker/containers
167936 /var/lib/docker/buildkit
163840 /var/lib/docker/network
12288 /var/lib/docker/plugins
8192 /var/lib/docker/tmp
4096 /var/lib/docker/swarm
4096 /var/lib/docker/runtimes
### journald
Archived and active journals take up 8M in the file system.
[Journal]
Nov 08 04:35:34 demo-felhom systemd-journald[46]: Runtime Journal (/run/log/journal/37964053e2684573b3ef4190cd3c3644) is 8M, max 315.4M, 307.4M free.
Nov 08 04:35:34 demo-felhom systemd-journald[46]: Runtime Journal (/run/log/journal/37964053e2684573b3ef4190cd3c3644) is 8M, max 315.4M, 307.4M free.
### /var/log
135168 /var/log
49152 /var/log/apt
4096 /var/log/private
### apt
95715328 /var/cache/apt
96104448 /var/lib/apt/lists
0
### tmp
0 /tmp
20480 /var/tmp
2026-11-08 40 /tmp/.ICE-unix
2026-11-08 40 /tmp/.X11-unix
2026-11-08 40 /tmp/.XIM-unix
2026-11-08 40 /tmp/.font-unix
2026-11-08 4096 /var/tmp/network
2026-11-08 4096 /var/tmp/systemd-private-84664f34e28e46b5ac5b6370374b6a84-systemd-logind.service-QWfar9
2026-11-08 60 /tmp/systemd-private-84664f34e28e46b5ac5b6370374b6a84-systemd-logind.service-cjq9uj
entries=7
### felhom dirs
/etc/felhom-bootstrap
/etc/felhom-controller-image
/opt/docker
/var/lib/felhom
### root filesystem top 45 to depth 4
1185918976 /
913002496 /usr
336863232 /usr/bin
266797056 /var
266072064 /usr/lib
182603776 /usr/share
165662720 /var/lib
154345472 /usr/lib/x86_64-linux-gnu
110944256 /usr/libexec
110768128 /usr/libexec/docker
110018560 /usr/libexec/docker/cli-plugins
99495936 /var/cache
96387072 /var/lib/apt
96104448 /var/lib/apt/lists
95715328 /var/cache/apt
67096576 /var/lib/dpkg
66502656 /usr/share/locale
30691328 /usr/share/doc
28696576 /usr/lib/python3.13
26927104 /usr/lib/x86_64-linux-gnu/perl
25632768 /usr/lib/git-core
23740416 /usr/share/perl
23736320 /usr/share/perl/5.40.1
23347200 /usr/lib/udev
17846272 /usr/share/man
16797696 /usr/share/i18n
16412672 /usr/sbin
13242368 /usr/share/i18n/locales
10358784 /usr/lib/file
9621504 /var/lib/dpkg/info
9428992 /usr/lib/udev/hwdb.d
9031680 /usr/lib/python3
9027584 /usr/lib/python3/dist-packages
8577024 /usr/lib/x86_64-linux-gnu/gconv
7770112 /usr/share/terminfo
6914048 /usr/lib/x86_64-linux-gnu/systemd
6012928 /usr/lib/systemd
5689344 /usr/lib/x86_64-linux-gnu/perl-base
5439488 /usr/lib/python3.13/__pycache__
5259264 /usr/share/doc/git
5152768 /usr/share/man/man1
4128768 /etc
3661824 /usr/lib/openssh
3620864 /usr/lib/python3.13/lib-dynload
3538944 /usr/share/i18n/charmaps
sort: write failed: 'standard output': Broken pipe
sort: write error
### /var/lib/felhom depth 3 (data volume)
2466926592 /var/lib/felhom
2419159040 /var/lib/felhom/docker
2258448384 /var/lib/felhom/docker/overlay2
135462912 /var/lib/felhom/docker/volumes
47747072 /var/lib/felhom/sys_drive
47742976 /var/lib/felhom/sys_drive/felhom-data
47730688 /var/lib/felhom/sys_drive/felhom-data/backups
14815232 /var/lib/felhom/docker/image
10063872 /var/lib/felhom/docker/containers
167936 /var/lib/felhom/docker/buildkit
163840 /var/lib/felhom/docker/network
159744 /var/lib/felhom/docker/network/files
16384 /var/lib/felhom/lost+found
12288 /var/lib/felhom/docker/plugins
8192 /var/lib/felhom/sys_drive/felhom-data/userdata
8192 /var/lib/felhom/docker/tmp
4096 /var/lib/felhom/docker/tmp/moby-detect-rro201841214
4096 /var/lib/felhom/docker/swarm
4096 /var/lib/felhom/docker/runtimes
4096 /var/lib/felhom/docker/plugins/tmp
4096 /var/lib/felhom/docker/plugins/storage
### /opt/docker depth 2
1863680 /opt/docker
1859584 /opt/docker/stacks
73728 /opt/docker/stacks/traefik
57344 /opt/docker/stacks/qbittorrent
45056 /opt/docker/stacks/sonarr
45056 /opt/docker/stacks/radarr
40960 /opt/docker/stacks/prowlarr
40960 /opt/docker/stacks/immich
36864 /opt/docker/stacks/sabnzbd
36864 /opt/docker/stacks/romm
36864 /opt/docker/stacks/opengist
32768 /opt/docker/stacks/wger
32768 /opt/docker/stacks/paperless-ngx
32768 /opt/docker/stacks/nextcloud
32768 /opt/docker/stacks/grocy
28672 /opt/docker/stacks/wishlist
28672 /opt/docker/stacks/wanderer
28672 /opt/docker/stacks/transmission
28672 /opt/docker/stacks/sparkyfitness
28672 /opt/docker/stacks/nzbget
28672 /opt/docker/stacks/lubelogger
28672 /opt/docker/stacks/karakeep
28672 /opt/docker/stacks/dawarich
28672 /opt/docker/stacks/adventurelog
24576 /opt/docker/stacks/zipline
24576 /opt/docker/stacks/vikunja
24576 /opt/docker/stacks/vaultwarden
24576 /opt/docker/stacks/uptime-kuma
24576 /opt/docker/stacks/termix
24576 /opt/docker/stacks/tandoor
24576 /opt/docker/stacks/recipe-importer
24576 /opt/docker/stacks/rallly
24576 /opt/docker/stacks/radicale
24576 /opt/docker/stacks/plex
24576 /opt/docker/stacks/papra
24576 /opt/docker/stacks/outline
24576 /opt/docker/stacks/navidrome
24576 /opt/docker/stacks/n8n
24576 /opt/docker/stacks/metube
24576 /opt/docker/stacks/mealie
24576 /opt/docker/stacks/komga
24576 /opt/docker/stacks/kimai
24576 /opt/docker/stacks/jellyfin
24576 /opt/docker/stacks/homebox
24576 /opt/docker/stacks/home-assistant
24576 /opt/docker/stacks/grimmory
24576 /opt/docker/stacks/grafana
24576 /opt/docker/stacks/gitea
24576 /opt/docker/stacks/ghost
24576 /opt/docker/stacks/docmost
### drives under /mnt/felhom-drives (depth 3 from each drive)
-- /mnt/felhom-drives/hdd_1
/dev/sdb 983351140352 8143593472 925180526592 1% /mnt/felhom-drives/hdd_1
3782635520 /mnt/felhom-drives/hdd_1
3339952128 /mnt/felhom-drives/hdd_1/backups
1654599680 /mnt/felhom-drives/hdd_1/backups/offsite-restore
1491480576 /mnt/felhom-drives/hdd_1/backups/offsite-restore/immich
1391616000 /mnt/felhom-drives/hdd_1/backups/primary
1350868992 /mnt/felhom-drives/hdd_1/backups/primary/immich
442085376 /mnt/felhom-drives/hdd_1/appdata
376659968 /mnt/felhom-drives/hdd_1/appdata/immich
293728256 /mnt/felhom-drives/hdd_1/backups/secondary
163590144 /mnt/felhom-drives/hdd_1/backups/secondary/bookstack
163115008 /mnt/felhom-drives/hdd_1/backups/offsite-restore/bookstack
123023360 /mnt/felhom-drives/hdd_1/backups/secondary/docmost
65384448 /mnt/felhom-drives/hdd_1/appdata/nextcloud
22638592 /mnt/felhom-drives/hdd_1/backups/primary/sonarr
10412032 /mnt/felhom-drives/hdd_1/backups/primary/qbittorrent
7294976 /mnt/felhom-drives/hdd_1/backups/primary/radarr
6881280 /mnt/felhom-drives/hdd_1/backups/secondary/prowlarr
561152 /mnt/felhom-drives/hdd_1/userdata
528384 /mnt/felhom-drives/hdd_1/userdata/media
397312 /mnt/felhom-drives/hdd_1/backups/primary/calibre-web
229376 /mnt/felhom-drives/hdd_1/backups/secondary/opengist
32768 /mnt/felhom-drives/hdd_1/shares
28672 /mnt/felhom-drives/hdd_1/shares/Share
24576 /mnt/felhom-drives/hdd_1/appdata/paperless
12288 /mnt/felhom-drives/hdd_1/userdata/import
12288 /mnt/felhom-drives/hdd_1/appdata/romm
4096 /mnt/felhom-drives/hdd_1/userdata/roms-bios
4096 /mnt/felhom-drives/hdd_1/userdata/roms
4096 /mnt/felhom-drives/hdd_1/userdata/downloads
4096 /mnt/felhom-drives/hdd_1/userdata/documents
4096 /mnt/felhom-drives/hdd_1/backups/offsite-proof
### timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2026-11-11 00:00:00 UTC 16h Tue 2026-11-10 00:00:14 UTC 7h ago dpkg-db-backup.timer dpkg-db-backup.service
Wed 2026-11-11 00:59:43 UTC 17h Tue 2026-11-10 00:54:14 UTC 6h ago logrotate.timer logrotate.service
Wed 2026-11-11 01:27:36 UTC 18h Tue 2026-11-10 06:49:16 UTC 37min ago apt-daily.timer apt-daily.service
Wed 2026-11-11 04:51:34 UTC 21h Tue 2026-11-10 04:51:34 UTC 2h 34min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Wed 2026-11-11 06:42:59 UTC 23h Tue 2026-11-10 06:32:16 UTC 54min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Wed 2026-11-11 09:21:08 UTC 1 day 1h Tue 2026-11-10 03:33:16 UTC 3h 53min ago man-db.timer man-db.service
Sun 2026-11-15 03:10:29 UTC 4 days Sun 2026-11-08 03:10:56 UTC - e2scrub_all.timer e2scrub_all.service
- - - - fstrim.timer fstrim.service
8 timers listed.
### cron
/etc/cron.d:
e2scrub_all
/etc/cron.daily:
apt-compat
dpkg
logrotate
man-db
/etc/cron.weekly:
man-db
### end
2026-11-10T07:26:23Z
@@ -0,0 +1,534 @@
### identity
demo-felhom
2026-11-10T07:33:04Z
### in flight
(end in flight)
### agent state dir (depth 2, bytes)
724164608 /var/lib/felhom-agent
723935232 /var/lib/felhom-agent/selfupdate
24576 /var/lib/felhom-agent/os
20480 /var/lib/felhom-agent/wg
16384 /var/lib/felhom-agent/guests
16384 /var/lib/felhom-agent/felhom-sshd
12288 /var/lib/felhom-agent/pbsdr
12288 /var/lib/felhom-agent/guests/9201
8192 /var/lib/felhom-agent/escrow-stage
### selfupdate listing
total 706972
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-07 23:05 .
drwxr-x--- 9 felhom-agent felhom-agent 4096 2026-11-10 06:07 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 14190233 2026-09-16 11:04 felhom-agent-0.131.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14195690 2026-09-17 10:34 felhom-agent-0.132.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14262881 2026-09-24 21:34 felhom-agent-0.133.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14271901 2026-09-24 22:51 felhom-agent-0.134.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14273488 2026-09-27 12:21 felhom-agent-0.135.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14274024 2026-09-27 13:51 felhom-agent-0.136.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14273888 2026-09-27 14:07 felhom-agent-0.137.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14278858 2026-09-30 10:37 felhom-agent-0.138.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14284239 2026-10-04 09:07 felhom-agent-0.139.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14355058 2026-10-04 11:37 felhom-agent-0.140.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14376457 2026-10-04 13:22 felhom-agent-0.141.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14376105 2026-10-04 13:52 felhom-agent-0.141.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14427309 2026-10-04 16:22 felhom-agent-0.142.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14427261 2026-10-04 18:22 felhom-agent-0.142.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14461055 2026-10-04 19:38 felhom-agent-0.143.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14493513 2026-10-05 07:23 felhom-agent-0.144.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14494141 2026-10-05 07:53 felhom-agent-0.144.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14494291 2026-10-05 09:38 felhom-agent-0.145.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14504200 2026-10-05 12:38 felhom-agent-0.146.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14504898 2026-10-05 19:09 felhom-agent-0.147.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14515292 2026-10-06 00:39 felhom-agent-0.148.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14569321 2026-10-06 11:51 felhom-agent-0.149.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14590727 2026-10-07 09:06 felhom-agent-0.150.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14631775 2026-10-07 13:45 felhom-agent-0.151.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14715119 2026-10-07 17:26 felhom-agent-0.152.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14719868 2026-10-07 18:57 felhom-agent-0.153.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14731041 2026-10-09 07:16 felhom-agent-0.154.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14783135 2026-10-13 20:36 felhom-agent-0.155.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14820843 2026-10-14 14:34 felhom-agent-0.156.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14874944 2026-10-15 21:19 felhom-agent-0.157.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14914581 2026-10-16 04:04 felhom-agent-0.158.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14919180 2026-10-16 04:20 felhom-agent-0.158.1
-rwxr-xr-x 1 felhom-agent felhom-agent 16007680 2026-10-17 17:39 felhom-agent-0.159.1
-rwxr-xr-x 1 felhom-agent felhom-agent 16007752 2026-10-18 14:25 felhom-agent-0.159.2
-rwxr-xr-x 1 felhom-agent felhom-agent 16071264 2026-10-19 03:55 felhom-agent-0.160.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16082682 2026-10-20 08:55 felhom-agent-0.161.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16094507 2026-10-22 09:54 felhom-agent-0.162.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16183935 2026-10-23 08:54 felhom-agent-0.163.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16272301 2026-10-24 09:39 felhom-agent-0.164.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16286780 2026-10-25 17:09 felhom-agent-0.165.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16291150 2026-10-26 09:10 felhom-agent-0.166.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16296021 2026-10-27 10:25 felhom-agent-0.167.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16320123 2026-10-29 10:03 felhom-agent-0.168.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16350287 2026-10-29 12:48 felhom-agent-0.169.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16379051 2026-10-29 17:04 felhom-agent-0.170.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16389260 2026-10-30 10:19 felhom-agent-0.171.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16393888 2026-11-03 08:49 felhom-agent-0.172.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16393888 2026-11-07 23:04 felhom-agent-0.172.1
### selfupdate subdirs listing
### agent state top-level listing (names, sizes, dates only)
total 160
drwxr-x--- 9 felhom-agent felhom-agent 4096 2026-11-10 06:07 .
drwxr-xr-x 48 root root 4096 2026-10-08 04:39 ..
-rw------- 1 felhom-agent felhom-agent 631 2026-11-10 05:40 backup-success-state.json
-rw------- 1 felhom-agent felhom-agent 290 2026-11-09 13:14 controller-swap-9201.json
-rw------- 1 felhom-agent felhom-agent 61 2026-08-03 09:15 drive-intents.json
drwx------ 2 felhom-agent felhom-agent 4096 2026-10-03 17:03 escrow-stage
drwx------ 2 felhom-agent felhom-agent 4096 2026-08-03 09:15 felhom-sshd
-rw------- 1 felhom-agent felhom-agent 67 2026-10-14 15:29 guest-binds.json
-rw------- 1 felhom-agent felhom-agent 243 2026-11-04 10:54 guest-disk-trim.json
drwx------ 3 felhom-agent felhom-agent 4096 2026-08-03 09:17 guests
-rw------- 1 felhom-agent felhom-agent 65125 2026-11-10 06:07 journal.log
-rw------- 1 felhom-agent felhom-agent 305 2026-11-03 10:22 line-test.json
-rw-r--r-- 1 felhom-agent felhom-agent 591 2026-08-03 09:15 local-api.crt
-rw------- 1 felhom-agent felhom-agent 227 2026-08-03 09:15 local-api.key
-rw------- 1 felhom-agent felhom-agent 164 2026-08-09 09:51 local-tokens.log
-rw------- 1 felhom-agent felhom-agent 5440 2026-11-07 23:20 nonces.log
drwx------ 2 felhom-agent felhom-agent 4096 2026-11-10 08:20 os
drwx------ 2 felhom-agent felhom-agent 4096 2026-08-03 09:15 pbsdr
-rw------- 1 felhom-agent felhom-agent 77 2026-10-13 20:52 poweroff.json
-rw------- 1 felhom-agent felhom-agent 383 2026-11-10 06:07 restore-test-state.json
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-07 23:05 selfupdate
-rw------- 1 felhom-agent felhom-agent 40 2026-11-03 08:49 store-grant-seen.json
drwx------ 2 felhom-agent felhom-agent 4096 2026-08-03 09:15 wg
### os-apply state
376832 /var/lib/felhom-os-apply
364544 /var/lib/felhom-os-apply/bundle-prev
159744 /var/lib/felhom-os-apply/bundle-prev/20261107T222001Z-before-0.172.1
159744 /var/lib/felhom-os-apply/bundle-prev/20261026T081007Z-before-0.166.0
40960 /var/lib/felhom-os-apply/bundle-prev/20261029T160408Z-before-0.170.0
4096 /var/lib/felhom-os-apply/agent-update
/var/lib/felhom-os-apply:
total 20
drwx------ 4 root root 4096 2026-11-07 23:20 .
drwxr-xr-x 48 root root 4096 2026-10-08 04:39 ..
drwxr-xr-x 2 root root 4096 2026-11-07 23:04 agent-update
drwxr-xr-x 5 root root 4096 2026-11-07 23:20 bundle-prev
-rw-r--r-- 1 root root 96 2026-11-07 23:20 nonces.json
/var/lib/felhom-os-apply/bundle-prev:
total 20
drwxr-xr-x 5 root root 4096 2026-11-07 23:20 .
drwx------ 4 root root 4096 2026-11-07 23:20 ..
drwxr-xr-x 3 root root 4096 2026-10-26 09:10 20261026T081007Z-before-0.166.0
drwxr-xr-x 3 root root 4096 2026-10-29 17:04 20261029T160408Z-before-0.170.0
drwxr-xr-x 3 root root 4096 2026-11-07 23:20 20261107T222001Z-before-0.172.1
### usr local bin + sbin
/usr/local/bin:
total 183644
drwxr-xr-x 2 root root 4096 2026-11-07 23:04 .
drwxr-xr-x 11 root root 4096 2026-05-21 13:31 ..
-rwxr-xr-x 1 root root 16393888 2026-11-07 23:04 felhom-agent
-rwxr-xr-x 1 root root 13984713 2026-08-03 15:06 felhom-agent.bak-0.120.0
-rwxr-xr-x 1 root root 14081336 2026-08-03 15:27 felhom-agent.bak-0.121.0
-rwxr-xr-x 1 root root 14085464 2026-08-03 16:41 felhom-agent.bak-0.121.1
-rwxr-xr-x 1 root root 14076649 2026-08-03 18:54 felhom-agent.bak-0.122.0
-rwxr-xr-x 1 root root 14081569 2026-08-04 09:38 felhom-agent.bak-0.123.0
-rwxr-xr-x 1 root root 14093378 2026-08-04 13:42 felhom-agent.bak-0.124.1
-rwxr-xr-x 1 root root 14115378 2026-08-06 16:58 felhom-agent.bak-0.125.0
-rwxr-xr-x 1 root root 14124876 2026-08-08 16:30 felhom-agent.bak-0.127.0
-rwxr-xr-x 1 root root 14141686 2026-08-12 18:45 felhom-agent.bak-0.128.0
-rwxr-xr-x 1 root root 14161760 2026-08-20 12:18 felhom-agent.bak-0.129.0
-rwxr-xr-x 1 root root 14284239 2026-10-04 11:02 felhom-agent.bak-0.139.0
-rwxr-xr-x 1 root root 16393888 2026-11-03 08:49 felhom-agent.prev
/usr/local/sbin:
total 336
drwxr-xr-x 2 root root 4096 2026-11-07 23:20 .
drwxr-xr-x 11 root root 4096 2026-05-21 13:31 ..
-rwxr-xr-x 1 root root 6654 2026-08-03 09:15 felhom-backup-target-apply
-rwxr-xr-x 1 root root 12986 2026-07-18 18:17 felhom-bootstrap.sh
-rwxr-xr-x 1 root root 8687 2026-10-04 16:12 felhom-crash-guard
-rwxr-xr-x 1 root root 6816 2026-10-16 04:35 felhom-internal-place
-rwxr-xr-x 1 root root 3393 2026-08-03 09:15 felhom-mgmt-watchdog
-rwxr-xr-x 1 root root 5439 2026-10-16 04:35 felhom-mkfs-guarded
-rwxr-xr-x 1 root root 142720 2026-11-07 23:20 felhom-os-apply
-rwxr-xr-x 1 root root 50933 2026-10-04 18:22 felhom-os-apply.pre-bundle
-rwxr-xr-x 1 root root 9087 2026-08-03 09:15 felhom-pbs-apply
-rwxr-xr-x 1 root root 6316 2026-07-18 18:30 felhom-pbs-apply.bak-20260718-preR39
-rwxr-xr-x 1 root root 6822 2026-07-18 20:21 felhom-pbs-apply.bak-20260721-101635
-rwxr-xr-x 1 root root 26062 2026-10-17 17:55 felhom-priv-apply
-rwxr-xr-x 1 root root 7806 2026-10-05 13:09 felhom-selfupdate-guarded
-rwxr-xr-x 1 root root 943 2026-08-03 09:15 felhom-shared-parent.sh
-rwxr-xr-x 1 root root 3844 2026-10-17 17:55 felhom-thin-check
-rwxr-xr-x 1 root root 5537 2026-10-14 15:04 felhom-vzdump-partial-clean
### dumps on drives
-- /mnt/hdd_1/dump
total 4256676
drwxr-xr-x 2 root root 4096 2026-11-10 05:35 .
drwxr-xr-x 5 root root 4096 2026-07-28 21:25 ..
-rw-r--r-- 1 root root 1506 2026-11-08 05:29 vzdump-lxc-9201-2026_11_08-05_28_21.log
-rw-r--r-- 1 root root 1530164897 2026-11-08 05:29 vzdump-lxc-9201-2026_11_08-05_28_21.tar.zst
-rw-r--r-- 1 root root 16 2026-11-08 05:29 vzdump-lxc-9201-2026_11_08-05_28_21.tar.zst.notes
-rw-r--r-- 1 root root 1506 2026-11-09 05:31 vzdump-lxc-9201-2026_11_09-05_31_03.log
-rw-r--r-- 1 root root 1528966739 2026-11-09 05:31 vzdump-lxc-9201-2026_11_09-05_31_03.tar.zst
-rw-r--r-- 1 root root 16 2026-11-09 05:31 vzdump-lxc-9201-2026_11_09-05_31_03.tar.zst.notes
-rw-r--r-- 1 root root 1506 2026-11-10 05:35 vzdump-lxc-9201-2026_11_10-05_34_26.log
-rw-r--r-- 1 root root 1299653068 2026-11-10 05:35 vzdump-lxc-9201-2026_11_10-05_34_26.tar.zst
-rw-r--r-- 1 root root 16 2026-11-10 05:35 vzdump-lxc-9201-2026_11_10-05_34_26.tar.zst.notes
-- /var/lib/vz/dump
total 19679272
drwxr-xr-x 2 root root 4096 2026-10-05 05:19 .
drwxr-xr-x 7 root root 4096 2026-07-18 18:32 ..
-rw-r--r-- 1 root root 612397450 2026-07-18 18:31 vzdump-lxc-9100-2026_07_18-18_30_52.tar.zst
-rw-r--r-- 1 root root 779 2026-07-20 17:51 vzdump-lxc-9100-2026_07_20-17_50_57.log
-rw-r--r-- 1 root root 648549366 2026-07-20 17:51 vzdump-lxc-9100-2026_07_20-17_50_57.tar.zst
-rw-r--r-- 1 root root 779 2026-07-23 15:54 vzdump-lxc-9100-2026_07_23-15_54_21.log
-rw-r--r-- 1 root root 698 2026-08-03 06:58 vzdump-lxc-9100-2026_08_03-06_58_00.log
-rw-r--r-- 1 root root 649547835 2026-08-03 06:58 vzdump-lxc-9100-2026_08_03-06_58_00.tar.zst
-rw-r--r-- 1 root root 649547835 2026-08-03 09:16 vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst
-rw-r--r-- 1 root root 1579 2026-07-28 01:32 vzdump-lxc-9201-2026_07_28-01_27_56.log
-rw-r--r-- 1 root root 5820882781 2026-07-28 01:32 vzdump-lxc-9201-2026_07_28-01_27_56.tar.zst
-rw-r--r-- 1 root root 16 2026-07-28 01:32 vzdump-lxc-9201-2026_07_28-01_27_56.tar.zst.notes
-rw-r--r-- 1 root root 1579 2026-07-28 06:49 vzdump-lxc-9201-2026_07_28-06_44_22.log
-rw-r--r-- 1 root root 5840583923 2026-07-28 06:49 vzdump-lxc-9201-2026_07_28-06_44_22.tar.zst
-rw-r--r-- 1 root root 16 2026-07-28 06:49 vzdump-lxc-9201-2026_07_28-06_44_22.tar.zst.notes
-rw-r--r-- 1 root root 1579 2026-07-28 17:58 vzdump-lxc-9201-2026_07_28-17_53_26.log
-rw-r--r-- 1 root root 5929975285 2026-07-28 17:58 vzdump-lxc-9201-2026_07_28-17_53_26.tar.zst
-rw-r--r-- 1 root root 16 2026-07-28 17:58 vzdump-lxc-9201-2026_07_28-17_53_26.tar.zst.notes
### template cache
/var/lib/vz/template/cache:
total 126920
drwxr-xr-x 2 root root 4096 2026-07-20 17:47 .
drwxr-xr-x 4 root root 4096 2026-07-18 18:15 ..
-rw-r--r-- 1 root root 129954319 2026-07-14 21:37 debian-13-standard_13.6-1_amd64.tar.zst
/var/lib/vz/template/iso:
total 8
drwxr-xr-x 2 root root 4096 2026-07-23 18:37 .
drwxr-xr-x 4 root root 4096 2026-07-18 18:15 ..
### journald disk usage + limits
Archived and active journals take up 3.9G in the file system.
Nov 08 05:35:21 demo-felhom systemd-journald[308]: Runtime Journal (/run/log/journal/2b66cc60125645b48684cec2f00e2137) is 8M, max 157.7M, 149.7M free.
Nov 08 05:35:21 demo-felhom systemd-journald[308]: System Journal (/var/log/journal/2b66cc60125645b48684cec2f00e2137) is 3.9G, max 4G, 48.3M free.
### journal oldest + newest entry
2026-11-10T08:33:04+01:00 demo-felhom sudo[3488171]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-10-04T21:43:33+02:00 demo-felhom systemd[3282788]: Queued start job for default target default.target.
2026-10-04T21:43:33+02:00 demo-felhom systemd[3282788]: Created slice app.slice - User Application Slice.
### journal files (count, oldest, newest)
files=74 bytes=5410652160 oldest=2026-10-04 newest=2026-11-10
### journal bytes exact
4282028032 /var/log/journal
### journal last hour: lines by identifier (top 12)
12753 "SYSLOG_IDENTIFIER":"sudo"
442 "SYSLOG_IDENTIFIER":"felhom-agent"
363 "SYSLOG_IDENTIFIER":"systemd"
66 "SYSLOG_IDENTIFIER":"postfix/smtp"
21 "SYSLOG_IDENTIFIER":"sshd-session"
17 "SYSLOG_IDENTIFIER":"systemd-logind"
12 "SYSLOG_IDENTIFIER":"pveproxy"
11 "SYSLOG_IDENTIFIER":"postfix/qmgr"
8 "SYSLOG_IDENTIFIER":"pvedaemon"
4 "SYSLOG_IDENTIFIER":"tailscaled"
4 "SYSLOG_IDENTIFIER":"(systemd)"
3 "SYSLOG_IDENTIFIER":"(sd-pam)"
### journal last hour: total lines
13708
### journald conf files
[Journal]
### thin-check script
-- /usr/local/sbin/felhom-thin-check
#!/bin/bash
# felhom-thin-check — list the thin device ids a thin pool holds next to the ones LVM knows (R-1012). READ-ONLY: it
# names an orphan, it never removes one. Installed as /usr/local/sbin/felhom-thin-check (config bundle).
#
# felhom-thin-check <vg> <pool>
#
# A full host root while vzdump takes its snapshot can leave a thin snapshot inside the pool that LVM no longer knows
# (demo-hp 2026-10-14: `thin_ls` device 4, LVM only 1 and 2); from then on every snapshot backup of that guest fails
# („Creation of new snapshot 4 of device 2 failed") and nothing said why. The repair (`dmsetup message … delete <id>`)
# stays a HUMAN step, decided by the operator — this helper only reads.
#
# Reading the live pool's device list needs a metadata snapshot: the kernel's documented way to read metadata while
# the pool is in use (`reserve_metadata_snap`, `thin_ls --metadata-snap`, `release_metadata_snap`). It changes no data
# and no mapping; it is held for the length of one `thin_ls -o DEV` (well under a second on these pools). Rules:
# 1. the arguments are an LVM name pair; <vg>/<pool> is a thin pool (lv_attr starts with `t`);
# 2. only one run at a time (a lock file); a metadata snapshot that someone ELSE holds is never touched — the run
# refuses (exit 3) and does not release it;
# 3. LVM's list is read before AND after the pool's, so a volume created in between is not reported as an orphan
# (the caller treats an id as an orphan only when it is in the pool and in neither LVM list).
# Output, one record per line: lvm-before <id> <lv> | pool <id> | lvm-after <id> <lv> | ok
# Exit: 0 read, 1 refused (arguments / not a thin pool), 3 a metadata snapshot is already held or cannot be reserved,
# 4 the pool could not be read. Pinned by configs/test_felhom_thin_check.py.
set -u
LVS=/usr/sbin/lvs
DMSETUP=/usr/sbin/dmsetup
THIN_LS=/usr/sbin/thin_ls
LOCK=/run/lock/felhom-thin-check.lock
NAME_RE='^[A-Za-z0-9_+][A-Za-z0-9_.+-]{0,63}$'
die() { echo "felhom-thin-check: REFUSED: $1" >&2; exit "${2:-1}"; } # $1 the reason, $2 the exit code
lvm_ids() { # $1 tag → "<tag> <id> <lv>" for every thin volume of this pool
"$LVS" --noheadings --separator , -o lv_name,thin_id --select "pool_lv=$POOL" -- "$VG" 2>/dev/null |
awk -F, -v tag="$1" '{ gsub(/ /, ""); if ($2 ~ /^[0-9]+$/) print tag, $2, $1 }'
}
release() { "$DMSETUP" message "$TPOOL" 0 release_metadata_snap >/dev/null 2>&1; }
main() {
[[ $# -eq 2 ]] || die "usage: felhom-thin-check <vg> <pool>"
VG=$1 POOL=$2
local attr held ids
[[ "$VG" =~ $NAME_RE && "$POOL" =~ $NAME_RE ]] || die "not an LVM name pair"
attr=$("$LVS" --noheadings -o lv_attr -- "$VG/$POOL" 2>/dev/null | tr -d ' ')
[[ "$attr" == t* ]] || die "$VG/$POOL is not a thin pool"
DM="${VG//-/--}-${POOL//-/--}"
TPOOL="$DM-tpool"
TMETA="/dev/mapper/${DM}_tmeta"
exec 9>"$LOCK" || die "cannot open the lock file" 4
flock -w 30 9 || die "another check holds the lock" 3
# Field 7 of the thin-pool status is the held metadata root: "-" when no metadata snapshot exists.
held=$("$DMSETUP" status "$TPOOL" 2>/dev/null | awk '$3 == "thin-pool" { print $7 }')
[[ -n "$held" ]] || die "cannot read the status of $TPOOL" 4
[[ "$held" == "-" ]] || die "a metadata snapshot of $TPOOL is already held (by someone else) — not touched" 3
lvm_ids lvm-before
"$DMSETUP" message "$TPOOL" 0 reserve_metadata_snap >/dev/null 2>&1 || die "could not reserve a metadata snapshot of $TPOOL" 3
trap release EXIT
trap 'release; trap - EXIT; exit 4' INT TERM HUP
if ! ids=$("$THIN_LS" --metadata-snap --no-headers -o DEV "$TMETA" 2>/dev/null); then
die "thin_ls could not read $TMETA" 4
fi
release
trap - EXIT INT TERM HUP
awk '$1 ~ /^[0-9]+$/ { print "pool", $1 }' <<<"$ids"
lvm_ids lvm-after
echo ok
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then main "$@"; fi
-- /usr/local/sbin/felhom-vzdump-partial-clean
#!/bin/bash
# felhom-vzdump-partial-clean — remove ONE dead, half-written vzdump backup (R-1006). The only delete the agent's
# sudoers allows for backup files. Installed as /usr/local/sbin/felhom-vzdump-partial-clean (config bundle).
#
# felhom-vzdump-partial-clean <storage> <vmid> <name>
#
# A power cut during the night backup leaves `vzdump-lxc-<vmid>-<date>.tar.dat` (+ its `.tmp/` directory) on the
# storage; nothing removes it, every later backup is then skipped „not enough space", and a full host root breaks
# LVM's own metadata writes (power-cut-fixes-2026-10-14 C4). This helper removes such a leftover — and NOTHING else.
# It trusts nothing the agent says; every rule is checked here, as root, and any doubt REFUSES (exit 1):
# 1. the arguments: a storage id, a vmid, and a name that is exactly
# vzdump-(lxc|qemu)-<THAT vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp) — never a finished archive
# (.tar.zst, .vma.zst, …), never a .log/.notes, never another vmid;
# 2. the storage: its section in /etc/pve/storage.cfg is `dir: <storage>` with a `path`; the target is
# <path>/dump/<name>, not a symlink, and its real directory IS <path>/dump;
# 3. not in use: no vzdump runs on the host at all (a task `…:vzdump:…` or a `vzdump` command — a job for several
# guests carries no vmid in its task name), and no
# process holds the target (or, for the .tmp directory, anything under it) open or as its working directory
# (C2: after a killed worker, tar and zstd kept writing into the .tar.dat for 4 min 40 s, none named „vzdump");
# 4. dead: last modified before this boot (`btime` in /proc/stat), or untouched for at least QUIET_S seconds.
# Prints one line on success: „removed <name> <bytes> B (<why>)". Pinned by configs/test_felhom_vzdump_partial_clean.py.
set -u
STORAGE_CFG=/etc/pve/storage.cfg
PROC=/proc
QUIET_S=120
die() { echo "felhom-vzdump-partial-clean: REFUSED: $*" >&2; exit 1; }
storage_path() { # $1 storage id → prints the dir storage's path, or nothing
awk -v want="$1" '
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (id == want); next }
in_s && /^[ \t]+path[ \t]/ { if (type == "dir") { sub(/^[ \t]+path[ \t]+/, ""); sub(/[ \t]+$/, ""); print; exit } }
' "$STORAGE_CFG"
}
storage_type() {
awk -v want="$1" '/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); id = h[2]; sub(/[ \t]+$/, "", id); if (id == want) { print h[1]; exit } }' "$STORAGE_CFG"
}
boot_time() { awk '$1 == "btime" { print $2 }' "$PROC/stat"; }
newest_mtime() { # a file's mtime, or the newest mtime of anything in a directory (the dir itself included)
if [[ -d "$1" ]]; then find "$1" -xdev -printf '%T@\n' 2>/dev/null | sort -n | tail -1 | cut -d. -f1
else stat -c %Y "$1"; fi
}
busy_reason() { # $1 vmid, $2 target → prints why it is in use, or nothing
local vmid=$1 target=$2 p link cmd
for p in "$PROC"/[0-9]*; do
[[ -r "$p/cmdline" ]] || continue
cmd=$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)
# ANY vzdump on the host refuses: a job for several guests runs as `task UPID:…:vzdump::…` (no vmid in it), and
# its tar may be between this guest's files. Waiting costs one backup run; deleting a live one costs the backup.
if [[ "$cmd" == *":vzdump:"* ]] || [[ "$cmd" =~ (^|/)vzdump([[:space:]]|$) ]]; then
echo "a vzdump runs on this host (pid ${p##*/})"; return
fi
for link in "$p/cwd" "$p"/fd/*; do
local to
to=$(readlink "$link" 2>/dev/null) || continue
if [[ "$to" == "$target" || "$to" == "$target/"* ]]; then
echo "pid ${p##*/} has it open"; return
fi
done
done
}
main() {
local storage=${1:-} vmid=${2:-} name=${3:-}
[[ $# -eq 3 ]] || die "usage: felhom-vzdump-partial-clean <storage> <vmid> <name>"
[[ "$storage" =~ ^[A-Za-z][A-Za-z0-9_-]{0,63}$ ]] || die "bad storage id"
[[ "$vmid" =~ ^[1-9][0-9]{2,8}$ ]] || die "bad vmid"
[[ "$name" =~ ^vzdump-(lxc|qemu)-${vmid}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$ ]] \
|| die "name is not a half-written vzdump of guest $vmid: $name"
[[ "$(storage_type "$storage")" == "dir" ]] || die "storage $storage is not a dir storage"
local path; path=$(storage_path "$storage")
[[ "$path" == /* ]] || die "storage $storage has no path"
local dump="$path/dump" target="$path/dump/$name"
[[ -e "$target" ]] || die "no such entry: $target"
[[ ! -L "$target" ]] || die "a symlink: $target"
[[ "$(realpath -e "$dump")" == "$(realpath -e "$(dirname "$target")")" ]] || die "not directly in $dump"
if [[ "$name" == *.tmp ]]; then [[ -d "$target" ]] || die ".tmp is not a directory"
else [[ -f "$target" ]] || die "not a regular file"; fi
local busy; busy=$(busy_reason "$vmid" "$target")
[[ -z "$busy" ]] || die "in use: $busy"
local bt mt now why
bt=$(boot_time); mt=$(newest_mtime "$target"); now=$(date +%s)
[[ "$bt" =~ ^[0-9]+$ && "$mt" =~ ^[0-9]+$ ]] || die "cannot read the boot time or the modification time"
if (( mt < bt )); then why="written before this boot"
elif (( now - mt >= QUIET_S )); then why="untouched for $(( now - mt )) s, no vzdump for $vmid, nothing holds it"
else die "modified $(( now - mt )) s ago — may still be written"; fi
local size; size=$(du -sb "$target" 2>/dev/null | cut -f1)
if [[ -d "$target" ]]; then rm -rf --one-file-system -- "$target"; else rm -f -- "$target"; fi
[[ ! -e "$target" ]] || die "remove failed: $target"
echo "removed $name ${size:-?} B ($why)"
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
main "$@"
fi
### units mentioning thin-check / partial-clean
/etc/sudoers.d/felhom-agent
### lvm thin settings
thin_pool_autoextend_threshold=100
thin_pool_autoextend_percent=20
monitoring=1
active
### agent config: selected non-secret keys
-- /etc/felhom-agent/agent.json
"local_backup_target": "felhom-backup"
retention": 3
"keep_last": 0
### vzdump.conf + storage prune settings
1:dir: local
2: path /var/lib/vz
3: content iso,backup,import,vztmpl
5:lvmthin: local-lvm
8: content images,rootdir
10:pbs: felhom-pbs
13: content backup
19:dir: felhom-backup
20: path /mnt/hdd_1
21: content backup
22: is_mountpoint 1
### kernels
proxmox-kernel-7.0.14-20-pve un
proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20 1006525 ii
proxmox-kernel-7.0.14-22-pve un
proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22 1006522 ii
proxmox-kernel-7.0.14-23-pve un
proxmox-kernel-7.0.14-23-pve-signed 7.0.14-23 1006525 ii
proxmox-kernel-7.0.14-24-pve un
proxmox-kernel-7.0.14-24-pve-signed 7.0.14-24 1006528 ii
proxmox-kernel-7.0.14-25-pve un
proxmox-kernel-7.0.14-25-pve-signed 7.0.14-25 1007819 ii
proxmox-kernel-7.0.14-26-pve un
proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26 1007815 ii
proxmox-kernel-7.0.14-28-pve un
proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28 1007817 ii
proxmox-kernel-7.0.2-6-pve un
proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6 1005219 ii
proxmox-kernel-helper 9.1.0+fde2 109 ii
7.0.14-28-pve
### apt history: kernel installs/removes
Start-Date: 2026-11-08 05:32:58
Start-Date: 2026-10-07 12:09:02
Start-Date: 2026-10-09 04:44:23
Start-Date: 2026-10-10 04:44:06
Start-Date: 2026-10-16 05:06:43
Start-Date: 2026-10-21 05:21:10
Start-Date: 2026-10-29 04:45:31
### apt autoremove config
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve-signed$";
### boot
Filesystem 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 100861726720 38093606912 57597382656 40% /
/dev/sda2 1071624192 9469952 1062154240 1% /boot/efi
total 901784
drwxr-xr-x 5 root root 4096 2026-11-08 05:33 .
drwxr-xr-x 19 root root 4096 2026-07-18 18:16 ..
-rw-r--r-- 1 root root 308363 2026-09-24 12:43 config-7.0.14-20-pve
-rw-r--r-- 1 root root 308363 2026-10-06 07:41 config-7.0.14-22-pve
-rw-r--r-- 1 root root 308363 2026-10-07 22:25 config-7.0.14-23-pve
-rw-r--r-- 1 root root 308363 2026-10-09 13:32 config-7.0.14-24-pve
-rw-r--r-- 1 root root 308500 2026-10-17 22:42 config-7.0.14-25-pve
-rw-r--r-- 1 root root 308500 2026-10-22 20:45 config-7.0.14-26-pve
-rw-r--r-- 1 root root 308500 2026-11-02 12:54 config-7.0.14-28-pve
-rw-r--r-- 1 root root 308342 2026-05-20 10:55 config-7.0.2-6-pve
drwxr-xr-x 3 root root 4096 1970-01-01 01:00 efi
drwxr-xr-x 6 root root 4096 2026-11-08 05:36 grub
-rw-r--r-- 1 root root 87751579 2026-10-07 14:16 initrd.img-7.0.14-20-pve
-rw-r--r-- 1 root root 87761381 2026-10-09 04:44 initrd.img-7.0.14-22-pve
-rw-r--r-- 1 root root 87787201 2026-10-10 04:44 initrd.img-7.0.14-23-pve
-rw-r--r-- 1 root root 87763821 2026-10-16 05:06 initrd.img-7.0.14-24-pve
-rw-r--r-- 1 root root 87852290 2026-10-21 05:21 initrd.img-7.0.14-25-pve
-rw-r--r-- 1 root root 87853652 2026-10-29 04:45 initrd.img-7.0.14-26-pve
-rw-r--r-- 1 root root 87847391 2026-11-08 05:33 initrd.img-7.0.14-28-pve
-rw-r--r-- 1 root root 91998300 2026-10-04 13:35 initrd.img-7.0.2-6-pve
-rw-r--r-- 1 root root 151020 2024-11-17 17:43 memtest86+ia32.bin
-rw-r--r-- 1 root root 152064 2024-11-17 17:43 memtest86+ia32.efi
-rw-r--r-- 1 root root 155992 2024-11-17 17:43 memtest86+x64.bin
-rw-r--r-- 1 root root 157184 2024-11-17 17:43 memtest86+x64.efi
drwxr-xr-x 2 root root 4096 2026-11-08 05:33 pve
-rw-r--r-- 1 root root 10483512 2026-09-24 12:43 System.map-7.0.14-20-pve
-rw-r--r-- 1 root root 10483429 2026-10-06 07:41 System.map-7.0.14-22-pve
-rw-r--r-- 1 root root 10483429 2026-10-07 22:25 System.map-7.0.14-23-pve
-rw-r--r-- 1 root root 10483455 2026-10-09 13:32 System.map-7.0.14-24-pve
-rw-r--r-- 1 root root 10341401 2026-10-17 22:42 System.map-7.0.14-25-pve
-rw-r--r-- 1 root root 10341401 2026-10-22 20:45 System.map-7.0.14-26-pve
-rw-r--r-- 1 root root 10341916 2026-11-02 12:54 System.map-7.0.14-28-pve
-rw-r--r-- 1 root root 10452992 2026-05-20 10:55 System.map-7.0.2-6-pve
-rw-r--r-- 1 root root 16284776 2026-09-24 12:43 vmlinuz-7.0.14-20-pve
-rw-r--r-- 1 root root 16288872 2026-10-06 07:41 vmlinuz-7.0.14-22-pve
-rw-r--r-- 1 root root 16288872 2026-10-07 22:25 vmlinuz-7.0.14-23-pve
-rw-r--r-- 1 root root 16288872 2026-10-09 13:32 vmlinuz-7.0.14-24-pve
-rw-r--r-- 1 root root 16284776 2026-10-17 22:42 vmlinuz-7.0.14-25-pve
-rw-r--r-- 1 root root 16280680 2026-10-22 20:45 vmlinuz-7.0.14-26-pve
### pve tasks / rrd
3162112 /var/log/pve
21041152 /var/lib/rrdcached
706
### root dir extras
16269312 /root
1007616 /root/n100-rehearsal
20480 /root/.ssh
12288 /root/.cache
8192 /root/.config
### df now
Filesystem 1B-blocks Used Avail Use% Mounted on
/dev/mapper/pve-root 100861726720 38093606912 57597382656 40% /
data 374538764288B 3.03 0.58
root 103079215104B
swap 8589934592B
vm-9201-disk-2 34359738368B 6.48
vm-9201-disk-3 268435456000B 3.40
### end
2026-11-10T07:33:05Z
@@ -0,0 +1,584 @@
### identity
demo-felhom
2026-11-10T07:24:54Z
pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve)
felhom-agent 0.172.1
### in-flight (must be empty)
### lsblk
NAME SIZE TYPE FSTYPE MOUNTPOINT MODEL TRAN
sda 512110190592 disk AirDisk 512GB SSD sata
|-sda1 1031168 part
|-sda2 1073741824 part vfat /boot/efi
`-sda3 511035383296 part LVM2_member
|-pve-swap 8589934592 lvm swap [SWAP]
|-pve-root 103079215104 lvm ext4 /
|-pve-data_tmeta 3821010944 lvm
| `-pve-data-tpool 374538764288 lvm
| |-pve-data 374538764288 lvm
| |-pve-vm--9201--disk--2 34359738368 lvm ext4
| `-pve-vm--9201--disk--3 268435456000 lvm ext4
`-pve-data_tdata 374538764288 lvm
`-pve-data-tpool 374538764288 lvm
|-pve-data 374538764288 lvm
|-pve-vm--9201--disk--2 34359738368 lvm ext4
`-pve-vm--9201--disk--3 268435456000 lvm ext4
sdb 1000204886016 disk ext4 /mnt/hdd_1 TOSHIBA MQ04ABF100 usb
### df bytes
Filesystem Type 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-root ext4 100861726720 38093578240 57597411328 40% /
/dev/sda2 vfat 1071624192 9469952 1062154240 1% /boot/efi
/dev/sdb ext4 983351140352 8143593472 925180526592 1% /mnt/hdd_1
/dev/fuse fuse 134217728 20480 134197248 1% /etc/pve
### df inodes
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pve-root 6291456 120798 6170658 2% /
/dev/sda2 0 0 0 - /boot/efi
/dev/sdb 61054976 731 61054245 1% /mnt/hdd_1
/dev/fuse 262144 39 262105 1% /etc/pve
### vgs
VG VSize VFree
pve 511033999360 17184063488
### lvs
LV VG Attr LSize Pool Data% Meta% MSize
data pve twi-aotz-- 374538764288 3.03 0.58 3821010944
root pve -wi-ao---- 103079215104
swap pve -wi-ao---- 8589934592
vm-9201-disk-2 pve Vwi-aotz-- 34359738368 data 6.48
vm-9201-disk-3 pve Vwi-aotz-- 268435456000 data 3.40
### thin pool autoextend + monitoring
Configuration node activation/thin_pool_autoextend_threshold not found
Configuration node activation/thin_pool_autoextend_percent not found
Configuration node activation/monitoring not found
### pvesm status
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
felhom-backup dir active 960303848 7952728 903496608 0.83%
felhom-pbs pbs active 0 0 0 0.00%
local dir active 98497780 37200760 56247472 37.77%
local-lvm lvmthin active 365760512 11082543 354677968 3.03%
### storage.cfg (no secrets live here; passwords are in /etc/pve/priv)
dir: local
path /var/lib/vz
content iso,backup,import,vztmpl
lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir
pbs: felhom-pbs
datastore felhom-offsite
server <redacted>
content backup
encryption-key <redacted>
fingerprint <redacted>
namespace demo-felhom
username <redacted>
dir: felhom-backup
path /mnt/hdd_1
content backup
is_mountpoint 1
### pvesm list per storage (count, bytes, oldest, newest)
felhom-backup total entries=3 bytes=4358784704
felhom-backup type=backup entries=3 bytes=4358784704
felhom-pbs total entries=3 bytes=11096707642
felhom-pbs type=backup entries=3 bytes=11096707642
local total entries=8 bytes=20281438794
local type=backup entries=7 bytes=20151484475
local type=vztmpl entries=1 bytes=129954319
local-lvm total entries=2 bytes=302795194368
local-lvm type=rootdir entries=2 bytes=302795194368
### guest configs (disk + mountpoint lines only)
-- /etc/pve/lxc/9201.conf
memory: 12288
mp0: local-lvm:vm-9201-disk-3,mp=/var/lib/felhom,backup=1,size=250G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
rootfs: local-lvm:vm-9201-disk-2,size=32G
swap: 12288
### vzdump.conf + jobs (keep rules)
### journald
Archived and active journals take up 3.9G in the file system.
[Journal]
76
IDX BOOT ID FIRST ENTRY LAST ENTRY
0 72899d2625244e069e8e8099775d1e51 Sun 2026-11-08 05:35:21 CET Tue 2026-11-10 08:24:54 CET
### /var/log top
4295942144 /var/log
4282028032 /var/log/journal
8921088 /var/log/pveproxy
3162112 /var/log/pve
1290240 /var/log/ifupdown2
102400 /var/log/apt
16384 /var/log/vzdump
12288 /var/log/runit
4096 /var/log/samba
4096 /var/log/private
4096 /var/log/lxc
4096 /var/log/frr
4096 /var/log/corosync
4096 /var/log/chrony
4096 /var/log/ceph
### /var/log big files
75497472 2026-11-10 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001ceeaf1-00065fdd17ea36cd.journal
75497472 2026-11-09 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001cd0806-00065fd56c646a80.journal
75497472 2026-11-09 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001cb2963-00065fcde14d0e1a.journal
75497472 2026-11-09 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001c9470a-00065fc6325a3cbf.journal
75497472 2026-11-08 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001c76471-00065fbe7fb14409.journal
75497472 2026-11-07 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001c3a790-00065fafb162e27e.journal
75497472 2026-11-07 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001c1c0b6-00065fa7f74ec2e8.journal
75497472 2026-11-07 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001bfd98f-00065fa03a744d43.journal
75497472 2026-11-06 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001bdf158-00065f987152125c.journal
75497472 2026-11-06 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001bc0c5d-00065f90c72342b8.journal
75497472 2026-11-06 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001ba2443-00065f890080cb91.journal
75497472 2026-11-05 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001b83cb8-00065f813e67e76a.journal
75497472 2026-11-05 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001b6577a-00065f798f5cf7c9.journal
75497472 2026-11-04 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001b470c0-00065f71d0e7e530.journal
75497472 2026-11-04 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001b28c21-00065f6a26b545c8.journal
75497472 2026-11-04 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001b0a6a8-00065f626d4b5971.journal
75497472 2026-11-03 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001aec1bf-00065f5ab8b956f0.journal
75497472 2026-11-03 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001ace431-00065f5366d557de.journal
75497472 2026-11-03 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001aafc79-00065f4ba25d0659.journal
75497472 2026-11-02 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001a9151c-00065f43e0185292.journal
75497472 2026-11-02 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001a72f9f-00065f3c3030bff7.journal
75497472 2026-11-01 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001a546fa-00065f346e2d97fb.journal
75497472 2026-11-01 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001a36222-00065f2cc4920d39.journal
75497472 2026-11-01 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001a17a1a-00065f24fc9bdcdc.journal
75497472 2026-10-31 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-00000000019f91dd-00065f1d32db0b4d.journal
75497472 2026-10-31 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-00000000019dad26-00065f15899b665c.journal
75497472 2026-10-30 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-00000000019bc4e3-00065f0dc1354914.journal
75497472 2026-10-30 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-000000000199e35f-00065f0633414f7a.journal
75497472 2026-10-30 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-0000000001980137-00065efe9afbf08e.journal
75497472 2026-10-29 /var/log/journal/2b66cc60125645b48684cec2f00e2137/system@4230bcd42751455ea688563670e5999d-000000000196258d-00065ef7369141a1.journal
### logrotate configs present
alternatives
apt
btmp
ceph-common
chrony
corosync
dpkg
frr
pve
pve-firewall
wtmp
wtmpdb
### kernels installed
linux-image-7.0.14-20-pve-amd64 un
linux-image-7.0.14-22-pve-amd64 un
linux-image-7.0.14-23-pve-amd64 un
linux-image-7.0.14-24-pve-amd64 un
linux-image-7.0.14-25-pve-amd64 un
linux-image-7.0.14-26-pve-amd64 un
linux-image-7.0.14-28-pve-amd64 un
linux-image-7.0.2-6-pve-amd64 un
linux-image-amd64 un
linux-image-generic un
proxmox-kernel-7.0 7.0.14-28 37 ii
proxmox-kernel-7.0.14-20-pve un
proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20 1006525 ii
proxmox-kernel-7.0.14-22-pve un
proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22 1006522 ii
proxmox-kernel-7.0.14-23-pve un
proxmox-kernel-7.0.14-23-pve-signed 7.0.14-23 1006525 ii
proxmox-kernel-7.0.14-24-pve un
proxmox-kernel-7.0.14-24-pve-signed 7.0.14-24 1006528 ii
proxmox-kernel-7.0.14-25-pve un
proxmox-kernel-7.0.14-25-pve-signed 7.0.14-25 1007819 ii
proxmox-kernel-7.0.14-26-pve un
proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26 1007815 ii
proxmox-kernel-7.0.14-28-pve un
proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28 1007817 ii
proxmox-kernel-7.0.2-6-pve un
proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6 1005219 ii
proxmox-kernel-helper 9.1.0+fde2 109 ii
pve-kernel-7.0 un
pve-kernel-helper un
7.0.14-28-pve
### proxmox-boot-tool
Manually selected kernels:
None.
Automatically selected kernels:
7.0.14-26-pve
7.0.14-28-pve
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
E: /etc/kernel/proxmox-boot-uuids does not exist.
### /boot
Filesystem 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 100861726720 38093574144 57597415424 40% /
/dev/sda2 1071624192 9469952 1062154240 1% /boot/efi
939851776 /boot
16437248 /boot/grub
4096 /boot/pve
91998300 Oct 4 13:35 initrd.img-7.0.2-6-pve
87853652 Oct 29 04:45 initrd.img-7.0.14-26-pve
87852290 Oct 21 05:21 initrd.img-7.0.14-25-pve
87847391 Nov 8 05:33 initrd.img-7.0.14-28-pve
87787201 Oct 10 04:44 initrd.img-7.0.14-23-pve
87763821 Oct 16 05:06 initrd.img-7.0.14-24-pve
87761381 Oct 9 04:44 initrd.img-7.0.14-22-pve
87751579 Oct 7 14:16 initrd.img-7.0.14-20-pve
16288872 Oct 9 13:32 vmlinuz-7.0.14-24-pve
16288872 Oct 7 22:25 vmlinuz-7.0.14-23-pve
16288872 Oct 6 07:41 vmlinuz-7.0.14-22-pve
16284776 Sep 24 12:43 vmlinuz-7.0.14-20-pve
16284776 Oct 17 22:42 vmlinuz-7.0.14-25-pve
16280680 Oct 22 20:45 vmlinuz-7.0.14-26-pve
16280680 Nov 2 12:54 vmlinuz-7.0.14-28-pve
16227432 May 20 10:55 vmlinuz-7.0.2-6-pve
10483512 Sep 24 12:43 System.map-7.0.14-20-pve
10483455 Oct 9 13:32 System.map-7.0.14-24-pve
10483429 Oct 7 22:25 System.map-7.0.14-23-pve
10483429 Oct 6 07:41 System.map-7.0.14-22-pve
10452992 May 20 10:55 System.map-7.0.2-6-pve
10341916 Nov 2 12:54 System.map-7.0.14-28-pve
10341401 Oct 22 20:45 System.map-7.0.14-26-pve
10341401 Oct 17 22:42 System.map-7.0.14-25-pve
308500 Oct 22 20:45 config-7.0.14-26-pve
308500 Oct 17 22:42 config-7.0.14-25-pve
308500 Nov 2 12:54 config-7.0.14-28-pve
308363 Sep 24 12:43 config-7.0.14-20-pve
308363 Oct 9 13:32 config-7.0.14-24-pve
308363 Oct 7 22:25 config-7.0.14-23-pve
### /lib/modules
8148774912 /lib/modules
1019707392 /lib/modules/7.0.14-26-pve
1019707392 /lib/modules/7.0.14-25-pve
1019703296 /lib/modules/7.0.14-28-pve
1018171392 /lib/modules/7.0.14-24-pve
1018171392 /lib/modules/7.0.14-23-pve
1018171392 /lib/modules/7.0.14-22-pve
1018171392 /lib/modules/7.0.14-20-pve
1016967168 /lib/modules/7.0.2-6-pve
### apt
92090368 /var/cache/apt
61947904 /var/lib/apt/lists
0
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve-signed$";
01autoremove
10pveapthook
20listchanges
70debconf
76pveconf
76pveproxy
99mmdebstrap
### apt autoremove candidates (dry read)
0
### tmp
0 /tmp
12288 /var/tmp
2026-11-08 40 /tmp/.ICE-unix
2026-11-08 40 /tmp/.X11-unix
2026-11-08 40 /tmp/.XIM-unix
2026-11-08 40 /tmp/.font-unix
2026-11-08 4096 /var/tmp/systemd-private-72899d2625244e069e8e8099775d1e51-systemd-logind.service-p1H9e5
2026-11-08 60 /tmp/systemd-private-72899d2625244e069e8e8099775d1e51-systemd-logind.service-2RHCoj
6
### systemd-tmpfiles rules touching tmp
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
### /var/lib/vz
20281561088 /var/lib/vz
20151570432 /var/lib/vz/dump
129970176 /var/lib/vz/template
129961984 /var/lib/vz/template/cache
8192 /var/lib/vz/snippets
4096 /var/lib/vz/template/iso
4096 /var/lib/vz/import
4096 /var/lib/vz/images
-- /var/lib/vz/dump
612397450 2026-07-18 vzdump-lxc-9100-2026_07_18-18_30_52.tar.zst
779 2026-07-20 vzdump-lxc-9100-2026_07_20-17_50_57.log
648549366 2026-07-20 vzdump-lxc-9100-2026_07_20-17_50_57.tar.zst
779 2026-07-23 vzdump-lxc-9100-2026_07_23-15_54_21.log
1579 2026-07-28 vzdump-lxc-9201-2026_07_28-01_27_56.log
5820882781 2026-07-28 vzdump-lxc-9201-2026_07_28-01_27_56.tar.zst
16 2026-07-28 vzdump-lxc-9201-2026_07_28-01_27_56.tar.zst.notes
1579 2026-07-28 vzdump-lxc-9201-2026_07_28-06_44_22.log
5840583923 2026-07-28 vzdump-lxc-9201-2026_07_28-06_44_22.tar.zst
16 2026-07-28 vzdump-lxc-9201-2026_07_28-06_44_22.tar.zst.notes
1579 2026-07-28 vzdump-lxc-9201-2026_07_28-17_53_26.log
5929975285 2026-07-28 vzdump-lxc-9201-2026_07_28-17_53_26.tar.zst
16 2026-07-28 vzdump-lxc-9201-2026_07_28-17_53_26.tar.zst.notes
698 2026-08-03 vzdump-lxc-9100-2026_08_03-06_58_00.log
649547835 2026-08-03 vzdump-lxc-9100-2026_08_03-06_58_00.tar.zst
649547835 2026-08-03 vzdump-lxc-9100-2026_08_03-09_15_34.tar.zst
-- /var/lib/vz/template/cache
129954319 2026-07-14 debian-13-standard_13.6-1_amd64.tar.zst
-- /var/lib/vz/template/iso
### felhom dirs (candidates)
/etc/felhom
/etc/felhom-agent
/etc/felhom-oob.nft
/etc/felhom-sshd
/run/felhom-sshd.pid
/var/lib/felhom-agent
/var/lib/felhom-crash-guard
/var/lib/felhom-install
/var/lib/felhom-kernel
/var/lib/felhom-os-apply
/var/log/felhom-first-boot.log
-- /etc/felhom
24576 /etc/felhom
files=6 oldest=2026-07-18 newest=2026-11-07
-- /etc/felhom-agent
40960 /etc/felhom-agent
files=9 oldest=2026-07-27 newest=2026-09-24
-- /etc/felhom-oob.nft
4096 /etc/felhom-oob.nft
files=1 oldest=2026-08-03 newest=2026-08-03
-- /etc/felhom-sshd
24576 /etc/felhom-sshd
8192 /etc/felhom-sshd/authorized_keys
files=4 oldest=2026-08-03 newest=2026-10-05
-- /var/lib/felhom-agent
724164608 /var/lib/felhom-agent
723935232 /var/lib/felhom-agent/selfupdate
24576 /var/lib/felhom-agent/os
20480 /var/lib/felhom-agent/wg
16384 /var/lib/felhom-agent/guests
16384 /var/lib/felhom-agent/felhom-sshd
12288 /var/lib/felhom-agent/pbsdr
12288 /var/lib/felhom-agent/guests/9201
8192 /var/lib/felhom-agent/escrow-stage
files=79 oldest=2026-08-03 newest=2026-11-10
-- /var/lib/felhom-crash-guard
8192 /var/lib/felhom-crash-guard
files=1 oldest=2026-11-10 newest=2026-11-10
-- /var/lib/felhom-install
8192 /var/lib/felhom-install
files=1 oldest=2026-08-03 newest=2026-08-03
-- /var/lib/felhom-kernel
8192 /var/lib/felhom-kernel
files=1 oldest=2026-11-08 newest=2026-11-08
-- /var/lib/felhom-os-apply
376832 /var/lib/felhom-os-apply
364544 /var/lib/felhom-os-apply/bundle-prev
159744 /var/lib/felhom-os-apply/bundle-prev/20261107T222001Z-before-0.172.1
159744 /var/lib/felhom-os-apply/bundle-prev/20261026T081007Z-before-0.166.0
40960 /var/lib/felhom-os-apply/bundle-prev/20261029T160408Z-before-0.170.0
4096 /var/lib/felhom-os-apply/agent-update
files=4 oldest=2026-10-26 newest=2026-11-07
-- /var/log/felhom-first-boot.log
4096 /var/log/felhom-first-boot.log
files=1 oldest=2026-07-18 newest=2026-07-18
### felhom binaries + slots
4096 Nov 7 23:04 .
4096 May 21 13:31 ..
16393888 Nov 7 23:04 felhom-agent
13984713 Aug 3 15:06 felhom-agent.bak-0.120.0
14081336 Aug 3 15:27 felhom-agent.bak-0.121.0
14085464 Aug 3 16:41 felhom-agent.bak-0.121.1
14076649 Aug 3 18:54 felhom-agent.bak-0.122.0
14081569 Aug 4 09:38 felhom-agent.bak-0.123.0
14093378 Aug 4 13:42 felhom-agent.bak-0.124.1
14115378 Aug 6 16:58 felhom-agent.bak-0.125.0
14124876 Aug 8 16:30 felhom-agent.bak-0.127.0
14141686 Aug 12 18:45 felhom-agent.bak-0.128.0
14161760 Aug 20 12:18 felhom-agent.bak-0.129.0
14284239 Oct 4 11:02 felhom-agent.bak-0.139.0
16393888 Nov 3 08:49 felhom-agent.prev
4096 Nov 7 23:20 .
4096 May 21 13:31 ..
6654 Aug 3 09:15 felhom-backup-target-apply
12986 Jul 18 18:17 felhom-bootstrap.sh
8687 Oct 4 16:12 felhom-crash-guard
6816 Oct 16 04:35 felhom-internal-place
3393 Aug 3 09:15 felhom-mgmt-watchdog
5439 Oct 16 04:35 felhom-mkfs-guarded
142720 Nov 7 23:20 felhom-os-apply
50933 Oct 4 18:22 felhom-os-apply.pre-bundle
9087 Aug 3 09:15 felhom-pbs-apply
6316 Jul 18 18:30 felhom-pbs-apply.bak-20260718-preR39
6822 Jul 18 20:21 felhom-pbs-apply.bak-20260721-101635
26062 Oct 17 17:55 felhom-priv-apply
7806 Oct 5 13:09 felhom-selfupdate-guarded
943 Aug 3 09:15 felhom-shared-parent.sh
3844 Oct 17 17:55 felhom-thin-check
5537 Oct 14 15:04 felhom-vzdump-partial-clean
### root filesystem, top 70 dirs to depth 4
38091399168 /
25604259840 /var
21134848000 /var/lib
20281561088 /var/lib/vz
20151570432 /var/lib/vz/dump
11516825600 /usr
9682477056 /usr/lib
8148774912 /usr/lib/modules
4295942144 /var/log
4282028032 /var/log/journal
4282019840 /var/log/journal/2b66cc60125645b48684cec2f00e2137
1165144064 /usr/share
1019707392 /usr/lib/modules/7.0.14-26-pve
1019707392 /usr/lib/modules/7.0.14-25-pve
1019703296 /usr/lib/modules/7.0.14-28-pve
1018171392 /usr/lib/modules/7.0.14-24-pve
1018171392 /usr/lib/modules/7.0.14-23-pve
1018171392 /usr/lib/modules/7.0.14-22-pve
1018171392 /usr/lib/modules/7.0.14-20-pve
1016967168 /usr/lib/modules/7.0.2-6-pve
939851776 /boot
724164608 /var/lib/felhom-agent
723935232 /var/lib/felhom-agent/selfupdate
596193280 /usr/lib/x86_64-linux-gnu
582549504 /usr/lib/firmware
374468608 /usr/bin
365469696 /usr/share/pve-edk2-firmware
327876608 /usr/share/kvm
188440576 /usr/local
188047360 /usr/local/bin
170545152 /var/cache
160776192 /usr/lib/firmware/nvidia
129970176 /var/lib/vz/template
118800384 /usr/share/locale
113082368 /usr/lib/firmware/amdgpu
92090368 /var/cache/apt
87576576 /usr/sbin
79351808 /usr/share/doc
73621504 /usr/lib/ceph
65830912 /usr/lib/x86_64-linux-gnu/proxmox-backup
62586880 /var/lib/apt
61947904 /var/lib/apt/lists
58040320 /var/cache/proxmox-backup
56102912 /usr/lib/python3
56098816 /usr/lib/python3/dist-packages
51273728 /usr/lib/ceph/denc
49405952 /usr/lib/x86_64-linux-gnu/perl5
46153728 /usr/lib/firmware/intel
45576192 /usr/lib/firmware/ath11k
38559744 /usr/share/pve-docs
36225024 /var/lib/dpkg
34611200 /var/lib/dpkg/info
34213888 /usr/lib/firmware/ath12k
28692480 /usr/lib/python3.13
27885568 /usr/lib/firmware/mediatek
27856896 /usr/share/javascript
26943488 /usr/lib/x86_64-linux-gnu/perl
26587136 /usr/lib/grub
25772032 /usr/share/man
25546752 /usr/share/javascript/extjs
23773184 /usr/share/perl
23769088 /usr/share/perl/5.40.1
23715840 /usr/share/nmap
23490560 /usr/lib/udev
22081536 /usr/share/doc/pve-qemu-kvm
21475328 /usr/lib/firmware/intel-ucode
21041152 /var/lib/rrdcached
21032960 /var/lib/rrdcached/db
18657280 /usr/libexec
17694720 /usr/lib/frr
### root fs: file count per top dir
4893 /var
93612 /usr
47 /root
4 /home
0 /opt
1730 /etc
0 /srv
0 /tmp
### root home
16269312 /root
1007616 /root/n100-rehearsal
20480 /root/.ssh
12288 /root/.cache
8192 /root/.config
### other mounted filesystems, depth 2
-- /mnt/hdd_1
8141488128 /mnt/hdd_1
4358832128 /mnt/hdd_1/dump
3782635520 /mnt/hdd_1/felhom-data
3339952128 /mnt/hdd_1/felhom-data/backups
442085376 /mnt/hdd_1/felhom-data/appdata
561152 /mnt/hdd_1/felhom-data/userdata
32768 /mnt/hdd_1/felhom-data/shares
16384 /mnt/hdd_1/lost+found
### pve task logs + rrd
3162112 /var/log/pve
21041152 /var/lib/rrdcached
4210688 /var/lib/pve-cluster
106496 /var/lib/pve-manager
706
2026-07-18
2026-11-10
### core dumps
4096 /var/lib/systemd/coredump
0
### swap
NAME TYPE SIZE USED PRIO
/dev/dm-0 partition 8589930496 8192 -1
### timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2026-11-10 08:25:34 CET 29s Tue 2026-11-10 08:24:34 CET 30s ago felhom-mgmt-watchdog.timer felhom-mgmt-watchdog.service
Tue 2026-11-10 08:54:52 CET 29min Tue 2026-11-10 07:54:52 CET 30min ago felhom-crash-guard-check.timer felhom-crash-guard-check.service
Tue 2026-11-10 13:12:04 CET 4h 46min Tue 2026-11-10 00:20:45 CET 8h ago apt-daily.timer apt-daily.service
Wed 2026-11-11 00:00:00 CET 15h Tue 2026-11-10 00:00:25 CET 8h ago dpkg-db-backup.timer dpkg-db-backup.service
Wed 2026-11-11 00:09:22 CET 15h Tue 2026-11-10 00:21:32 CET 8h ago logrotate.timer logrotate.service
Wed 2026-11-11 02:05:51 CET 17h Tue 2026-11-10 01:23:25 CET 7h ago pve-daily-update.timer pve-daily-update.service
Wed 2026-11-11 03:51:22 CET 19h Tue 2026-11-10 00:12:52 CET 8h ago man-db.timer man-db.service
Wed 2026-11-11 05:50:32 CET 21h Tue 2026-11-10 05:50:32 CET 2h 34min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Wed 2026-11-11 06:00:27 CET 21h Tue 2026-11-10 06:03:00 CET 2h 22min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Sun 2026-11-15 03:10:19 CET 4 days Sun 2026-11-08 03:10:44 CET - xfs_scrub_all.timer xfs_scrub_all.service
Sun 2026-11-15 03:10:39 CET 4 days Sun 2026-11-08 03:10:44 CET - e2scrub_all.timer e2scrub_all.service
Mon 2026-11-16 00:45:50 CET 5 days Mon 2026-11-09 01:32:30 CET 1 day 6h ago fstrim.timer fstrim.service
12 timers listed.
### end
2026-11-10T07:25:05Z
@@ -0,0 +1,537 @@
### identity
tester-1
2026-11-10T07:26:54Z
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
### df bytes
Filesystem Type 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-vm--9201--disk--0 ext4 33501757440 1182224384 30584770560 4% /
/dev/mapper/pve-vm--9201--disk--1 ext4 73793978368 8987340800 61031776256 13% /var/lib/felhom
/dev/mapper/pve-root ext4 62903865344 21642317824 38032982016 37% /mnt/felhom-drives
/dev/sdb ext4 52521566208 13112242176 36708192256 27% /mnt/felhom-drives/adatlemez
### df inodes
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pve-vm--9201--disk--0 2097152 24793 2072359 2% /
/dev/mapper/pve-vm--9201--disk--1 4587520 245891 4341629 6% /var/lib/felhom
/dev/mapper/pve-root 3923968 79581 3844387 3% /mnt/felhom-drives
/dev/sdb 3276800 193 3276607 1% /mnt/felhom-drives/adatlemez
### docker info
driver=overlay2 root=/var/lib/docker logdriver=json-file images=24 containers=10 running=10 server=29.8.2
### daemon.json
{
"features": { "containerd-snapshotter": false },
"live-restore": true,
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "3" }
}
total 12
drwxr-xr-x 2 root root 4096 2026-10-04 18:11 .
drwxr-xr-x 73 root root 4096 2026-11-10 07:03 ..
-rw-r--r-- 1 root root 161 2026-10-04 18:11 daemon.json
### docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 24 10 7.612GB 3.129GB (41%)
Containers 10 10 48.6MB 0B (0%)
Local Volumes 388 11 349MB 42.16kB (0%)
Build Cache 0 0 0B 0B
### containers (name image state logdriver logopts)
/bookstack lscr.io/linuxserver/bookstack:26.09.2@sha256:2fdeeb1a2028ce7cb1274696237e32b9f849e16b6d77314d08c78356a6016fca running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T04:30:37.914578409Z
/bookstack-db mariadb:12.3@sha256:d46e102fa8b9b72bfcdeb56f9dd538742092119dc8b11f31fe662605cd234cd9 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T04:30:37.807169359Z
/cloudflared cloudflare/cloudflared:2026.10.4 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-10-27T15:41:50.993866528Z
/felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.351.0 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T01:55:08.463694927Z
/filebrowser gtstef/filebrowser:1.5.8-stable running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-10-29T19:07:19.681019496Z
/paperless-postgres postgres:18-alpine@sha256:10a5c69a83035ab5239f5620d708520e8b28f5aa1f2b20143e6024532edadfb5 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T04:30:44.385917625Z
/paperless-redis redis:7-alpine@sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T04:30:44.385518151Z
/paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:3.4.1@sha256:1a08d8dfc2b51a08de09652dd86ad4d0c611c362382f944533ad38b73125fbc4 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T04:30:44.518139888Z
/privatebin privatebin/pdo:2.1.0@sha256:3400b0a4714d62161727a710c82a3e319cfea364ac6e591aed653f5bb6bb8055 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-10T04:30:55.650252549Z
/traefik traefik:v3.7.17 running json-file {"max-file":"3","max-size":"10m"} restart=0 created=2026-11-03T07:04:43.586810659Z
### container log files (bytes path) and total
5657717 2026-11-10 /var/lib/docker/containers/f97e429c269de6d43632dde90d2a15288c3db68a493a7130f2c61ffeb2e565d8/f97e429c269de6d43632dde90d2a15288c3db68a493a7130f2c61ffeb2e565d8-json.log
995218 2026-11-10 /var/lib/docker/containers/711c99c0882a4f205b572191e19d91f477ab3b956df4ae85d08025da48b623fd/711c99c0882a4f205b572191e19d91f477ab3b956df4ae85d08025da48b623fd-json.log
227801 2026-11-10 /var/lib/docker/containers/45d7a4eaa9143bb0f0a635af14e651ec075edf66e6006a0a8cfd59254946c74b/45d7a4eaa9143bb0f0a635af14e651ec075edf66e6006a0a8cfd59254946c74b-json.log
212452 2026-11-10 /var/lib/docker/containers/ea1383c240f18657a9ddb3a811e95c396e9191561600b6396f4cd9a2687dda97/ea1383c240f18657a9ddb3a811e95c396e9191561600b6396f4cd9a2687dda97-json.log
63745 2026-11-10 /var/lib/docker/containers/77c321c06798e98d6b2f1471da508f616f7b6f118bb3c33277295b369cd0d74e/77c321c06798e98d6b2f1471da508f616f7b6f118bb3c33277295b369cd0d74e-json.log
30323 2026-11-10 /var/lib/docker/containers/936aaa3ef351135cbb591deb7620c478f1a315281be578f1714a66e209c0a0a8/936aaa3ef351135cbb591deb7620c478f1a315281be578f1714a66e209c0a0a8-json.log
10638 2026-11-10 /var/lib/docker/containers/2c4dcb14458ae2fc318c2f22f856373f7a48ae7d122291e5a3cbffe0451a0fde/2c4dcb14458ae2fc318c2f22f856373f7a48ae7d122291e5a3cbffe0451a0fde-json.log
5168 2026-11-10 /var/lib/docker/containers/2d6afb8994b9e4afac3ab811e6820537fcb46fb0081949c947a16812bc608437/2d6afb8994b9e4afac3ab811e6820537fcb46fb0081949c947a16812bc608437-json.log
5104 2026-11-10 /var/lib/docker/containers/4eea751d3439e012a1af84da6066a5c4d87d9225285e7cc99b24bd3b94efcb23/4eea751d3439e012a1af84da6066a5c4d87d9225285e7cc99b24bd3b94efcb23-json.log
4600 2026-11-10 /var/lib/docker/containers/a0c362fb6eaaabf0e91c932d81555d24eb49d94496ce96bfa6852a19eee4bbef/a0c362fb6eaaabf0e91c932d81555d24eb49d94496ce96bfa6852a19eee4bbef-json.log
TOTAL bytes=7212766 files=10
### container writable layers (docker ps -as)
bookstack 28.3kB (virtual 454MB)
bookstack-db 2B (virtual 334MB)
cloudflared 0B (virtual 64.6MB)
felhom-controller 468kB (virtual 349MB)
filebrowser 59.7kB (virtual 195MB)
paperless-postgres 68B (virtual 304MB)
paperless-redis 0B (virtual 39.1MB)
paperless-webserver 48MB (virtual 2.53GB)
privatebin 0B (virtual 79.7MB)
traefik 0B (virtual 191MB)
### images
alpine:latest 320994c3b997 2026-09-17 20:37:20 +0000 UTC 8.42MB
cloudflare/cloudflared:2026.10.3 1475624a6d23 2026-10-21 13:26:38 +0000 UTC 64.6MB
cloudflare/cloudflared:2026.10.4 472341c39b02 2026-10-26 11:27:24 +0000 UTC 64.6MB
ghcr.io/paperless-ngx/paperless-ngx:<none> 354dc6ebeb86 2026-10-31 05:27:36 +0000 UTC 2.48GB
ghcr.io/paperless-ngx/paperless-ngx:<none> 623d64064a8d 2026-04-27 02:45:05 +0000 UTC 1.42GB
gitea.dooplex.hu/admin/felhom-controller:0.351.0 ed7f096c3ff7 2026-11-09 12:14:14 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-controller:0.351.0-rc.1 485f658c0460 2026-11-09 12:09:35 +0000 UTC 348MB
gitea.dooplex.hu/admin/felhom-samba:1.1.0 8e21163a1646 2026-07-20 11:36:16 +0000 UTC 103MB
gtstef/filebrowser:1.5.6-stable 96e26edeefc3 2026-09-04 21:33:58 +0000 UTC 195MB
gtstef/filebrowser:1.5.8-stable 0019084d4a6f 2026-10-04 21:15:40 +0000 UTC 195MB
hello-world:latest e2ac70e7319a 2026-03-23 21:33:59 +0000 UTC 10.1kB
lscr.io/linuxserver/bookstack:<none> b274bee3f111 2026-09-30 06:56:35 +0000 UTC 416MB
lscr.io/linuxserver/bookstack:<none> f6f94cac70ef 2026-10-26 22:30:17 +0000 UTC 453MB
mariadb:<none> 2215e9af9663 2026-10-26 23:24:58 +0000 UTC 334MB
mariadb:<none> dfff46ef3f9d 2026-09-16 03:26:24 +0000 UTC 334MB
postgres:<none> 57b23a0326ee 2026-10-21 22:56:40 +0000 UTC 304MB
postgres:<none> c293117fcecd 2026-09-17 21:29:50 +0000 UTC 304MB
privatebin/pdo:<none> 7aa03d985415 2026-08-08 06:24:54 +0000 UTC 73.2MB
privatebin/pdo:<none> d108bf546a18 2026-10-21 21:13:05 +0000 UTC 79.7MB
redis:<none> 98c6ca8f397f 2026-10-26 16:40:19 +0000 UTC 39.1MB
redis:<none> f84b0c467801 2026-09-17 21:32:46 +0000 UTC 39.1MB
traefik/whoami:v1.10.3 aeef15490f2b 2024-08-22 08:07:12 +0000 UTC 6.58MB
traefik:v3.7.16 3733df53e74c 2026-10-21 22:55:46 +0000 UTC 194MB
traefik:v3.7.17 a72cb2f4a210 2026-10-30 20:17:26 +0000 UTC 191MB
### images count / dangling
12
12
### images in use by a container
1 cloudflare/cloudflared:2026.10.4
1 ghcr.io/paperless-ngx/paperless-ngx:3.4.1
1 gitea.dooplex.hu/admin/felhom-controller:0.351.0
1 gtstef/filebrowser:1.5.8-stable
1 lscr.io/linuxserver/bookstack:26.09.2
1 mariadb:12.3
1 postgres:18-alpine
1 privatebin/pdo:2.1.0
1 redis:7-alpine
1 traefik:v3.7.17
### volumes (count, anonymous count, dangling count)
388
380
377
### volumes by size (system df -v, Local Volumes block)
Local Volumes space usage:
VOLUME NAME LINKS SIZE
1a0995ec1fa824099927c0cb37504b3ccd6d8e30a6f48f616dd1c9e27bd921c8 0 0B
77c9939676ec7e4c56d332d2f38b3454048758fbc9b70c144d255ebf78eff866 0 0B
c0f132dee2b24dbbafc6ff8ef94c40047973c328d50f19fbdc3c0d2a1009c1d0 0 0B
ed89b25f9b78775c9744c10a0b05214ecc61ee40a268917e0ffc05755c969e6c 0 0B
felhom-controller-data 1 74.9MB
paperless-ngx_paperless_postgres_data 1 72.46MB
6ccd394faddf2ff61ee0abbb95d95e6643f8daa31b922b35e47694b4f1a73a89 0 340B
8076fde747347156353daa1b46aa97fbc854d9e51c8754b3c833a6674fc8bcf2 0 0B
887e41bfc719ff6ad177e75a256ef9b1b53aa7db26077439c654796a468da364 0 0B
3573efc924d1f6ff9a44e8e9bf455bbf1ca809b8c49cbc468c54d29ef711c054 0 0B
3fdce3fd3489379cafe338eff1f282b30d61e687ac883d11cbb4c16c9cfb6d68 0 0B
79e589d058c34901e3ce366abfc34e9aee3431c12dc1685ad6dd70997e998d64 0 340B
98dc8db152bc07d7970d07e7898a7e490cece5ecf1aa47e495680652c313059a 0 0B
99c42826d08ce6243b9bf8a0f928a423771f044edb74735f360bd5bd5d2b08bf 0 340B
9f2ec3b0e3fb23885414604fb7da81ec0df88daea7d5a422a067d40f64791d24 0 340B
bad8fee32ac06ce5773a2fc31507636fb5c6ca0caf90a2fea8e7dda43103a8d6 0 0B
35eae9e23b7dbbe0068613f2a99308d3e8f5d81246b1956a14c84e932b7913de 0 0B
4022eb87f53a2ac0f1021fe60741a215f3325647e2692e6b9d4c1bba28ca5a63 0 340B
40adc1b526df2547bf7690b03f6b34655ede6496fc1fe3ba2b2a366e4c5d97a9 0 0B
51dc5eaee47e0d902546941ae5462363a57bab30f604f950f4f2efdb6c06dc94 0 340B
59e65f6aee1a077839c9ddc34902d3b18a6ec5491e6df8ab35bd23cc259c158e 0 0B
80691462fec8717413857aa76148b63b07fadd71ff38f8dbfc90cf3317b84437 0 0B
8c738b754e6adc44dcf89cbf54084d769c08e9bde2137b7da06de55e1565e945 0 340B
d26afd4d6a024e861081397e0733d659ab9df1777a0606246c6b203a3201306c 0 0B
0ed0950de66f59e390312976d2dcbc482fc46ccc6116eaedffefc8385ee569cd 0 340B
145536a1d1ffc7f44808aac78826cdb926aa2c7f5af4749df3dc8a2192827c3a 0 0B
31becedda54ffa01827a18a004c5bb5ee2b18a1bc97b1f42b90cc32f1b1e92ff 0 0B
7224dac5110be6bb73a640f813218c65497aca167f97423ec3ba7c5d573efa8e 0 340B
7e995104784c9154f7546b50227dfdc60bd4ada15ab6d4cd13ac677bbcb9921a 0 0B
a8fb644ad077f23bbabe269134ae675514811c30cb4aee2e1baa6fc60928a774 0 0B
b4c8a01fdc0d6890c8112d9a270de3fed5915011431ad8aee41cd480772c0b12 0 0B
bb4cb8ad03b92e415724e7b9ea371fa9a0719c3b244cf38e3465dc025f23e712 0 0B
597b93a8833088933ab7281c5df8b8a4e866620b1992e235d7b62bebe5209e6b 0 0B
930d24bdff15118065d54b712d82475d10c1a677886f47f7ac582b9a69a7ddaf 0 0B
c8d34ded435496db30ee00525fe00ec46d73c63e24341bd8835c080995283325 0 0B
ca98a3f6471eb0efab615e78dcc496f17fed147cfc573af648c184ff7e0aafa5 0 0B
cf172f6b323abc8225ac36b05f36c06da548f8edd5ce63c2c473173859edd618 0 0B
d72d9e389c168c257944b3dde0e807ed795fe77bcd36d755598b0536965df44a 0 0B
d8a67a033a39fe6f886903556f8f69c5d72cf13bf4860c2d50ecf808bcf84c81 0 340B
fd26547d4646042fead55ad21bafa7ac4ffa83b464adf7f0dba4385dabe7eb59 0 340B
0af1d7ba94faf62af4085edae6877f0eaf9e7d3327eea5c78ea7c1b374604e36 0 0B
1baf3ed3a09348f35f8149d153c64ad95e10ca08f53e3a10437cdc7a61dcc629 0 0B
2b823e6422beb7b278748ab90e7bd1fac8b6f841f433595c1f14c6ccbce8e3d9 0 0B
3cb0c23e4a4637e0b492706b713db22dcbe56011c53d834b2c1485f02a067996 0 0B
71307b4988994fea3c0e0dcb681d717715b112c9a49123ef54aa97f3cb4e2024 0 0B
7a29f347e972db28979f697dfd2107373cbe219633af4fb969f961cde572d492 0 0B
b22a97bce70026ce57c9b35a2397c00d883588c50b0f2011c36ae44643270590 0 0B
b45f16a60f568f6cbc272fc79fc55ab6c5014c25ebcec90f62c086b95acf4830 0 340B
05be881c3d6df732ac8accd17b8b60ae528f3d4dc3e82dbcc581ec0fc11788fd 0 0B
13cdc778636898ab0ae3f0608517b74b2d37f304ad27f38994c63bd2d1df88e1 0 0B
42c2250e9fe5b5f5cee8aed34443280b250141358580ac448324f03b7b164ffb 0 0B
4464ee2fd1ddf58e950a02d2ab0a8d08e01c616df16ebf34591c2cc927f09728 0 0B
4f9fe9cd611b046ebf408dbb4ab6bfd07b46064ddee5ce7dc42634c5b09828da 0 0B
92f78e44d608c3a070de162939af9248acfa2a1a6111d1a63fa68f0bc2acc597 0 0B
af2ef39dea8eac3871ee860874157389c1c509c3b70ff37c4b7d990f3cf0a9b7 0 0B
ce8c79b3a922c33549bc79ae6af5fe88e9a7200eee7915ae2b5811584261a320 0 0B
0dbbd609c9ffebb17db835a8e78c5e5055f4d406fd391f0a98df8da2acdc1c80 0 340B
2d6e67e3979b18ec184b09ec197bb175002f029c1ed59442c6f1817823236747 0 0B
53d45c8f089f14a9fdc3153ec9d932198002b4be0be58db547f56a6cd8c8317a 0 0B
93e3db6eef0cf8c3ac5beb3c6f966a9beaf4f176197746073fd4b1371d0afc4c 0 340B
af60110e4834043a76b333db1328a5c8040956a10e3b8843c26bdb20ab3e92d1 0 0B
d1c8823aebe8a3eb6a20242b685e466383f4a32b79b2b30c7d5fddb6d4ede325 0 0B
dc22052c15733d8881dacf85853c3c4fad7e20118601f7d29e941e76bae5b385 0 340B
12e7dd9792b6b1b7ea35e979ceb76a2b35cfcc34cb806936cc7db4570763162c 0 340B
9d92ff119044918fb0b44e1dcab2c36139e5176bb918fedd601ff1bdf88c355e 0 0B
f4c4852307615064464fd6b59990f031d1e0e6e926d76b25f538e5d7f1156156 0 340B
ff9fcd38ad6080875b5c121ea94957128f35c5e3300f0192521a40ccf055c0bd 0 0B
2ca5312fdca5b0e1062663571c1d22daa365ab9199320711cab64a6128be9a37 0 0B
300b348c97bc82d526b924691f5c2ec2cde931a1b32e57f403b29a132ff60d69 0 0B
3f1aad2915904443bc5f66d4afa2a12ae988b556938847774f4f5ab0bcdf878f 0 340B
470c426c756309d8e535066912817e52a1b3268f90b7f4936aa248637f6ff99e 0 0B
f4573daac82885e39585604297044e7677d4edff30eb4cdde2edbfb2b1b2d8ae 0 0B
fb1a557fd1e7a61643134e0cbc57d7c54306330a5ec4f673f7871488e8072556 0 0B
bcde75a4cb74057174c7ee309f5630f11b3d23262a731492a58f35e7a177bf0a 0 0B
0209ac2f801f4cbed0c53e319d1f3c66befc9c309f311ac425e205f6b74c857c 0 340B
29d4c05b0a58d8e3ddf299dfb77c312ef4604ce3580caf565f8b6effb03866ed 0 340B
3868029eef55eee16c75b8709bbd78fca6e3b08f7634d9b6d519575468feac66 0 340B
a6b7afddb51c7f4d48fa163b77a468644d66551783655c6b46ab40543ec9991a 0 0B
a74bbcf91ea34ba1558e4f92c400d00d5d1b4654c032bb77240f226d99be3ac2 0 340B
f58282c0d4b95da155ffe4eefa3778fdb50617dd918a733c93c7eb8c50d39c56 0 340B
a59452d57d553008c895847a4676f1e54df99412453f0440aa822c954b80310a 0 0B
bedacba089c4f8576191fe6f4010bf082a6594a9c516964a049ff879c1ac2230 0 340B
1efb8232cca07e79814c3312fb01742a04b1b4e9f685f2a05850971ee266bcd7 0 0B
3665996bb9b3d30e859cce10cd616d5030337d583b5ef98d80cebe0ea80d5ecc 0 340B
846e350bc99e50cb60b251f543febef0626796689963bd3c7aa5d6570c544470 0 0B
8ab72a51248d3275d4608440778c0cd7bff4614307c4a27554be7b51d4c58ea3 0 340B
cbef4ff3508c5ae98df6d910b14b8c52d66ca5f259cf966297a6fd818e24d695 0 340B
36e89857418bdb0db5fee915bcc57a706bdae7e7ffad640d9d7fea0aa106755b 0 340B
41fdfcdd15766c36af2f607a6dff16a5680fa447ce6f12af7879d92dd5d9516e 0 0B
73a4d574d043ef1c494870136ba650cf32e385fe6dc534bddb65256613a039d7 0 0B
77bdadc3b823c8c25b302297910f3d1000dab75144678800868fe9d0fdace90f 0 340B
8408f9809c348b11e1dcd8b0c769c46c6363780870e273a5c7c560d7d828cd7f 0 340B
f92b58bff009e768c65bc67bd75d410fe063414261fc841dbda26c2bae82c95c 0 340B
fb755952415d40ee7d2fa7af507dbbe9bb377510b4448a48b292778cfd6c2c83 0 0B
46b21935a0fbb9b18bc56ca72ea9ad8ba6876b024952a8613030bfb5911b4662 0 0B
5bd3cb958f8a42194a5980266c30b9eac522455d2d15832680282c85a77f24e6 0 340B
7768f5ba39cca9930845be079a963995271015a2bb461d2cbe1ba4ca13a017d0 0 340B
99c2977efbdbfd8abdc20694df171b41e0a0774c9b2016ecca829643358a8808 0 0B
d34419ee5659f7d8fcc590cc329cb26b2ff13a45970cb5cbebb199365894c715 0 0B
f93ea05339c0f6c48f5cd9bf18f14922555ce62b7fd06e7563946d4134f0c444 0 0B
40b9deceaca0f4d72070ed243fb8e0c230c473fe374a4e53c45b7ec74c96ab35 0 0B
625acf7fb8862c1065709e9734ffe99a89f1c9e01d4ddfd5f65cbb8670f203b8 0 340B
e0a6cf7542476d85181416e3a5b54588e692d5e2a36cb7439a0d4fc05ea75cdf 0 340B
0c60ac019df853c9a423de4d8df848c9ba55997fe972df8daee0cadd9cd0a8e9 0 0B
0e80fb67355307fe7a308c935143ebcffe92ef48c19c6ac8f683f1be2cdcaac4 0 340B
1736f9912df81d3baa0948c81e0c29d060257e5dc71bb48825e7ca0f2e332ec9 0 0B
39cdd55cee60a501a32a7bd9f13fe01f0903cf725927ccadf19ea8077770ea9d 0 340B
b866b4bc219c3ebd57f965ec78fe878708cf5edf7d52d881971b7e68e86c6d01 0 0B
23c16e83962478de5f949ad9ca3c87e2bd4dc1411f93def1054bc99102211d0f 0 0B
341e4cdfefa6769d31b3e1bcf50f941a560fa91545fff410e1ba2e06d0362c91 0 340B
6131874f33d3c3887516e68de0368769a8cac5348ccacfceea5ac54273a36611 0 0B
9e2452e7d60e31de7b2c83ad26d0a702f7d32679ce5404752b49e60eb68318e9 0 340B
ce313d72e040dcc30a4df849212511a3688d0764ed2caa814afd55dd6f8cb525 0 0B
d51fe1ac2fc10b9abb6d6e333fda91bd794cf0b5a36fe2060e30b2758a5507d3 0 0B
4b176b8b9860a157765395f3b2116bf6a8551a811d5e407b37ad607bb1c728e0 0 0B
10bd4a099b7f4ba1a6e630c03ecedf87c8c22e0f3e876f667b97e5bfb38850e1 0 0B
36691204214bc453053aaf9feb8db2a52378a5a09f69048cb5cf06a6d01d1f2a 0 0B
3878ff257f55ead8bd4a48864fd4d0bb694e971bc4e5e111c293d1445e0c6641 0 0B
58261c801d3c4f9b3c40481bb73513a12234f791c465469f23a8ae1af8b65069 0 340B
ad89ce83ef046ca4d1d9fee168dd9ea1cdf33afdec91bc658bb9cd6aabfafadf 0 340B
filebrowser_filebrowser_data 1 136.6kB
7090e6a6fb4233d6c7e06dac2d9fa31591388a14e5ff9af22177692951131811 0 0B
95357e4c1efb7d954eaf3f8b7a308fcd65aae4eea22f1988c65bd84bb0898294 0 0B
1515080837b23fcdd92e79ad74419d377e43d20a86f886e1c14e9092c6e5116b 0 0B
1b1cfa1031168bfe8f253b181cd596b2e26a0bc2d1f2bab62a06837dc7d4a6fd 0 0B
6f9ef861cc7e00994738e64d5ab640673a66f617d6933277c7110f1135beb5ae 0 340B
700a81f777b70e910d2097a1b0705866b56b4702ac980b943c0cc8ad60c3bc5f 0 0B
f8d6f1387935ae6316af476ffe7dba55288cfeb4fa95a30fed6b6fe4156647e2 0 0B
paperless-ngx_paperless_redis_data 1 7.586MB
6e77a58f856657d4b3da205dd99884eb97555482fd917dd42747f6c321a93283 0 340B
8ec69f1b16f4102e45c0ef8af56c49a6fe27a277bf830b1855d11f1f01b67bd8 0 340B
108c4a90665e840f619f04c4d969694e9900ef7dbe374ced2c1e8da5a38b7eca 0 0B
18d8583ed522380a43e6c3bd5ff8f71f3f182e1035cb58297d875efe0a2a9cdd 0 0B
3e5457013af4ac4c3a406926b7f0f10a8596c71fc300a7b71a4ff02acd4e0a3a 0 0B
448a8cf6b579abd1476ab3bb5c7e2d887daa2afeaed121417e49689d098d107b 0 0B
a8491aec0f0db0b6e66ce597180761937693100a3102bb60f48a7833b467897c 0 0B
46828d783ac19d210fa6df0aa959189a6185e8c3c1529a2bfcad5d134142cc2c 0 0B
7f38810dd3c7448c20420eb76b371be9f329d723813fd0e93fa575f19f5452cf 0 340B
5692d2745f6c5f3c01479ea934451fbc4ba8982f89e60d4fc6a44fc22c25b8e1 0 0B
8d02518fe445ffe5073953f6c59a91a75447e0d824155356fdfb723cbfc2bcf9 0 0B
c65d6a2777b8cfe7e09d6835e6b1ed122c0e49abfb79c9102bea24d17a7977c0 0 0B
e4d4dd066ab630f703fe3753b5fb00f6c88fa47c344bf51214c598b88d8afca8 0 0B
026676d188d863de11370e48d6decb4a3c9f18e291d47e6b6f93134c04fd8b8a 0 340B
0307ed61b95007e0e6c16b610151f970d48ff44aaf27107e142642f1542b0aff 0 0B
33a0c6604fdac2e307dc0110b583ecff5420ec98a0bc56555ebc206573e10933 0 0B
6ae24f571bd8d8e033432c05d8991b04619f871982053072d245714b04b4ff8b 0 0B
6f06aeea4a112e6f6ee3ee401dd3acf2189f233ba8010d259c13706d643d521e 0 0B
### build cache
Build cache usage: 0B
CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED
### docker root du depth 1
8715157504 /var/lib/docker
8315240448 /var/lib/docker/overlay2
370810880 /var/lib/docker/volumes
21069824 /var/lib/docker/image
7671808 /var/lib/docker/containers
167936 /var/lib/docker/buildkit
159744 /var/lib/docker/network
12288 /var/lib/docker/plugins
8192 /var/lib/docker/tmp
4096 /var/lib/docker/swarm
4096 /var/lib/docker/runtimes
### journald
Archived and active journals take up 8M in the file system.
[Journal]
Nov 10 01:54:57 tester-1 systemd-journald[47]: Runtime Journal (/run/log/journal/553840b734374c4490bce2b9c3474a9e) is 8M, max 157.4M, 149.4M free.
Nov 10 01:54:57 tester-1 systemd-journald[47]: Runtime Journal (/run/log/journal/553840b734374c4490bce2b9c3474a9e) is 8M, max 157.4M, 149.4M free.
### /var/log
114688 /var/log
32768 /var/log/apt
4096 /var/log/private
### apt
95719424 /var/cache/apt
96104448 /var/lib/apt/lists
0
### tmp
0 /tmp
20480 /var/tmp
2026-11-10 40 /tmp/.ICE-unix
2026-11-10 40 /tmp/.X11-unix
2026-11-10 40 /tmp/.XIM-unix
2026-11-10 40 /tmp/.font-unix
2026-11-10 4096 /var/tmp/network
2026-11-10 4096 /var/tmp/systemd-private-9c0247fd2659403fbafc3e8edacc25e7-systemd-logind.service-64Jtca
2026-11-10 60 /tmp/systemd-private-9c0247fd2659403fbafc3e8edacc25e7-systemd-logind.service-IVpm4X
entries=7
### felhom dirs
/etc/felhom-bootstrap
/etc/felhom-controller-image
/opt/docker
/var/lib/felhom
### root filesystem top 45 to depth 4
1179938816 /
907108352 /usr
335069184 /usr/bin
266764288 /var
266133504 /usr/lib
182730752 /usr/share
165650432 /var/lib
154402816 /usr/lib/x86_64-linux-gnu
106651648 /usr/libexec
106475520 /usr/libexec/docker
105725952 /usr/libexec/docker/cli-plugins
99500032 /var/cache
96387072 /var/lib/apt
96104448 /var/lib/apt/lists
95719424 /var/cache/apt
67100672 /var/lib/dpkg
66502656 /usr/share/locale
30691328 /usr/share/doc
28700672 /usr/lib/python3.13
26943488 /usr/lib/x86_64-linux-gnu/perl
25632768 /usr/lib/git-core
23773184 /usr/share/perl
23769088 /usr/share/perl/5.40.1
23347200 /usr/lib/udev
17883136 /usr/share/man
16855040 /usr/share/i18n
16412672 /usr/sbin
13271040 /usr/share/i18n/locales
10358784 /usr/lib/file
9625600 /var/lib/dpkg/info
9428992 /usr/lib/udev/hwdb.d
9031680 /usr/lib/python3
9027584 /usr/lib/python3/dist-packages
8601600 /usr/lib/x86_64-linux-gnu/gconv
7770112 /usr/share/terminfo
6914048 /usr/lib/x86_64-linux-gnu/systemd
6012928 /usr/lib/systemd
5705728 /usr/lib/x86_64-linux-gnu/perl-base
5439488 /usr/lib/python3.13/__pycache__
5259264 /usr/share/doc/git
5144576 /usr/share/man/man1
4128768 /etc
3661824 /usr/lib/openssh
3620864 /usr/lib/python3.13/lib-dynload
3567616 /usr/share/i18n/charmaps
sort: write failed: 'standard output': Broken pipe
sort: write error
### /var/lib/felhom depth 3 (data volume)
8985198592 /var/lib/felhom
8715161600 /var/lib/felhom/docker
8315240448 /var/lib/felhom/docker/overlay2
370810880 /var/lib/felhom/docker/volumes
270016512 /var/lib/felhom/sys_drive
270012416 /var/lib/felhom/sys_drive/felhom-data
269996032 /var/lib/felhom/sys_drive/felhom-data/backups
21069824 /var/lib/felhom/docker/image
7675904 /var/lib/felhom/docker/containers
167936 /var/lib/felhom/docker/buildkit
159744 /var/lib/felhom/docker/network
155648 /var/lib/felhom/docker/network/files
16384 /var/lib/felhom/lost+found
12288 /var/lib/felhom/sys_drive/felhom-data/userdata
12288 /var/lib/felhom/docker/plugins
8192 /var/lib/felhom/docker/tmp
4096 /var/lib/felhom/docker/tmp/moby-detect-rro3702549578
4096 /var/lib/felhom/docker/swarm
4096 /var/lib/felhom/docker/runtimes
4096 /var/lib/felhom/docker/plugins/tmp
4096 /var/lib/felhom/docker/plugins/storage
### /opt/docker depth 2
1851392 /opt/docker
1847296 /opt/docker/stacks
69632 /opt/docker/stacks/paperless-ngx
57344 /opt/docker/stacks/traefik
53248 /opt/docker/stacks/bookstack
45056 /opt/docker/stacks/privatebin
40960 /opt/docker/stacks/immich
36864 /opt/docker/stacks/sabnzbd
36864 /opt/docker/stacks/romm
32768 /opt/docker/stacks/wger
32768 /opt/docker/stacks/transmission
32768 /opt/docker/stacks/nextcloud
32768 /opt/docker/stacks/grocy
28672 /opt/docker/stacks/wishlist
28672 /opt/docker/stacks/wanderer
28672 /opt/docker/stacks/sparkyfitness
28672 /opt/docker/stacks/sonarr
28672 /opt/docker/stacks/radarr
28672 /opt/docker/stacks/qbittorrent
28672 /opt/docker/stacks/plex
28672 /opt/docker/stacks/nzbget
28672 /opt/docker/stacks/lubelogger
28672 /opt/docker/stacks/karakeep
28672 /opt/docker/stacks/jellyfin
28672 /opt/docker/stacks/dawarich
28672 /opt/docker/stacks/adventurelog
24576 /opt/docker/stacks/zipline
24576 /opt/docker/stacks/vikunja
24576 /opt/docker/stacks/vaultwarden
24576 /opt/docker/stacks/uptime-kuma
24576 /opt/docker/stacks/termix
24576 /opt/docker/stacks/tandoor
24576 /opt/docker/stacks/recipe-importer
24576 /opt/docker/stacks/rallly
24576 /opt/docker/stacks/radicale
24576 /opt/docker/stacks/papra
24576 /opt/docker/stacks/outline
24576 /opt/docker/stacks/navidrome
24576 /opt/docker/stacks/n8n
24576 /opt/docker/stacks/metube
24576 /opt/docker/stacks/mealie
24576 /opt/docker/stacks/komga
24576 /opt/docker/stacks/kimai
24576 /opt/docker/stacks/homebox
24576 /opt/docker/stacks/home-assistant
24576 /opt/docker/stacks/grimmory
24576 /opt/docker/stacks/grafana
24576 /opt/docker/stacks/gitea
24576 /opt/docker/stacks/ghost
24576 /opt/docker/stacks/docmost
### drives under /mnt/felhom-drives (depth 3 from each drive)
-- /mnt/felhom-drives/adatlemez
/dev/sdb 52521566208 13112242176 36708192256 27% /mnt/felhom-drives/adatlemez
13110120448 /mnt/felhom-drives/adatlemez
13109968896 /mnt/felhom-drives/adatlemez/backups
13010116608 /mnt/felhom-drives/adatlemez/backups/secondary
12494602240 /mnt/felhom-drives/adatlemez/backups/secondary/transmission
172351488 /mnt/felhom-drives/adatlemez/backups/secondary/plex
164175872 /mnt/felhom-drives/adatlemez/backups/secondary/bookstack
114933760 /mnt/felhom-drives/adatlemez/backups/secondary/jellyfin
99840000 /mnt/felhom-drives/adatlemez/backups/primary
99835904 /mnt/felhom-drives/adatlemez/backups/primary/paperless-ngx
41418752 /mnt/felhom-drives/adatlemez/backups/secondary/sonarr
16568320 /mnt/felhom-drives/adatlemez/backups/secondary/radarr
6000640 /mnt/felhom-drives/adatlemez/backups/secondary/prowlarr
102400 /mnt/felhom-drives/adatlemez/userdata
81920 /mnt/felhom-drives/adatlemez/userdata/media
61440 /mnt/felhom-drives/adatlemez/backups/secondary/privatebin
45056 /mnt/felhom-drives/adatlemez/appdata
40960 /mnt/felhom-drives/adatlemez/appdata/paperless
32768 /mnt/felhom-drives/adatlemez/appdata/paperless/media
16384 /mnt/felhom-drives/adatlemez/userdata/media/transmission
12288 /mnt/felhom-drives/adatlemez/userdata/media/torrents
4096 /mnt/felhom-drives/adatlemez/userdata/roms-bios
4096 /mnt/felhom-drives/adatlemez/userdata/roms
4096 /mnt/felhom-drives/adatlemez/userdata/media/usenet
4096 /mnt/felhom-drives/adatlemez/userdata/media/tv
4096 /mnt/felhom-drives/adatlemez/userdata/media/podcasts
4096 /mnt/felhom-drives/adatlemez/userdata/media/photos
4096 /mnt/felhom-drives/adatlemez/userdata/media/nzbget
4096 /mnt/felhom-drives/adatlemez/userdata/media/music
4096 /mnt/felhom-drives/adatlemez/userdata/media/movies
4096 /mnt/felhom-drives/adatlemez/userdata/media/metube
4096 /mnt/felhom-drives/adatlemez/userdata/media/grimmory
4096 /mnt/felhom-drives/adatlemez/userdata/media/comics
4096 /mnt/felhom-drives/adatlemez/userdata/media/books
4096 /mnt/felhom-drives/adatlemez/userdata/media/audiobooks
4096 /mnt/felhom-drives/adatlemez/userdata/downloads
4096 /mnt/felhom-drives/adatlemez/userdata/documents
4096 /mnt/felhom-drives/adatlemez/backups/offsite-restore
4096 /mnt/felhom-drives/adatlemez/backups/offsite-proof
4096 /mnt/felhom-drives/adatlemez/appdata/paperless/export
-- /mnt/felhom-drives/media_a
/dev/mapper/pve-root 62903865344 21642317824 38032982016 37% /mnt/felhom-drives
4096 /mnt/felhom-drives/media_a
-- /mnt/felhom-drives/media_b
/dev/mapper/pve-root 62903865344 21642317824 38032982016 37% /mnt/felhom-drives
4096 /mnt/felhom-drives/media_b
-- /mnt/felhom-drives/media_c
/dev/mapper/pve-root 62903865344 21642317824 38032982016 37% /mnt/felhom-drives
4096 /mnt/felhom-drives/media_c
-- /mnt/felhom-drives/media_d
/dev/mapper/pve-root 62903865344 21642317824 38032982016 37% /mnt/felhom-drives
4096 /mnt/felhom-drives/media_d
### timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2026-11-10 17:36:08 UTC 10h Mon 2026-11-09 18:35:38 UTC - apt-daily.timer apt-daily.service
Wed 2026-11-11 00:00:00 UTC 16h Tue 2026-11-10 00:00:57 UTC - dpkg-db-backup.timer dpkg-db-backup.service
Wed 2026-11-11 00:29:06 UTC 17h Tue 2026-11-10 00:58:57 UTC - logrotate.timer logrotate.service
Wed 2026-11-11 02:10:36 UTC 18h Tue 2026-11-10 02:10:36 UTC 5h 16min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Wed 2026-11-11 06:01:19 UTC 22h Tue 2026-11-10 06:14:46 UTC 1h 12min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Wed 2026-11-11 10:15:34 UTC 1 day 2h Tue 2026-11-10 00:09:57 UTC - man-db.timer man-db.service
Sun 2026-11-15 03:10:54 UTC 4 days Sun 2026-11-08 03:10:38 UTC - e2scrub_all.timer e2scrub_all.service
- - - - fstrim.timer fstrim.service
8 timers listed.
### cron
/etc/cron.d:
e2scrub_all
/etc/cron.daily:
apt-compat
dpkg
logrotate
man-db
/etc/cron.weekly:
man-db
### end
2026-11-10T07:27:09Z
@@ -0,0 +1,449 @@
### identity
felhom
2026-11-10T07:33:37Z
### in flight
(end in flight)
### agent state dir (depth 2, bytes)
509444096 /var/lib/felhom-agent
509177856 /var/lib/felhom-agent/selfupdate
73728 /var/lib/felhom-agent/os
24576 /var/lib/felhom-agent/units
24576 /var/lib/felhom-agent/guests
20480 /var/lib/felhom-agent/wg
20480 /var/lib/felhom-agent/guests/9201
16384 /var/lib/felhom-agent/felhom-sshd
12288 /var/lib/felhom-agent/pbsdr
4096 /var/lib/felhom-agent/escrow-stage
### selfupdate listing
total 497248
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-07 23:13 .
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10 05:33 ..
-rwxr-xr-x 1 felhom-agent felhom-agent 14493513 2026-10-05 07:25 felhom-agent-0.144.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14494141 2026-10-05 07:56 felhom-agent-0.144.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14494291 2026-10-05 09:31 felhom-agent-0.145.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14504200 2026-10-05 12:27 felhom-agent-0.146.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14504898 2026-10-05 19:12 felhom-agent-0.147.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14515292 2026-10-06 00:42 felhom-agent-0.148.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14569321 2026-10-06 11:51 felhom-agent-0.149.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14590727 2026-10-07 09:06 felhom-agent-0.150.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14631775 2026-10-07 13:51 felhom-agent-0.151.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14715119 2026-10-07 15:31 felhom-agent-0.152.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14719868 2026-10-07 19:03 felhom-agent-0.153.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14731041 2026-10-09 07:18 felhom-agent-0.154.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14783135 2026-10-13 20:46 felhom-agent-0.155.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14820843 2026-10-14 14:23 felhom-agent-0.156.1
-rwxr-xr-x 1 felhom-agent felhom-agent 14874944 2026-10-15 21:16 felhom-agent-0.157.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14914581 2026-10-16 04:02 felhom-agent-0.158.0
-rwxr-xr-x 1 felhom-agent felhom-agent 14919180 2026-10-16 04:17 felhom-agent-0.158.1
-rwxr-xr-x 1 felhom-agent felhom-agent 16007680 2026-10-17 17:38 felhom-agent-0.159.1
-rwxr-xr-x 1 felhom-agent felhom-agent 16007752 2026-10-18 14:23 felhom-agent-0.159.2
-rwxr-xr-x 1 felhom-agent felhom-agent 16071264 2026-10-19 03:54 felhom-agent-0.160.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16082682 2026-10-20 08:54 felhom-agent-0.161.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16094507 2026-10-22 09:57 felhom-agent-0.162.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16183935 2026-10-23 08:57 felhom-agent-0.163.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16272301 2026-10-24 09:32 felhom-agent-0.164.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16286780 2026-10-25 17:15 felhom-agent-0.165.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16291150 2026-10-26 09:16 felhom-agent-0.166.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16296021 2026-10-27 10:31 felhom-agent-0.167.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16320123 2026-10-29 09:56 felhom-agent-0.168.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16350287 2026-10-29 12:56 felhom-agent-0.169.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16379051 2026-10-29 17:11 felhom-agent-0.170.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16389260 2026-10-30 10:12 felhom-agent-0.171.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16393888 2026-11-03 08:57 felhom-agent-0.172.0
-rwxr-xr-x 1 felhom-agent felhom-agent 16393888 2026-11-07 23:12 felhom-agent-0.172.1
### selfupdate subdirs listing
### agent state top-level listing (names, sizes, dates only)
total 124
drwxr-x--- 10 felhom-agent felhom-agent 4096 2026-11-10 05:33 .
drwxr-xr-x 46 root root 4096 2026-10-07 16:17 ..
-rw------- 1 felhom-agent felhom-agent 593 2026-11-10 05:33 backup-success-state.json
-rw------- 1 felhom-agent felhom-agent 295 2026-11-09 13:14 controller-swap-9201.json
-rw------- 1 felhom-agent felhom-agent 321 2026-10-17 10:57 drive-intents.json
drwx------ 2 felhom-agent felhom-agent 4096 2026-10-29 10:34 escrow-stage
drwx------ 2 felhom-agent felhom-agent 4096 2026-10-04 21:40 felhom-sshd
-rw------- 1 felhom-agent felhom-agent 308 2026-10-15 10:04 format-job.json
-rw------- 1 felhom-agent felhom-agent 67 2026-10-17 10:57 guest-binds.json
-rw------- 1 felhom-agent felhom-agent 242 2026-11-04 10:02 guest-disk-trim.json
drwx------ 3 felhom-agent felhom-agent 4096 2026-10-04 21:41 guests
-rw------- 1 felhom-agent felhom-agent 23656 2026-11-09 23:46 journal.log
-rw------- 1 felhom-agent felhom-agent 302 2026-11-03 13:35 line-test.json
-rw-r--r-- 1 felhom-agent felhom-agent 591 2026-10-04 21:40 local-api.crt
-rw------- 1 felhom-agent felhom-agent 227 2026-10-04 21:40 local-api.key
-rw------- 1 felhom-agent felhom-agent 82 2026-10-04 21:41 local-tokens.log
-rw------- 1 felhom-agent felhom-agent 4420 2026-11-07 23:27 nonces.log
drwx------ 2 felhom-agent felhom-agent 4096 2026-11-10 08:24 os
drwx------ 2 felhom-agent felhom-agent 4096 2026-10-04 21:55 pbsdr
-rw------- 1 felhom-agent felhom-agent 367 2026-11-09 23:46 restore-test-state.json
drwxr-x--- 2 felhom-agent felhom-agent 4096 2026-11-07 23:13 selfupdate
-rw------- 1 felhom-agent felhom-agent 32 2026-11-03 08:57 store-grant-seen.json
drwx------ 2 felhom-agent felhom-agent 4096 2026-10-15 10:04 units
drwx------ 2 felhom-agent felhom-agent 4096 2026-10-04 21:40 wg
### os-apply state
376832 /var/lib/felhom-os-apply
364544 /var/lib/felhom-os-apply/bundle-prev
159744 /var/lib/felhom-os-apply/bundle-prev/20261107T222759Z-before-0.172.1
159744 /var/lib/felhom-os-apply/bundle-prev/20261026T081610Z-before-0.166.0
40960 /var/lib/felhom-os-apply/bundle-prev/20261029T161156Z-before-0.170.0
4096 /var/lib/felhom-os-apply/agent-update
/var/lib/felhom-os-apply:
total 20
drwx------ 4 root root 4096 2026-11-07 23:27 .
drwxr-xr-x 46 root root 4096 2026-10-07 16:17 ..
drwxr-xr-x 2 root root 4096 2026-11-07 23:12 agent-update
drwxr-xr-x 5 root root 4096 2026-11-07 23:28 bundle-prev
-rw-r--r-- 1 root root 96 2026-11-07 23:27 nonces.json
/var/lib/felhom-os-apply/bundle-prev:
total 20
drwxr-xr-x 5 root root 4096 2026-11-07 23:28 .
drwx------ 4 root root 4096 2026-11-07 23:27 ..
drwxr-xr-x 3 root root 4096 2026-10-26 09:16 20261026T081610Z-before-0.166.0
drwxr-xr-x 3 root root 4096 2026-10-29 17:11 20261029T161156Z-before-0.170.0
drwxr-xr-x 3 root root 4096 2026-11-07 23:27 20261107T222759Z-before-0.172.1
### usr local bin + sbin
/usr/local/bin:
total 32032
drwxr-xr-x 2 root root 4096 2026-11-07 23:12 .
drwxr-xr-x 11 root root 4096 2026-05-21 13:31 ..
-rwxr-xr-x 1 root root 16393888 2026-11-07 23:12 felhom-agent
-rwxr-xr-x 1 root root 16393888 2026-11-03 08:57 felhom-agent.prev
/usr/local/sbin:
total 284
drwxr-xr-x 2 root root 4096 2026-11-07 23:27 .
drwxr-xr-x 11 root root 4096 2026-05-21 13:31 ..
-rwxr-xr-x 1 root root 6654 2026-10-04 21:40 felhom-backup-target-apply
-rwxr-xr-x 1 root root 32206 2026-09-18 20:41 felhom-bootstrap.sh
-rwxr-xr-x 1 root root 8687 2026-10-04 21:40 felhom-crash-guard
-rwxr-xr-x 1 root root 6816 2026-10-16 04:32 felhom-internal-place
-rwxr-xr-x 1 root root 3393 2026-10-04 21:40 felhom-mgmt-watchdog
-rwxr-xr-x 1 root root 5439 2026-10-16 04:32 felhom-mkfs-guarded
-rwxr-xr-x 1 root root 142720 2026-11-07 23:27 felhom-os-apply
-rwxr-xr-x 1 root root 9087 2026-10-04 21:40 felhom-pbs-apply
-rwxr-xr-x 1 root root 26062 2026-10-17 17:53 felhom-priv-apply
-rwxr-xr-x 1 root root 7806 2026-10-05 13:27 felhom-selfupdate-guarded
-rwxr-xr-x 1 root root 943 2026-10-04 21:40 felhom-shared-parent.sh
-rwxr-xr-x 1 root root 3844 2026-10-17 17:53 felhom-thin-check
-rwxr-xr-x 1 root root 5537 2026-10-14 14:53 felhom-vzdump-partial-clean
### dumps on drives
-- /var/lib/vz/dump
total 9874132
drwxr-xr-x 2 root root 4096 2026-11-10 05:33 .
drwxr-xr-x 7 root root 4096 2026-10-04 21:41 ..
-rw-r--r-- 1 root root 648135998 2026-10-04 21:40 vzdump-lxc-9100-2026_10_04-21_40_45.tar.zst
-rw-r--r-- 1 root root 1496 2026-11-08 05:28 vzdump-lxc-9201-2026_11_08-05_24_54.log
-rw-r--r-- 1 root root 3311751246 2026-11-08 05:28 vzdump-lxc-9201-2026_11_08-05_24_54.tar.zst
-rw-r--r-- 1 root root 16 2026-11-08 05:28 vzdump-lxc-9201-2026_11_08-05_24_54.tar.zst.notes
-rw-r--r-- 1 root root 1496 2026-11-09 05:33 vzdump-lxc-9201-2026_11_09-05_29_55.log
-rw-r--r-- 1 root root 3311185174 2026-11-09 05:33 vzdump-lxc-9201-2026_11_09-05_29_55.tar.zst
-rw-r--r-- 1 root root 16 2026-11-09 05:33 vzdump-lxc-9201-2026_11_09-05_29_55.tar.zst.notes
-rw-r--r-- 1 root root 1496 2026-11-10 05:33 vzdump-lxc-9201-2026_11_10-05_30_28.log
-rw-r--r-- 1 root root 2839981414 2026-11-10 05:33 vzdump-lxc-9201-2026_11_10-05_30_28.tar.zst
-rw-r--r-- 1 root root 16 2026-11-10 05:33 vzdump-lxc-9201-2026_11_10-05_30_28.tar.zst.notes
### template cache
/var/lib/vz/template/cache:
total 8
drwxr-xr-x 2 root root 4096 2026-05-21 13:20 .
drwxr-xr-x 4 root root 4096 2026-10-04 21:32 ..
/var/lib/vz/template/iso:
total 8
drwxr-xr-x 2 root root 4096 2026-05-21 13:20 .
drwxr-xr-x 4 root root 4096 2026-10-04 21:32 ..
### journald disk usage + limits
Archived and active journals take up 3.9G in the file system.
Nov 10 02:54:30 felhom systemd-journald[327]: Runtime Journal (/run/log/journal/5bd060f8d0764d4387a4760b4c744152) is 8M, max 78.7M, 70.7M free.
Nov 10 02:54:30 felhom systemd-journald[327]: System Journal (/var/log/journal/5bd060f8d0764d4387a4760b4c744152) is 3.9G, max 4G, 37.5M free.
### journal oldest + newest entry
2026-11-10T08:33:37+01:00 felhom systemd[1]: Started session-23.scope - Session 23 of User root.
2026-10-13T01:30:35+02:00 felhom sudo[3525612]: pam_unix(sudo:session): session closed for user root
2026-10-13T01:30:35+02:00 felhom sudo[3525615]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev
### journal files (count, oldest, newest)
files=81 bytes=5276434432 oldest=2026-10-13 newest=2026-11-10
### journal bytes exact
4288311296 /var/log/journal
### journal last hour: lines by identifier (top 12)
10419 "SYSLOG_IDENTIFIER":"sudo"
442 "SYSLOG_IDENTIFIER":"felhom-agent"
363 "SYSLOG_IDENTIFIER":"systemd"
66 "SYSLOG_IDENTIFIER":"postfix/smtp"
21 "SYSLOG_IDENTIFIER":"sshd-session"
17 "SYSLOG_IDENTIFIER":"systemd-logind"
11 "SYSLOG_IDENTIFIER":"postfix/qmgr"
4 "SYSLOG_IDENTIFIER":"(systemd)"
3 "SYSLOG_IDENTIFIER":"(sd-pam)"
3 "SYSLOG_IDENTIFIER":"CRON"
### journal last hour: total lines
11349
### journald conf files
[Journal]
### thin-check script
-- /usr/local/sbin/felhom-thin-check
#!/bin/bash
# felhom-thin-check — list the thin device ids a thin pool holds next to the ones LVM knows (R-1012). READ-ONLY: it
# names an orphan, it never removes one. Installed as /usr/local/sbin/felhom-thin-check (config bundle).
#
# felhom-thin-check <vg> <pool>
#
# A full host root while vzdump takes its snapshot can leave a thin snapshot inside the pool that LVM no longer knows
# (demo-hp 2026-10-14: `thin_ls` device 4, LVM only 1 and 2); from then on every snapshot backup of that guest fails
# („Creation of new snapshot 4 of device 2 failed") and nothing said why. The repair (`dmsetup message … delete <id>`)
# stays a HUMAN step, decided by the operator — this helper only reads.
#
# Reading the live pool's device list needs a metadata snapshot: the kernel's documented way to read metadata while
# the pool is in use (`reserve_metadata_snap`, `thin_ls --metadata-snap`, `release_metadata_snap`). It changes no data
# and no mapping; it is held for the length of one `thin_ls -o DEV` (well under a second on these pools). Rules:
# 1. the arguments are an LVM name pair; <vg>/<pool> is a thin pool (lv_attr starts with `t`);
# 2. only one run at a time (a lock file); a metadata snapshot that someone ELSE holds is never touched — the run
# refuses (exit 3) and does not release it;
# 3. LVM's list is read before AND after the pool's, so a volume created in between is not reported as an orphan
# (the caller treats an id as an orphan only when it is in the pool and in neither LVM list).
# Output, one record per line: lvm-before <id> <lv> | pool <id> | lvm-after <id> <lv> | ok
# Exit: 0 read, 1 refused (arguments / not a thin pool), 3 a metadata snapshot is already held or cannot be reserved,
# 4 the pool could not be read. Pinned by configs/test_felhom_thin_check.py.
set -u
LVS=/usr/sbin/lvs
DMSETUP=/usr/sbin/dmsetup
THIN_LS=/usr/sbin/thin_ls
LOCK=/run/lock/felhom-thin-check.lock
NAME_RE='^[A-Za-z0-9_+][A-Za-z0-9_.+-]{0,63}$'
die() { echo "felhom-thin-check: REFUSED: $1" >&2; exit "${2:-1}"; } # $1 the reason, $2 the exit code
lvm_ids() { # $1 tag → "<tag> <id> <lv>" for every thin volume of this pool
"$LVS" --noheadings --separator , -o lv_name,thin_id --select "pool_lv=$POOL" -- "$VG" 2>/dev/null |
awk -F, -v tag="$1" '{ gsub(/ /, ""); if ($2 ~ /^[0-9]+$/) print tag, $2, $1 }'
}
release() { "$DMSETUP" message "$TPOOL" 0 release_metadata_snap >/dev/null 2>&1; }
main() {
[[ $# -eq 2 ]] || die "usage: felhom-thin-check <vg> <pool>"
VG=$1 POOL=$2
local attr held ids
[[ "$VG" =~ $NAME_RE && "$POOL" =~ $NAME_RE ]] || die "not an LVM name pair"
attr=$("$LVS" --noheadings -o lv_attr -- "$VG/$POOL" 2>/dev/null | tr -d ' ')
[[ "$attr" == t* ]] || die "$VG/$POOL is not a thin pool"
DM="${VG//-/--}-${POOL//-/--}"
TPOOL="$DM-tpool"
TMETA="/dev/mapper/${DM}_tmeta"
exec 9>"$LOCK" || die "cannot open the lock file" 4
flock -w 30 9 || die "another check holds the lock" 3
# Field 7 of the thin-pool status is the held metadata root: "-" when no metadata snapshot exists.
held=$("$DMSETUP" status "$TPOOL" 2>/dev/null | awk '$3 == "thin-pool" { print $7 }')
[[ -n "$held" ]] || die "cannot read the status of $TPOOL" 4
[[ "$held" == "-" ]] || die "a metadata snapshot of $TPOOL is already held (by someone else) — not touched" 3
lvm_ids lvm-before
"$DMSETUP" message "$TPOOL" 0 reserve_metadata_snap >/dev/null 2>&1 || die "could not reserve a metadata snapshot of $TPOOL" 3
trap release EXIT
trap 'release; trap - EXIT; exit 4' INT TERM HUP
if ! ids=$("$THIN_LS" --metadata-snap --no-headers -o DEV "$TMETA" 2>/dev/null); then
die "thin_ls could not read $TMETA" 4
fi
release
trap - EXIT INT TERM HUP
awk '$1 ~ /^[0-9]+$/ { print "pool", $1 }' <<<"$ids"
lvm_ids lvm-after
echo ok
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then main "$@"; fi
-- /usr/local/sbin/felhom-vzdump-partial-clean
#!/bin/bash
# felhom-vzdump-partial-clean — remove ONE dead, half-written vzdump backup (R-1006). The only delete the agent's
# sudoers allows for backup files. Installed as /usr/local/sbin/felhom-vzdump-partial-clean (config bundle).
#
# felhom-vzdump-partial-clean <storage> <vmid> <name>
#
# A power cut during the night backup leaves `vzdump-lxc-<vmid>-<date>.tar.dat` (+ its `.tmp/` directory) on the
# storage; nothing removes it, every later backup is then skipped „not enough space", and a full host root breaks
# LVM's own metadata writes (power-cut-fixes-2026-10-14 C4). This helper removes such a leftover — and NOTHING else.
# It trusts nothing the agent says; every rule is checked here, as root, and any doubt REFUSES (exit 1):
# 1. the arguments: a storage id, a vmid, and a name that is exactly
# vzdump-(lxc|qemu)-<THAT vmid>-YYYY_MM_DD-HH_MM_SS.(tar.dat|vma.dat|tmp) — never a finished archive
# (.tar.zst, .vma.zst, …), never a .log/.notes, never another vmid;
# 2. the storage: its section in /etc/pve/storage.cfg is `dir: <storage>` with a `path`; the target is
# <path>/dump/<name>, not a symlink, and its real directory IS <path>/dump;
# 3. not in use: no vzdump runs on the host at all (a task `…:vzdump:…` or a `vzdump` command — a job for several
# guests carries no vmid in its task name), and no
# process holds the target (or, for the .tmp directory, anything under it) open or as its working directory
# (C2: after a killed worker, tar and zstd kept writing into the .tar.dat for 4 min 40 s, none named „vzdump");
# 4. dead: last modified before this boot (`btime` in /proc/stat), or untouched for at least QUIET_S seconds.
# Prints one line on success: „removed <name> <bytes> B (<why>)". Pinned by configs/test_felhom_vzdump_partial_clean.py.
set -u
STORAGE_CFG=/etc/pve/storage.cfg
PROC=/proc
QUIET_S=120
die() { echo "felhom-vzdump-partial-clean: REFUSED: $*" >&2; exit 1; }
storage_path() { # $1 storage id → prints the dir storage's path, or nothing
awk -v want="$1" '
/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); type = h[1]; id = h[2]; sub(/[ \t]+$/, "", id); in_s = (id == want); next }
in_s && /^[ \t]+path[ \t]/ { if (type == "dir") { sub(/^[ \t]+path[ \t]+/, ""); sub(/[ \t]+$/, ""); print; exit } }
' "$STORAGE_CFG"
}
storage_type() {
awk -v want="$1" '/^[a-z]+:[ \t]/ { split($0, h, /:[ \t]+/); id = h[2]; sub(/[ \t]+$/, "", id); if (id == want) { print h[1]; exit } }' "$STORAGE_CFG"
}
boot_time() { awk '$1 == "btime" { print $2 }' "$PROC/stat"; }
newest_mtime() { # a file's mtime, or the newest mtime of anything in a directory (the dir itself included)
if [[ -d "$1" ]]; then find "$1" -xdev -printf '%T@\n' 2>/dev/null | sort -n | tail -1 | cut -d. -f1
else stat -c %Y "$1"; fi
}
busy_reason() { # $1 vmid, $2 target → prints why it is in use, or nothing
local vmid=$1 target=$2 p link cmd
for p in "$PROC"/[0-9]*; do
[[ -r "$p/cmdline" ]] || continue
cmd=$(tr '\0' ' ' < "$p/cmdline" 2>/dev/null)
# ANY vzdump on the host refuses: a job for several guests runs as `task UPID:…:vzdump::…` (no vmid in it), and
# its tar may be between this guest's files. Waiting costs one backup run; deleting a live one costs the backup.
if [[ "$cmd" == *":vzdump:"* ]] || [[ "$cmd" =~ (^|/)vzdump([[:space:]]|$) ]]; then
echo "a vzdump runs on this host (pid ${p##*/})"; return
fi
for link in "$p/cwd" "$p"/fd/*; do
local to
to=$(readlink "$link" 2>/dev/null) || continue
if [[ "$to" == "$target" || "$to" == "$target/"* ]]; then
echo "pid ${p##*/} has it open"; return
fi
done
done
}
main() {
local storage=${1:-} vmid=${2:-} name=${3:-}
[[ $# -eq 3 ]] || die "usage: felhom-vzdump-partial-clean <storage> <vmid> <name>"
[[ "$storage" =~ ^[A-Za-z][A-Za-z0-9_-]{0,63}$ ]] || die "bad storage id"
[[ "$vmid" =~ ^[1-9][0-9]{2,8}$ ]] || die "bad vmid"
[[ "$name" =~ ^vzdump-(lxc|qemu)-${vmid}-[0-9]{4}_[0-9]{2}_[0-9]{2}-[0-9]{2}_[0-9]{2}_[0-9]{2}\.(tar\.dat|vma\.dat|tmp)$ ]] \
|| die "name is not a half-written vzdump of guest $vmid: $name"
[[ "$(storage_type "$storage")" == "dir" ]] || die "storage $storage is not a dir storage"
local path; path=$(storage_path "$storage")
[[ "$path" == /* ]] || die "storage $storage has no path"
local dump="$path/dump" target="$path/dump/$name"
[[ -e "$target" ]] || die "no such entry: $target"
[[ ! -L "$target" ]] || die "a symlink: $target"
[[ "$(realpath -e "$dump")" == "$(realpath -e "$(dirname "$target")")" ]] || die "not directly in $dump"
if [[ "$name" == *.tmp ]]; then [[ -d "$target" ]] || die ".tmp is not a directory"
else [[ -f "$target" ]] || die "not a regular file"; fi
local busy; busy=$(busy_reason "$vmid" "$target")
[[ -z "$busy" ]] || die "in use: $busy"
local bt mt now why
bt=$(boot_time); mt=$(newest_mtime "$target"); now=$(date +%s)
[[ "$bt" =~ ^[0-9]+$ && "$mt" =~ ^[0-9]+$ ]] || die "cannot read the boot time or the modification time"
if (( mt < bt )); then why="written before this boot"
elif (( now - mt >= QUIET_S )); then why="untouched for $(( now - mt )) s, no vzdump for $vmid, nothing holds it"
else die "modified $(( now - mt )) s ago — may still be written"; fi
local size; size=$(du -sb "$target" 2>/dev/null | cut -f1)
if [[ -d "$target" ]]; then rm -rf --one-file-system -- "$target"; else rm -f -- "$target"; fi
[[ ! -e "$target" ]] || die "remove failed: $target"
echo "removed $name ${size:-?} B ($why)"
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
main "$@"
fi
### units mentioning thin-check / partial-clean
/etc/sudoers.d/felhom-agent
### lvm thin settings
thin_pool_autoextend_threshold=100
thin_pool_autoextend_percent=20
monitoring=1
active
### agent config: selected non-secret keys
-- /etc/felhom-agent/agent.json
"local_backup_target": "local"
retention": 3
"keep_last": 0
### vzdump.conf + storage prune settings
1:dir: local
2: path /var/lib/vz
3: content iso,vztmpl,backup,import
5:lvmthin: local-lvm
8: content rootdir,images
10:pbs: felhom-pbs
13: content backup
### kernels
proxmox-kernel-7.0.14-20-pve un
proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20 1006525 ii
proxmox-kernel-7.0.14-22-pve un
proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22 1006522 ii
proxmox-kernel-7.0.2-6-pve un
proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6 1005219 ii
proxmox-kernel-helper 9.1.0+fde2 109 ii
7.0.14-22-pve
### apt history: kernel installs/removes
Start-Date: 2026-10-07 10:02:07
Start-Date: 2026-10-07 17:14:54
### apt autoremove config
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-20-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-20-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-20-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-22-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-22-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-22-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.2-6-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.2-6-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.2-6-pve-signed$";
### boot
Filesystem 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 62903865344 21642342400 38032957440 37% /
/dev/sda2 1071624192 9469952 1062154240 1% /boot/efi
total 306836
drwxr-xr-x 5 root root 4096 2026-10-07 17:15 .
drwxr-xr-x 19 root root 4096 2026-10-04 21:33 ..
-rw-r--r-- 1 root root 308363 2026-09-24 12:43 config-7.0.14-20-pve
-rw-r--r-- 1 root root 308363 2026-10-06 07:41 config-7.0.14-22-pve
-rw-r--r-- 1 root root 308342 2026-05-20 10:55 config-7.0.2-6-pve
drwxr-xr-x 3 root root 4096 1970-01-01 01:00 efi
drwxr-xr-x 6 root root 4096 2026-10-07 17:19 grub
-rw-r--r-- 1 root root 77469765 2026-10-07 10:02 initrd.img-7.0.14-20-pve
-rw-r--r-- 1 root root 77483301 2026-10-07 17:15 initrd.img-7.0.14-22-pve
-rw-r--r-- 1 root root 77431550 2026-10-06 04:35 initrd.img-7.0.2-6-pve
-rw-r--r-- 1 root root 151020 2024-11-17 17:43 memtest86+ia32.bin
-rw-r--r-- 1 root root 152064 2024-11-17 17:43 memtest86+ia32.efi
-rw-r--r-- 1 root root 155992 2024-11-17 17:43 memtest86+x64.bin
-rw-r--r-- 1 root root 157184 2024-11-17 17:43 memtest86+x64.efi
drwxr-xr-x 2 root root 4096 2026-10-07 17:15 pve
-rw-r--r-- 1 root root 10483512 2026-09-24 12:43 System.map-7.0.14-20-pve
-rw-r--r-- 1 root root 10483429 2026-10-06 07:41 System.map-7.0.14-22-pve
-rw-r--r-- 1 root root 10452992 2026-05-20 10:55 System.map-7.0.2-6-pve
-rw-r--r-- 1 root root 16284776 2026-09-24 12:43 vmlinuz-7.0.14-20-pve
-rw-r--r-- 1 root root 16288872 2026-10-06 07:41 vmlinuz-7.0.14-22-pve
-rw-r--r-- 1 root root 16227432 2026-05-20 10:55 vmlinuz-7.0.2-6-pve
### pve tasks / rrd
3866624 /var/log/pve
4710400 /var/lib/rrdcached
877
### root dir extras
102400 /root
16384 /root/.ssh
12288 /root/.cache
### df now
Filesystem 1B-blocks Used Avail Use% Mounted on
/dev/mapper/pve-root 62903865344 21642342400 38032957440 37% /
data 121521569792B 13.10 1.84
root 64235765760B
swap 8254390272B
vm-9201-disk-0 34359738368B 7.03
vm-9201-disk-1 75161927680B 17.96
### end
2026-11-10T07:33:38Z
@@ -0,0 +1,526 @@
### identity
felhom
2026-11-10T07:25:27Z
pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.14-22-pve)
felhom-agent 0.172.1
### in-flight (must be empty)
### lsblk
NAME SIZE TYPE FSTYPE MOUNTPOINT MODEL TRAN
sda 214748364800 disk QEMU HARDDISK
|-sda1 1031168 part
|-sda2 1073741824 part vfat /boot/efi
`-sda3 213673557504 part LVM2_member
|-pve-swap 8254390272 lvm swap [SWAP]
|-pve-root 64235765760 lvm ext4 /
|-pve-data_tmeta 1237319680 lvm
| `-pve-data-tpool 121521569792 lvm
| |-pve-data 121521569792 lvm
| |-pve-vm--9201--disk--0 34359738368 lvm ext4
| `-pve-vm--9201--disk--1 75161927680 lvm ext4
`-pve-data_tdata 121521569792 lvm
`-pve-data-tpool 121521569792 lvm
|-pve-data 121521569792 lvm
|-pve-vm--9201--disk--0 34359738368 lvm ext4
`-pve-vm--9201--disk--1 75161927680 lvm ext4
sdb 53687091200 disk ext4 /mnt/adatlemez QEMU HARDDISK
### df bytes
Filesystem Type 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-root ext4 62903865344 21642309632 38032990208 37% /
/dev/sda2 vfat 1071624192 9469952 1062154240 1% /boot/efi
/dev/sdb ext4 52521566208 13112242176 36708192256 27% /mnt/adatlemez
/dev/fuse fuse 134217728 20480 134197248 1% /etc/pve
### df inodes
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pve-root 3923968 79581 3844387 3% /
/dev/sda2 0 0 0 - /boot/efi
/dev/sdb 3276800 193 3276607 1% /mnt/adatlemez
/dev/fuse 262144 38 262106 1% /etc/pve
### vgs
VG VSize VFree
pve 213670428672 17184063488
### lvs
LV VG Attr LSize Pool Data% Meta% MSize
data pve twi-aotz-- 121521569792 13.10 1.84 1237319680
root pve -wi-ao---- 64235765760
swap pve -wi-ao---- 8254390272
vm-9201-disk-0 pve Vwi-aotz-- 34359738368 data 7.03
vm-9201-disk-1 pve Vwi-aotz-- 75161927680 data 17.96
### thin pool autoextend + monitoring
Configuration node activation/thin_pool_autoextend_threshold not found
Configuration node activation/thin_pool_autoextend_percent not found
Configuration node activation/monitoring not found
### pvesm status
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
felhom-pbs pbs active 0 0 0 0.00%
local dir active 61429556 21135068 37141592 34.41%
local-lvm lvmthin active 118673408 15546216 103127191 13.10%
### storage.cfg (no secrets live here; passwords are in /etc/pve/priv)
dir: local
path /var/lib/vz
content iso,vztmpl,backup,import
lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images
pbs: felhom-pbs
datastore felhom-offsite
server <redacted>
content backup
encryption-key <redacted>
fingerprint <redacted>
namespace tester-1
username <redacted>
### pvesm list per storage (count, bytes, oldest, newest)
felhom-pbs total entries=2 bytes=17985092696
felhom-pbs type=backup entries=2 bytes=17985092696
local total entries=4 bytes=10111053832
local type=backup entries=4 bytes=10111053832
local-lvm total entries=2 bytes=109521666048
local-lvm type=rootdir entries=2 bytes=109521666048
### guest configs (disk + mountpoint lines only)
-- /etc/pve/lxc/9201.conf
memory: 4096
mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G
mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives
mp9: /var/lib/felhom-agent/guests/9201/bootstrap,mp=/etc/felhom-bootstrap,ro=1
rootfs: local-lvm:vm-9201-disk-0,size=32G
swap: 4096
### vzdump.conf + jobs (keep rules)
### journald
Archived and active journals take up 3.9G in the file system.
[Journal]
83
IDX BOOT ID FIRST ENTRY LAST ENTRY
0 bf6ac6018a8e4819beed5a725e3967f2 Tue 2026-11-10 02:54:30 CET Tue 2026-11-10 08:25:30 CET
### /var/log top
4302839808 /var/log
4288311296 /var/log/journal
8708096 /var/log/pveproxy
3866624 /var/log/pve
1290240 /var/log/ifupdown2
61440 /var/log/apt
12288 /var/log/runit
8192 /var/log/vzdump
4096 /var/log/samba
4096 /var/log/private
4096 /var/log/lxc
4096 /var/log/frr
4096 /var/log/corosync
4096 /var/log/chrony
4096 /var/log/ceph
### /var/log big files
75497472 2026-11-09 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000009f8976-00065fd432bed94d.journal
75497472 2026-11-09 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000009da9d0-00065fcb05408c4b.journal
75497472 2026-11-08 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000009bc6e1-00065fc1b772b774.journal
75497472 2026-11-08 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-000000000099e5f1-00065fb87f4df674.journal
75497472 2026-11-08 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000009805f1-00065faf7c6b60cd.journal
75497472 2026-11-07 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-0000000000962211-00065fa6325bc729.journal
75497472 2026-11-07 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-0000000000943d58-00065f9cdc8694e0.journal
75497472 2026-11-06 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000009258ab-00065f938adf3556.journal
75497472 2026-11-06 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000009074d8-00065f8a3ed48033.journal
75497472 2026-11-05 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000008e8f57-00065f80e226c43d.journal
75497472 2026-11-05 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000008cac59-00065f77a0455947.journal
75497472 2026-11-04 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000008ac69c-00065f6e45554053.journal
75497472 2026-11-04 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-000000000088e430-00065f65096faa95.journal
75497472 2026-11-03 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-000000000086fee7-00065f5bafae211a.journal
75497472 2026-11-03 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-0000000000853959-00065f53d75a6eda.journal
75497472 2026-11-03 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-0000000000835e88-00065f4ae2edade0.journal
75497472 2026-11-02 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-0000000000817ee9-00065f41af31af9e.journal
75497472 2026-11-02 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000007f9ee9-00065f3875175f58.journal
75497472 2026-11-01 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000007dbe87-00065f2f376391d7.journal
75497472 2026-11-01 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000007bdf7e-00065f26078bda5f.journal
75497472 2026-10-31 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-000000000079fddf-00065f1cbd51164b.journal
75497472 2026-10-31 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-0000000000781ee7-00065f1390e5f7b2.journal
75497472 2026-10-30 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000007640e1-00065f0a7ca45787.journal
75497472 2026-10-29 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000006c2653-00065ee4fb92f577.journal
75497472 2026-10-28 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000006a4628-00065edb56213992.journal
75497472 2026-10-28 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-000000000068656c-00065ed1b18f8b77.journal
75497472 2026-10-27 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-00000000006684ab-00065ec80a531f40.journal
75497472 2026-10-27 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-000000000064a144-00065ebe521f5abb.journal
75497472 2026-10-26 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-000000000062d9a8-00065eb5fa9c13dc.journal
75497472 2026-10-26 /var/log/journal/5bd060f8d0764d4387a4760b4c744152/system@df0318aa28b04eb496f376ec2f24c422-000000000060fe0f-00065eacab2e5445.journal
### logrotate configs present
alternatives
apt
btmp
ceph-common
chrony
corosync
dpkg
frr
pve
pve-firewall
wtmp
wtmpdb
### kernels installed
linux-image-7.0.14-20-pve-amd64 un
linux-image-7.0.14-22-pve-amd64 un
linux-image-7.0.2-6-pve-amd64 un
linux-image-amd64 un
linux-image-generic un
proxmox-kernel-7.0 7.0.14-22 31 ii
proxmox-kernel-7.0.14-20-pve un
proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20 1006525 ii
proxmox-kernel-7.0.14-22-pve un
proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22 1006522 ii
proxmox-kernel-7.0.2-6-pve un
proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6 1005219 ii
proxmox-kernel-helper 9.1.0+fde2 109 ii
pve-kernel-7.0 un
pve-kernel-helper un
7.0.14-22-pve
### proxmox-boot-tool
Manually selected kernels:
None.
Automatically selected kernels:
7.0.14-20-pve
7.0.14-22-pve
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
E: /etc/kernel/proxmox-boot-uuids does not exist.
### /boot
Filesystem 1B-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 62903865344 21642313728 38032986112 37% /
/dev/sda2 1071624192 9469952 1062154240 1% /boot/efi
330608640 /boot
16420864 /boot/grub
4096 /boot/pve
77483301 Oct 7 17:15 initrd.img-7.0.14-22-pve
77469765 Oct 7 10:02 initrd.img-7.0.14-20-pve
77431550 Oct 6 04:35 initrd.img-7.0.2-6-pve
16288872 Oct 6 07:41 vmlinuz-7.0.14-22-pve
16284776 Sep 24 12:43 vmlinuz-7.0.14-20-pve
16227432 May 20 10:55 vmlinuz-7.0.2-6-pve
10483512 Sep 24 12:43 System.map-7.0.14-20-pve
10483429 Oct 6 07:41 System.map-7.0.14-22-pve
10452992 May 20 10:55 System.map-7.0.2-6-pve
308363 Sep 24 12:43 config-7.0.14-20-pve
308363 Oct 6 07:41 config-7.0.14-22-pve
308342 May 20 10:55 config-7.0.2-6-pve
157184 Nov 17 2024 memtest86+x64.efi
155992 Nov 17 2024 memtest86+x64.bin
152064 Nov 17 2024 memtest86+ia32.efi
151020 Nov 17 2024 memtest86+ia32.bin
4096 Oct 7 17:19 grub
4096 Oct 7 17:15 pve
4096 Oct 7 17:15 .
4096 Oct 4 21:33 ..
4096 Jan 1 1970 efi
### /lib/modules
3053326336 /lib/modules
1018175488 /lib/modules/7.0.14-22-pve
1018175488 /lib/modules/7.0.14-20-pve
1016971264 /lib/modules/7.0.2-6-pve
### apt
92000256 /var/cache/apt
61833216 /var/lib/apt/lists
0
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-20-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-20-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-20-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-22-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-22-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-22-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.2-6-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.2-6-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.2-6-pve-signed$";
01autoremove
10pveapthook
20listchanges
70debconf
76pveconf
76pveproxy
99mmdebstrap
### apt autoremove candidates (dry read)
0
### tmp
0 /tmp
20480 /var/tmp
2026-11-10 40 /tmp/.ICE-unix
2026-11-10 40 /tmp/.X11-unix
2026-11-10 40 /tmp/.XIM-unix
2026-11-10 40 /tmp/.font-unix
2026-11-10 4096 /var/tmp/systemd-private-bf6ac6018a8e4819beed5a725e3967f2-chrony.service-PseIro
2026-11-10 60 /tmp/systemd-private-bf6ac6018a8e4819beed5a725e3967f2-systemd-logind.service-7mVQXv
8
### systemd-tmpfiles rules touching tmp
q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d
### /var/lib/vz
10111139840 /var/lib/vz
10111107072 /var/lib/vz/dump
12288 /var/lib/vz/template
8192 /var/lib/vz/snippets
4096 /var/lib/vz/template/iso
4096 /var/lib/vz/template/cache
4096 /var/lib/vz/import
4096 /var/lib/vz/images
-- /var/lib/vz/dump
648135998 2026-10-04 vzdump-lxc-9100-2026_10_04-21_40_45.tar.zst
1496 2026-11-08 vzdump-lxc-9201-2026_11_08-05_24_54.log
3311751246 2026-11-08 vzdump-lxc-9201-2026_11_08-05_24_54.tar.zst
16 2026-11-08 vzdump-lxc-9201-2026_11_08-05_24_54.tar.zst.notes
1496 2026-11-09 vzdump-lxc-9201-2026_11_09-05_29_55.log
3311185174 2026-11-09 vzdump-lxc-9201-2026_11_09-05_29_55.tar.zst
16 2026-11-09 vzdump-lxc-9201-2026_11_09-05_29_55.tar.zst.notes
1496 2026-11-10 vzdump-lxc-9201-2026_11_10-05_30_28.log
2839981414 2026-11-10 vzdump-lxc-9201-2026_11_10-05_30_28.tar.zst
16 2026-11-10 vzdump-lxc-9201-2026_11_10-05_30_28.tar.zst.notes
-- /var/lib/vz/template/cache
-- /var/lib/vz/template/iso
### felhom dirs (candidates)
/etc/felhom
/etc/felhom-agent
/etc/felhom-oob.nft
/etc/felhom-sshd
/run/felhom-sshd.pid
/var/lib/felhom-agent
/var/lib/felhom-crash-guard
/var/lib/felhom-install
/var/lib/felhom-kernel
/var/lib/felhom-os-apply
/var/log/felhom-bootstrap-postinst.log
-- /etc/felhom
24576 /etc/felhom
files=6 oldest=2026-10-04 newest=2026-11-07
-- /etc/felhom-agent
8192 /etc/felhom-agent
files=1 oldest=2026-10-04 newest=2026-10-04
-- /etc/felhom-oob.nft
4096 /etc/felhom-oob.nft
files=1 oldest=2026-10-04 newest=2026-10-04
-- /etc/felhom-sshd
24576 /etc/felhom-sshd
8192 /etc/felhom-sshd/authorized_keys
files=4 oldest=2026-10-04 newest=2026-10-05
-- /var/lib/felhom-agent
509444096 /var/lib/felhom-agent
509177856 /var/lib/felhom-agent/selfupdate
73728 /var/lib/felhom-agent/os
24576 /var/lib/felhom-agent/units
24576 /var/lib/felhom-agent/guests
20480 /var/lib/felhom-agent/wg
20480 /var/lib/felhom-agent/guests/9201
16384 /var/lib/felhom-agent/felhom-sshd
12288 /var/lib/felhom-agent/pbsdr
4096 /var/lib/felhom-agent/escrow-stage
files=68 oldest=2026-10-04 newest=2026-11-10
-- /var/lib/felhom-crash-guard
8192 /var/lib/felhom-crash-guard
files=1 oldest=2026-11-10 newest=2026-11-10
-- /var/lib/felhom-install
8192 /var/lib/felhom-install
files=1 oldest=2026-10-04 newest=2026-10-04
-- /var/lib/felhom-kernel
8192 /var/lib/felhom-kernel
files=1 oldest=2026-10-07 newest=2026-10-07
-- /var/lib/felhom-os-apply
376832 /var/lib/felhom-os-apply
364544 /var/lib/felhom-os-apply/bundle-prev
159744 /var/lib/felhom-os-apply/bundle-prev/20261107T222759Z-before-0.172.1
159744 /var/lib/felhom-os-apply/bundle-prev/20261026T081610Z-before-0.166.0
40960 /var/lib/felhom-os-apply/bundle-prev/20261029T161156Z-before-0.170.0
4096 /var/lib/felhom-os-apply/agent-update
files=4 oldest=2026-10-26 newest=2026-11-07
-- /var/log/felhom-bootstrap-postinst.log
4096 /var/log/felhom-bootstrap-postinst.log
files=1 oldest=2026-10-04 newest=2026-10-04
### felhom binaries + slots
4096 Nov 7 23:12 .
4096 May 21 13:31 ..
16393888 Nov 7 23:12 felhom-agent
16393888 Nov 3 08:57 felhom-agent.prev
4096 Nov 7 23:27 .
4096 May 21 13:31 ..
6654 Oct 4 21:40 felhom-backup-target-apply
32206 Sep 18 20:41 felhom-bootstrap.sh
8687 Oct 4 21:40 felhom-crash-guard
6816 Oct 16 04:32 felhom-internal-place
3393 Oct 4 21:40 felhom-mgmt-watchdog
5439 Oct 16 04:32 felhom-mkfs-guarded
142720 Nov 7 23:27 felhom-os-apply
9087 Oct 4 21:40 felhom-pbs-apply
26062 Oct 17 17:53 felhom-priv-apply
7806 Oct 5 13:27 felhom-selfupdate-guarded
943 Oct 4 21:40 felhom-shared-parent.sh
3844 Oct 17 17:53 felhom-thin-check
5537 Oct 14 14:53 felhom-vzdump-partial-clean
### root filesystem, top 70 dirs to depth 4
21640142848 /
15199895552 /var
10724442112 /var/lib
10111139840 /var/lib/vz
10111107072 /var/lib/vz/dump
6095261696 /usr
4565606400 /usr/lib
4302839808 /var/log
4288311296 /var/log/journal
4288303104 /var/log/journal/5bd060f8d0764d4387a4760b4c744152
3053326336 /usr/lib/modules
1095835648 /usr/share
1018175488 /usr/lib/modules/7.0.14-22-pve
1018175488 /usr/lib/modules/7.0.14-20-pve
1016971264 /usr/lib/modules/7.0.2-6-pve
595865600 /usr/lib/x86_64-linux-gnu
561766400 /usr/lib/firmware
509444096 /var/lib/felhom-agent
509177856 /var/lib/felhom-agent/selfupdate
338661376 /usr/bin
330608640 /boot
327843840 /usr/share/kvm
301502464 /usr/share/pve-edk2-firmware
169730048 /var/cache
160776192 /usr/lib/firmware/nvidia
118800384 /usr/share/locale
113082368 /usr/lib/firmware/amdgpu
92000256 /var/cache/apt
78229504 /usr/share/doc
73310208 /usr/lib/ceph
66158592 /usr/lib/x86_64-linux-gnu/proxmox-backup
62787584 /var/lib/apt
61833216 /var/lib/apt/lists
57892864 /var/cache/proxmox-backup
56111104 /usr/lib/python3
56107008 /usr/lib/python3/dist-packages
51236864 /usr/lib/ceph/denc
48840704 /usr/lib/x86_64-linux-gnu/perl5
46153728 /usr/lib/firmware/intel
45576192 /usr/lib/firmware/ath11k
43634688 /usr/sbin
37642240 /usr/share/pve-docs
34213888 /usr/lib/firmware/ath12k
33148928 /usr/local
32796672 /usr/local/bin
29851648 /var/lib/dpkg
28692480 /usr/lib/python3.13
28246016 /var/lib/dpkg/info
27885568 /usr/lib/firmware/mediatek
27578368 /usr/share/javascript
26943488 /usr/lib/x86_64-linux-gnu/perl
26587136 /usr/lib/grub
25718784 /usr/share/man
25358336 /usr/share/javascript/extjs
23773184 /usr/share/perl
23769088 /usr/share/perl/5.40.1
23715840 /usr/share/nmap
23490560 /usr/lib/udev
21630976 /usr/share/doc/pve-qemu-kvm
18313216 /usr/libexec
17694720 /usr/lib/frr
17686528 /usr/lib/podman
17391616 /usr/lib/x86_64-linux-gnu/samba
16855040 /usr/share/i18n
16420864 /boot/grub
15970304 /usr/lib/firmware/rtw89
14450688 /usr/share/perl5
14098432 /etc
13271040 /usr/share/i18n/locales
13160448 /usr/lib/grub/x86_64-efi
### root fs: file count per top dir
4942 /var
58804 /usr
18 /root
3 /home
0 /opt
1736 /etc
0 /srv
0 /tmp
### root home
102400 /root
16384 /root/.ssh
12288 /root/.cache
### other mounted filesystems, depth 2
-- /mnt/adatlemez
13110140928 /mnt/adatlemez
13110120448 /mnt/adatlemez/felhom-data
13109968896 /mnt/adatlemez/felhom-data/backups
102400 /mnt/adatlemez/felhom-data/userdata
45056 /mnt/adatlemez/felhom-data/appdata
16384 /mnt/adatlemez/lost+found
### pve task logs + rrd
3866624 /var/log/pve
4710400 /var/lib/rrdcached
4222976 /var/lib/pve-cluster
147456 /var/lib/pve-manager
877
2026-10-04
2026-11-10
### core dumps
4096 /var/lib/systemd/coredump
0
### swap
NAME TYPE SIZE USED PRIO
/dev/dm-0 partition 8254386176 141684736 -1
### timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2026-11-10 08:25:43 CET 1s Tue 2026-11-10 08:24:43 CET 58s ago felhom-mgmt-watchdog.timer felhom-mgmt-watchdog.service
Tue 2026-11-10 09:09:47 CET 44min Tue 2026-11-10 08:09:47 CET 15min ago felhom-crash-guard-check.timer felhom-crash-guard-check.service
Tue 2026-11-10 11:05:57 CET 2h 40min Mon 2026-11-09 05:25:20 CET - man-db.timer man-db.service
Wed 2026-11-11 00:00:00 CET 15h Tue 2026-11-10 00:00:05 CET - dpkg-db-backup.timer dpkg-db-backup.service
Wed 2026-11-11 00:54:03 CET 16h Tue 2026-11-10 00:42:19 CET - logrotate.timer logrotate.service
Wed 2026-11-11 01:00:26 CET 16h Tue 2026-11-10 03:05:00 CET 5h 20min ago pve-daily-update.timer pve-daily-update.service
Wed 2026-11-11 01:24:27 CET 16h Tue 2026-11-10 07:10:39 CET 1h 15min ago apt-daily.timer apt-daily.service
Wed 2026-11-11 03:09:30 CET 18h Tue 2026-11-10 03:09:30 CET 5h 16min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Wed 2026-11-11 06:21:56 CET 21h Tue 2026-11-10 06:06:30 CET 2h 19min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Sun 2026-11-15 03:10:23 CET 4 days Sun 2026-11-08 03:10:39 CET - e2scrub_all.timer e2scrub_all.service
Sun 2026-11-15 03:10:26 CET 4 days Sun 2026-11-08 03:11:23 CET - xfs_scrub_all.timer xfs_scrub_all.service
Mon 2026-11-16 00:54:57 CET 5 days Mon 2026-11-09 01:13:20 CET - fstrim.timer fstrim.service
12 timers listed.
### end
2026-11-10T07:25:41Z
@@ -0,0 +1,23 @@
guest:rootfs-used 1357545472 size=33501757440 2026-11-10T07:39:06Z
guest:data-volume-used:/var/lib/felhom 30090575872 size=73793978368 2026-11-10T07:39:06Z
guest:docker-images-layers:/var/lib/docker/overlay2 27008151552 images=22 untagged=21 2026-11-10T07:39:14Z
guest:docker-volumes:/var/lib/docker/volumes 1607647232 all=1914 anonymous=1887 unused=1885 2026-11-10T07:39:15Z
guest:container-logs:/var/lib/docker/containers/*/*-json.log* 26442837 files=27 containers=26 2026-11-10T07:39:15Z
guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T07:39:15Z
guest:controller-data-volume 145395712 NA 2026-11-10T07:39:15Z
guest:stacks:/opt/docker/stacks 2174976 69 2026-11-10T07:39:15Z
guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 1327214592 14 2026-11-10T07:39:15Z
guest:journal 8M storage=volatile 2026-11-10T07:39:15Z
guest:var-log:/var/log 253952 NA 2026-11-10T07:39:15Z
guest:apt-cache:/var/cache/apt 95723520 NA 2026-11-10T07:39:15Z
guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T07:39:15Z
guest:tmp:/tmp 0 5 2026-11-10T07:39:15Z
guest:var-tmp:/var/tmp 20480 2 2026-11-10T07:39:15Z
drive:/mnt/felhom-drives/hdd_1 used 322734333952 size=1006980812800 2026-11-10T07:39:15Z
drive:/mnt/felhom-drives/hdd_1/backups/primary 371859456 7 2026-11-10T07:39:15Z
drive:/mnt/felhom-drives/hdd_1/backups/secondary 919998464 8 2026-11-10T07:39:15Z
drive:/mnt/felhom-drives/hdd_1/backups/offsite-restore 4096 0 2026-11-10T07:39:15Z
drive:/mnt/felhom-drives/hdd_1/backups/offsite-proof 4096 0 2026-11-10T07:39:15Z
drive:/mnt/felhom-drives/hdd_1/kept 208642048 1 2026-11-10T07:39:15Z
drive:/mnt/felhom-drives/hdd_1/appdata 30277632 2 2026-11-10T07:39:15Z
drive:/mnt/felhom-drives/hdd_1/userdata 6164480 6 2026-11-10T07:39:15Z
1 guest:rootfs-used 1357545472 size=33501757440 2026-11-10T07:39:06Z
2 guest:data-volume-used:/var/lib/felhom 30090575872 size=73793978368 2026-11-10T07:39:06Z
3 guest:docker-images-layers:/var/lib/docker/overlay2 27008151552 images=22 untagged=21 2026-11-10T07:39:14Z
4 guest:docker-volumes:/var/lib/docker/volumes 1607647232 all=1914 anonymous=1887 unused=1885 2026-11-10T07:39:15Z
5 guest:container-logs:/var/lib/docker/containers/*/*-json.log* 26442837 files=27 containers=26 2026-11-10T07:39:15Z
6 guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T07:39:15Z
7 guest:controller-data-volume 145395712 NA 2026-11-10T07:39:15Z
8 guest:stacks:/opt/docker/stacks 2174976 69 2026-11-10T07:39:15Z
9 guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 1327214592 14 2026-11-10T07:39:15Z
10 guest:journal 8M storage=volatile 2026-11-10T07:39:15Z
11 guest:var-log:/var/log 253952 NA 2026-11-10T07:39:15Z
12 guest:apt-cache:/var/cache/apt 95723520 NA 2026-11-10T07:39:15Z
13 guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T07:39:15Z
14 guest:tmp:/tmp 0 5 2026-11-10T07:39:15Z
15 guest:var-tmp:/var/tmp 20480 2 2026-11-10T07:39:15Z
16 drive:/mnt/felhom-drives/hdd_1 used 322734333952 size=1006980812800 2026-11-10T07:39:15Z
17 drive:/mnt/felhom-drives/hdd_1/backups/primary 371859456 7 2026-11-10T07:39:15Z
18 drive:/mnt/felhom-drives/hdd_1/backups/secondary 919998464 8 2026-11-10T07:39:15Z
19 drive:/mnt/felhom-drives/hdd_1/backups/offsite-restore 4096 0 2026-11-10T07:39:15Z
20 drive:/mnt/felhom-drives/hdd_1/backups/offsite-proof 4096 0 2026-11-10T07:39:15Z
21 drive:/mnt/felhom-drives/hdd_1/kept 208642048 1 2026-11-10T07:39:15Z
22 drive:/mnt/felhom-drives/hdd_1/appdata 30277632 2 2026-11-10T07:39:15Z
23 drive:/mnt/felhom-drives/hdd_1/userdata 6164480 6 2026-11-10T07:39:15Z
@@ -0,0 +1,23 @@
guest:rootfs-used 1920802816 size=33501757440 2026-11-10T07:39:06Z
guest:data-volume-used:/var/lib/felhom 13990481920 size=73391005696 2026-11-10T07:39:06Z
guest:docker-images-layers:/var/lib/docker/overlay2 13676625920 images=41 untagged=7 2026-11-10T07:39:09Z
guest:docker-volumes:/var/lib/docker/volumes 247779328 all=368 anonymous=363 unused=363 2026-11-10T07:39:09Z
guest:container-logs:/var/lib/docker/containers/*/*-json.log* 17611662 files=6 containers=6 2026-11-10T07:39:09Z
guest:docker-buildcache:/var/lib/docker/buildkit 495616 NA 2026-11-10T07:39:09Z
guest:controller-data-volume 156348416 NA 2026-11-10T07:39:09Z
guest:stacks:/opt/docker/stacks 2990080 70 2026-11-10T07:39:09Z
guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 20480 0 2026-11-10T07:39:09Z
guest:journal 8M storage=volatile 2026-11-10T07:39:09Z
guest:var-log:/var/log 102400 NA 2026-11-10T07:39:09Z
guest:apt-cache:/var/cache/apt 273600512 NA 2026-11-10T07:39:09Z
guest:apt-lists:/var/lib/apt/lists 95510528 NA 2026-11-10T07:39:09Z
guest:tmp:/tmp 0 5 2026-11-10T07:39:09Z
guest:var-tmp:/var/tmp 20480 2 2026-11-10T07:39:09Z
drive:/mnt/felhom-drives/scratch_hdd used 322734333952 size=1006980812800 2026-11-10T07:39:09Z
drive:/mnt/felhom-drives/scratch_hdd/backups/primary 77488128 1 2026-11-10T07:39:09Z
drive:/mnt/felhom-drives/scratch_hdd/backups/secondary 4096 0 2026-11-10T07:39:09Z
drive:/mnt/felhom-drives/scratch_hdd/backups/offsite-restore 4096 0 2026-11-10T07:39:09Z
drive:/mnt/felhom-drives/scratch_hdd/backups/restore-undo 4096 0 2026-11-10T07:39:09Z
drive:/mnt/felhom-drives/scratch_hdd/kept 1355776 1 2026-11-10T07:39:09Z
drive:/mnt/felhom-drives/scratch_hdd/appdata 32768 2 2026-11-10T07:39:09Z
drive:/mnt/felhom-drives/scratch_hdd/userdata 33083392 18 2026-11-10T07:39:09Z
1 guest:rootfs-used 1920802816 size=33501757440 2026-11-10T07:39:06Z
2 guest:data-volume-used:/var/lib/felhom 13990481920 size=73391005696 2026-11-10T07:39:06Z
3 guest:docker-images-layers:/var/lib/docker/overlay2 13676625920 images=41 untagged=7 2026-11-10T07:39:09Z
4 guest:docker-volumes:/var/lib/docker/volumes 247779328 all=368 anonymous=363 unused=363 2026-11-10T07:39:09Z
5 guest:container-logs:/var/lib/docker/containers/*/*-json.log* 17611662 files=6 containers=6 2026-11-10T07:39:09Z
6 guest:docker-buildcache:/var/lib/docker/buildkit 495616 NA 2026-11-10T07:39:09Z
7 guest:controller-data-volume 156348416 NA 2026-11-10T07:39:09Z
8 guest:stacks:/opt/docker/stacks 2990080 70 2026-11-10T07:39:09Z
9 guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 20480 0 2026-11-10T07:39:09Z
10 guest:journal 8M storage=volatile 2026-11-10T07:39:09Z
11 guest:var-log:/var/log 102400 NA 2026-11-10T07:39:09Z
12 guest:apt-cache:/var/cache/apt 273600512 NA 2026-11-10T07:39:09Z
13 guest:apt-lists:/var/lib/apt/lists 95510528 NA 2026-11-10T07:39:09Z
14 guest:tmp:/tmp 0 5 2026-11-10T07:39:09Z
15 guest:var-tmp:/var/tmp 20480 2 2026-11-10T07:39:09Z
16 drive:/mnt/felhom-drives/scratch_hdd used 322734333952 size=1006980812800 2026-11-10T07:39:09Z
17 drive:/mnt/felhom-drives/scratch_hdd/backups/primary 77488128 1 2026-11-10T07:39:09Z
18 drive:/mnt/felhom-drives/scratch_hdd/backups/secondary 4096 0 2026-11-10T07:39:09Z
19 drive:/mnt/felhom-drives/scratch_hdd/backups/offsite-restore 4096 0 2026-11-10T07:39:09Z
20 drive:/mnt/felhom-drives/scratch_hdd/backups/restore-undo 4096 0 2026-11-10T07:39:09Z
21 drive:/mnt/felhom-drives/scratch_hdd/kept 1355776 1 2026-11-10T07:39:09Z
22 drive:/mnt/felhom-drives/scratch_hdd/appdata 32768 2 2026-11-10T07:39:09Z
23 drive:/mnt/felhom-drives/scratch_hdd/userdata 33083392 18 2026-11-10T07:39:09Z
@@ -0,0 +1,25 @@
host:rootfs-used 23968124928 size=41424257024 2026-11-10T07:39:05Z
host:thinpool-data-data 40878104956 size=57843646464 pct=70.67 2026-11-10T07:39:05Z
host:thinpool-data-metadata-pct NA pct=2.85 2026-11-10T07:39:05Z
host:dump:/var/lib/vz/dump 9750130688 archives=1 logs=10 all=13 2026-11-10T07:39:05Z
host:template-cache:/var/lib/vz/template 129974272 3 2026-11-10T07:39:05Z
host:dump:/mnt/hdd_1/dump 5099683840 archives=4 logs=3 all=10 2026-11-10T07:39:05Z
host:template-cache:/mnt/hdd_1/template NA NA 2026-11-10T07:39:05Z
host:journal:/var/log/journal 873717760 files=16 2026-11-10T07:39:05Z
host:var-log-other 14598144 pve-tasks=914 2026-11-10T07:39:05Z
host:rrdcached:/var/lib/rrdcached 64802816 NA 2026-11-10T07:39:05Z
host:kernels:/lib/modules 8148783104 kernels=8 2026-11-10T07:39:06Z
host:boot:/boot 823369728 initrd=8 2026-11-10T07:39:06Z
host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T07:39:06Z
host:apt-cache:/var/cache/apt 92004352 debs=0 2026-11-10T07:39:06Z
host:apt-lists:/var/lib/apt/lists 61833216 NA 2026-11-10T07:39:06Z
host:agent-state:/var/lib/felhom-agent 740171776 22 2026-11-10T07:39:06Z
host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 739938304 binaries=49 2026-11-10T07:39:06Z
host:agent-bak:/usr/local/bin/felhom-agent.bak-* 14284239 1 2026-11-10T07:39:06Z
host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T07:39:06Z
host:tmp:/tmp 0 6 2026-11-10T07:39:06Z
host:var-tmp:/var/tmp 28672 4 2026-11-10T07:39:06Z
host:root-home:/root 176128 NA 2026-11-10T07:39:06Z
host:swap-used 0 size=8589930496 2026-11-10T07:39:06Z
host:journal-lines-last-hour NA all=13946 sudo=12825 2026-11-10T07:39:06Z
host:journal-oldest-entry NA 2026-10-04T21:42:08+02:00 2026-11-10T07:39:06Z
1 host:rootfs-used 23968124928 size=41424257024 2026-11-10T07:39:05Z
2 host:thinpool-data-data 40878104956 size=57843646464 pct=70.67 2026-11-10T07:39:05Z
3 host:thinpool-data-metadata-pct NA pct=2.85 2026-11-10T07:39:05Z
4 host:dump:/var/lib/vz/dump 9750130688 archives=1 logs=10 all=13 2026-11-10T07:39:05Z
5 host:template-cache:/var/lib/vz/template 129974272 3 2026-11-10T07:39:05Z
6 host:dump:/mnt/hdd_1/dump 5099683840 archives=4 logs=3 all=10 2026-11-10T07:39:05Z
7 host:template-cache:/mnt/hdd_1/template NA NA 2026-11-10T07:39:05Z
8 host:journal:/var/log/journal 873717760 files=16 2026-11-10T07:39:05Z
9 host:var-log-other 14598144 pve-tasks=914 2026-11-10T07:39:05Z
10 host:rrdcached:/var/lib/rrdcached 64802816 NA 2026-11-10T07:39:05Z
11 host:kernels:/lib/modules 8148783104 kernels=8 2026-11-10T07:39:06Z
12 host:boot:/boot 823369728 initrd=8 2026-11-10T07:39:06Z
13 host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T07:39:06Z
14 host:apt-cache:/var/cache/apt 92004352 debs=0 2026-11-10T07:39:06Z
15 host:apt-lists:/var/lib/apt/lists 61833216 NA 2026-11-10T07:39:06Z
16 host:agent-state:/var/lib/felhom-agent 740171776 22 2026-11-10T07:39:06Z
17 host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 739938304 binaries=49 2026-11-10T07:39:06Z
18 host:agent-bak:/usr/local/bin/felhom-agent.bak-* 14284239 1 2026-11-10T07:39:06Z
19 host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T07:39:06Z
20 host:tmp:/tmp 0 6 2026-11-10T07:39:06Z
21 host:var-tmp:/var/tmp 28672 4 2026-11-10T07:39:06Z
22 host:root-home:/root 176128 NA 2026-11-10T07:39:06Z
23 host:swap-used 0 size=8589930496 2026-11-10T07:39:06Z
24 host:journal-lines-last-hour NA all=13946 sudo=12825 2026-11-10T07:39:06Z
25 host:journal-oldest-entry NA 2026-10-04T21:42:08+02:00 2026-11-10T07:39:06Z
@@ -0,0 +1,23 @@
guest:rootfs-used 1188171776 size=33501757440 2026-11-10T07:38:33Z
guest:data-volume-used:/var/lib/felhom 2470076416 size=263086084096 2026-11-10T07:38:33Z
guest:docker-images-layers:/var/lib/docker/overlay2 2258448384 images=11 untagged=5 2026-11-10T07:38:33Z
guest:docker-volumes:/var/lib/docker/volumes 136433664 all=8 anonymous=1 unused=1 2026-11-10T07:38:33Z
guest:container-logs:/var/lib/docker/containers/*/*-json.log* 9685130 files=9 containers=9 2026-11-10T07:38:33Z
guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T07:38:33Z
guest:controller-data-volume 84156416 NA 2026-11-10T07:38:33Z
guest:stacks:/opt/docker/stacks 1859584 68 2026-11-10T07:38:33Z
guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 47730688 5 2026-11-10T07:38:33Z
guest:journal 8M storage=volatile 2026-11-10T07:38:33Z
guest:var-log:/var/log 135168 NA 2026-11-10T07:38:33Z
guest:apt-cache:/var/cache/apt 95715328 NA 2026-11-10T07:38:33Z
guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T07:38:33Z
guest:tmp:/tmp 0 5 2026-11-10T07:38:33Z
guest:var-tmp:/var/tmp 20480 2 2026-11-10T07:38:33Z
drive:/mnt/felhom-drives/hdd_1 used 8143593472 size=983351140352 2026-11-10T07:38:33Z
drive:/mnt/felhom-drives/hdd_1/backups/primary 1391616000 5 2026-11-10T07:38:33Z
drive:/mnt/felhom-drives/hdd_1/backups/secondary 293728256 4 2026-11-10T07:38:33Z
drive:/mnt/felhom-drives/hdd_1/backups/offsite-restore 1654599680 2 2026-11-10T07:38:33Z
drive:/mnt/felhom-drives/hdd_1/backups/offsite-proof 4096 0 2026-11-10T07:38:33Z
drive:/mnt/felhom-drives/hdd_1/appdata 442085376 4 2026-11-10T07:38:33Z
drive:/mnt/felhom-drives/hdd_1/userdata 561152 6 2026-11-10T07:38:33Z
drive:/mnt/felhom-drives/hdd_1/shares 32768 1 2026-11-10T07:38:33Z
1 guest:rootfs-used 1188171776 size=33501757440 2026-11-10T07:38:33Z
2 guest:data-volume-used:/var/lib/felhom 2470076416 size=263086084096 2026-11-10T07:38:33Z
3 guest:docker-images-layers:/var/lib/docker/overlay2 2258448384 images=11 untagged=5 2026-11-10T07:38:33Z
4 guest:docker-volumes:/var/lib/docker/volumes 136433664 all=8 anonymous=1 unused=1 2026-11-10T07:38:33Z
5 guest:container-logs:/var/lib/docker/containers/*/*-json.log* 9685130 files=9 containers=9 2026-11-10T07:38:33Z
6 guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T07:38:33Z
7 guest:controller-data-volume 84156416 NA 2026-11-10T07:38:33Z
8 guest:stacks:/opt/docker/stacks 1859584 68 2026-11-10T07:38:33Z
9 guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 47730688 5 2026-11-10T07:38:33Z
10 guest:journal 8M storage=volatile 2026-11-10T07:38:33Z
11 guest:var-log:/var/log 135168 NA 2026-11-10T07:38:33Z
12 guest:apt-cache:/var/cache/apt 95715328 NA 2026-11-10T07:38:33Z
13 guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T07:38:33Z
14 guest:tmp:/tmp 0 5 2026-11-10T07:38:33Z
15 guest:var-tmp:/var/tmp 20480 2 2026-11-10T07:38:33Z
16 drive:/mnt/felhom-drives/hdd_1 used 8143593472 size=983351140352 2026-11-10T07:38:33Z
17 drive:/mnt/felhom-drives/hdd_1/backups/primary 1391616000 5 2026-11-10T07:38:33Z
18 drive:/mnt/felhom-drives/hdd_1/backups/secondary 293728256 4 2026-11-10T07:38:33Z
19 drive:/mnt/felhom-drives/hdd_1/backups/offsite-restore 1654599680 2 2026-11-10T07:38:33Z
20 drive:/mnt/felhom-drives/hdd_1/backups/offsite-proof 4096 0 2026-11-10T07:38:33Z
21 drive:/mnt/felhom-drives/hdd_1/appdata 442085376 4 2026-11-10T07:38:33Z
22 drive:/mnt/felhom-drives/hdd_1/userdata 561152 6 2026-11-10T07:38:33Z
23 drive:/mnt/felhom-drives/hdd_1/shares 32768 1 2026-11-10T07:38:33Z
@@ -0,0 +1,25 @@
host:rootfs-used 38093631488 size=100861726720 2026-11-10T07:38:32Z
host:thinpool-data-data 11348524558 size=374538764288 pct=3.03 2026-11-10T07:38:32Z
host:thinpool-data-metadata-pct NA pct=0.58 2026-11-10T07:38:32Z
host:dump:/var/lib/vz/dump 20151570432 archives=7 logs=6 all=16 2026-11-10T07:38:32Z
host:template-cache:/var/lib/vz/template 129970176 3 2026-11-10T07:38:32Z
host:dump:/mnt/hdd_1/dump 4358832128 archives=3 logs=3 all=9 2026-11-10T07:38:32Z
host:template-cache:/mnt/hdd_1/template NA NA 2026-11-10T07:38:32Z
host:journal:/var/log/journal 4282028032 files=74 2026-11-10T07:38:32Z
host:var-log-other 13967360 pve-tasks=706 2026-11-10T07:38:32Z
host:rrdcached:/var/lib/rrdcached 21041152 NA 2026-11-10T07:38:32Z
host:kernels:/lib/modules 8148774912 kernels=8 2026-11-10T07:38:32Z
host:boot:/boot 939851776 initrd=8 2026-11-10T07:38:32Z
host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T07:38:32Z
host:apt-cache:/var/cache/apt 92090368 debs=0 2026-11-10T07:38:32Z
host:apt-lists:/var/lib/apt/lists 61947904 NA 2026-11-10T07:38:32Z
host:agent-state:/var/lib/felhom-agent 724164608 21 2026-11-10T07:38:32Z
host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 723935232 binaries=48 2026-11-10T07:38:32Z
host:agent-bak:/usr/local/bin/felhom-agent.bak-* 155231048 11 2026-11-10T07:38:32Z
host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T07:38:32Z
host:tmp:/tmp 0 5 2026-11-10T07:38:32Z
host:var-tmp:/var/tmp 12288 1 2026-11-10T07:38:32Z
host:root-home:/root 16269312 NA 2026-11-10T07:38:32Z
host:swap-used 8192 size=8589930496 2026-11-10T07:38:32Z
host:journal-lines-last-hour NA all=13918 sudo=12750 2026-11-10T07:38:32Z
host:journal-oldest-entry NA 2026-10-04T21:43:33+02:00 2026-11-10T07:38:32Z
1 host:rootfs-used 38093631488 size=100861726720 2026-11-10T07:38:32Z
2 host:thinpool-data-data 11348524558 size=374538764288 pct=3.03 2026-11-10T07:38:32Z
3 host:thinpool-data-metadata-pct NA pct=0.58 2026-11-10T07:38:32Z
4 host:dump:/var/lib/vz/dump 20151570432 archives=7 logs=6 all=16 2026-11-10T07:38:32Z
5 host:template-cache:/var/lib/vz/template 129970176 3 2026-11-10T07:38:32Z
6 host:dump:/mnt/hdd_1/dump 4358832128 archives=3 logs=3 all=9 2026-11-10T07:38:32Z
7 host:template-cache:/mnt/hdd_1/template NA NA 2026-11-10T07:38:32Z
8 host:journal:/var/log/journal 4282028032 files=74 2026-11-10T07:38:32Z
9 host:var-log-other 13967360 pve-tasks=706 2026-11-10T07:38:32Z
10 host:rrdcached:/var/lib/rrdcached 21041152 NA 2026-11-10T07:38:32Z
11 host:kernels:/lib/modules 8148774912 kernels=8 2026-11-10T07:38:32Z
12 host:boot:/boot 939851776 initrd=8 2026-11-10T07:38:32Z
13 host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T07:38:32Z
14 host:apt-cache:/var/cache/apt 92090368 debs=0 2026-11-10T07:38:32Z
15 host:apt-lists:/var/lib/apt/lists 61947904 NA 2026-11-10T07:38:32Z
16 host:agent-state:/var/lib/felhom-agent 724164608 21 2026-11-10T07:38:32Z
17 host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 723935232 binaries=48 2026-11-10T07:38:32Z
18 host:agent-bak:/usr/local/bin/felhom-agent.bak-* 155231048 11 2026-11-10T07:38:32Z
19 host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T07:38:32Z
20 host:tmp:/tmp 0 5 2026-11-10T07:38:32Z
21 host:var-tmp:/var/tmp 12288 1 2026-11-10T07:38:32Z
22 host:root-home:/root 16269312 NA 2026-11-10T07:38:32Z
23 host:swap-used 8192 size=8589930496 2026-11-10T07:38:32Z
24 host:journal-lines-last-hour NA all=13918 sudo=12750 2026-11-10T07:38:32Z
25 host:journal-oldest-entry NA 2026-10-04T21:43:33+02:00 2026-11-10T07:38:32Z
@@ -0,0 +1,26 @@
guest:rootfs-used 1182224384 size=33501757440 2026-11-10T07:39:06Z
guest:data-volume-used:/var/lib/felhom 8987492352 size=73793978368 2026-11-10T07:39:06Z
guest:docker-images-layers:/var/lib/docker/overlay2 8315240448 images=12 untagged=12 2026-11-10T07:39:07Z
guest:docker-volumes:/var/lib/docker/volumes 370917376 all=388 anonymous=380 unused=377 2026-11-10T07:39:07Z
guest:container-logs:/var/lib/docker/containers/*/*-json.log* 7253286 files=10 containers=10 2026-11-10T07:39:07Z
guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T07:39:07Z
guest:controller-data-volume 79630336 NA 2026-11-10T07:39:07Z
guest:stacks:/opt/docker/stacks 1847296 68 2026-11-10T07:39:07Z
guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 269996032 4 2026-11-10T07:39:07Z
guest:journal 8M storage=volatile 2026-11-10T07:39:07Z
guest:var-log:/var/log 114688 NA 2026-11-10T07:39:07Z
guest:apt-cache:/var/cache/apt 95719424 NA 2026-11-10T07:39:07Z
guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T07:39:07Z
guest:tmp:/tmp 0 5 2026-11-10T07:39:07Z
guest:var-tmp:/var/tmp 20480 2 2026-11-10T07:39:07Z
drive:/mnt/felhom-drives/adatlemez used 13112242176 size=52521566208 2026-11-10T07:39:07Z
drive:/mnt/felhom-drives/adatlemez/backups/primary 99840000 1 2026-11-10T07:39:07Z
drive:/mnt/felhom-drives/adatlemez/backups/secondary 13010116608 8 2026-11-10T07:39:07Z
drive:/mnt/felhom-drives/adatlemez/backups/offsite-restore 4096 0 2026-11-10T07:39:07Z
drive:/mnt/felhom-drives/adatlemez/backups/offsite-proof 4096 0 2026-11-10T07:39:07Z
drive:/mnt/felhom-drives/adatlemez/appdata 45056 1 2026-11-10T07:39:07Z
drive:/mnt/felhom-drives/adatlemez/userdata 102400 5 2026-11-10T07:39:07Z
drive:/mnt/felhom-drives/media_a NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T07:39:07Z
drive:/mnt/felhom-drives/media_b NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T07:39:07Z
drive:/mnt/felhom-drives/media_c NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T07:39:08Z
drive:/mnt/felhom-drives/media_d NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T07:39:08Z
1 guest:rootfs-used 1182224384 size=33501757440 2026-11-10T07:39:06Z
2 guest:data-volume-used:/var/lib/felhom 8987492352 size=73793978368 2026-11-10T07:39:06Z
3 guest:docker-images-layers:/var/lib/docker/overlay2 8315240448 images=12 untagged=12 2026-11-10T07:39:07Z
4 guest:docker-volumes:/var/lib/docker/volumes 370917376 all=388 anonymous=380 unused=377 2026-11-10T07:39:07Z
5 guest:container-logs:/var/lib/docker/containers/*/*-json.log* 7253286 files=10 containers=10 2026-11-10T07:39:07Z
6 guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T07:39:07Z
7 guest:controller-data-volume 79630336 NA 2026-11-10T07:39:07Z
8 guest:stacks:/opt/docker/stacks 1847296 68 2026-11-10T07:39:07Z
9 guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 269996032 4 2026-11-10T07:39:07Z
10 guest:journal 8M storage=volatile 2026-11-10T07:39:07Z
11 guest:var-log:/var/log 114688 NA 2026-11-10T07:39:07Z
12 guest:apt-cache:/var/cache/apt 95719424 NA 2026-11-10T07:39:07Z
13 guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T07:39:07Z
14 guest:tmp:/tmp 0 5 2026-11-10T07:39:07Z
15 guest:var-tmp:/var/tmp 20480 2 2026-11-10T07:39:07Z
16 drive:/mnt/felhom-drives/adatlemez used 13112242176 size=52521566208 2026-11-10T07:39:07Z
17 drive:/mnt/felhom-drives/adatlemez/backups/primary 99840000 1 2026-11-10T07:39:07Z
18 drive:/mnt/felhom-drives/adatlemez/backups/secondary 13010116608 8 2026-11-10T07:39:07Z
19 drive:/mnt/felhom-drives/adatlemez/backups/offsite-restore 4096 0 2026-11-10T07:39:07Z
20 drive:/mnt/felhom-drives/adatlemez/backups/offsite-proof 4096 0 2026-11-10T07:39:07Z
21 drive:/mnt/felhom-drives/adatlemez/appdata 45056 1 2026-11-10T07:39:07Z
22 drive:/mnt/felhom-drives/adatlemez/userdata 102400 5 2026-11-10T07:39:07Z
23 drive:/mnt/felhom-drives/media_a NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T07:39:07Z
24 drive:/mnt/felhom-drives/media_b NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T07:39:07Z
25 drive:/mnt/felhom-drives/media_c NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T07:39:08Z
26 drive:/mnt/felhom-drives/media_d NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T07:39:08Z
@@ -0,0 +1,23 @@
host:rootfs-used 21642362880 size=62903865344 2026-11-10T07:39:05Z
host:thinpool-data-data 15919325643 size=121521569792 pct=13.10 2026-11-10T07:39:05Z
host:thinpool-data-metadata-pct NA pct=1.84 2026-11-10T07:39:05Z
host:dump:/var/lib/vz/dump 10111107072 archives=4 logs=3 all=10 2026-11-10T07:39:05Z
host:template-cache:/var/lib/vz/template 12288 2 2026-11-10T07:39:05Z
host:journal:/var/log/journal 4288311296 files=81 2026-11-10T07:39:05Z
host:var-log-other 14577664 pve-tasks=877 2026-11-10T07:39:05Z
host:rrdcached:/var/lib/rrdcached 4710400 NA 2026-11-10T07:39:05Z
host:kernels:/lib/modules 3053326336 kernels=3 2026-11-10T07:39:05Z
host:boot:/boot 330608640 initrd=3 2026-11-10T07:39:05Z
host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T07:39:05Z
host:apt-cache:/var/cache/apt 92000256 debs=0 2026-11-10T07:39:05Z
host:apt-lists:/var/lib/apt/lists 61833216 NA 2026-11-10T07:39:05Z
host:agent-state:/var/lib/felhom-agent 509444096 22 2026-11-10T07:39:05Z
host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 509177856 binaries=33 2026-11-10T07:39:05Z
host:agent-bak:/usr/local/bin/felhom-agent.bak-* 0 0 2026-11-10T07:39:05Z
host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T07:39:05Z
host:tmp:/tmp 0 6 2026-11-10T07:39:05Z
host:var-tmp:/var/tmp 20480 2 2026-11-10T07:39:05Z
host:root-home:/root 102400 NA 2026-11-10T07:39:05Z
host:swap-used 185950208 size=8254386176 2026-11-10T07:39:05Z
host:journal-lines-last-hour NA all=11558 sudo=10419 2026-11-10T07:39:05Z
host:journal-oldest-entry NA 2026-10-13T01:30:35+02:00 2026-11-10T07:39:05Z
1 host:rootfs-used 21642362880 size=62903865344 2026-11-10T07:39:05Z
2 host:thinpool-data-data 15919325643 size=121521569792 pct=13.10 2026-11-10T07:39:05Z
3 host:thinpool-data-metadata-pct NA pct=1.84 2026-11-10T07:39:05Z
4 host:dump:/var/lib/vz/dump 10111107072 archives=4 logs=3 all=10 2026-11-10T07:39:05Z
5 host:template-cache:/var/lib/vz/template 12288 2 2026-11-10T07:39:05Z
6 host:journal:/var/log/journal 4288311296 files=81 2026-11-10T07:39:05Z
7 host:var-log-other 14577664 pve-tasks=877 2026-11-10T07:39:05Z
8 host:rrdcached:/var/lib/rrdcached 4710400 NA 2026-11-10T07:39:05Z
9 host:kernels:/lib/modules 3053326336 kernels=3 2026-11-10T07:39:05Z
10 host:boot:/boot 330608640 initrd=3 2026-11-10T07:39:05Z
11 host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T07:39:05Z
12 host:apt-cache:/var/cache/apt 92000256 debs=0 2026-11-10T07:39:05Z
13 host:apt-lists:/var/lib/apt/lists 61833216 NA 2026-11-10T07:39:05Z
14 host:agent-state:/var/lib/felhom-agent 509444096 22 2026-11-10T07:39:05Z
15 host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 509177856 binaries=33 2026-11-10T07:39:05Z
16 host:agent-bak:/usr/local/bin/felhom-agent.bak-* 0 0 2026-11-10T07:39:05Z
17 host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T07:39:05Z
18 host:tmp:/tmp 0 6 2026-11-10T07:39:05Z
19 host:var-tmp:/var/tmp 20480 2 2026-11-10T07:39:05Z
20 host:root-home:/root 102400 NA 2026-11-10T07:39:05Z
21 host:swap-used 185950208 size=8254386176 2026-11-10T07:39:05Z
22 host:journal-lines-last-hour NA all=11558 sudo=10419 2026-11-10T07:39:05Z
23 host:journal-oldest-entry NA 2026-10-13T01:30:35+02:00 2026-11-10T07:39:05Z
@@ -0,0 +1,23 @@
guest:rootfs-used 1357545472 size=33501757440 2026-11-10T09:45:21Z
guest:data-volume-used:/var/lib/felhom 30096568320 size=73793978368 2026-11-10T09:45:21Z
guest:docker-images-layers:/var/lib/docker/overlay2 27008151552 images=22 untagged=21 2026-11-10T09:45:33Z
guest:docker-volumes:/var/lib/docker/volumes 1607942144 all=1914 anonymous=1887 unused=1885 2026-11-10T09:45:33Z
guest:container-logs:/var/lib/docker/containers/*/*-json.log* 32137125 files=27 containers=26 2026-11-10T09:45:33Z
guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T09:45:33Z
guest:controller-data-volume 145387520 NA 2026-11-10T09:45:33Z
guest:stacks:/opt/docker/stacks 2174976 69 2026-11-10T09:45:33Z
guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 1327214592 14 2026-11-10T09:45:33Z
guest:journal 8M storage=volatile 2026-11-10T09:45:33Z
guest:var-log:/var/log 253952 NA 2026-11-10T09:45:33Z
guest:apt-cache:/var/cache/apt 95723520 NA 2026-11-10T09:45:33Z
guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T09:45:33Z
guest:tmp:/tmp 0 5 2026-11-10T09:45:33Z
guest:var-tmp:/var/tmp 20480 2 2026-11-10T09:45:33Z
drive:/mnt/felhom-drives/hdd_1 used 322734358528 size=1006980812800 2026-11-10T09:45:33Z
drive:/mnt/felhom-drives/hdd_1/backups/primary 371859456 7 2026-11-10T09:45:33Z
drive:/mnt/felhom-drives/hdd_1/backups/secondary 919998464 8 2026-11-10T09:45:33Z
drive:/mnt/felhom-drives/hdd_1/backups/offsite-restore 4096 0 2026-11-10T09:45:33Z
drive:/mnt/felhom-drives/hdd_1/backups/offsite-proof 4096 0 2026-11-10T09:45:33Z
drive:/mnt/felhom-drives/hdd_1/kept 208642048 1 2026-11-10T09:45:33Z
drive:/mnt/felhom-drives/hdd_1/appdata 30277632 2 2026-11-10T09:45:33Z
drive:/mnt/felhom-drives/hdd_1/userdata 6164480 6 2026-11-10T09:45:33Z
1 guest:rootfs-used 1357545472 size=33501757440 2026-11-10T09:45:21Z
2 guest:data-volume-used:/var/lib/felhom 30096568320 size=73793978368 2026-11-10T09:45:21Z
3 guest:docker-images-layers:/var/lib/docker/overlay2 27008151552 images=22 untagged=21 2026-11-10T09:45:33Z
4 guest:docker-volumes:/var/lib/docker/volumes 1607942144 all=1914 anonymous=1887 unused=1885 2026-11-10T09:45:33Z
5 guest:container-logs:/var/lib/docker/containers/*/*-json.log* 32137125 files=27 containers=26 2026-11-10T09:45:33Z
6 guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T09:45:33Z
7 guest:controller-data-volume 145387520 NA 2026-11-10T09:45:33Z
8 guest:stacks:/opt/docker/stacks 2174976 69 2026-11-10T09:45:33Z
9 guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 1327214592 14 2026-11-10T09:45:33Z
10 guest:journal 8M storage=volatile 2026-11-10T09:45:33Z
11 guest:var-log:/var/log 253952 NA 2026-11-10T09:45:33Z
12 guest:apt-cache:/var/cache/apt 95723520 NA 2026-11-10T09:45:33Z
13 guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T09:45:33Z
14 guest:tmp:/tmp 0 5 2026-11-10T09:45:33Z
15 guest:var-tmp:/var/tmp 20480 2 2026-11-10T09:45:33Z
16 drive:/mnt/felhom-drives/hdd_1 used 322734358528 size=1006980812800 2026-11-10T09:45:33Z
17 drive:/mnt/felhom-drives/hdd_1/backups/primary 371859456 7 2026-11-10T09:45:33Z
18 drive:/mnt/felhom-drives/hdd_1/backups/secondary 919998464 8 2026-11-10T09:45:33Z
19 drive:/mnt/felhom-drives/hdd_1/backups/offsite-restore 4096 0 2026-11-10T09:45:33Z
20 drive:/mnt/felhom-drives/hdd_1/backups/offsite-proof 4096 0 2026-11-10T09:45:33Z
21 drive:/mnt/felhom-drives/hdd_1/kept 208642048 1 2026-11-10T09:45:33Z
22 drive:/mnt/felhom-drives/hdd_1/appdata 30277632 2 2026-11-10T09:45:33Z
23 drive:/mnt/felhom-drives/hdd_1/userdata 6164480 6 2026-11-10T09:45:33Z
@@ -0,0 +1,23 @@
guest:rootfs-used 1920802816 size=33501757440 2026-11-10T09:45:35Z
guest:data-volume-used:/var/lib/felhom 13993504768 size=73391005696 2026-11-10T09:45:35Z
guest:docker-images-layers:/var/lib/docker/overlay2 13676625920 images=41 untagged=7 2026-11-10T09:45:37Z
guest:docker-volumes:/var/lib/docker/volumes 247812096 all=368 anonymous=363 unused=363 2026-11-10T09:45:37Z
guest:container-logs:/var/lib/docker/containers/*/*-json.log* 20594551 files=7 containers=6 2026-11-10T09:45:37Z
guest:docker-buildcache:/var/lib/docker/buildkit 495616 NA 2026-11-10T09:45:37Z
guest:controller-data-volume 156348416 NA 2026-11-10T09:45:37Z
guest:stacks:/opt/docker/stacks 2990080 70 2026-11-10T09:45:37Z
guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 20480 0 2026-11-10T09:45:37Z
guest:journal 8M storage=volatile 2026-11-10T09:45:37Z
guest:var-log:/var/log 102400 NA 2026-11-10T09:45:37Z
guest:apt-cache:/var/cache/apt 273600512 NA 2026-11-10T09:45:37Z
guest:apt-lists:/var/lib/apt/lists 95510528 NA 2026-11-10T09:45:37Z
guest:tmp:/tmp 0 5 2026-11-10T09:45:37Z
guest:var-tmp:/var/tmp 20480 2 2026-11-10T09:45:37Z
drive:/mnt/felhom-drives/scratch_hdd used 322734358528 size=1006980812800 2026-11-10T09:45:37Z
drive:/mnt/felhom-drives/scratch_hdd/backups/primary 77488128 1 2026-11-10T09:45:37Z
drive:/mnt/felhom-drives/scratch_hdd/backups/secondary 4096 0 2026-11-10T09:45:37Z
drive:/mnt/felhom-drives/scratch_hdd/backups/offsite-restore 4096 0 2026-11-10T09:45:37Z
drive:/mnt/felhom-drives/scratch_hdd/backups/restore-undo 4096 0 2026-11-10T09:45:37Z
drive:/mnt/felhom-drives/scratch_hdd/kept 1355776 1 2026-11-10T09:45:37Z
drive:/mnt/felhom-drives/scratch_hdd/appdata 32768 2 2026-11-10T09:45:37Z
drive:/mnt/felhom-drives/scratch_hdd/userdata 33083392 18 2026-11-10T09:45:37Z
1 guest:rootfs-used 1920802816 size=33501757440 2026-11-10T09:45:35Z
2 guest:data-volume-used:/var/lib/felhom 13993504768 size=73391005696 2026-11-10T09:45:35Z
3 guest:docker-images-layers:/var/lib/docker/overlay2 13676625920 images=41 untagged=7 2026-11-10T09:45:37Z
4 guest:docker-volumes:/var/lib/docker/volumes 247812096 all=368 anonymous=363 unused=363 2026-11-10T09:45:37Z
5 guest:container-logs:/var/lib/docker/containers/*/*-json.log* 20594551 files=7 containers=6 2026-11-10T09:45:37Z
6 guest:docker-buildcache:/var/lib/docker/buildkit 495616 NA 2026-11-10T09:45:37Z
7 guest:controller-data-volume 156348416 NA 2026-11-10T09:45:37Z
8 guest:stacks:/opt/docker/stacks 2990080 70 2026-11-10T09:45:37Z
9 guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 20480 0 2026-11-10T09:45:37Z
10 guest:journal 8M storage=volatile 2026-11-10T09:45:37Z
11 guest:var-log:/var/log 102400 NA 2026-11-10T09:45:37Z
12 guest:apt-cache:/var/cache/apt 273600512 NA 2026-11-10T09:45:37Z
13 guest:apt-lists:/var/lib/apt/lists 95510528 NA 2026-11-10T09:45:37Z
14 guest:tmp:/tmp 0 5 2026-11-10T09:45:37Z
15 guest:var-tmp:/var/tmp 20480 2 2026-11-10T09:45:37Z
16 drive:/mnt/felhom-drives/scratch_hdd used 322734358528 size=1006980812800 2026-11-10T09:45:37Z
17 drive:/mnt/felhom-drives/scratch_hdd/backups/primary 77488128 1 2026-11-10T09:45:37Z
18 drive:/mnt/felhom-drives/scratch_hdd/backups/secondary 4096 0 2026-11-10T09:45:37Z
19 drive:/mnt/felhom-drives/scratch_hdd/backups/offsite-restore 4096 0 2026-11-10T09:45:37Z
20 drive:/mnt/felhom-drives/scratch_hdd/backups/restore-undo 4096 0 2026-11-10T09:45:37Z
21 drive:/mnt/felhom-drives/scratch_hdd/kept 1355776 1 2026-11-10T09:45:37Z
22 drive:/mnt/felhom-drives/scratch_hdd/appdata 32768 2 2026-11-10T09:45:37Z
23 drive:/mnt/felhom-drives/scratch_hdd/userdata 33083392 18 2026-11-10T09:45:37Z
@@ -0,0 +1,25 @@
host:rootfs-used 23993802752 size=41424257024 2026-11-10T09:45:14Z
host:thinpool-data-data 40878104956 size=57843646464 pct=70.67 2026-11-10T09:45:14Z
host:thinpool-data-metadata-pct NA pct=2.85 2026-11-10T09:45:14Z
host:dump:/var/lib/vz/dump 9750130688 archives=1 logs=10 all=13 2026-11-10T09:45:14Z
host:template-cache:/var/lib/vz/template 129974272 3 2026-11-10T09:45:14Z
host:dump:/mnt/hdd_1/dump 5099683840 archives=4 logs=3 all=10 2026-11-10T09:45:14Z
host:template-cache:/mnt/hdd_1/template NA NA 2026-11-10T09:45:14Z
host:journal:/var/log/journal 898883584 files=16 2026-11-10T09:45:14Z
host:var-log-other 15110144 pve-tasks=914 2026-11-10T09:45:14Z
host:rrdcached:/var/lib/rrdcached 64802816 NA 2026-11-10T09:45:14Z
host:kernels:/lib/modules 8148783104 kernels=8 2026-11-10T09:45:15Z
host:boot:/boot 823369728 initrd=8 2026-11-10T09:45:15Z
host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T09:45:15Z
host:apt-cache:/var/cache/apt 92004352 debs=0 2026-11-10T09:45:15Z
host:apt-lists:/var/lib/apt/lists 61833216 NA 2026-11-10T09:45:15Z
host:agent-state:/var/lib/felhom-agent 740171776 22 2026-11-10T09:45:15Z
host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 739938304 binaries=49 2026-11-10T09:45:15Z
host:agent-bak:/usr/local/bin/felhom-agent.bak-* 14284239 1 2026-11-10T09:45:15Z
host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T09:45:15Z
host:tmp:/tmp 0 6 2026-11-10T09:45:15Z
host:var-tmp:/var/tmp 28672 4 2026-11-10T09:45:15Z
host:root-home:/root 176128 NA 2026-11-10T09:45:15Z
host:swap-used 0 size=8589930496 2026-11-10T09:45:15Z
host:journal-lines-last-hour NA all=13521 sudo=12825 2026-11-10T09:45:15Z
host:journal-oldest-entry NA 2026-10-04T21:42:08+02:00 2026-11-10T09:45:15Z
1 host:rootfs-used 23993802752 size=41424257024 2026-11-10T09:45:14Z
2 host:thinpool-data-data 40878104956 size=57843646464 pct=70.67 2026-11-10T09:45:14Z
3 host:thinpool-data-metadata-pct NA pct=2.85 2026-11-10T09:45:14Z
4 host:dump:/var/lib/vz/dump 9750130688 archives=1 logs=10 all=13 2026-11-10T09:45:14Z
5 host:template-cache:/var/lib/vz/template 129974272 3 2026-11-10T09:45:14Z
6 host:dump:/mnt/hdd_1/dump 5099683840 archives=4 logs=3 all=10 2026-11-10T09:45:14Z
7 host:template-cache:/mnt/hdd_1/template NA NA 2026-11-10T09:45:14Z
8 host:journal:/var/log/journal 898883584 files=16 2026-11-10T09:45:14Z
9 host:var-log-other 15110144 pve-tasks=914 2026-11-10T09:45:14Z
10 host:rrdcached:/var/lib/rrdcached 64802816 NA 2026-11-10T09:45:14Z
11 host:kernels:/lib/modules 8148783104 kernels=8 2026-11-10T09:45:15Z
12 host:boot:/boot 823369728 initrd=8 2026-11-10T09:45:15Z
13 host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T09:45:15Z
14 host:apt-cache:/var/cache/apt 92004352 debs=0 2026-11-10T09:45:15Z
15 host:apt-lists:/var/lib/apt/lists 61833216 NA 2026-11-10T09:45:15Z
16 host:agent-state:/var/lib/felhom-agent 740171776 22 2026-11-10T09:45:15Z
17 host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 739938304 binaries=49 2026-11-10T09:45:15Z
18 host:agent-bak:/usr/local/bin/felhom-agent.bak-* 14284239 1 2026-11-10T09:45:15Z
19 host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T09:45:15Z
20 host:tmp:/tmp 0 6 2026-11-10T09:45:15Z
21 host:var-tmp:/var/tmp 28672 4 2026-11-10T09:45:15Z
22 host:root-home:/root 176128 NA 2026-11-10T09:45:15Z
23 host:swap-used 0 size=8589930496 2026-11-10T09:45:15Z
24 host:journal-lines-last-hour NA all=13521 sudo=12825 2026-11-10T09:45:15Z
25 host:journal-oldest-entry NA 2026-10-04T21:42:08+02:00 2026-11-10T09:45:15Z
@@ -0,0 +1,23 @@
guest:rootfs-used 1188163584 size=33501757440 2026-11-10T09:45:06Z
guest:data-volume-used:/var/lib/felhom 2474618880 size=263086084096 2026-11-10T09:45:06Z
guest:docker-images-layers:/var/lib/docker/overlay2 2258448384 images=11 untagged=5 2026-11-10T09:45:06Z
guest:docker-volumes:/var/lib/docker/volumes 140599296 all=8 anonymous=1 unused=1 2026-11-10T09:45:06Z
guest:container-logs:/var/lib/docker/containers/*/*-json.log* 10060111 files=9 containers=9 2026-11-10T09:45:06Z
guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T09:45:06Z
guest:controller-data-volume 84152320 NA 2026-11-10T09:45:06Z
guest:stacks:/opt/docker/stacks 1859584 68 2026-11-10T09:45:06Z
guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 47730688 5 2026-11-10T09:45:06Z
guest:journal 8M storage=volatile 2026-11-10T09:45:06Z
guest:var-log:/var/log 135168 NA 2026-11-10T09:45:06Z
guest:apt-cache:/var/cache/apt 95715328 NA 2026-11-10T09:45:06Z
guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T09:45:06Z
guest:tmp:/tmp 0 5 2026-11-10T09:45:06Z
guest:var-tmp:/var/tmp 20480 2 2026-11-10T09:45:06Z
drive:/mnt/felhom-drives/hdd_1 used 8143593472 size=983351140352 2026-11-10T09:45:06Z
drive:/mnt/felhom-drives/hdd_1/backups/primary 1391616000 5 2026-11-10T09:45:06Z
drive:/mnt/felhom-drives/hdd_1/backups/secondary 293728256 4 2026-11-10T09:45:06Z
drive:/mnt/felhom-drives/hdd_1/backups/offsite-restore 1654599680 2 2026-11-10T09:45:06Z
drive:/mnt/felhom-drives/hdd_1/backups/offsite-proof 4096 0 2026-11-10T09:45:06Z
drive:/mnt/felhom-drives/hdd_1/appdata 442085376 4 2026-11-10T09:45:06Z
drive:/mnt/felhom-drives/hdd_1/userdata 561152 6 2026-11-10T09:45:06Z
drive:/mnt/felhom-drives/hdd_1/shares 32768 1 2026-11-10T09:45:06Z
1 guest:rootfs-used 1188163584 size=33501757440 2026-11-10T09:45:06Z
2 guest:data-volume-used:/var/lib/felhom 2474618880 size=263086084096 2026-11-10T09:45:06Z
3 guest:docker-images-layers:/var/lib/docker/overlay2 2258448384 images=11 untagged=5 2026-11-10T09:45:06Z
4 guest:docker-volumes:/var/lib/docker/volumes 140599296 all=8 anonymous=1 unused=1 2026-11-10T09:45:06Z
5 guest:container-logs:/var/lib/docker/containers/*/*-json.log* 10060111 files=9 containers=9 2026-11-10T09:45:06Z
6 guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T09:45:06Z
7 guest:controller-data-volume 84152320 NA 2026-11-10T09:45:06Z
8 guest:stacks:/opt/docker/stacks 1859584 68 2026-11-10T09:45:06Z
9 guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 47730688 5 2026-11-10T09:45:06Z
10 guest:journal 8M storage=volatile 2026-11-10T09:45:06Z
11 guest:var-log:/var/log 135168 NA 2026-11-10T09:45:06Z
12 guest:apt-cache:/var/cache/apt 95715328 NA 2026-11-10T09:45:06Z
13 guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T09:45:06Z
14 guest:tmp:/tmp 0 5 2026-11-10T09:45:06Z
15 guest:var-tmp:/var/tmp 20480 2 2026-11-10T09:45:06Z
16 drive:/mnt/felhom-drives/hdd_1 used 8143593472 size=983351140352 2026-11-10T09:45:06Z
17 drive:/mnt/felhom-drives/hdd_1/backups/primary 1391616000 5 2026-11-10T09:45:06Z
18 drive:/mnt/felhom-drives/hdd_1/backups/secondary 293728256 4 2026-11-10T09:45:06Z
19 drive:/mnt/felhom-drives/hdd_1/backups/offsite-restore 1654599680 2 2026-11-10T09:45:06Z
20 drive:/mnt/felhom-drives/hdd_1/backups/offsite-proof 4096 0 2026-11-10T09:45:06Z
21 drive:/mnt/felhom-drives/hdd_1/appdata 442085376 4 2026-11-10T09:45:06Z
22 drive:/mnt/felhom-drives/hdd_1/userdata 561152 6 2026-11-10T09:45:06Z
23 drive:/mnt/felhom-drives/hdd_1/shares 32768 1 2026-11-10T09:45:06Z
@@ -0,0 +1,25 @@
host:rootfs-used 38110928896 size=100861726720 2026-11-10T09:44:42Z
host:thinpool-data-data 11348524558 size=374538764288 pct=3.03 2026-11-10T09:44:42Z
host:thinpool-data-metadata-pct NA pct=0.58 2026-11-10T09:44:42Z
host:dump:/var/lib/vz/dump 20151570432 archives=7 logs=6 all=16 2026-11-10T09:44:42Z
host:template-cache:/var/lib/vz/template 129970176 3 2026-11-10T09:44:42Z
host:dump:/mnt/hdd_1/dump 4358832128 archives=3 logs=3 all=9 2026-11-10T09:44:42Z
host:template-cache:/mnt/hdd_1/template NA NA 2026-11-10T09:44:42Z
host:journal:/var/log/journal 4298805248 files=74 2026-11-10T09:44:42Z
host:var-log-other 14487552 pve-tasks=706 2026-11-10T09:44:42Z
host:rrdcached:/var/lib/rrdcached 21041152 NA 2026-11-10T09:44:42Z
host:kernels:/lib/modules 8148774912 kernels=8 2026-11-10T09:44:43Z
host:boot:/boot 939851776 initrd=8 2026-11-10T09:44:43Z
host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T09:44:43Z
host:apt-cache:/var/cache/apt 92090368 debs=0 2026-11-10T09:44:43Z
host:apt-lists:/var/lib/apt/lists 61947904 NA 2026-11-10T09:44:43Z
host:agent-state:/var/lib/felhom-agent 724164608 21 2026-11-10T09:44:43Z
host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 723935232 binaries=48 2026-11-10T09:44:43Z
host:agent-bak:/usr/local/bin/felhom-agent.bak-* 155231048 11 2026-11-10T09:44:43Z
host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T09:44:43Z
host:tmp:/tmp 0 5 2026-11-10T09:44:43Z
host:var-tmp:/var/tmp 12288 1 2026-11-10T09:44:43Z
host:root-home:/root 16269312 NA 2026-11-10T09:44:43Z
host:swap-used 8192 size=8589930496 2026-11-10T09:44:43Z
host:journal-lines-last-hour NA all=13506 sudo=12753 2026-11-10T09:44:43Z
host:journal-oldest-entry NA 2026-10-04T21:43:33+02:00 2026-11-10T09:44:43Z
1 host:rootfs-used 38110928896 size=100861726720 2026-11-10T09:44:42Z
2 host:thinpool-data-data 11348524558 size=374538764288 pct=3.03 2026-11-10T09:44:42Z
3 host:thinpool-data-metadata-pct NA pct=0.58 2026-11-10T09:44:42Z
4 host:dump:/var/lib/vz/dump 20151570432 archives=7 logs=6 all=16 2026-11-10T09:44:42Z
5 host:template-cache:/var/lib/vz/template 129970176 3 2026-11-10T09:44:42Z
6 host:dump:/mnt/hdd_1/dump 4358832128 archives=3 logs=3 all=9 2026-11-10T09:44:42Z
7 host:template-cache:/mnt/hdd_1/template NA NA 2026-11-10T09:44:42Z
8 host:journal:/var/log/journal 4298805248 files=74 2026-11-10T09:44:42Z
9 host:var-log-other 14487552 pve-tasks=706 2026-11-10T09:44:42Z
10 host:rrdcached:/var/lib/rrdcached 21041152 NA 2026-11-10T09:44:42Z
11 host:kernels:/lib/modules 8148774912 kernels=8 2026-11-10T09:44:43Z
12 host:boot:/boot 939851776 initrd=8 2026-11-10T09:44:43Z
13 host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T09:44:43Z
14 host:apt-cache:/var/cache/apt 92090368 debs=0 2026-11-10T09:44:43Z
15 host:apt-lists:/var/lib/apt/lists 61947904 NA 2026-11-10T09:44:43Z
16 host:agent-state:/var/lib/felhom-agent 724164608 21 2026-11-10T09:44:43Z
17 host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 723935232 binaries=48 2026-11-10T09:44:43Z
18 host:agent-bak:/usr/local/bin/felhom-agent.bak-* 155231048 11 2026-11-10T09:44:43Z
19 host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T09:44:43Z
20 host:tmp:/tmp 0 5 2026-11-10T09:44:43Z
21 host:var-tmp:/var/tmp 12288 1 2026-11-10T09:44:43Z
22 host:root-home:/root 16269312 NA 2026-11-10T09:44:43Z
23 host:swap-used 8192 size=8589930496 2026-11-10T09:44:43Z
24 host:journal-lines-last-hour NA all=13506 sudo=12753 2026-11-10T09:44:43Z
25 host:journal-oldest-entry NA 2026-10-04T21:43:33+02:00 2026-11-10T09:44:43Z
@@ -0,0 +1,26 @@
guest:rootfs-used 1182224384 size=33501757440 2026-11-10T09:45:40Z
guest:data-volume-used:/var/lib/felhom 8987688960 size=73793978368 2026-11-10T09:45:40Z
guest:docker-images-layers:/var/lib/docker/overlay2 8315240448 images=12 untagged=12 2026-11-10T09:45:42Z
guest:docker-volumes:/var/lib/docker/volumes 370634752 all=388 anonymous=380 unused=377 2026-11-10T09:45:42Z
guest:container-logs:/var/lib/docker/containers/*/*-json.log* 7726107 files=10 containers=10 2026-11-10T09:45:42Z
guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T09:45:42Z
guest:controller-data-volume 79626240 NA 2026-11-10T09:45:42Z
guest:stacks:/opt/docker/stacks 1847296 68 2026-11-10T09:45:42Z
guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 269996032 4 2026-11-10T09:45:42Z
guest:journal 8M storage=volatile 2026-11-10T09:45:42Z
guest:var-log:/var/log 114688 NA 2026-11-10T09:45:42Z
guest:apt-cache:/var/cache/apt 95719424 NA 2026-11-10T09:45:42Z
guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T09:45:42Z
guest:tmp:/tmp 0 5 2026-11-10T09:45:42Z
guest:var-tmp:/var/tmp 20480 2 2026-11-10T09:45:42Z
drive:/mnt/felhom-drives/adatlemez used 13112242176 size=52521566208 2026-11-10T09:45:42Z
drive:/mnt/felhom-drives/adatlemez/backups/primary 99840000 1 2026-11-10T09:45:42Z
drive:/mnt/felhom-drives/adatlemez/backups/secondary 13010116608 8 2026-11-10T09:45:42Z
drive:/mnt/felhom-drives/adatlemez/backups/offsite-restore 4096 0 2026-11-10T09:45:42Z
drive:/mnt/felhom-drives/adatlemez/backups/offsite-proof 4096 0 2026-11-10T09:45:42Z
drive:/mnt/felhom-drives/adatlemez/appdata 45056 1 2026-11-10T09:45:42Z
drive:/mnt/felhom-drives/adatlemez/userdata 102400 5 2026-11-10T09:45:42Z
drive:/mnt/felhom-drives/media_a NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T09:45:42Z
drive:/mnt/felhom-drives/media_b NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T09:45:42Z
drive:/mnt/felhom-drives/media_c NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T09:45:42Z
drive:/mnt/felhom-drives/media_d NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T09:45:42Z
1 guest:rootfs-used 1182224384 size=33501757440 2026-11-10T09:45:40Z
2 guest:data-volume-used:/var/lib/felhom 8987688960 size=73793978368 2026-11-10T09:45:40Z
3 guest:docker-images-layers:/var/lib/docker/overlay2 8315240448 images=12 untagged=12 2026-11-10T09:45:42Z
4 guest:docker-volumes:/var/lib/docker/volumes 370634752 all=388 anonymous=380 unused=377 2026-11-10T09:45:42Z
5 guest:container-logs:/var/lib/docker/containers/*/*-json.log* 7726107 files=10 containers=10 2026-11-10T09:45:42Z
6 guest:docker-buildcache:/var/lib/docker/buildkit 167936 NA 2026-11-10T09:45:42Z
7 guest:controller-data-volume 79626240 NA 2026-11-10T09:45:42Z
8 guest:stacks:/opt/docker/stacks 1847296 68 2026-11-10T09:45:42Z
9 guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups 269996032 4 2026-11-10T09:45:42Z
10 guest:journal 8M storage=volatile 2026-11-10T09:45:42Z
11 guest:var-log:/var/log 114688 NA 2026-11-10T09:45:42Z
12 guest:apt-cache:/var/cache/apt 95719424 NA 2026-11-10T09:45:42Z
13 guest:apt-lists:/var/lib/apt/lists 96104448 NA 2026-11-10T09:45:42Z
14 guest:tmp:/tmp 0 5 2026-11-10T09:45:42Z
15 guest:var-tmp:/var/tmp 20480 2 2026-11-10T09:45:42Z
16 drive:/mnt/felhom-drives/adatlemez used 13112242176 size=52521566208 2026-11-10T09:45:42Z
17 drive:/mnt/felhom-drives/adatlemez/backups/primary 99840000 1 2026-11-10T09:45:42Z
18 drive:/mnt/felhom-drives/adatlemez/backups/secondary 13010116608 8 2026-11-10T09:45:42Z
19 drive:/mnt/felhom-drives/adatlemez/backups/offsite-restore 4096 0 2026-11-10T09:45:42Z
20 drive:/mnt/felhom-drives/adatlemez/backups/offsite-proof 4096 0 2026-11-10T09:45:42Z
21 drive:/mnt/felhom-drives/adatlemez/appdata 45056 1 2026-11-10T09:45:42Z
22 drive:/mnt/felhom-drives/adatlemez/userdata 102400 5 2026-11-10T09:45:42Z
23 drive:/mnt/felhom-drives/media_a NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T09:45:42Z
24 drive:/mnt/felhom-drives/media_b NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T09:45:42Z
25 drive:/mnt/felhom-drives/media_c NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T09:45:42Z
26 drive:/mnt/felhom-drives/media_d NOT-MOUNTED (bare folder on the host root disk) 4096 0 2026-11-10T09:45:42Z
@@ -0,0 +1,23 @@
host:rootfs-used 21651439616 size=62903865344 2026-11-10T09:45:16Z
host:thinpool-data-data 15919325643 size=121521569792 pct=13.10 2026-11-10T09:45:16Z
host:thinpool-data-metadata-pct NA pct=1.84 2026-11-10T09:45:16Z
host:dump:/var/lib/vz/dump 10111107072 archives=4 logs=3 all=10 2026-11-10T09:45:16Z
host:template-cache:/var/lib/vz/template 12288 2 2026-11-10T09:45:16Z
host:journal:/var/log/journal 4296699904 files=81 2026-11-10T09:45:16Z
host:var-log-other 15101952 pve-tasks=880 2026-11-10T09:45:16Z
host:rrdcached:/var/lib/rrdcached 4710400 NA 2026-11-10T09:45:16Z
host:kernels:/lib/modules 3053326336 kernels=3 2026-11-10T09:45:17Z
host:boot:/boot 330608640 initrd=3 2026-11-10T09:45:17Z
host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T09:45:17Z
host:apt-cache:/var/cache/apt 92000256 debs=0 2026-11-10T09:45:17Z
host:apt-lists:/var/lib/apt/lists 61833216 NA 2026-11-10T09:45:17Z
host:agent-state:/var/lib/felhom-agent 509444096 22 2026-11-10T09:45:17Z
host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 509177856 binaries=33 2026-11-10T09:45:17Z
host:agent-bak:/usr/local/bin/felhom-agent.bak-* 0 0 2026-11-10T09:45:17Z
host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T09:45:17Z
host:tmp:/tmp 0 6 2026-11-10T09:45:17Z
host:var-tmp:/var/tmp 20480 2 2026-11-10T09:45:17Z
host:root-home:/root 102400 NA 2026-11-10T09:45:17Z
host:swap-used 513003520 size=8254386176 2026-11-10T09:45:17Z
host:journal-lines-last-hour NA all=11177 sudo=10420 2026-11-10T09:45:19Z
host:journal-oldest-entry NA 2026-10-13T01:30:35+02:00 2026-11-10T09:45:19Z
1 host:rootfs-used 21651439616 size=62903865344 2026-11-10T09:45:16Z
2 host:thinpool-data-data 15919325643 size=121521569792 pct=13.10 2026-11-10T09:45:16Z
3 host:thinpool-data-metadata-pct NA pct=1.84 2026-11-10T09:45:16Z
4 host:dump:/var/lib/vz/dump 10111107072 archives=4 logs=3 all=10 2026-11-10T09:45:16Z
5 host:template-cache:/var/lib/vz/template 12288 2 2026-11-10T09:45:16Z
6 host:journal:/var/log/journal 4296699904 files=81 2026-11-10T09:45:16Z
7 host:var-log-other 15101952 pve-tasks=880 2026-11-10T09:45:16Z
8 host:rrdcached:/var/lib/rrdcached 4710400 NA 2026-11-10T09:45:16Z
9 host:kernels:/lib/modules 3053326336 kernels=3 2026-11-10T09:45:17Z
10 host:boot:/boot 330608640 initrd=3 2026-11-10T09:45:17Z
11 host:esp:/boot/efi 9469952 size=1071624192 2026-11-10T09:45:17Z
12 host:apt-cache:/var/cache/apt 92000256 debs=0 2026-11-10T09:45:17Z
13 host:apt-lists:/var/lib/apt/lists 61833216 NA 2026-11-10T09:45:17Z
14 host:agent-state:/var/lib/felhom-agent 509444096 22 2026-11-10T09:45:17Z
15 host:agent-selfupdate:/var/lib/felhom-agent/selfupdate 509177856 binaries=33 2026-11-10T09:45:17Z
16 host:agent-bak:/usr/local/bin/felhom-agent.bak-* 0 0 2026-11-10T09:45:17Z
17 host:os-apply:/var/lib/felhom-os-apply 376832 bundle-prev=3 2026-11-10T09:45:17Z
18 host:tmp:/tmp 0 6 2026-11-10T09:45:17Z
19 host:var-tmp:/var/tmp 20480 2 2026-11-10T09:45:17Z
20 host:root-home:/root 102400 NA 2026-11-10T09:45:17Z
21 host:swap-used 513003520 size=8254386176 2026-11-10T09:45:17Z
22 host:journal-lines-last-hour NA all=11177 sudo=10420 2026-11-10T09:45:19Z
23 host:journal-oldest-entry NA 2026-10-13T01:30:35+02:00 2026-11-10T09:45:19Z
@@ -0,0 +1,49 @@
# searches run 2026-11-10T08:15:31Z — counts are FILES for arch/runbooks/source and LINES for the two registers
## controls
thin arch=32 open=93 closed=213 runbooks=38 source=511
kernel arch=14 open=7 closed=10 runbooks=6 source=67
megtel arch=0 open=0 closed=1 runbooks=0 source=2
zzqxnope arch=0 open=0 closed=0 runbooks=0 source=0
## S1 old kernels: who removes them
autoremove arch=0 open=0 closed=0 runbooks=0 source=0
old kernel|kernel.*(remov|purge|prune)|(remov|purge|prune).*kernel arch=4 open=1 closed=3 runbooks=0 source=4
/lib/modules arch=0 open=0 closed=0 runbooks=0 source=0
## S2 agent self-update staging folder: who prunes it
selfupdate.*(prune|clean|remove old|keep)|(prune|clean).*selfupdate arch=1 open=2 closed=0 runbooks=0 source=1
felhom-agent/selfupdate arch=0 open=0 closed=0 runbooks=0 source=3
## S3 golden archive left in the dump folder
remove-golden|golden.*(archive|tar\.zst).*(delete|remove|prune|keep) arch=1 open=0 closed=1 runbooks=3 source=1
vzdump-lxc-9100 arch=0 open=0 closed=0 runbooks=0 source=1
## S4 host-disk alarm: who receives it
host_disk_(warning|critical) arch=0 open=0 closed=0 runbooks=0 source=3
storage_fill_(warning|critical) arch=0 open=0 closed=0 runbooks=0 source=4
host_disk_ arch=0 open=0 closed=0 runbooks=0 source=3
storage_fill_ arch=1 open=0 closed=0 runbooks=0 source=4
## S5 a space budget for the host system disk
space budget|disk budget|budget for (the )?(host|root|system) arch=0 open=0 closed=0 runbooks=0 source=0
## S6 a written way out of a full disk
disk[- ]full|No space left|ENOSPC arch=1 open=0 closed=2 runbooks=0 source=2
megtelt|betelt|tele van arch=0 open=0 closed=1 runbooks=0 source=2
## S7 helper images: a rule that deletes or refreshes them
helper image arch=0 open=2 closed=0 runbooks=0 source=0
hello-world arch=0 open=0 closed=0 runbooks=0 source=1
"alpine" arch=0 open=0 closed=0 runbooks=0 source=6
## S8 off-site restore copy: an age or size limit
offsite-restore arch=1 open=0 closed=0 runbooks=1 source=8
offsite-restore.*(days|limit|age|sweep|prune)|(sweep|prune).*offsite-restore arch=0 open=0 closed=0 runbooks=0 source=1
## S9 guest root filesystem: a watcher
guest.{0,12}root ?(fs|filesystem).*(warn|alarm|watch)|rootfs.*(warn|alarm|fill) arch=1 open=0 closed=1 runbooks=0 source=6
## S10 restic cache
restic.{0,6}cache|RESTIC_CACHE_DIR|--cache-dir|--no-cache arch=0 open=0 closed=0 runbooks=0 source=0
## S11 template cache
template/cache|pveam arch=0 open=0 closed=1 runbooks=2 source=0
## S12 container log cap re-check on a running box
max-size|log-opts arch=0 open=0 closed=0 runbooks=0 source=1
## S13 EFI partition: what happens when it fills, who watches it
/boot/efi arch=0 open=0 closed=0 runbooks=0 source=7
(boot/efi|\besp\b|efi partition).{0,80}(full|fills|free space|no space|percent|megtel) arch=0 open=0 closed=0 runbooks=0 source=0
## S14 the system log's storage and cap (host and guest)
journald\.conf|SystemMaxUse|RuntimeMaxUse|Storage=(volatile|persistent|auto) arch=0 open=1 closed=0 runbooks=0 source=0
guest.{0,40}journal.{0,60}(memory|volatile|/run|lost|persist) arch=1 open=0 closed=0 runbooks=0 source=0
## S15 the log of a failed whole-box backup run
vzdump.{0,60}\.log.{0,60}(remove|delete|prune|keep|stay)|failed.{0,30}vzdump.{0,30}log arch=0 open=0 closed=0 runbooks=0 source=0
@@ -0,0 +1,78 @@
#!/usr/bin/env python3
"""Print the text behind every pointer in a table file, so a person can read it.
Read-only by construction: it opens files for reading and prints. It runs on DooPlex against the
git clones only; it never touches a box.
Usage: check-pointers.py [FILE ...] (default: ../TABLE.md)
check-pointers.py --quiet [FILE] (only the verdict lines; rc 1 when a pointer is dead)
A pointer is a backticked token `PREFIX/path[:line[-line]]` with one of the prefixes below.
A pointer is DEAD when the file is missing or the line range lies beyond the end of the file.
The script cannot judge whether the line says what the table claims — that is the reader's job.
"""
import os
import re
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
AUDIT = os.path.dirname(HERE)
REPO = os.path.abspath(os.path.join(AUDIT, "..", "..", "..")) # felhom.eu
WORKSPACE = os.path.dirname(REPO)
PREFIX = {
"AG": os.path.join(WORKSPACE, "felhom-agent"),
"CT": os.path.join(WORKSPACE, "felhom-controller", "controller"),
"CAT": os.path.join(WORKSPACE, "app-catalog-felhom.eu"),
"HUB": os.path.join(REPO, "hub", "internal"),
"E": REPO,
"AR": os.path.join(REPO, "documentation", "architecture"),
"RB": os.path.join(REPO, "documentation", "runbooks"),
"EV": os.path.join(AUDIT, "evidence"),
}
TOKEN = re.compile(r"`((?:%s)/[^`:\s]+)(?::(\d+)(?:-(\d+))?)?`" % "|".join(PREFIX))
def main() -> int:
args = [a for a in sys.argv[1:] if a != "--quiet"]
quiet = "--quiet" in sys.argv[1:]
files = args or [os.path.join(AUDIT, "TABLE.md")]
seen, dead, total = set(), 0, 0
for table in files:
with open(table, encoding="utf-8") as fh:
text = fh.read()
for m in TOKEN.finditer(text):
key = m.group(0)
if key in seen:
continue
seen.add(key)
total += 1
pre, rest = m.group(1).split("/", 1)
path = os.path.join(PREFIX[pre], rest)
first = int(m.group(2)) if m.group(2) else None
last = int(m.group(3)) if m.group(3) else first
if not os.path.exists(path):
dead += 1
print(f"DEAD {key} (no such file: {path})")
continue
if first is None:
if not quiet:
print(f"OK {key} (file exists)")
continue
with open(path, encoding="utf-8", errors="replace") as fh:
lines = fh.readlines()
if last > len(lines):
dead += 1
print(f"DEAD {key} (file has {len(lines)} lines)")
continue
if quiet:
continue
print(f"OK {key}")
for n in range(first, min(last, first + 11) + 1):
print(f" {n:>5} {lines[n - 1].rstrip()[:200]}")
print(f"pointers: {total} dead: {dead}")
return 1 if dead else 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,36 @@
#!/bin/bash
# Space list 2026-11-10 — guest baseline. READ-ONLY by construction: df, du, find -printf, docker ls/inspect --format (no Env).
# Sizes are ALLOCATED bytes (du -B1), not apparent size.
# Output: place <TAB> bytes <TAB> entries <TAB> UTC time. Run inside the LXC guest. Never reads file contents; no names below tier/app folders.
set -u
T() { date -u +%Y-%m-%dT%H:%M:%SZ; }
row() { printf '%s\t%s\t%s\t%s\n' "$1" "$2" "$3" "$(T)"; }
dub() { [ -e "$1" ] && nice -n 19 ionice -c3 du -sx -B1 "$1" 2>/dev/null | cut -f1 || echo NA; }
cnt() { [ -d "$1" ] && find "$1" -xdev -maxdepth 1 -mindepth 1 2>/dev/null | wc -l || echo NA; }
if pgrep -x restic >/dev/null || pgrep -x rsync >/dev/null || pgrep -x apt-get >/dev/null; then row "IN-FLIGHT-skip-walk" NA NA; exit 3; fi
row "guest:rootfs-used" "$(df -B1 --output=used / | tail -1 | tr -d ' ')" "size=$(df -B1 --output=size / | tail -1 | tr -d ' ')"
row "guest:data-volume-used:/var/lib/felhom" "$(df -B1 --output=used /var/lib/felhom | tail -1 | tr -d ' ')" "size=$(df -B1 --output=size /var/lib/felhom | tail -1 | tr -d ' ')"
DR=$(docker info --format '{{.DockerRootDir}}' 2>/dev/null); DR=${DR:-/var/lib/docker}
row "guest:docker-images-layers:$DR/overlay2" "$(dub "$DR/overlay2")" "images=$(docker image ls -q --no-trunc | sort -u | wc -l) untagged=$(docker image ls -qf dangling=true | wc -l)"
row "guest:docker-volumes:$DR/volumes" "$(dub "$DR/volumes")" "all=$(docker volume ls -q | wc -l) anonymous=$(docker volume ls -q | grep -cE '^[0-9a-f]{64}$') unused=$(docker volume ls -qf dangling=true | wc -l)"
row "guest:container-logs:$DR/containers/*/*-json.log*" "$(find "$DR/containers" -name '*-json.log*' -printf '%s\n' 2>/dev/null | awk '{s+=$1} END{print s+0}')" "files=$(find "$DR/containers" -name '*-json.log*' 2>/dev/null | wc -l) containers=$(docker ps -aq | wc -l)"
row "guest:docker-buildcache:$DR/buildkit" "$(dub "$DR/buildkit")" "NA"
row "guest:controller-data-volume" "$(dub "$DR/volumes/felhom-controller-data")" "NA"
row "guest:stacks:/opt/docker/stacks" "$(dub /opt/docker/stacks)" "$(cnt /opt/docker/stacks)"
row "guest:sys-drive-backups:/var/lib/felhom/sys_drive/felhom-data/backups" "$(dub /var/lib/felhom/sys_drive/felhom-data/backups)" "$(find /var/lib/felhom/sys_drive/felhom-data/backups -maxdepth 2 -mindepth 2 -type d 2>/dev/null | wc -l)"
row "guest:journal" "$(journalctl --disk-usage 2>/dev/null | grep -oE '[0-9.]+[KMG]' | head -1)" "storage=$(ls -d /var/log/journal/* >/dev/null 2>&1 && echo persistent || echo volatile)"
row "guest:var-log:/var/log" "$(dub /var/log)" "NA"
row "guest:apt-cache:/var/cache/apt" "$(dub /var/cache/apt)" "NA"
row "guest:apt-lists:/var/lib/apt/lists" "$(dub /var/lib/apt/lists)" "NA"
row "guest:tmp:/tmp" "$(dub /tmp)" "$(cnt /tmp)"
row "guest:var-tmp:/var/tmp" "$(dub /var/tmp)" "$(cnt /var/tmp)"
for m in /mnt/felhom-drives/*; do
[ -d "$m" ] || continue
mountpoint -q "$m" || { row "drive:$m NOT-MOUNTED (bare folder on the host root disk)" "$(dub "$m")" "$(cnt "$m")"; continue; }
b="$m"; [ -d "$m/felhom-data" ] && b="$m/felhom-data" # two layouts exist in the fleet
row "drive:$m used" "$(df -B1 --output=used "$m" | tail -1 | tr -d ' ')" "size=$(df -B1 --output=size "$m" | tail -1 | tr -d ' ')"
for t in backups/primary backups/secondary backups/offsite-restore backups/offsite-proof backups/restore-undo kept appdata userdata shares; do
[ -d "$b/$t" ] && row "drive:$b/$t" "$(dub "$b/$t")" "$(cnt "$b/$t")"
done
done
exit 0
@@ -0,0 +1,39 @@
#!/bin/bash
# READ-ONLY discovery of every place that holds data inside a Felhom guest (LXC).
# Sizes, counts and dates only. No file contents of app or household data. Never dumps Env.
export LC_ALL=C
N="nice -n 19 ionice -c3"
sec() { echo; echo "### $*"; }
sec "identity"; hostname; date -u +%FT%TZ; grep PRETTY /etc/os-release; cat /etc/felhom-scratch-disposition 2>/dev/null | head -3
sec "df bytes"; df -B1 -T -x tmpfs -x devtmpfs -x overlay -x efivarfs 2>/dev/null
sec "df inodes"; df -i -x tmpfs -x devtmpfs -x overlay -x efivarfs 2>/dev/null
sec "docker info"; docker info --format 'driver={{.Driver}} root={{.DockerRootDir}} logdriver={{.LoggingDriver}} images={{.Images}} containers={{.Containers}} running={{.ContainersRunning}} server={{.ServerVersion}}'
sec "daemon.json"; cat /etc/docker/daemon.json 2>/dev/null; ls -la --time-style=long-iso /etc/docker/ 2>/dev/null
sec "docker system df"; docker system df 2>&1
sec "containers (name image state logdriver logopts)"
for c in $(docker ps -aq); do docker inspect --format '{{.Name}} {{.Config.Image}} {{.State.Status}} {{.HostConfig.LogConfig.Type}} {{json .HostConfig.LogConfig.Config}} restart={{.RestartCount}} created={{.Created}}' "$c"; done | sort
sec "container log files (bytes path) and total"
R=$(docker info --format '{{.DockerRootDir}}')
find "$R/containers" -maxdepth 2 -type f -name '*.log*' -printf '%s\t%TY-%Tm-%Td\t%p\n' 2>/dev/null | sort -rn | head -60
find "$R/containers" -maxdepth 2 -type f -name '*.log*' -printf '%s\n' 2>/dev/null | awk '{s+=$1;n++} END{print "TOTAL bytes="s" files="n}'
sec "container writable layers (docker ps -as)"; docker ps -as --format '{{.Names}} {{.Size}}' | sort
sec "images"; docker images -a --format '{{.Repository}}:{{.Tag}} {{.ID}} {{.CreatedAt}} {{.Size}}' | sort
sec "images count / dangling"; docker images -q | sort -u | wc -l; docker images -f dangling=true -q | wc -l
sec "images in use by a container"; docker ps -a --format '{{.Image}}' | sort | uniq -c
sec "volumes (count, anonymous count, dangling count)"; docker volume ls -q | wc -l; docker volume ls -q | grep -cE '^[0-9a-f]{64}$'; docker volume ls -f dangling=true -q | wc -l
sec "volumes by size (system df -v, Local Volumes block)"; docker system df -v 2>/dev/null | awk '/^Local Volumes/{p=1} /^Build cache/{p=0} p' | head -150
sec "build cache"; docker system df -v 2>/dev/null | awk '/^Build cache/{p=1} p' | head -8
sec "docker root du depth 1"; $N du -x -B1 --max-depth=1 "$R" 2>/dev/null | sort -rn
sec "journald"; journalctl --disk-usage 2>&1; grep -rhv '^#' /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ 2>/dev/null | grep -v '^$'; journalctl -u systemd-journald --no-pager 2>/dev/null | grep -i 'max\|vacuum\|deleted' | tail -4
sec "/var/log"; $N du -x -B1 --max-depth=1 /var/log 2>/dev/null | sort -rn | head -15; find /var/log -xdev -type f -size +5M -printf '%s\t%TY-%Tm-%Td\t%p\n' 2>/dev/null | sort -rn | head
sec "apt"; $N du -x -B1 -s /var/cache/apt /var/lib/apt/lists 2>/dev/null; ls /var/cache/apt/archives/*.deb 2>/dev/null | wc -l
sec "tmp"; $N du -x -B1 -s /tmp /var/tmp 2>/dev/null; find /tmp /var/tmp -xdev -mindepth 1 -maxdepth 1 -printf '%TY-%Tm-%Td\t%s\t%p\n' 2>/dev/null | sort | head -40; echo "entries=$(find /tmp /var/tmp -xdev -mindepth 1 -maxdepth 1 | wc -l)"
sec "felhom dirs"; ls -d /var/lib/felhom* /opt/felhom* /opt/docker /etc/felhom* /var/log/felhom* /var/cache/felhom* /srv/* 2>/dev/null
sec "root filesystem top 45 to depth 4"; $N du -x -B1 --max-depth=4 / 2>/dev/null | sort -rn | head -45
sec "/var/lib/felhom depth 3 (data volume)"; $N du -x -B1 --max-depth=3 /var/lib/felhom 2>/dev/null | sort -rn | grep -v "/docker/overlay2/\|/docker/volumes/\|/docker/containers/\|/docker/image/\|/docker/buildkit/" | head -70
sec "/opt/docker depth 2"; $N du -x -B1 --max-depth=2 /opt/docker 2>/dev/null | sort -rn | head -50
sec "drives under /mnt/felhom-drives (depth 3 from each drive)"
for d in /mnt/felhom-drives/*; do [ -d "$d" ] && { echo "-- $d"; df -B1 "$d" | tail -1; $N du -x -B1 --max-depth=3 "$d" 2>/dev/null | sort -rn | head -60; }; done
sec "timers"; systemctl list-timers --all --no-pager 2>/dev/null | head -20
sec "cron"; ls /etc/cron.d /etc/cron.daily /etc/cron.weekly 2>/dev/null
sec "end"; date -u +%FT%TZ
@@ -0,0 +1,18 @@
#!/bin/bash
# Read-only follow-up inside a guest: controller writable layer, restic cache, guest journal, guest timers.
echo "== utc $(date -u +%FT%TZ)"
echo "== controller container size (writable layer / virtual)"
docker ps -s --filter name=felhom-controller --format '{{.Names}} {{.Size}}'
echo "== restic cache folders inside the controller container (size, entries)"
docker exec felhom-controller sh -c 'for d in /root/.cache/restic /tmp /var/tmp; do [ -e "$d" ] && echo "$d $(du -sx -B1 "$d" 2>/dev/null | cut -f1) $(ls -A "$d" 2>/dev/null | wc -l)"; done; echo HOME=$HOME; echo XDG_CACHE_HOME=${XDG_CACHE_HOME:-unset}; echo RESTIC_CACHE_DIR=${RESTIC_CACHE_DIR:-unset}'
echo "== controller state volume top level (sizes only)"
docker exec felhom-controller sh -c 'du -sx -B1 /data/* 2>/dev/null | sort -rn | head -12'
echo "== guest journal"
ls -d /var/log/journal 2>/dev/null || echo "/var/log/journal absent"
journalctl --disk-usage 2>&1 | tail -1
df -B1 --output=target,size,used /run | tail -1
systemctl is-active logrotate.timer systemd-tmpfiles-clean.timer fstrim.timer apt-daily.timer 2>&1 | tr '\n' ' '; echo
findmnt -no FSTYPE /tmp
echo "== images with no container, by repository (name only)"
docker images --format '{{.Repository}}:{{.Tag}} {{.Size}} {{.CreatedSince}}' | grep -E '^(alpine|hello-world|curlimages|busybox|traefik/whoami)'
exit 0
@@ -0,0 +1,40 @@
#!/bin/bash
# Space list 2026-11-10 — host baseline. READ-ONLY by construction: df, du, ls, lvs, find -printf, journalctl --disk-usage.
# Sizes are ALLOCATED bytes (du -B1), not apparent size.
# Output: place <TAB> bytes <TAB> entries <TAB> UTC time. Run as root on a Proxmox host.
set -u
T() { date -u +%Y-%m-%dT%H:%M:%SZ; }
row() { printf '%s\t%s\t%s\t%s\n' "$1" "$2" "$3" "$(T)"; }
dub() { [ -e "$1" ] && nice -n 19 ionice -c3 du -sx -B1 "$1" 2>/dev/null | cut -f1 || echo NA; }
cnt() { [ -d "$1" ] && find "$1" -xdev -maxdepth "${3:-1}" -mindepth 1 ${2:+-name "$2"} 2>/dev/null | wc -l || echo NA; }
# refuse to walk while a backup / restore / update is in flight
if pgrep -x vzdump >/dev/null || pgrep -x apt-get >/dev/null || pgrep -x dpkg >/dev/null || pgrep -f 'pct restore' >/dev/null; then
row "IN-FLIGHT-skip-walk" NA NA; exit 3; fi
row "host:rootfs-used" "$(df -B1 --output=used / | tail -1 | tr -d ' ')" "size=$(df -B1 --output=size / | tail -1 | tr -d ' ')"
lvs --noheadings --units b --nosuffix -o lv_name,lv_size,data_percent,metadata_percent,lv_attr pve 2>/dev/null | awk '$5 ~ /^t/ {printf "%s %s %s %s\n",$1,$2,$3,$4}' | while read n s d m; do
row "host:thinpool-$n-data" "$(awk -v s="$s" -v d="$d" 'BEGIN{printf "%.0f", s*d/100}')" "size=$s pct=$d"
row "host:thinpool-$n-metadata-pct" "NA" "pct=$m"
done
for d in $(awk '/^dir:/{f=1;next} /^[a-z]+:/{f=0} f&&$1=="path"{print $2}' /etc/pve/storage.cfg 2>/dev/null); do
row "host:dump:$d/dump" "$(dub "$d/dump")" "archives=$(cnt "$d/dump" 'vzdump-*.tar.zst') logs=$(cnt "$d/dump" '*.log') all=$(cnt "$d/dump")"
row "host:template-cache:$d/template" "$(dub "$d/template")" "$(cnt "$d/template" '' 2)"
done
row "host:journal:/var/log/journal" "$(dub /var/log/journal)" "files=$(find /var/log/journal -type f 2>/dev/null | wc -l)"
row "host:var-log-other" "$(nice -n 19 ionice -c3 du -sx -B1 --exclude=/var/log/journal /var/log 2>/dev/null | cut -f1)" "pve-tasks=$(find /var/log/pve/tasks -type f 2>/dev/null | wc -l)"
row "host:rrdcached:/var/lib/rrdcached" "$(dub /var/lib/rrdcached)" "NA"
row "host:kernels:/lib/modules" "$(dub /lib/modules)" "kernels=$(ls /lib/modules 2>/dev/null | wc -l)"
row "host:boot:/boot" "$(nice du -sx -B1 /boot 2>/dev/null | cut -f1)" "initrd=$(ls /boot/initrd.img-* 2>/dev/null | wc -l)"
mountpoint -q /boot/efi && row "host:esp:/boot/efi" "$(df -B1 --output=used /boot/efi | tail -1 | tr -d ' ')" "size=$(df -B1 --output=size /boot/efi | tail -1 | tr -d ' ')"
row "host:apt-cache:/var/cache/apt" "$(dub /var/cache/apt)" "debs=$(cnt /var/cache/apt/archives '*.deb')"
row "host:apt-lists:/var/lib/apt/lists" "$(dub /var/lib/apt/lists)" "NA"
row "host:agent-state:/var/lib/felhom-agent" "$(dub /var/lib/felhom-agent)" "$(cnt /var/lib/felhom-agent)"
row "host:agent-selfupdate:/var/lib/felhom-agent/selfupdate" "$(dub /var/lib/felhom-agent/selfupdate)" "binaries=$(cnt /var/lib/felhom-agent/selfupdate 'felhom-agent-*')"
row "host:agent-bak:/usr/local/bin/felhom-agent.bak-*" "$(find /usr/local/bin -maxdepth 1 -name 'felhom-agent.bak-*' -printf '%s\n' 2>/dev/null | awk '{s+=$1} END{print s+0}')" "$(cnt /usr/local/bin 'felhom-agent.bak-*')"
row "host:os-apply:/var/lib/felhom-os-apply" "$(dub /var/lib/felhom-os-apply)" "bundle-prev=$(cnt /var/lib/felhom-os-apply/bundle-prev)"
row "host:tmp:/tmp" "$(dub /tmp)" "$(cnt /tmp)"
row "host:var-tmp:/var/tmp" "$(dub /var/tmp)" "$(cnt /var/tmp)"
row "host:root-home:/root" "$(dub /root)" "NA"
row "host:swap-used" "$(awk 'NR>1{s+=$4} END{print s*1024}' /proc/swaps)" "size=$(awk 'NR>1{s+=$3} END{print s*1024}' /proc/swaps)"
row "host:journal-lines-last-hour" "NA" "all=$(journalctl --since '-1h' -q --no-pager -o cat 2>/dev/null | wc -l) sudo=$(journalctl --since '-1h' -q --no-pager -t sudo -o cat 2>/dev/null | wc -l)"
row "host:journal-oldest-entry" "NA" "$(journalctl -q --no-pager -o short-iso 2>/dev/null | head -1 | awk '{print $1}')"
exit 0
@@ -0,0 +1,49 @@
#!/bin/bash
# READ-ONLY discovery of every place that holds data on a Felhom Proxmox host.
# Sizes, counts and dates only. No file contents. No command here can write.
export LC_ALL=C
N="nice -n 19 ionice -c3"
sec() { echo; echo "### $*"; }
sec "identity"; hostname; date -u +%FT%TZ; pveversion 2>/dev/null; felhom-agent --version 2>/dev/null
sec "in-flight (must be empty)"; pgrep -af 'vzdump|proxmox-backup-client|restic |apt-get|dpkg |pct restore|qmrestore' | grep -v pgrep
sec "lsblk"; lsblk -b -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL,TRAN
sec "df bytes"; df -B1 -T -x tmpfs -x devtmpfs -x overlay -x efivarfs
sec "df inodes"; df -i -x tmpfs -x devtmpfs -x overlay -x efivarfs
sec "vgs"; vgs --units b --nosuffix -o vg_name,vg_size,vg_free 2>/dev/null
sec "lvs"; lvs --units b --nosuffix -o lv_name,vg_name,lv_attr,lv_size,pool_lv,data_percent,metadata_percent,lv_metadata_size 2>/dev/null
sec "thin pool autoextend + monitoring"; lvmconfig activation/thin_pool_autoextend_threshold activation/thin_pool_autoextend_percent activation/monitoring 2>&1
sec "pvesm status"; pvesm status 2>&1
sec "storage.cfg (no secrets live here; passwords are in /etc/pve/priv)"; grep -v -i 'password\|keyring' /etc/pve/storage.cfg
sec "pvesm list per storage (count, bytes, oldest, newest)"
for s in $(pvesm status 2>/dev/null | awk 'NR>1 && $3=="active"{print $1}'); do
pvesm list "$s" 2>/dev/null | awk -v s="$s" 'NR>1{n++; b+=$4; split($1,a,"/"); k=$3; c[k]++; cb[k]+=$4} END{printf "%s total entries=%d bytes=%d\n", s, n, b; for(k in c) printf " %s type=%s entries=%d bytes=%d\n", s, k, c[k], cb[k]}'
done
sec "guest configs (disk + mountpoint lines only)"
for f in /etc/pve/lxc/*.conf /etc/pve/qemu-server/*.conf; do [ -f "$f" ] && { echo "-- $f"; grep -E '^(rootfs|mp[0-9]+|scsi[0-9]+|virtio[0-9]+|sata[0-9]+|efidisk|tpmstate|unused[0-9]+|memory|swap|\[)' "$f"; }; done
sec "vzdump.conf + jobs (keep rules)"; grep -v '^#' /etc/vzdump.conf 2>/dev/null; cat /etc/pve/jobs.cfg 2>/dev/null
sec "journald"; journalctl --disk-usage 2>&1; grep -rhv '^#' /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ 2>/dev/null | grep -v '^$'; ls -la /var/log/journal/*/ 2>/dev/null | awk 'NR>1' | wc -l; journalctl --list-boots 2>/dev/null | sed -n '1p;$p'
sec "/var/log top"; $N du -x -B1 --max-depth=1 /var/log 2>/dev/null | sort -rn | head -25
sec "/var/log big files"; find /var/log -xdev -type f -size +5M -printf '%s\t%TY-%Tm-%Td\t%p\n' 2>/dev/null | sort -rn | head -30
sec "logrotate configs present"; ls /etc/logrotate.d/
sec "kernels installed"; dpkg-query -W -f='${Package}\t${Version}\t${Installed-Size}\t${db:Status-Abbrev}\n' 'proxmox-kernel-*' 'pve-kernel-*' 'linux-image-*' 2>/dev/null; uname -r
sec "proxmox-boot-tool"; proxmox-boot-tool kernel list 2>&1; proxmox-boot-tool status 2>&1 | head -20
sec "/boot"; df -B1 /boot /boot/efi 2>/dev/null; $N du -x -B1 --max-depth=1 /boot 2>/dev/null | sort -rn; ls -la /boot | awk '{print $5"\t"$6" "$7" "$8"\t"$9}' | sort -rn | head -30
sec "/lib/modules"; $N du -x -B1 --max-depth=1 /lib/modules 2>/dev/null | sort -rn
sec "apt"; $N du -x -B1 -s /var/cache/apt /var/lib/apt/lists /var/cache/apt/archives 2>/dev/null; ls /var/cache/apt/archives/*.deb 2>/dev/null | wc -l; apt-config dump 2>/dev/null | grep -i -E 'Periodic|Unattended|AutoRemove|Clean' | head -30; ls /etc/apt/apt.conf.d/
sec "apt autoremove candidates (dry read)"; apt-get -s autoremove 2>/dev/null | grep -c '^Remv'
sec "tmp"; $N du -x -B1 -s /tmp /var/tmp 2>/dev/null; find /tmp /var/tmp -xdev -mindepth 1 -maxdepth 1 -printf '%TY-%Tm-%Td\t%s\t%p\n' 2>/dev/null | sort | sed -n '1,5p;$p'; find /tmp /var/tmp -xdev -mindepth 1 -maxdepth 1 | wc -l
sec "systemd-tmpfiles rules touching tmp"; grep -rh -E '^\s*[dDqQ]\s+/(var/)?tmp' /usr/lib/tmpfiles.d /etc/tmpfiles.d 2>/dev/null
sec "/var/lib/vz"; $N du -x -B1 --max-depth=2 /var/lib/vz 2>/dev/null | sort -rn | head -20; for d in /var/lib/vz/dump /var/lib/vz/template/cache /var/lib/vz/template/iso; do echo "-- $d"; find "$d" -maxdepth 1 -type f -printf '%s\t%TY-%Tm-%Td\t%f\n' 2>/dev/null | sort -k2; done
sec "felhom dirs (candidates)"; ls -d /var/lib/felhom* /opt/felhom* /etc/felhom* /var/log/felhom* /usr/local/lib/felhom* /var/cache/felhom* /root/.felhom* /srv/felhom* /run/felhom* 2>/dev/null
for d in $(ls -d /var/lib/felhom* /opt/felhom* /etc/felhom* /var/log/felhom* /var/cache/felhom* /usr/local/lib/felhom* 2>/dev/null); do echo "-- $d"; $N du -x -B1 --max-depth=2 "$d" 2>/dev/null | sort -rn | head -25; echo " files=$(find "$d" -xdev -type f 2>/dev/null | wc -l) oldest=$(find "$d" -xdev -type f -printf '%TY-%Tm-%Td\n' 2>/dev/null | sort | head -1) newest=$(find "$d" -xdev -type f -printf '%TY-%Tm-%Td\n' 2>/dev/null | sort | tail -1)"; done
sec "felhom binaries + slots"; ls -la /usr/local/bin/ /usr/local/sbin/ 2>/dev/null | awk '{print $5"\t"$6" "$7" "$8"\t"$9" "$10" "$11}'
sec "root filesystem, top 70 dirs to depth 4"; $N du -x -B1 --max-depth=4 / 2>/dev/null | sort -rn | head -70
sec "root fs: file count per top dir"; for d in /var /usr /root /home /opt /etc /srv /tmp; do printf '%s\t%s\n' "$(find $d -xdev -type f 2>/dev/null | wc -l)" "$d"; done
sec "root home"; $N du -x -B1 --max-depth=1 /root 2>/dev/null | sort -rn | head -15
sec "other mounted filesystems, depth 2"
for m in $(df --output=target -x tmpfs -x devtmpfs -x overlay -x efivarfs | awk 'NR>1 && $1!="/" && $1!~/^\/boot/ && $1!~/^\/etc\/pve/'); do echo "-- $m"; $N du -x -B1 --max-depth=2 "$m" 2>/dev/null | sort -rn | head -40; done
sec "pve task logs + rrd"; $N du -x -B1 -s /var/log/pve /var/lib/rrdcached /var/lib/pve-cluster /var/lib/pve-manager 2>/dev/null; find /var/log/pve/tasks -type f 2>/dev/null | wc -l; find /var/log/pve/tasks -type f -printf '%TY-%Tm-%Td\n' 2>/dev/null | sort | sed -n '1p;$p'
sec "core dumps"; $N du -x -B1 -s /var/lib/systemd/coredump 2>/dev/null; ls /var/lib/systemd/coredump 2>/dev/null | wc -l
sec "swap"; swapon --show --bytes 2>/dev/null
sec "timers"; systemctl list-timers --all --no-pager 2>/dev/null | head -40
sec "end"; date -u +%FT%TZ
@@ -0,0 +1,33 @@
#!/bin/bash
# Space list 2026-11-10 — host follow-up. READ-ONLY by construction: ls, stat, du, cat of scripts, journalctl reads.
N="nice -n 19 ionice -c3"
echo "### identity"; hostname; date -u +%FT%TZ
echo "### in flight"; pgrep -a 'vzdump|restic|proxmox-backup|apt|dpkg|pct restore' 2>/dev/null | grep -v pgrep | cut -c1-160; echo "(end in flight)"
echo "### agent state dir (depth 2, bytes)"; $N du -x -B1 --max-depth=2 /var/lib/felhom-agent 2>/dev/null | sort -rn | head -40
echo "### selfupdate listing"; ls -la --time-style=long-iso /var/lib/felhom-agent/selfupdate 2>/dev/null | head -60
echo "### selfupdate subdirs listing"; for d in /var/lib/felhom-agent/selfupdate/*/; do [ -d "$d" ] && { echo "-- $d"; ls -la --time-style=long-iso "$d" | head -40; }; done
echo "### agent state top-level listing (names, sizes, dates only)"; ls -la --time-style=long-iso /var/lib/felhom-agent 2>/dev/null
echo "### os-apply state"; $N du -x -B1 --max-depth=2 /var/lib/felhom-os-apply 2>/dev/null | sort -rn | head; ls -la --time-style=long-iso /var/lib/felhom-os-apply /var/lib/felhom-os-apply/bundle-prev 2>/dev/null
echo "### usr local bin + sbin"; ls -la --time-style=long-iso /usr/local/bin /usr/local/sbin 2>/dev/null
echo "### dumps on drives"; for d in /mnt/*/dump /var/lib/vz/dump; do [ -d "$d" ] && { echo "-- $d"; ls -la --time-style=long-iso "$d"; }; done
echo "### template cache"; ls -la --time-style=long-iso /var/lib/vz/template/cache /var/lib/vz/template/iso 2>/dev/null
echo "### journald disk usage + limits"; journalctl --disk-usage 2>&1; journalctl -b -u systemd-journald --no-pager 2>/dev/null | grep -i -E 'max|System Journal' | tail -3
echo "### journal oldest + newest entry"; journalctl --no-pager -o short-iso -n 1 --reverse 2>/dev/null | tail -1; journalctl --no-pager -o short-iso 2>/dev/null | head -2 | cut -c1-120
echo "### journal files (count, oldest, newest)"; find /var/log/journal -type f -printf '%TY-%Tm-%Td %s\n' 2>/dev/null | sort | awk 'NR==1{o=$1} {n=$1; c++; s+=$2} END{print "files="c" bytes="s" oldest="o" newest="n}'
echo "### journal bytes exact"; $N du -x -sB1 /var/log/journal 2>/dev/null
echo "### journal last hour: lines by identifier (top 12)"; journalctl --since '-1h' --no-pager -o json 2>/dev/null | grep -o '"SYSLOG_IDENTIFIER":"[^"]*"' | sort | uniq -c | sort -rn | head -12
echo "### journal last hour: total lines"; journalctl --since '-1h' --no-pager -o cat 2>/dev/null | wc -l
echo "### journald conf files"; ls /etc/systemd/journald.conf.d/ 2>/dev/null; grep -v '^#' /etc/systemd/journald.conf 2>/dev/null | grep -v '^$'
echo "### thin-check script"; for f in /usr/local/sbin/felhom-thin-check /usr/local/sbin/felhom-vzdump-partial-clean; do echo "-- $f"; [ -f "$f" ] && cat "$f" | head -120; done
echo "### units mentioning thin-check / partial-clean"; grep -rl -E 'felhom-thin-check|vzdump-partial-clean' /etc/systemd/system /etc/cron* /etc/sudoers.d 2>/dev/null
echo "### lvm thin settings"; lvmconfig --type full activation/thin_pool_autoextend_threshold activation/thin_pool_autoextend_percent activation/monitoring 2>/dev/null; systemctl is-active lvm2-monitor 2>/dev/null
echo "### agent config: selected non-secret keys"; for f in /etc/felhom-agent/*.json /etc/felhom-agent/*.yaml /etc/felhom-agent/*.yml; do [ -f "$f" ] && { echo "-- $f"; grep -o -E '"?(local_backup_target|backup_target|keep_daily|keep_last|retention[a-z_]*|local_keep[a-z_]*)"?[ ]*[:=][ ]*[^,}]*' "$f" | head; }; done
echo "### vzdump.conf + storage prune settings"; grep -v '^#' /etc/vzdump.conf 2>/dev/null | grep -v '^$'; grep -n -E '^(dir|lvmthin|pbs)|prune|maxfiles|content|path|is_mountpoint|mkdir' /etc/pve/storage.cfg 2>/dev/null
echo "### kernels"; dpkg-query -W -f='${Package}\t${Version}\t${Installed-Size}\t${db:Status-Abbrev}\n' 'proxmox-kernel-*' 2>/dev/null | grep -v '^proxmox-kernel-[0-9.]*\s' | head -30; uname -r
echo "### apt history: kernel installs/removes"; zgrep -h -E '^(Start-Date|Install|Remove|Purge):' /var/log/apt/history.log* 2>/dev/null | grep -B1 -E 'proxmox-kernel' | grep -E 'Start-Date|Remove|Purge' | cut -c1-200 | tail -24
echo "### apt autoremove config"; apt-config dump 2>/dev/null | grep -i -E 'AutoRemove|Unattended|Periodic' | cut -c1-200 | head -30
echo "### boot"; df -B1 /boot /boot/efi 2>/dev/null; ls -la --time-style=long-iso /boot | head -40
echo "### pve tasks / rrd"; $N du -x -sB1 /var/log/pve /var/lib/rrdcached 2>/dev/null; find /var/log/pve/tasks -type f 2>/dev/null | wc -l
echo "### root dir extras"; $N du -x -B1 --max-depth=1 /root 2>/dev/null | sort -rn | head -12
echo "### df now"; df -B1 --output=source,size,used,avail,pcent,target / 2>/dev/null; lvs --noheadings -o lv_name,lv_size,data_percent,metadata_percent --units b pve 2>/dev/null
echo "### end"; date -u +%FT%TZ
@@ -0,0 +1,127 @@
#!/usr/bin/env python3
"""Build MATRIX.md from TABLE.md, and check the table's shape.
Read-only on everything but MATRIX.md in this folder. It runs on DooPlex against the git clone only.
Usage: make-matrix.py write ../MATRIX.md, print the totals
make-matrix.py --check write nothing; rc 1 when the table breaks a rule
Rules checked (a broken rule is printed and makes rc 1):
- a place heading is `### <ID> — <name> · CUSTOMER · NAMED|MENTIONED|UNNAMED`
- each place has exactly eight answer rows, numbered 1..8, each with one mark A, D, K or N
- an A row carries a backticked pointer; a K row an `R-<n>`; an N row one class C1..C5 and a search S<n>
"""
import os
import re
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
AUDIT = os.path.dirname(HERE)
HEAD = re.compile(r"^### ([HGD]\d+) — (.+?) · CUSTOMER · (NAMED|MENTIONED|UNNAMED)\s*$")
ROW = re.compile(r"^\| ([1-8]) \| (.*) \| ([ADKN]) \| (.*) \|\s*$")
QUESTIONS = ["what is in it", "what makes it grow", "the limit", "who cleans up", "what happens when full",
"who gets an alarm", "self-repair or runbook", "how do we know"]
CLASS_ROW = {"C1": "R-1368", "C2": "R-1369", "C3": "R-1370", "C4": "R-1371", "C5": "R-1372"}
def parse(path):
places, errors, cur = [], [], None
with open(path, encoding="utf-8") as fh:
for ln, line in enumerate(fh, 1):
if line.startswith("## TEST-ONLY") or line.startswith("## DOC-ONLY"):
cur = None
continue
if line.startswith("### "):
m = HEAD.match(line)
if not m:
errors.append(f"line {ln}: heading has the wrong shape: {line.strip()}")
cur = None
continue
cur = {"id": m.group(1), "name": m.group(2), "named": m.group(3), "marks": {}, "n": {}, "k": {}}
places.append(cur)
continue
m = ROW.match(line)
if not m or cur is None:
continue
q, mark, ptr = int(m.group(1)), m.group(3), m.group(4)
if q in cur["marks"]:
errors.append(f"{cur['id']}: question {q} is answered twice")
cur["marks"][q] = mark
if mark == "A" and "`" not in ptr:
errors.append(f"{cur['id']}-{q}: A with no pointer")
if mark == "K":
rows = re.findall(r"R-\d+", ptr)
if not rows:
errors.append(f"{cur['id']}-{q}: K with no row")
cur["k"][q] = rows
if mark == "N":
cls = re.findall(r"\bC[1-5]\b", ptr)
if len(cls) != 1:
errors.append(f"{cur['id']}-{q}: N needs exactly one class, found {cls}")
elif not re.search(r"\bS\d+\b", ptr) and "NOBODY KNOWS" not in m.group(2):
errors.append(f"{cur['id']}-{q}: N with no search id")
cur["n"][q] = cls[0] if cls else "?"
for p in places:
if sorted(p["marks"]) != list(range(1, 9)):
errors.append(f"{p['id']}: has answers {sorted(p['marks'])}, wants 1..8")
return places, errors
def build(places):
out = ["# The Space list — the matrix", "",
"Made by `tools/make-matrix.py` from `TABLE.md`. Do not edit it by hand.", "",
"`A` answered · `D` decided, not built · `K` known gap (has a row) · `N` nobody asked. Columns are the eight",
"questions: 1 content · 2 growth · 3 limit · 4 cleaner · 5 when full · 6 alarm · 7 repair or runbook · 8 how we know.",
"", "| Place | Named | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | N |", "|---|---|---|---|---|---|---|---|---|---|---|"]
tot = {m: 0 for m in "ADKN"}
perq = {q: {m: 0 for m in "ADKN"} for q in range(1, 9)}
named = {"NAMED": 0, "MENTIONED": 0, "UNNAMED": 0}
members, krows = {c: [] for c in CLASS_ROW}, {}
for p in places:
named[p["named"]] += 1
cells = []
for q in range(1, 9):
mk = p["marks"].get(q, "?")
cells.append(f"**{mk}**" if mk == "N" else mk)
if mk in tot:
tot[mk] += 1
perq[q][mk] += 1
for q, c in p["n"].items():
members.setdefault(c, []).append(f"{p['id']}-{q}")
for q, rows in p["k"].items():
for r in rows:
krows.setdefault(r, []).append(f"{p['id']}-{q}")
out.append(f"| {p['id']} {p['name']} | {p['named'].lower()} | " + " | ".join(cells) + f" | {len(p['n'])} |")
n = len(places)
out += ["", "## Totals", "",
f"- Places: **{n}** customer places · named **{named['NAMED']}** · mentioned only **{named['MENTIONED']}** · "
f"named by no architecture document **{named['UNNAMED']}**.",
f"- Answers: **{n * 8}** · A **{tot['A']}** · D **{tot['D']}** · K **{tot['K']}** · N **{tot['N']}**.",
f"- Places with no N: **{sum(1 for p in places if not p['n'])}** of {n}.",
"", "| Question | A | D | K | N |", "|---|---|---|---|---|"]
for q in range(1, 9):
out.append(f"| {q} {QUESTIONS[q - 1]} | {perq[q]['A']} | {perq[q]['D']} | {perq[q]['K']} | {perq[q]['N']} |")
out += ["", "## The N answers, by class row", "", "| Class | Row | Count | Members (place-question) |", "|---|---|---|---|"]
for c in sorted(members):
out.append(f"| {c} | {CLASS_ROW.get(c, '?')} | {len(members[c])} | {', '.join(members[c])} |")
out += ["", "## The K answers, by existing row", "", "| Row | Count | Members |", "|---|---|---|"]
for r in sorted(krows, key=lambda x: int(x[2:])):
out.append(f"| {r} | {len(krows[r])} | {', '.join(krows[r])} |")
return "\n".join(out) + "\n", tot, named, members
def main() -> int:
places, errors = parse(os.path.join(AUDIT, "TABLE.md"))
for e in errors:
print("BROKEN " + e)
text, tot, named, members = build(places)
if "--check" not in sys.argv[1:]:
with open(os.path.join(AUDIT, "MATRIX.md"), "w", encoding="utf-8") as fh:
fh.write(text)
print(f"places {len(places)} named {named} marks {tot} "
+ " ".join(f"{c}={len(m)}" for c, m in sorted(members.items())) + f" broken {len(errors)}")
return 1 if errors else 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,13 @@
#!/bin/bash
# Space list 2026-11-10 — which OS-level cleaners exist. READ-ONLY: findmnt, systemctl is-active/show, ls, cat of OS config (no secrets), stat.
echo "### $(hostname) $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "## /tmp fs"; findmnt -no FSTYPE,SIZE,USED /tmp 2>/dev/null || echo "/tmp is on /"
echo "## tmpfiles rules for /tmp and /var/tmp"; systemd-tmpfiles --cat-config 2>/dev/null | grep -E "^[qQdD]\s+/(var/)?tmp\s"
echo "## timers"; for t in systemd-tmpfiles-clean.timer logrotate.timer apt-daily.timer apt-daily-upgrade.timer fstrim.timer; do printf '%s enabled=%s active=%s last=%s\n' $t "$(systemctl is-enabled $t 2>/dev/null)" "$(systemctl is-active $t 2>/dev/null)" "$(systemctl show $t -p LastTriggerUSec --value 2>/dev/null)"; done
echo "## logrotate.d"; ls /etc/logrotate.d 2>/dev/null | tr '\n' ' '; echo
echo "## journald effective"; systemd-analyze cat-config systemd/journald.conf 2>/dev/null | grep -E "^\s*(Storage|SystemMaxUse|SystemKeepFree|RuntimeMaxUse|MaxRetentionSec|MaxFileSec)=" || echo "(all defaults)"
echo "## apt periodic"; apt-config dump 2>/dev/null | grep -E "APT::Periodic|Unattended-Upgrade::Remove|Kernel-Packages|NeverAutoRemove" | cut -c1-160 | head -20
echo "## pve task index"; if [ -d /var/log/pve/tasks ]; then wc -l /var/log/pve/tasks/index /var/log/pve/tasks/index.1 2>/dev/null; find /var/log/pve/tasks -type f -name 'UPID*' -printf '%TY-%Tm-%Td\n' | sort | sed -n '1p;$p'; fi
echo "## lvm autoextend"; lvmconfig --type full activation/thin_pool_autoextend_threshold activation/thin_pool_autoextend_percent 2>/dev/null
echo "## docker daemon.json keys (guest)"; [ -f /etc/docker/daemon.json ] && cat /etc/docker/daemon.json
echo "## oldest files in /var/tmp and /tmp (dates only)"; find /tmp /var/tmp -xdev -mindepth 1 -maxdepth 1 -printf '%TY-%Tm-%Td %p\n' 2>/dev/null | sort | head -8
@@ -0,0 +1,66 @@
#!/usr/bin/env bash
# The searches behind every N ("nobody asked") answer. Read-only: grep over the git clones on DooPlex.
# Each search prints hit counts for: architecture folder, open register, closed register, runbooks, source.
# Controls first: a word that MUST hit and a word that must NOT — a search that fails its control proves nothing.
set -u
W=/mnt/5_hdd/felhom.eu/git
D=$W/felhom.eu/documentation
SRC="$W/felhom-agent/internal $W/felhom-agent/configs $W/felhom-agent/cmd $W/felhom-controller/controller/internal $W/felhom-controller/controller/cmd $W/felhom.eu/hub/internal $W/felhom.eu/scripts"
count() { # count <pattern>
local p="$1"
printf '%-58s arch=%-4s open=%-4s closed=%-4s runbooks=%-4s source=%s\n' "$p" \
"$(grep -rliE -- "$p" $D/architecture 2>/dev/null | wc -l)" \
"$(grep -ciE -- "$p" $D/backlog/OPEN-ITEMS.md)" \
"$(grep -ciE -- "$p" $D/backlog/CLOSED-ITEMS.md)" \
"$(grep -rliE -- "$p" $D/runbooks 2>/dev/null | wc -l)" \
"$(grep -rliE --include='*.go' --include='*.sh' --include='felhom-*' -- "$p" $SRC 2>/dev/null | grep -v _test.go | wc -l)"
}
echo "# searches run $(date -u +%FT%TZ) — counts are FILES for arch/runbooks/source and LINES for the two registers"
echo "## controls"
count 'thin'
count 'kernel'
count 'megtel' # Hungarian ASCII fragment, positive control ("megtelt" = it is full)
count 'zzqxnope' # negative control
echo "## S1 old kernels: who removes them"
count 'autoremove'
count 'old kernel|kernel.*(remov|purge|prune)|(remov|purge|prune).*kernel'
count '/lib/modules'
echo "## S2 agent self-update staging folder: who prunes it"
count 'selfupdate.*(prune|clean|remove old|keep)|(prune|clean).*selfupdate'
count 'felhom-agent/selfupdate'
echo "## S3 golden archive left in the dump folder"
count 'remove-golden|golden.*(archive|tar\.zst).*(delete|remove|prune|keep)'
count 'vzdump-lxc-9100'
echo "## S4 host-disk alarm: who receives it"
count 'host_disk_(warning|critical)'
count 'storage_fill_(warning|critical)'
count 'host_disk_'
count 'storage_fill_'
echo "## S5 a space budget for the host system disk"
count 'space budget|disk budget|budget for (the )?(host|root|system)'
echo "## S6 a written way out of a full disk"
count 'disk[- ]full|No space left|ENOSPC'
count 'megtelt|betelt|tele van'
echo "## S7 helper images: a rule that deletes or refreshes them"
count 'helper image'
count 'hello-world'
count '"alpine"'
echo "## S8 off-site restore copy: an age or size limit"
count 'offsite-restore'
count 'offsite-restore.*(days|limit|age|sweep|prune)|(sweep|prune).*offsite-restore'
echo "## S9 guest root filesystem: a watcher"
count 'guest.{0,12}root ?(fs|filesystem).*(warn|alarm|watch)|rootfs.*(warn|alarm|fill)'
echo "## S10 restic cache"
count 'restic.{0,6}cache|RESTIC_CACHE_DIR|--cache-dir|--no-cache'
echo "## S11 template cache"
count 'template/cache|pveam'
echo "## S12 container log cap re-check on a running box"
count 'max-size|log-opts'
echo "## S13 EFI partition: what happens when it fills, who watches it"
count '/boot/efi' # how often the place is named at all (boot mechanics)
count '(boot/efi|\besp\b|efi partition).{0,80}(full|fills|free space|no space|percent|megtel)'
echo "## S14 the system log's storage and cap (host and guest)"
count 'journald\.conf|SystemMaxUse|RuntimeMaxUse|Storage=(volatile|persistent|auto)'
count 'guest.{0,40}journal.{0,60}(memory|volatile|/run|lost|persist)'
echo "## S15 the log of a failed whole-box backup run"
count 'vzdump.{0,60}\.log.{0,60}(remove|delete|prune|keep|stay)|failed.{0,30}vzdump.{0,30}log'
+22 -17
View File
@@ -125,7 +125,7 @@ stopping line that lies.
| **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC |
| **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
## Apps & catalog — 27 rows (P2 1, P3 13, P4 13)
## Apps & catalog — 24 rows (P2 1, P3 10, P4 13)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -141,21 +141,21 @@ stopping line that lies.
| **R-76** | Apps & catalog | P4 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-26, size S, roadmap state `idea (surfaced by the R-75 spike, 2026-07-26)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: not a catalog fix** — the setgid chain is set by the controller's FileBrowser setup; a controller row. | — | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say t | CC |
| **R-707** | Apps & catalog | P4 | **[P2] 37 apps still start with a login a stranger can take (`09` §3 decision 45).** Audit of all 53 apps: `app-catalog-felhom.eu/FIRST-ADMIN.md` (class, fix route, status, measured or read). Open: **3 hard-coded defaults** — calibre-web (`admin / admin123`, measured working on demo-hp and 9202; its own `cps.py -s` route needs a generated password WITH a special character — our generator is letters+digits, a controller change), mealie (`changeme@example.com / MyPassword`), wger (`admin / adminadmin`); **34 open first-run screens** (the first visitor creates the admin: actualbudget, adventurelog, audiobookshelf, calcom, docmost, emby, ghost, gitea, gramps-web, home-assistant, homebox, immich, jellyfin, komga, n8n, navidrome, opengist, outline, papra, plant-it, radarr, rallly, recipe-importer, romm, seerr, sonarr, sparkyfitness, tandoor, termix, uptime-kuma, vikunja, wanderer, wishlist, zipline). **Stale notes:** romm's `default_creds` `admin / admin` answers 401 on demo-hp (like a wrong password) — the page now warns with a login that does not exist; zipline's looks stale too. **Measured on demo-hp 2026-09-28 (read-only):** bookstack's default still logs in on the INSTALLED app (the fix is for new installs; the page now warns). Each fix: route (a) env or (b) the app's own CLI/API via `after_install:`, proven on 9202 with the default failing and the generated password working; route (c) a page sentence. Several sessions (operator, 2026-09-28). **2026-09-29 (controller v0.280.0, catalog `d0e7e2e`):** every class-3 app fixed — mealie, wger, calibre-web by `after_install` (calibre-web with the new `password:24:special`), proven on 9202 fresh installs (`audits/login-gate-2026-09-29/D/`); the setup gate (decision 46, spike PASSED) built and live on immich, n8n, audiobookshelf (probes measured) and uptime-kuma (button) (`…/C/`); romm's and zipline's stale notes removed. **Left: 30 class-4 apps** — gate each (probe measured on 9202 where one exists — 11 upstream candidates listed in `…/B/B-VERDICT.md` §3; the button otherwise). **2026-09-29 afternoon (controller v0.281.0, catalog `6faf432`):** 28 more class-4 apps gated — 32 of 34 — each proven on 9202 (`audits/gate-rollout-2026-09-29/`B): stranger → gate page / 401, household reached the first-setup screen, the gate opened (9 by a measured probe, the rest by the press), the app answered after. seerr, outline, rallly: gated, their opening needs a media server / e-mail (not proven). **Left:** wanderer (R-714); plant-it is not installable. | **NARROWED** (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — seerr, outline and rallly are gated, but the gate OPENING is not proven (needs a media server / e-mail)) — **CLOSED — 2026-09-29 (the rest → R-714)** | — | — | CC |
| **R-905** | Apps & catalog | P4 | **wger's collected style files (283 MB) go into every wger backup, though the app rebuilds them at every start.** Since R-762's fix (`DJANGO_DEBUG=False` + `collectstatic`, catalog `cf1ed43`) the static files sit in a named volume that the backup legs copy like data (measured 283 MB, `audits/design-build-2026-10-06/`); `collectstatic` regenerates them, so they are rebuildable bytes in every Tier-1 unit, Tier-2 copy and off-site snapshot. wger is `lifecycle: hidden` and no box runs it (hub read 2026-10-08). Options to weigh: a non-backed-up volume class for regenerable data, or an anonymous volume. Found by the R-762 helpers 2026-10-08. | **OPEN — needs a design before wger is shown** | — | Design the „regenerable volume" exclusion (catalog + controller backup legs) | CC |
| **R-995** | Apps & catalog | P4 | **Removing an app through the product can leave unnamed Docker volumes and an empty import folder behind.** MEASURED 2026-10-13 on demo-felhom after 61 removals (`documentation/audits/sizing-2026-10-13/METHOD.md` „Teardown”, `evidence/leftover-volumes-after-bench.txt`): 8 unnamed volumes (4–72 KB; immich 1, romm 2, privatebin 3, onlyoffice 2 — volumes the images declare themselves) and `userdata/import/{paperless,calibre,grimmory}` (empty, created by the deploy). The removal answered 200 and its own leftover check was clean, because it looks only at labelled volumes. No data risk; disk and tidiness only. Removed by name on the bench. | **READY** — filed 2026-10-13 | — | Make the removal account for the containers' unnamed volumes (and an empty import folder it created), with a test that deploys an image that declares a VOLUME | CC |
| **R-995** | Apps & catalog | P4 | **Removing an app through the product can leave unnamed Docker volumes and an empty import folder behind.** MEASURED 2026-10-13 on demo-felhom after 61 removals (`documentation/audits/sizing-2026-10-13/METHOD.md` „Teardown”, `evidence/leftover-volumes-after-bench.txt`): 8 unnamed volumes (4–72 KB; immich 1, romm 2, privatebin 3, onlyoffice 2 — volumes the images declare themselves) and `userdata/import/{paperless,calibre,grimmory}` (empty, created by the deploy). The removal answered 200 and its own leftover check was clean, because it looks only at labelled volumes. No data risk; disk and tidiness only. Removed by name on the bench. | **READY** — filed 2026-10-13 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** places G8, D8: the unnamed volumes are counted under R-1267's line; no new measurement of the import folder. | — | Make the removal account for the containers' unnamed volumes (and an empty import folder it created), with a test that deploys an image that declares a VOLUME | CC |
| **R-1067** | Apps & catalog | P4 | **The catalog still carries `pi_compatible:` on every template, and nothing reads it any more.** controller v0.315.0 removed the „Pi kompatibilis" badge and the „csak x86" tag from every page (Raspberry Pi is no longer supported, so one was noise and the other was a warning on almost every app). The `ResourceHints.PiCompatible` struct field and the catalog key were deliberately LEFT: a box still on 0.314.0 reads the key, and removing it would make that box show „csak x86" on every app. **Gated on the floor:** once the fleet floor is at or above 0.315.0, drop `pi_compatible:` from the templates and the field from `stacks.ResourceHints` in the same pair of commits. Harmless until then — an unread key costs nothing. | **BLOCKED** — filed 2026-10-15. Checked 2026-10-16 (apps burndown): the floor is 0.321.0 and demo-hp, Tester 1 and the N100 run 0.321.0, but **Tester-2 still reports 0.292.0** (its newest report 2026-10-04; hub DB with WAL) — so nothing removed. | every box on ≥ 0.315.0 — today Tester-2 | Drop the catalog key and the struct field; grep the hub and the website for Pi wording at the same time (none found 2026-10-15) | CC |
| **R-1043** | Apps & catalog | P4 | **Sonarr, Radarr and Prowlarr build their login redirect as an absolute `http://` URL.** They trust forwarded headers only from their „Trusted networks” (the household set `192.168.0.0/24`; traefik is `172.18.x`), so they ignore `X-Forwarded-Proto: https` and answer `Location: http://<name>/login` to an https visitor. Since controller v0.312.0 that hop is sent back to https by traefik (a GET, no password), so it is cosmetic; a template could seed traefik's network as trusted. (`audits/https-only-2026-10-14/``1A-3-4.txt`) | **OPEN** — filed 2026-10-14 | — | Seed the docker network as a trusted proxy in the *arr templates (with the R-1030 seed), measure the redirect | CC |
| **R-1099** | Apps & catalog | P4 | **The N100 may hold RomM games in the old upload place, now shown under a fake „roms” platform.** Until 2026-10-16 RomM's own upload wrote to the drive's `roms/roms/<platform>/`; after the mapping fix (catalog `fb18471`, R-1089) those files are at RomM's `library/roms/roms/…`. Measured on 9201: such a game showed as missing in its platform and again under a platform „roms” (`audits/romm-ghost-2026-10-16/B/b-after.txt`). Files put in with the file manager are not affected — they are now found. The N100 runs RomM (hub `/apps/romm`); CC does not log in to the lived-in box. | OPEN — filed 2026-10-16 | Operator: look at `roms/roms/` on the N100's drive in the file manager | If it holds platform folders: move them up one level into `roms/`, then run „Scan” in RomM; if it is empty, close | Operator |
| **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | WATCHING — **2026-10-22 night:** not reproducible tonight; the Immich server log of 19:38 is gone (Immich restarted at 02:16 for the off-site dump). What the household sees: the upload fails after 10 minutes and the Immich phone app sends it again. | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC |
| **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. **Seen again 2026-10-24 night on Tester 1:** privatebin's image declares `/run`, `/tmp` and `/var/lib/nginx/tmp` as volumes — each recreate (the night dump's stop/start) leaves three empty anonymous volumes; the same class as Immich's `/data` (`audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt`). | OPEN **2026-10-24 (catalog sweep):** the same class for OnlyOffice and RomM (RomM's image volume `/romm` holds only the empty mount points of its binds); after every catalog app had been removed WITH data, the sweep box still held 52 such volumes (PrivateBin 21, OnlyOffice 8, RomM 8, Immich 3, and 12 from 12:50–12:52 before the batch, holding the same nginx folders as PrivateBin's) — remove-with-data does not see them (no compose project label). All small (4–72 KB), no household data (`audits/catalog-sweep-2026-10-24/_box/anon-volumes.txt`). | — | declare the image's VOLUME paths as tmpfs or named volumes in the catalog (Immich, privatebin), or remove the dangling ones a recreate leaves, by name | CC |
| **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo/<app>/`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC |
| **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. **Seen again 2026-10-24 night on Tester 1:** privatebin's image declares `/run`, `/tmp` and `/var/lib/nginx/tmp` as volumes — each recreate (the night dump's stop/start) leaves three empty anonymous volumes; the same class as Immich's `/data` (`audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt`). | OPEN **2026-10-24 (catalog sweep):** the same class for OnlyOffice and RomM (RomM's image volume `/romm` holds only the empty mount points of its binds); after every catalog app had been removed WITH data, the sweep box still held 52 such volumes (PrivateBin 21, OnlyOffice 8, RomM 8, Immich 3, and 12 from 12:50–12:52 before the batch, holding the same nginx folders as PrivateBin's) — remove-with-data does not see them (no compose project label). All small (4–72 KB), no household data (`audits/catalog-sweep-2026-10-24/_box/anon-volumes.txt`). **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place G8: demo-hp holds 1,887 unnamed volumes (1,885 unused; 1,157 on 2026-10-15 — about 29 more each day), Tester 1 holds 380 (132 on 10-15 — about 10 a day), the N100 holds 1. Nobody measured what many thousands do to Docker. | — | declare the image's VOLUME paths as tmpfs or named volumes in the catalog (Immich, privatebin), or remove the dangling ones a recreate leaves, by name | CC |
| **R-1329** | Apps & catalog | P3 | **cal.com runs on Node.js 20, which gets no security fixes any more** — v6.2.0 carries Node 20.20.0 (no fixed release on the 20 line for CVE-2026-111497) and Debian 12's unpatched `libbrotli1` 1.0.9-2+b6 (fixed: 1.0.9-2+deb12u1); the same image runs in the catalog and on DooPlex. Read from the image 2026-10-27; 2026-10-28: v6.2.0 is still the newest tag, no Node 22/24 image (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN **2026-11-02:** v6.2.0 is still the newest; its image still holds Node 20.20.0. | — | a cal.com release on Node 22/24 (a big step: its own walk); until then the watch lists it nightly | CC |
| **R-1330** | Apps & catalog | P2 | **RomM's newest release (5.4.1) still ships Node.js 24.18.1 (CVE-2026-111497, fixed 24.22.1)** — no newer RomM exists (tags read 2026-10-27). Exposure: Node is IN the image but no Node process runs (RomM serves through Python gunicorn/rq and nginx — `ps` on demo-hp 9201), and RomM sits behind the setup gate. | OPEN — **RULED D61 = keep RomM offered** (operator 2026-10-28, starting the day brief; `09` §3 decision 277): the first RomM release with Node ≥ 24.22.1 is a lane step. 2026-10-28: 5.4.1 is still the newest tag (read from the registry). Earlier: CC's pick 2026-10-27 night **2026-11-02:** 5.4.1 is still the newest tag; its image still holds Node 24.18.1. | — | the watch flags it nightly (R-1327); the step when RomM ships | CC |
| **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) **2026-11-02:** v4.0.8 is still the newest; its image still holds Brotli 1.2.0. | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC |
| **R-1357** | Apps & catalog | P3 | **PrivateBin 2.1.0 has two public notices fixed in 2.1.1 (`GHSA-hw9p` comment replay by anyone who knows a paste id; `GHSA-c4gc` 2.1.0's XSS fix is ineffective), but no 2.1.1 exists: no release on GitHub and no `privatebin/pdo` tag (read 2026-11-02).** Public by design on a box; Tester 1 runs it. | OPEN — filed 2026-11-03 | upstream: a 2.1.1 image | the watch keeps it open; take the release as a lane step | CC |
| **R-1360** | Apps & catalog | P4 | **Meilisearch v1.36.0 (Wanderer) and v1.41.0 (Karakeep) are 15–20 versions behind releases with „important security fixes" (v1.55.1, v1.56.1: long filters, rustls).** The search engine sits on the app's private network, so a stranger cannot reach it; its data needs a dump migration across such a jump: a big step. | OPEN — filed 2026-11-03 | — | a big step when one of the two apps is next moved | CC |
| **R-1372** | Apps & catalog | P4 | **Four small places in the guest that nobody described: no limit, no cleaner or no check.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places G6, G10, G15, D3). (1) **Helper images.** The product runs its helper jobs in `alpine` with no tag (seven call sites, e.g. `internal/backup/backup.go:988`); the bake pulls `hello-world`. None of the three image rules (decisions 53, 56, 285) can delete or refresh a helper image, so a box keeps the `alpine` of its first pull (4 months old on two boxes). (2) **Copies fetched from the off-site store** (`backups/offsite-restore/<app>`) have no age limit: the N100 holds 1.65 GB of two apps that are not installed any more. (3) **Container log caps** (10 MB × 3 files) are written once when the template is baked and never checked again on a running box. (4) **The guest's system log lives in memory** and is empty after each guest restart — a default nobody chose. **Members (answers marked N):** G6-3, G6-4, G10-7, G15-3, D3-3. | OPEN — filed 2026-11-10 (space list) | — | Controller: pin the helper image by tag in one place; delete an off-site restore copy when its app is removed with „delete backups too” or after N days. Docs: say in `11-os-updates.md` whether the guest log in memory is wanted. | CC |
## App updates — 9 rows (P3 9)
## App updates — 8 rows (P3 8)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -180,12 +180,14 @@ stopping line that lies.
| **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC |
| **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.** **What is NOT claimed here:** `restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC |
| **R-433** | Backup & restore | P3 | **A sub-account cannot reach ANY Storage Box snapshot, by any name — so clause (b) of the 2026-09-01 R-95 re-scope ("the rest is recoverable file by file") is NOT SUPPORTED.** MEASURED 2026-09-01 on `demo-hp` over the credential the box already holds, read-only, no delete verb issued. **The sweep:** a batched `stat -c %n` over Hetzner's port-23 restricted shell (500–600 paths per round trip, stdout carrying only paths that exist) tried **777,600** names of the vendor form `YYYY-MM-DDTHH-MM-SS` across nine full days at second granularity, and 126 alternative shapes — **zero resolved.** **The control is what makes the zero mean anything:** the identical 600-name batch with one real path appended returned it in 6 of 6 batches. **The structural cause:** `/home` (the customer data, `u629488-sub3`) is **st_dev 0,82**; `/.zfs/snapshot` is **st_dev 0,276**; `/home/.zfs` does not exist. A snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding `felhom-repo`. **What still stands:** clause (a) — the box can delete its live repository but cannot WRITE into `/.zfs/snapshot` — is unchanged and re-confirmed. **What is now open again:** the only routes to the older copy are a panel rollback of the WHOLE Storage Box (deletes newer snapshots, hits every customer on it) and the provider API (fenced by §11-D, and `hub/internal/hetznerapi/hetznerapi.go` has **no snapshot method at all**, so it needs new code regardless). **NOT ESTABLISHED, and it is the question that decides whether per-file recovery exists for anyone:** whether the MAIN account can see the snapshots. No main-account credential exists in this project. **The ranking is Viktor's; I am not re-ranking R-95 — but the argument that moved it down is the argument this drill removed.** `audits/evidence-drill-r95-recovery-2026-09-01/` **BLOCKED-ON-PROVIDER 2026-09-01.** The one question that can move this is drafted and ready to send: **Question 1** of `documentation/runbooks/provider-questions-2026-09-01.md` — *can the MAIN account retrieve individual files from a snapshot, without a whole-box restore?* **Neither answer leaves this row where it is:** "yes" makes per-file recovery real but permanently operator-only, and closes this; "no, full restore only" means the snapshots do not bound a single customer's exposure at all, because using them costs every other customer on the box their newer snapshots — **and R-95 becomes urgent.** Nothing here can progress without it, and it is not CC's to send (§11-D). Tracked by a dated check in the DUE-CHECKS block. **⚠ A TRAP ON THE WAY TO THIS ANSWER, found by the operator 2026-09-01 and armed against in the questions file: HETZNER HAS TWO SIMILARLY-NAMED PRODUCTS WHOSE DOCS SAY OPPOSITE THINGS.** **Storage SHARE** (a managed Nextcloud, NOT us) documents *"Currently, we only support restores for the full backup ZFS snapshot to a specific point in time"* (`docs.hetzner.com/storage/storage-share/faq/backup-snapshot/`). **Storage BOX** (ours) documents the opposite — *"You can download individual files or entire directories as usual"* (`docs.hetzner.com/storage/storage-box/snapshots/`). **A web search for the obvious phrasing surfaces the SHARE page, and it reads like a definitive NO.** Tell them apart by the giveaways: the Share page talks about *Nextcloud's data cache*, a *database dump* and the *konsoleH* interface, and never mentions Storage Box. **Why this is filed and not left as trivia: a support agent could answer Question 1 from the wrong page, and that answer would push this row to the top of the register for no reason.** Question 1 now names the product, quotes the Storage Box line, and says explicitly that we know what the Share FAQ says. **If an answer cites full-snapshot-only, check WHICH PRODUCT it is about before acting on it.** **Nothing in this repository ever leaned on the Share claim** — verified by grep at the time; the only vendor line we cite is the Storage Box one. **RE-DATED 2026-09-15 → 2026-09-22 (due-checks gate):** the operator mailbox read through the Gmail connector (`(from:hetzner OR subject:hetzner OR "Storage Box") after:2026/09/01`) holds no Hetzner reply — one match, our own `offsite_snapshots_dropped` alarm. Whether the two tickets were ever sent is not visible to CC; if they were not, sending them is the operator's act. **-- ANSWERED. Hetzner replied on ticket #2026090103040671; the operator supplied the thread on 2026-09-22 after this session searched for it and WRONGLY reported no answer (see the instrument note below and R-628).** **Q1 - can the MAIN account read individual files and directories out of a specific snapshot, without restoring the whole box?** Hetzner: *"With the main account it should be possible to download files and directories from the snapshot as it is stated in the documentation"*, citing `docs.hetzner.com/storage/storage-box/snapshots#access-to-snapshots`; and separately *"A restLine truncated
| **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC |
| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. **2026-11-03: demo-hp reached it too.** Its local tier has refused every try since 2026-10-30 („local has 9.4 GiB free; the last archive of guest 9201 was 9.1 GiB"; the disk is 39 GB, the newest archive 10-29), every 15 minutes: the mails `whole_guest_backup_failed` (08:11 CET, 2026-11-03) and `expected_backup_missed` (05:00 CET, „newest backup is 120h old"). PBS and off-site are not this tier. Not caused by the release. **2026-11-07 — the box half again (operator's yes):** still refused every ~4 h („local has 9.4 GiB free … needs about 12.4 GiB"), `expected_backup_missed` daily to 216 h, and the OS updates held behind it (`os_update_stale`). The journal had grown to 3.8 GB (journald's default cap on a 39 GB root); shrunk to 500 MB (`journalctl --vacuum-size=500M`, tLine truncated
| **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)** **2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place O1, listed from documents only (the off-site box was not read in this run — second pass). | — | — | CC |
| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. **2026-11-03: demo-hp reached it too.** Its local tier has refused every try since 2026-10-30 („local has 9.4 GiB free; the last archive of guest 9201 was 9.1 GiB"; the disk is 39 GB, the newest archive 10-29), every 15 minutes: the mails `whole_guest_backup_failed` (08:11 CET, 2026-11-03) and `expected_backup_missed` (05:00 CET, „newest backup is 120h old"). PBS and off-site are not this tier. Not caused by the release. **2026-11-07 — the box half again (operator's yes):** still refused every ~4 h („local has 9.4 GiB free … needs about 12.4 GiB"), `expected_backup_missed` daily to 216 h, and the OS updates held behind it (`os_update_stale`). The journal had grown to 3.8 GB (journald's default cap on a 39 GB root); shrunk to 500 MB (`journalctl --vacuum-size=500M`, tLine truncated
| **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) |
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
| **R-1366** | Backup & restore | P3 | **The whole-guest copy carries every Docker image, so it is mostly programs we could download again.** MEASURED 2026-11-09 on demo-hp guest 9201 (`docker system df -v`): the 32 GB source holds 27.5 GB of images — 13.2 GB used by containers, 9.8 GB the apps' undo images (decision 53, on purpose), 3.5 GB old controller and infrastructure images (deleted by hand the same day; controller v0.351.0 deletes them by itself) — and about 1.5 GB of Docker volumes plus 1.3 GB `sys_drive`. Packed: 9.8 GB, on a 39 GB system disk (R-548). The N100's copy is 1.5 GB (few apps). A copy without the images would be about a tenth of the size, but a whole-guest restore is the disaster path and would then need every image pulled again — an upstream tag can be gone (R-933 is the mirror). This is a design choice about the disaster path, not a defect. | OPEN — filed 2026-11-09 | operator decision | Options to put to the operator: (a) keep the images in every tier and size the local target (a backup drive, or the local tier skipped when it cannot fit — R-548); (b) the LOCAL tier only leaves the images out (`vzdump --exclude-path` on Docker's `image/` and `overlay2/`) and its restore re-pulls, PBS and off-site keep them; (c) no tier keeps them, after R-933. Measure first on scratch 9202: a whole-guest restore of a copy without images, every app back and the time it took. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/` (off-git, DooPlex) | CC |
| **R-1366** | Backup & restore | P3 | **The whole-guest copy carries every Docker image, so it is mostly programs we could download again.** MEASURED 2026-11-09 on demo-hp guest 9201 (`docker system df -v`): the 32 GB source holds 27.5 GB of images — 13.2 GB used by containers, 9.8 GB the apps' undo images (decision 53, on purpose), 3.5 GB old controller and infrastructure images (deleted by hand the same day; controller v0.351.0 deletes them by itself) — and about 1.5 GB of Docker volumes plus 1.3 GB `sys_drive`. Packed: 9.8 GB, on a 39 GB system disk (R-548). The N100's copy is 1.5 GB (few apps). A copy without the images would be about a tenth of the size, but a whole-guest restore is the disaster path and would then need every image pulled again — an upstream tag can be gone (R-933 is the mirror). This is a design choice about the disaster path, not a defect. | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H3 (answer H3-2): no new measurement; the row is the only answer to „what makes the whole-guest copy grow”. | operator decision | Options to put to the operator: (a) keep the images in every tier and size the local target (a backup drive, or the local tier skipped when it cannot fit — R-548); (b) the LOCAL tier only leaves the images out (`vzdump --exclude-path` on Docker's `image/` and `overlay2/`) and its restore re-pulls, PBS and off-site keep them; (c) no tier keeps them, after R-933. Measure first on scratch 9202: a whole-guest restore of a copy without images, every app back and the time it took. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/` (off-git, DooPlex) | CC |
| **R-1312** | Backup & restore | P3 | **The hub customer page's off-site switch did not reach the box: on Tester 1 (2026-10-27, operator-attended, ~14:00–14:12 UTC) the box's saved off-site target stayed `enabled` and its `controller.yaml` `offsite.enabled: true` (unchanged since 2026-10-04) while the hub showed off-site OFF.** So what the switch stops on the box side is not known: the controller reads `cfg.Offsite.Enabled` only at start (`cmd/controller/main.go:890`), and nothing rewrote the file. Either the switch is hub-side only by design (then the page must say so), or a box keeps pushing off-site while the operator believes it is off. Found by the R-1218 live attempt (R-1218 closed 2026-10-27 with its live proof waived by the operator). | OPEN | — | Read the hub handler behind the switch and what it changes (config, tenancy, storage access); then one attended run: switch off, watch the box's next off-site push (does it run, does it fail at the storage) | CC |
| **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo/<app>/`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place D5 (answer D5-4): the folder does not exist on any of the three boxes today; nothing new measured. | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC |
| **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
| **R-164** | Backup & restore | P4 | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
| **R-213** | Backup & restore | P4 | **Putting files back in place — the half the recovery screen deliberately does not do.** The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: *a screen that unlocks and then offers to overwrite is two decisions wearing one button*. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. **The operator named its requirement: a live-versus-backup comparison** — the customer must be able to see what would change before anything is overwritten | **DEFERRED** — operator ruling 2026-10-12 (S18, `09` §3 decision 213): designed after the first customers. Stays P4; re-dated: look again when the first paying household has run 30 days. **OPEN — not started, deliberately** | the comparison design (nothing exists for it yet) | After the first customers | Operator + CC |
@@ -196,7 +198,7 @@ stopping line that lies.
| **R-832** | Backup & restore | P4 | **ep0's copy in a place outside both Hetzner and the operator's home (roadmap).** Today DooPlex (the operator's home) holds it (decision 71). A Hetzner Storage Box would share a provider with ep0 and with every household's file backups, and cannot run PBS, so the copy could not be verified or restored from directly. | **DEFERRED — later, if the product grows** | — | — | operator |
| **R-878** | Backup & restore | P4 | **A catch-up (R-871) runs the database-dump leg in the DAY, and that leg stops an app with a volume for its copy — the household may notice the stop, and a large volume makes it longer.** MEASURED 2026-10-05 on demo-felhom: the catch-up at 08:25:02 stopped opengist, copied 182.5 KB, started it again — about 1 s, then a few seconds of `health: starting`; the night does exactly the same, unseen. Nothing measured for a large volume. Fix direction (if it matters): skip the volume copy of a running app in a DAYTIME catch-up and leave it to the next night, or warn. `audits/catchup-2026-10-05/partA/live-demo-felhom.txt` | **READY — owner: CC** **2026-10-05 (burn-down night): NEEDS A LIVE MEASUREMENT** — a timed catch-up on 9202 for an app with a multi-GB volume, then the skip-or-warn choice. | — | measure a large volume first | CC |
| **R-32** | Backup & restore | P4 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-32.md`) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at `hub/internal/offsite/offsite.go:276-280` is false for the shared tier; part (3) waits on a read-only measurement (`du` on port 23). Operator: agree to route A. **2026-10-07 07:58: `09` §3 decision 167 — option A yes, and delete the old RESET leftovers once.** **2026-10-07 (Part D step 2, read only):** the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → **~2.7 GB belongs to no live customer**; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. **The one-time clean-up is STOPPED:** no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (`audits/day-2026-10-07/D/D2-listing-readonly.txt`). **2026-10-07: option A DELIVERED (hub 0.142.0):** RESET purges the repo and every `.orphaned-*` through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, `audits/day-2026-10-07/D/`). Not yet seen live (no scratch RESET today). **The one-time clean-up of the ~2.7 GB of leftovers is STOPPED:** it needs the pool box's main-account login (see above). **2026-10-07 14:43 — `09` §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4.** Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. **Blocker: a safe main-account path.** **2026-10-22: Tester 2's planned RESET (operator ruling 2026-10-22 14:11, `pilot/TESTER2-handover.md`) will be the purge's first live run — CC reads its leg journal after.** **2026-10-22 12:56 UTC — the first live run (Tester 2's RESET, `audits/tester2-reset-2026-10-22/`): PASS for what it could show.** The hetzner leg logged in with the sub-account's own password, listed the folder, found NO repository (`purged … 0 path(s) []` — the box never ran an off-site backup; the hub never held a listing for it), re-listed, then deleted sub-account 322460; leg `ok`. The other customers' repositories are unchanged (each listed after the reset with its box's own key: demo-hp 303, tester-1 62, demo-felhom 43 = the hub's 05:10 listing; Tester4 has no box — the hub's own listing on 2026-10-23 05:10 compares it). **Not yet seen live: the removal of a folder that holds data** (the `rm -rf` branch) — the next RESET of a customer with off-site history shows it. Open as before: only the old leftovers. **Dated check 2026-10-23 (DUE-CHECKS):** read the hub's 05:10 UTC listing of Tester4 (72 entries on 2026-10-22) and its `offsite_snapshots_vanished` events — the one other customer whose folder CC could not re-list. **2026-10-23 dated check — DONE:** the hub's 05:10 UTC listing of Tester4 holds 80 snapshot files = the 72 of 2026-10-22 plus 8 new ones from the night's box (`8 new snapshot file(s), each inside a reported box run`), none vanished, no `offsite_snapshots_vanished` (hub log 07:10:03 CEST, `audits/night-2026-10-23/P4-testnight/`). Tester 2's RESET touched no other customer's folder. | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on theLine truncated
| **R-1087** | Backup & restore | P4 | **Whole-box copies left on the OLD local target are never pruned after the target moves.** MEASURED 2026-10-15 (storage-identity build B space check): the N100's host root (`local`) holds three copies from 2026-07-28 (5.5–5.7 GB each, ≈ 16.8 GB); its agent's `local_backup_target` is now `felhom-backup` (the USB disk, retention 3), and retention prunes only the current target. On a small-root box such leftovers would starve the local copies (R-548). Evidence: `audits/storage-identity-buildB-2026-10-15/space/felhom-pve-9201.txt`. | OPEN | — | Decide whether the agent prunes its own `vzdump-lxc-<vmid>-*` files on a former target (after a newer copy exists on the new one), or the target move asks; until then, the N100's three July files can be removed by hand with the operator's word. | CC |
| **R-1087** | Backup & restore | P4 | **Whole-box copies left on the OLD local target are never pruned after the target moves.** MEASURED 2026-10-15 (storage-identity build B space check): the N100's host root (`local`) holds three copies from 2026-07-28 (5.5–5.7 GB each, ≈ 16.8 GB); its agent's `local_backup_target` is now `felhom-backup` (the USB disk, retention 3), and retention prunes only the current target. On a small-root box such leftovers would starve the local copies (R-548). Evidence: `audits/storage-identity-buildB-2026-10-15/space/felhom-pve-9201.txt`. | OPEN **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H4: the N100 still holds the three copies of 2026-07-28 (17.59 GB with their logs). A second instance: demo-hp holds four copies of 2026-08-03…08-21 (5.10 GB) on `/mnt/hdd_1/dump`, its former target. | — | Decide whether the agent prunes its own `vzdump-lxc-<vmid>-*` files on a former target (after a newer copy exists on the new one), or the target move asks; until then, the N100's three July files can be removed by hand with the operator's word. | CC |
| **R-1171** | Backup & restore | P3 | **demo-hp's LOCAL whole-guest copy is never restore-tested: the box keeps one copy (keep-daily 1) and the restore test wants a copy 24 h old, so the only copy turns 24 h old exactly when the next night replaces it.** Read 2026-10-20 on demo-hp (agent journal): every 6-hourly evaluation since 2026-10-18 says `local: no settled archive yet — nothing to prove`; the local storage holds one 9201 archive; the last local proof was 2026-10-14 03:42 (`settle=24h`, `keep_last=1`, `prune=keep-daily=1`). The hub's `restore_test_stale` (168 h) will alarm from ~2026-10-21 03:42. A real household with retention 1 would be the same. | OPEN — filed 2026-10-20 **2026-11-02 read (journal):** demo-hp's local copy passed a scheduled restore-test on 2026-10-30 10:44 (`audits/security-2026-11-02/part4/restore-test-demo-hp.txt`), so „never” no longer holds; one pass, read once. Close after the next one. | demo-hp `journalctl -u felhom-agent` („restore-test evaluated"), the hub's host report `restore_tests` | Agent: a tier that keeps ONE copy proves its newest archive once it is older than the settle time measured from the BACKUP's end, not 24 h — or a settle below the cadence (e.g. 6 h) for a 1-copy tier; a test; then read the next evaluation | CC |
| **R-1172** | Backup & restore | P4 | **After option A (R-893): a failed off-site restore that was PUT BACK is logged at ERROR, and an empty `backups/restore-undo/` folder stays on disk.** Measured on 9202 2026-10-20 (rc.1): the web handler logs `[ERROR] [web] off-box reconstitute rallly (async): … Nem változott semmi…` — good news at ERROR; the put-back removes the per-app folder but not the empty `restore-undo/` parent. Small; 0.336.0 was already pushed (a version is pushed once). | OPEN — filed 2026-10-20 | `audits/night2-2026-10-20/P2/live/11-controller-log-window.txt`, `12-after-failed-restore.txt` | Next controller release: the put-back outcome logged at WARN (its own sentence is the household's), the empty parent removed; a test | CC |
| **R-1173** | Backup & restore | P3 | **R-921's consequence (no app stop for an off-site tier the agent will refuse) has not been seen live.** Agent 0.161.0 serves `host_busy` (proven live on demo-hp 2026-10-20 during a press: busy `backup:local`), and controller ≥ 0.335.0 reads it — but the case only arises on the night schedule (a manual press backs up the local tier only), so the no-stop branch is proven by its red-proved test alone. | OPEN — filed 2026-10-20 **2026-10-23 dated check — DONE (the other outcome):** demo-hp's off-site (PBS) tier ran 2026-10-22 20:42:53–20:49:48 UTC with nothing in flight (quiesce → started → snapshotted → early resume → done); no `host is busy` case arose, so the R-921 no-stop branch is still proven by its test alone. Tonight only the local tier was due (refused by space, R-548). Kept open; no new date — it reads itself the night a busy host meets a due off-site tier. | `audits/night2-2026-10-20/P1/1b-host-busy-poll.txt` | Read demo-hp's controller log after the night its off-site (pbs) tier is due (~2026-10-22): a `host is busy … no app is stopped (R-921)` line, or the off-site tier starting with nothing in flight | CC |
@@ -215,10 +217,10 @@ stopping line that lies.
| **R-1231** | Backup & restore | P4 | **The recovery page says the box's host service is „older” when the controller only cannot tell.** Test night 2026-10-23 (`P4-testnight/07a-unlock-refused-fixture-version.txt`): with the agent's capability `unknown` (a version string it could not compare), the page read „…a hozzá tartozó házon belüli szolgáltatás régebbi… A gép magától frissül”. The same page is reached by a merely-down agent, for which „older” and „updates itself” are not true. | OPEN | — | A separate sentence for `unknown` („a gép most nem éri el a házon belüli szolgáltatást — próbáld újra pár perc múlva”); a render test per state; rides a controller release | CC |
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place O1, listed from documents only (the off-site box was not read in this run — second pass). | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
| **R-1362** | Backup & restore | P3 | **With the internet down, the Backups page takes over 2 minutes (135 s and 145 s measured; 5 s with only the hub down).** Walked 2026-11-03 on Tester 1: every other page answers at once on the home network. The page waits on something off-site inside its handler; a household that lost its line and opens exactly this page sees a hang. `audits/security-2026-11-02/part5/outage-wan.log` | OPEN — filed 2026-11-03 | — | name the wait with a timing, then give it a short deadline and show „the off-site state could not be read" | CC |
## Storage & devices — 14 rows (P3 4, P4 10)
## Storage & devices — 13 rows (P3 4, P4 9)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -236,7 +238,7 @@ stopping line that lies.
| **R-1275** | Storage & devices | P4 | **An English household sees the drive names in Hungarian** on the install page (and wherever a storage label shows): „Belső tárhely — 52.0 GB free (default)”, „Külső tárhely (hdd_1)” (catalog sweep 2026-10-24, `audits/catalog-sweep-2026-10-24/navidrome/page-install-en.txt`). The label is stored once, in Hungarian, when the drive is found (`felhom-controller/controller/internal/settings/settings.go` `InferStorageLabel`); a household may rename it. | OPEN | — | Render a label that still equals the inferred default in the reader's language; a test per language | CC |
| **R-1276** | Storage & devices | P4 | **„Remove with data” leaves the app's empty parent folder** (`<drive>/appdata/<app>`) when the template binds a subfolder: Grimmory, RomM, Paperless-ngx (catalog sweep 2026-10-24; 4 KB each; the bound folders were removed). Harmless (empty, not counted as old data), untidy for a household that browses the drive. | OPEN | — | Remove the parent when it is empty after the bound folders go; a test | CC |
## Security & access — 30 rows (P1 2, P2 6, P3 15, P4 7)
## Security & access — 28 rows (P1 2, P2 5, P3 14, P4 7)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -268,15 +270,17 @@ stopping line that lies.
| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | **OPEN** — 2026-10-27 (operator yes): the dead names were REMOVED from the store with `sec rm` (46, one commit each; `audits/r1299-window-button-2026-10-27/sec-rm.txt`, names only): every one-time code this row names is gone; R_DEMO_FELHOM stays (unsure). Consequence: the nightly sweep no longer watches for the removed values — they open nothing (their boxes are destroyed). The session transcripts themselves are unchanged — that question stays here. **Earlier:** OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC |
| **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator |
| **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC |
| **R-1312** | Backup & restore | P3 | **The hub customer page's off-site switch did not reach the box: on Tester 1 (2026-10-27, operator-attended, ~14:00–14:12 UTC) the box's saved off-site target stayed `enabled` and its `controller.yaml` `offsite.enabled: true` (unchanged since 2026-10-04) while the hub showed off-site OFF.** So what the switch stops on the box side is not known: the controller reads `cfg.Offsite.Enabled` only at start (`cmd/controller/main.go:890`), and nothing rewrote the file. Either the switch is hub-side only by design (then the page must say so), or a box keeps pushing off-site while the operator believes it is off. Found by the R-1218 live attempt (R-1218 closed 2026-10-27 with its live proof waived by the operator). | OPEN | — | Read the hub handler behind the switch and what it changes (config, tenancy, storage access); then one attended run: switch off, watch the box's next off-site push (does it run, does it fail at the storage) | CC |
## Box system & updates — 17 rows (P3 11, P4 6)
## Box system & updates — 18 rows (P3 13, P4 5)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) | the kernel part needs the operator's word (it changes what rule R4 allows) — STATUS D67 | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) | an operator pick: a written budget or not — STATUS D69 | Docs: a short „disk space” section in an architecture document: each part of the system disk with its limit and its cleaner, pointing at the audit table. Then the installer can refuse a disk that cannot hold the sum. | CC |
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
| **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC |
| **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC |
@@ -291,7 +295,7 @@ stopping line that lies.
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
## Monitoring & notifications — 25 rows (P2 3, P3 14, P4 8)
## Monitoring & notifications — 25 rows (P2 3, P3 15, P4 7)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -319,6 +323,7 @@ stopping line that lies.
| **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC |
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) | part 3 needs an operator pick — STATUS D68 | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
## Hub & operator — 14 rows (P3 3, P4 11)
@@ -358,7 +363,7 @@ stopping line that lies.
| **R-990** | Business & legal | P4 | **Operator: read `documentation/audits/forum-research-2026-10-12/FELHOM.md` §5 and rule on each recommendation.** READ-ONLY research 2026-10-12: three years (#7943–#11092, 2,908 posts, 195 people, 82 beginner threads) of the public PROHARDVER! topic „Home server / házi szerver építése”, tagged and counted by person (`FINDINGS.md`, method and spot-check in `METHOD.md`; no username or post text in git). Ten ranked recommendations, none acted on; the top three: a „which machine works” page (8 of 33 beginners who name their hardware own only a Synology, a router disk or a Raspberry Pi — the installer cannot run there; the same gap is R-989's first recommendation from a different source, and the shipped-page half is R-991), „you do not choose an operating system — we do” (the biggest beginner question, 20 of 82 threads), and „bring your old office PC” as the cheap entry. Catalog gap (≥ 3 people): a DLNA server (7), an ad/DNS blocker (5), a camera recorder (5). | **WAITING-ON-OPERATOR** — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages | operator |
| **R-1035** | Business & legal | P4 | **Did the „felhő tárhely” page earn anything? Two dated checks.** **Baseline (reported by the operator from Search Console, 2026-10-14, last 3 months):** 1 click, 48 impressions, CTR 2.1 %, average position **73.6**; „felhő tárhely” 19 impressions, „felhőtárhely” 5, „felhő alapú tárhely” 3. CC measured separately that the phrase „felhő tárhely” appeared **0 times** on the home, FAQ and backup pages — Google was showing the home page for those queries by accident. `/sajat-felho-tarhely` + `/en/own-cloud-storage` published **2026-10-14**; full baseline, price sources and the power provenance in `audits/seo-2026-10-14/BASELINE.md`. **What to measure, both times:** impressions and average position for „felhő tárhely” and „saját felhő tárhely”, plus the new page's own impressions, clicks and position. **A null result at +4 weeks means nothing on its own** — a new page on a site at position 73.6 takes weeks to months to move, and „felhő tárhely” alone is held by Microsoft, Google and the big list sites; the page aims at the two longer phrases. That is why there are two checks and not one, and why neither should be read as a verdict on the copy. **There is no Search Console connector in this project, so CC cannot read these numbers** — the operator reads them, CC records and compares. **Blocked until the operator asks for indexing** (Search Console → URL-ellenőrzés → the URL → „Indexelés kérése”): until that happens, or Google finds the page itself, both checks read zero for a reason that is not the page. **This is a slow channel and does not replace talking to people.** | **WATCHING** | operator: request indexing, then supply the Search Console read | 2026-11-11: first read (+4 weeks); 2026-12-09: second (+8 weeks), compared with the first | operator + CC |
## Process & tooling — 33 rows (P3 8, P4 25)
## Process & tooling — 32 rows (P3 8, P4 24)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
+4
View File
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-11-10
**Updated 2026-11-09.** demo-hp did not restart, because its local whole-box copy was refused for space again and the night update waited; CC freed space (old images, an old installer file), so tonight's copy fits. Controller 0.351.0 is released to every box (floor 12:18 UTC): it deletes old test builds and old system images by itself. Two new rows: the whole-box copy is mostly app programs (your decision, R-1366), and the host log noise (R-1367).
## Moved from STATUS on 2026-11-09
**Updated 2026-11-08.** The night went right: the N100 installed its Proxmox updates and a new kernel and restarted itself; demo-hp made its first local whole-box copy in 10 days. Hub 0.179.1 no longer says „needs a restart" for a box that has restarted; demo-hp still needs one (its kernel step, likely tonight).
+1
View File
@@ -53,3 +53,4 @@ what happened, who noticed, and the register row it became. Prose here is not a
2026-10-16 („Could not send notification to $NOTIFY_SOCKET"); demo-hp, same package and options, runs it. Register:
R-1300 (the release), R-1301 (the clock). Evidence: `audits/cloudflared-2026.10.3-2026-10-26/` (`obs-n100-clock.txt`). (CC)
- **2026-11-05..07 — the nightly Proxmox update step refused three nights in a row.** `proxmox-grub` 2.12-9+pmx4 needs `grub-efi-amd64` of the same version; the update wrapper put `proxmox-grub` in the Proxmox group (its boot-package rule matched only names that START with `grub`), so the whole step was refused (R6) and 34 Proxmox updates waited; an `os_update_failed` mail each morning. Apps not touched. Agent 0.172.1 puts `proxmox-grub` with the boot packages; it reaches the box by the operator's vouch and two signed jobs. Register: fixed in the session (size rule), recorded in the agent's CHANGELOG. Evidence: `audits/demo-check-2026-11-07/n100-pve-step-R6.txt`. (CC)
- **2026-11-10 — every place that holds data was measured (read-only; the space list).** Host system disk 38.09 of 100.86 GB (38 %), 4.6 GiB more than on 10-15: the box holds 8 kernels (8.15 GB) and none was ever removed → R-1368. The agent's self-update folder holds 48 old binaries (724 MB) and there are 11 old agent copies beside the binary (155 MB) → R-1368. The host journal sits at its cap (4.28 GB, 74 files, oldest entry 10-04) → R-1367. The three whole-box copies of 2026-07-28 still lie on the system disk (17.59 GB) → R-1087; four template archives (2.56 GB) → R-1368. On the USB drive, `backups/offsite-restore` holds 1.65 GB of two apps that are not installed any more → R-1372. Nothing was deleted or changed. Table: `audits/space-list-2026-11-10/TABLE.md`. (CC)