From 025a1bcd40c8e8f80f730d8f48701e13bd534977 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 10 Nov 2026 14:47:25 +0100 Subject: [PATCH] =?UTF-8?q?docs:=20the=20disk-space=20section=20(16=20?= =?UTF-8?q?=C2=A714),=20the=20runbook=20"a=20disk=20is=20full",=20the=20ru?= =?UTF-8?q?n's=20report?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit C4: documentation/runbooks/disk-full.md (R-1369). Five disks; each hand step is PROVEN with its date and evidence, or NOT PROVEN. C5: 16-storage-physical-identity.md §14 (D69, decision 288). 45 places, each with limit, cleaner and alarm; the rulings behind "no limit"; the host disk budget; the open design points. PROMPT-TEMPLATE §N.5 gets the keep-it-true line. Close-out: STATUS (D75 new), CONTEXT, dated lines on R-1369, R-1370, R-1371, three small stale-document fixes, the report of the run. Documents only. No release, no build, no deploy. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- CONTEXT.md | 7 + REPORT-endings-kernels-fixes-2026-11-10.md | 146 +++++++++++++++ STATUS.md | 3 +- documentation/PROMPT-TEMPLATE.md | 4 + .../16-storage-physical-identity.md | 173 ++++++++++++++++++ .../architecture/capability-map-detail/A.md | 2 + documentation/backlog/OPEN-ITEMS.md | 6 +- documentation/backlog/STATUS-HISTORY.md | 4 + documentation/runbooks/day0-install.md | 5 +- documentation/runbooks/disk-full.md | 156 ++++++++++++++++ documentation/runbooks/ep0-datastore-copy.md | 6 +- 11 files changed, 504 insertions(+), 8 deletions(-) create mode 100644 REPORT-endings-kernels-fixes-2026-11-10.md create mode 100644 documentation/runbooks/disk-full.md diff --git a/CONTEXT.md b/CONTEXT.md index 9434527b..924cbf23 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -23,6 +23,13 @@ > household's; and moving the WARNING with the critical. No hub build, no deploy. (2) Agent `776025c8`: the self-update > folder prunes itself after a good update. (3) Controller `844f19f1`: one fill warning per filesystem. (4) Kernel > clean-up proven by hand on demo-hp (`audits/kernel-cleanup-spike-2026-11-10/`); the rule is DESIGN in `11` §5.11.1. +> **Also decided by CC in this run — operator may reverse:** (2) the staged binary of a FAILED version stays until the +> next good update (nothing is removed after a rollback); (3) of two targets on one filesystem the first keeps its name +> (app area, system area, then drives), and the container's `/` is identified by a Docker-kept `/etc` file; (4) the +> hand step is `apt-get purge` of exact package names, not `remove`. +> (5) Disk space has a design home: `16-storage-physical-identity.md` §14 (45 places with limit, cleaner, alarm; the +> host disk budget — about 14 GB with a backup drive, 28–54 GB without; the smallest disk is D75, not ruled). The hand +> steps are `runbooks/disk-full.md`, each PROVEN or NOT PROVEN. (6) The endings list: `audits/endings-list-2026-11-10/`. > Report: `REPORT-endings-kernels-fixes-2026-11-10.md`. > > **2026-11-03 (day, attended) — RELEASED: controller v0.350.0 (the global floor since 08:18 CET, MinAgent 0.158.1, `sha256:9ee53908…` from `5c336d82`), agent v0.172.0, golden 0.350.0 (both VOUCHED 10:18:10 CET), hub v0.179.0 (09:05 CET), catalog `f4c5c79`.** Traefik v3.7.17 on every box (R-1354): on 9202 v3.7.16 grew 18.7 → 79.0 MiB after 920 NTLM connections, v3.7.17 stayed near 25 MiB for 25,116. Golden 0.349.0 was never vouched. The fresh install ran the public installer with one added line naming the unvouched pair, because an install refuses any golden but the vouched one (R-297). A reinstall of a customer whose controller has reported is refused until the operator deletes the host, however long one waits (R-1159). R-1302's ERROR is a BYO box's: the appliance path pre-positions the grant. Report: `REPORT-release-2026-11-03.md`; evidence `documentation/audits/release-2026-11-03/`. diff --git a/REPORT-endings-kernels-fixes-2026-11-10.md b/REPORT-endings-kernels-fixes-2026-11-10.md new file mode 100644 index 00000000..1e7c495c --- /dev/null +++ b/REPORT-endings-kernels-fixes-2026-11-10.md @@ -0,0 +1,146 @@ +# REPORT — the Endings list, the kernel clean-up test, the first Space fixes (2026-11-10, operator brief, daytime) + +**Plain words first.** CC listed every way a thing ends on a box or in the hub. CC removed six old kernels from the HP +box by hand, with the keep rule the operator ruled, and restarted the box once. CC wrote three small fixes for disk +space. They are in source only. Nothing was released, built or deployed. No household got a mail. + +## 1. Claims in the brief that were wrong, or that needed a check + +| Claim | Verdict | +|---|---| +| "`07` §6.6 (retiring a drive)" | **Wrong.** §6.6 is "which version a restore brings back". Retiring a drive is in `03-host-agent.md:390` and `capability-map-detail/D.md:184` | +| "The kernel step names its packages, and that may be why old kernels stay" | **True, and it is one of two causes.** The other: nothing runs an autoremove, and rule R4 refuses a removal (`audits/kernel-cleanup-spike-2026-11-10/evidence/01-why-still-there.txt`) | +| "The operator-only register blocks a type a household already saved" | **True.** Proven by a test seen failing: with the register lines out, the saved type reached the household (`TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt`) | +| "The fill watch counts one filesystem twice on a one-volume box" | **True as measured** on three guests (`audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`). **Not seen as two real mails**: no disk was filled | + +## 2. The parts + +| Part | State | Note | +|---|---|---| +| 0 — the rulings of 12:22 written down | done | `449b065f` | +| A — the Endings list, with the ageing drill | done | `807175c2`; `documentation/audits/endings-list-2026-11-10/` | +| B — kernel clean-up by hand on demo-hp | done | `10ea4c2c`; `documentation/audits/kernel-cleanup-spike-2026-11-10/` | +| C1 — hub: a full host disk is the operator's alarm | done, commit only | `b14eb0fa` | +| C2 — agent: the self-update folder cleans itself | done, commit only | felhom-agent `776025c8` | +| C3 — controller: one fill warning for one filesystem | done, commit only | felhom-controller `844f19f1` | +| C4 — the runbook "a disk is full" | done | `documentation/runbooks/disk-full.md` | +| C5 — the "disk space" section | done | `documentation/architecture/16-storage-physical-identity.md` §14 | + +Nothing was dropped. Nothing was changed in scope. + +## 3. Results + +**Part A, five lines** (`documentation/audits/endings-list-2026-11-10/README.md`): + +1. 74 endings: 67 with a code path, 7 with none. 20 more are tiny. +2. 9 of the 67 are named in no architecture document; 24 are only mentioned. +3. 536 answers: `A` 499, `D` 0, `K` 15, `N` 22. +4. The worst `N`: the operator unticks a household's off-site tier, then resets or deletes the household. The paid + account stays at the provider and the hub has no record of it. +5. Four questions for the operator: D71–D74 in `STATUS.md`. + +**Method verdict.** It worked. Weak point: a tool opened all 691 pointers, but a person-like read covered only 86 of +536 cells (every `K`, every `N`, the main `A` cells). The other `A` cells are a map, not proof. The ageing drill: ten +cheap rounds found nothing; the drive round and the update round each found or proved something. Repeat it only with +drive apps, on a box with an agent. + +**Part B, two lines.** The ruled rule (keep the running kernel, the one before it and the newest) is safe by hand on +demo-hp: 8 → 2 kernels, 6,411 MB back, 50 seconds, one restart, all containers healthy after 311 seconds. `apt` alone +is not safe: it did not refuse the running kernel, and one meta-package pulled `proxmox-ve` — so the build must name +exact packages and check the plan first (`11-os-updates.md` §5.11.1). Not proven: a second box; a boot of the kept +older kernel. + +**Part C, by fix.** + +| Fix | Commit | Tests | Red seen | +|---|---|---|---| +| C1 hub | `b14eb0fa` | `TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt`, `TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree`, `TestD68_RegisterAndGate`, `TestD68_SeedHoldsNoOperatorOnlyType` | 3 times: the register lines out; the gate reading the register only; the type back in the first list | +| C2 agent | `776025c8` | `TestPrune_*`, 6 tests; one runs the wrapper's real commit and rollback | 3 times: no prune call; no refusal while an update is pending; no regular-file check | +| C3 controller | `844f19f1` | `TestR1371_OneFillSendsOneWarning`, `TestR1371_TargetsByFilesystem`, `TestR1371_ProductionListUsesTheRealIdentity`, `TestFilesystemKey_AnOverlayRootIsTheDockerDataRoot`, `TestFilesystemKey_RealSyscalls` | 3 times: no filesystem check; the production list not wired; an overlay keeping its own device number | +| C4 runbook | this commit | — (document). 5 parts; each hand step `PROVEN` with date and evidence, or `NOT PROVEN` | — | +| C5 section | this commit | — (document). 45 places; 8 "no limit, by ruling"; the budget; open points with rows | — | + +**Limits, said plainly.** None of C1–C3 runs on any box. C1: a household that saved the type keeps the saved text; the +register blocks it at send time, with no migration. C2: a box cleans its folder at the first good update after the +release; the `.bak-*` files are not reached. C3: not proven on a running box; the guest's root disk still has no +watcher. C4: every `PROVEN` step was proven on demo-hp only; the whole runbook was not rehearsed. C5: the budget is +from three boxes; the size of the root between the three measured disks is not known. + +**The budget in one line (C5).** The host's own files need about 14 GB when the whole-box copies go to a backup drive, +and 28–54 GB when three copies stay on the system disk. The smallest disk is not decided: it is D75. + +**Decisions CC took in this run (the operator may reverse each).** Part B: `purge`, not `remove`. C1: the warning moves +with the critical; the kinds `usb`, `local-dir`, `nfs`, `cifs` are the household's, every other kind and a missing kind +are the operator's. C2: the staged file of a failed version stays until the next good update. C3: the first target of a +filesystem keeps its name; the identity of the container's `/` comes from a file Docker keeps. All four are in +`CONTEXT.md` of `felhom.eu`, today's note, with the tag. The notes in the agent's and the controller's `CONTEXT.md` +describe the change and do not carry the tag. + +## 4. Machines changed, and how each was put back + +| Machine | Change | State now | +|---|---|---| +| Scratch guest 9202 | The drill: 5 rounds of install, update, restore, remove | As found (`documentation/audits/endings-list-2026-11-10/evidence/drill/putback-1.txt`, `putback-2.txt`) | +| demo-hp host | Six kernel packages purged. One orderly restart, sent 12:54:26 UTC | Kernels 8 → 2 (7.0.14-28 runs, 7.0.14-26 kept). `/` 62 % → 45 %. Hub: online, tunnel connected. Read again at 13:36 UTC: up, 45 %. **The kernels are not put back; that was the test** | +| demo-hp guest 9201, N100, Tester 1 | Read only | Unchanged | +| ep0, DooPlex workloads, Cloudflare, Hetzner, mail | Not touched. No login to ep0 | — | + +`felhom-os-apply`, the config bundle, the agent on the box and rule R4 did not change. Teardown: machine — 9202 put +back; host — nothing provisioned; hub — nothing provisioned. + +## 5. Commits, CI, register + +| Repo | Commit | CI (read by commit on the public Actions page) | +|---|---|---| +| felhom.eu | `449b065f` Part 0 | run 2606, `gates #1412`, success | +| felhom.eu | `807175c2` Part A | run 2607, `gates #1413`, success | +| felhom.eu | `10ea4c2c` Part B | run 2609, `gates #1414`, success | +| felhom.eu | `b14eb0fa` C1 | run 2611, `gates #1415`, success | +| felhom-agent | `776025c8` C2 | run 2608, `gates #311`, success | +| felhom-controller | `844f19f1` C3 | run 2610, `gates #557`, success | +| felhom.eu | the commit that holds this report (C4, C5, close-out) | read after the push; the result is in the session's closing note | + +The run's job detail could not be read: the `jobs` address answers 401 and no stored secret has Actions rights. The +run status on the public page is what was read. + +**Register:** 230 open at the start, 236 at the end. Opened 6 (R-1373…R-1378). Closed 0. Dated lines were added to +existing rows; in this last commit to R-1369, R-1370 and R-1371. + +**Other close-out facts.** Capability map: no status changed, because an unreleased commit proves nothing on a box; one +dated note was added under A21 (see §7). System poster: none of the three actions; no listed fact changed. +`unproven.py --summary`: 55 claims, 32 walked, 23 not walked — unchanged. CHANGELOG entries, marked unreleased: hub, +agent, controller. `STATUS.md`: new "Updated" paragraph, the old one moved to `documentation/backlog/STATUS-HISTORY.md`; +D71–D75 in the decide table. + +**Instruction file edited:** `documentation/PROMPT-TEMPLATE.md` §N.5, one new paragraph after the poster rule (before: +no line; after: "A place that holds data … updates the 'disk space' section (§14) …"). Why: the brief asked for it +(D69, decision 288). No rule was loosened. + +## 6. Architecture documents read + +**By the main session:** `11-os-updates.md` (§5.4.1, §5.5, §5.6, §5.9, §5.11) · `09-update-architecture.md` §3 +(decisions 40, 139, 286–290) · `08-alarm-ladder.md` (the operator-only register, the fill alarms) · +`16-storage-physical-identity.md` (in full) · `03-host-agent.md` (the restore-test space note, the self-update part) · +`07-backup-architecture.md` §6. + +**By the read-only helpers, for Part A** (their pointers were opened by a tool; see the method verdict): +`07-backup-architecture.md` (§6.5, §8.2a–§8.2e) · `05-hub-architecture.md` · `06-offsite-connectivity.md` · +`01-topology-and-trust.md` · `00-capability-map.md` with its detail files. + +No architecture document covers endings as a whole. That is the finding the list tested. + +## 7. Observations outside the scope + +| Observation | Verdict | +|---|---| +| Two "SMART read failed … context canceled" lines in the agent's log at 12:57:35 UTC, just after the agent started | NOT-A-FINDING: the first read at start was cancelled; 680 reads ran after it with no failure line (read 13:36 UTC) | +| `host_swap_short` warning after the restart | NOT-A-FINDING: report only, by ruling (closed R-1207, D27) | +| `apt`'s own keep list still names 7.0.14-23, a kernel that is gone | NOT-A-FINDING: it had no effect on the plan; noted in the spike's README | +| CC's hub token reads the host list, not a host's detail page or the System page (403) | NOT-A-FINDING: the token is limited on purpose | +| `runbooks/day0-install.md` said the hub has no host delete | Small; FIXED in this commit (the hub has `POST /hosts/{id}/delete`) | +| `runbooks/ep0-datastore-copy.md` heading said "NOT INSTALLED"; closed R-901 says the job was installed on 2026-10-09 | Small; FIXED in this commit, from the closed row. Not re-checked on DooPlex (fence) | +| Map row A21 says "tunnel + zone removed"; no code calls Cloudflare (closed R-688, accepted) | Small; a dated note added under A21 in `capability-map-detail/A.md`. No status change | +| R-1268 (a restore's undo copy stays after "delete backups too") reads as fixed: the source removes it and the drill saw it go | FILED: a dated line on R-1268; left open for a session that can red-prove it | +| Closed R-521 and a test comment call the container's `/` "the guest's root filesystem"; it is an overlay on the data volume | FILED: R-1371 (the guest's real root disk has no watcher), in its dated line of this run | +| R-810 lives in `ROADMAP.md` only | NOT-A-FINDING: the roadmap is its home; D74 points at it | +| The drill sent an install with no kept-data choice and expected a refusal (409). The product answered 202 and installed, because the kept folder was empty | NOT-A-FINDING for the product; the tool's expectation was wrong. It was on scratch guest 9202, inside a round that installs and removes that same app. The app was removed through the product, and 9202 is back as found. Said here because `.claude/rules/live-probes.md` asks for it | diff --git a/STATUS.md b/STATUS.md index c14ca68c..83770f2d 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-11-10.** CC listed every place that holds data on a box — 45 places, read-only on the three test boxes — and asked eight questions of each: 47 answers were „nobody asked”, now in five new rows (R-1368…R-1372). The worst one: nothing removes old kernels from the host; a ring-0 box holds 8 of them (8 GB) and gets about one more each week. You ruled on its four questions at 12:22 (D67–D70: yes, operator only, yes, yes — `09` §3 decisions 286–290); the list is in `documentation/audits/space-list-2026-11-10/README.md`. +**Updated 2026-11-10 (afternoon).** CC listed every way a thing ends on a box or in the hub (74 endings, eight questions each): 22 answers were „nobody asked”, now in six new rows (R-1373…R-1378) and four questions for you (D71–D74). CC removed six old kernels from the HP box by hand with the keep rule you ruled (8 → 2, 6.4 GB back, one restart, all apps healthy after 5 minutes); nothing is built for it yet. Three first space fixes wait in source for a release, the disk-space design (`documentation/architecture/16-storage-physical-identity.md` §14) and the runbook `documentation/runbooks/disk-full.md` (R-1369) are written, and D75 is one more question for you; the report is `REPORT-endings-kernels-fixes-2026-11-10.md`. ## What works - **The Plex attack did not reach Felhom's secrets** (R-1241). @@ -32,6 +32,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| +| D75 | R-1370 | What is the smallest system disk Felhom accepts? Measured: the host's own files need about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB when they stay on the system disk. | **128 GB only with a backup drive; 200 GB without one** (a 128 GB disk gives the host 41 GB: enough for 14, not for three local copies of a box with many apps) | 128 GB always, as today: a box with many apps and no backup drive stops its local whole-box copies | the installer accepts any disk; a small box can lose its local copies | R-1370 | | D74 | R-1375 | Shall CC write the runbook „a household or a tester leaves” now? | **Yes** (five code doors exist; documents only, one evening) | wait for the first leaver: the first run is by memory | no runbook; a leaver is handled by hand | — | | D73 | R-1376 | When you delete a household or a box in the hub, shall the hub delete every record of it? Today 13 tables keep rows. | **Yes** (add the tables to the purge; one hub change) | keep them as history: say so in the privacy text | rows of deleted households and boxes stay in the hub for ever | R-1376 | | D72 | R-1374 | Shall the off-site snapshots of a removed app end after a fixed time, with a notice first? Today they stay for ever. | **Yes — give the number of days** (paid space stops growing) | keep for ever: the household can restore a removed app at any time; paid space only grows | they stay for ever | R-1374 | diff --git a/documentation/PROMPT-TEMPLATE.md b/documentation/PROMPT-TEMPLATE.md index 019581a3..9026731e 100644 --- a/documentation/PROMPT-TEMPLATE.md +++ b/documentation/PROMPT-TEMPLATE.md @@ -375,6 +375,10 @@ Decisions made, architectural state, what's next. > "waiting on the operator" list. **The report names which of the three it did.** The poster is drawn > in Claude Design and nothing here renders it, so `scripts/poster_facts_gate.py` only WARNS when the > facts outrun the drawing — it never fails a push. Same wording in `.claude/rules/unprompted-work.md` §6. +> **A place that holds data.** A task that adds a place that holds data on a box, or changes a place's limit, its +> cleaner or its alarm, updates the "disk space" section (§14) of `architecture/16-storage-physical-identity.md` in the +> same session (D69, `09` §3 decision 288). + If the task **changed what the platform can do** (new/removed capability, a capability moved status, or **what is open changed**), update **all four** in the SAME session: - **`architecture/00-capability-map.md`** — add/adjust the affected *scenario* row with the correct diff --git a/documentation/architecture/16-storage-physical-identity.md b/documentation/architecture/16-storage-physical-identity.md index c2495fcb..09d43d67 100644 --- a/documentation/architecture/16-storage-physical-identity.md +++ b/documentation/architecture/16-storage-physical-identity.md @@ -240,6 +240,179 @@ Also in build B: R-1084 (the whole-box backup offer asks the same-disk question) offered to attach, by device). A bundle that adds a path needs a step bundle on boxes with an older wrapper (R16, R-880): 0.158.1 was delivered through `0.158.1-step1`. +## 14. Disk space — every place, its limit, its cleaner, its alarm (D69, decision 288) + +**What this is.** The design of disk space on a box. One line for each place that holds data: what makes it grow, its +limit, what cleans it, and which alarm fires. Written 2026-11-10 from the space list. **The evidence is not here** — it +is `audits/space-list-2026-11-10/` (`TABLE.md` holds each place with its `file:line` and its measurement; the place ids +below are the pointers into it). Counts on that day: 45 places; 21 have a name in this folder, 16 a mention, 8 none. + +**[DESIGN] The rule that keeps this true.** A change that adds a place that holds data, or changes the limit, the +cleaner or the alarm of one, changes this section in the same session (`PROMPT-TEMPLATE.md` §N.5). A place with no +limit is on the list in §14.2 with its ruling, or it has a register row. + +### 14.1 The places [FACT — read and measured 2026-11-10, `audits/space-list-2026-11-10/TABLE.md`] + +The alarms, by short name: + +- **host disk** — the hub's `host_disk_warning` at 90 % and `host_disk_critical` at 95 %. To the operator only (D68, + decision 287, `08-alarm-ladder.md`). That is in the hub's source and not in the hub that runs: until the next hub + release the household can still get the critical one. +- **pool** — the hub's `storage_fill_warning` at 85 % and `storage_fill_critical` at 90 %, on the worse of the pool's + data and metadata. To the operator. +- **data volume** — the controller, every 10 minutes: warning at 85 % or under 5 GiB free, critical at 95 % or under + 2 GiB. To the household. +- **drive** — the same watch of the controller, on each registered drive. To the household. + +**Host system disk** (`/`, the volume `pve/root`) + +| Id | Place | Grows with | Limit | Cleaner | Alarm | +|---|---|---|---|---|---| +| H1 | the system disk as a whole | its parts below (72–82 % of the used bytes) | the size Proxmox gave it; no budget per part (§14.3, R-1370) | each part's own; the whole disk only by the operator's hand (`runbooks/disk-full.md`) | host disk | +| H3 | whole-box copies on the current local target | the guest's size (R-1366) | the newest N (default 3) | Proxmox after each good run; the agent removes half-written copies | a skipped run is reported to the hub and shown on the backup page; host disk | +| H4 | whole-box copies on a former target | a move of the target | none (R-1087) | nobody | host disk | +| H5 | guest template archives | each install and re-install | none (R-1368) | nobody; only an uninstall with `--remove-golden` | host disk | +| H6 | backup logs and half-written copies | each run | a failed run's log: none (R-1368) | Proxmox; the agent | host disk | +| H7 | the host journal | the agent's `sudo` calls (R-1367) | the journald default: 10 % of the disk, at most 4 GB | journald | host disk | +| H8 | other host logs, task logs, graphs | each task | rotation | `logrotate`, Proxmox | host disk | +| H9 | old kernels | each approved kernel | none today. Ruled: keep the running one, the one before it and the newest (decision 286, `11-os-updates.md` §5.11.1). Proven by hand, not built | nobody today | host disk; later `os_update_stale` | +| H10 | the EFI partition | a boot-loader update | 1.07 GB, 1 % used | none needed | none (R-1371) | +| H11 | the host package cache | each OS update | emptied in each run | the update wrapper | host disk | +| H12 | the agent's self-update folder | each agent update | none on a released agent; one binary after the next agent release (in source, R-1368) | the agent, after that release | host disk | +| H13 | old agent binaries | `.prev`: one file. `.bak-*`: each re-run of the installer | none for `.bak-*` (R-1368) | nobody | host disk | +| H14 | wrapper state and previous config bundles | each bundle | the last 3 | the wrapper | host disk | +| H15 | host `/tmp` and `/var/tmp` | nothing that lasts | `/tmp` is in memory: half of memory | systemd, after 10 and 30 days | none for `/tmp` (R-1371) | +| H16 | a bare folder under a drive that is not mounted | a write while the drive is away | the write is refused | nothing to clean | `storage_disconnected`, to the household | +| H18 | swap | fixed | its volume | the kernel | `host_swap_short`, to the operator | + +**Thin pool** (`pve/data`) + +| Id | Place | Grows with | Limit | Cleaner | Alarm | +|---|---|---|---|---|---| +| H2 | the pool | each block a guest writes | the pool's size; auto-extend is off | the agent's weekly trim (decision 139); a left-over scratch guest: a retry every 10 minutes | pool | +| H17 | the restore test's scratch guest | each restore test | it needs restored size × 1.2 + 5 GiB free, off the tested guest's pool when it can | the agent at the end; a retry every 10 minutes | pool | + +**Guest root disk** (32 GiB; 4–5 % used on three boxes) + +| Id | Place | Grows with | Limit | Cleaner | Alarm | +|---|---|---|---|---|---| +| G1 | the root disk as a whole | guest OS updates | 32 GiB; the agent can grow it, never shrink it | the update wrapper, `logrotate` | none (R-1371) | +| G13 | app definitions, `/opt/docker/stacks` | the catalog's size | under 2 MB measured | an app removal deletes its `app.yaml` | none (R-1371) | +| G15 | the guest's system log | each line | the journald default; it is in memory (R-1372) | journald; a restart | none | +| G16 | the guest package cache | each guest OS update | emptied in each run | the wrapper | none (R-1371) | +| G17 | guest `/tmp` | nothing that lasts | in memory | systemd; a restart | none | + +**Guest data volume** (`mp0`, `/var/lib/felhom`) + +| Id | Place | Grows with | Limit | Cleaner | Alarm | +|---|---|---|---|---|---| +| G2 | the data volume as a whole | G3–G12 and G14 | 50 / 100 / 130 GiB (§7; older boxes 70 or 250). A new app is refused under max(5 GB, 10 %) free; a backup is refused at 97 % or under 1 GiB | the three image rules below; the weekly trim | data volume. A box with one volume counts it twice until the next controller release: two warnings for one fill (R-1371) | +| G3 | app images, and one undo image for each app | each install and update | a count: the current and the one before, for each app. No limit in bytes, by ruling (§14.2) | the controller | data volume | +| G4 | old controller images | each controller update | the running one and the one before | the controller | data volume | +| G5 | infrastructure images | a change of the pinned version | the pinned one and the one before | the controller | data volume | +| G6 | helper images (`alpine`, `hello-world`) | pulled once | no version is pinned (R-1372) | nobody | data volume | +| G7 | named app volumes | the app's use | none for each app; the volume's | only an app removal | data volume | +| G8 | volumes with no name | each re-create of a container | none (R-995, R-1267) | nobody | data volume (bytes only) | +| G9 | containers' writable layers; the off-site cache | temporary files; the off-site store's size | none of its own | a re-create drops it | data volume | +| G10 | container logs | each line | 10 MB × 3 for each container | Docker | data volume | +| G11 | the build cache | nothing | empty | — | data volume | +| G12 | the controller's state volume | graph rows; the catalog | rows older than 30 days are deleted | the controller | data volume | +| G14 | backups of apps with no drive (`sys_drive`) | dumps | one point for each app; refused at 97 % or under 1 GiB | the nightly replace; moved units are pruned | data volume | + +**A data drive** + +| Id | Place | Grows with | Limit | Cleaner | Alarm | +|---|---|---|---|---|---| +| D1 | `backups/primary` | dumps | one point for each app. A removed app's kept unit: none, by ruling (§14.2) | the nightly replace; a removal with "delete the backups too" | drive | +| D2 | `backups/secondary` | the apps' data | it needs max(2 GB, 20 %) free. A removed app's copy: none, by ruling (§14.2) | the mirror; the household, from the backup page | drive | +| D3 | `backups/offsite-restore` | each off-site restore or check | none (R-1372) | the household's delete button | drive | +| D4 | `backups/offsite-proof` | one unit a night, for minutes | a headroom check | the proof itself | drive | +| D5 | `backups/restore-undo` | each restore | 30 days unused; none under a support hold, by ruling (§14.2) | the controller's sweep | drive; the household is told on a delete | +| D6 | `kept` | each "remove, keep my data" | none, by ruling (§14.2) | the household | drive | +| D7 | `appdata` | the app's use | the drive | a removal "with data" | drive | +| D8 | `userdata` | the household | none, by ruling (§14.2) | the household | drive | +| D9 | `shares` | the household's copies | the drive | the household | drive | +| D10 | `backups/set-aside` | each re-install that finds a unit | none, by ruling (§14.2) | the household; a removal with "delete its backups too" | drive | + +### 14.2 What can grow without a limit, by ruling [FACT — each line is a ruling, not a defect] + +Do not file these as defects (R-370). A change to one is a decision for the operator. + +1. Undo images: one for each app, no limit in bytes (decision 53). +2. The kept data of a removed app: only the household deletes it, with the app's name typed (decision 40). +3. The household's own files (decision 67). +4. A set-aside backup unit (decision 256). +5. A restore's undo copy while a support hold is on it (decision 281). +6. The second-drive copy of a removed app (D25). +7. The backup unit of a removed app that the household kept (`07-backup-architecture.md` §6). +8. Images in the registry (decision 62). This is not on a box. + +### 14.3 The host system disk — a budget from measured numbers [FACT for the numbers; the answer is OPEN — D75, R-1370] + +Measured 2026-11-10 (`audits/space-list-2026-11-10/evidence/reading-1/`, GB = 10⁹ bytes): + +| | 128 GB disk (demo-hp) | 200 GB disk (Tester 1) | 512 GB disk (N100) | +|---|---|---|---| +| root volume, as Proxmox made it | 41.4 | 62.9 | 100.9 (the 96 GiB cap) | +| used | 24.0 | 21.6 | 38.1 | +| Proxmox, Debian and all that has no line of its own | 3.3 | 3.2 | 3.4 | + +No rule is written here for a size between these three: the Proxmox installer's formula was not read. + +**What the root needs when the whole-box copies go to a backup drive:** + +| Part | GB | From | +|---|---|---| +| Proxmox, Debian, small parts | 4 | the last line above | +| kernels: the three the ruling keeps, and one in install | 4.4 | one kernel = 1.02 GB of modules + 0.10 GB in `/boot` | +| the host journal at its cap | 4.3 | measured on two boxes (H7) | +| one guest template archive | 0.65 | H5 | +| the agent's files, after the self-update fix | 0.1 | H12 (0.75 before the fix) | +| free space an OS update asks for | 0.5 | `felhom-os-apply`, the free-space check | +| **Sum** | **≈ 14** | | + +With the parts nobody cleans today (8 kernels, 49 agent binaries, 4 templates) the same sum is about 21 GB, and it grows +by 1.1 GB with each kernel. + +**What local whole-box copies add.** Let C be one packed copy. The copies need keep × C. A new copy also needs +1.25 × C + 1.07 GB free before it starts, because Proxmox prunes only after a good run (A `internal/backup/runner.go`, +the space check). Measured C: 9.1 GB (demo-hp, 26 containers), 3.15 GB (Tester 1), 1.45 GB (N100). + +| Keep | C | Copies + the free space the next one needs | With the 14 GB above | +|---|---|---|---| +| 3 | 3.15 GB | 14.5 GB | 28.5 GB | +| 3 | 9.1 GB | 39.7 GB | 53.7 GB | +| 1 | 9.1 GB | 21.5 GB | 35.5 GB | + +**What follows from the numbers. This is not a decision.** + +- **With a backup drive** the root needs about 14 GB. The 41 GB root of a 128 GB disk holds that. The limit is then the + pool: a 128 GB disk has a 54 GiB pool, and the guest is promised 32 + 50 GiB of it (§7). That is over-promised by + design and it is watched (the pool alarm). The first measured disk whose pool holds 32 + 50 + the 16 GiB reserve is + the 200 GB one (113 GiB). +- **With no backup drive** the root needs 28 to 54 GB at keep 3, by the number of apps. A 128 GB disk holds a small + household. It does not hold demo-hp's app set at keep 3. demo-hp keeps 1, and 35.5 GB in a 41.4 GB root leaves + little room: a copy that cannot fit is skipped before it starts (R-548). The 63 GB root of a 200 GB disk holds the + 54 GB case with 9 GB to spare. +- The installer refuses no disk today, and nothing checks this sum at install time. + +The question for the operator is D75 in `STATUS.md`: the smallest system disk Felhom accepts, with and without a backup +drive. + +### 14.4 Open design points [each has a row] + +| Point | Row | +|---|---| +| Nobody watches the guest's root disk (G1, G13, G16). It is 4–5 % used today; app definitions lie on it | R-1371 | +| No alarm for the EFI partition (H10) or for the places in memory (H15, G15, G17) | R-1371 | +| A drive in the backup role reports as `usb` or `local-dir`. So the household gets the `storage_fill` mail for a drive it must not clean by hand | R-1371 | +| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. Not measured on the running hub in the 2026-11-10 run | R-1371 | +| Helper images carry no version; the guest's journal has no stated cap; off-site restores have no limit | R-1372 | +| Old kernels: the keep rule is ruled and proven by hand, not built | `11-os-updates.md` §5.11.1 | +| The parts of the host disk with no limit and no cleaner (H4, H5, H6, H13) | R-1087, R-1368 | +| No budget is enforced; no smallest disk is ruled | R-1370, D75 | +| What to do by hand when a disk is full | `runbooks/disk-full.md` (R-1369) | + ## Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)] Read at `file:line` on `main` by the main session: diff --git a/documentation/architecture/capability-map-detail/A.md b/documentation/architecture/capability-map-detail/A.md index 7cd20ce4..efc2de24 100644 --- a/documentation/architecture/capability-map-detail/A.md +++ b/documentation/architecture/capability-map-detail/A.md @@ -435,6 +435,8 @@ stayed. **No other customer touched:** the three live boxes' repositories re-lis **Gap / roadmap (as written):** Three acknowledgements + typed customer-id + stale-preview check + ONLINE-host refusal, ALL before any write. Ruling-3 preserved BY CONSTRUCTION (leg 2 never sees a host row); custody purged EXACTLY ONCE, in leg 3. **Coupling:** hub-only — no agent/controller/catalog change; the cascade calls the same service paths as manual host-delete and standalone RESET, so their rules move together. **v0.70.0 (2026-07-21):** added the **residue** leg — `GetCustomers()` is REPORT-derived, so before it a fully deleted customer stayed on the Customers list and its report stream kept the staleness/offsite checkers alerting (live: `demo-vm-felhom` deleted 07-18, still emailing `offsite_stale` on 07-21). The leg also purges the credential-bearing `appliance_registrations` + `selfbind_tokens`. **Ghost customers (config row already gone) are now deletable** — 404 means "nothing here", not "no config row"; the Hetzner/descriptor legs record `skipped_no_config`. **Gap:** the end-to-end live leg on a scratch customer (external Hetzner teardown observed from outside) is not yet run +**Noted 2026-11-10 (endings list):** „tunnel + zone removed” is not what the code does. No leg of the delete calls Cloudflare; the operator removes the tunnel and the DNS record by hand, guided by the dialog. The operator accepted that on 2026-10-05 (closed R-688). The words above stay as written. + --- diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index c6e1953d..6e5405b3 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -282,8 +282,8 @@ stopping line that lies. | **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | | **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC | | **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | -| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC | -| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. | — | Docs: a short „disk space” section in an architecture document: each part of the system disk with its limit and its cleaner, pointing at the audit table. Then the installer can refuse a disk that cannot hold the sum. | CC | +| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC | +| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. | — | Docs: a short „disk space” section in an architecture document: each part of the system disk with its limit and its cleaner, pointing at the audit table. Then the installer can refuse a disk that cannot hold the sum. | CC | | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC | | **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC | | **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC | @@ -326,7 +326,7 @@ stopping line that lies. | **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC | | **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC | | **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) | -| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC | +| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC | | **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC | ## Hub & operator — 16 rows (P2 1, P3 4, P4 11) diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index 60b46d21..58af3344 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.** +## Moved from STATUS on 2026-11-10 (afternoon) + +**Updated 2026-11-10.** CC listed every place that holds data on a box — 45 places, read-only on the three test boxes — and asked eight questions of each: 47 answers were „nobody asked”, now in five new rows (R-1368…R-1372). The worst one: nothing removes old kernels from the host; a ring-0 box holds 8 of them (8 GB) and gets about one more each week. You ruled on its four questions at 12:22 (D67–D70: yes, operator only, yes, yes — `09` §3 decisions 286–290); the list is in `documentation/audits/space-list-2026-11-10/README.md`. + ## Moved from STATUS on 2026-11-10 **Updated 2026-11-09.** demo-hp did not restart, because its local whole-box copy was refused for space again and the night update waited; CC freed space (old images, an old installer file), so tonight's copy fits. Controller 0.351.0 is released to every box (floor 12:18 UTC): it deletes old test builds and old system images by itself. Two new rows: the whole-box copy is mostly app programs (your decision, R-1366), and the host log noise (R-1367). diff --git a/documentation/runbooks/day0-install.md b/documentation/runbooks/day0-install.md index 87e0aa8c..2c0a1d18 100644 --- a/documentation/runbooks/day0-install.md +++ b/documentation/runbooks/day0-install.md @@ -525,8 +525,9 @@ recovery credential). **Expected remnants** (documented, not residue): -- The **hub host record** (+ its report/guest history) — the hub currently has **no host-delete**; - removing it is a direct-DB operator action (recorded follow-up). +- The **hub host record** (+ its report/guest history) — the operator removes it with the host's delete action in + the hub (`POST /hosts/{id}/delete`; `architecture/05-hub-architecture.md`, the host-delete table). *(Corrected + 2026-11-10: this line said the hub had no host delete.)* - The `sudo` and `dnsmasq` packages (system packages installed as dependencies; inert). - The **golden vzdump** on the archive storage — pass `--remove-golden` to delete it too. - System noise: LVM archive metadata, dpkg lists, PVE's own rrd/`/var/lib/lxc` scratch entries. diff --git a/documentation/runbooks/disk-full.md b/documentation/runbooks/disk-full.md new file mode 100644 index 00000000..f2b1806b --- /dev/null +++ b/documentation/runbooks/disk-full.md @@ -0,0 +1,156 @@ +# RUNBOOK — a disk on a box is full, or nearly full (R-1369) + +**When:** a space alarm came (`host_disk_*`, `storage_fill_*`, the controller's fill warning), a whole-box copy was +skipped for space, or an update was refused for space. + +**What this is:** one short part for each disk. Each part says how to see what is big, what is safe to remove and by +which path, what a hand must never remove, and what the product does by itself. **The design is not here**: each place +with its limit, cleaner and alarm is `architecture/16-storage-physical-identity.md` §14. The place ids (H1, G2, D6 …) +are the same there and in `audits/space-list-2026-11-10/TABLE.md`. + +**Each hand step carries a mark.** `PROVEN` = a person or CC did exactly this on a box, with the date and the +evidence. `NOT PROVEN` = nobody did it; it is not a step yet. **Every `PROVEN` step below was proven on demo-hp, a test +box.** None was done on a household's box. + +**Before any hand step, on any disk:** + +1. Read which box it is and what you may do there: `runbooks/target-selection.md`. +2. Check that no backup, restore, restore test or update runs: `pct list` shows no lock; `ps -ef | grep -E + 'vzdump|pct restore|apt'` shows nothing. +3. Save what you see before you remove anything (R-320): the `df`, the list, the file's checksum. +4. On a household's box a hand removes nothing without the operator's word. + +--- + +## 1. Host system disk (`/`) + +**See what is big (read-only):** + +```bash +df -h / +du -xsh /var/lib/vz/dump /var/lib/vz/template /var/log/journal /lib/modules /boot /var/lib/felhom-agent/selfupdate +journalctl --disk-usage +pvesm list local +dpkg -l 'proxmox-kernel-*-pve-signed' | grep ^ii +``` + +**The product does by itself:** it empties the package cache in each OS update. It keeps the newest N whole-box copies +(default 3) and removes half-written ones. journald caps the journal (10 % of the disk, at most 4 GB). It refuses an OS +update under 500 MB free. It skips a local whole-box copy that cannot fit, and says so on the backup page (R-548). The +hub alarms at 90 % and 95 %. + +**Safe to remove, by which path:** + +| What | Step | Mark | +|---|---|---| +| Old kernels | `ssh python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). **NOT PROVEN:** on a second box; a boot of the kept older kernel | +| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) | +| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free `, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only | +| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) | +| Old binaries in the agent's self-update folder (H12) | — | **NOT PROVEN.** The agent cleans the folder after the release that carries the fix (R-1368) | +| Extra guest template archives (H5); `/usr/local/bin/felhom-agent.bak-*` (H13) | — | **NOT PROVEN** (R-1368) | + +**Never by hand:** + +- The running kernel and the one before it. Never a kernel with a plain `apt autoremove` or by a meta-package: `apt` + did not refuse the running kernel, and one meta-package pulled `proxmox-ve` with it + (`audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt`). +- `pending.json` in the self-update folder, and `/usr/local/bin/felhom-agent.prev`. The agent's rollback needs both. +- The newest whole-box copy. +- Anything under `/var/lib/felhom-os-apply/`: the wrapper's state and the last config bundles. + +## 2. Thin pool (`pve/data`) + +**See what is big (read-only):** + +```bash +lvs -o lv_name,lv_size,data_percent,metadata_percent pve +pct list +``` + +**The product does by itself:** the agent trims each guest once a week (decision 139). It removes a restore test's +scratch guest at the end and tries again every 10 minutes when that fails. A restore test starts only with enough free +space. The hub alarms at 85 % and 90 % on the worse of data and metadata. + +**At 100 %** the guest's disks go read-only. A write in progress can be cut off: on 2026-09-24 two apps' Redis files +were (`architecture/03-host-agent.md` §8, the R-672 note; `audits/r672-2026-09-24/README.md`). + +| What | Step | Mark | +|---|---|---| +| Space the guest freed but the pool still holds | `pct fstrim ` | **PROVEN 2026-10-06**: pool 65.5 % → 33.4 % in 24 s, the box answered all the time (`audits/ten-answers-2026-10-06/r444-measure.txt`). The product does this weekly | +| A guest that is read-only after a full pool | Free pool space first. Stop the guest. `pct fsck `, then `pct fsck --device mp0`. Start it. Check each app | **PROVEN once, 2026-09-24** (`audits/r672-2026-09-24/README.md`, Part B; `B4-fsck.txt`). Two Redis files then needed `redis-check-aof --fix`, with the operator's yes and a copy kept first | +| A left-over scratch guest or a thin device with no volume | `runbooks/whole-guest-backup-orphan-thin.md` | see that runbook | + +**Never by hand:** `lvremove` of a `vm-9201-disk-*` volume. `lvextend` of the pool into the space that belongs to the +internal place. A thin device, except by the runbook above. + +## 3. Guest root disk (32 GiB) + +**See what is big (read-only):** + +```bash +pct exec -- df -h / +pct exec -- du -xsh /var /opt /usr /root +``` + +**The product does by itself:** it empties the guest's package cache in each guest OS update. Nothing else. **Nobody +watches this disk** (R-1371). It was 4–5 % used on three boxes on 2026-11-10. + +**Safe to remove:** nothing is proven. **NOT PROVEN** for every hand step on this disk. When it is full, the agent can +grow it; that path is the product's, not a hand's. + +**Never by hand:** `/opt/docker/stacks//app.yaml`. It holds the app's settings and its records. + +## 4. Guest data volume (`/var/lib/felhom`) + +**See what is big (read-only):** + +```bash +pct exec -- df -h /var/lib/felhom +pct exec -- docker system df +pct exec -- docker image ls --format '{{.Repository}}:{{.Tag}} {{.Size}}' +``` + +Never dump a container's `Env`. + +**The product does by itself:** it keeps the current image and one undo image for each app, and removes older ones. It +keeps the running controller image and the one before; the same for infrastructure images. Docker caps each container's +log at 10 MB × 3. It deletes graph rows older than 30 days. It refuses a new app under max(5 GB, 10 %) free. It refuses +a backup at 97 % or under 1 GiB, app by app, and deletes nothing. It warns the household at 85 % and 95 %. + +**Safe to remove, by which path:** + +| What | Step | Mark | +|---|---|---| +| An app the household does not want | The household removes it on the app's page. That is the only path that frees an app's volume | product path | +| Old controller, infrastructure and helper images that no container uses | `docker image rm `, one exact name at a time, after `docker ps -a` shows no container on it | **PROVEN 2026-11-09**: 36 images, 3.4 GB (off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/cleanup.txt`). The controller now removes old controller and infrastructure images by itself. Helper images stay (R-1372) | +| Volumes with no name (G8) | — | **NOT PROVEN** (R-995, R-1267) | + +**Never by hand:** `docker system prune`, `docker volume prune`, `docker image prune -a`. The undo images are kept on +purpose (decision 53), and volumes hold the apps' data. An app's undo image. A named volume. + +## 5. A data drive + +**See what is big:** sizes of the top folders only. Do not list file names below a household's data. + +```bash +pct exec -- df -h +pct exec -- du -xsh /* /backups/* +``` + +**The product does by itself:** the nightly backup replaces each app's one point. The second-drive copy needs +max(2 GB, 20 %) free. The controller deletes a restore's undo copy after 30 days without use, and tells the household. +The off-site proof cleans its own scratch. The controller warns the household at 85 % and 95 %. + +**Safe to remove:** only what the household removes on the product's pages: the kept data of a removed app (with the +app's name typed), a set-aside unit, an off-site restore, its own files. **A hand step on a data drive: NOT PROVEN, and +not wanted.** + +**Never by hand:** `userdata`, `appdata`, `kept`, `shares`, `backups/primary`, `backups/set-aside`, +`backups/restore-undo`. These grow without a limit by ruling (`16` §14.2; decisions 40, 67, 256, 281). Only the +household deletes them. + +--- + +**After any hand step:** read `df` again, save the before and after to the evidence folder, and check the box on the +hub. Then write a dated line on the row that names the place. diff --git a/documentation/runbooks/ep0-datastore-copy.md b/documentation/runbooks/ep0-datastore-copy.md index 3bba480e..666c062d 100644 --- a/documentation/runbooks/ep0-datastore-copy.md +++ b/documentation/runbooks/ep0-datastore-copy.md @@ -81,7 +81,7 @@ Provision a new ep0 (06 §5), then on it add DooPlex as a remote and run Do **not** prune the copy tighter than decision 71 (8 weekly copies); never tighter than ep0's own retention. -## Removing a deleted customer's copy (R-901, `09` §3 decision 181) — WRITTEN 2026-10-08, NOT INSTALLED +## Removing a deleted customer's copy (R-901, `09` §3 decision 181) — WRITTEN 2026-10-08, INSTALLED 2026-10-09 The sync keeps what ep0 removed (`remove-vanished false`), so a deleted customer's namespace stayed here for ever. Ruling: removed **within 30 days**. The job `scripts/ep0-copy-gc/felhom-ep0-copy-gc` (tests beside it) runs daily at @@ -92,7 +92,9 @@ nothing; if more than 2 namespaces are absent at once (a partial list, e.g. a re deletes at most one namespace per run; a HOLD file (`/etc/felhom/ep0-copy-gc/HOLD`) stops it. Without `--apply` it only prints. Worst case: ≤ 1 day to notice + 7 days grace + 1 day = inside 30 days. -**Not installed. To install (needs the operator's word — it is a change on DooPlex and it deletes data):** +**Installed 2026-10-09 with the operator present (closed R-901: key root-only, a dry run, then the 08:00 timer). Not +re-checked on DooPlex on 2026-11-10, when this heading was corrected from „NOT INSTALLED”. The steps, for an install +from nothing (needs the operator's word — it is a change on DooPlex and it deletes data):** 1. Two tokens, secrets in root-only files under `/etc/felhom/ep0-copy-gc/` (0600), never printed: - `ep0-reader.secret` — the existing ep0 read token `root@pam!dooplex-sync` (its secret is in `remote.cfg`, base64); `ep0.fingerprint` — ep0's pinned certificate fingerprint (also in `remote.cfg`).