From 327a7668e62b2439c18fa2c50e83f8632ea04639 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 14 Nov 2026 16:20:49 +0100 Subject: [PATCH] =?UTF-8?q?D82=20recorded=20(09=20=C2=A73=20decision=20307?= =?UTF-8?q?):=20a=20household's=20box=20may=20remove=20old=20kernels=20aft?= =?UTF-8?q?er=20two=20real=20night=20removals=20on=20each=20ring-0=20box?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- STATUS.md | 1 - documentation/architecture/09-update-architecture.md | 10 ++++++++++ documentation/backlog/OPEN-ITEMS.md | 2 +- 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/STATUS.md b/STATUS.md index 91cf41ad..9b41a83e 100644 --- a/STATUS.md +++ b/STATUS.md @@ -32,7 +32,6 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| -| D82 | R-1368 | May a household's box (ring 1) also remove its old kernels by itself at night? In release 2a only our ring-0 test boxes do. | **Yes, after 7 clean nights on the N100 and demo-hp** (the rule is proven by hand, by tests and on Tester 1; a household's disk fills by 1 GB per kernel) | no: a household's box keeps every kernel; someone removes them by hand before the disk is full | ring-1 boxes keep every kernel (Tester 1: 2 now, Tester 2: unknown) | the last part of R-1368 | | D66 | R-1365 | Should a security-watch source that stays down mail again every 7 days, as an open notice does? Today it mails once, then only the Monday summary lists it. | **Yes** (a real outage then cannot be forgotten; the seven pins that can never be read no longer count, so it stays quiet when all is well) | no: one line in a long Monday mail stays the only reminder | a source that stays down mails once; the Monday list still shows it | — | | D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | | D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — | diff --git a/documentation/architecture/09-update-architecture.md b/documentation/architecture/09-update-architecture.md index 787f4c13..80c868df 100644 --- a/documentation/architecture/09-update-architecture.md +++ b/documentation/architecture/09-update-architecture.md @@ -1265,6 +1265,16 @@ Recorded before the work (`audits/space-list-2026-11-10/README.md` §5 held the Hetzner and the mail service are read from source and documents only: no hub database read, no account call. *Operator, 2026-11-10 12:22.* +### 2026-11-14 (16:06, operator) — D82: a household's box may remove its old kernels + +307. **D82 = yes: a household's box (ring 1) may remove its old kernels by itself — after two REAL removals at night on + each of the two ring-0 boxes (the N100 and demo-hp).** A quiet night with nothing to remove does not count. How the + count is read: a kernel-layer report with mode `kernel-cleanup` and outcome `cleaned`, trigger `night`, per box + (the hub's stored reports; the box's own journal line „old kernels removed” is the second channel). It needs a + later agent release to switch it on; nothing is built for it in release 2a. **Cost of the other option:** a + household's box keeps every kernel, about 1 GB each, until someone removes them by hand. R-1368. + *Operator, 2026-11-14 16:06.* + ### 2026-11-14 (09:55, operator, planning chat) — D80, D81, the N100's time service, the build order Recorded before any build. **None of these is built at the time of writing.** The questions were D80 and D81 in diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 8c450e49..27fdc5d6 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -281,7 +281,7 @@ stopping line that lies. |---|---|---|---|---|---|---|---| | **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC02, CC06-6, CC07-6, CC22-6). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | | **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC | -| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | NARROWED — **2026-11-14 (release 2a): the old-kernel part is BUILT — agent 0.174.0, vouched 15:21 CET (`11` §5.11.1 `[FACT — built]`).** A ring-0 box removes its old kernels at night by the wrapper's own rule. On boxes: a real removal on Tester 1 (3 → 2 kernels, 1.12 GB, 8.5 s; the default, the pinned file, the generators and the state file unchanged; a second run removed nothing; one clean restart) — started by hand with a ring-0 mark set for the test on the operator's yes, because a night cannot be started by day (R-1394); dry runs on demo-hp (nothing to remove; refusals P1, P2, P3) and the N100 (6 of 8 will go). **LEFT for this part: the step by the agent at night — read the N100's and demo-hp's night of 2026-11-15; then a ruling for ring-1 boxes (D82).** Evidence `audits/release-2a-2026-11-14/`. The other four places of this row are unchanged. Earlier: OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | +| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | NARROWED — **2026-11-14 16:06 — RULED (D82, `09` §3 decision 307): a household's box may remove its old kernels, after two REAL night removals on each ring-0 box (the N100 and demo-hp); quiet nights do not count. Count so far: the N100 0, demo-hp 0 (the Tester 1 removal of 2026-11-14 was started by hand and is not a night). It needs a later agent release to switch it on.** Earlier: NARROWED — **2026-11-14 (release 2a): the old-kernel part is BUILT — agent 0.174.0, vouched 15:21 CET (`11` §5.11.1 `[FACT — built]`).** A ring-0 box removes its old kernels at night by the wrapper's own rule. On boxes: a real removal on Tester 1 (3 → 2 kernels, 1.12 GB, 8.5 s; the default, the pinned file, the generators and the state file unchanged; a second run removed nothing; one clean restart) — started by hand with a ring-0 mark set for the test on the operator's yes, because a night cannot be started by day (R-1394); dry runs on demo-hp (nothing to remove; refusals P1, P2, P3) and the N100 (6 of 8 will go). **LEFT for this part: the step by the agent at night — read the N100's and demo-hp's night of 2026-11-15; then a ruling for ring-1 boxes (D82).** Evidence `audits/release-2a-2026-11-14/`. The other four places of this row are unchanged. Earlier: OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | | **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC | | **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. **2026-11-10 15:19 — RULED (D75 = one number, 200 GB, with or without a backup drive; `09` §3 decision 295).** This replaces CC's two-step pick. **Open, not ruled:** what the installer does below 200 GB — refuse, or warn. demo-hp (128 GB) stays as a test box. `16` §14.3 carries the ruling. | — | Design first: what the installer does below 200 GB (refuse, or warn) — a later session. Then the installer. | CC | | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |