From fffbb40ce0137ba1fb4fae91b7ff58752e14d064 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 15 Nov 2026 06:41:17 +0100 Subject: [PATCH] =?UTF-8?q?night=20build=202026-11-14:=20the=20N100's=20fi?= =?UTF-8?q?rst=20night=20kernel=20removal=20read=20(D82:=201=20of=202),=20?= =?UTF-8?q?close-out=20=E2=80=94=20report,=20status,=20register?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reads only on the N100 and demo-hp; no box changed, nothing released. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- CONTEXT.md | 19 ++ REPORT-night-build-2026-11-14.md | 130 +++++++++ STATUS.md | 7 +- .../audits/night-build-2026-11-14/E/README.md | 56 ++++ .../night-build-2026-11-14/E/demo-hp-2.txt | 37 +++ .../night-build-2026-11-14/E/demo-hp-3.txt | 50 ++++ .../night-build-2026-11-14/E/demo-hp.txt | 33 +++ .../night-build-2026-11-14/E/felhom-pve-2.txt | 65 +++++ .../night-build-2026-11-14/E/felhom-pve-3.txt | 76 ++++++ .../night-build-2026-11-14/E/felhom-pve.txt | 31 +++ .../E/hub-system-page-0634.txt | 248 ++++++++++++++++++ .../night-build-2026-11-14/E/partE-read.sh | 38 +++ .../night-build-2026-11-14/E/partE-read2.sh | 23 ++ .../night-build-2026-11-14/E/partE-read3.sh | 12 + documentation/backlog/OPEN-ITEMS.md | 4 +- documentation/backlog/STATUS-HISTORY.md | 4 + documentation/lived-in/n100.md | 1 + 17 files changed, 830 insertions(+), 4 deletions(-) create mode 100644 REPORT-night-build-2026-11-14.md create mode 100644 documentation/audits/night-build-2026-11-14/E/README.md create mode 100644 documentation/audits/night-build-2026-11-14/E/demo-hp-2.txt create mode 100644 documentation/audits/night-build-2026-11-14/E/demo-hp-3.txt create mode 100644 documentation/audits/night-build-2026-11-14/E/demo-hp.txt create mode 100644 documentation/audits/night-build-2026-11-14/E/felhom-pve-2.txt create mode 100644 documentation/audits/night-build-2026-11-14/E/felhom-pve-3.txt create mode 100644 documentation/audits/night-build-2026-11-14/E/felhom-pve.txt create mode 100644 documentation/audits/night-build-2026-11-14/E/hub-system-page-0634.txt create mode 100644 documentation/audits/night-build-2026-11-14/E/partE-read.sh create mode 100644 documentation/audits/night-build-2026-11-14/E/partE-read2.sh create mode 100644 documentation/audits/night-build-2026-11-14/E/partE-read3.sh diff --git a/CONTEXT.md b/CONTEXT.md index eb4a5063..0fd7cef1 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,25 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-11-14 night → 11-15 morning (unattended) — releases 2b and 3 are BUILT IN SOURCE; nothing is released.** +> Report: `REPORT-night-build-2026-11-14.md`. Heads: `felhom.eu` `d2683b28`, `felhom-agent` `54ac2c2`, +> `felhom-controller` `ee3b05e`; the installer is not tagged. Live versions did not move (hub 0.181.0, agent 0.174.0 +> vouched, controller floor 0.352.1). What a session must know before it touches this code: +> (1) **Trust in Proxmox** (`03` §12a): the root step keeps a copy of the box's root CA; the agent moves a pinned box +> in the order copy → proof → config and clears the fingerprint LAST (a set fingerprint wins over a CA file). Every +> writer of `agent.json` takes `config.FileMu`. (2) **A report carries `sent_at`** (stamped at sending; `reported_at` +> is the start of collecting) and the hub alarms on more than one minute. (3) **A hub checker that tells a thing once** +> puts its event type in `notify.perSubjectCooldownEvents` and a `subject` in the details, or the one-hour limit eats +> the second thing's mail; marks that a box can mint are capped per box. (4) `store.SetOSAlarmRaised` with a zero time +> DELETES the row. (5) The boot token is the AGENT's (`guest_boot_key`); the controller compares. (6) `DeleteHost` +> now deletes `os_host_settings`: a box installed again under the same id returns as ring 1. +> **Decided by CC unattended — operator may reverse:** the operator's action log (`operator_actions`) of a deleted +> household goes one year after the deletion, with `events` and `notification_log` (`05` §14.2). Options: at once +> (loses „who did what” on the day of the deletion) or one year (the rule decision 181 already gives the other audit +> rows). Reversible: the rows are not deleted before the year ends. +> **The N100's first night removal** (2026-11-15 05:58 CET): 8 → 2 kernels, 6.81 GB, healthy; D82 count N100 1 of 2, +> demo-hp 0 of 2 (R-1368; `audits/night-build-2026-11-14/E/`). Open decisions: D83, D84, D85 (`STATUS.md`). + > **2026-11-10 (afternoon, attended release run) — the three Space fixes are RELEASED; the note below is history.** > (1) Agent 0.173.0, vouched by the operator 15:29 UTC: the self-update folder cleans itself at the commit of the > update that brings the fix (49 / 48 / 33 files → 1 on the three test boxes); a rollback after the clean-up was proven diff --git a/REPORT-night-build-2026-11-14.md b/REPORT-night-build-2026-11-14.md new file mode 100644 index 00000000..d141d193 --- /dev/null +++ b/REPORT-night-build-2026-11-14.md @@ -0,0 +1,130 @@ +# Night build 2026-11-14 → 11-15: releases 2b and 3 in source, then the N100's night + +**No box was changed and nothing was released.** No release, no deploy, no vouch, no floor change, no manifest bump, +no golden, no installer tag. No ep0 change. No call to the off-site provider, Cloudflare or the mail service (the +tests use the repo's fakes). No test alarm that can reach a household. Nothing removed from the `sec` store. No legal +or household-facing text published. The only commands on a test box were reads, in Part E, on the N100 and demo-hp +(`documentation/audits/night-build-2026-11-14/E/`). Provisioned nothing: no machine, no host entry, no hub entry to +tear down. + +## 1. The N100's night + +- It removed 6 of its 8 old kernels by itself at 05:58 CET, in 35 seconds, and gained 6.81 GB. It kept 7.0.14-26 + and the running 7.0.14-28. +- The default kernel and the pinned file did not change. The box is healthy: 0 failed units, 9 of 9 containers, + clock +0.06 s to DooPlex. The hub's „Kernel step” cell says „cleaned”. +- D82 count: **the N100 1 of 2, demo-hp 0 of 2** (demo-hp had nothing to remove; a quiet night does not count). + Both boxes hold 2 kernels now, so the next count needs new kernels to arrive. + +Tester 1: the hub says „Crash guard: armed”. The line „crash guard not installed” is not there. No row. + +## 2. Ready to release + +| Part | Repo and commit | Proof still needed on a box | Who must say yes | +|---|---|---|---| +| **Hub**: tells the operator about a wrong box clock, a refused signed job, a skipped daily check, a lost operator mail, a guest-power give-up, a Proxmox trust problem, an operator token or a box web certificate that ends in 14 days; shows the kernel clean-up and the profile refresh; keeps the off-site account record after an untick; a deleted box or household leaves no rows | `felhom.eu` `d2683b28` (CI run 2666 green) | After the deploy: the System page shows the two new cells for the N100; one staged wrong clock on scratch 9202 raises one mail to the operator only; the daily sweep's first run is read in the log (it deletes rows of boxes deleted earlier) | The operator, present (a hub deploy is never unattended) | +| **Agent**: trusts Proxmox by a copy of the box's root CA and moves a pinned box by itself; stamps its clock at sending; tells the hub about a refused job and a give-up; sends the clean-up detail; the boot key | `felhom-agent` `54ac2c2` (CI run 2667 green) | demo-hp: the move from the pin to the CA copy (copy, proof, then the config), then one new Proxmox certificate with the agent still in control; a cut power at the config write is NOT proven (the low open item, R-1380) | The operator's vouch | +| **Controller**: a step of the host's clock is not a new boot; HSTS one year; the web certificate's end date in the report | `felhom-controller` `ee3b05e` (CI run 2665 green) | The three test boxes through the normal update path; the box walk and the https-only check green on all three; the release must be built on or after 2026-11-15 | The operator says „HSTS now”; then CC raises the floor in a session the operator attends | +| **Installer**: a new box trusts Proxmox by the CA copy, the pin is the fallback | `felhom.eu` `fe824766` + `f618c591`, **not tagged** | One fresh install (Tester4) that ends with no fingerprint in the agent's config and a working agent | The operator (a tag publishes to every new install) | + +Order that is safe: hub first (it must know the new fields), then agent, then controller, then the installer tag. + +## 3. What stopped at a design + +- **The off-site copies of a removed app end after 30 days (D72)** — `07-backup-architecture.md` §8.2f. Five parts, + 14 edge cases, the texts in Hungarian and English as a proposal. One decision is open (D84). +- **The installer below 200 GB (D75)** — `16-storage-physical-identity.md` §14.5. One decision is open (D83). +- **The other sixteen end dates (R-1379)** — a table in `08-alarm-ladder.md` §12.2. One is small and waits for a yes + (D85). +- **A fully blind box** (neither the pin nor the CA copy works) sends no report, so the hub cannot name the cause + (R-1396, gap 2). Not designed. + +## 4. Decisions + +**Taken by CC unattended — the operator may reverse.** The operator's own action log of a deleted household goes one +year after the deletion, with the other two audit logs. Options: at once, or after one year. At once loses „who did +what” on the day of the deletion; one year is the rule the design already has for the other audit rows (decision +181). Recorded in `05-hub-architecture.md` §14.2 and `CONTEXT.md`. + +**For the operator (in `STATUS.md`):** + +| # | Question | CC's pick | The other option | If nothing is decided | +|---|---|---|---|---| +| D83 | Below 200 GB of system disk: refuse, or warn? | Refuse, with one named flag for the operator | Warn: a small box installs and stops its local copies months later | The installer takes any disk | +| D84 | Do a removed app's off-site copies leave outside the weekly deletion cap? | Yes, their own allowance, only for copies the hub agreed to | They count: a refused window and a one-shot grant at each large removal | The design is not built; the copies stay for ever | +| D85 | Should the hub read the end dates of its own five public names daily? | Yes | No: the operator keeps the dates in a list | Nobody warns before a hub name ends | + +**A privacy sentence — a proposal, NOT published (the operator publishes or not):** + +- HU: „A fiók törlése után a rendszer egy évig megőrzi a műveleti naplót (ki, mikor, mit tett), ezután törli. A + törlés tényét — egy azonosítót és az időpontot — a rendszer határidő nélkül megőrzi.” +- EN: "After an account is deleted, the system keeps the action log (who did what, and when) for one year, then + deletes it. The fact of the deletion — an identifier and the time — is kept with no end date." +- **A caveat:** the design text says the delete journal holds „no personal data”. A customer id can look like a name + (`peti-felhom`), so that claim may not hold. The sentence above does not make it. + +## 5. Claims in the prompt that turned out wrong + +1. „A report carries a time stamped at sending, or can.” It did not: the time in a report was the start of + collecting. The agent now stamps `sent_at` at the moment of sending. +2. „The controller's boot identity.” The boot token is the agent's; the controller only compares it. The fix needed + both repos. +3. „2a's clean-up detail is shown nowhere.” It was shown, as a hover text. And the cell had a defect: a clean-up hid + a kernel that fell back the night before. Fixed. +4. „13 tables” keep rows after a delete. Seven name a box or a household. + +Two claims were true: a set fingerprint wins over a CA file (so the move clears it last), and the second off-site +tier keeps its record when switched off. + +## 6. Per part + +**Baselines at the start (2026-11-14, 16:30 CET):** hub 0.181.0, agent 0.174.0 (vouched), controller 0.352.1 (floor). +Evidence for every red-proof: `documentation/audits/night-build-2026-11-14/{A,B,C,review}/`. + +| Part | Commits | Tests and red-proofs | Document read and changed | Not proven | +|---|---|---|---|---| +| A — trust in Proxmox by the root CA copy (2b; R-1380) | agent `386a0c4`; installer `fe824766`; docs `d809a1d4` | wrapper, agent and installer suites; `A/redproof-*.txt` | `03-host-agent.md` §12a | on a box: nothing. The move, a renewed certificate, a cut write | +| A — the hub names the trust problem (R-1396 gap 1) | `26f4f766` | `B/redproof-hub-proxmox-trust.txt` | `08-alarm-ladder.md` | on a deployed hub | +| B1 — the box's clock (R-1384) | agent `ede5d0e`; hub `88e63644` | `B/redproof-agent-sent-at.txt`, `redproof-hub-box-clock.txt` | `08-alarm-ladder.md` | a real wrong clock | +| B2 — four silent failures (R-1383) | agent `a113501`; hub `18860307`, `186f3fca` | `B/redproof-hub-daily-ledger.txt`, `-signed-job`, `-lost-mail`, `-box-told`, `redproof-agent-job-outcome.txt`, `-guest-power` | `05-hub-architecture.md` §16.6, `08` ladder rows | a real hub restart with a mail in retry | +| B3 — the clean-up and the profile refresh are shown (R-1383) | agent `61feb95`; hub `68292e40`, `78c08dc9` | `B/redproof-agent-b3.txt`, `redproof-hub-b3.txt` | `11-os-updates.md` §5.11.1, §5.12 | the two cells with a real report (the N100's report of tonight is the first candidate) | +| B5 — the off-site account record (R-1375, D71); the runbook (D74) | `e5c643d7` | `B/redproof-hub-offsite-record.txt` | `05` §14.1; `runbooks/household-leaves.md` | a real account; the runbook as a whole | +| C — the boot key (R-1395); HSTS one year (R-1050) | controller `95ed631`, `fcf7347`, `fdc0e29`; agent `d5ce5c3`, `c0bf97d` | `C/redproof-*.txt` | controller README; `09` §3 decision 249 | a real clock step on a box; a browser with the one-year header | +| B4 — two end dates warn (R-1379, D77) | controller `7daf3d2`, `ee3b05e`; hub `86bb912a`, `0ce752a3` | `B/redproof-controller-webcert.txt`, `redproof-hub-end-dates.txt` | `08-alarm-ladder.md` §12 | a real certificate 14 days from its end | +| B6 — a delete leaves no rows (R-1376, D73) | `73d99d06` | `B/redproof-hub-d73.txt` | `05` §14.2 | on the live database (the first sweep deletes real left-over rows) | +| D — two designs | `a044eb8c` | none (documents) | `07` §8.2f, `16` §14.5 | all of it; not built | +| Review fixes (two independent reviews, and the security review of each commit) | agent `ec7e699`, `54ac2c2`; hub `f618c591`, `d2683b28`; controller `ee3b05e` | `review/redproof-*.txt` | `03` §12a, `05`, `08` | one fix (the installer under strict mode) answers a defect that was reasoned and NOT reproduced; the evidence says so | + +About 220 mutations were made in all; each was seen red by the test that guards it. A mutant that did not compile, or +that matched no place or two places, was redone and the evidence says so. + +**What the reviews found, in one line each:** a checker that tells a thing once lost the second thing's mail to the +one-hour limit (seven new checkers; fixed, one limit per thing); three writers of the agent's config had no shared +lock (fixed); a box could make the hub write words of the box's choice into a mail (fixed, the words are bounded); +the trust run could run twice at once (fixed). + +**CI, by commit hash:** `felhom.eu` `d2683b28` run 2666 success; `felhom-agent` `54ac2c2` run 2667 success; +`felhom-controller` `ee3b05e` run 2665 success. One push was refused by a gate before it left DooPlex (a memory index +file 85 bytes over its limit); the index was shortened and the same commits were pushed. + +**Instruction files changed:** none. +**System poster:** no fact on the poster changed (nothing is released); no edit, no refresh asked. +**Capability map:** no row moves. `unproven.py --summary`: 32 walked, 15 partial, 4 built, 4 missing, 23 not walked — +no number moved. + +## 7. The register + +250 rows at the end; 249 at the start of the night. Opened: 1 (the hub does not yet say why a box's agent does not +trust Proxmox; its first half was built the same night). Closed: 0 — every built part still needs its release and its +proof on a box, so each row is narrowed, not closed. + +Rows with a dated line tonight: R-1368, R-1380, R-1383, R-1384, R-1375, R-1376, R-1379, R-1395, R-1396, R-1050, +R-1374, R-1370. + +## 8. For the operator, not CC's to do + +- Hosts page: delete host `Tester4-46836f`. +- After the hub release, once: at the off-site provider read each sub-account's label against the hub's list (an + account whose record was lost before the fix is found only this way). +- A note for the release review: deleting a box now deletes its update settings too, so a box that is installed + again under the same id comes back as a normal box (ring 1), not as a test box. diff --git a/STATUS.md b/STATUS.md index 9b41a83e..64795538 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-11-14 (afternoon).** Release 2a is out: agent 0.174.0, vouched. A ring-0 box now removes its old kernels at night, and a box reloads its security profiles after a kernel change. The N100 keeps time again; the product repaired it, with no hand step. +**Updated 2026-11-15 (morning).** The N100 removed 6 of its 8 old kernels by itself at night and is healthy: 1 of the 2 nights D82 asks for; demo-hp had nothing to remove (0 of 2). Releases 2b and 3 are built and tested in source; nothing is released (`REPORT-night-build-2026-11-14.md`). ## What works - **The Plex attack did not reach Felhom's secrets** (R-1241). @@ -32,6 +32,9 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| +| D85 | R-1379 | Should the hub read the end dates of its own five public names each day and warn you 14 days before? | **Yes** (small; the same check that is built for a box's certificate; a hub name that ends takes every box's link down) | no: you keep the five dates in your own list | nobody warns you before a hub name ends | — | +| D84 | R-1374 | When a household removes an app with „delete backup data”, do that app's off-site copies leave outside the weekly deletion cap? | **Yes — their own allowance, only for copies the hub agreed to** (the hub's own 30-day check does the cap's job here; under the cap one big removal stops every clean-up for weeks) | they count: no new rule, but a refused window and your one-shot grant at each large removal | the design stays a design; the off-site copies of removed apps stay for ever | the build of D72 | +| D83 | R-1370 | Below 200 GB of system disk: does the installer refuse, or warn? | **Refuse, with one named flag for you** (a warning on an install that nobody watches is read by nobody; the flag keeps demo-hp possible) | warn: a 128 GB box installs and stops its local copies months later | the installer takes any disk; the 200 GB is only a sentence | the build of D75 | | D66 | R-1365 | Should a security-watch source that stays down mail again every 7 days, as an open notice does? Today it mails once, then only the Monday summary lists it. | **Yes** (a real outage then cannot be forgotten; the seven pins that can never be read no longer count, so it stays quiet when all is well) | no: one line in a long Monday mail stays the only reminder | a source that stays down mails once; the Monday list still shows it | — | | D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | | D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — | @@ -39,7 +42,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — | ## Waiting on you — do -**2026-11-15 morning — read:** the N100 removed 6 of its 8 kernels at night, the hub's „Kernel step” cell says „cleaned”, the box is healthy. If not: tell CC (R-1368). **2026-11-14.** Next builds: 2b (trust in Proxmox), „hub tells you”, promises; no new gap list first. Hosts page: delete host `Tester4-46836f` (else it alarms). Say „HSTS now” from 11-15 (R-1050). **2026-11-11.** 2. Hosts page: discard `3fb5fed3…`. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18). +**2026-11-15 morning.** 1. Read `REPORT-night-build-2026-11-14.md`, „Ready to release”: hub, agent, controller and installer wait for your yes. If not: nothing reaches a box. 2. Say „HSTS now” (R-1050). 3. Hosts page: delete host `Tester4-46836f` (else it alarms). 4. The report has a privacy sentence as a proposal; you publish it or not. **2026-11-11.** 2. Hosts page: discard `3fb5fed3…`. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18). **Still needed? (yes/no)** — answer per line. 1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16) diff --git a/documentation/audits/night-build-2026-11-14/E/README.md b/documentation/audits/night-build-2026-11-14/E/README.md new file mode 100644 index 00000000..8170cfb7 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/E/README.md @@ -0,0 +1,56 @@ +# Part E — the morning read after the first night a box removed kernels by itself (2026-11-15, 06:33 CET) + +Reads only. Nothing was written on a box: no service action, no package action, no guest started or stopped, no file +made. The three read scripts are in this folder as they were run (`partE-read.sh`, `partE-read2.sh`, +`partE-read3.sh`), each as `ssh 'bash -s' < script`. Tester 1 was not read on the box: the hub's page did not +show the line that would have asked for it. + +## The N100 (`demo-felhom-8363b5`) — a real removal, healthy + +| Fact | Before | After | Where | +|---|---|---|---| +| Kernel packages installed | 8 (the wrapper's own count at 05:58:01) | 2: 7.0.14-26 and 7.0.14-28 | `felhom-pve-3.txt`, `felhom-pve-2.txt` (dpkg state) | +| Removed | — | 6: 7.0.2-6, 7.0.14-20, -22, -23, -24, -25; purged, 0 left-over paths | `felhom-pve-2.txt` (dpkg log, apt history: requested by the agent's user) | +| Time | 05:58:01 | 05:58:35 (34.6 s) | the same | +| Space gained | — | 6,810,329,088 bytes (6.81 GB, 6.3 GiB) by the wrapper's figure; 65.10 GB free after | `felhom-pve-3.txt`, `felhom-pve.txt` | +| Running kernel | 7.0.14-28 | 7.0.14-28 (no reboot; up 1 week) | `felhom-pve.txt` | +| Default kernel | 7.0.14-28 | 7.0.14-28, „proved from grub.cfg” by the wrapper and read again from `grub.cfg` | both | +| The pinned file and the kernel state file | written 2026-11-08 05:36 | the same times | `felhom-pve.txt` | +| Health | — | 0 failed units; guest 9201 running, 9 containers, 0 unhealthy, 0 exited; dpkg audit empty | `felhom-pve.txt` | +| Time service | — | active, leap normal; the box is +0.064 s from DooPlex (round trip 0.139 s) | `felhom-pve.txt` | +| The hub's „Kernel step” cell | — | „cleaned 36 min ago”, the hover text names the 6 packages and the 2 kept | `hub-system-page-0634.txt` line 97 | + +The clean-up ran at the end of the night's update run, after the Proxmox step applied 12 packages. One line in the +journal reads „dd invoked oom-killer” at 05:57:20: that is the update run's own memory-kill check (it reports +`OOM-CHECK result=pass`), not a fault. + +## demo-hp (`demo-hp-bb76ea`) — a quiet night + +The night run at 05:02–05:04 found 2 kernels installed, 2 to keep, 0 to remove („nothing to remove”). The box's six +old kernels were purged on 2026-11-10 at 13:51, by day (apt history). Health: 0 failed units; guests 9201 (26 +containers) and 9202 (6 containers) healthy; clock +0.071 s from DooPlex. The hub's cell still shows its newest kernel +step („applied 5 days ago”), which is correct for a clean-up that removed nothing. + +## Tester 1 (`tester-1-d70be4`) — the hub's line + +„Crash guard: armed”; root files 0.174.0; kernel.panic 10 s; health „fine”. The line „crash guard not installed” is +not there, so no read on the box and no row. + +## The D82 count (R-1368) + +Two real night removals on each ring-0 box. **The N100: 1 of 2. demo-hp: 0 of 2.** Both boxes now hold 2 kernels and +keep 2, so the next real removal needs new kernels to arrive first (the hub shows 7.0.14-29 as due on both). + +## Two instrument defects in the first read, said plainly + +1. The first script's kernel count printed 0 on both boxes. Its pattern looked for the package name at the end of a + line that ends with the state. The second read lists the dpkg states directly. +2. The first and second scripts' journal filters printed nothing, and the positive control (a line count) read 0: + `--since "yesterday 20:00"` gave no lines over this ssh form, with the error hidden. The third read uses a full + date and shows 139,573 and 140,523 lines. + +## One answer to a question of the night's review + +The folder that holds the profile-refresh record is `drwx------ root`, so the agent's user cannot read the record +file (read on both boxes). The night's build takes the result from the root step's answer and treats the file as a +bonus, so this does not break it; the file read stays dead until the folder's mode changes. diff --git a/documentation/audits/night-build-2026-11-14/E/demo-hp-2.txt b/documentation/audits/night-build-2026-11-14/E/demo-hp-2.txt new file mode 100644 index 00000000..c16dc19d --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/E/demo-hp-2.txt @@ -0,0 +1,37 @@ +#### demo-hp 2026-11-15 06:33:42 CET +## kernel packages by dpkg state + ii |proxmox-kernel-7.0|7.0.14-28 + un |proxmox-kernel-7.0.14-26-pve| + ii |proxmox-kernel-7.0.14-26-pve-signed|7.0.14-26 + un |proxmox-kernel-7.0.14-28-pve| + ii |proxmox-kernel-7.0.14-28-pve-signed|7.0.14-28 + ii |proxmox-kernel-helper|9.1.0+fde2 +## dpkg log: kernel package removals and purges, last 3 days +## apt history: last 3 entries that name a kernel + +Start-Date: 2026-11-10 13:51:11 +Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef purge proxmox-kernel-7.0.2-6-pve-signed proxmox-kernel-7.0.14-20-pve-signed proxmox-kernel-7.0.14-22-pve-signed proxmox-kernel-7.0.14-23-pve-signed proxmox-kernel-7.0.14-24-pve-signed proxmox-kernel-7.0.14-25-pve-signed +Purge: proxmox-kernel-7.0.14-20-pve-signed:amd64 (7.0.14-20), proxmox-kernel-7.0.14-22-pve-signed:amd64 (7.0.14-22), proxmox-kernel-7.0.14-23-pve-signed:amd64 (7.0.14-23), proxmox-kernel-7.0.14-24-pve-signed:amd64 (7.0.14-24), proxmox-kernel-7.0.2-6-pve-signed:amd64 (7.0.2-6), proxmox-kernel-7.0.14-25-pve-signed:amd64 (7.0.14-25) +End-Date: 2026-11-10 13:52:00 + +Start-Date: 2026-11-14 05:00:45 +## agent journal since yesterday 20:00: every line with osupdate / kernel / clean +## agent journal line count since yesterday 20:00 (a positive control that the journal is readable) +0 +## the whole journal since yesterday 20:00: lines with felhom-os-apply or KERNEL +## wrapper state folder (names and times only) +total 24 +drwx------ 4 root root 4096 2026-11-14 12:33 . +drwxr-xr-x 46 root root 4096 2026-10-10 04:42 .. +drwxr-xr-x 2 root root 4096 2026-11-14 12:18 agent-update +drwxr-xr-x 5 root root 4096 2026-11-14 12:33 bundle-prev +-rw-r--r-- 1 root root 96 2026-11-14 12:33 nonces.json +-rw-r--r-- 1 root root 137 2026-11-14 12:33 profile-refresh.json +total 16 +drwxr-xr-x 2 root root 4096 2026-11-10 02:56 . +drwxr-xr-x 46 root root 4096 2026-10-10 04:42 .. +-rw-r--r-- 1 root root 5701 2026-11-10 02:56 state.json +## disk +21968302080 17318748160 45% +21968302080 17318748160 45% +#### end diff --git a/documentation/audits/night-build-2026-11-14/E/demo-hp-3.txt b/documentation/audits/night-build-2026-11-14/E/demo-hp-3.txt new file mode 100644 index 00000000..a6f7a969 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/E/demo-hp-3.txt @@ -0,0 +1,50 @@ +#### demo-hp 2026-11-15 06:34:01 CET +## control: agent journal lines since 2026-11-14 20:00 +140523 +Nov 14 20:00:01 demo-hp sudo[3471829]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +Nov 14 20:00:01 demo-hp sudo[3471829]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +## agent journal: osupdate / kernel / clean-up lines +2026-11-15T05:02:48+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:02:48.557+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z +2026-11-15T05:03:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:02.991+01:00 level=INFO msg="osupdate: a pass is running — its start sends any kept report" +2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0" +2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.209+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=19.5 +2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.303+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z +2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=host lane=fast mode=apply select=pending-fast packages=0" +2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-15T05:03:26+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:26.644+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=30 not_covered=30 restart_needed=1 reboot_needed=false wrapper_seconds=17.1 +2026-11-15T05:03:29+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:29.145+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on" +2026-11-15T05:03:29+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:29.145+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}" +2026-11-15T05:03:29+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:29.145+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z +2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0" +2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.352.1 — the engine reported the memory kill: OOMKilled=true and the oom event" +2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.416+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=30.2 +2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.423+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261115T040248Z layer=pve vmid=9201 trigger=night +2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.423+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z +2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0" +2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0" +2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=25.4 upgraded=12 restart-needed=kvm reboot-needed=no" +2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.480+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=pve vmid=9201 ring=0 trigger=night outcome=applied healthy=true reason="" upgraded=12 pending=2 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=46.8 +2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.509+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261115T040248Z layer=pve vmid=9201 trigger=night +2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.509+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261115T040248Z layer=kernel vmid=9201 trigger=night ring=0 +2026-11-15T05:04:47+01:00 demo-hp sudo[1134694]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T040248Z-kernel-kernel-cleanup.json +2026-11-15T05:04:47+01:00 demo-hp felhom-os-apply[1134710]: os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[] +2026-11-15T05:04:47+01:00 demo-hp felhom-os-apply[1134711]: os-apply: KERNEL CLEANUP nothing to remove +2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[]" +2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP nothing to remove" +2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: kernel clean-up — nothing to remove" run=20261115T040248Z layer=kernel mode=kernel-cleanup vmid=9201 trigger=night kept=7.0.14-26-pve,7.0.14-28-pve took=0s +## whole journal 05:30-06:30: wrapper and sudo lines +2026-11-15T05:33:13+01:00 demo-hp sudo[1229107]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts043312-host-facts.json +2026-11-15T05:48:13+01:00 demo-hp sudo[1279117]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts044813-host-facts.json +2026-11-15T06:03:13+01:00 demo-hp sudo[1329378]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts050313-host-facts.json +2026-11-15T06:18:13+01:00 demo-hp sudo[1379432]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts051813-host-facts.json +#### end diff --git a/documentation/audits/night-build-2026-11-14/E/demo-hp.txt b/documentation/audits/night-build-2026-11-14/E/demo-hp.txt new file mode 100644 index 00000000..09255064 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/E/demo-hp.txt @@ -0,0 +1,33 @@ +#### demo-hp 2026-11-15T05:33:09Z local 2026-11-15 06:33:09 CET epoch 1794720789.075455874 +agent: felhom-agent 0.174.0; uptime: up 4 days, 14 hours, 50 minutes; running kernel: 7.0.14-28-pve +## kernels installed (ii) +count: 0 +/boot: 7.0.14-26-pve 7.0.14-28-pve +/lib/modules: 7.0.14-26-pve 7.0.14-28-pve +## default kernel and the pinned file +grub.cfg default: gnulinux-7.0.14-28-pve-advanced + 6b35d9aeddc1ddd0 2026-11-10 02:56:43 /etc/default/grub.d/zz-felhom-kernel-default.cfg + 607a2ce7ce3b6ca1 2026-11-10 02:56:47 /var/lib/felhom-kernel/state.json + f2f56d59c6356067 2026-10-07 18:01:30 /etc/grub.d/01_felhom_oneshot + c34907ce68599873 2026-10-07 18:01:30 /etc/grub.d/42_felhom_oneshot +pinned file content: # felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-a +kernel state: { "authority": "ring0", "default_before": "7.0.14-26-pve", "from": "7.0.14-26-pve", "health_before": { "guest": { "containers": { "adventurelog": { "health": "healthy", "id": "50af393c36a37d01a10847456d391d70572c121045e28626f220f83cec1cefcc", "state": "running" }, "adventurelog-frontend": { "health": "healthy", "id": "ab +## disk + / free 21.97 GB, used 45% +## the night's lines: the agent's OS leg and the root wrapper (since yesterday 22:00 local) +## the wrapper's own journal lines about the clean-up +## health +failed units: 0 + guest 9201: running; containers 26 running, 0 unhealthy, 0 exited + guest 9202: running; containers 6 running, 0 unhealthy, 0 exited + guest 9401: stopped +dpkg audit: [] +## time service +chrony: active; Reference ID : C1292F54 (ntp1.iszt.hu);System time : 0.000095636 seconds slow of NTP time;Leap status : Normal; +## the profile-refresh record's folder (a question of the night's review: can the agent read it?) +drwx------ 4 root root 4096 Nov 14 12:33 /var/lib/felhom-os-apply +-rw-r--r-- 1 root root 137 Nov 14 12:33 /var/lib/felhom-os-apply/profile-refresh.json +can the agent user read it: no +#### end demo-hp epoch 1794720799.360448067 +#### DooPlex: ssh rc=0, started epoch 1794720788.467514838, ended epoch 1794720799.365384080 +#### clock: box minus DooPlex (middle of the round trip) = +0.071 s; round trip 0.151 s diff --git a/documentation/audits/night-build-2026-11-14/E/felhom-pve-2.txt b/documentation/audits/night-build-2026-11-14/E/felhom-pve-2.txt new file mode 100644 index 00000000..142c175c --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/E/felhom-pve-2.txt @@ -0,0 +1,65 @@ +#### demo-felhom 2026-11-15 06:33:41 CET +## kernel packages by dpkg state + ii |proxmox-kernel-7.0|7.0.14-28 + un |proxmox-kernel-7.0.14-26-pve| + ii |proxmox-kernel-7.0.14-26-pve-signed|7.0.14-26 + un |proxmox-kernel-7.0.14-28-pve| + ii |proxmox-kernel-7.0.14-28-pve-signed|7.0.14-28 + ii |proxmox-kernel-helper|9.1.0+fde2 +## dpkg log: kernel package removals and purges, last 3 days +2026-11-15 05:58:01 remove proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 +2026-11-15 05:58:04 status config-files proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 +2026-11-15 05:58:04 remove proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22 +2026-11-15 05:58:07 status config-files proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22 +2026-11-15 05:58:07 remove proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23 +2026-11-15 05:58:10 status config-files proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23 +2026-11-15 05:58:10 remove proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24 +2026-11-15 05:58:13 status config-files proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24 +2026-11-15 05:58:13 remove proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25 +2026-11-15 05:58:16 status config-files proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25 +2026-11-15 05:58:16 remove proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 +2026-11-15 05:58:19 status config-files proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 +2026-11-15 05:58:19 purge proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22 +2026-11-15 05:58:19 status config-files proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22 +2026-11-15 05:58:22 status not-installed proxmox-kernel-7.0.14-22-pve-signed:amd64 +2026-11-15 05:58:22 purge proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 +2026-11-15 05:58:22 status config-files proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 +2026-11-15 05:58:24 status not-installed proxmox-kernel-7.0.14-20-pve-signed:amd64 +2026-11-15 05:58:24 purge proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 +2026-11-15 05:58:24 status config-files proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 +2026-11-15 05:58:27 status not-installed proxmox-kernel-7.0.2-6-pve-signed:amd64 +2026-11-15 05:58:27 purge proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25 +2026-11-15 05:58:27 status config-files proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25 +2026-11-15 05:58:30 status not-installed proxmox-kernel-7.0.14-25-pve-signed:amd64 +2026-11-15 05:58:30 purge proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24 +2026-11-15 05:58:30 status config-files proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24 +2026-11-15 05:58:32 status not-installed proxmox-kernel-7.0.14-24-pve-signed:amd64 +2026-11-15 05:58:32 purge proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23 +2026-11-15 05:58:32 status config-files proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23 +2026-11-15 05:58:35 status not-installed proxmox-kernel-7.0.14-23-pve-signed:amd64 +## apt history: last 3 entries that name a kernel +Start-Date: 2026-11-15 05:58:01 +Commandline: apt-get -y -q purge proxmox-kernel-7.0.14-20-pve-signed proxmox-kernel-7.0.14-22-pve-signed proxmox-kernel-7.0.14-23-pve-signed proxmox-kernel-7.0.14-24-pve-signed proxmox-kernel-7.0.14-25-pve-signed proxmox-kernel-7.0.2-6-pve-signed +Requested-By: felhom-agent (999) +Purge: proxmox-kernel-7.0.14-20-pve-signed:amd64 (7.0.14-20), proxmox-kernel-7.0.14-22-pve-signed:amd64 (7.0.14-22), proxmox-kernel-7.0.14-23-pve-signed:amd64 (7.0.14-23), proxmox-kernel-7.0.14-24-pve-signed:amd64 (7.0.14-24), proxmox-kernel-7.0.2-6-pve-signed:amd64 (7.0.2-6), proxmox-kernel-7.0.14-25-pve-signed:amd64 (7.0.14-25) +End-Date: 2026-11-15 05:58:35 +## agent journal since yesterday 20:00: every line with osupdate / kernel / clean +## agent journal line count since yesterday 20:00 (a positive control that the journal is readable) +0 +## the whole journal since yesterday 20:00: lines with felhom-os-apply or KERNEL +## wrapper state folder (names and times only) +total 24 +drwx------ 4 root root 4096 2026-11-14 12:51 . +drwxr-xr-x 48 root root 4096 2026-10-08 04:39 .. +drwxr-xr-x 2 root root 4096 2026-11-14 12:35 agent-update +drwxr-xr-x 5 root root 4096 2026-11-14 12:51 bundle-prev +-rw-r--r-- 1 root root 96 2026-11-14 12:51 nonces.json +-rw-r--r-- 1 root root 196 2026-11-14 12:51 profile-refresh.json +total 12 +drwxr-xr-x 2 root root 4096 2026-11-08 05:36 . +drwxr-xr-x 48 root root 4096 2026-10-08 04:39 .. +-rw-r--r-- 1 root root 2615 2026-11-08 05:36 state.json +## disk +65095643136 30595346432 32% +65095643136 30595346432 32% +#### end diff --git a/documentation/audits/night-build-2026-11-14/E/felhom-pve-3.txt b/documentation/audits/night-build-2026-11-14/E/felhom-pve-3.txt new file mode 100644 index 00000000..76e9e57e --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/E/felhom-pve-3.txt @@ -0,0 +1,76 @@ +#### demo-felhom 2026-11-15 06:33:58 CET +## control: agent journal lines since 2026-11-14 20:00 +139573 +Nov 14 20:00:00 demo-felhom sudo[2436686]: pam_unix(sudo:session): session closed for user root +Nov 14 20:00:00 demo-felhom sudo[2436691]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +## agent journal: osupdate / kernel / clean-up lines +2026-11-15T05:56:34+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:34.495+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z +2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.839+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0" +2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.839+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.840+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.840+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.840+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=14.3 +2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.901+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z +2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=host lane=fast mode=apply select=pending-fast packages=0" +2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.867+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=29 not_covered=29 restart_needed=0 reboot_needed=false wrapper_seconds=11.1 +2026-11-15T05:57:02+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:02.546+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on" +2026-11-15T05:57:02+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:02.546+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}" +2026-11-15T05:57:02+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:02.546+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z +2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0" +2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.352.1 — the engine reported the memory kill: OOMKilled=true and the oom event" +2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.218+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=22.6 +2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.225+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261115T045634Z layer=pve vmid=9201 trigger=night +2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.225+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z +2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0" +2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0" +2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=20.1 upgraded=12 restart-needed=- reboot-needed=no" +2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.998+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=pve vmid=9201 ring=0 trigger=night outcome=applied healthy=true reason="" upgraded=12 pending=1 not_covered=0 restart_needed=0 reboot_needed=false wrapper_seconds=33.9 +2026-11-15T05:58:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:00.007+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261115T045634Z layer=pve vmid=9201 trigger=night +2026-11-15T05:58:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:00.007+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261115T045634Z layer=kernel vmid=9201 trigger=night ring=0 +2026-11-15T05:58:00+01:00 demo-felhom sudo[3123080]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-kernel-kernel-cleanup.json +2026-11-15T05:58:01+01:00 demo-felhom felhom-os-apply[3123143]: os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[] +2026-11-15T05:58:35+01:00 demo-felhom felhom-os-apply[3132623]: os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve (proved from grub.cfg) +2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0" +2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]" +2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve (proved from grub.cfg)" +2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: kernel clean-up — old kernels removed" run=20261115T045634Z layer=kernel mode=kernel-cleanup vmid=9201 trigger=night removed=6 names=proxmox-kernel-7.0.14-20-pve-signed,proxmox-kernel-7.0.14-22-pve-signed,proxmox-kernel-7.0.14-23-pve-signed,proxmox-kernel-7.0.14-24-pve-signed,proxmox-kernel-7.0.14-25-pve-signed,proxmox-kernel-7.0.2-6-pve-signed kept=7.0.14-26-pve,7.0.14-28-pve bytes_gained=0x118111de4540 leftovers=0 took=36s +2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=kernel mode=kernel-cleanup vmid=9201 trigger=night outcome=cleaned healthy=true reason="removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)" upgraded=0 pending=0 not_covered +## whole journal 05:30-06:30: wrapper and sudo lines +2026-11-15T05:56:39+01:00 demo-felhom felhom-os-apply[3119297]: os-apply: REPAIR configured=0 journal=0 fixed=0 +2026-11-15T05:56:43+01:00 demo-felhom felhom-os-apply[3119391]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0 +2026-11-15T05:56:43+01:00 demo-felhom felhom-os-apply[3119392]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +2026-11-15T05:56:48+01:00 demo-felhom sudo[3119650]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-host-apply.json +2026-11-15T05:56:49+01:00 demo-felhom felhom-os-apply[3119656]: os-apply: START release=ring0-20261115T045634Z layer=host lane=fast mode=apply select=pending-fast packages=0 +2026-11-15T05:56:52+01:00 demo-felhom felhom-os-apply[3119745]: os-apply: REPAIR configured=0 journal=0 fixed=0 +2026-11-15T05:56:54+01:00 demo-felhom felhom-os-apply[3119801]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0 +2026-11-15T05:56:54+01:00 demo-felhom felhom-os-apply[3119802]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +2026-11-15T05:57:00+01:00 demo-felhom sudo[3120431]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-guest-live-restore-on.json +2026-11-15T05:57:02+01:00 demo-felhom felhom-os-apply[3120495]: os-apply: LIVE-RESTORE already on +2026-11-15T05:57:02+01:00 demo-felhom sudo[3120496]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-docker-apply.json +2026-11-15T05:57:04+01:00 demo-felhom felhom-os-apply[3120626]: os-apply: START release=ring0-20261115T045634Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0 +2026-11-15T05:57:08+01:00 demo-felhom felhom-os-apply[3120807]: os-apply: REPAIR configured=0 journal=0 fixed=0 +2026-11-15T05:57:11+01:00 demo-felhom felhom-os-apply[3120839]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0 +2026-11-15T05:57:11+01:00 demo-felhom felhom-os-apply[3120840]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +2026-11-15T05:57:20+01:00 demo-felhom kernel: dd invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0 +2026-11-15T05:57:25+01:00 demo-felhom felhom-os-apply[3121371]: os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.352.1 — the engine reported the memory kill: OOMKilled=true and the oom event +2026-11-15T05:57:25+01:00 demo-felhom sudo[3121372]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-pve-apply.json +2026-11-15T05:57:26+01:00 demo-felhom felhom-os-apply[3121379]: os-apply: START release=ring0-20261115T045634Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0 +2026-11-15T05:57:29+01:00 demo-felhom felhom-os-apply[3121477]: os-apply: REPAIR configured=0 journal=0 fixed=0 +2026-11-15T05:57:31+01:00 demo-felhom felhom-os-apply[3121582]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0 +2026-11-15T05:57:53+01:00 demo-felhom felhom-os-apply[3122820]: os-apply: DONE rc=0 seconds=20.1 upgraded=12 restart-needed=- reboot-needed=no +2026-11-15T05:58:00+01:00 demo-felhom sudo[3123080]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-kernel-kernel-cleanup.json +2026-11-15T05:58:00+01:00 demo-felhom felhom-os-apply[3123087]: os-apply: REPAIR configured=0 journal=0 fixed=0 +2026-11-15T05:58:01+01:00 demo-felhom felhom-os-apply[3123143]: os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[] +2026-11-15T05:58:35+01:00 demo-felhom felhom-os-apply[3132623]: os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve (proved from grub.cfg) +2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]" +2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve +2026-11-15T06:06:42+01:00 demo-felhom sudo[3142189]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts050641-host-facts.json +2026-11-15T06:21:42+01:00 demo-felhom sudo[3159228]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts052142-host-facts.json +#### end diff --git a/documentation/audits/night-build-2026-11-14/E/felhom-pve.txt b/documentation/audits/night-build-2026-11-14/E/felhom-pve.txt new file mode 100644 index 00000000..f4824c5b --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/E/felhom-pve.txt @@ -0,0 +1,31 @@ +#### demo-felhom 2026-11-15T05:33:04Z local 2026-11-15 06:33:04 CET epoch 1794720784.224381391 +agent: felhom-agent 0.174.0; uptime: up 1 week, 57 minutes; running kernel: 7.0.14-28-pve +## kernels installed (ii) +count: 0 +/boot: 7.0.14-26-pve 7.0.14-28-pve +/lib/modules: 7.0.14-26-pve 7.0.14-28-pve +## default kernel and the pinned file +grub.cfg default: gnulinux-7.0.14-28-pve-advanced + 513c1822cc58fc81 2026-11-08 05:36:39 /etc/default/grub.d/zz-felhom-kernel-default.cfg + 4301585f3b987114 2026-11-08 05:36:42 /var/lib/felhom-kernel/state.json + f2f56d59c6356067 2026-10-07 17:57:10 /etc/grub.d/01_felhom_oneshot + c34907ce68599873 2026-10-07 17:57:10 /etc/grub.d/42_felhom_oneshot +pinned file content: # felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily GRUB_DEFAULT="gnulinux-advanced-1af1fcc6-639c-416b-a7e5-c4470d41a502>gnulinux-7.0.14-28-pve-a +kernel state: { "authority": "ring0", "default_before": "7.0.14-26-pve", "from": "7.0.14-26-pve", "health_before": { "guest": { "containers": { "cloudflared": { "health": "healthy", "id": "372fb8861b8465781dd368079d5e49753e4fe9435f062efa363bf5dd0f60e493", "state": "running" }, "felhom-controller": { "health": "healthy", "id": "13db198 +## disk + / free 65.10 GB, used 32% +## the night's lines: the agent's OS leg and the root wrapper (since yesterday 22:00 local) +## the wrapper's own journal lines about the clean-up +## health +failed units: 0 + guest 9201: running; containers 9 running, 0 unhealthy, 0 exited +dpkg audit: [] +## time service +chrony: active; Reference ID : C2255A28 (h90-40.pool194-37.dyn.tolna.net);System time : 0.000012041 seconds slow of NTP time;Leap status : Normal; +## the profile-refresh record's folder (a question of the night's review: can the agent read it?) +drwx------ 4 root root 4096 Nov 14 12:51 /var/lib/felhom-os-apply +-rw-r--r-- 1 root root 196 Nov 14 12:51 /var/lib/felhom-os-apply/profile-refresh.json +can the agent user read it: no +#### end demo-felhom epoch 1794720788.294832561 +#### DooPlex: ssh rc=0, started epoch 1794720783.844141907, ended epoch 1794720788.299159339 +#### clock: box minus DooPlex (middle of the round trip) = +0.064 s; round trip 0.139 s diff --git a/documentation/audits/night-build-2026-11-14/E/hub-system-page-0634.txt b/documentation/audits/night-build-2026-11-14/E/hub-system-page-0634.txt new file mode 100644 index 00000000..4fa7d6ab --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/E/hub-system-page-0634.txt @@ -0,0 +1,248 @@ +System — Felhom Hub +Felhom Hub +Dashboard +Customers +Apps +Hosts +System +Offsite +Configuration +System — versions and OS updates +Needs attention +One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3). +[hover: worth a look] look +Tester4-46836f Tester4 +[hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] kernel step: — · the host needs a restart since 2026-11-14 · [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] root files behind the vouched agent's · [hover: 1 minor release behind — sign an agent_update for this box] agent behind: 0.173.0 → 0.174.0 (since 2026-11-14) · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: default kernel +Waiting for you +Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes. +Docker engine 29.9.3-1 +Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed. +first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job. +Approve Docker set +Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set +Boxes +Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess. +Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night +Tester4-46836f Tester4 +[hover: Ring 1: a normal box — it installs only approved releases] 1 +[hover: worth a look] look +0.352.1 [hover: 1 minor release behind — sign an agent_update for this box] agent 0.173.0 → 0.174.0 (since 2026-11-14) +[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind +7.0.2-6-pve +[hover: 19 h ago · applied · 93 s — last successful leg: 19 h ago] ok · 19 h ago +Ring and updates +Ring ring 1 +→ demo +OS updates updates ON +switch off +Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up +Controller 0.352.1 +Agent [hover: 1 minor release behind — sign an agent_update for this box] 0.173.0 → 0.174.0 (since 2026-11-14) +Last OS leg [hover: last successful leg: 19 h ago] 19 h ago · applied · 93 s +Docker engine +Docker 29.9.2 +containerd 2.4.2-3~debian.13~trixie +live-restore on +Docker release — +Host +Proxmox 9.2.2 +Kernel (running) 7.0.2-6-pve +Kernel (next boot) 7.0.2-6-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] — +Debian 13.7 +Felhom release os-host-20261111-034654 +Pending 100 +Not covered 0 +Held none +Reboot needed since 2026-11-14 +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 +Guest +Guest Debian 13.7 +Felhom release os-guest-20261111-034654 +Pending 6 +Restart needed 7 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 19 h ago · 78.4 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet +demo-felhom-8363b5 Demo Ügyfél +[hover: Ring 0: a demo box — it installs every new fix first] 0 +[hover: nothing amber or red] fine +0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind +7.0.14-28-pve +[hover: 37 min ago · nothing · 23 s — last successful leg: 37 min ago] ok · 37 min ago +Ring and updates +Ring ring 0 +→ normal +OS updates updates ON +switch off +Tunnel running +Controller 0.352.1 +Agent [hover: current (vouched 0.174.0)] 0.174.0 +Last OS leg [hover: last successful leg: 37 min ago] 37 min ago · nothing · 23 s +Docker engine +Docker 29.9.3 +containerd 2.4.2-4~debian.13~trixie +live-restore on +Docker release ring0-20261115T045634Z +Host +Proxmox 9.2.27 +Kernel (running) 7.0.14-28-pve +Kernel (next boot) 7.0.14-28-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 36 min ago · due 7.0.14-29-pve — not told yet (mails 09–20 h) · phase good +Debian 13.7 +Felhom release ring0-20261115T045634Z +Pending 29 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0 +Guest +Guest Debian 13.7 +Felhom release ring0-20261115T045634Z +Pending 0 +Restart needed 1 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 272.7 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 4 days ago +demo-hp-bb76ea Demo HP +[hover: Ring 0: a demo box — it installs every new fix first] 0 +[hover: nothing amber or red] fine +0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind +7.0.14-28-pve +[hover: 1 h ago · nothing · 30 s — last successful leg: 1 h ago] ok · 1 h ago +Ring and updates +Ring ring 0 +→ normal +OS updates updates ON +switch off +Tunnel running +Controller 0.352.1 +Agent [hover: current (vouched 0.174.0)] 0.174.0 +Last OS leg [hover: last successful leg: 1 h ago] 1 h ago · nothing · 30 s +Docker engine +Docker 29.9.3 +containerd 2.4.2-4~debian.13~trixie +live-restore on +Docker release ring0-20261115T040248Z +Host +Proxmox 9.2.27 +Kernel (running) 7.0.14-28-pve +Kernel (next boot) 7.0.14-28-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — not told yet (mails 09–20 h) · phase good +Debian 13.7 +Felhom release ring0-20261115T040248Z +Pending 30 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0 +Guest +Guest Debian 13.7 +Felhom release ring0-20261115T040248Z +Pending 0 +Restart needed 1 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 70.5 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 4 days ago +tester-1-d70be4 Tester 1 +[hover: Ring 1: a normal box — it installs only approved releases] 1 +[hover: nothing amber or red] fine +0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0 +[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind +7.0.14-22-pve +[hover: 39 min ago · nothing · 16 s — last successful leg: 39 min ago] ok · 39 min ago +Ring and updates +Ring ring 1 +→ demo +OS updates updates ON +switch off +Tunnel running +Controller 0.352.1 +Agent [hover: current (vouched 0.174.0)] 0.174.0 +Last OS leg [hover: last successful leg: 39 min ago] 39 min ago · nothing · 16 s +Docker engine +Docker 29.8.2 +containerd 2.3.6-1~debian.13~trixie +live-restore on +Docker release — +Host +Proxmox 9.2.2 +Kernel (running) 7.0.14-22-pve +Kernel (next boot) 7.0.14-22-pve +[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve +[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good +Debian 13.7 +Felhom release os-host-20261111-034654 +Pending 101 +Not covered 0 +Held none +Reboot needed no +kernel.panic 10 s +Oops no +Crash restarts 24 h 0 +Crash guard armed +[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0 +Guest +Guest Debian 13.7 +Felhom release os-guest-20261111-034654 +Pending 6 +Restart needed 0 +[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 90.4 GiB +[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago +Details +Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files. +Approved releases +The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber. +guest : os-guest-20261111-034654 +272 packages · 2026-11-11 03:46 UTC · by auto +host : os-host-20261115-045718 +607 packages · 2026-11-15 04:57 UTC · by auto +docker : os-docker-20261026-190351 +6 packages · 2026-10-26 19:03 UTC · by operator +pve : os-pve-20261114-075712 +121 packages · 2026-11-14 07:57 UTC · by operator +kernel : os-kernel-20261114-075716 +2 packages · 2026-11-14 07:57 UTC · by operator +Approve now (guest + host) +An urgent fix only — normally the hub approves after 24 h and one night. +What ring 0 runs now +guest : +272 packages, first seen 2026-11-10 01:49 UTC +approved as os-guest-20261111-034654 +host : +607 packages, first seen 2026-11-14 04:52 UTC +approved as os-host-20261115-045718 +docker : +6 packages, first seen 2026-11-13 04:49 UTC +ready — see "Waiting for you" +pve : +121 packages, first seen 2026-11-15 04:58 UTC +demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set +kernel : +2 packages, first seen 2026-11-10 01:57 UTC +approved as os-kernel-20261114-075716 +Version floors +Global controller floor: 0.352.1 · vouched agent: 0.174.0 +[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent? +No per-customer floors: every box follows the global floor. +Crash guard and root files +Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦ +Tester4-46836f ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 ¦ +demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦ +demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦ +tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦ +Felhom Hub 0.181.0 diff --git a/documentation/audits/night-build-2026-11-14/E/partE-read.sh b/documentation/audits/night-build-2026-11-14/E/partE-read.sh new file mode 100644 index 00000000..77f28d87 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/E/partE-read.sh @@ -0,0 +1,38 @@ +# Part E — READS ONLY on a test box, the morning after the first night a box may remove kernels by itself. +# Nothing here writes: no systemctl action, no apt, no pct start/stop, no file made on the box. +set -u +echo "#### $(hostname) $(date -u +%FT%TZ) local $(date +%F\ %T\ %Z) epoch $(date +%s.%N)" +echo "agent: $(felhom-agent --version 2>&1 | head -1); uptime: $(uptime -p); running kernel: $(uname -r)" +echo "## kernels installed (ii)" +dpkg-query -W -f '${Package} ${db:Status-Abbrev}\n' 'proxmox-kernel-*' 2>/dev/null | awk '$2=="ii" && /pve(-signed)?$/ {print " " $1}' +echo "count: $(dpkg-query -W -f '${Package} ${db:Status-Abbrev}\n' 'proxmox-kernel-*' 2>/dev/null | awk '$2=="ii" && /pve(-signed)?$/' | wc -l)" +echo "/boot: $(ls /boot/vmlinuz-* 2>/dev/null | sed 's#/boot/vmlinuz-##' | tr '\n' ' ')" +echo "/lib/modules: $(ls /lib/modules | tr '\n' ' ')" +echo "## default kernel and the pinned file" +echo "grub.cfg default: $(grep -m1 'set default="gnu' /boot/grub/grub.cfg | grep -o 'gnulinux-[0-9][^"]*-pve-advanced' | head -1)" +for f in /etc/default/grub.d/zz-felhom-kernel-default.cfg /var/lib/felhom-kernel/state.json /etc/grub.d/01_felhom_oneshot /etc/grub.d/42_felhom_oneshot; do + [ -e "$f" ] && echo " $(sha256sum "$f" | cut -c1-16) $(stat -c '%y' "$f" | cut -c1-19) $f" || echo " (absent) $f"; done +echo "pinned file content: $(tr '\n' ' ' < /etc/default/grub.d/zz-felhom-kernel-default.cfg 2>/dev/null | cut -c1-200)" +echo "kernel state: $(tr -d '\n' < /var/lib/felhom-kernel/state.json 2>/dev/null | cut -c1-400)" +echo "## disk" +df -B1 --output=avail,pcent / | tail -1 | awk '{printf " / free %.2f GB, used %s\n", $1/1e9, $2}' +echo "## the night's lines: the agent's OS leg and the root wrapper (since yesterday 22:00 local)" +journalctl -u felhom-agent --since "yesterday 22:00" --no-pager -o short-iso 2>/dev/null | grep -E "kernel clean-up|osupdate: (pass|leg|night|kernel|host step|the night)|osupdate: wrapper.*(KERNEL CLEANUP|HOST|KERNEL)|whole-guest backup (done|finished|FAILED)|os leg" | cut -c1-420 | tail -40 +echo "## the wrapper's own journal lines about the clean-up" +journalctl -t felhom-os-apply --since "yesterday 22:00" --no-pager -o short-iso 2>/dev/null | grep -E "KERNEL CLEANUP|KERNEL |PROFILE REFRESH|REFUSED|FAILED" | cut -c1-420 | tail -20 +echo "## health" +echo "failed units: $(systemctl --failed --no-legend 2>/dev/null | wc -l) $(systemctl --failed --no-legend 2>/dev/null | awk '{print $1}' | tr '\n' ' ')" +for id in $(pct list 2>/dev/null | awk 'NR>1 {print $1}'); do + st=$(pct status $id 2>/dev/null | awk '{print $2}') + if [ "$st" = running ]; then + echo " guest $id: running; containers $(pct exec $id -- docker ps -q 2>/dev/null | wc -l) running, $(pct exec $id -- docker ps --filter health=unhealthy -q 2>/dev/null | wc -l) unhealthy, $(pct exec $id -- docker ps -a --filter status=exited -q 2>/dev/null | wc -l) exited" + else echo " guest $id: $st"; fi +done +echo "dpkg audit: [$(dpkg --audit 2>/dev/null | head -1)]" +echo "## time service" +echo "chrony: $(systemctl is-active chrony 2>&1); $(chronyc tracking 2>/dev/null | grep -E 'System time|Leap status|Reference ID' | tr '\n' ';' | tr -s ' ')" +echo "## the profile-refresh record's folder (a question of the night's review: can the agent read it?)" +ls -ld /var/lib/felhom-os-apply 2>&1 | cut -c1-120 +ls -l /var/lib/felhom-os-apply/profile-refresh.json 2>&1 | cut -c1-140 +echo "can the agent user read it: $(sudo -n -u felhom-agent test -r /var/lib/felhom-os-apply/profile-refresh.json 2>/dev/null && echo yes || echo no)" +echo "#### end $(hostname) epoch $(date +%s.%N)" diff --git a/documentation/audits/night-build-2026-11-14/E/partE-read2.sh b/documentation/audits/night-build-2026-11-14/E/partE-read2.sh new file mode 100644 index 00000000..1a80ef55 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/E/partE-read2.sh @@ -0,0 +1,23 @@ +# Part E, second read — READS ONLY. The first script's kernel count and journal filters matched nothing +# (the count's pattern was wrong; the journal words were guessed). This one reads wider. +set -u +echo "#### $(hostname) $(date +%F\ %T\ %Z)" +echo "## kernel packages by dpkg state" +dpkg-query -W -f '${db:Status-Abbrev}|${Package}|${Version}\n' 'proxmox-kernel-*' 2>/dev/null | sed 's/^/ /' +echo "## dpkg log: kernel package removals and purges, last 3 days" +grep -hE " (remove|purge) proxmox-kernel| status (not-installed|config-files) proxmox-kernel" /var/log/dpkg.log 2>/dev/null | grep -E "^2026-11-1[345]" | cut -c1-160 | tail -40 +echo "## apt history: last 3 entries that name a kernel" +grep -B3 -A3 -E "^(Remove|Purge):.*proxmox-kernel" /var/log/apt/history.log 2>/dev/null | cut -c1-600 | tail -30 +echo "## agent journal since yesterday 20:00: every line with osupdate / kernel / clean" +journalctl -u felhom-agent --since "yesterday 20:00" --no-pager -o short-iso 2>/dev/null | grep -iE "osupdate|kernel|clean-?up|cleanup" | cut -c1-500 | tail -60 +echo "## agent journal line count since yesterday 20:00 (a positive control that the journal is readable)" +journalctl -u felhom-agent --since "yesterday 20:00" --no-pager 2>/dev/null | wc -l +echo "## the whole journal since yesterday 20:00: lines with felhom-os-apply or KERNEL" +journalctl --since "yesterday 20:00" --no-pager -o short-iso 2>/dev/null | grep -E "felhom-os-apply|KERNEL" | cut -c1-500 | tail -40 +echo "## wrapper state folder (names and times only)" +ls -la --time-style=long-iso /var/lib/felhom-os-apply/ 2>&1 | cut -c1-160 +ls -la --time-style=long-iso /var/lib/felhom-kernel/ 2>&1 | cut -c1-160 +echo "## disk" +df -B1 --output=avail,used,pcent / | tail -1 +df -B1 --output=avail,used,pcent /boot 2>/dev/null | tail -1 +echo "#### end" diff --git a/documentation/audits/night-build-2026-11-14/E/partE-read3.sh b/documentation/audits/night-build-2026-11-14/E/partE-read3.sh new file mode 100644 index 00000000..58084593 --- /dev/null +++ b/documentation/audits/night-build-2026-11-14/E/partE-read3.sh @@ -0,0 +1,12 @@ +# Part E, third read — READS ONLY. The journal filters of reads 1 and 2 gave 0 lines, positive control included: +# the --since form was wrong. This one uses a full date and shows journalctl's own error. +set -u +echo "#### $(hostname) $(date +%F\ %T\ %Z)" +echo "## control: agent journal lines since 2026-11-14 20:00" +journalctl -u felhom-agent --since "2026-11-14 20:00:00" --no-pager 2>&1 | wc -l +journalctl -u felhom-agent --since "2026-11-14 20:00:00" --no-pager 2>&1 | head -2 | cut -c1-200 +echo "## agent journal: osupdate / kernel / clean-up lines" +journalctl -u felhom-agent --since "2026-11-14 20:00:00" --no-pager -o short-iso 2>&1 | grep -iE "osupdate|kernel|clean-?up|cleanup" | cut -c1-600 | tail -50 +echo "## whole journal 05:30-06:30: wrapper and sudo lines" +journalctl --since "2026-11-15 05:30:00" --until "2026-11-15 06:30:00" --no-pager -o short-iso 2>&1 | grep -E "felhom-os-apply|KERNEL|apt-get" | cut -c1-500 | tail -30 +echo "#### end" diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 76a81fe6..a4c33bdb 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -281,7 +281,7 @@ stopping line that lies. |---|---|---|---|---|---|---|---| | **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC02, CC06-6, CC07-6, CC22-6). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | | **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC | -| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | NARROWED — **2026-11-14 16:06 — RULED (D82, `09` §3 decision 307): a household's box may remove its old kernels, after two REAL night removals on each ring-0 box (the N100 and demo-hp); quiet nights do not count. Count so far: the N100 0, demo-hp 0 (the Tester 1 removal of 2026-11-14 was started by hand and is not a night). It needs a later agent release to switch it on.** Earlier: NARROWED — **2026-11-14 (release 2a): the old-kernel part is BUILT — agent 0.174.0, vouched 15:21 CET (`11` §5.11.1 `[FACT — built]`).** A ring-0 box removes its old kernels at night by the wrapper's own rule. On boxes: a real removal on Tester 1 (3 → 2 kernels, 1.12 GB, 8.5 s; the default, the pinned file, the generators and the state file unchanged; a second run removed nothing; one clean restart) — started by hand with a ring-0 mark set for the test on the operator's yes, because a night cannot be started by day (R-1394); dry runs on demo-hp (nothing to remove; refusals P1, P2, P3) and the N100 (6 of 8 will go). **LEFT for this part: the step by the agent at night — read the N100's and demo-hp's night of 2026-11-15; then a ruling for ring-1 boxes (D82).** Evidence `audits/release-2a-2026-11-14/`. The other four places of this row are unchanged. Earlier: OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | +| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | NARROWED — **2026-11-15 06:35 — the FIRST real night removal: the N100, by itself, at 05:58 CET.** 8 kernels before, 2 after (it kept 7.0.14-26 and 7.0.14-28, the running one), 6 packages purged in 35 s, 6.81 GB gained (the wrapper's own figure; 65.1 GB free after), 0 left-over paths; the default kernel and the pinned file are unchanged (the file's time is still 2026-11-08); the hub's „Kernel step” cell reads „cleaned”; 0 failed units, 9 of 9 containers healthy, clock +0.06 s to DooPlex. demo-hp had a QUIET night (2 installed, nothing to remove) — it does not count; its 6 old kernels went on 2026-11-10 at 13:51, by day. **D82 count: the N100 1 of 2, demo-hp 0 of 2.** demo-hp can count only after two more kernels arrive (it holds 2 and keeps 2). Evidence: `audits/night-build-2026-11-14/E/`. Earlier: NARROWED — **2026-11-14 16:06 — RULED (D82, `09` §3 decision 307): a household's box may remove its old kernels, after two REAL night removals on each ring-0 box (the N100 and demo-hp); quiet nights do not count. Count so far: the N100 0, demo-hp 0 (the Tester 1 removal of 2026-11-14 was started by hand and is not a night). It needs a later agent release to switch it on.** Earlier: NARROWED — **2026-11-14 (release 2a): the old-kernel part is BUILT — agent 0.174.0, vouched 15:21 CET (`11` §5.11.1 `[FACT — built]`).** A ring-0 box removes its old kernels at night by the wrapper's own rule. On boxes: a real removal on Tester 1 (3 → 2 kernels, 1.12 GB, 8.5 s; the default, the pinned file, the generators and the state file unchanged; a second run removed nothing; one clean restart) — started by hand with a ring-0 mark set for the test on the operator's yes, because a night cannot be started by day (R-1394); dry runs on demo-hp (nothing to remove; refusals P1, P2, P3) and the N100 (6 of 8 will go). **LEFT for this part: the step by the agent at night — read the N100's and demo-hp's night of 2026-11-15; then a ruling for ring-1 boxes (D82).** Evidence `audits/release-2a-2026-11-14/`. The other four places of this row are unchanged. Earlier: OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | | **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC | | **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — **2026-11-14 (night build):** the open point of D75 has a written design with a pick, not built: `16-storage-physical-identity.md` §14.5 — refuse below 200 GB on an appliance install, with one named flag for the operator; warn only on a BYO host; 8 edge cases. It waits for the operator's yes. OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. **2026-11-10 15:19 — RULED (D75 = one number, 200 GB, with or without a backup drive; `09` §3 decision 295).** This replaces CC's two-step pick. **Open, not ruled:** what the installer does below 200 GB — refuse, or warn. demo-hp (128 GB) stays as a test box. `16` §14.3 carries the ruling. | — | Design first: what the installer does below 200 GB (refuse, or warn) — a later session. Then the installer. | CC | | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC | @@ -292,7 +292,7 @@ stopping line that lies. | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC | | **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. **2026-11-11: golden 0.352.0 baked and published; one fresh box installed from it (`documentation/tests/golden-0.352.0-2026-11-11/README.md`); the gate is green (`python3 scripts/golden_currency_gate.py`, rc 0). The operator vouched it on 2026-11-11 at 14:48:11 CET.** | WATCHING — **2026-11-14: golden 0.352.1 baked (sha256 `4f411cbe…346d`), one fresh install proven on a nested VM on demo-hp, vouched by the operator 10:25:43 UTC (`documentation/tests/golden-0.352.1-2026-11-14/`). The gate judges the bake's age: the next bake is due by 2026-11-21.** Earlier: **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC | | **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC | -| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | READY — **2026-11-14 night: BUILT, not released** (felhom-agent `386a0c4` + `a9ab47e`; the installer on felhom.eu `main`, not tagged). The root step keeps the copy; the agent moves a pinned box in the order copy → proof → config, clears the fingerprint last, and a pinned box whose certificate changed heals at the first failed call; the installer gives a new box the copy and no fingerprint; the cause of a trust failure is named. `03` §12a is `[FACT — built, not released]`. **LEFT: the proof on a box** (demo-hp, a forced new certificate, twice; then the release with both signed jobs and the installer tag). What the hub does not tell yet: R-1396. Earlier: READY — **2026-11-14 09:55 — RULED (D80, `09` §3 decision 303): the root CA copy.** Built in release 2 (the host release). OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC | +| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | READY — **2026-11-14 night, after an independent review (agent `ec7e699`, `54ac2c2`):** one lock for every writer of `agent.json`; the trust run is serialised and ends after 90 s; the root step keeps the previous root only during a changeover and compares exactly. **One low item stays open:** the move writes `agent.json` in place and nothing restores `agent.json.pre-pve-trust` by itself after a cut write (`03` §12a says so) — to weigh at the box proof. Earlier: READY — **2026-11-14 night: BUILT, not released** (felhom-agent `386a0c4` + `a9ab47e`; the installer on felhom.eu `main`, not tagged). The root step keeps the copy; the agent moves a pinned box in the order copy → proof → config, clears the fingerprint last, and a pinned box whose certificate changed heals at the first failed call; the installer gives a new box the copy and no fingerprint; the cause of a trust failure is named. `03` §12a is `[FACT — built, not released]`. **LEFT: the proof on a box** (demo-hp, a forced new certificate, twice; then the release with both signed jobs and the installer tag). What the hub does not tell yet: R-1396. Earlier: READY — **2026-11-14 09:55 — RULED (D80, `09` §3 decision 303): the root CA copy.** Built in release 2 (the host release). OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC | | **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC | | **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | NARROWED — the hub's check is built, not released. **2026-11-14 (night build):** the agent stamps its clock at the moment of sending (`sent_at`) and the hub alarms the operator when it is more than one minute from the hub's clock, in either direction, and says so when it is back (`08-alarm-ladder.md`, `box_clock_wrong`). A report that waited on the box raises nothing (tested). Not released, not seen on a box. LEFT after the release: the 30 answers themselves still trust the clock; the alarm only makes a wrong clock known. OPEN — **2026-11-14: the one box whose clock was wrong (the N100, 32.6 s) is right again (R-1301 closed); this row's check is still not built.** Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC | | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC | diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index a89a3ea3..8a374c22 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.** +## Moved from STATUS on 2026-11-15 (morning) + +**Updated 2026-11-14 (afternoon).** Release 2a is out: agent 0.174.0, vouched. A ring-0 box now removes its old kernels at night, and a box reloads its security profiles after a kernel change. The N100 keeps time again; the product repaired it, with no hand step. + ## Moved from STATUS on 2026-11-14 (afternoon) **Updated 2026-11-14 (midday).** Release 1 („dates”) is out: controller 0.352.1 and hub 0.181.0 with the Go security fix; golden 0.352.1 is vouched. The false „root files behind” warning is fixed; no alarm mail is due on 11-17. The one-year HTTPS setting is NOT in (next controller release, from 11-15). diff --git a/documentation/lived-in/n100.md b/documentation/lived-in/n100.md index c7586af5..b9d7cb43 100644 --- a/documentation/lived-in/n100.md +++ b/documentation/lived-in/n100.md @@ -55,3 +55,4 @@ what happened, who noticed, and the register row it became. Prose here is not a - **2026-11-05..07 — the nightly Proxmox update step refused three nights in a row.** `proxmox-grub` 2.12-9+pmx4 needs `grub-efi-amd64` of the same version; the update wrapper put `proxmox-grub` in the Proxmox group (its boot-package rule matched only names that START with `grub`), so the whole step was refused (R6) and 34 Proxmox updates waited; an `os_update_failed` mail each morning. Apps not touched. Agent 0.172.1 puts `proxmox-grub` with the boot packages; it reaches the box by the operator's vouch and two signed jobs. Register: fixed in the session (size rule), recorded in the agent's CHANGELOG. Evidence: `audits/demo-check-2026-11-07/n100-pve-step-R6.txt`. (CC) - **2026-11-10 — every place that holds data was measured (read-only; the space list).** Host system disk 38.09 of 100.86 GB (38 %), 4.6 GiB more than on 10-15: the box holds 8 kernels (8.15 GB) and none was ever removed → R-1368. The agent's self-update folder holds 48 old binaries (724 MB) and there are 11 old agent copies beside the binary (155 MB) → R-1368. The host journal sits at its cap (4.28 GB, 74 files, oldest entry 10-04) → R-1367. The three whole-box copies of 2026-07-28 still lie on the system disk (17.59 GB) → R-1087; four template archives (2.56 GB) → R-1368. On the USB drive, `backups/offsite-restore` holds 1.65 GB of two apps that are not installed any more → R-1372. Nothing was deleted or changed. Table: `audits/space-list-2026-11-10/TABLE.md`. (CC) - **2026-11-14 — the clock is right again, repaired by the product.** When the box took agent 0.174.0 and its root files (11:36 and 11:51 UTC), the new „profile refresh" found the old security profile of the time service, loaded the right one and started the service: the clock stepped forward 32.7 s (offset to DooPlex −32.55 s → +0.18 s; still so 2.5 hours later). Nine seconds after the step the controller read a new boot and recreated Radarr, Sonarr and qBittorrent once (each down for a few seconds; all healthy). A dry run says tonight's night removes 6 of the 8 kernels. Register: R-1301 and R-1388 closed, R-1395 (the controller and the clock step), R-1368 (the kernels — read the night of 11-15). Evidence: `audits/release-2a-2026-11-14/D/` (`09`, `13`, `13b`, `14`). (CC) +- **2026-11-15 — the box removed its old kernels by itself, the first time.** At 05:58 CET, at the end of the night's update run (the Proxmox step applied 12 packages first), the clean-up purged 6 of 8 kernel packages in 35 s and kept 7.0.14-26 and the running 7.0.14-28: 6.81 GB gained, 65.1 GB free after, 0 left-over paths. The default kernel and the pinned file did not change; 0 failed units, 9 of 9 containers healthy, the clock +0.06 s to DooPlex; the hub's „Kernel step” cell reads „cleaned”. Nothing was done by hand. Register: R-1368 (D82: 1 of 2 nights). Evidence: `audits/night-build-2026-11-14/E/`. (CC, read-only)