Release 2a: agent 0.174.0 vouched — old kernels removed by the box (11 §5.11.1 built), security profiles refreshed after a kernel change (11 §5.12, spike + build); R-1388, R-1301, R-1391 closed; R-1394, R-1395 filed; D82
gates / gates (push) Successful in 2m1s

This commit is contained in:
admin committed 2026-11-14 15:30:44 +01:00
1 parent a2173830c1
commit 8955b4e55f
51 files changed
+1385 -19

No files matched your search

+131
View File
@@ -0,0 +1,131 @@
# REPORT — release 2a: the box removes its old kernels, and refreshes its security profiles after a kernel change (2026-11-14, attended at one STOP)
## 1. Claims in the prompt that turned out wrong, named first
1. **„A night can be started on a test box by an existing path" — FALSE.** The agent starts the night leg only
after the night's OWN whole-guest backup. A manual backup does not start it, by design (R-899), and the debug pass
runs with trigger `debug`, which the kernel step and the clean-up refuse, by design. So the real clean-up on
Tester 1 was **started by hand** with the agent's own command line, not „through the night leg's own path". The
step by the agent at night is **not proven yet**. Filed: R-1394.
2. **„Tester 1 — the first real clean-up by the program" needed something the prompt did not name.** Tester 1 is
ring 1 and has **no root-owned ring-0 mark**, and the release runs the clean-up on ring-0 boxes only (the prompt's
own scenario 8). The program answered „not run" there, correctly. CC wanted to set the mark by hand; Claude Code's
own safety check refused that, rightly. CC asked at the STOP; **the operator said yes** to the mark for the test.
It was set for under a minute and is back as before (proven in the evidence).
3. **„The wrapper is reached by `--plan` only, so a new mode needs no new sudo line" — TRUE.** Pinned by a test, and
seen live: both new modes ran through the one existing line.
4. **„The result can ride the existing OS-update report" — TRUE in part.** A clean-up that removed kernels reaches
the hub as a kernel-layer report; the hub stores it and shows „cleaned <time ago>" in the „Kernel step" cell. What
was removed, kept and gained is stored and shown nowhere. A refusal or failure raises the existing operator event.
**One trap, avoided and pinned by a test:** the report must carry NO kernel version, or the hub reads it as a
kernel step for that kernel. The profile refresh sends nothing to the hub. Both gaps are on R-1383 (release 3).
5. **„One stale profile can be made on demo-hp without changing a package" — TRUE.** One old compiled copy was put
into the running kernel's cache folder; the time service then failed exactly as on the N100.
6. **The diagnosis' method needed a control it did not have.** „Loaded hash against a fresh compile" is wrong for one
profile always (`lxc-containers` compiles differently every time) and for 17 of 119 after a fresh load (files with
several profiles). The built mode does not use it; it reads the cached FILE changing.
7. **The prompt's „about 4 hours after the start" for the vouch:** the STOP was reached after about 2 hours; the
answer came about 2.5 hours later. No part waited on it except the Tester 1 proof and the vouch.
## 2. Five lines
1. **The clean-up on Tester 1:** 3 kernels → 2 (removed `proxmox-kernel-7.0.14-20-pve-signed`), **1.12 GB** gained,
8.5 s; the default, the pinned file, both generators and the state file unchanged; no left-over; the second call
removed nothing. **The reboot:** back on 7.0.14-22-pve, all 10 containers healthy after 239 s, a clean stop for the
crash guard.
2. **What the N100 removes tonight:** 6 of its 8 kernels (7.0.2-6, 7.0.14-20, -22, -23, -24, -25); it keeps 7.0.14-26
and -28. About 6 GB.
3. **Part B — BUILT.** The spike on demo-hp was clean (0.5 s, both guests and 32 containers untouched, no new kernel
refusal in 10 minutes with a positive control), so the mode was built, tested and released.
4. **The N100's clock:** offset to DooPlex **−32.55 s before, +0.18 s after** and 2.5 hours later; the time service is
active. The product repaired it when the N100 took the new agent's root files; the clock stepped +32.7 s once.
5. **Agent 0.174.0: vouched** by the operator, 2026-11-14 15:21:45 CET (the hub's log and its System page).
## 3. The parts
| Part | Result |
|---|---|
| A — the kernel clean-up | BUILT. Wrapper mode `kernel-cleanup` (layer kernel) and the agent's night step. `11` §5.11.1 is `[FACT — built]` |
| B — security profiles | SPIKED, then BUILT. Wrapper mode `profile-refresh` (layer host) and the agent's call at start and after a bundle install. `11` §5.12 (new) |
| C — Go 1.27.4 | done; the running binary reads `go1.27.4` on the three boxes. R-1391 closed |
| D — the release | 0.174.0 released, on Tester 1, demo-hp and the N100 by the two signed jobs each, vouched |
| E — the one-year HTTPS setting | **skipped, nothing changed:** the build day is 2026-11-14, before the ruled 2026-11-15 |
**Versions, digests, CI by commit.** felhom-agent code `cc63dd5` — CI run id 2641 success (read before the release);
release records `2706e35` — run id 2643 success. Agent sha256 `6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92`,
bundle `7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f`. No hub build, no hub deploy, no controller
release. felhom.eu: documents only (the CI run of this push is quoted in the chat note).
**Design decisions taken by CC in the build (each follows the design; the operator may reverse):**
- The clean-up asks „may this box?" from the **root-owned ring-0 mark only**; the agent does not gate on the hub's
ring. (`11` §5.4.1: the protection is what the wrapper refuses.)
- „Nothing to remove" and „not run" send **no hub report** — one a night would bury the kernel step's own line.
- The wrapper adds its own 20-hour window after a kernel step (the agent also never calls it in a step's night).
- Profile refresh: **once per kernel** (not at every agent start), and the **narrow restart rule** (a failed service
whose own program's cached profile was stale). Both with the other option and its cost in the spike's README.
**Tests, each with its red-proof** (`audits/release-2a-2026-11-14/A`, `B`):
| Suite | Cases | Mutations seen red |
|---|---|---|
| wrapper, clean-up (`KernelCleanup`) | 31: one per scenario 1–11, one per kept-kernel kind for scenario 6, the window, the dry run, the after-purge proof, the lock, two image packages, the sudo line | 14, incl. the four the prompt names: K1 removed → the kept kernel reaches the purge line; the simulation check removed → the purge runs although apt would do more; the phase check removed → a purge during `judging`; the ring-0 check removed → a box without the mark purges |
| wrapper, profile refresh (`ProfileRefresh`) | 11 | 10 |
| agent, clean-up | 5 (the night's order, once, no names; no report for nothing / not run; refused and failed reach the hub; when it may not run; the dry run) | 3 |
| agent, profile refresh | 4 | 3 (one did not compile at first and proved nothing; done again) |
Two existing agent tests list the night's calls exactly and gained the clean-up call; three kernel-step tests now
count only the step's own calls. Wrapper suite 191 green; the agent's `go test ./...`, `go vet` and the gates green.
**Rows:** 250 before, 249 after. Opened R-1394 (a night cannot be started by day), R-1395 (the controller and a clock
step). Closed R-1388, R-1301, R-1391. Narrowed R-1368 (the kernel part: the night is left, then the ring-1 ruling).
Dated lines: R-1383, R-1384. New decision in `STATUS.md`: D82 (may a household's box clean up its kernels).
**Machines changed, and how each was left.**
- **demo-hp:** the spike planted one stale profile and removed it (cache = the fresh compile, loaded hashes back as
found by one cache load, the spike's files gone, chrony active — `profile-refresh-spike-2026-11-14/evidence/07`).
Three test states for the refusal dry runs, each put back; checksums identical after (`D/10`). Agent 0.174.0 + bundle.
- **Tester 1:** agent 0.174.0 + bundle; one kernel removed; the ring-0 mark set for the test and put back (`false`,
root:root 0644, no saved copy left); one restart. The hub's System page showed a partial line for it after the
restart (see Observations).
- **The N100:** agent 0.174.0 + bundle by the normal signed jobs; reads and two dry runs by CC; no hand step. The
product restarted its time service and the clock stepped; the controller recreated three apps once.
- **DooPlex:** one release build, the signing tool built in the scratchpad. Nothing else. **ep0, the hub:** no change.
- **Scratch guest 9202, bench VMs:** not used.
**Architecture documents read:** `11-os-updates.md` §5.4.1, §5.4.2, §5.5 (the night's order, through the leg's code),
§5.9, §5.11, §5.11.1; `09-update-architecture.md` §3 decisions 286, 305, 306; `runbooks/config-bundle.md`;
`audits/kernel-cleanup-spike-2026-11-10/` (README and the rule script); `audits/n100-clock-2026-11-11/`; the
`felhom-build-deploy` skill. **Not read in full:** `03-host-agent.md` (the sudo rules were read in the sudoers file
itself), `05-hub-architecture.md` (the vouch flow was unchanged), `09` decisions 164, 171, 172, 195 (the kernel
lane's own rulings; its behaviour was read in `11` §5.11 and in source).
**Documents changed:** `11` §5.11.1 (`[FACT — built]`, with the proofs and the „not proven" lines) and §5.12 (new);
`16` §14 (the kernel line: its limit and its cleaner); `runbooks/disk-full.md` (the hand step is no longer the only
way); the capability map row G18; `lived-in/n100.md`; the register; `STATUS.md`. **System poster:** no listed fact
changed; none of the three acts was needed. **Instruction files:** none edited. **`unproven.py --summary`:** 32
walked, 15 partial, 4 built, 4 missing, 23 not walked — no number moved.
## 4. What is not proven
- **The clean-up by the agent at night** — the N100's and demo-hp's night of 2026-11-15 (`STATUS.md` says what to read).
- A box whose running kernel is not its newest, and a self-revert into the kept kernel chosen by the product (tests only).
- Refusal K1 on a box (it guards a wrong computation; tests only). Scenarios 7, 9, 10 and 11 on a box (tests only).
- A restart by the profile-refresh rule of any program but chrony; the refresh after a real fall-back of the kernel lane.
- A box that boots with `proxmox-boot-tool`; a BYO host on a box (tests only: R12).
- That no audit line that mattered was lost in the seconds after a refresh (the kernel suppresses them; measured).
## 5. Observations
- A night cannot be started by day on a test box. `FILED: R-1394`.
- A clock step makes the controller read a new boot and recreate its drive-backed apps. `FILED: R-1395`.
- The hub shows no detail of a clean-up and nothing of a profile refresh. `FILED: a dated line on R-1383`.
- `lxc-containers` never compiles to the same binary twice, so every refresh „rewrites" it. `NOT-A-FINDING: AppArmor's behaviour; written into `11` §5.12; it names no program, so the restart rule never matches it.`
- After a profile reload the kernel suppresses audit lines for some seconds. `NOT-A-FINDING: the kernel's rate limit; it is the reason for „once per kernel", written into `11` §5.12.`
- In the customer guest AppArmor is installed and inactive, and the containers read „unconfined" from inside. `NOT-A-FINDING for this release: no cache exists there, so the stale-profile fault cannot. Not checked: whether the register already holds a row on container confinement inside the guest.`
- The old wrapper refuses the new mode at the first start of the new agent; the refresh then runs after the bundle. `NOT-A-FINDING: built for (the agent asks again after a bundle install); seen on three boxes.`
- The release script prints the repository's address in its output. `NOT-A-FINDING: the address holds no login since 2026-11-11; the output was not committed.`
- The signing tool prints the whole signed job when its output is not filtered; one evidence file held two such jobs for a minute. `NOT-A-FINDING: a signed job is not a secret and is single-use; the lines were removed before any commit and the script now filters.`
- Tester 1's hub line read „partial … crash guard not installed" after its restart. See the chat note for the last read.
- The hub still lists host `Tester4-46836f` (this morning's test VM, destroyed). `NOT-A-FINDING: the operator's delete, already in STATUS.md.`
+3 -2
View File
@@ -2,7 +2,7 @@
**Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-11-14 (midday).** Release 1 („dates”) is out: controller 0.352.1 and hub 0.181.0 with the Go security fix; golden 0.352.1 is vouched. The false „root files behind” warning is fixed; no alarm mail is due on 11-17. The one-year HTTPS setting is NOT in (next controller release, from 11-15).
**Updated 2026-11-14 (afternoon).** Release 2a is out: agent 0.174.0, vouched. A ring-0 box now removes its old kernels at night, and a box reloads its security profiles after a kernel change. The N100 keeps time again; the product repaired it, with no hand step.
## What works
- **The Plex attack did not reach Felhom's secrets** (R-1241).
@@ -32,6 +32,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|---|---|---|---|---|---|---|
| D82 | R-1368 | May a household's box (ring 1) also remove its old kernels by itself at night? In release 2a only our ring-0 test boxes do. | **Yes, after 7 clean nights on the N100 and demo-hp** (the rule is proven by hand, by tests and on Tester 1; a household's disk fills by 1 GB per kernel) | no: a household's box keeps every kernel; someone removes them by hand before the disk is full | ring-1 boxes keep every kernel (Tester 1: 2 now, Tester 2: unknown) | the last part of R-1368 |
| D66 | R-1365 | Should a security-watch source that stays down mail again every 7 days, as an open notice does? Today it mails once, then only the Monday summary lists it. | **Yes** (a real outage then cannot be forgotten; the seven pins that can never be read no longer count, so it stays quiet when all is well) | no: one line in a long Monday mail stays the only reminder | a source that stays down mails once; the Monday list still shows it | — |
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
@@ -39,7 +40,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
## Waiting on you — do
**2026-11-14.** Next builds: host, „hub tells you”, promises; no new gap list first. Hub Hosts page: delete host `Tester4-46836f` (after 12:15 CET; else it alarms). Say „HSTS now” from 11-15 (R-1050). **2026-11-11.** 1. N100's time service: no hand step now; release 2 repairs it. If not live by 2026-12-15: the hand step in R-1301. 2. Hosts page: discard `3fb5fed3…`. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18).
**2026-11-15 morning — read:** the N100 removed 6 of its 8 kernels at night, the hub's „Kernel step” cell says „cleaned”, the box is healthy. If not: tell CC (R-1368). **2026-11-14.** Next builds: 2b (trust in Proxmox), „hub tells you”, promises; no new gap list first. Hosts page: delete host `Tester4-46836f` (else it alarms). Say „HSTS now” from 11-15 (R-1050). **2026-11-11.** 2. Hosts page: discard `3fb5fed3…`. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18).
**Still needed? (yes/no)** — answer per line.
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16)
@@ -249,7 +249,7 @@
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a consistent copy every 6 hours since D34 (nightly before 2026-10-23; corrected 2026-10-24), encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **added 2026-11-14 (agent v0.174.0):** a box reloads its security profiles after a kernel change — PROVEN-LIVE on three boxes, it repaired the N100's time service (`audits/release-2a-2026-11-14/D/13-n100-after.txt`); a ring-0 box removes its old kernels — the removal PROVEN on Tester 1, started by hand (`D/08-tester1-cleanup.txt`), the step at night NOT yet; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
| <a id="g19"></a>[G19](capability-map-detail/G.md#g19) | An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps | controller v0.259.0 + hub v0.119.0 + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE** | walked on 0.258.0 with one blocker; fixed, the walk not repeated | `audits/DRILL-first-hour-en-0258-2026-09-20.md` (2026-09-20); `audits/i18n-closing-2026-09-21/live/` (2026-09-21) |
| <a id="g20"></a>[G20](capability-map-detail/G.md#g20) | A deletion of a customer's off-site history is NOTICED within a day | hub v0.111.0 (R-431) | **IMPLEMENTED** | not yet proven live | R-431 |
+97 -4
View File
@@ -609,7 +609,7 @@ another kernel), and the kernel step (the newest result, the kernel due, whether
7.0.14-20), so a soft lockup turning into a panic is recorded as unmeasured (no tool was built). A true dead freeze
needs a person (accepted, decision 172). Evidence and proofs: `audits/kernel-lane-2026-10-07/`.
### 5.11.1 Removing old kernels `[DESIGN — proven by hand on demo-hp 2026-11-10; not built]`
### 5.11.1 Removing old kernels — BUILT 2026-11-14 (agent v0.174.0; R-1368, `09` §3 decision 286) `[FACT — built]`
**The ruling** (`09` §3 decision 286): the box may remove old kernels by itself, as its own step with its own narrow
rule. **Rule R4 does not change**: an update plan that removes a package is still refused. Why a step is needed: the
@@ -642,14 +642,107 @@ self-revert into `from`, must be over first. In the night's order it belongs whe
whole-guest backup. It needs no reboot and stops no guest. A box that fell back or reverted keeps all its kernels
until a later step ends `good`.
**Proven** on demo-hp by hand: 8 kernels → 2, 6.4 GB gained on the system disk, the default, the pinned file and both
**As built** (`felhom-agent` `configs/felhom-os-apply`, layer `kernel`, mode `kernel-cleanup`; the agent's
`internal/osupdate/kernel.go`):
- **The plan names nothing.** The wrapper computes the set from the box; a plan that carries packages, a select, a
signed job or an expected kernel is refused (C1). `dry_run: true` is the same code path: it reports the keep set, the
removal set and what it would do, and changes nothing (`felhom-agent --selftest=kernel-cleanup-dry-run`).
- **Who may:** an appliance (R12, the root-owned install record) with the **root-owned ring-0 mark**
(`/etc/felhom/os-trust.json`), read by the wrapper itself — never the agent's config, never the hub's ring. A box
without the mark answers „not run". Ring 1 needs an operator ruling first.
- **„Not run"** (exit 0, nothing changed, a log line): no ring-0 mark; no kernel step yet (no state file); the newest
step ended in another phase than `good` (`fell_back`, `self_reverted`, `revert_failed`, `cancelled`); the box does
not run its newest installed kernel; a step was staged, booted or ended within the last 20 hours.
- **Refused** (exit 2, nothing changed): the table above, each with its own code; the wrapper repairs dpkg first
(§5.4.1), so a purge that a power cut interrupted is made whole and the rule is computed again from the box.
- **K1 reads the box a second time.** The removal set is built from the keep set; K1 then reads the running kernel, the
step's `from`/`to`/`default_before`, both defaults, the flag and the boot pin again and refuses a name that any of
them still needs — also before apt's simulation.
- **After the purge** the wrapper proves that the default in `grub.cfg`, the pinned file, both generators and the
step's state file did not change, and that every kept kernel is still installed. Anything else is `failed`, said
loudly; nothing is repaired in code. A left-over `/lib/modules/<version>` or `/boot` file is reported, never deleted.
- **When (the agent):** the night leg's own step, after a healthy host step (and Proxmox step, when one ran), the last
call of the night, never in a night with a kernel step (due, staged or reported), never in a debug pass, never on a
BYO host. Not retried the same night.
- **The sudo rule did not change:** the wrapper is reached by the one `--plan` line.
- **What the hub shows (hub v0.181.0, no hub change in this release):** a clean-up that removed kernels arrives as a
kernel-layer report with outcome `cleaned`; the System page's „Kernel step" cell then reads „cleaned <time ago>", and
the report's text (what was removed, kept and gained) is stored but shown nowhere. A refusal or a failure raises the
existing operator event „Kernel step on <box> refused/failed: …". „Nothing to remove" and „not run" are log lines
on the box only. The report carries no kernel version on purpose: the hub reads that as a kernel step. What is missing
is in R-1383 (release 3).
**Proven on boxes with the program** (2026-11-14, `audits/release-2a-2026-11-14/D/`): Tester 1 — 3 kernels → 2
(`proxmox-kernel-7.0.14-20-pve-signed`), 1.12 GB gained, 8.5 s, nothing else changed, no left-over, the second call
removed nothing, one orderly restart back onto the same kernel with a clean stop for the crash guard. **That run was
started by hand** with the agent's own command line, and with the ring-0 mark set for the test on the operator's yes
(Tester 1 is ring 1; a night cannot be started by day, R-1394). demo-hp — „nothing to remove"; P1, P2 and P3 refused by
dry run against states set and put back. Tester 1 without the mark — „not run". The N100 — the dry run names 6 of 8.
**Not proven:** the step by the agent at night (the night of 2026-11-15 on the N100 and demo-hp); K1 on a box (it
guards a wrong computation — tests only); a box whose running kernel is not its newest, and a self-revert into the
kept kernel chosen by the product (tests only); a box that boots with `proxmox-boot-tool`.
**Proven by hand** on demo-hp (2026-11-10): 8 kernels → 2, 6.4 GB gained on the system disk, the default, the pinned file and both
generators unchanged, five refusals by simulation, one orderly reboot back onto the same kernel with a clean stop for
the crash guard (§5.9), a second run that removed nothing. The kept old kernel then booted the box to healthy through
the one-shot flag (313 s), and the box went back to its newest kernel (324 s); the pinned file, `grub.cfg` and the
step's state file did not change. **Not proven:** a box whose running kernel is not its newest; a self-revert into
the kept kernel chosen by the product; the step as a program at night; a box that boots with `proxmox-boot-tool`. Evidence, the rule as a read-only script, and the full "not measured" list:
step's state file did not change. Evidence, the rule as a read-only script, and the full "not measured" list:
`audits/kernel-cleanup-spike-2026-11-10/`.
### 5.12 Security profiles after a kernel change — BUILT 2026-11-14 (agent v0.174.0; R-1388, R-1301) `[FACT — built]` `[FACT — measured on demo-hp]`
**The fault** (`audits/n100-clock-2026-11-11/`). AppArmor keeps one cache folder for each kernel feature set. When a
package changes a profile while one kernel runs, only that kernel's folder gets the new compiled copy. A box that then
starts on another kernel loads the OLD copy from that kernel's folder — the copy's time looks newer than its source.
The order „start a new kernel once → go back → a profile is updated → start the new kernel again" is made by a kernel
test, and by the kernel lane's own fall-back or self-revert (§5.11). On the N100 it stopped the time service.
**The mode** (`felhom-os-apply`, layer `host`, mode `profile-refresh`; an appliance only; it changes no package and
takes nothing from the plan but `dry_run`, F1):
1. It runs **once per kernel the box starts on**: the wrapper keeps the kernel of the last refresh in a root-owned
record (`/var/lib/felhom-os-apply/profile-refresh.json`) and answers „not needed" for the same kernel. A box with no
record counts as changed.
2. `apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d` — every profile from a fresh
compile, the cache ignored and rewritten.
3. **The narrow restart rule:** a service in state `failed`, whose `ExecStart` program is the path-name of a profile
whose cached copy the rewrite CHANGED, gets one `systemctl restart`. At most 3 a run. Never „whatever failed", never
a unit a plan names. A running program needs nothing: the kernel swaps its profile in place.
4. A parser failure writes no record; the next start tries again.
**The agent** asks at every start and after a config bundle was installed (a new wrapper may know the mode the old one
refused), under the heavy-op gate (up to 12 tries, 5 minutes apart) and never beside an OS-update pass. Nothing goes to
the hub (no field exists for it; R-1383).
**Measured on demo-hp, both guests running** (`audits/profile-refresh-spike-2026-11-14/`): the fault was reproduced
with one planted stale profile (chrony failed with the N100's own error); the refresh took 0.51 s; the same 119
profiles stayed loaded in the same modes; no container restarted; no new kernel refusal in 10 minutes (with a positive
control — see below); the restart rule brought chrony back.
**Three facts the spike found, each built into the mode:**
- A stale copy is told by the **cached file changing at the rewrite**. The kernel's loaded hash is NOT a signal: for a
file with several profiles it differs between a cache load and a fresh load of the same policy (17 of 119 profiles).
- `lxc-containers` compiles to a different binary every time, so it is „rewritten" at every refresh. It names no
program path, so the restart rule never matches it.
- A refresh writes about 119 audit lines, and the kernel then **suppresses other audit lines for a few seconds**
(a real refusal in those seconds is not in the kernel log). This is the cost that makes „once per kernel" the right
trigger and „at every agent start" the wrong one.
**Proven on boxes by the product's own path** (2026-11-14, `audits/release-2a-2026-11-14/D/`): on Tester 1, demo-hp
and the N100 the old wrapper refused the mode at the agent's start (logged), and the refresh ran right after the
bundle install — 0.5 s each. Tester 1 and demo-hp: only `lxc-containers` rewritten, nothing restarted. **The N100:**
the stale chrony profile was rewritten, `chrony.service` (failed since 2026-10-16) was restarted, the clock stepped
+32.717 s; offset to DooPlex −32.55 s → +0.18 s, the same 2.5 hours later. After a restart of Tester 1 the agent
answered „not needed" for the same kernel.
**Seen on the N100, a consequence of the clock step, not of the refresh:** the controller read a new boot and
recreated its three drive-backed apps once (R-1395).
**Not proven:** a restart by this rule of any program but chrony; the refresh after a real fall-back or self-revert of
the kernel lane (the trigger is the same: another kernel than at the last refresh); the spike's „Not measured" list.
## 6. Risks and edge cases
| # | What can go wrong | What the design does |
@@ -275,7 +275,7 @@ The alarms, by short name:
| H6 | backup logs and half-written copies | each run | a failed run's log: none (R-1368) | Proxmox; the agent | host disk |
| H7 | the host journal | the agent's `sudo` calls (R-1367) | the journald default: 10 % of the disk, at most 4 GB | journald | host disk |
| H8 | other host logs, task logs, graphs | each task | rotation | `logrotate`, Proxmox | host disk |
| H9 | old kernels | each approved kernel | none today. Ruled: keep the running one, the one before it and the newest (decision 286, `11-os-updates.md` §5.11.1). Proven by hand, not built | nobody today | host disk; later `os_update_stale` |
| H9 | old kernels | each approved kernel | the running one, the one before it and the newest: 3 at most, about 3.4 GB (decision 286, `11-os-updates.md` §5.11.1) | the box itself at night, on a ring-0 box (agent v0.174.0); a ring-1 box: nobody yet (needs a ruling) | host disk; a refused or failed clean-up is an operator event |
| H10 | the EFI partition | a boot-loader update | 1.07 GB, 1 % used | none needed | none (R-1371) |
| H11 | the host package cache | each OS update | emptied in each run | the update wrapper | host disk |
| H12 | the agent's self-update folder | each agent update | the running version's binary, and the file of a failed update until the next good one (about 16 MB each; measured 2026-11-10 on three boxes, R-1368) | the agent, at the end of each good update | host disk |
@@ -410,7 +410,7 @@ below 200 GB — refuse, or warn (R-1370). demo-hp (128 GB) stays as a test box.
| A drive in the backup role reports as `usb` or `local-dir`. So the household gets the `storage_fill` mail for a drive it must not clean by hand | R-1371 |
| The hub's first list for a household still holds `storage_fill_critical`. It is replaced when the box sends its own list. **Not measured on the running hub:** the release run of 2026-11-10 deployed the hub that blocks the host-disk alarm at send time, but the pages that show a household's list refuse a token („session only"), and no test alarm was made. The proof is the tests and the deployed commit (`audits/release-space-fixes-2026-11-10/hub/`) | R-1371 |
| Helper images carry no version; the guest's journal has no stated cap; off-site restores have no limit | R-1372 |
| Old kernels: the keep rule is ruled and proven by hand, not built | `11-os-updates.md` §5.11.1 |
| Old kernels: the clean-up is built for ring-0 boxes (agent v0.174.0); a ring-1 box still keeps every kernel | `11-os-updates.md` §5.11.1 |
| The parts of the host disk with no limit and no cleaner (H4, H5, H6, H13) | R-1087, R-1368 |
| The smallest disk is ruled (200 GB, `09` §3 decision 295). Nothing enforces it: what the installer does below 200 GB (refuse, or warn) is open | R-1370 |
| What to do by hand when a disk is full | `runbooks/disk-full.md` (R-1369) |
@@ -0,0 +1,60 @@
# Spike — can a box refresh its security profiles by itself? — 2026-11-14 (release 2a, Part B)
**Answer: yes. The spike is clean, so the mode was built in the same run** (agent v0.174.0, `11-os-updates.md` §5.12).
Venue: demo-hp (Tier 0), host kernel 7.0.14-28-pve, guests 9201 (26 containers) and 9202 (6) running the whole time;
reads on Tester 1. Nothing on the N100 was touched. No package was changed. Documents read:
`audits/n100-clock-2026-11-11/README.md` (the diagnosis and its method), `architecture/11-os-updates.md` §5.4.1, §5.9,
§5.11, rows R-1388, R-1301.
## The six questions
| # | Question | Result | File |
|---|---|---|---|
| 1 | Does the detector (loaded hash against a fresh compile) work, and why does `lxc-containers` always "differ"? | It works for one-profile files. **`lxc-containers` compiles to a different binary every time** (two fresh compiles in a row differ; it includes a whole folder). So its hash can never judge it; the detector now compiles twice and says so. On demo-hp and Tester 1: 106 files, 0 stale, 1 not judgeable, 4.5 s | `evidence/01-detect-*.txt`, `tools/detect.sh` |
| 2 | Can one stale profile be made by the real mechanism, with no package change? | Yes. The old compiled chrony profile (of 4.6.1, sha `dbf67f16…`) was copied from the old kernel's cache folder into the running kernel's; its time (2026-08-21) is newer than the source's (2026-07-01). The system's own reload then **loaded the old one**, and a restart of chrony **failed exactly as on the N100** („Could not send notification to $NOTIFY_SOCKET", twice) | `evidence/02-make-stale.txt` |
| 3 | The smallest repair | `apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d` — **0.51 s**. After it: the same 119 profiles in the same modes; chrony's loaded profile and its cached copy are the fresh compile (`83417e83…`); both guests running, 26 and 6 containers, none restarted (newest start time unchanged), every guest init still confined by its own profile | `evidence/03-refresh.txt` |
| 3b | Why did 17 other profiles change their loaded hash? | Not staleness. For a file that defines several profiles the kernel's hash differs between a cache load and a fresh load of the **same** policy. **The only reliable sign of a stale copy is the cached FILE changing when it is rewritten** (here: only the planted one, and `lxc-containers`) | `evidence/03b-why-others-changed.txt` |
| 4 | What starts the failed program again? | The refresh does not (chrony stayed `failed`). The narrow rule, tried by hand: a **failed service** whose **ExecStart program is the path-name of a profile whose cached copy the refresh rewrote** → one `systemctl restart`. chrony came back, 0.0003 s off | `evidence/04-restart-rule.txt` |
| 5 | When should it run? | See the options below; the pick is „when the box runs another kernel than at the last refresh" | — |
| 6 | Can the guest have the same staleness? | Not today: in guest 9201 the AppArmor package is installed, its service is inactive, there is no profile cache, and the kernel's profile list is not visible | `evidence/06-guest-read.txt` |
**No new refusal by the kernel in 10 minutes** (`evidence/05-watch.txt`): 1 `DENIED` line, and it is the control.
**The control was needed** (`evidence/05a-denied-watch-control.txt`): the kernel rate-limits its audit lines, each
reload writes about 119 `STATUS` lines, and the `DENIED` line that the planted stale profile made at 10:58:07 UTC is
**not in the kernel log at all** („kauditd_printk_skb: 111 callbacks suppressed"). A count of 0 alone would have proved
nothing. The second channel is behaviour: no container restarted, no failed unit.
## Decision 4 — what restarts the program (two options)
| | Option A (picked) | Option B |
|---|---|---|
| Rule | a failed service whose own program's cached profile was stale | a fixed list: `chrony.service` |
| Covers | every program this fault can stop | the one case seen |
| Risk | a unit that failed for another reason, and whose profile was also stale, gets ONE start try | none beyond chrony |
| Cost | 3 `systemctl` reads per failed unit, only when a path-named profile was rewritten | a new release for each next program |
Never „restart whatever failed", never a unit named by the plan, at most 3 units a run.
## Decision 5 — when it runs (two options)
| | When the kernel differs from the last refresh's (picked) | After every agent start |
|---|---|---|
| Catches the fault | yes: a stale copy can only appear for a kernel that did NOT run when a profile changed, so the box always starts on „another kernel" before the kernel loads it. A box with no record yet counts as changed (this is what repairs the N100) | yes |
| Cost | one refresh per kernel change (about weekly) | one per agent start (about daily): 0.5 s, and each run floods the kernel log with ~119 lines and **hides other audit lines for a few seconds** (measured above) |
The record is the wrapper's own root-owned file; the agent asks at every start and after a bundle install, and the
wrapper answers „not needed".
## Put back
`evidence/07-put-back.txt`: the spike's files removed from `/root`; the planted copy is gone (the cache holds the fresh
compile, the old kernel's folder is untouched); one cache load (`systemctl reload apparmor`) put the loaded hashes
back as found; the detector reads 0 stale; chrony active; both guests running; no failed unit.
## Not measured
- A box that boots with `proxmox-boot-tool` or without AppArmor (none in the fleet).
- A profile in `complain` mode or in `/etc/apparmor.d/disable` (demo-hp has none; the parser is given the folder, as the
system's own loader is).
- The refresh while a backup or a restore-test runs (the agent takes the heavy-op gate so it never does).
- The real case: the N100. It is the proof of the built mode, in the release.
@@ -0,0 +1,5 @@
#### demo-hp 2026-11-14T10:57:26Z kernel 7.0.14-28-pve
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
loaded profiles: 119
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 03ad7998e16e a5f0e0eaf4e4
files=106 cache-differs=0 loaded-differs=0 profiles-not-loaded=4 not-deterministic=1 seconds=4.556914622
@@ -0,0 +1,5 @@
#### felhom 2026-11-14T10:57:31Z kernel 7.0.14-22-pve
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
loaded profiles: 118
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 7ea3f12cf701 b1b2529f2c17
files=106 cache-differs=0 loaded-differs=0 profiles-not-loaded=4 not-deterministic=1 seconds=4.179027032
@@ -0,0 +1,19 @@
#### demo-hp 2026-11-14T10:58:06Z kernel 7.0.14-28-pve
## before: chrony active; loaded 83417e83b538; cache(cur) 83417e83b538; cache(old folder) dbf67f16ac25; fresh 83417e83b538
## source mtime 2026-07-01 22:28:16; old copy mtime 2026-08-21 17:42:38
saved the good cache file to /root/spike-profile-usr.sbin.chronyd.good (83417e83b538)
PLANTED: the old compiled copy is now in the running kernel's folder; its time 2026-08-21 17:42:38 is newer than the source
## reload the way the system does at a start (systemctl reload apparmor = apparmor_parser --write-cache --replace)
reload rc=0
## after the reload: loaded dbf67f16ac25 (dbf67f16 = the profile of chrony 4.6.1; 83417e83 = of 4.8)
## chrony still runs under the replaced profile? active; its confinement: /usr/sbin/chronyd (enforce)
Job for chrony.service failed because the control process exited with error code.
See "systemctl status chrony.service" and "journalctl -xeu chrony.service" for details.
restart rc=1
## after a restart: chrony failed; result=exit-code
2026-11-14T11:58:07+01:00 demo-hp chronyd[1853258]: Loaded seccomp filter (level 1)
2026-11-14T11:58:07+01:00 demo-hp chronyd[1853258]: Fatal error : Could not send notification to $NOTIFY_SOCKET
2026-11-14T11:58:07+01:00 demo-hp chronyd[1853258]: Fatal error : Could not send notification to $NOTIFY_SOCKET
2026-11-14T11:58:07+01:00 demo-hp systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
2026-11-14T11:58:07+01:00 demo-hp systemd[1]: chrony.service: Failed with result 'exit-code'.
2026-11-14T11:58:07+01:00 demo-hp systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
@@ -0,0 +1,55 @@
#### demo-hp 2026-11-14T10:58:32Z
## BEFORE
guest 9201: status: running init pid 2159 confinement=lxc-9201_</var/lib/lxc>//&:lxc-9201_<-var-lib-lxc>:unconfined (enforce)
containers: 26 running, 0 unhealthy, started-at-newest=2026-11-14T09:18:59.079611972Z
the controller container's process (host pid ?): confinement=
guest 9202: status: running init pid 2117 confinement=lxc-9202_</var/lib/lxc>//&:lxc-9202_<-var-lib-lxc>:unconfined (enforce)
containers: 6 running, 0 unhealthy, started-at-newest=2026-11-14T09:56:41.303141464Z
the controller container's process (host pid ?): confinement=
VM 341 (Tester 1): status: running
loaded profiles: 119
chrony: failed; loaded chrony profile dbf67f16ac25; cache dbf67f16ac25
disabled profiles (must stay unloaded):
## THE COMMAND: apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d
rc=0 seconds=.514403371
## AFTER
guest 9201: status: running init pid 2159 confinement=lxc-9201_</var/lib/lxc>//&:lxc-9201_<-var-lib-lxc>:unconfined (enforce)
containers: 26 running, 0 unhealthy, started-at-newest=2026-11-14T09:18:59.079611972Z
the controller container's process (host pid ?): confinement=
guest 9202: status: running init pid 2117 confinement=lxc-9202_</var/lib/lxc>//&:lxc-9202_<-var-lib-lxc>:unconfined (enforce)
containers: 6 running, 0 unhealthy, started-at-newest=2026-11-14T09:56:41.303141464Z
the controller container's process (host pid ?): confinement=
VM 341 (Tester 1): status: running
loaded profiles: 119
profiles whose loaded hash changed:
lxc-container-default 68d1e8408c2e9241ee6081869502ed73059174251ebb1af548f86afc72b06bec
lxc-container-default-cgns 2e6dec9bc9637ea459d1675047ef6a887bd41bc4511447cbbb61cbf23d6b6fed
lxc-container-default-with-mounting caeab50e1e3007a362e37a64c9adab1334aa4a9decacbba35a69301ed7bcf9bf
lxc-container-default-with-nesting 659040965dfea57ecb76f61643ce276e7ee35550e5089e6ed34862b6ae963d33
man_filter 6dc1077b7fa57c23a446f7bdcca7e27457182dcb12a635219604c23a16d414c9
man_groff 0f6549c22ad03ed89f4c857aa4d332f2a0f2b777f6f07939cad9486291645815
nvidia_modprobe f4ae61617079f7e868204f1bd5046778570fc5ed2c9a3cad9aa6c8ff34c657e7
plasmashell 17cfa46264cdf3b2ca1181848a56604d4efc64b328c8d57b221529e4d22045ea
transmission-cli 04b531d8a589bfa5b8c154d961f58835a57f4d35d17520915b971f9a15d3af3d
transmission-daemon f46a1d2db62218fa9f9be1afacfe256c2b1b0c6ed971117f38f883e1d61d49cb
transmission-gtk ab371958b58947d437928b327736e4c49cb83b3db1f2531d9765a936dc50ffe6
transmission-qt 0077a589d81e621e1faa806a460110820e10cccb8b1916b0e2a3b747a99d0894
/usr/bin/man 4ee549c2ed8742dbe5c1be3fc0b9b3e08ea7f77463366094ad454123e8fc03ed
/usr/lib/connman/scripts/dhclient-script fcadbc62c5777ecee8b1ec1d6f0abab4131df7fc1dc7f7b99635e689f3e60ee8
/usr/{lib/NetworkManager,libexec}/nm-dhcp-helper 6ca7645b1031d446bc65bf4bc9eec5cad0f7e70bf1d841271e972d79ebdf10d3
/usr/lib/NetworkManager/nm-dhcp-client.action b5532441db4544da68a7ac5bc6980f0aaa34dc28fb9ab702221d8ec04cca38b3
/usr/sbin/chronyd 83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1
/{,usr/}sbin/dhclient 3b3e18f5a89e78eb104fd6377ab41ec76dcb9823f799a95dc6d3dac787e429e2
profiles added or gone (names):
none
modes changed:
none
chrony profile: loaded 83417e83b538; cache 83417e83b538 mtime 2026-08-21 17:42:38; fresh 83417e83b538
chrony unit, NOT restarted by the command: failed
kernel lines since the command:
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.683:265): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="brave" pid=1855998 comm="apparmor_parser"
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.684:266): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="buildah" pid=1855999 comm="apparmor_parser"
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.686:267): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="busybox" pid=1856000 comm="apparmor_parser"
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.686:268): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="cam" pid=1856001 comm="apparmor_parser"
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.686:269): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="ch-run" pid=1856003 comm="apparmor_parser"
(end)
@@ -0,0 +1,5 @@
#### demo-hp 2026-11-14T10:59:22Z
loaded: fresh-load vs cache-load differ for: 17 profile(s): lxc-container-default lxc-container-default-cgns lxc-container-default-with-mounting lxc-container-default-with-nesting man_filter man_groff nvidia_modprobe plasmashell transmission-cli transmission-daemon transmission-gtk transmission-qt /usr/bin/man /usr/lib/connman/scripts/dhclient-script /usr/{lib/NetworkManager,libexec}/nm-dhcp-helper /usr/lib/NetworkManager/nm-dhcp-client.action /{,usr/}sbin/dhclient
loaded: fresh-load vs second fresh-load differ for: 2 profile(s): lxc-container-default-cgns lxc-container-default-with-nesting
CACHE FILES: before vs after a rewrite differ for: 1 file(s): lxc-containers
files that define more than one profile: lxc-containers nvidia_modprobe plasmashell transmission usr.bin.man usr.sbin.dhclient
@@ -0,0 +1,11 @@
#### demo-hp 2026-11-14T10:59:56Z
failed service units now: chrony.service
chrony.service ExecStart path: path=/usr/sbin/chronyd
profile names of the rewritten cache file usr.sbin.chronyd: /usr/sbin/chronyd
## match: chrony.service (failed, ExecStart /usr/sbin/chronyd = the profile's name) -> systemctl restart chrony.service
rc=0
chrony: active; confinement /usr/sbin/chronyd (enforce)
Reference ID : 25DDD139 (37.221.209.57)
System time : 0.000319822 seconds slow of NTP time
Leap status : Normal
failed service units after: (end)
@@ -0,0 +1,10 @@
#### demo-hp 2026-11-14T11:09:32Z — the watch, 10 minutes after the LAST fresh load (10:59:22Z; the first was 10:58:44Z)
apparmor DENIED lines since 10:58:40Z: 1 — expected exactly 1: the control of 11:04:38Z
2026-11-14T12:04:38+01:00 demo-hp kernel: audit: type=1400 audit(1794654278.061:617): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/chronyd" name="/etc/shadow" pid=1879323 comm="c
BLIND SECONDS (the kernel suppressed audit lines): 2026-11-14T11:58:44+01:00 demo-hp kernel: kauditd_printk_skb: 111 callbacks supp;2026-11-14T11:59:22+01:00 demo-hp kernel: kauditd_printk_skb: 109 callbacks supp;2026-11-14T12:04:38+01:00 demo-hp kernel: kauditd_printk_skb: 228 callbacks supp;
## second channel (behaviour, not the kernel log):
guest 9201: status: running; containers 26 running, 0 unhealthy, 0 restarting, newest start 2026-11-14T09:18:59.079611972Z; init confinement lxc-9201_</var/lib/lxc>//&:lxc-9201_<-var-lib-lxc>:unconfined (enforce)
guest 9202: status: running; containers 6 running, 0 unhealthy, 0 restarting, newest start 2026-11-14T09:56:41.303141464Z; init confinement lxc-9202_</var/lib/lxc>//&:lxc-9202_<-var-lib-lxc>:unconfined (enforce)
failed units on the host: 0
chrony: active System time : 0.000691365 seconds slow of NTP time
agent: active
@@ -0,0 +1,5 @@
#### demo-hp 2026-11-14T11:04:38Z — positive control for the DENIED watch
why: the kernel rate-limits its audit lines (kauditd_printk_skb: 111 callbacks suppressed at 11:58:44 and 109 at 11:59:22 local) — the reload floods ~119 STATUS lines, and the DENIED line of the planted stale profile at 11:58:07 is NOT in the log. So a count of 0 proves nothing without this control.
control command rc=1 (cat: /etc/shadow: Permission denied)
DENIED lines the kernel log shows for the control: 1
2026-11-14T12:04:38+01:00 demo-hp kernel: audit: type=1400 audit(1794654278.061:617): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/chronyd" name="/etc/shadow" pid=1879323 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
@@ -0,0 +1,9 @@
#### guest 9201 2026-11-14T11:00:04Z
apparmor userspace: apparmor 4.1.0-1 ii
ls: cannot access '/var/cache/apparmor/*/': No such file or directory
105
inactive
cat: /sys/kernel/security/apparmor/profiles: No such file or directory
container confinement: unconfined
ls: cannot access '/sys/kernel/security/apparmor/policy/profiles': No such file or directory
7.0.14-28-pve
@@ -0,0 +1,33 @@
#### demo-hp 2026-11-14T11:09:55Z — put back as found
ls: cannot access '/root/spike-profile*': No such file or directory
ls: cannot access '/root/spike-loaded*': No such file or directory
ls: cannot access '/root/spike-refresh-at': No such file or directory
cache, running kernel folder: chrony 83417e83b538 (83417e83 = the fresh compile); old folder untouched: dbf67f16ac25
loaded: 83417e83b538
#### demo-hp 2026-11-14T11:09:55Z kernel 7.0.14-28-pve
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
loaded profiles: 119
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 6c024cf01fc0 88d69a2bb62c
LOADED-DIFFERS: file=nvidia_modprobe profile=nvidia_modprobe loaded=f4ae61617079 fresh=34f355a47446
LOADED-DIFFERS: file=plasmashell profile=plasmashell loaded=17cfa46264cd fresh=2f95a57eab51
LOADED-DIFFERS: file=transmission profile=transmission-cli loaded=04b531d8a589 fresh=d9230550cd5f
LOADED-DIFFERS: file=transmission profile=transmission-daemon loaded=f46a1d2db622 fresh=d9230550cd5f
LOADED-DIFFERS: file=transmission profile=transmission-gtk loaded=ab371958b589 fresh=d9230550cd5f
LOADED-DIFFERS: file=transmission profile=transmission-qt loaded=0077a589d81e fresh=d9230550cd5f
LOADED-DIFFERS: file=usr.bin.man profile=/usr/bin/man loaded=4ee549c2ed87 fresh=3fe9acfaeb92
LOADED-DIFFERS: file=usr.bin.man profile=man_filter loaded=6dc1077b7fa5 fresh=3fe9acfaeb92
LOADED-DIFFERS: file=usr.bin.man profile=man_groff loaded=0f6549c22ad0 fresh=3fe9acfaeb92
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/usr/lib/NetworkManager/nm-dhcp-client.action loaded=b5532441db45 fresh=99b03dd8451f
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/usr/lib/connman/scripts/dhclient-script loaded=fcadbc62c577 fresh=99b03dd8451f
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/usr/{lib/NetworkManager,libexec}/nm-dhcp-helper loaded=6ca7645b1031 fresh=99b03dd8451f
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/{,usr/}sbin/dhclient loaded=3b3e18f5a89e fresh=99b03dd8451f
files=106 cache-differs=0 loaded-differs=13 profiles-not-loaded=4 not-deterministic=1 seconds=4.562202656
## the 13 LOADED-DIFFERS above are the multi-profile files after a FRESH load (03b: same policy, another hash). To leave the box exactly as found, one cache load, as at a boot: systemctl reload apparmor
reload rc=0
#### demo-hp 2026-11-14T11:10:08Z kernel 7.0.14-28-pve
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
loaded profiles: 119
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 303716fc0b9a 03ad7998e16e
files=106 cache-differs=0 loaded-differs=0 profiles-not-loaded=4 not-deterministic=1 seconds=4.587272637
chrony active; guests: 2 running; failed units: 0
@@ -0,0 +1,19 @@
# 02-make-stale.sh — demo-hp (Tier 0). Make ONE stale profile by the real mechanism: an old compiled copy in the
# running kernel's cache folder whose time is newer than its source. No package is changed.
set -u
C=/var/cache/apparmor; CUR=$C/f6da0254.0; OLD=$C/ae1a4050.0; F=usr.sbin.chronyd
LOADED=/sys/kernel/security/apparmor/policy/profiles
echo "#### $(hostname) $(date -u +%FT%TZ) kernel $(uname -r)"
echo "## before: chrony $(systemctl is-active chrony); loaded $(cat $LOADED/usr.sbin.chronyd*/raw_sha256 | cut -c1-12); cache(cur) $(sha256sum < $CUR/$F | cut -c1-12); cache(old folder) $(sha256sum < $OLD/$F | cut -c1-12); fresh $(apparmor_parser -S -K /etc/apparmor.d/$F | sha256sum | cut -c1-12)"
echo "## source mtime $(stat -c %y /etc/apparmor.d/$F | cut -c1-19); old copy mtime $(stat -c %y $OLD/$F | cut -c1-19)"
cp -a $CUR/$F /root/spike-profile-$F.good && echo "saved the good cache file to /root/spike-profile-$F.good ($(sha256sum < /root/spike-profile-$F.good | cut -c1-12))"
cp -a $OLD/$F $CUR/$F && echo "PLANTED: the old compiled copy is now in the running kernel's folder; its time $(stat -c %y $CUR/$F | cut -c1-19) is newer than the source"
echo "## reload the way the system does at a start (systemctl reload apparmor = apparmor_parser --write-cache --replace)"
systemctl reload apparmor; echo "reload rc=$?"
echo "## after the reload: loaded $(cat $LOADED/usr.sbin.chronyd*/raw_sha256 | cut -c1-12) (dbf67f16 = the profile of chrony 4.6.1; 83417e83 = of 4.8)"
echo "## chrony still runs under the replaced profile? $(systemctl is-active chrony); its confinement: $(cat /proc/$(pidof -s chronyd)/attr/current 2>/dev/null)"
T=$(date +"%Y-%m-%d %H:%M:%S")
systemctl restart chrony; echo "restart rc=$?"; sleep 6
echo "## after a restart: chrony $(systemctl is-active chrony); result=$(systemctl show chrony -p Result --value)"
journalctl -u chrony --since "$T" --no-pager -o short-iso | tail -6
journalctl -k --since "$T" --no-pager -o short-iso | grep -i 'apparmor="DENIED"' | tail -3
@@ -0,0 +1,30 @@
# 03-refresh.sh — demo-hp. The repair candidate: load every profile from a FRESH compile, cache ignored and rewritten.
set -u
P=/sys/kernel/security/apparmor/policy/profiles; CUR=/var/cache/apparmor/f6da0254.0
snap() { # name -> loaded hash, every profile
for d in $P/*/; do echo "$(cat $d/name) $(cat $d/raw_sha256 2>/dev/null) $(cat $d/mode 2>/dev/null)"; done | sort
}
guests() {
for v in 9201 9202; do
ip=$(lxc-info -n $v -p -H); echo "guest $v: $(pct status $v) init pid $ip confinement=$(cat /proc/$ip/attr/current)"
echo " containers: $(pct exec $v -- docker ps -q | wc -l) running, $(pct exec $v -- docker ps --filter health=unhealthy -q | wc -l) unhealthy, started-at-newest=$(pct exec $v -- sh -c 'docker inspect --format "{{.State.StartedAt}}" $(docker ps -q) | sort | tail -1')"
cp=$(pct exec $v -- sh -c 'docker inspect --format "{{.State.Pid}}" felhom-controller'); hp=$(grep -l "NSpid:.*[[:space:]]$cp$" /proc/[0-9]*/status 2>/dev/null | head -1 | cut -d/ -f3)
echo " the controller container's process (host pid ${hp:-?}): confinement=$(cat /proc/${hp:-0}/attr/current 2>/dev/null)"
done
echo "VM 341 (Tester 1): $(qm status 341)"
}
echo "#### $(hostname) $(date -u +%FT%TZ)"
echo "## BEFORE"; guests; snap > /root/spike-loaded-before.txt; echo "loaded profiles: $(wc -l < /root/spike-loaded-before.txt)"
echo "chrony: $(systemctl is-active chrony); loaded chrony profile $(cat $P/usr.sbin.chronyd*/raw_sha256 | cut -c1-12); cache $(sha256sum < $CUR/usr.sbin.chronyd | cut -c1-12)"
echo "disabled profiles (must stay unloaded): $(ls /etc/apparmor.d/disable 2>/dev/null | tr '\n' ' ')"
T=$(date +"%Y-%m-%d %H:%M:%S"); date +%s > /root/spike-refresh-at
echo "## THE COMMAND: apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d"
s=$(date +%s.%N); apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d; rc=$?; e=$(date +%s.%N)
echo "rc=$rc seconds=$(echo "$e - $s" | bc)"
echo "## AFTER"; guests; snap > /root/spike-loaded-after.txt; echo "loaded profiles: $(wc -l < /root/spike-loaded-after.txt)"
echo "profiles whose loaded hash changed:"; diff <(cut -d' ' -f1,2 /root/spike-loaded-before.txt) <(cut -d' ' -f1,2 /root/spike-loaded-after.txt) | grep '^>' | sed 's/^> / /'
echo "profiles added or gone (names):"; diff <(cut -d' ' -f1 /root/spike-loaded-before.txt) <(cut -d' ' -f1 /root/spike-loaded-after.txt) | grep '^[<>]' || echo " none"
echo "modes changed:"; diff <(cut -d' ' -f1,3 /root/spike-loaded-before.txt) <(cut -d' ' -f1,3 /root/spike-loaded-after.txt) | grep '^[<>]' || echo " none"
echo "chrony profile: loaded $(cat $P/usr.sbin.chronyd*/raw_sha256 | cut -c1-12); cache $(sha256sum < $CUR/usr.sbin.chronyd | cut -c1-12) mtime $(stat -c %y $CUR/usr.sbin.chronyd | cut -c1-19); fresh $(apparmor_parser -S -K /etc/apparmor.d/usr.sbin.chronyd | sha256sum | cut -c1-12)"
echo "chrony unit, NOT restarted by the command: $(systemctl is-active chrony)"
echo "kernel lines since the command:"; journalctl -k --since "$T" --no-pager -o short-iso | grep -i apparmor | tail -5; echo "(end)"
@@ -0,0 +1,14 @@
# 03b — why did 13 other profiles change their loaded hash in 03? Compare: loaded after a CACHE load (the system's
# reload) against loaded after a FRESH load, and the cache files before/after a rewrite. Nothing is stale now.
P=/sys/kernel/security/apparmor/policy/profiles; CUR=/var/cache/apparmor/f6da0254.0
snap() { for d in $P/*/; do echo "$(cat $d/name) $(cat $d/raw_sha256 2>/dev/null)"; done | sort; }
csnap() { sha256sum $CUR/* | sed "s#$CUR/##" | awk '{print $2, $1}' | sort; }
echo "#### $(hostname) $(date -u +%FT%TZ)"
snap > /tmp/s-fresh1; csnap > /tmp/c1
systemctl reload apparmor; snap > /tmp/s-cache
apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d; snap > /tmp/s-fresh2; csnap > /tmp/c2
echo "loaded: fresh-load vs cache-load differ for: $(diff /tmp/s-fresh1 /tmp/s-cache | grep -c '^>') profile(s): $(diff /tmp/s-fresh1 /tmp/s-cache | grep '^>' | cut -d' ' -f2 | tr '\n' ' ')"
echo "loaded: fresh-load vs second fresh-load differ for: $(diff /tmp/s-fresh1 /tmp/s-fresh2 | grep -c '^>') profile(s): $(diff /tmp/s-fresh1 /tmp/s-fresh2 | grep '^>' | cut -d' ' -f2 | tr '\n' ' ')"
echo "CACHE FILES: before vs after a rewrite differ for: $(diff /tmp/c1 /tmp/c2 | grep -c '^>') file(s): $(diff /tmp/c1 /tmp/c2 | grep '^>' | cut -d' ' -f2 | tr '\n' ' ')"
echo "files that define more than one profile: $(for f in /etc/apparmor.d/*; do [ -f $f ] && [ $(apparmor_parser -N $f 2>/dev/null | wc -l) -gt 1 ] && basename $f; done | tr '\n' ' ')"
rm -f /tmp/s-fresh1 /tmp/s-cache /tmp/s-fresh2 /tmp/c1 /tmp/c2
@@ -0,0 +1,12 @@
# 04 — the narrow restart rule, by hand: a FAILED service unit whose ExecStart program is the path-name of a profile
# whose cached copy the refresh had to rewrite. (On this box: chrony, made stale in 02.)
echo "#### $(hostname) $(date -u +%FT%TZ)"
echo "failed service units now: $(systemctl list-units --type=service --state=failed --no-legend --plain | awk '{print $1}' | tr '\n' ' ')"
for u in $(systemctl list-units --type=service --state=failed --no-legend --plain | awk '{print $1}'); do
echo " $u ExecStart path: $(systemctl show -p ExecStart --value $u | grep -o 'path=[^ ;]*' | head -1)"
done
echo "profile names of the rewritten cache file usr.sbin.chronyd: $(apparmor_parser -N /etc/apparmor.d/usr.sbin.chronyd | tr '\n' ' ')"
echo "## match: chrony.service (failed, ExecStart /usr/sbin/chronyd = the profile's name) -> systemctl restart chrony.service"
systemctl restart chrony.service; echo "rc=$?"; sleep 8
echo "chrony: $(systemctl is-active chrony); confinement $(cat /proc/$(pidof -s chronyd)/attr/current)"; chronyc -n tracking | grep -E "System time|Leap|Reference"
echo "failed service units after: $(systemctl list-units --type=service --state=failed --no-legend --plain | awk '{print $1}' | tr '\n' ' ')(end)"
@@ -0,0 +1,29 @@
# detect.sh — READ ONLY. For every profile file in /etc/apparmor.d: does the profile the KERNEL has loaded equal a
# fresh compile of the file on disk? (the n100-clock-2026-11-11 method, per file AND per loaded profile)
# fresh = sha256 of `apparmor_parser -S -K <file>` (compile to stdout: no load, cache not read, not written)
# loaded = policy/profiles/*/raw_sha256 of each profile name the file defines (`apparmor_parser -N`)
# cache = sha256 of the cached binary in the running kernel's cache folder
main() {
echo "#### $(hostname) $(date -u +%FT%TZ) kernel $(uname -r)"
P=/sys/kernel/security/apparmor/policy/profiles
cur=""; for d in /var/cache/apparmor/*/; do [ "$(sha256sum < $d/.features | cut -d' ' -f1)" = "$(aa-features-abi -x 2>/dev/null | sha256sum | cut -d' ' -f1)" ] && cur=$d; done
echo "cache folder of the running kernel: ${cur:-none} all: $(ls -d /var/cache/apparmor/*/ | tr '\n' ' ')"
declare -A L; for d in $P/*/; do n=$(cat $d/name 2>/dev/null); [ -n "$n" ] && L["$n"]=$(cat $d/raw_sha256 2>/dev/null); done
echo "loaded profiles: ${#L[@]}"
n=0; stale=0; cdiff=0; notloaded=0; nd=0; t0=$(date +%s.%N)
for f in /etc/apparmor.d/*; do [ -f "$f" ] || continue; b=$(basename "$f"); [ -e /etc/apparmor.d/disable/$b ] && continue
fresh=$(apparmor_parser -S -K "$f" 2>/dev/null | sha256sum | cut -d' ' -f1); n=$((n+1))
again=$(apparmor_parser -S -K "$f" 2>/dev/null | sha256sum | cut -d' ' -f1)
# CONTROL ON THE METHOD (found 2026-11-14): a file whose two fresh compiles differ cannot be judged by a hash at all
[ "$fresh" = "$again" ] || { nd=$((nd+1)); echo "NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): $b ${fresh:0:12} ${again:0:12}"; continue; }
c="-"; [ -n "$cur" ] && [ -f "$cur$b" ] && c=$(sha256sum < "$cur$b" | cut -d' ' -f1)
[ "$c" != "-" ] && [ "$c" != "$fresh" ] && { cdiff=$((cdiff+1)); echo "CACHE-DIFFERS: $b cache=${c:0:12} fresh=${fresh:0:12} cache-mtime=$(stat -c %y "$cur$b" | cut -c1-19) source-mtime=$(stat -c %y "$f" | cut -c1-19)"; }
for name in $(apparmor_parser -N "$f" 2>/dev/null); do
l=${L["$name"]:-}
if [ -z "$l" ]; then notloaded=$((notloaded+1)); continue; fi
[ "$l" = "$fresh" ] || { stale=$((stale+1)); echo "LOADED-DIFFERS: file=$b profile=$name loaded=${l:0:12} fresh=${fresh:0:12}"; }
done
done
echo "files=$n cache-differs=$cdiff loaded-differs=$stale profiles-not-loaded=$notloaded not-deterministic=$nd seconds=$(echo "$(date +%s.%N) - $t0" | bc)"
}
main </dev/null
@@ -0,0 +1,43 @@
### demo-hp 2026-11-14T10:55:37Z — dry run of the working-copy wrapper (sha256 8d5052e4548341e0…)
os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[]
{
"refused": null,
"failed": null,
"kernel_cleanup": {
"default": "7.0.14-28-pve",
"dry_run": true,
"installed": {
"7.0.14-26-pve": [
"proxmox-kernel-7.0.14-26-pve-signed"
],
"7.0.14-28-pve": [
"proxmox-kernel-7.0.14-28-pve-signed"
]
},
"kept": {
"7.0.14-26-pve": [
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
"the kernel step's `default_before`"
],
"7.0.14-28-pve": [
"running",
"the kernel step's `to`",
"the default in grub.cfg",
"the pinned default file",
"newest installed"
]
},
"leftovers": [],
"newest": "7.0.14-28-pve",
"not_run": [],
"outcome": "dry_run",
"phase": "good",
"pinned_default": "7.0.14-28-pve",
"refusals": [],
"removal": [],
"removed": [],
"running": "7.0.14-28-pve",
"would": "nothing"
},
"pass_seconds": 0.2
}
@@ -0,0 +1,60 @@
### root@192.168.0.154 2026-11-14T10:55:38Z — dry run of the working-copy wrapper (sha256 8d5052e4548341e0…)
os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=3 keep=2 removal=1 not_run=['no-ring0-mark'] refusals=[]
{
"refused": null,
"failed": null,
"kernel_cleanup": {
"default": "7.0.14-22-pve",
"dry_run": true,
"installed": {
"7.0.14-20-pve": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"7.0.14-22-pve": [
"proxmox-kernel-7.0.14-22-pve-signed"
],
"7.0.2-6-pve": [
"proxmox-kernel-7.0.2-6-pve-signed"
]
},
"kept": {
"7.0.14-22-pve": [
"running",
"the kernel step's `to`",
"the default in grub.cfg",
"the pinned default file",
"newest installed"
],
"7.0.2-6-pve": [
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
"the kernel step's `default_before`"
]
},
"leftovers": [],
"newest": "7.0.14-22-pve",
"not_run": [
{
"code": "no-ring0-mark",
"reason": "this box has no root-owned ring-0 mark \u2014 the clean-up runs on ring 0 only in this release"
}
],
"outcome": "dry_run",
"phase": "good",
"pinned_default": "7.0.14-22-pve",
"refusals": [],
"removal": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"removed": [],
"running": "7.0.14-22-pve",
"simulation": {
"installs": [],
"rc": 0,
"removes": [
"proxmox-kernel-7.0.14-20-pve-signed"
]
},
"would": "not_run"
},
"pass_seconds": 1.4
}
@@ -0,0 +1,23 @@
== agent red-proofs (internal/osupdate), 2026-11-14T10:54:58Z
-- MUTATION 1: kernelCleanupSkip always answers "" (every 'may not run' rule removed)
--- FAIL: TestKernel_Ring0ToldNightStagesThenReboots (0.00s)
kernel_test.go:83: the reboot must end the night, last call = map[lane:slow layer:kernel mode:kernel-cleanup packages:[] release_id:kernel-cleanup ring:0 run_id:20261004T040000Z trigger:night vmid:9201]
--- FAIL: TestKernelCleanup_WhenItMayNotRun (0.00s)
kernel_test.go:489: a kernel step is due tonight (ring 0): the clean-up ran (1 call(s)); calls = guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply,kernel:kernel-status,kernel:apply,kernel:kernel-reb
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.022s
FAIL
-- MUTATION 2: 'nothing' sends a report (return l.finish in the nothing case)
--- FAIL: TestKernelCleanup_NothingAndNotRunSendNoReport (0.00s)
kernel_test.go:423: nothing: a report was made: {RunID:20261004T040000Z Layer:kernel Trigger:night Mode:kernel-cleanup Ring:1 ReleaseID:kernel-cleanup Outcome:nothing Healthy:false HealthReason: VMID:9201 Upgraded:[]
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.021s
FAIL
-- MUTATION 3: the report carries the wrapper's kernel view
--- FAIL: TestKernelCleanup_NightOrderOnceAndNoNames (0.00s)
kernel_test.go:401: the clean-up's report names a kernel (the hub reads that as a kernel step): {RunID:20261004T040000Z Layer:kernel Trigger:night Mode:kernel-cleanup Ring:0 ReleaseID:kernel-cleanup Outcome:cleaned Healthy:true HealthReason:removed 1 old k
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.010s
FAIL
-- the real file:
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 4.973s
@@ -0,0 +1,74 @@
== MUTATION: K1 removed (the rule's own check of the removal set)
elif by_name[n] in protected or by_name[n] == newest: -> elif False:
tests: 7 mutant: Ran 7 tests in 0.119s | FAILED (failures=7) real: Ran 7 tests in 0.116s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : running: {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running':
AssertionError: 0 != 2 : from: {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7
== MUTATION: the simulation check removed (S1/S2)
if removal and not [x for x in refuse if x["code"] in ("K1", "N1")]: -> if False:
tests: 3 mutant: Ran 3 tests in 0.055s | FAILED (failures=3) real: Ran 3 tests in 0.047s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
== MUTATION: the phase check removed (P1)
elif phase != "good": -> elif False:
tests: 1 mutant: Ran 1 test in 0.019s | FAILED (failures=1) real: Ran 1 test in 0.072s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'staged', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.1
== MUTATION: the ring-0 check removed
if not ring0 and not not_run: -> if False:
tests: 1 mutant: Ran 1 test in 0.021s | FAILED (failures=1) real: Ran 1 test in 0.047s | OK
RED on the mutant, GREEN on the real file
AssertionError: 'removed' != 'not_run'
== MUTATION: the flag check removed (P2)
if flag: -> if False:
tests: 1 mutant: Ran 1 test in 0.021s | FAILED (failures=1) real: Ran 1 test in 0.022s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '7.0.14-26-pve', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'runni
== MUTATION: the default check removed (P3)
if pinned is None or pinned != dflt or pinned not in images: -> if False:
tests: 1 mutant: Ran 1 test in 0.024s | FAILED (failures=1) real: Ran 1 test in 0.044s | OK
RED on the mutant, GREEN on the real file
AssertionError: 3 != 2 : {'failed': {'rc': 3, 'reason': 'the default in grub.cfg moved from 7.0.14-28-pve to 7.0.14-26-pve', 'step': 'verify'}, 'kernel': {'default': '7.0.14-26-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': '
== MUTATION: the not-on-newest rule removed
if running and newest and running in images and running != newest: -> if False:
tests: 1 mutant: Ran 1 test in 0.032s | FAILED (failures=1) real: Ran 1 test in 0.028s | OK
RED on the mutant, GREEN on the real file
AssertionError: 'removed' != 'not_run'
== MUTATION: the fell-back rule removed
elif phase in KERNEL_ENDED_NOT_GOOD: -> elif False:
tests: 1 mutant: Ran 1 test in 0.035s | FAILED (failures=1) real: Ran 1 test in 0.101s | OK
RED on the mutant, GREEN on the real file
AssertionError: 'step-not-good' not found in ['not-on-newest'] : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'fell_back', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at'
== MUTATION: the plan-names check removed (C1)
if mode == "kernel-cleanup": -> if False:
tests: 1 mutant: Ran 1 test in 0.020s | FAILED (failures=1) real: Ran 1 test in 0.001s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
== MUTATION: the no-state-file rule removed
if raw_state is None or not st: -> if False:
tests: 1 mutant: Ran 1 test in 0.036s | FAILED (failures=1) real: Ran 1 test in 0.031s | OK
RED on the mutant, GREEN on the real file
AssertionError: 2 != 0 : {'kernel_cleanup': {'default': '7.0.14-28-pve', 'dry_run': False, 'installed': {'7.0.14-20-pve': ['proxmox-kernel-7.0.14-20-pve-signed'], '7.0.14-22-pve': ['proxmox-kernel-7.0.14-22-pve-signed'], '7.0.14-2
== MUTATION: the 20-hour window removed
if times and self.r.now() - max(times) < KERNEL_MIN_GAP: -> if False:
tests: 1 mutant: Ran 1 test in 0.022s | FAILED (failures=1) real: Ran 1 test in 0.016s | OK
RED on the mutant, GREEN on the real file
AssertionError: 'removed' != 'not_run'
== MUTATION: the dry run purges (dry-run verdict removed)
if dry: -> if False:
tests: 1 mutant: Ran 1 test in 0.022s | FAILED (errors=1) real: Ran 1 test in 0.049s | OK
RED on the mutant, GREEN on the real file
KeyError: 'would'
== MUTATION: repair-first removed
if not dry and not not_run: -> if False:
tests: 1 mutant: Ran 1 test in 0.032s | FAILED (errors=1) real: Ran 1 test in 0.031s | OK
RED on the mutant, GREEN on the real file
KeyError: 'repair'
== MUTATION: the after-purge proof removed
if problems: -> if False:
tests: 1 mutant: Ran 1 test in 0.024s | FAILED (failures=1) real: Ran 1 test in 0.022s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 3 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
14 mutations, 0 not a red-proof
@@ -0,0 +1,25 @@
== agent red-proofs, profile refresh, 2026-11-14T11:04:02Z
-- MUTATION 1: the appliance check removed
--- FAIL: TestProfileRefresh_ByoHostIsNeverAsked (0.00s)
profilerefresh_test.go:54: a BYO host was asked: "refreshed" [map[layer:host mode:profile-refresh release_id:profile-refresh]]
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.010s
FAIL
-- MUTATION 2: the gate's answer ignored (always go on)
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate [build failed]
FAIL
-- MUTATION 3: the plan names a unit
--- FAIL: TestProfileRefresh_OneCallNoNamesNoHubReport (0.00s)
profilerefresh_test.go:42: the plan carries more than release_id, layer and mode: map[layer:host mode:profile-refresh release_id:profile-refresh units:[chrony.service]]
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.011s
FAIL
-- the real file:
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.009s
-- MUTATION 2, done again (the first try did not compile, so it proved nothing): the gate's answer ignored
--- FAIL: TestProfileRefresh_WaitsForTheHeavyOpGate (0.00s)
profilerefresh_test.go:75: gate tries 1, released 0, calls 1
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.012s
FAIL
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate (cached)
@@ -0,0 +1,53 @@
== MUTATION: the restart rule widened: any failed unit restarts (the program check removed)
if prog not in programs: -> if False:
tests: 2 mutant: Ran 2 tests in 0.010s | FAILED (errors=2) real: Ran 2 tests in 0.006s | OK
RED on the mutant, GREEN on the real file
KeyError: '/usr/sbin/postfix'
KeyError: '/usr/sbin/chronyd'
== MUTATION: the stale-copy condition removed: every profile file counts, not only a rewritten one
for key in res["rewritten"]: -> for key in after:
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: Lists differ: ['chrony.service'] != []
== MUTATION: a profile name that is not a path counts as a program
if PROFILE_PATH_NAME_RE.match(n): -> if True:
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: Tuples differ: (0, ['nvidia.service']) != (0, [])
== MUTATION: once-per-kernel removed
needed = rec.get("kernel") != kernel -> needed = True
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: Tuples differ: (0, 'refreshed', 2) != (0, 'not_needed', 1)
== MUTATION: the cache is not skipped (the system's own reload)
PROFILE_REFRESH_CMD = ["apparmor_parser", "--replace", "--skip-read-cache", "--write-cache", "--", PROFILE_DIR -> PROFILE_REFRESH_CMD = ["apparmor_parser", "--replace", "--write-cache", "--", PROFILE_DIR]
tests: 1 mutant: Ran 1 test in 0.006s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: ('host', ['apparmor_parser', '--replace', '--skip-read-cache', '--write-cache', '--', '/etc/apparmor.d']) not found in [('host', ['uname', '-r']), ('host', ['apparmor_parser', '--replace', '--write-cache', '--', '/
== MUTATION: a failed parser still writes the record
r.log(f"os-apply: FAILED rc={rc} step=profile-refresh (no record written: the next start tries again)") -> r.log(f"os-apply: FAILED rc={rc} step=profile-refresh (no record written: the next start tries again)")
tests: 1 mutant: Ran 1 test in 0.006s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: '/var/lib/felhom-os-apply/profile-refresh.json' unexpectedly found in {'/var/lib/felhom-os-apply/profile-refresh.json': '{"kernel": "7.0.14-28-pve"}'}
== MUTATION: the plan may name things (F1 removed)
for k in ("packages", "allow_new", "signed", "select", "undo", "snapshot", "profiles", "units", "force"): -> for k in ("packages", "allow_new", "signed", "select", "undo", "snapshot", "profiles", "units", "force"):
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.001s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : {'layer': 'host', 'mode': 'profile-refresh', 'pass_seconds': 0.0, 'profile_refresh': {'dry_run': False, 'kernel': '7.0.14-28-pve', 'last_refresh_at': None, 'last_refresh_kernel': None, 'not_restarted': [],
== MUTATION: the appliance rule removed
self.check_appliance() -> return self.profile_refresh(plan)
tests: 1 mutant: Ran 1 test in 0.005s | FAILED (errors=1) real: Ran 1 test in 0.000s | OK
RED on the mutant, GREEN on the real file
TypeError: 'NoneType' object is not subscriptable
== MUTATION: the dry run refreshes
if dry: -> if False:
tests: 1 mutant: Ran 1 test in 0.006s | FAILED (errors=1) real: Ran 1 test in 0.000s | OK
RED on the mutant, GREEN on the real file
KeyError: 'would'
== MUTATION: the restart cap removed
if len(res["restarted"]) >= PROFILE_RESTART_MAX: -> if False:
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: Tuples differ: (0, 5, 0) != (0, 3, 2)
10 mutations, 0 not a red-proof
@@ -0,0 +1,4 @@
== the published binary, downloaded with no login 2026-11-14T11:13:30Z
6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92
agent-0.174.0: go1.27.4
felhom-agent 0.174.0
@@ -0,0 +1,2 @@
felhom: agent felhom-agent 0.173.0 wrapper sha a5f67f015fbc bundle 0.172.1 3e7acaee4500 host_id tester-1-d70be4
demo-hp: agent felhom-agent 0.173.0 wrapper sha a5f67f015fbc bundle 0.172.1 3e7acaee4500 host_id demo-hp-bb76ea
@@ -0,0 +1,8 @@
== tester-1-d70be4 2026-11-14T11:13:39Z agent_update 0.174.0
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=3a80f37e50d321ee00db919bb6c659fc expires=2026-11-14T11:58:39Z
uploaded signed op to the hub jobs queue
rc=0
== demo-hp-bb76ea 2026-11-14T11:13:39Z agent_update 0.174.0
signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=c9fb994bd9952af35d137219442067b4 expires=2026-11-14T11:58:39Z
uploaded signed op to the hub jobs queue
rc=0
@@ -0,0 +1,13 @@
### felhom 2026-11-14T11:18:17Z
2026-11-14T12:14:53+01:00 felhom felhom-agent[51939]: time=2026-11-14T12:14:53.096+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=df59dd26428b9120 op=agent_update
2026-11-14T12:14:55+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:14:55.235+01:00 level=INFO msg="felhom-agent daemon starting" version=0.174.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-14T12:14:55+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:14:55.356+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
2026-11-14T12:15:56+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:15:56.990+01:00 level=WARN msg="selfupdate: update committed" version=0.174.0 wrapper=""
active
0
### demo-hp 2026-11-14T11:18:21Z
2026-11-14T12:18:00+01:00 demo-hp felhom-agent[535281]: time=2026-11-14T12:18:00.785+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=0ef54dd1311c3a04 op=agent_update
2026-11-14T12:18:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:18:02.833+01:00 level=INFO msg="felhom-agent daemon starting" version=0.174.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
2026-11-14T12:18:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:18:02.956+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
active
0
@@ -0,0 +1,8 @@
== tester-1-d70be4 2026-11-14T11:19:06Z agent_config_update 0.174.0
signed: op=agent_config_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=f632b7fb9a3ae1667d0050ce176a86d5 expires=2026-11-14T12:04:06Z
uploaded signed op to the hub jobs queue
rc=0
== demo-hp-bb76ea 2026-11-14T11:19:06Z agent_config_update 0.174.0
signed: op=agent_config_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=61a7b130297809cab3b5e26dfc1f7288 expires=2026-11-14T12:04:06Z
uploaded signed op to the hub jobs queue
rc=0
@@ -0,0 +1,24 @@
### felhom 2026-11-14T11:30:25Z
2026-11-14T12:14:55+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:14:55.356+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
2026-11-14T12:30:09+01:00 felhom felhom-os-apply[3433197]: os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False
2026-11-14T12:30:09+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:09.431+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False"
2026-11-14T12:30:09+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:09.431+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.174.0 bundle="{\"agent_version\": \"0.174.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261114T113008Z-before-0.174.0\", \"removed\": [], \"same\": 30,
2026-11-14T12:30:10+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:10.471+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded=""
2026-11-14T12:30:10+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:10.471+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=de2c899634da22aa op=agent_config_update
2026-11-14T12:30:11+01:00 felhom felhom-os-apply[3433509]: os-apply: PROFILE REFRESH DONE kernel=7.0.14-22-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]
2026-11-14T12:30:11+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:11.181+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PROFILE REFRESH DONE kernel=7.0.14-22-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]"
2026-11-14T12:30:11+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:11.181+01:00 level=INFO msg="osupdate: profile refresh — security profiles reloaded from a fresh compile" mode=profile-refresh why="config bundle installed" kernel=7.0.14-22-pve rewritten=1 wrapper_seconds=0.5 took=700ms
wrapper sha 9d03203766040a97 record: { "at": "2026-11-14T11:30:11Z", "kernel": "7.0.14-22-pve", "restarted": [], "rewritten": [ "f6da0254.0/lxc-containers" ]}
chrony active; failed units 0; guests running 1
### demo-hp 2026-11-14T11:33:26Z
2026-11-14T12:18:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:18:02.956+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
2026-11-14T12:33:16+01:00 demo-hp felhom-os-apply[1980789]: os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False
2026-11-14T12:33:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:16.771+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False"
2026-11-14T12:33:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:16.771+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.174.0 bundle="{\"agent_version\": \"0.174.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261114T113315Z-before-0.174.0\", \"removed\": [], \"same\": 30,
2026-11-14T12:33:17+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:17.923+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded=""
2026-11-14T12:33:17+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:17.923+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=524eef508b29fa0c op=agent_config_update
2026-11-14T12:33:18+01:00 demo-hp felhom-os-apply[1981111]: os-apply: PROFILE REFRESH DONE kernel=7.0.14-28-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]
2026-11-14T12:33:18+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:18.531+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PROFILE REFRESH DONE kernel=7.0.14-28-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]"
2026-11-14T12:33:18+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:18.532+01:00 level=INFO msg="osupdate: profile refresh — security profiles reloaded from a fresh compile" mode=profile-refresh why="config bundle installed" kernel=7.0.14-28-pve rewritten=1 wrapper_seconds=0.5 took=600ms
wrapper sha 9d03203766040a97 record: { "at": "2026-11-14T11:33:18Z", "kernel": "7.0.14-28-pve", "restarted": [], "rewritten": [ "f6da0254.0/lxc-containers" ]}
chrony active; failed units 0; guests running 2
@@ -0,0 +1,57 @@
### felhom (Tester 1) 2026-11-14T11:30:43Z — the dry run through the released agent and its own sudo line
agent felhom-agent 0.174.0; wrapper 9d03203766040a97; mark: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false}
time=2026-11-14T12:30:45.231+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=3 keep=2 removal=1 not_run=['no-ring0-mark'] refusals=[]"
--- kernel clean-up — DRY RUN (nothing was changed) ---
{
"default": "7.0.14-22-pve",
"dry_run": true,
"installed": {
"7.0.14-20-pve": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"7.0.14-22-pve": [
"proxmox-kernel-7.0.14-22-pve-signed"
],
"7.0.2-6-pve": [
"proxmox-kernel-7.0.2-6-pve-signed"
]
},
"kept": {
"7.0.14-22-pve": [
"running",
"the kernel step's `to`",
"the default in grub.cfg",
"the pinned default file",
"newest installed"
],
"7.0.2-6-pve": [
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
"the kernel step's `default_before`"
]
},
"leftovers": [],
"newest": "7.0.14-22-pve",
"not_run": [
{
"code": "no-ring0-mark",
"reason": "this box has no root-owned ring-0 mark — the clean-up runs on ring 0 only in this release"
}
],
"outcome": "dry_run",
"phase": "good",
"pinned_default": "7.0.14-22-pve",
"refusals": [],
"removal": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"removed": [],
"running": "7.0.14-22-pve",
"simulation": {
"installs": [],
"rc": 0,
"removes": [
"proxmox-kernel-7.0.14-20-pve-signed"
]
},
"would": "not_run"
}
@@ -0,0 +1,116 @@
OPERATOR'S YES for the temporary ring-0 mark on Tester 1: given in the session, 2026-11-14 (the STOP's first question).
#### felhom (Tester 1) 2026-11-14T14:16:19Z agent felhom-agent 0.174.0 wrapper 9d03203766040a97
busy? vzdump 0, apt 0, guest status: running, containers 10 running / 0 unhealthy
## BEFORE
kernels:
running 7.0.14-22-pve; grub.cfg default: gnulinux-7.0.14-22-pve-advanced
sha256: pinned d96c890b9749 state 8008e8e2192c gen01 f2f56d59c635 gen42 c34907ce6859 flag [felhom_next= ]
/ free bytes: 39449448448; /boot kernels: 7.0.14-20-pve 7.0.14-22-pve 7.0.2-6-pve ; /lib/modules: 7.0.14-20-pve 7.0.14-22-pve 7.0.2-6-pve
one-shot entries in grub.cfg: felhom-oneshot-7.0.14-20-pve felhom-oneshot-7.0.14-22-pve felhom-oneshot-7.0.2-6-pve
## the ring-0 mark, set by hand FOR THIS TEST (Tester 1 is ring 1; the clean-up runs on ring 0 only in this release)
mark before: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false} (root:root 644)
mark set: { "host_id": "tester-1-d70be4", "ring0_slow_lane": true} (root:root 644)
## SCENARIO 1 — the clean-up
wrapper exit code: 0
os-apply: REPAIR configured=0 journal=0 fixed=0
os-apply: KERNEL CLEANUP running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=3 keep=2 removal=1 not_run=[] refusals=[]
os-apply: KERNEL CLEANUP DONE removed=1 (proxmox-kernel-7.0.14-20-pve-signed) kept=7.0.14-22-pve, 7.0.2-6-pve gained=1124024320 B seconds=8.5 leftovers=0; default 7.0.14-22-pve (proved from grub.cfg)
{
"refused": null,
"failed": null,
"repair": {
"clean_after": true,
"fixed": 0,
"half_configured_before": 0,
"journal_before": 0
},
"pass_seconds": 10.1,
"kernel_cleanup": {
"outcome": "removed",
"not_run": [],
"refusals": [],
"kept": {
"7.0.14-22-pve": [
"running",
"the kernel step's `to`",
"the default in grub.cfg",
"the pinned default file",
"newest installed"
],
"7.0.2-6-pve": [
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
"the kernel step's `default_before`"
]
},
"removal": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"removed": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"bytes_gained": 1124024320,
"leftovers": [],
"seconds": 8.5,
"simulation": {
"installs": [],
"rc": 0,
"removes": [
"proxmox-kernel-7.0.14-20-pve-signed"
]
}
}
}
## AFTER
kernels:
running 7.0.14-22-pve; grub.cfg default: gnulinux-7.0.14-22-pve-advanced
sha256: pinned d96c890b9749 state 8008e8e2192c gen01 f2f56d59c635 gen42 c34907ce6859 flag [felhom_next= ]
/ free bytes: 40573468672; /boot kernels: 7.0.14-22-pve 7.0.2-6-pve ; /lib/modules: 7.0.14-22-pve 7.0.2-6-pve
one-shot entries in grub.cfg: felhom-oneshot-7.0.14-22-pve felhom-oneshot-7.0.2-6-pve
## SCENARIO 2 — the same call again
wrapper exit code: 0
os-apply: REPAIR configured=0 journal=0 fixed=0
os-apply: KERNEL CLEANUP running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[]
os-apply: KERNEL CLEANUP nothing to remove
{
"refused": null,
"failed": null,
"repair": {
"clean_after": true,
"fixed": 0,
"half_configured_before": 0,
"journal_before": 0
},
"pass_seconds": 0.1,
"kernel_cleanup": {
"outcome": "nothing",
"not_run": [],
"refusals": [],
"kept": {
"7.0.14-22-pve": [
"running",
"the kernel step's `to`",
"the default in grub.cfg",
"the pinned default file",
"newest installed"
],
"7.0.2-6-pve": [
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
"the kernel step's `default_before`"
]
},
"removal": [],
"removed": [],
"bytes_gained": null,
"leftovers": [],
"seconds": null,
"simulation": null
}
}
## the mark put back
mark now: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false} (root:root 644); saved copy: ls: cannot access '/root/os-trust.json.before-2a': No such file or directory
## and with the mark back, the program answers:
wrapper exit code: 0
os-apply: KERNEL CLEANUP running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=2 keep=2 removal=0 not_run=['no-ring0-mark'] refusals=[]
"outcome": "not_run",
"code": "no-ring0-mark",
guest status: running, containers 10 running / 0 unhealthy; dpkg audit: []
@@ -0,0 +1,7 @@
#### the N100 (demo-felhom) BEFORE the release — reads only; DooPlex clock 2026-11-14T11:32:03.295Z
offset of the N100 to DooPlex: -32.55 s (negative = the N100 is behind); the ssh round trip was 0.35 s
agent felhom-agent 0.173.0; kernel 7.0.14-28-pve; chrony: failed (result exit-code); failed units: chrony.service
chrony profile: loaded dbf67f16ac25, cache (7.0.14 folder) dbf67f16ac25, fresh compile 83417e83b538
busy? vzdump 0, apt 0, task lock files: pve-migrate-9201 pve-storage-local-lvm ; guest: 9201 running demo-felhom ;
kernels:
/dev/mapper/pve-root 94G 35G 55G 40% /
@@ -0,0 +1,19 @@
#### demo-hp 2026-11-14T11:33:49Z agent felhom-agent 0.174.0
## 0. the box as it is
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
would: nothing | refusals: [] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
## 1. scenario 3 — a step that is not over: phase set to 'judging' in the state file
would: refuse | refusals: ["P1 a kernel step is in phase 'judging' — not ended (or not a phase this wrapper knows)"] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
put back:
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
## 2. scenario 4a — the one-shot flag names the kept older kernel
would: refuse | refusals: ["P2 the one-shot flag names '7.0.14-26-pve'"] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
put back:
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
## 3. scenario 4b — the pinned default file names another kernel than grub.cfg
would: refuse | refusals: ["P3 the pinned default '7.0.14-26-pve' and grub.cfg's default '7.0.14-28-pve' do not name the same installed kerne"] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
put back:
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
## 4. as found?
IDENTICAL to step 0: state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
ls: cannot access '/root/2a-*': No such file or directory
@@ -0,0 +1,4 @@
== demo-felhom-8363b5 2026-11-14T11:34:09Z agent_update 0.174.0
signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=20e3bd9eb794fa7137294ed8ca1390c3 expires=2026-11-14T12:19:09Z
uploaded signed op to the hub jobs queue
rc=0
@@ -0,0 +1,4 @@
== demo-felhom-8363b5 2026-11-14T11:37:49Z agent_config_update 0.174.0
signed: op=agent_config_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=29c5412f563d8131d25c69aa55f6fd17 expires=2026-11-14T12:22:49Z
uploaded signed op to the hub jobs queue
rc=0
@@ -0,0 +1,59 @@
#### the N100 (demo-felhom) AFTER the release — DooPlex clock 2026-11-14T11:52:11Z
offset of the N100 to DooPlex: +0.177 s (before: -32.55 s); the ssh round trip was 0.36 s
agent felhom-agent 0.174.0; wrapper 9d03203766040a97; chrony: active; failed units: 0
Reference ID : C1E3C502 (193.227.197.2)
System time : 0.000001782 seconds slow of NTP time
Last offset : +0.002664797 seconds
Leap status : Normal
chrony profile: loaded 83417e83b538, cache 83417e83b538, fresh 83417e83b538
record: { "at": "2026-11-14T11:51:13Z", "kernel": "7.0.14-28-pve", "restarted": [ "chrony.service" ], "rewritten": [ "f6da0254.0/lxc-containers", "f6da0254.0/usr.sbin.chronyd" ]}
guest: status: running; containers 9 running, 0 unhealthy, newest container start 2026-11-14T11:52:10.648640606Z
## the full minute, the host journal (agent, wrapper, chrony, systemd):
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949497]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949539]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- pgrep -x dhclient
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949539]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949542]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- ip -4 -o addr show dev eth0
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949542]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949539]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949542]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949553]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker exec felhom-controller cat /opt/docker/felhom-controller/controller.yaml
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949553]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949573]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949573]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949573]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949586]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949586]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949553]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949586]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949608]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949608]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949608]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949611]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949611]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949611]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949619]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' cloudflared
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949619]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:08+01:00 demo-felhom sudo[1949619]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:09+01:00 demo-felhom sudo[1949801]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts115109-host-facts.json
2026-11-14T12:51:09+01:00 demo-felhom sudo[1949801]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949801]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949874]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949874]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949874]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:10+01:00 demo-felhom sshd-session[1949878]: Connection closed by 127.0.0.1 port 50390
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949879]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949879]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949879]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949882]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949882]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949882]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949886]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949886]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
## the controller in the guest, the same minutes (WARN/ERROR lines, and any line on time):
(end)
## the hub, events of demo-felhom since 11:50Z:
(end)
#### the N100, 2026-11-14T14:23:46Z — 2.5 hours after the repair
offset of the N100 to DooPlex: +0.178 s; the ssh round trip was 0.36 s
chrony: active since Sat 2026-11-14 12:51:13 CET; System time : 0.000158301 seconds fast of NTP time
failed units: 0; guest containers: 9 running, 0 unhealthy; controller boot-gate lines since: 0
@@ -0,0 +1,101 @@
#### demo-felhom 2026-11-14T11:52:28Z — which container started near the clock step?
2026-11-14T04:49:14.901603074Z restarts=0 status=running health=healthy prowlarr
2026-11-14T09:18:23.620814926Z restarts=0 status=running health=healthy felhom-controller
2026-11-14T11:52:05.808655933Z restarts=0 status=running health=healthy radarr
2026-11-14T11:52:09.614378919Z restarts=0 status=running health=healthy sonarr
2026-11-14T11:52:10.648640606Z restarts=0 status=running health=healthy qbittorrent
## docker events 11:50–11:54Z (start/die/restart/health):
2026-11-14T11:52:00.406970341Z container kill fc2cd3bd36b94bbae38a99e8b476cf40b9e6fd142811f067647ad0f96077fd99 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com.docker.compose.co
2026-11-14T11:52:05.629973236Z container die fc2cd3bd36b94bbae38a99e8b476cf40b9e6fd142811f067647ad0f96077fd99 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com.docker.compose.con
2026-11-14T11:52:05.927443897Z container start e7b713a45c209013c5bc914d4f3cb9c8697627e3e6e1cb1aba6f99a7679a9de9 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com.docker.compose.c
2026-11-14T11:52:06.175065238Z container kill e5351e8bed94fabe527f7cb26760a106f6ae1eb0c4a577d3c1cd5b03fa3b6266 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com.docker.compose.co
2026-11-14T11:52:09.418552075Z container die e5351e8bed94fabe527f7cb26760a106f6ae1eb0c4a577d3c1cd5b03fa3b6266 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com.docker.compose.con
2026-11-14T11:52:09.752602177Z container start 90d435fcdb3ed6e1731469312725c7edae2f6363da11999c0fcb34fd47fe7452 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com.docker.compose.c
2026-11-14T11:52:09.977460590Z container kill 82bd9f8caf4124d1fff6b9b9fa1799088805815c63bb6835c9557c160ef51b3a (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com.docker.compose.co
2026-11-14T11:52:10.407038534Z container die 82bd9f8caf4124d1fff6b9b9fa1799088805815c63bb6835c9557c160ef51b3a (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com.docker.compose.con
2026-11-14T11:52:10.787260774Z container start 16e8e756487f2ea6ede83737f01b6c707292aa1b2f9004d3d71d11bd0131b8a1 (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com.docker.compose.c
2026-11-14T11:52:11.003795092Z container health_status: healthy e7b713a45c209013c5bc914d4f3cb9c8697627e3e6e1cb1aba6f99a7679a9de9 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com
2026-11-14T11:52:14.829102987Z container health_status: healthy 90d435fcdb3ed6e1731469312725c7edae2f6363da11999c0fcb34fd47fe7452 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com
2026-11-14T11:52:15.847713728Z container health_status: healthy 16e8e756487f2ea6ede83737f01b6c707292aa1b2f9004d3d71d11bd0131b8a1 (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com
## the controller around it:
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml
2026/11/14 11:52:11 [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (1 storage path(s))
2026/11/14 11:52:12 [INFO] [stacks] Stack sonarr post-start status:
2026/11/14 11:52:12 [INFO] [stacks] sonarr lscr.io/linuxserver/sonarr:4.0.20@sha256:f247545d23ba8b233d6604575347e48a623fe6ad75dda02348bf81917f3b5c06 running Up 3 seconds (health: starting)
2026/11/14 11:52:14 [INFO] [stacks] Stack qbittorrent post-start status:
2026/11/14 11:52:14 [INFO] [stacks] qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1@sha256:92bfd78d731e254ba64f62b77b0696add6447a33642ee5feda106a26b285aa7f running Up 3 seconds (health: starting)
2026/11/14 11:52:17 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
2026/11/14 11:52:27 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
2026/11/14 11:52:27 [INFO] Health probes: 3 ok (of 3 probed)
2026/11/14 11:52:37 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
2026/11/14 11:52:47 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
2026/11/14 11:50:24 [INFO] [scheduler] Running job: stack-scan
2026/11/14 11:50:24 [INFO] [scheduler] Running job: hold-report
2026/11/14 11:50:24 [INFO] [scheduler] Job hold-report completed (took 0s)
2026/11/14 11:50:24 [INFO] [stacks] ScanStacks complete: 68 stacks found (5 deployed, 63 available)
2026/11/14 11:50:24 [INFO] [scheduler] Job stack-scan completed (took 87ms)
2026/11/14 11:50:24 [INFO] [scheduler] Running job: agent-channel-health
2026/11/14 11:50:25 [INFO] [scheduler] Job agent-channel-health completed (took 890ms)
2026/11/14 11:51:14 [INFO] Health probes: 1 ok (of 1 probed)
2026/11/14 11:51:57 [INFO] [scheduler] Running job: hold-report
2026/11/14 11:51:57 [INFO] [scheduler] Job hold-report completed (took 0s)
2026/11/14 11:51:57 [INFO] Health probes: 4 ok (of 4 probed)
2026/11/14 11:51:57 [INFO] [scheduler] Running job: agent-channel-health
2026/11/14 11:51:58 [INFO] [scheduler] Job agent-channel-health completed (took 818ms)
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/hdd_1] before recreating drive-backed apps
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app radarr (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:00 [INFO] [stacks] Stopping stack: radarr
2026/11/14 11:52:05 [INFO] [stacks] Stack radarr stopped successfully (took 5.3s)
2026/11/14 11:52:05 [INFO] [stacks] Starting stack: radarr
2026/11/14 11:52:05 [INFO] [stacks] media_library radarr: per-drive binds 0 []; drives not bound: []
2026/11/14 11:52:05 [INFO] [stacks] Stack radarr started successfully (took 0.3s)
2026/11/14 11:52:06 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app sonarr (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:06 [INFO] [stacks] Stopping stack: sonarr
2026/11/14 11:52:09 [INFO] [stacks] Stack radarr post-start status:
2026/11/14 11:52:09 [INFO] [stacks] radarr lscr.io/linuxserver/radarr:6.4.4@sha256:adb6c09d6b729ea5e642c99cea35af72702ef476bf4763f153299ac5db9f0b4f running Up 3 seconds (health: starting)
2026/11/14 11:52:09 [INFO] [stacks] Stack sonarr stopped successfully (took 3.4s)
2026/11/14 11:52:09 [INFO] [stacks] Starting stack: sonarr
2026/11/14 11:52:09 [INFO] [stacks] media_library sonarr: per-drive binds 0 []; drives not bound: []
2026/11/14 11:52:09 [INFO] [stacks] Stack sonarr started successfully (took 0.3s)
2026/11/14 11:52:09 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app qbittorrent (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:09 [INFO] [stacks] Stopping stack: qbittorrent
2026/11/14 11:52:10 [INFO] [stacks] Stack qbittorrent stopped successfully (took 0.5s)
2026/11/14 11:52:10 [INFO] [stacks] Starting stack: qbittorrent
2026/11/14 11:52:10 [INFO] [stacks] media_library qbittorrent: per-drive binds 0 []; drives not bound: []
2026/11/14 11:52:10 [INFO] [stacks] Stack qbittorrent started successfully (took 0.3s)
2026/11/14 11:52:10 [INFO] [gate] boot 1794112550-1438: re-syncing FileBrowser mounts against the live binds
2026/11/14 11:52:10 [INFO] [settings] Settings saved
2026/11/14 11:52:11 [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (1 storage path(s))
2026/11/14 11:52:12 [INFO] [stacks] Stack sonarr post-start status:
2026/11/14 11:52:12 [INFO] [stacks] sonarr lscr.io/linuxserver/sonarr:4.0.20@sha256:f247545d23ba8b233d6604575347e48a623fe6ad75dda02348bf81917f3b5c06 running Up 3 seconds (health: starting)
#### demo-felhom 2026-11-14T11:53:19Z — did anything else read the clock step as a restart?
crash guard: {"armed":true,"boot_id":"72899d26-2524-4e06-9e8e-8099775d1e51","config":{"LIMIT":3,"PANIC_SECONDS":10,"REARM_HOURS":24,"WINDOW_MINUTES":60},"kernel_panic":10,"last_boot_at":"2026-11-08T04:35:18Z","last_boot_unclean":false,"tripped":false,"unclean_boots":[],"unclean_boots_24h":0,"unclean_boots_in_window":0,"updated_at":"2026-11-14T11:03:05Z","version":1}
uptime since: 2026-11-08 05:35:50; /proc/stat btime: btime 1794112550 = 2026-11-08T04:35:50Z; boot_id 72899d26-2524-4e06-9e8e-8099775d1e51
agent journal since the step (WARN/ERROR, boot, power, restart words):
2026-11-14T12:51:56+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:51:56.457+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
2026-11-14T12:52:16+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:52:16.469+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
2026-11-14T12:52:36+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:52:36.511+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
2026-11-14T12:52:56+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:52:56.477+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
2026-11-14T12:53:16+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:53:16.525+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
(end)
kernel step state phase: good
controller boot-gate lines today:
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/hdd_1] before recreating drive-backed apps
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app radarr (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:06 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app sonarr (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:09 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app qbittorrent (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:10 [INFO] [gate] boot 1794112550-1438: re-syncing FileBrowser mounts against the live binds
@@ -0,0 +1,19 @@
#### demo-felhom (the N100) 2026-11-14T11:53:34Z — the DRY RUN through the released agent: what tonight removes. Nothing is changed.
time=2026-11-14T12:53:35.431+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]"
would: remove | not_run: [] | refusals: []
installed: ['7.0.14-20-pve', '7.0.14-22-pve', '7.0.14-23-pve', '7.0.14-24-pve', '7.0.14-25-pve', '7.0.14-26-pve', '7.0.14-28-pve', '7.0.2-6-pve']
KEPT:
7.0.14-26-pve ["the kernel step's `from` (the last kernel that booted well; a self-revert boots it)", "the kernel step's `default_before`"]
7.0.14-28-pve ['running', "the kernel step's `to`", 'the default in grub.cfg', 'the pinned default file', 'newest installed']
WILL BE REMOVED ( 6 ):
proxmox-kernel-7.0.14-20-pve-signed
proxmox-kernel-7.0.14-22-pve-signed
proxmox-kernel-7.0.14-23-pve-signed
proxmox-kernel-7.0.14-24-pve-signed
proxmox-kernel-7.0.14-25-pve-signed
proxmox-kernel-7.0.2-6-pve-signed
simulation: {'installs': [], 'rc': 0, 'removes': ['proxmox-kernel-7.0.14-20-pve-signed', 'proxmox-kernel-7.0.14-22-pve-signed', 'proxmox-kernel-7.0.14-23-pve-signed', 'proxmox-kernel-7.0.14-24-pve-signed', 'proxmox-kernel-7.0.14-25-pve-signed', 'proxmox-kernel-7.0.2-6-pve-signed']}
sizes now: /lib/modules 7.6G, /boot 906M, / free 55G
972M /lib/modules/7.0.14-20-pve;972M /lib/modules/7.0.14-22-pve;972M /lib/modules/7.0.14-23-pve;972M /lib/modules/7.0.14-24-pve;973M /lib/modules/7.0.14-25-pve;973M /lib/modules/7.0.14-26-pve;973M /lib/modules/7.0.14-28-pve;970M /lib/modules/7.0.2-6-pve;
the hub block for this box (kernel step due tonight?): {'ring': 0, 'enabled': True, 'kernel': None}
last night run marker: 2026-11-14T04:51:34Z
@@ -0,0 +1,17 @@
#### felhom (Tester 1) BEFORE the restart 2026-11-14T14:16:51Z
kernel 7.0.14-22-pve; boot_id 6407d71b-1d89-4748-8655-49c6de8cf61b; up since 2026-11-10 15:43:22
crash guard: {'armed': True, 'tripped': False, 'last_boot_at': '2026-11-10T14:43:22Z', 'last_boot_unclean': False, 'unclean_boots_in_window': 0, 'boot_id': '6407d71b-1d89-4748-8655-49c6de8cf61b'}
busy? vzdump 0 apt 0; guest status: running; containers 10 running / 0 unhealthy
RESTART (systemctl reboot) at 2026-11-14T14:16:54Z
host answers, agent active, guest running after 156 s
containers: 10 running, 0 starting, 0 unhealthy after 239 s (or the 540 s limit)
#### felhom (Tester 1) AFTER the restart 2026-11-14T14:21:03Z
kernel 7.0.14-22-pve (before: 7.0.14-22-pve); boot_id 7a721957-7e95-476c-be40-52d0dafc7284; up since 2026-11-14 15:19:11
crash guard: {'armed': True, 'tripped': False, 'last_boot_at': '2026-11-14T14:19:11Z', 'last_boot_unclean': False, 'unclean_boots_in_window': 0, 'boot_id': '7a721957-7e95-476c-be40-52d0dafc7284'}
## the host health rule: daemons active active active active active ; guest status: running; containers 10 running / 0 unhealthy; controller healthy; tunnel container: running
kernels: /boot 7.0.14-22-pve 7.0.2-6-pve ; grub default gnulinux-7.0.14-22-pve-advanced; failed units 0; chrony active
mark: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false}
## the agent at this start:
2026-11-14T15:19:22+01:00 felhom felhom-agent[1148]: time=2026-11-14T15:19:22.533+01:00 level=INFO msg="felhom-agent daemon starting" version=0.174.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-14T15:19:22+01:00 felhom felhom-agent[1148]: time=2026-11-14T15:19:22.731+01:00 level=INFO msg="osupdate: profile refresh not needed" mode=profile-refresh why="agent start" kernel=7.0.14-22-pve reason="the profiles were refreshed on this kernel (7.0.14-22-pve) at 2026-11-14T11:30:11Z"
profile-refresh record: { "at": "2026-11-14T11:30:11Z", "kernel": "7.0.14-22-pve", "restarted": [], "rewritten": [ "f6da0254.0/lxc-containers" ]}
@@ -0,0 +1,38 @@
## vouch read-back 2026-11-14T14:21:57Z — channel 1: the hub's log (the pod's clock is CET)
2026/11/14 15:21:45 [INFO] Artifact manifest set: agent=0.174.0 golden=0.352.1 min_agent="0.158.1" wrapper_sha=false bundle_sha="7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f"
2026/11/14 15:21:45 [INFO] artifact vouch timing (R-1346): registry probes 119ms, sha lookups 76ms, bundle sha 10ms, fleet check + save 222ms, total 428ms
## channel 2: the hub's System page (CC's token)
Global controller floor: 0.352.1 · vouched agent: 0.174.0
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
look
Tester4-46836f Tester4
kernel step: — · the host needs a restart since 2026-11-14 · root files behind the vouched agent's · agent behind: 0.173.0 → 0.174.0 · could not read: default kernel
look
tester-1-d70be4 Tester 1
partial: no running customer guest · crash guard not installed · could not read: running kernel, next-boot kernel, default kernel, host Debian, held packages, kernel.panic, crash restarts, guest Debian, Docker engine, containerd, Docker live-restore
demo-felhom-8363b5 10 s no 0 armed 0.174.0
demo-hp-bb76ea 10 s no 0 armed 0.174.0
tester-1-d70be4 unknown no unknown not installed 0.174.0
root-files cell titles: ['behind the vouched agent 0.174.0&#39;s bundle — send it with a signed agent_config_upd']
demo-felhom-8363b5 Demo Ügyfél 0.174.0 9.2.26 7.0.14-28-pve ONLINE 1/2 5% 23% 37% running (connected) 37% local demo-hp-bb76ea D
demo-hp-bb76ea Demo HP 0.174.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 48% 42% running (connected) 62% local-lvm tester-1-d70be4
tester-1-d70be4 Tester 1 0.174.0 9.2.2 7.0.14-22-pve ONLINE 1/2 0% 33% 30% running (connected) 30% local Felhom Hub 0.181.0
Tester4-46836f Tester4 0.173.0 9.2.2 7.0.2-6-pve DOWN 1/1 11% 20% 21% absent (no cloudflared container in the guest (no tunn
## the hub's System page, read 2026-11-14T14:25:58Z
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
look
Tester4-46836f Tester4
kernel step: — · the host needs a restart since 2026-11-14 · root files behind the vouched agent's · agent behind: 0.173.0 → 0.174.0 · could not read: default kernel
look
tester-1-d70be4 Tester 1
partial: no running customer guest · crash guard not installed · could not read: running kernel, next-boot kernel, default kernel, host Debian, held packages, kernel.panic, crash restarts, guest Debian, Docker engine, containerd, Docker live-restore
Crash guard and root files
Box kernel.panic Oops Crash restarts 24 h Crash guard Root files
Tester4-46836f 10 s no 0 armed 0.173.0
demo-felhom-8363b5 10 s no 0 armed 0.174.0
demo-hp-bb76ea 10 s no 0 armed 0.174.0
tester-1-d70be4 unknown no unknown not installed 0.174.0
Felhom Hub 0.181.0
(function(){var h=location.hash&&document.getElementById(location.hash.slice(1));if(!h)return;var d=h.closest('details');while(d){d.open=true;d=d.parentElement&
@@ -0,0 +1,3 @@
felhom: felhom-agent 0.174.0; sha256 6efbb97809135483; go in the running binary: 2 go1.27.4; unit active restarts 0
demo-hp: felhom-agent 0.174.0; sha256 6efbb97809135483; go in the running binary: 2 go1.27.4; unit active restarts 0
demo-felhom: felhom-agent 0.174.0; sha256 6efbb97809135483; go in the running binary: 2 go1.27.4; unit active restarts 0
@@ -0,0 +1,25 @@
# Release 2a — the box removes its old kernels, and refreshes its security profiles — evidence (2026-11-14)
Report: `REPORT-release-2a-2026-11-14.md` (repo root). The profile spike: `../profile-refresh-spike-2026-11-14/`.
Agent v0.174.0 (felhom-agent `cc63dd5`), vouched 2026-11-14 15:21:45 CET.
| File | What it shows |
|---|---|
| `A/redproof.txt` | the clean-up's wrapper tests: 14 mutations (K1, the simulation, the phase, the ring-0 mark, the flag, the default, not-on-newest, fell back, the plan's names, no state file, the 20-hour window, the dry run, repair-first, the after-purge proof) — each RED on the mutant, GREEN on the real file |
| `A/redproof-agent.txt` | the agent's clean-up tests: 3 mutations, each red |
| `A/01-dryrun-preview-*.txt` | the working-copy wrapper's dry run on demo-hp and Tester 1, before the commit |
| `B/redproof.txt`, `B/redproof-agent.txt` | the profile refresh: 10 + 3 mutations, each red (one agent mutation did not compile at the first try and was done again; both are in the file) |
| `D/01-published-binary.txt` | the published agent, downloaded with no login: sha256, `go1.27.4` |
| `D/02`–`D/06` | delivery to Tester 1 and demo-hp: before; the signed `agent_update`; the journals (the old wrapper refuses the new mode at the agent's start); the signed bundle; bundle installed 77/77 and the profile refresh right after it |
| `D/07-tester1-dryrun.txt` | Tester 1 through the released agent: „not run", it has no ring-0 mark (scenario 8, live) |
| `D/08-tester1-cleanup.txt` | **the first real removal by the program** (scenario 1), the second call (scenario 2), the mark set for the test and put back. Started by hand — see the report |
| `D/09-n100-before.txt`, `D/11`, `D/12` | the N100 before (−32.55 s, chrony failed, the stale profile); its two signed jobs |
| `D/13-n100-after.txt` | the N100 after: +0.18 s, chrony active, the profile is the fresh compile; the same 2.5 hours later |
| `D/13b-n100-container-near-the-step.txt` | the controller recreated three apps 9 s after the clock step, and why (R-1395); the crash guard was not fooled |
| `D/14-n100-dryrun-tonight.txt` | what the N100 removes tonight: 6 of 8 kernels |
| `D/10-demo-hp-dryruns-and-refusals.txt` | demo-hp: nothing to remove; P1, P2, P3 by dry run against states set and put back; identical after |
| `D/15-tester1-reboot.txt` | one orderly restart of Tester 1: the same kernel, healthy after 239 s, a clean stop, „profile refresh not needed" |
| `D/16-vouch-readback.txt` | the vouch from the hub's log and from its System page |
| `D/17-go-version-on-boxes.txt` | the running agent on the three boxes: sha256 and `go1.27.4` |
Not kept as files: the release script's own output (it prints a repository address; the CHANGELOG has the two sha256s).
+10
View File
@@ -26,6 +26,16 @@
---
## 2026-11-14 — release 2a: R-1388, R-1301, R-1391 closed (full text of the rows: `git show a2173830:documentation/backlog/OPEN-ITEMS.md`)
| Row | What | Closed | Evidence |
|---|---|---|---|
| **R-1388** | **After a kernel went back and forth, a box could load an old security profile for an updated package — on the N100 this stopped the time service for good.** (P3) | CLOSED 2026-11-14 — agent 0.174.0 (vouched 15:21:45 CET): the root wrapper's mode `profile-refresh` loads every AppArmor profile from a fresh compile once per kernel the box starts on, and restarts only a failed service whose own program's cached profile was stale. Spiked first on demo-hp with both guests running (the fault reproduced with one planted stale profile; 0.5 s; no container touched). Ran by the product's own path on Tester 1, demo-hp and the N100 | `audits/profile-refresh-spike-2026-11-14/`; `audits/release-2a-2026-11-14/` `B/`, `D/06`, `D/13`; `11-os-updates.md` §5.12; felhom-agent `cc63dd5` |
| **R-1301** | **The N100's Proxmox host kept no time: `chrony` failed at every boot since 2026-10-16.** (P3) | CLOSED 2026-11-14 — repaired by the product, no hand step (`09` §3 decision 305): when the N100 took agent 0.174.0's bundle, the profile refresh rewrote the stale chrony profile and restarted `chrony.service`; the clock stepped +32.717 s. Offset to DooPlex −32.55 s before, +0.18 s after and 2.5 hours later; chrony active, 0.0002 s off | `audits/release-2a-2026-11-14/D/09-n100-before.txt`, `D/13-n100-after.txt` |
| **R-1391** | **The hub, the controller and the agent were built on Go 1.27.3, which has GO-2026-6859.** (P3) | CLOSED 2026-11-14 — all three on Go 1.27.4: controller 0.352.1 and hub 0.181.0 (release 1), agent 0.174.0 (the running binary reads `go1.27.4` on Tester 1, demo-hp and the N100). No handler of the three matched the advisory's crash pattern | `audits/release-1-dates-2026-11-14/`; `audits/release-2a-2026-11-14/D/17-go-version-on-boxes.txt` |
**Reasoning kept (R-1388).** A stale copy is told by the CACHED FILE changing when it is rewritten — never by the kernel's loaded hash, which differs between a cache load and a fresh load of the same policy for a file with several profiles. The refresh runs once per kernel, not at every agent start: each run floods the kernel log and hides other audit lines for a few seconds.
## 2026-11-14 — R-1392 closed: root files are „behind” only when the files differ (full text of the row: `git show 561a4d98:documentation/backlog/OPEN-ITEMS.md`)
| Row | What | Closed | Evidence |
+8 -9
View File
@@ -242,7 +242,7 @@ stopping line that lies.
| **R-1373** | Storage & devices | P3 | **Three endings have a second door that does less than the first.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC13, EA08; read in source, not measured on a box). (1) **Orphan delete** of an app (`internal/stacks/delete.go:305-460`, `internal/api/router.go:1150-1178`) removes the containers, the volumes and the stack record; it does not remove the undo copy, the update record or the backup folders that the normal remove takes away. (2) **A move of ONE app to another drive** (`internal/stacks/migrate.go:616-654`) removes the app's data and its backup unit from the old drive; the app's restore copy, a set-aside backup and kept data stay there. Only the move of all apps clears the old drive. (3) **A drive retired by a signed operator job** (agent `internal/signedjobs/decommission.go:86-103`) sets the record and does not take the drive out of the running guest, which the household's own door does (`internal/localapi/disks.go:559-567`). **Members (answers marked N):** EC03-4, EC13-4, EA08-4. | OPEN — filed 2026-11-10 (endings list) | — | Controller: make orphan delete call the same clean-up steps as the normal remove; make a one-app move take the app's other folders with it, or list them. Agent: say in `03-host-agent.md` whether the signed retire is meant to leave the drive in the guest. | CC |
| **R-1378** | Storage & devices | P3 | **Three endings can stop half-way with no finish, or have no way back that nobody chose.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC15, EC03, EC27; read in source, not measured). (1) **Wipe of a drive** (`internal/web/storage_handlers.go:911-1000`): the handler removes the drive from the registered places BEFORE the format. If the format fails, or the controller stops between the two steps, the drive is not registered and still holds the old data; there is no record of a wipe in progress, and a failed eject is only a warning. The handler does not look for apps on the drive. (2) **Orphan delete** writes no 'remove pending' record, so a controller that stops in the middle does not finish it at the next start (the normal remove does). (3) **Import over an installed app** (`internal/web/handler_export.go:312-356`, `internal/appexport/restore.go:288`) replaces the app's volumes and makes no undo copy first. **Members (answers marked N):** EC15-4, EC15-7, EC03-7, EC27-5. | OPEN — filed 2026-11-10 (endings list) | — | Controller: format first and unregister after, or write a wipe-in-progress record that the next start reads; give orphan delete the pending record; decide if an import makes an undo copy (a restore does). | CC |
## Security & access — 29 rows (P1 2, P2 5, P3 15, P4 7)
## Security & access — 28 rows (P1 2, P2 5, P3 14, P4 7)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -273,17 +273,15 @@ stopping line that lies.
| **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f <conf>` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC |
| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | **OPEN** — 2026-10-27 (operator yes): the dead names were REMOVED from the store with `sec rm` (46, one commit each; `audits/r1299-window-button-2026-10-27/sec-rm.txt`, names only): every one-time code this row names is gone; R_DEMO_FELHOM stays (unsure). Consequence: the nightly sweep no longer watches for the removed values — they open nothing (their boxes are destroyed). The session transcripts themselves are unchanged — that question stays here. **Earlier:** OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC |
| **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator |
| **R-1391** | Security & access | P3 | **The hub, the controller and the agent are built on Go 1.27.3, which has GO-2026-6859 (an HTTP/1 server can crash with a runtime panic when a `MaxBytesReader` limit is hit while the handler reads the body and writes a header at the same time); fixed in Go 1.27.4.** Found 2026-11-14 (health-check session) from the security watch's 2026-11-11 mail, which names only the hub image (`golang 1.27.3-alpine3.24`); `go.mod` in all three repos reads `go 1.27.3`. Reach not measured: the crash needs a handler that reads the request body concurrently with writing the response header — whether any Felhom handler does is not read yet. A crash restarts the process (the controller's supervisor, the agent's unit, the hub's pod), so the effect is an outage of seconds, not data loss. Source: https://vuln.go.dev/ID/GO-2026-6859.json | NARROWED — **2026-11-14: the controller (0.352.1, global floor 09:21 UTC, on every box that reports) and the hub (0.181.0, live 09:58 UTC) are built on Go 1.27.4. LEFT: the agent (still 1.27.3), release 2.** Reach read the same day: no controller handler matches the crash pattern (seven `MaxBytesReader` places, each reads the body to its end before it answers; no `TimeoutHandler`); the hub has no `MaxBytesReader` at all; the agent has ONE (`internal/localapi/server.go`, a JSON decode in the handler's own goroutine — not read further). So the agent's turn is hygiene, not urgent. Evidence `audits/release-1-dates-2026-11-14/`. Earlier: OPEN — filed 2026-11-14 | — | build each component's next release on Go 1.27.4 (`go.mod` + the build image); until then, grep each repo for `MaxBytesReader`/`MaxBytesHandler` and a handler that writes headers while a goroutine reads the body | CC |
| **R-1309** | Security & access | P4 | **What no automated source shows: a fix announced only by mail, forum or a blog we do not read; a security fix nobody calls one (cloudflared 2026.10.2's „bump distroless image"); closed-source apps (Plex, Emby).** The watch (R-1303) shows these only as BEHIND (weekly). The vendor mails the operator receives stay a source. `audits/security-watch-2026-10-26/SPIKE.md` | OPEN — filed 2026-10-26 (a stated limit) | — | none until a vendor offers a feed; add a vendor's feed to `security_watch.FEEDS` when one exists | CC |
## Box system & updates — 21 rows (P3 16, P4 5)
## Box system & updates — 20 rows (P3 14, P4 6)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` **2026-11-11:** the Clock list measured it again (32.6 s behind, time service not running) and leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CA03-6; class R-1384). **2026-11-11 (afternoon, read only): the cause is found.** The kernel loaded an OLD cached AppArmor profile for chrony (the one of chrony 4.6.1); chrony 4.8 writes to `/run/systemd/notify`, the old profile refuses that write, chrony treats it as fatal. Proof: loaded profile sha256 `dbf67f16…` against a fresh compile `83417e83…` on the N100, equal on demo-hp; the kernel's `DENIED` line of 2026-10-16. The order that made it: a new kernel started once (2026-10-07 12:11), the box went back to the old kernel, chrony was updated there (14:15), the new kernel started again and used its stale cache folder (`documentation/audits/n100-clock-2026-11-11/README.md`). The product side is R-1388. The hand step is in `STATUS.md` „do”; it was not run and not tested. | READY — **2026-11-14 09:55 — RULED (`09` §3 decision 305): no hand repair now; release 2 repairs it through the product. If release 2 is not live by 2026-12-15, the operator runs the hand step.** OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | the operator runs the one hand step on the N100 (`STATUS.md`); then read `chronyc tracking` and close. The product fix is R-1388 | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC02, CC06-6, CC07-6, CC22-6). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | NARROWED — **2026-11-14 (release 2a): the old-kernel part is BUILT — agent 0.174.0, vouched 15:21 CET (`11` §5.11.1 `[FACT — built]`).** A ring-0 box removes its old kernels at night by the wrapper's own rule. On boxes: a real removal on Tester 1 (3 → 2 kernels, 1.12 GB, 8.5 s; the default, the pinned file, the generators and the state file unchanged; a second run removed nothing; one clean restart) — started by hand with a ring-0 mark set for the test on the operator's yes, because a night cannot be started by day (R-1394); dry runs on demo-hp (nothing to remove; refusals P1, P2, P3) and the N100 (6 of 8 will go). **LEFT for this part: the step by the agent at night — read the N100's and demo-hp's night of 2026-11-15; then a ruling for ring-1 boxes (D82).** Evidence `audits/release-2a-2026-11-14/`. The other four places of this row are unchanged. Earlier: OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallbLine truncated
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. **2026-11-10 15:19 — RULED (D75 = one number, 200 GB, with or without a backup drive; `09` §3 decision 295).** This replaces CC's two-step pick. **Open, not ruled:** what the installer does below 200 GB — refuse, or warn. demo-hp (128 GB) stays as a test box. `16` §14.3 carries the ruling. | — | Design first: what the installer does below 200 GB (refuse, or warn) — a later session. Then the installer. | CC |
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
@@ -296,12 +294,12 @@ stopping line that lies.
| **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC |
| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | READY — **2026-11-14 09:55 — RULED (D80, `09` §3 decision 303): the root CA copy.** Built in release 2 (the host release). OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC |
| **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC |
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
| **R-1388** | Box system & updates | P3 | **After a kernel went back and forth, the box can load an old security profile for an updated package — on the N100 this stopped the time service for good.** Found 2026-11-11 (`documentation/audits/n100-clock-2026-11-11/README.md`; read only, on three boxes). AppArmor keeps one cache folder for each kernel. A package update writes the new profile only into the folder of the kernel that runs. If the box then starts another kernel that it started once before, that kernel's folder still holds the old profile, and AppArmor uses it (the cached copy looks newer than the package's file). MEASURED on the N100: chrony 4.8 under the profile of chrony 4.6.1; the kernel refuses its write to `/run/systemd/notify`; the unit fails at every start since 2026-10-16. On a household box the product's own kernel rollback (the crash guard) makes the same order, and it is true for every profile, not only chrony's. Not measured: a box where the rollback really ran. Also seen: Tester 1's host still has chrony 4.6.1 while the other two have 4.8 from the OS-update leg; why was not looked at. | READY — **2026-11-14 09:55 — RULED (`09` §3 decision 305): the N100 is this fix's real case in release 2; no hand step before.** OPEN — filed 2026-11-11 (N100 clock diagnosis) | — | Design first, in `11-os-updates.md`: after an OS-update leg and after a kernel rollback, reload the profiles with the cache ignored (`apparmor_parser -r -T -W`), or clear the cache folders of the kernels that do not run. Red-proof on a scratch host: make the order, see the stale profile, see the fix remove it. Then say why Tester 1 did not take chrony 4.8. | CC |
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | OPEN — **2026-11-14: the one box whose clock was wrong (the N100, 32.6 s) is right again (R-1301 closed); this row's check is still not built.** Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
| **R-1395** | Box system & updates | P4 | **A step of the host's clock makes the controller read a new boot: it then recreates every drive-backed app, once.** Found 2026-11-14 on the N100 (`audits/release-2a-2026-11-14/D/13b-n100-container-near-the-step.txt`): the time service came back and stepped the clock by +32.7 s; 9 seconds later the controller logged „[gate] boot 1794112550-…: recreating drive-backed app” for radarr, sonarr and qbittorrent (each down for 1 to 5 s; all healthy after). Cause, read on the box: the controller's boot identity carries `/proc/stat` `btime`, and the kernel computes `btime` from the wall clock, so it moved by the same 32 s (the crash guard's `last_boot_at` reads 04:35:18Z, `btime` now 04:35:50Z). The crash guard was NOT fooled: it reads the kernel's `boot_id`. A step this large needs a clock that was wrong for weeks, so a healthy box does not meet it; a fresh box with a wrong hardware clock can, at its first time sync. | OPEN — filed 2026-11-14 | — | make the controller's boot identity the kernel's `boot_id` (as the crash guard's is), with a test that a moved `btime` is not a new boot | CC |
## Monitoring & notifications — 28 rows (P2 3, P3 17, P4 8)
@@ -334,7 +332,7 @@ stopping line that lies.
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
## Hub & operator — 18 rows (P2 1, P3 5, P4 12)
@@ -379,7 +377,7 @@ stopping line that lies.
| **R-990** | Business & legal | P4 | **Operator: read `documentation/audits/forum-research-2026-10-12/FELHOM.md` §5 and rule on each recommendation.** READ-ONLY research 2026-10-12: three years (#7943–#11092, 2,908 posts, 195 people, 82 beginner threads) of the public PROHARDVER! topic „Home server / házi szerver építése”, tagged and counted by person (`FINDINGS.md`, method and spot-check in `METHOD.md`; no username or post text in git). Ten ranked recommendations, none acted on; the top three: a „which machine works” page (8 of 33 beginners who name their hardware own only a Synology, a router disk or a Raspberry Pi — the installer cannot run there; the same gap is R-989's first recommendation from a different source, and the shipped-page half is R-991), „you do not choose an operating system — we do” (the biggest beginner question, 20 of 82 threads), and „bring your old office PC” as the cheap entry. Catalog gap (≥ 3 people): a DLNA server (7), an ad/DNS blocker (5), a camera recorder (5). | **WAITING-ON-OPERATOR** — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages | operator |
| **R-1035** | Business & legal | P4 | **Did the „felhő tárhely” page earn anything? Two dated checks.** **Baseline (reported by the operator from Search Console, 2026-10-14, last 3 months):** 1 click, 48 impressions, CTR 2.1 %, average position **73.6**; „felhő tárhely” 19 impressions, „felhőtárhely” 5, „felhő alapú tárhely” 3. CC measured separately that the phrase „felhő tárhely” appeared **0 times** on the home, FAQ and backup pages — Google was showing the home page for those queries by accident. `/sajat-felho-tarhely` + `/en/own-cloud-storage` published **2026-10-14**; full baseline, price sources and the power provenance in `audits/seo-2026-10-14/BASELINE.md`. **What to measure, both times:** impressions and average position for „felhő tárhely” and „saját felhő tárhely”, plus the new page's own impressions, clicks and position. **A null result at +4 weeks means nothing on its own** — a new page on a site at position 73.6 takes weeks to months to move, and „felhő tárhely” alone is held by Microsoft, Google and the big list sites; the page aims at the two longer phrases. That is why there are two checks and not one, and why neither should be read as a verdict on the copy. **There is no Search Console connector in this project, so CC cannot read these numbers** — the operator reads them, CC records and compares. **Blocked until the operator asks for indexing** (Search Console → URL-ellenőrzés → the URL → „Indexelés kérése”): until that happens, or Google finds the page itself, both checks read zero for a reason that is not the page. **This is a slow channel and does not replace talking to people.** **2026-11-11:** the 4-week check came due today in a run with no operator present; only the operator can read Search Console, so the date moved to 2026-11-18 (the gate's rule allows a move with a reason). The 8-week check keeps its date. | **WATCHING** | operator: request indexing, then supply the Search Console read | 2026-11-11: first read (+4 weeks); 2026-12-09: second (+8 weeks), compared with the first | operator + CC |
## Process & tooling — 34 rows (P3 8, P4 26)
## Process & tooling — 35 rows (P3 8, P4 27)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -434,3 +432,4 @@ stopping line that lies.
| **R-1387** | Process & tooling | P4 | **A document and the source disagree on a time, in four places (4 answers).** Found 2026-11-11 (the Clock list, class W5, `documentation/audits/clock-list-2026-11-11/TABLE.md`). (1) `documentation/architecture/03-host-agent.md` names `Restart=always` and a watchdog for the agent's unit; the unit has `Restart=on-failure` and no watchdog (CA18-6, CA18-8). Which one is the design is not stated, so this session did not edit either. (2) The hub-database copy alarm: 13 hours in one place, 26 in another (CH23-7; a DooPlex procedure, listed for the operator, not edited). (3) Old „24 h / 02:30” text for the hub's daily job (CH17-6). **Seen beside the lists, same kind, not in the count:** (4) the comment at `felhom-controller/controller/cmd/controller/main.go:1412` says 14 days where the code keeps restore copies 30 days (D62); (5) `documentation/architecture/07-backup-architecture.md:455-457` cites `main.go` lines that have moved; (6) `documentation/operations/tailscale.md` says „both nodes” where three test hosts exist, and Tailscale was not running on demo-hp and Tester 1 on 2026-11-11 (not known if that is meant); (7) the threat model and the recovery inventory do not say the same about how often the hub bearer is rotated. | OPEN — filed 2026-11-11 (clock list) | — | Operator or a task: say which side is right for (1) and (2); then one docs commit. (3) is a text fix in the same commit. | CC |
| **R-1389** | Process & tooling | P4 | **From the operator's home network the installer address gives 404.** Found 2026-11-11 (`documentation/tests/golden-0.352.0-2026-11-11/fresh-install/installer-address-404-on-lan.txt`). The home router's DNS (192.168.0.250) answers `felhom.eu` with 192.168.0.192, the site copy on DooPlex that was retired on 2026-11-07 (R-1228). So `https://felhom.eu/scripts/…` gives 404 for every machine that uses that DNS; from DooPlex (public DNS) it gives 200. A household is not touched. A test box on the home network, and Tester 2 if it sits there, cannot get the installer by its public address. | OPEN — filed 2026-11-11 (golden fresh install) | — | The operator removes the `felhom.eu` override from the home DNS (it is his router), then one `curl -sI https://felhom.eu/scripts/` from a machine on that network reads 200. | operator |
| **R-1390** | Process & tooling | P4 | **`sec put` cannot push the store any more, so a rotated token is saved on DooPlex only.** Found 2026-11-11. `sec put` ends with a plain `git push` of the `cc-secrets` clone (`cc-secrets/ccsec/cli.py:51-56`). Since this run that clone's address holds no login, and its helper answers only inside `sec run GITEA_GIT_TOKEN -- …`. A plain push now fails for lack of a login; the new entry is committed in the clone and not pushed. Until it is fixed, the push is a second command: `sec run GITEA_GIT_TOKEN -- git -C /mnt/5_hdd/felhom.eu/git/cc-secrets push origin HEAD:main`. Not tried in this run (no store change was allowed). | OPEN — filed 2026-11-11 (token fix) | — | In `cc-secrets`: `sec put` runs its push through the store's own token (or says plainly „committed, not pushed: run …”); a test that a `put` on a clone with no login in the address ends pushed or says so. | CC |
| **R-1394** | Process & tooling | P4 | **The night's OS-update leg cannot be started by day on a test box, so a step that lives in it is proven only by waiting for a night.** Found 2026-11-14 (release 2a). The agent starts the leg only after the night's OWN whole-guest backup; a manual backup does not start it, by design (R-899: a household press must not start OS updates), and the debug pass (`--selftest=os-update`) runs with trigger `debug`, which the kernel step and the kernel clean-up refuse, also by design. So the first real kernel clean-up „by the program” on Tester 1 was started by hand with the agent's own command line (`audits/release-2a-2026-11-14/D/08-tester1-cleanup.txt`), and „the step at night by the agent” waits for a real night. The same held for the kernel lane's own night tests. | OPEN — filed 2026-11-14 | — | decide if a test box needs a signed „run the night leg now” (operator-signed, never a household's press), or if waiting one night stays the method | CC |
+4
View File
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-11-14 (afternoon)
**Updated 2026-11-14 (midday).** Release 1 („dates”) is out: controller 0.352.1 and hub 0.181.0 with the Go security fix; golden 0.352.1 is vouched. The false „root files behind” warning is fixed; no alarm mail is due on 11-17. The one-year HTTPS setting is NOT in (next controller release, from 11-15).
## Moved from STATUS on 2026-11-14 (midday)
**Updated 2026-11-14 (morning).** Health check: the three test boxes are healthy; last night's backups and restore proofs passed. CI was red three nights for one due check; it is read now. The yellow „root files behind" is a false alarm that mails on 11-17 unless you answer D81.
+1
View File
@@ -54,3 +54,4 @@ what happened, who noticed, and the register row it became. Prose here is not a
R-1300 (the release), R-1301 (the clock). Evidence: `audits/cloudflared-2026.10.3-2026-10-26/` (`obs-n100-clock.txt`). (CC)
- **2026-11-05..07 — the nightly Proxmox update step refused three nights in a row.** `proxmox-grub` 2.12-9+pmx4 needs `grub-efi-amd64` of the same version; the update wrapper put `proxmox-grub` in the Proxmox group (its boot-package rule matched only names that START with `grub`), so the whole step was refused (R6) and 34 Proxmox updates waited; an `os_update_failed` mail each morning. Apps not touched. Agent 0.172.1 puts `proxmox-grub` with the boot packages; it reaches the box by the operator's vouch and two signed jobs. Register: fixed in the session (size rule), recorded in the agent's CHANGELOG. Evidence: `audits/demo-check-2026-11-07/n100-pve-step-R6.txt`. (CC)
- **2026-11-10 — every place that holds data was measured (read-only; the space list).** Host system disk 38.09 of 100.86 GB (38 %), 4.6 GiB more than on 10-15: the box holds 8 kernels (8.15 GB) and none was ever removed → R-1368. The agent's self-update folder holds 48 old binaries (724 MB) and there are 11 old agent copies beside the binary (155 MB) → R-1368. The host journal sits at its cap (4.28 GB, 74 files, oldest entry 10-04) → R-1367. The three whole-box copies of 2026-07-28 still lie on the system disk (17.59 GB) → R-1087; four template archives (2.56 GB) → R-1368. On the USB drive, `backups/offsite-restore` holds 1.65 GB of two apps that are not installed any more → R-1372. Nothing was deleted or changed. Table: `audits/space-list-2026-11-10/TABLE.md`. (CC)
- **2026-11-14 — the clock is right again, repaired by the product.** When the box took agent 0.174.0 and its root files (11:36 and 11:51 UTC), the new „profile refresh" found the old security profile of the time service, loaded the right one and started the service: the clock stepped forward 32.7 s (offset to DooPlex −32.55 s → +0.18 s; still so 2.5 hours later). Nine seconds after the step the controller read a new boot and recreated Radarr, Sonarr and qBittorrent once (each down for a few seconds; all healthy). A dry run says tonight's night removes 6 of the 8 kernels. Register: R-1301 and R-1388 closed, R-1395 (the controller and the clock step), R-1368 (the kernels — read the night of 11-15). Evidence: `audits/release-2a-2026-11-14/D/` (`09`, `13`, `13b`, `14`). (CC)
+1 -1
View File
@@ -44,7 +44,7 @@ hub alarms at 90 % and 95 %.
| What | Step | Mark |
|---|---|---|
| Old kernels | `ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). The kept older kernel then booted the box to healthy and the box went back (`audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **NOT PROVEN:** on a second box |
| Old kernels | **A ring-0 box removes them by itself at night since agent v0.174.0** (`11-os-updates.md` §5.11.1). First read what it would do: `sudo -u felhom-agent felhom-agent --config /etc/felhom-agent/agent.json --selftest=kernel-cleanup-dry-run` (it changes nothing; „would": remove / nothing / refuse / not_run, with the reason). The hand step below is for a box where the program does not run (a ring-1 box, a box that fell back) and for an urgent case: `ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). The kept older kernel then booted the box to healthy and the box went back (`audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **NOT PROVEN:** on a second box |
| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) |
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free <volid>`, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only |
| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |