Release 2a: agent 0.174.0 vouched — old kernels removed by the box (11 §5.11.1 built), security profiles refreshed after a kernel change (11 §5.12, spike + build); R-1388, R-1301, R-1391 closed; R-1394, R-1395 filed; D82
gates / gates (push) Successful in 2m1s

This commit is contained in:
admin committed 2026-11-14 15:30:44 +01:00
1 parent a2173830c1
commit 8955b4e55f
51 files changed
+1385 -19

No files matched your search

@@ -0,0 +1,60 @@
# Spike — can a box refresh its security profiles by itself? — 2026-11-14 (release 2a, Part B)
**Answer: yes. The spike is clean, so the mode was built in the same run** (agent v0.174.0, `11-os-updates.md` §5.12).
Venue: demo-hp (Tier 0), host kernel 7.0.14-28-pve, guests 9201 (26 containers) and 9202 (6) running the whole time;
reads on Tester 1. Nothing on the N100 was touched. No package was changed. Documents read:
`audits/n100-clock-2026-11-11/README.md` (the diagnosis and its method), `architecture/11-os-updates.md` §5.4.1, §5.9,
§5.11, rows R-1388, R-1301.
## The six questions
| # | Question | Result | File |
|---|---|---|---|
| 1 | Does the detector (loaded hash against a fresh compile) work, and why does `lxc-containers` always "differ"? | It works for one-profile files. **`lxc-containers` compiles to a different binary every time** (two fresh compiles in a row differ; it includes a whole folder). So its hash can never judge it; the detector now compiles twice and says so. On demo-hp and Tester 1: 106 files, 0 stale, 1 not judgeable, 4.5 s | `evidence/01-detect-*.txt`, `tools/detect.sh` |
| 2 | Can one stale profile be made by the real mechanism, with no package change? | Yes. The old compiled chrony profile (of 4.6.1, sha `dbf67f16…`) was copied from the old kernel's cache folder into the running kernel's; its time (2026-08-21) is newer than the source's (2026-07-01). The system's own reload then **loaded the old one**, and a restart of chrony **failed exactly as on the N100** („Could not send notification to $NOTIFY_SOCKET", twice) | `evidence/02-make-stale.txt` |
| 3 | The smallest repair | `apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d` — **0.51 s**. After it: the same 119 profiles in the same modes; chrony's loaded profile and its cached copy are the fresh compile (`83417e83…`); both guests running, 26 and 6 containers, none restarted (newest start time unchanged), every guest init still confined by its own profile | `evidence/03-refresh.txt` |
| 3b | Why did 17 other profiles change their loaded hash? | Not staleness. For a file that defines several profiles the kernel's hash differs between a cache load and a fresh load of the **same** policy. **The only reliable sign of a stale copy is the cached FILE changing when it is rewritten** (here: only the planted one, and `lxc-containers`) | `evidence/03b-why-others-changed.txt` |
| 4 | What starts the failed program again? | The refresh does not (chrony stayed `failed`). The narrow rule, tried by hand: a **failed service** whose **ExecStart program is the path-name of a profile whose cached copy the refresh rewrote** → one `systemctl restart`. chrony came back, 0.0003 s off | `evidence/04-restart-rule.txt` |
| 5 | When should it run? | See the options below; the pick is „when the box runs another kernel than at the last refresh" | — |
| 6 | Can the guest have the same staleness? | Not today: in guest 9201 the AppArmor package is installed, its service is inactive, there is no profile cache, and the kernel's profile list is not visible | `evidence/06-guest-read.txt` |
**No new refusal by the kernel in 10 minutes** (`evidence/05-watch.txt`): 1 `DENIED` line, and it is the control.
**The control was needed** (`evidence/05a-denied-watch-control.txt`): the kernel rate-limits its audit lines, each
reload writes about 119 `STATUS` lines, and the `DENIED` line that the planted stale profile made at 10:58:07 UTC is
**not in the kernel log at all** („kauditd_printk_skb: 111 callbacks suppressed"). A count of 0 alone would have proved
nothing. The second channel is behaviour: no container restarted, no failed unit.
## Decision 4 — what restarts the program (two options)
| | Option A (picked) | Option B |
|---|---|---|
| Rule | a failed service whose own program's cached profile was stale | a fixed list: `chrony.service` |
| Covers | every program this fault can stop | the one case seen |
| Risk | a unit that failed for another reason, and whose profile was also stale, gets ONE start try | none beyond chrony |
| Cost | 3 `systemctl` reads per failed unit, only when a path-named profile was rewritten | a new release for each next program |
Never „restart whatever failed", never a unit named by the plan, at most 3 units a run.
## Decision 5 — when it runs (two options)
| | When the kernel differs from the last refresh's (picked) | After every agent start |
|---|---|---|
| Catches the fault | yes: a stale copy can only appear for a kernel that did NOT run when a profile changed, so the box always starts on „another kernel" before the kernel loads it. A box with no record yet counts as changed (this is what repairs the N100) | yes |
| Cost | one refresh per kernel change (about weekly) | one per agent start (about daily): 0.5 s, and each run floods the kernel log with ~119 lines and **hides other audit lines for a few seconds** (measured above) |
The record is the wrapper's own root-owned file; the agent asks at every start and after a bundle install, and the
wrapper answers „not needed".
## Put back
`evidence/07-put-back.txt`: the spike's files removed from `/root`; the planted copy is gone (the cache holds the fresh
compile, the old kernel's folder is untouched); one cache load (`systemctl reload apparmor`) put the loaded hashes
back as found; the detector reads 0 stale; chrony active; both guests running; no failed unit.
## Not measured
- A box that boots with `proxmox-boot-tool` or without AppArmor (none in the fleet).
- A profile in `complain` mode or in `/etc/apparmor.d/disable` (demo-hp has none; the parser is given the folder, as the
system's own loader is).
- The refresh while a backup or a restore-test runs (the agent takes the heavy-op gate so it never does).
- The real case: the N100. It is the proof of the built mode, in the release.
@@ -0,0 +1,5 @@
#### demo-hp 2026-11-14T10:57:26Z kernel 7.0.14-28-pve
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
loaded profiles: 119
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 03ad7998e16e a5f0e0eaf4e4
files=106 cache-differs=0 loaded-differs=0 profiles-not-loaded=4 not-deterministic=1 seconds=4.556914622
@@ -0,0 +1,5 @@
#### felhom 2026-11-14T10:57:31Z kernel 7.0.14-22-pve
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
loaded profiles: 118
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 7ea3f12cf701 b1b2529f2c17
files=106 cache-differs=0 loaded-differs=0 profiles-not-loaded=4 not-deterministic=1 seconds=4.179027032
@@ -0,0 +1,19 @@
#### demo-hp 2026-11-14T10:58:06Z kernel 7.0.14-28-pve
## before: chrony active; loaded 83417e83b538; cache(cur) 83417e83b538; cache(old folder) dbf67f16ac25; fresh 83417e83b538
## source mtime 2026-07-01 22:28:16; old copy mtime 2026-08-21 17:42:38
saved the good cache file to /root/spike-profile-usr.sbin.chronyd.good (83417e83b538)
PLANTED: the old compiled copy is now in the running kernel's folder; its time 2026-08-21 17:42:38 is newer than the source
## reload the way the system does at a start (systemctl reload apparmor = apparmor_parser --write-cache --replace)
reload rc=0
## after the reload: loaded dbf67f16ac25 (dbf67f16 = the profile of chrony 4.6.1; 83417e83 = of 4.8)
## chrony still runs under the replaced profile? active; its confinement: /usr/sbin/chronyd (enforce)
Job for chrony.service failed because the control process exited with error code.
See "systemctl status chrony.service" and "journalctl -xeu chrony.service" for details.
restart rc=1
## after a restart: chrony failed; result=exit-code
2026-11-14T11:58:07+01:00 demo-hp chronyd[1853258]: Loaded seccomp filter (level 1)
2026-11-14T11:58:07+01:00 demo-hp chronyd[1853258]: Fatal error : Could not send notification to $NOTIFY_SOCKET
2026-11-14T11:58:07+01:00 demo-hp chronyd[1853258]: Fatal error : Could not send notification to $NOTIFY_SOCKET
2026-11-14T11:58:07+01:00 demo-hp systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
2026-11-14T11:58:07+01:00 demo-hp systemd[1]: chrony.service: Failed with result 'exit-code'.
2026-11-14T11:58:07+01:00 demo-hp systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
@@ -0,0 +1,55 @@
#### demo-hp 2026-11-14T10:58:32Z
## BEFORE
guest 9201: status: running init pid 2159 confinement=lxc-9201_</var/lib/lxc>//&:lxc-9201_<-var-lib-lxc>:unconfined (enforce)
containers: 26 running, 0 unhealthy, started-at-newest=2026-11-14T09:18:59.079611972Z
the controller container's process (host pid ?): confinement=
guest 9202: status: running init pid 2117 confinement=lxc-9202_</var/lib/lxc>//&:lxc-9202_<-var-lib-lxc>:unconfined (enforce)
containers: 6 running, 0 unhealthy, started-at-newest=2026-11-14T09:56:41.303141464Z
the controller container's process (host pid ?): confinement=
VM 341 (Tester 1): status: running
loaded profiles: 119
chrony: failed; loaded chrony profile dbf67f16ac25; cache dbf67f16ac25
disabled profiles (must stay unloaded):
## THE COMMAND: apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d
rc=0 seconds=.514403371
## AFTER
guest 9201: status: running init pid 2159 confinement=lxc-9201_</var/lib/lxc>//&:lxc-9201_<-var-lib-lxc>:unconfined (enforce)
containers: 26 running, 0 unhealthy, started-at-newest=2026-11-14T09:18:59.079611972Z
the controller container's process (host pid ?): confinement=
guest 9202: status: running init pid 2117 confinement=lxc-9202_</var/lib/lxc>//&:lxc-9202_<-var-lib-lxc>:unconfined (enforce)
containers: 6 running, 0 unhealthy, started-at-newest=2026-11-14T09:56:41.303141464Z
the controller container's process (host pid ?): confinement=
VM 341 (Tester 1): status: running
loaded profiles: 119
profiles whose loaded hash changed:
lxc-container-default 68d1e8408c2e9241ee6081869502ed73059174251ebb1af548f86afc72b06bec
lxc-container-default-cgns 2e6dec9bc9637ea459d1675047ef6a887bd41bc4511447cbbb61cbf23d6b6fed
lxc-container-default-with-mounting caeab50e1e3007a362e37a64c9adab1334aa4a9decacbba35a69301ed7bcf9bf
lxc-container-default-with-nesting 659040965dfea57ecb76f61643ce276e7ee35550e5089e6ed34862b6ae963d33
man_filter 6dc1077b7fa57c23a446f7bdcca7e27457182dcb12a635219604c23a16d414c9
man_groff 0f6549c22ad03ed89f4c857aa4d332f2a0f2b777f6f07939cad9486291645815
nvidia_modprobe f4ae61617079f7e868204f1bd5046778570fc5ed2c9a3cad9aa6c8ff34c657e7
plasmashell 17cfa46264cdf3b2ca1181848a56604d4efc64b328c8d57b221529e4d22045ea
transmission-cli 04b531d8a589bfa5b8c154d961f58835a57f4d35d17520915b971f9a15d3af3d
transmission-daemon f46a1d2db62218fa9f9be1afacfe256c2b1b0c6ed971117f38f883e1d61d49cb
transmission-gtk ab371958b58947d437928b327736e4c49cb83b3db1f2531d9765a936dc50ffe6
transmission-qt 0077a589d81e621e1faa806a460110820e10cccb8b1916b0e2a3b747a99d0894
/usr/bin/man 4ee549c2ed8742dbe5c1be3fc0b9b3e08ea7f77463366094ad454123e8fc03ed
/usr/lib/connman/scripts/dhclient-script fcadbc62c5777ecee8b1ec1d6f0abab4131df7fc1dc7f7b99635e689f3e60ee8
/usr/{lib/NetworkManager,libexec}/nm-dhcp-helper 6ca7645b1031d446bc65bf4bc9eec5cad0f7e70bf1d841271e972d79ebdf10d3
/usr/lib/NetworkManager/nm-dhcp-client.action b5532441db4544da68a7ac5bc6980f0aaa34dc28fb9ab702221d8ec04cca38b3
/usr/sbin/chronyd 83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1
/{,usr/}sbin/dhclient 3b3e18f5a89e78eb104fd6377ab41ec76dcb9823f799a95dc6d3dac787e429e2
profiles added or gone (names):
none
modes changed:
none
chrony profile: loaded 83417e83b538; cache 83417e83b538 mtime 2026-08-21 17:42:38; fresh 83417e83b538
chrony unit, NOT restarted by the command: failed
kernel lines since the command:
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.683:265): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="brave" pid=1855998 comm="apparmor_parser"
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.684:266): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="buildah" pid=1855999 comm="apparmor_parser"
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.686:267): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="busybox" pid=1856000 comm="apparmor_parser"
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.686:268): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="cam" pid=1856001 comm="apparmor_parser"
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.686:269): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="ch-run" pid=1856003 comm="apparmor_parser"
(end)
@@ -0,0 +1,5 @@
#### demo-hp 2026-11-14T10:59:22Z
loaded: fresh-load vs cache-load differ for: 17 profile(s): lxc-container-default lxc-container-default-cgns lxc-container-default-with-mounting lxc-container-default-with-nesting man_filter man_groff nvidia_modprobe plasmashell transmission-cli transmission-daemon transmission-gtk transmission-qt /usr/bin/man /usr/lib/connman/scripts/dhclient-script /usr/{lib/NetworkManager,libexec}/nm-dhcp-helper /usr/lib/NetworkManager/nm-dhcp-client.action /{,usr/}sbin/dhclient
loaded: fresh-load vs second fresh-load differ for: 2 profile(s): lxc-container-default-cgns lxc-container-default-with-nesting
CACHE FILES: before vs after a rewrite differ for: 1 file(s): lxc-containers
files that define more than one profile: lxc-containers nvidia_modprobe plasmashell transmission usr.bin.man usr.sbin.dhclient
@@ -0,0 +1,11 @@
#### demo-hp 2026-11-14T10:59:56Z
failed service units now: chrony.service
chrony.service ExecStart path: path=/usr/sbin/chronyd
profile names of the rewritten cache file usr.sbin.chronyd: /usr/sbin/chronyd
## match: chrony.service (failed, ExecStart /usr/sbin/chronyd = the profile's name) -> systemctl restart chrony.service
rc=0
chrony: active; confinement /usr/sbin/chronyd (enforce)
Reference ID : 25DDD139 (37.221.209.57)
System time : 0.000319822 seconds slow of NTP time
Leap status : Normal
failed service units after: (end)
@@ -0,0 +1,10 @@
#### demo-hp 2026-11-14T11:09:32Z — the watch, 10 minutes after the LAST fresh load (10:59:22Z; the first was 10:58:44Z)
apparmor DENIED lines since 10:58:40Z: 1 — expected exactly 1: the control of 11:04:38Z
2026-11-14T12:04:38+01:00 demo-hp kernel: audit: type=1400 audit(1794654278.061:617): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/chronyd" name="/etc/shadow" pid=1879323 comm="c
BLIND SECONDS (the kernel suppressed audit lines): 2026-11-14T11:58:44+01:00 demo-hp kernel: kauditd_printk_skb: 111 callbacks supp;2026-11-14T11:59:22+01:00 demo-hp kernel: kauditd_printk_skb: 109 callbacks supp;2026-11-14T12:04:38+01:00 demo-hp kernel: kauditd_printk_skb: 228 callbacks supp;
## second channel (behaviour, not the kernel log):
guest 9201: status: running; containers 26 running, 0 unhealthy, 0 restarting, newest start 2026-11-14T09:18:59.079611972Z; init confinement lxc-9201_</var/lib/lxc>//&:lxc-9201_<-var-lib-lxc>:unconfined (enforce)
guest 9202: status: running; containers 6 running, 0 unhealthy, 0 restarting, newest start 2026-11-14T09:56:41.303141464Z; init confinement lxc-9202_</var/lib/lxc>//&:lxc-9202_<-var-lib-lxc>:unconfined (enforce)
failed units on the host: 0
chrony: active System time : 0.000691365 seconds slow of NTP time
agent: active
@@ -0,0 +1,5 @@
#### demo-hp 2026-11-14T11:04:38Z — positive control for the DENIED watch
why: the kernel rate-limits its audit lines (kauditd_printk_skb: 111 callbacks suppressed at 11:58:44 and 109 at 11:59:22 local) — the reload floods ~119 STATUS lines, and the DENIED line of the planted stale profile at 11:58:07 is NOT in the log. So a count of 0 proves nothing without this control.
control command rc=1 (cat: /etc/shadow: Permission denied)
DENIED lines the kernel log shows for the control: 1
2026-11-14T12:04:38+01:00 demo-hp kernel: audit: type=1400 audit(1794654278.061:617): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/chronyd" name="/etc/shadow" pid=1879323 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
@@ -0,0 +1,9 @@
#### guest 9201 2026-11-14T11:00:04Z
apparmor userspace: apparmor 4.1.0-1 ii
ls: cannot access '/var/cache/apparmor/*/': No such file or directory
105
inactive
cat: /sys/kernel/security/apparmor/profiles: No such file or directory
container confinement: unconfined
ls: cannot access '/sys/kernel/security/apparmor/policy/profiles': No such file or directory
7.0.14-28-pve
@@ -0,0 +1,33 @@
#### demo-hp 2026-11-14T11:09:55Z — put back as found
ls: cannot access '/root/spike-profile*': No such file or directory
ls: cannot access '/root/spike-loaded*': No such file or directory
ls: cannot access '/root/spike-refresh-at': No such file or directory
cache, running kernel folder: chrony 83417e83b538 (83417e83 = the fresh compile); old folder untouched: dbf67f16ac25
loaded: 83417e83b538
#### demo-hp 2026-11-14T11:09:55Z kernel 7.0.14-28-pve
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
loaded profiles: 119
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 6c024cf01fc0 88d69a2bb62c
LOADED-DIFFERS: file=nvidia_modprobe profile=nvidia_modprobe loaded=f4ae61617079 fresh=34f355a47446
LOADED-DIFFERS: file=plasmashell profile=plasmashell loaded=17cfa46264cd fresh=2f95a57eab51
LOADED-DIFFERS: file=transmission profile=transmission-cli loaded=04b531d8a589 fresh=d9230550cd5f
LOADED-DIFFERS: file=transmission profile=transmission-daemon loaded=f46a1d2db622 fresh=d9230550cd5f
LOADED-DIFFERS: file=transmission profile=transmission-gtk loaded=ab371958b589 fresh=d9230550cd5f
LOADED-DIFFERS: file=transmission profile=transmission-qt loaded=0077a589d81e fresh=d9230550cd5f
LOADED-DIFFERS: file=usr.bin.man profile=/usr/bin/man loaded=4ee549c2ed87 fresh=3fe9acfaeb92
LOADED-DIFFERS: file=usr.bin.man profile=man_filter loaded=6dc1077b7fa5 fresh=3fe9acfaeb92
LOADED-DIFFERS: file=usr.bin.man profile=man_groff loaded=0f6549c22ad0 fresh=3fe9acfaeb92
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/usr/lib/NetworkManager/nm-dhcp-client.action loaded=b5532441db45 fresh=99b03dd8451f
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/usr/lib/connman/scripts/dhclient-script loaded=fcadbc62c577 fresh=99b03dd8451f
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/usr/{lib/NetworkManager,libexec}/nm-dhcp-helper loaded=6ca7645b1031 fresh=99b03dd8451f
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/{,usr/}sbin/dhclient loaded=3b3e18f5a89e fresh=99b03dd8451f
files=106 cache-differs=0 loaded-differs=13 profiles-not-loaded=4 not-deterministic=1 seconds=4.562202656
## the 13 LOADED-DIFFERS above are the multi-profile files after a FRESH load (03b: same policy, another hash). To leave the box exactly as found, one cache load, as at a boot: systemctl reload apparmor
reload rc=0
#### demo-hp 2026-11-14T11:10:08Z kernel 7.0.14-28-pve
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
loaded profiles: 119
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 303716fc0b9a 03ad7998e16e
files=106 cache-differs=0 loaded-differs=0 profiles-not-loaded=4 not-deterministic=1 seconds=4.587272637
chrony active; guests: 2 running; failed units: 0
@@ -0,0 +1,19 @@
# 02-make-stale.sh — demo-hp (Tier 0). Make ONE stale profile by the real mechanism: an old compiled copy in the
# running kernel's cache folder whose time is newer than its source. No package is changed.
set -u
C=/var/cache/apparmor; CUR=$C/f6da0254.0; OLD=$C/ae1a4050.0; F=usr.sbin.chronyd
LOADED=/sys/kernel/security/apparmor/policy/profiles
echo "#### $(hostname) $(date -u +%FT%TZ) kernel $(uname -r)"
echo "## before: chrony $(systemctl is-active chrony); loaded $(cat $LOADED/usr.sbin.chronyd*/raw_sha256 | cut -c1-12); cache(cur) $(sha256sum < $CUR/$F | cut -c1-12); cache(old folder) $(sha256sum < $OLD/$F | cut -c1-12); fresh $(apparmor_parser -S -K /etc/apparmor.d/$F | sha256sum | cut -c1-12)"
echo "## source mtime $(stat -c %y /etc/apparmor.d/$F | cut -c1-19); old copy mtime $(stat -c %y $OLD/$F | cut -c1-19)"
cp -a $CUR/$F /root/spike-profile-$F.good && echo "saved the good cache file to /root/spike-profile-$F.good ($(sha256sum < /root/spike-profile-$F.good | cut -c1-12))"
cp -a $OLD/$F $CUR/$F && echo "PLANTED: the old compiled copy is now in the running kernel's folder; its time $(stat -c %y $CUR/$F | cut -c1-19) is newer than the source"
echo "## reload the way the system does at a start (systemctl reload apparmor = apparmor_parser --write-cache --replace)"
systemctl reload apparmor; echo "reload rc=$?"
echo "## after the reload: loaded $(cat $LOADED/usr.sbin.chronyd*/raw_sha256 | cut -c1-12) (dbf67f16 = the profile of chrony 4.6.1; 83417e83 = of 4.8)"
echo "## chrony still runs under the replaced profile? $(systemctl is-active chrony); its confinement: $(cat /proc/$(pidof -s chronyd)/attr/current 2>/dev/null)"
T=$(date +"%Y-%m-%d %H:%M:%S")
systemctl restart chrony; echo "restart rc=$?"; sleep 6
echo "## after a restart: chrony $(systemctl is-active chrony); result=$(systemctl show chrony -p Result --value)"
journalctl -u chrony --since "$T" --no-pager -o short-iso | tail -6
journalctl -k --since "$T" --no-pager -o short-iso | grep -i 'apparmor="DENIED"' | tail -3
@@ -0,0 +1,30 @@
# 03-refresh.sh — demo-hp. The repair candidate: load every profile from a FRESH compile, cache ignored and rewritten.
set -u
P=/sys/kernel/security/apparmor/policy/profiles; CUR=/var/cache/apparmor/f6da0254.0
snap() { # name -> loaded hash, every profile
for d in $P/*/; do echo "$(cat $d/name) $(cat $d/raw_sha256 2>/dev/null) $(cat $d/mode 2>/dev/null)"; done | sort
}
guests() {
for v in 9201 9202; do
ip=$(lxc-info -n $v -p -H); echo "guest $v: $(pct status $v) init pid $ip confinement=$(cat /proc/$ip/attr/current)"
echo " containers: $(pct exec $v -- docker ps -q | wc -l) running, $(pct exec $v -- docker ps --filter health=unhealthy -q | wc -l) unhealthy, started-at-newest=$(pct exec $v -- sh -c 'docker inspect --format "{{.State.StartedAt}}" $(docker ps -q) | sort | tail -1')"
cp=$(pct exec $v -- sh -c 'docker inspect --format "{{.State.Pid}}" felhom-controller'); hp=$(grep -l "NSpid:.*[[:space:]]$cp$" /proc/[0-9]*/status 2>/dev/null | head -1 | cut -d/ -f3)
echo " the controller container's process (host pid ${hp:-?}): confinement=$(cat /proc/${hp:-0}/attr/current 2>/dev/null)"
done
echo "VM 341 (Tester 1): $(qm status 341)"
}
echo "#### $(hostname) $(date -u +%FT%TZ)"
echo "## BEFORE"; guests; snap > /root/spike-loaded-before.txt; echo "loaded profiles: $(wc -l < /root/spike-loaded-before.txt)"
echo "chrony: $(systemctl is-active chrony); loaded chrony profile $(cat $P/usr.sbin.chronyd*/raw_sha256 | cut -c1-12); cache $(sha256sum < $CUR/usr.sbin.chronyd | cut -c1-12)"
echo "disabled profiles (must stay unloaded): $(ls /etc/apparmor.d/disable 2>/dev/null | tr '\n' ' ')"
T=$(date +"%Y-%m-%d %H:%M:%S"); date +%s > /root/spike-refresh-at
echo "## THE COMMAND: apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d"
s=$(date +%s.%N); apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d; rc=$?; e=$(date +%s.%N)
echo "rc=$rc seconds=$(echo "$e - $s" | bc)"
echo "## AFTER"; guests; snap > /root/spike-loaded-after.txt; echo "loaded profiles: $(wc -l < /root/spike-loaded-after.txt)"
echo "profiles whose loaded hash changed:"; diff <(cut -d' ' -f1,2 /root/spike-loaded-before.txt) <(cut -d' ' -f1,2 /root/spike-loaded-after.txt) | grep '^>' | sed 's/^> / /'
echo "profiles added or gone (names):"; diff <(cut -d' ' -f1 /root/spike-loaded-before.txt) <(cut -d' ' -f1 /root/spike-loaded-after.txt) | grep '^[<>]' || echo " none"
echo "modes changed:"; diff <(cut -d' ' -f1,3 /root/spike-loaded-before.txt) <(cut -d' ' -f1,3 /root/spike-loaded-after.txt) | grep '^[<>]' || echo " none"
echo "chrony profile: loaded $(cat $P/usr.sbin.chronyd*/raw_sha256 | cut -c1-12); cache $(sha256sum < $CUR/usr.sbin.chronyd | cut -c1-12) mtime $(stat -c %y $CUR/usr.sbin.chronyd | cut -c1-19); fresh $(apparmor_parser -S -K /etc/apparmor.d/usr.sbin.chronyd | sha256sum | cut -c1-12)"
echo "chrony unit, NOT restarted by the command: $(systemctl is-active chrony)"
echo "kernel lines since the command:"; journalctl -k --since "$T" --no-pager -o short-iso | grep -i apparmor | tail -5; echo "(end)"
@@ -0,0 +1,14 @@
# 03b — why did 13 other profiles change their loaded hash in 03? Compare: loaded after a CACHE load (the system's
# reload) against loaded after a FRESH load, and the cache files before/after a rewrite. Nothing is stale now.
P=/sys/kernel/security/apparmor/policy/profiles; CUR=/var/cache/apparmor/f6da0254.0
snap() { for d in $P/*/; do echo "$(cat $d/name) $(cat $d/raw_sha256 2>/dev/null)"; done | sort; }
csnap() { sha256sum $CUR/* | sed "s#$CUR/##" | awk '{print $2, $1}' | sort; }
echo "#### $(hostname) $(date -u +%FT%TZ)"
snap > /tmp/s-fresh1; csnap > /tmp/c1
systemctl reload apparmor; snap > /tmp/s-cache
apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d; snap > /tmp/s-fresh2; csnap > /tmp/c2
echo "loaded: fresh-load vs cache-load differ for: $(diff /tmp/s-fresh1 /tmp/s-cache | grep -c '^>') profile(s): $(diff /tmp/s-fresh1 /tmp/s-cache | grep '^>' | cut -d' ' -f2 | tr '\n' ' ')"
echo "loaded: fresh-load vs second fresh-load differ for: $(diff /tmp/s-fresh1 /tmp/s-fresh2 | grep -c '^>') profile(s): $(diff /tmp/s-fresh1 /tmp/s-fresh2 | grep '^>' | cut -d' ' -f2 | tr '\n' ' ')"
echo "CACHE FILES: before vs after a rewrite differ for: $(diff /tmp/c1 /tmp/c2 | grep -c '^>') file(s): $(diff /tmp/c1 /tmp/c2 | grep '^>' | cut -d' ' -f2 | tr '\n' ' ')"
echo "files that define more than one profile: $(for f in /etc/apparmor.d/*; do [ -f $f ] && [ $(apparmor_parser -N $f 2>/dev/null | wc -l) -gt 1 ] && basename $f; done | tr '\n' ' ')"
rm -f /tmp/s-fresh1 /tmp/s-cache /tmp/s-fresh2 /tmp/c1 /tmp/c2
@@ -0,0 +1,12 @@
# 04 — the narrow restart rule, by hand: a FAILED service unit whose ExecStart program is the path-name of a profile
# whose cached copy the refresh had to rewrite. (On this box: chrony, made stale in 02.)
echo "#### $(hostname) $(date -u +%FT%TZ)"
echo "failed service units now: $(systemctl list-units --type=service --state=failed --no-legend --plain | awk '{print $1}' | tr '\n' ' ')"
for u in $(systemctl list-units --type=service --state=failed --no-legend --plain | awk '{print $1}'); do
echo " $u ExecStart path: $(systemctl show -p ExecStart --value $u | grep -o 'path=[^ ;]*' | head -1)"
done
echo "profile names of the rewritten cache file usr.sbin.chronyd: $(apparmor_parser -N /etc/apparmor.d/usr.sbin.chronyd | tr '\n' ' ')"
echo "## match: chrony.service (failed, ExecStart /usr/sbin/chronyd = the profile's name) -> systemctl restart chrony.service"
systemctl restart chrony.service; echo "rc=$?"; sleep 8
echo "chrony: $(systemctl is-active chrony); confinement $(cat /proc/$(pidof -s chronyd)/attr/current)"; chronyc -n tracking | grep -E "System time|Leap|Reference"
echo "failed service units after: $(systemctl list-units --type=service --state=failed --no-legend --plain | awk '{print $1}' | tr '\n' ' ')(end)"
@@ -0,0 +1,29 @@
# detect.sh — READ ONLY. For every profile file in /etc/apparmor.d: does the profile the KERNEL has loaded equal a
# fresh compile of the file on disk? (the n100-clock-2026-11-11 method, per file AND per loaded profile)
# fresh = sha256 of `apparmor_parser -S -K <file>` (compile to stdout: no load, cache not read, not written)
# loaded = policy/profiles/*/raw_sha256 of each profile name the file defines (`apparmor_parser -N`)
# cache = sha256 of the cached binary in the running kernel's cache folder
main() {
echo "#### $(hostname) $(date -u +%FT%TZ) kernel $(uname -r)"
P=/sys/kernel/security/apparmor/policy/profiles
cur=""; for d in /var/cache/apparmor/*/; do [ "$(sha256sum < $d/.features | cut -d' ' -f1)" = "$(aa-features-abi -x 2>/dev/null | sha256sum | cut -d' ' -f1)" ] && cur=$d; done
echo "cache folder of the running kernel: ${cur:-none} all: $(ls -d /var/cache/apparmor/*/ | tr '\n' ' ')"
declare -A L; for d in $P/*/; do n=$(cat $d/name 2>/dev/null); [ -n "$n" ] && L["$n"]=$(cat $d/raw_sha256 2>/dev/null); done
echo "loaded profiles: ${#L[@]}"
n=0; stale=0; cdiff=0; notloaded=0; nd=0; t0=$(date +%s.%N)
for f in /etc/apparmor.d/*; do [ -f "$f" ] || continue; b=$(basename "$f"); [ -e /etc/apparmor.d/disable/$b ] && continue
fresh=$(apparmor_parser -S -K "$f" 2>/dev/null | sha256sum | cut -d' ' -f1); n=$((n+1))
again=$(apparmor_parser -S -K "$f" 2>/dev/null | sha256sum | cut -d' ' -f1)
# CONTROL ON THE METHOD (found 2026-11-14): a file whose two fresh compiles differ cannot be judged by a hash at all
[ "$fresh" = "$again" ] || { nd=$((nd+1)); echo "NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): $b ${fresh:0:12} ${again:0:12}"; continue; }
c="-"; [ -n "$cur" ] && [ -f "$cur$b" ] && c=$(sha256sum < "$cur$b" | cut -d' ' -f1)
[ "$c" != "-" ] && [ "$c" != "$fresh" ] && { cdiff=$((cdiff+1)); echo "CACHE-DIFFERS: $b cache=${c:0:12} fresh=${fresh:0:12} cache-mtime=$(stat -c %y "$cur$b" | cut -c1-19) source-mtime=$(stat -c %y "$f" | cut -c1-19)"; }
for name in $(apparmor_parser -N "$f" 2>/dev/null); do
l=${L["$name"]:-}
if [ -z "$l" ]; then notloaded=$((notloaded+1)); continue; fi
[ "$l" = "$fresh" ] || { stale=$((stale+1)); echo "LOADED-DIFFERS: file=$b profile=$name loaded=${l:0:12} fresh=${fresh:0:12}"; }
done
done
echo "files=$n cache-differs=$cdiff loaded-differs=$stale profiles-not-loaded=$notloaded not-deterministic=$nd seconds=$(echo "$(date +%s.%N) - $t0" | bc)"
}
main </dev/null
@@ -0,0 +1,43 @@
### demo-hp 2026-11-14T10:55:37Z — dry run of the working-copy wrapper (sha256 8d5052e4548341e0…)
os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[]
{
"refused": null,
"failed": null,
"kernel_cleanup": {
"default": "7.0.14-28-pve",
"dry_run": true,
"installed": {
"7.0.14-26-pve": [
"proxmox-kernel-7.0.14-26-pve-signed"
],
"7.0.14-28-pve": [
"proxmox-kernel-7.0.14-28-pve-signed"
]
},
"kept": {
"7.0.14-26-pve": [
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
"the kernel step's `default_before`"
],
"7.0.14-28-pve": [
"running",
"the kernel step's `to`",
"the default in grub.cfg",
"the pinned default file",
"newest installed"
]
},
"leftovers": [],
"newest": "7.0.14-28-pve",
"not_run": [],
"outcome": "dry_run",
"phase": "good",
"pinned_default": "7.0.14-28-pve",
"refusals": [],
"removal": [],
"removed": [],
"running": "7.0.14-28-pve",
"would": "nothing"
},
"pass_seconds": 0.2
}
@@ -0,0 +1,60 @@
### root@192.168.0.154 2026-11-14T10:55:38Z — dry run of the working-copy wrapper (sha256 8d5052e4548341e0…)
os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=3 keep=2 removal=1 not_run=['no-ring0-mark'] refusals=[]
{
"refused": null,
"failed": null,
"kernel_cleanup": {
"default": "7.0.14-22-pve",
"dry_run": true,
"installed": {
"7.0.14-20-pve": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"7.0.14-22-pve": [
"proxmox-kernel-7.0.14-22-pve-signed"
],
"7.0.2-6-pve": [
"proxmox-kernel-7.0.2-6-pve-signed"
]
},
"kept": {
"7.0.14-22-pve": [
"running",
"the kernel step's `to`",
"the default in grub.cfg",
"the pinned default file",
"newest installed"
],
"7.0.2-6-pve": [
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
"the kernel step's `default_before`"
]
},
"leftovers": [],
"newest": "7.0.14-22-pve",
"not_run": [
{
"code": "no-ring0-mark",
"reason": "this box has no root-owned ring-0 mark \u2014 the clean-up runs on ring 0 only in this release"
}
],
"outcome": "dry_run",
"phase": "good",
"pinned_default": "7.0.14-22-pve",
"refusals": [],
"removal": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"removed": [],
"running": "7.0.14-22-pve",
"simulation": {
"installs": [],
"rc": 0,
"removes": [
"proxmox-kernel-7.0.14-20-pve-signed"
]
},
"would": "not_run"
},
"pass_seconds": 1.4
}
@@ -0,0 +1,23 @@
== agent red-proofs (internal/osupdate), 2026-11-14T10:54:58Z
-- MUTATION 1: kernelCleanupSkip always answers "" (every 'may not run' rule removed)
--- FAIL: TestKernel_Ring0ToldNightStagesThenReboots (0.00s)
kernel_test.go:83: the reboot must end the night, last call = map[lane:slow layer:kernel mode:kernel-cleanup packages:[] release_id:kernel-cleanup ring:0 run_id:20261004T040000Z trigger:night vmid:9201]
--- FAIL: TestKernelCleanup_WhenItMayNotRun (0.00s)
kernel_test.go:489: a kernel step is due tonight (ring 0): the clean-up ran (1 call(s)); calls = guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply,kernel:kernel-status,kernel:apply,kernel:kernel-reb
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.022s
FAIL
-- MUTATION 2: 'nothing' sends a report (return l.finish in the nothing case)
--- FAIL: TestKernelCleanup_NothingAndNotRunSendNoReport (0.00s)
kernel_test.go:423: nothing: a report was made: {RunID:20261004T040000Z Layer:kernel Trigger:night Mode:kernel-cleanup Ring:1 ReleaseID:kernel-cleanup Outcome:nothing Healthy:false HealthReason: VMID:9201 Upgraded:[]
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.021s
FAIL
-- MUTATION 3: the report carries the wrapper's kernel view
--- FAIL: TestKernelCleanup_NightOrderOnceAndNoNames (0.00s)
kernel_test.go:401: the clean-up's report names a kernel (the hub reads that as a kernel step): {RunID:20261004T040000Z Layer:kernel Trigger:night Mode:kernel-cleanup Ring:0 ReleaseID:kernel-cleanup Outcome:cleaned Healthy:true HealthReason:removed 1 old k
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.010s
FAIL
-- the real file:
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 4.973s
@@ -0,0 +1,74 @@
== MUTATION: K1 removed (the rule's own check of the removal set)
elif by_name[n] in protected or by_name[n] == newest: -> elif False:
tests: 7 mutant: Ran 7 tests in 0.119s | FAILED (failures=7) real: Ran 7 tests in 0.116s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : running: {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running':
AssertionError: 0 != 2 : from: {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7
== MUTATION: the simulation check removed (S1/S2)
if removal and not [x for x in refuse if x["code"] in ("K1", "N1")]: -> if False:
tests: 3 mutant: Ran 3 tests in 0.055s | FAILED (failures=3) real: Ran 3 tests in 0.047s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
== MUTATION: the phase check removed (P1)
elif phase != "good": -> elif False:
tests: 1 mutant: Ran 1 test in 0.019s | FAILED (failures=1) real: Ran 1 test in 0.072s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'staged', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.1
== MUTATION: the ring-0 check removed
if not ring0 and not not_run: -> if False:
tests: 1 mutant: Ran 1 test in 0.021s | FAILED (failures=1) real: Ran 1 test in 0.047s | OK
RED on the mutant, GREEN on the real file
AssertionError: 'removed' != 'not_run'
== MUTATION: the flag check removed (P2)
if flag: -> if False:
tests: 1 mutant: Ran 1 test in 0.021s | FAILED (failures=1) real: Ran 1 test in 0.022s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '7.0.14-26-pve', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'runni
== MUTATION: the default check removed (P3)
if pinned is None or pinned != dflt or pinned not in images: -> if False:
tests: 1 mutant: Ran 1 test in 0.024s | FAILED (failures=1) real: Ran 1 test in 0.044s | OK
RED on the mutant, GREEN on the real file
AssertionError: 3 != 2 : {'failed': {'rc': 3, 'reason': 'the default in grub.cfg moved from 7.0.14-28-pve to 7.0.14-26-pve', 'step': 'verify'}, 'kernel': {'default': '7.0.14-26-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': '
== MUTATION: the not-on-newest rule removed
if running and newest and running in images and running != newest: -> if False:
tests: 1 mutant: Ran 1 test in 0.032s | FAILED (failures=1) real: Ran 1 test in 0.028s | OK
RED on the mutant, GREEN on the real file
AssertionError: 'removed' != 'not_run'
== MUTATION: the fell-back rule removed
elif phase in KERNEL_ENDED_NOT_GOOD: -> elif False:
tests: 1 mutant: Ran 1 test in 0.035s | FAILED (failures=1) real: Ran 1 test in 0.101s | OK
RED on the mutant, GREEN on the real file
AssertionError: 'step-not-good' not found in ['not-on-newest'] : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'fell_back', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at'
== MUTATION: the plan-names check removed (C1)
if mode == "kernel-cleanup": -> if False:
tests: 1 mutant: Ran 1 test in 0.020s | FAILED (failures=1) real: Ran 1 test in 0.001s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
== MUTATION: the no-state-file rule removed
if raw_state is None or not st: -> if False:
tests: 1 mutant: Ran 1 test in 0.036s | FAILED (failures=1) real: Ran 1 test in 0.031s | OK
RED on the mutant, GREEN on the real file
AssertionError: 2 != 0 : {'kernel_cleanup': {'default': '7.0.14-28-pve', 'dry_run': False, 'installed': {'7.0.14-20-pve': ['proxmox-kernel-7.0.14-20-pve-signed'], '7.0.14-22-pve': ['proxmox-kernel-7.0.14-22-pve-signed'], '7.0.14-2
== MUTATION: the 20-hour window removed
if times and self.r.now() - max(times) < KERNEL_MIN_GAP: -> if False:
tests: 1 mutant: Ran 1 test in 0.022s | FAILED (failures=1) real: Ran 1 test in 0.016s | OK
RED on the mutant, GREEN on the real file
AssertionError: 'removed' != 'not_run'
== MUTATION: the dry run purges (dry-run verdict removed)
if dry: -> if False:
tests: 1 mutant: Ran 1 test in 0.022s | FAILED (errors=1) real: Ran 1 test in 0.049s | OK
RED on the mutant, GREEN on the real file
KeyError: 'would'
== MUTATION: repair-first removed
if not dry and not not_run: -> if False:
tests: 1 mutant: Ran 1 test in 0.032s | FAILED (errors=1) real: Ran 1 test in 0.031s | OK
RED on the mutant, GREEN on the real file
KeyError: 'repair'
== MUTATION: the after-purge proof removed
if problems: -> if False:
tests: 1 mutant: Ran 1 test in 0.024s | FAILED (failures=1) real: Ran 1 test in 0.022s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 3 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
14 mutations, 0 not a red-proof
@@ -0,0 +1,25 @@
== agent red-proofs, profile refresh, 2026-11-14T11:04:02Z
-- MUTATION 1: the appliance check removed
--- FAIL: TestProfileRefresh_ByoHostIsNeverAsked (0.00s)
profilerefresh_test.go:54: a BYO host was asked: "refreshed" [map[layer:host mode:profile-refresh release_id:profile-refresh]]
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.010s
FAIL
-- MUTATION 2: the gate's answer ignored (always go on)
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate [build failed]
FAIL
-- MUTATION 3: the plan names a unit
--- FAIL: TestProfileRefresh_OneCallNoNamesNoHubReport (0.00s)
profilerefresh_test.go:42: the plan carries more than release_id, layer and mode: map[layer:host mode:profile-refresh release_id:profile-refresh units:[chrony.service]]
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.011s
FAIL
-- the real file:
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.009s
-- MUTATION 2, done again (the first try did not compile, so it proved nothing): the gate's answer ignored
--- FAIL: TestProfileRefresh_WaitsForTheHeavyOpGate (0.00s)
profilerefresh_test.go:75: gate tries 1, released 0, calls 1
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.012s
FAIL
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate (cached)
@@ -0,0 +1,53 @@
== MUTATION: the restart rule widened: any failed unit restarts (the program check removed)
if prog not in programs: -> if False:
tests: 2 mutant: Ran 2 tests in 0.010s | FAILED (errors=2) real: Ran 2 tests in 0.006s | OK
RED on the mutant, GREEN on the real file
KeyError: '/usr/sbin/postfix'
KeyError: '/usr/sbin/chronyd'
== MUTATION: the stale-copy condition removed: every profile file counts, not only a rewritten one
for key in res["rewritten"]: -> for key in after:
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: Lists differ: ['chrony.service'] != []
== MUTATION: a profile name that is not a path counts as a program
if PROFILE_PATH_NAME_RE.match(n): -> if True:
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: Tuples differ: (0, ['nvidia.service']) != (0, [])
== MUTATION: once-per-kernel removed
needed = rec.get("kernel") != kernel -> needed = True
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: Tuples differ: (0, 'refreshed', 2) != (0, 'not_needed', 1)
== MUTATION: the cache is not skipped (the system's own reload)
PROFILE_REFRESH_CMD = ["apparmor_parser", "--replace", "--skip-read-cache", "--write-cache", "--", PROFILE_DIR -> PROFILE_REFRESH_CMD = ["apparmor_parser", "--replace", "--write-cache", "--", PROFILE_DIR]
tests: 1 mutant: Ran 1 test in 0.006s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: ('host', ['apparmor_parser', '--replace', '--skip-read-cache', '--write-cache', '--', '/etc/apparmor.d']) not found in [('host', ['uname', '-r']), ('host', ['apparmor_parser', '--replace', '--write-cache', '--', '/
== MUTATION: a failed parser still writes the record
r.log(f"os-apply: FAILED rc={rc} step=profile-refresh (no record written: the next start tries again)") -> r.log(f"os-apply: FAILED rc={rc} step=profile-refresh (no record written: the next start tries again)")
tests: 1 mutant: Ran 1 test in 0.006s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: '/var/lib/felhom-os-apply/profile-refresh.json' unexpectedly found in {'/var/lib/felhom-os-apply/profile-refresh.json': '{"kernel": "7.0.14-28-pve"}'}
== MUTATION: the plan may name things (F1 removed)
for k in ("packages", "allow_new", "signed", "select", "undo", "snapshot", "profiles", "units", "force"): -> for k in ("packages", "allow_new", "signed", "select", "undo", "snapshot", "profiles", "units", "force"):
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.001s | OK
RED on the mutant, GREEN on the real file
AssertionError: 0 != 2 : {'layer': 'host', 'mode': 'profile-refresh', 'pass_seconds': 0.0, 'profile_refresh': {'dry_run': False, 'kernel': '7.0.14-28-pve', 'last_refresh_at': None, 'last_refresh_kernel': None, 'not_restarted': [],
== MUTATION: the appliance rule removed
self.check_appliance() -> return self.profile_refresh(plan)
tests: 1 mutant: Ran 1 test in 0.005s | FAILED (errors=1) real: Ran 1 test in 0.000s | OK
RED on the mutant, GREEN on the real file
TypeError: 'NoneType' object is not subscriptable
== MUTATION: the dry run refreshes
if dry: -> if False:
tests: 1 mutant: Ran 1 test in 0.006s | FAILED (errors=1) real: Ran 1 test in 0.000s | OK
RED on the mutant, GREEN on the real file
KeyError: 'would'
== MUTATION: the restart cap removed
if len(res["restarted"]) >= PROFILE_RESTART_MAX: -> if False:
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
RED on the mutant, GREEN on the real file
AssertionError: Tuples differ: (0, 5, 0) != (0, 3, 2)
10 mutations, 0 not a red-proof
@@ -0,0 +1,4 @@
== the published binary, downloaded with no login 2026-11-14T11:13:30Z
6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92
agent-0.174.0: go1.27.4
felhom-agent 0.174.0
@@ -0,0 +1,2 @@
felhom: agent felhom-agent 0.173.0 wrapper sha a5f67f015fbc bundle 0.172.1 3e7acaee4500 host_id tester-1-d70be4
demo-hp: agent felhom-agent 0.173.0 wrapper sha a5f67f015fbc bundle 0.172.1 3e7acaee4500 host_id demo-hp-bb76ea
@@ -0,0 +1,8 @@
== tester-1-d70be4 2026-11-14T11:13:39Z agent_update 0.174.0
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=3a80f37e50d321ee00db919bb6c659fc expires=2026-11-14T11:58:39Z
uploaded signed op to the hub jobs queue
rc=0
== demo-hp-bb76ea 2026-11-14T11:13:39Z agent_update 0.174.0
signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=c9fb994bd9952af35d137219442067b4 expires=2026-11-14T11:58:39Z
uploaded signed op to the hub jobs queue
rc=0
@@ -0,0 +1,13 @@
### felhom 2026-11-14T11:18:17Z
2026-11-14T12:14:53+01:00 felhom felhom-agent[51939]: time=2026-11-14T12:14:53.096+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=df59dd26428b9120 op=agent_update
2026-11-14T12:14:55+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:14:55.235+01:00 level=INFO msg="felhom-agent daemon starting" version=0.174.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-14T12:14:55+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:14:55.356+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
2026-11-14T12:15:56+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:15:56.990+01:00 level=WARN msg="selfupdate: update committed" version=0.174.0 wrapper=""
active
0
### demo-hp 2026-11-14T11:18:21Z
2026-11-14T12:18:00+01:00 demo-hp felhom-agent[535281]: time=2026-11-14T12:18:00.785+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=0ef54dd1311c3a04 op=agent_update
2026-11-14T12:18:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:18:02.833+01:00 level=INFO msg="felhom-agent daemon starting" version=0.174.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
2026-11-14T12:18:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:18:02.956+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
active
0
@@ -0,0 +1,8 @@
== tester-1-d70be4 2026-11-14T11:19:06Z agent_config_update 0.174.0
signed: op=agent_config_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=f632b7fb9a3ae1667d0050ce176a86d5 expires=2026-11-14T12:04:06Z
uploaded signed op to the hub jobs queue
rc=0
== demo-hp-bb76ea 2026-11-14T11:19:06Z agent_config_update 0.174.0
signed: op=agent_config_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=61a7b130297809cab3b5e26dfc1f7288 expires=2026-11-14T12:04:06Z
uploaded signed op to the hub jobs queue
rc=0
@@ -0,0 +1,24 @@
### felhom 2026-11-14T11:30:25Z
2026-11-14T12:14:55+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:14:55.356+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
2026-11-14T12:30:09+01:00 felhom felhom-os-apply[3433197]: os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False
2026-11-14T12:30:09+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:09.431+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False"
2026-11-14T12:30:09+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:09.431+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.174.0 bundle="{\"agent_version\": \"0.174.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261114T113008Z-before-0.174.0\", \"removed\": [], \"same\": 30,
2026-11-14T12:30:10+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:10.471+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded=""
2026-11-14T12:30:10+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:10.471+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=de2c899634da22aa op=agent_config_update
2026-11-14T12:30:11+01:00 felhom felhom-os-apply[3433509]: os-apply: PROFILE REFRESH DONE kernel=7.0.14-22-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]
2026-11-14T12:30:11+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:11.181+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PROFILE REFRESH DONE kernel=7.0.14-22-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]"
2026-11-14T12:30:11+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:11.181+01:00 level=INFO msg="osupdate: profile refresh — security profiles reloaded from a fresh compile" mode=profile-refresh why="config bundle installed" kernel=7.0.14-22-pve rewritten=1 wrapper_seconds=0.5 took=700ms
wrapper sha 9d03203766040a97 record: { "at": "2026-11-14T11:30:11Z", "kernel": "7.0.14-22-pve", "restarted": [], "rewritten": [ "f6da0254.0/lxc-containers" ]}
chrony active; failed units 0; guests running 1
### demo-hp 2026-11-14T11:33:26Z
2026-11-14T12:18:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:18:02.956+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
2026-11-14T12:33:16+01:00 demo-hp felhom-os-apply[1980789]: os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False
2026-11-14T12:33:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:16.771+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False"
2026-11-14T12:33:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:16.771+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.174.0 bundle="{\"agent_version\": \"0.174.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261114T113315Z-before-0.174.0\", \"removed\": [], \"same\": 30,
2026-11-14T12:33:17+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:17.923+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded=""
2026-11-14T12:33:17+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:17.923+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=524eef508b29fa0c op=agent_config_update
2026-11-14T12:33:18+01:00 demo-hp felhom-os-apply[1981111]: os-apply: PROFILE REFRESH DONE kernel=7.0.14-28-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]
2026-11-14T12:33:18+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:18.531+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PROFILE REFRESH DONE kernel=7.0.14-28-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]"
2026-11-14T12:33:18+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:18.532+01:00 level=INFO msg="osupdate: profile refresh — security profiles reloaded from a fresh compile" mode=profile-refresh why="config bundle installed" kernel=7.0.14-28-pve rewritten=1 wrapper_seconds=0.5 took=600ms
wrapper sha 9d03203766040a97 record: { "at": "2026-11-14T11:33:18Z", "kernel": "7.0.14-28-pve", "restarted": [], "rewritten": [ "f6da0254.0/lxc-containers" ]}
chrony active; failed units 0; guests running 2
@@ -0,0 +1,57 @@
### felhom (Tester 1) 2026-11-14T11:30:43Z — the dry run through the released agent and its own sudo line
agent felhom-agent 0.174.0; wrapper 9d03203766040a97; mark: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false}
time=2026-11-14T12:30:45.231+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=3 keep=2 removal=1 not_run=['no-ring0-mark'] refusals=[]"
--- kernel clean-up — DRY RUN (nothing was changed) ---
{
"default": "7.0.14-22-pve",
"dry_run": true,
"installed": {
"7.0.14-20-pve": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"7.0.14-22-pve": [
"proxmox-kernel-7.0.14-22-pve-signed"
],
"7.0.2-6-pve": [
"proxmox-kernel-7.0.2-6-pve-signed"
]
},
"kept": {
"7.0.14-22-pve": [
"running",
"the kernel step's `to`",
"the default in grub.cfg",
"the pinned default file",
"newest installed"
],
"7.0.2-6-pve": [
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
"the kernel step's `default_before`"
]
},
"leftovers": [],
"newest": "7.0.14-22-pve",
"not_run": [
{
"code": "no-ring0-mark",
"reason": "this box has no root-owned ring-0 mark — the clean-up runs on ring 0 only in this release"
}
],
"outcome": "dry_run",
"phase": "good",
"pinned_default": "7.0.14-22-pve",
"refusals": [],
"removal": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"removed": [],
"running": "7.0.14-22-pve",
"simulation": {
"installs": [],
"rc": 0,
"removes": [
"proxmox-kernel-7.0.14-20-pve-signed"
]
},
"would": "not_run"
}
@@ -0,0 +1,116 @@
OPERATOR'S YES for the temporary ring-0 mark on Tester 1: given in the session, 2026-11-14 (the STOP's first question).
#### felhom (Tester 1) 2026-11-14T14:16:19Z agent felhom-agent 0.174.0 wrapper 9d03203766040a97
busy? vzdump 0, apt 0, guest status: running, containers 10 running / 0 unhealthy
## BEFORE
kernels:
running 7.0.14-22-pve; grub.cfg default: gnulinux-7.0.14-22-pve-advanced
sha256: pinned d96c890b9749 state 8008e8e2192c gen01 f2f56d59c635 gen42 c34907ce6859 flag [felhom_next= ]
/ free bytes: 39449448448; /boot kernels: 7.0.14-20-pve 7.0.14-22-pve 7.0.2-6-pve ; /lib/modules: 7.0.14-20-pve 7.0.14-22-pve 7.0.2-6-pve
one-shot entries in grub.cfg: felhom-oneshot-7.0.14-20-pve felhom-oneshot-7.0.14-22-pve felhom-oneshot-7.0.2-6-pve
## the ring-0 mark, set by hand FOR THIS TEST (Tester 1 is ring 1; the clean-up runs on ring 0 only in this release)
mark before: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false} (root:root 644)
mark set: { "host_id": "tester-1-d70be4", "ring0_slow_lane": true} (root:root 644)
## SCENARIO 1 — the clean-up
wrapper exit code: 0
os-apply: REPAIR configured=0 journal=0 fixed=0
os-apply: KERNEL CLEANUP running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=3 keep=2 removal=1 not_run=[] refusals=[]
os-apply: KERNEL CLEANUP DONE removed=1 (proxmox-kernel-7.0.14-20-pve-signed) kept=7.0.14-22-pve, 7.0.2-6-pve gained=1124024320 B seconds=8.5 leftovers=0; default 7.0.14-22-pve (proved from grub.cfg)
{
"refused": null,
"failed": null,
"repair": {
"clean_after": true,
"fixed": 0,
"half_configured_before": 0,
"journal_before": 0
},
"pass_seconds": 10.1,
"kernel_cleanup": {
"outcome": "removed",
"not_run": [],
"refusals": [],
"kept": {
"7.0.14-22-pve": [
"running",
"the kernel step's `to`",
"the default in grub.cfg",
"the pinned default file",
"newest installed"
],
"7.0.2-6-pve": [
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
"the kernel step's `default_before`"
]
},
"removal": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"removed": [
"proxmox-kernel-7.0.14-20-pve-signed"
],
"bytes_gained": 1124024320,
"leftovers": [],
"seconds": 8.5,
"simulation": {
"installs": [],
"rc": 0,
"removes": [
"proxmox-kernel-7.0.14-20-pve-signed"
]
}
}
}
## AFTER
kernels:
running 7.0.14-22-pve; grub.cfg default: gnulinux-7.0.14-22-pve-advanced
sha256: pinned d96c890b9749 state 8008e8e2192c gen01 f2f56d59c635 gen42 c34907ce6859 flag [felhom_next= ]
/ free bytes: 40573468672; /boot kernels: 7.0.14-22-pve 7.0.2-6-pve ; /lib/modules: 7.0.14-22-pve 7.0.2-6-pve
one-shot entries in grub.cfg: felhom-oneshot-7.0.14-22-pve felhom-oneshot-7.0.2-6-pve
## SCENARIO 2 — the same call again
wrapper exit code: 0
os-apply: REPAIR configured=0 journal=0 fixed=0
os-apply: KERNEL CLEANUP running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[]
os-apply: KERNEL CLEANUP nothing to remove
{
"refused": null,
"failed": null,
"repair": {
"clean_after": true,
"fixed": 0,
"half_configured_before": 0,
"journal_before": 0
},
"pass_seconds": 0.1,
"kernel_cleanup": {
"outcome": "nothing",
"not_run": [],
"refusals": [],
"kept": {
"7.0.14-22-pve": [
"running",
"the kernel step's `to`",
"the default in grub.cfg",
"the pinned default file",
"newest installed"
],
"7.0.2-6-pve": [
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
"the kernel step's `default_before`"
]
},
"removal": [],
"removed": [],
"bytes_gained": null,
"leftovers": [],
"seconds": null,
"simulation": null
}
}
## the mark put back
mark now: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false} (root:root 644); saved copy: ls: cannot access '/root/os-trust.json.before-2a': No such file or directory
## and with the mark back, the program answers:
wrapper exit code: 0
os-apply: KERNEL CLEANUP running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=2 keep=2 removal=0 not_run=['no-ring0-mark'] refusals=[]
"outcome": "not_run",
"code": "no-ring0-mark",
guest status: running, containers 10 running / 0 unhealthy; dpkg audit: []
@@ -0,0 +1,7 @@
#### the N100 (demo-felhom) BEFORE the release — reads only; DooPlex clock 2026-11-14T11:32:03.295Z
offset of the N100 to DooPlex: -32.55 s (negative = the N100 is behind); the ssh round trip was 0.35 s
agent felhom-agent 0.173.0; kernel 7.0.14-28-pve; chrony: failed (result exit-code); failed units: chrony.service
chrony profile: loaded dbf67f16ac25, cache (7.0.14 folder) dbf67f16ac25, fresh compile 83417e83b538
busy? vzdump 0, apt 0, task lock files: pve-migrate-9201 pve-storage-local-lvm ; guest: 9201 running demo-felhom ;
kernels:
/dev/mapper/pve-root 94G 35G 55G 40% /
@@ -0,0 +1,19 @@
#### demo-hp 2026-11-14T11:33:49Z agent felhom-agent 0.174.0
## 0. the box as it is
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
would: nothing | refusals: [] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
## 1. scenario 3 — a step that is not over: phase set to 'judging' in the state file
would: refuse | refusals: ["P1 a kernel step is in phase 'judging' — not ended (or not a phase this wrapper knows)"] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
put back:
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
## 2. scenario 4a — the one-shot flag names the kept older kernel
would: refuse | refusals: ["P2 the one-shot flag names '7.0.14-26-pve'"] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
put back:
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
## 3. scenario 4b — the pinned default file names another kernel than grub.cfg
would: refuse | refusals: ["P3 the pinned default '7.0.14-26-pve' and grub.cfg's default '7.0.14-28-pve' do not name the same installed kerne"] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
put back:
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
## 4. as found?
IDENTICAL to step 0: state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
ls: cannot access '/root/2a-*': No such file or directory
@@ -0,0 +1,4 @@
== demo-felhom-8363b5 2026-11-14T11:34:09Z agent_update 0.174.0
signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=20e3bd9eb794fa7137294ed8ca1390c3 expires=2026-11-14T12:19:09Z
uploaded signed op to the hub jobs queue
rc=0
@@ -0,0 +1,4 @@
== demo-felhom-8363b5 2026-11-14T11:37:49Z agent_config_update 0.174.0
signed: op=agent_config_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=29c5412f563d8131d25c69aa55f6fd17 expires=2026-11-14T12:22:49Z
uploaded signed op to the hub jobs queue
rc=0
@@ -0,0 +1,59 @@
#### the N100 (demo-felhom) AFTER the release — DooPlex clock 2026-11-14T11:52:11Z
offset of the N100 to DooPlex: +0.177 s (before: -32.55 s); the ssh round trip was 0.36 s
agent felhom-agent 0.174.0; wrapper 9d03203766040a97; chrony: active; failed units: 0
Reference ID : C1E3C502 (193.227.197.2)
System time : 0.000001782 seconds slow of NTP time
Last offset : +0.002664797 seconds
Leap status : Normal
chrony profile: loaded 83417e83b538, cache 83417e83b538, fresh 83417e83b538
record: { "at": "2026-11-14T11:51:13Z", "kernel": "7.0.14-28-pve", "restarted": [ "chrony.service" ], "rewritten": [ "f6da0254.0/lxc-containers", "f6da0254.0/usr.sbin.chronyd" ]}
guest: status: running; containers 9 running, 0 unhealthy, newest container start 2026-11-14T11:52:10.648640606Z
## the full minute, the host journal (agent, wrapper, chrony, systemd):
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949497]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949539]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- pgrep -x dhclient
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949539]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949542]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- ip -4 -o addr show dev eth0
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949542]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949539]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949542]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949553]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker exec felhom-controller cat /opt/docker/felhom-controller/controller.yaml
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949553]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949573]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949573]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949573]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949586]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949586]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949553]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949586]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949608]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949608]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949608]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949611]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949611]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949611]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949619]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' cloudflared
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949619]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:08+01:00 demo-felhom sudo[1949619]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:09+01:00 demo-felhom sudo[1949801]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts115109-host-facts.json
2026-11-14T12:51:09+01:00 demo-felhom sudo[1949801]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949801]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949874]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949874]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949874]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:10+01:00 demo-felhom sshd-session[1949878]: Connection closed by 127.0.0.1 port 50390
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949879]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949879]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949879]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949882]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949882]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949882]: pam_unix(sudo:session): session closed for user root
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949886]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949886]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
## the controller in the guest, the same minutes (WARN/ERROR lines, and any line on time):
(end)
## the hub, events of demo-felhom since 11:50Z:
(end)
#### the N100, 2026-11-14T14:23:46Z — 2.5 hours after the repair
offset of the N100 to DooPlex: +0.178 s; the ssh round trip was 0.36 s
chrony: active since Sat 2026-11-14 12:51:13 CET; System time : 0.000158301 seconds fast of NTP time
failed units: 0; guest containers: 9 running, 0 unhealthy; controller boot-gate lines since: 0
@@ -0,0 +1,101 @@
#### demo-felhom 2026-11-14T11:52:28Z — which container started near the clock step?
2026-11-14T04:49:14.901603074Z restarts=0 status=running health=healthy prowlarr
2026-11-14T09:18:23.620814926Z restarts=0 status=running health=healthy felhom-controller
2026-11-14T11:52:05.808655933Z restarts=0 status=running health=healthy radarr
2026-11-14T11:52:09.614378919Z restarts=0 status=running health=healthy sonarr
2026-11-14T11:52:10.648640606Z restarts=0 status=running health=healthy qbittorrent
## docker events 11:50–11:54Z (start/die/restart/health):
2026-11-14T11:52:00.406970341Z container kill fc2cd3bd36b94bbae38a99e8b476cf40b9e6fd142811f067647ad0f96077fd99 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com.docker.compose.co
2026-11-14T11:52:05.629973236Z container die fc2cd3bd36b94bbae38a99e8b476cf40b9e6fd142811f067647ad0f96077fd99 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com.docker.compose.con
2026-11-14T11:52:05.927443897Z container start e7b713a45c209013c5bc914d4f3cb9c8697627e3e6e1cb1aba6f99a7679a9de9 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com.docker.compose.c
2026-11-14T11:52:06.175065238Z container kill e5351e8bed94fabe527f7cb26760a106f6ae1eb0c4a577d3c1cd5b03fa3b6266 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com.docker.compose.co
2026-11-14T11:52:09.418552075Z container die e5351e8bed94fabe527f7cb26760a106f6ae1eb0c4a577d3c1cd5b03fa3b6266 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com.docker.compose.con
2026-11-14T11:52:09.752602177Z container start 90d435fcdb3ed6e1731469312725c7edae2f6363da11999c0fcb34fd47fe7452 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com.docker.compose.c
2026-11-14T11:52:09.977460590Z container kill 82bd9f8caf4124d1fff6b9b9fa1799088805815c63bb6835c9557c160ef51b3a (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com.docker.compose.co
2026-11-14T11:52:10.407038534Z container die 82bd9f8caf4124d1fff6b9b9fa1799088805815c63bb6835c9557c160ef51b3a (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com.docker.compose.con
2026-11-14T11:52:10.787260774Z container start 16e8e756487f2ea6ede83737f01b6c707292aa1b2f9004d3d71d11bd0131b8a1 (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com.docker.compose.c
2026-11-14T11:52:11.003795092Z container health_status: healthy e7b713a45c209013c5bc914d4f3cb9c8697627e3e6e1cb1aba6f99a7679a9de9 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com
2026-11-14T11:52:14.829102987Z container health_status: healthy 90d435fcdb3ed6e1731469312725c7edae2f6363da11999c0fcb34fd47fe7452 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com
2026-11-14T11:52:15.847713728Z container health_status: healthy 16e8e756487f2ea6ede83737f01b6c707292aa1b2f9004d3d71d11bd0131b8a1 (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com
## the controller around it:
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml
2026/11/14 11:52:11 [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (1 storage path(s))
2026/11/14 11:52:12 [INFO] [stacks] Stack sonarr post-start status:
2026/11/14 11:52:12 [INFO] [stacks] sonarr lscr.io/linuxserver/sonarr:4.0.20@sha256:f247545d23ba8b233d6604575347e48a623fe6ad75dda02348bf81917f3b5c06 running Up 3 seconds (health: starting)
2026/11/14 11:52:14 [INFO] [stacks] Stack qbittorrent post-start status:
2026/11/14 11:52:14 [INFO] [stacks] qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1@sha256:92bfd78d731e254ba64f62b77b0696add6447a33642ee5feda106a26b285aa7f running Up 3 seconds (health: starting)
2026/11/14 11:52:17 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
2026/11/14 11:52:27 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
2026/11/14 11:52:27 [INFO] Health probes: 3 ok (of 3 probed)
2026/11/14 11:52:37 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
2026/11/14 11:52:47 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
2026/11/14 11:50:24 [INFO] [scheduler] Running job: stack-scan
2026/11/14 11:50:24 [INFO] [scheduler] Running job: hold-report
2026/11/14 11:50:24 [INFO] [scheduler] Job hold-report completed (took 0s)
2026/11/14 11:50:24 [INFO] [stacks] ScanStacks complete: 68 stacks found (5 deployed, 63 available)
2026/11/14 11:50:24 [INFO] [scheduler] Job stack-scan completed (took 87ms)
2026/11/14 11:50:24 [INFO] [scheduler] Running job: agent-channel-health
2026/11/14 11:50:25 [INFO] [scheduler] Job agent-channel-health completed (took 890ms)
2026/11/14 11:51:14 [INFO] Health probes: 1 ok (of 1 probed)
2026/11/14 11:51:57 [INFO] [scheduler] Running job: hold-report
2026/11/14 11:51:57 [INFO] [scheduler] Job hold-report completed (took 0s)
2026/11/14 11:51:57 [INFO] Health probes: 4 ok (of 4 probed)
2026/11/14 11:51:57 [INFO] [scheduler] Running job: agent-channel-health
2026/11/14 11:51:58 [INFO] [scheduler] Job agent-channel-health completed (took 818ms)
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/hdd_1] before recreating drive-backed apps
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app radarr (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:00 [INFO] [stacks] Stopping stack: radarr
2026/11/14 11:52:05 [INFO] [stacks] Stack radarr stopped successfully (took 5.3s)
2026/11/14 11:52:05 [INFO] [stacks] Starting stack: radarr
2026/11/14 11:52:05 [INFO] [stacks] media_library radarr: per-drive binds 0 []; drives not bound: []
2026/11/14 11:52:05 [INFO] [stacks] Stack radarr started successfully (took 0.3s)
2026/11/14 11:52:06 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app sonarr (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:06 [INFO] [stacks] Stopping stack: sonarr
2026/11/14 11:52:09 [INFO] [stacks] Stack radarr post-start status:
2026/11/14 11:52:09 [INFO] [stacks] radarr lscr.io/linuxserver/radarr:6.4.4@sha256:adb6c09d6b729ea5e642c99cea35af72702ef476bf4763f153299ac5db9f0b4f running Up 3 seconds (health: starting)
2026/11/14 11:52:09 [INFO] [stacks] Stack sonarr stopped successfully (took 3.4s)
2026/11/14 11:52:09 [INFO] [stacks] Starting stack: sonarr
2026/11/14 11:52:09 [INFO] [stacks] media_library sonarr: per-drive binds 0 []; drives not bound: []
2026/11/14 11:52:09 [INFO] [stacks] Stack sonarr started successfully (took 0.3s)
2026/11/14 11:52:09 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app qbittorrent (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:09 [INFO] [stacks] Stopping stack: qbittorrent
2026/11/14 11:52:10 [INFO] [stacks] Stack qbittorrent stopped successfully (took 0.5s)
2026/11/14 11:52:10 [INFO] [stacks] Starting stack: qbittorrent
2026/11/14 11:52:10 [INFO] [stacks] media_library qbittorrent: per-drive binds 0 []; drives not bound: []
2026/11/14 11:52:10 [INFO] [stacks] Stack qbittorrent started successfully (took 0.3s)
2026/11/14 11:52:10 [INFO] [gate] boot 1794112550-1438: re-syncing FileBrowser mounts against the live binds
2026/11/14 11:52:10 [INFO] [settings] Settings saved
2026/11/14 11:52:11 [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (1 storage path(s))
2026/11/14 11:52:12 [INFO] [stacks] Stack sonarr post-start status:
2026/11/14 11:52:12 [INFO] [stacks] sonarr lscr.io/linuxserver/sonarr:4.0.20@sha256:f247545d23ba8b233d6604575347e48a623fe6ad75dda02348bf81917f3b5c06 running Up 3 seconds (health: starting)
#### demo-felhom 2026-11-14T11:53:19Z — did anything else read the clock step as a restart?
crash guard: {"armed":true,"boot_id":"72899d26-2524-4e06-9e8e-8099775d1e51","config":{"LIMIT":3,"PANIC_SECONDS":10,"REARM_HOURS":24,"WINDOW_MINUTES":60},"kernel_panic":10,"last_boot_at":"2026-11-08T04:35:18Z","last_boot_unclean":false,"tripped":false,"unclean_boots":[],"unclean_boots_24h":0,"unclean_boots_in_window":0,"updated_at":"2026-11-14T11:03:05Z","version":1}
uptime since: 2026-11-08 05:35:50; /proc/stat btime: btime 1794112550 = 2026-11-08T04:35:50Z; boot_id 72899d26-2524-4e06-9e8e-8099775d1e51
agent journal since the step (WARN/ERROR, boot, power, restart words):
2026-11-14T12:51:56+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:51:56.457+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
2026-11-14T12:52:16+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:52:16.469+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
2026-11-14T12:52:36+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:52:36.511+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
2026-11-14T12:52:56+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:52:56.477+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
2026-11-14T12:53:16+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:53:16.525+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
(end)
kernel step state phase: good
controller boot-gate lines today:
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/hdd_1] before recreating drive-backed apps
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app radarr (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:06 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app sonarr (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:09 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app qbittorrent (state=running) onto /mnt/felhom-drives/hdd_1
2026/11/14 11:52:10 [INFO] [gate] boot 1794112550-1438: re-syncing FileBrowser mounts against the live binds
@@ -0,0 +1,19 @@
#### demo-felhom (the N100) 2026-11-14T11:53:34Z — the DRY RUN through the released agent: what tonight removes. Nothing is changed.
time=2026-11-14T12:53:35.431+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]"
would: remove | not_run: [] | refusals: []
installed: ['7.0.14-20-pve', '7.0.14-22-pve', '7.0.14-23-pve', '7.0.14-24-pve', '7.0.14-25-pve', '7.0.14-26-pve', '7.0.14-28-pve', '7.0.2-6-pve']
KEPT:
7.0.14-26-pve ["the kernel step's `from` (the last kernel that booted well; a self-revert boots it)", "the kernel step's `default_before`"]
7.0.14-28-pve ['running', "the kernel step's `to`", 'the default in grub.cfg', 'the pinned default file', 'newest installed']
WILL BE REMOVED ( 6 ):
proxmox-kernel-7.0.14-20-pve-signed
proxmox-kernel-7.0.14-22-pve-signed
proxmox-kernel-7.0.14-23-pve-signed
proxmox-kernel-7.0.14-24-pve-signed
proxmox-kernel-7.0.14-25-pve-signed
proxmox-kernel-7.0.2-6-pve-signed
simulation: {'installs': [], 'rc': 0, 'removes': ['proxmox-kernel-7.0.14-20-pve-signed', 'proxmox-kernel-7.0.14-22-pve-signed', 'proxmox-kernel-7.0.14-23-pve-signed', 'proxmox-kernel-7.0.14-24-pve-signed', 'proxmox-kernel-7.0.14-25-pve-signed', 'proxmox-kernel-7.0.2-6-pve-signed']}
sizes now: /lib/modules 7.6G, /boot 906M, / free 55G
972M /lib/modules/7.0.14-20-pve;972M /lib/modules/7.0.14-22-pve;972M /lib/modules/7.0.14-23-pve;972M /lib/modules/7.0.14-24-pve;973M /lib/modules/7.0.14-25-pve;973M /lib/modules/7.0.14-26-pve;973M /lib/modules/7.0.14-28-pve;970M /lib/modules/7.0.2-6-pve;
the hub block for this box (kernel step due tonight?): {'ring': 0, 'enabled': True, 'kernel': None}
last night run marker: 2026-11-14T04:51:34Z
@@ -0,0 +1,17 @@
#### felhom (Tester 1) BEFORE the restart 2026-11-14T14:16:51Z
kernel 7.0.14-22-pve; boot_id 6407d71b-1d89-4748-8655-49c6de8cf61b; up since 2026-11-10 15:43:22
crash guard: {'armed': True, 'tripped': False, 'last_boot_at': '2026-11-10T14:43:22Z', 'last_boot_unclean': False, 'unclean_boots_in_window': 0, 'boot_id': '6407d71b-1d89-4748-8655-49c6de8cf61b'}
busy? vzdump 0 apt 0; guest status: running; containers 10 running / 0 unhealthy
RESTART (systemctl reboot) at 2026-11-14T14:16:54Z
host answers, agent active, guest running after 156 s
containers: 10 running, 0 starting, 0 unhealthy after 239 s (or the 540 s limit)
#### felhom (Tester 1) AFTER the restart 2026-11-14T14:21:03Z
kernel 7.0.14-22-pve (before: 7.0.14-22-pve); boot_id 7a721957-7e95-476c-be40-52d0dafc7284; up since 2026-11-14 15:19:11
crash guard: {'armed': True, 'tripped': False, 'last_boot_at': '2026-11-14T14:19:11Z', 'last_boot_unclean': False, 'unclean_boots_in_window': 0, 'boot_id': '7a721957-7e95-476c-be40-52d0dafc7284'}
## the host health rule: daemons active active active active active ; guest status: running; containers 10 running / 0 unhealthy; controller healthy; tunnel container: running
kernels: /boot 7.0.14-22-pve 7.0.2-6-pve ; grub default gnulinux-7.0.14-22-pve-advanced; failed units 0; chrony active
mark: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false}
## the agent at this start:
2026-11-14T15:19:22+01:00 felhom felhom-agent[1148]: time=2026-11-14T15:19:22.533+01:00 level=INFO msg="felhom-agent daemon starting" version=0.174.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
2026-11-14T15:19:22+01:00 felhom felhom-agent[1148]: time=2026-11-14T15:19:22.731+01:00 level=INFO msg="osupdate: profile refresh not needed" mode=profile-refresh why="agent start" kernel=7.0.14-22-pve reason="the profiles were refreshed on this kernel (7.0.14-22-pve) at 2026-11-14T11:30:11Z"
profile-refresh record: { "at": "2026-11-14T11:30:11Z", "kernel": "7.0.14-22-pve", "restarted": [], "rewritten": [ "f6da0254.0/lxc-containers" ]}
@@ -0,0 +1,38 @@
## vouch read-back 2026-11-14T14:21:57Z — channel 1: the hub's log (the pod's clock is CET)
2026/11/14 15:21:45 [INFO] Artifact manifest set: agent=0.174.0 golden=0.352.1 min_agent="0.158.1" wrapper_sha=false bundle_sha="7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f"
2026/11/14 15:21:45 [INFO] artifact vouch timing (R-1346): registry probes 119ms, sha lookups 76ms, bundle sha 10ms, fleet check + save 222ms, total 428ms
## channel 2: the hub's System page (CC's token)
Global controller floor: 0.352.1 · vouched agent: 0.174.0
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
look
Tester4-46836f Tester4
kernel step: — · the host needs a restart since 2026-11-14 · root files behind the vouched agent's · agent behind: 0.173.0 → 0.174.0 · could not read: default kernel
look
tester-1-d70be4 Tester 1
partial: no running customer guest · crash guard not installed · could not read: running kernel, next-boot kernel, default kernel, host Debian, held packages, kernel.panic, crash restarts, guest Debian, Docker engine, containerd, Docker live-restore
demo-felhom-8363b5 10 s no 0 armed 0.174.0
demo-hp-bb76ea 10 s no 0 armed 0.174.0
tester-1-d70be4 unknown no unknown not installed 0.174.0
root-files cell titles: ['behind the vouched agent 0.174.0&#39;s bundle — send it with a signed agent_config_upd']
demo-felhom-8363b5 Demo Ügyfél 0.174.0 9.2.26 7.0.14-28-pve ONLINE 1/2 5% 23% 37% running (connected) 37% local demo-hp-bb76ea D
demo-hp-bb76ea Demo HP 0.174.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 48% 42% running (connected) 62% local-lvm tester-1-d70be4
tester-1-d70be4 Tester 1 0.174.0 9.2.2 7.0.14-22-pve ONLINE 1/2 0% 33% 30% running (connected) 30% local Felhom Hub 0.181.0
Tester4-46836f Tester4 0.173.0 9.2.2 7.0.2-6-pve DOWN 1/1 11% 20% 21% absent (no cloudflared container in the guest (no tunn
## the hub's System page, read 2026-11-14T14:25:58Z
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
look
Tester4-46836f Tester4
kernel step: — · the host needs a restart since 2026-11-14 · root files behind the vouched agent's · agent behind: 0.173.0 → 0.174.0 · could not read: default kernel
look
tester-1-d70be4 Tester 1
partial: no running customer guest · crash guard not installed · could not read: running kernel, next-boot kernel, default kernel, host Debian, held packages, kernel.panic, crash restarts, guest Debian, Docker engine, containerd, Docker live-restore
Crash guard and root files
Box kernel.panic Oops Crash restarts 24 h Crash guard Root files
Tester4-46836f 10 s no 0 armed 0.173.0
demo-felhom-8363b5 10 s no 0 armed 0.174.0
demo-hp-bb76ea 10 s no 0 armed 0.174.0
tester-1-d70be4 unknown no unknown not installed 0.174.0
Felhom Hub 0.181.0
(function(){var h=location.hash&&document.getElementById(location.hash.slice(1));if(!h)return;var d=h.closest('details');while(d){d.open=true;d=d.parentElement&
@@ -0,0 +1,3 @@
felhom: felhom-agent 0.174.0; sha256 6efbb97809135483; go in the running binary: 2 go1.27.4; unit active restarts 0
demo-hp: felhom-agent 0.174.0; sha256 6efbb97809135483; go in the running binary: 2 go1.27.4; unit active restarts 0
demo-felhom: felhom-agent 0.174.0; sha256 6efbb97809135483; go in the running binary: 2 go1.27.4; unit active restarts 0
@@ -0,0 +1,25 @@
# Release 2a — the box removes its old kernels, and refreshes its security profiles — evidence (2026-11-14)
Report: `REPORT-release-2a-2026-11-14.md` (repo root). The profile spike: `../profile-refresh-spike-2026-11-14/`.
Agent v0.174.0 (felhom-agent `cc63dd5`), vouched 2026-11-14 15:21:45 CET.
| File | What it shows |
|---|---|
| `A/redproof.txt` | the clean-up's wrapper tests: 14 mutations (K1, the simulation, the phase, the ring-0 mark, the flag, the default, not-on-newest, fell back, the plan's names, no state file, the 20-hour window, the dry run, repair-first, the after-purge proof) — each RED on the mutant, GREEN on the real file |
| `A/redproof-agent.txt` | the agent's clean-up tests: 3 mutations, each red |
| `A/01-dryrun-preview-*.txt` | the working-copy wrapper's dry run on demo-hp and Tester 1, before the commit |
| `B/redproof.txt`, `B/redproof-agent.txt` | the profile refresh: 10 + 3 mutations, each red (one agent mutation did not compile at the first try and was done again; both are in the file) |
| `D/01-published-binary.txt` | the published agent, downloaded with no login: sha256, `go1.27.4` |
| `D/02`–`D/06` | delivery to Tester 1 and demo-hp: before; the signed `agent_update`; the journals (the old wrapper refuses the new mode at the agent's start); the signed bundle; bundle installed 77/77 and the profile refresh right after it |
| `D/07-tester1-dryrun.txt` | Tester 1 through the released agent: „not run", it has no ring-0 mark (scenario 8, live) |
| `D/08-tester1-cleanup.txt` | **the first real removal by the program** (scenario 1), the second call (scenario 2), the mark set for the test and put back. Started by hand — see the report |
| `D/09-n100-before.txt`, `D/11`, `D/12` | the N100 before (−32.55 s, chrony failed, the stale profile); its two signed jobs |
| `D/13-n100-after.txt` | the N100 after: +0.18 s, chrony active, the profile is the fresh compile; the same 2.5 hours later |
| `D/13b-n100-container-near-the-step.txt` | the controller recreated three apps 9 s after the clock step, and why (R-1395); the crash guard was not fooled |
| `D/14-n100-dryrun-tonight.txt` | what the N100 removes tonight: 6 of 8 kernels |
| `D/10-demo-hp-dryruns-and-refusals.txt` | demo-hp: nothing to remove; P1, P2, P3 by dry run against states set and put back; identical after |
| `D/15-tester1-reboot.txt` | one orderly restart of Tester 1: the same kernel, healthy after 239 s, a clean stop, „profile refresh not needed" |
| `D/16-vouch-readback.txt` | the vouch from the hub's log and from its System page |
| `D/17-go-version-on-boxes.txt` | the running agent on the three boxes: sha256 and `go1.27.4` |
Not kept as files: the release script's own output (it prints a repository address; the CHANGELOG has the two sha256s).