Release 2a: agent 0.174.0 vouched — old kernels removed by the box (11 §5.11.1 built), security profiles refreshed after a kernel change (11 §5.12, spike + build); R-1388, R-1301, R-1391 closed; R-1394, R-1395 filed; D82
gates / gates (push) Successful in 2m1s
gates / gates (push) Successful in 2m1s
This commit is contained in:
1 parent
a2173830c1
commit
8955b4e55f
51 files changed
+1385
-19
No files matched your search
@@ -0,0 +1,60 @@
|
||||
# Spike — can a box refresh its security profiles by itself? — 2026-11-14 (release 2a, Part B)
|
||||
|
||||
**Answer: yes. The spike is clean, so the mode was built in the same run** (agent v0.174.0, `11-os-updates.md` §5.12).
|
||||
Venue: demo-hp (Tier 0), host kernel 7.0.14-28-pve, guests 9201 (26 containers) and 9202 (6) running the whole time;
|
||||
reads on Tester 1. Nothing on the N100 was touched. No package was changed. Documents read:
|
||||
`audits/n100-clock-2026-11-11/README.md` (the diagnosis and its method), `architecture/11-os-updates.md` §5.4.1, §5.9,
|
||||
§5.11, rows R-1388, R-1301.
|
||||
|
||||
## The six questions
|
||||
|
||||
| # | Question | Result | File |
|
||||
|---|---|---|---|
|
||||
| 1 | Does the detector (loaded hash against a fresh compile) work, and why does `lxc-containers` always "differ"? | It works for one-profile files. **`lxc-containers` compiles to a different binary every time** (two fresh compiles in a row differ; it includes a whole folder). So its hash can never judge it; the detector now compiles twice and says so. On demo-hp and Tester 1: 106 files, 0 stale, 1 not judgeable, 4.5 s | `evidence/01-detect-*.txt`, `tools/detect.sh` |
|
||||
| 2 | Can one stale profile be made by the real mechanism, with no package change? | Yes. The old compiled chrony profile (of 4.6.1, sha `dbf67f16…`) was copied from the old kernel's cache folder into the running kernel's; its time (2026-08-21) is newer than the source's (2026-07-01). The system's own reload then **loaded the old one**, and a restart of chrony **failed exactly as on the N100** („Could not send notification to $NOTIFY_SOCKET", twice) | `evidence/02-make-stale.txt` |
|
||||
| 3 | The smallest repair | `apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d` — **0.51 s**. After it: the same 119 profiles in the same modes; chrony's loaded profile and its cached copy are the fresh compile (`83417e83…`); both guests running, 26 and 6 containers, none restarted (newest start time unchanged), every guest init still confined by its own profile | `evidence/03-refresh.txt` |
|
||||
| 3b | Why did 17 other profiles change their loaded hash? | Not staleness. For a file that defines several profiles the kernel's hash differs between a cache load and a fresh load of the **same** policy. **The only reliable sign of a stale copy is the cached FILE changing when it is rewritten** (here: only the planted one, and `lxc-containers`) | `evidence/03b-why-others-changed.txt` |
|
||||
| 4 | What starts the failed program again? | The refresh does not (chrony stayed `failed`). The narrow rule, tried by hand: a **failed service** whose **ExecStart program is the path-name of a profile whose cached copy the refresh rewrote** → one `systemctl restart`. chrony came back, 0.0003 s off | `evidence/04-restart-rule.txt` |
|
||||
| 5 | When should it run? | See the options below; the pick is „when the box runs another kernel than at the last refresh" | — |
|
||||
| 6 | Can the guest have the same staleness? | Not today: in guest 9201 the AppArmor package is installed, its service is inactive, there is no profile cache, and the kernel's profile list is not visible | `evidence/06-guest-read.txt` |
|
||||
|
||||
**No new refusal by the kernel in 10 minutes** (`evidence/05-watch.txt`): 1 `DENIED` line, and it is the control.
|
||||
**The control was needed** (`evidence/05a-denied-watch-control.txt`): the kernel rate-limits its audit lines, each
|
||||
reload writes about 119 `STATUS` lines, and the `DENIED` line that the planted stale profile made at 10:58:07 UTC is
|
||||
**not in the kernel log at all** („kauditd_printk_skb: 111 callbacks suppressed"). A count of 0 alone would have proved
|
||||
nothing. The second channel is behaviour: no container restarted, no failed unit.
|
||||
|
||||
## Decision 4 — what restarts the program (two options)
|
||||
|
||||
| | Option A (picked) | Option B |
|
||||
|---|---|---|
|
||||
| Rule | a failed service whose own program's cached profile was stale | a fixed list: `chrony.service` |
|
||||
| Covers | every program this fault can stop | the one case seen |
|
||||
| Risk | a unit that failed for another reason, and whose profile was also stale, gets ONE start try | none beyond chrony |
|
||||
| Cost | 3 `systemctl` reads per failed unit, only when a path-named profile was rewritten | a new release for each next program |
|
||||
|
||||
Never „restart whatever failed", never a unit named by the plan, at most 3 units a run.
|
||||
|
||||
## Decision 5 — when it runs (two options)
|
||||
|
||||
| | When the kernel differs from the last refresh's (picked) | After every agent start |
|
||||
|---|---|---|
|
||||
| Catches the fault | yes: a stale copy can only appear for a kernel that did NOT run when a profile changed, so the box always starts on „another kernel" before the kernel loads it. A box with no record yet counts as changed (this is what repairs the N100) | yes |
|
||||
| Cost | one refresh per kernel change (about weekly) | one per agent start (about daily): 0.5 s, and each run floods the kernel log with ~119 lines and **hides other audit lines for a few seconds** (measured above) |
|
||||
|
||||
The record is the wrapper's own root-owned file; the agent asks at every start and after a bundle install, and the
|
||||
wrapper answers „not needed".
|
||||
|
||||
## Put back
|
||||
|
||||
`evidence/07-put-back.txt`: the spike's files removed from `/root`; the planted copy is gone (the cache holds the fresh
|
||||
compile, the old kernel's folder is untouched); one cache load (`systemctl reload apparmor`) put the loaded hashes
|
||||
back as found; the detector reads 0 stale; chrony active; both guests running; no failed unit.
|
||||
|
||||
## Not measured
|
||||
|
||||
- A box that boots with `proxmox-boot-tool` or without AppArmor (none in the fleet).
|
||||
- A profile in `complain` mode or in `/etc/apparmor.d/disable` (demo-hp has none; the parser is given the folder, as the
|
||||
system's own loader is).
|
||||
- The refresh while a backup or a restore-test runs (the agent takes the heavy-op gate so it never does).
|
||||
- The real case: the N100. It is the proof of the built mode, in the release.
|
||||
@@ -0,0 +1,5 @@
|
||||
#### demo-hp 2026-11-14T10:57:26Z kernel 7.0.14-28-pve
|
||||
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
|
||||
loaded profiles: 119
|
||||
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 03ad7998e16e a5f0e0eaf4e4
|
||||
files=106 cache-differs=0 loaded-differs=0 profiles-not-loaded=4 not-deterministic=1 seconds=4.556914622
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
#### felhom 2026-11-14T10:57:31Z kernel 7.0.14-22-pve
|
||||
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
|
||||
loaded profiles: 118
|
||||
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 7ea3f12cf701 b1b2529f2c17
|
||||
files=106 cache-differs=0 loaded-differs=0 profiles-not-loaded=4 not-deterministic=1 seconds=4.179027032
|
||||
@@ -0,0 +1,19 @@
|
||||
#### demo-hp 2026-11-14T10:58:06Z kernel 7.0.14-28-pve
|
||||
## before: chrony active; loaded 83417e83b538; cache(cur) 83417e83b538; cache(old folder) dbf67f16ac25; fresh 83417e83b538
|
||||
## source mtime 2026-07-01 22:28:16; old copy mtime 2026-08-21 17:42:38
|
||||
saved the good cache file to /root/spike-profile-usr.sbin.chronyd.good (83417e83b538)
|
||||
PLANTED: the old compiled copy is now in the running kernel's folder; its time 2026-08-21 17:42:38 is newer than the source
|
||||
## reload the way the system does at a start (systemctl reload apparmor = apparmor_parser --write-cache --replace)
|
||||
reload rc=0
|
||||
## after the reload: loaded dbf67f16ac25 (dbf67f16 = the profile of chrony 4.6.1; 83417e83 = of 4.8)
|
||||
## chrony still runs under the replaced profile? active; its confinement: /usr/sbin/chronyd (enforce)
|
||||
Job for chrony.service failed because the control process exited with error code.
|
||||
See "systemctl status chrony.service" and "journalctl -xeu chrony.service" for details.
|
||||
restart rc=1
|
||||
## after a restart: chrony failed; result=exit-code
|
||||
2026-11-14T11:58:07+01:00 demo-hp chronyd[1853258]: Loaded seccomp filter (level 1)
|
||||
2026-11-14T11:58:07+01:00 demo-hp chronyd[1853258]: Fatal error : Could not send notification to $NOTIFY_SOCKET
|
||||
2026-11-14T11:58:07+01:00 demo-hp chronyd[1853258]: Fatal error : Could not send notification to $NOTIFY_SOCKET
|
||||
2026-11-14T11:58:07+01:00 demo-hp systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
|
||||
2026-11-14T11:58:07+01:00 demo-hp systemd[1]: chrony.service: Failed with result 'exit-code'.
|
||||
2026-11-14T11:58:07+01:00 demo-hp systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
|
||||
@@ -0,0 +1,55 @@
|
||||
#### demo-hp 2026-11-14T10:58:32Z
|
||||
## BEFORE
|
||||
guest 9201: status: running init pid 2159 confinement=lxc-9201_</var/lib/lxc>//&:lxc-9201_<-var-lib-lxc>:unconfined (enforce)
|
||||
containers: 26 running, 0 unhealthy, started-at-newest=2026-11-14T09:18:59.079611972Z
|
||||
the controller container's process (host pid ?): confinement=
|
||||
guest 9202: status: running init pid 2117 confinement=lxc-9202_</var/lib/lxc>//&:lxc-9202_<-var-lib-lxc>:unconfined (enforce)
|
||||
containers: 6 running, 0 unhealthy, started-at-newest=2026-11-14T09:56:41.303141464Z
|
||||
the controller container's process (host pid ?): confinement=
|
||||
VM 341 (Tester 1): status: running
|
||||
loaded profiles: 119
|
||||
chrony: failed; loaded chrony profile dbf67f16ac25; cache dbf67f16ac25
|
||||
disabled profiles (must stay unloaded):
|
||||
## THE COMMAND: apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d
|
||||
rc=0 seconds=.514403371
|
||||
## AFTER
|
||||
guest 9201: status: running init pid 2159 confinement=lxc-9201_</var/lib/lxc>//&:lxc-9201_<-var-lib-lxc>:unconfined (enforce)
|
||||
containers: 26 running, 0 unhealthy, started-at-newest=2026-11-14T09:18:59.079611972Z
|
||||
the controller container's process (host pid ?): confinement=
|
||||
guest 9202: status: running init pid 2117 confinement=lxc-9202_</var/lib/lxc>//&:lxc-9202_<-var-lib-lxc>:unconfined (enforce)
|
||||
containers: 6 running, 0 unhealthy, started-at-newest=2026-11-14T09:56:41.303141464Z
|
||||
the controller container's process (host pid ?): confinement=
|
||||
VM 341 (Tester 1): status: running
|
||||
loaded profiles: 119
|
||||
profiles whose loaded hash changed:
|
||||
lxc-container-default 68d1e8408c2e9241ee6081869502ed73059174251ebb1af548f86afc72b06bec
|
||||
lxc-container-default-cgns 2e6dec9bc9637ea459d1675047ef6a887bd41bc4511447cbbb61cbf23d6b6fed
|
||||
lxc-container-default-with-mounting caeab50e1e3007a362e37a64c9adab1334aa4a9decacbba35a69301ed7bcf9bf
|
||||
lxc-container-default-with-nesting 659040965dfea57ecb76f61643ce276e7ee35550e5089e6ed34862b6ae963d33
|
||||
man_filter 6dc1077b7fa57c23a446f7bdcca7e27457182dcb12a635219604c23a16d414c9
|
||||
man_groff 0f6549c22ad03ed89f4c857aa4d332f2a0f2b777f6f07939cad9486291645815
|
||||
nvidia_modprobe f4ae61617079f7e868204f1bd5046778570fc5ed2c9a3cad9aa6c8ff34c657e7
|
||||
plasmashell 17cfa46264cdf3b2ca1181848a56604d4efc64b328c8d57b221529e4d22045ea
|
||||
transmission-cli 04b531d8a589bfa5b8c154d961f58835a57f4d35d17520915b971f9a15d3af3d
|
||||
transmission-daemon f46a1d2db62218fa9f9be1afacfe256c2b1b0c6ed971117f38f883e1d61d49cb
|
||||
transmission-gtk ab371958b58947d437928b327736e4c49cb83b3db1f2531d9765a936dc50ffe6
|
||||
transmission-qt 0077a589d81e621e1faa806a460110820e10cccb8b1916b0e2a3b747a99d0894
|
||||
/usr/bin/man 4ee549c2ed8742dbe5c1be3fc0b9b3e08ea7f77463366094ad454123e8fc03ed
|
||||
/usr/lib/connman/scripts/dhclient-script fcadbc62c5777ecee8b1ec1d6f0abab4131df7fc1dc7f7b99635e689f3e60ee8
|
||||
/usr/{lib/NetworkManager,libexec}/nm-dhcp-helper 6ca7645b1031d446bc65bf4bc9eec5cad0f7e70bf1d841271e972d79ebdf10d3
|
||||
/usr/lib/NetworkManager/nm-dhcp-client.action b5532441db4544da68a7ac5bc6980f0aaa34dc28fb9ab702221d8ec04cca38b3
|
||||
/usr/sbin/chronyd 83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1
|
||||
/{,usr/}sbin/dhclient 3b3e18f5a89e78eb104fd6377ab41ec76dcb9823f799a95dc6d3dac787e429e2
|
||||
profiles added or gone (names):
|
||||
none
|
||||
modes changed:
|
||||
none
|
||||
chrony profile: loaded 83417e83b538; cache 83417e83b538 mtime 2026-08-21 17:42:38; fresh 83417e83b538
|
||||
chrony unit, NOT restarted by the command: failed
|
||||
kernel lines since the command:
|
||||
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.683:265): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="brave" pid=1855998 comm="apparmor_parser"
|
||||
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.684:266): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="buildah" pid=1855999 comm="apparmor_parser"
|
||||
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.686:267): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="busybox" pid=1856000 comm="apparmor_parser"
|
||||
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.686:268): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="cam" pid=1856001 comm="apparmor_parser"
|
||||
2026-11-14T11:58:44+01:00 demo-hp kernel: audit: type=1400 audit(1794653924.686:269): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="ch-run" pid=1856003 comm="apparmor_parser"
|
||||
(end)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
#### demo-hp 2026-11-14T10:59:22Z
|
||||
loaded: fresh-load vs cache-load differ for: 17 profile(s): lxc-container-default lxc-container-default-cgns lxc-container-default-with-mounting lxc-container-default-with-nesting man_filter man_groff nvidia_modprobe plasmashell transmission-cli transmission-daemon transmission-gtk transmission-qt /usr/bin/man /usr/lib/connman/scripts/dhclient-script /usr/{lib/NetworkManager,libexec}/nm-dhcp-helper /usr/lib/NetworkManager/nm-dhcp-client.action /{,usr/}sbin/dhclient
|
||||
loaded: fresh-load vs second fresh-load differ for: 2 profile(s): lxc-container-default-cgns lxc-container-default-with-nesting
|
||||
CACHE FILES: before vs after a rewrite differ for: 1 file(s): lxc-containers
|
||||
files that define more than one profile: lxc-containers nvidia_modprobe plasmashell transmission usr.bin.man usr.sbin.dhclient
|
||||
@@ -0,0 +1,11 @@
|
||||
#### demo-hp 2026-11-14T10:59:56Z
|
||||
failed service units now: chrony.service
|
||||
chrony.service ExecStart path: path=/usr/sbin/chronyd
|
||||
profile names of the rewritten cache file usr.sbin.chronyd: /usr/sbin/chronyd
|
||||
## match: chrony.service (failed, ExecStart /usr/sbin/chronyd = the profile's name) -> systemctl restart chrony.service
|
||||
rc=0
|
||||
chrony: active; confinement /usr/sbin/chronyd (enforce)
|
||||
Reference ID : 25DDD139 (37.221.209.57)
|
||||
System time : 0.000319822 seconds slow of NTP time
|
||||
Leap status : Normal
|
||||
failed service units after: (end)
|
||||
@@ -0,0 +1,10 @@
|
||||
#### demo-hp 2026-11-14T11:09:32Z — the watch, 10 minutes after the LAST fresh load (10:59:22Z; the first was 10:58:44Z)
|
||||
apparmor DENIED lines since 10:58:40Z: 1 — expected exactly 1: the control of 11:04:38Z
|
||||
2026-11-14T12:04:38+01:00 demo-hp kernel: audit: type=1400 audit(1794654278.061:617): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/chronyd" name="/etc/shadow" pid=1879323 comm="c
|
||||
BLIND SECONDS (the kernel suppressed audit lines): 2026-11-14T11:58:44+01:00 demo-hp kernel: kauditd_printk_skb: 111 callbacks supp;2026-11-14T11:59:22+01:00 demo-hp kernel: kauditd_printk_skb: 109 callbacks supp;2026-11-14T12:04:38+01:00 demo-hp kernel: kauditd_printk_skb: 228 callbacks supp;
|
||||
## second channel (behaviour, not the kernel log):
|
||||
guest 9201: status: running; containers 26 running, 0 unhealthy, 0 restarting, newest start 2026-11-14T09:18:59.079611972Z; init confinement lxc-9201_</var/lib/lxc>//&:lxc-9201_<-var-lib-lxc>:unconfined (enforce)
|
||||
guest 9202: status: running; containers 6 running, 0 unhealthy, 0 restarting, newest start 2026-11-14T09:56:41.303141464Z; init confinement lxc-9202_</var/lib/lxc>//&:lxc-9202_<-var-lib-lxc>:unconfined (enforce)
|
||||
failed units on the host: 0
|
||||
chrony: active System time : 0.000691365 seconds slow of NTP time
|
||||
agent: active
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
#### demo-hp 2026-11-14T11:04:38Z — positive control for the DENIED watch
|
||||
why: the kernel rate-limits its audit lines (kauditd_printk_skb: 111 callbacks suppressed at 11:58:44 and 109 at 11:59:22 local) — the reload floods ~119 STATUS lines, and the DENIED line of the planted stale profile at 11:58:07 is NOT in the log. So a count of 0 proves nothing without this control.
|
||||
control command rc=1 (cat: /etc/shadow: Permission denied)
|
||||
DENIED lines the kernel log shows for the control: 1
|
||||
2026-11-14T12:04:38+01:00 demo-hp kernel: audit: type=1400 audit(1794654278.061:617): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/chronyd" name="/etc/shadow" pid=1879323 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
|
||||
@@ -0,0 +1,9 @@
|
||||
#### guest 9201 2026-11-14T11:00:04Z
|
||||
apparmor userspace: apparmor 4.1.0-1 ii
|
||||
ls: cannot access '/var/cache/apparmor/*/': No such file or directory
|
||||
105
|
||||
inactive
|
||||
cat: /sys/kernel/security/apparmor/profiles: No such file or directory
|
||||
container confinement: unconfined
|
||||
ls: cannot access '/sys/kernel/security/apparmor/policy/profiles': No such file or directory
|
||||
7.0.14-28-pve
|
||||
@@ -0,0 +1,33 @@
|
||||
#### demo-hp 2026-11-14T11:09:55Z — put back as found
|
||||
ls: cannot access '/root/spike-profile*': No such file or directory
|
||||
ls: cannot access '/root/spike-loaded*': No such file or directory
|
||||
ls: cannot access '/root/spike-refresh-at': No such file or directory
|
||||
cache, running kernel folder: chrony 83417e83b538 (83417e83 = the fresh compile); old folder untouched: dbf67f16ac25
|
||||
loaded: 83417e83b538
|
||||
#### demo-hp 2026-11-14T11:09:55Z kernel 7.0.14-28-pve
|
||||
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
|
||||
loaded profiles: 119
|
||||
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 6c024cf01fc0 88d69a2bb62c
|
||||
LOADED-DIFFERS: file=nvidia_modprobe profile=nvidia_modprobe loaded=f4ae61617079 fresh=34f355a47446
|
||||
LOADED-DIFFERS: file=plasmashell profile=plasmashell loaded=17cfa46264cd fresh=2f95a57eab51
|
||||
LOADED-DIFFERS: file=transmission profile=transmission-cli loaded=04b531d8a589 fresh=d9230550cd5f
|
||||
LOADED-DIFFERS: file=transmission profile=transmission-daemon loaded=f46a1d2db622 fresh=d9230550cd5f
|
||||
LOADED-DIFFERS: file=transmission profile=transmission-gtk loaded=ab371958b589 fresh=d9230550cd5f
|
||||
LOADED-DIFFERS: file=transmission profile=transmission-qt loaded=0077a589d81e fresh=d9230550cd5f
|
||||
LOADED-DIFFERS: file=usr.bin.man profile=/usr/bin/man loaded=4ee549c2ed87 fresh=3fe9acfaeb92
|
||||
LOADED-DIFFERS: file=usr.bin.man profile=man_filter loaded=6dc1077b7fa5 fresh=3fe9acfaeb92
|
||||
LOADED-DIFFERS: file=usr.bin.man profile=man_groff loaded=0f6549c22ad0 fresh=3fe9acfaeb92
|
||||
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/usr/lib/NetworkManager/nm-dhcp-client.action loaded=b5532441db45 fresh=99b03dd8451f
|
||||
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/usr/lib/connman/scripts/dhclient-script loaded=fcadbc62c577 fresh=99b03dd8451f
|
||||
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/usr/{lib/NetworkManager,libexec}/nm-dhcp-helper loaded=6ca7645b1031 fresh=99b03dd8451f
|
||||
LOADED-DIFFERS: file=usr.sbin.dhclient profile=/{,usr/}sbin/dhclient loaded=3b3e18f5a89e fresh=99b03dd8451f
|
||||
files=106 cache-differs=0 loaded-differs=13 profiles-not-loaded=4 not-deterministic=1 seconds=4.562202656
|
||||
|
||||
## the 13 LOADED-DIFFERS above are the multi-profile files after a FRESH load (03b: same policy, another hash). To leave the box exactly as found, one cache load, as at a boot: systemctl reload apparmor
|
||||
reload rc=0
|
||||
#### demo-hp 2026-11-14T11:10:08Z kernel 7.0.14-28-pve
|
||||
cache folder of the running kernel: /var/cache/apparmor/f6da0254.0/ all: /var/cache/apparmor/ae1a4050.0/ /var/cache/apparmor/f6da0254.0/
|
||||
loaded profiles: 119
|
||||
NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): lxc-containers 303716fc0b9a 03ad7998e16e
|
||||
files=106 cache-differs=0 loaded-differs=0 profiles-not-loaded=4 not-deterministic=1 seconds=4.587272637
|
||||
chrony active; guests: 2 running; failed units: 0
|
||||
@@ -0,0 +1,19 @@
|
||||
# 02-make-stale.sh — demo-hp (Tier 0). Make ONE stale profile by the real mechanism: an old compiled copy in the
|
||||
# running kernel's cache folder whose time is newer than its source. No package is changed.
|
||||
set -u
|
||||
C=/var/cache/apparmor; CUR=$C/f6da0254.0; OLD=$C/ae1a4050.0; F=usr.sbin.chronyd
|
||||
LOADED=/sys/kernel/security/apparmor/policy/profiles
|
||||
echo "#### $(hostname) $(date -u +%FT%TZ) kernel $(uname -r)"
|
||||
echo "## before: chrony $(systemctl is-active chrony); loaded $(cat $LOADED/usr.sbin.chronyd*/raw_sha256 | cut -c1-12); cache(cur) $(sha256sum < $CUR/$F | cut -c1-12); cache(old folder) $(sha256sum < $OLD/$F | cut -c1-12); fresh $(apparmor_parser -S -K /etc/apparmor.d/$F | sha256sum | cut -c1-12)"
|
||||
echo "## source mtime $(stat -c %y /etc/apparmor.d/$F | cut -c1-19); old copy mtime $(stat -c %y $OLD/$F | cut -c1-19)"
|
||||
cp -a $CUR/$F /root/spike-profile-$F.good && echo "saved the good cache file to /root/spike-profile-$F.good ($(sha256sum < /root/spike-profile-$F.good | cut -c1-12))"
|
||||
cp -a $OLD/$F $CUR/$F && echo "PLANTED: the old compiled copy is now in the running kernel's folder; its time $(stat -c %y $CUR/$F | cut -c1-19) is newer than the source"
|
||||
echo "## reload the way the system does at a start (systemctl reload apparmor = apparmor_parser --write-cache --replace)"
|
||||
systemctl reload apparmor; echo "reload rc=$?"
|
||||
echo "## after the reload: loaded $(cat $LOADED/usr.sbin.chronyd*/raw_sha256 | cut -c1-12) (dbf67f16 = the profile of chrony 4.6.1; 83417e83 = of 4.8)"
|
||||
echo "## chrony still runs under the replaced profile? $(systemctl is-active chrony); its confinement: $(cat /proc/$(pidof -s chronyd)/attr/current 2>/dev/null)"
|
||||
T=$(date +"%Y-%m-%d %H:%M:%S")
|
||||
systemctl restart chrony; echo "restart rc=$?"; sleep 6
|
||||
echo "## after a restart: chrony $(systemctl is-active chrony); result=$(systemctl show chrony -p Result --value)"
|
||||
journalctl -u chrony --since "$T" --no-pager -o short-iso | tail -6
|
||||
journalctl -k --since "$T" --no-pager -o short-iso | grep -i 'apparmor="DENIED"' | tail -3
|
||||
@@ -0,0 +1,30 @@
|
||||
# 03-refresh.sh — demo-hp. The repair candidate: load every profile from a FRESH compile, cache ignored and rewritten.
|
||||
set -u
|
||||
P=/sys/kernel/security/apparmor/policy/profiles; CUR=/var/cache/apparmor/f6da0254.0
|
||||
snap() { # name -> loaded hash, every profile
|
||||
for d in $P/*/; do echo "$(cat $d/name) $(cat $d/raw_sha256 2>/dev/null) $(cat $d/mode 2>/dev/null)"; done | sort
|
||||
}
|
||||
guests() {
|
||||
for v in 9201 9202; do
|
||||
ip=$(lxc-info -n $v -p -H); echo "guest $v: $(pct status $v) init pid $ip confinement=$(cat /proc/$ip/attr/current)"
|
||||
echo " containers: $(pct exec $v -- docker ps -q | wc -l) running, $(pct exec $v -- docker ps --filter health=unhealthy -q | wc -l) unhealthy, started-at-newest=$(pct exec $v -- sh -c 'docker inspect --format "{{.State.StartedAt}}" $(docker ps -q) | sort | tail -1')"
|
||||
cp=$(pct exec $v -- sh -c 'docker inspect --format "{{.State.Pid}}" felhom-controller'); hp=$(grep -l "NSpid:.*[[:space:]]$cp$" /proc/[0-9]*/status 2>/dev/null | head -1 | cut -d/ -f3)
|
||||
echo " the controller container's process (host pid ${hp:-?}): confinement=$(cat /proc/${hp:-0}/attr/current 2>/dev/null)"
|
||||
done
|
||||
echo "VM 341 (Tester 1): $(qm status 341)"
|
||||
}
|
||||
echo "#### $(hostname) $(date -u +%FT%TZ)"
|
||||
echo "## BEFORE"; guests; snap > /root/spike-loaded-before.txt; echo "loaded profiles: $(wc -l < /root/spike-loaded-before.txt)"
|
||||
echo "chrony: $(systemctl is-active chrony); loaded chrony profile $(cat $P/usr.sbin.chronyd*/raw_sha256 | cut -c1-12); cache $(sha256sum < $CUR/usr.sbin.chronyd | cut -c1-12)"
|
||||
echo "disabled profiles (must stay unloaded): $(ls /etc/apparmor.d/disable 2>/dev/null | tr '\n' ' ')"
|
||||
T=$(date +"%Y-%m-%d %H:%M:%S"); date +%s > /root/spike-refresh-at
|
||||
echo "## THE COMMAND: apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d"
|
||||
s=$(date +%s.%N); apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d; rc=$?; e=$(date +%s.%N)
|
||||
echo "rc=$rc seconds=$(echo "$e - $s" | bc)"
|
||||
echo "## AFTER"; guests; snap > /root/spike-loaded-after.txt; echo "loaded profiles: $(wc -l < /root/spike-loaded-after.txt)"
|
||||
echo "profiles whose loaded hash changed:"; diff <(cut -d' ' -f1,2 /root/spike-loaded-before.txt) <(cut -d' ' -f1,2 /root/spike-loaded-after.txt) | grep '^>' | sed 's/^> / /'
|
||||
echo "profiles added or gone (names):"; diff <(cut -d' ' -f1 /root/spike-loaded-before.txt) <(cut -d' ' -f1 /root/spike-loaded-after.txt) | grep '^[<>]' || echo " none"
|
||||
echo "modes changed:"; diff <(cut -d' ' -f1,3 /root/spike-loaded-before.txt) <(cut -d' ' -f1,3 /root/spike-loaded-after.txt) | grep '^[<>]' || echo " none"
|
||||
echo "chrony profile: loaded $(cat $P/usr.sbin.chronyd*/raw_sha256 | cut -c1-12); cache $(sha256sum < $CUR/usr.sbin.chronyd | cut -c1-12) mtime $(stat -c %y $CUR/usr.sbin.chronyd | cut -c1-19); fresh $(apparmor_parser -S -K /etc/apparmor.d/usr.sbin.chronyd | sha256sum | cut -c1-12)"
|
||||
echo "chrony unit, NOT restarted by the command: $(systemctl is-active chrony)"
|
||||
echo "kernel lines since the command:"; journalctl -k --since "$T" --no-pager -o short-iso | grep -i apparmor | tail -5; echo "(end)"
|
||||
@@ -0,0 +1,14 @@
|
||||
# 03b — why did 13 other profiles change their loaded hash in 03? Compare: loaded after a CACHE load (the system's
|
||||
# reload) against loaded after a FRESH load, and the cache files before/after a rewrite. Nothing is stale now.
|
||||
P=/sys/kernel/security/apparmor/policy/profiles; CUR=/var/cache/apparmor/f6da0254.0
|
||||
snap() { for d in $P/*/; do echo "$(cat $d/name) $(cat $d/raw_sha256 2>/dev/null)"; done | sort; }
|
||||
csnap() { sha256sum $CUR/* | sed "s#$CUR/##" | awk '{print $2, $1}' | sort; }
|
||||
echo "#### $(hostname) $(date -u +%FT%TZ)"
|
||||
snap > /tmp/s-fresh1; csnap > /tmp/c1
|
||||
systemctl reload apparmor; snap > /tmp/s-cache
|
||||
apparmor_parser --replace --skip-read-cache --write-cache -- /etc/apparmor.d; snap > /tmp/s-fresh2; csnap > /tmp/c2
|
||||
echo "loaded: fresh-load vs cache-load differ for: $(diff /tmp/s-fresh1 /tmp/s-cache | grep -c '^>') profile(s): $(diff /tmp/s-fresh1 /tmp/s-cache | grep '^>' | cut -d' ' -f2 | tr '\n' ' ')"
|
||||
echo "loaded: fresh-load vs second fresh-load differ for: $(diff /tmp/s-fresh1 /tmp/s-fresh2 | grep -c '^>') profile(s): $(diff /tmp/s-fresh1 /tmp/s-fresh2 | grep '^>' | cut -d' ' -f2 | tr '\n' ' ')"
|
||||
echo "CACHE FILES: before vs after a rewrite differ for: $(diff /tmp/c1 /tmp/c2 | grep -c '^>') file(s): $(diff /tmp/c1 /tmp/c2 | grep '^>' | cut -d' ' -f2 | tr '\n' ' ')"
|
||||
echo "files that define more than one profile: $(for f in /etc/apparmor.d/*; do [ -f $f ] && [ $(apparmor_parser -N $f 2>/dev/null | wc -l) -gt 1 ] && basename $f; done | tr '\n' ' ')"
|
||||
rm -f /tmp/s-fresh1 /tmp/s-cache /tmp/s-fresh2 /tmp/c1 /tmp/c2
|
||||
@@ -0,0 +1,12 @@
|
||||
# 04 — the narrow restart rule, by hand: a FAILED service unit whose ExecStart program is the path-name of a profile
|
||||
# whose cached copy the refresh had to rewrite. (On this box: chrony, made stale in 02.)
|
||||
echo "#### $(hostname) $(date -u +%FT%TZ)"
|
||||
echo "failed service units now: $(systemctl list-units --type=service --state=failed --no-legend --plain | awk '{print $1}' | tr '\n' ' ')"
|
||||
for u in $(systemctl list-units --type=service --state=failed --no-legend --plain | awk '{print $1}'); do
|
||||
echo " $u ExecStart path: $(systemctl show -p ExecStart --value $u | grep -o 'path=[^ ;]*' | head -1)"
|
||||
done
|
||||
echo "profile names of the rewritten cache file usr.sbin.chronyd: $(apparmor_parser -N /etc/apparmor.d/usr.sbin.chronyd | tr '\n' ' ')"
|
||||
echo "## match: chrony.service (failed, ExecStart /usr/sbin/chronyd = the profile's name) -> systemctl restart chrony.service"
|
||||
systemctl restart chrony.service; echo "rc=$?"; sleep 8
|
||||
echo "chrony: $(systemctl is-active chrony); confinement $(cat /proc/$(pidof -s chronyd)/attr/current)"; chronyc -n tracking | grep -E "System time|Leap|Reference"
|
||||
echo "failed service units after: $(systemctl list-units --type=service --state=failed --no-legend --plain | awk '{print $1}' | tr '\n' ' ')(end)"
|
||||
@@ -0,0 +1,29 @@
|
||||
# detect.sh — READ ONLY. For every profile file in /etc/apparmor.d: does the profile the KERNEL has loaded equal a
|
||||
# fresh compile of the file on disk? (the n100-clock-2026-11-11 method, per file AND per loaded profile)
|
||||
# fresh = sha256 of `apparmor_parser -S -K <file>` (compile to stdout: no load, cache not read, not written)
|
||||
# loaded = policy/profiles/*/raw_sha256 of each profile name the file defines (`apparmor_parser -N`)
|
||||
# cache = sha256 of the cached binary in the running kernel's cache folder
|
||||
main() {
|
||||
echo "#### $(hostname) $(date -u +%FT%TZ) kernel $(uname -r)"
|
||||
P=/sys/kernel/security/apparmor/policy/profiles
|
||||
cur=""; for d in /var/cache/apparmor/*/; do [ "$(sha256sum < $d/.features | cut -d' ' -f1)" = "$(aa-features-abi -x 2>/dev/null | sha256sum | cut -d' ' -f1)" ] && cur=$d; done
|
||||
echo "cache folder of the running kernel: ${cur:-none} all: $(ls -d /var/cache/apparmor/*/ | tr '\n' ' ')"
|
||||
declare -A L; for d in $P/*/; do n=$(cat $d/name 2>/dev/null); [ -n "$n" ] && L["$n"]=$(cat $d/raw_sha256 2>/dev/null); done
|
||||
echo "loaded profiles: ${#L[@]}"
|
||||
n=0; stale=0; cdiff=0; notloaded=0; nd=0; t0=$(date +%s.%N)
|
||||
for f in /etc/apparmor.d/*; do [ -f "$f" ] || continue; b=$(basename "$f"); [ -e /etc/apparmor.d/disable/$b ] && continue
|
||||
fresh=$(apparmor_parser -S -K "$f" 2>/dev/null | sha256sum | cut -d' ' -f1); n=$((n+1))
|
||||
again=$(apparmor_parser -S -K "$f" 2>/dev/null | sha256sum | cut -d' ' -f1)
|
||||
# CONTROL ON THE METHOD (found 2026-11-14): a file whose two fresh compiles differ cannot be judged by a hash at all
|
||||
[ "$fresh" = "$again" ] || { nd=$((nd+1)); echo "NOT-DETERMINISTIC (two fresh compiles differ — not a staleness signal): $b ${fresh:0:12} ${again:0:12}"; continue; }
|
||||
c="-"; [ -n "$cur" ] && [ -f "$cur$b" ] && c=$(sha256sum < "$cur$b" | cut -d' ' -f1)
|
||||
[ "$c" != "-" ] && [ "$c" != "$fresh" ] && { cdiff=$((cdiff+1)); echo "CACHE-DIFFERS: $b cache=${c:0:12} fresh=${fresh:0:12} cache-mtime=$(stat -c %y "$cur$b" | cut -c1-19) source-mtime=$(stat -c %y "$f" | cut -c1-19)"; }
|
||||
for name in $(apparmor_parser -N "$f" 2>/dev/null); do
|
||||
l=${L["$name"]:-}
|
||||
if [ -z "$l" ]; then notloaded=$((notloaded+1)); continue; fi
|
||||
[ "$l" = "$fresh" ] || { stale=$((stale+1)); echo "LOADED-DIFFERS: file=$b profile=$name loaded=${l:0:12} fresh=${fresh:0:12}"; }
|
||||
done
|
||||
done
|
||||
echo "files=$n cache-differs=$cdiff loaded-differs=$stale profiles-not-loaded=$notloaded not-deterministic=$nd seconds=$(echo "$(date +%s.%N) - $t0" | bc)"
|
||||
}
|
||||
main </dev/null
|
||||
@@ -0,0 +1,43 @@
|
||||
### demo-hp 2026-11-14T10:55:37Z — dry run of the working-copy wrapper (sha256 8d5052e4548341e0…)
|
||||
os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[]
|
||||
{
|
||||
"refused": null,
|
||||
"failed": null,
|
||||
"kernel_cleanup": {
|
||||
"default": "7.0.14-28-pve",
|
||||
"dry_run": true,
|
||||
"installed": {
|
||||
"7.0.14-26-pve": [
|
||||
"proxmox-kernel-7.0.14-26-pve-signed"
|
||||
],
|
||||
"7.0.14-28-pve": [
|
||||
"proxmox-kernel-7.0.14-28-pve-signed"
|
||||
]
|
||||
},
|
||||
"kept": {
|
||||
"7.0.14-26-pve": [
|
||||
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
|
||||
"the kernel step's `default_before`"
|
||||
],
|
||||
"7.0.14-28-pve": [
|
||||
"running",
|
||||
"the kernel step's `to`",
|
||||
"the default in grub.cfg",
|
||||
"the pinned default file",
|
||||
"newest installed"
|
||||
]
|
||||
},
|
||||
"leftovers": [],
|
||||
"newest": "7.0.14-28-pve",
|
||||
"not_run": [],
|
||||
"outcome": "dry_run",
|
||||
"phase": "good",
|
||||
"pinned_default": "7.0.14-28-pve",
|
||||
"refusals": [],
|
||||
"removal": [],
|
||||
"removed": [],
|
||||
"running": "7.0.14-28-pve",
|
||||
"would": "nothing"
|
||||
},
|
||||
"pass_seconds": 0.2
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
### root@192.168.0.154 2026-11-14T10:55:38Z — dry run of the working-copy wrapper (sha256 8d5052e4548341e0…)
|
||||
os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=3 keep=2 removal=1 not_run=['no-ring0-mark'] refusals=[]
|
||||
{
|
||||
"refused": null,
|
||||
"failed": null,
|
||||
"kernel_cleanup": {
|
||||
"default": "7.0.14-22-pve",
|
||||
"dry_run": true,
|
||||
"installed": {
|
||||
"7.0.14-20-pve": [
|
||||
"proxmox-kernel-7.0.14-20-pve-signed"
|
||||
],
|
||||
"7.0.14-22-pve": [
|
||||
"proxmox-kernel-7.0.14-22-pve-signed"
|
||||
],
|
||||
"7.0.2-6-pve": [
|
||||
"proxmox-kernel-7.0.2-6-pve-signed"
|
||||
]
|
||||
},
|
||||
"kept": {
|
||||
"7.0.14-22-pve": [
|
||||
"running",
|
||||
"the kernel step's `to`",
|
||||
"the default in grub.cfg",
|
||||
"the pinned default file",
|
||||
"newest installed"
|
||||
],
|
||||
"7.0.2-6-pve": [
|
||||
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
|
||||
"the kernel step's `default_before`"
|
||||
]
|
||||
},
|
||||
"leftovers": [],
|
||||
"newest": "7.0.14-22-pve",
|
||||
"not_run": [
|
||||
{
|
||||
"code": "no-ring0-mark",
|
||||
"reason": "this box has no root-owned ring-0 mark \u2014 the clean-up runs on ring 0 only in this release"
|
||||
}
|
||||
],
|
||||
"outcome": "dry_run",
|
||||
"phase": "good",
|
||||
"pinned_default": "7.0.14-22-pve",
|
||||
"refusals": [],
|
||||
"removal": [
|
||||
"proxmox-kernel-7.0.14-20-pve-signed"
|
||||
],
|
||||
"removed": [],
|
||||
"running": "7.0.14-22-pve",
|
||||
"simulation": {
|
||||
"installs": [],
|
||||
"rc": 0,
|
||||
"removes": [
|
||||
"proxmox-kernel-7.0.14-20-pve-signed"
|
||||
]
|
||||
},
|
||||
"would": "not_run"
|
||||
},
|
||||
"pass_seconds": 1.4
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
== agent red-proofs (internal/osupdate), 2026-11-14T10:54:58Z
|
||||
-- MUTATION 1: kernelCleanupSkip always answers "" (every 'may not run' rule removed)
|
||||
--- FAIL: TestKernel_Ring0ToldNightStagesThenReboots (0.00s)
|
||||
kernel_test.go:83: the reboot must end the night, last call = map[lane:slow layer:kernel mode:kernel-cleanup packages:[] release_id:kernel-cleanup ring:0 run_id:20261004T040000Z trigger:night vmid:9201]
|
||||
--- FAIL: TestKernelCleanup_WhenItMayNotRun (0.00s)
|
||||
kernel_test.go:489: a kernel step is due tonight (ring 0): the clean-up ran (1 call(s)); calls = guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply,kernel:kernel-status,kernel:apply,kernel:kernel-reb
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.022s
|
||||
FAIL
|
||||
-- MUTATION 2: 'nothing' sends a report (return l.finish in the nothing case)
|
||||
--- FAIL: TestKernelCleanup_NothingAndNotRunSendNoReport (0.00s)
|
||||
kernel_test.go:423: nothing: a report was made: {RunID:20261004T040000Z Layer:kernel Trigger:night Mode:kernel-cleanup Ring:1 ReleaseID:kernel-cleanup Outcome:nothing Healthy:false HealthReason: VMID:9201 Upgraded:[]
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.021s
|
||||
FAIL
|
||||
-- MUTATION 3: the report carries the wrapper's kernel view
|
||||
--- FAIL: TestKernelCleanup_NightOrderOnceAndNoNames (0.00s)
|
||||
kernel_test.go:401: the clean-up's report names a kernel (the hub reads that as a kernel step): {RunID:20261004T040000Z Layer:kernel Trigger:night Mode:kernel-cleanup Ring:0 ReleaseID:kernel-cleanup Outcome:cleaned Healthy:true HealthReason:removed 1 old k
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.010s
|
||||
FAIL
|
||||
-- the real file:
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 4.973s
|
||||
@@ -0,0 +1,74 @@
|
||||
== MUTATION: K1 removed (the rule's own check of the removal set)
|
||||
elif by_name[n] in protected or by_name[n] == newest: -> elif False:
|
||||
tests: 7 mutant: Ran 7 tests in 0.119s | FAILED (failures=7) real: Ran 7 tests in 0.116s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 0 != 2 : running: {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running':
|
||||
AssertionError: 0 != 2 : from: {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7
|
||||
== MUTATION: the simulation check removed (S1/S2)
|
||||
if removal and not [x for x in refuse if x["code"] in ("K1", "N1")]: -> if False:
|
||||
tests: 3 mutant: Ran 3 tests in 0.055s | FAILED (failures=3) real: Ran 3 tests in 0.047s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
|
||||
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
|
||||
== MUTATION: the phase check removed (P1)
|
||||
elif phase != "good": -> elif False:
|
||||
tests: 1 mutant: Ran 1 test in 0.019s | FAILED (failures=1) real: Ran 1 test in 0.072s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'staged', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.1
|
||||
== MUTATION: the ring-0 check removed
|
||||
if not ring0 and not not_run: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.021s | FAILED (failures=1) real: Ran 1 test in 0.047s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 'removed' != 'not_run'
|
||||
== MUTATION: the flag check removed (P2)
|
||||
if flag: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.021s | FAILED (failures=1) real: Ran 1 test in 0.022s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '7.0.14-26-pve', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'runni
|
||||
== MUTATION: the default check removed (P3)
|
||||
if pinned is None or pinned != dflt or pinned not in images: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.024s | FAILED (failures=1) real: Ran 1 test in 0.044s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 3 != 2 : {'failed': {'rc': 3, 'reason': 'the default in grub.cfg moved from 7.0.14-28-pve to 7.0.14-26-pve', 'step': 'verify'}, 'kernel': {'default': '7.0.14-26-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': '
|
||||
== MUTATION: the not-on-newest rule removed
|
||||
if running and newest and running in images and running != newest: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.032s | FAILED (failures=1) real: Ran 1 test in 0.028s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 'removed' != 'not_run'
|
||||
== MUTATION: the fell-back rule removed
|
||||
elif phase in KERNEL_ENDED_NOT_GOOD: -> elif False:
|
||||
tests: 1 mutant: Ran 1 test in 0.035s | FAILED (failures=1) real: Ran 1 test in 0.101s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 'step-not-good' not found in ['not-on-newest'] : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'fell_back', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at'
|
||||
== MUTATION: the plan-names check removed (C1)
|
||||
if mode == "kernel-cleanup": -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.020s | FAILED (failures=1) real: Ran 1 test in 0.001s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 0 != 2 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
|
||||
== MUTATION: the no-state-file rule removed
|
||||
if raw_state is None or not st: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.036s | FAILED (failures=1) real: Ran 1 test in 0.031s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 2 != 0 : {'kernel_cleanup': {'default': '7.0.14-28-pve', 'dry_run': False, 'installed': {'7.0.14-20-pve': ['proxmox-kernel-7.0.14-20-pve-signed'], '7.0.14-22-pve': ['proxmox-kernel-7.0.14-22-pve-signed'], '7.0.14-2
|
||||
== MUTATION: the 20-hour window removed
|
||||
if times and self.r.now() - max(times) < KERNEL_MIN_GAP: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.022s | FAILED (failures=1) real: Ran 1 test in 0.016s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 'removed' != 'not_run'
|
||||
== MUTATION: the dry run purges (dry-run verdict removed)
|
||||
if dry: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.022s | FAILED (errors=1) real: Ran 1 test in 0.049s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
KeyError: 'would'
|
||||
== MUTATION: repair-first removed
|
||||
if not dry and not not_run: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.032s | FAILED (errors=1) real: Ran 1 test in 0.031s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
KeyError: 'repair'
|
||||
== MUTATION: the after-purge proof removed
|
||||
if problems: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.024s | FAILED (failures=1) real: Ran 1 test in 0.022s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 0 != 3 : {'kernel': {'default': '7.0.14-28-pve', 'flag': '', 'from': '7.0.14-26-pve', 'phase': 'good', 'reason': None, 'rebooted_at': '2026-10-01T02:51:00Z', 'result_at': '2026-10-01T03:10:00Z', 'running': '7.0.14-
|
||||
|
||||
14 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,25 @@
|
||||
== agent red-proofs, profile refresh, 2026-11-14T11:04:02Z
|
||||
-- MUTATION 1: the appliance check removed
|
||||
--- FAIL: TestProfileRefresh_ByoHostIsNeverAsked (0.00s)
|
||||
profilerefresh_test.go:54: a BYO host was asked: "refreshed" [map[layer:host mode:profile-refresh release_id:profile-refresh]]
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.010s
|
||||
FAIL
|
||||
-- MUTATION 2: the gate's answer ignored (always go on)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate [build failed]
|
||||
FAIL
|
||||
-- MUTATION 3: the plan names a unit
|
||||
--- FAIL: TestProfileRefresh_OneCallNoNamesNoHubReport (0.00s)
|
||||
profilerefresh_test.go:42: the plan carries more than release_id, layer and mode: map[layer:host mode:profile-refresh release_id:profile-refresh units:[chrony.service]]
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.011s
|
||||
FAIL
|
||||
-- the real file:
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.009s
|
||||
-- MUTATION 2, done again (the first try did not compile, so it proved nothing): the gate's answer ignored
|
||||
--- FAIL: TestProfileRefresh_WaitsForTheHeavyOpGate (0.00s)
|
||||
profilerefresh_test.go:75: gate tries 1, released 0, calls 1
|
||||
FAIL
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.012s
|
||||
FAIL
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate (cached)
|
||||
@@ -0,0 +1,53 @@
|
||||
== MUTATION: the restart rule widened: any failed unit restarts (the program check removed)
|
||||
if prog not in programs: -> if False:
|
||||
tests: 2 mutant: Ran 2 tests in 0.010s | FAILED (errors=2) real: Ran 2 tests in 0.006s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
KeyError: '/usr/sbin/postfix'
|
||||
KeyError: '/usr/sbin/chronyd'
|
||||
== MUTATION: the stale-copy condition removed: every profile file counts, not only a rewritten one
|
||||
for key in res["rewritten"]: -> for key in after:
|
||||
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: Lists differ: ['chrony.service'] != []
|
||||
== MUTATION: a profile name that is not a path counts as a program
|
||||
if PROFILE_PATH_NAME_RE.match(n): -> if True:
|
||||
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: Tuples differ: (0, ['nvidia.service']) != (0, [])
|
||||
== MUTATION: once-per-kernel removed
|
||||
needed = rec.get("kernel") != kernel -> needed = True
|
||||
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: Tuples differ: (0, 'refreshed', 2) != (0, 'not_needed', 1)
|
||||
== MUTATION: the cache is not skipped (the system's own reload)
|
||||
PROFILE_REFRESH_CMD = ["apparmor_parser", "--replace", "--skip-read-cache", "--write-cache", "--", PROFILE_DIR -> PROFILE_REFRESH_CMD = ["apparmor_parser", "--replace", "--write-cache", "--", PROFILE_DIR]
|
||||
tests: 1 mutant: Ran 1 test in 0.006s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: ('host', ['apparmor_parser', '--replace', '--skip-read-cache', '--write-cache', '--', '/etc/apparmor.d']) not found in [('host', ['uname', '-r']), ('host', ['apparmor_parser', '--replace', '--write-cache', '--', '/
|
||||
== MUTATION: a failed parser still writes the record
|
||||
r.log(f"os-apply: FAILED rc={rc} step=profile-refresh (no record written: the next start tries again)") -> r.log(f"os-apply: FAILED rc={rc} step=profile-refresh (no record written: the next start tries again)")
|
||||
tests: 1 mutant: Ran 1 test in 0.006s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: '/var/lib/felhom-os-apply/profile-refresh.json' unexpectedly found in {'/var/lib/felhom-os-apply/profile-refresh.json': '{"kernel": "7.0.14-28-pve"}'}
|
||||
== MUTATION: the plan may name things (F1 removed)
|
||||
for k in ("packages", "allow_new", "signed", "select", "undo", "snapshot", "profiles", "units", "force"): -> for k in ("packages", "allow_new", "signed", "select", "undo", "snapshot", "profiles", "units", "force"):
|
||||
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.001s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: 0 != 2 : {'layer': 'host', 'mode': 'profile-refresh', 'pass_seconds': 0.0, 'profile_refresh': {'dry_run': False, 'kernel': '7.0.14-28-pve', 'last_refresh_at': None, 'last_refresh_kernel': None, 'not_restarted': [],
|
||||
== MUTATION: the appliance rule removed
|
||||
self.check_appliance() -> return self.profile_refresh(plan)
|
||||
tests: 1 mutant: Ran 1 test in 0.005s | FAILED (errors=1) real: Ran 1 test in 0.000s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
TypeError: 'NoneType' object is not subscriptable
|
||||
== MUTATION: the dry run refreshes
|
||||
if dry: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.006s | FAILED (errors=1) real: Ran 1 test in 0.000s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
KeyError: 'would'
|
||||
== MUTATION: the restart cap removed
|
||||
if len(res["restarted"]) >= PROFILE_RESTART_MAX: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.007s | FAILED (failures=1) real: Ran 1 test in 0.005s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: Tuples differ: (0, 5, 0) != (0, 3, 2)
|
||||
|
||||
10 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,4 @@
|
||||
== the published binary, downloaded with no login 2026-11-14T11:13:30Z
|
||||
6efbb97809135483bc446d71d1321f9e2f7fc3268fe9b20e7fe742a2eb13db92
|
||||
agent-0.174.0: go1.27.4
|
||||
felhom-agent 0.174.0
|
||||
@@ -0,0 +1,2 @@
|
||||
felhom: agent felhom-agent 0.173.0 wrapper sha a5f67f015fbc bundle 0.172.1 3e7acaee4500 host_id tester-1-d70be4
|
||||
demo-hp: agent felhom-agent 0.173.0 wrapper sha a5f67f015fbc bundle 0.172.1 3e7acaee4500 host_id demo-hp-bb76ea
|
||||
@@ -0,0 +1,8 @@
|
||||
== tester-1-d70be4 2026-11-14T11:13:39Z agent_update 0.174.0
|
||||
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=3a80f37e50d321ee00db919bb6c659fc expires=2026-11-14T11:58:39Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
rc=0
|
||||
== demo-hp-bb76ea 2026-11-14T11:13:39Z agent_update 0.174.0
|
||||
signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=c9fb994bd9952af35d137219442067b4 expires=2026-11-14T11:58:39Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
rc=0
|
||||
@@ -0,0 +1,13 @@
|
||||
### felhom 2026-11-14T11:18:17Z
|
||||
2026-11-14T12:14:53+01:00 felhom felhom-agent[51939]: time=2026-11-14T12:14:53.096+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=df59dd26428b9120 op=agent_update
|
||||
2026-11-14T12:14:55+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:14:55.235+01:00 level=INFO msg="felhom-agent daemon starting" version=0.174.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
|
||||
2026-11-14T12:14:55+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:14:55.356+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
|
||||
2026-11-14T12:15:56+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:15:56.990+01:00 level=WARN msg="selfupdate: update committed" version=0.174.0 wrapper=""
|
||||
active
|
||||
0
|
||||
### demo-hp 2026-11-14T11:18:21Z
|
||||
2026-11-14T12:18:00+01:00 demo-hp felhom-agent[535281]: time=2026-11-14T12:18:00.785+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=0ef54dd1311c3a04 op=agent_update
|
||||
2026-11-14T12:18:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:18:02.833+01:00 level=INFO msg="felhom-agent daemon starting" version=0.174.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
|
||||
2026-11-14T12:18:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:18:02.956+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
|
||||
active
|
||||
0
|
||||
@@ -0,0 +1,8 @@
|
||||
== tester-1-d70be4 2026-11-14T11:19:06Z agent_config_update 0.174.0
|
||||
signed: op=agent_config_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=f632b7fb9a3ae1667d0050ce176a86d5 expires=2026-11-14T12:04:06Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
rc=0
|
||||
== demo-hp-bb76ea 2026-11-14T11:19:06Z agent_config_update 0.174.0
|
||||
signed: op=agent_config_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=61a7b130297809cab3b5e26dfc1f7288 expires=2026-11-14T12:04:06Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
rc=0
|
||||
@@ -0,0 +1,24 @@
|
||||
### felhom 2026-11-14T11:30:25Z
|
||||
2026-11-14T12:14:55+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:14:55.356+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
|
||||
2026-11-14T12:30:09+01:00 felhom felhom-os-apply[3433197]: os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False
|
||||
2026-11-14T12:30:09+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:09.431+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False"
|
||||
2026-11-14T12:30:09+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:09.431+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.174.0 bundle="{\"agent_version\": \"0.174.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261114T113008Z-before-0.174.0\", \"removed\": [], \"same\": 30,
|
||||
2026-11-14T12:30:10+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:10.471+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded=""
|
||||
2026-11-14T12:30:10+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:10.471+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=de2c899634da22aa op=agent_config_update
|
||||
2026-11-14T12:30:11+01:00 felhom felhom-os-apply[3433509]: os-apply: PROFILE REFRESH DONE kernel=7.0.14-22-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]
|
||||
2026-11-14T12:30:11+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:11.181+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PROFILE REFRESH DONE kernel=7.0.14-22-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]"
|
||||
2026-11-14T12:30:11+01:00 felhom felhom-agent[3411307]: time=2026-11-14T12:30:11.181+01:00 level=INFO msg="osupdate: profile refresh — security profiles reloaded from a fresh compile" mode=profile-refresh why="config bundle installed" kernel=7.0.14-22-pve rewritten=1 wrapper_seconds=0.5 took=700ms
|
||||
wrapper sha 9d03203766040a97 record: { "at": "2026-11-14T11:30:11Z", "kernel": "7.0.14-22-pve", "restarted": [], "rewritten": [ "f6da0254.0/lxc-containers" ]}
|
||||
chrony active; failed units 0; guests running 1
|
||||
### demo-hp 2026-11-14T11:33:26Z
|
||||
2026-11-14T12:18:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:18:02.956+01:00 level=INFO msg="osupdate: profile refresh refused (an older wrapper that does not know the mode, or a BYO host)" mode=profile-refresh why="agent start" refused="{\"code\": \"R11\", \"reason\": \"unknown mode 'profile-refresh'\"}"
|
||||
2026-11-14T12:33:16+01:00 demo-hp felhom-os-apply[1980789]: os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False
|
||||
2026-11-14T12:33:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:16.771+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: BUNDLE DONE agent=0.174.0 written=1 same=30 self-check=ok signers-created=False"
|
||||
2026-11-14T12:33:16+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:16.771+01:00 level=WARN msg="osupdate: config bundle INSTALLED" op=agent_config_update agent_version=0.174.0 bundle="{\"agent_version\": \"0.174.0\", \"authority\": \"signed\", \"kept\": [\"/etc/felhom/crash-guard.conf\"], \"prev_dir\": \"/var/lib/felhom-os-apply/bundle-prev/20261114T113315Z-before-0.174.0\", \"removed\": [], \"same\": 30,
|
||||
2026-11-14T12:33:17+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:17.923+01:00 level=WARN msg="osupdate: capability probe after the config bundle" ok=77 total=77 degraded=""
|
||||
2026-11-14T12:33:17+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:17.923+01:00 level=WARN msg="signedjobs: signed op COMPLETED" job=524eef508b29fa0c op=agent_config_update
|
||||
2026-11-14T12:33:18+01:00 demo-hp felhom-os-apply[1981111]: os-apply: PROFILE REFRESH DONE kernel=7.0.14-28-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]
|
||||
2026-11-14T12:33:18+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:18.531+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PROFILE REFRESH DONE kernel=7.0.14-28-pve (last never) seconds=0.5 rewritten=1 (f6da0254.0/lxc-containers) restarted=[]"
|
||||
2026-11-14T12:33:18+01:00 demo-hp felhom-agent[1929140]: time=2026-11-14T12:33:18.532+01:00 level=INFO msg="osupdate: profile refresh — security profiles reloaded from a fresh compile" mode=profile-refresh why="config bundle installed" kernel=7.0.14-28-pve rewritten=1 wrapper_seconds=0.5 took=600ms
|
||||
wrapper sha 9d03203766040a97 record: { "at": "2026-11-14T11:33:18Z", "kernel": "7.0.14-28-pve", "restarted": [], "rewritten": [ "f6da0254.0/lxc-containers" ]}
|
||||
chrony active; failed units 0; guests running 2
|
||||
@@ -0,0 +1,57 @@
|
||||
### felhom (Tester 1) 2026-11-14T11:30:43Z — the dry run through the released agent and its own sudo line
|
||||
agent felhom-agent 0.174.0; wrapper 9d03203766040a97; mark: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false}
|
||||
time=2026-11-14T12:30:45.231+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=3 keep=2 removal=1 not_run=['no-ring0-mark'] refusals=[]"
|
||||
--- kernel clean-up — DRY RUN (nothing was changed) ---
|
||||
{
|
||||
"default": "7.0.14-22-pve",
|
||||
"dry_run": true,
|
||||
"installed": {
|
||||
"7.0.14-20-pve": [
|
||||
"proxmox-kernel-7.0.14-20-pve-signed"
|
||||
],
|
||||
"7.0.14-22-pve": [
|
||||
"proxmox-kernel-7.0.14-22-pve-signed"
|
||||
],
|
||||
"7.0.2-6-pve": [
|
||||
"proxmox-kernel-7.0.2-6-pve-signed"
|
||||
]
|
||||
},
|
||||
"kept": {
|
||||
"7.0.14-22-pve": [
|
||||
"running",
|
||||
"the kernel step's `to`",
|
||||
"the default in grub.cfg",
|
||||
"the pinned default file",
|
||||
"newest installed"
|
||||
],
|
||||
"7.0.2-6-pve": [
|
||||
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
|
||||
"the kernel step's `default_before`"
|
||||
]
|
||||
},
|
||||
"leftovers": [],
|
||||
"newest": "7.0.14-22-pve",
|
||||
"not_run": [
|
||||
{
|
||||
"code": "no-ring0-mark",
|
||||
"reason": "this box has no root-owned ring-0 mark — the clean-up runs on ring 0 only in this release"
|
||||
}
|
||||
],
|
||||
"outcome": "dry_run",
|
||||
"phase": "good",
|
||||
"pinned_default": "7.0.14-22-pve",
|
||||
"refusals": [],
|
||||
"removal": [
|
||||
"proxmox-kernel-7.0.14-20-pve-signed"
|
||||
],
|
||||
"removed": [],
|
||||
"running": "7.0.14-22-pve",
|
||||
"simulation": {
|
||||
"installs": [],
|
||||
"rc": 0,
|
||||
"removes": [
|
||||
"proxmox-kernel-7.0.14-20-pve-signed"
|
||||
]
|
||||
},
|
||||
"would": "not_run"
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
OPERATOR'S YES for the temporary ring-0 mark on Tester 1: given in the session, 2026-11-14 (the STOP's first question).
|
||||
#### felhom (Tester 1) 2026-11-14T14:16:19Z agent felhom-agent 0.174.0 wrapper 9d03203766040a97
|
||||
busy? vzdump 0, apt 0, guest status: running, containers 10 running / 0 unhealthy
|
||||
## BEFORE
|
||||
kernels:
|
||||
running 7.0.14-22-pve; grub.cfg default: gnulinux-7.0.14-22-pve-advanced
|
||||
sha256: pinned d96c890b9749 state 8008e8e2192c gen01 f2f56d59c635 gen42 c34907ce6859 flag [felhom_next= ]
|
||||
/ free bytes: 39449448448; /boot kernels: 7.0.14-20-pve 7.0.14-22-pve 7.0.2-6-pve ; /lib/modules: 7.0.14-20-pve 7.0.14-22-pve 7.0.2-6-pve
|
||||
one-shot entries in grub.cfg: felhom-oneshot-7.0.14-20-pve felhom-oneshot-7.0.14-22-pve felhom-oneshot-7.0.2-6-pve
|
||||
## the ring-0 mark, set by hand FOR THIS TEST (Tester 1 is ring 1; the clean-up runs on ring 0 only in this release)
|
||||
mark before: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false} (root:root 644)
|
||||
mark set: { "host_id": "tester-1-d70be4", "ring0_slow_lane": true} (root:root 644)
|
||||
## SCENARIO 1 — the clean-up
|
||||
wrapper exit code: 0
|
||||
os-apply: REPAIR configured=0 journal=0 fixed=0
|
||||
os-apply: KERNEL CLEANUP running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=3 keep=2 removal=1 not_run=[] refusals=[]
|
||||
os-apply: KERNEL CLEANUP DONE removed=1 (proxmox-kernel-7.0.14-20-pve-signed) kept=7.0.14-22-pve, 7.0.2-6-pve gained=1124024320 B seconds=8.5 leftovers=0; default 7.0.14-22-pve (proved from grub.cfg)
|
||||
{
|
||||
"refused": null,
|
||||
"failed": null,
|
||||
"repair": {
|
||||
"clean_after": true,
|
||||
"fixed": 0,
|
||||
"half_configured_before": 0,
|
||||
"journal_before": 0
|
||||
},
|
||||
"pass_seconds": 10.1,
|
||||
"kernel_cleanup": {
|
||||
"outcome": "removed",
|
||||
"not_run": [],
|
||||
"refusals": [],
|
||||
"kept": {
|
||||
"7.0.14-22-pve": [
|
||||
"running",
|
||||
"the kernel step's `to`",
|
||||
"the default in grub.cfg",
|
||||
"the pinned default file",
|
||||
"newest installed"
|
||||
],
|
||||
"7.0.2-6-pve": [
|
||||
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
|
||||
"the kernel step's `default_before`"
|
||||
]
|
||||
},
|
||||
"removal": [
|
||||
"proxmox-kernel-7.0.14-20-pve-signed"
|
||||
],
|
||||
"removed": [
|
||||
"proxmox-kernel-7.0.14-20-pve-signed"
|
||||
],
|
||||
"bytes_gained": 1124024320,
|
||||
"leftovers": [],
|
||||
"seconds": 8.5,
|
||||
"simulation": {
|
||||
"installs": [],
|
||||
"rc": 0,
|
||||
"removes": [
|
||||
"proxmox-kernel-7.0.14-20-pve-signed"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
## AFTER
|
||||
kernels:
|
||||
running 7.0.14-22-pve; grub.cfg default: gnulinux-7.0.14-22-pve-advanced
|
||||
sha256: pinned d96c890b9749 state 8008e8e2192c gen01 f2f56d59c635 gen42 c34907ce6859 flag [felhom_next= ]
|
||||
/ free bytes: 40573468672; /boot kernels: 7.0.14-22-pve 7.0.2-6-pve ; /lib/modules: 7.0.14-22-pve 7.0.2-6-pve
|
||||
one-shot entries in grub.cfg: felhom-oneshot-7.0.14-22-pve felhom-oneshot-7.0.2-6-pve
|
||||
## SCENARIO 2 — the same call again
|
||||
wrapper exit code: 0
|
||||
os-apply: REPAIR configured=0 journal=0 fixed=0
|
||||
os-apply: KERNEL CLEANUP running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[]
|
||||
os-apply: KERNEL CLEANUP nothing to remove
|
||||
{
|
||||
"refused": null,
|
||||
"failed": null,
|
||||
"repair": {
|
||||
"clean_after": true,
|
||||
"fixed": 0,
|
||||
"half_configured_before": 0,
|
||||
"journal_before": 0
|
||||
},
|
||||
"pass_seconds": 0.1,
|
||||
"kernel_cleanup": {
|
||||
"outcome": "nothing",
|
||||
"not_run": [],
|
||||
"refusals": [],
|
||||
"kept": {
|
||||
"7.0.14-22-pve": [
|
||||
"running",
|
||||
"the kernel step's `to`",
|
||||
"the default in grub.cfg",
|
||||
"the pinned default file",
|
||||
"newest installed"
|
||||
],
|
||||
"7.0.2-6-pve": [
|
||||
"the kernel step's `from` (the last kernel that booted well; a self-revert boots it)",
|
||||
"the kernel step's `default_before`"
|
||||
]
|
||||
},
|
||||
"removal": [],
|
||||
"removed": [],
|
||||
"bytes_gained": null,
|
||||
"leftovers": [],
|
||||
"seconds": null,
|
||||
"simulation": null
|
||||
}
|
||||
}
|
||||
## the mark put back
|
||||
mark now: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false} (root:root 644); saved copy: ls: cannot access '/root/os-trust.json.before-2a': No such file or directory
|
||||
## and with the mark back, the program answers:
|
||||
wrapper exit code: 0
|
||||
os-apply: KERNEL CLEANUP running=7.0.14-22-pve newest=7.0.14-22-pve phase=good installed=2 keep=2 removal=0 not_run=['no-ring0-mark'] refusals=[]
|
||||
"outcome": "not_run",
|
||||
"code": "no-ring0-mark",
|
||||
guest status: running, containers 10 running / 0 unhealthy; dpkg audit: []
|
||||
@@ -0,0 +1,7 @@
|
||||
#### the N100 (demo-felhom) BEFORE the release — reads only; DooPlex clock 2026-11-14T11:32:03.295Z
|
||||
offset of the N100 to DooPlex: -32.55 s (negative = the N100 is behind); the ssh round trip was 0.35 s
|
||||
agent felhom-agent 0.173.0; kernel 7.0.14-28-pve; chrony: failed (result exit-code); failed units: chrony.service
|
||||
chrony profile: loaded dbf67f16ac25, cache (7.0.14 folder) dbf67f16ac25, fresh compile 83417e83b538
|
||||
busy? vzdump 0, apt 0, task lock files: pve-migrate-9201 pve-storage-local-lvm ; guest: 9201 running demo-felhom ;
|
||||
kernels:
|
||||
/dev/mapper/pve-root 94G 35G 55G 40% /
|
||||
@@ -0,0 +1,19 @@
|
||||
#### demo-hp 2026-11-14T11:33:49Z agent felhom-agent 0.174.0
|
||||
## 0. the box as it is
|
||||
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
|
||||
would: nothing | refusals: [] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
|
||||
## 1. scenario 3 — a step that is not over: phase set to 'judging' in the state file
|
||||
would: refuse | refusals: ["P1 a kernel step is in phase 'judging' — not ended (or not a phase this wrapper knows)"] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
|
||||
put back:
|
||||
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
|
||||
## 2. scenario 4a — the one-shot flag names the kept older kernel
|
||||
would: refuse | refusals: ["P2 the one-shot flag names '7.0.14-26-pve'"] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
|
||||
put back:
|
||||
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
|
||||
## 3. scenario 4b — the pinned default file names another kernel than grub.cfg
|
||||
would: refuse | refusals: ["P3 the pinned default '7.0.14-26-pve' and grub.cfg's default '7.0.14-28-pve' do not name the same installed kerne"] | not_run: [] | removal: [] | kept: ['7.0.14-26-pve', '7.0.14-28-pve']
|
||||
put back:
|
||||
state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
|
||||
## 4. as found?
|
||||
IDENTICAL to step 0: state 607a2ce7ce3b pinned 6b35d9aeddc1 flag [felhom_next= ] grub.cfg 805cc2ba2da6 kernels 2
|
||||
ls: cannot access '/root/2a-*': No such file or directory
|
||||
@@ -0,0 +1,4 @@
|
||||
== demo-felhom-8363b5 2026-11-14T11:34:09Z agent_update 0.174.0
|
||||
signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=20e3bd9eb794fa7137294ed8ca1390c3 expires=2026-11-14T12:19:09Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
rc=0
|
||||
@@ -0,0 +1,4 @@
|
||||
== demo-felhom-8363b5 2026-11-14T11:37:49Z agent_config_update 0.174.0
|
||||
signed: op=agent_config_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=29c5412f563d8131d25c69aa55f6fd17 expires=2026-11-14T12:22:49Z
|
||||
uploaded signed op to the hub jobs queue
|
||||
rc=0
|
||||
@@ -0,0 +1,59 @@
|
||||
#### the N100 (demo-felhom) AFTER the release — DooPlex clock 2026-11-14T11:52:11Z
|
||||
offset of the N100 to DooPlex: +0.177 s (before: -32.55 s); the ssh round trip was 0.36 s
|
||||
agent felhom-agent 0.174.0; wrapper 9d03203766040a97; chrony: active; failed units: 0
|
||||
Reference ID : C1E3C502 (193.227.197.2)
|
||||
System time : 0.000001782 seconds slow of NTP time
|
||||
Last offset : +0.002664797 seconds
|
||||
Leap status : Normal
|
||||
chrony profile: loaded 83417e83b538, cache 83417e83b538, fresh 83417e83b538
|
||||
record: { "at": "2026-11-14T11:51:13Z", "kernel": "7.0.14-28-pve", "restarted": [ "chrony.service" ], "rewritten": [ "f6da0254.0/lxc-containers", "f6da0254.0/usr.sbin.chronyd" ]}
|
||||
guest: status: running; containers 9 running, 0 unhealthy, newest container start 2026-11-14T11:52:10.648640606Z
|
||||
## the full minute, the host journal (agent, wrapper, chrony, systemd):
|
||||
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949497]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949539]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- pgrep -x dhclient
|
||||
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949539]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949542]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- ip -4 -o addr show dev eth0
|
||||
2026-11-14T12:51:05+01:00 demo-felhom sudo[1949542]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949539]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949542]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949553]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker exec felhom-controller cat /opt/docker/felhom-controller/controller.yaml
|
||||
2026-11-14T12:51:06+01:00 demo-felhom sudo[1949553]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949573]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949573]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949573]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949586]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sdb
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949586]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949553]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949586]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949608]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a -j /dev/sda
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949608]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949608]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949611]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-pbs-apply read felhom-pbs /etc/pve/priv/storage
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949611]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949611]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949619]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- docker inspect -f '{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' cloudflared
|
||||
2026-11-14T12:51:07+01:00 demo-felhom sudo[1949619]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:08+01:00 demo-felhom sudo[1949619]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:09+01:00 demo-felhom sudo[1949801]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts115109-host-facts.json
|
||||
2026-11-14T12:51:09+01:00 demo-felhom sudo[1949801]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949801]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949874]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949874]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949874]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sshd-session[1949878]: Connection closed by 127.0.0.1 port 50390
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949879]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -T -f /etc/felhom-sshd/sshd_config
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949879]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949879]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949882]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/sshd -t -f /etc/felhom-sshd/sshd_config
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949882]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949882]: pam_unix(sudo:session): session closed for user root
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949886]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/bin/wg show wg-felhom latest-handshakes
|
||||
2026-11-14T12:51:10+01:00 demo-felhom sudo[1949886]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
## the controller in the guest, the same minutes (WARN/ERROR lines, and any line on time):
|
||||
(end)
|
||||
## the hub, events of demo-felhom since 11:50Z:
|
||||
(end)
|
||||
#### the N100, 2026-11-14T14:23:46Z — 2.5 hours after the repair
|
||||
offset of the N100 to DooPlex: +0.178 s; the ssh round trip was 0.36 s
|
||||
chrony: active since Sat 2026-11-14 12:51:13 CET; System time : 0.000158301 seconds fast of NTP time
|
||||
failed units: 0; guest containers: 9 running, 0 unhealthy; controller boot-gate lines since: 0
|
||||
@@ -0,0 +1,101 @@
|
||||
#### demo-felhom 2026-11-14T11:52:28Z — which container started near the clock step?
|
||||
2026-11-14T04:49:14.901603074Z restarts=0 status=running health=healthy prowlarr
|
||||
2026-11-14T09:18:23.620814926Z restarts=0 status=running health=healthy felhom-controller
|
||||
2026-11-14T11:52:05.808655933Z restarts=0 status=running health=healthy radarr
|
||||
2026-11-14T11:52:09.614378919Z restarts=0 status=running health=healthy sonarr
|
||||
2026-11-14T11:52:10.648640606Z restarts=0 status=running health=healthy qbittorrent
|
||||
## docker events 11:50–11:54Z (start/die/restart/health):
|
||||
2026-11-14T11:52:00.406970341Z container kill fc2cd3bd36b94bbae38a99e8b476cf40b9e6fd142811f067647ad0f96077fd99 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com.docker.compose.co
|
||||
2026-11-14T11:52:05.629973236Z container die fc2cd3bd36b94bbae38a99e8b476cf40b9e6fd142811f067647ad0f96077fd99 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com.docker.compose.con
|
||||
2026-11-14T11:52:05.927443897Z container start e7b713a45c209013c5bc914d4f3cb9c8697627e3e6e1cb1aba6f99a7679a9de9 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com.docker.compose.c
|
||||
2026-11-14T11:52:06.175065238Z container kill e5351e8bed94fabe527f7cb26760a106f6ae1eb0c4a577d3c1cd5b03fa3b6266 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com.docker.compose.co
|
||||
2026-11-14T11:52:09.418552075Z container die e5351e8bed94fabe527f7cb26760a106f6ae1eb0c4a577d3c1cd5b03fa3b6266 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com.docker.compose.con
|
||||
2026-11-14T11:52:09.752602177Z container start 90d435fcdb3ed6e1731469312725c7edae2f6363da11999c0fcb34fd47fe7452 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com.docker.compose.c
|
||||
2026-11-14T11:52:09.977460590Z container kill 82bd9f8caf4124d1fff6b9b9fa1799088805815c63bb6835c9557c160ef51b3a (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com.docker.compose.co
|
||||
2026-11-14T11:52:10.407038534Z container die 82bd9f8caf4124d1fff6b9b9fa1799088805815c63bb6835c9557c160ef51b3a (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com.docker.compose.con
|
||||
2026-11-14T11:52:10.787260774Z container start 16e8e756487f2ea6ede83737f01b6c707292aa1b2f9004d3d71d11bd0131b8a1 (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com.docker.compose.c
|
||||
2026-11-14T11:52:11.003795092Z container health_status: healthy e7b713a45c209013c5bc914d4f3cb9c8697627e3e6e1cb1aba6f99a7679a9de9 (build_version=Linuxserver.io version:- 6.4.4.10685-ls318 Build-date:- 2026-09-20T19:09:21+00:00, com
|
||||
2026-11-14T11:52:14.829102987Z container health_status: healthy 90d435fcdb3ed6e1731469312725c7edae2f6363da11999c0fcb34fd47fe7452 (build_version=Linuxserver.io version:- 4.0.20.3014-ls326 Build-date:- 2026-09-26T01:45:08+00:00, com
|
||||
2026-11-14T11:52:15.847713728Z container health_status: healthy 16e8e756487f2ea6ede83737f01b6c707292aa1b2f9004d3d71d11bd0131b8a1 (com.docker.compose.config-hash=6a9fdfafecc6b0f48dfd822f165e48908900a1673b0d2ee724bb7d6b448a6051, com
|
||||
## the controller around it:
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/grimmory/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/immich/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/nextcloud/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 2 HDD mounts for /opt/docker/stacks/paperless-ngx/docker-compose.yml
|
||||
2026/11/14 11:52:10 [INFO] [stacks] ParseComposeHDDMounts: found 1 HDD mounts for /opt/docker/stacks/romm/docker-compose.yml
|
||||
2026/11/14 11:52:11 [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (1 storage path(s))
|
||||
2026/11/14 11:52:12 [INFO] [stacks] Stack sonarr post-start status:
|
||||
2026/11/14 11:52:12 [INFO] [stacks] sonarr lscr.io/linuxserver/sonarr:4.0.20@sha256:f247545d23ba8b233d6604575347e48a623fe6ad75dda02348bf81917f3b5c06 running Up 3 seconds (health: starting)
|
||||
2026/11/14 11:52:14 [INFO] [stacks] Stack qbittorrent post-start status:
|
||||
2026/11/14 11:52:14 [INFO] [stacks] qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1@sha256:92bfd78d731e254ba64f62b77b0696add6447a33642ee5feda106a26b285aa7f running Up 3 seconds (health: starting)
|
||||
2026/11/14 11:52:17 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
|
||||
2026/11/14 11:52:27 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
|
||||
2026/11/14 11:52:27 [INFO] Health probes: 3 ok (of 3 probed)
|
||||
2026/11/14 11:52:37 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
|
||||
2026/11/14 11:52:47 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
|
||||
2026/11/14 11:50:24 [INFO] [scheduler] Running job: stack-scan
|
||||
2026/11/14 11:50:24 [INFO] [scheduler] Running job: hold-report
|
||||
2026/11/14 11:50:24 [INFO] [scheduler] Job hold-report completed (took 0s)
|
||||
2026/11/14 11:50:24 [INFO] [stacks] ScanStacks complete: 68 stacks found (5 deployed, 63 available)
|
||||
2026/11/14 11:50:24 [INFO] [scheduler] Job stack-scan completed (took 87ms)
|
||||
2026/11/14 11:50:24 [INFO] [scheduler] Running job: agent-channel-health
|
||||
2026/11/14 11:50:25 [INFO] [scheduler] Job agent-channel-health completed (took 890ms)
|
||||
2026/11/14 11:51:14 [INFO] Health probes: 1 ok (of 1 probed)
|
||||
2026/11/14 11:51:57 [INFO] [scheduler] Running job: hold-report
|
||||
2026/11/14 11:51:57 [INFO] [scheduler] Job hold-report completed (took 0s)
|
||||
2026/11/14 11:51:57 [INFO] Health probes: 4 ok (of 4 probed)
|
||||
2026/11/14 11:51:57 [INFO] [scheduler] Running job: agent-channel-health
|
||||
2026/11/14 11:51:58 [INFO] [scheduler] Job agent-channel-health completed (took 818ms)
|
||||
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/hdd_1] before recreating drive-backed apps
|
||||
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app radarr (state=running) onto /mnt/felhom-drives/hdd_1
|
||||
2026/11/14 11:52:00 [INFO] [stacks] Stopping stack: radarr
|
||||
2026/11/14 11:52:05 [INFO] [stacks] Stack radarr stopped successfully (took 5.3s)
|
||||
2026/11/14 11:52:05 [INFO] [stacks] Starting stack: radarr
|
||||
2026/11/14 11:52:05 [INFO] [stacks] media_library radarr: per-drive binds 0 []; drives not bound: []
|
||||
2026/11/14 11:52:05 [INFO] [stacks] Stack radarr started successfully (took 0.3s)
|
||||
2026/11/14 11:52:06 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app sonarr (state=running) onto /mnt/felhom-drives/hdd_1
|
||||
2026/11/14 11:52:06 [INFO] [stacks] Stopping stack: sonarr
|
||||
2026/11/14 11:52:09 [INFO] [stacks] Stack radarr post-start status:
|
||||
2026/11/14 11:52:09 [INFO] [stacks] radarr lscr.io/linuxserver/radarr:6.4.4@sha256:adb6c09d6b729ea5e642c99cea35af72702ef476bf4763f153299ac5db9f0b4f running Up 3 seconds (health: starting)
|
||||
2026/11/14 11:52:09 [INFO] [stacks] Stack sonarr stopped successfully (took 3.4s)
|
||||
2026/11/14 11:52:09 [INFO] [stacks] Starting stack: sonarr
|
||||
2026/11/14 11:52:09 [INFO] [stacks] media_library sonarr: per-drive binds 0 []; drives not bound: []
|
||||
2026/11/14 11:52:09 [INFO] [stacks] Stack sonarr started successfully (took 0.3s)
|
||||
2026/11/14 11:52:09 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app qbittorrent (state=running) onto /mnt/felhom-drives/hdd_1
|
||||
2026/11/14 11:52:09 [INFO] [stacks] Stopping stack: qbittorrent
|
||||
2026/11/14 11:52:10 [INFO] [stacks] Stack qbittorrent stopped successfully (took 0.5s)
|
||||
2026/11/14 11:52:10 [INFO] [stacks] Starting stack: qbittorrent
|
||||
2026/11/14 11:52:10 [INFO] [stacks] media_library qbittorrent: per-drive binds 0 []; drives not bound: []
|
||||
2026/11/14 11:52:10 [INFO] [stacks] Stack qbittorrent started successfully (took 0.3s)
|
||||
2026/11/14 11:52:10 [INFO] [gate] boot 1794112550-1438: re-syncing FileBrowser mounts against the live binds
|
||||
2026/11/14 11:52:10 [INFO] [settings] Settings saved
|
||||
2026/11/14 11:52:11 [INFO] [web] FileBrowser sync — no config/compose change, ensured running without recreate (1 storage path(s))
|
||||
2026/11/14 11:52:12 [INFO] [stacks] Stack sonarr post-start status:
|
||||
2026/11/14 11:52:12 [INFO] [stacks] sonarr lscr.io/linuxserver/sonarr:4.0.20@sha256:f247545d23ba8b233d6604575347e48a623fe6ad75dda02348bf81917f3b5c06 running Up 3 seconds (health: starting)
|
||||
#### demo-felhom 2026-11-14T11:53:19Z — did anything else read the clock step as a restart?
|
||||
crash guard: {"armed":true,"boot_id":"72899d26-2524-4e06-9e8e-8099775d1e51","config":{"LIMIT":3,"PANIC_SECONDS":10,"REARM_HOURS":24,"WINDOW_MINUTES":60},"kernel_panic":10,"last_boot_at":"2026-11-08T04:35:18Z","last_boot_unclean":false,"tripped":false,"unclean_boots":[],"unclean_boots_24h":0,"unclean_boots_in_window":0,"updated_at":"2026-11-14T11:03:05Z","version":1}
|
||||
uptime since: 2026-11-08 05:35:50; /proc/stat btime: btime 1794112550 = 2026-11-08T04:35:50Z; boot_id 72899d26-2524-4e06-9e8e-8099775d1e51
|
||||
agent journal since the step (WARN/ERROR, boot, power, restart words):
|
||||
2026-11-14T12:51:56+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:51:56.457+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
|
||||
2026-11-14T12:52:16+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:52:16.469+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
|
||||
2026-11-14T12:52:36+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:52:36.511+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
|
||||
2026-11-14T12:52:56+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:52:56.477+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
|
||||
2026-11-14T12:53:16+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-14T12:53:16.525+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uui
|
||||
(end)
|
||||
kernel step state phase: good
|
||||
controller boot-gate lines today:
|
||||
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: waiting (≤2m0s) for live drive bind(s) [/mnt/felhom-drives/hdd_1] before recreating drive-backed apps
|
||||
2026/11/14 11:52:00 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app radarr (state=running) onto /mnt/felhom-drives/hdd_1
|
||||
2026/11/14 11:52:06 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app sonarr (state=running) onto /mnt/felhom-drives/hdd_1
|
||||
2026/11/14 11:52:09 [INFO] [gate] boot 1794112550-1438: live bind confirmed — recreating drive-backed app qbittorrent (state=running) onto /mnt/felhom-drives/hdd_1
|
||||
2026/11/14 11:52:10 [INFO] [gate] boot 1794112550-1438: re-syncing FileBrowser mounts against the live binds
|
||||
@@ -0,0 +1,19 @@
|
||||
#### demo-felhom (the N100) 2026-11-14T11:53:34Z — the DRY RUN through the released agent: what tonight removes. Nothing is changed.
|
||||
time=2026-11-14T12:53:35.431+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DRY RUN running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]"
|
||||
would: remove | not_run: [] | refusals: []
|
||||
installed: ['7.0.14-20-pve', '7.0.14-22-pve', '7.0.14-23-pve', '7.0.14-24-pve', '7.0.14-25-pve', '7.0.14-26-pve', '7.0.14-28-pve', '7.0.2-6-pve']
|
||||
KEPT:
|
||||
7.0.14-26-pve ["the kernel step's `from` (the last kernel that booted well; a self-revert boots it)", "the kernel step's `default_before`"]
|
||||
7.0.14-28-pve ['running', "the kernel step's `to`", 'the default in grub.cfg', 'the pinned default file', 'newest installed']
|
||||
WILL BE REMOVED ( 6 ):
|
||||
proxmox-kernel-7.0.14-20-pve-signed
|
||||
proxmox-kernel-7.0.14-22-pve-signed
|
||||
proxmox-kernel-7.0.14-23-pve-signed
|
||||
proxmox-kernel-7.0.14-24-pve-signed
|
||||
proxmox-kernel-7.0.14-25-pve-signed
|
||||
proxmox-kernel-7.0.2-6-pve-signed
|
||||
simulation: {'installs': [], 'rc': 0, 'removes': ['proxmox-kernel-7.0.14-20-pve-signed', 'proxmox-kernel-7.0.14-22-pve-signed', 'proxmox-kernel-7.0.14-23-pve-signed', 'proxmox-kernel-7.0.14-24-pve-signed', 'proxmox-kernel-7.0.14-25-pve-signed', 'proxmox-kernel-7.0.2-6-pve-signed']}
|
||||
sizes now: /lib/modules 7.6G, /boot 906M, / free 55G
|
||||
972M /lib/modules/7.0.14-20-pve;972M /lib/modules/7.0.14-22-pve;972M /lib/modules/7.0.14-23-pve;972M /lib/modules/7.0.14-24-pve;973M /lib/modules/7.0.14-25-pve;973M /lib/modules/7.0.14-26-pve;973M /lib/modules/7.0.14-28-pve;970M /lib/modules/7.0.2-6-pve;
|
||||
the hub block for this box (kernel step due tonight?): {'ring': 0, 'enabled': True, 'kernel': None}
|
||||
last night run marker: 2026-11-14T04:51:34Z
|
||||
@@ -0,0 +1,17 @@
|
||||
#### felhom (Tester 1) BEFORE the restart 2026-11-14T14:16:51Z
|
||||
kernel 7.0.14-22-pve; boot_id 6407d71b-1d89-4748-8655-49c6de8cf61b; up since 2026-11-10 15:43:22
|
||||
crash guard: {'armed': True, 'tripped': False, 'last_boot_at': '2026-11-10T14:43:22Z', 'last_boot_unclean': False, 'unclean_boots_in_window': 0, 'boot_id': '6407d71b-1d89-4748-8655-49c6de8cf61b'}
|
||||
busy? vzdump 0 apt 0; guest status: running; containers 10 running / 0 unhealthy
|
||||
RESTART (systemctl reboot) at 2026-11-14T14:16:54Z
|
||||
host answers, agent active, guest running after 156 s
|
||||
containers: 10 running, 0 starting, 0 unhealthy after 239 s (or the 540 s limit)
|
||||
#### felhom (Tester 1) AFTER the restart 2026-11-14T14:21:03Z
|
||||
kernel 7.0.14-22-pve (before: 7.0.14-22-pve); boot_id 7a721957-7e95-476c-be40-52d0dafc7284; up since 2026-11-14 15:19:11
|
||||
crash guard: {'armed': True, 'tripped': False, 'last_boot_at': '2026-11-14T14:19:11Z', 'last_boot_unclean': False, 'unclean_boots_in_window': 0, 'boot_id': '7a721957-7e95-476c-be40-52d0dafc7284'}
|
||||
## the host health rule: daemons active active active active active ; guest status: running; containers 10 running / 0 unhealthy; controller healthy; tunnel container: running
|
||||
kernels: /boot 7.0.14-22-pve 7.0.2-6-pve ; grub default gnulinux-7.0.14-22-pve-advanced; failed units 0; chrony active
|
||||
mark: { "host_id": "tester-1-d70be4", "ring0_slow_lane": false}
|
||||
## the agent at this start:
|
||||
2026-11-14T15:19:22+01:00 felhom felhom-agent[1148]: time=2026-11-14T15:19:22.533+01:00 level=INFO msg="felhom-agent daemon starting" version=0.174.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
|
||||
2026-11-14T15:19:22+01:00 felhom felhom-agent[1148]: time=2026-11-14T15:19:22.731+01:00 level=INFO msg="osupdate: profile refresh not needed" mode=profile-refresh why="agent start" kernel=7.0.14-22-pve reason="the profiles were refreshed on this kernel (7.0.14-22-pve) at 2026-11-14T11:30:11Z"
|
||||
profile-refresh record: { "at": "2026-11-14T11:30:11Z", "kernel": "7.0.14-22-pve", "restarted": [], "rewritten": [ "f6da0254.0/lxc-containers" ]}
|
||||
@@ -0,0 +1,38 @@
|
||||
## vouch read-back 2026-11-14T14:21:57Z — channel 1: the hub's log (the pod's clock is CET)
|
||||
2026/11/14 15:21:45 [INFO] Artifact manifest set: agent=0.174.0 golden=0.352.1 min_agent="0.158.1" wrapper_sha=false bundle_sha="7d87d0584dee79eee766f01c27b9e89273d39c89a14b0725a6e5564a55b3ec5f"
|
||||
2026/11/14 15:21:45 [INFO] artifact vouch timing (R-1346): registry probes 119ms, sha lookups 76ms, bundle sha 10ms, fleet check + save 222ms, total 428ms
|
||||
## channel 2: the hub's System page (CC's token)
|
||||
Global controller floor: 0.352.1 · vouched agent: 0.174.0
|
||||
Needs attention
|
||||
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
|
||||
look
|
||||
Tester4-46836f Tester4
|
||||
kernel step: — · the host needs a restart since 2026-11-14 · root files behind the vouched agent's · agent behind: 0.173.0 → 0.174.0 · could not read: default kernel
|
||||
look
|
||||
tester-1-d70be4 Tester 1
|
||||
partial: no running customer guest · crash guard not installed · could not read: running kernel, next-boot kernel, default kernel, host Debian, held packages, kernel.panic, crash restarts, guest Debian, Docker engine, containerd, Docker live-restore
|
||||
demo-felhom-8363b5 10 s no 0 armed 0.174.0
|
||||
demo-hp-bb76ea 10 s no 0 armed 0.174.0
|
||||
tester-1-d70be4 unknown no unknown not installed 0.174.0
|
||||
root-files cell titles: ['behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_upd']
|
||||
demo-felhom-8363b5 Demo Ügyfél 0.174.0 9.2.26 7.0.14-28-pve ONLINE 1/2 5% 23% 37% running (connected) 37% local demo-hp-bb76ea D
|
||||
demo-hp-bb76ea Demo HP 0.174.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 48% 42% running (connected) 62% local-lvm tester-1-d70be4
|
||||
tester-1-d70be4 Tester 1 0.174.0 9.2.2 7.0.14-22-pve ONLINE 1/2 0% 33% 30% running (connected) 30% local Felhom Hub 0.181.0
|
||||
Tester4-46836f Tester4 0.173.0 9.2.2 7.0.2-6-pve DOWN 1/1 11% 20% 21% absent (no cloudflared container in the guest (no tunn
|
||||
## the hub's System page, read 2026-11-14T14:25:58Z
|
||||
Needs attention
|
||||
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
|
||||
look
|
||||
Tester4-46836f Tester4
|
||||
kernel step: — · the host needs a restart since 2026-11-14 · root files behind the vouched agent's · agent behind: 0.173.0 → 0.174.0 · could not read: default kernel
|
||||
look
|
||||
tester-1-d70be4 Tester 1
|
||||
partial: no running customer guest · crash guard not installed · could not read: running kernel, next-boot kernel, default kernel, host Debian, held packages, kernel.panic, crash restarts, guest Debian, Docker engine, containerd, Docker live-restore
|
||||
Crash guard and root files
|
||||
Box kernel.panic Oops Crash restarts 24 h Crash guard Root files
|
||||
Tester4-46836f 10 s no 0 armed 0.173.0
|
||||
demo-felhom-8363b5 10 s no 0 armed 0.174.0
|
||||
demo-hp-bb76ea 10 s no 0 armed 0.174.0
|
||||
tester-1-d70be4 unknown no unknown not installed 0.174.0
|
||||
Felhom Hub 0.181.0
|
||||
(function(){var h=location.hash&&document.getElementById(location.hash.slice(1));if(!h)return;var d=h.closest('details');while(d){d.open=true;d=d.parentElement&
|
||||
@@ -0,0 +1,3 @@
|
||||
felhom: felhom-agent 0.174.0; sha256 6efbb97809135483; go in the running binary: 2 go1.27.4; unit active restarts 0
|
||||
demo-hp: felhom-agent 0.174.0; sha256 6efbb97809135483; go in the running binary: 2 go1.27.4; unit active restarts 0
|
||||
demo-felhom: felhom-agent 0.174.0; sha256 6efbb97809135483; go in the running binary: 2 go1.27.4; unit active restarts 0
|
||||
@@ -0,0 +1,25 @@
|
||||
# Release 2a — the box removes its old kernels, and refreshes its security profiles — evidence (2026-11-14)
|
||||
|
||||
Report: `REPORT-release-2a-2026-11-14.md` (repo root). The profile spike: `../profile-refresh-spike-2026-11-14/`.
|
||||
Agent v0.174.0 (felhom-agent `cc63dd5`), vouched 2026-11-14 15:21:45 CET.
|
||||
|
||||
| File | What it shows |
|
||||
|---|---|
|
||||
| `A/redproof.txt` | the clean-up's wrapper tests: 14 mutations (K1, the simulation, the phase, the ring-0 mark, the flag, the default, not-on-newest, fell back, the plan's names, no state file, the 20-hour window, the dry run, repair-first, the after-purge proof) — each RED on the mutant, GREEN on the real file |
|
||||
| `A/redproof-agent.txt` | the agent's clean-up tests: 3 mutations, each red |
|
||||
| `A/01-dryrun-preview-*.txt` | the working-copy wrapper's dry run on demo-hp and Tester 1, before the commit |
|
||||
| `B/redproof.txt`, `B/redproof-agent.txt` | the profile refresh: 10 + 3 mutations, each red (one agent mutation did not compile at the first try and was done again; both are in the file) |
|
||||
| `D/01-published-binary.txt` | the published agent, downloaded with no login: sha256, `go1.27.4` |
|
||||
| `D/02`–`D/06` | delivery to Tester 1 and demo-hp: before; the signed `agent_update`; the journals (the old wrapper refuses the new mode at the agent's start); the signed bundle; bundle installed 77/77 and the profile refresh right after it |
|
||||
| `D/07-tester1-dryrun.txt` | Tester 1 through the released agent: „not run", it has no ring-0 mark (scenario 8, live) |
|
||||
| `D/08-tester1-cleanup.txt` | **the first real removal by the program** (scenario 1), the second call (scenario 2), the mark set for the test and put back. Started by hand — see the report |
|
||||
| `D/09-n100-before.txt`, `D/11`, `D/12` | the N100 before (−32.55 s, chrony failed, the stale profile); its two signed jobs |
|
||||
| `D/13-n100-after.txt` | the N100 after: +0.18 s, chrony active, the profile is the fresh compile; the same 2.5 hours later |
|
||||
| `D/13b-n100-container-near-the-step.txt` | the controller recreated three apps 9 s after the clock step, and why (R-1395); the crash guard was not fooled |
|
||||
| `D/14-n100-dryrun-tonight.txt` | what the N100 removes tonight: 6 of 8 kernels |
|
||||
| `D/10-demo-hp-dryruns-and-refusals.txt` | demo-hp: nothing to remove; P1, P2, P3 by dry run against states set and put back; identical after |
|
||||
| `D/15-tester1-reboot.txt` | one orderly restart of Tester 1: the same kernel, healthy after 239 s, a clean stop, „profile refresh not needed" |
|
||||
| `D/16-vouch-readback.txt` | the vouch from the hub's log and from its System page |
|
||||
| `D/17-go-version-on-boxes.txt` | the running agent on the three boxes: sha256 and `go1.27.4` |
|
||||
|
||||
Not kept as files: the release script's own output (it prints a repository address; the CHANGELOG has the two sha256s).
|
||||
Reference in new issue
Block a user