admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

hub v0.132.0: the System page (versions + OS updates with the ring/switch/approve buttons, R-852), the Hosts Proxmox/kernel column, the operator-approved Docker engine release (2 healthy ring-0 nights), the crash-guard events (R-851); evidence audits/os-docker-crash-2026-10-04
gates / gates (push) Successful in 30s
Details

Browse Source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
authentik Default Admin
2026-10-04 16:16:02 +02:00
parent bee277ffad
commit 175ecfcdd2
45 changed files with 2198 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/os-docker-crash-2026-10-04/partA/hub-redproofs.txt
+31
View File
@@ -0,0 +1,31 @@
# Hub v0.132.0 red-proofs, 2026-10-04T14:04:22Z
# First run: 2 of 10 NOT CAUGHT (the parser test never fed empty facts fields; the page test counted the word 'unknown' but not its amber class). Both tests strengthened; the rerun below catches all 10.
[CAUGHT] unknown, never empty (parser): internal/sysfacts/sysfacts.go mutated; TestParse_AbsentAndPartialAreUnknown ran=True failed=True
sysfacts_test.go:38: empty facts fields: {Present:true PVEVersion:x KernelVersion:unknown VMID:0 FactsError: ReadAt: Host:{Debian:unknown KernelRunning:unknown KernelNextBoot:unknown KernelNextBootSou
[CAUGHT] Docker never auto-approved: internal/osupdates/service.go mutated; TestDocker_NeverAutoApproved ran=True failed=True
docker_test.go:33: a Docker set was auto-approved: &{ID:os-docker-20261006-140000 Layer:docker Fingerprint:a4ec54235cdd17a5 ApprovedAt:2026-10-06 14:00:00 +0000 UTC ApprovedBy:auto PackagesJSON:[{"nam
[CAUGHT] Docker button needs 2 nights: internal/osupdates/service.go mutated; TestDocker_ApproveNeedsTwoHealthyNightsOnEveryRing0Box ran=True failed=True
docker_test.go:45: approved after one night: <nil>
[CAUGHT] a Docker approval nudges no box: internal/osupdates/service.go mutated; TestDocker_ApproveNeedsTwoHealthyNightsOnEveryRing0Box ran=True failed=True
docker_test.go:62: a Docker approval must nudge no box (ring 1 takes it by a signed job): [cust1]
[CAUGHT] crash event once per boot: internal/api/crash.go mutated; TestCheckCrash_EventsOncePerFact ran=True failed=True
crash_test.go:45: restart events: op=4 household=4
[CAUGHT] first sighting is quiet: internal/api/crash.go mutated; TestCheckCrash_EventsOncePerFact ran=True failed=True
crash_test.go:39: first sighting flooded 1 events
[CAUGHT] page: unknown cell: internal/web/system.go mutated; TestSystemPage_FullPartialUnknown ran=True failed=True
system_test.go:65: an unknown value must be shown amber with its reason, got 0
[CAUGHT] page: Docker button gate: internal/web/templates/system.html mutated; TestSystemPage_DockerButtonOnlyWhenReady ran=True failed=True
system_test.go:87: button shown after ONE night
[CAUGHT] buttons need the operator login: internal/web/server.go mutated; TestSystemButtons_NeedTheOperatorLogin ran=True failed=True
system_test.go:117: POST /os/ring/full-1 with auth "" = 303 — must be refused
system_test.go:117: POST /os/ring/full-1 with auth "Bearer k-full-1" = 303 — must be refused
system_test.go:117: POST /os/enabled/full-1 with auth "" = 303 — must be refused
system_test.go:117: POST /os/enabled/full-1 with auth "Bearer k-full-1" = 303 — must be refused
system_test.go:117: POST /os/approve-now with auth "" = 303 — must be refused
system_test.go:117: POST /os/approve-now with auth "Bearer k-full-1" = 303 — must be refused
system_test.go:117: POST /os/approve-docker with auth "" = 303 — must be refused
system_test.go:117: POST /os/approve-docker with auth "Bearer k-full-1" = 303 — must be refused
system_test.go:122: an unauthenticated POST changed a box
[CAUGHT] Hosts column: internal/web/templates/hosts.html mutated; TestHostsPage_ProxmoxKernelColumn ran=True failed=True
system_test.go:153: hosts column missing:
after restore: ok gitea.dooplex.hu/admin/felhom-hub/internal/sysfacts 0.003s | ok gitea.dooplex.hu/admin/felhom-hub/internal/osupdates 0.682s | ok gitea.dooplex.hu/admin/felhom-hub/internal/api 4.765s | ok gitea.dooplex.hu/admin/felhom-hub/internal/web 15.831s rc= 0
documentation/audits/os-docker-crash-2026-10-04/partA/hub_rp2.py
+54
View File
@@ -0,0 +1,54 @@
import subprocess, shutil, sys
M = [
("unknown, never empty (parser)", "internal/sysfacts/sysfacts.go",
"out.Guest.Containerd, out.Guest.LiveRestore = orUnknown(out.Guest.Containerd), orUnknown(out.Guest.LiveRestore)",
"out.Guest.Containerd, out.Guest.LiveRestore = out.Guest.Containerd, out.Guest.LiveRestore",
"./internal/sysfacts/", "TestParse_AbsentAndPartialAreUnknown"),
("Docker never auto-approved", "internal/osupdates/service.go",
"var Layers = []string{LayerGuest, LayerHost}", "var Layers = []string{LayerGuest, LayerHost, LayerDocker}",
"./internal/osupdates/", "TestDocker_NeverAutoApproved"),
("Docker button needs 2 nights", "internal/osupdates/service.go",
"\tcase need == 0:\n\t\tneed = 2", "\tcase need == 0:\n\t\tneed = 1",
"./internal/osupdates/", "TestDocker_ApproveNeedsTwoHealthyNightsOnEveryRing0Box"),
("a Docker approval nudges no box", "internal/osupdates/service.go",
"if s.Bump != nil && layer != LayerDocker {", "if s.Bump != nil {",
"./internal/osupdates/", "TestDocker_ApproveNeedsTwoHealthyNightsOnEveryRing0Box"),
("crash event once per boot", "internal/api/crash.go",
"\t\tif known[b] {\n\t\t\tcontinue\n\t\t}", "\t\tif false {\n\t\t\tcontinue\n\t\t}",
"./internal/api/", "TestCheckCrash_EventsOncePerFact"),
("first sighting is quiet", "internal/api/crash.go",
"\t\tif first {\n\t\t\treturn\n\t\t}\n\t\td := map", "\t\td := map",
"./internal/api/", "TestCheckCrash_EventsOncePerFact"),
("page: unknown cell", "internal/web/system.go",
'if s == "" || s == sysfacts.Unknown {', 'if false {',
"./internal/web/", "TestSystemPage_FullPartialUnknown"),
("page: Docker button gate", "internal/web/templates/system.html",
'(not .Approved) (eq .Waiting "")}}', '(not .Approved)}}',
"./internal/web/", "TestSystemPage_DockerButtonOnlyWhenReady"),
("buttons need the operator login", "internal/web/server.go",
'if r.URL.Path == "/login" || isPublicBindPath(r.URL.Path) {', 'if r.URL.Path == "/login" || isPublicBindPath(r.URL.Path) || strings.HasPrefix(r.URL.Path, "/os/") {',
"./internal/web/", "TestSystemButtons_NeedTheOperatorLogin"),
("Hosts column", "internal/web/templates/hosts.html",
"<th>Proxmox / kernel</th>", "<th>Proxmox</th>",
"./internal/web/", "TestHostsPage_ProxmoxKernelColumn"),
]
ok = True
for name, f, old, new, pkg, test in M:
src = open(f).read()
assert src.count(old) == 1, (name, src.count(old))
shutil.copy(f, f + ".bak")
open(f, "w").write(src.replace(old, new))
r = subprocess.run(["go", "test", pkg, "-run", "^" + test + "$", "-v", "-count=1"], capture_output=True, text=True)
shutil.move(f + ".bak", f)
out = r.stdout + r.stderr
ran = ("=== RUN " + test) in out
failed = ("--- FAIL: " + test) in out
v = "CAUGHT" if ran and failed else "NOT CAUGHT"
ok &= v == "CAUGHT"
print(f"[{v}] {name}: {f} mutated; {test} ran={ran} failed={failed}")
for l in out.splitlines():
if "_test.go:" in l or "build failed" in l or l.startswith("#"):
print(" " + l.strip()[:200])
r = subprocess.run(["go", "test", "./internal/sysfacts/", "./internal/osupdates/", "./internal/api/", "./internal/web/", "-count=1"], capture_output=True, text=True)
print("after restore:", r.stdout.strip().replace("\n", " | "), "rc=", r.returncode)
sys.exit(0 if ok and r.returncode == 0 else 1)
documentation/audits/os-docker-crash-2026-10-04/partA/live/facts-demo-hp-rc1.txt
+39
View File
@@ -0,0 +1,39 @@
--- facts (vmid 9201, 2.1s) ---
{
"guest": {
"containerd": "2.3.3-1~debian.13~trixie",
"debian": "13.7",
"docker_engine": "29.7.2",
"live_restore": "on"
},
"host": {
"crash_guard": {
"armed": true,
"boot_id": "0defc390-c295-4e9f-aa6e-13d69a1f24c3",
"config": {
"LIMIT": 3,
"PANIC_SECONDS": 10,
"REARM_HOURS": 24,
"WINDOW_MINUTES": 60
},
"kernel_panic": 10,
"last_boot_at": "2026-10-04T13:39:06Z",
"last_boot_unclean": false,
"tripped": false,
"unclean_boots": [],
"unclean_boots_24h": 0,
"unclean_boots_in_window": 0,
"updated_at": "2026-10-04T14:08:23Z",
"version": 1
},
"debian": "13.7",
"held": [],
"kernel_next_boot": "7.0.14-20-pve",
"kernel_next_boot_source": "saved default",
"kernel_panic": 10,
"kernel_running": "7.0.14-20-pve",
"oops_this_boot": false,
"tainted": 4097,
"warn_this_boot": false
}
}
documentation/audits/os-docker-crash-2026-10-04/partB/agent-redproofs.txt
+106
View File
@@ -0,0 +1,106 @@
# Agent v0.142.0 red-proofs, 2026-10-04T13:54:18Z
[CAUGHT] docker pkg in a fast plan refused (R2): configs/felhom-os-apply mutated; DockerLane.test_docker_package_in_a_fast_plan_is_refused ran=True rc=1
test_docker_package_in_a_fast_plan_is_refused (__main__.DockerLane.test_docker_package_in_a_fast_plan_is_refused) ... FAIL
FAIL: test_docker_package_in_a_fast_plan_is_refused (__main__.DockerLane.test_docker_package_in_a_fast_plan_is_refused)
self.assertEqual(rc, 2, rep)
AssertionError: 0 != 2 : {'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'running'}, 'felhom-controller': {'health': 'healthy', '
FAILED (failures=1)
[CAUGHT] slow lane without authority refused (R3): configs/felhom-os-apply mutated; DockerLane.test_no_authority_is_refused ran=True rc=1
test_no_authority_is_refused (__main__.DockerLane.test_no_authority_is_refused) ... FAIL
FAIL: test_no_authority_is_refused (__main__.DockerLane.test_no_authority_is_refused)
self.assertEqual(rc, 2, rep)
AssertionError: 0 != 2 : {'authority': 'ring0', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'runnin
FAILED (failures=1)
[CAUGHT] bad signature refused (R3): configs/felhom-os-apply mutated; DockerLane.test_bad_signature_is_refused ran=True rc=1
test_bad_signature_is_refused (__main__.DockerLane.test_bad_signature_is_refused) ... FAIL
FAIL: test_bad_signature_is_refused (__main__.DockerLane.test_bad_signature_is_refused)
self.assertEqual(rc, 2, rep)
AssertionError: 0 != 2 : {'authority': 'signed', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'runni
FAILED (failures=1)
[CAUGHT] replay refused (R3): configs/felhom-os-apply mutated; DockerLane.test_replayed_signed_job_is_refused ran=True rc=1
test_replayed_signed_job_is_refused (__main__.DockerLane.test_replayed_signed_job_is_refused) ... FAIL
FAIL: test_replayed_signed_job_is_refused (__main__.DockerLane.test_replayed_signed_job_is_refused)
self.assertEqual(rc, 2, rep)
AssertionError: 0 != 2 : {'authority': 'signed', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'runni
FAILED (failures=1)
[CAUGHT] agent-writable trust file refused (R3): configs/felhom-os-apply mutated; DockerLane.test_agent_writable_trust_file_is_refused ran=True rc=1
test_agent_writable_trust_file_is_refused (__main__.DockerLane.test_agent_writable_trust_file_is_refused) ... FAIL
FAIL: test_agent_writable_trust_file_is_refused (__main__.DockerLane.test_agent_writable_trust_file_is_refused)
self.assertEqual(rc, 2, rep)
AssertionError: 0 != 2 : {'authority': 'ring0', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'runnin
FAILED (failures=1)
[CAUGHT] live-restore off refused (R15): configs/felhom-os-apply mutated; DockerLane.test_live_restore_off_is_refused ran=True rc=1
test_live_restore_off_is_refused (__main__.DockerLane.test_live_restore_off_is_refused) ... FAIL
FAIL: test_live_restore_off_is_refused (__main__.DockerLane.test_live_restore_off_is_refused)
self.assertEqual(rc, 2, rep)
AssertionError: 0 != 2 : {'authority': 'signed', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'runni
FAILED (failures=1)
[CAUGHT] live-restore uses reload, never restart: configs/felhom-os-apply mutated; LiveRestore.test_turns_it_on_with_a_reload_never_a_restart ran=True rc=1
test_turns_it_on_with_a_reload_never_a_restart (__main__.LiveRestore.test_turns_it_on_with_a_reload_never_a_restart) ... FAIL
FAIL: test_turns_it_on_with_a_reload_never_a_restart (__main__.LiveRestore.test_turns_it_on_with_a_reload_never_a_restart)
self.assertEqual(rc, 0, rep)
AssertionError: 3 != 0 : {'failed': {'rc': 3, 'reason': '', 'step': 'live-restore'}, 'layer': 'guest', 'live_restore': {'containers_before': 2, 'reload_rc': 0, 'result': 'failed', 'same_ids': True}, '
FAILED (failures=1)
[CAUGHT] guest scanned every pass (R-849): configs/felhom-os-apply mutated; HostLayer.test_every_layer_scans_every_pass ran=True rc=1
test_every_layer_scans_every_pass (__main__.HostLayer.test_every_layer_scans_every_pass) ... FAIL
FAIL: test_every_layer_scans_every_pass (__main__.HostLayer.test_every_layer_scans_every_pass)
self.assertTrue(rep["reboot_scanned"] and rep["reboot_needed"], (layer, rep))
AssertionError: False is not true : ('guest', {'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'running'}, 'felhom-controller': {'health': 'healthy', 'id': 'aaa11
FAILED (failures=1)
[CAUGHT] facts: next-boot by dpkg order: configs/felhom-os-apply mutated; Facts.test_next_entry_wins_and_default_zero_is_the_newest ran=True rc=1
test_next_entry_wins_and_default_zero_is_the_newest (__main__.Facts.test_next_entry_wins_and_default_zero_is_the_newest) ... FAIL
FAIL: test_next_entry_wins_and_default_zero_is_the_newest (__main__.Facts.test_next_entry_wins_and_default_zero_is_the_newest)
self.assertEqual(fa["host"]["kernel_next_boot"], "7.0.14-20-pve", "dpkg order, not string order (7.0.2 < 7.0.14)")
AssertionError: '7.0.2-6-pve' != '7.0.14-20-pve'
FAILED (failures=1)
[CAUGHT] crash guard: 3rd crash leaves the box off: configs/felhom-crash-guard mutated; Guard.test_third_crash_in_an_hour_leaves_the_box_off ran=True rc=1
FAIL
FAIL: test_third_crash_in_an_hour_leaves_the_box_off (__main__.Guard.test_third_crash_in_an_hour_leaves_the_box_off)
self.assertTrue(s["tripped"], s)
AssertionError: False is not true : {'armed': True, 'boot_id': 'unknown', 'config': {'LIMIT': 3, 'PANIC_SECONDS': 10, 'REARM_HOURS': 24, 'WINDOW_MINUTES': 60}, 'kernel_panic': 10, 'last_boot_at': '202
FAILED (failures=1)
[CAUGHT] crash guard: clean reboots never count: configs/felhom-crash-guard mutated; Guard.test_clean_reboots_never_count ran=True rc=1
FAIL
FAIL: test_clean_reboots_never_count (__main__.Guard.test_clean_reboots_never_count)
self.assertEqual(s["unclean_boots_in_window"], 0)
AssertionError: 5 != 0
FAILED (failures=1)
[CAUGHT] crash guard: 24 h re-arm not earlier: configs/felhom-crash-guard mutated; Guard.test_rearms_after_24h_of_normal_running ran=True rc=1
FAIL
FAIL: test_rearms_after_24h_of_normal_running (__main__.Guard.test_rearms_after_24h_of_normal_running)
self.assertTrue(self.e.state()["tripped"], "not before 24 h")
AssertionError: False is not true : not before 24 h
FAILED (failures=1)
[CAUGHT] docker health: changed id fails: internal/osupdate/leg.go mutated; TestDockerHealthVerdict ran=True failed=True
leg_test.go:442: a changed container id passed — live-restore failed and the apps restarted
--- FAIL: TestDockerHealthVerdict (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.008s
FAIL
[CAUGHT] docker health: changed id -> health_failed: internal/osupdate/leg.go mutated; TestDocker_ChangedIDIsHealthFailed ran=True failed=True
leg_test.go:465: docker = {RunID:20261004T040000Z Layer:docker Trigger:night Mode:apply Ring:0 ReleaseID:ring0-20261004T040000Z Outcome:applied Healthy:true HealthReason: VMID:9201 Upgraded:[{Name:doc
--- FAIL: TestDocker_ChangedIDIsHealthFailed (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.011s
FAIL
[CAUGHT] docker health: engine must move: internal/osupdate/leg.go mutated; TestDockerHealthVerdict ran=True failed=True
leg_test.go:445: the engine did not move and the step passed
--- FAIL: TestDockerHealthVerdict (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.007s
FAIL
[CAUGHT] ring 1 never steps in the night leg: internal/osupdate/leg.go mutated; TestDocker_Ring1NightLegNeverSteps ran=True failed=True
leg_test.go:390: ring 1 took a docker step: guest:inventory,host:inventory,guest:live-restore-on,docker:apply
--- FAIL: TestDocker_Ring1NightLegNeverSteps (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.009s
FAIL
after restore: ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.628s rc=0
[executor] signed envelope reaches the plan — mutation: drop plan["signed"]
=== RUN TestDockerStepExecutor_PassesTheSignedEnvelope
dockerjob_test.go:31: docker plan = map[lane:slow layer:docker mode:apply packages:[map[name:docker-ce origin:Docker CE version:5:29.8.2-1~debian.13~trixie]] release_id:os-docker-1 select:listed s
--- FAIL: TestDockerStepExecutor_PassesTheSignedEnvelope (0.00s)
FAIL
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.009s
FAIL
documentation/audits/os-docker-crash-2026-10-04/partB/b1-live-restore-9202.txt
+80
View File
@@ -0,0 +1,80 @@
+ pct exec 9202 -- docker ps --no-trunc --format '{{.ID}} {{.Names}} {{.Status}}'
+ sort
+ cat /root/9202-ids-before.txt
+ cut -c1-12,65-
1b8dc9032f36 paperless-webserver Up 27 minutes (healthy)
2d6019eb0042 paperless-redis Up 27 minutes (healthy)
3f8f9651ee32 felhom-controller Up 27 minutes (healthy)
468aa784345b paperless-postgres Up 27 minutes (healthy)
7504fa621b39 filebrowser Up 27 minutes (healthy)
8d8778d4dcf8 traefik Up 27 minutes
+ pct exec 9202 -- docker info --format 'live={{.LiveRestoreEnabled}}'
live=false
+ pct exec 9202 -- cat /etc/docker/daemon.json
{
"features": { "containerd-snapshotter": false },
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "3" }
}
++ date +%s.%N
+ S=1791122916.729550605
+ sudo -u felhom-agent /usr/local/bin/felhom-agent-0.142.0-rc1 --config /etc/felhom-agent/agent.json --selftest=live-restore -vmid 9202
+ grep -E 'LIVE-RESTORE|live-restore|error|refused'
selftest=live-restore: refused: {"code": "R10", "reason": "vmid 9202 does not bind /mnt/felhom-drives \u2014 it is not this box's customer guest"}
++ bc
+++ date +%s.%N
++ echo 1791122916.827586880-1791122916.729550605
+ echo WALL=.098036275
WALL=.098036275
+ pct exec 9202 -- docker ps --no-trunc --format '{{.ID}} {{.Names}} {{.Status}}'
+ sort
+ cut '-d ' -f1,2 /root/9202-ids-before.txt
+ cut '-d ' -f1,2 /root/9202-ids-after.txt
+ diff /tmp/a /tmp/b
++ wc -l
IDS IDENTICAL (6 containers)
+ echo 'IDS IDENTICAL (6 containers)'
+ pct exec 9202 -- docker info --format 'live={{.LiveRestoreEnabled}}'
live=false
+ pct exec 9202 -- cat /etc/docker/daemon.json
{
"features": { "containerd-snapshotter": false },
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "3" }
}
+ pct exec 9202 -- systemctl show docker -p ActiveEnterTimestamp -p NRestarts
NRestarts=0
ActiveEnterTimestamp=Sun 2026-10-04 13:40:57 UTC
+ pct exec 9202 -- cp -p /etc/docker/daemon.json /root/daemon.json.bak-2026-10-04
+ pct exec 9202 -- python3 -c '
import json; p="/etc/docker/daemon.json"; d=json.load(open(p)); d["live-restore"]=True; open(p,"w").write(json.dumps(d,indent=2,sort_keys=True)+"\n")'
++ date +%s.%N
+ S=1791122938.796302354
+ pct exec 9202 -- systemctl reload docker
++ bc
+++ date +%s.%N
++ echo 1791122939.730342275-1791122938.796302354
reload_rc=0 took=.934039921
+ echo 'reload_rc=0 took=.934039921'
+ sleep 3
+ pct exec 9202 -- docker info --format 'live={{.LiveRestoreEnabled}}'
live=true
+ pct exec 9202 -- docker ps --no-trunc --format '{{.ID}} {{.Names}} {{.Status}}'
+ sort
+ cut '-d ' -f1,2 /root/9202-ids-before.txt
+ cut '-d ' -f1,2 /root/9202-ids-after.txt
+ diff /tmp/a /tmp/b
++ wc -l
IDS IDENTICAL (6 containers)
+ echo 'IDS IDENTICAL (6 containers)'
+ cut -c1-12,65- /root/9202-ids-after.txt
1b8dc9032f36 paperless-webserver Up 28 minutes (healthy)
2d6019eb0042 paperless-redis Up 28 minutes (healthy)
3f8f9651ee32 felhom-controller Up 28 minutes (healthy)
468aa784345b paperless-postgres Up 28 minutes (healthy)
7504fa621b39 filebrowser Up 28 minutes (healthy)
8d8778d4dcf8 traefik Up 28 minutes
+ pct exec 9202 -- systemctl show docker -p ActiveEnterTimestamp -p NRestarts -p ExecMainPID
NRestarts=0
ExecMainPID=227
ActiveEnterTimestamp=Sun 2026-10-04 13:40:57 UTC
documentation/audits/os-docker-crash-2026-10-04/partB/b2-live-restore-demo-hp-9201.txt
+7
View File
@@ -0,0 +1,7 @@
before: 24 containers, live=false
os-apply: LIVE-RESTORE reload_rc=0 state=true containers=24 same-ids=yes
live-restore: on (see the wrapper's LIVE-RESTORE line above for the container ids)
WALL=7.532254689
IDS IDENTICAL (24 containers)
after: live=true
NRestarts=0 ActiveEnterTimestamp=Sun 2026-10-04 13:41:02 UTC
documentation/audits/os-docker-crash-2026-10-04/partB/b3-ring0-docker-demo-hp.log
+202
View File
@@ -0,0 +1,202 @@
=== felhom-agent 0.142.0-rc1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true ===
time=2026-10-04T16:09:47.878+02:00 level=INFO msg="osupdate: START" run=20261004T140947Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T140947Z
time=2026-10-04T16:10:04.518+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T140947Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
time=2026-10-04T16:10:04.518+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
time=2026-10-04T16:10:04.518+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
time=2026-10-04T16:10:04.518+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
time=2026-10-04T16:10:04.519+02:00 level=INFO msg="osupdate: DONE" run=20261004T140947Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=0 reboot_needed=false wrapper_seconds=16.6
time=2026-10-04T16:10:04.530+02:00 level=INFO msg="osupdate: START" run=20261004T140947Z layer=host vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T140947Z
time=2026-10-04T16:10:19.645+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T140947Z layer=host lane=fast mode=apply select=pending-fast packages=0"
time=2026-10-04T16:10:19.645+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
time=2026-10-04T16:10:19.645+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
time=2026-10-04T16:10:19.645+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
time=2026-10-04T16:10:20.691+02:00 level=INFO msg="osupdate: DONE" run=20261004T140947Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=78 not_covered=78 restart_needed=0 reboot_needed=false wrapper_seconds=15
time=2026-10-04T16:10:22.649+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
time=2026-10-04T16:10:22.649+02:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}"
time=2026-10-04T16:10:22.649+02:00 level=INFO msg="osupdate: START" run=20261004T140947Z layer=docker vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T140947Z
time=2026-10-04T16:11:14.270+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T140947Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0"
time=2026-10-04T16:11:14.270+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
time=2026-10-04T16:11:14.270+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=6 already=0 not-installed=0 from-snapshot=0"
time=2026-10-04T16:11:14.271+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=24.4 upgraded=6 restart-needed=containerd-shim reboot-needed=no"
time=2026-10-04T16:11:50.070+02:00 level=INFO msg="osupdate: DONE" run=20261004T140947Z layer=docker vmid=9201 ring=0 trigger=debug outcome=applied healthy=true reason="" upgraded=6 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=51.5
--- os-update report (guest) ---
{
"authority": "",
"docker_engine": "",
"health_reason": "",
"healthy": true,
"mode": "apply",
"not_covered": [
"docker-ce-cli",
"containerd.io",
"docker-ce",
"docker-buildx-plugin",
"docker-ce-rootless-extras",
"docker-compose-plugin"
],
"outcome": "nothing",
"pending": 6,
"reboot_needed": false,
"refused": null,
"release_id": "ring0-20261004T140947Z",
"restart_needed": [],
"ring": 0,
"run_id": "20261004T140947Z",
"upgraded": [],
"wrapper_seconds": 16.6
}
--- os-update report (host) ---
{
"authority": "",
"docker_engine": "",
"health_reason": "",
"healthy": true,
"mode": "apply",
"not_covered": [
"frr",
"shim-signed-common",
"proxmox-secure-boot-support",
"shim-unsigned",
"shim-helpers-amd64-signed",
"shim-signed",
"amd64-microcode",
"libradosstriper1",
"librgw2",
"ceph-common",
"librbd1",
"librados2",
"python3-cephfs",
"libcephfs2",
"python3-rgw",
"python3-rados",
"python3-ceph-argparse",
"python3-ceph-common",
"python3-rbd",
"ceph-fuse",
"chrony",
"libcorosync-common4",
"libcfg7",
"libcmap4",
"libcpg4",
"libknet1t64",
"libnozzle1t64",
"libquorum5",
"libvotequorum8",
"corosync",
"frr-pythontools",
"libjs-extjs",
"libnvpair3linux",
"libproxmox-acme-plugins",
"libproxmox-backup-qemu0",
"pve-qemu-kvm",
"libpve-notify-perl",
"libpve-cluster-api-perl",
"libpve-cluster-perl",
"pve-cluster",
"libpve-access-control",
"libpve-apiclient-perl",
"librados2-perl",
"proxmox-backup-client",
"proxmox-backup-file-restore",
"pve-manager",
"libproxmox-acme-perl",
"libpve-common-perl",
"libpve-guest-common-perl",
"qemu-server",
"libpve-storage-perl",
"pve-edk2-firmware-legacy",
"pve-edk2-firmware-ovmf",
"libpve-network-api-perl",
"libpve-network-perl",
"proxmox-firewall-data",
"pve-firewall",
"pve-container",
"pve-ha-manager",
"novnc-pve",
"proxmox-enterprise-support-keyring",
"proxmox-mini-journalreader",
"proxmox-widget-toolkit",
"pve-docs",
"pve-i18n",
"pve-xtermjs",
"pve-yew-mobile-i18n",
"pve-yew-mobile-gui",
"libuutil3linux",
"libzfs7linux",
"libzpool7linux",
"proxmox-kernel-helper",
"pve-edk2-firmware-aarch64",
"pve-edk2-firmware",
"pve-firmware",
"zfs-initramfs",
"zfsutils-linux",
"zfs-zed"
],
"outcome": "nothing",
"pending": 78,
"reboot_needed": false,
"refused": null,
"release_id": "ring0-20261004T140947Z",
"restart_needed": [],
"ring": 0,
"run_id": "20261004T140947Z",
"upgraded": [],
"wrapper_seconds": 15
}
--- os-update report (docker) ---
{
"authority": "ring0",
"docker_engine": "29.8.2",
"health_reason": "",
"healthy": true,
"mode": "apply",
"not_covered": null,
"outcome": "applied",
"pending": 0,
"reboot_needed": false,
"refused": null,
"release_id": "ring0-20261004T140947Z",
"restart_needed": [
"containerd-shim"
],
"ring": 0,
"run_id": "20261004T140947Z",
"upgraded": [
{
"name": "docker-ce-cli",
"version": "5:29.8.2-1~debian.13~trixie",
"origin": ""
},
{
"name": "containerd.io",
"version": "2.3.6-1~debian.13~trixie",
"origin": ""
},
{
"name": "docker-ce",
"version": "5:29.8.2-1~debian.13~trixie",
"origin": ""
},
{
"name": "docker-buildx-plugin",
"version": "0.37.1-1~debian.13~trixie",
"origin": ""
},
{
"name": "docker-ce-rootless-extras",
"version": "5:29.8.2-1~debian.13~trixie",
"origin": ""
},
{
"name": "docker-compose-plugin",
"version": "5.6.0-1~debian.13~trixie",
"origin": ""
}
],
"wrapper_seconds": 51.5
}
pass took 2m2.2s
WALL=122.327970106
IDS IDENTICAL (24 containers)
engine=29.8.2
live=true
documentation/audits/os-docker-crash-2026-10-04/partB/b4-ring0-docker-demo-felhom.log
+259
View File
@@ -0,0 +1,259 @@
before: 5 containers, engine=29.7.1, live=false
=== felhom-agent 0.142.0-rc1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true ===
time=2026-10-04T16:12:37.832+02:00 level=INFO msg="osupdate: START" run=20261004T141237Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T141237Z
time=2026-10-04T16:12:50.352+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T141237Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
time=2026-10-04T16:12:50.352+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
time=2026-10-04T16:12:50.352+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
time=2026-10-04T16:12:50.352+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
time=2026-10-04T16:12:50.352+02:00 level=INFO msg="osupdate: DONE" run=20261004T141237Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=10 reboot_needed=true wrapper_seconds=12.5
time=2026-10-04T16:12:50.364+02:00 level=INFO msg="osupdate: START" run=20261004T141237Z layer=host vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T141237Z
time=2026-10-04T16:13:00.773+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T141237Z layer=host lane=fast mode=apply select=pending-fast packages=0"
time=2026-10-04T16:13:00.773+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
time=2026-10-04T16:13:00.773+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
time=2026-10-04T16:13:00.773+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
time=2026-10-04T16:13:01.549+02:00 level=INFO msg="osupdate: DONE" run=20261004T141237Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=80 not_covered=80 restart_needed=34 reboot_needed=true wrapper_seconds=10.4
time=2026-10-04T16:13:08.002+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE reload_rc=0 state=true containers=5 same-ids=yes"
time=2026-10-04T16:13:08.002+02:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"containers_before\": 5, \"reload_rc\": 0, \"result\": \"on\", \"same_ids\": true}"
time=2026-10-04T16:13:08.002+02:00 level=INFO msg="osupdate: START" run=20261004T141237Z layer=docker vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T141237Z
time=2026-10-04T16:13:42.395+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T141237Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0"
time=2026-10-04T16:13:42.395+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
time=2026-10-04T16:13:42.395+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=6 already=0 not-installed=0 from-snapshot=0"
time=2026-10-04T16:13:42.395+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=13.7 upgraded=6 restart-needed=agetty,containerd-shim,cron,dbus-daemon,dhclient,sshd,systemd,systemd-journal,systemd-logind,systemd-network reboot-needed=yes"
time=2026-10-04T16:14:16.982+02:00 level=INFO msg="osupdate: DONE" run=20261004T141237Z layer=docker vmid=9201 ring=0 trigger=debug outcome=applied healthy=true reason="" upgraded=6 pending=0 not_covered=0 restart_needed=10 reboot_needed=true wrapper_seconds=34.3
--- os-update report (guest) ---
{
"authority": "",
"docker_engine": "",
"health_reason": "",
"healthy": true,
"mode": "apply",
"not_covered": [
"docker-ce-cli",
"containerd.io",
"docker-ce",
"docker-buildx-plugin",
"docker-ce-rootless-extras",
"docker-compose-plugin"
],
"outcome": "nothing",
"pending": 6,
"reboot_needed": true,
"refused": null,
"release_id": "ring0-20261004T141237Z",
"restart_needed": [
"agetty",
"containerd",
"cron",
"dbus-daemon",
"dhclient",
"sshd",
"systemd",
"systemd-journal",
"systemd-logind",
"systemd-network"
],
"ring": 0,
"run_id": "20261004T141237Z",
"upgraded": [],
"wrapper_seconds": 12.5
}
--- os-update report (host) ---
{
"authority": "",
"docker_engine": "",
"health_reason": "",
"healthy": true,
"mode": "apply",
"not_covered": [
"frr",
"shim-signed-common",
"shim-unsigned",
"shim-helpers-amd64-signed",
"shim-signed",
"libradosstriper1",
"librgw2",
"ceph-common",
"librbd1",
"librados2",
"python3-cephfs",
"libcephfs2",
"python3-rgw",
"python3-rados",
"python3-ceph-argparse",
"python3-ceph-common",
"python3-rbd",
"ceph-fuse",
"chrony",
"libcorosync-common4",
"libcfg7",
"libcmap4",
"libcpg4",
"libknet1t64",
"libnozzle1t64",
"libquorum5",
"libvotequorum8",
"corosync",
"frr-pythontools",
"libjs-extjs",
"libnvpair3linux",
"libproxmox-acme-plugins",
"libproxmox-backup-qemu0",
"pve-qemu-kvm",
"libpve-notify-perl",
"libpve-cluster-api-perl",
"libpve-cluster-perl",
"pve-cluster",
"libpve-access-control",
"libpve-apiclient-perl",
"librados2-perl",
"proxmox-backup-client",
"proxmox-backup-file-restore",
"pve-manager",
"libproxmox-acme-perl",
"libpve-common-perl",
"libpve-guest-common-perl",
"qemu-server",
"libpve-storage-perl",
"pve-edk2-firmware-legacy",
"pve-edk2-firmware-ovmf",
"libpve-network-api-perl",
"libpve-network-perl",
"proxmox-firewall-data",
"pve-firewall",
"pve-container",
"pve-ha-manager",
"novnc-pve",
"proxmox-enterprise-support-keyring",
"proxmox-mini-journalreader",
"proxmox-widget-toolkit",
"pve-docs",
"pve-i18n",
"pve-xtermjs",
"pve-yew-mobile-i18n",
"pve-yew-mobile-gui",
"libuutil3linux",
"libzfs7linux",
"libzpool7linux",
"proxmox-first-boot",
"pve-firmware",
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0",
"proxmox-kernel-helper",
"pve-edk2-firmware-aarch64",
"pve-edk2-firmware",
"zfs-initramfs",
"zfsutils-linux",
"zfs-zed",
"tailscale"
],
"outcome": "nothing",
"pending": 80,
"reboot_needed": true,
"refused": null,
"release_id": "ring0-20261004T141237Z",
"restart_needed": [
"agetty",
"blkmapd",
"chronyd",
"cron",
"dbus-daemon",
"dmeventd",
"ksmtuned",
"lxc-monitord",
"lxc-start",
"lxcfs",
"pmxcfs",
"proxmox-firewal",
"pve-firewall",
"pve-ha-crm",
"pve-ha-lrm",
"pve-lxc-syscall",
"pvedaemon",
"pvedaemon worke",
"pvefw-logger",
"pveproxy",
"pveproxy worker",
"pvescheduler",
"pvestatd",
"qmeventd",
"rpcbind",
"rrdcached",
"smartd",
"spiceproxy",
"spiceproxy work",
"sshd",
"systemd-logind",
"systemd-udevd",
"watchdog-mux",
"zed"
],
"ring": 0,
"run_id": "20261004T141237Z",
"upgraded": [],
"wrapper_seconds": 10.4
}
--- os-update report (docker) ---
{
"authority": "ring0",
"docker_engine": "29.8.2",
"health_reason": "",
"healthy": true,
"mode": "apply",
"not_covered": null,
"outcome": "applied",
"pending": 0,
"reboot_needed": true,
"refused": null,
"release_id": "ring0-20261004T141237Z",
"restart_needed": [
"agetty",
"containerd-shim",
"cron",
"dbus-daemon",
"dhclient",
"sshd",
"systemd",
"systemd-journal",
"systemd-logind",
"systemd-network"
],
"ring": 0,
"run_id": "20261004T141237Z",
"upgraded": [
{
"name": "docker-ce-cli",
"version": "5:29.8.2-1~debian.13~trixie",
"origin": ""
},
{
"name": "containerd.io",
"version": "2.3.6-1~debian.13~trixie",
"origin": ""
},
{
"name": "docker-ce",
"version": "5:29.8.2-1~debian.13~trixie",
"origin": ""
},
{
"name": "docker-buildx-plugin",
"version": "0.37.1-1~debian.13~trixie",
"origin": ""
},
{
"name": "docker-ce-rootless-extras",
"version": "5:29.8.2-1~debian.13~trixie",
"origin": ""
},
{
"name": "docker-compose-plugin",
"version": "5.6.0-1~debian.13~trixie",
"origin": ""
}
],
"wrapper_seconds": 34.3
}
pass took 1m39.2s
WALL=99.343682135
IDS IDENTICAL (5 containers)
after: engine=29.8.2 live=true
documentation/audits/os-docker-crash-2026-10-04/partC/c0-readonly-demo-hp.txt
+53
View File
@@ -0,0 +1,53 @@
+ sysctl kernel.panic kernel.panic_on_oops kernel.sysrq kernel.panic_on_warn kernel.softlockup_panic kernel.hung_task_panic
kernel.panic = 0
kernel.panic_on_oops = 0
kernel.sysrq = 438
kernel.panic_on_warn = 0
kernel.softlockup_panic = 0
kernel.hung_task_panic = 0
+ ls -la /sys/fs/pstore/ /var/lib/systemd/pstore/
/sys/fs/pstore/:
total 0
drwxr-x--- 2 root root 0 Oct 4 14:36 .
drwxr-xr-x 10 root root 0 Oct 4 14:36 ..
/var/lib/systemd/pstore/:
total 8
drwxr-xr-x 2 root root 4096 May 21 13:32 .
drwxr-xr-x 12 root root 4096 Aug 21 17:44 ..
+ mount
+ grep pstore
none on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
+ cat /sys/module/pstore/parameters/backend
efi_pstore
+ dmesg
+ grep -iE 'pstore|efi_pstore|ramoops|erst'
+ head
[ 4.129612] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
[ 4.153714] pstore: Using crash dump compression: deflate
[ 4.206148] pstore: Registered efi_pstore as persistent store backend
[ 4.207174] systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
[ 4.207514] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
[ 4.210265] systemd[1]: systemd-pstore.service - Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore
+ grep -E '^#?Storage' /etc/systemd/journald.conf
#Storage=auto
+ ls /var/log/journal
+ head -2
9756a908fef046fda6a56428d3fefe3c
+ journalctl --list-boots --no-pager
+ tail -4
-3 9cfa86adcf9f4dec99e324f5e2645032 Sun 2026-10-04 09:44:31 CEST Sun 2026-10-04 09:45:46 CEST
-2 1407ed1ad3ba4a37ac84ddae7cb05b3b Sun 2026-10-04 09:46:18 CEST Sun 2026-10-04 14:26:34 CEST
-1 ffd2f9aa02a344399338755a7958219f Sun 2026-10-04 14:27:08 CEST Sun 2026-10-04 14:35:43 CEST
0 7e320e5daf1f4ee79760c2de1aaf39ff Sun 2026-10-04 14:36:16 CEST Sun 2026-10-04 15:26:56 CEST
+ systemctl is-enabled systemd-pstore.service
enabled
+ lsmod
+ grep -iE 'efi_pstore|ramoops'
efi_pstore 12288 0
+ ls /sys/firmware/efi/efivars
+ grep -ci dump
0
+ which kdump-config
+ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet
documentation/audits/os-docker-crash-2026-10-04/partC/c1-crash1.txt
+85
View File
@@ -0,0 +1,85 @@
== before crash 1, 2026-10-04T13:38:26Z
kernel.panic = 10
7e320e5d-af1f-4ee7-9760-c2de1aaf39ff
7.0.14-20-pve
2026-10-04 14:36:11
crash issued 2026-10-04T13:38:27Z
== after crash 1, 2026-10-04T13:39:27Z
+ uname -r
7.0.14-20-pve
+ uptime -s
2026-10-04 15:39:06
+ cat /proc/sys/kernel/random/boot_id
0defc390-c295-4e9f-aa6e-13d69a1f24c3
+ sysctl kernel.panic
kernel.panic = 0
+ ls -la /sys/fs/pstore/ /var/lib/systemd/pstore/
/sys/fs/pstore/:
total 0
drwxr-x--- 2 root root 0 Oct 4 15:39 .
drwxr-xr-x 10 root root 0 Oct 4 15:39 ..
/var/lib/systemd/pstore/:
total 8
drwxr-xr-x 2 root root 4096 May 21 13:32 .
drwxr-xr-x 12 root root 4096 Aug 21 17:44 ..
+ find /var/lib/systemd/pstore -type f
+ head
++ find /var/lib/systemd/pstore -type f
++ head -2
+ journalctl -b -1 --no-pager -n 8 -o short-iso
2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Listening on gpg-agent.socket - GnuPG cryptographic agent and passphrase cache.
2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Listening on ssh-agent.socket - OpenSSH Agent socket.
2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Reached target sockets.target - Sockets.
2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Reached target basic.target - Basic System.
2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Reached target default.target - Main User Target.
2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Startup finished in 310ms.
2026-10-04T15:38:27+02:00 demo-hp systemd[1]: Started user@0.service - User Manager for UID 0.
2026-10-04T15:38:27+02:00 demo-hp systemd[1]: Started session-15.scope - Session 15 of User root.
+ journalctl -b -2 --no-pager -n 4 -o short-iso
2026-10-04T14:35:43+02:00 demo-hp systemd-shutdown[1]: Syncing filesystems and block devices.
2026-10-04T14:35:43+02:00 demo-hp systemd-shutdown[1]: Sending SIGTERM to remaining processes...
2026-10-04T14:35:43+02:00 demo-hp systemd-journald[349]: Received SIGTERM from PID 1 (systemd-shutdow).
2026-10-04T14:35:43+02:00 demo-hp systemd-journald[349]: Journal stopped
+ journalctl -b 0 -u systemd-pstore --no-pager -o cat
+ head
+ last -x
+ head -5
reboot system boot 7.0.14-20-pve Sun Oct 4 15:39 - still running
reboot system boot 7.0.14-20-pve Sun Oct 4 14:36 - crash
reboot system boot 7.0.14-20-pve Sun Oct 4 14:27 - 14:35 (00:08)
shutdown system down 7.0.14-20-pve Sun Oct 4 14:35 - 14:36 (00:00)
reboot system boot 7.0.2-6-pve Sun Oct 4 09:46 - 14:26 (04:39)
+ cat /sys/module/efi_pstore/parameters/pstore_disable
N
+ cat /sys/module/pstore/parameters/max_reason
cat: /sys/module/pstore/parameters/max_reason: No such file or directory
+ cat /sys/module/pstore/parameters/kmsg_bytes
10240
++ uname -r
+ grep -E 'EFI_VARS_PSTORE|PSTORE_DEFAULT|CONFIG_PSTORE=' /boot/config-7.0.14-20-pve
CONFIG_EFI_VARS_PSTORE=m
# CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE is not set
CONFIG_PSTORE=y
CONFIG_PSTORE_DEFAULT_KMSG_BYTES=10240
+ cat /sys/module/printk/parameters/always_kmsg_dump
N
+ journalctl -b 0 -u felhom-agent --no-pager -o short-iso
+ grep -iE 'started|report|hub:'
+ head -5
+ cut -c1-200
2026-10-04T15:39:22+02:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-10-04T15:39:23+02:00 demo-hp sudo[1492]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
2026-10-04T15:39:24+02:00 demo-hp felhom-agent[1304]: time=2026-10-04T15:39:24.149+02:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
2026-10-04T15:39:24+02:00 demo-hp felhom-agent[1304]: time=2026-10-04T15:39:24.149+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=
2026-10-04T15:39:24+02:00 demo-hp sudo[1585]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
+ pct status 9201
status: running
+ journalctl -k -b -2 --no-pager
+ grep -ciE 'oops|BUG:'
0
+ journalctl -k --no-pager
+ grep -iE 'Oops|BUG:|Call Trace'
+ head -3
+ cat /proc/sys/kernel/tainted
4097
documentation/audits/os-docker-crash-2026-10-04/partD/copied-by-hand-demo-felhom.txt
+10
View File
@@ -0,0 +1,10 @@
755 root 41c759e4c195d349 /usr/local/sbin/felhom-os-apply
755 root f45b99817fe8ff50 /usr/local/sbin/felhom-crash-guard
644 root 5580ad6cdd51990b /etc/systemd/system/felhom-crash-guard.service
644 root 93d2c4749fe63ab1 /etc/systemd/system/felhom-crash-guard-check.service
644 root b214e79e483c339f /etc/systemd/system/felhom-crash-guard-check.timer
644 root 9b9d305b421f81d2 /etc/felhom/crash-guard.conf
644 root d17cb8e4456575b1 /etc/felhom/operator-signers
644 root 338a564cc8392bad /etc/felhom/os-trust.json
755 root 18acdf69f9c915be /usr/local/bin/felhom-agent-0.142.0-rc1
kernel.panic = 10
documentation/audits/os-docker-crash-2026-10-04/partD/copied-by-hand-demo-hp.txt
+33
View File
@@ -0,0 +1,33 @@
Created symlink '/etc/systemd/system/sysinit.target.wants/felhom-crash-guard.service' -> '/etc/systemd/system/felhom-crash-guard.service'.
Created symlink '/etc/systemd/system/timers.target.wants/felhom-crash-guard-check.timer' -> '/etc/systemd/system/felhom-crash-guard-check.timer'.
755 root 41c759e4c195d349 /usr/local/sbin/felhom-os-apply
755 root f45b99817fe8ff50 /usr/local/sbin/felhom-crash-guard
644 root 5580ad6cdd51990b /etc/systemd/system/felhom-crash-guard.service
644 root 93d2c4749fe63ab1 /etc/systemd/system/felhom-crash-guard-check.service
644 root b214e79e483c339f /etc/systemd/system/felhom-crash-guard-check.timer
644 root 9b9d305b421f81d2 /etc/felhom/crash-guard.conf
644 root d17cb8e4456575b1 /etc/felhom/operator-signers
644 root af0890ef7a5778d0 /etc/felhom/os-trust.json
755 root 18acdf69f9c915be /usr/local/bin/felhom-agent-0.142.0-rc1
active
active
kernel.panic = 10
{
"armed": true,
"boot_id": "0defc390-c295-4e9f-aa6e-13d69a1f24c3",
"config": {
"LIMIT": 3,
"PANIC_SECONDS": 10,
"REARM_HOURS": 24,
"WINDOW_MINUTES": 60
},
"kernel_panic": 10,
"last_boot_at": "2026-10-04T13:39:06Z",
"last_boot_unclean": false,
"tripped": false,
"unclean_boots": [],
"unclean_boots_24h": 0,
"unclean_boots_in_window": 0,
"updated_at": "2026-10-04T14:08:23Z",
"version": 1
}