kernel clean-up spike: the kept old kernel boots demo-hp to healthy, and the box goes back (R-1368)
The one-shot flag booted 7.0.14-26-pve: every health point good after 313 s. An orderly restart brought 7.0.14-28-pve back: 324 s. The pinned default, grub.cfg and the step's state file did not change. No kernel report and no alarm at the hub. Evidence: evidence/fallback-boot/. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
e83a78ce61
commit
5f408caca8
12 files changed
+408
-7
No files matched your search
@@ -635,9 +635,10 @@ until a later step ends `good`.
|
||||
|
||||
**Proven** on demo-hp by hand: 8 kernels → 2, 6.4 GB gained on the system disk, the default, the pinned file and both
|
||||
generators unchanged, five refusals by simulation, one orderly reboot back onto the same kernel with a clean stop for
|
||||
the crash guard (§5.9), a second run that removed nothing. **Not proven:** a box whose running kernel is not its
|
||||
newest; a boot of the kept old kernel after the clean-up; the step as a program at night; a box that boots with
|
||||
`proxmox-boot-tool`. Evidence, the rule as a read-only script, and the full "not measured" list:
|
||||
the crash guard (§5.9), a second run that removed nothing. The kept old kernel then booted the box to healthy through
|
||||
the one-shot flag (313 s), and the box went back to its newest kernel (324 s); the pinned file, `grub.cfg` and the
|
||||
step's state file did not change. **Not proven:** a box whose running kernel is not its newest; a self-revert into
|
||||
the kept kernel chosen by the product; the step as a program at night; a box that boots with `proxmox-boot-tool`. Evidence, the rule as a read-only script, and the full "not measured" list:
|
||||
`audits/kernel-cleanup-spike-2026-11-10/`.
|
||||
|
||||
## 6. Risks and edge cases
|
||||
|
||||
@@ -81,13 +81,39 @@ and 42% after.
|
||||
- `felhom-os-apply --kernel-status` needs a plan file in the agent's folder. The run read its inputs directly (the
|
||||
state file, the flag, `grub.cfg`).
|
||||
|
||||
## Measured later the same day: the kept old kernel boots (`evidence/fallback-boot/`)
|
||||
|
||||
**Yes. The kept old kernel (7.0.14-26-pve) booted demo-hp to healthy, and the box went back to its newest kernel.**
|
||||
A second run, 2026-11-10 14:29–14:46 UTC, two orderly restarts.
|
||||
|
||||
| | Old kernel (26) | Back on the newest (28) |
|
||||
|---|---|---|
|
||||
| Restart sent (UTC) | 14:30:24 | 14:40:09 |
|
||||
| SSH answered after | 166 s | 176 s |
|
||||
| Every health point good after | 313 s | 324 s |
|
||||
|
||||
- **Method.** The product's own boot-side switch, set by hand: `grub-editenv /boot/efi/EFI/felhom/oneshot.env set
|
||||
felhom_next=7.0.14-26-pve`, then `systemctl reboot`. GRUB cleared the flag and booted 26 once, with the panic
|
||||
parameters of a trial boot (`softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10`). The second restart
|
||||
had no flag, so the box booted its default, 28. No wrapper verb ran; nothing was built or changed in the product.
|
||||
- **Healthy** means the host health rule of `11-os-updates.md` §8.2, read point by point: the agent and the four
|
||||
Proxmox services active, both guests running, Docker answering, both guests resolving `deb.debian.org`, the
|
||||
controllers healthy, every container back, the tunnel connected. The limit was 15 minutes.
|
||||
- **Nothing was left behind.** The pinned default file, `grub.cfg` and the kernel-step state file have the same sha256
|
||||
before, on the old kernel and after. The step state stayed `good`. The crash guard saw two clean stops.
|
||||
- **The hub.** No kernel-step report came from the test (3 lines for this host before and after, all from the
|
||||
morning's step). No alarm line. Only the "controller started" notes of the two restarts.
|
||||
- **A different channel.** On the old kernel both public dashboards answered through Cloudflare (302). The hub's row
|
||||
for the host said "unknown (no running customer guest)" then: it was the agent's first report after the boot, and no
|
||||
newer report came in the six minutes the box stayed on 26.
|
||||
- **Not covered by this.** A step where the product itself chooses the old kernel (a self-revert after a bad trial
|
||||
boot). Here a person set the flag.
|
||||
|
||||
## Not measured
|
||||
|
||||
- A box whose running kernel is **not** its newest one (after a fall-back or a self-revert). The rule keeps both; no
|
||||
such box was tried.
|
||||
- A real kernel step that has not ended. Control C is a pretended phase.
|
||||
- **A boot of the kept old kernel (26) after the clean-up.** Its files and its menu entries exist and `grub.cfg`
|
||||
passes the syntax check; it was not booted.
|
||||
- The N100 and the Tester 1 VM (8 and 3 kernels). Read-only in this run.
|
||||
- The step as a program, at night, next to the agent's own apt runs. A person checked the lock by hand.
|
||||
- A box that boots with `proxmox-boot-tool` (demo-hp does not use it), and a box with out-of-tree kernel modules
|
||||
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
### BEFORE — 2026-11-10T14:29:24Z (read on demo-hp, read only)
|
||||
uname -r: 7.0.14-28-pve
|
||||
boot_id: d1167d94-31bf-459a-bf2a-569e8997a569
|
||||
up since (UTC): 2026-11-10T12:57:00Z
|
||||
cmdline: BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet
|
||||
== one-shot flag (grub-editenv /boot/efi/EFI/felhom/oneshot.env list)
|
||||
felhom_next=
|
||||
== pinned default file
|
||||
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
|
||||
GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
|
||||
== grub.cfg
|
||||
805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg
|
||||
22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
|
||||
== kernel step state
|
||||
607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json
|
||||
{'phase': 'good', 'from': '7.0.14-26-pve', 'to': '7.0.14-28-pve', 'step_id': 'ring0-20261110T014802Z', 'result_at': '2026-11-10T01:56:47Z', 'updated_at': '2026-11-10T01:56:47Z'}
|
||||
== kernels installed
|
||||
/boot/vmlinuz-7.0.14-26-pve
|
||||
/boot/vmlinuz-7.0.14-28-pve
|
||||
== hand pin files
|
||||
ls: cannot access '/etc/kernel/proxmox-boot-pin': No such file or directory
|
||||
ls: cannot access '/etc/kernel/next-boot-pin': No such file or directory
|
||||
== crash guard
|
||||
{'armed': True, 'tripped': False, 'boot_id': 'd1167d94-31bf-459a-bf2a-569e8997a569', 'last_boot_at': '2026-11-10T12:56:59Z', 'last_boot_unclean': False, 'unclean_boots': [], 'unclean_boots_in_window': 0, 'unclean_boots_24h': 0, 'updated_at': '2026-11-10T14:12:09Z'}
|
||||
== host services
|
||||
felhom-agent: active
|
||||
pve-cluster: active
|
||||
pvedaemon: active
|
||||
pveproxy: active
|
||||
pvestatd: active
|
||||
failed units: 0
|
||||
== tunnel
|
||||
handshake age s: 87
|
||||
== guests
|
||||
VMID Status Lock Name
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
9401 stopped upgrade-harness
|
||||
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
|
||||
341 night1004-tester1 running 8192 200.00 1847
|
||||
== running jobs (vzdump, restore, apt, dpkg, backup client, wrapper)
|
||||
321567 00:00 /usr/bin/proxmox-backup-client status --output-format=json --repository felhom@pbs!demo-hp@10.77.0.1:felhom-offsite
|
||||
(end)
|
||||
== containers in 9201
|
||||
adventurelog-frontend Up 2 hours (healthy)
|
||||
adventurelog-postgres Up 2 hours (healthy)
|
||||
adventurelog Up 2 hours (healthy)
|
||||
bentopdf Up 2 hours (healthy)
|
||||
bookstack-db Up 2 hours (healthy)
|
||||
bookstack Up 2 hours (healthy)
|
||||
calibre-web Up 2 hours (healthy)
|
||||
cloudflared Up 2 hours (healthy)
|
||||
docmost-postgres Up 2 hours (healthy)
|
||||
docmost-redis Up 2 hours (healthy)
|
||||
docmost Up 2 hours (healthy)
|
||||
felhom-controller Up 2 hours (healthy)
|
||||
filebrowser Up 2 hours (healthy)
|
||||
home-assistant Up 2 hours (healthy)
|
||||
kimai-db Up 2 hours (healthy)
|
||||
kimai Up 2 hours (healthy)
|
||||
opengist Up 2 hours (healthy)
|
||||
paperless-postgres Up 2 hours (healthy)
|
||||
paperless-redis Up 2 hours (healthy)
|
||||
paperless-webserver Up 2 hours (healthy)
|
||||
plex Up 2 hours (healthy)
|
||||
privatebin Up 2 hours (healthy)
|
||||
romm-db Up 2 hours (healthy)
|
||||
romm-redis Up 2 hours (healthy)
|
||||
romm Up 2 hours (healthy)
|
||||
traefik Up 2 hours
|
||||
count: 26 running; not healthy: 1
|
||||
== containers in 9202
|
||||
felhom-controller Up 2 hours (healthy)
|
||||
filebrowser Up 2 hours (healthy)
|
||||
paperless-postgres Up 2 hours (healthy)
|
||||
paperless-redis Up 2 hours (healthy)
|
||||
paperless-webserver Up 2 hours (healthy)
|
||||
traefik Up 2 hours
|
||||
count: 6 running; not healthy: 1
|
||||
== Tester 1 VM 341
|
||||
status: running
|
||||
guest agent: no answer (rc 255)
|
||||
== disk
|
||||
/dev/mapper/pve-root 39G 17G 21G 45% /
|
||||
== agent log, kernel lines since boot
|
||||
Nov 10 13:57:15 demo-hp sudo[1275]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T125715Z-kernel-kernel-boot.json
|
||||
(end)
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
### before — 2026-11-10T14:29:35Z — hub, read only: GET /hosts -> 200, GET /os/fleet -> 403
|
||||
== /hosts rows
|
||||
demo-felhom-8363b5 Demo Ügyfél 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 5% 23% 38% running (connected) 38% local
|
||||
demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 11% 38% 42% running (connected) 71% local-lvm
|
||||
tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 1/2 0% 38% 34% running (connected) 34% local
|
||||
== /os/fleet (kernel parts)
|
||||
not JSON: Expecting value: line 1 column 1 (char 0)
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
### Boot the kept older kernel ONCE — the product's own boot flag, set by hand (no wrapper verb, no step state)
|
||||
# command on demo-hp: grub-editenv /boot/efi/EFI/felhom/oneshot.env set felhom_next=7.0.14-26-pve ; then: systemctl reboot
|
||||
set rc=0
|
||||
read back: felhom_next=7.0.14-26-pve
|
||||
607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json
|
||||
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
|
||||
805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg
|
||||
reboot sent at 2026-11-10T14:30:24Z (T0=1794321024), old boot_id d1167d94-31bf-459a-bf2a-569e8997a569
|
||||
+129s ping DOWN
|
||||
+166s ssh answers, NEW boot_id=e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf, uname -r=7.0.14-26-pve
|
||||
+171s 9201 n=0 healthy=0 other=0 | 9202 n=0 healthy=0 other=0 | vm341=stopped | services-not-active=0 | 9201=stopped 9202=stopped
|
||||
+191s 9201 n=26 healthy=0 other=25 | 9202 n=6 healthy=3 other=2 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+210s 9201 n=26 healthy=15 other=10 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+228s 9201 n=26 healthy=15 other=10 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+246s 9201 n=26 healthy=20 other=5 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+264s 9201 n=26 healthy=22 other=3 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+282s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+297s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+313s 9201 n=26 healthy=25 other=0 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
WATCH RESULT=ok kernel=7.0.14-26-pve want=7.0.14-26-pve ping-down=+129s ssh=+166s all-healthy=+313s
|
||||
+98
@@ -0,0 +1,98 @@
|
||||
### ON-OLD-KERNEL — 2026-11-10T14:35:44Z (read on demo-hp, read only)
|
||||
uname -r: 7.0.14-26-pve
|
||||
boot_id: e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf
|
||||
up since (UTC): 2026-11-10T14:32:55Z
|
||||
cmdline: BOOT_IMAGE=/boot/vmlinuz-7.0.14-26-pve root=/dev/mapper/pve-root ro quiet softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10
|
||||
== one-shot flag (grub-editenv /boot/efi/EFI/felhom/oneshot.env list)
|
||||
felhom_next=
|
||||
== pinned default file
|
||||
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
|
||||
GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
|
||||
== grub.cfg
|
||||
805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg
|
||||
22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
|
||||
== kernel step state
|
||||
607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json
|
||||
{'phase': 'good', 'from': '7.0.14-26-pve', 'to': '7.0.14-28-pve', 'step_id': 'ring0-20261110T014802Z', 'result_at': '2026-11-10T01:56:47Z', 'updated_at': '2026-11-10T01:56:47Z'}
|
||||
== kernels installed
|
||||
/boot/vmlinuz-7.0.14-26-pve
|
||||
/boot/vmlinuz-7.0.14-28-pve
|
||||
== hand pin files
|
||||
ls: cannot access '/etc/kernel/proxmox-boot-pin': No such file or directory
|
||||
ls: cannot access '/etc/kernel/next-boot-pin': No such file or directory
|
||||
== crash guard
|
||||
{'armed': True, 'tripped': False, 'boot_id': 'e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf', 'last_boot_at': '2026-11-10T14:32:54Z', 'last_boot_unclean': False, 'unclean_boots': [], 'unclean_boots_in_window': 0, 'unclean_boots_24h': 0, 'updated_at': '2026-11-10T14:33:04Z'}
|
||||
== host services
|
||||
felhom-agent: active
|
||||
pve-cluster: active
|
||||
pvedaemon: active
|
||||
pveproxy: active
|
||||
pvestatd: active
|
||||
failed units: 0
|
||||
== tunnel
|
||||
handshake age s: 32
|
||||
== guests
|
||||
VMID Status Lock Name
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
9401 stopped upgrade-harness
|
||||
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
|
||||
341 night1004-tester1 running 8192 200.00 1871
|
||||
== running jobs (vzdump, restore, apt, dpkg, backup client, wrapper)
|
||||
(end)
|
||||
== containers in 9201
|
||||
adventurelog-frontend Up 2 minutes (healthy)
|
||||
adventurelog-postgres Up 2 minutes (healthy)
|
||||
adventurelog Up 2 minutes (healthy)
|
||||
bentopdf Up 2 minutes (healthy)
|
||||
bookstack-db Up 2 minutes (healthy)
|
||||
bookstack Up 2 minutes (healthy)
|
||||
calibre-web Up 2 minutes (healthy)
|
||||
cloudflared Up 2 minutes (healthy)
|
||||
docmost-postgres Up 2 minutes (healthy)
|
||||
docmost-redis Up 2 minutes (healthy)
|
||||
docmost Up 2 minutes (healthy)
|
||||
felhom-controller Up 2 minutes (healthy)
|
||||
filebrowser Up 2 minutes (healthy)
|
||||
home-assistant Up 2 minutes (healthy)
|
||||
kimai-db Up 2 minutes (healthy)
|
||||
kimai Up 2 minutes (healthy)
|
||||
opengist Up 2 minutes (healthy)
|
||||
paperless-postgres Up 2 minutes (healthy)
|
||||
paperless-redis Up 2 minutes (healthy)
|
||||
paperless-webserver Up About a minute (healthy)
|
||||
plex Up About a minute (healthy)
|
||||
privatebin Up 2 minutes (healthy)
|
||||
romm-db Up About a minute (healthy)
|
||||
romm-redis Up About a minute (healthy)
|
||||
romm Up About a minute (healthy)
|
||||
traefik Up 2 minutes
|
||||
count: 26 running; not healthy: 1
|
||||
== containers in 9202
|
||||
felhom-controller Up 2 minutes (healthy)
|
||||
filebrowser Up 2 minutes (healthy)
|
||||
paperless-postgres Up 2 minutes (healthy)
|
||||
paperless-redis Up 2 minutes (healthy)
|
||||
paperless-webserver Up 2 minutes (healthy)
|
||||
traefik Up 2 minutes
|
||||
count: 6 running; not healthy: 1
|
||||
== Tester 1 VM 341
|
||||
status: running
|
||||
guest agent: no answer (rc 255)
|
||||
== disk
|
||||
/dev/mapper/pve-root 39G 17G 21G 45% /
|
||||
== agent log, kernel lines since boot
|
||||
Nov 10 15:33:10 demo-hp sudo[1318]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T143309Z-kernel-kernel-boot.json
|
||||
(end)
|
||||
== host health rule, the two points snap.sh does not read (2026-11-10T14:36:21Z)
|
||||
uname -r: 7.0.14-26-pve
|
||||
-- 9201
|
||||
2a04:4e42:400::644 debian.map.fastlydns.net deb.debian.org
|
||||
felhom-controller Up 2 minutes (healthy)
|
||||
cloudflared Up 2 minutes (healthy)
|
||||
-- 9202
|
||||
2a04:4e42:400::644 debian.map.fastlydns.net deb.debian.org
|
||||
felhom-controller Up 3 minutes (healthy)
|
||||
== hub row
|
||||
demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm
|
||||
tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
== hub snapshot on the old kernel 2026-11-10T14:36:01Z
|
||||
### ON-OLD-KERNEL — 2026-11-10T14:36:01Z — hub, read only: GET /hosts -> 200, GET /os/fleet -> 403
|
||||
== /hosts rows
|
||||
demo-felhom-8363b5 Demo Ügyfél 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 5% 23% 38% running (connected) 38% local
|
||||
demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm
|
||||
tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local
|
||||
== /os/fleet (kernel parts)
|
||||
not JSON: Expecting value: line 1 column 1 (char 0)
|
||||
== hub pod kernel lines (channel 2; 3 before the test)
|
||||
3
|
||||
2026/11/10 02:50:50 [INFO] osupdates: demo-hp-bb76ea reported kernel run 20261110T014802Z: ring=0 mode=apply outcome=staged healthy=true upgraded=2 pending=0 not-covered=0 restart-needed=0 reboot-need
|
||||
2026/11/10 02:54:08 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-boot outcome=judging healthy=false upgraded=0 pending=0 not-covered=0 restart-needed=
|
||||
2026/11/10 02:56:47 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-good outcome=applied healthy=true upgraded=0 pending=0 not-covered=0 restart-needed=0
|
||||
== hub row again 2026-11-10T14:36:50Z
|
||||
demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm
|
||||
tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local
|
||||
== hub row again 2026-11-10T14:37:08Z
|
||||
demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm
|
||||
tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local
|
||||
NO MATCH by 2026-11-10T14:39:31Z: demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm
|
||||
== from outside, through Cloudflare (2026-11-10T14:39:45Z); the hub row had not taken a new report yet (cadence up to 15 min)
|
||||
https://felhom.enkisfelhom.hu/ -> 302
|
||||
https://felhom.enkicsifelhom.hu/ -> 302
|
||||
Tester 1 host by SSH: 7.0.14-22-pve 9201 running tester-1 ;9
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
### BOOT BACK — demo-hp, one orderly reboot, no flag set
|
||||
uname -r: 7.0.14-26-pve
|
||||
boot_id: e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf
|
||||
flag: felhom_next=
|
||||
jobs: 0
|
||||
locks: 1
|
||||
607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json
|
||||
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
|
||||
805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg
|
||||
note: 'locks: 1' above counted the header line of 'pct list' (column 'Lock'); no guest config holds a lock line (checked /etc/pve/lxc/*.conf and qemu-server/*.conf), no job process runs
|
||||
reboot sent at 2026-11-10T14:40:09Z (T0=1794321609), old boot_id e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf
|
||||
+139s ping DOWN
|
||||
+176s ssh answers, NEW boot_id=bed106f7-9e80-4eb1-ace3-14a26da7f049, uname -r=7.0.14-28-pve
|
||||
+181s 9201 n=0 healthy=0 other=0 | 9202 n=0 healthy=0 other=0 | vm341=stopped | services-not-active=0 | 9201=stopped 9202=stopped
|
||||
+201s 9201 n=26 healthy=0 other=25 | 9202 n=6 healthy=3 other=2 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+220s 9201 n=26 healthy=14 other=11 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+239s 9201 n=26 healthy=15 other=10 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+256s 9201 n=26 healthy=20 other=5 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+275s 9201 n=26 healthy=22 other=3 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+293s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+309s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
+324s 9201 n=26 healthy=25 other=0 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
|
||||
WATCH RESULT=ok kernel=7.0.14-28-pve want=7.0.14-28-pve ping-down=+139s ssh=+176s all-healthy=+324s
|
||||
+91
@@ -0,0 +1,91 @@
|
||||
### AFTER — 2026-11-10T14:45:36Z (read on demo-hp, read only)
|
||||
uname -r: 7.0.14-28-pve
|
||||
boot_id: bed106f7-9e80-4eb1-ace3-14a26da7f049
|
||||
up since (UTC): 2026-11-10T14:42:50Z
|
||||
cmdline: BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet
|
||||
== one-shot flag (grub-editenv /boot/efi/EFI/felhom/oneshot.env list)
|
||||
felhom_next=
|
||||
== pinned default file
|
||||
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
|
||||
GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
|
||||
== grub.cfg
|
||||
805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg
|
||||
22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
|
||||
== kernel step state
|
||||
607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json
|
||||
{'phase': 'good', 'from': '7.0.14-26-pve', 'to': '7.0.14-28-pve', 'step_id': 'ring0-20261110T014802Z', 'result_at': '2026-11-10T01:56:47Z', 'updated_at': '2026-11-10T01:56:47Z'}
|
||||
== kernels installed
|
||||
/boot/vmlinuz-7.0.14-26-pve
|
||||
/boot/vmlinuz-7.0.14-28-pve
|
||||
== hand pin files
|
||||
ls: cannot access '/etc/kernel/proxmox-boot-pin': No such file or directory
|
||||
ls: cannot access '/etc/kernel/next-boot-pin': No such file or directory
|
||||
== crash guard
|
||||
{'armed': True, 'tripped': False, 'boot_id': 'bed106f7-9e80-4eb1-ace3-14a26da7f049', 'last_boot_at': '2026-11-10T14:42:49Z', 'last_boot_unclean': False, 'unclean_boots': [], 'unclean_boots_in_window': 0, 'unclean_boots_24h': 0, 'updated_at': '2026-11-10T14:42:59Z'}
|
||||
== host services
|
||||
felhom-agent: active
|
||||
pve-cluster: active
|
||||
pvedaemon: active
|
||||
pveproxy: active
|
||||
pvestatd: active
|
||||
failed units: 0
|
||||
== tunnel
|
||||
handshake age s: 30
|
||||
== guests
|
||||
VMID Status Lock Name
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
9401 stopped upgrade-harness
|
||||
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
|
||||
341 night1004-tester1 running 8192 200.00 1950
|
||||
== running jobs (vzdump, restore, apt, dpkg, backup client, wrapper)
|
||||
(end)
|
||||
== containers in 9201
|
||||
adventurelog-frontend Up 2 minutes (healthy)
|
||||
adventurelog-postgres Up 2 minutes (healthy)
|
||||
adventurelog Up 2 minutes (healthy)
|
||||
bentopdf Up 2 minutes (healthy)
|
||||
bookstack-db Up 2 minutes (healthy)
|
||||
bookstack Up 2 minutes (healthy)
|
||||
calibre-web Up 2 minutes (healthy)
|
||||
cloudflared Up 2 minutes (healthy)
|
||||
docmost-postgres Up 2 minutes (healthy)
|
||||
docmost-redis Up 2 minutes (healthy)
|
||||
docmost Up 2 minutes (healthy)
|
||||
felhom-controller Up 2 minutes (healthy)
|
||||
filebrowser Up 2 minutes (healthy)
|
||||
home-assistant Up 2 minutes (healthy)
|
||||
kimai-db Up 2 minutes (healthy)
|
||||
kimai Up 2 minutes (healthy)
|
||||
opengist Up 2 minutes (healthy)
|
||||
paperless-postgres Up About a minute (healthy)
|
||||
paperless-redis Up 2 minutes (healthy)
|
||||
paperless-webserver Up About a minute (healthy)
|
||||
plex Up About a minute (healthy)
|
||||
privatebin Up 2 minutes (healthy)
|
||||
romm-db Up About a minute (healthy)
|
||||
romm-redis Up About a minute (healthy)
|
||||
romm Up About a minute (healthy)
|
||||
traefik Up 2 minutes
|
||||
count: 26 running; not healthy: 1
|
||||
== containers in 9202
|
||||
felhom-controller Up 2 minutes (healthy)
|
||||
filebrowser Up 2 minutes (healthy)
|
||||
paperless-postgres Up 2 minutes (healthy)
|
||||
paperless-redis Up 2 minutes (healthy)
|
||||
paperless-webserver Up 2 minutes (healthy)
|
||||
traefik Up 2 minutes
|
||||
count: 6 running; not healthy: 1
|
||||
== Tester 1 VM 341
|
||||
status: running
|
||||
guest agent: no answer (rc 255)
|
||||
== disk
|
||||
/dev/mapper/pve-root 39G 17G 21G 45% /
|
||||
== agent log, kernel lines since boot
|
||||
Nov 10 15:43:05 demo-hp sudo[1323]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T144304Z-kernel-kernel-boot.json
|
||||
(end)
|
||||
2a04:4e42:200::644 debian.map.fastlydns.net deb.debian.org
|
||||
felhom-controller Up 2 minutes (healthy)
|
||||
cloudflared Up 2 minutes (healthy)
|
||||
2a04:4e42::644 debian.map.fastlydns.net deb.debian.org
|
||||
felhom-controller Up 2 minutes (healthy)
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
### HUB AFTER — 2026-11-10T14:46:02Z
|
||||
== hub pod kernel lines for demo-hp, last 14 h (3 before the test; the pod log prints CET)
|
||||
3
|
||||
2026/11/10 02:50:50 [INFO] osupdates: demo-hp-bb76ea reported kernel run 20261110T014802Z: ring=0 mode=apply outcome=staged healthy=true upgraded=2 pending=0 no
|
||||
2026/11/10 02:54:08 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-boot outcome=judging healthy=false upgraded=0
|
||||
2026/11/10 02:56:47 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-good outcome=applied healthy=true upgraded=0
|
||||
== positive control: the same log holds lines for demo-hp since the test began (15:29 CET)
|
||||
6
|
||||
== every hub log line since the test began that names an event, an alarm or a mail
|
||||
2026/11/10 15:33:40 [INFO] Event from demo-hp: controller_started (info) — Controller elindult (0.351.0)
|
||||
2026/11/10 15:34:18 [INFO] Notification prefs push for tester-1: household cleared its mail address — deleted
|
||||
2026/11/10 15:34:18 [INFO] Notification preferences updated for tester-1: address set=false, household=true, events=[]
|
||||
2026/11/10 15:34:23 [INFO] Event from tester-1: controller_started (info) — Controller elindult (0.351.0)
|
||||
2026/11/10 15:43:36 [INFO] Event from demo-hp: controller_started (info) — Controller elindult (0.351.0)
|
||||
2026/11/10 15:44:12 [INFO] Notification prefs push for tester-1: household cleared its mail address — deleted
|
||||
2026/11/10 15:44:12 [INFO] Notification preferences updated for tester-1: address set=false, household=true, events=[]
|
||||
2026/11/10 15:44:17 [INFO] Event from tester-1: controller_started (info) — Controller elindult (0.351.0)
|
||||
== 'household cleared its mail address' lines for tester-1, last 30 h (is it the same at every controller start?)
|
||||
2026/11/09 13:09:54 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
|
||||
2026/11/09 13:14:39 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
|
||||
2026/11/10 02:55:11 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
|
||||
2026/11/10 13:58:18 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
|
||||
2026/11/10 15:34:18 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
|
||||
2026/11/10 15:44:12 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
|
||||
@@ -281,7 +281,7 @@ stopping line that lies.
|
||||
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
|
||||
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
|
||||
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
|
||||
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
|
||||
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
|
||||
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
|
||||
| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. **2026-11-10 15:19 — RULED (D75 = one number, 200 GB, with or without a backup drive; `09` §3 decision 295).** This replaces CC's two-step pick. **Open, not ruled:** what the installer does below 200 GB — refuse, or warn. demo-hp (128 GB) stays as a test box. `16` §14.3 carries the ruling. | — | Design first: what the installer does below 200 GB (refuse, or warn) — a later session. Then the installer. | CC |
|
||||
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
|
||||
|
||||
@@ -43,7 +43,7 @@ hub alarms at 90 % and 95 %.
|
||||
|
||||
| What | Step | Mark |
|
||||
|---|---|---|
|
||||
| Old kernels | `ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). **NOT PROVEN:** on a second box; a boot of the kept older kernel |
|
||||
| Old kernels | `ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). The kept older kernel then booted the box to healthy and the box went back (`audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **NOT PROVEN:** on a second box |
|
||||
| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) |
|
||||
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free <volid>`, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only |
|
||||
| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |
|
||||
|
||||
Reference in new issue
Block a user