admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

hub v0.143.0 (code): the kernel lane — the day-before household mail, the night instruction, the operator's kernel set (R-836, decision 172)
gates / gates (push) Successful in 2m47s
Details

Browse Source 
KernelDue / KernelNotify (09-20 h Budapest, one per 20 h, max 3, registered
address, only an accepted mail counts) / os_update.kernel {kver, tonight}
(no mail, no step) / layer kernel ingest + operator events / Approve kernel
set after every ring-0 box booted it healthily after a night stage / two
System page cells. 11 §5.11 written; §5.10 status corrected (proven).
Installer uninstall knows the two GRUB generators (unreleased).
Evidence: audits/kernel-lane-2026-10-07/ (red-proofs, boot timing).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
authentik Default Admin
2026-10-07 15:34:14 +02:00
parent 0c5dbfbfc1
commit ab3b7ea2f4
29 changed files with 1453 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/architecture/11-os-updates.md
+76 -3
View File
@@ -469,7 +469,7 @@ Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-do
---
### 5.10 The Proxmox package lane — BUILT 2026-10-07, unreleased, not yet proven live (R-812 option A, `09` §3 decision 163) `[FACT]`
### 5.10 The Proxmox package lane — BUILT 2026-10-07 (agent v0.151.0, hub v0.142.0), proven on demo-felhom (R-812 option A, `09` §3 decision 163) `[FACT]`
**What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve,
libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on
@@ -498,9 +498,81 @@ libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78
when every ring-0 box ran the set in `DockerNightsEffective` (2) healthy night steps since it was first seen and none
was unhealthy — „healthy" is the box's own verdict above (no memory-kill check; that is the Docker engine's, R-528).
An approval nudges no box: ring 1 takes the set only by a signed `os_pve_step` (signed per box, as `os_docker_step`).
- **Not yet:** the live proof on demo-felhom (ring 0); the undo runbook (by hand: `apt-get install <name>=<old>` —
- **Proven live 2026-10-07** on demo-felhom (ring 0, a signed `os_pve_step`): 65 packages in 70 s, pve-manager
9.2.2 → 9.2.21, healthy, every container kept its id, the hub logged the report (`audits/day-2026-10-07/B/RESULT.md`).
- **Not yet:** the undo runbook (by hand: `apt-get install <name>=<old>` —
Proxmox keeps 30–66 old versions, C2).
### 5.11 The kernel lane — BUILT 2026-10-07 (agent v0.152.0, hub v0.143.0; R-836, `09` §3 decisions 164, 171, 172) `[FACT]`
**The ruling (decision 172):** design option A (a one-shot flag on the ESP) with option C (lockup → panic) on the one-shot
entry. A box may restart at night for a kernel update; **the household is mailed the day before** with what to do if
the box is not back by morning; each kernel set is approved by the operator after ring 0 ran it; **a frozen new kernel
that needs a person's power cycle is accepted** for the first customers. Why A: measured in the spike on the Tester 1
VM, demo-felhom and demo-hp (Secure Boot on) — the ESP flag and UEFI `BootNext` both boot a kernel once and fall back
after a panic; a hardware watchdog armed during the reboot never fired on any of the three
(`audits/kernel-spike-2026-10-07/DESIGN-kernel-lane.md`). A writes nothing to firmware.
**The boot side** (config bundle, two GRUB generators): `/etc/grub.d/01_felhom_oneshot` reads `felhom_next` from a GRUB
env block on the ESP (`EFI/felhom/oneshot.env`), clears and saves it BEFORE the menu, and — only if it names an
installed kernel — boots that kernel's one-shot entry. `/etc/grub.d/42_felhom_oneshot` gives each installed kernel an
entry `felhom-oneshot-<ver>`: the normal entry plus `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10`
(option C; sorted after `10_linux`, so never entry 0 and never the default). A panic on the one-shot restarts the box in
10 s into the default — the old kernel. Without a vfat ESP both print nothing.
**The default** is the kernel the box RUNS, pinned in `/etc/default/grub.d/zz-felhom-kernel-default.cfg` before the
install and proved from grub.cfg after it (an install would otherwise make the newest kernel the default — R-836's
defect). Only a healthy one-shot boot moves it.
**The root wrapper** (`felhom-os-apply`, layer `kernel`, lane slow, an appliance, authority = a signed `os_kernel_step`
or the root-owned ring-0 mark): `apply` STAGES (installs the set — the series meta-package, at most one new kernel
image, the boot helper and firmware — pins the default, writes the flag; never reboots); `kernel-reboot` reboots a
staged step; `kernel-boot` says what became of it after a boot; `kernel-good` moves the default; `kernel-revert`
reboots ONCE into the old kernel; `kernel-cancel`; `kernel-status`. Refusals R20 (the box cannot do a one-shot: not
UEFI, no vfat ESP, a separate /boot, GRUB without fat/loadenv, the generators missing, a hand pin), R21 (the crash
guard tripped or saw an unclean boot within its window), R22 (the step's phase does not allow it; never two steps
within 20 h), R23 (not exactly one newer kernel, or not the one signed / told). State `/var/lib/felhom-kernel/state.json`.
**The night slot** (agent): the kernel step ENDS the night leg — after the host step (and the Proxmox step when one ran)
ended healthy, after the whole-guest backup (the leg runs only after it), under the same heavy-op gate, trigger `night`
only, at most one per night. Ring 0 stages the pending kernel and reboots; ring 1 reboots only a kernel a signed
`os_kernel_step` staged by day (the job never reboots). The hub hears `staged` before the reboot.
**"Boot good" and the self-revert** (agent, at every start): on the new kernel the agent judges the boot for **20
minutes** — the host health rule (§8.2: the Proxmox daemons and the agent active, the customer guest running and its
own rule passing, the tunnel running) AND the box reached the hub (the `judging` report itself). Healthy → the new
kernel becomes the default (`applied`). Not healthy by the deadline → `health_failed`, then ONE self-revert into the old
kernel (`self_reverted` after it boots; a revert that comes back on the new kernel is `revert_failed` and never
retried). The wait, measured 2026-10-07 by a plain reboot (`audits/kernel-lane-2026-10-07/B/`): every container healthy
68 s after the reboot on demo-felhom (the hub reached at 63 s; Tailscale back at 53 s — the spike's > 6 min did not
recur) and 272 s on demo-hp (the hub at 189 s). A box that never comes back raises the hub's existing `host_stale`
(45 min, `alerting.stale_threshold`) and `host_down` (90 min) to the operator.
**The crash guard** (§5.9): the step's reboot and the self-revert are orderly (the clean-stop marker), so a step adds at
most ONE unclean boot (a panic before userspace adds none: the planned reboot's marker is still there); with R21 the
step starts only with none in the window — the box cannot reach the 3rd unclean boot that leaves it off. Pinned by
`KernelStepCannotLeaveTheBoxOff`.
**The household mail** (hub): a box is DUE when ring 0 has a pending kernel (its host report's `proxmox-kernel-X.Y`
upgrade) or a ring-1 box runs a kernel a signed job staged, and no step for that kernel has ended (`applied`,
`fell_back`, `health_failed`, `self_reverted`, `revert_failed`, or a refusal R20/R23/R3/R12 — the operator decides; never
retried by itself). Its household gets ONE mail (`mail.kernel.*`, its language, informal; to the registered address)
between 09:00 and 20:00 Budapest time, at most once per 20 h and three times per kernel. The box's `os_update.kernel`
block says `tonight` only within 24 h of a mail the mail service ACCEPTED — **no mail, no step**. Operator events
`os_kernel_step` and `os_kernel_notice`; the household's timeline gets the usual "security fixes installed" line after
`applied`.
**Approval** (hub): "Approve kernel set" on the System page appears when every ring-0 box's newest ended kernel step is
`applied` for the same kernel, after a NIGHT stage. The approved set is the series meta-package and the signed image. An
approval nudges no box: ring 1 takes it only through a signed `os_kernel_step` (stage), then its own told night.
**The System page** shows per box: the running kernel, the next boot, the default (amber while a one-shot flag names
another kernel), and the kernel step (the newest result, the kernel due, whether the household was told for tonight).
**Not measured:** option C itself — the Proxmox kernel ships no lockup test module (`CONFIG_TEST_LOCKUP` is not set on
7.0.14-20), so a soft lockup turning into a panic is recorded as unmeasured (no tool was built). A true dead freeze
needs a person (accepted, decision 172). Evidence and proofs: `audits/kernel-lane-2026-10-07/`.
## 6. Risks and edge cases
| # | What can go wrong | What the design does |
@@ -580,7 +652,8 @@ Each step returns to the operator for go or no-go.
after enrolment (16:24/16:25 UTC: 49 guest + 106 host packages, 110 s + 44 s, healthy). An installer pass would cost
~45 s and run BEFORE any whole-guest backup exists (no undo) — not built.
6. **Slow lane: host kernel and Proxmox packages, with the reboot.** **Split 2026-10-07 (decisions 163–164):** the Proxmox
USERSPACE packages are §5.10 (BUILT, unreleased, no reboot); the kernel lane is R-836 (a spike with reboots first).
USERSPACE packages are §5.10 (BUILT, proven on demo-felhom); the kernel lane is §5.11 (**BUILT 2026-10-07**, agent
v0.152.0 + hub v0.143.0, decision 172; the spike with reboots came first, `audits/kernel-spike-2026-10-07/`).
7. **Later:** the Proxmox major upgrade (PVE 9 → 10), drilled on ring 0 first.
---
documentation/audits/kernel-lane-2026-10-07/A/redproof.txt
+18
View File
@@ -0,0 +1,18 @@
# kernel lane red-proof 2026-10-07T13:08:35Z — each mutation must turn its test red (FAILED); the clean tree is green
clean: OK
no default pin before the install -> test_installing_a_kernel_does_not_change_the_grub_default: FAILED (failures=1)
no boot-setup check -> test_a_box_without_the_esp_flag_is_refused: FAILED (failures=1)
reboot without the flag check -> test_a_reboot_without_the_flag_is_refused: FAILED (failures=1)
no signed-set image check -> test_a_kernel_outside_the_approved_set_is_refused: FAILED (failures=1) (re-run after the test was sharpened: the first run stayed green because a second image was ALSO refused by the one-kernel rule)
no expect_kver check -> test_a_kernel_outside_the_approved_set_is_refused: FAILED (failures=1)
snippet sets the default before clearing the flag -> test_the_snippet_clears_the_flag_on_use: FAILED (errors=1)
no crash-guard check -> test_the_crash_guard_must_be_armed_and_quiet: FAILED (failures=1)
self-revert used twice -> test_one_self_revert_then_never_again: FAILED (failures=1)
guard trips one unclean boot early (LIMIT-2) -> KernelStepCannotLeaveTheBoxOff.test_planned_reboot_one_crash_one_self_revert: FAILED (failures=1)
go: kernelDue ignores Tonight -> TestKernel_NoMailNoStep: --- FAIL: TestKernel_NoMailNoStep (0.00s) FAIL
go: KernelVerdict ignores the hub -> TestKernelVerdict: --- FAIL: TestKernelVerdict (0.00s) FAIL
go: no self-revert call -> TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait: --- FAIL: TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait (0.00s) FAIL
hub: tonight without a mail -> TestKernel_DueButNotToldIsNotTonight: --- FAIL: TestKernel_DueButNotToldIsNotTonight (0.04s)
hub: a failed mail still recorded -> TestKernel_NoMailNoStep: --- FAIL: TestKernel_NoMailNoStep (0.04s)
hub: approval without a night stage -> TestKernel_ApproveNeedsEveryRing0BoxHealthyAfterANightStep: --- FAIL: TestKernel_ApproveNeedsEveryRing0BoxHealthyAfterANightStep (0.03s)
hub: kernel button without its readiness gate (.Fingerprint and Waiting dropped) -> TestSystemPage_KernelButtonOnlyWhenReady: --- FAIL: TestSystemPage_KernelButtonOnlyWhenReady (0.04s)
documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/agent-journal.txt
+36
View File
@@ -0,0 +1,36 @@
2026-10-07T15:11:39+02:00 demo-felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.785+02:00 level=INFO msg="felhom-agent daemon starting" version=0.151.0 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.795+02:00 level=INFO msg="daemon: operator signers pinned" count=2
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="local-api leaf LOADED" fingerprint_sha256=c4b96bf0f97efcd462b11d33bd39a28133b4b3648535ee5b497c005eadf3e51e cert=/var/lib/felhom-agent/local-api.crt
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="backup tier armed" target=felhom-backup cadence=24h0m0s keep_last=3 wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="lanresolver: enabled" host_ip=192.168.0.162 upstreams="[1.1.1.1 8.8.8.8]" interval_s=300
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="wgtunnel: enabled" interval_s=60 state_dir=/var/lib/felhom-agent
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="poke: agent-plane sync listener enabled" port=51822
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="felhomsshd (OOB): enabled" interval_s=60 state_dir=/var/lib/felhom-agent
2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="capabilities self-check" ok=68 total=68 degraded=0 inactive=0
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.531+02:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="fast-tick armed: 30s out-of-band cadence while desired-state is unapplied" interval=30s
2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.534+02:00 level=INFO msg="poke: listening for hub sync-pokes (WG-confined, contentless)" addr=10.77.0.2:51822
2026-10-07T15:11:43+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:43.004+02:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-10-07T15:11:43+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:43.004+02:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.915+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.934+02:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.934+02:00 level=INFO msg="local-api server listening" addr=169.254.253.1:8443
2026-10-07T15:11:45+02:00 demo-felhom felhom-priv-apply[1711]: felhom-priv-apply: SAME sshd-config /etc/felhom-sshd/sshd_config
2026-10-07T15:11:46+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:46.084+02:00 level=INFO msg="felhomsshd: config applied" port=8822 action=reload
2026-10-07T15:11:46+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:46.142+02:00 level=INFO msg="felhomsshd belt: set synced" set=ssh_port elements=8822
2026-10-07T15:11:49+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:49.890+02:00 level=INFO msg="guestnet: guest network unhealthy but not acting" vmid=9201 reason="agent started less than 3m0s ago" detail="no IPv4 address on eth0"
2026-10-07T15:11:50+02:00 demo-felhom felhom-agent[1198]: 2026/10/07 15:11:50 http: TLS handshake error from 169.254.253.2:55264: EOF
2026-10-07T15:11:50+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:50.475+02:00 level=WARN msg="lanresolver: reconcile failed" vmid=9201 err="discover guest 9201 domain: pct exec cat controller.yaml: : exit status 137"
2026-10-07T15:11:51+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:51.814+02:00 level=INFO msg="desired: updated from hub" generation=18 guests=0
2026-10-07T15:11:51+02:00 demo-felhom felhom-priv-apply[3439]: felhom-priv-apply: SAME sshd-key /etc/felhom-sshd/authorized_keys/felhom-op
2026-10-07T15:11:51+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:51.868+02:00 level=INFO msg="felhomsshd: operator authorized_keys updated" user=felhom-op present=true
2026-10-07T15:11:51+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:51.930+02:00 level=INFO msg="felhomsshd belt: set synced" set=operator_ips elements=10.77.0.250
documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/timing.log
+11
View File
@@ -0,0 +1,11 @@
old boot b7c1573e-e8d3-4921-9a98-4ec43dcbad70
reboot sent 2026-10-07T13:10:48Z
+53s LAN ssh back
+53s guest running
+53s containers running=5/5 starting-or-unhealthy=4
+53s tunnel container running
+53s tailnet ssh back
+68s agent reached the hub (2026-10-07T15:11:51+02:00)
+68s containers running=5/5 starting-or-unhealthy=0
+68s ALL containers running and healthy
SUMMARY lan=53 hub=68 guest=53 apps=68 tunnel=53 tailnet=53
documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/units-journal.txt
+138
View File
@@ -0,0 +1,138 @@
2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)...
2026-10-07T15:11:34+02:00 demo-felhom felhom-crash-guard[532]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
2026-10-07T15:11:34+02:00 demo-felhom felhom-crash-guard[653]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Finished felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit).
2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Starting tailscaled.service - Tailscale node agent...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: TPM: successfully read all properties
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Program starting: v1.102.2-t6cac91817-g6ff0ddc72, Go 1.26.5: []string{"/usr/sbin/tailscaled", "--state=/var/lib/tailscale/tailscaled.state", "--socket=/run/tailscale/tailscaled.sock", "--port=41641"}
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: LogID: 94a2aaa0b2c8fac06a42677d5acc9c23a59b45535ba49413ff0e63882f48728e
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale"
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: [rc=unknown ret=direct]
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using "direct" mode
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using *dns.directManager
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: inotify: NewDirWatcher: context canceled
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: logtail: dial "log.tailscale.com:443" failed: dial tcp: lookup log.tailscale.com on 192.168.0.1:53: dial udp 192.168.0.1:53: connect: network is unreachable (in 2ms), trying bootstrap...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "log.tailscale.com" error: Get "https://derp6.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 68.183.90.120:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4e.tailscale.com", "2a03:b0c0:3:d0::29:9001") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4e.tailscale.com", "2a03:b0c0:3:d0::29:9001") for "log.tailscale.com" error: Get "https://derp4e.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2a03:b0c0:3:d0::29:9001]:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4c.tailscale.com", "134.122.77.138") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4c.tailscale.com", "134.122.77.138") for "log.tailscale.com" error: Get "https://derp4c.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 134.122.77.138:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp7.tailscale.com", "2401:c080:1000:467f:5400:2ff:feee:22aa") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp7.tailscale.com", "2401:c080:1000:467f:5400:2ff:feee:22aa") for "log.tailscale.com" error: Get "https://derp7.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2401:c080:1000:467f:5400:2ff:feee:22aa]:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4e.tailscale.com", "134.122.74.153") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4e.tailscale.com", "134.122.74.153") for "log.tailscale.com" error: Get "https://derp4e.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 134.122.74.153:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp6.tailscale.com", "2400:6180:100:d0::982:d001") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp6.tailscale.com", "2400:6180:100:d0::982:d001") for "log.tailscale.com" error: Get "https://derp6.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2400:6180:100:d0::982:d001]:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp7.tailscale.com", "167.179.89.145") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp7.tailscale.com", "167.179.89.145") for "log.tailscale.com" error: Get "https://derp7.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 167.179.89.145:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp12b.tailscale.com", "2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp12b.tailscale.com", "2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f") for "log.tailscale.com" error: Get "https://derp12b.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f]:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp8c.tailscale.com", "206.189.16.32") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp8c.tailscale.com", "206.189.16.32") for "log.tailscale.com" error: Get "https://derp8c.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 206.189.16.32:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4d.tailscale.com", "2a03:b0c0:3:d0::1501:b001") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4d.tailscale.com", "2a03:b0c0:3:d0::1501:b001") for "log.tailscale.com" error: Get "https://derp4d.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2a03:b0c0:3:d0::1501:b001]:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp1e.tailscale.com", "64.225.56.166") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp1e.tailscale.com", "64.225.56.166") for "log.tailscale.com" error: Get "https://derp1e.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 64.225.56.166:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp12.tailscale.com", "2001:19f0:5c01:289:5400:3ff:fe8d:cb5e") for "log.tailscale.com" ...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp12.tailscale.com", "2001:19f0:5c01:289:5400:3ff:fe8d:cb5e") for "log.tailscale.com" error: Get "https://derp12.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2001:19f0:5c01:289:5400:3ff:fe8d:cb5e]:443: connect: network is unreachable
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: logtail: upload: log upload of 365 bytes compressed failed: Post "https://log.tailscale.com/c/tailnode.log.tailscale.io/a8874038d193b4737b146408485f2db2abd40834fbfa94f59c39122bb7f51754": failed to resolve "log.tailscale.com": no DNS fallback candidates remain for "log.tailscale.com"
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: enumerating tailscale0 addresses for cleanup failed: failed to look up link "tailscale0": Link not found
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: wgengine.NewUserspaceEngine(tun "tailscale0") ...
2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Started tailscaled.service - Tailscale node agent.
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: [rc=unknown ret=direct]
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using "direct" mode
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using *dns.directManager
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: link state: interfaces.State{defaultRoute= ifs={} v4=false v6=false}
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp6)
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: magicsock: disco key = d:f75f0468399c429d
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: using firewall mode pref
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: magicsock: SetNetworkUp(false)
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Creating WireGuard device...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Bringing WireGuard device up...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: default choosing iptables
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Bringing router up...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: external route: up
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: netfilter running in iptables mode v6 = true, v6filter = true, v6nat = true
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp4)
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Clearing router settings...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Starting network monitor...
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Engine created.
2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: LinkChange: all links down; pausing: old: interfaces.State{defaultRoute= ifs={} v4=false v6=false} new: interfaces.State{defaultRoute= ifs={} v4=false v6=false} diff: numInterfaces: 3->4; numInterfaceIPs: 3->4; if tailscale0: added; ips tailscale0: added [fe80::fbbf:d4ae:bf83:b0ed/64]
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: pm: using backend prefs for "profile-10be": Prefs{ra=false dns=false want=true routes=[] statefulFiltering=false nf=on host="felhom-pve" update=on Persist{o=, n=[CE5tx] u="nagyfenyvesi.viktor@gmail.com" ak=-}}
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: envknob: PORT="41641"
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale"
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: linkChange: in state NoState; PAC or proxyConfig changed; updating routes
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: got LocalBackend in 35ms
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: Start
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: offline auto-update: starting update checks
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: ipnext: "conn25": skipping extension
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: ipnext: active extensions: captiveportal, acme, portlist, posture, clientupdate, relayserver, routecheck, serviceclientprefs, taildrop
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: load netmap from cache: netmap cache is not available
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: setPaused(true)
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: authRoutine: awaiting unpause
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: mapRoutine: awaiting unpause
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: Backend: logs: be:94a2aaa0b2c8fac06a42677d5acc9c23a59b45535ba49413ff0e63882f48728e fe:
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: updateRoutine: awaiting unpause
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: client.Login(0)
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: health(warnable=warming-up): error: Tailscale is starting. Please wait.
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=192.168.0.162/0, dst=192.168.0.162/32, gw=, outif=5, table=255
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=192.168.0.162/0, dst=192.168.0.0/24, gw=, outif=5, table=254
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=192.168.0.162/0, dst=192.168.0.255/32, gw=, outif=5, table=255
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=, dst=, gw=192.168.0.1, outif=5, table=254
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=10.77.0.2/0, dst=10.77.0.2/32, gw=, outif=7, table=255
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=, dst=10.77.0.250/32, gw=, outif=7, table=254
2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=, dst=10.77.0.1/32, gw=, outif=7, table=254
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: control: setPaused(false)
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: LinkChange: major, rebinding: old: interfaces.State{defaultRoute= ifs={} v4=false v6=false} new: interfaces.State{defaultRoute=vmbr0 ifs={vmbr0:[192.168.0.162/24 llu6] wg-felhom:[10.77.0.2/32]} v4=true v6=false} diff: HaveV4: false->true; DefaultRoute: ""->"vmbr0"; numInterfaces: 4->7; numInterfaceIPs: 4->7; if enp1s0 flags: broadcast|multicast->up|broadcast|multicast|running; if vmbr0: added; if vmbr9: added; if wg-felhom: added; ips vmbr0: added [192.168.0.162/24 fe80::6a1d:efff:fe5d:a664/64]; ips vmbr9: added [169.254.253.1/30 fe80::c0fc:cff:feca:d18c/64]; ips wg-felhom: added [10.77.0.2/32] rebind-reason=[default-if-changed,ips-changed,protocols-changed]
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: magicsock: SetNetworkUp(true)
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: control: doLogin(regen=false, hasUrl=false)
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp6)
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: Rebind; defIf="vmbr0", ips=[192.168.0.162/24 fe80::6a1d:efff:fe5d:a664/64]
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: magicsock: 0 active derp conns
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp4)
2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: monitor: gateway and self IP changed: gw=192.168.0.1 self=192.168.0.162
2026-10-07T15:11:40+02:00 demo-felhom tailscaled[711]: health(warnable=warming-up): ok
2026-10-07T15:11:40+02:00 demo-felhom systemd[1]: Starting pve-guests.service - PVE guests...
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: control server key from https://controlplane.tailscale.com: ts2021=[fSeS+], legacy=[nlFWp]
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: RegisterReq: onode= node=[CE5tx] fup=false nks=false
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: RegisterReq: got response; nodeKeyExpired=false, machineAuthorized=true; authURL=false
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: netmap: got new dial plan from control
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: new contact: control-netmap usec=6317945 cached=false
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: active login: nagyfenyvesi.viktor@gmail.com
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: netmap: suggested exit node: no preferred DERP, try again later
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: offline auto-update: stopping update checks
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: Switching ipn state NoState -> Starting (WantRunning=true, nm=true)
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: SetPrivateKey called (init)
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: wgengine: Reconfig: configuring router
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: router: enabling connmark-based rp_filter workaround
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: wgengine: Reconfig: user dialer
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: tsdial: bart table size: 5
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: wgengine: Reconfig: configuring DNS
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: dns: Set: {DefaultResolvers:[] Routes:{} SearchDomains:[] Hosts:0}
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: dns: Resolvercfg: {Routes:{} Hosts:0 LocalDomains:[]}
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: dns: OScfg: {}
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: peerapi: serving on http://100.70.170.35:36209
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: peerapi: serving on http://[fd7a:115c:a1e0::5236:aa24]:61570
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: portmapper: UPnP discovery response from 192.168.0.254, but gateway IP is 192.168.0.1
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: portmapper: UPnP meta changed: [{Location:http://192.168.0.1:1900/rootDesc.xml Server:TP-Link/TP-LINK UPnP/1.1 MiniUPnPd/1.8 USN:uuid:e4c797e0-82ce-4f84-b56d-085d9d77c8cd::urn:schemas-upnp-org:device:InternetGatewayDevice:1} {Location:http://192.168.0.254:1900/igd.xml Server:ipos/7.0 UPnP/1.0 Archer_C5/2.0 USN:uuid:060b7353-fca6-4070-85f4-1fbfb9add62c::urn:schemas-upnp-org:device:InternetGatewayDevice:1}]
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: home DERP changing from derp-0 [0ms] to derp-4 [39ms] (forced=false)
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: home is now derp-4 (fra)
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: adding connection to derp-4 for home-keep-alive
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: 1 active derp conns: derp-4=cr0s,wr0s
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: derphttp.Client.Connect: connecting to derp-4 (fra)
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: updating netmap in disk cache
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: endpoints changed: 37.191.56.193:41641 (portmap), 10.77.0.2:41641 (local), 192.168.0.162:41641 (local)
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: Switching ipn state Starting -> Running (WantRunning=true, nm=true)
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: NetInfo: NetInfo{varies=false ipv6=false ipv6os=true udp=true icmpv4=false derp=#4 portmap=active-UM link="" firewallmode="ipt-default"}
2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: derp-4 connected; connGen=1
2026-10-07T15:11:41+02:00 demo-felhom pve-guests[1375]: <root@pam> starting task UPID:demo-felhom:000005D5:00000504:6AC6450D:startall::root@pam:
2026-10-07T15:11:41+02:00 demo-felhom pvesh[1375]: Starting CT 9201
2026-10-07T15:11:41+02:00 demo-felhom pve-guests[1493]: <root@pam> starting task UPID:demo-felhom:000005D6:00000505:6AC6450D:vzstart:9201:root@pam:
2026-10-07T15:11:41+02:00 demo-felhom pve-guests[1494]: starting CT 9201: UPID:demo-felhom:000005D6:00000505:6AC6450D:vzstart:9201:root@pam:
2026-10-07T15:11:45+02:00 demo-felhom tailscaled[711]: magicsock: new contact: peer=[G+z+N] usec=10136339 cached=false via=derp
2026-10-07T15:11:45+02:00 demo-felhom tailscaled[711]: magicsock: disco: node [G+z+N] d:ac1b0afc01e49a40 now using 192.168.0.180:35032 mtu=1360 tx=3f73e074118a
2026-10-07T15:11:45+02:00 demo-felhom tailscaled[711]: magicsock: new contact: peer=[G+z+N] usec=10137837 cached=false via=direct
2026-10-07T15:11:46+02:00 demo-felhom pve-guests[1375]: <root@pam> end task UPID:demo-felhom:000005D5:00000504:6AC6450D:startall::root@pam: OK
2026-10-07T15:11:46+02:00 demo-felhom systemd[1]: Finished pve-guests.service - PVE guests.
2026-10-07T15:11:51+02:00 demo-felhom tailscaled[711]: magicsock: disco: node [G+z+N] d:ac1b0afc01e49a40 now using 192.168.0.180:35032 mtu=1360 tx=2b416b182a12
documentation/audits/kernel-lane-2026-10-07/B/demo-hp/agent-journal.txt
+50
View File
@@ -0,0 +1,50 @@
2026-10-07T15:16:05+02:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.853+02:00 level=INFO msg="felhom-agent daemon starting" version=0.151.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.864+02:00 level=INFO msg="daemon: operator signers pinned" count=2
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="local-api leaf LOADED" fingerprint_sha256=34122ac2ace991685ad9adaaf882702581ae931837203dbeb25f0d9e493a837d cert=/var/lib/felhom-agent/local-api.crt
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 wait_timeout=30m0s prune_pbs_allowed=false primary=true
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 wait_timeout=12h0m0s prune_pbs_allowed=false primary=false
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="lanresolver: enabled" host_ip=192.168.0.104 upstreams="[1.1.1.1 8.8.8.8]" interval_s=300
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="wgtunnel: enabled" interval_s=60 state_dir=/var/lib/felhom-agent
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="poke: agent-plane sync listener enabled" port=51822
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="felhomsshd (OOB): enabled" interval_s=60 state_dir=/var/lib/felhom-agent
2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.872+02:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)"
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.808+02:00 level=INFO msg="capabilities self-check" ok=68 total=68 degraded=0 inactive=0
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="poke: listening for hub sync-pokes (WG-confined, contentless)" addr=10.77.0.3:51822
2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="fast-tick armed: 30s out-of-band cadence while desired-state is unapplied" interval=30s
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/hdd_1 stable=/mnt/felhom-drives/hdd_1 prior_binds=0
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.613+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.616+02:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.616+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.616+02:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.617+02:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test"
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.617+02:00 level=INFO msg="local-api server listening" addr=169.254.253.1:8443
2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.619+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
2026-10-07T15:16:07+02:00 demo-hp felhom-priv-apply[1599]: felhom-priv-apply: SAME sshd-config /etc/felhom-sshd/sshd_config
2026-10-07T15:16:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:08.339+02:00 level=INFO msg="felhomsshd: config applied" port=8822 action=reload
2026-10-07T15:16:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:08.395+02:00 level=INFO msg="felhomsshd belt: set synced" set=ssh_port elements=8822
2026-10-07T15:16:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:08.930+02:00 level=INFO msg="desired: updated from hub" generation=9 guests=0
2026-10-07T15:16:08+02:00 demo-hp felhom-priv-apply[1911]: felhom-priv-apply: SAME sshd-key /etc/felhom-sshd/authorized_keys/felhom-op
2026-10-07T15:16:09+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:09.008+02:00 level=INFO msg="felhomsshd: operator authorized_keys updated" user=felhom-op present=true
2026-10-07T15:16:09+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:09.083+02:00 level=INFO msg="felhomsshd belt: set synced" set=operator_ips elements=10.77.0.250
2026-10-07T15:16:09+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:09.581+02:00 level=INFO msg="lanresolver: guest IP not yet leased — skipping (will retry)" vmid=9201
2026-10-07T15:16:10+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:10.562+02:00 level=INFO msg="lanresolver: guest IP not yet leased — skipping (will retry)" vmid=9202
2026-10-07T15:16:23+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:23.620+02:00 level=WARN msg="local-api: storage view unavailable for class hints" err="storage: NodeStorage: proxmox: GET /nodes/demo-hp/storage: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage\": context canceled"
2026-10-07T15:16:28+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:28.656+02:00 level=INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/hdd_1 stable=/mnt/felhom-drives/hdd_1 prior_binds=1
2026-10-07T15:16:28+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:28.657+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-10-07T15:16:37+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:37.744+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
2026-10-07T15:16:39+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:39.272+02:00 level=INFO msg="controller-supervisor: slow counter restored from disk" vmid=9201 restarts_24h=0 slow_crashloop_since=2026-09-17T09:22:29Z
2026-10-07T15:16:48+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:48.483+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-10-07T15:16:55+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:55.099+02:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2
2026-10-07T15:17:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:07.712+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
2026-10-07T15:17:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:07.725+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
2026-10-07T15:17:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:07.737+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
2026-10-07T15:17:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:08.416+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-10-07T15:17:28+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:28.462+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-10-07T15:17:37+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:37.717+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
documentation/audits/kernel-lane-2026-10-07/B/demo-hp/timing.log
+21
View File
@@ -0,0 +1,21 @@
old boot 8ea2cdff-d166-4af6-9937-156f4ee659a5
reboot sent 2026-10-07T13:13:06Z
+176s LAN ssh back
+176s containers running=0/0 starting-or-unhealthy=0
+189s agent reached the hub (2026-10-07T15:16:08+02:00)
+189s guest running
+189s containers running=18/21 starting-or-unhealthy=17
+189s tunnel container running
+204s containers running=20/21 starting-or-unhealthy=11
+211s containers running=20/21 starting-or-unhealthy=7
+217s containers running=21/21 starting-or-unhealthy=6
+224s containers running=21/21 starting-or-unhealthy=6
+231s containers running=21/21 starting-or-unhealthy=6
+238s containers running=21/21 starting-or-unhealthy=6
+245s containers running=21/21 starting-or-unhealthy=6
+252s containers running=21/21 starting-or-unhealthy=6
+258s containers running=21/21 starting-or-unhealthy=2
+265s containers running=21/21 starting-or-unhealthy=1
+272s containers running=21/21 starting-or-unhealthy=0
+272s ALL containers running and healthy
SUMMARY lan=176 hub=189 guest=189 apps=272 tunnel=189 tailnet=n/a
documentation/audits/kernel-lane-2026-10-07/B/demo-hp/units-journal.txt
+18
View File
@@ -0,0 +1,18 @@
2026-10-07T15:16:00+02:00 demo-hp systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)...
2026-10-07T15:16:00+02:00 demo-hp felhom-crash-guard[628]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
2026-10-07T15:16:00+02:00 demo-hp felhom-crash-guard[773]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
2026-10-07T15:16:00+02:00 demo-hp systemd[1]: Finished felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit).
2026-10-07T15:16:06+02:00 demo-hp systemd[1]: Starting pve-guests.service - PVE guests...
2026-10-07T15:16:08+02:00 demo-hp pve-guests[1493]: <root@pam> starting task UPID:demo-hp:000006CD:000006D3:6AC64618:startall::root@pam:
2026-10-07T15:16:08+02:00 demo-hp pvesh[1493]: Starting VM 341
2026-10-07T15:16:08+02:00 demo-hp pve-guests[1741]: <root@pam> starting task UPID:demo-hp:000006D4:000006D5:6AC64618:qmstart:341:root@pam:
2026-10-07T15:16:08+02:00 demo-hp pve-guests[1748]: start VM 341: UPID:demo-hp:000006D4:000006D5:6AC64618:qmstart:341:root@pam:
2026-10-07T15:16:09+02:00 demo-hp pve-guests[1748]: VM 341 started with PID 1891.
2026-10-07T15:16:11+02:00 demo-hp pvesh[1493]: Starting CT 9201
2026-10-07T15:16:11+02:00 demo-hp pve-guests[1741]: <root@pam> starting task UPID:demo-hp:000007A3:00000804:6AC6461B:vzstart:9201:root@pam:
2026-10-07T15:16:11+02:00 demo-hp pve-guests[1955]: starting CT 9201: UPID:demo-hp:000007A3:00000804:6AC6461B:vzstart:9201:root@pam:
2026-10-07T15:16:11+02:00 demo-hp pvesh[1493]: Starting CT 9202
2026-10-07T15:16:11+02:00 demo-hp pve-guests[1741]: <root@pam> starting task UPID:demo-hp:000007A5:00000808:6AC6461B:vzstart:9202:root@pam:
2026-10-07T15:16:11+02:00 demo-hp pve-guests[1957]: starting CT 9202: UPID:demo-hp:000007A5:00000808:6AC6461B:vzstart:9202:root@pam:
2026-10-07T15:16:16+02:00 demo-hp pve-guests[1493]: <root@pam> end task UPID:demo-hp:000006CD:000006D3:6AC64618:startall::root@pam: OK
2026-10-07T15:16:16+02:00 demo-hp systemd[1]: Finished pve-guests.service - PVE guests.
documentation/audits/kernel-lane-2026-10-07/delivery/tester1.txt
+4
View File
@@ -0,0 +1,4 @@
== agent_update 0.152.0 → tester-1, 2026-10-07T13:21:33Z
signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=31589d832a25a458ce3878c0fcbd3ae3 expires=2026-10-07T14:06:33Z
wrote envelope to /dev/null
uploaded signed op to the hub jobs queue
hub/CHANGELOG.md
+27
View File
@@ -1,3 +1,30 @@
## v0.143.0 — the kernel lane: the day-before household mail, the night instruction, the operator's kernel set (R-836; `09` §3 decision 172; `11` §5.11) (2026-10-07)
**Operator action on deploy: none.** Built and deployed only in a session the operator attends (decision 162). Needs
agent v0.152.0 on a box for anything to happen there; an older agent reports no kernel lane and is never due.
- **Due** (`osupdates/kernel.go` `KernelDue`): a ring-0 box with a pending `proxmox-kernel-X.Y` upgrade in its host
report, or a ring-1 box whose facts say a signed `os_kernel_step` STAGED a kernel; never a switched-off box, a box
whose wrapper names a setup problem, a box already running it, or a kernel whose step ended (`applied`, `fell_back`,
`health_failed`, `self_reverted`, `revert_failed`, a refusal R20/R23/R3/R12) — the operator decides; never retried.
- **The day-before mail** (`KernelNotify`, on the minute tick): ONE mail per due box between 09:00 and 20:00
Budapest, never twice within 20 h, at most 3 per kernel (then one operator event); to the REGISTERED customer
address (the notification address when none), in the household's language, informal (`mail.kernel.*` in both
locales; `notify.SendKernelNotice`). Only a mail the mail service accepted is recorded (`os_kernel_notices`).
- **The instruction** (`os_update.kernel` {kver, tonight, notified_at}): `tonight` only within 24 h of a recorded
mail — **no mail, no step**. The box is bumped when the mail goes out.
- **Reports**: layer `kernel` is its own layer (an older hub filed unknown layers as the guest's). Outcomes staged,
judging, applied, fell_back, health_failed, self_reverted, revert_failed, refused, failed → operator events
`os_kernel_step` / `os_update_failed` (operator-only, registered in the same commit); `applied` also gives the
household its "security fixes installed" timeline line.
- **Approval**: "Approve kernel set" (`/os/approve-kernel`) appears when every ring-0 box's newest ended kernel step is
`applied` for the same kernel after a NIGHT stage; the set is the series meta-package and the signed image; it nudges
no box (ring 1: a signed `os_kernel_step` only).
- **System page**: two cells — "Kernel (default)" (amber while a one-shot flag names another kernel) and "Kernel step"
(the newest result, the kernel due, whether the household was told for tonight).
- Tests: `osupdates/kernel_test.go` (8), `web/system_test.go` (2), `notify/kernel_notice_test.go`; red-proofs
`documentation/audits/kernel-lane-2026-10-07/A/redproof.txt`.
## v0.142.0 — the Proxmox package set; RESET purges the off-site folder; one line when old copies use another key; the two never-built DR fields retired (R-812 A, R-32, R-366, R-105; `09` §3 163, 167–169) (2026-10-07)
**Operator action on deploy: none.** Built and deployed in the attended session (decision 162).
hub/cmd/hub/main.go
+3 -1
View File
@@ -420,7 +420,9 @@ func main() {
if _, err := dataStore.BumpHostDesired(hostID); err != nil {
logger.Printf("[WARN] osupdates: bump desired for %s: %v", hostID, err)
}
}}
},
// R-836 (`09` §3 decision 172): the household's day-before kernel mail; no accepted mail → no kernel step.
KernelMail: dispatcher.SendKernelNotice}
if v := os.Getenv("OS_APPROVE_AFTER"); v != "" {
if d, derr := time.ParseDuration(v); derr == nil && d >= 0 {
osSvc.ApproveAfter = d
hub/internal/i18n/locales/en.json
+2
View File
@@ -57,6 +57,8 @@
"mail.claim.claim.body": "Dear Customer,\n\nYour Felhom server is up. To use the dashboard for the first time, enter the\nsetup code below, then choose your own password:\n\nSetup code: %s\n\nThe code is valid for 72 hours and can be used once. You can reach the\ndashboard here:\n\n%s\n\nIf it did not reach you in time, the dashboard's \"Request a new code\" button\ngets you a fresh one — it always arrives at this e-mail address.\n\nBest regards,\nFelhom.eu",
"mail.selfbind.subject": "[Felhom] Link your Felhom box to your account",
"mail.selfbind.body": "Dear Customer,\n\nYour Felhom box is ready and waiting to be linked. You can link it to your\naccount yourself using the link below — no sign-in needed:\n\n%s\n\nAfter you open the link you will need to enter two things:\n\n 1. The pairing code, shown on the box's own screen (the monitor).\n 2. Your owner passphrase (the 5-word phrase) that you received from your\n Felhom operator — in person or by telephone, never by e-mail. It proves\n the account is yours.\n\nThe link is valid for 7 days. For safety it locks after 5 failed attempts —\nif that happens, contact support.\n\nIf you did not ask for this, ignore this e-mail.\n\nBest regards,\nFelhom.eu",
"mail.kernel.subject": "[Felhom] Your Felhom box restarts tonight",
"mail.kernel.body": "Hi,\n\nTonight your Felhom box restarts for a security update. Your apps will be away for a few\nminutes, then come back by themselves. You do not need to do anything.\n\nIf the box is not back by morning: unplug its power cable, wait 10 seconds, and plug it back\nin. The box then starts its earlier, proven system by itself.\n\nIf that does not help, reply to this e-mail and we will help.\n\nBest,\nFelhom.eu",
"bind.title": "Felhom — Link your box",
"bind.heading": "Link your Felhom <span>box</span>",
"bind.lead": "Link the Felhom box you have just installed to your account. Enter the pairing code shown on the box's screen, and your owner passphrase.",
hub/internal/i18n/locales/hu.json
+2
View File
@@ -57,6 +57,8 @@
"mail.claim.claim.body": "Kedves Ügyfél!\n\nElindult a Felhom szervered. A vezérlőpult első használatához add meg az\nalábbi beállító kódot, majd válassz saját jelszót:\n\nBeállító kód: %s\n\nA kód 72 óráig érvényes, és egyszer használható fel. A vezérlőpultot itt éred el:\n\n%s\n\nHa nem kaptad volna meg időben, a vezérlőpult \"Új kód kérése\" gombjával\nkérhetsz frisset — az mindig erre az e-mail címre érkezik.\n\nÜdvözlettel,\nFelhom.eu",
"mail.selfbind.subject": "[Felhom] Kösd össze a Felhom dobozodat",
"mail.selfbind.body": "Kedves Ügyfél!\n\nElkészült a Felhom dobozod, és készen áll az összekötésre. Az alábbi hivatkozáson\ntudod te magad összekötni a fiókoddal — nincs szükség bejelentkezésre:\n\n%s\n\nA hivatkozás megnyitása után két adatot kell megadnod:\n\n 1. A párosító kódot, amely a doboz képernyőjén (a monitoron) látható.\n 2. A tulajdonosi jelmondatodat (az 5 szóból álló kifejezést), amelyet a\n Felhom üzemeltetőjétől kaptál — személyesen vagy telefonon, e-mailben\n soha. Ez igazolja, hogy a fiók a tiéd.\n\nA hivatkozás 7 napig érvényes. Biztonsági okból 5 sikertelen próbálkozás után\nzárolódik — ilyenkor vedd fel a kapcsolatot az ügyfélszolgálattal.\n\nHa nem te kérted ezt, hagyd figyelmen kívül ezt az e-mailt.\n\nÜdvözlettel,\nFelhom.eu",
"mail.kernel.subject": "[Felhom] Ma éjjel újraindul a Felhom dobozod",
"mail.kernel.body": "Szia!\n\nMa éjjel a Felhom dobozod egy biztonsági frissítés miatt újraindul. Ilyenkor az alkalmazásaid\nnéhány percig nem érhetők el, utána maguktól visszajönnek. Neked nem kell semmit tenned.\n\nHa reggelre a doboz mégsem érhető el: húzd ki a tápkábelét, várj 10 másodpercet, és dugd\nvissza. A doboz ilyenkor magától a korábbi, bevált rendszerrel indul el.\n\nHa ez sem segít, válaszolj erre a levélre, és segítünk.\n\nÜdv,\nFelhom.eu",
"bind.title": "Felhom — Doboz összekötése",
"bind.heading": "Felhom <span>doboz</span> összekötése",
"bind.lead": "Kösd össze a most telepített Felhom dobozodat a fiókoddal. Add meg a doboz képernyőjén látható párosító kódot és a tulajdonosi jelmondatodat.",
hub/internal/notify/dispatcher.go
+37
View File
@@ -733,6 +733,10 @@ var operatorOnlyEvents = map[string]bool{
// R-841 (hub v0.131.0): the tunnel alarm — a box fact the household can do nothing about from inside.
"tunnel_down": true,
"tunnel_recovered": true,
// R-836 (hub v0.143.0): the kernel lane's step results and the day-before notice's own record — operator facts. The
// household's word is the day-before mail itself (SendKernelNotice), never these events.
"os_kernel_step": true,
"os_kernel_notice": true,
// `11` §8.3 (hub v0.131.0): the four OS-update alarms — fleet facts only the operator can act on.
"os_update_stale": true,
"os_reboot_needed": true,
@@ -969,6 +973,39 @@ func (d *Dispatcher) SendSelfBindEmail(customerID, email, link string) error {
return nil
}
// SendKernelNotice mails the household that the box restarts tonight for a kernel update (R-836, `09` §3 decision 172).
// To the REGISTERED customer address (the notification address when none is registered) — a promise to the household,
// so it does not depend on which event types the household switched on. Returns the language used. An error means the
// mail service did not accept it, and the caller then runs NO kernel step (no mail, no step).
func (d *Dispatcher) SendKernelNotice(customerID, kver string) (string, error) {
if d.resendAPIKey == "" {
d.logger.Printf("[ERROR] kernel notice for %s NOT sent: no Resend API key configured", customerID)
return "", fmt.Errorf("notify: no resend api key")
}
to := ""
if cc, err := d.store.GetCustomerConfig(customerID); err == nil && cc != nil {
to = strings.TrimSpace(cc.Email)
}
if to == "" {
if p, err := d.store.GetNotificationPrefs(customerID); err == nil && p != nil {
to = strings.TrimSpace(p.Email)
}
}
if to == "" {
return "", fmt.Errorf("the household has no e-mail address")
}
lang := d.store.CustomerLanguage(customerID)
subject, body := FormatKernelNoticeEmail(lang)
if err := d.sendEmailFn(to, subject, body, nil); err != nil {
d.logger.Printf("[ERROR] kernel notice (%s) to customer %s failed: %v", kver, customerID, err)
d.store.LogNotification(customerID, "kernel_notice", "info", subject, "failed", err.Error(), "customer")
return lang, err
}
d.logger.Printf("[INFO] kernel notice (%s) emailed to the household of %s (lang=%s)", kver, customerID, lang)
d.store.LogNotification(customerID, "kernel_notice", "info", subject, "sent", "", "customer")
return lang, nil
}
// firstHourOfABox is how long after enrolment a whole-guest tier skip is provisioning, not a fault (R-723).
const firstHourOfABox = time.Hour
hub/internal/notify/kernel_notice_test.go
+33
View File
@@ -0,0 +1,33 @@
package notify
import (
"strings"
"testing"
)
// R-836, `09` §3 decision 172: the household's day-before mail — both languages, informal, says the box restarts
// tonight and exactly what to do if it is not back by morning. ASCII fragments with a positive and a negative control
// (rule 8: an accented search can return nothing for text that is there).
func TestKernelNotice_SaysWhatToDo(t *testing.T) {
for lang, want := range map[string][]string{
"hu": {"jjel", "jraindul", "tápkábel"[:2], "10 m", "dugd", "vissza", "korábbi"[:3]},
"en": {"Tonight", "restarts", "unplug", "10 seconds", "plug it back", "earlier, proven"},
} {
subj, body := FormatKernelNoticeEmail(lang)
if subj == "" || body == "" || strings.HasPrefix(subj, "mail.") || strings.HasPrefix(body, "mail.") {
t.Fatalf("%s: missing text: %q / %q", lang, subj, body)
}
for _, w := range want {
if !strings.Contains(body, w) {
t.Fatalf("%s: body lacks %q:\n%s", lang, w, body)
}
}
if strings.Contains(body, "7.0.") || strings.Contains(body, "%") {
t.Fatalf("%s: the household needs what to do, not a kernel version or a format verb:\n%s", lang, body)
}
}
// negative control: a key that does not exist comes back as its own name
if s, _ := FormatClaimEmail("en", "claim", "c", "d", "x"); strings.Contains(s, "unplug") {
t.Fatal("negative control matched")
}
}
hub/internal/notify/templates.go
+9
View File
@@ -266,6 +266,15 @@ func FormatSelfBindEmail(lang, customerID, link string) (string, string) {
return b.Msg(lang, "mail.selfbind.subject"), b.Msgf(lang, "mail.selfbind.body", link)
}
// FormatKernelNoticeEmail is the household's day-before mail of the kernel lane (R-836, `09` §3 decision 172): the box
// restarts tonight for a security update; if it is not back by morning, unplug it, wait 10 seconds, plug it back in —
// it then starts the earlier, proven kernel by itself. Informal voice, both languages from the bundle. It names no
// kernel version: the household needs what to do, not which kernel. Pinned by TestKernelNotice_*.
func FormatKernelNoticeEmail(lang string) (string, string) {
b := i18n.Shared()
return b.Msg(lang, "mail.kernel.subject"), b.Msg(lang, "mail.kernel.body")
}
// ──────────────────────────────────────────────────────────────────────
// R-182 — the backup run digest
// ──────────────────────────────────────────────────────────────────────
hub/internal/osupdates/kernel.go
+424
View File
@@ -0,0 +1,424 @@
package osupdates
// The kernel lane, hub half (hub v0.143.0; R-836, `09` §3 decision 172, `11` §5.11). The agent stages a kernel, boots
// it ONCE through a flag on the ESP, and makes it the default only after a healthy boot (felhom-agent internal/osupdate
// kernel.go + the wrapper's layer "kernel"). The hub decides WHEN:
//
// - A box is DUE when ring 0 has a pending kernel (its host report's `proxmox-kernel-X.Y` upgrade), or a ring-1 box
// runs a kernel a signed os_kernel_step STAGED — and no step for that kernel has ended yet (an ended step is the
// operator's to judge; the hub never retries it by itself).
// - The household of a due box gets ONE mail the day before, in its language, between 09:00 and 20:00 Budapest
// time (KernelNotify): the box restarts tonight; if it is not back by morning, unplug it, wait 10 seconds, plug it
// back in. Only a mail the mail service ACCEPTED is recorded.
// - The box's os_update block carries `kernel: {kver, tonight}`; tonight is true only within 24 h of such a mail. No
// mail → no step: the agent's night leg refuses a kernel without `tonight`.
// - The operator approves a kernel set ("Approve kernel set") once every ring-0 box booted it healthily as the
// default after a night step; a ring-1 box takes it only through a signed os_kernel_step.
//
// Pinned by kernel_test.go.
import (
"encoding/json"
"fmt"
"regexp"
"sort"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
"gitea.dooplex.hu/admin/felhom-hub/internal/sysfacts"
)
// LayerKernel is the host's kernel (agent v0.152.0).
const LayerKernel = "kernel"
// Kernel-lane events — operator only (the household's one word is the day-before mail).
const (
EventKernelStep = "os_kernel_step" // what became of a kernel step (staged, the default moved, fell back, …)
EventKernelNotice = "os_kernel_notice" // the household was told — or could not be told (then no step)
)
// The day-before mail's window and limits (decided by CC — operator may reverse): mailed only in the household's
// daytime, valid for the night that follows, never twice within 20 h, at most 3 times for one kernel on one box.
const (
KernelNoticeFromHour = 9
KernelNoticeToHour = 20
KernelNoticeValid = 24 * time.Hour
KernelNoticeGap = 20 * time.Hour
KernelNoticeMax = 3
)
// KernelBlock is a box's kernel instruction (felhom-agent hub.WireKernelStep — field-exact, cross-repo).
type KernelBlock struct {
Kver string `json:"kver"`
Tonight bool `json:"tonight"`
NotifiedAt string `json:"notified_at,omitempty"`
}
var (
kverRE = regexp.MustCompile(`^([0-9]+\.[0-9]+)\.[0-9]+-[0-9]+-pve$`)
kernelMetaRE = regexp.MustCompile(`^proxmox-kernel-[0-9]+\.[0-9]+$`)
)
// kernelEnded are the outcomes after which a kernel is never stepped again by itself (the operator decides).
var kernelEnded = map[string]bool{"applied": true, "fell_back": true, "health_failed": true, "self_reverted": true, "revert_failed": true}
// refusals that will not go away by waiting a night (the box cannot do a one-shot, the set is wrong, no authority,
// not an appliance) — such a refusal ends the step too. A transient one (a lock, the crash guard's window) is retried
// within KernelNoticeMax mails.
var kernelRefusedForGood = map[string]bool{"R20": true, "R23": true, "R3": true, "R12": true}
func budapestLoc() *time.Location {
if l, err := time.LoadLocation("Europe/Budapest"); err == nil {
return l
}
return time.FixedZone("CET", 3600)
}
// kernelOf is the kernel a kernel-layer report is about: the wrapper's view's "to", else the release id.
func kernelOf(r Report) string {
var v struct {
To string `json:"to"`
}
if len(r.Kernel) > 0 && json.Unmarshal(r.Kernel, &v) == nil && kverRE.MatchString(v.To) {
return v.To
}
if kverRE.MatchString(r.ReleaseID) {
return r.ReleaseID
}
return ""
}
func refusedCode(raw json.RawMessage) string {
var c struct {
Code string `json:"code"`
}
_ = json.Unmarshal(raw, &c)
return c.Code
}
// kernelLane is the box's newest kernel-lane facts (nil: an older agent, or no host report).
func (s *Service) kernelLane(hostID string) *sysfacts.KernelLane {
h, err := s.Store.GetHost(hostID)
if err != nil || h == nil {
return nil
}
rj, _ := s.Store.GetLatestHostReportJSON(h.CustomerID)
if rj == "" {
return nil
}
return sysfacts.Parse(rj).Host.KernelLane
}
// KernelDue says which kernel the box is due to step to, or why none.
func (s *Service) KernelDue(hostID string) (kver, why string) {
st := s.Store.GetOSHostSettings(hostID)
if !st.Enabled {
return "", "OS updates are switched off"
}
rep, _ := s.Store.LatestOSReport(hostID, LayerHost)
if rep == nil {
return "", "no host step reported (the kernel lane is for appliances)"
}
lane := s.kernelLane(hostID)
if lane == nil {
return "", "the box reports no kernel lane (agent older than v0.152.0)"
}
if len(lane.SetupProblems) > 0 {
return "", "the box cannot do a one-shot boot: " + strings.Join(lane.SetupProblems, "; ")
}
if st.Ring == 1 {
if lane.Phase != "staged" || !kverRE.MatchString(lane.To) {
return "", "ring 1: no kernel staged by a signed os_kernel_step"
}
kver = lane.To
} else {
var hr Report
_ = json.Unmarshal([]byte(rep.ReportJSON), &hr)
for _, p := range hr.Pending {
if kernelMetaRE.MatchString(p.Name) && p.From != "" && kverRE.MatchString(p.To+"-pve") {
kver = p.To + "-pve"
}
}
if kver == "" && lane.Phase == "staged" && kverRE.MatchString(lane.To) {
kver = lane.To
}
if kver == "" {
return "", "no pending kernel"
}
}
if lane.Running == kver {
return "", "the box already runs " + kver
}
reps, _ := s.Store.OSReportsDesc(hostID, LayerKernel, 100)
for _, r := range reps {
var kr Report
if json.Unmarshal([]byte(r.ReportJSON), &kr) != nil || kernelOf(kr) != kver {
continue
}
if kernelEnded[r.Outcome] {
return "", fmt.Sprintf("the kernel step to %s ended %s at %s — the operator decides", kver, r.Outcome,
r.ReceivedAt.UTC().Format("2006-01-02 15:04"))
}
if r.Outcome == "refused" && kernelRefusedForGood[refusedCode(kr.Refused)] {
return "", fmt.Sprintf("the kernel step to %s was refused (%s) — the operator decides", kver, refusedCode(kr.Refused))
}
}
return kver, ""
}
// KernelBlockFor is the box's kernel instruction (nil: not due).
func (s *Service) KernelBlockFor(hostID string) *KernelBlock {
kver, _ := s.KernelDue(hostID)
if kver == "" {
return nil
}
b := &KernelBlock{Kver: kver}
ns, _ := s.Store.KernelNotices(hostID, kver)
if len(ns) > 0 {
last := ns[len(ns)-1].SentAt
if s.now().Sub(last) < KernelNoticeValid {
b.Tonight, b.NotifiedAt = true, last.UTC().Format(time.RFC3339)
}
}
return b
}
// KernelNotify tells each due box's household, the day before (`09` §3 decision 172). Called every minute; it acts only
// between KernelNoticeFromHour and KernelNoticeToHour Budapest time. Returns the boxes it told.
func (s *Service) KernelNotify() []string {
now := s.now()
lt := now.In(budapestLoc())
if lt.Hour() < KernelNoticeFromHour || lt.Hour() >= KernelNoticeToHour {
return nil
}
hosts, err := s.Store.ListHosts()
if err != nil {
return nil
}
var told []string
for _, h := range hosts {
kver, _ := s.KernelDue(h.HostID)
if kver == "" {
continue
}
ns, _ := s.Store.KernelNotices(h.HostID, kver)
if len(ns) > 0 && now.Sub(ns[len(ns)-1].SentAt) < KernelNoticeGap {
continue
}
if len(ns) >= KernelNoticeMax {
key := "kernel_notice_max:" + h.HostID + ":" + kver
if s.Store.OSAlarmRaised(key).IsZero() {
_ = s.Store.SetOSAlarmRaised(key, now)
s.event(h.CustomerID, EventKernelNotice, "warning", fmt.Sprintf("Kernel %s on %s: the household was told %d times "+
"and no kernel step ran — no further mail; the operator decides (the System page shows why).", kver, h.HostID, len(ns)),
map[string]any{"host_id": h.HostID, "kver": kver, "notices": len(ns)})
}
continue
}
if s.KernelMail == nil {
continue
}
lang, err := s.KernelMail(h.CustomerID, kver)
if err != nil {
key := "kernel_notice_failed:" + h.HostID + ":" + kver + ":" + lt.Format("2006-01-02")
if s.Store.OSAlarmRaised(key).IsZero() {
_ = s.Store.SetOSAlarmRaised(key, now)
s.event(h.CustomerID, EventKernelNotice, "warning", fmt.Sprintf("Kernel %s on %s: the household could NOT be told "+
"(%v) — so no kernel step tonight (no mail, no step).", kver, h.HostID, err),
map[string]any{"host_id": h.HostID, "kver": kver, "error": err.Error()})
}
continue
}
if err := s.Store.SaveKernelNotice(store.KernelNotice{HostID: h.HostID, Kver: kver, SentAt: now, Lang: lang}); err != nil {
s.logf("[ERROR] osupdates: kernel notice for %s sent but NOT recorded (%v) — no step tonight", h.HostID, err)
continue
}
told = append(told, h.HostID)
s.logf("[INFO] osupdates: kernel %s — the household of %s was told the box restarts tonight (lang=%s)", kver, h.HostID, lang)
s.event(h.CustomerID, EventKernelNotice, "info", fmt.Sprintf("Kernel %s on %s: the household was told the box "+
"restarts tonight (mail %d of at most %d).", kver, h.HostID, len(ns)+1, KernelNoticeMax),
map[string]any{"host_id": h.HostID, "kver": kver, "lang": lang})
if s.Bump != nil {
s.Bump(h.HostID)
}
}
sort.Strings(told)
return told
}
// kernelIngest raises the kernel layer's operator events (called by Ingest for layer "kernel").
func (s *Service) kernelIngest(hostID, customerID string, r Report, details map[string]any) {
var v struct {
From, To, Reason string
}
_ = json.Unmarshal(r.Kernel, &v)
to := kernelOf(r)
details["kver"] = to
switch r.Outcome {
case "staged":
s.event(customerID, EventKernelStep, "info", fmt.Sprintf("Kernel %s staged on %s (trigger %s): installed, never the "+
"default; the box boots it ONCE at its night reboot.", to, hostID, r.Trigger), details)
case "applied":
s.event(customerID, EventKernelStep, "info", fmt.Sprintf("Kernel %s booted healthily on %s and is the default now "+
"(was %s). %s", to, hostID, v.From, r.HealthReason), details)
s.event(customerID, EventApplied, "info", "System security fixes installed on the box's kernel (the box restarted at night).", details)
case "fell_back":
s.event(customerID, EventKernelStep, "error", fmt.Sprintf("Kernel %s did NOT come up on %s: the box came back on %s "+
"by itself. %s is installed but never the default; the operator decides.", to, hostID, v.From, to), details)
case "health_failed":
s.event(customerID, EventKernelStep, "error", fmt.Sprintf("Kernel %s booted on %s but is NOT healthy (%s): the box "+
"restarts ONCE into %s by itself.", to, hostID, r.HealthReason, v.From), details)
case "self_reverted":
s.event(customerID, EventKernelStep, "error", fmt.Sprintf("Kernel %s on %s: back on %s after the self-revert (%s). "+
"The operator decides.", to, hostID, v.From, r.HealthReason), details)
case "revert_failed":
s.event(customerID, EventKernelStep, "critical", fmt.Sprintf("Kernel %s on %s: the self-revert did NOT bring back %s "+
"(%s). No second revert — look at the box.", to, hostID, v.From, r.HealthReason), details)
case "refused", "failed":
s.event(customerID, EventFailed, "error", fmt.Sprintf("Kernel step on %s %s: %s", hostID, r.Outcome,
strings.TrimSpace(string(r.Refused)+" "+r.HealthReason)), details)
}
}
// kernelSet is the approved set for a kernel version: the series meta-package and the signed image.
func kernelSet(kver string) []Package {
m := kverRE.FindStringSubmatch(kver)
if m == nil {
return nil
}
v := strings.TrimSuffix(kver, "-pve")
return []Package{{Name: "proxmox-kernel-" + m[1], Version: v, Origin: "Proxmox Debian Repository"},
{Name: "proxmox-kernel-" + kver + "-signed", Version: v, Origin: "Proxmox Debian Repository"}}
}
// KernelStatus is the kernel set ring 0 booted and whether the operator's button may approve it: every ring-0 box's
// newest ended kernel step is "applied" (a healthy one-shot boot made it the default) for the SAME kernel, after a
// night stage; none fell back or reverted.
func (s *Service) KernelStatus() (Status, error) {
st := Status{Layer: LayerKernel}
ring0, err := s.ring0Hosts()
if err != nil || len(ring0) == 0 {
st.Waiting = "no ring-0 box"
return st, err
}
kver := ""
for _, h := range ring0 {
reps, err := s.Store.OSReportsDesc(h, LayerKernel, 100)
if err != nil {
return st, err
}
var applied Report
var appliedAt time.Time
for _, r := range reps {
var kr Report
if json.Unmarshal([]byte(r.ReportJSON), &kr) != nil {
continue
}
if kernelEnded[r.Outcome] {
if r.Outcome != "applied" || !r.Healthy {
st.Waiting = fmt.Sprintf("%s: the kernel step to %s ended %s", h, kernelOf(kr), r.Outcome)
return st, nil
}
applied, appliedAt = kr, r.ReceivedAt
break
}
}
k := kernelOf(applied)
if k == "" {
st.Waiting = h + " has not booted a new kernel healthily yet"
return st, nil
}
night := false
for _, r := range reps {
var kr Report
if json.Unmarshal([]byte(r.ReportJSON), &kr) == nil && r.Outcome == "staged" && kr.Trigger == "night" &&
kernelOf(kr) == k && !r.ReceivedAt.After(appliedAt) {
night = true
break
}
}
if !night {
st.Waiting = fmt.Sprintf("%s booted %s healthily, but not after a night step", h, k)
return st, nil
}
if kver != "" && k != kver {
st.Waiting = fmt.Sprintf("the ring-0 boxes booted different kernels (%s, %s)", kver, k)
return st, nil
}
kver = k
}
set := kernelSet(kver)
c := map[string]Package{}
for _, p := range set {
c[p.Name] = p
}
fp, list := fingerprint(LayerKernel, c)
pj, _ := json.Marshal(list)
first, err := s.Store.OSCandidateFirstSeen(fp, string(pj), s.now())
if err != nil {
return st, err
}
st.Fingerprint, st.FirstSeen, st.Packages = fp, first, len(list)
if rel, _ := s.Store.LatestOSRelease(LayerKernel); rel != nil && rel.Fingerprint == fp {
st.Approved, st.Waiting = rel.ID, "already approved"
}
return st, nil
}
// ApproveKernel is the operator's "Approve kernel set" (`09` §3 decision 172). A ring-1 box takes it only through a
// signed os_kernel_step — approval alone installs nothing and restarts nothing.
func (s *Service) ApproveKernel() (string, error) {
st, err := s.KernelStatus()
if err != nil {
return "", err
}
if st.Waiting != "" || st.Fingerprint == "" {
return "", fmt.Errorf("osupdates: the kernel set cannot be approved yet: %s", st.Waiting)
}
var list []Package
_ = json.Unmarshal([]byte(s.candidatePackages(st.Fingerprint)), &list)
if err := s.approve(LayerKernel, st.Fingerprint, list, "operator"); err != nil {
return "", err
}
rel, _ := s.Store.LatestOSRelease(LayerKernel)
return rel.ID, nil
}
func (s *Service) candidatePackages(fp string) string {
pj, _ := s.Store.OSCandidatePackages(fp)
return pj
}
// KernelLine is one box's kernel view for the System page.
type KernelLine struct {
Due, Why string // the kernel it is due to step to, or why none
LastOutcome string
LastKernel string
LastAt time.Time
LastReason string
NoticeAt time.Time // the newest day-before mail (any kernel)
NoticeKver string
Tonight bool
}
// KernelLineFor reads one box's kernel line.
func (s *Service) KernelLineFor(hostID string) KernelLine {
var l KernelLine
l.Due, l.Why = s.KernelDue(hostID)
if b := s.KernelBlockFor(hostID); b != nil {
l.Tonight = b.Tonight
}
if rep, _ := s.Store.LatestOSReport(hostID, LayerKernel); rep != nil {
var kr Report
_ = json.Unmarshal([]byte(rep.ReportJSON), &kr)
l.LastOutcome, l.LastKernel, l.LastAt, l.LastReason = rep.Outcome, kernelOf(kr), rep.ReceivedAt, kr.HealthReason
if l.LastReason == "" && len(kr.Refused) > 0 {
l.LastReason = string(kr.Refused)
}
}
if n, _ := s.Store.LatestKernelNotice(hostID); n != nil {
l.NoticeAt, l.NoticeKver = n.SentAt, n.Kver
}
return l
}
hub/internal/osupdates/kernel_test.go
+284
View File
@@ -0,0 +1,284 @@
package osupdates
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// R-836, `09` §3 decision 172: the kernel lane's hub half — due boxes, the day-before mail (no mail, no step), the
// block, the events, the operator's approval.
const kOld, kNew = "7.0.2-6-pve", "7.0.14-22-pve"
// laneFacts stores a host report whose system facts carry the kernel lane.
func (f *fix) laneFacts(t *testing.T, host, cust, running, phase, to string, problems ...string) {
t.Helper()
lane := map[string]any{"running": running, "default": running, "phase": phase, "to": to, "from": kOld}
if len(problems) > 0 {
lane["setup_problems"] = problems
}
body, _ := json.Marshal(map[string]any{"system": map[string]any{"facts": map[string]any{"host": map[string]any{
"kernel_running": running, "kernel_lane": lane}}}})
if err := f.s.Store.SaveHostReport(host, cust, body, store.HostReportDenorm{AgentVersion: "0.152.0"}); err != nil {
t.Fatal(err)
}
}
// hostWithPendingKernel reports a host step whose pending list carries the kernel meta-package upgrade.
func (f *fix) hostWithPendingKernel(t *testing.T, host string) {
f.ingest(t, host, Report{Layer: LayerHost, Trigger: "night", Mode: "apply", Outcome: "nothing", Healthy: true,
Pending: []PendingPkg{{Name: "proxmox-kernel-7.0", From: "7.0.2-6", To: "7.0.14-22", Origin: []string{"Proxmox Debian Repository"}},
{Name: "pve-manager", From: "9.2.2", To: "9.2.21", Origin: []string{"Proxmox Debian Repository"}}}})
}
func kreport(outcome, trigger string, healthy bool) Report {
return Report{Layer: LayerKernel, Trigger: trigger, Mode: "apply", Outcome: outcome, Healthy: healthy, ReleaseID: kNew,
Kernel: json.RawMessage(`{"from":"` + kOld + `","to":"` + kNew + `"}`)}
}
func (f *fix) mail(sent *[]string, fail error) {
f.s.KernelMail = func(cust, kver string) (string, error) {
if fail != nil {
return "", fail
}
*sent = append(*sent, cust+":"+kver)
return "hu", nil
}
}
func budapest(t *testing.T, h int) time.Time {
loc, err := time.LoadLocation("Europe/Budapest")
if err != nil {
t.Skip("no tzdata")
}
return time.Date(2026, 10, 8, h, 30, 0, 0, loc)
}
// A ring-0 box with a pending kernel is due; its block names the kernel, NOT tonight, until the household was mailed.
// COMPANION RED-PROOF (observed): set Tonight without a notice in KernelBlockFor → "tonight before any mail".
func TestKernel_DueButNotToldIsNotTonight(t *testing.T) {
f := newFix(t)
f.hostWithPendingKernel(t, "hp")
f.laneFacts(t, "hp", "c-hp", kOld, "none", "")
k, why := f.s.KernelDue("hp")
if k != kNew {
t.Fatalf("due = %q (%s)", k, why)
}
b := f.s.DesiredBlock("hp").Kernel
if b == nil || b.Kver != kNew || b.Tonight {
t.Fatalf("tonight before any mail: %+v", b)
}
}
// The mail goes out only 09:00–20:00 Budapest; then the block says tonight (for 24 h); never two within 20 h.
func TestKernel_DayBeforeMailMakesTonight(t *testing.T) {
f := newFix(t)
f.hostWithPendingKernel(t, "hp")
f.laneFacts(t, "hp", "c-hp", kOld, "none", "")
var sent []string
f.mail(&sent, nil)
f.now = budapest(t, 3) // night: no mail
if told := f.s.KernelNotify(); len(told) != 0 || len(sent) != 0 {
t.Fatalf("mailed at 03:30: %v", sent)
}
f.now = budapest(t, 21)
if f.s.KernelNotify(); len(sent) != 0 {
t.Fatalf("mailed at 21:30: %v", sent)
}
f.now = budapest(t, 10)
if told := f.s.KernelNotify(); len(told) != 1 || len(sent) != 1 || sent[0] != "c-hp:"+kNew {
t.Fatalf("told=%v sent=%v", told, sent)
}
if b := f.s.DesiredBlock("hp").Kernel; b == nil || !b.Tonight || b.NotifiedAt == "" {
t.Fatalf("after the mail the block must say tonight: %+v", b)
}
if !contains(f.bumps, "hp") || !contains(f.events, EventKernelNotice) {
t.Fatalf("the box must be bumped and the operator told: bumps=%v events=%v", f.bumps, f.events)
}
f.now = f.now.Add(5 * time.Hour)
if f.s.KernelNotify(); len(sent) != 1 {
t.Fatalf("a second mail within 20 h: %v", sent)
}
f.now = budapest(t, 10).Add(25 * time.Hour)
if b := f.s.DesiredBlock("hp").Kernel; b == nil || b.Tonight {
t.Fatalf("a mail older than 24 h no longer covers tonight: %+v", b)
}
}
// No mail → no step: a mail the service did not accept is not recorded; the block stays not-tonight; the operator hears.
// COMPANION RED-PROOF (observed): record the notice before checking the send error → "a failed mail made tonight".
func TestKernel_NoMailNoStep(t *testing.T) {
f := newFix(t)
f.hostWithPendingKernel(t, "hp")
f.laneFacts(t, "hp", "c-hp", kOld, "none", "")
var sent []string
f.mail(&sent, errors.New("the household has no e-mail address"))
f.now = budapest(t, 11)
f.s.KernelNotify()
if b := f.s.DesiredBlock("hp").Kernel; b == nil || b.Tonight {
t.Fatalf("a failed mail made tonight: %+v", b)
}
if !contains(f.events, EventKernelNotice) {
t.Fatalf("the operator must hear the household could not be told: %v", f.events)
}
f.s.KernelMail = nil
f.s.KernelNotify()
if b := f.s.DesiredBlock("hp").Kernel; b == nil || b.Tonight {
t.Fatalf("no mailer → never tonight: %+v", b)
}
}
// An ended step is never retried by itself; the operator decides.
func TestKernel_EndedStepIsNeverRetried(t *testing.T) {
for _, out := range []string{"applied", "fell_back", "health_failed", "self_reverted", "revert_failed"} {
f := newFix(t)
f.hostWithPendingKernel(t, "hp")
f.laneFacts(t, "hp", "c-hp", kOld, "fell_back", kNew)
f.ingest(t, "hp", kreport(out, "boot", out == "applied"))
if k, why := f.s.KernelDue("hp"); k != "" || !strings.Contains(why, "operator decides") {
t.Fatalf("%s: still due (%q %q)", out, k, why)
}
if f.s.DesiredBlock("hp").Kernel != nil {
t.Fatalf("%s: block still names a kernel", out)
}
}
// a refusal for good ends it; a passing one (a lock) does not
f := newFix(t)
f.hostWithPendingKernel(t, "hp")
f.laneFacts(t, "hp", "c-hp", kOld, "none", "")
r := kreport("refused", "night", false)
r.Refused = json.RawMessage(`{"code":"R9","reason":"another apt holds the lock"}`)
f.ingest(t, "hp", r)
if k, _ := f.s.KernelDue("hp"); k != kNew {
t.Fatal("a transient refusal (R9) must not end the step")
}
r.Refused = json.RawMessage(`{"code":"R20","reason":"/boot/efi is not vfat"}`)
f.ingest(t, "hp", r)
if k, _ := f.s.KernelDue("hp"); k != "" {
t.Fatal("R20 must end the step")
}
}
// Ring 1 is due ONLY for a kernel a signed job staged; a box with a setup problem or switched off never is.
func TestKernel_Ring1OnlyWhenStagedAndSetupMustBeFine(t *testing.T) {
f := newFix(t)
f.hostWithPendingKernel(t, "cust1")
f.laneFacts(t, "cust1", "c-1", kOld, "none", "")
if k, _ := f.s.KernelDue("cust1"); k != "" {
t.Fatal("ring 1 due without a staged kernel")
}
f.laneFacts(t, "cust1", "c-1", kOld, "staged", kNew)
if k, why := f.s.KernelDue("cust1"); k != kNew {
t.Fatalf("ring 1 with a staged kernel not due: %s", why)
}
f.laneFacts(t, "cust1", "c-1", kOld, "staged", kNew, "/boot/efi is not a mounted vfat ESP")
if k, _ := f.s.KernelDue("cust1"); k != "" {
t.Fatal("a box that cannot do a one-shot is never due")
}
_ = f.s.Store.SetOSEnabled("hp", false)
f.hostWithPendingKernel(t, "hp")
f.laneFacts(t, "hp", "c-hp", kOld, "none", "")
if k, _ := f.s.KernelDue("hp"); k != "" {
t.Fatal("switched off but due")
}
}
// The household is told at most KernelNoticeMax times for one kernel; then the operator hears once.
func TestKernel_AtMostThreeMails(t *testing.T) {
f := newFix(t)
f.hostWithPendingKernel(t, "hp")
f.laneFacts(t, "hp", "c-hp", kOld, "none", "")
var sent []string
f.mail(&sent, nil)
day := budapest(t, 10)
for i := 0; i < 5; i++ {
f.now = day.Add(time.Duration(i) * 24 * time.Hour)
f.s.KernelNotify()
}
if len(sent) != KernelNoticeMax {
t.Fatalf("mails = %d, want %d", len(sent), KernelNoticeMax)
}
n := 0
for _, e := range f.events {
if e == EventKernelNotice {
n++
}
}
if n != KernelNoticeMax+1 {
t.Fatalf("notice events = %d (3 told + 1 'no further mail')", n)
}
}
// Each outcome is an operator event; "applied" also gives the household its timeline line; ingest never treats a
// kernel report as a guest report.
func TestKernel_IngestEvents(t *testing.T) {
f := newFix(t)
f.ingest(t, "hp", kreport("staged", "night", true))
f.ingest(t, "hp", kreport("applied", "boot", true))
f.ingest(t, "hp", kreport("fell_back", "boot", false))
if c := countOf(f.events, EventKernelStep); c != 3 {
t.Fatalf("kernel events = %d: %v", c, f.events)
}
if !contains(f.events, EventApplied) {
t.Fatal("an applied kernel gives the household its line")
}
if rep, _ := f.s.Store.LatestOSReport("hp", LayerGuest); rep != nil {
t.Fatalf("a kernel report was stored as a guest report: %+v", rep)
}
if rep, _ := f.s.Store.LatestOSReport("hp", LayerKernel); rep == nil || rep.Outcome != "fell_back" {
t.Fatalf("kernel report not stored on its layer: %+v", rep)
}
}
// The operator's approval: every ring-0 box booted the same kernel healthily after a night stage — never before.
// COMPANION RED-PROOF (observed): drop the `night` check in KernelStatus → "approved without a night stage".
func TestKernel_ApproveNeedsEveryRing0BoxHealthyAfterANightStep(t *testing.T) {
f := newFix(t)
if _, err := f.s.ApproveKernel(); err == nil {
t.Fatal("approved with nothing booted")
}
f.ingest(t, "hp", kreport("staged", "night", true))
f.ingest(t, "hp", kreport("applied", "boot", true))
f.ingest(t, "n100", kreport("staged", "signed", true)) // a by-day stage is not a night step
f.ingest(t, "n100", kreport("applied", "boot", true))
if _, err := f.s.ApproveKernel(); err == nil || !strings.Contains(err.Error(), "night") {
t.Fatalf("approved without a night stage: %v", err)
}
f.ingest(t, "n100", kreport("staged", "night", true))
f.ingest(t, "n100", kreport("applied", "boot", true))
id, err := f.s.ApproveKernel()
if err != nil || !strings.HasPrefix(id, "os-kernel-") {
t.Fatalf("%q %v", id, err)
}
rel, _ := f.s.Store.LatestOSRelease(LayerKernel)
if !strings.Contains(rel.PackagesJSON, `"proxmox-kernel-7.0"`) || !strings.Contains(rel.PackagesJSON, "proxmox-kernel-7.0.14-22-pve-signed") {
t.Fatalf("release = %s", rel.PackagesJSON)
}
if contains(f.bumps, "cust1") {
t.Fatal("an approved kernel set must nudge no ring-1 box (a signed job only)")
}
// a fell-back ring-0 box blocks the button
g := newFix(t)
g.ingest(t, "hp", kreport("staged", "night", true))
g.ingest(t, "hp", kreport("fell_back", "boot", false))
if st, _ := g.s.KernelStatus(); !strings.Contains(st.Waiting, "fell_back") {
t.Fatalf("waiting = %q", st.Waiting)
}
}
func contains(xs []string, x string) bool { return countOf(xs, x) > 0 }
func countOf(xs []string, x string) int {
n := 0
for _, v := range xs {
if v == x {
n++
}
}
return n
}
hub/internal/osupdates/service.go
+20 -4
View File
@@ -47,7 +47,7 @@ const (
var Layers = []string{LayerGuest, LayerHost}
// AllLayers adds the Docker engine set (approved only by the operator's button, ApproveDocker).
var AllLayers = []string{LayerGuest, LayerHost, LayerDocker, LayerPVE}
var AllLayers = []string{LayerGuest, LayerHost, LayerDocker, LayerPVE, LayerKernel}
// dockerNames are the six packages of the Docker engine set (the agent wrapper's DOCKER_NAMES).
var dockerNames = map[string]bool{"containerd.io": true, "docker-buildx-plugin": true, "docker-ce": true,
@@ -117,6 +117,8 @@ type Report struct {
// OOMCheck (agent v0.150.0, decision 157): the Docker step's memory-kill check, kept verbatim — oomCheckWaiting
// reads it from the stored report.
OOMCheck json.RawMessage `json:"oom_check,omitempty"`
// Kernel (agent v0.152.0, R-836): the kernel layer's view {running, default, flag, phase, from, to, …}, kept verbatim.
Kernel json.RawMessage `json:"kernel,omitempty"`
}
// PendingPkg is one update the sources offer.
@@ -156,6 +158,7 @@ type Block struct {
Enabled bool `json:"enabled"`
Release *ReleaseBlock `json:"release,omitempty"`
HostRelease *ReleaseBlock `json:"host_release,omitempty"`
Kernel *KernelBlock `json:"kernel,omitempty"` // R-836 (hub v0.143.0)
}
// ReleaseBlock is a layer's newest approved release, for ring 1.
@@ -205,6 +208,9 @@ type Service struct {
// TestOverride names the TEST overrides active at start ("" = none, the ruled waits). Every approval made while it
// is set carries the `test` mark; CancelTestReleases cancels them at a start without it (`11` §5.3.1).
TestOverride string
// KernelMail sends a household the day-before kernel mail (R-836, `09` §3 decision 172) and returns the language
// it used; an error means the mail service did not accept it — then there is no step. nil = no mail, no step.
KernelMail func(customerID, kver string) (lang string, err error)
}
func (s *Service) now() time.Time {
@@ -244,7 +250,7 @@ func (s *Service) event(customerID, typ, sev, msg string, details any) {
func layerOf(r Report) string {
switch r.Layer {
case LayerHost, LayerDocker, LayerPVE:
case LayerHost, LayerDocker, LayerPVE, LayerKernel:
return r.Layer
}
return LayerGuest
@@ -267,6 +273,10 @@ func (s *Service) Ingest(hostID string, r Report) error {
hostID, layer, r.RunID, r.Ring, r.Mode, r.Outcome, r.Healthy, len(r.Upgraded), len(r.Pending), len(r.NotCovered), len(r.RestartNeeded), r.RebootNeeded, r.PassSeconds)
details := map[string]any{"host_id": hostID, "layer": layer, "run_id": r.RunID, "ring": r.Ring, "outcome": r.Outcome,
"upgraded": len(r.Upgraded), "release_id": r.ReleaseID, "health_reason": r.HealthReason}
if layer == LayerKernel {
s.kernelIngest(hostID, h.CustomerID, r, details) // R-836: its own outcomes and events (kernel.go)
return nil
}
where := "the box"
switch layer {
case LayerHost:
@@ -420,6 +430,10 @@ func (s *Service) Evaluate() ([]Status, error) {
if _, err := s.PVEStatus(); err != nil { // R-812 option A: stamped on the tick too, like the Docker set
return out, err
}
if _, err := s.KernelStatus(); err != nil { // R-836: stamped on the tick too
return out, err
}
s.KernelNotify() // R-836: the day-before mails (acts only 09:00–20:00 Budapest time)
return out, nil
}
@@ -527,7 +541,8 @@ func (s *Service) Candidates() []Status {
}
d, _ := s.DockerStatus()
p, _ := s.PVEStatus()
return append(out, d, p)
k, _ := s.KernelStatus()
return append(out, d, p, k)
}
// BundleThreshold is the config-bundle alarm's wait (the System page turns the cell red at it).
@@ -776,7 +791,7 @@ func (s *Service) approve(layer, fp string, list []Package, by string) error {
s.logf("[WARN] osupdates: OS release %s (%s) APPROVED by %s (%d packages, fingerprint %s)%s", id, layer, by, len(list), fp, mark)
s.event("", EventReleaseApprove, "info", fmt.Sprintf("OS release %s (%s) approved (%s, %d packages)%s.", id, layer, by, len(list), mark),
map[string]any{"release_id": id, "layer": layer, "approved_by": by, "packages": len(list), "fingerprint": fp, "test": test})
if s.Bump != nil && layer != LayerDocker && layer != LayerPVE { // the slow-lane sets reach ring 1 only by a signed job
if s.Bump != nil && layer != LayerDocker && layer != LayerPVE && layer != LayerKernel { // the slow-lane sets reach ring 1 only by a signed job
hosts, _ := s.Store.ListHosts()
for _, h := range hosts {
if st := s.Store.GetOSHostSettings(h.HostID); st.Ring == 1 && st.Enabled {
@@ -845,6 +860,7 @@ func (s *Service) DesiredBlock(hostID string) Block {
if st.Ring == 1 {
b.Release, b.HostRelease = s.releaseBlock(LayerGuest), s.releaseBlock(LayerHost)
}
b.Kernel = s.KernelBlockFor(hostID) // R-836: nil unless a kernel is due; tonight only after the day-before mail
return b
}
hub/internal/store/os_updates.go
+71
View File
@@ -70,9 +70,70 @@ func (s *Store) migrateOSUpdates() error {
}
s.db.Exec(`ALTER TABLE os_releases ADD COLUMN cancelled_at TEXT NOT NULL DEFAULT ''`)
s.db.Exec(`ALTER TABLE os_releases ADD COLUMN cancel_reason TEXT NOT NULL DEFAULT ''`)
// The kernel lane (hub v0.143.0, R-836, `09` §3 decision 172): every mail that told a household "the box restarts
// tonight". A kernel step runs only on a night such a mail announced — no mail, no step.
if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS os_kernel_notices (
id INTEGER PRIMARY KEY AUTOINCREMENT,
host_id TEXT NOT NULL,
kver TEXT NOT NULL,
sent_at DATETIME NOT NULL,
lang TEXT NOT NULL DEFAULT ''
);
CREATE INDEX IF NOT EXISTS idx_os_kernel_notices ON os_kernel_notices(host_id, kver, id);`); err != nil {
return err
}
return nil
}
// KernelNotice is one "the box restarts tonight" mail that reached the mail service.
type KernelNotice struct {
HostID, Kver, Lang string
SentAt time.Time
}
// SaveKernelNotice records a sent notice (only after the mail service accepted it).
func (s *Store) SaveKernelNotice(n KernelNotice) error {
_, err := s.db.Exec(`INSERT INTO os_kernel_notices (host_id, kver, sent_at, lang) VALUES (?, ?, ?, ?)`,
n.HostID, n.Kver, n.SentAt.UTC().Format("2006-01-02 15:04:05"), n.Lang)
return err
}
// KernelNotices lists the notices for one box and kernel, oldest first.
func (s *Store) KernelNotices(hostID, kver string) ([]KernelNotice, error) {
rows, err := s.db.Query(`SELECT host_id, kver, sent_at, lang FROM os_kernel_notices WHERE host_id = ? AND kver = ? ORDER BY id`, hostID, kver)
if err != nil {
return nil, err
}
defer rows.Close()
var out []KernelNotice
for rows.Next() {
var n KernelNotice
var at string
if err := rows.Scan(&n.HostID, &n.Kver, &at, &n.Lang); err != nil {
return nil, err
}
n.SentAt = parseSQLiteTime(at)
out = append(out, n)
}
return out, rows.Err()
}
// LatestKernelNotice is the newest notice for a box (any kernel); nil when there was none.
func (s *Store) LatestKernelNotice(hostID string) (*KernelNotice, error) {
var n KernelNotice
var at string
err := s.db.QueryRow(`SELECT host_id, kver, sent_at, lang FROM os_kernel_notices WHERE host_id = ? ORDER BY id DESC LIMIT 1`, hostID).
Scan(&n.HostID, &n.Kver, &at, &n.Lang)
if err == sql.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, err
}
n.SentAt = parseSQLiteTime(at)
return &n, nil
}
// hasColumn reports whether table has the column (PRAGMA table_info).
func (s *Store) hasColumn(table, col string) bool {
rows, err := s.db.Query(`SELECT name FROM pragma_table_info(?)`, table)
@@ -216,6 +277,16 @@ func (s *Store) OSCandidateFirstSeen(fingerprint, packagesJSON string, now time.
return parseSQLiteTime(at), nil
}
// OSCandidatePackages is the package list a candidate fingerprint was first seen with ("" when unknown).
func (s *Store) OSCandidatePackages(fingerprint string) (string, error) {
var pj string
err := s.db.QueryRow(`SELECT packages_json FROM os_candidates WHERE fingerprint = ?`, fingerprint).Scan(&pj)
if err == sql.ErrNoRows {
return "", nil
}
return pj, err
}
// OSRelease is one approved version set.
type OSRelease struct {
ID string
hub/internal/sysfacts/sysfacts.go
+17
View File
@@ -39,6 +39,23 @@ type Host struct {
OopsThisBoot *bool `json:"oops_this_boot"`
KernelPanic *int `json:"kernel_panic"`
CrashGuard *CrashGuard `json:"crash_guard"`
KernelLane *KernelLane `json:"kernel_lane"` // agent ≥ v0.152.0 (R-836); nil = an older agent
}
// KernelLane is the box's kernel-lane state (agent v0.152.0, R-836, `11` §5.11), read by the wrapper from grub.cfg, the
// ESP flag and its own step record. Default is the kernel GRUB boots normally; Flag (non-empty) is a kernel that boots
// ONCE at the next boot; Phase is the step's: none | staged | oneshot | judging | good | fell_back | reverting |
// self_reverted | revert_failed | cancelled.
type KernelLane struct {
Running string `json:"running"`
Default string `json:"default"`
Flag *string `json:"flag"`
Phase string `json:"phase"`
From string `json:"from"`
To string `json:"to"`
Reason string `json:"reason"`
SetupProblems []string `json:"setup_problems"`
Error string `json:"error"`
}
// Guest is the customer guest's half.
hub/internal/web/os_updates.go
+9
View File
@@ -79,6 +79,15 @@ func (s *Server) handleOSAdmin(w http.ResponseWriter, r *http.Request, path stri
}
id, err := view.ApprovePVE()
reply(map[string]string{"release_id": id}, err)
case r.Method == http.MethodPost && path == "/os/approve-kernel":
// R-836 (`09` §3 decision 172): the operator approves the kernel set every ring-0 box booted healthily.
view, ok := s.osUpdates.(OSSystemView)
if !ok {
reply(nil, fmt.Errorf("kernel approval not available"))
return
}
id, err := view.ApproveKernel()
reply(map[string]string{"release_id": id}, err)
default:
http.Error(w, "not found", http.StatusNotFound)
}
hub/internal/web/r135_csrf_test.go
+1
View File
@@ -60,6 +60,7 @@ var r135PostRoutes = []string{
"/os/approve-now",
"/os/approve-docker",
"/os/approve-pve",
"/os/approve-kernel",
// Not a route: the gate must refuse BEFORE routing, so even an unknown path is 403, never 404.
"/no-such-route",
}
hub/internal/web/system.go
+50
View File
@@ -36,6 +36,7 @@ type systemRow struct {
HostRelease, HostPending, HostNotCovered cell
Held, RebootSince, KernelPanic, Oops cell
CrashRestarts24h, Guard cell
KernelDefault, KernelStep cell // R-836: the GRUB default kernel; the kernel lane's step
Bundle cell // R-840: the root-owned config bundle
Agent cell // R-530: the box's agent against the vouched one
// guest
@@ -58,6 +59,54 @@ type OSSystemView interface {
AgentThreshold() time.Duration
ApproveDocker() (string, error)
ApprovePVE() (string, error) // R-812 option A: the Proxmox package set
ApproveKernel() (string, error) // R-836: the kernel set
KernelLineFor(hostID string) osupdates.KernelLine // R-836: one box's kernel step and day-before mail
}
// kernelCells are the System page's two kernel-lane cells (R-836, `11` §5.11): the GRUB default (amber while a one-shot
// flag names another kernel for the next boot), and the step — the newest result, the kernel the box is due, whether
// the household was told for tonight. "unknown" when the box reports no kernel lane (an older agent).
func kernelCells(f sysfacts.System, kl osupdates.KernelLine, now time.Time) (dflt, step cell) {
lane := f.Host.KernelLane
if lane == nil {
if kl.LastOutcome == "" {
return unknownCell(""), cell{Text: "—", Title: "the box reports no kernel lane (agent older than v0.152.0)"}
}
lane = &sysfacts.KernelLane{} // a step result without facts (an older facts read) still shows
}
dflt = unknownCell(lane.Default)
if lane.Flag != nil && *lane.Flag != "" {
dflt.Text += " (once: " + *lane.Flag + ")"
dflt.Class, dflt.Title = "warn", "the next boot runs "+*lane.Flag+" ONCE; the boot after it the default again"
}
var parts []string
if kl.LastOutcome != "" {
parts = append(parts, fmt.Sprintf("%s %s %s", kl.LastKernel, kl.LastOutcome, ago(kl.LastAt, now)))
}
switch {
case kl.Due != "" && kl.Tonight:
parts = append(parts, "due "+kl.Due+" — household told "+ago(kl.NoticeAt, now)+": TONIGHT")
case kl.Due != "":
parts = append(parts, "due "+kl.Due+" — not told yet (mails 09–20 h)")
}
if lane.Phase != "" && lane.Phase != "none" {
parts = append(parts, "phase "+lane.Phase)
}
if len(parts) == 0 {
parts = append(parts, "—")
}
step = cell{Text: strings.Join(parts, " · "), Title: strings.TrimSpace(kl.Why + " " + kl.LastReason)}
switch kl.LastOutcome {
case "fell_back", "health_failed", "self_reverted", "refused", "failed":
step.Class = "warn"
case "revert_failed":
step.Class = "bad"
}
if len(lane.SetupProblems) > 0 {
step.Class = "warn"
step.Title = "cannot do a one-shot boot: " + strings.Join(lane.SetupProblems, "; ")
}
return dflt, step
}
// agentCell is the "Agent" cell (R-530, hub v0.135.0): the box's agent against the vouched one, how far behind and
@@ -375,6 +424,7 @@ func (s *Server) handleSystem(w http.ResponseWriter, r *http.Request) {
agents[h.HostID] = h.AgentVersion
}
for i := range rows {
rows[i].KernelDefault, rows[i].KernelStep = kernelCells(facts[rows[i].HostID], view.KernelLineFor(rows[i].HostID), time.Now())
rows[i].Bundle = bundleCell(facts[rows[i].HostID], man.AgentVersion, man.BundleSHA256,
s.store.BundleBehindSince(rows[i].HostID), view.BundleThreshold(), time.Now())
rows[i].Agent = agentCell(agents[rows[i].HostID], man.AgentVersion,
hub/internal/web/system_test.go
+59
View File
@@ -12,6 +12,7 @@ import (
"gitea.dooplex.hu/admin/felhom-hub/internal/osupdates"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
"gitea.dooplex.hu/admin/felhom-hub/internal/sysfacts"
"golang.org/x/crypto/bcrypt"
)
@@ -213,3 +214,61 @@ func TestSystemPage_PVEButtonOnlyWhenReady(t *testing.T) {
t.Fatal("button missing after two healthy nights")
}
}
// R-836: the "Approve kernel set" button appears ONLY when every ring-0 box booted the kernel healthily after a night
// step (one render test per branch of the gate), and the two kernel cells render the lane — "—" for an older agent.
//
// COMPANION RED-PROOF (observed): drop the readiness gate (`.Fingerprint` and `(eq .Waiting "")` — KernelStatus sets the
// fingerprint only when ready, so each alone suffices) → "button shown with no kernel booted".
func TestSystemPage_KernelButtonOnlyWhenReady(t *testing.T) {
s, st, svc := systemServer(t)
if strings.Contains(getSystem(t, s), `action="/os/approve-kernel"`) {
t.Fatal("button shown with no kernel booted")
}
_ = st.SetOSRing("full-1", 0)
k := func(outcome, trigger string) {
r := osupdates.Report{RunID: time.Now().String() + outcome, Layer: "kernel", Trigger: trigger, Mode: "apply",
Outcome: outcome, Healthy: outcome != "fell_back", ReleaseID: "7.0.14-22-pve",
Kernel: []byte(`{"from":"7.0.2-6-pve","to":"7.0.14-22-pve"}`)}
if err := svc.Ingest("full-1", r); err != nil {
t.Fatal(err)
}
}
k("staged", "night")
if strings.Contains(getSystem(t, s), `action="/os/approve-kernel"`) {
t.Fatal("button shown before a healthy night boot")
}
k("applied", "boot")
b := getSystem(t, s)
if !strings.Contains(b, `action="/os/approve-kernel"`) {
t.Fatal("button missing after a healthy boot following a night step")
}
if !strings.Contains(b, "7.0.14-22-pve applied") {
t.Fatalf("the kernel step cell does not show the newest result:\n%s", b[strings.Index(b, "<tbody>"):min(len(b), strings.Index(b, "<tbody>")+3000)])
}
}
func TestSystemPage_KernelCells(t *testing.T) {
flag := "7.0.14-22-pve"
f := sysfacts.System{Present: true, Host: sysfacts.Host{KernelLane: &sysfacts.KernelLane{Running: "7.0.2-6-pve",
Default: "7.0.2-6-pve", Flag: &flag, Phase: "staged", To: "7.0.14-22-pve"}}}
now := time.Date(2026, 10, 8, 3, 0, 0, 0, time.UTC)
d, step := kernelCells(f, osupdates.KernelLine{Due: "7.0.14-22-pve", Tonight: true, NoticeAt: now.Add(-14 * time.Hour)}, now)
if d.Class != "warn" || !strings.Contains(d.Text, "once: 7.0.14-22-pve") {
t.Fatalf("default cell = %+v", d)
}
if !strings.Contains(step.Text, "TONIGHT") || !strings.Contains(step.Text, "phase staged") {
t.Fatalf("step cell = %+v", step)
}
_, step = kernelCells(f, osupdates.KernelLine{Due: "7.0.14-22-pve"}, now)
if !strings.Contains(step.Text, "not told yet") {
t.Fatalf("not-told step cell = %+v", step)
}
_, step = kernelCells(f, osupdates.KernelLine{LastOutcome: "revert_failed", LastKernel: "7.0.14-22-pve", LastAt: now}, now)
if step.Class != "bad" {
t.Fatalf("a failed revert must be red: %+v", step)
}
if d, step := kernelCells(sysfacts.System{Present: true}, osupdates.KernelLine{}, now); d.Text != "unknown" || step.Text != "—" {
t.Fatalf("an older agent: %+v %+v", d, step)
}
}
hub/internal/web/templates/system.html
+8 -3
View File
@@ -72,6 +72,11 @@
<input type="hidden" name="_csrf" value="{{$.CSRFToken}}"><input type="hidden" name="return" value="/system">
<button type="submit" class="btn btn-sm" data-confirm="Approve this Proxmox package set (no kernel)? Ring-1 boxes take it only through a signed operator job.">Approve Proxmox set</button>
</form>{{end}}
{{if and (eq .Layer "kernel") .Fingerprint (not .Approved) (eq .Waiting "")}}
<form method="POST" action="/os/approve-kernel" style="margin-top: 0.3rem;">
<input type="hidden" name="_csrf" value="{{$.CSRFToken}}"><input type="hidden" name="return" value="/system">
<button type="submit" class="btn btn-sm" data-confirm="Approve this kernel set? Every ring-0 box booted it healthily after a night step. Ring-1 boxes take it only through a signed os_kernel_step, and restart only on a night their household was told about.">Approve kernel set</button>
</form>{{end}}
</div>
{{end}}
</div>
@@ -108,12 +113,12 @@
<thead>
<tr>
<th>Box</th><th>Ring / updates</th><th>Tunnel</th>
<th class="grp">Proxmox</th><th>Kernel (running)</th><th>Kernel (next boot)</th><th>Debian</th><th>Felhom release</th><th>Pending</th><th>Not covered</th><th>Held</th><th>Reboot needed</th><th>kernel.panic</th><th>Oops</th><th>Crash restarts 24 h</th><th>Crash guard</th><th title="The root-owned config bundle: sudoers, wrappers, units (R-840)">Root files</th><th title="The box's agent against the vouched one (R-530). Agents update only by a per-box signed job.">Agent</th>
<th class="grp">Proxmox</th><th>Kernel (running)</th><th>Kernel (next boot)</th><th title="The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.">Kernel (default)</th><th title="The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).">Kernel step</th><th>Debian</th><th>Felhom release</th><th>Pending</th><th>Not covered</th><th>Held</th><th>Reboot needed</th><th>kernel.panic</th><th>Oops</th><th>Crash restarts 24 h</th><th>Crash guard</th><th title="The root-owned config bundle: sudoers, wrappers, units (R-840)">Root files</th><th title="The box's agent against the vouched one (R-530). Agents update only by a per-box signed job.">Agent</th>
<th class="grp">Guest Debian</th><th>Felhom release</th><th>Pending</th><th>Restart needed</th><th title="The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.">Last disk trim</th>
<th class="grp">Docker</th><th>containerd</th><th>live-restore</th><th>Docker release</th>
<th class="grp">Last OS leg</th>
</tr>
<tr class="text-muted"><th></th><th></th><th></th><th class="grp" colspan="15">host</th><th class="grp" colspan="5">guest</th><th class="grp" colspan="4">Docker engine</th><th class="grp"></th></tr>
<tr class="text-muted"><th></th><th></th><th></th><th class="grp" colspan="17">host</th><th class="grp" colspan="5">guest</th><th class="grp" colspan="4">Docker engine</th><th class="grp"></th></tr>
</thead>
<tbody>
{{range .Rows}}
@@ -136,7 +141,7 @@
</td>
{{template "sys_cell" .Tunnel}}
<td class="grp {{if .PVE.Class}}c-{{.PVE.Class}}{{end}}">{{.PVE.Text}}</td>
{{template "sys_cell" .KernelRunning}}{{template "sys_cell" .KernelNextBoot}}{{template "sys_cell" .HostDebian}}
{{template "sys_cell" .KernelRunning}}{{template "sys_cell" .KernelNextBoot}}{{template "sys_cell" .KernelDefault}}{{template "sys_cell" .KernelStep}}{{template "sys_cell" .HostDebian}}
{{template "sys_cell" .HostRelease}}{{template "sys_cell" .HostPending}}{{template "sys_cell" .HostNotCovered}}
{{template "sys_cell" .Held}}{{template "sys_cell" .RebootSince}}{{template "sys_cell" .KernelPanic}}{{template "sys_cell" .Oops}}
{{template "sys_cell" .CrashRestarts24h}}{{template "sys_cell" .Guard}}{{template "sys_cell" .Bundle}}{{template "sys_cell" .Agent}}
scripts/CHANGELOG.md
+9
View File
@@ -1,3 +1,12 @@
## installer — the uninstall knows the kernel lane's files (R-836; unreleased, lands with the next installer tag) (2026-10-07)
- `felhom-host-install.sh` uninstall: removes the two GRUB generators agent v0.152.0's bundle installs
(`/etc/grub.d/01_felhom_oneshot`, `/etc/grub.d/42_felhom_oneshot`), the one-shot env block's directory on the ESP
(`/boot/efi/EFI/felhom`) and `/var/lib/felhom-kernel`, then `update-grub`. It KEEPS
`/etc/default/grub.d/zz-felhom-kernel-default.cfg` and says so: it names the kernel the box booted healthily, and
without it GRUB would boot the newest installed kernel, which may be one that fell back. Found by
`test_bundle_list_is_current` (the frozen list now names both). SCRIPT_VERSION unchanged (no tag cut this session).
## gates — the ISO first-boot test is a gate, full runs only (R-502, `09` §3 decision 147) (2026-10-06)
- scripts: R-502 — new gate `iso-bootstrap` (scripts/iso_bootstrap_gate.py) runs the ISO first-boot harness in felhom-iso-assistant:trixie on FULL runs only (fast=False: never the pre-push hook, never CI); no docker/no image/docker error = exit 2 'not checked', never a pass; every green run is followed by a built-in decoy (a pairing banner that never paints) the harness must fail. Docker-free decoys in scripts/test_iso_bootstrap_gate.py, covered in test_gate_decoys.py (decision 147).
scripts/felhom-host-install.sh
+14
View File
@@ -1367,6 +1367,20 @@ run_uninstall() {
if [[ -e "$cgf" ]]; then run rm -f "$cgf"; fi
done
if [[ -d /var/lib/felhom-crash-guard ]]; then run rm -rf /var/lib/felhom-crash-guard; fi
# Agent v0.152.0 (R-836, the kernel lane): the two GRUB generators the bundle installs, the one-shot flag's env
# block on the ESP and the step record — then grub.cfg is regenerated without them. The GRUB default pin
# (/etc/default/grub.d/zz-felhom-kernel-default.cfg) is KEPT on purpose: it names the kernel this box booted
# healthily, and without it GRUB would boot the newest installed kernel, which may be one that fell back.
local kgf _grub_touched=false
for kgf in /etc/grub.d/01_felhom_oneshot /etc/grub.d/42_felhom_oneshot; do
if [[ -e "$kgf" ]]; then run rm -f "$kgf"; _grub_touched=true; fi
done
if [[ -d /boot/efi/EFI/felhom ]]; then run rm -rf /boot/efi/EFI/felhom; fi
if [[ -d /var/lib/felhom-kernel ]]; then run rm -rf /var/lib/felhom-kernel; fi
if $_grub_touched && command -v update-grub >/dev/null 2>&1; then run update-grub; fi
if [[ -f /etc/default/grub.d/zz-felhom-kernel-default.cfg ]]; then
log_info " kept /etc/default/grub.d/zz-felhom-kernel-default.cfg (the kernel this box booted healthily stays the default)"
fi
# 1.31.0 (R-840): the bundle's previous copies and the wrapper's nonce record.
if [[ -d /var/lib/felhom-os-apply ]]; then run rm -rf /var/lib/felhom-os-apply; fi
if [[ -f /var/lib/vz/snippets/felhom-guest-hook.sh ]]; then run rm -f /var/lib/vz/snippets/felhom-guest-hook.sh; fi
scripts/test_hostinstall.py
+2
View File
@@ -44,6 +44,8 @@ BUNDLE_DESTS = [
"/etc/systemd/system/felhom-crash-guard-check.service",
"/etc/systemd/system/felhom-crash-guard-check.timer",
"/etc/felhom/crash-guard.conf",
"/etc/grub.d/01_felhom_oneshot",
"/etc/grub.d/42_felhom_oneshot",
"/etc/systemd/system/felhom-agent.service",
"/etc/systemd/system/felhom-agent-rollback.service",
"/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf",