diff --git a/documentation/architecture/11-os-updates.md b/documentation/architecture/11-os-updates.md index 63f5cf70..99c0af79 100644 --- a/documentation/architecture/11-os-updates.md +++ b/documentation/architecture/11-os-updates.md @@ -469,7 +469,7 @@ Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-do --- -### 5.10 The Proxmox package lane — BUILT 2026-10-07, unreleased, not yet proven live (R-812 option A, `09` §3 decision 163) `[FACT]` +### 5.10 The Proxmox package lane — BUILT 2026-10-07 (agent v0.151.0, hub v0.142.0), proven on demo-felhom (R-812 option A, `09` §3 decision 163) `[FACT]` **What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve, libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on @@ -498,9 +498,81 @@ libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 when every ring-0 box ran the set in `DockerNightsEffective` (2) healthy night steps since it was first seen and none was unhealthy — „healthy" is the box's own verdict above (no memory-kill check; that is the Docker engine's, R-528). An approval nudges no box: ring 1 takes the set only by a signed `os_pve_step` (signed per box, as `os_docker_step`). -- **Not yet:** the live proof on demo-felhom (ring 0); the undo runbook (by hand: `apt-get install =` — +- **Proven live 2026-10-07** on demo-felhom (ring 0, a signed `os_pve_step`): 65 packages in 70 s, pve-manager + 9.2.2 → 9.2.21, healthy, every container kept its id, the hub logged the report (`audits/day-2026-10-07/B/RESULT.md`). +- **Not yet:** the undo runbook (by hand: `apt-get install =` — Proxmox keeps 30–66 old versions, C2). +### 5.11 The kernel lane — BUILT 2026-10-07 (agent v0.152.0, hub v0.143.0; R-836, `09` §3 decisions 164, 171, 172) `[FACT]` + +**The ruling (decision 172):** design option A (a one-shot flag on the ESP) with option C (lockup → panic) on the one-shot +entry. A box may restart at night for a kernel update; **the household is mailed the day before** with what to do if +the box is not back by morning; each kernel set is approved by the operator after ring 0 ran it; **a frozen new kernel +that needs a person's power cycle is accepted** for the first customers. Why A: measured in the spike on the Tester 1 +VM, demo-felhom and demo-hp (Secure Boot on) — the ESP flag and UEFI `BootNext` both boot a kernel once and fall back +after a panic; a hardware watchdog armed during the reboot never fired on any of the three +(`audits/kernel-spike-2026-10-07/DESIGN-kernel-lane.md`). A writes nothing to firmware. + +**The boot side** (config bundle, two GRUB generators): `/etc/grub.d/01_felhom_oneshot` reads `felhom_next` from a GRUB +env block on the ESP (`EFI/felhom/oneshot.env`), clears and saves it BEFORE the menu, and — only if it names an +installed kernel — boots that kernel's one-shot entry. `/etc/grub.d/42_felhom_oneshot` gives each installed kernel an +entry `felhom-oneshot-`: the normal entry plus `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10` +(option C; sorted after `10_linux`, so never entry 0 and never the default). A panic on the one-shot restarts the box in +10 s into the default — the old kernel. Without a vfat ESP both print nothing. + +**The default** is the kernel the box RUNS, pinned in `/etc/default/grub.d/zz-felhom-kernel-default.cfg` before the +install and proved from grub.cfg after it (an install would otherwise make the newest kernel the default — R-836's +defect). Only a healthy one-shot boot moves it. + +**The root wrapper** (`felhom-os-apply`, layer `kernel`, lane slow, an appliance, authority = a signed `os_kernel_step` +or the root-owned ring-0 mark): `apply` STAGES (installs the set — the series meta-package, at most one new kernel +image, the boot helper and firmware — pins the default, writes the flag; never reboots); `kernel-reboot` reboots a +staged step; `kernel-boot` says what became of it after a boot; `kernel-good` moves the default; `kernel-revert` +reboots ONCE into the old kernel; `kernel-cancel`; `kernel-status`. Refusals R20 (the box cannot do a one-shot: not +UEFI, no vfat ESP, a separate /boot, GRUB without fat/loadenv, the generators missing, a hand pin), R21 (the crash +guard tripped or saw an unclean boot within its window), R22 (the step's phase does not allow it; never two steps +within 20 h), R23 (not exactly one newer kernel, or not the one signed / told). State `/var/lib/felhom-kernel/state.json`. + +**The night slot** (agent): the kernel step ENDS the night leg — after the host step (and the Proxmox step when one ran) +ended healthy, after the whole-guest backup (the leg runs only after it), under the same heavy-op gate, trigger `night` +only, at most one per night. Ring 0 stages the pending kernel and reboots; ring 1 reboots only a kernel a signed +`os_kernel_step` staged by day (the job never reboots). The hub hears `staged` before the reboot. + +**"Boot good" and the self-revert** (agent, at every start): on the new kernel the agent judges the boot for **20 +minutes** — the host health rule (§8.2: the Proxmox daemons and the agent active, the customer guest running and its +own rule passing, the tunnel running) AND the box reached the hub (the `judging` report itself). Healthy → the new +kernel becomes the default (`applied`). Not healthy by the deadline → `health_failed`, then ONE self-revert into the old +kernel (`self_reverted` after it boots; a revert that comes back on the new kernel is `revert_failed` and never +retried). The wait, measured 2026-10-07 by a plain reboot (`audits/kernel-lane-2026-10-07/B/`): every container healthy +68 s after the reboot on demo-felhom (the hub reached at 63 s; Tailscale back at 53 s — the spike's > 6 min did not +recur) and 272 s on demo-hp (the hub at 189 s). A box that never comes back raises the hub's existing `host_stale` +(45 min, `alerting.stale_threshold`) and `host_down` (90 min) to the operator. + +**The crash guard** (§5.9): the step's reboot and the self-revert are orderly (the clean-stop marker), so a step adds at +most ONE unclean boot (a panic before userspace adds none: the planned reboot's marker is still there); with R21 the +step starts only with none in the window — the box cannot reach the 3rd unclean boot that leaves it off. Pinned by +`KernelStepCannotLeaveTheBoxOff`. + +**The household mail** (hub): a box is DUE when ring 0 has a pending kernel (its host report's `proxmox-kernel-X.Y` +upgrade) or a ring-1 box runs a kernel a signed job staged, and no step for that kernel has ended (`applied`, +`fell_back`, `health_failed`, `self_reverted`, `revert_failed`, or a refusal R20/R23/R3/R12 — the operator decides; never +retried by itself). Its household gets ONE mail (`mail.kernel.*`, its language, informal; to the registered address) +between 09:00 and 20:00 Budapest time, at most once per 20 h and three times per kernel. The box's `os_update.kernel` +block says `tonight` only within 24 h of a mail the mail service ACCEPTED — **no mail, no step**. Operator events +`os_kernel_step` and `os_kernel_notice`; the household's timeline gets the usual "security fixes installed" line after +`applied`. + +**Approval** (hub): "Approve kernel set" on the System page appears when every ring-0 box's newest ended kernel step is +`applied` for the same kernel, after a NIGHT stage. The approved set is the series meta-package and the signed image. An +approval nudges no box: ring 1 takes it only through a signed `os_kernel_step` (stage), then its own told night. + +**The System page** shows per box: the running kernel, the next boot, the default (amber while a one-shot flag names +another kernel), and the kernel step (the newest result, the kernel due, whether the household was told for tonight). + +**Not measured:** option C itself — the Proxmox kernel ships no lockup test module (`CONFIG_TEST_LOCKUP` is not set on +7.0.14-20), so a soft lockup turning into a panic is recorded as unmeasured (no tool was built). A true dead freeze +needs a person (accepted, decision 172). Evidence and proofs: `audits/kernel-lane-2026-10-07/`. + ## 6. Risks and edge cases | # | What can go wrong | What the design does | @@ -580,7 +652,8 @@ Each step returns to the operator for go or no-go. after enrolment (16:24/16:25 UTC: 49 guest + 106 host packages, 110 s + 44 s, healthy). An installer pass would cost ~45 s and run BEFORE any whole-guest backup exists (no undo) — not built. 6. **Slow lane: host kernel and Proxmox packages, with the reboot.** **Split 2026-10-07 (decisions 163–164):** the Proxmox - USERSPACE packages are §5.10 (BUILT, unreleased, no reboot); the kernel lane is R-836 (a spike with reboots first). + USERSPACE packages are §5.10 (BUILT, proven on demo-felhom); the kernel lane is §5.11 (**BUILT 2026-10-07**, agent + v0.152.0 + hub v0.143.0, decision 172; the spike with reboots came first, `audits/kernel-spike-2026-10-07/`). 7. **Later:** the Proxmox major upgrade (PVE 9 → 10), drilled on ring 0 first. --- diff --git a/documentation/audits/kernel-lane-2026-10-07/A/redproof.txt b/documentation/audits/kernel-lane-2026-10-07/A/redproof.txt new file mode 100644 index 00000000..d69d83b0 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/A/redproof.txt @@ -0,0 +1,18 @@ +# kernel lane red-proof 2026-10-07T13:08:35Z — each mutation must turn its test red (FAILED); the clean tree is green +clean: OK +no default pin before the install -> test_installing_a_kernel_does_not_change_the_grub_default: FAILED (failures=1) +no boot-setup check -> test_a_box_without_the_esp_flag_is_refused: FAILED (failures=1) +reboot without the flag check -> test_a_reboot_without_the_flag_is_refused: FAILED (failures=1) +no signed-set image check -> test_a_kernel_outside_the_approved_set_is_refused: FAILED (failures=1) (re-run after the test was sharpened: the first run stayed green because a second image was ALSO refused by the one-kernel rule) +no expect_kver check -> test_a_kernel_outside_the_approved_set_is_refused: FAILED (failures=1) +snippet sets the default before clearing the flag -> test_the_snippet_clears_the_flag_on_use: FAILED (errors=1) +no crash-guard check -> test_the_crash_guard_must_be_armed_and_quiet: FAILED (failures=1) +self-revert used twice -> test_one_self_revert_then_never_again: FAILED (failures=1) +guard trips one unclean boot early (LIMIT-2) -> KernelStepCannotLeaveTheBoxOff.test_planned_reboot_one_crash_one_self_revert: FAILED (failures=1) +go: kernelDue ignores Tonight -> TestKernel_NoMailNoStep: --- FAIL: TestKernel_NoMailNoStep (0.00s) FAIL +go: KernelVerdict ignores the hub -> TestKernelVerdict: --- FAIL: TestKernelVerdict (0.00s) FAIL +go: no self-revert call -> TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait: --- FAIL: TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait (0.00s) FAIL +hub: tonight without a mail -> TestKernel_DueButNotToldIsNotTonight: --- FAIL: TestKernel_DueButNotToldIsNotTonight (0.04s) +hub: a failed mail still recorded -> TestKernel_NoMailNoStep: --- FAIL: TestKernel_NoMailNoStep (0.04s) +hub: approval without a night stage -> TestKernel_ApproveNeedsEveryRing0BoxHealthyAfterANightStep: --- FAIL: TestKernel_ApproveNeedsEveryRing0BoxHealthyAfterANightStep (0.03s) +hub: kernel button without its readiness gate (.Fingerprint and Waiting dropped) -> TestSystemPage_KernelButtonOnlyWhenReady: --- FAIL: TestSystemPage_KernelButtonOnlyWhenReady (0.04s) diff --git a/documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/agent-journal.txt b/documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/agent-journal.txt new file mode 100644 index 00000000..cf856e6a --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/agent-journal.txt @@ -0,0 +1,36 @@ +2026-10-07T15:11:39+02:00 demo-felhom systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.785+02:00 level=INFO msg="felhom-agent daemon starting" version=0.151.0 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900 +2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.795+02:00 level=INFO msg="daemon: operator signers pinned" count=2 +2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="local-api leaf LOADED" fingerprint_sha256=c4b96bf0f97efcd462b11d33bd39a28133b4b3648535ee5b497c005eadf3e51e cert=/var/lib/felhom-agent/local-api.crt +2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="backup tier armed" target=felhom-backup cadence=24h0m0s keep_last=3 wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="lanresolver: enabled" host_ip=192.168.0.162 upstreams="[1.1.1.1 8.8.8.8]" interval_s=300 +2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="wgtunnel: enabled" interval_s=60 state_dir=/var/lib/felhom-agent +2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="poke: agent-plane sync listener enabled" port=51822 +2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="felhomsshd (OOB): enabled" interval_s=60 state_dir=/var/lib/felhom-agent +2026-10-07T15:11:39+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:39.799+02:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="capabilities self-check" ok=68 total=68 degraded=0 inactive=0 +2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.531+02:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s +2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.530+02:00 level=INFO msg="fast-tick armed: 30s out-of-band cadence while desired-state is unapplied" interval=30s +2026-10-07T15:11:40+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:40.534+02:00 level=INFO msg="poke: listening for hub sync-pokes (WG-confined, contentless)" addr=10.77.0.2:51822 +2026-10-07T15:11:43+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:43.004+02:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60 +2026-10-07T15:11:43+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:43.004+02:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s +2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.915+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests +2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3 +2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.933+02:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.934+02:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test" +2026-10-07T15:11:44+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:44.934+02:00 level=INFO msg="local-api server listening" addr=169.254.253.1:8443 +2026-10-07T15:11:45+02:00 demo-felhom felhom-priv-apply[1711]: felhom-priv-apply: SAME sshd-config /etc/felhom-sshd/sshd_config +2026-10-07T15:11:46+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:46.084+02:00 level=INFO msg="felhomsshd: config applied" port=8822 action=reload +2026-10-07T15:11:46+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:46.142+02:00 level=INFO msg="felhomsshd belt: set synced" set=ssh_port elements=8822 +2026-10-07T15:11:49+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:49.890+02:00 level=INFO msg="guestnet: guest network unhealthy but not acting" vmid=9201 reason="agent started less than 3m0s ago" detail="no IPv4 address on eth0" +2026-10-07T15:11:50+02:00 demo-felhom felhom-agent[1198]: 2026/10/07 15:11:50 http: TLS handshake error from 169.254.253.2:55264: EOF +2026-10-07T15:11:50+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:50.475+02:00 level=WARN msg="lanresolver: reconcile failed" vmid=9201 err="discover guest 9201 domain: pct exec cat controller.yaml: : exit status 137" +2026-10-07T15:11:51+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:51.814+02:00 level=INFO msg="desired: updated from hub" generation=18 guests=0 +2026-10-07T15:11:51+02:00 demo-felhom felhom-priv-apply[3439]: felhom-priv-apply: SAME sshd-key /etc/felhom-sshd/authorized_keys/felhom-op +2026-10-07T15:11:51+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:51.868+02:00 level=INFO msg="felhomsshd: operator authorized_keys updated" user=felhom-op present=true +2026-10-07T15:11:51+02:00 demo-felhom felhom-agent[1198]: time=2026-10-07T15:11:51.930+02:00 level=INFO msg="felhomsshd belt: set synced" set=operator_ips elements=10.77.0.250 diff --git a/documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/timing.log b/documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/timing.log new file mode 100644 index 00000000..3735e952 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/timing.log @@ -0,0 +1,11 @@ +old boot b7c1573e-e8d3-4921-9a98-4ec43dcbad70 +reboot sent 2026-10-07T13:10:48Z ++53s LAN ssh back ++53s guest running ++53s containers running=5/5 starting-or-unhealthy=4 ++53s tunnel container running ++53s tailnet ssh back ++68s agent reached the hub (2026-10-07T15:11:51+02:00) ++68s containers running=5/5 starting-or-unhealthy=0 ++68s ALL containers running and healthy +SUMMARY lan=53 hub=68 guest=53 apps=68 tunnel=53 tailnet=53 diff --git a/documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/units-journal.txt b/documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/units-journal.txt new file mode 100644 index 00000000..96c95e2b --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/B/demo-felhom/units-journal.txt @@ -0,0 +1,138 @@ +2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)... +2026-10-07T15:11:34+02:00 demo-felhom felhom-crash-guard[532]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10 +2026-10-07T15:11:34+02:00 demo-felhom felhom-crash-guard[653]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10 +2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Finished felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit). +2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Starting tailscaled.service - Tailscale node agent... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: TPM: successfully read all properties +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Program starting: v1.102.2-t6cac91817-g6ff0ddc72, Go 1.26.5: []string{"/usr/sbin/tailscaled", "--state=/var/lib/tailscale/tailscaled.state", "--socket=/run/tailscale/tailscaled.sock", "--port=41641"} +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: LogID: 94a2aaa0b2c8fac06a42677d5acc9c23a59b45535ba49413ff0e63882f48728e +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale" +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: [rc=unknown ret=direct] +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using "direct" mode +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using *dns.directManager +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: inotify: NewDirWatcher: context canceled +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: logtail: dial "log.tailscale.com:443" failed: dial tcp: lookup log.tailscale.com on 192.168.0.1:53: dial udp 192.168.0.1:53: connect: network is unreachable (in 2ms), trying bootstrap... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp6.tailscale.com", "68.183.90.120") for "log.tailscale.com" error: Get "https://derp6.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 68.183.90.120:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4e.tailscale.com", "2a03:b0c0:3:d0::29:9001") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4e.tailscale.com", "2a03:b0c0:3:d0::29:9001") for "log.tailscale.com" error: Get "https://derp4e.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2a03:b0c0:3:d0::29:9001]:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4c.tailscale.com", "134.122.77.138") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4c.tailscale.com", "134.122.77.138") for "log.tailscale.com" error: Get "https://derp4c.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 134.122.77.138:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp7.tailscale.com", "2401:c080:1000:467f:5400:2ff:feee:22aa") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp7.tailscale.com", "2401:c080:1000:467f:5400:2ff:feee:22aa") for "log.tailscale.com" error: Get "https://derp7.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2401:c080:1000:467f:5400:2ff:feee:22aa]:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4e.tailscale.com", "134.122.74.153") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4e.tailscale.com", "134.122.74.153") for "log.tailscale.com" error: Get "https://derp4e.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 134.122.74.153:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp6.tailscale.com", "2400:6180:100:d0::982:d001") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp6.tailscale.com", "2400:6180:100:d0::982:d001") for "log.tailscale.com" error: Get "https://derp6.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2400:6180:100:d0::982:d001]:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp7.tailscale.com", "167.179.89.145") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp7.tailscale.com", "167.179.89.145") for "log.tailscale.com" error: Get "https://derp7.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 167.179.89.145:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp12b.tailscale.com", "2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp12b.tailscale.com", "2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f") for "log.tailscale.com" error: Get "https://derp12b.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2001:19f0:5c01:48a:5400:3ff:fe8d:cb5f]:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp8c.tailscale.com", "206.189.16.32") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp8c.tailscale.com", "206.189.16.32") for "log.tailscale.com" error: Get "https://derp8c.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 206.189.16.32:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp4d.tailscale.com", "2a03:b0c0:3:d0::1501:b001") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp4d.tailscale.com", "2a03:b0c0:3:d0::1501:b001") for "log.tailscale.com" error: Get "https://derp4d.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2a03:b0c0:3:d0::1501:b001]:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp1e.tailscale.com", "64.225.56.166") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp1e.tailscale.com", "64.225.56.166") for "log.tailscale.com" error: Get "https://derp1e.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp 64.225.56.166:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: trying bootstrapDNS("derp12.tailscale.com", "2001:19f0:5c01:289:5400:3ff:fe8d:cb5e") for "log.tailscale.com" ... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: bootstrapDNS("derp12.tailscale.com", "2001:19f0:5c01:289:5400:3ff:fe8d:cb5e") for "log.tailscale.com" error: Get "https://derp12.tailscale.com/bootstrap-dns?q=log.tailscale.com": dial tcp [2001:19f0:5c01:289:5400:3ff:fe8d:cb5e]:443: connect: network is unreachable +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: logtail: upload: log upload of 365 bytes compressed failed: Post "https://log.tailscale.com/c/tailnode.log.tailscale.io/a8874038d193b4737b146408485f2db2abd40834fbfa94f59c39122bb7f51754": failed to resolve "log.tailscale.com": no DNS fallback candidates remain for "log.tailscale.com" +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: enumerating tailscale0 addresses for cleanup failed: failed to look up link "tailscale0": Link not found +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: wgengine.NewUserspaceEngine(tun "tailscale0") ... +2026-10-07T15:11:34+02:00 demo-felhom systemd[1]: Started tailscaled.service - Tailscale node agent. +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: [rc=unknown ret=direct] +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using "direct" mode +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: dns: using *dns.directManager +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: link state: interfaces.State{defaultRoute= ifs={} v4=false v6=false} +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp6) +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: magicsock: disco key = d:f75f0468399c429d +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: using firewall mode pref +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: magicsock: SetNetworkUp(false) +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Creating WireGuard device... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Bringing WireGuard device up... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: default choosing iptables +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Bringing router up... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: external route: up +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: netfilter running in iptables mode v6 = true, v6filter = true, v6nat = true +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp4) +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Clearing router settings... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Starting network monitor... +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: Engine created. +2026-10-07T15:11:34+02:00 demo-felhom tailscaled[711]: LinkChange: all links down; pausing: old: interfaces.State{defaultRoute= ifs={} v4=false v6=false} new: interfaces.State{defaultRoute= ifs={} v4=false v6=false} diff: numInterfaces: 3->4; numInterfaceIPs: 3->4; if tailscale0: added; ips tailscale0: added [fe80::fbbf:d4ae:bf83:b0ed/64] +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: pm: using backend prefs for "profile-10be": Prefs{ra=false dns=false want=true routes=[] statefulFiltering=false nf=on host="felhom-pve" update=on Persist{o=, n=[CE5tx] u="nagyfenyvesi.viktor@gmail.com" ak=-}} +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: envknob: PORT="41641" +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale" +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: linkChange: in state NoState; PAC or proxyConfig changed; updating routes +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: got LocalBackend in 35ms +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: Start +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: offline auto-update: starting update checks +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: ipnext: "conn25": skipping extension +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: ipnext: active extensions: captiveportal, acme, portlist, posture, clientupdate, relayserver, routecheck, serviceclientprefs, taildrop +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: load netmap from cache: netmap cache is not available +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: setPaused(true) +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: authRoutine: awaiting unpause +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: mapRoutine: awaiting unpause +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: Backend: logs: be:94a2aaa0b2c8fac06a42677d5acc9c23a59b45535ba49413ff0e63882f48728e fe: +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: updateRoutine: awaiting unpause +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: control: client.Login(0) +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: health(warnable=warming-up): error: Tailscale is starting. Please wait. +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=192.168.0.162/0, dst=192.168.0.162/32, gw=, outif=5, table=255 +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=192.168.0.162/0, dst=192.168.0.0/24, gw=, outif=5, table=254 +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=192.168.0.162/0, dst=192.168.0.255/32, gw=, outif=5, table=255 +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=, dst=, gw=192.168.0.1, outif=5, table=254 +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=10.77.0.2/0, dst=10.77.0.2/32, gw=, outif=7, table=255 +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=, dst=10.77.0.250/32, gw=, outif=7, table=254 +2026-10-07T15:11:35+02:00 demo-felhom tailscaled[711]: monitor: RTM_NEWROUTE: src=, dst=10.77.0.1/32, gw=, outif=7, table=254 +2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: control: setPaused(false) +2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: LinkChange: major, rebinding: old: interfaces.State{defaultRoute= ifs={} v4=false v6=false} new: interfaces.State{defaultRoute=vmbr0 ifs={vmbr0:[192.168.0.162/24 llu6] wg-felhom:[10.77.0.2/32]} v4=true v6=false} diff: HaveV4: false->true; DefaultRoute: ""->"vmbr0"; numInterfaces: 4->7; numInterfaceIPs: 4->7; if enp1s0 flags: broadcast|multicast->up|broadcast|multicast|running; if vmbr0: added; if vmbr9: added; if wg-felhom: added; ips vmbr0: added [192.168.0.162/24 fe80::6a1d:efff:fe5d:a664/64]; ips vmbr9: added [169.254.253.1/30 fe80::c0fc:cff:feca:d18c/64]; ips wg-felhom: added [10.77.0.2/32] rebind-reason=[default-if-changed,ips-changed,protocols-changed] +2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: magicsock: SetNetworkUp(true) +2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: control: doLogin(regen=false, hasUrl=false) +2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp6) +2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: Rebind; defIf="vmbr0", ips=[192.168.0.162/24 fe80::6a1d:efff:fe5d:a664/64] +2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: magicsock: 0 active derp conns +2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: router: portUpdate(port=41641, network=udp4) +2026-10-07T15:11:36+02:00 demo-felhom tailscaled[711]: monitor: gateway and self IP changed: gw=192.168.0.1 self=192.168.0.162 +2026-10-07T15:11:40+02:00 demo-felhom tailscaled[711]: health(warnable=warming-up): ok +2026-10-07T15:11:40+02:00 demo-felhom systemd[1]: Starting pve-guests.service - PVE guests... +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: control server key from https://controlplane.tailscale.com: ts2021=[fSeS+], legacy=[nlFWp] +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: RegisterReq: onode= node=[CE5tx] fup=false nks=false +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: RegisterReq: got response; nodeKeyExpired=false, machineAuthorized=true; authURL=false +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: netmap: got new dial plan from control +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: new contact: control-netmap usec=6317945 cached=false +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: active login: nagyfenyvesi.viktor@gmail.com +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: netmap: suggested exit node: no preferred DERP, try again later +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: offline auto-update: stopping update checks +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: Switching ipn state NoState -> Starting (WantRunning=true, nm=true) +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: SetPrivateKey called (init) +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: wgengine: Reconfig: configuring router +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: router: enabling connmark-based rp_filter workaround +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: wgengine: Reconfig: user dialer +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: tsdial: bart table size: 5 +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: wgengine: Reconfig: configuring DNS +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: dns: Set: {DefaultResolvers:[] Routes:{} SearchDomains:[] Hosts:0} +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: dns: Resolvercfg: {Routes:{} Hosts:0 LocalDomains:[]} +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: dns: OScfg: {} +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: peerapi: serving on http://100.70.170.35:36209 +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: peerapi: serving on http://[fd7a:115c:a1e0::5236:aa24]:61570 +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: portmapper: UPnP discovery response from 192.168.0.254, but gateway IP is 192.168.0.1 +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: portmapper: UPnP meta changed: [{Location:http://192.168.0.1:1900/rootDesc.xml Server:TP-Link/TP-LINK UPnP/1.1 MiniUPnPd/1.8 USN:uuid:e4c797e0-82ce-4f84-b56d-085d9d77c8cd::urn:schemas-upnp-org:device:InternetGatewayDevice:1} {Location:http://192.168.0.254:1900/igd.xml Server:ipos/7.0 UPnP/1.0 Archer_C5/2.0 USN:uuid:060b7353-fca6-4070-85f4-1fbfb9add62c::urn:schemas-upnp-org:device:InternetGatewayDevice:1}] +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: home DERP changing from derp-0 [0ms] to derp-4 [39ms] (forced=false) +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: home is now derp-4 (fra) +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: adding connection to derp-4 for home-keep-alive +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: 1 active derp conns: derp-4=cr0s,wr0s +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: derphttp.Client.Connect: connecting to derp-4 (fra) +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: updating netmap in disk cache +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: endpoints changed: 37.191.56.193:41641 (portmap), 10.77.0.2:41641 (local), 192.168.0.162:41641 (local) +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: Switching ipn state Starting -> Running (WantRunning=true, nm=true) +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: control: NetInfo: NetInfo{varies=false ipv6=false ipv6os=true udp=true icmpv4=false derp=#4 portmap=active-UM link="" firewallmode="ipt-default"} +2026-10-07T15:11:41+02:00 demo-felhom tailscaled[711]: magicsock: derp-4 connected; connGen=1 +2026-10-07T15:11:41+02:00 demo-felhom pve-guests[1375]: starting task UPID:demo-felhom:000005D5:00000504:6AC6450D:startall::root@pam: +2026-10-07T15:11:41+02:00 demo-felhom pvesh[1375]: Starting CT 9201 +2026-10-07T15:11:41+02:00 demo-felhom pve-guests[1493]: starting task UPID:demo-felhom:000005D6:00000505:6AC6450D:vzstart:9201:root@pam: +2026-10-07T15:11:41+02:00 demo-felhom pve-guests[1494]: starting CT 9201: UPID:demo-felhom:000005D6:00000505:6AC6450D:vzstart:9201:root@pam: +2026-10-07T15:11:45+02:00 demo-felhom tailscaled[711]: magicsock: new contact: peer=[G+z+N] usec=10136339 cached=false via=derp +2026-10-07T15:11:45+02:00 demo-felhom tailscaled[711]: magicsock: disco: node [G+z+N] d:ac1b0afc01e49a40 now using 192.168.0.180:35032 mtu=1360 tx=3f73e074118a +2026-10-07T15:11:45+02:00 demo-felhom tailscaled[711]: magicsock: new contact: peer=[G+z+N] usec=10137837 cached=false via=direct +2026-10-07T15:11:46+02:00 demo-felhom pve-guests[1375]: end task UPID:demo-felhom:000005D5:00000504:6AC6450D:startall::root@pam: OK +2026-10-07T15:11:46+02:00 demo-felhom systemd[1]: Finished pve-guests.service - PVE guests. +2026-10-07T15:11:51+02:00 demo-felhom tailscaled[711]: magicsock: disco: node [G+z+N] d:ac1b0afc01e49a40 now using 192.168.0.180:35032 mtu=1360 tx=2b416b182a12 diff --git a/documentation/audits/kernel-lane-2026-10-07/B/demo-hp/agent-journal.txt b/documentation/audits/kernel-lane-2026-10-07/B/demo-hp/agent-journal.txt new file mode 100644 index 00000000..846623df --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/B/demo-hp/agent-journal.txt @@ -0,0 +1,50 @@ +2026-10-07T15:16:05+02:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.853+02:00 level=INFO msg="felhom-agent daemon starting" version=0.151.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900 +2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.864+02:00 level=INFO msg="daemon: operator signers pinned" count=2 +2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="local-api leaf LOADED" fingerprint_sha256=34122ac2ace991685ad9adaaf882702581ae931837203dbeb25f0d9e493a837d cert=/var/lib/felhom-agent/local-api.crt +2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="backup tier armed" target=local cadence=24h0m0s keep_last=1 wait_timeout=30m0s prune_pbs_allowed=false primary=true +2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="backup tier armed" target=felhom-pbs cadence=168h0m0s keep_last=0 wait_timeout=12h0m0s prune_pbs_allowed=false primary=false +2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="lanresolver: enabled" host_ip=192.168.0.104 upstreams="[1.1.1.1 8.8.8.8]" interval_s=300 +2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="wgtunnel: enabled" interval_s=60 state_dir=/var/lib/felhom-agent +2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="poke: agent-plane sync listener enabled" port=51822 +2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.871+02:00 level=INFO msg="felhomsshd (OOB): enabled" interval_s=60 state_dir=/var/lib/felhom-agent +2026-10-07T15:16:05+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:05.872+02:00 level=INFO msg="pbsdr: bridge enabled (hub-driven; no-op until a pbs_dr descriptor arrives)" +2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.808+02:00 level=INFO msg="capabilities self-check" ok=68 total=68 degraded=0 inactive=0 +2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="backup: restore-test scheduler starting (per-archive due-check)" eval_interval=6h0m0s settle=24h0m0s +2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s +2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="poke: listening for hub sync-pokes (WG-confined, contentless)" addr=10.77.0.3:51822 +2026-10-07T15:16:06+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:06.809+02:00 level=INFO msg="fast-tick armed: 30s out-of-band cadence while desired-state is unapplied" interval=30s +2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/hdd_1 stable=/mnt/felhom-drives/hdd_1 prior_binds=0 +2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60 +2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.595+02:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s +2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.613+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.616+02:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3 +2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.616+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s slow_crashloop_max=5 slow_crashloop_window=24h0m0s guests_dir=/var/lib/felhom-agent/guests +2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.616+02:00 level=INFO msg="janitor: starting (restore-test scratch retry + stale-lock sweep)" interval=10m0s +2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.617+02:00 level=INFO msg="fstrim: weekly guest disk trim starting" schedule="weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test" +2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.617+02:00 level=INFO msg="local-api server listening" addr=169.254.253.1:8443 +2026-10-07T15:16:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:07.619+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +2026-10-07T15:16:07+02:00 demo-hp felhom-priv-apply[1599]: felhom-priv-apply: SAME sshd-config /etc/felhom-sshd/sshd_config +2026-10-07T15:16:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:08.339+02:00 level=INFO msg="felhomsshd: config applied" port=8822 action=reload +2026-10-07T15:16:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:08.395+02:00 level=INFO msg="felhomsshd belt: set synced" set=ssh_port elements=8822 +2026-10-07T15:16:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:08.930+02:00 level=INFO msg="desired: updated from hub" generation=9 guests=0 +2026-10-07T15:16:08+02:00 demo-hp felhom-priv-apply[1911]: felhom-priv-apply: SAME sshd-key /etc/felhom-sshd/authorized_keys/felhom-op +2026-10-07T15:16:09+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:09.008+02:00 level=INFO msg="felhomsshd: operator authorized_keys updated" user=felhom-op present=true +2026-10-07T15:16:09+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:09.083+02:00 level=INFO msg="felhomsshd belt: set synced" set=operator_ips elements=10.77.0.250 +2026-10-07T15:16:09+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:09.581+02:00 level=INFO msg="lanresolver: guest IP not yet leased — skipping (will retry)" vmid=9201 +2026-10-07T15:16:10+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:10.562+02:00 level=INFO msg="lanresolver: guest IP not yet leased — skipping (will retry)" vmid=9202 +2026-10-07T15:16:23+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:23.620+02:00 level=WARN msg="local-api: storage view unavailable for class hints" err="storage: NodeStorage: proxmox: GET /nodes/demo-hp/storage: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage\": context canceled" +2026-10-07T15:16:28+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:28.656+02:00 level=INFO msg="guest-attach: drive bound under shared parent (normalized to one bind, live)" vmid=9201 where=/mnt/hdd_1 stable=/mnt/felhom-drives/hdd_1 prior_binds=1 +2026-10-07T15:16:28+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:28.657+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +2026-10-07T15:16:37+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:37.744+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +2026-10-07T15:16:39+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:39.272+02:00 level=INFO msg="controller-supervisor: slow counter restored from disk" vmid=9201 restarts_24h=0 slow_crashloop_since=2026-09-17T09:22:29Z +2026-10-07T15:16:48+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:48.483+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +2026-10-07T15:16:55+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:16:55.099+02:00 level=INFO msg="pbs: verify cycle complete" datastore=felhom-offsite snapshots=2 +2026-10-07T15:17:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:07.712+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +2026-10-07T15:17:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:07.725+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +2026-10-07T15:17:07+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:07.737+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +2026-10-07T15:17:08+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:08.416+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +2026-10-07T15:17:28+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:28.462+02:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +2026-10-07T15:17:37+02:00 demo-hp felhom-agent[1306]: time=2026-10-07T15:17:37.717+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 diff --git a/documentation/audits/kernel-lane-2026-10-07/B/demo-hp/timing.log b/documentation/audits/kernel-lane-2026-10-07/B/demo-hp/timing.log new file mode 100644 index 00000000..ada5e412 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/B/demo-hp/timing.log @@ -0,0 +1,21 @@ +old boot 8ea2cdff-d166-4af6-9937-156f4ee659a5 +reboot sent 2026-10-07T13:13:06Z ++176s LAN ssh back ++176s containers running=0/0 starting-or-unhealthy=0 ++189s agent reached the hub (2026-10-07T15:16:08+02:00) ++189s guest running ++189s containers running=18/21 starting-or-unhealthy=17 ++189s tunnel container running ++204s containers running=20/21 starting-or-unhealthy=11 ++211s containers running=20/21 starting-or-unhealthy=7 ++217s containers running=21/21 starting-or-unhealthy=6 ++224s containers running=21/21 starting-or-unhealthy=6 ++231s containers running=21/21 starting-or-unhealthy=6 ++238s containers running=21/21 starting-or-unhealthy=6 ++245s containers running=21/21 starting-or-unhealthy=6 ++252s containers running=21/21 starting-or-unhealthy=6 ++258s containers running=21/21 starting-or-unhealthy=2 ++265s containers running=21/21 starting-or-unhealthy=1 ++272s containers running=21/21 starting-or-unhealthy=0 ++272s ALL containers running and healthy +SUMMARY lan=176 hub=189 guest=189 apps=272 tunnel=189 tailnet=n/a diff --git a/documentation/audits/kernel-lane-2026-10-07/B/demo-hp/units-journal.txt b/documentation/audits/kernel-lane-2026-10-07/B/demo-hp/units-journal.txt new file mode 100644 index 00000000..4f547f33 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/B/demo-hp/units-journal.txt @@ -0,0 +1,18 @@ +2026-10-07T15:16:00+02:00 demo-hp systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)... +2026-10-07T15:16:00+02:00 demo-hp felhom-crash-guard[628]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10 +2026-10-07T15:16:00+02:00 demo-hp felhom-crash-guard[773]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10 +2026-10-07T15:16:00+02:00 demo-hp systemd[1]: Finished felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit). +2026-10-07T15:16:06+02:00 demo-hp systemd[1]: Starting pve-guests.service - PVE guests... +2026-10-07T15:16:08+02:00 demo-hp pve-guests[1493]: starting task UPID:demo-hp:000006CD:000006D3:6AC64618:startall::root@pam: +2026-10-07T15:16:08+02:00 demo-hp pvesh[1493]: Starting VM 341 +2026-10-07T15:16:08+02:00 demo-hp pve-guests[1741]: starting task UPID:demo-hp:000006D4:000006D5:6AC64618:qmstart:341:root@pam: +2026-10-07T15:16:08+02:00 demo-hp pve-guests[1748]: start VM 341: UPID:demo-hp:000006D4:000006D5:6AC64618:qmstart:341:root@pam: +2026-10-07T15:16:09+02:00 demo-hp pve-guests[1748]: VM 341 started with PID 1891. +2026-10-07T15:16:11+02:00 demo-hp pvesh[1493]: Starting CT 9201 +2026-10-07T15:16:11+02:00 demo-hp pve-guests[1741]: starting task UPID:demo-hp:000007A3:00000804:6AC6461B:vzstart:9201:root@pam: +2026-10-07T15:16:11+02:00 demo-hp pve-guests[1955]: starting CT 9201: UPID:demo-hp:000007A3:00000804:6AC6461B:vzstart:9201:root@pam: +2026-10-07T15:16:11+02:00 demo-hp pvesh[1493]: Starting CT 9202 +2026-10-07T15:16:11+02:00 demo-hp pve-guests[1741]: starting task UPID:demo-hp:000007A5:00000808:6AC6461B:vzstart:9202:root@pam: +2026-10-07T15:16:11+02:00 demo-hp pve-guests[1957]: starting CT 9202: UPID:demo-hp:000007A5:00000808:6AC6461B:vzstart:9202:root@pam: +2026-10-07T15:16:16+02:00 demo-hp pve-guests[1493]: end task UPID:demo-hp:000006CD:000006D3:6AC64618:startall::root@pam: OK +2026-10-07T15:16:16+02:00 demo-hp systemd[1]: Finished pve-guests.service - PVE guests. diff --git a/documentation/audits/kernel-lane-2026-10-07/delivery/tester1.txt b/documentation/audits/kernel-lane-2026-10-07/delivery/tester1.txt new file mode 100644 index 00000000..88cda163 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/delivery/tester1.txt @@ -0,0 +1,4 @@ +== agent_update 0.152.0 → tester-1, 2026-10-07T13:21:33Z +signed: op=agent_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=31589d832a25a458ce3878c0fcbd3ae3 expires=2026-10-07T14:06:33Z +wrote envelope to /dev/null +uploaded signed op to the hub jobs queue diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 5451d236..0fb8cb81 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,30 @@ +## v0.143.0 — the kernel lane: the day-before household mail, the night instruction, the operator's kernel set (R-836; `09` §3 decision 172; `11` §5.11) (2026-10-07) + +**Operator action on deploy: none.** Built and deployed only in a session the operator attends (decision 162). Needs +agent v0.152.0 on a box for anything to happen there; an older agent reports no kernel lane and is never due. + +- **Due** (`osupdates/kernel.go` `KernelDue`): a ring-0 box with a pending `proxmox-kernel-X.Y` upgrade in its host + report, or a ring-1 box whose facts say a signed `os_kernel_step` STAGED a kernel; never a switched-off box, a box + whose wrapper names a setup problem, a box already running it, or a kernel whose step ended (`applied`, `fell_back`, + `health_failed`, `self_reverted`, `revert_failed`, a refusal R20/R23/R3/R12) — the operator decides; never retried. +- **The day-before mail** (`KernelNotify`, on the minute tick): ONE mail per due box between 09:00 and 20:00 + Budapest, never twice within 20 h, at most 3 per kernel (then one operator event); to the REGISTERED customer + address (the notification address when none), in the household's language, informal (`mail.kernel.*` in both + locales; `notify.SendKernelNotice`). Only a mail the mail service accepted is recorded (`os_kernel_notices`). +- **The instruction** (`os_update.kernel` {kver, tonight, notified_at}): `tonight` only within 24 h of a recorded + mail — **no mail, no step**. The box is bumped when the mail goes out. +- **Reports**: layer `kernel` is its own layer (an older hub filed unknown layers as the guest's). Outcomes staged, + judging, applied, fell_back, health_failed, self_reverted, revert_failed, refused, failed → operator events + `os_kernel_step` / `os_update_failed` (operator-only, registered in the same commit); `applied` also gives the + household its "security fixes installed" timeline line. +- **Approval**: "Approve kernel set" (`/os/approve-kernel`) appears when every ring-0 box's newest ended kernel step is + `applied` for the same kernel after a NIGHT stage; the set is the series meta-package and the signed image; it nudges + no box (ring 1: a signed `os_kernel_step` only). +- **System page**: two cells — "Kernel (default)" (amber while a one-shot flag names another kernel) and "Kernel step" + (the newest result, the kernel due, whether the household was told for tonight). +- Tests: `osupdates/kernel_test.go` (8), `web/system_test.go` (2), `notify/kernel_notice_test.go`; red-proofs + `documentation/audits/kernel-lane-2026-10-07/A/redproof.txt`. + ## v0.142.0 — the Proxmox package set; RESET purges the off-site folder; one line when old copies use another key; the two never-built DR fields retired (R-812 A, R-32, R-366, R-105; `09` §3 163, 167–169) (2026-10-07) **Operator action on deploy: none.** Built and deployed in the attended session (decision 162). diff --git a/hub/cmd/hub/main.go b/hub/cmd/hub/main.go index da43229e..64f07188 100644 --- a/hub/cmd/hub/main.go +++ b/hub/cmd/hub/main.go @@ -420,7 +420,9 @@ func main() { if _, err := dataStore.BumpHostDesired(hostID); err != nil { logger.Printf("[WARN] osupdates: bump desired for %s: %v", hostID, err) } - }} + }, + // R-836 (`09` §3 decision 172): the household's day-before kernel mail; no accepted mail → no kernel step. + KernelMail: dispatcher.SendKernelNotice} if v := os.Getenv("OS_APPROVE_AFTER"); v != "" { if d, derr := time.ParseDuration(v); derr == nil && d >= 0 { osSvc.ApproveAfter = d diff --git a/hub/internal/i18n/locales/en.json b/hub/internal/i18n/locales/en.json index 75cbee11..c3bc17e0 100644 --- a/hub/internal/i18n/locales/en.json +++ b/hub/internal/i18n/locales/en.json @@ -57,6 +57,8 @@ "mail.claim.claim.body": "Dear Customer,\n\nYour Felhom server is up. To use the dashboard for the first time, enter the\nsetup code below, then choose your own password:\n\nSetup code: %s\n\nThe code is valid for 72 hours and can be used once. You can reach the\ndashboard here:\n\n%s\n\nIf it did not reach you in time, the dashboard's \"Request a new code\" button\ngets you a fresh one — it always arrives at this e-mail address.\n\nBest regards,\nFelhom.eu", "mail.selfbind.subject": "[Felhom] Link your Felhom box to your account", "mail.selfbind.body": "Dear Customer,\n\nYour Felhom box is ready and waiting to be linked. You can link it to your\naccount yourself using the link below — no sign-in needed:\n\n%s\n\nAfter you open the link you will need to enter two things:\n\n 1. The pairing code, shown on the box's own screen (the monitor).\n 2. Your owner passphrase (the 5-word phrase) that you received from your\n Felhom operator — in person or by telephone, never by e-mail. It proves\n the account is yours.\n\nThe link is valid for 7 days. For safety it locks after 5 failed attempts —\nif that happens, contact support.\n\nIf you did not ask for this, ignore this e-mail.\n\nBest regards,\nFelhom.eu", + "mail.kernel.subject": "[Felhom] Your Felhom box restarts tonight", + "mail.kernel.body": "Hi,\n\nTonight your Felhom box restarts for a security update. Your apps will be away for a few\nminutes, then come back by themselves. You do not need to do anything.\n\nIf the box is not back by morning: unplug its power cable, wait 10 seconds, and plug it back\nin. The box then starts its earlier, proven system by itself.\n\nIf that does not help, reply to this e-mail and we will help.\n\nBest,\nFelhom.eu", "bind.title": "Felhom — Link your box", "bind.heading": "Link your Felhom box", "bind.lead": "Link the Felhom box you have just installed to your account. Enter the pairing code shown on the box's screen, and your owner passphrase.", diff --git a/hub/internal/i18n/locales/hu.json b/hub/internal/i18n/locales/hu.json index aa279f20..54521675 100644 --- a/hub/internal/i18n/locales/hu.json +++ b/hub/internal/i18n/locales/hu.json @@ -57,6 +57,8 @@ "mail.claim.claim.body": "Kedves Ügyfél!\n\nElindult a Felhom szervered. A vezérlőpult első használatához add meg az\nalábbi beállító kódot, majd válassz saját jelszót:\n\nBeállító kód: %s\n\nA kód 72 óráig érvényes, és egyszer használható fel. A vezérlőpultot itt éred el:\n\n%s\n\nHa nem kaptad volna meg időben, a vezérlőpult \"Új kód kérése\" gombjával\nkérhetsz frisset — az mindig erre az e-mail címre érkezik.\n\nÜdvözlettel,\nFelhom.eu", "mail.selfbind.subject": "[Felhom] Kösd össze a Felhom dobozodat", "mail.selfbind.body": "Kedves Ügyfél!\n\nElkészült a Felhom dobozod, és készen áll az összekötésre. Az alábbi hivatkozáson\ntudod te magad összekötni a fiókoddal — nincs szükség bejelentkezésre:\n\n%s\n\nA hivatkozás megnyitása után két adatot kell megadnod:\n\n 1. A párosító kódot, amely a doboz képernyőjén (a monitoron) látható.\n 2. A tulajdonosi jelmondatodat (az 5 szóból álló kifejezést), amelyet a\n Felhom üzemeltetőjétől kaptál — személyesen vagy telefonon, e-mailben\n soha. Ez igazolja, hogy a fiók a tiéd.\n\nA hivatkozás 7 napig érvényes. Biztonsági okból 5 sikertelen próbálkozás után\nzárolódik — ilyenkor vedd fel a kapcsolatot az ügyfélszolgálattal.\n\nHa nem te kérted ezt, hagyd figyelmen kívül ezt az e-mailt.\n\nÜdvözlettel,\nFelhom.eu", + "mail.kernel.subject": "[Felhom] Ma éjjel újraindul a Felhom dobozod", + "mail.kernel.body": "Szia!\n\nMa éjjel a Felhom dobozod egy biztonsági frissítés miatt újraindul. Ilyenkor az alkalmazásaid\nnéhány percig nem érhetők el, utána maguktól visszajönnek. Neked nem kell semmit tenned.\n\nHa reggelre a doboz mégsem érhető el: húzd ki a tápkábelét, várj 10 másodpercet, és dugd\nvissza. A doboz ilyenkor magától a korábbi, bevált rendszerrel indul el.\n\nHa ez sem segít, válaszolj erre a levélre, és segítünk.\n\nÜdv,\nFelhom.eu", "bind.title": "Felhom — Doboz összekötése", "bind.heading": "Felhom doboz összekötése", "bind.lead": "Kösd össze a most telepített Felhom dobozodat a fiókoddal. Add meg a doboz képernyőjén látható párosító kódot és a tulajdonosi jelmondatodat.", diff --git a/hub/internal/notify/dispatcher.go b/hub/internal/notify/dispatcher.go index f86ae258..5f94e864 100644 --- a/hub/internal/notify/dispatcher.go +++ b/hub/internal/notify/dispatcher.go @@ -733,6 +733,10 @@ var operatorOnlyEvents = map[string]bool{ // R-841 (hub v0.131.0): the tunnel alarm — a box fact the household can do nothing about from inside. "tunnel_down": true, "tunnel_recovered": true, + // R-836 (hub v0.143.0): the kernel lane's step results and the day-before notice's own record — operator facts. The + // household's word is the day-before mail itself (SendKernelNotice), never these events. + "os_kernel_step": true, + "os_kernel_notice": true, // `11` §8.3 (hub v0.131.0): the four OS-update alarms — fleet facts only the operator can act on. "os_update_stale": true, "os_reboot_needed": true, @@ -969,6 +973,39 @@ func (d *Dispatcher) SendSelfBindEmail(customerID, email, link string) error { return nil } +// SendKernelNotice mails the household that the box restarts tonight for a kernel update (R-836, `09` §3 decision 172). +// To the REGISTERED customer address (the notification address when none is registered) — a promise to the household, +// so it does not depend on which event types the household switched on. Returns the language used. An error means the +// mail service did not accept it, and the caller then runs NO kernel step (no mail, no step). +func (d *Dispatcher) SendKernelNotice(customerID, kver string) (string, error) { + if d.resendAPIKey == "" { + d.logger.Printf("[ERROR] kernel notice for %s NOT sent: no Resend API key configured", customerID) + return "", fmt.Errorf("notify: no resend api key") + } + to := "" + if cc, err := d.store.GetCustomerConfig(customerID); err == nil && cc != nil { + to = strings.TrimSpace(cc.Email) + } + if to == "" { + if p, err := d.store.GetNotificationPrefs(customerID); err == nil && p != nil { + to = strings.TrimSpace(p.Email) + } + } + if to == "" { + return "", fmt.Errorf("the household has no e-mail address") + } + lang := d.store.CustomerLanguage(customerID) + subject, body := FormatKernelNoticeEmail(lang) + if err := d.sendEmailFn(to, subject, body, nil); err != nil { + d.logger.Printf("[ERROR] kernel notice (%s) to customer %s failed: %v", kver, customerID, err) + d.store.LogNotification(customerID, "kernel_notice", "info", subject, "failed", err.Error(), "customer") + return lang, err + } + d.logger.Printf("[INFO] kernel notice (%s) emailed to the household of %s (lang=%s)", kver, customerID, lang) + d.store.LogNotification(customerID, "kernel_notice", "info", subject, "sent", "", "customer") + return lang, nil +} + // firstHourOfABox is how long after enrolment a whole-guest tier skip is provisioning, not a fault (R-723). const firstHourOfABox = time.Hour diff --git a/hub/internal/notify/kernel_notice_test.go b/hub/internal/notify/kernel_notice_test.go new file mode 100644 index 00000000..50cddca7 --- /dev/null +++ b/hub/internal/notify/kernel_notice_test.go @@ -0,0 +1,33 @@ +package notify + +import ( + "strings" + "testing" +) + +// R-836, `09` §3 decision 172: the household's day-before mail — both languages, informal, says the box restarts +// tonight and exactly what to do if it is not back by morning. ASCII fragments with a positive and a negative control +// (rule 8: an accented search can return nothing for text that is there). +func TestKernelNotice_SaysWhatToDo(t *testing.T) { + for lang, want := range map[string][]string{ + "hu": {"jjel", "jraindul", "tápkábel"[:2], "10 m", "dugd", "vissza", "korábbi"[:3]}, + "en": {"Tonight", "restarts", "unplug", "10 seconds", "plug it back", "earlier, proven"}, + } { + subj, body := FormatKernelNoticeEmail(lang) + if subj == "" || body == "" || strings.HasPrefix(subj, "mail.") || strings.HasPrefix(body, "mail.") { + t.Fatalf("%s: missing text: %q / %q", lang, subj, body) + } + for _, w := range want { + if !strings.Contains(body, w) { + t.Fatalf("%s: body lacks %q:\n%s", lang, w, body) + } + } + if strings.Contains(body, "7.0.") || strings.Contains(body, "%") { + t.Fatalf("%s: the household needs what to do, not a kernel version or a format verb:\n%s", lang, body) + } + } + // negative control: a key that does not exist comes back as its own name + if s, _ := FormatClaimEmail("en", "claim", "c", "d", "x"); strings.Contains(s, "unplug") { + t.Fatal("negative control matched") + } +} diff --git a/hub/internal/notify/templates.go b/hub/internal/notify/templates.go index 03c2fce0..d266e682 100644 --- a/hub/internal/notify/templates.go +++ b/hub/internal/notify/templates.go @@ -266,6 +266,15 @@ func FormatSelfBindEmail(lang, customerID, link string) (string, string) { return b.Msg(lang, "mail.selfbind.subject"), b.Msgf(lang, "mail.selfbind.body", link) } +// FormatKernelNoticeEmail is the household's day-before mail of the kernel lane (R-836, `09` §3 decision 172): the box +// restarts tonight for a security update; if it is not back by morning, unplug it, wait 10 seconds, plug it back in — +// it then starts the earlier, proven kernel by itself. Informal voice, both languages from the bundle. It names no +// kernel version: the household needs what to do, not which kernel. Pinned by TestKernelNotice_*. +func FormatKernelNoticeEmail(lang string) (string, string) { + b := i18n.Shared() + return b.Msg(lang, "mail.kernel.subject"), b.Msg(lang, "mail.kernel.body") +} + // ────────────────────────────────────────────────────────────────────── // R-182 — the backup run digest // ────────────────────────────────────────────────────────────────────── diff --git a/hub/internal/osupdates/kernel.go b/hub/internal/osupdates/kernel.go new file mode 100644 index 00000000..10d7e37b --- /dev/null +++ b/hub/internal/osupdates/kernel.go @@ -0,0 +1,424 @@ +package osupdates + +// The kernel lane, hub half (hub v0.143.0; R-836, `09` §3 decision 172, `11` §5.11). The agent stages a kernel, boots +// it ONCE through a flag on the ESP, and makes it the default only after a healthy boot (felhom-agent internal/osupdate +// kernel.go + the wrapper's layer "kernel"). The hub decides WHEN: +// +// - A box is DUE when ring 0 has a pending kernel (its host report's `proxmox-kernel-X.Y` upgrade), or a ring-1 box +// runs a kernel a signed os_kernel_step STAGED — and no step for that kernel has ended yet (an ended step is the +// operator's to judge; the hub never retries it by itself). +// - The household of a due box gets ONE mail the day before, in its language, between 09:00 and 20:00 Budapest +// time (KernelNotify): the box restarts tonight; if it is not back by morning, unplug it, wait 10 seconds, plug it +// back in. Only a mail the mail service ACCEPTED is recorded. +// - The box's os_update block carries `kernel: {kver, tonight}`; tonight is true only within 24 h of such a mail. No +// mail → no step: the agent's night leg refuses a kernel without `tonight`. +// - The operator approves a kernel set ("Approve kernel set") once every ring-0 box booted it healthily as the +// default after a night step; a ring-1 box takes it only through a signed os_kernel_step. +// +// Pinned by kernel_test.go. + +import ( + "encoding/json" + "fmt" + "regexp" + "sort" + "strings" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" + "gitea.dooplex.hu/admin/felhom-hub/internal/sysfacts" +) + +// LayerKernel is the host's kernel (agent v0.152.0). +const LayerKernel = "kernel" + +// Kernel-lane events — operator only (the household's one word is the day-before mail). +const ( + EventKernelStep = "os_kernel_step" // what became of a kernel step (staged, the default moved, fell back, …) + EventKernelNotice = "os_kernel_notice" // the household was told — or could not be told (then no step) +) + +// The day-before mail's window and limits (decided by CC — operator may reverse): mailed only in the household's +// daytime, valid for the night that follows, never twice within 20 h, at most 3 times for one kernel on one box. +const ( + KernelNoticeFromHour = 9 + KernelNoticeToHour = 20 + KernelNoticeValid = 24 * time.Hour + KernelNoticeGap = 20 * time.Hour + KernelNoticeMax = 3 +) + +// KernelBlock is a box's kernel instruction (felhom-agent hub.WireKernelStep — field-exact, cross-repo). +type KernelBlock struct { + Kver string `json:"kver"` + Tonight bool `json:"tonight"` + NotifiedAt string `json:"notified_at,omitempty"` +} + +var ( + kverRE = regexp.MustCompile(`^([0-9]+\.[0-9]+)\.[0-9]+-[0-9]+-pve$`) + kernelMetaRE = regexp.MustCompile(`^proxmox-kernel-[0-9]+\.[0-9]+$`) +) + +// kernelEnded are the outcomes after which a kernel is never stepped again by itself (the operator decides). +var kernelEnded = map[string]bool{"applied": true, "fell_back": true, "health_failed": true, "self_reverted": true, "revert_failed": true} + +// refusals that will not go away by waiting a night (the box cannot do a one-shot, the set is wrong, no authority, +// not an appliance) — such a refusal ends the step too. A transient one (a lock, the crash guard's window) is retried +// within KernelNoticeMax mails. +var kernelRefusedForGood = map[string]bool{"R20": true, "R23": true, "R3": true, "R12": true} + +func budapestLoc() *time.Location { + if l, err := time.LoadLocation("Europe/Budapest"); err == nil { + return l + } + return time.FixedZone("CET", 3600) +} + +// kernelOf is the kernel a kernel-layer report is about: the wrapper's view's "to", else the release id. +func kernelOf(r Report) string { + var v struct { + To string `json:"to"` + } + if len(r.Kernel) > 0 && json.Unmarshal(r.Kernel, &v) == nil && kverRE.MatchString(v.To) { + return v.To + } + if kverRE.MatchString(r.ReleaseID) { + return r.ReleaseID + } + return "" +} + +func refusedCode(raw json.RawMessage) string { + var c struct { + Code string `json:"code"` + } + _ = json.Unmarshal(raw, &c) + return c.Code +} + +// kernelLane is the box's newest kernel-lane facts (nil: an older agent, or no host report). +func (s *Service) kernelLane(hostID string) *sysfacts.KernelLane { + h, err := s.Store.GetHost(hostID) + if err != nil || h == nil { + return nil + } + rj, _ := s.Store.GetLatestHostReportJSON(h.CustomerID) + if rj == "" { + return nil + } + return sysfacts.Parse(rj).Host.KernelLane +} + +// KernelDue says which kernel the box is due to step to, or why none. +func (s *Service) KernelDue(hostID string) (kver, why string) { + st := s.Store.GetOSHostSettings(hostID) + if !st.Enabled { + return "", "OS updates are switched off" + } + rep, _ := s.Store.LatestOSReport(hostID, LayerHost) + if rep == nil { + return "", "no host step reported (the kernel lane is for appliances)" + } + lane := s.kernelLane(hostID) + if lane == nil { + return "", "the box reports no kernel lane (agent older than v0.152.0)" + } + if len(lane.SetupProblems) > 0 { + return "", "the box cannot do a one-shot boot: " + strings.Join(lane.SetupProblems, "; ") + } + if st.Ring == 1 { + if lane.Phase != "staged" || !kverRE.MatchString(lane.To) { + return "", "ring 1: no kernel staged by a signed os_kernel_step" + } + kver = lane.To + } else { + var hr Report + _ = json.Unmarshal([]byte(rep.ReportJSON), &hr) + for _, p := range hr.Pending { + if kernelMetaRE.MatchString(p.Name) && p.From != "" && kverRE.MatchString(p.To+"-pve") { + kver = p.To + "-pve" + } + } + if kver == "" && lane.Phase == "staged" && kverRE.MatchString(lane.To) { + kver = lane.To + } + if kver == "" { + return "", "no pending kernel" + } + } + if lane.Running == kver { + return "", "the box already runs " + kver + } + reps, _ := s.Store.OSReportsDesc(hostID, LayerKernel, 100) + for _, r := range reps { + var kr Report + if json.Unmarshal([]byte(r.ReportJSON), &kr) != nil || kernelOf(kr) != kver { + continue + } + if kernelEnded[r.Outcome] { + return "", fmt.Sprintf("the kernel step to %s ended %s at %s — the operator decides", kver, r.Outcome, + r.ReceivedAt.UTC().Format("2006-01-02 15:04")) + } + if r.Outcome == "refused" && kernelRefusedForGood[refusedCode(kr.Refused)] { + return "", fmt.Sprintf("the kernel step to %s was refused (%s) — the operator decides", kver, refusedCode(kr.Refused)) + } + } + return kver, "" +} + +// KernelBlockFor is the box's kernel instruction (nil: not due). +func (s *Service) KernelBlockFor(hostID string) *KernelBlock { + kver, _ := s.KernelDue(hostID) + if kver == "" { + return nil + } + b := &KernelBlock{Kver: kver} + ns, _ := s.Store.KernelNotices(hostID, kver) + if len(ns) > 0 { + last := ns[len(ns)-1].SentAt + if s.now().Sub(last) < KernelNoticeValid { + b.Tonight, b.NotifiedAt = true, last.UTC().Format(time.RFC3339) + } + } + return b +} + +// KernelNotify tells each due box's household, the day before (`09` §3 decision 172). Called every minute; it acts only +// between KernelNoticeFromHour and KernelNoticeToHour Budapest time. Returns the boxes it told. +func (s *Service) KernelNotify() []string { + now := s.now() + lt := now.In(budapestLoc()) + if lt.Hour() < KernelNoticeFromHour || lt.Hour() >= KernelNoticeToHour { + return nil + } + hosts, err := s.Store.ListHosts() + if err != nil { + return nil + } + var told []string + for _, h := range hosts { + kver, _ := s.KernelDue(h.HostID) + if kver == "" { + continue + } + ns, _ := s.Store.KernelNotices(h.HostID, kver) + if len(ns) > 0 && now.Sub(ns[len(ns)-1].SentAt) < KernelNoticeGap { + continue + } + if len(ns) >= KernelNoticeMax { + key := "kernel_notice_max:" + h.HostID + ":" + kver + if s.Store.OSAlarmRaised(key).IsZero() { + _ = s.Store.SetOSAlarmRaised(key, now) + s.event(h.CustomerID, EventKernelNotice, "warning", fmt.Sprintf("Kernel %s on %s: the household was told %d times "+ + "and no kernel step ran — no further mail; the operator decides (the System page shows why).", kver, h.HostID, len(ns)), + map[string]any{"host_id": h.HostID, "kver": kver, "notices": len(ns)}) + } + continue + } + if s.KernelMail == nil { + continue + } + lang, err := s.KernelMail(h.CustomerID, kver) + if err != nil { + key := "kernel_notice_failed:" + h.HostID + ":" + kver + ":" + lt.Format("2006-01-02") + if s.Store.OSAlarmRaised(key).IsZero() { + _ = s.Store.SetOSAlarmRaised(key, now) + s.event(h.CustomerID, EventKernelNotice, "warning", fmt.Sprintf("Kernel %s on %s: the household could NOT be told "+ + "(%v) — so no kernel step tonight (no mail, no step).", kver, h.HostID, err), + map[string]any{"host_id": h.HostID, "kver": kver, "error": err.Error()}) + } + continue + } + if err := s.Store.SaveKernelNotice(store.KernelNotice{HostID: h.HostID, Kver: kver, SentAt: now, Lang: lang}); err != nil { + s.logf("[ERROR] osupdates: kernel notice for %s sent but NOT recorded (%v) — no step tonight", h.HostID, err) + continue + } + told = append(told, h.HostID) + s.logf("[INFO] osupdates: kernel %s — the household of %s was told the box restarts tonight (lang=%s)", kver, h.HostID, lang) + s.event(h.CustomerID, EventKernelNotice, "info", fmt.Sprintf("Kernel %s on %s: the household was told the box "+ + "restarts tonight (mail %d of at most %d).", kver, h.HostID, len(ns)+1, KernelNoticeMax), + map[string]any{"host_id": h.HostID, "kver": kver, "lang": lang}) + if s.Bump != nil { + s.Bump(h.HostID) + } + } + sort.Strings(told) + return told +} + +// kernelIngest raises the kernel layer's operator events (called by Ingest for layer "kernel"). +func (s *Service) kernelIngest(hostID, customerID string, r Report, details map[string]any) { + var v struct { + From, To, Reason string + } + _ = json.Unmarshal(r.Kernel, &v) + to := kernelOf(r) + details["kver"] = to + switch r.Outcome { + case "staged": + s.event(customerID, EventKernelStep, "info", fmt.Sprintf("Kernel %s staged on %s (trigger %s): installed, never the "+ + "default; the box boots it ONCE at its night reboot.", to, hostID, r.Trigger), details) + case "applied": + s.event(customerID, EventKernelStep, "info", fmt.Sprintf("Kernel %s booted healthily on %s and is the default now "+ + "(was %s). %s", to, hostID, v.From, r.HealthReason), details) + s.event(customerID, EventApplied, "info", "System security fixes installed on the box's kernel (the box restarted at night).", details) + case "fell_back": + s.event(customerID, EventKernelStep, "error", fmt.Sprintf("Kernel %s did NOT come up on %s: the box came back on %s "+ + "by itself. %s is installed but never the default; the operator decides.", to, hostID, v.From, to), details) + case "health_failed": + s.event(customerID, EventKernelStep, "error", fmt.Sprintf("Kernel %s booted on %s but is NOT healthy (%s): the box "+ + "restarts ONCE into %s by itself.", to, hostID, r.HealthReason, v.From), details) + case "self_reverted": + s.event(customerID, EventKernelStep, "error", fmt.Sprintf("Kernel %s on %s: back on %s after the self-revert (%s). "+ + "The operator decides.", to, hostID, v.From, r.HealthReason), details) + case "revert_failed": + s.event(customerID, EventKernelStep, "critical", fmt.Sprintf("Kernel %s on %s: the self-revert did NOT bring back %s "+ + "(%s). No second revert — look at the box.", to, hostID, v.From, r.HealthReason), details) + case "refused", "failed": + s.event(customerID, EventFailed, "error", fmt.Sprintf("Kernel step on %s %s: %s", hostID, r.Outcome, + strings.TrimSpace(string(r.Refused)+" "+r.HealthReason)), details) + } +} + +// kernelSet is the approved set for a kernel version: the series meta-package and the signed image. +func kernelSet(kver string) []Package { + m := kverRE.FindStringSubmatch(kver) + if m == nil { + return nil + } + v := strings.TrimSuffix(kver, "-pve") + return []Package{{Name: "proxmox-kernel-" + m[1], Version: v, Origin: "Proxmox Debian Repository"}, + {Name: "proxmox-kernel-" + kver + "-signed", Version: v, Origin: "Proxmox Debian Repository"}} +} + +// KernelStatus is the kernel set ring 0 booted and whether the operator's button may approve it: every ring-0 box's +// newest ended kernel step is "applied" (a healthy one-shot boot made it the default) for the SAME kernel, after a +// night stage; none fell back or reverted. +func (s *Service) KernelStatus() (Status, error) { + st := Status{Layer: LayerKernel} + ring0, err := s.ring0Hosts() + if err != nil || len(ring0) == 0 { + st.Waiting = "no ring-0 box" + return st, err + } + kver := "" + for _, h := range ring0 { + reps, err := s.Store.OSReportsDesc(h, LayerKernel, 100) + if err != nil { + return st, err + } + var applied Report + var appliedAt time.Time + for _, r := range reps { + var kr Report + if json.Unmarshal([]byte(r.ReportJSON), &kr) != nil { + continue + } + if kernelEnded[r.Outcome] { + if r.Outcome != "applied" || !r.Healthy { + st.Waiting = fmt.Sprintf("%s: the kernel step to %s ended %s", h, kernelOf(kr), r.Outcome) + return st, nil + } + applied, appliedAt = kr, r.ReceivedAt + break + } + } + k := kernelOf(applied) + if k == "" { + st.Waiting = h + " has not booted a new kernel healthily yet" + return st, nil + } + night := false + for _, r := range reps { + var kr Report + if json.Unmarshal([]byte(r.ReportJSON), &kr) == nil && r.Outcome == "staged" && kr.Trigger == "night" && + kernelOf(kr) == k && !r.ReceivedAt.After(appliedAt) { + night = true + break + } + } + if !night { + st.Waiting = fmt.Sprintf("%s booted %s healthily, but not after a night step", h, k) + return st, nil + } + if kver != "" && k != kver { + st.Waiting = fmt.Sprintf("the ring-0 boxes booted different kernels (%s, %s)", kver, k) + return st, nil + } + kver = k + } + set := kernelSet(kver) + c := map[string]Package{} + for _, p := range set { + c[p.Name] = p + } + fp, list := fingerprint(LayerKernel, c) + pj, _ := json.Marshal(list) + first, err := s.Store.OSCandidateFirstSeen(fp, string(pj), s.now()) + if err != nil { + return st, err + } + st.Fingerprint, st.FirstSeen, st.Packages = fp, first, len(list) + if rel, _ := s.Store.LatestOSRelease(LayerKernel); rel != nil && rel.Fingerprint == fp { + st.Approved, st.Waiting = rel.ID, "already approved" + } + return st, nil +} + +// ApproveKernel is the operator's "Approve kernel set" (`09` §3 decision 172). A ring-1 box takes it only through a +// signed os_kernel_step — approval alone installs nothing and restarts nothing. +func (s *Service) ApproveKernel() (string, error) { + st, err := s.KernelStatus() + if err != nil { + return "", err + } + if st.Waiting != "" || st.Fingerprint == "" { + return "", fmt.Errorf("osupdates: the kernel set cannot be approved yet: %s", st.Waiting) + } + var list []Package + _ = json.Unmarshal([]byte(s.candidatePackages(st.Fingerprint)), &list) + if err := s.approve(LayerKernel, st.Fingerprint, list, "operator"); err != nil { + return "", err + } + rel, _ := s.Store.LatestOSRelease(LayerKernel) + return rel.ID, nil +} + +func (s *Service) candidatePackages(fp string) string { + pj, _ := s.Store.OSCandidatePackages(fp) + return pj +} + +// KernelLine is one box's kernel view for the System page. +type KernelLine struct { + Due, Why string // the kernel it is due to step to, or why none + LastOutcome string + LastKernel string + LastAt time.Time + LastReason string + NoticeAt time.Time // the newest day-before mail (any kernel) + NoticeKver string + Tonight bool +} + +// KernelLineFor reads one box's kernel line. +func (s *Service) KernelLineFor(hostID string) KernelLine { + var l KernelLine + l.Due, l.Why = s.KernelDue(hostID) + if b := s.KernelBlockFor(hostID); b != nil { + l.Tonight = b.Tonight + } + if rep, _ := s.Store.LatestOSReport(hostID, LayerKernel); rep != nil { + var kr Report + _ = json.Unmarshal([]byte(rep.ReportJSON), &kr) + l.LastOutcome, l.LastKernel, l.LastAt, l.LastReason = rep.Outcome, kernelOf(kr), rep.ReceivedAt, kr.HealthReason + if l.LastReason == "" && len(kr.Refused) > 0 { + l.LastReason = string(kr.Refused) + } + } + if n, _ := s.Store.LatestKernelNotice(hostID); n != nil { + l.NoticeAt, l.NoticeKver = n.SentAt, n.Kver + } + return l +} diff --git a/hub/internal/osupdates/kernel_test.go b/hub/internal/osupdates/kernel_test.go new file mode 100644 index 00000000..f58849eb --- /dev/null +++ b/hub/internal/osupdates/kernel_test.go @@ -0,0 +1,284 @@ +package osupdates + +import ( + "encoding/json" + "errors" + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// R-836, `09` §3 decision 172: the kernel lane's hub half — due boxes, the day-before mail (no mail, no step), the +// block, the events, the operator's approval. + +const kOld, kNew = "7.0.2-6-pve", "7.0.14-22-pve" + +// laneFacts stores a host report whose system facts carry the kernel lane. +func (f *fix) laneFacts(t *testing.T, host, cust, running, phase, to string, problems ...string) { + t.Helper() + lane := map[string]any{"running": running, "default": running, "phase": phase, "to": to, "from": kOld} + if len(problems) > 0 { + lane["setup_problems"] = problems + } + body, _ := json.Marshal(map[string]any{"system": map[string]any{"facts": map[string]any{"host": map[string]any{ + "kernel_running": running, "kernel_lane": lane}}}}) + if err := f.s.Store.SaveHostReport(host, cust, body, store.HostReportDenorm{AgentVersion: "0.152.0"}); err != nil { + t.Fatal(err) + } +} + +// hostWithPendingKernel reports a host step whose pending list carries the kernel meta-package upgrade. +func (f *fix) hostWithPendingKernel(t *testing.T, host string) { + f.ingest(t, host, Report{Layer: LayerHost, Trigger: "night", Mode: "apply", Outcome: "nothing", Healthy: true, + Pending: []PendingPkg{{Name: "proxmox-kernel-7.0", From: "7.0.2-6", To: "7.0.14-22", Origin: []string{"Proxmox Debian Repository"}}, + {Name: "pve-manager", From: "9.2.2", To: "9.2.21", Origin: []string{"Proxmox Debian Repository"}}}}) +} + +func kreport(outcome, trigger string, healthy bool) Report { + return Report{Layer: LayerKernel, Trigger: trigger, Mode: "apply", Outcome: outcome, Healthy: healthy, ReleaseID: kNew, + Kernel: json.RawMessage(`{"from":"` + kOld + `","to":"` + kNew + `"}`)} +} + +func (f *fix) mail(sent *[]string, fail error) { + f.s.KernelMail = func(cust, kver string) (string, error) { + if fail != nil { + return "", fail + } + *sent = append(*sent, cust+":"+kver) + return "hu", nil + } +} + +func budapest(t *testing.T, h int) time.Time { + loc, err := time.LoadLocation("Europe/Budapest") + if err != nil { + t.Skip("no tzdata") + } + return time.Date(2026, 10, 8, h, 30, 0, 0, loc) +} + +// A ring-0 box with a pending kernel is due; its block names the kernel, NOT tonight, until the household was mailed. +// COMPANION RED-PROOF (observed): set Tonight without a notice in KernelBlockFor → "tonight before any mail". +func TestKernel_DueButNotToldIsNotTonight(t *testing.T) { + f := newFix(t) + f.hostWithPendingKernel(t, "hp") + f.laneFacts(t, "hp", "c-hp", kOld, "none", "") + k, why := f.s.KernelDue("hp") + if k != kNew { + t.Fatalf("due = %q (%s)", k, why) + } + b := f.s.DesiredBlock("hp").Kernel + if b == nil || b.Kver != kNew || b.Tonight { + t.Fatalf("tonight before any mail: %+v", b) + } +} + +// The mail goes out only 09:00–20:00 Budapest; then the block says tonight (for 24 h); never two within 20 h. +func TestKernel_DayBeforeMailMakesTonight(t *testing.T) { + f := newFix(t) + f.hostWithPendingKernel(t, "hp") + f.laneFacts(t, "hp", "c-hp", kOld, "none", "") + var sent []string + f.mail(&sent, nil) + f.now = budapest(t, 3) // night: no mail + if told := f.s.KernelNotify(); len(told) != 0 || len(sent) != 0 { + t.Fatalf("mailed at 03:30: %v", sent) + } + f.now = budapest(t, 21) + if f.s.KernelNotify(); len(sent) != 0 { + t.Fatalf("mailed at 21:30: %v", sent) + } + f.now = budapest(t, 10) + if told := f.s.KernelNotify(); len(told) != 1 || len(sent) != 1 || sent[0] != "c-hp:"+kNew { + t.Fatalf("told=%v sent=%v", told, sent) + } + if b := f.s.DesiredBlock("hp").Kernel; b == nil || !b.Tonight || b.NotifiedAt == "" { + t.Fatalf("after the mail the block must say tonight: %+v", b) + } + if !contains(f.bumps, "hp") || !contains(f.events, EventKernelNotice) { + t.Fatalf("the box must be bumped and the operator told: bumps=%v events=%v", f.bumps, f.events) + } + f.now = f.now.Add(5 * time.Hour) + if f.s.KernelNotify(); len(sent) != 1 { + t.Fatalf("a second mail within 20 h: %v", sent) + } + f.now = budapest(t, 10).Add(25 * time.Hour) + if b := f.s.DesiredBlock("hp").Kernel; b == nil || b.Tonight { + t.Fatalf("a mail older than 24 h no longer covers tonight: %+v", b) + } +} + +// No mail → no step: a mail the service did not accept is not recorded; the block stays not-tonight; the operator hears. +// COMPANION RED-PROOF (observed): record the notice before checking the send error → "a failed mail made tonight". +func TestKernel_NoMailNoStep(t *testing.T) { + f := newFix(t) + f.hostWithPendingKernel(t, "hp") + f.laneFacts(t, "hp", "c-hp", kOld, "none", "") + var sent []string + f.mail(&sent, errors.New("the household has no e-mail address")) + f.now = budapest(t, 11) + f.s.KernelNotify() + if b := f.s.DesiredBlock("hp").Kernel; b == nil || b.Tonight { + t.Fatalf("a failed mail made tonight: %+v", b) + } + if !contains(f.events, EventKernelNotice) { + t.Fatalf("the operator must hear the household could not be told: %v", f.events) + } + f.s.KernelMail = nil + f.s.KernelNotify() + if b := f.s.DesiredBlock("hp").Kernel; b == nil || b.Tonight { + t.Fatalf("no mailer → never tonight: %+v", b) + } +} + +// An ended step is never retried by itself; the operator decides. +func TestKernel_EndedStepIsNeverRetried(t *testing.T) { + for _, out := range []string{"applied", "fell_back", "health_failed", "self_reverted", "revert_failed"} { + f := newFix(t) + f.hostWithPendingKernel(t, "hp") + f.laneFacts(t, "hp", "c-hp", kOld, "fell_back", kNew) + f.ingest(t, "hp", kreport(out, "boot", out == "applied")) + if k, why := f.s.KernelDue("hp"); k != "" || !strings.Contains(why, "operator decides") { + t.Fatalf("%s: still due (%q %q)", out, k, why) + } + if f.s.DesiredBlock("hp").Kernel != nil { + t.Fatalf("%s: block still names a kernel", out) + } + } + // a refusal for good ends it; a passing one (a lock) does not + f := newFix(t) + f.hostWithPendingKernel(t, "hp") + f.laneFacts(t, "hp", "c-hp", kOld, "none", "") + r := kreport("refused", "night", false) + r.Refused = json.RawMessage(`{"code":"R9","reason":"another apt holds the lock"}`) + f.ingest(t, "hp", r) + if k, _ := f.s.KernelDue("hp"); k != kNew { + t.Fatal("a transient refusal (R9) must not end the step") + } + r.Refused = json.RawMessage(`{"code":"R20","reason":"/boot/efi is not vfat"}`) + f.ingest(t, "hp", r) + if k, _ := f.s.KernelDue("hp"); k != "" { + t.Fatal("R20 must end the step") + } +} + +// Ring 1 is due ONLY for a kernel a signed job staged; a box with a setup problem or switched off never is. +func TestKernel_Ring1OnlyWhenStagedAndSetupMustBeFine(t *testing.T) { + f := newFix(t) + f.hostWithPendingKernel(t, "cust1") + f.laneFacts(t, "cust1", "c-1", kOld, "none", "") + if k, _ := f.s.KernelDue("cust1"); k != "" { + t.Fatal("ring 1 due without a staged kernel") + } + f.laneFacts(t, "cust1", "c-1", kOld, "staged", kNew) + if k, why := f.s.KernelDue("cust1"); k != kNew { + t.Fatalf("ring 1 with a staged kernel not due: %s", why) + } + f.laneFacts(t, "cust1", "c-1", kOld, "staged", kNew, "/boot/efi is not a mounted vfat ESP") + if k, _ := f.s.KernelDue("cust1"); k != "" { + t.Fatal("a box that cannot do a one-shot is never due") + } + _ = f.s.Store.SetOSEnabled("hp", false) + f.hostWithPendingKernel(t, "hp") + f.laneFacts(t, "hp", "c-hp", kOld, "none", "") + if k, _ := f.s.KernelDue("hp"); k != "" { + t.Fatal("switched off but due") + } +} + +// The household is told at most KernelNoticeMax times for one kernel; then the operator hears once. +func TestKernel_AtMostThreeMails(t *testing.T) { + f := newFix(t) + f.hostWithPendingKernel(t, "hp") + f.laneFacts(t, "hp", "c-hp", kOld, "none", "") + var sent []string + f.mail(&sent, nil) + day := budapest(t, 10) + for i := 0; i < 5; i++ { + f.now = day.Add(time.Duration(i) * 24 * time.Hour) + f.s.KernelNotify() + } + if len(sent) != KernelNoticeMax { + t.Fatalf("mails = %d, want %d", len(sent), KernelNoticeMax) + } + n := 0 + for _, e := range f.events { + if e == EventKernelNotice { + n++ + } + } + if n != KernelNoticeMax+1 { + t.Fatalf("notice events = %d (3 told + 1 'no further mail')", n) + } +} + +// Each outcome is an operator event; "applied" also gives the household its timeline line; ingest never treats a +// kernel report as a guest report. +func TestKernel_IngestEvents(t *testing.T) { + f := newFix(t) + f.ingest(t, "hp", kreport("staged", "night", true)) + f.ingest(t, "hp", kreport("applied", "boot", true)) + f.ingest(t, "hp", kreport("fell_back", "boot", false)) + if c := countOf(f.events, EventKernelStep); c != 3 { + t.Fatalf("kernel events = %d: %v", c, f.events) + } + if !contains(f.events, EventApplied) { + t.Fatal("an applied kernel gives the household its line") + } + if rep, _ := f.s.Store.LatestOSReport("hp", LayerGuest); rep != nil { + t.Fatalf("a kernel report was stored as a guest report: %+v", rep) + } + if rep, _ := f.s.Store.LatestOSReport("hp", LayerKernel); rep == nil || rep.Outcome != "fell_back" { + t.Fatalf("kernel report not stored on its layer: %+v", rep) + } +} + +// The operator's approval: every ring-0 box booted the same kernel healthily after a night stage — never before. +// COMPANION RED-PROOF (observed): drop the `night` check in KernelStatus → "approved without a night stage". +func TestKernel_ApproveNeedsEveryRing0BoxHealthyAfterANightStep(t *testing.T) { + f := newFix(t) + if _, err := f.s.ApproveKernel(); err == nil { + t.Fatal("approved with nothing booted") + } + f.ingest(t, "hp", kreport("staged", "night", true)) + f.ingest(t, "hp", kreport("applied", "boot", true)) + f.ingest(t, "n100", kreport("staged", "signed", true)) // a by-day stage is not a night step + f.ingest(t, "n100", kreport("applied", "boot", true)) + if _, err := f.s.ApproveKernel(); err == nil || !strings.Contains(err.Error(), "night") { + t.Fatalf("approved without a night stage: %v", err) + } + f.ingest(t, "n100", kreport("staged", "night", true)) + f.ingest(t, "n100", kreport("applied", "boot", true)) + id, err := f.s.ApproveKernel() + if err != nil || !strings.HasPrefix(id, "os-kernel-") { + t.Fatalf("%q %v", id, err) + } + rel, _ := f.s.Store.LatestOSRelease(LayerKernel) + if !strings.Contains(rel.PackagesJSON, `"proxmox-kernel-7.0"`) || !strings.Contains(rel.PackagesJSON, "proxmox-kernel-7.0.14-22-pve-signed") { + t.Fatalf("release = %s", rel.PackagesJSON) + } + if contains(f.bumps, "cust1") { + t.Fatal("an approved kernel set must nudge no ring-1 box (a signed job only)") + } + // a fell-back ring-0 box blocks the button + g := newFix(t) + g.ingest(t, "hp", kreport("staged", "night", true)) + g.ingest(t, "hp", kreport("fell_back", "boot", false)) + if st, _ := g.s.KernelStatus(); !strings.Contains(st.Waiting, "fell_back") { + t.Fatalf("waiting = %q", st.Waiting) + } +} + +func contains(xs []string, x string) bool { return countOf(xs, x) > 0 } + +func countOf(xs []string, x string) int { + n := 0 + for _, v := range xs { + if v == x { + n++ + } + } + return n +} diff --git a/hub/internal/osupdates/service.go b/hub/internal/osupdates/service.go index 6eb4ca4c..c8f447cc 100644 --- a/hub/internal/osupdates/service.go +++ b/hub/internal/osupdates/service.go @@ -47,7 +47,7 @@ const ( var Layers = []string{LayerGuest, LayerHost} // AllLayers adds the Docker engine set (approved only by the operator's button, ApproveDocker). -var AllLayers = []string{LayerGuest, LayerHost, LayerDocker, LayerPVE} +var AllLayers = []string{LayerGuest, LayerHost, LayerDocker, LayerPVE, LayerKernel} // dockerNames are the six packages of the Docker engine set (the agent wrapper's DOCKER_NAMES). var dockerNames = map[string]bool{"containerd.io": true, "docker-buildx-plugin": true, "docker-ce": true, @@ -117,6 +117,8 @@ type Report struct { // OOMCheck (agent v0.150.0, decision 157): the Docker step's memory-kill check, kept verbatim — oomCheckWaiting // reads it from the stored report. OOMCheck json.RawMessage `json:"oom_check,omitempty"` + // Kernel (agent v0.152.0, R-836): the kernel layer's view {running, default, flag, phase, from, to, …}, kept verbatim. + Kernel json.RawMessage `json:"kernel,omitempty"` } // PendingPkg is one update the sources offer. @@ -156,6 +158,7 @@ type Block struct { Enabled bool `json:"enabled"` Release *ReleaseBlock `json:"release,omitempty"` HostRelease *ReleaseBlock `json:"host_release,omitempty"` + Kernel *KernelBlock `json:"kernel,omitempty"` // R-836 (hub v0.143.0) } // ReleaseBlock is a layer's newest approved release, for ring 1. @@ -205,6 +208,9 @@ type Service struct { // TestOverride names the TEST overrides active at start ("" = none, the ruled waits). Every approval made while it // is set carries the `test` mark; CancelTestReleases cancels them at a start without it (`11` §5.3.1). TestOverride string + // KernelMail sends a household the day-before kernel mail (R-836, `09` §3 decision 172) and returns the language + // it used; an error means the mail service did not accept it — then there is no step. nil = no mail, no step. + KernelMail func(customerID, kver string) (lang string, err error) } func (s *Service) now() time.Time { @@ -244,7 +250,7 @@ func (s *Service) event(customerID, typ, sev, msg string, details any) { func layerOf(r Report) string { switch r.Layer { - case LayerHost, LayerDocker, LayerPVE: + case LayerHost, LayerDocker, LayerPVE, LayerKernel: return r.Layer } return LayerGuest @@ -267,6 +273,10 @@ func (s *Service) Ingest(hostID string, r Report) error { hostID, layer, r.RunID, r.Ring, r.Mode, r.Outcome, r.Healthy, len(r.Upgraded), len(r.Pending), len(r.NotCovered), len(r.RestartNeeded), r.RebootNeeded, r.PassSeconds) details := map[string]any{"host_id": hostID, "layer": layer, "run_id": r.RunID, "ring": r.Ring, "outcome": r.Outcome, "upgraded": len(r.Upgraded), "release_id": r.ReleaseID, "health_reason": r.HealthReason} + if layer == LayerKernel { + s.kernelIngest(hostID, h.CustomerID, r, details) // R-836: its own outcomes and events (kernel.go) + return nil + } where := "the box" switch layer { case LayerHost: @@ -420,6 +430,10 @@ func (s *Service) Evaluate() ([]Status, error) { if _, err := s.PVEStatus(); err != nil { // R-812 option A: stamped on the tick too, like the Docker set return out, err } + if _, err := s.KernelStatus(); err != nil { // R-836: stamped on the tick too + return out, err + } + s.KernelNotify() // R-836: the day-before mails (acts only 09:00–20:00 Budapest time) return out, nil } @@ -527,7 +541,8 @@ func (s *Service) Candidates() []Status { } d, _ := s.DockerStatus() p, _ := s.PVEStatus() - return append(out, d, p) + k, _ := s.KernelStatus() + return append(out, d, p, k) } // BundleThreshold is the config-bundle alarm's wait (the System page turns the cell red at it). @@ -776,7 +791,7 @@ func (s *Service) approve(layer, fp string, list []Package, by string) error { s.logf("[WARN] osupdates: OS release %s (%s) APPROVED by %s (%d packages, fingerprint %s)%s", id, layer, by, len(list), fp, mark) s.event("", EventReleaseApprove, "info", fmt.Sprintf("OS release %s (%s) approved (%s, %d packages)%s.", id, layer, by, len(list), mark), map[string]any{"release_id": id, "layer": layer, "approved_by": by, "packages": len(list), "fingerprint": fp, "test": test}) - if s.Bump != nil && layer != LayerDocker && layer != LayerPVE { // the slow-lane sets reach ring 1 only by a signed job + if s.Bump != nil && layer != LayerDocker && layer != LayerPVE && layer != LayerKernel { // the slow-lane sets reach ring 1 only by a signed job hosts, _ := s.Store.ListHosts() for _, h := range hosts { if st := s.Store.GetOSHostSettings(h.HostID); st.Ring == 1 && st.Enabled { @@ -845,6 +860,7 @@ func (s *Service) DesiredBlock(hostID string) Block { if st.Ring == 1 { b.Release, b.HostRelease = s.releaseBlock(LayerGuest), s.releaseBlock(LayerHost) } + b.Kernel = s.KernelBlockFor(hostID) // R-836: nil unless a kernel is due; tonight only after the day-before mail return b } diff --git a/hub/internal/store/os_updates.go b/hub/internal/store/os_updates.go index e76779e2..0c735f72 100644 --- a/hub/internal/store/os_updates.go +++ b/hub/internal/store/os_updates.go @@ -70,9 +70,70 @@ func (s *Store) migrateOSUpdates() error { } s.db.Exec(`ALTER TABLE os_releases ADD COLUMN cancelled_at TEXT NOT NULL DEFAULT ''`) s.db.Exec(`ALTER TABLE os_releases ADD COLUMN cancel_reason TEXT NOT NULL DEFAULT ''`) + // The kernel lane (hub v0.143.0, R-836, `09` §3 decision 172): every mail that told a household "the box restarts + // tonight". A kernel step runs only on a night such a mail announced — no mail, no step. + if _, err := s.db.Exec(`CREATE TABLE IF NOT EXISTS os_kernel_notices ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + host_id TEXT NOT NULL, + kver TEXT NOT NULL, + sent_at DATETIME NOT NULL, + lang TEXT NOT NULL DEFAULT '' + ); + CREATE INDEX IF NOT EXISTS idx_os_kernel_notices ON os_kernel_notices(host_id, kver, id);`); err != nil { + return err + } return nil } +// KernelNotice is one "the box restarts tonight" mail that reached the mail service. +type KernelNotice struct { + HostID, Kver, Lang string + SentAt time.Time +} + +// SaveKernelNotice records a sent notice (only after the mail service accepted it). +func (s *Store) SaveKernelNotice(n KernelNotice) error { + _, err := s.db.Exec(`INSERT INTO os_kernel_notices (host_id, kver, sent_at, lang) VALUES (?, ?, ?, ?)`, + n.HostID, n.Kver, n.SentAt.UTC().Format("2006-01-02 15:04:05"), n.Lang) + return err +} + +// KernelNotices lists the notices for one box and kernel, oldest first. +func (s *Store) KernelNotices(hostID, kver string) ([]KernelNotice, error) { + rows, err := s.db.Query(`SELECT host_id, kver, sent_at, lang FROM os_kernel_notices WHERE host_id = ? AND kver = ? ORDER BY id`, hostID, kver) + if err != nil { + return nil, err + } + defer rows.Close() + var out []KernelNotice + for rows.Next() { + var n KernelNotice + var at string + if err := rows.Scan(&n.HostID, &n.Kver, &at, &n.Lang); err != nil { + return nil, err + } + n.SentAt = parseSQLiteTime(at) + out = append(out, n) + } + return out, rows.Err() +} + +// LatestKernelNotice is the newest notice for a box (any kernel); nil when there was none. +func (s *Store) LatestKernelNotice(hostID string) (*KernelNotice, error) { + var n KernelNotice + var at string + err := s.db.QueryRow(`SELECT host_id, kver, sent_at, lang FROM os_kernel_notices WHERE host_id = ? ORDER BY id DESC LIMIT 1`, hostID). + Scan(&n.HostID, &n.Kver, &at, &n.Lang) + if err == sql.ErrNoRows { + return nil, nil + } + if err != nil { + return nil, err + } + n.SentAt = parseSQLiteTime(at) + return &n, nil +} + // hasColumn reports whether table has the column (PRAGMA table_info). func (s *Store) hasColumn(table, col string) bool { rows, err := s.db.Query(`SELECT name FROM pragma_table_info(?)`, table) @@ -216,6 +277,16 @@ func (s *Store) OSCandidateFirstSeen(fingerprint, packagesJSON string, now time. return parseSQLiteTime(at), nil } +// OSCandidatePackages is the package list a candidate fingerprint was first seen with ("" when unknown). +func (s *Store) OSCandidatePackages(fingerprint string) (string, error) { + var pj string + err := s.db.QueryRow(`SELECT packages_json FROM os_candidates WHERE fingerprint = ?`, fingerprint).Scan(&pj) + if err == sql.ErrNoRows { + return "", nil + } + return pj, err +} + // OSRelease is one approved version set. type OSRelease struct { ID string diff --git a/hub/internal/sysfacts/sysfacts.go b/hub/internal/sysfacts/sysfacts.go index 00e2b0a6..625a9819 100644 --- a/hub/internal/sysfacts/sysfacts.go +++ b/hub/internal/sysfacts/sysfacts.go @@ -39,6 +39,23 @@ type Host struct { OopsThisBoot *bool `json:"oops_this_boot"` KernelPanic *int `json:"kernel_panic"` CrashGuard *CrashGuard `json:"crash_guard"` + KernelLane *KernelLane `json:"kernel_lane"` // agent ≥ v0.152.0 (R-836); nil = an older agent +} + +// KernelLane is the box's kernel-lane state (agent v0.152.0, R-836, `11` §5.11), read by the wrapper from grub.cfg, the +// ESP flag and its own step record. Default is the kernel GRUB boots normally; Flag (non-empty) is a kernel that boots +// ONCE at the next boot; Phase is the step's: none | staged | oneshot | judging | good | fell_back | reverting | +// self_reverted | revert_failed | cancelled. +type KernelLane struct { + Running string `json:"running"` + Default string `json:"default"` + Flag *string `json:"flag"` + Phase string `json:"phase"` + From string `json:"from"` + To string `json:"to"` + Reason string `json:"reason"` + SetupProblems []string `json:"setup_problems"` + Error string `json:"error"` } // Guest is the customer guest's half. diff --git a/hub/internal/web/os_updates.go b/hub/internal/web/os_updates.go index f107249a..f14da839 100644 --- a/hub/internal/web/os_updates.go +++ b/hub/internal/web/os_updates.go @@ -79,6 +79,15 @@ func (s *Server) handleOSAdmin(w http.ResponseWriter, r *http.Request, path stri } id, err := view.ApprovePVE() reply(map[string]string{"release_id": id}, err) + case r.Method == http.MethodPost && path == "/os/approve-kernel": + // R-836 (`09` §3 decision 172): the operator approves the kernel set every ring-0 box booted healthily. + view, ok := s.osUpdates.(OSSystemView) + if !ok { + reply(nil, fmt.Errorf("kernel approval not available")) + return + } + id, err := view.ApproveKernel() + reply(map[string]string{"release_id": id}, err) default: http.Error(w, "not found", http.StatusNotFound) } diff --git a/hub/internal/web/r135_csrf_test.go b/hub/internal/web/r135_csrf_test.go index 743c3fc6..a2fdb306 100644 --- a/hub/internal/web/r135_csrf_test.go +++ b/hub/internal/web/r135_csrf_test.go @@ -60,6 +60,7 @@ var r135PostRoutes = []string{ "/os/approve-now", "/os/approve-docker", "/os/approve-pve", + "/os/approve-kernel", // Not a route: the gate must refuse BEFORE routing, so even an unknown path is 403, never 404. "/no-such-route", } diff --git a/hub/internal/web/system.go b/hub/internal/web/system.go index cea759aa..27840b93 100644 --- a/hub/internal/web/system.go +++ b/hub/internal/web/system.go @@ -36,6 +36,7 @@ type systemRow struct { HostRelease, HostPending, HostNotCovered cell Held, RebootSince, KernelPanic, Oops cell CrashRestarts24h, Guard cell + KernelDefault, KernelStep cell // R-836: the GRUB default kernel; the kernel lane's step Bundle cell // R-840: the root-owned config bundle Agent cell // R-530: the box's agent against the vouched one // guest @@ -58,6 +59,54 @@ type OSSystemView interface { AgentThreshold() time.Duration ApproveDocker() (string, error) ApprovePVE() (string, error) // R-812 option A: the Proxmox package set + ApproveKernel() (string, error) // R-836: the kernel set + KernelLineFor(hostID string) osupdates.KernelLine // R-836: one box's kernel step and day-before mail +} + +// kernelCells are the System page's two kernel-lane cells (R-836, `11` §5.11): the GRUB default (amber while a one-shot +// flag names another kernel for the next boot), and the step — the newest result, the kernel the box is due, whether +// the household was told for tonight. "unknown" when the box reports no kernel lane (an older agent). +func kernelCells(f sysfacts.System, kl osupdates.KernelLine, now time.Time) (dflt, step cell) { + lane := f.Host.KernelLane + if lane == nil { + if kl.LastOutcome == "" { + return unknownCell(""), cell{Text: "—", Title: "the box reports no kernel lane (agent older than v0.152.0)"} + } + lane = &sysfacts.KernelLane{} // a step result without facts (an older facts read) still shows + } + dflt = unknownCell(lane.Default) + if lane.Flag != nil && *lane.Flag != "" { + dflt.Text += " (once: " + *lane.Flag + ")" + dflt.Class, dflt.Title = "warn", "the next boot runs "+*lane.Flag+" ONCE; the boot after it the default again" + } + var parts []string + if kl.LastOutcome != "" { + parts = append(parts, fmt.Sprintf("%s %s %s", kl.LastKernel, kl.LastOutcome, ago(kl.LastAt, now))) + } + switch { + case kl.Due != "" && kl.Tonight: + parts = append(parts, "due "+kl.Due+" — household told "+ago(kl.NoticeAt, now)+": TONIGHT") + case kl.Due != "": + parts = append(parts, "due "+kl.Due+" — not told yet (mails 09–20 h)") + } + if lane.Phase != "" && lane.Phase != "none" { + parts = append(parts, "phase "+lane.Phase) + } + if len(parts) == 0 { + parts = append(parts, "—") + } + step = cell{Text: strings.Join(parts, " · "), Title: strings.TrimSpace(kl.Why + " " + kl.LastReason)} + switch kl.LastOutcome { + case "fell_back", "health_failed", "self_reverted", "refused", "failed": + step.Class = "warn" + case "revert_failed": + step.Class = "bad" + } + if len(lane.SetupProblems) > 0 { + step.Class = "warn" + step.Title = "cannot do a one-shot boot: " + strings.Join(lane.SetupProblems, "; ") + } + return dflt, step } // agentCell is the "Agent" cell (R-530, hub v0.135.0): the box's agent against the vouched one, how far behind and @@ -375,6 +424,7 @@ func (s *Server) handleSystem(w http.ResponseWriter, r *http.Request) { agents[h.HostID] = h.AgentVersion } for i := range rows { + rows[i].KernelDefault, rows[i].KernelStep = kernelCells(facts[rows[i].HostID], view.KernelLineFor(rows[i].HostID), time.Now()) rows[i].Bundle = bundleCell(facts[rows[i].HostID], man.AgentVersion, man.BundleSHA256, s.store.BundleBehindSince(rows[i].HostID), view.BundleThreshold(), time.Now()) rows[i].Agent = agentCell(agents[rows[i].HostID], man.AgentVersion, diff --git a/hub/internal/web/system_test.go b/hub/internal/web/system_test.go index 0a2627c5..d5c152eb 100644 --- a/hub/internal/web/system_test.go +++ b/hub/internal/web/system_test.go @@ -12,6 +12,7 @@ import ( "gitea.dooplex.hu/admin/felhom-hub/internal/osupdates" "gitea.dooplex.hu/admin/felhom-hub/internal/store" + "gitea.dooplex.hu/admin/felhom-hub/internal/sysfacts" "golang.org/x/crypto/bcrypt" ) @@ -213,3 +214,61 @@ func TestSystemPage_PVEButtonOnlyWhenReady(t *testing.T) { t.Fatal("button missing after two healthy nights") } } + +// R-836: the "Approve kernel set" button appears ONLY when every ring-0 box booted the kernel healthily after a night +// step (one render test per branch of the gate), and the two kernel cells render the lane — "—" for an older agent. +// +// COMPANION RED-PROOF (observed): drop the readiness gate (`.Fingerprint` and `(eq .Waiting "")` — KernelStatus sets the +// fingerprint only when ready, so each alone suffices) → "button shown with no kernel booted". +func TestSystemPage_KernelButtonOnlyWhenReady(t *testing.T) { + s, st, svc := systemServer(t) + if strings.Contains(getSystem(t, s), `action="/os/approve-kernel"`) { + t.Fatal("button shown with no kernel booted") + } + _ = st.SetOSRing("full-1", 0) + k := func(outcome, trigger string) { + r := osupdates.Report{RunID: time.Now().String() + outcome, Layer: "kernel", Trigger: trigger, Mode: "apply", + Outcome: outcome, Healthy: outcome != "fell_back", ReleaseID: "7.0.14-22-pve", + Kernel: []byte(`{"from":"7.0.2-6-pve","to":"7.0.14-22-pve"}`)} + if err := svc.Ingest("full-1", r); err != nil { + t.Fatal(err) + } + } + k("staged", "night") + if strings.Contains(getSystem(t, s), `action="/os/approve-kernel"`) { + t.Fatal("button shown before a healthy night boot") + } + k("applied", "boot") + b := getSystem(t, s) + if !strings.Contains(b, `action="/os/approve-kernel"`) { + t.Fatal("button missing after a healthy boot following a night step") + } + if !strings.Contains(b, "7.0.14-22-pve applied") { + t.Fatalf("the kernel step cell does not show the newest result:\n%s", b[strings.Index(b, ""):min(len(b), strings.Index(b, "")+3000)]) + } +} + +func TestSystemPage_KernelCells(t *testing.T) { + flag := "7.0.14-22-pve" + f := sysfacts.System{Present: true, Host: sysfacts.Host{KernelLane: &sysfacts.KernelLane{Running: "7.0.2-6-pve", + Default: "7.0.2-6-pve", Flag: &flag, Phase: "staged", To: "7.0.14-22-pve"}}} + now := time.Date(2026, 10, 8, 3, 0, 0, 0, time.UTC) + d, step := kernelCells(f, osupdates.KernelLine{Due: "7.0.14-22-pve", Tonight: true, NoticeAt: now.Add(-14 * time.Hour)}, now) + if d.Class != "warn" || !strings.Contains(d.Text, "once: 7.0.14-22-pve") { + t.Fatalf("default cell = %+v", d) + } + if !strings.Contains(step.Text, "TONIGHT") || !strings.Contains(step.Text, "phase staged") { + t.Fatalf("step cell = %+v", step) + } + _, step = kernelCells(f, osupdates.KernelLine{Due: "7.0.14-22-pve"}, now) + if !strings.Contains(step.Text, "not told yet") { + t.Fatalf("not-told step cell = %+v", step) + } + _, step = kernelCells(f, osupdates.KernelLine{LastOutcome: "revert_failed", LastKernel: "7.0.14-22-pve", LastAt: now}, now) + if step.Class != "bad" { + t.Fatalf("a failed revert must be red: %+v", step) + } + if d, step := kernelCells(sysfacts.System{Present: true}, osupdates.KernelLine{}, now); d.Text != "unknown" || step.Text != "—" { + t.Fatalf("an older agent: %+v %+v", d, step) + } +} diff --git a/hub/internal/web/templates/system.html b/hub/internal/web/templates/system.html index b12d8159..74cec460 100644 --- a/hub/internal/web/templates/system.html +++ b/hub/internal/web/templates/system.html @@ -72,6 +72,11 @@ {{end}} + {{if and (eq .Layer "kernel") .Fingerprint (not .Approved) (eq .Waiting "")}} +
+ + +
{{end}} {{end}} @@ -108,12 +113,12 @@ BoxRing / updatesTunnel - ProxmoxKernel (running)Kernel (next boot)DebianFelhom releasePendingNot coveredHeldReboot neededkernel.panicOopsCrash restarts 24 hCrash guardRoot filesAgent + ProxmoxKernel (running)Kernel (next boot)Kernel (default)Kernel stepDebianFelhom releasePendingNot coveredHeldReboot neededkernel.panicOopsCrash restarts 24 hCrash guardRoot filesAgent Guest DebianFelhom releasePendingRestart neededLast disk trim Dockercontainerdlive-restoreDocker release Last OS leg - hostguestDocker engine + hostguestDocker engine {{range .Rows}} @@ -136,7 +141,7 @@ {{template "sys_cell" .Tunnel}} {{.PVE.Text}} - {{template "sys_cell" .KernelRunning}}{{template "sys_cell" .KernelNextBoot}}{{template "sys_cell" .HostDebian}} + {{template "sys_cell" .KernelRunning}}{{template "sys_cell" .KernelNextBoot}}{{template "sys_cell" .KernelDefault}}{{template "sys_cell" .KernelStep}}{{template "sys_cell" .HostDebian}} {{template "sys_cell" .HostRelease}}{{template "sys_cell" .HostPending}}{{template "sys_cell" .HostNotCovered}} {{template "sys_cell" .Held}}{{template "sys_cell" .RebootSince}}{{template "sys_cell" .KernelPanic}}{{template "sys_cell" .Oops}} {{template "sys_cell" .CrashRestarts24h}}{{template "sys_cell" .Guard}}{{template "sys_cell" .Bundle}}{{template "sys_cell" .Agent}} diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index b9e90ffe..9a78555d 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,3 +1,12 @@ +## installer — the uninstall knows the kernel lane's files (R-836; unreleased, lands with the next installer tag) (2026-10-07) + +- `felhom-host-install.sh` uninstall: removes the two GRUB generators agent v0.152.0's bundle installs + (`/etc/grub.d/01_felhom_oneshot`, `/etc/grub.d/42_felhom_oneshot`), the one-shot env block's directory on the ESP + (`/boot/efi/EFI/felhom`) and `/var/lib/felhom-kernel`, then `update-grub`. It KEEPS + `/etc/default/grub.d/zz-felhom-kernel-default.cfg` and says so: it names the kernel the box booted healthily, and + without it GRUB would boot the newest installed kernel, which may be one that fell back. Found by + `test_bundle_list_is_current` (the frozen list now names both). SCRIPT_VERSION unchanged (no tag cut this session). + ## gates — the ISO first-boot test is a gate, full runs only (R-502, `09` §3 decision 147) (2026-10-06) - scripts: R-502 — new gate `iso-bootstrap` (scripts/iso_bootstrap_gate.py) runs the ISO first-boot harness in felhom-iso-assistant:trixie on FULL runs only (fast=False: never the pre-push hook, never CI); no docker/no image/docker error = exit 2 'not checked', never a pass; every green run is followed by a built-in decoy (a pairing banner that never paints) the harness must fail. Docker-free decoys in scripts/test_iso_bootstrap_gate.py, covered in test_gate_decoys.py (decision 147). diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh index 9b2f784c..71a7a4d8 100644 --- a/scripts/felhom-host-install.sh +++ b/scripts/felhom-host-install.sh @@ -1367,6 +1367,20 @@ run_uninstall() { if [[ -e "$cgf" ]]; then run rm -f "$cgf"; fi done if [[ -d /var/lib/felhom-crash-guard ]]; then run rm -rf /var/lib/felhom-crash-guard; fi + # Agent v0.152.0 (R-836, the kernel lane): the two GRUB generators the bundle installs, the one-shot flag's env + # block on the ESP and the step record — then grub.cfg is regenerated without them. The GRUB default pin + # (/etc/default/grub.d/zz-felhom-kernel-default.cfg) is KEPT on purpose: it names the kernel this box booted + # healthily, and without it GRUB would boot the newest installed kernel, which may be one that fell back. + local kgf _grub_touched=false + for kgf in /etc/grub.d/01_felhom_oneshot /etc/grub.d/42_felhom_oneshot; do + if [[ -e "$kgf" ]]; then run rm -f "$kgf"; _grub_touched=true; fi + done + if [[ -d /boot/efi/EFI/felhom ]]; then run rm -rf /boot/efi/EFI/felhom; fi + if [[ -d /var/lib/felhom-kernel ]]; then run rm -rf /var/lib/felhom-kernel; fi + if $_grub_touched && command -v update-grub >/dev/null 2>&1; then run update-grub; fi + if [[ -f /etc/default/grub.d/zz-felhom-kernel-default.cfg ]]; then + log_info " kept /etc/default/grub.d/zz-felhom-kernel-default.cfg (the kernel this box booted healthily stays the default)" + fi # 1.31.0 (R-840): the bundle's previous copies and the wrapper's nonce record. if [[ -d /var/lib/felhom-os-apply ]]; then run rm -rf /var/lib/felhom-os-apply; fi if [[ -f /var/lib/vz/snippets/felhom-guest-hook.sh ]]; then run rm -f /var/lib/vz/snippets/felhom-guest-hook.sh; fi diff --git a/scripts/test_hostinstall.py b/scripts/test_hostinstall.py index 91d33fcd..729b867f 100644 --- a/scripts/test_hostinstall.py +++ b/scripts/test_hostinstall.py @@ -44,6 +44,8 @@ BUNDLE_DESTS = [ "/etc/systemd/system/felhom-crash-guard-check.service", "/etc/systemd/system/felhom-crash-guard-check.timer", "/etc/felhom/crash-guard.conf", + "/etc/grub.d/01_felhom_oneshot", + "/etc/grub.d/42_felhom_oneshot", "/etc/systemd/system/felhom-agent.service", "/etc/systemd/system/felhom-agent-rollback.service", "/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf",